Fail to Reproduce WebArena Results with GenericAgent-GPT-4o

[bpucla]

Dear AgentLab Authors,

Thank you for the great work! I'm trying to reproduce the WebArena Results with GenericAgent-GPT-4o. In particular, I used the following code. Everything should just follow AgentLab's default. However the number I got is 25 which is significantly lower than 31.4 as shown on the BrowserGym Leaderboard. Do you have any suggestions for the reproduction? Any code available to reproduce the performance ~31?

Thanks again for you great contribution to the community!

from agentlab.agents.generic_agent import AGENT_4o 

from agentlab.experiments.study import make_study
from agentlab.experiments.study import Study

study = make_study(
    benchmark="webarena", 
    agent_args=[AGENT_4o],
    comment="repo 4o agent",
)



study.run(n_jobs=5)

[recursix]

I'm sorry for not getting back to you sooner, and thank you for reporting this.

Could you give us some more details about your experiments? I.e., which os, which version of packages? Info should be in the experiment directory.

We recently realized that when running on a laptop, e.g., MacBook, a retina display is activated, doubling the DPI of the playwright and causing some actions to fail, e.g., click. This can cause a significant drop in performance.

In a later version, we may run everything in a container.