OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tianbao Xie1, Danyang Zhang1, Jixuan Chen1, Xiaochuan Li1,
Siheng Zhao1, Ruisheng Cao1, Toh Jing Hua1, Zhoujun Cheng1, Dongchan Shin1, Fangyu Lei1, Yitao Liu1, Yiheng Xu1, Shuyan Zhou3, Silvio Savarese2, Caiming Xiong2, Victor Zhong4, Tao Yu1
1The University of Hong Kong, 2Salesforce Research, 3Carnegie Mellon University, 4University of Waterloo

2025-07-28: Major Upgrade! OSWorld has been enhanced and is now OSWorld-Verified with comprehensive improvements: fixed community-reported examples, AWS support reducing evaluation time to within 1 hour, and updated benchmark results. See the verified benchmark results in the Benchmark section below. Please compare your OSWorld results with the new benchmark results when running the latest version.

Paper Code Doc Data Data Viewer Slides Twitter Discord

osworld task_demonstration **OSWorld** is a first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems. It can serve as a unified environment for evaluating open-ended computer tasks that involve arbitrary apps (e.g., task examples in the above Fig). We also create a benchmark of 369 real-world computer tasks in **OSWorld** with reliable, reproducible setup and evaluation scripts.

Abstract

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce **OSWorld**, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. **OSWorld** can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon **OSWorld**, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on **OSWorld** reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using **OSWorld** provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks.

OSWorld Environment Infrastructure

environment infrastructure The **OSWorld** environment uses a configuration file for initializing tasks *(highlighted in red)*, agent interaction, post-processing upon agent completion *(highlighted in orange)*, retrieving files and information *(highlighted in yellow)*, and executing the evaluation function *(highlighted in green)*. The corresponding configuration items are highlighted in colors that match their respective components within the environment. Environments can run in parallel on a single host machine for learning or evaluation purposes. Headless operation is supported.

Data Statistics and Comparison

Below we present an overview of the main statistics of **OSWorld**, showcasing the outline and a broad spectrum of tasks. **OSWorld** contains a total of 369 tasks (and an additional 43 tasks on Windows for analysis).



Key statistics of OSWorld.
The "Supp. tasks" refers to the Windows-based tasks, that could only be used after activation due to copyright restrictions.

data-overview
data-composition

Distribution of task instructions in OSWorld
based on the app domains and operation types to showcase the content intuitively.

We make a comparison of **OSWorld** against some other different benchmarks for digital agents as presented below.
**The columns indicate:** whether they provide a controllable executable environment *(Control. Exec. Env.)*, the ease of adding new tasks involving arbitrary applications in open domains *(Environment Scalability)*, support for multimodal agent evaluation *(Multimodal Support)*, support for and inclusion of cross-app tasks *(Cross-App)*, capability to start tasks from an intermediate initial state *(Intermediate Init. State)*, and the number of execution-based evaluation functions *(# Exec.-based Eval. Func.)*.
  OSWorld
# Instances
(# Templates)		 369
Control.
Exec. Env.		 Computer
Environment Scalability? ✔️
Multimodal
Support?		 ✔️
Cross-App? ✔️
Intermediate
Init. State?		 ✔️
# Exec.-based
Eval. Func.		 134
GAIA Mind2Web WebLINX PixelHelp MetaGUI AitW OmniAct ScreenAgent AgentBench InterCode MiniWoB++ WebShop WebArena VisualWebArena WorkArena WikiHow AssistGUI
466
 		 2350 2337 187 1125 30k 9802 70 1091 1350(3) 125 12k(1) 812(241) 910(314) 23k(29) 150(16) 100
❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ Multi-isolated Code Web Web Web Web Web Mobile ❌
- - - - - - - - ❌
 		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
❌
 		 ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ❌ ❌ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️
❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
❌ ✔️ ✔️ ❌
 		 ❌ ✔️ ✔️ ✔️ ❌ ❌ ❌ ❌ ❌ ❌ ✔️ ❌ ✔️
0 0 0 0 0 0 0 0 7 3 125 1 5 6 7 16 2

Benchmark

We adopt state-of-the-art LLM and VLM from open-source representatives such as UI-TARS, Agent-S, Qwen, Mixtral and CogAgent, and closed-source ones from Operator, GPT, Gemini, and Claude families on **OSWorld**, as LLM and VLM agent baselines. We also explore methods such as the Set-of-Marks aided approach, which has been demonstrated to improve spatial capabilities for visual reasoning. **We are actively updating the benchmark with new LLMs, VLMs and methods. Pull requests welcomed!**

Results

These are official results evaluated by our team under unified settings and environment. All models are tested with consistent evaluation protocols to ensure fair comparison.

For self-reported results and progress trends across different modalities, click here.

Loading verified benchmark data...

Analysis

We conduct a qualitative analysis in the aspect of models, methods, and human to find out factors influencing the performance of VLMs in digital agent tasks and their underlying behavioral logic. We investigate the impact of task attributes (such as difficulty, feasibility, visual requirement, and GUI complexity), input measurements (such as screenshot resolution, the influence of trajectory history, and the effect of UI layout), and explore whether there are patterns in the agent's performance across different operating systems. Here is an overview of our analysis outcome.

Videos

Special thanks to the following YouTubers and enthusiasts for their reports. We are delighted to see the community's interest. If you would like a brief video introduction and their thoughts, feel free to check them out!

Acknowledgement

We thank Sida Wang, Peter Shaw, Alane Suhr, Luke Zettlemoyer, Haoyuan Wu, Junli Wang, Chengyou Jia, Junlin Yang, Junlei Zhang, Chen Henry Wu, Pengcheng Yin, Shunyu Yao, Xing Han Lu, Siva Reddy, Ruoxi Sun, Zhiyuan Zeng, and Lei Li for their helpful feedback on this work

Acknowledgement for OSWorld-Verified

Special thanks to the following institutions that provided feedback and participated in the fixes (as well as institutions that provided feedback during the process): [MoonShot AI, a.k.a. Kimi](https://www.moonshot.ai/)，[Human Data](https://www.hud.so/), [OpenAI](https://openai.com/), [ByteDance Seed TARS](https://seed-tars.com/), [Anthropic](https://www.anthropic.com/), [Simular](https://www.simular.ai/), [HKU Data Intelligence Lab](https://sites.google.com/view/chaoh)
Special thanks to the following students who participated in the specific fixes: [Mengqi Yuan](https://yuanmengqi.github.io/), [Danyang Zhang](https://zdy023.github.io/), [Xinzhuang Xiong](https://thisisxxz.com/), [Zhennan Shen](https://scholar.google.com/citations?user=JPwg5MwAAAAJ&hl=en), [Zilong Zhou](https://github.com/adlsdztony), Yanxu Chen, [Jiaqi Deng](https://millank0817.github.io/), [Tianbao Xie](https://tianbaoxie.com/), Junda Chen, [Jixuan Chen](https://chenjix.github.io/), [Haoyuan Wu](https://www.linkedin.com/in/haoyuan-wu-240878291/).
Special thanks to the following students who participated in running the re-evaluation: [Mengqi Yuan](https://yuanmengqi.github.io/), [Zilong Zhou](https://github.com/adlsdztony), [Xinyuan Wang](https://xinyuanwangcs.github.io/), [Bowen Wang](https://bowenbryanwang.github.io/).

Evaluation

Local Evaluation

Please start by reading through the [agent interface](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/README.md). Correctly implement the agent interface and import your customized version in the `run.py` or `run_multienv.py` file. Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.

Public Evaluation

If you want your results to be verified and displayed on the verified leaderboard, you need to schedule a meeting with us (current maintainer: tianbaoxiexxx@gmail.com, yuanmengqi732@gmail.com) to run your agent code on our side and have us report the results. You need to upload and allow us to disclose your agent implementation under the OSWorld framework (you may choose not to expose your model API to the public), along with a report that allows the public to understand what's happening behind the scenes. Alternatively, if you are from a trusted institution, you can share your monitoring data and trajectories with us. Please carefully follow the [Public Evaluation Guideline](https://github.com/xlang-ai/OSWorld/blob/main/PUBLIC_EVALUATION_GUIDELINE.md) to get results.

FAQ

What is the username and password for the virtual machines?

The username and password for the virtual machines are as follows (for provider vmware, virtualbox and docker): we set the account credentials for Ubuntu as user / password.
For cloud service providers like aws, to prevent attacks due to weak passwords, we default to osworld-public-evaluation.
If you make further modifications, remember to set the client_password variable and pass it to DesktopEnv and Agent (if supported) when running experiments.
Some features like setting up proxy require the environment to have the client VM password to obtain sudo privileges, and for some OSWorld tasks, the agent needs the password to obtain sudo privileges to complete them.

How to setup the account and credentials for Google and Google Drive?

See Account Guideline.

How can I configure a proxy for the VM (if I'm behind the GFW, or I don't want some of my tasks to be identified as bot and get lower scores)?

If you want to set it up yourself, please refer to Proxy Guideline.
We also provide a pre-configured solution based on dataimpulse, please refer to proxy-setup section in PUBLIC_EVALUATION_GUIDELINE.

BibTeX

@misc{OSWorld,
      title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments},
      author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},
      year={2024},
      eprint={2404.07972},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}