# OSWorld-V2

> A benchmark that tests AI agents on long, real-world computer tasks

- Repository: https://github.com/xlang-ai/OSWorld-V2
- Category: AI agents
- Stars: 381 · Forks: 56 · Language: Python · License: Apache-2.0
- Experience: Advanced
- Page: https://ir.adyog.com/repos/xlang-ai/osworld-v2/

Computer-use agents are models that look at a screen and operate a computer with the mouse and keyboard. To compare them fairly, you need a shared set of tasks and a way to score them. OSWorld-V2 supplies both, with a focus on long tasks that resemble real work.

An agent runs inside a virtual machine, either a Docker image or an AWS machine. Some tasks also use mocked websites and a self-hosted GitLab. The Python task classes and assets are not in the GitHub repo. They live in gated Hugging Face datasets, which the authors say reduces answer leakage. Each benchmark release pins the code, tasks, assets, websites and VM images so runs stay comparable.

It succeeds the original OSWorld (xlang-ai/OSWorld), and the README includes a migration guide from 1.0. It recommends the osworld-v2.1 release, not the main branch, for reproducible results.

## What it does

- **Versioned benchmark releases**: Each release manifest pins the code, task files, assets, websites and provider images, so results can be compared between runs.
- **Docker and AWS providers**: Docker suits Linux servers, especially ones with KVM. AWS suits large parallel evaluation or training. Images are provided only for these two.
- **Mocked websites and GitLab**: Some tasks run against mocked sites you can self-host. GitLab tasks need your own instance because they use a private token.
- **Gated tasks and assets**: Tasks and assets download from Hugging Face after you accept a gated access request, which the README says is approved automatically.
- **Setup skill for coding agents**: A skill called setup-osworld lets an agent help provision the checkout, asking before any cloud spend, DNS, SSH or secret step.

## What you could build

- Evaluate a computer-use agent on long-horizon desktop tasks and compare it with published results.
- Run large parallel evaluations of an agent on AWS.
- Inspect agent trajectories in the trajectory viewer or download them for analysis.
- Move an existing OSWorld 1.0 setup and tasks to the 2.1 release.
- Collect agent runs on realistic tasks as data for training or research.

## Who it's for

Researchers and engineers building or evaluating computer-use agents, who should be comfortable with Python, uv, Docker or AWS, and Hugging Face accounts.

## Get started

Clone the osworld-v2.1 tag and install dependencies with uv (Python 3.12 or newer). Then accept the gated dataset requests on Hugging Face, download the task files and assets, and set up a Docker or AWS provider as the docs describe.

```
git clone --branch osworld-v2.1 https://github.com/xlang-ai/OSWorld-V2
```
```
cd OSWorld-V2
```
```
uv sync --frozen
```

Tags: benchmark, computer-use, agents, gui, evaluation