xlang-ai /
OSWorld-V2
A benchmark that tests AI agents on long, real-world computer tasks
- 381 stars
- Python
- Updated today
- Advanced
- AI agents
Computer-use agents are models that look at a screen and operate a computer with the mouse and keyboard. To compare them fairly, you need a shared set of tasks and a way to score them. OSWorld-V2 supplies both, with a focus on long tasks that resemble real work.
An agent runs inside a virtual machine, either a Docker image or an AWS machine. Some tasks also use mocked websites and a self-hosted GitLab. The Python task classes and assets are not in the GitHub repo. They live in gated Hugging Face datasets, which the authors say reduces answer leakage. Each benchmark release pins the code, tasks, assets, websites and VM images so runs stay comparable.
It succeeds the original OSWorld (xlang-ai/OSWorld), and the README includes a migration guide from 1.0. It recommends the osworld-v2.1 release, not the main branch, for reproducible results.
What it does
Versioned benchmark releases
Each release manifest pins the code, task files, assets, websites and provider images, so results can be compared between runs.
Docker and AWS providers
Docker suits Linux servers, especially ones with KVM. AWS suits large parallel evaluation or training. Images are provided only for these two.
Mocked websites and GitLab
Some tasks run against mocked sites you can self-host. GitLab tasks need your own instance because they use a private token.
Gated tasks and assets
Tasks and assets download from Hugging Face after you accept a gated access request, which the README says is approved automatically.
Setup skill for coding agents
A skill called setup-osworld lets an agent help provision the checkout, asking before any cloud spend, DNS, SSH or secret step.
What you could build
- Evaluate a computer-use agent on long-horizon desktop tasks and compare it with published results.
- Run large parallel evaluations of an agent on AWS.
- Inspect agent trajectories in the trajectory viewer or download them for analysis.
- Move an existing OSWorld 1.0 setup and tasks to the 2.1 release.
- Collect agent runs on realistic tasks as data for training or research.
Get started
Clone the osworld-v2.1 tag and install dependencies with uv (Python 3.12 or newer). Then accept the gated dataset requests on Hugging Face, download the task files and assets, and set up a Docker or AWS provider as the docs describe.
git clone --branch osworld-v2.1 https://github.com/xlang-ai/OSWorld-V2cd OSWorld-V2uv sync --frozenCommands are copied from the project's README. Read it on GitHub for the full setup.
More in AI agents
openclaw
Self-hosted AI assistant that answers in your chat apps and runs on your own machine
hermes-agent
Self-hosted AI agent that keeps skills and memory, and chats via terminal or Telegram
TradingAgents
A team of LLM agents that debate and decide on stock trades, for research
paperclip
A Node.js server and React dashboard for running a team of AI agents like a company
autoresearch
Let an AI agent edit a small LLM training script overnight and keep what improves it
ruflo
Agent harness adding swarms, memory and plugins around Claude Code and Codex
OpenManus
Open-source general AI agent that runs tasks from your terminal with your own LLM key
multica
A shared board where you assign issues to AI coding agents like teammates
openhuman
Rust agent harness with a desktop app, terminal client and embeddable library
buzz
Self-hostable team workspace where people and AI agents share channels on a Nostr relay