Who is AutomationBench for?

Suits AI engineers and researchers who evaluate or train agents; you should be comfortable with Python, API keys and the command line.

What can you build with AutomationBench?

Compare several LLMs on realistic business automation tasks before choosing one for your product. Measure whether a prompt or agent framework change improves workflow completion in a given domain. Use partial_credit as a reward signal when training or tuning agents on business workflows. Run the simple domain to check that a new model handles basic tool calling. Inspect failed tasks in the visualizer to see where an agent goes wrong.

zapier /

AutomationBench

Benchmark that tests AI agents on business tasks across 47 simulated SaaS tools

352 stars
33 stars since 3 Oct
Python
Updated 3 months ago
Some experience
AI agents

Open on GitHub (opens in a new tab) Website (opens in a new tab)

Most agent tests check whether a model can call a tool. AutomationBench asks a harder question: can the agent finish a real business workflow, like routing a support ticket or onboarding an employee, and leave the systems in the right state?

Each task starts with trigger data and a pre-filled simulated environment spanning 47 SaaS tools. The agent gets tools to work with it and runs on its own, with no user to talk to. When it stops, the final state is compared with the expected state using assertions. Every task has to be checkable by code.

The public repo has 600 scored tasks, 100 each in sales, marketing, operations, support, finance and HR, plus 200 easy tasks as a warm-up. The official leaderboard uses a separate, harder private task set, so local scores point the same way but won't match exactly.

What it does

Six business domains
Sales, marketing, operations, support, finance and HR, with 100 public tasks each. A separate simple domain of 200 basic tasks checks basic tool use and is not scored.
State-based checking
Tasks pass or fail by comparing the final simulated environment with the expected one through assertions, not by judging the agent's text.
Two scores per task
partial_credit is the fraction of assertions met and works as a training reward. task_completed_correctly is strict and is 1.0 only if every assertion passes.
Configurable CLI
Run any model by name, and choose domains, reasoning effort, step limits, concurrency and tool style (api, zapier or limited_zapier). OpenAI and Anthropic keys are supported.
Visualizer and hosted option
A local web visualizer shows charts and filterable results and can compare 2 to 5 runs. The benchmark can also be run on the Prime Intellect Environments Hub.

What you could build

Compare several LLMs on realistic business automation tasks before choosing one for your product.
Measure whether a prompt or agent framework change improves workflow completion in a given domain.
Use partial_credit as a reward signal when training or tuning agents on business workflows.
Run the simple domain to check that a new model handles basic tool calling.
Inspect failed tasks in the visualizer to see where an agent goes wrong.

Get started

Clone the repo, install dependencies with uv, and set your model provider's API key (for example OPENAI_API_KEY). Then run auto-bench with a model name, optionally limiting it to one domain with --domains.

git clone https://github.com/zapier/AutomationBench.git

uv sync

uv run auto-bench --model gpt-5-mini

Commands are copied from the project's README. Read it on GitHub for the full setup.