AI repos for LLM evals
Open-source AI projects built around LLM evals or genuinely useful if you work with it, each with what it does for you. 5 repos, most relevant first.
-
OSWorld-V2
Reproducible, versioned benchmark for testing computer-use agents on long multi-step GUI tasks.
-
AutomationBench
Compare models on sales, finance, HR and support tasks with state-based checking across simulated SaaS tools.
-
dexter
Includes an evaluation suite for testing the financial research agent.
-
maka
Publishes evaluations of agent runs you can compare against.
-
reasoning-from-scratch
Chapters evaluate reasoning accuracy of the model as techniques are added.