AgentBench vs evals-skills
AgentBench is much bigger: 3.8k stars against 1.7k. Over the days we have tracked them AgentBench moved +4.5% and evals-skills +271.1%, so evals-skills is growing faster right now.
They split the axes: AgentBench leads on model freedom, evals-skills on noise control and customization and privacy and setup ease.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Provides an end-to-end, multi-environment benchmark specifically for evaluating LLMs as interactive agents with containerized task workers and function-calling support.
- Stars
- 3.8k
- Tracked growth
- +4.5%
- Maturity
- ●●●●●
- Last commit
- 265d ago
- Language
- Python
- License
- Apache-2.0
- Cost to run
- Your OpenAI API key required
Provides modular, reusable agent 'skills' (not just benchmarks), including an error-discovery skill that builds a dependency-free single-file review app for guided failure-mode analysis.
- Stars
- 1.7k
- Tracked growth
- +271.1%
- Maturity
- ●●●●●
- Last commit
- 7d ago
- Language
- mixed
- License
- Apache-2.0
- Cost to run
- Free, local
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | AgentBench | evals-skills |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Env/task only | ●●●●● Single-file analysis |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● None mentioned | ●●●●● Severity prioritization |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config files | ●●●●● Custom skills/plugins |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● API key cloud | ●●●●● Fully local |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Configurable provider | ●●●●● Fixed provider |
Setup ease What it takes to get a first useful run out of it. | ●●●●● Config + API key | ●●●●● One-command install |
Which one to pick
Pick AgentBench if…
Comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.
- Model freedom: Configurable provider (3/5 against 1/5)
Pick evals-skills if…
Easy setup — one-command install via npx and a local, dependency-free review app for interactive error discovery and building product-specific evals.
- Noise control: Severity prioritization (3/5 against 1/5)
- Customization: Custom skills/plugins (5/5 against 3/5)
- Privacy: Fully local (5/5 against 3/5)
- Setup ease: One-command install (5/5 against 3/5)
What people want from each one
THUDM/AgentBench
Hacker News: AgentBench: Evaluating LLMs as Agents drew 2 points and 1 comments.
Questions people ask
Is AgentBench better than evals-skills?
They split the axes: AgentBench leads on model freedom, evals-skills on noise control and customization and privacy and setup ease. AgentBench is worth picking when comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.
Which of AgentBench and evals-skills keeps my code private?
AgentBench: API key cloud (3/5). evals-skills: Fully local (5/5).
What does each one cost to run?
AgentBench: Your OpenAI API key required. evals-skills: Free, local.
Full profiles: THUDM/AgentBench and ai-evals-course/evals-skills. Everything else in Evals & benchmarks.