AgentBench vs chinese-llm-benchmark
The two are close in size: 3.8k stars for AgentBench, 6.5k for chinese-llm-benchmark. Over the days we have tracked them AgentBench moved +4.5% and chinese-llm-benchmark +2.3%, so AgentBench is growing faster right now.
chinese-llm-benchmark leads on noise control. AgentBench does not take any axis by a clear margin.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Provides an end-to-end, multi-environment benchmark specifically for evaluating LLMs as interactive agents with containerized task workers and function-calling support.
- Stars
- 3.8k
- Tracked growth
- +4.5%
- Maturity
- ●●●●●
- Last commit
- 265d ago
- Language
- Python
- License
- Apache-2.0
- Cost to run
- Your OpenAI API key required
Chinese-first, large-scale benchmark that covers hundreds of commercial and open-source models and ships a >2M-entry defect database for model diagnosis.
- Stars
- 6.5k
- Tracked growth
- +2.3%
- Maturity
- ●●●●●
- Last commit
- 5d ago
- Language
- mixed
- License
- none declared
- Cost to run
- Requires Nonelinear API key (cloud service)
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | AgentBench | chinese-llm-benchmark |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Env/task only | ●●●●● Diff only |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● None mentioned | ●●●●● Basic scoring filters |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config files | ●●●●● Config & custom data |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● API key cloud | ●●●●● Cloud API (your key) |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Configurable provider | ●●●●● Gateway to many models |
Setup ease What it takes to get a first useful run out of it. | ●●●●● Config + API key | ●●●●● API key required |
Which one to pick
Pick AgentBench if…
Comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.
Pick chinese-llm-benchmark if…
Chinese-focused — pick this when you need the most extensive, regularly updated multi-domain Chinese LLM benchmark and a huge defect corpus for analysis.
- Noise control: Basic scoring filters (3/5 against 1/5)
What people want from each one
THUDM/AgentBench
Hacker News: AgentBench: Evaluating LLMs as Agents drew 2 points and 1 comments.
jeinlee1991/chinese-llm-benchmark
Questions people ask
Is AgentBench better than chinese-llm-benchmark?
chinese-llm-benchmark leads on noise control. AgentBench does not take any axis by a clear margin. AgentBench is worth picking when comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.
Which of AgentBench and chinese-llm-benchmark keeps my code private?
AgentBench: API key cloud (3/5). chinese-llm-benchmark: Cloud API (your key) (3/5).
What does each one cost to run?
AgentBench: Your OpenAI API key required. chinese-llm-benchmark: Requires Nonelinear API key (cloud service).
Full profiles: THUDM/AgentBench and jeinlee1991/chinese-llm-benchmark. Everything else in Evals & benchmarks.