AgentBench vs better-harness

The two are close in size: 3.8k stars for AgentBench, 2.4k for better-harness. Over the days we have tracked them AgentBench moved +4.5% and better-harness +54.5%, so better-harness is growing faster right now.

better-harness leads on context depth and noise control. AgentBench does not take any axis by a clear margin.

Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.

Where they stand today

THUDM/AgentBench

Provides an end-to-end, multi-environment benchmark specifically for evaluating LLMs as interactive agents with containerized task workers and function-calling support.

Stars: 3.8k
Tracked growth: +4.5%
Maturity: ●●●●●
Last commit: 265d ago
Language: Python
License: Apache-2.0
Cost to run: Your OpenAI API key required

QoderAI/better-harness

Reviews AI coding workflows end-to-end and produces evidence-bounded, prioritized findings across five Agent Work Loop dimensions rather than only analyzing final diffs.

Stars: 2.4k
Tracked growth: +54.5%
Maturity: ●●●●●
Last commit: 12d ago
Language: JavaScript
License: MIT
Cost to run: Depends on host and agent (may use paid services)

THUDM/AgentBenchQoderAI/better-harness

Six axes, head to head

Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.

Axis	AgentBench	better-harness
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository.	●●●●● Env/task only	●●●●● Whole-repo analysis
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only.	●●●●● None mentioned	●●●●● Evidence-backed findings
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config.	●●●●● Config files	●●●●● Extensible configs & plugins
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only.	●●●●● API key cloud	●●●●● Self-hostable
Model freedom Whether you can point it at any provider, or it is wired to one.	●●●●● Configurable provider	●●●●● Multiple host providers
Setup ease What it takes to get a first useful run out of it.	●●●●● Config + API key	●●●●● Host-specific quick start

Which one to pick

Pick AgentBench if…

Comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.

Runs in cli. Works with openai, other-fixed.

Pick better-harness if…

Workflow-focused — pick Better Harness when you want honest, evidence-backed reviews of your agents' task understanding, execution, validation, delivery, and learning capture.

Context depth: Whole-repo analysis (5/5 against 1/5)
Noise control: Evidence-backed findings (3/5 against 1/5)

Runs in cli, ide, coding-agent-plugin. Works with other-fixed.

What people want from each one

THUDM/AgentBench

Hacker News: AgentBench: Evaluating LLMs as Agents drew 2 points and 1 comments.

Most-wanted open issues

QoderAI/better-harness

Most-wanted open issues

Questions people ask

Is AgentBench better than better-harness?

better-harness leads on context depth and noise control. AgentBench does not take any axis by a clear margin. AgentBench is worth picking when comprehensive evaluation — choose AgentBench when you need an end-to-end, multi-task benchmark to measure and compare LLMs as autonomous agents across diverse interactive environments.

Which of AgentBench and better-harness keeps my code private?

AgentBench: API key cloud (3/5). better-harness: Self-hostable (4/5).

What does each one cost to run?

AgentBench: Your OpenAI API key required. better-harness: Depends on host and agent (may use paid services).

Full profiles: THUDM/AgentBench and QoderAI/better-harness. Everything else in Evals & benchmarks.