chinese-llm-benchmark vs deep-swe
chinese-llm-benchmark is much bigger: 6.5k stars against 1.8k. Over the days we have tracked them chinese-llm-benchmark moved +2.8% and deep-swe +52.9%, so deep-swe is growing faster right now.
deep-swe leads on context depth. chinese-llm-benchmark does not take any axis by a clear margin.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Chinese-first, large-scale benchmark that covers hundreds of commercial and open-source models and ships a >2M-entry defect database for model diagnosis.
- Stars
- 6.5k
- Tracked growth
- +2.8%
- Maturity
- ●●●●●
- Last commit
- 17h ago
- Language
- mixed
- License
- none declared
- Cost to run
- Requires Nonelinear API key (cloud service)
Provides long-horizon, behaviorally-graded software-engineering tasks with isolated sandbox execution and separate verifier environments (via Pier).
- Stars
- 1.8k
- Tracked growth
- +52.9%
- Maturity
- ●●●●●
- Last commit
- 54d ago
- Language
- Python
- License
- Apache-2.0
- Cost to run
- Your API key (OpenAI/Anthropic), pay-per-run model costs
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | chinese-llm-benchmark | deep-swe |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Diff only | ●●●●● Whole-repo analysis |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● Basic scoring filters | ●●●●● Behavioral verification |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config & custom data | ●●●●● Config + prompts |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● Cloud API (your key) | ●●●●● Cloud via API key |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Gateway to many models | ●●●●● Multiple providers |
Setup ease What it takes to get a first useful run out of it. | ●●●●● API key required | ●●●●● CLI + API key |
Which one to pick
Pick chinese-llm-benchmark if…
Chinese-focused — pick this when you need the most extensive, regularly updated multi-domain Chinese LLM benchmark and a huge defect corpus for analysis.
Pick deep-swe if…
Long-horizon evaluation — choose DeepSWE when you need realistic, multi-step engineering tasks with programmatic verifiers and sandboxed grading to measure end-to-end agent behavior.
- Context depth: Whole-repo analysis (5/5 against 1/5)
What people want from each one
jeinlee1991/chinese-llm-benchmark
datacurve-ai/deep-swe
Hacker News: DeepSWE results are unreliable – 3/3 DSv4 "failed" tasks solved with same model drew 3 points and 0 comments.
Questions people ask
Is chinese-llm-benchmark better than deep-swe?
deep-swe leads on context depth. chinese-llm-benchmark does not take any axis by a clear margin. chinese-llm-benchmark is worth picking when chinese-focused — pick this when you need the most extensive, regularly updated multi-domain Chinese LLM benchmark and a huge defect corpus for analysis.
Which of chinese-llm-benchmark and deep-swe keeps my code private?
chinese-llm-benchmark: Cloud API (your key) (3/5). deep-swe: Cloud via API key (3/5).
What does each one cost to run?
chinese-llm-benchmark: Requires Nonelinear API key (cloud service). deep-swe: Your API key (OpenAI/Anthropic), pay-per-run model costs.
Full profiles: jeinlee1991/chinese-llm-benchmark and datacurve-ai/deep-swe. Everything else in Evals & benchmarks.