chinese-llm-benchmark vs evals-skills
chinese-llm-benchmark is much bigger: 6.4k stars against 548.
They split the axes: chinese-llm-benchmark leads on model freedom, evals-skills on customization and privacy and setup ease.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Chinese-first, large-scale benchmark that covers hundreds of commercial and open-source models and ships a >2M-entry defect database for model diagnosis.
- Stars
- 6.4k
- Tracked growth
- +5%
- Maturity
- ●●●●●
- Last commit
- 3d ago
- Language
- mixed
- License
- none declared
- Cost to run
- Requires Nonelinear API key (cloud service)
Provides modular, reusable agent 'skills' (not just benchmarks), including an error-discovery skill that builds a dependency-free single-file review app for guided failure-mode analysis.
- Stars
- 548
- Tracked growth
- not tracked long enough
- Maturity
- ●●●●●
- Last commit
- 6d ago
- Language
- mixed
- License
- none declared
- Cost to run
- Free, local
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | chinese-llm-benchmark | evals-skills |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Diff only | ●●●●● Single-file analysis |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● Basic scoring filters | ●●●●● Severity prioritization |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config & custom data | ●●●●● Custom skills/plugins |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● Cloud API (your key) | ●●●●● Fully local |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Gateway to many models | ●●●●● Fixed provider |
Setup ease What it takes to get a first useful run out of it. | ●●●●● API key required | ●●●●● One-command install |
Which one to pick
Pick chinese-llm-benchmark if…
Chinese-focused — pick this when you need the most extensive, regularly updated multi-domain Chinese LLM benchmark and a huge defect corpus for analysis.
- Model freedom: Gateway to many models (3/5 against 1/5)
Pick evals-skills if…
Easy setup — one-command install via npx and a local, dependency-free review app for interactive error discovery and building product-specific evals.
- Customization: Custom skills/plugins (5/5 against 3/5)
- Privacy: Fully local (5/5 against 3/5)
- Setup ease: One-command install (5/5 against 3/5)
What people want from each one
jeinlee1991/chinese-llm-benchmark
Questions people ask
Is chinese-llm-benchmark better than evals-skills?
They split the axes: chinese-llm-benchmark leads on model freedom, evals-skills on customization and privacy and setup ease. chinese-llm-benchmark is worth picking when chinese-focused — pick this when you need the most extensive, regularly updated multi-domain Chinese LLM benchmark and a huge defect corpus for analysis.
Which of chinese-llm-benchmark and evals-skills keeps my code private?
chinese-llm-benchmark: Cloud API (your key) (3/5). evals-skills: Fully local (5/5).
What does each one cost to run?
chinese-llm-benchmark: Requires Nonelinear API key (cloud service). evals-skills: Free, local.
Full profiles: jeinlee1991/chinese-llm-benchmark and ai-evals-course/evals-skills. Everything else in Evals & benchmarks.