deep-swe vs evals-skills
The two are close in size: 1.9k stars for deep-swe, 1.5k for evals-skills. Over the days we have tracked them deep-swe moved +51.3% and evals-skills +238.5%, so evals-skills is growing faster right now.
They split the axes: deep-swe leads on context depth and model freedom, evals-skills on customization and privacy and setup ease.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Provides long-horizon, behaviorally-graded software-engineering tasks with isolated sandbox execution and separate verifier environments (via Pier).
- Stars
- 1.9k
- Tracked growth
- +51.3%
- Maturity
- ●●●●●
- Last commit
- 56d ago
- Language
- Python
- License
- Apache-2.0
- Cost to run
- Your API key (OpenAI/Anthropic), pay-per-run model costs
Provides modular, reusable agent 'skills' (not just benchmarks), including an error-discovery skill that builds a dependency-free single-file review app for guided failure-mode analysis.
- Stars
- 1.5k
- Tracked growth
- +238.5%
- Maturity
- ●●●●●
- Last commit
- 27d ago
- Language
- mixed
- License
- Apache-2.0
- Cost to run
- Free, local
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | deep-swe | evals-skills |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Whole-repo analysis | ●●●●● Single-file analysis |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● Behavioral verification | ●●●●● Severity prioritization |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config + prompts | ●●●●● Custom skills/plugins |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● Cloud via API key | ●●●●● Fully local |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Multiple providers | ●●●●● Fixed provider |
Setup ease What it takes to get a first useful run out of it. | ●●●●● CLI + API key | ●●●●● One-command install |
Which one to pick
Pick deep-swe if…
Long-horizon evaluation — choose DeepSWE when you need realistic, multi-step engineering tasks with programmatic verifiers and sandboxed grading to measure end-to-end agent behavior.
- Context depth: Whole-repo analysis (5/5 against 1/5)
- Model freedom: Multiple providers (3/5 against 1/5)
Pick evals-skills if…
Easy setup — one-command install via npx and a local, dependency-free review app for interactive error discovery and building product-specific evals.
- Customization: Custom skills/plugins (5/5 against 3/5)
- Privacy: Fully local (5/5 against 3/5)
- Setup ease: One-command install (5/5 against 3/5)
What people want from each one
datacurve-ai/deep-swe
Hacker News: DeepSWE results are unreliable – 3/3 DSv4 "failed" tasks solved with same model drew 3 points and 0 comments.
Questions people ask
Is deep-swe better than evals-skills?
They split the axes: deep-swe leads on context depth and model freedom, evals-skills on customization and privacy and setup ease. deep-swe is worth picking when long-horizon evaluation — choose DeepSWE when you need realistic, multi-step engineering tasks with programmatic verifiers and sandboxed grading to measure end-to-end agent behavior.
Which of deep-swe and evals-skills keeps my code private?
deep-swe: Cloud via API key (3/5). evals-skills: Fully local (5/5).
What does each one cost to run?
deep-swe: Your API key (OpenAI/Anthropic), pay-per-run model costs. evals-skills: Free, local.
Full profiles: datacurve-ai/deep-swe and ai-evals-course/evals-skills. Everything else in Evals & benchmarks.