deep-swe vs PostTrainBench
deep-swe is much bigger: 1.6k stars against 546. Over the days we have tracked them deep-swe moved +130.1% and PostTrainBench +55.6%, so deep-swe is growing faster right now.
PostTrainBench leads on model freedom. deep-swe does not take any axis by a clear margin.
Stars and commit dates come from our own daily tracking. The six axes are read off each project's documentation by our review pipeline, so they describe what a project says about itself, not what we measured in its code.
Where they stand today
Provides long-horizon, behaviorally-graded software-engineering tasks with isolated sandbox execution and separate verifier environments (via Pier).
- Stars
- 1.6k
- Tracked growth
- +130.1%
- Maturity
- ●●●●●
- Last commit
- 9d ago
- Language
- Python
- License
- Apache-2.0
- Cost to run
- Your API key (OpenAI/Anthropic), pay-per-run model costs
Measures autonomous CLI agents' ability to post-train base LLMs within a 10‑hour H100 budget, evaluating agent-driven R&D rather than only inference.
- Stars
- 546
- Tracked growth
- +55.6%
- Maturity
- ●●●●●
- Last commit
- 2d ago
- Language
- Python
- License
- MIT
- Cost to run
- Your API keys + H100 GPU (cluster or rented)
Six axes, head to head
Each axis runs 0 to 5. The label under a score is what that project's own docs claim, not a category average.
| Axis | deep-swe | PostTrainBench |
|---|---|---|
Context depth How much of your codebase it sees before it answers: the open diff, the diff plus related files, or the whole repository. | ●●●●● Whole-repo analysis | ●●●●● Whole-repo access |
Noise control How it keeps output volume down — severity thresholds, deduplication, incremental runs over new commits only. | ●●●●● Behavioral verification | ●●●●● Judge + rules |
Customization How far it bends to your team: custom rules, prompts, style guides, per-path config. | ●●●●● Config + prompts | ●●●●● Configurable CLI options |
Privacy Whether your code stays on your own infrastructure: fully local, self-hostable, or cloud API only. | ●●●●● Cloud via API key | ●●●●● APIs with your key |
Model freedom Whether you can point it at any provider, or it is wired to one. | ●●●●● Multiple providers | ●●●●● BYO key + local models |
Setup ease What it takes to get a first useful run out of it. | ●●●●● CLI + API key | ●●●●● Container + cluster setup |
Which one to pick
Pick deep-swe if…
Long-horizon evaluation — choose DeepSWE when you need realistic, multi-step engineering tasks with programmatic verifiers and sandboxed grading to measure end-to-end agent behavior.
Pick PostTrainBench if…
Autonomy-focused — use this to benchmark end-to-end agent-driven post-training of base models on an H100 with built-in evaluation tasks and reward‑hacking mitigations.
- Model freedom: BYO key + local models (5/5 against 3/5)
What people want from each one
datacurve-ai/deep-swe
Hacker News: DeepSWE results are unreliable – 3/3 DSv4 "failed" tasks solved with same model drew 3 points and 0 comments.
aisa-group/PostTrainBench
Questions people ask
Is deep-swe better than PostTrainBench?
PostTrainBench leads on model freedom. deep-swe does not take any axis by a clear margin. deep-swe is worth picking when long-horizon evaluation — choose DeepSWE when you need realistic, multi-step engineering tasks with programmatic verifiers and sandboxed grading to measure end-to-end agent behavior.
Which of deep-swe and PostTrainBench keeps my code private?
deep-swe: Cloud via API key (3/5). PostTrainBench: APIs with your key (3/5).
What does each one cost to run?
deep-swe: Your API key (OpenAI/Anthropic), pay-per-run model costs. PostTrainBench: Your API keys + H100 GPU (cluster or rented).
Full profiles: datacurve-ai/deep-swe and aisa-group/PostTrainBench. Everything else in Evals & benchmarks.