Numbers·Partial·Devin·Software companies
One month testing Devin
Devin was connected to the RunLLM GitHub repo and given small bug fixes and one larger full-stack task over a month.
Small scoped fixes took about five minutes to test and merge versus roughly an hour by hand; the full-stack task failed and was rewritten.
Overview
How it was set up
Devin connected to the GitHub repo, scanned it to infer build steps, and opened pull requests. Tasks were typed into its chat interface.
Worth knowing
Devin struggled with a multi-layer task, ignored taught practices, and its code was unworkable. Backend work went worse than frontend.
A prompt to start from
The source does not show the prompt.
Sources 1
History
- Source published
- 2025-01-23
- Added to Agentic Feats
- 2026-10-13
- Pick
- No
- Markdown copy
- /c/one-month-testing-devin.md
More in software companies
15 featsResultDevin found nearly every old term in a docs-wide rename, but the edits still needed 20+ manual commits.
ResultThe author shipped 120 models, an MCP server for Claude Desktop, and 140 chaos tests with a 100% pass rate over 22 months.
ResultCognition merged 659 Devin PRs into its own codebase in one week, up from 154 in its best week of 2025.