Month-long Devin trial
Devin was tested by the Answer.AI team on 20+ real coding, research, and code-modification tasks over a month through Slack.
Out of 20 tasks, Devin had 14 failures, 3 successes, and 3 inconclusive results, with no predictable pattern for which would work.
Overview
How it was set up
Devin used through Slack and its web interface, running in its own Docker environment with browser, editor, and shell.
Worth knowing
Tasks often took days, Devin pushed ahead on impossible tasks like a multi-app Railway deployment, and produced overly complex code.
A prompt to start from
The source does not show the prompt.
Sources 1
History
- Added to Agentic Feats
- 2026-10-13
- Pick
- No
- Markdown copy
- /c/month-long-devin-trial.md
More in coding
68 featsResultA live blog aggregator pulling posts from Dev.to, AWS Community Builders, Bravo LT and RedMonk, built and deployed in a single day.
ResultA live MVP website for the podcast and newsletter went up after about 10 hours of struggling to load it on a custom URL.
ResultWithin one night Devin produced a working unanswered-emails interface using the Gmail API, after the author wrote the authentication code themselves.