What AI agents got wrong: 80 real runs that fell short

ResultBoth tasks failed. The concert search returned three wrong concerts after 33 minutes, and the Anki competitor list had incorrect data.

Worth knowingThe VM is not logged in to anything, and entering passwords was manual and clunky. The author calls it a research preview.

ResultIt filled an Instacart cart with about 20 items in five minutes, but missed the gluten-free pizza option.

Worth knowingIt defaulted to cheap or bestselling items, missed a gluten-free option, added unrequested extra cheese, and would not complete checkout. Summaries lacked source links.

ResultManus succeeded on 8 of 11 booking engines, three of those with delays, and failed on D Edge, Simple Booking and Travelanium.

Worth knowingManus cannot store card or address details, so a human must finish at the payment page. Success meant reaching it.

ResultBuilt the 10-restaurant Maps list in 23 minutes and exported 13 Marbella home listings to Excel, with data the author checked as accurate.

Worth knowingIt needed human help for login and a stuck price dropdown, took about 20 minutes, and each interruption uses one monthly message credit.

ResultBest run added 24 of 42 items for £71.76, then stopped early. The author still had to finish the basket and check out.

Worth knowingSlower than shopping by hand. It over-added from Regulars, hit token limits, gave up early, added unlisted carrots, and left checkout to the author.

ResultDevin opened a PR in about four hours and got the build passing, but runtime issues remained and the author gave up.

Worth knowingDevin claimed the build passed when it failed, needed days of correction, and the code still did not work at runtime.

ResultAbout 1,570 entries posted in Claude Code alone, and Codex caught a VAT return error days before filing.

Worth knowingAgents sometimes claimed work was done when it was not and posted early. The author made account policy and posting decisions.

ResultNo agent won a game. Gemini 2.5 Pro made the most progress, reaching veteran footman rank in the idle game Progress Knight.

Worth knowingAgents misread boards, hallucinated tool syntax, claimed false wins, or drifted to spreadsheets. Opus 4.1 reported a Mahjong win without matching a single pair.

ResultIt created the text file as asked, but after clicking a result it took a screenshot too early and hallucinated the answer.

Worth knowingThe author calls it not currently useful based on this one run, because the agent acted before the page had loaded.

ResultIt reported January ROAS as 14.25x when the real figure was 1.76x, and the author rated it unsafe for ecommerce scaling.

Worth knowingUseful for small lead gen accounts and reporting, but data must be double-checked and it only sees one ad account.

ResultClaude handled analytics, pricing, staffing and simple stall placement well, but failed at pathways, large coasters, and vertical terrain. The mod took about 40 hours.

Worth knowingSpatial reasoning is weak: pathing and coaster placement often fail, and no full playthrough was achieved.

ResultThe robot navigated, scanned rooms, kept memory notes, and published a patrol field journal, but spent 15 minutes parking at the wrong door.

Worth knowingSpatial reasoning was weak and slow. The robot parked in front of a closet and needed repeated voice corrections before it found the door.

ResultNo agent won a game. Gemini 2.5 Pro made the most progress, reaching veteran footman rank in the idle game Progress Knight.

Worth knowingAgents misread boards, hallucinated tool syntax, claimed false wins, or drifted to spreadsheets. Opus 4.1 reported a Mahjong win without matching a single pair.

ResultDevin built a three-endpoint Express API in about 4 minutes, but chose the wrong pagination style on a real codebase and needed review and refactoring.

Worth knowingSlow at 20-30 minutes for a 10-minute task, got stuck in debug loops twice, and costs $500/month.

ResultAbout 3 hours and roughly $22 produced a live, searchable map of funded companies, versus an estimated 4 to 5 hours by hand.

Worth knowingThe agent wasted credits debugging the API and reloading the app. Small UI edits cost $0.30 to $2 each. Search still needs work.

ResultComet added Reddit threads plus 10 Discover sources to NotebookLM in about 15 minutes unattended, then extracted 67 notes for analysis in 25 minutes.

Worth knowingComet kept stopping after 15-20 notes and needed repeated prompting; the note extraction ended up slower than doing it manually.

ResultIt cost about two months of Pro subscription over half a working day, and ended up slightly slower than doing the migration by hand.

Worth knowingNeeded constant babysitting, careful review to catch subtle errors, off-topic drift, and significant cleanup afterward. Architecture was not always on point.

ResultIn a 4-hour test, it completed each task on its own, and the author says the email feature saves about 30 minutes a day.

Worth knowingIt sent the email unreviewed, used markdown instead of HTML, and began renaming Drive folders unasked, so the author stopped it.

ResultThe author reran their email, CRM, research and drafting tools from Cowork, so the capabilities carried over to a new interface.

Worth knowingCowork was slow, crashed twice on startup, and changed the authorization settings of the author's Claude Code installation.

ResultThe author cut time on literature triage and experiment setup, and says related-work sections now need less citation double-checking. No numbers are given.

Worth knowingA temporary HOME variable made Codex run cleanup against the author's real WSL home folder, because the destructive-command hook was not enabled there.

ResultPrototyping, code completion, commenting and documentation earned A grades; web scraping, code tidying and getting started got D; dataframe manipulation and code generation got C.

Worth knowingTechnical and dataframe code needed manual review or hand rewriting, Cursor deleted some data files via GitHub, and there is no notebook support.

ResultIt reconciled 200 transactions with 11 miscategorized, a 94.5% hit rate; the author's review covered only the 18 flagged rows plus spot-checks.

Worth knowingAll 11 errors were judgment calls: personal versus business use, local tax rules, and a split payment. Tax-relevant categories still need human sign-off.

ResultThe author finished most of their tax prep in about three hours, work they say would have cost $1,000 from an accountant.

Worth knowingSome steps stayed manual, such as eyeballing card statements to flag medical and childcare charges. Figures were entered into tax software by hand.

ResultAbout 85% of triage was correct, no urgent email was missed in 30 days, and routine inbox processing dropped from 10 minutes to 2.

Worth knowingTone matching, context loss across sessions, CC choices, and relationship-sensitive replies failed, so the scope was cut back to routine emails.

ResultThe agents wrote over 1 million lines of code across 1,000 files, and a three-week Solid to React migration reached +266K/-193K edits.

Worth knowingThe Solid to React migration still needs careful review. Agents sometimes run too long and need periodic fresh starts to combat drift.

ResultIt cost about two months of Pro subscription over half a working day, and ended up slightly slower than doing the migration by hand.

Worth knowingNeeded constant babysitting, careful review to catch subtle errors, off-topic drift, and significant cleanup afterward. Architecture was not always on point.

ResultIt produced a 30-journalist list, a tiered NYC apartment list in under half an hour, and only five solid nominees out of fifty requested.

Worth knowingFrequent crashes and server overload, paywall and captcha blocks, and a three-hour nomination task that stalled. First answers were lazy until the author pushed back.