Devin cloud testing of PRs

Devin writes a test plan, runs the app on a cloud VM with computer use, and returns annotated test videos and reports.

Resultrc_54HT9P8Y

Test runs approved per day on Devin more than doubled in a couple of months, and engineers run 10 to 20 Devins in parallel.

NumbersBlog · May 29 · Last checked Oct 13, 2026

Overview

How it was set up

Devin's computer use tools on a cloud VM, a code-grounded test plan, login scripts saved as repo testing skills, and a YAML blueprint snapshot.

Worth knowing

Timing issues can miss transient UI like toasts, and models sometimes run JavaScript instead of clicking through the UI. First setup often needs credentials.

A prompt to start from

The source does not show the prompt.

Sources 1

Originalcognition.ai/blog/testing-development

History

Source published: 2026-05-29
Added to Agentic Feats: 2026-10-13
Pick: No
Markdown copy: /c/devin-cloud-testing-of-prs.md

More in software companies

15 feats

Numbers·Devin·Partial

One month testing Devin

ResultSmall scoped fixes took about five minutes to test and merge versus roughly an hour by hand; the full-stack task failed and was rewritten.

Blog · Jan 23

Numbers·Devin·Partial

Docs maintenance with Devin

ResultDevin found nearly every old term in a docs-wide rename, but the edits still needed 20+ manual commits.

Blog · Jun 23

Numbers·Claude Code, Cursor, Other

Mental models framework built with agents

ResultThe author shipped 120 models, an MCP server for Claude Desktop, and 140 chaos tests with a 100% pass rate over 22 months.

HN · Nov 26