Fixing flaky CI tests overnight
Claude Code monitored 30 days of CI results, found the worst flaky tests, narrowed reproductions, and reran fixes overnight to verify them statistically.
A legacy pipeline with a roughly 40% pass rate lost 20 minutes of runtime and gained 35% pass rate in weeks.
Overview
How it was set up
Claude with read access to CI artifacts, plus monitoring tooling Claude built to track the rolling pass/fail rate over 30 days.
Worth knowing
Some of the proposed fixes were bad and the author had to throw them away. Pull requests still got human review.
A prompt to start from
Read the CI artifacts for [repo] and track the pass/fail rate over the last [30] days. Find the worst flaky tests, decide whether each is a test or production issue, and build a minimal reproduction. If it cannot be reproduced, propose a fix and rerun the tests until you can judge the fix statistically.
Written by Agentic Feats from the source. Not the author's exact prompt.
Sources 1
History
- Source published
- 2026-05-01
- Added to Agentic Feats
- 2026-10-13
- Pick
- No
- Markdown copy
- /c/fixing-flaky-ci-tests-overnight-claude-code.md
More in coding
68 featsResultA live blog aggregator pulling posts from Dev.to, AWS Community Builders, Bravo LT and RedMonk, built and deployed in a single day.
ResultA live MVP website for the podcast and newsletter went up after about 10 hours of struggling to load it on a custom URL.
ResultWithin one night Devin produced a working unanswered-emails interface using the Gmail API, after the author wrote the authentication code themselves.