---
title: "Fixing flaky CI tests overnight"
canonical: "https://agenticfeats.com/c/fixing-flaky-ci-tests-overnight-claude-code/"
agents: ["Claude Code"]
evidence_grade: "B"
outcome: "completed"
job_type: "Coding"
roles: ["Developer", "Operations"]
source_url: "https://news.ycombinator.com/item?id=47970174"
author: "SeanAnderson"
source_date: "2026-05-01"
published: "2026-10-13"
updated: "2026-10-13"
---

> From Agentic Feats: sourced examples of AI agents at work. Map of the site for agents: https://agenticfeats.com/llms.txt

# Fixing flaky CI tests overnight

Claude Code monitored 30 days of CI results, found the worst flaky tests, narrowed reproductions, and reran fixes overnight to verify them statistically.

Result: A legacy pipeline with a roughly 40% pass rate lost 20 minutes of runtime and gained 35% pass rate in weeks.

## How it was set up

Claude with read access to CI artifacts, plus monitoring tooling Claude built to track the rolling pass/fail rate over 30 days.

## Worth knowing

Some of the proposed fixes were bad and the author had to throw them away. Pull requests still got human review.

## Prompt

```text
Read the CI artifacts for [repo] and track the pass/fail rate over the last [30] days. Find the worst flaky tests, decide whether each is a test or production issue, and build a minimal reproduction. If it cannot be reproduced, propose a fix and rerun the tests until you can judge the fix statistically.
```

## Evidence note

Author states pipeline and pass-rate numbers; no repo or artifact linked

## Sources

- Original: https://news.ycombinator.com/item?id=47970174

## The receipt

- Agent: Claude Code
- Evidence: Numbers (grade B, a specific number stated)
- Outcome: Completed
- Job type: Coding
- Roles: Developer, Operations
- Source: HN
- Author: SeanAnderson
- Published: 2026-05-01
- Added: 2026-10-13