File 07 · IBM Bob 2.0 Hackathon · lablab.ai · Sep 2026
Deflake
Flaky tests get fixed, not retried. It finds tests that pass and fail with no code change, and sends each to an agent that fixes the root cause.
- 10 → 100%CI green rate, 40 runs before, 100 afterMeasured
- 5/5root causes matched blindMeasured
- 0masking findingsMeasured
The demo is a small storefront backend with 18 tests. Five are flaky, each for a different reason: shared state, an unordered collection, a race, randomness and the clock.
The agent worked blind. The answer key was written before it saw the code and kept out of the repo, with its SHA-256 in the first commit. The key was added only after the fixes.
Some fixes landed in the product, not the test, because that is where the bug was.
What it refuses to doAccepting a fix that only hides the problem. A gate rejects retries, longer sleeps, skips and weaker asserts, then proves the rest over 100 randomized runs.
How it runs
- ScanRuns pytest dozens of times in randomized order.
- HuntOne parallel subagent per flaky test, in an IBM Bob custom mode.
- GateRejects anything that masks the failure.
- Prove100 randomized runs and a written certificate.
Honest caveatResults are on a demo project built with known causes. That is what the sealed answer key is for.
Next fileFocus Freeze →