Challenge
The client’s automated test suite for apps and web portals runs around the clock in CI and on real devices. Two engineers used to handle every failure by hand: watching CI, collecting evidence, attributing the cause, writing the ticket - about 16 hours a day.
Solution
We built an agent system that runs this loop:
- Monitoring and evidence - every failure is captured, documented with logs and screenshots and matched against known cases via fingerprints.
- Triage agent - attributes each new failure: test automation, system under test or blocked.
- Repair agent - fixes approved test defects and re-runs the tests on real devices.
- Decisions stay with the team - tickets go to the client’s system; fix approval and merge are done by people via an internal dashboard.
How the system works
deterministicagenthuman decision
Detection & triage
Test run, continuous
→
Monitor: failed N times
→
Evidence assembly
→
Fingerprint match (vector search)
→
Triage agent: verdict · confidence · fix proposal
→
Incident on the internal dashboard
Known cause (~40% of incidents recur) → added to the existing ticket automatically.
Category 1 · Agentic fix
Review: approve fix
→
Repair agent in a container
→
Pull request, tests re-run
→
Review & merge
Category 2 · Human assistance
No agentic fix possible
→
Engineer takes over and resolves
→
Ticket to the team
Category 3 · Stakeholder reporting
Assess impact, decide whom to inform
→
Automated message
→
Ticket or e-mail to stakeholder
~2,000failed test runs checked
~100incidents per month
20-45tickets per week
~93%of causes lie in the system under test
Result
- The system takes over around 14 of 16 hours of routine work a day (≈ 88%). The team only reviews and decides - about 2 hours a day - and spends the rest on work that moves things forward.
- Around 2,000 failed test runs checked in the sample month; about 40% of incidents recur and are matched to the existing ticket automatically.
- The same evidence set and a reasoned verdict for every failure.
Team assessment for a sample month compared with fully manual operation.
Deliberately not yet measured: verdict accuracy, repair outcomes and time per incident. The build started with around 1.5 FTE; the freed capacity went into further development. Model costs are around €200-400 per month for the whole system.
* Project: we owned the delivery · ● in operation: we run the solution today
