Agent-driven QA is here to stay

Adapted from my LinkedIn post. Publication date is approximate.

Browser automation is no longer the bottleneck in our setup. With agents using tools like Playwright, getting them to click, type, and navigate the UI has become surprisingly reliable.

The real challenge is teaching the agent to think and behave like a disciplined QA tester.

Here’s what’s actually making the biggest difference in our setup.

Check what else the change could affect

An agent will verify the exact checks you gave it and stop there. We train it to go wider: What else could this change have impacted? Does anything look visually off? What would a real user notice?

Start with a real test plan

We use a companion skill that turns a feature, PR description, or code diff into a solid test plan first. The execution agent then follows a thoughtful strategy instead of improvising.

Define “pass” explicitly and ruthlessly

Left on its own, an agent will settle for the weakest possible bar—usually “it didn’t crash.” We spell it out clearly: no console errors, no silently swallowed exceptions, and no “handled gracefully” excuses. An error is an error.

Once this definition lives in the agent’s skill, the quality bar stops drifting.

Test against realistic environments

An agent will happily pass against stubbed responses that real users will never see. We make it stand up realistic conditions instead of relying on mocks and fake data for these checks.

Separate issue detection from triage

The testing agent’s only job is to surface every single issue it finds—no filtering, no “probably pre-existing,” no excuses.

A separate step, or a second agent, handles triage: Is this new? Does it block the PR? Should it be fixed now or tracked?

Keeping these roles separate keeps the report honest.

Be deliberate about test data

How data gets into the system matters as much as the test itself. Sometimes synthetic data is acceptable. Other times it must be created through the UI exactly as a real user would. We give the agent clear rules for both cases.