Separate the marketing claim from the actual mechanism
"AI testing" covers several genuinely different things bundled under one buzzword. Understanding which mechanism a tool actually uses tells you a lot more about its real limits than its marketing page does.
Self-healing locators
These tools capture multiple locator strategies for each element at record time, and when the primary one breaks, fall back to alternates (or a similarity heuristic) rather than failing outright. Genuinely useful for absorbing minor markup churn (an id changing, a class renamed) — but it's a band-aid for locator brittleness, not a substitute for a well-designed Page Object layer. A self-healing locator that silently "heals" onto the wrong element is a worse failure mode than an honest, loud test failure — worth verifying how conservative a given tool actually is before trusting it in CI.
AI-assisted test generation
LLM-based tools that generate test code from a plain-English description or a recorded user flow are legitimately useful for a first draft — turning "test that a user can add an item to cart and checkout" into a runnable Playwright/Selenium skeleton in seconds. What they're not good at yet: reliably generating correct, comprehensive edge cases without a human reviewing and extending them, or understanding your specific app's business rules well enough to write assertions you'd trust unreviewed in production.
Agentic/autonomous testing
The newest category: an AI agent that explores an app on its own (clicking, filling forms, navigating) with only a high-level goal, rather than a scripted set of steps. This is genuinely promising for exploratory testing and catching unexpected UI states a scripted suite wouldn't think to check — but it comes with real trade-offs today: runs aren't fully deterministic (two runs can explore differently), debugging a failure the agent found often takes longer than debugging a scripted test failure, and it's not yet a replacement for a deterministic regression suite that needs to reliably re-run the same checks every time.
Where agentic testing fits in a real QA stack
- As a discovery tool run periodically to surface issues a scripted suite doesn't cover — not as the primary regression gate.
- Alongside, not instead of, your deterministic suite — the two answer different questions ("does the expected flow still work" vs. "what unexpected things might break").
- With guardrails — scoping what the agent is allowed to do (read-only exploration vs. actions with real side effects) matters a lot once it's pointed at anything beyond a sandbox.
Evaluating an AI testing tool honestly
Ask specifically: what happens when it's wrong? Does a self-healing tool tell you when it healed onto a different element, or silently pass? Does a generated test come with a confidence signal, or do you have to manually verify every line regardless? The tools that are honest about their failure modes are usually the ones worth adopting; the ones that only market their success cases usually aren't ready for a production regression suite yet.