← QA Library
Chapter 8 / 9 · 7 min read

AI in testing

Two sides of the same coin: using AI to test faster, and testing products that contain AI. Benefits, limits and safe practices.

AI affects QA in two ways. First, AI assistants and agents help testers work faster. Second, more and more products contain AI features themselves, and those need new testing methods. Both require the same mindset: use AI's speed, but keep humans responsible for decisions.

Using AI to test faster

TaskHow AI helpsWhat a human must check
Test ideasSuggests scenarios and edge cases from a requirementRelevance, missing business rules
Test dataGenerates realistic names, addresses, texts in many languagesNo real personal data, correct formats
Test cases and checklistsDrafts structured cases from user storiesAccuracy, duplicates, priorities
Automation codeWrites and refactors test scriptsSelectors, assertions, maintainability
Log and failure analysisSummarises long logs, groups similar failuresRoot cause, not just symptoms
Bug reportsTurns notes into a clean reportFacts, steps actually performed

A prompt that works

Good results come from good context: the requirement, the product's rules, and the output format you need.

You are a senior QA engineer. Requirement:
“Users can apply one promo code per order. Codes have an expiry date
 and a minimum order value. Promo codes can't be combined with sale items.”

Create a checklist of test conditions. Include boundary values, invalid
inputs, combinations with sale items and time zones. Output a table:
ID | condition | test data | expected result | priority.

Limits and risks

  • Confident mistakes: AI can invent features, rules or API fields that don't exist; every output needs review
  • Shallow coverage: generated tests often follow the happy path; experienced testers add the risky cases
  • Privacy: never paste customer data, credentials or confidential code into tools that aren't approved by your company
  • False productivity: a thousand generated test cases nobody runs or maintains are worse than fifty good ones

Human in the loop

We use AI as a fast junior colleague: it drafts, a human reviews and decides. AI agents can now run tools, browse the application and write code; they're powerful for repetitive tasks, but their actions need clear limits — test environments only, no access to production data, and review of every change before merge.

Testing products that contain AI

AI features need properties-based checks, evaluation sets, red-team attacks and monitoring in production instead of exact-match assertions. We describe the full process in How to test AI products.

Classic featureAI feature
Same input → same outputSame input → different, often acceptable outputs
Assert exact valuesScore answers against a rubric
Fixed regression suiteVersioned evaluation set, re-run on every prompt or model change
Bugs are reproducibleFailures can be statistical; measure rates
Security: inputs and accessPlus prompt injection, data leakage, tool misuse
AI makes good testers faster. It doesn't make untested assumptions safe.