AI in testing
Two sides of the same coin: using AI to test faster, and testing products that contain AI. Benefits, limits and safe practices.
AI affects QA in two ways. First, AI assistants and agents help testers work faster. Second, more and more products contain AI features themselves, and those need new testing methods. Both require the same mindset: use AI's speed, but keep humans responsible for decisions.
Using AI to test faster
| Task | How AI helps | What a human must check |
|---|---|---|
| Test ideas | Suggests scenarios and edge cases from a requirement | Relevance, missing business rules |
| Test data | Generates realistic names, addresses, texts in many languages | No real personal data, correct formats |
| Test cases and checklists | Drafts structured cases from user stories | Accuracy, duplicates, priorities |
| Automation code | Writes and refactors test scripts | Selectors, assertions, maintainability |
| Log and failure analysis | Summarises long logs, groups similar failures | Root cause, not just symptoms |
| Bug reports | Turns notes into a clean report | Facts, steps actually performed |
A prompt that works
Good results come from good context: the requirement, the product's rules, and the output format you need.
You are a senior QA engineer. Requirement:
“Users can apply one promo code per order. Codes have an expiry date
and a minimum order value. Promo codes can't be combined with sale items.”
Create a checklist of test conditions. Include boundary values, invalid
inputs, combinations with sale items and time zones. Output a table:
ID | condition | test data | expected result | priority.
Limits and risks
- Confident mistakes: AI can invent features, rules or API fields that don't exist; every output needs review
- Shallow coverage: generated tests often follow the happy path; experienced testers add the risky cases
- Privacy: never paste customer data, credentials or confidential code into tools that aren't approved by your company
- False productivity: a thousand generated test cases nobody runs or maintains are worse than fifty good ones
Human in the loop
We use AI as a fast junior colleague: it drafts, a human reviews and decides. AI agents can now run tools, browse the application and write code; they're powerful for repetitive tasks, but their actions need clear limits — test environments only, no access to production data, and review of every change before merge.
Testing products that contain AI
AI features need properties-based checks, evaluation sets, red-team attacks and monitoring in production instead of exact-match assertions. We describe the full process in How to test AI products.
| Classic feature | AI feature |
|---|---|
| Same input → same output | Same input → different, often acceptable outputs |
| Assert exact values | Score answers against a rubric |
| Fixed regression suite | Versioned evaluation set, re-run on every prompt or model change |
| Bugs are reproducible | Failures can be statistical; measure rates |
| Security: inputs and access | Plus prompt injection, data leakage, tool misuse |