AI in Software Testing Introduction
Where AI genuinely helps QA teams today, where it does not, and how to use it without losing test quality.
AI and software testing: two different topics
"AI in testing" means two things, and this track covers both:
- Using AI to test software faster — generating test cases and data, writing automation code, healing broken locators, triaging failures.
- Testing software that uses AI — chatbots, summarisers, RAG search and agents, whose outputs are non-deterministic and can't be checked with a simple
assertEqual.
Both are now part of a QA engineer's job.
Where AI genuinely helps QA teams
| Task | How AI helps | Lesson |
|---|---|---|
| Test design | Turns a user story into a first draft of scenarios, edge cases and negative tests | Generating Test Cases with LLMs |
| Test data | Produces realistic, varied data (names, addresses, edge-case strings) on demand | Same lesson |
| Automation code | Writes Playwright, Cypress or Selenium tests and page objects from a description | AI-Assisted Test Automation |
| Maintenance | "Self-healing" locators find the element when a selector changes | Same lesson |
| Visual checks | Compares screenshots and ignores irrelevant pixel noise | Same lesson |
| Failure triage | Summarises long logs, groups similar failures, suggests likely causes | Same lesson |
| Bug reports | Drafts clear reproduction steps from notes or a recording | — |
Where AI does not replace testers
- Knowing what matters. An LLM doesn't know your users, your business risks or last quarter's production incident.
- Judging correctness. AI-generated tests can assert the current (buggy) behaviour as correct.
- Exploratory testing. Curiosity and domain intuition still find the most valuable bugs.
- Accountability. Someone must decide a release is safe.
Treat AI output like a junior colleague's first draft: useful, fast, and always reviewed.
The main risks
- Hallucinated requirements. The model invents rules ("passwords expire every 90 days") that nobody specified. Always trace generated tests back to real requirements.
- Plausible but wrong assertions. Generated code can pass while checking the wrong thing. Make a generated test fail on purpose once to prove it can.
- Data privacy. Never paste customer data, credentials or proprietary code into tools your company hasn't approved. Use synthetic data.
- Over-trusting "self-healing". A locator that "heals" onto the wrong button turns a real bug into a passing test.
- Cost and speed. Calling a model inside every test run is slow and expensive; use AI to create tests, then run plain, deterministic tests in CI.
A sensible way to adopt AI in QA
- Start with drafting: test ideas, test data, first versions of automation code — reviewed by a person.
- Add triage: summarising failed CI runs and grouping duplicates.
- Measure: did review time drop? Did coverage of edge cases improve? Did escaped defects fall?
- Keep the CI suite deterministic. AI helps write and maintain tests; the tests themselves should give the same answer every time.
Testing AI-powered features
When the product itself uses an LLM, traditional "expected output" tests break down — the same input can produce different (all acceptable) answers. You test properties instead: valid format, required facts present, no forbidden content, grounded in the source, resistant to prompt injection. The Testing LLM Applications lesson shows how, with runnable examples.
If you're new to how LLMs work, the LLM and Prompt Engineering tracks are useful background.