AI Testing Interview Questions
Interview questions on AI in QA and on testing LLM applications, with concise answers.
AI testing interview questions
Interviewers increasingly ask how you use AI as a tester and how you'd test AI features. Show enthusiasm and healthy scepticism.
1. How do you use AI in your testing work?
To draft test ideas and test data from requirements, scaffold automation code and page objects, summarise logs and group CI failures — always reviewing the output, and never pasting confidential data into unapproved tools.
2. What are the risks of AI-generated test cases?
Invented requirements, plausible but wrong expected results, duplicates, missing context-specific risks and wrong priorities. Every generated case must trace back to a real requirement and be reviewed.
3. How do you verify an AI-generated automated test is correct?
Read it, run it, and make it fail deliberately (change the expected value) to prove it can catch a bug. Check locators, missing awaits, sleeps and whether the assertion tests the actual requirement.
4. What are self-healing locators and what's the risk?
Tools that find an element by alternative attributes when the primary locator breaks. The risk is healing onto the wrong element and hiding a real regression, so heals must be reported and reviewed.
5. What is visual AI testing?
Comparing screenshots against a baseline in a way that ignores rendering noise but flags meaningful layout and content changes (e.g. Applitools), versus plain pixel diffs.
6. Why are LLM applications hard to test?
Outputs are non-deterministic and many answers can be correct, so exact-match assertions fail. You test properties, score quality over datasets and track trends.
7. How would you test a chatbot built on an LLM?
Unit-test the surrounding code with the model stubbed; build a golden dataset with property assertions (format, required facts, length, tone); use LLM-as-judge for faithfulness and helpfulness; run a red-team suite for prompt injection, data leakage and harmful content; monitor production feedback.
8. What is a golden dataset?
A curated set of representative and tricky inputs with the properties each output must satisfy, used to measure quality and catch regressions when prompts or models change.
9. What is LLM-as-judge? What are its limitations?
Using a model to grade outputs against a rubric. It's approximate — judges can be biased toward longer answers or their own style — so use clear rubrics, structured output and calibrate against human ratings.
10. What is prompt injection and how do you test for it?
Input that tries to override the system's instructions — directly ("ignore previous instructions") or indirectly via documents and web pages. Test with a suite of known attacks and assert the system refuses, doesn't leak its prompt and takes no unauthorised actions.
11. How do you test a RAG system?
Separately evaluate retrieval (did it fetch the right documents?) and generation (is the answer faithful to them?), plus answer relevance, citation accuracy and behaviour when no relevant document exists.
12. Should AI decide pass/fail in your CI pipeline?
Not for regular product tests — keep them deterministic so a red build means a real problem. AI evaluation belongs in scheduled eval runs for AI features, with results tracked as trends.
13. How do you generate test data with AI safely?
Use synthetic data only, ask for specific edge cases (scripts, lengths, languages, injection strings), and use libraries like Faker with a fixed seed for large, reproducible datasets.
14. What is MCP and how does it relate to testing?
The Model Context Protocol lets AI assistants use tools. With the Playwright MCP server, an assistant can drive a real browser to explore an app and generate tests from what it observes — still reviewed by a person.
15. How do you measure whether AI tooling improved your QA process?
Compare before and after: time to design and automate tests, review effort, edge-case coverage, escaped defects and flaky-test rates — not just the number of tests produced.