AI-Assisted Test Automation

AI as an automation pair-programmer#

Coding assistants (GitHub Copilot, Claude, Cursor and others) are now very good at writing test code: page objects, Playwright and Cypress specs, API tests and fixtures. Used well, they remove the boilerplate so you can focus on what to test.

Writing a Playwright test with an assistant#

Give the assistant the same context a new teammate would need: the page, the behaviour, the conventions.

Write a Playwright Test (TypeScript) spec for https://the-internet.herokuapp.com/login.
- Use role- and label-based locators (getByRole, getByLabel), no CSS classes.
- Put locators and actions in a LoginPage page object in pages/LoginPage.ts.
- Tests: valid login reaches /secure with the success message; wrong password
  shows "Your password is invalid!"; unknown user shows "Your username is invalid!".
- Use web-first assertions only (await expect(...)), no waitForTimeout.
- Read the password from process.env.TEST_PASSWORD with a fallback.

A typical result (after review):

// pages/LoginPage.ts
import { type Page, type Locator } from '@playwright/test';

export class LoginPage {
  readonly username: Locator;
  readonly password: Locator;
  readonly submit: Locator;
  readonly flash: Locator;

  constructor(readonly page: Page) {
    this.username = page.getByLabel('Username');
    this.password = page.getByLabel('Password');
    this.submit = page.getByRole('button', { name: 'Login' });
    this.flash = page.locator('#flash');
  }

  async goto() {
    await this.page.goto('https://the-internet.herokuapp.com/login');
  }

  async login(username: string, password: string) {
    await this.username.fill(username);
    await this.password.fill(password);
    await this.submit.click();
  }
}

// tests/login.spec.ts
import { test, expect } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';

const PASSWORD = process.env.TEST_PASSWORD ?? 'SuperSecretPassword!';

test.describe('Login', () => {
  let login: LoginPage;

  test.beforeEach(async ({ page }) => {
    login = new LoginPage(page);
    await login.goto();
  });

  test('valid user reaches the secure area', async ({ page }) => {
    await login.login('tomsmith', PASSWORD);
    await expect(page).toHaveURL(/\/secure$/);
    await expect(login.flash).toContainText('You logged into a secure area!');
  });

  test('wrong password is rejected', async () => {
    await login.login('tomsmith', 'wrong-password');
    await expect(login.flash).toContainText('Your password is invalid!');
  });

  test('unknown user is rejected', async () => {
    await login.login('nobody', PASSWORD);
    await expect(login.flash).toContainText('Your username is invalid!');
  });
});

Review generated tests like any code#

• Run it and make it fail. Change an expected message and confirm the test goes red. A test that can't fail is worthless.
• Check locators. Prefer roles and labels; reject brittle CSS chains and XPaths.
• Look for missing await on assertions — a classic generated-code bug.
• Remove sleeps (waitForTimeout) — replace with web-first assertions.
• Check the assertions test the requirement, not just "something is visible".

AI agents that drive a browser (MCP)#

The Model Context Protocol (MCP) lets AI assistants use tools. Microsoft's Playwright MCP server gives an assistant a real browser it can navigate, click and read — through the page's accessibility tree rather than screenshots.

Add it to an MCP-capable client (Claude Desktop, Claude Code, VS Code and others):

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

You can then ask: "Open the-internet.herokuapp.com/login, log in as tomsmith, and write a Playwright test that reproduces what you did." The assistant explores the real page and writes the test from what it actually saw — locators included. Recent Playwright releases also ship test agents (a planner, a generator and a healer) built on the same idea.

It's powerful for exploration and scaffolding. The tests it produces still go through the same review as any other code. Learn more about MCP in the MCP track.

Self-healing locators#

When a developer renames #submit-btn to #login-button, traditional tests fail. Self-healing tools record several attributes of each element (ID, text, position, neighbours) and, when the primary locator fails, pick the most similar element and continue — often suggesting an updated locator.

Examples: Healenium (open source, for Selenium), and commercial tools such as mabl, Testim and testRigor.

Use with care: a heal can land on the wrong element and hide a real regression. Good tools report every heal for human review instead of silently passing. Stable data-testid attributes and role-based locators prevent most breakages in the first place.

Visual testing#

Visual tests compare screenshots against an approved baseline:

// Playwright's built-in visual comparison (pixel-based, not AI)
await expect(page).toHaveScreenshot('checkout.png', { maxDiffPixelRatio: 0.01 });

Pixel comparison is sensitive to fonts, anti-aliasing and dynamic content. Visual AI tools (e.g. Applitools Eyes) compare pages the way a person would — ignoring rendering noise but catching real layout shifts, overlapping text and missing elements. Mask dynamic regions (dates, ads, avatars) in either approach.

AI for failure triage#

Large suites produce noisy failures. AI helps by:

• Summarising long CI logs into "what failed and the likely cause".
• Grouping failures with the same root cause (one broken API → 40 failed UI tests).
• Spotting flaky patterns ("this test fails only on WebKit, 1 in 15 runs").
• Drafting bug reports from the failure, trace and logs.

A practical setup: after a failed CI run, a script sends the failure messages and the relevant log excerpt to an LLM and posts a short summary to the pull request — while the test results themselves stay deterministic.

Keep the suite deterministic#

The golden rule: AI helps you write and maintain tests; it shouldn't decide pass/fail in your regular CI suite. Generated tests are reviewed, committed and run like any other code. The exception is when the product contains AI — covered next in Testing LLM Applications.