CI Integration & Flaky Tests

Why run tests in CI?#

Tests that only run on a tester's laptop protect nobody. Continuous Integration (CI) runs your suite automatically on every pull request, so a change that breaks login is caught before it's merged — not two weeks later in regression.

Running Selenium tests in GitHub Actions#

GitHub-hosted Ubuntu runners already have Chrome installed, and Selenium Manager finds the matching driver, so the workflow is short:

# .github/workflows/ui-tests.yml
name: UI tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  selenium:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install selenium pytest pytest-html

      - name: Run tests
        run: pytest -v --html=report.html --self-contained-html

      - name: Upload report
        if: always()                      # upload even when tests fail
        uses: actions/upload-artifact@v4
        with:
          name: test-report
          path: report.html

Two things make browser tests CI-friendly:

• Headless mode — options.add_argument("--headless=new") in your driver fixture.
• A fixed window size — --window-size=1366,768, so responsive layouts behave the same everywhere.

Screenshots on failure#

A failing CI test without evidence is hard to debug. A pytest hook can save a screenshot automatically whenever a test fails:

# conftest.py (add to the existing fixtures)
from pathlib import Path

import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call" and report.failed and "driver" in item.fixturenames:
        Path("screenshots").mkdir(exist_ok=True)
        item.funcargs["driver"].save_screenshot(f"screenshots/{item.name}.png")

Upload the screenshots/ folder as an artifact the same way as the report.

Speeding up the suite#

• Run in parallel with pytest-xdist: pytest -n 4 runs four browsers at once. Tests must be independent — no shared accounts or ordering assumptions.
• Tag and split: mark smoke tests with @pytest.mark.smoke and run pytest -m smoke on every PR, the full suite nightly.
• Set up data through the API, not the UI. Creating a user with one HTTP call is far faster than clicking through registration in every test.

Flaky tests#

A flaky test passes and fails on the same code. Flaky tests are worse than no tests: people stop trusting red builds and start ignoring real failures.

Common causes and fixes#

Finding flaky tests#

Run a suspect test many times:

pip install pytest-repeat
pytest test_checkout.py -v --count=20 -x     # stop at the first failure

If it fails 1 time in 20, it's flaky — investigate before trusting it.

Retries: a safety net, not a fix#

pytest-rerunfailures can retry failed tests:

pip install pytest-rerunfailures
pytest --reruns 2 --reruns-delay 1

Retries hide the symptom. Use them sparingly, track which tests needed a retry, and fix the root cause. A good team policy: a flaky test is quarantined (skipped with a ticket) within a day, and fixed within a sprint.

A healthy pipeline#

Pull request  → lint + unit tests + API tests + UI smoke (≈ 5–10 min)
Merge to main → full UI regression in parallel (≈ 15–20 min)
Nightly       → cross-browser run + performance checks
Deploy        → production smoke test

Fast feedback on every change, complete coverage before release. Test yourself with the test automation interview questions.