Quality Metrics & Reporting

Why measure quality?#

Metrics turn "the release feels risky" into something a team can discuss and act on. Good metrics answer three questions: Are we testing enough? Are we finding problems early? Is quality improving over time? Bad metrics get gamed — so always pair a number with context.

Core QA metrics#

Test execution metrics#

• Pass rate = passed tests ÷ executed tests × 100.
• Execution progress = executed ÷ planned × 100.
• Blocked tests — tests that couldn't run (environment down, blocking bug). A rising number is an early warning.

Defect metrics#

• Defect density = defects ÷ size (per 1,000 lines of code, per feature or per story point). Highlights the modules that need attention.
• Defect leakage = defects found in production ÷ total defects found × 100. The single best measure of how well testing protects users.
• Defect removal efficiency (DRE) = defects found before release ÷ (before + after release) × 100. The inverse view of leakage.
• Defect age / time to fix — how long defects stay open, by severity.
• Reopen rate — fixed defects that come back. High numbers suggest rushed fixes or unclear bug reports.

Coverage metrics#

• Requirement coverage = requirements with at least one test ÷ total requirements.
• Code coverage = lines (or branches) executed by tests ÷ total. Useful for finding untested code; misleading as a target — 90% coverage with weak assertions proves little.

Delivery and reliability metrics (DORA)#

Many teams also track the four DORA metrics: deployment frequency, lead time for changes, change failure rate and mean time to restore (MTTR). They show whether quality practices let the team release safely and often.

Calculating metrics from real data#

A short Python script that computes the key numbers from a release's defect export:

# release_metrics.py — run with: python release_metrics.py
defects = [
    {"id": "BUG-101", "severity": "High",     "found_in": "QA"},
    {"id": "BUG-102", "severity": "Medium",   "found_in": "QA"},
    {"id": "BUG-103", "severity": "Low",      "found_in": "QA"},
    {"id": "BUG-104", "severity": "Critical", "found_in": "Production"},
    {"id": "BUG-105", "severity": "Medium",   "found_in": "QA"},
]
executed, passed, kloc = 240, 228, 18.5

found_before = sum(1 for d in defects if d["found_in"] != "Production")
found_after = len(defects) - found_before

print(f"Pass rate:        {passed / executed:.1%}")
print(f"Defect density:   {len(defects) / kloc:.2f} defects per KLOC")
print(f"Defect leakage:   {found_after / len(defects):.1%}")
print(f"DRE:              {found_before / len(defects):.1%}")

Output:

Pass rate:        95.0%
Defect density:   0.27 defects per KLOC
Defect leakage:   20.0%
DRE:              80.0%

A 20% leakage with a Critical bug in production says more about release readiness than a 95% pass rate.

Writing a test summary report#

A test summary report tells stakeholders whether to release. Keep it short and lead with the decision:

Release 4.2 — Test Summary

Recommendation: GO, with one known issue (BUG-88, Medium, workaround documented).

Scope tested:   Checkout redesign, discount codes, Apple Pay.
Execution:      240 / 250 planned tests run (96%); 228 passed (95%).
                10 not run: Apple Pay on Android (device unavailable) — covered by mocks.
Defects:        31 found, 30 fixed and verified, 1 open (Medium).
Regression:     Automated suite green on staging (412 tests, 0 failures).
Risks:          Apple Pay real-device coverage limited → monitor payment errors for 48h.

Metric anti-patterns#

• Counting test cases as productivity. Encourages many tiny, low-value tests.
• Rewarding bug counts. Encourages duplicate or trivial bugs and conflict with developers.
• 100% code coverage as a target. Teams write assertion-free tests to hit the number.
• One number in isolation. A falling defect count might mean better quality — or less testing.

Use metrics to ask better questions, not to rank people. Next, test yourself with the SQA interview questions.