DevOps QA / Basic: delivery evidence and suite design / Lesson 2

Risk-based CI strategy and gate placement

Choose checks by failure impact, detection strength, and feedback cost instead of accumulating an undifferentiated test list.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Map changed behavior to risk and suitable evidence.
Place fast checks and deeper gates along a delivery path.

Describe a failure in user terms

For each change, describe an unwanted outcome, its affected users, likelihood drivers, detectability, and recovery difficulty. A reservation algorithm that oversells seats differs from a cosmetic spacing change even when both touch one file. Numerical risk scores can support discussion, but multiplying subjective numbers does not make the result objective. Use categories with examples and review unusual cases. The goal is to select evidence that would reveal a plausible harmful failure, not to assign a decorative risk label after the pipeline is already designed.

Map risks to checks with boundaries

A unit test can check a capacity invariant cheaply, an API test can check request validation, and a concurrent integration test can expose race behavior in the real persistence path. A browser journey verifies whether users can complete a booking through the visible interface. No layer automatically replaces the others. Record the blind spot of each selected check. Coverage reports reveal executed code regions under the instrumentation rules; they do not show whether assertions are meaningful or missing business scenarios were considered. Keep risk coverage separate from line coverage.

json A teaching policy for a high-impact change

{
  "changeClass": "capacity-rule",
  "requiredEvidence": ["unit-boundary", "api-validation", "concurrent-store"],
  "advisoryEvidence": ["browser-layout"],
  "owner": "booking-maintainers",
  "missingEvidenceDecision": "hold"
}

This JSON is an application-defined policy record, not a built-in feature of a CI provider. A trusted gate implementation must evaluate it. The owner is a synthetic team label; the record shows that absent required evidence blocks promotion instead of silently disappearing.

Optimize the sequence of learning

Run cheap deterministic checks early so obvious defects receive rapid feedback. Deeper integration, browser, accessibility, and performance checks belong where their environments and costs are justified. Some can run in parallel, while others depend on a built artifact. Design a dependency graph rather than a single queue of every test. A slow suite that repeatedly finds critical integration failures may be valuable; a fast redundant suite may not be. Measure time to actionable diagnosis, including queue and investigation time, instead of optimizing test execution duration alone.

Treat selection rules as tested software

Path filters and change classification can shorten feedback, but they are executable policy with their own failure modes. A shared dependency or migration may affect behavior outside the files selected by a naïve filter. A required check that never runs can also interact badly with branch protection. Keep a stable gate entry point that evaluates expected evidence and selection reasons. Periodic broader runs can reveal gaps, but they do not retroactively prove an earlier candidate was safe. Review selection changes with the same care as the tests they control.

python A deliberately narrow change-selection function

def required_suites(changed_paths):
    suites = {"unit"}
    if any(path.startswith(("api/", "shared/", "migrations/")) for path in changed_paths):
        suites.update({"api", "component"})
    if any(path.startswith(("web/", "shared/")) for path in changed_paths):
        suites.add("browser-smoke")
    return sorted(suites)

The repository must define and test these path conventions, including renames and generated inputs. An empty or incomplete diff must not be assumed safe. This function is an excerpt of policy evaluation, not a complete dependency-aware selection engine.

Make exceptions bounded and visible

A gate exception needs a specific risk statement, compensating evidence, accountable owner, expiration, and recovery plan. It should not become a permanent continue-on-error setting. Separate advisory experiments from mandatory controls so teams can introduce a new check without pretending its signal is mature. Review escaped defects and false blocks to improve the strategy. If a threshold mostly rewards skipping hard tests, the incentive is wrong. Evaluate the strategy by prevented harm and useful learning, not by a rising percentage of green runs.

Common errors and fixes

A shared-library change skips affected suites

Why it happens: Path rules model folders rather than dependencies.
How to fix it: Expand selection for shared inputs and add regression cases.
Prevent it next time: Test selection rules on historical changes.

Coverage rises while booking defects escape

Why it happens: Execution coverage is used as a proxy for risk coverage.
How to fix it: Add assertions and scenarios tied to escaped failure modes.
Prevent it next time: Review behavior claims alongside coverage reports.

Temporary bypass remains indefinitely

Why it happens: The exception has no expiry or owner.
How to fix it: Create a bounded exception record and restore enforcement.
Prevent it next time: Alert on expired exceptions and review their rationale.

Check your understanding

1. A database migration changes no API files. Should an api-folder filter skip compatibility tests?

Show worked solution

No automatic conclusion follows from the path alone. The migration can change data shape and runtime behavior. Require migration and relevant API/component evidence, and teach the selector that schema inputs affect those suites. Verify upgrade behavior against a representative old schema.

2. A new performance check has unstable baselines. Decide its initial gate status.