CI/CD / Integrity, coordination, and improvement / Lesson 18

Delivery measurement and improvement experiments

Use service-level measures to locate constraints and test improvements without rewarding gaming.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Interpret current delivery metrics with explicit event definitions.
Design one improvement experiment with throughput and stability guardrails.

Define events before computing rates

A delivery dashboard depends on what counts as a change, deployment, failure, recovery, and incident-driven rework. Parcel’s package upload is not necessarily production adoption, and a failed test is not automatically a failed production deployment. Write event definitions with service owners and preserve timestamps from authoritative records. Use consistent timezone and a declared observation window. Without this groundwork, two teams can compute numerically precise but incomparable rates because one counts builds while another counts user-visible adoption.

Use a balanced current model

DORA’s current guidance describes five software delivery performance measures: change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate. They combine throughput and instability perspectives and are best interpreted for a particular application or service in context. Older material often describes four measures or uses a broad MTTR label; state the definition and baseline being used instead of mixing generations. These measures guide improvement and conversation, not a universal ranking of individual developers.

State denominators and missing cases

The complete local model computes a synthetic change-fail proportion from ten production adoptions, two of which required immediate intervention under the declared definition. It also computes a mean recovery duration for those two failures. A mean can hide a long tail, and two observations are too few for a stable conclusion. Keep raw events and use appropriate distributions over a useful window. Do not count a missing recovery timestamp as zero minutes or exclude a long-running incident merely because it makes the chart less attractive.

python Calculate declared synthetic measures

deployments = 10
failed = 2
recovery_minutes = [12, 48]
print(f"change fail fraction: {failed / deployments:.0%}")
print(f"mean recovery minutes: {sum(recovery_minutes) / len(recovery_minutes):.0f}")

Expected output

change fail fraction: 20%
mean recovery minutes: 30

These invented observations illustrate arithmetic and denominators. They are not benchmarks, service results, or a claim that the mean alone satisfies DORA measurement guidance.

Change one constraint with guardrails

Suppose Parcel’s lead time is dominated by waiting for a scarce test runner. An experiment might add appropriately isolated capacity or reduce unnecessary serial barriers, while holding required evidence constant. State the hypothesis, comparison window, workload context, and stability guardrails before changing the process. If deployment frequency increases while failure and rework rise, investigate the combined outcome. Merely splitting one release into many empty deployments can improve a count without improving user value. Metrics should reveal constraints, not teach teams to manipulate denominators.

text A bounded improvement experiment

hypothesis: runner queueing dominates feedback
change: remove one unnecessary dependency barrier
observe: queue time, candidate lead time, failures, rework
retain: the same required release checks

This is an analysis plan with no infrastructure mutation. Compare sufficiently similar workloads and record confounders such as traffic, staffing, dependency incidents, and major product changes.

Keep human and service context

Use review discussions to connect numbers with concrete delayed changes and failed releases. A small service with rare deployments may need longer windows and qualitative evidence; a highly regulated product may have legitimate release timing constraints. Delivery speed also interacts with reliability objectives and the team’s capacity to respond. An error-budget policy can make those tradeoffs explicit under local governance, but copying another organization’s thresholds is not a substitute for defining user expectations. Improve the system around the team rather than using dashboards as personal scorecards.

Common errors and fixes

Build uploads are counted as production deployments.

Why it happens: The event definition does not match actual adoption.
How to fix it: Use authoritative production-adoption records and state exclusions.
Prevent it next time: Review metric definitions with service and operational owners.

Recovery durations with missing end times become zero.

Why it happens: Unknown incidents were silently converted into fast recovery.
How to fix it: Preserve incomplete cases and report their measurement limitation.
Prevent it next time: Validate timestamp completeness and retain raw event lineage.

A team increases deployment count with no-op releases.

Why it happens: A target rewards a proxy independently of user value and stability.
How to fix it: Review lead time, failure, rework, and real delivery outcomes together.
Prevent it next time: Use metrics for improvement experiments rather than individual quotas.

Check your understanding

1. Ten deployments include two rollbacks and one incident-driven hotfix. How should they be counted?

Show worked solution

Apply the documented local definitions consistently and retain the event relationships. DORA’s measures distinguish deployment failure and incident-driven rework, so do not erase recovery actions or invent a denominator after seeing the result. The exact classification requires the declared production-adoption and rework rules.

2. Lead time improves after changing runner capacity and test scope simultaneously. Can capacity alone receive credit?