Observability / Advanced: bias, trust, and evidence engineering / Lesson 15
Telemetry cost, retention, and pipeline capacity
Estimate telemetry volume with explicit assumptions and review which evidence to retain, aggregate, or discard without silently changing its diagnostic purpose.
On this page
Lesson goals
- Model event volume and retention separately.
- Compare cost controls with evidence loss.
- Recognize stateful collection and scaling constraints.
Estimate before changing the pipeline
Telemetry cost depends on event or sample rate, record size, dimensions, replication, indexing, query activity, and retention. A raw byte estimate is a useful starting point but not a bill. Compression and backend overhead can change the stored size, while high-cardinality indexing can dominate cost even when raw payloads are small. Record assumptions and compare them with approved observations before making a capacity claim. The offline model calculates uncompressed daily bytes for a fixed synthetic log rate. It performs no benchmark, billing query, traffic generation, or storage measurement.
events_per_second = 200
bytes_per_event = 500
seconds_per_day = 24 * 60 * 60
daily_bytes = events_per_second * bytes_per_event * seconds_per_day
print("daily_bytes", daily_bytes)
print("daily_GB_decimal", f"{daily_bytes / 1_000_000_000:.2f}")
print("seven_day_GB_decimal", f"{7 * daily_bytes / 1_000_000_000:.2f}")
daily_bytes 8640000000
daily_GB_decimal 8.64
seven_day_GB_decimal 60.48The estimate is 8.64 decimal GB per day before compression, indexing, replication, and other overhead. It is not a provider price or measured capacity.
Retain evidence for a declared use
Different evidence may need different retention because debugging a recent release differs from reviewing a seasonal trend. Aggregated metrics can support longer-term trends while detailed events remain available for a shorter approved period. Legal, privacy, and organizational requirements must be reviewed through the proper process rather than inferred from a tutorial. A backup or secondary export can retain data after the primary backend expires it, so inventory the complete lifecycle. Expiration should not be described as deletion from every copy unless the system and policy actually establish that claim.
Choose controls according to lost information
Aggregation, dimension removal, sampling, filtering, and shorter retention reduce different kinds of evidence. Downsampling can hide short spikes; dropping an attribute can merge distinct populations; outcome-biased sampling can invalidate raw rate interpretation. Start with unnecessary or unsafe fields, duplicate instrumentation, and unused high-cost data, but verify the effect on real diagnostic questions. The editorial policy below requires an owner and a named question for retained evidence. It is not a universal retention schedule or a command to delete data. Consumers need notice when a cost change alters their evidence contract.
retention_review: parcel-observability
retained_question: recent-failure-diagnosis-and-reviewed-trends
controls: [remove-unsafe-fields, bound-dimensions, remove-duplicates]
changes_require: [consumer-impact-review, policy-owner, verification]
copy_inventory: [primary-store, secondary-export, backup-policy]
claim_limit: expiry-in-one-store-is-not-global-deletion
The record documents decisions without selecting durations or modifying data. Actual retention and deletion need the organization’s authorized policy.
Scale the constrained stage and preserve state
Adding collector replicas does not necessarily increase useful throughput if an exporter destination, shared quota, or storage path is the bottleneck. Stateful processing such as certain sampling or metric aggregation designs can require consistent routing and careful state handling. Scraping the same target independently from multiple collectors can create duplicate observations unless the architecture accounts for ownership. Review agent and gateway patterns according to fault isolation, resource use, configuration ownership, and transport. Do not assume a load balancer makes all processors stateless or that a persistent queue provides unlimited outage tolerance.
Verify usefulness after a cost change
Compare volume, dropped evidence, freshness, query usefulness, and incident coverage after an authorized controlled change. A lower bill accompanied by unusable failure evidence is not automatically a successful improvement. Track backlog and recovery behavior during destination unavailability using only approved isolated tests. Keep rollback or adjustment criteria and data-policy review explicit. This lesson executes a fixed volume calculation and no collector or backend. It supplies neither operational scaling settings nor proof of retention enforcement, deletion, resilience, or a financial saving for a real deployment.
Common errors and fixes
Raw bytes are presented as a final storage bill
- Why it happens
- Compression, indexing, replication, and pricing were omitted.
- How to fix it
- Label the estimate and include actual system factors.
- Prevent it next time
- Separate volume models from financial claims.
More replicas duplicate metric observations
- Why it happens
- Collection ownership or stateful routing was ignored.
- How to fix it
- Review the data model and distribute ownership deliberately.
- Prevent it next time
- Test uniqueness and state handling in the approved environment.
Primary expiry is called complete deletion
- Why it happens
- Secondary exports and backups were omitted.
- How to fix it
- Inventory copies and apply the authorized lifecycle policy.
- Prevent it next time
- Review retention and deletion claims across the full path.
Check your understanding
1. A model predicts ten GB daily. Does seven-day retention require exactly seventy GB?
Show worked solution
Only for the stated raw linear model. Actual storage includes compression, indexing, replication, metadata, deletion delay, and varying traffic. State decimal or binary units and obtain appropriate backend evidence before sizing. Do not present the arithmetic as a capacity guarantee or price estimate.
2. Which cost reduction should be considered before discarding all error traces?
Show worked solution
Review unsafe or unused fields, duplicate instrumentation, unbounded dimensions, unnecessary destinations, and retention purpose first. Then evaluate a sampling policy that preserves required diagnostic mechanisms with explicit bias limits. The choice depends on workload, privacy, and investigation needs, not a universal percentage.
Key takeaways
- Volume models require explicit units and overhead assumptions.
- Cost controls change which evidence survives.
- Scaling must preserve collection ownership and required state.