AWS / Intermediate: application service contracts / Lesson 11
Monitoring, audit, and incident evidence
Connect CloudWatch signals, audit activity, application telemetry, and release context without treating any one dashboard as the complete truth.
On this page
Lesson goals
- Explain the assumptions, responsibilities, and trade-offs of monitoring, audit, and incident evidence.
- Interpret both examples and distinguish declared intent from verified evidence.
- Diagnose the listed failures and justify the worked decisions without using a real AWS account, credential, or service.
Start from the question being investigated
Monitoring should answer whether users receive the intended service and which boundary may be failing. Resource metrics, application logs, traces, and API audit records have different purposes and coverage. CloudWatch can collect and evaluate supported metrics and logs, while CloudTrail records supported account and service activity under its configured scope. A low CPU graph does not prove a healthy booking path, and an API audit event does not prove a payment completed. The first synthetic record links user impact, deployment revision, and a dependency hypothesis while keeping uncertainty visible. It contains no actual account activity.
impact: synthetic-booking-timeout-cohort
release: lab-build-42
observation: application-wait-time-increased
hypothesis: database-connection-budget-saturation
required_evidence: pool-wait-query-and-dependency-metrics
not_established: EC2-capacity-or-DNS-root-cause
The record preserves uncertainty and names the next evidence. It does not query CloudWatch, CloudTrail, a database, or a real incident system.
Define metrics and alarms with units
A metric needs namespace, dimensions, unit, aggregation, and time window. Percentiles and averages answer different latency questions, and missing data is not automatically zero demand or zero failure. Service metric availability and collection delay depend on the offering and configuration. An alarmโs evaluation and action policy must be reviewed separately from the metric. Avoid labeling every request ID or full URL as a metric dimension, which creates high cardinality and cost. Use bounded service and route categories, then correlate detailed events through safe logs or traces when needed. A green alarm does not validate an unmeasured user cohort.
Interpret CloudTrail and data-event scope
Management activity and data access have different logging configuration and coverage. Event history and a configured trail or event data store are not interchangeable archives, and data events may need explicit selection. Verify service support, Region, account, retention, delivery, and access policy before treating absence as proof that nothing happened. Audit records can reveal sensitive context and require protected storage and review access. A captured request can show who attempted an operation and its result at that boundary, but business reconciliation may still need database or downstream evidence. This lesson does not enable, query, or retrieve an audit log.
Use safe structured application context
Record approved event type, operation reference, release identity, error class, and duration rather than full credentials, request bodies, or environment dumps. The second synthetic JSON event illustrates bounded fields and a templated route. A correlation ID is diagnostic data, not an authenticated user identity. Distributed tracing can explain time across selected dependencies when supported instrumentation and sampling are configured, but missing traces can result from sampling or export failure. Keep telemetry queues, retention, cardinality, and access bounded so investigation does not create a second outage or a data-disclosure incident.
{"event":"operation_failed","service":"booking-api","route":"/bookings/:id","release":"lab42","operation_id":"lab-operation-a","duration_ms":900,"error_class":"DEPENDENCY_TIMEOUT"}
The event has fictional identifiers and no user payload or secret. Its timeout class is a diagnostic clue, not proof of a particular AWS service or permission failure.
Make evidence actionable and accountable
A useful incident record separates observations, hypotheses, actions, owners, and next verification. Align timestamps and exact scope before joining evidence across services. Define alert ownership, severity, response entry points, and what confirms recovery for the affected journey. Test alarm and redaction logic with synthetic records before adopting it in an authorized environment. No alarm, log query, trace export, account audit, metric API call, notification, or incident message is executed here. Local JSON parsing demonstrates structure only; it does not prove service delivery, audit completeness, or a measured production reliability result.
Common errors and fixes
Missing audit data is called no activity
- Why it happens
- Configured coverage and delivery were not verified.
- How to fix it
- Review event type, account, Region, selection, and retention.
- Prevent it next time
- Test expected audit visibility and safe access.
An average hides a failing cohort
- Why it happens
- Aggregation replaced the user-facing distribution.
- How to fix it
- Inspect relevant latency and error segments with denominators.
- Prevent it next time
- Define important journeys and bounded dimensions.
Debug logs expose credentials
- Why it happens
- Full request or environment was serialized.
- How to fix it
- Contain exposure and use allowlisted metadata.
- Prevent it next time
- Review and test redaction and retention.
Check your understanding
1. An operation has no CloudTrail data event. Can you conclude it never happened?
Show worked solution
No. Verify whether that event type and service were selected and delivered in the correct account, Region, and retention window. Correlate authorized application and downstream evidence. Audit absence can be a coverage gap rather than a business fact.
2. Why should a unique booking ID not become a latency-metric dimension?
Show worked solution
It creates a series per identifier and can increase cardinality, cost, and exposure. Use a bounded route or operation category for aggregation and approved diagnostic correlation for individual investigations, with retention and access appropriate to the data.
Key takeaways
- Monitoring should answer whether users receive the intended service and which boundary may be failing.
- Management activity and data access have different logging configuration and coverage.
- A useful incident record separates observations, hypotheses, actions, owners, and next verification.