MongoDB / Advanced: distributed and recoverable operations / Lesson 17

Windows, change streams and maintained results

Compute per-document analytics and design resumable change processing without promising automatic exactly-once external effects. Bound pipelines and derived-result maintenance.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Use explicit partitions, order and document windows.
Explain resume tokens, delivery identity and derived-result lifecycle.

A window adds context without grouping away detail

$setWindowFields, available from MongoDB 5.0, can calculate values over related documents while preserving detail. Partitioning defines independent groups, sortBy defines analytic order and the window specifies the input neighborhood. Use stable event identifiers when timestamps tie. A final $sort is still appropriate when the displayed order must be guaranteed; do not assume every transformation preserves presentation order. Window calculations can require substantial processing even when the final report is small, so bound the source population and inspect execution evidence.

Calculate a reasoned cumulative total

The collectionless fixture uses three numeric amounts and a unique sequence. The document window from unbounded to current produces five, thirteen and fifteen. Each original sequence remains visible, unlike a $group that collapses the input. The literals use $documents, requiring MongoDB 5.1 or later, and the baseline examples target 8.0-compatible operators. No persistent collection or observed timing is involved. Missing, nonnumeric and duplicate-order values need a separate business rule before applying the pattern to a financial stream.

javascript Illustrative mongosh; results are reasoned, not measured.

db.getSiblingDB('quay_lab').aggregate([
  { $documents: [{ seq: 1, amount: 5 }, { seq: 2, amount: 8 }, { seq: 3, amount: 2 }] },
  { $setWindowFields: { sortBy: { seq: 1 }, output: {
    running: { $sum: '$amount', window: { documents: ['unbounded', 'current'] } }
  } } },
  { $project: { _id: 0, seq: 1, running: 1 } },
  { $sort: { seq: 1 } }
]);

Expected output

{ seq: 1, running: 5 }
{ seq: 2, running: 13 }
{ seq: 3, running: 15 }

The explicit window is cumulative over ordered documents, not a wall-clock duration.

A stream needs retention and consumer state

Change streams observe supported changes on replica sets and sharded clusters, not an arbitrary standalone deployment. Consumers receive resume tokens and must retain them according to a deliberate processing/checkpoint protocol. History is finite: if the needed history is no longer available, resuming can fail and require a documented rebuild. A resume token is not permission to ignore business idempotency. Delivery can be repeated around failures, and a crash between external effect and checkpoint can duplicate that effect without an explicit deduplication design.

Keep a stream consumer bounded and explicit

The Node.js fragment assumes a compatible installed driver and an existing synthetic collection handle. It opens an insert-only stream, processes at most one available event and closes it in finally; waiting for an event can still block, so a real service needs a shutdown and deadline policy. The placeholder function represents application-defined durable processing, not an implemented exactly-once guarantee. This snippet is non-runnable and no stream, connection or external effect is started during authoring. Driver and server compatibility must be checked together.

javascript Illustrative Node.js driver fragment; prerequisites are stated in the explanation.

// Driver fragment: collection and processIdempotently are supplied by the application.
const stream = collection.watch([{ $match: { operationType: 'insert' } }]);
try {
  for await (const event of stream) {
    await processIdempotently(event._id, event.fullDocument);
    break;
  }
} finally { await stream.close(); }

A real implementation must durably coordinate event identity, effects and checkpointing. The function name alone proves none of those guarantees.

Materialization and specialized features have contracts

$merge and $out maintain stored results and therefore change a pipeline from observation into a write workflow with privileges, uniqueness and recovery requirements. $facet has its own memory/output limits and cannot treat allowDiskUse as unlimited spilling for its facet state. Search and vector stages require supported deployment features and indexes; their availability differs across current products, versions and tiers, so do not assume every Community or Atlas deployment has the same feature. Keep source authority, refresh lag, rebuild procedure and consumer compatibility explicit for every derived collection.

Common errors and fixes

A resumed consumer repeats an external action

Why it happens: Checkpointing was mistaken for coordinated idempotent effect delivery.
How to fix it: Persist event identity and design effect/checkpoint recovery explicitly.
Prevent it next time: Test crashes on both sides of checkpoint persistence.

ChangeStreamHistoryLost

Why it happens: The required resume history is no longer available.
How to fix it: Use the documented rebuild or reseed procedure from a consistent source.
Prevent it next time: Monitor consumer lag against retained history and rehearse rebuilds.

allowDiskUse does not solve a facet limit

Why it happens: Stage-specific memory and result limits were ignored.
How to fix it: Bound or redesign the facet and inspect the exact stage constraints.
Prevent it next time: Review limits per stage rather than assuming one global escape option.

Check your understanding

1. A consumer sends an email, crashes before saving its resume token, then resumes. What can happen?

Show worked solution

The same event can be processed again and send another email. Resume capability does not atomically coordinate an external service with MongoDB consumer state. Use a durable event identity and an idempotent delivery or outbox protocol, then test the failure window. A helper named processIdempotently is only a requirement until its storage and delivery behavior are implemented and verified.

2. Why is a cumulative document window different from a rolling last-hour metric?