Apache Kafka / Advanced: atomic processing and recovery design / Lesson 17
Capacity, partitions and storage budgets
Estimate bounded resource requirements and identify the assumptions that require real evidence.
On this page
Lesson goals
- Calculate rough retained replicated bytes from explicit units.
- Review partition and throughput decisions against skew and recovery cost.
Describe the workload before sizing
A useful workload envelope names records per second, serialized byte distribution, key skew, retention, replication and reader fan-out. Peak and sustained rates answer different questions. A system that handles a brief burst can still run out of disk during a prolonged backlog or lose recovery margin while replicas rebuild. Compression depends on actual batches and content, so an assumed ratio should be clearly labeled until measured. Include record headers and protocol overhead where relevant instead of treating a payload's character count as physical disk usage. No single partition count or broker size is correct for all workloads.
Keep a first estimate dimensionally correct
The arithmetic multiplies two hundred records per second by five hundred serialized bytes, one hour and three replicas. The result is a simple uncompressed payload-copy estimate of 1.08 billion bytes. It omits indexes, segments, metadata, filesystem overhead, compression variation and operational free-space margin, so it must not be used directly as a disk allocation. Decimal gigabytes and binary gibibytes are different units; label whichever is used. Retention is also subject to segment and cleanup behavior rather than an exact per-record deadline. Use the estimate to expose order of magnitude and missing assumptions, then gather authorized measurements.
records_per_s = 200
bytes_per_record = 500
retention_s = 3600
replicas = 3
payload_copy_bytes = records_per_s * bytes_per_record * retention_s * replicas
assert payload_copy_bytes == 1080000000
print(payload_copy_bytes)1080000000The result excludes overhead, compression and reserve; it is not a physical disk requirement.
Partitions buy parallelism at a cost
More partitions can increase potential consumer parallelism and distribute traffic, but they also add metadata, files, replication work and recovery coordination. They do not split one business key's required sequence automatically. A partition change can alter key routing, so capacity planning and ordering design are linked. Too many tiny partitions can create overhead without useful throughput; too few can constrain a hot workload. Evaluate expected task distribution and failure recovery, not merely the number of CPU cores. Managed products may impose additional quotas or billing boundaries, which require service-specific review rather than extrapolation from self-managed Kafka.
Include consumer fan-out in network reasoning
The second model takes a declared source byte rate and three independent full readers. It calculates the application-read traffic alone, then compares it with one reader. Replication traffic, request overhead, compression and retries remain outside the arithmetic. Three consumers in one group do not necessarily mean three full copies of every record; three independent groups reading all records do. This difference matters when adding analytics or replay readers. Capacity reserve should also cover one failed broker's redistributed traffic and replica catch-up under the chosen failure model, rather than assuming normal steady-state utilization is an adequate ceiling.
source_bytes_per_s = 100000
full_reader_groups = 3
read_bytes_per_s = source_bytes_per_s * full_reader_groups
assert read_bytes_per_s == 300000
print(read_bytes_per_s)300000Only application read payload is counted; replication and protocol costs remain excluded.
Measure the bottleneck without unsafe experimentation
An authorized capacity study would vary representative record sizes, key distributions and failure conditions while observing end-to-end latency, disk use, replication health and destination completion. This course performs none of those service or load operations. Its local calculations are bounded arithmetic over synthetic inputs. Operational decisions should preserve rollback and compare like-for-like intervals after any change. Storage growth, network saturation, CPU compression cost and slow external consumers call for different remedies. Avoid universal tuning prescriptions: identify the limiting resource, the user consequence and the recovery margin before proposing more partitions, larger batches or longer retention.
Common errors and fixes
Estimated payload bytes are allocated as the entire disk capacity.
- Why it happens
- Overhead and recovery reserve were omitted.
- How to fix it
- Measure physical usage and budget explicit operational margin.
- Prevent it next time
- Label estimates and list excluded costs.
Three group members are counted as three complete readers.
- Why it happens
- Worker parallelism was confused with independent group fan-out.
- How to fix it
- Count independent full-reading groups and actual subscription behavior.
- Prevent it next time
- Record group topology in network estimates.
A normal-capacity design collapses during replica recovery.
- Why it happens
- Failure traffic and catch-up were excluded from the envelope.
- How to fix it
- Budget the selected degraded scenario and recovery objective.
- Prevent it next time
- Review steady state and recovery capacity separately.
Check your understanding
1. If retention doubles with all other model inputs fixed, what happens to the payload-copy estimate?
Show worked solution
It doubles from 1.08 billion to 2.16 billion bytes in this linear model. Physical storage need will not necessarily scale with exactly the same ratio because segment structure, compaction and overhead can vary. The calculation is useful for a first sensitivity check, followed by measurements and an explicit reserve budget.
2. Does adding ten partitions solve a single slow downstream API call for every record?
Show worked solution
Not necessarily. The dependency may remain the bottleneck or become more overloaded by added concurrency. Measure its sustainable capacity and the applicationโs wait budget, then review batching, caching or workload policy if appropriate. Partition changes also affect ordering and recovery, so they should follow an identified need rather than serve as a universal remedy.
Key takeaways
- Sizing is a set of explicit workload assumptions and measured limits.
- Include fan-out, skew and degraded recovery costs.