Linux performance / Intermediate: mechanism and scoped diagnosis / Lesson 11
Read effective cgroup and systemd budgets
Explain how hierarchical resource policy can create latency while the host still appears to have spare capacity.
On this page
Lesson goals
- Distinguish CPU weights, quotas, cpusets, and task limits.
- Inspect effective limits and ancestor constraints.
- Plan a policy experiment through the owning manager.
A service’s capacity can differ from the host’s
Cgroup v2 organizes resource control hierarchically, and systemd maps unit policy onto that hierarchy where supported. A service can be CPU-throttled or memory-constrained while the host retains idle CPUs or available RAM. Inspect the service’s actual cgroup and its ancestors rather than relying on a host dashboard. A container can expose a delegated subtree that hides some outer policy. Record the manager context, unit name, kernel, systemd version, and controller availability. Directly editing a manager-owned control file can conflict with its lifecycle and is not an appropriate diagnostic shortcut.
Weight, quota, and placement are different
CPUWeight expresses relative share among competing siblings under applicable controller behavior. It does not reserve a fixed fraction when no competitor exists. CPUQuota expresses a time budget, and usage can reach that budget before a period ends, creating throttled intervals. AllowedCPUs or cpusets constrain placement rather than promising uninterrupted execution. A quota equivalent to several CPUs is not the same as several dedicated cores. Compare cpu.stat event deltas, cpu.max, effective allowed CPUs, and request timing before interpreting a CPU percentage as the complete availability story.
systemctl show example-report.service -p ControlGroup -p CPUWeight -p CPUQuotaPerSecUSec -p AllowedCPUs -p TasksMax
systemctl cat example-report.serviceThis non-runnable diagnostic targets an existing prepared lab unit. Properties vary by systemd release and context; no limit is changed and no service is restarted.
Resource controls interact across the request path
MemoryHigh can introduce reclaim pressure, MemoryMax can enforce a final boundary, and TasksMax includes threads as well as processes. I/O weights and device limits depend on controller and storage support and should be interpreted at the relevant device layer. Restricting a background writer may protect foreground latency, but it can also grow a persistent queue elsewhere. Read effective settings together with observed pressure and queue age. Do not assume a resource unit’s configuration fragment proves the kernel applied every requested controller on every filesystem or execution context.
Find the effective enclosing policy
A child cannot escape a stricter ancestor by displaying an unlimited local value. Walk the authorized hierarchy and identify where policy is owned, but avoid collecting unrelated tenant details. Event counters can be hierarchical, so distinguish local and aggregate observations using the documented file semantics. Compare the same cgroup lifetime: unit recreation or container replacement can reset the counters. A normal host load average does not rule out scoped throttling. Preserve unit-to-cgroup mapping with timestamps so a later process move does not invalidate the investigation.
cat /sys/fs/cgroup/lab-report/cpu.max
cat /sys/fs/cgroup/lab-report/cpu.stat
cat /sys/fs/cgroup/lab-report/cpuset.cpus.effectivelab-report is a fictional prepared path. Correlate usage and throttling deltas with latency and inspect the parent policy separately; these reads do not move tasks or write controller settings.
Treat a budget change as a shared-capacity experiment
A justified candidate might change admission or a service-specific budget in a disposable system. Predict which event rate and queue should change, keep the workload fixed, and monitor siblings and host reserve. Removing a quota can improve one service while harming another, so acceptance must include the shared objective and error budget. Use the supported manager configuration path and a reviewed rollback for any later authorized action. No systemctl set-property, direct cgroup write, or host service mutation is executed in this course; the lesson teaches evidence and policy design.
Common errors and fixes
Unlimited child is assumed unconstrained
- Why it happens
- Ancestor policy was omitted.
- How to fix it
- Inspect the effective hierarchy and manager context.
- Prevent it next time
- Record enclosing policy with local values.
CPU weight is treated as a hard quota
- Why it happens
- Relative competition and time budgets were conflated.
- How to fix it
- Read weight, quota, and placement separately.
- Prevent it next time
- Describe each control’s intended mechanism.
Manual cgroup edits fight the manager
- Why it happens
- Two owners changed the same policy boundary.
- How to fix it
- Use the documented manager interface in the lab.
- Prevent it next time
- Define ownership and delegation before tuning.
Check your understanding
1. A service has high throttling deltas while the host is half idle. Is this contradictory?
Show worked solution
No. A scoped quota or placement boundary can restrict the service independently of total host capacity. Inspect its time budget and ancestors, correlate the request interval, and test a justified policy or admission change only on an isolated copy.
2. Why should a successful budget increase include sibling-service checks?
Show worked solution
The service shares finite resources. Its gain can consume time or memory previously available to other workloads. Acceptance needs their latency, errors, and recovery reserve as well as the target service’s improvement.
Key takeaways
- Effective capacity is hierarchical and workload-specific.
- Use manager-owned policy and verify shared impact.