Linux performance / Intermediate: mechanism and scoped diagnosis / Lesson 8
Find scheduling and synchronization delay
Distinguish runnable waiting, blocked synchronization, oversubscription, and affinity constraints before changing scheduling policy.
On this page
Lesson goals
- Separate on-CPU execution from runnable and blocked time.
- Interpret context switches without assuming they are defects.
- Evaluate thread-pool and placement hypotheses safely.
Elapsed time includes several thread states
A request thread can execute on a CPU, remain runnable while waiting for a CPU, or sleep while awaiting data, a timer, or synchronization. These states need different remedies. A CPU profile primarily samples execution, so it may show little activity for a request delayed on a lock. A runnable queue suggests scheduling competition, but CPU quotas and affinity can narrow usable capacity. Write a state-based hypothesis before changing thread count: extra threads cannot make a serialized critical section parallel and may instead increase contention and memory demand.
Context switches are events to explain
Voluntary switches commonly follow a thread yielding or blocking, while involuntary switches reflect scheduler preemption under the applicable accounting. Their counts are not a direct penalty bill: frequent switching can be normal for an I/O-heavy workload and costly for another. Normalize by useful work and examine latency correlation. CPU migration can affect locality but is not automatically wrong; balancing can help share work. Tool versions and kernel configurations affect available scheduling fields. Preserve interval and lifetime boundaries so a cumulative switch total is not mistaken for current contention.
pidstat -w -t -p 4242 1 5
ps -L -p 4242 -o tid,psr,stat,ni,commThis diagnostic syntax requires sysstat in a prepared lab and a fictional target PID. The output is not fabricated; compare measured switches with request rates and thread states.
Fair scheduling is version-sensitive
Linux’s fair scheduler has evolved, including the transition toward EEVDF in newer kernels. An old CFS tuning article may not describe the installed implementation or supported interfaces. Nice values express relative scheduling priority for applicable policies, not a guaranteed response deadline or a reserved CPU. Real-time policies have different rules and can starve ordinary work if misused. Do not infer that selecting a more aggressive policy fixes a latency issue. First identify the scheduling class, effective cgroup policy, allowed CPUs, and whether the delayed work is actually runnable.
Find the owner and the critical section
Many waiting threads can indicate a single owner performing slow work under a lock. A futex wait is a mechanism, not the application-level reason for holding the lock. Correlate stacks, lock instrumentation where available, and the owner’s state. Logging, allocation, or remote calls inside a critical section can serialize unrelated requests. Avoid assuming every sleeping thread needs more CPU. In a disposable application experiment, shorten protected work or reduce contention while preserving correctness, then compare useful throughput and tail latency rather than only the number of blocked threads.
cat /proc/4242/task/4243/status
cat /proc/4242/task/4243/schedstatThe paths refer to a prepared lab PID/TID pair. schedstat availability and field interpretation depend on kernel accounting support; use the installed documentation and compare deltas rather than inventing units.
Tune concurrency as admission control
A pool size determines how much work can compete at once; it is not simply a request to use all cores. More workers can hide independent I/O waits until a dependency, lock, or memory boundary becomes limiting. Too few can leave useful capacity idle; too many can increase queues and timeouts. Test a small set of candidate sizes in an isolated model with the same arrival pattern and bounded queues. Keep a rollback and observe dependent services, because apparent local throughput gains can overload the next stage and worsen end-to-end latency.
Common errors and fixes
Adding threads to a serialized path
- Why it happens
- The lock owner remains the limiting stage.
- How to fix it
- Inspect owner work and reduce justified critical-section cost.
- Prevent it next time
- Measure blocked and executing states before pool changes.
Treating every context switch as waste
- Why it happens
- Necessary blocking was not distinguished from harmful competition.
- How to fix it
- Normalize switches and correlate user-visible effects.
- Prevent it next time
- Evaluate workload behavior rather than a universal switch target.
Copying scheduler knobs from another kernel
- Why it happens
- Implementation and supported policy differ.
- How to fix it
- Read installed kernel and scheduler documentation.
- Prevent it next time
- Record version and avoid undocumented host changes.
Check your understanding
1. Most workers wait on a lock while its owner blocks on storage. What is the next useful hypothesis?
Show worked solution
The critical section may include slow storage work and serialize requests. Inspect why that operation must hold the lock, then test a correctness-preserving redesign on a copy. Adding CPU priority to waiting threads does not remove the owner’s dependency.
2. Why might a larger pool improve local throughput but worsen client p99?
Show worked solution
It can increase simultaneous demand on a downstream queue or memory limit. Measure the whole request path and offered load, include rejection and timeouts, and compare dependency queues before accepting the local improvement.
Key takeaways
- Distinguish executing, runnable, and blocked work.
- Treat concurrency and scheduling as measured policies.