Linux Kernel cgroups v2 Under the Hood: A Step-by-Step Walkthrough
A comprehensive system engineer's guide to control groups v2, unified hierarchy architecture, CPU/Memory/IO resource controllers, Pressure Stall Information (PSI), OOM killer mechanics, and Kubernetes integration.
Every modern container runtime—from Docker and containerd to Kubernetes pods, systemd services, and cloud serverless engines—relies on the Linux kernel to enforce hardware resource boundaries. Yet, few developers understand the kernel mechanisms preventing one runaway process from crashing an entire multi-tenant host server.
When a containerized Java application, Go microservice, or Python ML worker consumes excess RAM, how does the kernel determine whether to reclaim page cache memory or invoke the dreaded **Out-Of-Memory (OOM) Killer**? Why did Linux transition from the legacy multi-hierarchy cgroups v1 model to the unified **cgroups v2** architecture? How do Pressure Stall Information (PSI) metrics quantify microsecond-level hardware starvation? How do `cpu.max`, `memory.high`, and `io.max` sysfs pseudo-filesystem nodes translate into Completely Fair Scheduler (CFS) bandwidth throttling and block layer IOPS throttling? In this deep dive, Professor Pixel breaks down Linux cgroups v2 from first principles: unified hierarchy design, resource controller mechanics, OOM score algorithms, production C++ and Python cgroup managers, performance benchmarks, and interactive visual simulators.
1. The Intuition: Why Resource Isolation Matters
1.1 The Multi-Tenant Noisy Neighbor Problem
Imagine an apartment building sharing a single main water pipe. If one tenant opens all their taps full blast to fill a swimming pool, water pressure drops to zero for every other resident in the building. In operating system architecture, this is the classic Noisy Neighbor Problem: an unconstrained background worker process executing an infinite loop or allocating gigabytes of physical RAM starves latency-sensitive API gateways sharing the same Linux kernel.
To solve this fundamental multi-tenancy challenge, the Linux kernel relies on two distinct, complementary building blocks:
- Linux Namespaces (Visibility Control): Restricts what a process can see. For example, the PID namespace hides processes running in other containers, the NET namespace isolates network interfaces and routing tables, and the MOUNT namespace isolates file system directory trees.
- Control Groups / cgroups (Resource Budget Control): Restricts what a process can use. For example, capping CPU execution time, limiting physical RAM allocations, throttling block I/O throughput, and restricting maximum process fork counts.
Together, Namespaces and cgroups form the foundational architecture of Linux containers. Namespaces provide the security illusion of running on a dedicated operating system, while cgroups enforce physical hardware budget limits across shared multi-tenant infrastructure!
Diagram 1: The dual foundation of Linux containers. Namespaces isolate process visibility, while cgroups v2 controllers enforce hardware boundaries.
Developer Pitfall — Confusing Namespaces with Resource Limits:
A common misconception among software engineers is believing that creating a Docker container automatically limits its CPU or memory usage. Linux namespaces only isolate process IDs, mount points, and network stacks—they place zero limits on CPU or memory consumption! Without explicit cgroup configurations (`--memory` or `resources.limits`), a single container process can consume 100% of host CPU cores and trigger host-wide physical OOM crashes!
2. Architectural Deep Dive: cgroups v1 vs cgroups v2 Unified Hierarchy
2.1 The Architectural Flaws of cgroups v1
First introduced by Google engineers in 2006 (originally named "Process Containers"), cgroups v1 allowed each resource subsystem (CPU, Memory, Block I/O, PIDs, Devices) to maintain its own independent pseudo-filesystem hierarchy under `/sys/fs/cgroup/`:
This orthogonal multi-hierarchy design created severe kernel architectural flaws that plagued enterprise Linux deployments for a decade:
- Priority Inversion in Writeback I/O: When a application process wrote dirty memory pages to disk, the memory controller tracked the RAM allocation under `memory/container_A`, but the background kernel writeback thread (`kworker`) flushing pages to disk ran in the root `blkio` cgroup. Block I/O throttling could not correlate dirty memory pages to the originating cgroup!
- Complex Process Membership & Lock Contention: A process could belong to `cpu/group1` but `memory/group2`, leading to impossible-to-debug kernel deadlock scenarios during resource reclamation and heavy RCU lock contention across controller hierarchies.
- Uncoordinated OOM Kills: The memory controller would trigger OOM kills without notifying the CPU or PID controllers, leaving orphaned processes running indefinitely in partial cgroup states.
2.2 The cgroups v2 Single Unified Hierarchy
Redesigned by Linux kernel developer Tejun Heo and merged in Linux 4.5, cgroups v2 enforces a single, unified tree structure rooted at `/sys/fs/cgroup/`. Every process in the system belongs to exactly one leaf cgroup node in the tree hierarchy!
2.3 The "No Internal Processes" Rule
A fundamental structural rule in cgroups v2 is the No Internal Processes Rule: a non-root cgroup node cannot contain both child cgroups AND active processes (`cgroup.procs`) simultaneously!
Processes can only reside in leaf nodes. This eliminates accounting ambiguity when parent cgroups distribute resource limits among child cgroups, guaranteeing clean hierarchical budgeting across systemd unit scopes!
Developer Pitfall — Mixing cgroups v1 and v2 Mode on Modern Linux Distributions:
Modern Enterprise Linux distributions (Ubuntu 22.04+, RHEL 9+, Debian 11+) default to cgroups v2 unified mode (`systemd.unified_cgroup_hierarchy=1`). If a legacy Docker daemon or container agent attempts to mount cgroups v1 filesystems alongside v2, systemd cgroup drivers will fail to track container limits, resulting in unthrottled CPU consumption or silent container startup failures!
3. Under the Hood: CPU, Memory, and I/O Controller Subsystems
3.1 CPU Controller (`cpu.max` & `cpu.weight`)
The CPU controller manages CPU execution time using the Completely Fair Scheduler (CFS) bandwidth control mechanism. It provides two distinct control knobs:
- Quota Throttling (`cpu.max`): Defines absolute hard execution limits using two parameters: `$Quota$` and `$Period$` in microseconds:
$$ \text{Allowed CPU Cores} = \frac{Quota}{Period} $$For example, writing
200000 100000tocpu.maxpermits processes in the cgroup to run for 200,000 microseconds ($200\text{ ms}$) within every 100,000 microsecond ($100\text{ ms}$) period—allocating exactly 2.0 CPU cores! - Proportional Shares (`cpu.weight`): Configures relative CPU priority shares between 1 and 10,000 (default: 100). When CPU demand exceeds 100% capacity, available CPU cycles are split proportionally based on `cpu.weight`.
3.2 Memory Controller (`memory.min`, `memory.low`, `memory.high`, `memory.max`)
cgroups v2 introduces a sophisticated 4-tier memory protection and limit hierarchy, replacing the simplistic single memory limit of v1:
| Control File | Protection Level | Kernel Behavior When Exceeded |
|---|---|---|
| memory.min | Hard Protection | Memory below this threshold is never reclaimed by the kernel, guaranteeing dedicated RAM for critical services. |
| memory.low | Soft Protection | Memory below this threshold is reclaimed only if no un-protected cgroup memory can be reclaimed. |
| memory.high | Throttle Limit | Exceeding this limit triggers aggressive background page reclaim and throttles allocating processes. Does NOT trigger OOM killer! |
| memory.max | Absolute Limit | Hard RAM limit. If reclaim fails to reduce usage below `memory.max`, the OOM Killer is invoked to terminate processes! |
3.3 Detailed Step-by-Step Numerical Walkthrough: CFS Quota Refill Cycle
Let us trace how the Linux CFS scheduler handles a cgroup limited to 1.5 CPU cores (`cpu.max = 150000 100000`):
- Period Start ($t = 0\text{ms}$): The kernel timer initializes `cfs_rq->runtime = 150,000\,\mu s`. Four threads belonging to the cgroup begin executing simultaneously across 4 physical CPU cores.
- Execution Phase ($t = 37.5\text{ms}$): Since 4 threads run concurrently for $37.5\text{ ms}$, total accumulated CPU runtime reaches $4 \times 37,500\,\mu s = 150,000\,\mu s$. The quota is completely exhausted!
- Throttling Phase ($t = 37.5\text{ms} \to 100\text{ms}$): The kernel deschedules all 4 threads and places the cgroup on the `throttled_list`. For the remaining $62.5\text{ ms}$ of the period, the threads cannot execute, causing a latency spike!
- Quota Refill ($t = 100\text{ms}$): The timer fires, refilling `runtime` back to $150,000\,\mu s$. The 4 threads are unthrottled and resumed.
3.4 I/O Controller (`io.max` & `io.weight`)
The I/O controller limits storage device throughput directly at the block layer (`blk-mq`), preventing disk IOPS saturation:
Developer Pitfall — Setting `memory.max` Without `memory.high` Protection:
If you set `memory.max = 2GB` without setting `memory.high = 1.8GB`, a process allocating memory rapidly will instantly slam into `memory.max`. Because the kernel has zero warning lead time to reclaim page cache memory, the container will suffer abrupt OOM crashes! Always set `memory.high` at roughly 85–90% of `memory.max` to allow smooth kernel page reclamation before hard OOM invocation!
4. Kernel Traps: Out-Of-Memory (OOM) Killer & Memory Pressure Stalls (PSI)
4.1 The Kernel OOM Killer Scoring Algorithm
When a cgroup's memory usage reaches `memory.max` and direct page reclaim fails to free memory, the kernel invokes `oom_kill_process()`. To select which task to terminate, the kernel calculates a heuristic `badness score` (ranging from 0 to 1000) for every process in the target cgroup:
Where `oom_score_adj` is a user-configurable offset stored in `/proc/$PID/oom_score_adj` (ranging from -1000 to +1000):
oom_score_adj = -1000: Completely disables the OOM killer for this process (e.g. systemd or sshd daemons).oom_score_adj = +1000: Makes the process the first choice for termination when memory is exhausted.
4.2 Pressure Stall Information (PSI): Quantifying Hardware Starvation
Before cgroups v2, engineers could only inspect raw usage metrics (e.g. "80% CPU used"). However, high usage does not necessarily mean an application is suffering latency degradation! Linux kernel 4.20 introduced Pressure Stall Information (PSI) to measure actual microsecond-level process execution stalls caused by hardware starvation.
PSI tracks stall pressure across three hardware resources under `/sys/fs/cgroup/my_app/`:
cpu.pressure: Measures time spent waiting for runnable CPU threads.memory.pressure: Measures time spent waiting for memory page reclaim and swap allocation.io.pressure: Measures time spent waiting for block device I/O completion.
Developer Pitfall — Setting `oom_score_adj = -1000` on Container Workers:
If a developer sets `oom_score_adj = -1000` inside a containerized background worker to prevent OOM termination, the kernel memory controller will be unable to kill the offending process when `memory.max` is exceeded. This forces the kernel to escalate OOM termination up to parent cgroups, causing catastrophic unexpected kills of host infrastructure services!
5. Step-by-Step Production C++ & Python cgroups v2 Controllers from Scratch
5.1 Production C++ cgroups v2 Manager System Program
Below is a complete, production-ready C++ system program that interacts directly with the Linux `/sys/fs/cgroup` sysfs pseudo-filesystem to create a cgroup, enable controllers, apply CPU/Memory limits, and move a target process into the cgroup:
5.2 Production Python cgroups v2 Inspector and Monitoring Script
Below is a complete Python script to inspect active cgroups v2 limits, memory pressure stalls, and process memberships:
Developer Pitfall — Insufficient Privileges for Subtree Controller Delegation:
Writing to `/sys/fs/cgroup/my_group/cgroup.subtree_control` requires root privileges or explicit systemd cgroup delegation (`Delegate=yes` in systemd service units). Attempting to enable controllers without root access will raise `PermissionError: [Errno 13] Permission denied`!
6. Advanced: Kubernetes Integration, eBPF Hooks, and JVM Container Awareness
6.1 How Kubernetes Kubelet Configures cgroups v2
In a Kubernetes cluster, Kubelet communicates with container runtimes (containerd/CRI-O) to configure cgroup boundaries based on Pod Quality of Service (QoS) classes:
- Guaranteed QoS (`requests == limits`): Pods receive dedicated cgroups with `memory.min` equal to requested RAM and `cpu.weight` allocated proportionally. OOM score adj is set to `-997`, making them the last targets for host eviction.
- Burstable QoS (`requests < limits`): Pods receive `memory.low` protection equal to requested RAM, with `memory.max` set to the limit. OOM score adj is scaled between `2` and `999`.
- BestEffort QoS (No limits): Pods receive no memory protection. OOM score adj is set to `1000`, ensuring they are the very first tasks killed under memory pressure.
6.2 Java JVM Container Support (`-XX:+UseContainerSupport`)
Historically, Java applications running inside Docker containers inspected `/proc/meminfo` and `/proc/cpuinfo` to size the JVM Max Heap (`-Xmx`). Because those `/proc` files expose the host server's hardware (e.g. 128GB RAM), the JVM would size its heap to 32GB inside a container limited by cgroups to 4GB—causing instant OOM crashes on startup!
Java 10+ introduced `-XX:+UseContainerSupport` (backported to Java 8u191), directing the JVM to parse `/sys/fs/cgroup/memory.max` and `/sys/fs/cgroup/cpu.max` directly, sizing its thread pools and garbage collector ergonomics accurately!
6.3 eBPF Integration with cgroups v2
Modern Linux kernels allow attaching Extended Berkeley Packet Filter (eBPF) programs directly to cgroups v2 nodes (`BPF_PROG_TYPE_CGROUP_SKB`, `BPF_PROG_TYPE_CGROUP_SOCK_ADDR`, and `BPF_PROG_TYPE_CGROUP_DEVICE`). This enables high-performance network security filtering, socket load balancing, and hardware device access control without modifying container image code!
When a socket packet passes through a network interface, the kernel checks whether the associated socket's process belongs to a cgroup with attached eBPF programs. If present, the eBPF program executes in-kernel at near-native hardware speed, filtering or redirecting packets before they reach the main network stack. This architecture powers modern Kubernetes CNI plugins like Cilium and Calico eBPF data paths!
6.4 Kernel Page List Management: Active vs Inactive LRU Lists
To understand how `memory.high` reclaims page cache memory efficiently without triggering unnecessary disk swap thrashing, we must inspect the kernel's Least Recently Used (LRU) page lists. The Linux memory management subsystem maintains two primary double-linked lists for physical memory pages:
- Active LRU List (`shrink_active_list()`): Contains physical memory pages that have been accessed recently by active processes. Pages on this list are protected from immediate eviction.
- Inactive LRU List (`shrink_inactive_list()`): Contains pages that have not been accessed recently. When memory pressure rises above `memory.high`, the kernel page reclaim algorithm scans the inactive list first, reclaiming file-backed page cache pages without touching anonymous process memory!
If a page on the inactive list is accessed again before eviction, the kernel promotes it back to the active list (`mark_page_accessed()`). This dual-list LRU algorithm ensures that active application working sets remain in physical RAM while cold file buffers are aggressively recycled!
Developer Pitfall — Running Old Java Versions in Containers Without Heap Caps:
If you run an older Java 8 runtime (prior to update 191) inside a container without explicitly specifying `-Xmx`, the JVM will ignore cgroup memory limits and attempt to allocate a heap based on host physical RAM, triggering instant OOM container kills. Always pass `-Xmx` explicitly or upgrade to modern JVM versions with container awareness!
7. Industry Comparison Matrix of Container Isolation Technologies
The table below compares core Linux container and sandbox isolation mechanisms across security, performance overhead, and resource limits:
| Isolation Technology | Isolation Boundary | CPU/RAM Overhead | Startup Latency | Primary Industry Use Case |
|---|---|---|---|---|
| cgroups v2 + Namespaces | Kernel Shared (Namespaces + cgroup) | < 1% (Near Native) | < 10ms | Standard Docker / Kubernetes Workloads |
| gVisor (Google) | User-Space Kernel Sentry (Go) | 5% - 15% Syscall Overhead | ~50ms | Multi-Tenant Untrusted Code Execution (Google Cloud Run) |
| Kata Containers | Hardware Virtualization (QEMU / Cloud-Hypervisor) | 5% - 10% RAM Overhead | 100ms - 300ms | Strict Multi-Tenant Financial / Enterprise Isolation |
| Firecracker MicroVMs (AWS) | KVM Hypervisor MicroVM | < 5MB RAM Overhead | < 5ms | Serverless Function Runtimes (AWS Lambda, Fargate) |
8. Interactive: cgroups v2 Memory Reclamation & OOM Simulator
Click "Step Memory Growth" to trace process RAM expansion $\to$ `memory.high` Page Reclaim $\to$ `memory.max` OOM Killer Trigger:
9. Performance Benchmarks: Application Latency Under CPU Throttling
The chart below compares API response latency (p99 in milliseconds) when a service is unconstrained versus when it experiences CFS bandwidth throttling under `cpu.max` limits:
10. Frequently Asked Questions
Q1: What is the fundamental difference between Linux Namespaces and cgroups v2?
Linux Namespaces isolate process visibility (restricting what a process can see, such as mount points, network adapters, and PID trees). cgroups v2 isolate hardware resource consumption (restricting what a process can use, such as CPU cycles, RAM allocations, disk I/O bandwidth, and process counts). Together, they form container runtimes.
Q2: Why did the Linux kernel community move from cgroups v1 to cgroups v2?
cgroups v1 permitted separate, orthogonal hierarchy trees for each resource controller (e.g. CPU, memory, blkio). This caused severe priority inversion bugs during dirty page writebacks to storage and uncoordinated OOM kills. cgroups v2 enforces a single unified hierarchy where every process belongs to exactly one cgroup leaf node.
Q3: How does `cpu.max` enforce CPU limits in cgroups v2?
`cpu.max` accepts two values: `$Quota$` and `$Period$` in microseconds (e.g. `150000 100000`). The ratio $Quota / Period$ defines the allowed CPU core capacity. In this example, processes in the cgroup are allowed 150ms of CPU runtime every 100ms period, allocating exactly 1.5 CPU cores.
Q4: What happens when a container exceeds `memory.high` versus `memory.max`?
Exceeding `memory.high` triggers aggressive background page cache reclamation and throttles allocating processes, but does NOT invoke the OOM killer. Exceeding `memory.max` forces immediate synchronous page reclamation; if usage remains above `memory.max`, the kernel OOM killer terminates tasks in the cgroup.
Q5: What is Pressure Stall Information (PSI) and why is it better than simple RAM usage metrics?
High memory usage alone does not indicate performance degradation. Pressure Stall Information (PSI) measures the exact microsecond percentage of execution time that tasks spend stalled waiting for CPU, memory reclamation, or block I/O completion, providing actionable hardware starvation metrics.
Q6: What is the "No Internal Processes Rule" in cgroups v2?
The No Internal Processes Rule dictates that a non-root cgroup node cannot contain both active processes (`cgroup.procs`) and child cgroup directories simultaneously. Processes must reside strictly in leaf nodes to ensure clean hierarchical resource distribution.
Q7: How does `oom_score_adj` influence kernel process termination?
`oom_score_adj` is a kernel knob ranging from -1000 to +1000 added to a process's OOM badness score. Setting `-1000` completely immune-guards a process from OOM termination, while `+1000` marks it for immediate eviction when memory is exhausted.
Q8: How does Kubernetes Kubelet use cgroups v2 for Pod Quality of Service (QoS)?
Kubelet maps Guaranteed pods to cgroups with `memory.min` set to requested RAM and low OOM scores (-997). Burstable pods use `memory.low` and `memory.max`. BestEffort pods receive zero memory protection and high OOM scores (+1000), making them the first evicted during host memory pressure.
Q9: How do I check if my Linux host is running cgroups v2 in unified mode?
Run `stat -fc %T /sys/fs/cgroup`. If the output displays `cgroup2fs`, your system is running full cgroups v2 unified hierarchy mode. If it displays `tmpfs`, your system is running legacy cgroups v1 multi-hierarchy mode.
Q10: Why do Java applications crash with OOM errors in containers if `-XX:+UseContainerSupport` is missing?
Without container awareness, older JVMs inspect `/proc/meminfo` and read physical host RAM (e.g. 64GB) instead of cgroup limits (e.g. 2GB). The JVM allocates a massive heap based on host RAM, causing the kernel to instantly terminate the container when it breaches `memory.max`.