Linux Kernel Cgroups v2 Under the Hood: A Step-by-Step Walkthrough
There is no such thing as a "container" inside the Linux kernel. When you run docker run or apply a Kubernetes Pod manifest, the kernel does not spawn a special "container" object. Instead, the illusion of a container is constructed in user space using two fundamental kernel primitives: Namespaces and Control Groups (cgroups). While namespaces dictate what a process can see (hiding other processes, network interfaces, and mount points), cgroups dictate what a process can use (restricting CPU time, memory consumption, and I/O bandwidth).
For a decade, the container ecosystem relied on cgroups v1, a system notorious for its chaotic, overlapping hierarchies and inconsistency. With the advent of cgroups v2 (unified hierarchy), the Linux kernel completely overhauled resource management, making it safer, more predictable, and easier to delegate to unprivileged users. In this deep dive, we will bypass the Docker abstraction entirely. We will directly interact with the Linux virtual filesystem to create our own cgroups, trace the exact mechanisms the kernel uses to enforce memory limits, dissect the Out-Of-Memory (OOM) killer, and explore why systems engineering requires a deep understanding of these low-level primitives.
1. The Evolution: Why Cgroups v1 Was Broken
1.1 The Multiple Hierarchy Chaos
In cgroups v1, every resource controller (CPU, memory, blkio, pids) lived in its own independent hierarchy tree. You could have a process belonging to a cgroup named "web-server" for memory limits, but a completely different cgroup named "high-priority" for CPU limits. Because the hierarchies were entirely orthogonal, coordinating limits across resources was practically impossible.
For example, what happens if you want to limit the total disk I/O a process generates when it is aggressively swapping memory? The blkio controller (managing disk I/O) had no idea which memory cgroup the process belonged to. When the kernel swapped a page to disk, the I/O was charged to the root cgroup, effectively bypassing the container's I/O limits entirely.
1.2 The Unified Hierarchy of v2
Cgroups v2 solved this by enforcing a Single Unified Hierarchy. A process belongs to exactly one cgroup in the tree. All controllers (CPU, memory, IO) are enabled on that single node. This means the kernel always knows exactly which cgroup is responsible for a resource action. If a process in `cgroup A` causes a page swap, the kernel charges both the memory controller and the IO controller associated with `cgroup A`.
If you run an older Linux distribution or an older version of Docker, you might be running in "hybrid mode" where some controllers are v1 and some are v2. This leads to extremely confusing edge cases where Kubernetes resource limits silently fail to apply. Always verify your system is fully migrated by checking stat -fc %T /sys/fs/cgroup/. If it returns cgroup2fs, you are running pure v2.
2. Interacting with the Virtual Filesystem
2.1 Everything is a File
The interface to cgroups is not a system call or a binary utility; it is a virtual filesystem (cgroup2fs) typically mounted at /sys/fs/cgroup. The kernel exposes data structures as files and directories. Creating a directory creates a new cgroup. Writing a PID to a file moves a process into that cgroup. Writing an integer to a file sets a limit.
Let's trace a concrete example directly on a Linux terminal. We will create a cgroup named sandbox.
# cd /sys/fs/cgroup # mkdir sandbox # ls sandbox/ cgroup.controllers memory.current cpu.stat cgroup.events memory.high io.stat cgroup.freeze memory.max pids.current cgroup.procs memory.swap.max pids.max cgroup.subtree_control memory.stat ...
By simply creating the sandbox directory, the kernel automatically populated it with dozens of control files. Notice cgroup.procs—this file contains the list of PIDs currently bound to this cgroup. memory.max sets the absolute memory limit.
2.2 Activating Controllers
In v2, controllers must be explicitly enabled down the tree using the cgroup.subtree_control file. By default, a newly created cgroup might not have the memory controller enabled for its children. You must write +memory to the parent's subtree_control file to enable it.
A fundamental rule of cgroups v2 is that a non-root cgroup can either contain processes (leaf node) OR distribute controllers to its children (interior node), but NOT both simultaneously. If you try to enable controllers for sub-directories in a cgroup that already contains processes in its cgroup.procs, the kernel will reject the write with EBUSY. You must move processes into leaf nodes.
3. Memory Limits: Hard Max vs. Soft High
3.1 memory.max (The OOM Trigger)
When you define resources.limits.memory in a Kubernetes Pod, the kubelet ultimately writes that byte value into the memory.max file of the container's cgroup. This is a hard limit. If a process attempts to allocate memory (via malloc, which triggers a page fault) that pushes the cgroup over memory.max, the kernel halts the allocation.
The kernel will first furiously attempt to reclaim memory (flushing file caches, swapping out anonymous memory). If reclaim fails to bring usage below the limit, the kernel invokes the OOM (Out-Of-Memory) killer. The OOM killer scans the processes in the cgroup, selects the one with the highest memory score (often the main application), and sends it a synchronous SIGKILL (9).
3.2 memory.high (The Throttle)
Cgroups v2 introduced a vastly superior mechanism: memory.high. This acts as a soft threshold. When a cgroup's memory usage crosses memory.high, the kernel does not kill the process. Instead, it aggressively throttles the process, forcing the allocating process itself to perform synchronous memory reclaim before it is granted the new page.
This causes the process to significantly slow down, providing a natural backpressure mechanism. It allows the system administrator (or orchestrator) time to detect the memory pressure and gracefully scale up or restart the service, avoiding sudden catastrophic application crashes.
# Setting a 100MB high limit, and 150MB hard limit echo 100M > /sys/fs/cgroup/sandbox/memory.high echo 150M > /sys/fs/cgroup/sandbox/memory.max # Moving our current shell into the cgroup echo $$ > /sys/fs/cgroup/sandbox/cgroup.procs
4. The Worked Trace: Inducing an OOM Kill
Let's see exactly what happens when we exceed the memory.max threshold. We have placed our shell into the sandbox cgroup with a 150MB limit.
python3 -c "x = b'A' * (200 * 1024 * 1024)".malloc succeeds virtually, but as Python writes 'A' to the memory, CPU page faults occur. The kernel's page fault handler steps in to allocate physical RAM.mem_cgroup_charge) checks the sandbox cgroup. The new page would push usage to 151MB, exceeding 150MB.out_of_memory(), targets the Python process, and sends SIGKILL.Killed $ dmesg | tail -n 5 [ 1345.678] memory cgrom oom kill: cgroup /sandbox [ 1345.679] Task in /sandbox killed as a result of limit of /sandbox [ 1345.680] memory: usage 153600kB, limit 153600kB, failcnt 32 [ 1345.681] Out of memory: Killed process 14201 (python3) total-vm:220100kB, anon-rss:153120kB
In the kernel log above, total-vm represents the virtual memory requested. Virtual memory is functionally infinite and does not count towards cgroup limits. The anon-rss (Anonymous Resident Set Size) represents the actual physical RAM backed by the process. Cgroups limit physical page allocations (RSS and Cache), not virtual address space. Never set container limits based on virtual memory sizes observed in top.
5. CPU Scheduling: Weight vs Max
5.1 CPU Weight (Proportional Shares)
Unlike memory, which is a spatial resource, CPU is a temporal resource. Cgroups manage CPU using two distinct models. The first is cpu.weight, which defines proportional fair scheduling. If cgroup A has a weight of 100, and cgroup B has a weight of 200, the kernel's Completely Fair Scheduler (CFS) ensures that when both groups are actively demanding CPU, cgroup B gets exactly twice as much CPU time as cgroup A.
Crucially, if cgroup A is idle, cgroup B can consume 100% of the CPU. Weights only apply when there is contention. This correlates directly to Kubernetes resources.requests.cpu.
5.2 CPU Max (Hard Quotas)
The second model is cpu.max, which enforces an absolute bandwidth limit, correlating to Kubernetes resources.limits.cpu. The file accepts two values: quota and period. For example, writing 100000 100000 means the cgroup is allowed 100,000 microseconds of CPU time every 100,000 microsecond period (exactly 1 CPU core). Writing 50000 100000 restricts it to 50% of a single core.
If a multi-threaded application burns through its 50,000 microsecond quota in the first 20ms of the period, the kernel CFS simply removes all threads of that cgroup from the runqueue for the remaining 80ms. The application is "throttled," appearing to completely freeze and spike in latency, even if the host machine is 99% idle.
Setting low CPU limits (e.g., 0.1 cores) on highly concurrent web servers (like Node.js or Go) is a disaster. A single inbound request might wake up multiple threads, instantly exhausting the tiny quota window. The application pauses for 80ms, destroying p99 latencies. In modern infrastructure, many companies completely disable CPU limits (only using CPU requests/weight) because CFS quota throttling is far more detrimental to latency than natural proportional contention.
6. Advanced: Cgroups and eBPF
The unified hierarchy of cgroups v2 enabled a revolutionary capability in the Linux kernel: attaching eBPF programs directly to cgroups. Using bpf() syscalls, you can attach an eBPF program of type BPF_PROG_TYPE_CGROUP_SKB to a specific cgroup path. The kernel will automatically execute this eBPF program for every single network packet sent or received by any process inside that cgroup, or any of its descendants.
This enables incredibly powerful security and observability tooling. For example, Cilium uses cgroup eBPF hooks to enforce fine-grained network policies and identity-aware packet routing without needing complex iptables rules or user-space proxies. The packet is simply evaluated as it leaves the process socket, matched against the cgroup ID, and allowed or dropped natively within the kernel.
7. Frequently Asked Questions
Q1: How do I know if my process is hitting its cgroup limits?
You should read the `memory.events` and `cpu.stat` files in the cgroup directory. `memory.events` contains a counter for `oom_kill` and `max` (times the limit was hit causing reclaim). `cpu.stat` contains `nr_throttled` (number of times the process was suspended) and `throttled_time` (total time spent frozen). These metrics are exactly what tools like cAdvisor read to populate Prometheus metrics.
Q2: Why does the `free` command inside a Docker container show the host's total RAM?
Standard Linux utilities like `free`, `top`, and `htop` read their data from `/proc/meminfo` and `/proc/stat`. Unfortunately, namespaces do not isolate these global `/proc` files. The container sees the global host metrics, leading Java virtual machines (prior to JDK 10) to incorrectly calculate heap sizes based on the host RAM rather than the cgroup `memory.max`, causing immediate OOM kills. Modern runtimes read `/sys/fs/cgroup` directly to detect limits, or use tools like `lxcfs` to bind-mount fake `/proc` files over the originals.
Q3: What is the systemd cgroup driver?
Systemd acts as the init system (PID 1) and manages the root cgroup hierarchy for the entire OS. If Docker or Kubernetes bypasses systemd and creates its own cgroup directories directly under `/sys/fs/cgroup/docker` (the `cgroupfs` driver), systemd is unaware of them and might aggressively delete them during a daemon reload, causing containers to instantly die. Modern Kubernetes clusters strongly enforce the use of the `systemd` cgroup driver, delegating all cgroup creation to systemd via DBus calls to ensure consistency.
Q4: Can a process escape its cgroup?
A process cannot modify its own `cgroup.procs` file to move itself to a higher-privileged cgroup unless it has root access and the `/sys/fs/cgroup` filesystem is mounted read-write. This is why container runtimes mount the cgroup fs as read-only inside the container's mount namespace, preventing privilege escalation.
Q5: How does Rootless Docker use cgroups v2?
Cgroups v1 required full root privileges to administer. Cgroups v2 introduces safe delegation. An administrator can change the ownership of a cgroup directory (e.g., `/sys/fs/cgroup/user.slice/user-1000.slice`) to a regular user. That user can then create sub-cgroups and move their own processes around within that subtree. This fundamentally enables secure, rootless container execution (Rootless Docker, Podman) without requiring `sudo`.
Q6: How does the IO controller handle Buffered Writes?
In v1, buffered writes (writing to the page cache rather than O_DIRECT disk writes) were completely unmanaged because the actual writeback to disk happened asynchronously by a kernel thread. Because v2 unified the memory and IO hierarchies, the kernel can now track exactly which cgroup "dirties" a page in memory, and accurately charge the subsequent disk I/O to the originating cgroup's `io.max` limit.
Written by Professor Pixel · CodingPancake · Linux Kernel & Systems Series
1. The Evolution: Why Cgroups v1 Was Broken
1.1 The Multiple Hierarchy Chaos
In cgroups v1, every resource controller (CPU, memory, blkio, pids) lived in its own independent hierarchy tree. You could have a process belonging to a cgroup named "web-server" for memory limits, but a completely different cgroup named "high-priority" for CPU limits. Because the hierarchies were entirely orthogonal, coordinating limits across resources was practically impossible.
For example, what happens if you want to limit the total disk I/O a process generates when it is aggressively swapping memory? The blkio controller (managing disk I/O) had no idea which memory cgroup the process belonged to. When the kernel swapped a page to disk, the I/O was charged to the root cgroup, effectively bypassing the container's I/O limits entirely.
1.2 The Unified Hierarchy of v2
Cgroups v2 solved this by enforcing a Single Unified Hierarchy. A process belongs to exactly one cgroup in the tree. All controllers (CPU, memory, IO) are enabled on that single node. This means the kernel always knows exactly which cgroup is responsible for a resource action. If a process in `cgroup A` causes a page swap, the kernel charges both the memory controller and the IO controller associated with `cgroup A`.
If you run an older Linux distribution or an older version of Docker, you might be running in "hybrid mode" where some controllers are v1 and some are v2. This leads to extremely confusing edge cases where Kubernetes resource limits silently fail to apply. Always verify your system is fully migrated by checking stat -fc %T /sys/fs/cgroup/. If it returns cgroup2fs, you are running pure v2.
2. Interacting with the Virtual Filesystem
2.1 Everything is a File
The interface to cgroups is not a system call or a binary utility; it is a virtual filesystem (cgroup2fs) typically mounted at /sys/fs/cgroup. The kernel exposes data structures as files and directories. Creating a directory creates a new cgroup. Writing a PID to a file moves a process into that cgroup. Writing an integer to a file sets a limit.
Let's trace a concrete example directly on a Linux terminal. We will create a cgroup named sandbox.
# cd /sys/fs/cgroup # mkdir sandbox # ls sandbox/ cgroup.controllers memory.current cpu.stat cgroup.events memory.high io.stat cgroup.freeze memory.max pids.current cgroup.procs memory.swap.max pids.max cgroup.subtree_control memory.stat ...
By simply creating the sandbox directory, the kernel automatically populated it with dozens of control files. Notice cgroup.procs—this file contains the list of PIDs currently bound to this cgroup. memory.max sets the absolute memory limit.
2.2 Activating Controllers
In v2, controllers must be explicitly enabled down the tree using the cgroup.subtree_control file. By default, a newly created cgroup might not have the memory controller enabled for its children. You must write +memory to the parent's subtree_control file to enable it.
A fundamental rule of cgroups v2 is that a non-root cgroup can either contain processes (leaf node) OR distribute controllers to its children (interior node), but NOT both simultaneously. If you try to enable controllers for sub-directories in a cgroup that already contains processes in its cgroup.procs, the kernel will reject the write with EBUSY. You must move processes into leaf nodes.
3. Memory Limits: Hard Max vs. Soft High
3.1 memory.max (The OOM Trigger)
When you define resources.limits.memory in a Kubernetes Pod, the kubelet ultimately writes that byte value into the memory.max file of the container's cgroup. This is a hard limit. If a process attempts to allocate memory (via malloc, which triggers a page fault) that pushes the cgroup over memory.max, the kernel halts the allocation.
The kernel will first furiously attempt to reclaim memory (flushing file caches, swapping out anonymous memory). If reclaim fails to bring usage below the limit, the kernel invokes the OOM (Out-Of-Memory) killer. The OOM killer scans the processes in the cgroup, selects the one with the highest memory score (often the main application), and sends it a synchronous SIGKILL (9).
3.2 memory.high (The Throttle)
Cgroups v2 introduced a vastly superior mechanism: memory.high. This acts as a soft threshold. When a cgroup's memory usage crosses memory.high, the kernel does not kill the process. Instead, it aggressively throttles the process, forcing the allocating process itself to perform synchronous memory reclaim before it is granted the new page.
This causes the process to significantly slow down, providing a natural backpressure mechanism. It allows the system administrator (or orchestrator) time to detect the memory pressure and gracefully scale up or restart the service, avoiding sudden catastrophic application crashes.
# Setting a 100MB high limit, and 150MB hard limit echo 100M > /sys/fs/cgroup/sandbox/memory.high echo 150M > /sys/fs/cgroup/sandbox/memory.max # Moving our current shell into the cgroup echo $$ > /sys/fs/cgroup/sandbox/cgroup.procs
4. The Worked Trace: Inducing an OOM Kill
Let's see exactly what happens when we exceed the memory.max threshold. We have placed our shell into the sandbox cgroup with a 150MB limit.
python3 -c "x = b'A' * (200 * 1024 * 1024)".malloc succeeds virtually, but as Python writes 'A' to the memory, CPU page faults occur. The kernel's page fault handler steps in to allocate physical RAM.mem_cgroup_charge) checks the sandbox cgroup. The new page would push usage to 151MB, exceeding 150MB.out_of_memory(), targets the Python process, and sends SIGKILL.Killed $ dmesg | tail -n 5 [ 1345.678] memory cgrom oom kill: cgroup /sandbox [ 1345.679] Task in /sandbox killed as a result of limit of /sandbox [ 1345.680] memory: usage 153600kB, limit 153600kB, failcnt 32 [ 1345.681] Out of memory: Killed process 14201 (python3) total-vm:220100kB, anon-rss:153120kB
In the kernel log above, total-vm represents the virtual memory requested. Virtual memory is functionally infinite and does not count towards cgroup limits. The anon-rss (Anonymous Resident Set Size) represents the actual physical RAM backed by the process. Cgroups limit physical page allocations (RSS and Cache), not virtual address space. Never set container limits based on virtual memory sizes observed in top.
5. CPU Scheduling: Weight vs Max
5.1 CPU Weight (Proportional Shares)
Unlike memory, which is a spatial resource, CPU is a temporal resource. Cgroups manage CPU using two distinct models. The first is cpu.weight, which defines proportional fair scheduling. If cgroup A has a weight of 100, and cgroup B has a weight of 200, the kernel's Completely Fair Scheduler (CFS) ensures that when both groups are actively demanding CPU, cgroup B gets exactly twice as much CPU time as cgroup A.
Crucially, if cgroup A is idle, cgroup B can consume 100% of the CPU. Weights only apply when there is contention. This correlates directly to Kubernetes resources.requests.cpu.
5.2 CPU Max (Hard Quotas)
The second model is cpu.max, which enforces an absolute bandwidth limit, correlating to Kubernetes resources.limits.cpu. The file accepts two values: quota and period. For example, writing 100000 100000 means the cgroup is allowed 100,000 microseconds of CPU time every 100,000 microsecond period (exactly 1 CPU core). Writing 50000 100000 restricts it to 50% of a single core.
If a multi-threaded application burns through its 50,000 microsecond quota in the first 20ms of the period, the kernel CFS simply removes all threads of that cgroup from the runqueue for the remaining 80ms. The application is "throttled," appearing to completely freeze and spike in latency, even if the host machine is 99% idle.
Setting low CPU limits (e.g., 0.1 cores) on highly concurrent web servers (like Node.js or Go) is a disaster. A single inbound request might wake up multiple threads, instantly exhausting the tiny quota window. The application pauses for 80ms, destroying p99 latencies. In modern infrastructure, many companies completely disable CPU limits (only using CPU requests/weight) because CFS quota throttling is far more detrimental to latency than natural proportional contention.
6. Advanced: Cgroups and eBPF
The unified hierarchy of cgroups v2 enabled a revolutionary capability in the Linux kernel: attaching eBPF programs directly to cgroups. Using bpf() syscalls, you can attach an eBPF program of type BPF_PROG_TYPE_CGROUP_SKB to a specific cgroup path. The kernel will automatically execute this eBPF program for every single network packet sent or received by any process inside that cgroup, or any of its descendants.
This enables incredibly powerful security and observability tooling. For example, Cilium uses cgroup eBPF hooks to enforce fine-grained network policies and identity-aware packet routing without needing complex iptables rules or user-space proxies. The packet is simply evaluated as it leaves the process socket, matched against the cgroup ID, and allowed or dropped natively within the kernel.
7. Frequently Asked Questions
Q1: How do I know if my process is hitting its cgroup limits?
You should read the `memory.events` and `cpu.stat` files in the cgroup directory. `memory.events` contains a counter for `oom_kill` and `max` (times the limit was hit causing reclaim). `cpu.stat` contains `nr_throttled` (number of times the process was suspended) and `throttled_time` (total time spent frozen). These metrics are exactly what tools like cAdvisor read to populate Prometheus metrics.
Q2: Why does the `free` command inside a Docker container show the host's total RAM?
Standard Linux utilities like `free`, `top`, and `htop` read their data from `/proc/meminfo` and `/proc/stat`. Unfortunately, namespaces do not isolate these global `/proc` files. The container sees the global host metrics, leading Java virtual machines (prior to JDK 10) to incorrectly calculate heap sizes based on the host RAM rather than the cgroup `memory.max`, causing immediate OOM kills. Modern runtimes read `/sys/fs/cgroup` directly to detect limits, or use tools like `lxcfs` to bind-mount fake `/proc` files over the originals.
Q3: What is the systemd cgroup driver?
Systemd acts as the init system (PID 1) and manages the root cgroup hierarchy for the entire OS. If Docker or Kubernetes bypasses systemd and creates its own cgroup directories directly under `/sys/fs/cgroup/docker` (the `cgroupfs` driver), systemd is unaware of them and might aggressively delete them during a daemon reload, causing containers to instantly die. Modern Kubernetes clusters strongly enforce the use of the `systemd` cgroup driver, delegating all cgroup creation to systemd via DBus calls to ensure consistency.
Q4: Can a process escape its cgroup?
A process cannot modify its own `cgroup.procs` file to move itself to a higher-privileged cgroup unless it has root access and the `/sys/fs/cgroup` filesystem is mounted read-write. This is why container runtimes mount the cgroup fs as read-only inside the container's mount namespace, preventing privilege escalation.
Q5: How does Rootless Docker use cgroups v2?
Cgroups v1 required full root privileges to administer. Cgroups v2 introduces safe delegation. An administrator can change the ownership of a cgroup directory (e.g., `/sys/fs/cgroup/user.slice/user-1000.slice`) to a regular user. That user can then create sub-cgroups and move their own processes around within that subtree. This fundamentally enables secure, rootless container execution (Rootless Docker, Podman) without requiring `sudo`.
Q6: How does the IO controller handle Buffered Writes?
In v1, buffered writes (writing to the page cache rather than O_DIRECT disk writes) were completely unmanaged because the actual writeback to disk happened asynchronously by a kernel thread. Because v2 unified the memory and IO hierarchies, the kernel can now track exactly which cgroup "dirties" a page in memory, and accurately charge the subsequent disk I/O to the originating cgroup's `io.max` limit.
Written by Professor Pixel · CodingPancake · Linux Kernel & Systems Series
1. The Evolution: Why Cgroups v1 Was Broken
1.1 The Multiple Hierarchy Chaos
In cgroups v1, every resource controller (CPU, memory, blkio, pids) lived in its own independent hierarchy tree. You could have a process belonging to a cgroup named "web-server" for memory limits, but a completely different cgroup named "high-priority" for CPU limits. Because the hierarchies were entirely orthogonal, coordinating limits across resources was practically impossible.
For example, what happens if you want to limit the total disk I/O a process generates when it is aggressively swapping memory? The blkio controller (managing disk I/O) had no idea which memory cgroup the process belonged to. When the kernel swapped a page to disk, the I/O was charged to the root cgroup, effectively bypassing the container's I/O limits entirely.
1.2 The Unified Hierarchy of v2
Cgroups v2 solved this by enforcing a Single Unified Hierarchy. A process belongs to exactly one cgroup in the tree. All controllers (CPU, memory, IO) are enabled on that single node. This means the kernel always knows exactly which cgroup is responsible for a resource action. If a process in `cgroup A` causes a page swap, the kernel charges both the memory controller and the IO controller associated with `cgroup A`.
If you run an older Linux distribution or an older version of Docker, you might be running in "hybrid mode" where some controllers are v1 and some are v2. This leads to extremely confusing edge cases where Kubernetes resource limits silently fail to apply. Always verify your system is fully migrated by checking stat -fc %T /sys/fs/cgroup/. If it returns cgroup2fs, you are running pure v2.
2. Interacting with the Virtual Filesystem
2.1 Everything is a File
The interface to cgroups is not a system call or a binary utility; it is a virtual filesystem (cgroup2fs) typically mounted at /sys/fs/cgroup. The kernel exposes data structures as files and directories. Creating a directory creates a new cgroup. Writing a PID to a file moves a process into that cgroup. Writing an integer to a file sets a limit.
Let's trace a concrete example directly on a Linux terminal. We will create a cgroup named sandbox.
# cd /sys/fs/cgroup # mkdir sandbox # ls sandbox/ cgroup.controllers memory.current cpu.stat cgroup.events memory.high io.stat cgroup.freeze memory.max pids.current cgroup.procs memory.swap.max pids.max cgroup.subtree_control memory.stat ...
By simply creating the sandbox directory, the kernel automatically populated it with dozens of control files. Notice cgroup.procs—this file contains the list of PIDs currently bound to this cgroup. memory.max sets the absolute memory limit.
2.2 Activating Controllers
In v2, controllers must be explicitly enabled down the tree using the cgroup.subtree_control file. By default, a newly created cgroup might not have the memory controller enabled for its children. You must write +memory to the parent's subtree_control file to enable it.
A fundamental rule of cgroups v2 is that a non-root cgroup can either contain processes (leaf node) OR distribute controllers to its children (interior node), but NOT both simultaneously. If you try to enable controllers for sub-directories in a cgroup that already contains processes in its cgroup.procs, the kernel will reject the write with EBUSY. You must move processes into leaf nodes.
3. Memory Limits: Hard Max vs. Soft High
3.1 memory.max (The OOM Trigger)
When you define resources.limits.memory in a Kubernetes Pod, the kubelet ultimately writes that byte value into the memory.max file of the container's cgroup. This is a hard limit. If a process attempts to allocate memory (via malloc, which triggers a page fault) that pushes the cgroup over memory.max, the kernel halts the allocation.
The kernel will first furiously attempt to reclaim memory (flushing file caches, swapping out anonymous memory). If reclaim fails to bring usage below the limit, the kernel invokes the OOM (Out-Of-Memory) killer. The OOM killer scans the processes in the cgroup, selects the one with the highest memory score (often the main application), and sends it a synchronous SIGKILL (9).
3.2 memory.high (The Throttle)
Cgroups v2 introduced a vastly superior mechanism: memory.high. This acts as a soft threshold. When a cgroup's memory usage crosses memory.high, the kernel does not kill the process. Instead, it aggressively throttles the process, forcing the allocating process itself to perform synchronous memory reclaim before it is granted the new page.
This causes the process to significantly slow down, providing a natural backpressure mechanism. It allows the system administrator (or orchestrator) time to detect the memory pressure and gracefully scale up or restart the service, avoiding sudden catastrophic application crashes.
# Setting a 100MB high limit, and 150MB hard limit echo 100M > /sys/fs/cgroup/sandbox/memory.high echo 150M > /sys/fs/cgroup/sandbox/memory.max # Moving our current shell into the cgroup echo $$ > /sys/fs/cgroup/sandbox/cgroup.procs
4. The Worked Trace: Inducing an OOM Kill
Let's see exactly what happens when we exceed the memory.max threshold. We have placed our shell into the sandbox cgroup with a 150MB limit.
python3 -c "x = b'A' * (200 * 1024 * 1024)".malloc succeeds virtually, but as Python writes 'A' to the memory, CPU page faults occur. The kernel's page fault handler steps in to allocate physical RAM.mem_cgroup_charge) checks the sandbox cgroup. The new page would push usage to 151MB, exceeding 150MB.out_of_memory(), targets the Python process, and sends SIGKILL.Killed $ dmesg | tail -n 5 [ 1345.678] memory cgrom oom kill: cgroup /sandbox [ 1345.679] Task in /sandbox killed as a result of limit of /sandbox [ 1345.680] memory: usage 153600kB, limit 153600kB, failcnt 32 [ 1345.681] Out of memory: Killed process 14201 (python3) total-vm:220100kB, anon-rss:153120kB
In the kernel log above, total-vm represents the virtual memory requested. Virtual memory is functionally infinite and does not count towards cgroup limits. The anon-rss (Anonymous Resident Set Size) represents the actual physical RAM backed by the process. Cgroups limit physical page allocations (RSS and Cache), not virtual address space. Never set container limits based on virtual memory sizes observed in top.
5. CPU Scheduling: Weight vs Max
5.1 CPU Weight (Proportional Shares)
Unlike memory, which is a spatial resource, CPU is a temporal resource. Cgroups manage CPU using two distinct models. The first is cpu.weight, which defines proportional fair scheduling. If cgroup A has a weight of 100, and cgroup B has a weight of 200, the kernel's Completely Fair Scheduler (CFS) ensures that when both groups are actively demanding CPU, cgroup B gets exactly twice as much CPU time as cgroup A.
Crucially, if cgroup A is idle, cgroup B can consume 100% of the CPU. Weights only apply when there is contention. This correlates directly to Kubernetes resources.requests.cpu.
5.2 CPU Max (Hard Quotas)
The second model is cpu.max, which enforces an absolute bandwidth limit, correlating to Kubernetes resources.limits.cpu. The file accepts two values: quota and period. For example, writing 100000 100000 means the cgroup is allowed 100,000 microseconds of CPU time every 100,000 microsecond period (exactly 1 CPU core). Writing 50000 100000 restricts it to 50% of a single core.
If a multi-threaded application burns through its 50,000 microsecond quota in the first 20ms of the period, the kernel CFS simply removes all threads of that cgroup from the runqueue for the remaining 80ms. The application is "throttled," appearing to completely freeze and spike in latency, even if the host machine is 99% idle.
Setting low CPU limits (e.g., 0.1 cores) on highly concurrent web servers (like Node.js or Go) is a disaster. A single inbound request might wake up multiple threads, instantly exhausting the tiny quota window. The application pauses for 80ms, destroying p99 latencies. In modern infrastructure, many companies completely disable CPU limits (only using CPU requests/weight) because CFS quota throttling is far more detrimental to latency than natural proportional contention.
6. Advanced: Cgroups and eBPF
The unified hierarchy of cgroups v2 enabled a revolutionary capability in the Linux kernel: attaching eBPF programs directly to cgroups. Using bpf() syscalls, you can attach an eBPF program of type BPF_PROG_TYPE_CGROUP_SKB to a specific cgroup path. The kernel will automatically execute this eBPF program for every single network packet sent or received by any process inside that cgroup, or any of its descendants.
This enables incredibly powerful security and observability tooling. For example, Cilium uses cgroup eBPF hooks to enforce fine-grained network policies and identity-aware packet routing without needing complex iptables rules or user-space proxies. The packet is simply evaluated as it leaves the process socket, matched against the cgroup ID, and allowed or dropped natively within the kernel.
7. Frequently Asked Questions
Q1: How do I know if my process is hitting its cgroup limits?
You should read the `memory.events` and `cpu.stat` files in the cgroup directory. `memory.events` contains a counter for `oom_kill` and `max` (times the limit was hit causing reclaim). `cpu.stat` contains `nr_throttled` (number of times the process was suspended) and `throttled_time` (total time spent frozen). These metrics are exactly what tools like cAdvisor read to populate Prometheus metrics.
Q2: Why does the `free` command inside a Docker container show the host's total RAM?
Standard Linux utilities like `free`, `top`, and `htop` read their data from `/proc/meminfo` and `/proc/stat`. Unfortunately, namespaces do not isolate these global `/proc` files. The container sees the global host metrics, leading Java virtual machines (prior to JDK 10) to incorrectly calculate heap sizes based on the host RAM rather than the cgroup `memory.max`, causing immediate OOM kills. Modern runtimes read `/sys/fs/cgroup` directly to detect limits, or use tools like `lxcfs` to bind-mount fake `/proc` files over the originals.
Q3: What is the systemd cgroup driver?
Systemd acts as the init system (PID 1) and manages the root cgroup hierarchy for the entire OS. If Docker or Kubernetes bypasses systemd and creates its own cgroup directories directly under `/sys/fs/cgroup/docker` (the `cgroupfs` driver), systemd is unaware of them and might aggressively delete them during a daemon reload, causing containers to instantly die. Modern Kubernetes clusters strongly enforce the use of the `systemd` cgroup driver, delegating all cgroup creation to systemd via DBus calls to ensure consistency.
Q4: Can a process escape its cgroup?
A process cannot modify its own `cgroup.procs` file to move itself to a higher-privileged cgroup unless it has root access and the `/sys/fs/cgroup` filesystem is mounted read-write. This is why container runtimes mount the cgroup fs as read-only inside the container's mount namespace, preventing privilege escalation.
Q5: How does Rootless Docker use cgroups v2?
Cgroups v1 required full root privileges to administer. Cgroups v2 introduces safe delegation. An administrator can change the ownership of a cgroup directory (e.g., `/sys/fs/cgroup/user.slice/user-1000.slice`) to a regular user. That user can then create sub-cgroups and move their own processes around within that subtree. This fundamentally enables secure, rootless container execution (Rootless Docker, Podman) without requiring `sudo`.
Q6: How does the IO controller handle Buffered Writes?
In v1, buffered writes (writing to the page cache rather than O_DIRECT disk writes) were completely unmanaged because the actual writeback to disk happened asynchronously by a kernel thread. Because v2 unified the memory and IO hierarchies, the kernel can now track exactly which cgroup "dirties" a page in memory, and accurately charge the subsequent disk I/O to the originating cgroup's `io.max` limit.
Written by Professor Pixel · CodingPancake · Linux Kernel & Systems Series