eBPF in the Linux Kernel: How It Works, Verifier Safety, and Production Observability
eBPF lets you run verified, sandboxed programs inside the Linux kernel at near-native speed — without writing kernel modules, recompiling the kernel, or rebooting. In this walkthrough, you'll understand exactly how the eBPF verifier proves safety, how maps share state between kernel and userspace, and how production systems use eBPF for zero-overhead observability, XDP packet processing, and runtime security.
1. Why eBPF Rewired Linux Observability
1.1 The Problem eBPF Solves
For decades, observing a running Linux system meant choosing between two painful options. You could add printk() calls and recompile the kernel, or you could write a kernel module — a shared library that runs at kernel privilege level with no safety guarantees whatsoever. A single null pointer dereference in a kernel module panics the entire machine. Neither approach is remotely acceptable in a production environment serving millions of requests per second.
eBPF (Extended Berkeley Packet Filter) solves this with a radically different model: you compile a small C program into eBPF bytecode, hand it to the kernel, and a built-in static analyzer called the Verifier mathematically proves it cannot crash, loop infinitely, or access memory out of bounds — before it ever runs. Once verified, a JIT compiler converts the bytecode directly into native CPU instructions. The result is a program running inside the kernel at nearly zero overhead, attached to any kernel event you care about.
This mental model is the key intuition: eBPF is to the Linux kernel what JavaScript is to a browser sandbox — you write code that runs in a privileged, constrained environment with strict safety rules enforced by the host runtime. The difference is that eBPF programs can observe kernel internals that are invisible to any userspace tool.
1.2 A Brief History: BPF → eBPF
The original Berkeley Packet Filter (BPF), introduced in 1992, was a simple in-kernel bytecode VM with 2 registers designed for filtering network packets. The tcpdump tool still uses it. In 2014, Alexei Starovoitov and Daniel Borkmann extended BPF into what we now call eBPF: 10 64-bit general-purpose registers, a 512-byte stack, a verifier, a JIT compiler, and support for far more than packet filtering. eBPF was merged into Linux 3.18 and has grown explosively — today it powers Cilium (Kubernetes CNI), Pixie (observability), Falco (security), Meta's Katran load balancer, and Cloudflare's DDoS mitigation stack.
Developer Pitfall — Confusing "eBPF" with Classic BPF Socket Filters:
When you run tcpdump -d 'tcp port 80', you see classic BPF assembly — 32-bit registers, no maps, no helpers. Modern eBPF programs use the extended instruction set (64-bit registers, 10 registers vs 2, BPF maps, helper calls). Always verify you're using libbpf / BTF / CO-RE for new development — not the legacy socket filter API. Tools like bpftool prog list will show you program types that distinguish them.
2. The eBPF Architecture: Maps, Programs, Helpers, and the Verifier
2.1 Four Pillars of the eBPF Runtime
Every eBPF deployment consists of four interacting components. Understanding how they connect is essential before you write your first program.
1. eBPF Programs — C code compiled to eBPF bytecode via Clang/LLVM (using -target bpf). Programs are event-driven: they execute when a specific kernel event occurs (a syscall entry, a network packet arriving, a function return). They are stateless by themselves — any state that must persist between invocations, or that must be shared with userspace, lives in Maps.
2. eBPF Maps — Persistent key-value data structures allocated in kernel memory, accessible by both the eBPF program (via helper calls) and a userspace process (via file descriptors). Maps are the shared memory between kernel instrumentation and your userspace monitoring dashboard.
3. Helper Functions — A curated set of in-kernel functions that eBPF programs can call. They provide safe, mediated access to kernel services: reading process metadata (bpf_get_current_pid_tgid()), sending data to userspace (bpf_perf_event_output()), performing map lookups (bpf_map_lookup_elem()), and timing events (bpf_ktime_get_ns()). Calling arbitrary kernel functions directly is forbidden — only approved helpers are permitted.
4. The Verifier — The kernel's static analysis engine that validates every eBPF program before it executes. It proves absence of unbounded loops, invalid memory accesses, uninitialized register reads, and unsafe pointer arithmetic. If any of these conditions cannot be proven safe, the program is rejected with a detailed error message.
Diagram 1: eBPF Runtime Architecture. Userspace loads bytecode via libbpf → Verifier proves safety → JIT compiles to native code → Programs attach to kernel events and communicate with userspace through Maps.
Developer Pitfall — eBPF Programs Are NOT Kernel Modules:
A kernel module (.ko file) runs at full kernel privilege with zero safety constraints — one bad pointer write corrupts kernel memory and causes a system panic. An eBPF program is sandboxed by the verifier, cannot call arbitrary kernel functions, cannot access out-of-bounds memory, and cannot execute indefinitely. The performance difference is minimal (both run in kernel context) but the safety difference is absolute. Never load an eBPF program as a kernel module, or vice versa.
3. The eBPF Verifier: Safety Without Sacrifice
3.1 How the Verifier Thinks About Your Code
The Verifier performs abstract interpretation — it symbolically executes your program tracking not just code paths but also the type and range of every register's value at every instruction. This is why it can reject programs before they ever run.
The analysis works in two passes. The first pass builds a Directed Acyclic Graph (DAG) of all reachable basic blocks and verifies that no unbounded back-edges exist — this is how it proves termination. The second, deeper pass symbolically executes every instruction along every possible code path, carrying a register state that tracks whether each register holds: an uninitialized value, a scalar integer with known bounds, a pointer to map data, a pointer to packet data, or a pointer to the stack.
3.2 Pointer Tracking: The Hardest Part
The most sophisticated part of the verifier is pointer tracking. When you call bpf_map_lookup_elem(), it returns either a pointer to map value memory, or NULL. The verifier forces you to check for NULL before dereferencing the pointer. If you attempt to dereference the return value without a NULL check, the program is rejected:
The verifier also tracks pointer arithmetic bounds. You can advance a pointer into a map value, but only within the bounds declared in the map's value size. Any pointer arithmetic that could produce an out-of-bounds address — even speculatively — is rejected. This prevents the entire class of out-of-bounds kernel memory read/write vulnerabilities that plague traditional kernel code.
3.3 Bounded Loops and the Loop Iteration Limit
Early eBPF had zero loop support — all loops had to be manually unrolled at compile time with #pragma unroll. Since Linux 5.3, bounded loops are permitted, but the verifier must be able to prove the loop terminates in a bounded number of iterations. The maximum loop depth is currently 1,000,000 instructions total across the entire program execution path. Programs exceeding this limit are rejected even if they would terminate correctly at runtime.
Developer Pitfall — Verifier Complexity Explosion on Wide Value Ranges:
The verifier's state space explodes when register value ranges are wide. If the verifier tracks r1 as a scalar in range [0, UINT32_MAX], it must simulate the loop for every possible value in that range. This causes "verifier complexity limit exceeded" rejections even for simple loops. The fix: narrow register ranges with explicit bounds checks (e.g., if (len > 64) return -1;) before using the value as a loop counter. This tells the verifier the range is [0, 64] — tractable to analyze.
4. eBPF Program Types and Attachment Points
4.1 The Taxonomy of eBPF Program Types
Every eBPF program has a declared program type that determines which kernel subsystem it attaches to, which context struct it receives as argument, and which helper functions it is permitted to call. Choosing the wrong program type for your use case is the most common architectural mistake in eBPF development.
| Program Type | Attachment Point | Context Struct | Typical Use Case |
|---|---|---|---|
| BPF_PROG_TYPE_XDP | NIC driver (pre-stack) | xdp_md | DDoS mitigation, load balancing at line rate |
| BPF_PROG_TYPE_KPROBE | Any kernel function entry/exit | pt_regs | Syscall latency tracing, argument inspection |
| BPF_PROG_TYPE_TRACEPOINT | Static kernel tracepoints | tracepoint args struct | Scheduler events, block I/O, network events |
| BPF_PROG_TYPE_SCHED_CLS | TC (Traffic Control) hook | __sk_buff | Pod-to-pod network policy enforcement |
| BPF_PROG_TYPE_LSM | Linux Security Module hooks | LSM hook args | Runtime security policy (Tetragon, Falco) |
| BPF_PROG_TYPE_SOCKET_FILTER | Socket receive path | __sk_buff | Packet filtering (classic BPF replacement) |
| BPF_PROG_TYPE_FENTRY/FEXIT | BTF-typed kernel functions | BTF function args | Low-overhead kernel function tracing |
4.2 XDP vs TC: Which Network Hook Do You Need?
The most performance-sensitive decision in eBPF networking is choosing between XDP and TC hooks. XDP (eXpress Data Path) runs at the earliest possible point — the NIC driver's receive function, before the kernel even allocates an sk_buff struct. This means XDP programs execute on raw DMA memory directly off the NIC, achieving packet processing rates exceeding 24 Mpps on a single core.
TC (Traffic Control) hooks run later, after the kernel has allocated an sk_buff and after the network stack has begun processing. TC hooks see a richer, fully-parsed packet context and can both filter incoming (ingress) and outgoing (egress) traffic. This makes TC the right choice for Kubernetes network policy enforcement (where you need to inspect pod IP routing decisions), while XDP is the right choice for raw packet dropping (DDoS mitigation) where every nanosecond matters.
Developer Pitfall — XDP in SKB Mode vs Native Mode:
XDP has three operating modes: native (the driver directly calls your program — fastest), offloaded (runs on the NIC's FPGA — even faster but hardware-specific), and generic/SKB mode (kernel emulates XDP for NICs without native support — much slower, defeats the purpose). Check your NIC driver's XDP support with ip link show. If native mode isn't supported, consider using TC ingress hooks instead, which provide near-equivalent performance without the mode confusion.
5. eBPF Maps: Shared State Between Kernel and Userspace
5.1 Maps Are the Memory of eBPF Programs
An eBPF program by itself is stateless — when the event that triggered it finishes, all register values are gone. Maps provide persistent, shared state. From the eBPF program's side, maps are accessed via helper calls. From the userspace side, maps are accessed via file descriptors obtained from the bpf(BPF_MAP_CREATE, ...) syscall or via bpf_obj_get() on pinned paths in the BPF filesystem (/sys/fs/bpf/).
Map access from eBPF code is lock-free by design: the kernel provides atomic per-CPU map variants and spin-lock primitives for correctness in concurrent scenarios. The per-CPU hash map and per-CPU array are the go-to choices for high-frequency instrumentation — each CPU core writes to its own copy of the map, eliminating contention. Userspace then aggregates values across cores.
5.2 Map Types and When to Use Each
5.3 Ring Buffer vs Perf Event Array: The Modern Choice
The BPF_MAP_TYPE_RINGBUF (Linux 5.8+) is now the strongly preferred mechanism for sending events from kernel to userspace, replacing the older BPF_MAP_TYPE_PERF_EVENT_ARRAY. The key difference: the ring buffer uses a single shared circular buffer across all CPUs, while perf event arrays use one buffer per CPU. The ring buffer reduces memory waste, provides guaranteed ordering of events from different CPUs, and supports a reserve-commit protocol that eliminates memory copies for large records.
Developer Pitfall — Map Pinning and Lifecycle Management:
By default, eBPF maps are reference-counted and destroyed when all file descriptors to them are closed. This means your map data disappears when your loader process exits! To persist a map across process restarts (e.g., for stateful packet flow tracking), you must pin the map to the BPF filesystem: bpf_obj_pin(map_fd, "/sys/fs/bpf/my_flow_table"). A subsequent process retrieves it via bpf_obj_get("/sys/fs/bpf/my_flow_table"). Forgetting to pin maps that should survive restarts is a common production bug that causes silent data loss after a monitoring agent crashes and restarts.
6. Step-by-Step: Writing a Production kprobe eBPF Program
6.1 The Goal: Trace sys_read Latency by PID
Let's walk through a complete, production-quality eBPF program that measures the latency of the sys_read system call broken down by process ID. This involves two kprobe attachments: one at sys_read entry (to record the start timestamp) and one at sys_read exit (to compute elapsed time and write it to a histogram map).
6.2 The Userspace Loader and Histogram Reader
The kernel-side eBPF C code is compiled to a .o ELF file, then loaded by a userspace program using libbpf. The loader opens the ELF, instantiates the maps, loads programs through the verifier, and attaches them to their target kprobe hook points.
Developer Pitfall — kprobe vs tracepoint: Prefer Tracepoints for Stability:
kprobes attach to kernel function names — but kernel function names can be renamed, inlined by the compiler, or removed between kernel versions. A kprobe that works on Linux 5.15 may silently do nothing on Linux 6.1 if the function was inlined. Stable kernel tracepoints (e.g., tracepoint:syscalls:sys_enter_read) are part of the kernel's stable ABI and are explicitly versioned. Use tracepoints for syscalls and scheduler events whenever possible, and reserve kprobes for internal kernel functions with no tracepoint equivalent.
7. Advanced: eBPF CO-RE — Compile Once, Run Everywhere
7.1 The Portability Problem eBPF Historically Had
Classic eBPF programs had a painful portability problem: they directly accessed kernel struct fields by hardcoded byte offsets. The offset of task_struct.pid is not guaranteed to be the same between Linux 5.4 and Linux 5.15 — kernel developers add, remove, and reorder struct fields. A program compiled against kernel header version A would silently read wrong memory on kernel version B, producing corrupted data with no error.
The solution is CO-RE (Compile Once — Run Everywhere), introduced in libbpf 0.1 alongside the BTF (BPF Type Format) kernel feature. BTF embeds rich kernel type information (essentially DWARF debug info, but compact) directly into the running kernel at /sys/kernel/btf/vmlinux. When your eBPF loader starts, it reads the host kernel's BTF type info and rewrites the byte offsets in your pre-compiled eBPF bytecode to match the running kernel's actual struct layout. This happens transparently in the loader before the program reaches the verifier.
7.2 Generating vmlinux.h from BTF
The vmlinux.h header — which contains every kernel type, struct, enum, and typedef — is generated from the running kernel's BTF data with a single command. This eliminates the need to install kernel header packages on your build machine:
Developer Pitfall — CO-RE Requires CONFIG_DEBUG_INFO_BTF=y in the Host Kernel:
CO-RE portability depends entirely on the host kernel being compiled with CONFIG_DEBUG_INFO_BTF=y. All major distributions (Ubuntu 20.04+, Fedora 32+, RHEL 9, Debian 12) ship this enabled by default. However, minimal embedded Linux builds, custom kernel configurations, and some cloud VM images may not have it. Before distributing a CO-RE eBPF tool, always add a startup check: access("/sys/kernel/btf/vmlinux", F_OK). If absent, emit a clear error — your tool will silently fail to load otherwise.
8. Production Use Case 1: XDP for Line-Rate Packet Processing
8.1 How Cloudflare Drops DDoS Traffic at the NIC
Cloudflare's production DDoS mitigation system — Gatebot and the underlying XDP framework — makes an XDP-forwarding decision for every packet before the kernel TCP/IP stack allocates a single byte of memory. This is the core value proposition of XDP: the action decision happens in the NIC driver's NAPI receive loop, at memory that's still in DMA cache, before any kernel data structures are touched.
An XDP program returns one of five verdict codes that tell the NIC driver what to do with the packet: XDP_DROP (free the DMA buffer immediately), XDP_PASS (hand it to the kernel stack normally), XDP_TX (bounce it back out the same NIC — useful for reflectors), XDP_REDIRECT (send it to a different NIC or CPU queue via BPF_MAP_TYPE_DEVMAP/CPUMAP), or XDP_ABORTED (drop with tracepoint — for debugging).
Notice the mandatory bounds checks before every pointer advance (if ((void *)(eth + 1) > data_end)). These are required by the verifier — every packet pointer dereference must be preceded by an explicit bounds check proving the access is within the packet's DMA buffer. This is the XDP equivalent of the NULL pointer check for map lookups. Without these checks, the verifier rejects the program.
Developer Pitfall — Forgetting data_end Bounds Checks on Every Pointer Advance:
Every time you advance a packet pointer (e.g., from Ethernet header to IP header to TCP header), you must re-check the new pointer against ctx->data_end before dereferencing. Forgetting a single bounds check causes immediate verifier rejection with the message: "invalid access to packet, off=X size=Y, R[0] max value is beyond packet end." This is intentional: the verifier cannot statically prove how long your packet is, so it requires you to prove it dynamically at each step.
9. Production Use Case 2: Distributed Tracing Without Sidecars
9.1 The Sidecar Proxy Tax and How eBPF Eliminates It
Traditional service mesh observability (Istio + Envoy sidecars) injects a proxy container into every pod that intercepts all network traffic. This sidecar model extracts rich L7 telemetry — HTTP method, URL, response code, latency — but at a cost: every pod now runs an additional process consuming 50–150MB RAM, adds 2–5ms of additional latency per request due to L4 socket bouncing, and requires privileged container injection that creates RBAC complexity.
eBPF-based observability tools like Cilium Hubble and Pixie attach to kernel socket system calls and protocol parsing functions to extract the same L7 telemetry from within the kernel, without any sidecar. A single eBPF agent running as a DaemonSet (one per node) observes all pod network traffic by hooking into tcp_sendmsg / tcp_recvmsg kprobes and parsing HTTP/1.1, HTTP/2, gRPC, and MySQL protocol frames from raw socket data.
9.2 How Pixie Traces HTTP/2 gRPC Without Instrumentation
Pixie's protocol parser attaches eBPF uprobes (userspace probes, the userspace equivalent of kprobes) to Go's crypto/tls library functions to capture cleartext HTTP/2 frames after TLS decryption but before the data is written to the network. This works because TLS termination happens in the application's TLS library — the plaintext exists briefly in application memory between the decrypt call and the send call. Pixie's uprobe captures this plaintext window with zero changes to application code and zero redeployment required.
The latency overhead of Pixie's full-service telemetry is typically 2–4% CPU across the node — compared to 15–30% CPU consumed by sidecar proxies. This is the quantitative reason the cloud-native community is shifting from sidecar-based to eBPF-based observability for high-scale deployments.
Developer Pitfall — uprobes Break When Application Binaries Are Stripped:
Uprobes attach to function addresses resolved from the binary's symbol table. If your Go, Rust, or C++ binary is built with strip -s (no symbols), uprobe-based tools cannot resolve function names and silently attach to nothing. Production binaries are often stripped for security and size. The workaround: use Go's runtime.ReadMemStats uprobes (runtime functions are never stripped in Go), or build with -w -s (remove DWARF) but not stripped symbols, or rely on kernel kprobe-only instrumentation where possible. Always verify your uprobe attached successfully by checking bpftool prog list and confirming non-zero event counts.
10. Production Use Case 3: Runtime Security with LSM + eBPF
10.1 The Linux Security Module Hook System
The Linux Security Module (LSM) framework inserts security decision hooks at critical kernel operations: file open, process exec, socket bind, capability checks, network connect. Traditionally, only statically compiled security modules (SELinux, AppArmor) could hook these points. Since Linux 5.7, eBPF programs with type BPF_PROG_TYPE_LSM can attach to any LSM hook — giving you programmable, kernel-enforced security policies without modifying the kernel or rebooting.
Tetragon (by Isovalent/Cilium) uses LSM eBPF hooks to enforce fine-grained Kubernetes security policies: blocking specific syscalls, preventing process privilege escalation (setuid to root), restricting which files a pod can open, and detecting and blocking fileless malware execution (memfd_create + exec of anonymous memory regions). Critically, these blocks happen inside the kernel — a compromised container process cannot bypass them even with full container root access.
Returning a negative errno from an LSM eBPF hook causes the kernel to deny the operation outright. The denied process receives an EPERM error from its execve() call. The key security property: this decision is made inside the kernel, so even a process running as UID 0 inside a container cannot bypass it — the kernel enforces the denial before any privileged process has a chance to intercept.
Developer Pitfall — LSM eBPF Requires CAP_BPF + CAP_PERFMON (or CAP_SYS_ADMIN):
Loading eBPF programs — especially LSM type programs — requires elevated Linux capabilities. On Linux 5.8+, the CAP_BPF and CAP_PERFMON capabilities were split from CAP_SYS_ADMIN to enable least-privilege eBPF agents. However, many production environments still run eBPF agents with CAP_SYS_ADMIN for compatibility. Always scope capabilities to the minimum required: CAP_BPF (load programs + create maps), CAP_PERFMON (attach to perf events and kprobes), and NET_ADMIN (attach XDP and TC programs). Never give your eBPF agent full CAP_SYS_ADMIN in a Kubernetes DaemonSet without explicit justification and a security review.
11. eBPF vs Traditional Kernel Modules: An Honest Comparison
eBPF is often described as a replacement for kernel modules, but that's an oversimplification. Both have valid use cases. Here's an honest engineering comparison to help you decide which to reach for:
| Dimension | eBPF Program | Kernel Module (.ko) |
|---|---|---|
| Safety | Verifier prevents crashes, OOB accesses, infinite loops | Zero guarantees — one bad ptr write panics the machine |
| Kernel API access | Restricted to approved helper functions (~200 helpers) | Full access to all exported kernel symbols |
| Portability | CO-RE: compile once, run on any kernel with BTF | Must recompile for every kernel version |
| Deployment | No reboot — load/unload at runtime in milliseconds | No reboot required, but modprobe is privileged |
| Performance | JIT-compiled — effectively native speed | Native speed (C code compiled directly) |
| Custom kernel I/O drivers | Not possible — no DMA, IRQ registration, or device I/O | Yes — full driver development possible |
| Observability | Excellent — attaches to any event dynamically | Possible but requires manual hook implementation |
| Ideal use case | Observability, networking policy, security enforcement | Hardware drivers, new filesystem implementations, FUSE alternatives |
The summary: reach for eBPF when you need to observe or filter kernel events. Reach for a kernel module when you need to extend kernel capabilities with new hardware drivers, custom filesystems, or functionality that requires kernel APIs the eBPF helper set doesn't expose. For almost all cloud-native infrastructure use cases (observability, networking, security), eBPF is the correct choice.
Developer Pitfall — eBPF Cannot Replace Everything a Kernel Module Can Do:
eBPF programs cannot register interrupt handlers, perform DMA, implement a character or block device, implement a network protocol stack, or call arbitrary unexported kernel functions. If your use case requires any of these, you need a kernel module. A common mistake is trying to use eBPF to implement functionality that fundamentally requires driver-level kernel access — the verifier's helper restriction will block you at compilation, and the attempt will be a dead end. Audit your requirements against the full list of eBPF helpers in linux/bpf.h before committing to eBPF.
12. Frequently Asked Questions
Q1: Can eBPF programs panic the kernel?
A correctly verified eBPF program cannot panic the Linux kernel — the verifier's safety guarantees cover out-of-bounds memory access, null pointer dereferences, uninitialized register reads, and infinite loops. However, there have been verifier bugs (CVE-level vulnerabilities) where programs that should have been rejected were incorrectly allowed, leading to kernel exploits. The Linux kernel security team treats verifier bypasses as critical vulnerabilities. In practice, for non-adversarial production use, verified eBPF programs are significantly safer than kernel modules. Always run the latest kernel with security patches applied, and use seccomp to restrict who can call the bpf() syscall in your environment.
Q2: What is the actual overhead of an eBPF kprobe on a frequently called syscall?
A minimal eBPF kprobe program (get timestamp, store to map, return) adds approximately 50–200 nanoseconds per invocation on modern x86_64 hardware after JIT compilation. For a syscall called 100,000 times per second, this is 5–20ms of added CPU time per second per core — roughly 0.5–2% of a core's capacity. However, kprobes on very hot syscalls like write() or futex() (millions of calls/second) can add measurable CPU overhead. Prefer tracepoints (lower overhead than kprobes) and per-CPU maps (eliminates map lock contention) for high-frequency syscall tracing. Always measure overhead in a staging environment before deploying to production.
Q3: How does Cilium use eBPF to replace kube-proxy?
Traditional kube-proxy implements Kubernetes Service IP routing using iptables or IPVS rules — updating tens of thousands of rules every time a Pod endpoint changes. Cilium replaces this with eBPF TC (Traffic Control) and socket-level programs that implement service load balancing directly in the kernel's socket connect path. When a Pod connects to a Service ClusterIP, an eBPF sock_ops program intercepts the connect() call and rewrites the destination address to a healthy backend Pod IP — before the packet ever enters the network stack. This eliminates iptables rule chains entirely, reducing Service update propagation time from seconds (iptables) to milliseconds (eBPF map updates) and reducing per-packet latency by avoiding iptables traversal overhead.
Q4: What is a tail call in eBPF and when should I use it?
An eBPF tail call (bpf_tail_call(ctx, &prog_array_map, index)) transfers execution from the current eBPF program to another eBPF program stored in a BPF_MAP_TYPE_PROG_ARRAY, replacing the current stack frame rather than adding to it. This is eBPF's answer to the 512-byte stack limit — complex programs that would require more stack space are split into a chain of smaller programs linked by tail calls. Tail calls are also useful for protocol dispatching: an XDP program can parse the outer Ethernet/IP header, look up the inner protocol in a prog_array, and tail-call the appropriate protocol handler. The limit is 32 consecutive tail calls to prevent infinite chains; the verifier enforces this at load time.
Q5: Is eBPF available on Windows?
Yes — Microsoft's eBPF for Windows project (github.com/microsoft/ebpf-for-windows) implements the eBPF instruction set, verifier, and a subset of Linux eBPF program types (XDP and socket filter) on top of Windows kernel extension mechanisms. It uses PREVAIL (a sound polynomial-time verifier) rather than the Linux verifier, and programs run via LLVM JIT or as native Windows driver extensions. The Windows implementation supports a growing subset of Linux eBPF helpers and map types. The primary use case is network filtering and security policy — the same bpftrace and Cilium tooling cannot be directly ported, but tools that compile to eBPF bytecode can target Windows with appropriate framework changes.
Q6: How do I debug an eBPF verifier rejection?
The verifier produces a detailed rejection log that describes the exact instruction that failed verification, the register state at that point, and the reason for rejection. To see this log, call bpf(BPF_PROG_LOAD, ...) with a non-null log_buf and sufficient log_size (1MB is a good starting size), then print the buffer on EACCES/EINVAL. When using libbpf, set the environment variable LIBBPF_LOG_LEVEL=debug or use libbpf_set_print(libbpf_print_fn). The most common rejection patterns are: uninitialized register reads (use __builtin_memset() to zero structs), missing NULL checks after bpf_map_lookup_elem, and pointer arithmetic that the verifier cannot bound (add explicit size checks).
Q7: What is the minimum Linux kernel version required for a useful eBPF development environment?
The practical minimum for modern eBPF development is Linux 5.8, which introduced BPF_MAP_TYPE_RINGBUF (the preferred event delivery mechanism), CAP_BPF/CAP_PERFMON capability splitting, and BTF improvements. For CO-RE (the portability framework), 5.4 is the minimum. For LSM eBPF hooks (runtime security), you need 5.7+. For BPF_PROG_TYPE_FENTRY/FEXIT (fast BTF-typed kernel function probes), you need 5.5+. Most production Linux distributions ship 5.15 LTS or newer (Ubuntu 22.04, RHEL 9, Amazon Linux 2023), making all of these features available. Avoid developing on kernels older than 5.4 — you'll be working around too many missing features.
Q8: Can eBPF programs make network connections or write to disk?
No — eBPF programs cannot initiate network connections, open files, write to disk, or perform any blocking I/O. This is by design: the verifier prohibits calls to arbitrary kernel functions, and all approved helper functions are non-blocking. The correct architecture for eBPF-based monitoring is: the eBPF program writes events to a ring buffer or perf event array, and a separate userspace process reads from that buffer and is responsible for all I/O (writing to databases, sending to Kafka, posting to APIs). Attempting to perform I/O from within an eBPF program is a common mistake by developers coming from a traditional kernel module background, and the verifier will reject the program at load time.
Q9: What is the difference between bpftrace, BCC, and libbpf?
bpftrace is a high-level scripting language for eBPF — ideal for rapid ad-hoc investigation (one-liners for tracing latency, counting events, printing stack traces). It compiles bpftrace scripts to eBPF bytecode at runtime via LLVM and loads them immediately. BCC (BPF Compiler Collection) provides Python and Lua bindings that let you embed eBPF C code as strings in a Python program, compile it at runtime, and interact with maps via Python objects — good for moderate-complexity tooling but requires Clang/LLVM at runtime on the target host. libbpf is the low-level C library used by the kernel itself — you compile eBPF C code offline, use CO-RE for portability, and distribute a self-contained binary with no runtime compiler dependency. For production deployments, libbpf + CO-RE is the professional standard; bpftrace is the first-responder debugging tool.
Q10: How does eBPF interact with container and Kubernetes isolation (namespaces, cgroups)?
eBPF programs run in the host kernel's namespace — they have visibility into all containers and Pods on the node, regardless of network namespace or PID namespace isolation. This is simultaneously their greatest observability strength and a key security consideration. An eBPF agent with CAP_BPF can read memory from any process on the node, trace any container's syscalls, and inspect all inter-pod network traffic. This is why tools like Tetragon and Falco can provide host-wide runtime security: they observe everything. From a security-in-depth perspective, you should audit what capabilities you grant eBPF DaemonSets, restrict who can load eBPF programs (via seccomp, syscall filtering), and be aware that a compromised eBPF agent has broad visibility into all tenant workloads on the node.