ebpf-observability skill by bagelhole

eBPF Observability

eBPF (extended Berkeley Packet Filter) allows you to run sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. This skill covers using eBPF for deep observability, network monitoring, and security enforcement across cloud-native infrastructure.

1. When to Use

Use eBPF-based observability when you need:

Deep performance debugging -- trace kernel-level latency, syscall overhead, and scheduling delays that application-level metrics cannot reveal.
Network observability without sidecars -- capture L3/L4/L7 flows, DNS queries, and TCP state transitions directly from the kernel, eliminating the CPU and memory overhead of sidecar proxies.
Security monitoring at the kernel boundary -- detect container escapes, unexpected process execution, sensitive file access, and anomalous syscall patterns in real time.
Continuous profiling in production -- generate CPU flame graphs and memory allocation profiles with negligible overhead (typically under 1% CPU).
Service mesh replacement or augmentation -- Cilium can replace kube-proxy and provide identity-aware network policies enforced at the kernel level.

Avoid eBPF when your kernel version is below 4.19, when you are running on managed platforms that restrict BPF capabilities, or when your debugging needs are fully met by application-level tracing.

2. Prerequisites

Kernel Version Requirements

Feature	Minimum Kernel	Recommended Kernel
Basic BPF maps & probes	4.9	5.10+
BPF CO-RE (BTF support)	5.2	5.10+
BPF ring buffer	5.8	5.10+
BPF LSM hooks	5.7	5.15+
Cilium full features	4.19	5.10+
Tetragon	4.19	5.13+

Verify Kernel Support

# Check kernel version
uname -r
 
# Verify BTF (BPF Type Format) is enabled -- required for CO-RE
ls /sys/kernel/btf/vmlinux
 
# Check BPF filesystem is mounted
mount | grep

Install Toolchain

3. Cilium Setup

Cilium replaces kube-proxy with eBPF-based networking, providing identity-aware security and deep network observability via Hubble.

Install Cilium on Kubernetes

Install the Cilium CLI and Hubble CLI

Hubble Network Observability

Hubble UI Access

4. Tetragon for Security

Tetragon is Cilium's runtime security enforcement engine. It uses eBPF to observe and enforce security policies at the kernel level with zero application changes.

Install Tetragon

Process Execution Monitoring

File Access Tracking

Network Connection Enforcement

Privileged Escalation Detection

5. bpftrace One-Liners

These are practical bpftrace commands you can run directly in production for targeted debugging.

Syscall Latency

DNS Tracing

TCP Retransmits

Disk I/O Latency

Container-Aware Tracing

6. Prometheus Integration

Hubble Metrics for Prometheus

Hubble automatically exposes Prometheus metrics when configured in the Cilium Helm install. Verify the metrics endpoint:

Create a ServiceMonitor for Prometheus Operator:

eBPF Exporter for Custom Kernel Metrics

Example ebpf_exporter config for tracking OOM kills and run queue latency:

Grafana Dashboard

Import these community dashboards for eBPF metrics:

Key Prometheus queries for eBPF-sourced metrics:

7. Network Observability

L3/L4 Flow Logging

L7 Protocol Visibility

Enable L7 visibility with Cilium annotations on target pods:

DNS Monitoring

Service Dependency Map Generation

Hubble UI automatically generates service maps. For programmatic access:

8. Security Monitoring

Detect Container Escapes

Unexpected Syscall Detection

File Integrity Monitoring

9. Performance Profiling

Continuous Profiling with Parca

Parca uses eBPF to collect CPU profiles continuously with minimal overhead.

Continuous Profiling with Pyroscope

CPU Flame Graphs with bpftrace

Off-CPU Analysis

Memory Leak Detection

10. Troubleshooting

Common eBPF Issues

BPF verifier rejects program:

BTF not available:

Permission denied:

Cilium pods not starting:

Tetragon events missing:

High overhead from eBPF programs:

Kernel Compatibility Matrix