Kernel‑Level Ground Truth: Why eBPF Is Replacing User‑Space Agents for Security Observability

What changed

Traditional Kubernetes security stacks rely on sidecar containers or DaemonSets that run in user space alongside the workloads they protect. Because the agent and the protected process share the same privilege level, a compromised container can simply kill the agent, truncate its logs, or bypass file‑based monitors. The result is a blind spot that attackers exploit to hide file‑less payloads, process injection and metadata exfiltration.

eBPF flips this model. By attaching sandboxed programs to kernel‑level tracepoints and syscalls, observability lives where a container‑level attacker cannot reach it. Disabling a probe requires a host‑kernel escape, a far more difficult problem than sending kill -9. The shift also eliminates the repeated user‑kernel context switches that dominate CPU usage in classic agents.

Provider comparison

Feature	Falco (CNCF)	Tetragon (Cilium)	Sysdig Secure (commercial)	Datadog Security (commercial)
Primary mode	Observation‑only, optional enforcement via bpf_send_signal	Full enforcement, tight Cilium integration	Observation + enforcement, UI‑driven policy editor	Observation + enforcement, SaaS analytics layer
Rule language	YAML, MITRE ATT&CK mapping	YAML + CiliumNetworkPolicy syntax	Policy DSL, GUI builder	Rule engine with templating
Kernel version support	4.15+ (full feature set from 5.2)	5.4+ (requires BTF)	5.4+ (bundled driver)	5.6+ (managed agents)
CPU impact (reported)	60‑80 % lower than multi‑agent stacks	65 % lower, micro‑second enforcement latency	55 % lower, depends on rule complexity	50‑70 % lower, SaaS overhead varies
Telemetry volume	In‑kernel filtering, only relevant events shipped	Same, plus optional Cilium flow logs	Centralized ingestion, higher raw volume	Centralized ingestion, optional edge filtering
Ease of deployment	Helm chart, privileged DaemonSet, no kernel code required	DaemonSet, requires Cilium CNI, privileged mode	Managed SaaS agent, requires license	Managed agent, requires API key
Cost model	Free, open source; operational cost is staff time	Free, open source; operational cost similar to Falco	License per node, includes SIEM integration	Subscription per host, includes analytics platform

Why the open‑source options matter

Both Falco and Tetragon run without a license fee, which makes them attractive for organizations that already operate a CNCF‑centric stack. Falco offers a mature rule set that maps directly to MITRE ATT&CK techniques, simplifying the jump from threat modeling to detection. Tetragon, being a sub‑project of Cilium, can enforce policies synchronously in the kernel, giving sub‑millisecond response times for critical blocks such as metadata‑service access.

Commercial vendors have re‑implemented their agents on top of eBPF, but they still charge per‑node or per‑GB of ingested telemetry. The primary advantage of those services is a polished UI, integrated threat intel and managed rule updates. For teams that already have a DevOps platform and want to avoid additional SaaS spend, the open‑source stack provides comparable security coverage with a fraction of the recurring cost.

Business impact

CPU and cost savings

Clusters that migrated a traditional stack (multiple sidecar agents, log shippers, network proxies) to a single eBPF‑based agent reported 60‑80 % reduction in security‑related CPU consumption. On a 100‑node production fleet, that translates to roughly 200 CPU‑core hours saved per day, which can be re‑allocated to workload capacity or result in lower cloud billings.

Because filtering occurs in the kernel, only events that match a rule leave the node. Teams typically see a 70‑90 % drop in log volume sent to the SIEM, cutting ingestion fees that are often billed per‑gigabyte.

Faster incident response

When enforcement is enabled, the kernel can abort a malicious syscall before it completes. A connect attempt to the cloud metadata endpoint can be blocked in microseconds, preventing credential leakage that would otherwise require a multi‑minute forensic investigation.

Risk reduction

Since the eBPF probe survives a container‑level compromise, attackers cannot simply kill the monitoring component. The only way to tamper with the probe is to break out of the container and gain host‑kernel access, a scenario that most threat models already consider a high‑severity breach. This raises the bar for adversaries and reduces the attack surface of the observability layer itself.

Rolling out eBPF safely

1. Phase 1 – Observe: Deploy Falco or Tetragon as a privileged DaemonSet in passive mode. Capture all syscalls, stream events to cheap storage, and build a baseline of normal process, file and network activity.
2. Phase 2 – Alert: Write detection rules against the baseline. Start with high‑severity patterns (e.g., unexpected binaries, outbound traffic to 169.254.169.254). Tune false positives until the alert volume is manageable.
3. Phase 3 – Enforce: Enable bpf_send_signal or Cilium LSM hooks to kill offending processes. Only enable enforcement after confidence in rule accuracy is high; a mis‑fire can cause a production outage.

Security hardening of the eBPF deployment

• Grant the container only the capabilities it truly needs: CAP_BPF, CAP_PERFMON, CAP_SYS_RESOURCE.
• Use admission controllers (OPA Gatekeeper, Kyverno) to restrict privileged pods to a dedicated namespace.
• Pin agent images to immutable digests.
• Monitor the namespace for unauthorized changes.

Choosing the right tool for your organization

Situation	Recommended tool
Existing CNCF stack, need quick start, no enforcement required	Falco – Helm chart, rich rule set, free
Already using Cilium CNI, need kernel‑level enforcement	Tetragon – Tight Cilium integration, sub‑millisecond blocking
Preference for SaaS UI, integrated threat intel, willing to pay per node	Sysdig Secure or Datadog Security – Managed agents built on eBPF
Highly regulated environment, need formal support contracts	Commercial vendor with SLA (Sysdig, Datadog)

Bottom line

Application‑level logs remain essential for debugging, but for security you need a telemetry source that cannot be silenced from inside a compromised container. eBPF provides immutable, low‑overhead visibility at the syscall boundary, cuts CPU and data‑ingestion costs, and enables rapid enforcement. Start with observation, mature your rule set, then move to enforcement – and you will close the blind spot that traditional user‑space agents leave wide open.

---

For a hands‑on trial, deploy the Falco Helm chart in observation mode on a staging cluster and compare the volume of events captured to your existing log shipper. The contrast will make the case for eBPF clear.