Introduction The story began with a tweet about a new eBPF-based Security Observability and Runtime Enforcement solution, named Tetragon, posted by the CTO of the company (Isovalent) that created it. With a very cute logo, eBPF, Kubernetes, Linux kernel runtime and real-time security enforcer, capable of hooking into all layers of the operating system and application stack, the solution seemed to hit the ground running right from the start. All the features the tool provides can be found at its website: [1]. The "transparent security observability across the stack from the lower level up into the applications. File access, networking, storage, syscalls, escalations, function tracers, ..." sounds interesting, but what really caught our eye was the enforcement capabilities. On the website, in the section "Automatic Mitigation of Privilege & Container Escapes", the authors claim that "Tetragon adds the ability to prevent privilege, capability, and namespace escalations in the kernel by detecting them and stopping the involved processes". To demonstrate its effectiveness, the authors show a portion of an apparently simple policy that is supposed to automatically detect a capability change to CAP_SYS_ADMIN and kill the responsible process. As a worthy opponent, an exploit (authored by theflow) for the CVE-2021-22555 vulnerability [5] (Netfilter bug leading to the privilege escalation, more about the bug can be found in an excellent write up at [7]) in the Linux kernel had been selected. The original version of the Tetragon blog post showed how the exploit process gets killed upon executing the execve() system call with the escalated privilege, which was quickly pointed out to be far too late [2]. Now, the authors claim to be using a similar policy attached to all system calls, in order to catch and kill the exploit earlier. Why the change? We will discuss it in the next section. Regardless of the exact policy and exact moment of detecting the escalation, the exploit process gets ultimately killed. That is excellent, right!? Well, there is a small (read: huge) caveat, that authors of the Tetragon blog post perhaps unknowingly admit: "The process was killed right when the vulnerability was exploited to escalate privileges". "vulnerability was exploited"?! This can't be good. Why it can't work To quickly recap the situation: in attempting to mitigate container escapes, Tetragon tries using advanced Linux kernel features like eBPF and kprobes not to protect the very same kernel from getting exploited, but instead to stop an already successful exploit from using its gains. Going back to security fundamentals, this approach is simply infeasible: post-exploitation detection/mitigation is at the mercy of an exploit writer putting little to no effort into avoiding tripping these detection mechanisms. To help illustrate this point, it helps to think in terms of the graphic below: At point 1, a defense employs methods like attack surface reduction to prevent a vulnerability from being reached in the first place. This is highly (even perfectly) effective where it is possible. Once a vulnerability can be triggered, however, an attacker will invoke a series of steps in order to achieve both greater reliability and greater control over the vulnerability. The initial steps (points 2 and 3) are precarious and essential: disrupting these either require significant reworking of the exploit or renders it infeasible (due to a high probability of detection), impossible without an additional vulnerability, or simply impossible. As illustrated in the graphic, the further along the exploit is able to operate in achieving that reliability and control, the more possibilities (size of the circles in the graphic) open up for it at a lower marginal cost to the attacker. In the case of the exploit used for Tetragon's demo in its blog, the exploit was able to achieve ROP, allowing it to execute any code in the kernel, modify any memory -- the possibilities are virtually endless. The attacker is in full control of the environment in which Tetragon attempts to perform its enforcement actions. The core message here is: Tetragon (in its aspect of post-exploitation container escape mitigation) simply tries to address the problem too late. Post-exploitation "effectiveness" Tetragon aims in its container escape defense for a post-exploitation "mitigation". In order to be even slightly successful without strong and comprehensive pre-exploitation defenses/hardening, it has to adhere to certain principles. We can evaluate Tetragon's effectiveness here by looking at modern principles for its closest post-exploitation comparison: integrity checking / anti-persistence. First, one can only expect some guarantees from a higher privilege level component monitoring or enforcing policies for a lower privilege level component (for example: OS kernel vs user-land programs). Let's call it "privilege domain separation". Second, "Nemo iudex in causa sua": there must be role separation between a monitoring/enforcing component and the one being watched. One cannot expect any reasonable guarantees from a component monitoring or enforcing policies on itself, regardless of the component's privilege level, once an attacker has significant control over that component. How does Tetragon adhere to the basic principles (in the example above specifically)? Tetragon tries to use Linux kernel privilege level features (eBPF/kprobes) to enforce security policies on... the Linux kernel. Quite immediately this creates a connotation with another realm of security software: old-style Antivirus (AV). What Tetragon claims to do is no different from an AV vendor with only a userland hook library injected into malware processes, claiming to be able to detect or stop malware, when the malware gets to run first. Simply, after compromising the core controller, using that (already compromised!) controller's features to mitigate the already successful attack just can't work out well. Furthermore, it does not matter how fine-grained the applied policy might be, it cannot reliably mitigate against a kernel memory corruption bug and all the opportunities it gives to the attacker. To quote Mathias Krause: "When you give control to the weird machine, it is game over". An attacker can use all sorts of evading/bypassing techniques. Some of them might become a standard part of any exploit by default. Let's take a look at the original exploit used in the Tetragon demonstration. By default, it avoids very effectively a handful of popular mitigations: W^X - avoids by using ROP SMEP - avoids by using ROP SMAP - avoids by using data in sprayed kernel-land objects only Knowing if the mitigation is or is not enabled oftentimes does not make any big difference. An attacker can simply assume it is there and accommodate the exploit to bypass it just in case. For instance, in the example exploit, one doesn't need to actually confirm the presence of an SMAP-capable CPU before using a technique that avoids tripping over SMAP. The same is the case for Tetragon. There is however a significant difference between the mitigations listed above and Tetragon. Ideally, useful pre-exploitation mitigations should require, at minimum, significant reworking of exploits and not allow an attacker to circumvent them by generally-applicable additions to an exploit library. They stand in the way of successful or reliable exploitation before or while it occurs. Tetragon however, does not prevent the exploitation at all (it does not make the W^X, SMEP nor SMAP bypasses any harder), it lets it happen and hopes to be able to detect the fallout. It just cannot do that very well (or at all to be frank), because at this point it is fully at the mercy of the attacker, who can quite easily (deliberately or accidently) wipe out all Tetragon's capabilities right away. Since the attacker can, the attacker will. Mitigation side-effects Speaking of the fine-grained policy: additional hooks, checks, probes and the like come at a cost. Adding more and more is not free, as the performance hit