It’s time to return to the continuation of the series of articles devoted to Polish projects in the security industry. In this article we will focus on the Linux Runtime Kernel Guard solution, which is gaining more and more recognition from industry experts with each release. It is a significant obstacle to, for example, Metasploit, thereby making it more difficult to exploit Linux.
What is LKRG
LKRG is a Linux kernel module that is responsible for protecting the kernel from vulnerability exploitation techniques and from rootkits. Its function is to verify the integrity of the kernel in the mode of informing about suspicious anomalies or blocking them when detected. Its effectiveness is confirmed by the fact that it has been recommended for use to protect against vulnerabilities by e.g. CISA (the U.S. CyberSecurity and Infrastructure Security Agency) and Red Canary. You can also find scientific studies on the web that take up the aspect of using LKRG in specific environments, e.g. TEE. Also not to be overlooked is the study by Alexander Popov, who spent some time and work to get around LKRG.
Linux Kernel Runtime Guard is usable in virtually every Linux distribution, some of them deliver LKRG in their repositories (e.g Rocky Linux, Gentoo, Arch, NixOS, OpenBMC, Yocto and others), and in some it is installed natively e.g. Whonix (known as TorBox) or Kicksecure.
Who creates LKRG?
The LKRG was created by Adam “pi3” Zabrocki, known to our readers, who is a well-known IT security expert in the industry. Among other things, Adam discovered bugs in RISC-V architecture, KVM hypervisors, Hyper-V, Intel microcode, Linux kernel, OpenSSH or Apache software and others. He is also the author of an article on exploitation and circumvention of various mitigation mechanisms in Phrack magazine, a speaker at conferences such as BlackHat, DefCon, and has been nominated twice for the Pwnie Award. He currently works at NVIDIA as “Director of Offensive Security” and is working on new ISA extensions to improve the security of RISC-V processors while being Vice Chair of the J-ext working group for that architecture. You can hear more about the author in an episode of Polish podcast “Rozmowa Kontrolowana”.
Solar Designer, a legend of the security scene and the creator of the John the Ripper software and the Openwall project, also takes a leadership role in the project. He has developed several well-known exploitation techniques like the “return-to-libc attack.” In 2009, he was awarded the Pwnie Award in the Lifetime Achievement Award category at the Black Hat conference.
How can I use LKRG in practice?
Instructions on how to install, run and configure LKRG are available, for example, in the project’s repository on Github. However, before you start testing or using LKRG in practice, I highly recommend reading Adam’s and Solar’s answers and then scrolling through the documentation.
LKRG is not an invasive solution, but its effectiveness is based on adjusting the configuration for the environment in which it is intended to work. We therefore recommend familiarizing yourself with the “theory” before using it for the first time.
What’s up with the project?
I decided to ask project creator Adam, as well as project leader Solara, about the development of LKRG. Enjoy your reading!
Adwersarz.pl: How do you assess the current development of LKRG – are you at a stage that satisfies you? Does the project meet your original assumptions?
Adam: LKRG is at the stage where it is stable enough to be used but we are not feeling comfortable to release version 1.0 yet. There are a few areas which need improvement and some parts of the code could be written in a more stable and elegant way. Some of the features that we plan to add perhaps should be part of the 1.0 release but we haven’t decided on this yet.
As to the “assumptions”, they are changing over time. Some of them are certainly met, some of them are not.
What successes can LKRG already boast about in your opinion?
Adam: LKRG is certainly used in production in many places as well as being explored in others. Personally, I think it’s a big success that this project is running and protecting production servers (not a single server but sometimes data centers).
Moreover, people and the community recognize the project. It made a stamp in the community and it is being taken into account during evaluation of Linux security. I’m aware of various university papers which reference LKRG, some people made their thesis evaluating or comparing LKRG to other solutions. Additionally, LKRG has been researched by well-known security researchers and some exploitation frameworks warn or bail out when LKRG is detected. Taking all of this into account, I would assume it has at least moderate success 🙂
Should LKRG be launched by default in the most popular distributions, intended for both desktops and servers? Until now, I saw that you have recommended the use of LKRG to individuals and distributions strongly associated with security and privacy.
Adam, Solar: We didn’t change our opinion here. Generally, LKRG is not intended to run on Linux desktops (although I’m using it on my desktop and my friends who run Linux on desktop use it too – Adam). There isn’t much local security on typical Linux desktop systems/distros anyway, in the way those systems are configured and used, irrespective of any locally exploitable Linux kernel vulnerabilities. Thus, adding LKRG would result in inconsistent security. Specifically, on a typical single-user Linux desktop system there’s that one user account that uses “sudo” to access root, so it is effectively root-equivalent (only different from root in terms of safety from mistakes, not really in terms of security). LKRG does provide some relevant security hardening to such systems anyway – it might detect a web browser vulnerability exploit trying to escape the web browser’s sandbox – so it could help e.g. a vulnerable system withstand web browser attacks in the Pwn2Own contest. However, that’s not how most end-user systems are actually compromised (when they are).
A security enthusiast can very well reasonably use LKRG on their desktop system, though, primarily for consistency with their other systems, and for advance testing of kernel+LKRG combinations prior to deployment on the servers the person manages.
Nevertheless, we generally think it should be beneficial to run LKRG on various Internet-facing servers. LKRG helps on systems where there are multiple users (or at least pseudo-users for multiple services), and especially where kernels might not be updated or rebooted into in a timely manner after new vulnerabilities are discovered.
Since you started the project, have you ever received information from users that LKRG saved them from a serious attack?
Adam, Solar: Many users reported LKRG blocking exploit programs and container escapes during testing or with attack simulation. We’re not currently aware of cases of it blocking real attacks. We suspect this is for several reasons including Linux system compromises not being that frequent (especially where someone cared to deploy LKRG and thus probably also cared to secure the system in other ways), and log monitoring being rare.
Further, during LKRG development we discovered multiple issues in the Linux kernel. In 2020, when Adam was actively adding new LKRG features, he found 5+ Linux kernel bugs several of which were vulnerabilities and got CVEs assigned.
LKRG has an active community – have you already received any interesting information about unusual applications of LKRG in practice or its use in unusual environments?
Adam: Certainly we are aware of people deploying LKRG in IoT devices / products sold in the Chinese market. Some of them are integrating LKRG with Android (which is not our standard use case) compiled with LLVM (instead of gcc).
Adam, Solar: Some people were very interested in running LKRG under realtime (RT) Linux kernel builds on their desktop systems, suggesting we could want to support those kernels also for medical devices. RT kernels would be non-trivial for us to support and the medical use case is inconsistent with LKRG favoring security over safety. So at least for now we instead added (controversial) guards to prevent compiling LKRG against RT kernels. We could reconsider RT kernel support later.
Does Metasploit still refrain from exploiting when it detects LKRG on the targeted system, or has it gotten better at handling it?
Adam: It is left to the individual exploit developer. If an exploit ignores such an event from the Metasploit framework it’s unlikely to succeed.
How do you evaluate the currently available methods for matching the security protections provided by LKRG?
Adam: To fully answer that question I feel obligated to answer a bit different question. Why did we develop LKRG?
The idea of writing something like LKRG had been growing in my mind for quite a long time. The first serious attempts to write something similar date back to 2009 and my work at CERN. However, the decision to write the first line of LKRG code with its current functionality (more advanced compared to 2009) I made in 2015. The first version of LKRG (0.0) was released in January 2018 under the Openwall umbrella.
As you can see, I was struggling with myself as to whether there was a need for such a project. It took me some time to convince myself that it’s something important and needed. The answer to the question “why did I write LKRG” is the same as what arguments convinced me to pull the trigger and create LKRG. There are several of them:
- I’ve been an Offensive Security Engineer focusing on rootkits and exploits for some time (since 2003/2004). I’ve realized that in principle many of the techniques of “infecting” the Linux kernel did not change at all for the last 20+ years. The same applies to many kernel exploitation techniques.
- Other OSes have been facing similar problems as well. However, they tried to address them and some of the solutions have been developed. Those are not perfect, there are some ways to bypass them (although it’s not cheap), but it’s better than being open to the same 20+ years old techniques.
- Over the years, I’ve learned that even if some of the protections are imperfect, they can significantly hurt the reliability of exploitation or make it harder. In most cases, this is good enough to stop mass attacks – those not targeting specific users – and thus protect the users.
- I’ve also learned that most of the time people who write rootkits or exploits are not the same people who use them. If during an attack the latter people face non-trivial protection or an exploit is not reliable enough (e.g., due to LKRG), they won’t be able to or won’t bother to adjust the tools and will move on to another target.
- However, even if they are able and willing to adjust the tools, it might be already too late since LKRG will take appropriate action (e.g., panic the kernel), which will be visible to the owner of the system.
It is worth adding that as far as I’m aware, all of the anti-exploitation / security technologies (including mitigations) follow the same path and are generally bypassable, e.g.:
- Firewalls are commonly trivial to bypass, but you still use them
- Antiviruses are trivial to bypass, but people pay money for them
- Mitigations:
- ASLR and especially KASLR are easy to bypass, but people still want to have them enabled in the system
- Non-executable memory is easy to bypass, but people still want to have it enabled in the system
- SMEP is easy to bypass, but people still want to have it enabled in the system
- Stack cookies are bypassable but people still compile binaries with them
- RelRO are bypassable but people still enable that flag
- Any anti-threat technology (e.g., Capsule8, Crowdstrike, WDATP, and many others) can be bypassed or completely disabled (some of them run in user mode), but people pay a lot of money to use them
LKRG is no different here, except for the “price” part, since many of these technologies are not available for free.
Taking all these into account, I came to the conclusion that LKRG can improve the Linux ecosystem and bring useful security functionalities that everyone can benefit from. It will also bring some of the “comparable functionality” from other OSes into the Linux world, like Patch Guard, Advanced Threat Protection (ATP), monitoring, and more:
– https://en.wikipedia.org/wiki/Kernel_Patch_Protection
– https://digitalguardian.com/blog/what-advanced-threat-protection-atp
In short, LKRG should not be seen as some self-contained protection layer but rather as an additional security layer which can be added on the top of existing one, not replacing it.
In 2020, Michael Larabel conducted interesting performance tests of LKRG, indicating a potential 5% performance loss in systems running LKRG. Have you implemented any optimizations since then, or do you believe that such a result is acceptable?
Adam, Solar: The Phoronix project and Michael himself did a great job evaluating 2 different versions of LKRG. We implemented optimizations between these 2 versions.
Performance impact is a controversial topic because it really depends on the use case and on the environment where LKRG is deployed.
On average, with default configuration, LKRG performance cost is around 2.5% (which was confirmed by Phoronix tests). However, in some use cases where new tasks and temporary files are frequently created and removed, the performance impact is moderately higher (and was a lot higher prior to the mentioned optimizations). In some synthetic benchmarks repeating only a certain most-affected operation in a tight loop, the performance impact could look like it’s a disaster (even twice slower), but this does not reflect overall impact in real-world use cases.
Surprisingly to some, a few benchmarks tend to receive a slight performance boost under LKRG – that is, LKRG kind of makes them faster. It is difficult to figure out exactly why. This may indicate that LKRG’s typical performance impact (at least for those use cases) is within the usual variance in system performance seen between otherwise similar system configurations, software builds, system reboots, etc. – or in our case loading of LKRG vs. not. Such variance almost always exists e.g. due to relative timing of events, ASLR, and CPU cache aliasing limitations.
With the increasing use of technologies like Docker, Kubernetes, and similar solutions, do you believe LKRG will be excellent support for environments with such solutions?
Adam: Yes, LKRG can be beneficial in any environment with various isolation layers which needs to be protected and hardened. Namespaces are a great example of that and some of the attacks that were blocked in users’ simulations (mentioned in one of the previous answers) were exactly the attempts of Docker escapes through the UMH interface, which LKRG locks down. The same applies to sandboxes too (e.g., based on seccomp).
Are you considering implementing dedicated support for monitoring services (e.g., SIEM), which could aggregate alerts from LKRG to warn of anomalies?
Solar: Yes. With sponsorship from Binarly, we’ve implemented LKRG built-in remote logging of kernel messages, including LKRG’s own alerts, over a simple transport security layer. This can become part of a SIEM/EDR solution, although we do not currently have our own plans for that.
– https://www.openwall.com/presentations/BSidesZagreb2024-Linux-remote-logging/
Have you encountered any instances where LKRG caused problems in someone’s infrastructure and led to issues?
Solar, Adam: We’ve had some people report false positive detection of “attacks”, which was caused by incompatibility of the given versions/builds of LKRG and the kernel. The potential of such compatibility issues is why on a new system we recommend testing LKRG with its enforcement actions disabled, and only enabling it fully after having confirmed that no false positives occur for a little while. LKRG documentation includes specific instructions for this.
We are also aware that LKRG may cause unacceptable performance impact in some unintended environments, such as with real-time audio or video processing on a desktop system, which is a more latency-sensitive use case than the typical server workloads that LKRG is intended for.
Other than that, there are generally no issues.
Have you received any offers from companies, corporations, or other third parties who liked LKRG and wanted to support the project’s development?
Adam, Solar: Yes, such as from Binarly as mentioned above. Also, CIQ supported the packaging of LKRG for Rocky Linux (and Enterprise Linux distros derived from Red Hat’s in general) via SIG/Security, and the public release and talk on the remote logging functionality referenced above.
Besides yourselves, is anyone else involved in the development of the project?
Adam: Yes, during various stages of the project there were different people involved in LKRG development. Other than me and Alexander, I’d like to mention Mariusz Zaborski (oshogbo), Vitaly Chikunov, Jakub ‘mrl5’ Kołodziejczak, Mikhail Morfikov, and more.
As of today, GitHub lists 17 total LKRG contributors: https://github.com/lkrg-org/lkrg/graphs/contributors
What are the next plans for the development of LKRG?
Solar: The 1.0 release will be an important milestone, not so much in terms of improvements over the current 0.9.8, but hopefully in terms of how LKRG is viewed by others. We may gain not only more users, but also more contributors. This may in turn affect the development pace and focus.
Ideally, we’d rework the LKRG source code and tree in certain major ways (such as making the coding style similar to Linux kernel’s), but it is one of those things that keeps getting mostly postponed.
We’ve been planning to have LKRG try and hide its own and the kernel’s build specifics, to make it harder for exploits to reliably target LKRG and the kernel. This is something Ilya V. Matveychikov has also worked on and demonstrated in his own Kiddy project. Ilya and independently also Alexander Popov had previously demonstrated attacks on LKRG that relied on its exposure, which such changes would reduce.
We could also try and reduce the kernel’s attack surface, with extra knobs to control that.
We could even start detecting and blocking certain userspace attacks that rely on malicious input passing through the kernel (primarily through the execve syscall).
Adam: We were also working on hardening the (e)BPF interface, which may be released or not. 🙂
Summary
We thank Adam and Solar very much for their answers, and encourage readers to test and implement LKRG in their environments – it can be a valuable reinforcement of your infrastructure against attacks!