Linux Rootkit Detection: Tools and Techniques for Server Security

Rootkits give attackers persistent, hidden access to Linux systems. They manipulate kernel operations, hide files and processes, and dodge standard security tools. Some have gone undetected for years. Detecting them requires a layered approach, because no single tool catches everything.

This post covers the main types of Linux rootkits, how to scan for them, and the advanced monitoring techniques that catch what basic scanners miss.

Types of Linux Rootkits

Rootkits operate at different privilege levels, and the deeper they sit, the harder they are to find.

User-mode rootkits run at the application level (Ring 3). They hijack dynamic linking using LD_PRELOAD to inject malicious libraries that intercept standard C library functions like readdir or fopen, hiding files and processes from userland tools.

Kernel-mode rootkits run at Ring 0 by loading as Loadable Kernel Modules (LKMs). They intercept system calls, manipulate kernel memory, and hide their own presence. Because they're tied to specific kernel versions, a misconfigured one can cause a kernel panic, which ironically exposes it.

eBPF-based rootkits exploit the Extended Berkeley Packet Filter subsystem to run in kernel space without loading a traditional module. They attach to syscall hooks, tracepoints, or LSM events. Standard LKM scanners won't see them. Boopkit is a well-known proof of concept that creates a covert C2 channel using this approach.

io_uring-based rootkits are the newest variant. They use the io_uring asynchronous I/O interface to batch operations, generating fewer observable syscall events. RingReaper, an experimental rootkit, demonstrated how this can silently replace calls like read, write, and connect while evading EDR tools.

Rootkit Type	Privilege Level	Hooking Method	Detection Difficulty
User-mode	Ring 3 (User)	LD_PRELOAD, library hijacking	Moderate
Kernel-mode	Ring 0 (Kernel)	Syscall table, LKM, inline hooking	High
eBPF-based	Ring 0 (Kernel)	eBPF program attachment	Very high
io_uring-based	User/Kernel	Asynchronous I/O batching	Very high

Scanning with chkrootkit and rkhunter

Two tools form the baseline for rootkit detection on Linux servers: chkrootkit for quick scans and rkhunter for deeper checks.

chkrootkit

chkrootkit is a shell script that checks critical system binaries (ls, ps, netstat, sshd, ifconfig) for signs of tampering. It also detects promiscuous network interfaces and deleted logs. As of version 0.59 (January 2026), it can identify over 75 rootkits, worms, and LKMs, including newer threats like Linux BPFDoor, Syslogk, and the XZ Backdoor.

Run it after any suspicious activity. Pay attention to warnings from the ifpromisc component and alerts about deleted lastlog or wtmp files.

rkhunter

rkhunter goes further. It compares SHA-1 hashes of system binaries against known-good values, monitors file permissions and hidden files, checks startup configurations, and performs kernel and network analysis.

Set it up properly from the start:

# Establish a baseline after a clean install or update
rkhunter --propupd
 
# Update rootkit definitions
rkhunter --update
 
# Run a full scan (use --novl on production servers to reduce I/O)
rkhunter --check --enable all

For automated daily scans, set CRON_DAILY_RUN="true" in /etc/rkhunter.conf and use --skip-keypress and --report-warnings-only for clean output. Review logs at /var/log/rkhunter.log and whitelist confirmed false positives.

Detecting LKM rootkits

LKM rootkits are particularly dangerous because they operate as kernel extensions, intercepting system calls and hiding processes at the kernel level. Standard tools like lsmod won't see them, but there are ways to spot them.

Compare lsmod output against /sys/module/ listings. Check system logs for suspicious kernel messages:

# Look for out-of-tree module warnings
sudo dmesg | grep "loading out-of-tree module taints kernel"
 
# Check for module verification failures
grep "module verification failed" /var/log/syslog
 
# Run chkrootkit's LKM-specific check
sudo chkrootkit lkm

Even rootkits that hide from lsmod and /proc/modules (like Diamorphine) can still be found by checking /sys/module/diamorphine/coresize or reviewing syslog warnings.

Advanced Detection: Behavior Monitoring and Integrity Checks

Static scanners have a fundamental weakness. In a 2026 experiment, adding a single null byte to a rootkit binary, a change that doesn't affect functionality, cut detection rates dramatically. Diamorphine's detections dropped from 33/66 to 8/64 just by stripping its symbol tables. Relying on signatures alone isn't enough.

Behavior analysis

Instead of asking "does this file match a known rootkit?", behavior analysis asks "is this process doing something unusual?" Monitor these signals:

• Use Auditd to watch init_module() and finit_module() system calls, which load kernel modules regardless of method.
• Monitor kill() calls with signals above 31, which can indicate covert rootkit communication.
• Check /proc/sys/kernel/tainted for unauthorized kernel module activity.
• Watch for unexpected .so files in /tmp or /dev/shm.
• Track eBPF activity, particularly bpf_probe_write_user calls. Tools like BCC can trace io_uring operations through tracepoints like sys_enter_io_uring_enter.

For eBPF and io_uring rootkits specifically, runtime monitoring tools like Tetragon, Falco, and Tracee can observe in-kernel activity in real time.

File integrity monitoring

AIDE (Advanced Intrusion Detection Environment) creates a baseline of trusted system files and checks for changes. Initialize with aide --init, then schedule aide --check via cron. It tracks checksums, permissions, ownership, and timestamps on critical binaries like /bin/login and /usr/bin/sshd.

For package-level verification, debsums (Debian/Ubuntu) or rpm -Va (RHEL/CentOS) can confirm system file integrity. For the most reliable results, boot from trusted rescue media and inspect the filesystem offline, since rootkits can tamper with a running kernel's reporting.

Also monitor persistence mechanisms. Rootkits often modify /etc/ld.so.preload to inject shared objects, or alter .bashrc and .profile. Legitimate changes to these files are rare, so alerts here have a high signal-to-noise ratio.

Automated monitoring with Auditd

Add these rules to /etc/audit/rules.d/rootkit.rules for real-time detection of suspicious kernel activity:

# Detect unauthorized kernel module loading
-a always,exit -F arch=b64 -S finit_module -S init_module
 
# Catch unusual high-range kill signals
-a always,exit -F arch=b64 -S kill -F a1>=32

Combine these with rkhunter's automatic updates. Set UPDATE_MIRRORS=1 and MIRRORS_MODE=0 in the config, and run rkhunter --propupd after legitimate system updates to refresh the baseline.

Hardening Your Server Against Rootkits

Detection matters, but prevention is better. Most rootkits require elevated privileges to install, so reducing the attack surface makes a real difference.

Keep the kernel and packages updated. Attackers exploit unpatched vulnerabilities to escalate privileges and deploy LKM or eBPF-based rootkits. After patching, update your detection baselines with rkhunter --propupd.

Enforce least privilege. Don't give users or processes more access than they need. Use SELinux or AppArmor for mandatory access controls that block unauthorized actions even if a process is compromised.

Disable kernel module loading post-boot. On dedicated servers, you can prevent LKM rootkits entirely by locking down module loading after the system starts. This isn't possible on shared hosting, which is one reason dedicated or VPS environments offer stronger security posture.

Segment your network. Isolating parts of your infrastructure limits lateral movement if one machine is compromised.

Audit regularly. Tools like Lynis can identify permission errors and misconfigurations before attackers exploit them.

Conclusion

Rootkits are built to hide in plain sight. By the time symptoms show up, the system may already be compromised beyond easy recovery. No single scanner catches everything, and basic signature-based detection is easy to evade.

A practical defense combines multiple layers:

• Regular scans with chkrootkit and rkhunter for known threats
• Auditd rules and behavior monitoring for suspicious kernel activity
• File integrity monitoring with AIDE to catch unauthorized changes
• Runtime tools like Tetragon or Falco for eBPF and io_uring threats
• Hardened access controls, patching, and network segmentation to reduce the attack surface

Automate what you can, keep baselines current, and start from a clean OS install you trust.

FDC Servers offers dedicated and VPS hosting with full root access and customizable kernel configurations. Explore dedicated server options to build a hardened Linux environment.