Tutorials 28 min read

Linux Incident Response and Live Forensics: A Hands-On Playbook

A hands-on, command-driven playbook for investigating compromised Linux servers. Covers memory acquisition with AVML, process and network forensics, persistence hunting, log analysis, filesystem imaging, and automated collection with Velociraptor and Cat-Scale.

Editorial Team

Linux Incident Response and Live Forensics: A Hands-On Playbook

Linux runs the backbone of the internet. It powers over 96% of the world's top one million web servers, dominates the public cloud, and serves as the go-to operating system for databases, containers, and critical infrastructure. So when a Linux system gets compromised, the stakes are high — and the window to collect evidence is painfully narrow. The Linux kernel alone had 5,530 CVEs reported in 2025, roughly a 28% jump year-over-year. Ransomware groups, cryptominers, and state-sponsored threat actors are going after Linux infrastructure directly now, not just as a stepping stone but as the primary target.

Yet Linux incident response remains harder than its Windows counterpart in several important ways. Windows hands you rich forensic artifacts out of the box — Prefetch files record application execution, ShimCache and Amcache track binary compatibility data, UserAssist logs GUI interactions, and the NTFS $MFT captures detailed file metadata with nanosecond timestamps. On Linux? None of these equivalents exist natively. The bash_history file is trivially wiped or disabled. The ext4 filesystem doesn't maintain a built-in execution history. File access timestamps (atime) are often disabled for performance via noatime mount options. An attacker with root access can effectively erase their tracks in minutes.

This guide gives you a structured, practical playbook for investigating a compromised Linux server. Every command is real, tested, and copy-pasteable. The focus here is entirely on what to do after an incident is detected — the actual investigation and forensics workflow. If you're looking to build out your detection infrastructure with AIDE, auditd, Wazuh, and Suricata, see our separate guide on Building a Multi-Layer Linux Intrusion Detection System. Here, we assume an alert has fired or suspicious activity has been spotted, and you need to figure out what happened, how it happened, and how far the attacker got.

Before the Breach: Forensic Readiness

The quality of your incident response is largely determined by what you have in place before the incident happens. Forensic readiness isn't optional — it's the difference between a successful investigation and guesswork.

The following should already be configured on every production Linux system:

  • NTP synchronization to UTC. Every system must sync to a common time source using chrony or systemd-timesyncd. All logging should use UTC. Without consistent timestamps across systems, correlating events across a multi-server compromise is basically impossible. Configure this in /etc/chrony/chrony.conf or via timedatectl set-timezone UTC.
  • Centralized logging. Local logs on a compromised host can't be trusted. Ship all logs to a remote SIEM or log aggregation platform (Wazuh, Elastic, Splunk, Graylog) over an encrypted channel. The attacker may wipe /var/log/ on the compromised machine, but the SIEM copy persists.
  • Auditd configured and active. The Linux Audit Framework is the closest equivalent to Windows Event Logging for security events. Make sure auditd is running with rules covering: file access to sensitive paths, privilege escalation events (execve with uid=0), module loading, and changes to authentication configuration. Immutable audit rules (using -e 2) prevent an attacker from disabling audit logging even with root access.
  • AIDE baselines. A current AIDE (Advanced Intrusion Detection Environment) database provides a known-good filesystem state to compare against during investigation. Without a baseline, you can't reliably determine which files were modified.
  • Memory capture tools pre-staged. You can't install forensic tools on a compromised system — doing so overwrites evidence and introduces untrusted binaries. Keep a static, pre-compiled toolkit on a read-only USB drive or network share. This toolkit should include AVML (for memory acquisition), a statically compiled busybox, and your standard investigation scripts.
  • IR runbook documented. The middle of an active incident is not the time to decide on procedure. Document your response steps, escalation paths, communication plans, evidence handling procedures, and legal/compliance notification requirements in advance.

Phase 1: Initial Triage — The First 15 Minutes

The first minutes after detecting a potential compromise are critical. Every action you take (or fail to take) affects the integrity of the evidence.

Follow this sequence precisely.

Do Not Reboot

This is the single most important rule. I can't stress this enough.

Rebooting destroys volatile evidence: running processes, network connections, contents of /dev/shm and /tmp (on tmpfs), kernel state, and the contents of RAM. Fileless malware that exists only in memory will be permanently lost. If there's pressure from management to "just restart it," push back — explain that you need 30 to 60 minutes to capture volatile evidence before any recovery action.

Isolate the System

Immediately cut the compromised system off from the network to prevent lateral movement, data exfiltration, or command-and-control communication. Do this at the network level, not by powering off:

# Disable the network interface (preserves all running processes)
ip link set eth0 down

# Alternatively, if the server has multiple interfaces, disable all
for iface in $(ls /sys/class/net/ | grep -v lo); do
    ip link set "$iface" down
done

In cloud environments, modify the security group or network ACL to deny all inbound and outbound traffic instead. This achieves isolation without requiring console access to the instance.

Document the Starting State

# Record the current UTC time and your timezone offset
date -u
# Example output: Mon Mar 16 14:23:45 UTC 2026

# Record system uptime (tells you when it was last rebooted)
uptime

# Start recording your entire terminal session with timestamps
# This creates a verifiable log of every command you run
script -t 2>timing.log session.log

The script command is essential for chain of custody. It records every command you execute and every output you see, with timing information. If your investigation ever reaches a courtroom or a formal review, this log proves exactly what you did and when.

Capture Volatile Data in Order of Volatility

The order of volatility, as defined in RFC 3227, dictates the sequence of evidence collection. Collect the most volatile data first:

  1. Memory (RAM) — disappears on power loss
  2. Running processes and network state — changes by the second
  3. Logged-in users and active sessions — sessions may timeout
  4. Temporary filesystems — /tmp, /dev/shm (often tmpfs)
  5. Disk contents — relatively stable but can be wiped
  6. Remote logs and backups — most stable

Phase 2: Memory Acquisition and Analysis

Honestly, memory forensics is often the single most valuable phase of a Linux investigation. Fileless malware, injected code, encryption keys, active network connections, and command history all live in RAM. Once the system is powered off or rebooted, this evidence is gone. Permanently.

Capturing Memory with AVML

AVML (Acquire Volatile Memory for Linux), developed by Microsoft, is the recommended tool for modern Linux memory acquisition. Unlike LiME, AVML doesn't require loading a kernel module — it reads directly from /dev/mem or /proc/kcore and works on most kernels without compilation. That alone makes it worth keeping in your toolkit.

# Download AVML (pre-stage this on your IR toolkit USB)
# https://github.com/microsoft/avml/releases
# Use the statically compiled binary -- no dependencies needed

# Capture full system memory to a file
./avml memory.lime

# Capture with compression (significantly smaller output)
./avml --compress memory.lime.compressed

# Verify the capture completed successfully
ls -lh memory.lime

If AVML isn't available, LiME (Linux Memory Extractor) is the alternative. LiME requires inserting a kernel module, which is more invasive but more widely supported on older kernels:

# LiME must be compiled against the target kernel headers
# Pre-compile for all your server kernel versions as part of forensic readiness
insmod lime-$(uname -r).ko "path=/evidence/memory.lime format=lime"

Analyzing Memory with Volatility 3

Volatility 3 (current release v2.27.0) is the standard framework for memory analysis. It's been completely rewritten from Volatility 2 and uses ISF (Intermediate Symbol Format) symbol tables instead of the old profile system. For Linux analysis, you'll need the appropriate symbol table for the target kernel, which can be generated using dwarf2json from the kernel's debug symbols.

# List all running processes from the memory dump
vol -f memory.lime linux.pslist

# Extract bash command history from memory
# (works even if .bash_history has been wiped on disk)
vol -f memory.lime linux.bash

# Scan for network connections
vol -f memory.lime linux.netscan

# Detect injected or hidden code in process memory
vol -f memory.lime linux.malfind

# List loaded kernel modules (detect rootkit modules)
vol -f memory.lime linux.lsmod

# Extract environment variables for all processes
vol -f memory.lime linux.envars

What to look for in memory analysis results:

  • Processes with no corresponding binary on disk — this indicates fileless malware or a deleted executable that's still running.
  • Injected code regions — linux.malfind identifies memory regions with executable permissions that weren't loaded from a file. That's a strong indicator of code injection.
  • Hidden processes — compare the process list from memory against ps output. Rootkits often hide processes from userspace tools but can't hide them from raw memory analysis.
  • Suspicious bash history — the in-memory bash history includes commands that the attacker may have deleted from ~/.bash_history.
  • Unexpected kernel modules — rootkit kernel modules that don't correspond to any legitimate driver.

Phase 3: Process and Network Investigation

With memory captured, turn your attention to live process and network analysis. This phase identifies what's actively running on the system, what it's communicating with, and whether processes are masquerading as legitimate services.

Process Investigation

# Full process tree with wide output (shows parent-child relationships)
ps auxwf

# For a specific suspicious process (replace PID), investigate thoroughly:

# Verify the actual binary path -- attackers rename malware to look like system processes
ls -la /proc/PID/exe
# If this shows "(deleted)", the binary was deleted after execution -- highly suspicious

# Get the full command line with all arguments
cat /proc/PID/cmdline | tr '
# View all open file descriptors (reveals files, sockets, pipes)
ls -la /proc/PID/fd/

# Examine memory maps -- look for suspicious shared objects
cat /proc/PID/maps

# Check the environment variables for the process
cat /proc/PID/environ | tr ''

# View the current working directory of the process
ls -la /proc/PID/cwd

# Check when the process started
stat /proc/PID

# If the binary was deleted but process is still running, recover it:
cp /proc/PID/exe /evidence/recovered_binary

Red flags during process investigation:

  • Processes running from /tmp, /dev/shm, /var/tmp, or any world-writable directory.
  • The /proc/PID/exe symlink pointing to "(deleted)" — the binary was removed after execution.
  • Processes named to mimic system services (e.g., [kworker/0:0] with square brackets but running in userspace, or sshd running from an unusual path).
  • Processes with abnormally high CPU usage (cryptominers) or unusual memory consumption.
  • Child processes spawned by web servers (apache, nginx) or database processes that shouldn't be spawning shells.

Network Investigation

# Show all listening sockets with process information
ss -tulnp

# Show all established connections with process info
ss -tnp

# Show connections in all states (including TIME_WAIT, CLOSE_WAIT)
ss -tanp

# If you suspect ss itself may be compromised (rootkit replaced it),
# read the raw kernel data directly -- this cannot be spoofed without
# a kernel-level rootkit:
cat /proc/net/tcp    # IPv4 TCP connections (hex encoded)
cat /proc/net/tcp6   # IPv6 TCP connections
cat /proc/net/udp    # UDP sockets

# Decode the hex addresses from /proc/net/tcp for readable output:
awk '{print $2, $3}' /proc/net/tcp | while read local remote; do
    local_ip=$(printf "%d.%d.%d.%d" \
        0x${local:6:2} 0x${local:4:2} 0x${local:2:2} 0x${local:0:2})
    local_port=$((16#${local:9:4}))
    remote_ip=$(printf "%d.%d.%d.%d" \
        0x${remote:6:2} 0x${remote:4:2} 0x${remote:2:2} 0x${remote:0:2})
    remote_port=$((16#${remote:9:4}))
    echo "$local_ip:$local_port -> $remote_ip:$remote_port"
done

Suspicious network indicators to watch for:

  • Connections to known malicious ports: 4444 (Metasploit default), 1337, 31337 (common backdoor ports), or high-numbered ephemeral ports with established connections.
  • Outbound connections on ports 53 (DNS tunneling), 443 (HTTPS C2 blending), or 8080/8443 from processes that shouldn't be making such connections.
  • Connections to IP addresses in unusual geographies or to known Tor exit nodes.
  • Listening services that you don't recognize or that weren't part of the system's intended configuration.
  • Unusually large amounts of data transferred on a single connection (potential exfiltration).

Phase 4: User Account and Authentication Forensics

Attackers need access, and access means user accounts. This phase examines who has logged in, what accounts exist, and whether any unauthorized accounts or privilege escalation has occurred.

# Comprehensive login history with full timestamps, IP addresses, and hostnames
last -Faiwx

# Last login time for every account on the system
lastlog

# Find all accounts with UID 0 (root-equivalent) -- should ONLY be root
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Find all accounts that have a login shell (not nologin or false)
awk -F: '$7 != "/usr/sbin/nologin" && $7 != "/bin/false" && $7 != "/sbin/nologin" {print $1, $7}' /etc/passwd

# Check for recently modified user accounts
ls -la /etc/passwd /etc/shadow /etc/group /etc/gshadow

# Review sudo configuration for unexpected escalation paths
grep -v '^#' /etc/sudoers 2>/dev/null
cat /etc/sudoers.d/* 2>/dev/null

# Check for password-less sudo (NOPASSWD entries are high-risk)
grep -r 'NOPASSWD' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

# Find SSH authorized_keys files across the entire system
find / -name authorized_keys -exec echo "=== {} ===" \; -exec cat {} \; 2>/dev/null

# Look for SSH keys with forced commands (command= option can hide backdoors)
grep -r 'command=' /home/*/.ssh/authorized_keys /root/.ssh/authorized_keys 2>/dev/null

# Check for accounts with empty passwords
awk -F: '$2 == "" {print $1, "HAS EMPTY PASSWORD"}' /etc/shadow

Authentication Log Analysis

# On Debian/Ubuntu -- check auth.log for brute force, su, sudo events
grep -E '(Failed password|Accepted password|session opened|sudo:)' /var/log/auth.log | tail -100

# On RHEL/CentOS -- same analysis on /var/log/secure
grep -E '(Failed password|Accepted password|session opened|sudo:)' /var/log/secure | tail -100

# Identify brute force patterns -- multiple failed logins from same IP
grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Check for successful logins following brute force attempts from the same IP
grep 'Accepted password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Review privilege escalation events
grep -E '(su\[|sudo:)' /var/log/auth.log | tail -50

Key findings to watch for: new UID 0 accounts, authorized_keys files in service accounts that shouldn't have SSH access, NOPASSWD sudo rules that weren't part of your configuration management, and successful SSH logins from IP addresses that don't match your team or known jump hosts.

Phase 5: Persistence Mechanism Hunting

Once an attacker gains access, their immediate priority is keeping it. Persistence mechanisms ensure they can come back even if the initial entry point is closed. Linux offers dozens of persistence locations, and a thorough investigation needs to check all of them.

Cron Jobs

# System-wide crontab
cat /etc/crontab

# Cron directories (any file here runs on schedule)
ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
ls -la /etc/cron.weekly/
ls -la /etc/cron.monthly/

# Per-user crontabs
ls -la /var/spool/cron/crontabs/    # Debian/Ubuntu
ls -la /var/spool/cron/             # RHEL/CentOS

# Display content of all user crontabs
for user in $(cut -f1 -d: /etc/passwd); do
    crontab_output=$(crontab -l -u "$user" 2>/dev/null)
    if [ -n "$crontab_output" ]; then
        echo "=== Crontab for $user ==="
        echo "$crontab_output"
    fi
done

Systemd Services and Timers

# List all running services -- look for anything unfamiliar
systemctl list-units --type=service --state=running

# List all enabled services (will start on boot)
systemctl list-unit-files --type=service --state=enabled

# List active timers (systemd equivalent of cron)
systemctl list-timers --all

# Check for recently created or modified unit files
find /etc/systemd/system/ /run/systemd/system/ /usr/lib/systemd/system/ \
    -type f -mtime -30 -ls 2>/dev/null

# Look for user-level systemd services (often overlooked)
find /home -path '*/.config/systemd/user/*.service' -ls 2>/dev/null

Shell Initialization Files

# Attackers add malicious commands to shell startup files
# Check system-wide and per-user initialization scripts

# System-wide
cat /etc/profile
ls -la /etc/profile.d/
cat /etc/bash.bashrc    # Debian/Ubuntu
cat /etc/bashrc         # RHEL/CentOS

# Per-user (check for ALL users, not just root)
for home in /home/* /root; do
    for rc in .bashrc .bash_profile .profile .bash_login .bash_logout; do
        if [ -f "$home/$rc" ]; then
            echo "=== $home/$rc ==="
            cat "$home/$rc"
        fi
    done
done

SSH Configuration Backdoors

# SSH per-user configuration can specify ProxyCommand for code execution
find / -name "config" -path "*/.ssh/*" -exec echo "=== {} ===" \; -exec cat {} \; 2>/dev/null

# SSH rc files execute on every SSH login
find / -name "rc" -path "*/.ssh/*" -exec echo "=== {} ===" \; -exec cat {} \; 2>/dev/null

# Check sshd_config for backdoor settings
grep -E '(AuthorizedKeysFile|PermitRootLogin|PasswordAuthentication|ForceCommand)' /etc/ssh/sshd_config

Kernel Modules and Library Injection

# List loaded kernel modules -- look for anything unrecognized
lsmod

# Check modules configured to load at boot
cat /etc/modules 2>/dev/null
ls -la /etc/modules-load.d/ 2>/dev/null

# LD_PRELOAD persistence -- a shared library loaded into EVERY process
# This is a powerful backdoor technique
cat /etc/ld.so.preload 2>/dev/null
# If this file exists and contains content, investigate immediately

# Check the dynamic linker configuration
ldconfig -p | wc -l   # baseline count
cat /etc/ld.so.conf
ls -la /etc/ld.so.conf.d/

PAM Backdoors and At Jobs

# Check PAM configuration for modifications
# PAM backdoors can allow any password to authenticate
ls -la /etc/pam.d/
# Compare against a known-good system or your configuration management

# Check for modifications to critical PAM modules
find /lib/x86_64-linux-gnu/security/ /lib64/security/ -name '*.so' \
    -newer /etc/os-release -ls 2>/dev/null

# Check at job queue
atq
ls -la /var/spool/at/ 2>/dev/null

Finding Recently Modified Files System-Wide

# Find all files modified in the last 7 days (adjust timeframe as needed)
# Excludes virtual filesystems to avoid false positives
find / -mtime -7 -type f \
    -not -path '/proc/*' \
    -not -path '/sys/*' \
    -not -path '/run/*' \
    -not -path '/dev/*' \
    -not -path '/var/log/*' \
    2>/dev/null | sort

# Find files modified in a specific time window
# First, create reference timestamps
touch -t 202603150000 /tmp/start_time
touch -t 202603160000 /tmp/end_time
find / -newer /tmp/start_time -not -newer /tmp/end_time -type f \
    -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null

Phase 6: Log Analysis and Timeline Reconstruction

Log analysis is where individual findings start coming together into a coherent story. The goal is to reconstruct a timeline: when did the attacker gain access, what did they do, and in what order? This timeline is essential for understanding the full scope of the compromise — and it'll form the backbone of your incident report.

Key Log Locations

Log File Contents Distribution
/var/log/auth.log Authentication events, sudo, su, SSH logins Debian/Ubuntu
/var/log/secure Authentication events, sudo, su, SSH logins RHEL/CentOS
/var/log/syslog General system messages Debian/Ubuntu
/var/log/messages General system messages RHEL/CentOS
/var/log/kern.log Kernel messages, module loading, hardware events All
/var/log/cron.log Cron job execution records Varies
/var/log/audit/audit.log Auditd events (syscalls, file access, exec) All (if auditd installed)
/var/log/btmp Failed login attempts (binary, read with lastb) All
/var/log/wtmp Login/logout records (binary, read with last) All

Using journalctl for Targeted Investigation

# View all logs within a specific time window
journalctl --since "2026-03-15 00:00:00" --until "2026-03-16 00:00:00"

# Filter by unit (service)
journalctl -u sshd --since "2026-03-15" --no-pager

# Show only kernel messages (useful for detecting module loading)
journalctl -k --since "2026-03-14" --no-pager

# Show logs at priority "warning" and above
journalctl -p warning --since "2026-03-14" --no-pager

# Export logs in JSON format for external analysis
journalctl --since "2026-03-14" -o json --no-pager > /evidence/journal_export.json

Using ausearch for Auditd Events

# Search for recent audit events in human-readable format
ausearch -ts recent -i

# Search for all executed commands (requires execve audit rule)
ausearch -sc execve -ts today -i

# Search for file access to sensitive paths
ausearch -f /etc/passwd -i
ausearch -f /etc/shadow -i

# Search for events by specific user
ausearch -ua 1001 -i    # by UID
ausearch -ua root -i     # by username

# Generate a summary report of audit events
aureport --summary
aureport --auth          # authentication attempts
aureport --login         # login events
aureport --file          # file access events

Building a Unified Timeline

# Create a master timeline by combining multiple sources
# Step 1: Create a reference timestamp for the suspected compromise window
touch -t 202603140000 /tmp/reference_timestamp

# Step 2: Find all files modified since that reference point
find / -newer /tmp/reference_timestamp -type f \
    -not -path '/proc/*' -not -path '/sys/*' -not -path '/run/*' \
    -printf '%T+ %p
' 2>/dev/null | sort > /evidence/filesystem_timeline.txt

# Step 3: Extract timestamped events from logs
journalctl --since "2026-03-14" --output=short-iso --no-pager \
    > /evidence/journal_timeline.txt

# Step 4: Extract auditd timeline
ausearch -ts '03/14/2026' -i --format text > /evidence/audit_timeline.txt

Detecting Log Tampering

A sophisticated attacker will often try to tamper with logs. Here's what to look for:

  • Gaps in timestamps. If logs show entries at 14:23:45 and then skip to 14:47:12 with nothing in between on a busy system, that gap is suspicious.
  • Truncated log files. Check file sizes — a /var/log/auth.log that's only a few kilobytes on a server that's been running for months strongly suggests truncation.
  • Missing log rotation files. If auth.log.1 exists but auth.log.2.gz through auth.log.4.gz are missing, older logs may have been deleted.
  • Journal corruption. Run journalctl --verify to check the integrity of the systemd journal.
  • Inconsistency between local and remote logs. If your SIEM has entries that don't appear in the local logs, the local logs have been tampered with. This is one of the strongest reasons to always have centralized logging in place.

Phase 7: Filesystem Forensics

Filesystem analysis provides the most durable evidence, but it needs to be done carefully to maintain forensic integrity. Always work on a copy, never the original.

Creating a Forensic Disk Image

# Standard disk image with progress reporting
dd if=/dev/sda of=/evidence/disk.img bs=4M status=progress

# Better: use dc3dd for concurrent hashing (forensic standard)
dc3dd if=/dev/sda of=/evidence/disk.img hash=sha256 log=/evidence/dc3dd.log

# For a live system, image individual partitions if full disk is too large
lsblk    # identify partition layout
dc3dd if=/dev/sda1 of=/evidence/sda1.img hash=sha256 log=/evidence/sda1.log

# Verify image integrity
sha256sum /evidence/disk.img

SUID/SGID Binary Audit

# Find all SUID and SGID binaries on the system
# SUID binaries run as the file owner (often root) regardless of who executes them
# Attackers add SUID bits to shells or backdoors for easy privilege escalation
find / -perm -4000 -o -perm -2000 -type f 2>/dev/null | sort

# Compare against a known-good list from a clean system
# Save this output and diff it against your baseline

Hidden Files and World-Writable Directories

# Find hidden files (starting with a dot) outside of normal locations
find / -name ".*" -type f \
    -not -path '/home/*' \
    -not -path '/root/*' \
    -not -path '/proc/*' \
    -not -path '/sys/*' \
    -not -path '/run/*' \
    2>/dev/null

# Examine world-writable directories -- common staging areas for attackers
ls -laR /tmp/
ls -laR /dev/shm/
ls -laR /var/tmp/

# Look for executable files in world-writable locations
find /tmp /dev/shm /var/tmp -type f -executable 2>/dev/null

Deleted but Open Files

# Find deleted files that are still held open by running processes
# These are files the attacker deleted to hide evidence, but the process
# still has a file descriptor open
lsof +L1

# Recover the content of a deleted-but-open file
# Find the PID and file descriptor from lsof output, then:
cp /proc/PID/fd/FD_NUMBER /evidence/recovered_file

Package Integrity Verification

# On RHEL/CentOS/Fedora -- verify all installed RPM packages
# Output shows modifications: S=size, 5=MD5, T=mtime, etc.
rpm -Va

# On Debian/Ubuntu -- verify package file checksums
debsums -c    # show only changed files
debsums -a    # show all files with verification status

# Focus on critical packages
rpm -V openssh-server coreutils util-linux    # RHEL
debsums openssh-server coreutils              # Debian

Modifications to core system binaries (ls, ps, ss, netstat, lsof) are a strong indicator of a userspace rootkit. If rpm -Va or debsums -c shows that these binaries have been modified, don't trust any output from those commands — switch to your pre-staged static toolkit immediately.

Phase 8: Automated Collection with Open-Source Tools

Manual investigation is thorough but slow. When you're responding to an incident across multiple systems, or when you need to make sure nothing gets missed, automated collection tools become essential. They codify the manual steps above into repeatable, auditable workflows.

Velociraptor

Velociraptor is the leading open-source endpoint visibility and forensic collection platform. It uses VQL (Velociraptor Query Language) to define artifact collections and can scale to over 15,000 endpoints. On Linux, it supports eBPF-based real-time detection in addition to forensic collection.

# Run a VQL hunt across your fleet to collect process listings
# This example collects running processes from all enrolled Linux endpoints

# VQL query for process tree with network connections:
SELECT Pid, Name, Exe, CommandLine, Username,
       { SELECT Laddr, Raddr, Status
         FROM netstat() WHERE Pid = Pid } AS Connections
FROM pslist()
WHERE Name =~ "."

# Collect SSH authorized_keys from all endpoints:
SELECT OSPath, Mtime, Data.Blob
FROM glob(globs='/home/*/.ssh/authorized_keys')

Deploy Velociraptor as a server with lightweight agents on each endpoint. During an incident, you can push collection artifacts to specific hosts or run hunts across the entire fleet in minutes. It's genuinely one of the most powerful tools you can have in your IR arsenal.

Cat-Scale (WithSecure)

Cat-Scale is WithSecure's (formerly F-Secure) open-source Linux forensic artifact collection script. It's a standalone bash script that collects system state, logs, configuration files, and persistence mechanisms into a compressed archive — ideal for situations where you need quick collection without deploying an agent.

# Download and run Cat-Scale on the target system
# https://github.com/WithSecureLabs/LinuxCatScale
chmod +x Cat-Scale.sh
sudo ./Cat-Scale.sh

# Output is saved to a timestamped tar.gz archive in the current directory
# Transfer the archive to your analysis workstation for review

UAC (Unix-like Artifacts Collector)

UAC is purpose-built for collecting forensic artifacts from Unix-like systems including Linux, macOS, and the BSDs. It collects over 200 different artifact types and outputs them in a structured format suitable for timeline analysis.

# Run a full collection
./uac -p full /evidence/uac_output

# Run a quick triage collection (faster, fewer artifacts)
./uac -p ir_triage /evidence/uac_output

SIFT Workstation

The SANS Investigative Forensics Toolkit (SIFT) Workstation is a full forensic analysis platform based on Ubuntu. It comes pre-installed with The Sleuth Kit, Volatility, Plaso (log2timeline), Autopsy, and dozens of other forensic tools. Use SIFT as your analysis workstation — transfer disk images and memory dumps to it for examination. It's not something you install on production servers; it lives on the analyst's machine.

Phase 9: Containment, Eradication, and Recovery

Once the investigation has established the scope of the compromise, you shift from understanding the incident to resolving it. This is where the findings from Phases 1 through 8 translate into action.

Containment

  • Network containment. Move the compromised system to a quarantine VLAN that allows only forensic access. Block all identified IOCs (IP addresses, domains) at the perimeter firewall and DNS level. If the attacker has compromised multiple systems, isolate all of them simultaneously — staggered isolation just gives the attacker time to pivot.
  • Account containment. Disable all accounts that showed unauthorized access. Force password resets for any account that was active on the compromised system. Revoke and rotate SSH keys, API tokens, and service account credentials that were present on the compromised host.
  • Scope assessment. Use the IOCs from your investigation (IP addresses, file hashes, SSH keys, user agents) to search your SIEM and other systems for signs of lateral movement. The compromised system you found may not be the only one.

Eradication

For any compromise where the attacker achieved root access, the only safe eradication strategy is to rebuild the system from a clean image. Don't attempt to "clean" a root-compromised system. Rootkits can modify the kernel, replace system binaries, and install persistence mechanisms in locations you may not think to check. A rebuild from a known-good base image, followed by redeployment through your configuration management system (Ansible, Puppet, Chef, SaltStack), is the only way to be certain the attacker's access has been fully removed.

# Verify backup integrity before restoring
# Check that the backup predates the compromise
sha256sum /backups/application_data_20260312.tar.gz

# Restore application data to the freshly built system
tar -xzf /backups/application_data_20260312.tar.gz -C /var/www/app/

# Do NOT restore system configuration files from backup -- they may contain
# attacker modifications. Regenerate all configuration through your
# configuration management tooling instead.

Post-Recovery Hardening

  • Patch the entry point. If the investigation identified the initial access vector (an unpatched CVE, a weak password, an exposed service), remediate it before bringing the system back online. This sounds obvious, but I've seen teams skip it under pressure to restore service.
  • Rotate all credentials. This includes SSH keys, database passwords, API tokens, TLS certificates, and any other secrets that were present on the compromised system. Assume the attacker exfiltrated every credential on the host.
  • Add specific detections. Create SIEM rules, Falco rules, or auditd rules that detect the specific TTPs (Tactics, Techniques, and Procedures) observed during the investigation. If the attacker used a cron-based persistence mechanism, add monitoring for crontab modifications. If they used LD_PRELOAD, add a rule to detect writes to /etc/ld.so.preload.
  • Verify the rebuild. Run your AIDE baseline, vulnerability scanner, and compliance checks against the rebuilt system before returning it to production.

Documentation

Create a formal incident report. It doesn't have to be a novel, but it should contain:

  1. Executive summary — one paragraph describing what happened, the impact, and the resolution.
  2. Timeline — a chronological record of the attacker's actions and your response actions, with UTC timestamps.
  3. Indicators of Compromise (IOCs) — IP addresses, file hashes (SHA-256), domains, user accounts, file paths, and any other observables associated with the attacker.
  4. Root cause analysis — how the attacker gained initial access and what conditions allowed it.
  5. Remediation actions taken — what you did to contain, eradicate, and recover.
  6. Lessons learned — what could be improved in detection, prevention, or response processes.

Building Your IR Toolkit

Here's a quick reference of essential tools organized by investigation phase. Pre-stage the acquisition tools on a read-only USB or network share so they're available immediately when needed.

Category Tools Purpose
Memory Acquisition AVML, LiME Capture full system RAM for offline analysis
Memory Analysis Volatility 3 (v2.27.0) Analyze memory dumps for processes, connections, injected code
Process/Network ps, ss, lsof, /proc filesystem Live process tree, network connections, open files
Log Analysis journalctl, ausearch, aureport Query systemd journal and auditd event logs
Filesystem Forensics The Sleuth Kit, dc3dd, Autopsy Disk imaging, file carving, timeline analysis
Package Integrity debsums, rpm -Va Verify installed packages against known-good checksums
Rootkit Detection chkrootkit, rkhunter Scan for known rootkit signatures and suspicious modifications
Automated Collection Velociraptor, Cat-Scale, UAC Scalable, repeatable artifact collection across endpoints
Analysis Workstation SIFT Workstation, Autopsy Full forensic analysis environment with pre-installed tools

Frequently Asked Questions

How do I check if my Linux server has been hacked?

Start by checking several indicators in sequence. Run last -Faiwx to review login history for unfamiliar IP addresses or unusual login times. Check for unauthorized user accounts with awk -F: '$3 == 0' /etc/passwd to find UID 0 accounts beyond root. Review running processes with ps auxwf and look for anything running from /tmp, /dev/shm, or showing as "(deleted)" in /proc/PID/exe. Examine network connections with ss -tnp for unexpected outbound connections. Check cron jobs, systemd services, and SSH authorized_keys for unauthorized entries. Run rpm -Va or debsums -c to detect modified system binaries. If any of these checks reveal anomalies, treat the system as compromised and begin the full investigation workflow described above.

What is the first thing to do when a Linux server is compromised?

Isolate the system from the network without powering it off. Run ip link set eth0 down to disable the network interface, or modify cloud security groups to deny all traffic. Don't reboot — rebooting destroys volatile evidence in memory, including running processes, network connections, and fileless malware. After isolation, start recording your terminal session with script -t 2>timing.log session.log, document the current UTC time with date -u, and then proceed to capture memory using AVML before it can be lost. The complete triage workflow is covered in Phase 1 of this guide.

What tools are used for Linux memory forensics?

For memory acquisition, the two primary tools are AVML (Acquire Volatile Memory for Linux) and LiME (Linux Memory Extractor). AVML is preferred for modern systems because it doesn't require loading a kernel module — it reads from /proc/kcore or /dev/mem and works across kernel versions without compilation. LiME requires a kernel module compiled against the target kernel's headers but is more widely supported on older systems. For analysis, Volatility 3 (current version v2.27.0) is the industry standard. It can extract process lists, bash history, network connections, loaded kernel modules, and injected code regions from memory dumps. Volatility 3 uses ISF symbol tables, which you can generate with dwarf2json from the kernel's debug symbols.

How do I preserve evidence on a compromised Linux system?

Evidence preservation follows the order of volatility. First, capture system memory with AVML (./avml memory.lime) — this is the most volatile and valuable evidence. Second, record the running state: process listings, network connections, logged-in users, and open files. Third, create a forensic disk image using dc3dd with hash verification (dc3dd if=/dev/sda of=/evidence/disk.img hash=sha256). Throughout the process, record your terminal session with the script command to maintain chain of custody. Store all evidence on a separate, write-protected medium — never write evidence files to the compromised system's disk. Document every action with timestamps. If the evidence may be used in legal proceedings, follow your organization's evidence handling procedures and consider engaging a certified forensic examiner.

What logs should I check during Linux incident response?

The critical logs to examine are: /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS) for authentication events including SSH logins, sudo usage, and failed login attempts; /var/log/syslog or /var/log/messages for general system events; /var/log/kern.log for kernel messages including module loading events that might indicate rootkit installation; and /var/log/audit/audit.log for auditd events if the audit framework is configured. Use journalctl with time-range filters for systemd-based systems. Binary log files /var/log/wtmp (read with last) and /var/log/btmp (read with lastb) record successful and failed login attempts respectively. Always compare local logs against your centralized SIEM copy — attackers frequently tamper with local log files to cover their tracks.

About the Author Editorial Team

Our team of expert writers and editors.

Related Articles