Chapter 45

Linux Forensics

Linux servers are common targets in enterprise environments — web servers, database servers, build systems, and cloud instances run Linux. This chapter covers the Linux forensic artifact landscape: shell history, authentication logs, systemd journal, auditd, /proc pseudo-filesystem, cron jobs, and persistence mechanisms.

Scenario

A Linux web server was compromised via a vulnerable PHP application. The attacker gained a webshell, escalated to root, installed a backdoor, and added a cron job for persistence. The server logs, bash history, and auditd records all survived because the attacker only cleared the auth.log — not realizing that auditd writes to a separate file and the systemd journal is a binary format they didn't know to clear. This chapter covers finding all of these artifacts.

Linux Forensic Artifact Locations

ArtifactLocationWhat it records
Authentication log/var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS)SSH logins, sudo usage, PAM authentication, su commands
Systemd journal/var/log/journal/ (binary)All systemd service events, boot messages, kernel messages
Syslog/var/log/syslog or /var/log/messagesGeneral system log aggregation
Web server logs/var/log/apache2/ or /var/log/nginx/HTTP requests: IP, URI, status, user-agent
Bash history~/.bash_history per user; /root/.bash_history for rootCommand-line history (can be disabled or cleared)
Auditd logs/var/log/audit/audit.logSyscall auditing — file access, process execution, network connections
Cron jobs/etc/cron*, /var/spool/cron/, /etc/crontabScheduled task definitions (persistence)
Systemd services/etc/systemd/system/, /lib/systemd/system/Service definitions (persistence via new .service files)
SSH authorized_keys~/.ssh/authorized_keys per userPre-authorized SSH public keys (persistence)
Bashlinux-triage.sh
#!/bin/bash
# Linux forensic triage collection
# Run as root on the compromised system

CASE="CASE-2026-009"
HOST=$(hostname)
OUT="/mnt/evidence/$CASE/$HOST"
mkdir -p $OUT/logs $OUT/users $OUT/persistence $OUT/webserver

# Preserve timestamps of source files (important: use cp -a)
echo "[1/8] Collecting authentication logs..."
cp -a /var/log/auth.log* $OUT/logs/ 2>/dev/null
cp -a /var/log/secure* $OUT/logs/ 2>/dev/null
cp -a /var/log/syslog* $OUT/logs/ 2>/dev/null

echo "[2/8] Collecting systemd journal (binary)..."
journalctl --no-pager -o json > $OUT/logs/journal.json
journalctl --no-pager > $OUT/logs/journal.txt

echo "[3/8] Collecting auditd logs..."
cp -a /var/log/audit/ $OUT/logs/audit/ 2>/dev/null

echo "[4/8] Collecting web server logs..."
cp -a /var/log/apache2/ $OUT/logs/apache2/ 2>/dev/null
cp -a /var/log/nginx/ $OUT/logs/nginx/ 2>/dev/null

echo "[5/8] Collecting user history and config..."
for user_home in /home/* /root; do
    if [ -d "$user_home" ]; then
        user=$(basename $user_home)
        mkdir -p $OUT/users/$user
        cp -a $user_home/.bash_history $OUT/users/$user/ 2>/dev/null
        cp -a $user_home/.ssh/ $OUT/users/$user/ 2>/dev/null
        cp -a $user_home/.bashrc $user_home/.profile $OUT/users/$user/ 2>/dev/null
    fi
done

echo "[6/8] Collecting persistence mechanisms..."
cp -a /etc/cron* $OUT/persistence/
cp -a /var/spool/cron/ $OUT/persistence/
cp -a /etc/systemd/system/ $OUT/persistence/systemd-custom/
cp -a /etc/rc.local $OUT/persistence/ 2>/dev/null

echo "[7/8] Collecting /etc and installed packages..."
cp -a /etc/passwd /etc/shadow /etc/group $OUT/
dpkg -l > $OUT/installed-packages.txt 2>/dev/null || rpm -qa > $OUT/installed-packages.txt

echo "[8/8] Collecting /proc snapshot (live only)..."
ls -la /proc/[0-9]*/exe 2>/dev/null > $OUT/proc-exe-list.txt
for pid in $(ls /proc/ | grep '^[0-9]'); do
    readlink /proc/$pid/exe 2>/dev/null
done | sort | uniq -c | sort -rn > $OUT/running-exe-summary.txt

sha256sum -r $OUT/**/* > $OUT/COLLECTION_HASHES.txt
echo "Collection complete: $OUT"

SSH Attack Investigation

Bashssh-investigation.sh
AUTH_LOG="/cases/CASE-2026-009/web-srv01/logs/auth.log"

# Find successful SSH logins
grep "Accepted" $AUTH_LOG | \
    awk '{print $1,$2,$3,"user:",$9,"from:",$11,"port:",$13}' | \
    tail -50

# Find SSH brute force (many failed attempts before success)
grep "Failed password" $AUTH_LOG | \
    awk '{print $11}' | sort | uniq -c | sort -rn | head -20

# Find root account usage
grep "sudo\|su " $AUTH_LOG | tail -30

# Find new user account creation
grep "useradd\|adduser" $AUTH_LOG | tail -20

# Check for known attacker backdoor accounts
awk -F: '$3 == 0' /etc/passwd  # UID 0 = root privilege
# Should only show root — any other account with UID 0 is suspicious

Webshell Detection from Access Logs

Bashwebshell-hunt.sh
ACCESS_LOG="/cases/CASE-2026-009/web-srv01/logs/apache2/access.log"

# Find POST requests to PHP files (common webshell interaction)
grep "POST" $ACCESS_LOG | grep "\.php" | \
    awk '{print $1,$7,$9}' | sort | uniq -c | sort -rn | head -20

# Find requests with suspicious patterns in URI
grep -iP "(cmd=|exec=|system(|passthru(|shell_exec(|eval(|base64|wget|curl)" $ACCESS_LOG | \
    tail -30

# Find large response codes for non-existent paths (404 pattern hunting)
awk '$9==200 && $7 ~ /\.php/' $ACCESS_LOG | \
    awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# Newly created PHP files with 200 responses are webshell candidates

# Find by status code pattern — webshell setup often creates 200s from paths
# that never existed before attack
grep "200" $ACCESS_LOG | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

Q & A

Q: .bash_history was cleared (file is empty or doesn't exist). What alternatives exist for recovering command history?

Multiple fallback sources: (1) Auditd with exec auditing: if auditd has a rule for execve syscalls (-a exit,always -F arch=b64 -S execve), every command execution is logged in /var/log/audit/audit.log — including the arguments. Even if bash_history is cleared, auditd captures the same information at the kernel level. (2) Systemd journal: sudo commands and some other privileged actions appear in the journal even without auditd. journalctl _COMM=sudo shows sudo usage with full command. (3) Shell history file in /proc: if the bash session is still open (process not killed), /proc/[bash_pid]/fd/ may reference the history file in the inode. (4) Filesystem timeline: ext4 maintains atime/mtime/ctime. Even if .bash_history was truncated, the file's ctime (inode change) and mtime show when it was last modified — sudden update at 03:00 UTC = cleared at that time (itself an indicator). (5) VSS equivalent (LVM snapshots): Linux environments using LVM may have snapshots predating the clearance. (6) Recover deleted file: if .bash_history was deleted (not truncated), TSK/Autopsy may recover it from unallocated space.