Container Forensics
Container environments (Docker, Kubernetes) present unique forensic challenges: containers are ephemeral (evidence disappears when they're restarted), overlay filesystems layer changes on read-only base images, and traditional host-level forensic tools may not see into container namespaces. This chapter covers evidence collection from running and stopped containers before they're lost.
A Kubernetes pod running a web application was compromised. The attacker exploited a RCE vulnerability, gained shell access, and installed a cryptominer. The pod hasn't been restarted yet — all evidence is still live. You have 30 minutes before the DevOps team restarts the pod as part of their standard remediation. This chapter covers what to collect before that restart destroys the container.
In traditional IR, a compromised server can sit isolated for 12–24 hours while you prepare your forensic workstation. Containers give you no such luxury. Kubernetes deployments are typically configured with health checks, auto-restart policies, and automated remediation. A pod that fails a liveness probe restarts within 30–90 seconds. Even without automation, the DevOps team's standard remediation is "restart the pod" — and they will do this within minutes of being notified of a compromise. When a container restarts, the entire read-write overlay layer is destroyed: your attacker's files, shell history, and live network connections are gone. Every minute you spend in a meeting rather than collecting evidence is evidence you'll never recover. The first action on a compromised container is not isolation — it is collection.
Container Evidence Sources
Docker/Kubernetes Forensic Evidence Sources
═══════════════════════════════════════════════════════════════════
1. Container runtime logs (docker logs / kubectl logs)
├── STDOUT/STDERR of the container process
└── Available even after restart (for a retention window)
2. Container filesystem (overlay layers)
├── Read-only base layers (from the image)
├── Read-write upper layer (container's changes since start)
└── Attacker's files are ONLY in the upper layer — easy to identify
3. Container inspect metadata
├── When container started
├── Mounts, network settings, environment variables
└── Image used + any volume mounts
4. Host-level evidence (from the underlying Linux host)
├── /proc/[container_pid]/: process info from host perspective
├── Container filesystem at /var/lib/docker/overlay2/[hash]/
└── Host network connections: netstat sees container traffic
5. Kubernetes-specific
├── Pod events: kubectl describe pod
├── API server audit logs: every kubectl call is logged
└── Kubelet logs: container lifecycle events
Docker vs Kubernetes Evidence Comparison
| Evidence type | Docker (standalone) | Kubernetes | Survives restart? |
|---|---|---|---|
| Container STDOUT/STDERR | docker logs | kubectl logs --previous | Yes (retention window) |
| Filesystem changes (attacker files) | UpperDir overlay layer | UpperDir on node where pod ran | No — lost on restart |
| Live process list | docker top or nsenter -n | kubectl exec -- ps | No — lost on restart |
| Active network connections | nsenter -t [pid] -n -- ss | kubectl exec -- ss | No — lost on restart |
| Container metadata/config | docker inspect | kubectl describe pod / pod YAML | Yes |
| Orchestration audit log | Docker daemon log (journald) | K8s API server audit log | Yes (per retention policy) |
| Persistent storage | Volume mounts on host | PersistentVolumeClaims | Yes |
| Network traffic | Host netflow/PCAP | Node netflow/PCAP or service mesh logs | Yes (per retention policy) |
CONTAINER_ID="b3f12a9c8e1d"
CASE_DIR="/cases/CASE-2026-009/containers"
mkdir -p $CASE_DIR
# Step 1: Save container metadata before it changes
echo "Collecting container metadata..."
docker inspect $CONTAINER_ID > $CASE_DIR/inspect.json
docker logs $CONTAINER_ID > $CASE_DIR/container-stdout.log 2>&1
# Step 2: Export the container filesystem (read-write layer = attacker changes)
echo "Exporting container filesystem..."
docker export $CONTAINER_ID > $CASE_DIR/container-filesystem.tar
# Extract and look for attacker files:
mkdir $CASE_DIR/filesystem
tar xf $CASE_DIR/container-filesystem.tar -C $CASE_DIR/filesystem
# Step 3: Find files added to the container (not in the base image)
# Mount the container's diff (overlay upper layer) on the host:
CONTAINER_GRAPH=$(docker inspect $CONTAINER_ID | jq -r '.[0].GraphDriver.Data.UpperDir')
echo "Attacker-modified files (upper overlay layer):"
find $CONTAINER_GRAPH -type f | sort
# Step 4: Live process snapshot (before restart)
CONTAINER_PID=$(docker inspect $CONTAINER_ID | jq -r '.[0].State.Pid')
echo "Container PID: $CONTAINER_PID"
# Get process tree inside container from host perspective
ps auxf | grep -A100 "container" | head -30
# Network connections from the container
nsenter -t $CONTAINER_PID -n -- ss -tulpn > $CASE_DIR/container-netstat.txt
nsenter -t $CONTAINER_PID -n -- ss -tp > $CASE_DIR/container-connections.txt
# Step 5: Memory dump of container PID
cd $CASE_DIR && avml --pid $CONTAINER_PID memory-container.lime 2>/dev/null || \
gcore -o $CASE_DIR/core $CONTAINER_PID # fallback: gcore dumpNAMESPACE="production"
POD="webapp-abc123"
CASE_DIR="/cases/CASE-2026-009/kubernetes"
mkdir -p $CASE_DIR
# Kubernetes forensic collection before pod restart
# Step 1: Capture pod metadata and events
kubectl describe pod $POD -n $NAMESPACE > $CASE_DIR/pod-describe.txt
kubectl get pod $POD -n $NAMESPACE -o yaml > $CASE_DIR/pod-manifest.yaml
kubectl get events -n $NAMESPACE --field-selector involvedObject.name=$POD \
> $CASE_DIR/pod-events.txt
# Step 2: Capture pod logs
kubectl logs $POD -n $NAMESPACE --all-containers > $CASE_DIR/pod-logs.txt
kubectl logs $POD -n $NAMESPACE --previous > $CASE_DIR/pod-logs-previous.txt 2>/dev/null
# Step 3: Run forensic commands inside the pod (before restart)
# Only if container is still running
kubectl exec $POD -n $NAMESPACE -- ps auxf > $CASE_DIR/processes.txt
kubectl exec $POD -n $NAMESPACE -- ss -tulpn > $CASE_DIR/network.txt
kubectl exec $POD -n $NAMESPACE -- find / -type f -newer /proc -mmin -1440 \
2>/dev/null > $CASE_DIR/recently-modified.txt # files modified in last 24h
kubectl exec $POD -n $NAMESPACE -- cat /etc/crontab > $CASE_DIR/crontab.txt 2>/dev/null
kubectl exec $POD -n $NAMESPACE -- history > $CASE_DIR/shell-history.txt 2>/dev/null
# Step 4: Copy suspicious files from container
kubectl cp $NAMESPACE/$POD:/tmp/ $CASE_DIR/container-tmp/
kubectl cp $NAMESPACE/$POD:/var/www/html/ $CASE_DIR/webroot/ 2>/dev/null
# Step 5: Check Kubernetes API audit logs
# API server audit log location varies by cluster deployment
# For kubeadm clusters:
kubectl logs -n kube-system kube-apiserver-$(hostname) --tail=1000 | \
grep -i "$POD\|$NAMESPACE" > $CASE_DIR/apiserver-logs.txtAnalysts new to container forensics often kubectl exec into a compromised container and run commands like find, ps, and netstat directly inside it. The problem: those tools come from the container image. If the attacker replaced them with modified versions (a common persistence technique), your forensic output is attacker-controlled. More critically, running commands inside the container modifies the container's filesystem timestamps — you've compromised the very evidence you're collecting. The correct approach is to collect from the outside: use nsenter -t [pid] on the host to enter only the container's network or PID namespace while running host binaries, and use the overlay UpperDir on the host filesystem to examine what the attacker added without touching the container. Reserve kubectl exec for commands that cannot be run from the host and document each one explicitly in your case notes.
Q & A
Q: The compromised pod was already restarted before you could collect evidence. What's left?
Significant evidence remains after a pod restart: (1) kubectl logs --previous: Kubernetes retains the previous container's log output. The --previous flag retrieves the logs from the last crashed or terminated container instance. This captures the attacker's commands if they appeared in STDOUT/STDERR. (2) Kubernetes API server audit log: every kubectl command the attacker ran (exec, cp, port-forward) is logged in the API server audit log with the user, command, and timestamp. This survives pod restarts completely. (3) Container image layers: the base image layers are unchanged. If the attacker modified files that were part of the original image, those modifications are gone. But the image itself is available for analysis. (4) Host-level logs: the underlying Kubernetes node's auditd and systemd journal log container lifecycle events. The docker daemon log (journalctl -u docker) records container start/stop and exec events. (5) Network telemetry: NetFlow/PCAP from the node captures all traffic the container sent/received, regardless of pod lifecycle. If you have 72 hours of packet capture on the node, the exfiltration traffic is still there. (6) Persistent volumes: if the pod had a PersistentVolumeClaim, the attacker may have written to the persistent storage, which survived the restart. Check the PVC for modified files.