Chapter 04

Evidence Preservation During IR

The single most common IR mistake is also one of the most irreversible: destroying evidence before collecting it. Everything an investigation can tell you is bounded by what was preserved.

Scenario

A significant breach. A hundred compromised hosts. Six months of incident investigation. And then outside counsel sends a letter: they need forensic evidence to establish what customer data the attacker accessed. The IR team confirms: they reimaged the hosts during recovery. No forensic images were taken first. The memory — which would have shown exactly what processes were running and what data was in memory at the time of compromise — was lost when the hosts were rebooted during triage. The Velociraptor deployment they discussed during Preparation planning was never actually deployed. The investigation can answer many questions, but it cannot answer the one the lawyers need answered. That evidence is gone permanently. Not stolen, not corrupted — simply never collected.

Order of Volatility: Why Sequence Matters

Evidence exists in different states of volatility — some evidence disappears the moment the system is rebooted, some disappears when a process terminates, some persists for days after the system is shut down. The order of volatility determines the order of collection: most volatile first. Collecting in the wrong order means the act of collection destroys what you were trying to collect.

  Evidence Order of Volatility — Most Volatile to Least
  ══════════════════════════════════════════════════════════════════════

  MOST VOLATILE — lost on reboot or power cycle
  ─────────────────────────────────────────────────────────────────
  1. CPU registers, cache, running processes
     → Contains: currently executing code, encryption keys in use,
       decrypted data that is encrypted on disk
     → Lost: immediately when process terminates or system reboots
     → Collect via: memory acquisition (live system)

  2. RAM (physical memory)
     → Contains: all running processes, injected code (fileless malware),
       network connections, recently typed passwords, decrypted content,
       attacker tools that only live in memory
     → Lost: on reboot or power cycle (some remnants survive briefly
       on some hardware — cold boot attack — but treat as lost)
     → Collect via: FTK Imager Live, WinPmem, Magnet RAM Capture,
       Velociraptor memory acquisition

  3. Network connections (current state)
     → Contains: active C2 connections (PID, remote IP/port, state),
       listening ports, ARP table (recently seen hosts)
     → Lost: connections close, ARP entries expire (minutes to hours)
     → Collect via: netstat output, EDR network telemetry snapshot

  4. Running processes and loaded modules
     → Contains: process list with command lines and parent PIDs,
       loaded DLLs (including injected ones), handles held open
     → Lost: on process termination or reboot
     → Collect via: EDR process tree snapshot, Velociraptor live query

  MODERATELY VOLATILE — survives reboot, not permanent
  ─────────────────────────────────────────────────────────────────
  5. Windows Event Logs (in-memory buffer before write to disk)
     → Some events are in memory buffer; most are written to disk
     → Collect via: wevtutil export, EDR log forwarding

  6. Temporary files, browser cache, recent file access (LRU lists)
     → Survives reboot but may be cleaned by OS or applications
     → Collect via: triage collection (KAPE, Velociraptor artifact)

  LEAST VOLATILE — survives reboots
  ─────────────────────────────────────────────────────────────────
  7. File system (disk) — registry, logs, malware files, artifacts
     → Survives until overwritten. Deleted files may be recoverable.
     → Collect via: disk image (dd, FTK Imager), targeted artifact
       collection (KAPE triage), VSS shadow copy access

  8. Cloud logs, SIEM, EDR telemetry (already shipped off-host)
     → Already preserved centrally; not affected by host state
     → Subject to retention window expiration (see below)
The most common IR evidence mistake: rebooting the host

Rebooting is the single action most likely to destroy critical evidence. Analysts who discover a compromised host often reboot it instinctively — to "apply updates," "clear the infection," or "start fresh." This destroys RAM (which may contain the only copy of a fileless malware implant, encryption keys, plaintext credentials, or injected shellcode), terminates running processes (which may be the only evidence of what was executing), and closes network connections (which may be the only real-time evidence of active C2). If a host may be compromised, the rule is: do not touch it until the IR lead gives direction. If touching it is unavoidable, acquire memory first. If acquiring memory is not possible, document the state as thoroughly as possible before doing anything that might destroy it.

Memory Acquisition: Tools and What They Capture

Memory acquisition is the process of creating a bit-for-bit copy of a system's physical RAM to a file that can be analyzed offline. A memory image contains everything that was in memory at the moment of acquisition: running processes, injected code, network socket state, kernel data structures, and — critically — malware that only exists in memory and leaves no disk artifacts.

ToolAcquisition methodStrengthsLimitations
FTK Imager (Live) Load as process, read physical memory via device driver Widely accepted in forensic/legal contexts. Includes memory and can also image disk simultaneously. Requires running an executable on the compromised host (modifies evidence). Needs admin rights.
WinPmem Kernel driver that maps physical memory; output to raw or AFF4 format Open source, command-line, scriptable. Creates raw image suitable for Volatility. Small footprint. Driver must be loaded (generates Windows event). Some EDRs may flag or block the driver.
Magnet RAM Capture Loads kernel driver, writes physical memory to file Free, GUI and CLI modes, widely used in corporate IR. Good documentation. Same driver-loading considerations as WinPmem.
Velociraptor Agent-based acquisition triggered remotely from IR console Scalable to hundreds of hosts simultaneously. No analyst interaction with each host. Already deployed as part of preparation. Requires Velociraptor pre-deployed. Memory acquisition artifact creates significant network traffic for large images.
EDR memory dump EDR platform capability (CrowdStrike RTR, Defender XDR) No additional tool deployment required. Fully remote. Not all EDR platforms support full physical memory acquisition. May capture process memory, not full physical memory image.
Why fileless malware makes memory acquisition non-optional

Fileless malware lives entirely in memory — it is injected into a legitimate process, runs as shellcode or a reflectively-loaded DLL, and leaves no file on disk that traditional antivirus or EDR file-scanning can find. Examples: Cobalt Strike in-memory beacon injected into svchost.exe, PowerShell Empire payloads executed via WMI, process hollowing where legitimate process memory is replaced with attacker code. If you do not acquire memory before rebooting or reimaging, this class of malware leaves no recoverable artifact. The investigation can confirm the host was compromised (based on network indicators and process events) but cannot establish what specific code was executing, what it was doing, or whether it contained functionality beyond C2 communication. For legal and regulatory purposes, "we cannot determine what the attacker's tool was doing" is a far weaker position than "the memory image shows the attacker's tool was only performing network beaconing with no data access." Memory acquisition converts an unknowable into a known.

Disk Imaging vs Live Triage: When Each Is Appropriate

A full forensic disk image is a bit-for-bit copy of the entire drive — it captures everything: allocated files, deleted files, unallocated space, file system metadata, slack space. It is the gold standard for forensic completeness. A triage collection is a targeted collection of known-important artifacts — event logs, registry, prefetch, recent files, browser history — without capturing the full disk. Each has its place.

  Full Disk Image vs Triage Collection Decision
  ══════════════════════════════════════════════════════════════════════

  FULL DISK IMAGE
  ─────────────────────────────────────────────────────────────────
  When to use:
  • Litigation is likely — legal counsel requires forensic-grade evidence
  • Deleted file recovery is needed (attacker deleted their tools)
  • Complete file system timeline analysis required
  • Slack space or unallocated space analysis needed
  • Low-cost endpoint where the full image is practical

  Time: 500GB SSD → 2–4 hours via USB3. 1TB → 4–8 hours.
        Impractical for mass collection (100 hosts × 4 hours each).
  Tools: FTK Imager, dd, dc3dd, Guymager

  TRIAGE COLLECTION
  ─────────────────────────────────────────────────────────────────
  When to use:
  • Large incident with many compromised hosts (triage first, image later
    if specific hosts warrant full analysis)
  • Evidence collection during active incident where speed matters
  • Cloud/virtual environments where full disk image is expensive
  • When specific artifact categories are known to be relevant

  Time: KAPE triage collection → 5–20 minutes per host.
        Velociraptor remote artifact collection → scalable to 100+ hosts
        simultaneously.
  Tools: KAPE (Kroll Artifact Parser and Extractor), Velociraptor,
         EZ Tools collection scripts

  RECOMMENDED WORKFLOW FOR LARGE INCIDENTS:
  1. Triage collection on all confirmed hosts (fast, scalable)
  2. Memory acquisition on all confirmed hosts before isolation
  3. Full disk image on highest-priority hosts (patient zero,
     data repositories, hosts with strongest evidence)
  4. EDR telemetry fills gaps for hosts where images are impractical

Chain of Custody

Chain of custody is the documented, unbroken record of who collected an item of evidence, how it was handled, and who had access to it. It exists to establish that evidence has not been tampered with between collection and presentation. In a criminal or civil case, an unbroken chain of custody is often required for evidence to be admissible. In a regulatory investigation, it demonstrates that your evidence collection process was controlled and reliable.

Chain of custody elementWhat to document
CollectionWho collected it, what tool was used, timestamp (UTC), hash of the evidence file at time of collection
TransferWho transferred it to whom, by what method, timestamp, hash verified on receipt
StorageWhere it is stored (physical location or cloud storage), who has access, encryption status
AnalysisWho analyzed it, on what workstation, what tools were used, timestamp
Hash verificationSHA-256 hash of original evidence verified before and after each stage to confirm integrity
Mental model: every action on a live system modifies evidence

Running any process on a compromised system modifies the system state: it changes the process list, modifies memory, may write to disk (temp files, prefetch entries), and updates timestamps. This is unavoidable — you cannot collect evidence without interacting with the system. The goal is not zero modification but documented, minimized, and consistent modification. Every tool you run on a compromised host should be logged: what tool, what version, what time, what it accessed. This documents the modifications you made so that any analyst reviewing the evidence later can distinguish your artifacts from the attacker's artifacts. Forensic analysts call this "maintaining forensic soundness" — not the impossible goal of zero impact, but the achievable goal of known and documented impact.

Log Source Expiration Windows

Log data preserved centrally (in SIEM, in EDR, in cloud audit services) is bounded by retention windows. These windows are often shorter than IR teams assume, and once they expire, the data is gone. Understanding these windows before an incident determines what evidence will be available to you when you need it.

Log sourceDefault retentionRecommended retentionWhat expires that matters
Windows Event Log (on host)20MB circular buffer (~7 days of busy system)N/A — forward to SIEM, don't rely on localAuthentication events, process creation, security policy changes
SIEM (forwarded events)Varies: 30–90 days common, often less1 year minimum, 3 years for regulated industriesEverything — SIEM is often the single source of truth for historical events
EDR telemetry30–90 days depending on vendor/tier90 days minimum, 1 year preferredProcess tree history, network connections, file events
AWS CloudTrail90 days in Event History; S3 indefinite if configured1 year in S3; enable all regions and all accountsIAM actions, API calls, configuration changes
Azure Audit Logs30 days in portal; Sentinel/Log Analytics: 90 days1 year in Log Analytics or archive storageSign-in events, Azure AD changes, resource modifications
Microsoft 365 Unified Audit Log90 days (E3); 1 year (E5/Purview)1 year minimumEmail access, SharePoint access, Teams activity — critical for BEC investigations
DNS query logsOften not collected at all90 days, collected at resolverC2 domain queries — often the only pre-detection evidence of malware
Wild: the investigation that was impossible because of a 30-day retention window

A financial services firm was investigating a suspected insider threat. The security team believed an employee had been accessing and exfiltrating sensitive customer records for several months before their departure. The M365 Unified Audit Log — the authoritative record of which files were accessed, downloaded, and emailed — had a 90-day retention window. The suspected activity had occurred more than 90 days before the investigation began. The data access records were gone. The investigation could establish that the employee had left with a company laptop, but it could not establish what data was accessed, in what quantity, or when. The regulatory notification decision — which turned on whether customer data was confirmed exfiltrated — could not be made with confidence. All of this would have been resolved with a 1-year retention window. The cost difference between 90-day and 1-year retention for M365 audit logs is trivial. The consequence of the shorter window was an unresolvable regulatory situation.

Velociraptor: Live Triage at Scale

Velociraptor is an open-source endpoint visibility and IR platform that transforms live triage from a host-by-host manual operation into a fleet-wide operation executed from a central console. Where traditional forensics required a USB drive, physical access, and 30 minutes per host, Velociraptor enables collection from 100 hosts simultaneously from a remote console in minutes. For large IR engagements, it changes what is operationally possible.

  Velociraptor IR Use Cases
  ══════════════════════════════════════════════════════════════════════

  TRIAGE AT SCALE
  ─────────────────────────────────────────────────────────────────
  Hunt across all endpoints simultaneously:
  • "Show me all hosts with this process running right now"
  • "Show me all hosts that have executed this file hash in the last 30 days"
  • "Show me all hosts with a network connection to this IP range"
  • "Collect Windows.KapeFiles.Targets from all hosts in this group"

  MEMORY ACQUISITION
  ─────────────────────────────────────────────────────────────────
  Windows.Memory.Acquisition artifact → triggers WinPmem on remote host,
  uploads memory image to Velociraptor server.
  For 100 hosts: run simultaneously, images arrive as they complete.
  No physical access. No USB drives. No per-host manual steps.

  ARTIFACT COLLECTION
  ─────────────────────────────────────────────────────────────────
  Pre-built artifacts cover: event logs, registry, prefetch, Amcache,
  Shimcache, LNK files, browser history, scheduled tasks, services,
  network connections, DNS cache, WMI subscriptions, startup items.
  Custom artifacts can be written in VQL (Velociraptor Query Language)
  for environment-specific collection needs.

  REAL-TIME HUNTING
  ─────────────────────────────────────────────────────────────────
  During active IR: hunt for indicators as they are discovered.
  "We found Cobalt Strike pipe name \\.\pipe\mojo.XXXX — let's hunt
  for all hosts with this pipe name right now."
  Results in under 60 seconds across the entire fleet.

Forensic Notes as Legal Documents

Every action taken during an IR investigation is a potential exhibit in future litigation, regulatory proceedings, or insurance claims. Forensic notes are not informal scratch paper — they are contemporaneous records of what was done, when, by whom, and what was found. They document the investigation's integrity. They explain the analyst's reasoning. And they protect the investigation team when their methods are challenged.

Note elementWhy it matters legally/forensically
Timestamp (UTC)Establishes the time sequence of the investigation relative to the incident timeline. All timestamps must be UTC — inconsistent time zones are a source of error and confusion in multi-system investigations.
Analyst name and roleIdentifies who performed each action for chain of custody and for expert witness qualification if testimony is required.
System state at start of analysisDocuments what the system was doing when analysis began — whether it was live or isolated, what processes were running, whether containment had already been applied.
Tool name and versionRequired for reproducibility. "I used Volatility 3.2.1 with the windows.cmdline plugin" allows another analyst to replicate and verify findings.
Finding verbatimThe raw output, hash values, and artifact text — not just the conclusion. "I saw malware" is an assertion. The actual command output or screenshot is evidence.
Analyst interpretationClearly separated from the raw finding: "The above registry entry represents persistence via a run key, consistent with attacker-established persistence at account login."

Q & A

Why is SHA-256 used for evidence hashing instead of MD5?

MD5 is cryptographically broken — it is possible to construct two different files that produce the same MD5 hash (a collision). This means MD5 hash matching cannot prove that two files are identical, because an adversary (or a careless analyst) could theoretically produce a modified file with the same MD5 as the original. SHA-256 has no known practical collisions and is the current forensic standard for evidence integrity verification. In practice, most forensic tools still compute both MD5 and SHA-256 for legacy compatibility with older case management systems, but SHA-256 is the hash you would rely on in a legal challenge. For large evidence sets (hundreds of disk images), hash computation time is a consideration — SHA-256 is slower than MD5 — but the integrity guarantee justifies the overhead.

A compromised host must be kept running for business continuity — you cannot take it offline to image it. How do you collect forensic evidence from a live running system without disrupting it?

Live forensics — collecting from a running system — is a standard technique with well-established tools and accepted methodology. The key principle: use tools designed for live forensic acquisition (FTK Imager Live, WinPmem, Velociraptor) rather than ad-hoc approaches. For memory: run WinPmem or Magnet RAM Capture; the acquisition takes 5–15 minutes and causes minimal performance impact (I/O bound, not CPU bound). For disk artifacts: use Velociraptor artifact collection or KAPE; targeted artifact collection can complete in under 5 minutes. For network state: capture current connections and DNS cache before the state changes. Document everything you ran on the system and when. The one limitation of live forensics is that the acquired memory image includes the memory used by your acquisition tool — this is expected and well-understood, and does not invalidate the forensic value of the image. What you cannot do from a live running system is acquire an unmodified disk image — but in most corporate IR cases, targeted artifact collection is sufficient for the investigation.

How do you preserve evidence from a cloud instance that will be terminated?

Cloud instances are ephemeral by design — when terminated, their instance storage is gone. Evidence preservation requires action before termination. For AWS EC2: take an EBS snapshot (captures disk state at the moment of snapshot creation), export memory via SSM Run Command to an S3 bucket using WinPmem or similar if SSM agent is installed, capture instance metadata (IAM role, security groups, VPC flow logs). For Azure VMs: create a disk snapshot via Azure portal or CLI before deallocating, capture memory if Azure's serial console access or Run Command is available. In both cases: CloudTrail/Activity Log records for the instance and any associated IAM principal should be explicitly exported to durable storage before they are relied on solely from the cloud provider's retention window. One common mistake: capturing a snapshot of a terminated instance. Once the instance is terminated and the EBS volume is released, snapshots cannot be created from it. Preservation must happen before termination, not after.

What is the right balance between thorough evidence collection and the speed needed during active IR?

Prioritize by evidence volatility and investigation need. The first 30 minutes of any IR engagement should focus exclusively on volatile evidence: memory acquisition from confirmed compromised hosts and documentation of current network state. These cannot be recovered if delayed. Disk artifacts (event logs, registry, prefetch) are less volatile and can be collected in parallel with containment work over the next hour. Full disk images can follow at a measured pace based on investigative priority — not every compromised host needs a full image, and full images from lower-priority hosts can wait until the active phase is over. The worst outcome is letting the pressure of active IR prevent any evidence collection until after recovery, at which point reimaging has destroyed everything. Set a floor: minimum collection is memory (before isolation) and triage artifacts (within 2 hours of confirmation). Everything else is a function of time and resources.

A manager says "we don't need forensic images — we have EDR telemetry." Is this sufficient?

EDR telemetry is excellent for many investigative questions and is often sufficient for operational IR — determining scope, understanding the attack chain, supporting containment and eradication decisions. It is not a complete substitute for forensic acquisition in cases where legal proceedings are possible. EDR telemetry captures events as they happen — process creation, network connections, file writes — but it does not capture the state of memory at a specific moment, deleted files, unallocated disk space, or artifacts that occurred before EDR was deployed. For a routine corporate security incident with no litigation anticipated, EDR-only investigation is often pragmatically appropriate. For incidents involving: likely legal proceedings, regulatory investigation, insider threat, significant data exfiltration, or nation-state actors — full forensic acquisition from key hosts is appropriate and should be established in policy before the incident happens, not debated during it.

How long should forensic evidence be retained after an incident is closed?

The answer is driven by legal and regulatory requirements, not operational convenience. General guidance: incident records (timeline, decision log, reports) should be retained for 7 years — litigation can arise years after an incident, and statutes of limitations in many jurisdictions extend to 5–7 years. Forensic images and raw evidence should be retained for the duration of any active legal, regulatory, or insurance proceedings, plus 1 year. Where no proceedings are anticipated, 3–5 years is a reasonable minimum for significant incidents. Shorter-term operational evidence (triage collections from low-priority hosts in a large incident) may be retained for 1–2 years. Before destroying any evidence: confirm with legal counsel that no litigation holds are in place and that all regulatory reporting timelines have been satisfied. Document the destruction with a hash verification log showing what was destroyed, when, and by what method.