Chapter 15

Lessons Learned and Detection Uplift

The post-incident review is not a report — it is a change management exercise. Every incident that does not change the security program in a measurable way has wasted its best learning opportunity. Done correctly, lessons learned convert incident pain into permanent detection and resilience improvements.

Scenario

A mid-sized manufacturing company suffers a ransomware attack in January. The IR team writes an excellent post-incident review. The PIR identifies the root causes: an unpatched remote access server, no MFA on VPN, and absence of detection rules for the malware family's lateral movement techniques. The PIR is filed, reviewed by the CISO, and shelved. Patching and MFA get added to a project list that never gets prioritized against other competing work.

Six months later, the same ransomware group — a different affiliate, same toolkit — attacks via the same unpatched remote access server. Same initial access vector. Same lateral movement. Same ransomware deployment. The only difference is the ransom demand has increased because the group now knows this organization paid the first time.

The PIR didn't fail because it was bad. It failed because it had no accountability mechanism, no timeline, and no owner who was responsible for the outcome — only authors who were responsible for writing it. The PIR that drives change is structurally different from the PIR that generates a PDF that goes into a folder.

PIR Structure That Drives Change

The post-incident review is a specific type of document. It has a defined structure because its audience (IR team, security leadership, IT operations, sometimes legal) has different information needs from each section. More importantly, its recommendations must be written in a format that allows each one to be tracked to completion by a specific person by a specific date. Recommendations without owners and dates are wishes.

  Post-Incident Review Document Structure
  ═══════════════════════════════════════════════════════════════════

  Section 1: Executive Summary (1 page)
  ─────────────────────────────────────────────────────────────────
  → What happened (3 sentences, no jargon)
  → Business impact (systems affected, data at risk, downtime)
  → Current status (recovered, monitoring, open questions)
  → Top 3 action items with owner and date

  Section 2: Incident Timeline
  ─────────────────────────────────────────────────────────────────
  → Full chronological timeline from earliest known attacker activity
    to recovery complete
  → Every entry: timestamp, event, data source, analyst who confirmed
  → Annotated with: when was this detected? when was this identified
    forensically? what was the gap between occurrence and detection?
  → Detection gap column: for each event, the time between occurrence
    and when the IR team became aware of it

  Section 3: Root Cause Analysis
  ─────────────────────────────────────────────────────────────────
  → 5 Whys for the initial access vector
  → 5 Whys for the detection failure(s)
  → 5 Whys for the response failure(s) (if any)
  → Root causes are technical AND process — both are equally important

  Section 4: What Went Well
  ─────────────────────────────────────────────────────────────────
  → Specific things the team did right
  → These should be reinforced, not just mentioned
  → Include: detection that worked, decisions that were correct,
    communication that was effective

  Section 5: What Needs to Change
  ─────────────────────────────────────────────────────────────────
  → Every gap identified — technical, process, tooling, people
  → Each item has: description, priority (P1/P2/P3), owner, due date
  → P1 items require completion report; P2/P3 tracked in next quarterly review

  Section 6: MITRE ATT&CK Coverage Map
  ─────────────────────────────────────────────────────────────────
  → Every technique observed in this incident mapped to ATT&CK
  → For each technique: did we have a detection? Did it fire?
  → Gaps where we had no detection are P1 remediation items

Applying 5 Whys to Root Cause Analysis

The 5 Whys technique is a structured root cause analysis method: ask "why" recursively until you reach a root cause that is addressable. In IR, the temptation is to stop at the first technical answer — "the server was unpatched." But "the server was unpatched" is a symptom, not a root cause. The root cause explains why it was unpatched.

  5 Whys: Ransomware via Unpatched VPN Server Example
  ═══════════════════════════════════════════════════════════════════

  Why 1: Why did the attacker get initial access?
  → The VPN server was running an unpatched version of Pulse Secure with
    CVE-2021-22893 (unauthenticated file read / RCE). The patch was available
    for 8 months before the attack.

  Why 2: Why was the patch not applied?
  → The VPN server was in a "critical infrastructure" exception category that
    required additional testing before patches could be applied. The patch was
    in the testing queue.

  Why 3: Why was the patch in the testing queue for 8 months?
  → The testing queue for critical infrastructure is handled by the IT operations
    team. They have 23 systems in the same exception category. The VPN server
    was prioritized below Active Directory and email server patches.

  Why 4: Why was the VPN server deprioritized relative to AD and email?
  → Patch prioritization was based on perceived business criticality, not on
    exploitability. The VPN server was considered "less critical" because it
    had an external WAF in front of it (the WAF did not prevent this exploit).

  Why 5: Why was exploitability not used as a prioritization factor?
  → The patching process predates the organization's use of vulnerability
    scoring that incorporates exploitability (CVSS has EPSS as a complement).
    No one updated the prioritization process when the organization adopted
    the critical infrastructure exception category 3 years ago.

  ROOT CAUSE: The patch prioritization process does not incorporate
  exploitability data (EPSS), and the critical infrastructure exception
  category does not have a maximum dwell-time-in-queue rule.

  REMEDIATION: Update patching SLA: any CVE with EPSS > 0.3 on an
  internet-facing system must be patched within 14 days regardless of
  exception category. Add SLA enforcement alerts to the patch management
  system. Owner: IT Operations Lead. Due: 30 days.
Why most PIRs stop before the real root cause

The first technical answer is comfortable. "The server was unpatched" is true, and it points to a technical action (patch it). But stopping there means the same root cause — a flawed prioritization process — will produce the next unpatched server in 8 months. The 5 Whys forces you past the technical symptom to the process failure. Process failures are harder to articulate and require cross-team coordination to fix, which is exactly why people avoid them. But they are the only things that prevent recurrence. The PIR that only produces technical remediation items will produce the next incident.

Converting Timeline Gaps to Detection Rules

Every item in the incident timeline that was discovered forensically — not detected in real time — represents a detection gap. The timeline gap is the time between when the attacker performed the action and when the IR team became aware of it. The goal of detection uplift is to close these gaps permanently: write and deploy detection rules such that if the same action is repeated in the future, it fires an alert within minutes, not hours or days.

Timeline item (discovered forensically) Detection gap Log source Detection rule concept
Attacker used comsvcs.dll MiniDump to dump LSASS (T-72h before detection) 72 hours Sysmon Event ID 10 (process access) Process access to lsass.exe from non-system processes; or rundll32 with comsvcs.dll argument
Attacker ran net group "Domain Admins" (T-48h before detection) 48 hours Windows Security Event 4688 (process creation) net.exe or net1.exe with "group" + "Domain" arguments; baseline for frequency, alert on spikes
Attacker created local admin account "support" on 12 hosts (T-36h before detection) 36 hours Windows Security Event 4720 (account created) New local account creation outside of IT provisioning hours or from non-admin source
Attacker modified LSA registry key to enable WDigest plaintext auth (T-24h before detection) 24 hours Sysmon Event ID 13 (registry value set) Registry set on HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\WDigest\UseLogonCredential = 1
Attacker deployed ransomware via domain GPO (T-0) 0 (detected at deployment) EDR file creation events; Sysmon Event ID 11 Existing detection fired — no new rule needed for this stage
  From Timeline Gap to Sigma Rule
  ═══════════════════════════════════════════════════════════════════

  Timeline finding: Attacker modified WDigest registry key to enable
  cleartext credential caching. Discovered forensically, gap = 24 hours.

  Sigma rule:
  ───────────────────────────────────────────────────────────────────
  title: WDigest Cleartext Credential Caching Enabled
  id: 614f8f3b-d6e2-4c24-b84e-4a6a21b3a18d
  status: stable
  description: Detects modification of WDigest UseLogonCredential to
               enable cleartext credential caching. Used by attackers
               to harvest credentials from LSASS.
  references:
      - https://attack.mitre.org/techniques/T1112/
  tags:
      - attack.credential_access
      - attack.t1003.001
  logsource:
      product: windows
      category: registry_set
  detection:
      selection:
          TargetObject|endswith: '\WDigest\UseLogonCredential'
          Details: 'DWORD (0x00000001)'
      condition: selection
  falsepositives:
      - Legitimate credential management tools (very rare)
  level: high

  Deployment note: This rule was written because it would have detected
  the specific attacker action that occurred 24 hours before detection.
  It has zero false positives in this environment (tested against 90 days
  of historical telemetry — zero legitimate occurrences).

Measuring MTTD Improvement Over Time

Detection uplift is only valuable if it can be measured. Mean Time to Detect (MTTD) — the average time between an attacker action and an alert firing — is the core metric. After each incident, recalculate the MTTD for each stage of the attack chain. After implementing the new detection rules, backtest against historical telemetry to verify the rules would have fired earlier.

Attack stage MTTD (incident) Root cause of gap New detection rule Expected MTTD (post-uplift)
Initial access (VPN exploit) 0 (no detection until attacker moved laterally) No detection rule for VPN server exploitation; VPN logs not ingested to SIEM Ingest VPN server authentication logs; alert on auth from IP reputation hits; alert on auth followed by immediate admin command execution ~15 minutes (VPN log ingestion latency)
Credential dumping (LSASS) 72 hours (forensic discovery) Sysmon Event ID 10 not deployed; no LSASS access monitoring Deploy Sysmon with EventID 10; rule for comsvcs.dll MiniDump ~3 minutes (real-time EDR/Sysmon alert)
Lateral movement (PsExec to DCs) 48 hours (forensic discovery) DC event logs not centralized; no alert for SMB lateral movement to DCs Centralize DC event logs; alert on non-standard admin tool execution on DCs ~10 minutes (log forwarding latency)
Ransomware deployment 0 (EDR caught this) N/A — existing detection was effective No new rule needed ~2 minutes (no change)
Mental model: MTTD as a program-level metric

Organizations that track MTTD per incident across multiple years develop a data-driven view of whether their security program is improving. A declining MTTD trend means detections are getting better and attacker dwell time is shrinking. A flat MTTD trend despite multiple incidents means lessons are being learned but not implemented. An increasing MTTD trend (getting slower) is a red flag — it means the attack surface is growing faster than the detection program. MTTD is not perfect (it depends on what you can measure), but it is far better than "we haven't had a breach recently" as an indicator of program health.

Addressing Security Debt Exposed by Incidents

Almost every incident exposes security debt — technical or process deficiencies that existed before the incident and created the conditions that allowed the incident to occur or expand. The post-incident review is the moment when security debt becomes visible to business leadership in a way that ordinary security assessments cannot achieve. The incident provides the concrete evidence that the debt is real and that the cost of not addressing it has now been paid.

  Security Debt Visibility Framework
  ═══════════════════════════════════════════════════════════════════

  BEFORE THE INCIDENT (typical security assessment finding):
  ─────────────────────────────────────────────────────────────────
  "The organization has 47 servers running EOL operating systems.
   This represents a risk of unauthorized access or malware spread."

  Business response: Added to the backlog. Scheduled for "next fiscal year."

  AFTER THE INCIDENT (same finding, new context):
  ─────────────────────────────────────────────────────────────────
  "The 12 servers in the lateral movement path — all running Windows
   Server 2012 R2 (EOL since October 2023) — could not be monitored
   with our EDR agent. We had no visibility into attacker activity on
   these systems during the entire 3-week dwell period. Upgrading
   these servers would have given us the telemetry that would have
   reduced MTTD by approximately 2 weeks."

  Business response: Budget approved in the next board meeting.
  Cost justification: Incident cost $1.2M. Server upgrade costs $180k.

  The incident is the business case. Use it.

Recommendation Accountability: Making the PIR Drive Change

The PIR with no accountability mechanism produces no change. Every PIR recommendation must have four fields to be actionable: what, who, when, and how we know it's done.

FieldWhat it meansBad exampleGood example
What Specific, verifiable action — not a vague improvement "Improve patching" "Update patch SLA: CVE with EPSS > 0.3 on internet-facing systems must be patched within 14 days"
Who A named individual, not a team or department "IT Operations" "J. Smith, IT Operations Lead"
When A specific calendar date, not a relative period "Within 60 days" "By November 15, 2026"
Done when Verifiable completion criteria — not "task completed" "Patch SLA updated" "Updated SLA policy published in IT wiki; patch management system configured to alert on violations; 30-day compliance report showing zero violations"

Q & A

Q: How soon after an incident should the PIR be conducted?

The PIR meeting should be scheduled within 5–7 business days of recovery completion. Waiting longer causes memory decay — specific details about why decisions were made become harder to reconstruct. The full PIR document can be drafted over the following week based on the meeting discussion and the incident documentation. However, the one-week meeting window is important: if PIR is treated as a low-priority document to write at leisure, it competes with every other work demand and gets delayed indefinitely. The PIR should be written and reviewed within 2 weeks of recovery. Recommendations should be assigned and due dates set in the same meeting, not in a follow-up.

Q: A critical team member who made some poor decisions during the incident is in the PIR meeting. How do you handle the lessons-learned discussion without it becoming a blame session?

The PIR is explicitly blameless with respect to individuals and explicitly accountable with respect to systems and processes. The ground rule, stated at the start of the meeting: "We are here to understand what happened and why, with the goal of preventing recurrence. We are not here to assign individual blame. Every person in this incident did the best they could with the information and tools they had at the time." The 5 Whys technique naturally redirects blame from individuals to systems: the question is never "why did Person X make this decision" but "why did the system allow this outcome." If the IR lead had incomplete information and made a decision based on it, the PIR question is "why did the lead have incomplete information" (logging gap, communication failure, tool limitation), not "why did the lead make this call."

Q: After writing 12 new Sigma rules from the timeline gaps, how do you prioritize which ones to deploy first?

Prioritize by two factors: the detection gap length (how long did the attacker operate without being detected using this technique) and the specificity-to-noise ratio (how unique is this technique to malicious activity). Rules with the longest detection gaps and the lowest false positive rate should be deployed first. The WDigest registry modification rule (gap: 24 hours, false positive rate: near-zero in typical environments) is a higher priority than a rule for net.exe execution (gap: 48 hours but net.exe is used by legitimate IT processes daily, so tuning is required). High-fidelity, low-noise rules should go live immediately. Rules that require tuning should be deployed in a monitoring-only mode first, with thresholds tuned against a 30-day baseline before being set to alert.

Q: The CISO wants to share the full PIR with the board. What do you include?

The board gets the executive summary only — Section 1. The board needs: what happened (3 sentences), business impact (financial, operational, reputational), current status (is this over), the top 3 action items being taken, and the cost. The board does not need and should not receive the full technical timeline, the complete IOC list, or the detailed attack chain. Two reasons: (1) board members are not security technical practitioners and detailed technical sections obscure the business message; (2) a full technical PIR sent to 12 board members is a significant information security risk in itself — if any board member's email is compromised, the attacker now has the full technical playbook for your environment. The detailed technical sections should have restricted distribution: IR team, CISO, legal counsel, and external IR firm if applicable.

Q: Six months after the incident, how do you know whether the PIR recommendations actually improved security?

Three measurements: (1) Completion audit — have all P1 and P2 recommendations been implemented by their due dates? Assign an owner to check completion criteria (not just "task closed"). (2) Detection coverage — take the MITRE ATT&CK techniques from the incident and run a purple team or detection gap analysis against them. Do the new rules fire on simulated attacker behavior? (3) Incident comparison — when the next incident occurs, compare the MTTD for stages that had detection gaps in the previous incident. If the new rules are deployed and working, the MTTD for those stages should be shorter. This is the ultimate test: not "did we implement the recommendation" but "did the recommendation make us faster at detecting the same technique."

Q: The same attacker group is targeting your industry. Other organizations are sharing IOCs from their incidents. How do you use this intelligence in your PIR and detection uplift?

Shared intelligence from sector peers is extremely valuable for detection uplift and should be integrated into the PIR in two ways. First, add the shared IOCs to your detection rules immediately — C2 domains, malware hashes, infrastructure IPs, and TTPs. These are validated indicators from real incidents against similar organizations; their fidelity is high. Second, compare your incident timeline with the shared TTP descriptions. If the same actor used WDigest registry modification, BloodHound domain enumeration, and GPO-based ransomware deployment against another organization, and you only had one of those three techniques covered by detection rules, you now have a specific detection gap list for the other two — even if they weren't used in your specific incident. Proactively covering techniques this actor uses builds detection ahead of the next attack, not just in response to the current one.