Lessons Learned and Detection Uplift
The post-incident review is not a report — it is a change management exercise. Every incident that does not change the security program in a measurable way has wasted its best learning opportunity. Done correctly, lessons learned convert incident pain into permanent detection and resilience improvements.
A mid-sized manufacturing company suffers a ransomware attack in January. The IR team writes an excellent post-incident review. The PIR identifies the root causes: an unpatched remote access server, no MFA on VPN, and absence of detection rules for the malware family's lateral movement techniques. The PIR is filed, reviewed by the CISO, and shelved. Patching and MFA get added to a project list that never gets prioritized against other competing work.
Six months later, the same ransomware group — a different affiliate, same toolkit — attacks via the same unpatched remote access server. Same initial access vector. Same lateral movement. Same ransomware deployment. The only difference is the ransom demand has increased because the group now knows this organization paid the first time.
The PIR didn't fail because it was bad. It failed because it had no accountability mechanism, no timeline, and no owner who was responsible for the outcome — only authors who were responsible for writing it. The PIR that drives change is structurally different from the PIR that generates a PDF that goes into a folder.
PIR Structure That Drives Change
The post-incident review is a specific type of document. It has a defined structure because its audience (IR team, security leadership, IT operations, sometimes legal) has different information needs from each section. More importantly, its recommendations must be written in a format that allows each one to be tracked to completion by a specific person by a specific date. Recommendations without owners and dates are wishes.
Post-Incident Review Document Structure
═══════════════════════════════════════════════════════════════════
Section 1: Executive Summary (1 page)
─────────────────────────────────────────────────────────────────
→ What happened (3 sentences, no jargon)
→ Business impact (systems affected, data at risk, downtime)
→ Current status (recovered, monitoring, open questions)
→ Top 3 action items with owner and date
Section 2: Incident Timeline
─────────────────────────────────────────────────────────────────
→ Full chronological timeline from earliest known attacker activity
to recovery complete
→ Every entry: timestamp, event, data source, analyst who confirmed
→ Annotated with: when was this detected? when was this identified
forensically? what was the gap between occurrence and detection?
→ Detection gap column: for each event, the time between occurrence
and when the IR team became aware of it
Section 3: Root Cause Analysis
─────────────────────────────────────────────────────────────────
→ 5 Whys for the initial access vector
→ 5 Whys for the detection failure(s)
→ 5 Whys for the response failure(s) (if any)
→ Root causes are technical AND process — both are equally important
Section 4: What Went Well
─────────────────────────────────────────────────────────────────
→ Specific things the team did right
→ These should be reinforced, not just mentioned
→ Include: detection that worked, decisions that were correct,
communication that was effective
Section 5: What Needs to Change
─────────────────────────────────────────────────────────────────
→ Every gap identified — technical, process, tooling, people
→ Each item has: description, priority (P1/P2/P3), owner, due date
→ P1 items require completion report; P2/P3 tracked in next quarterly review
Section 6: MITRE ATT&CK Coverage Map
─────────────────────────────────────────────────────────────────
→ Every technique observed in this incident mapped to ATT&CK
→ For each technique: did we have a detection? Did it fire?
→ Gaps where we had no detection are P1 remediation items
Applying 5 Whys to Root Cause Analysis
The 5 Whys technique is a structured root cause analysis method: ask "why" recursively until you reach a root cause that is addressable. In IR, the temptation is to stop at the first technical answer — "the server was unpatched." But "the server was unpatched" is a symptom, not a root cause. The root cause explains why it was unpatched.
5 Whys: Ransomware via Unpatched VPN Server Example
═══════════════════════════════════════════════════════════════════
Why 1: Why did the attacker get initial access?
→ The VPN server was running an unpatched version of Pulse Secure with
CVE-2021-22893 (unauthenticated file read / RCE). The patch was available
for 8 months before the attack.
Why 2: Why was the patch not applied?
→ The VPN server was in a "critical infrastructure" exception category that
required additional testing before patches could be applied. The patch was
in the testing queue.
Why 3: Why was the patch in the testing queue for 8 months?
→ The testing queue for critical infrastructure is handled by the IT operations
team. They have 23 systems in the same exception category. The VPN server
was prioritized below Active Directory and email server patches.
Why 4: Why was the VPN server deprioritized relative to AD and email?
→ Patch prioritization was based on perceived business criticality, not on
exploitability. The VPN server was considered "less critical" because it
had an external WAF in front of it (the WAF did not prevent this exploit).
Why 5: Why was exploitability not used as a prioritization factor?
→ The patching process predates the organization's use of vulnerability
scoring that incorporates exploitability (CVSS has EPSS as a complement).
No one updated the prioritization process when the organization adopted
the critical infrastructure exception category 3 years ago.
ROOT CAUSE: The patch prioritization process does not incorporate
exploitability data (EPSS), and the critical infrastructure exception
category does not have a maximum dwell-time-in-queue rule.
REMEDIATION: Update patching SLA: any CVE with EPSS > 0.3 on an
internet-facing system must be patched within 14 days regardless of
exception category. Add SLA enforcement alerts to the patch management
system. Owner: IT Operations Lead. Due: 30 days.
The first technical answer is comfortable. "The server was unpatched" is true, and it points to a technical action (patch it). But stopping there means the same root cause — a flawed prioritization process — will produce the next unpatched server in 8 months. The 5 Whys forces you past the technical symptom to the process failure. Process failures are harder to articulate and require cross-team coordination to fix, which is exactly why people avoid them. But they are the only things that prevent recurrence. The PIR that only produces technical remediation items will produce the next incident.
Converting Timeline Gaps to Detection Rules
Every item in the incident timeline that was discovered forensically — not detected in real time — represents a detection gap. The timeline gap is the time between when the attacker performed the action and when the IR team became aware of it. The goal of detection uplift is to close these gaps permanently: write and deploy detection rules such that if the same action is repeated in the future, it fires an alert within minutes, not hours or days.
| Timeline item (discovered forensically) | Detection gap | Log source | Detection rule concept |
|---|---|---|---|
| Attacker used comsvcs.dll MiniDump to dump LSASS (T-72h before detection) | 72 hours | Sysmon Event ID 10 (process access) | Process access to lsass.exe from non-system processes; or rundll32 with comsvcs.dll argument |
| Attacker ran net group "Domain Admins" (T-48h before detection) | 48 hours | Windows Security Event 4688 (process creation) | net.exe or net1.exe with "group" + "Domain" arguments; baseline for frequency, alert on spikes |
| Attacker created local admin account "support" on 12 hosts (T-36h before detection) | 36 hours | Windows Security Event 4720 (account created) | New local account creation outside of IT provisioning hours or from non-admin source |
| Attacker modified LSA registry key to enable WDigest plaintext auth (T-24h before detection) | 24 hours | Sysmon Event ID 13 (registry value set) | Registry set on HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\WDigest\UseLogonCredential = 1 |
| Attacker deployed ransomware via domain GPO (T-0) | 0 (detected at deployment) | EDR file creation events; Sysmon Event ID 11 | Existing detection fired — no new rule needed for this stage |
From Timeline Gap to Sigma Rule
═══════════════════════════════════════════════════════════════════
Timeline finding: Attacker modified WDigest registry key to enable
cleartext credential caching. Discovered forensically, gap = 24 hours.
Sigma rule:
───────────────────────────────────────────────────────────────────
title: WDigest Cleartext Credential Caching Enabled
id: 614f8f3b-d6e2-4c24-b84e-4a6a21b3a18d
status: stable
description: Detects modification of WDigest UseLogonCredential to
enable cleartext credential caching. Used by attackers
to harvest credentials from LSASS.
references:
- https://attack.mitre.org/techniques/T1112/
tags:
- attack.credential_access
- attack.t1003.001
logsource:
product: windows
category: registry_set
detection:
selection:
TargetObject|endswith: '\WDigest\UseLogonCredential'
Details: 'DWORD (0x00000001)'
condition: selection
falsepositives:
- Legitimate credential management tools (very rare)
level: high
Deployment note: This rule was written because it would have detected
the specific attacker action that occurred 24 hours before detection.
It has zero false positives in this environment (tested against 90 days
of historical telemetry — zero legitimate occurrences).
Measuring MTTD Improvement Over Time
Detection uplift is only valuable if it can be measured. Mean Time to Detect (MTTD) — the average time between an attacker action and an alert firing — is the core metric. After each incident, recalculate the MTTD for each stage of the attack chain. After implementing the new detection rules, backtest against historical telemetry to verify the rules would have fired earlier.
| Attack stage | MTTD (incident) | Root cause of gap | New detection rule | Expected MTTD (post-uplift) |
|---|---|---|---|---|
| Initial access (VPN exploit) | 0 (no detection until attacker moved laterally) | No detection rule for VPN server exploitation; VPN logs not ingested to SIEM | Ingest VPN server authentication logs; alert on auth from IP reputation hits; alert on auth followed by immediate admin command execution | ~15 minutes (VPN log ingestion latency) |
| Credential dumping (LSASS) | 72 hours (forensic discovery) | Sysmon Event ID 10 not deployed; no LSASS access monitoring | Deploy Sysmon with EventID 10; rule for comsvcs.dll MiniDump | ~3 minutes (real-time EDR/Sysmon alert) |
| Lateral movement (PsExec to DCs) | 48 hours (forensic discovery) | DC event logs not centralized; no alert for SMB lateral movement to DCs | Centralize DC event logs; alert on non-standard admin tool execution on DCs | ~10 minutes (log forwarding latency) |
| Ransomware deployment | 0 (EDR caught this) | N/A — existing detection was effective | No new rule needed | ~2 minutes (no change) |
Organizations that track MTTD per incident across multiple years develop a data-driven view of whether their security program is improving. A declining MTTD trend means detections are getting better and attacker dwell time is shrinking. A flat MTTD trend despite multiple incidents means lessons are being learned but not implemented. An increasing MTTD trend (getting slower) is a red flag — it means the attack surface is growing faster than the detection program. MTTD is not perfect (it depends on what you can measure), but it is far better than "we haven't had a breach recently" as an indicator of program health.
Addressing Security Debt Exposed by Incidents
Almost every incident exposes security debt — technical or process deficiencies that existed before the incident and created the conditions that allowed the incident to occur or expand. The post-incident review is the moment when security debt becomes visible to business leadership in a way that ordinary security assessments cannot achieve. The incident provides the concrete evidence that the debt is real and that the cost of not addressing it has now been paid.
Security Debt Visibility Framework ═══════════════════════════════════════════════════════════════════ BEFORE THE INCIDENT (typical security assessment finding): ───────────────────────────────────────────────────────────────── "The organization has 47 servers running EOL operating systems. This represents a risk of unauthorized access or malware spread." Business response: Added to the backlog. Scheduled for "next fiscal year." AFTER THE INCIDENT (same finding, new context): ───────────────────────────────────────────────────────────────── "The 12 servers in the lateral movement path — all running Windows Server 2012 R2 (EOL since October 2023) — could not be monitored with our EDR agent. We had no visibility into attacker activity on these systems during the entire 3-week dwell period. Upgrading these servers would have given us the telemetry that would have reduced MTTD by approximately 2 weeks." Business response: Budget approved in the next board meeting. Cost justification: Incident cost $1.2M. Server upgrade costs $180k. The incident is the business case. Use it.
Recommendation Accountability: Making the PIR Drive Change
The PIR with no accountability mechanism produces no change. Every PIR recommendation must have four fields to be actionable: what, who, when, and how we know it's done.
| Field | What it means | Bad example | Good example |
|---|---|---|---|
| What | Specific, verifiable action — not a vague improvement | "Improve patching" | "Update patch SLA: CVE with EPSS > 0.3 on internet-facing systems must be patched within 14 days" |
| Who | A named individual, not a team or department | "IT Operations" | "J. Smith, IT Operations Lead" |
| When | A specific calendar date, not a relative period | "Within 60 days" | "By November 15, 2026" |
| Done when | Verifiable completion criteria — not "task completed" | "Patch SLA updated" | "Updated SLA policy published in IT wiki; patch management system configured to alert on violations; 30-day compliance report showing zero violations" |
Q & A
Q: How soon after an incident should the PIR be conducted?
The PIR meeting should be scheduled within 5–7 business days of recovery completion. Waiting longer causes memory decay — specific details about why decisions were made become harder to reconstruct. The full PIR document can be drafted over the following week based on the meeting discussion and the incident documentation. However, the one-week meeting window is important: if PIR is treated as a low-priority document to write at leisure, it competes with every other work demand and gets delayed indefinitely. The PIR should be written and reviewed within 2 weeks of recovery. Recommendations should be assigned and due dates set in the same meeting, not in a follow-up.
Q: A critical team member who made some poor decisions during the incident is in the PIR meeting. How do you handle the lessons-learned discussion without it becoming a blame session?
The PIR is explicitly blameless with respect to individuals and explicitly accountable with respect to systems and processes. The ground rule, stated at the start of the meeting: "We are here to understand what happened and why, with the goal of preventing recurrence. We are not here to assign individual blame. Every person in this incident did the best they could with the information and tools they had at the time." The 5 Whys technique naturally redirects blame from individuals to systems: the question is never "why did Person X make this decision" but "why did the system allow this outcome." If the IR lead had incomplete information and made a decision based on it, the PIR question is "why did the lead have incomplete information" (logging gap, communication failure, tool limitation), not "why did the lead make this call."
Q: After writing 12 new Sigma rules from the timeline gaps, how do you prioritize which ones to deploy first?
Prioritize by two factors: the detection gap length (how long did the attacker operate without being detected using this technique) and the specificity-to-noise ratio (how unique is this technique to malicious activity). Rules with the longest detection gaps and the lowest false positive rate should be deployed first. The WDigest registry modification rule (gap: 24 hours, false positive rate: near-zero in typical environments) is a higher priority than a rule for net.exe execution (gap: 48 hours but net.exe is used by legitimate IT processes daily, so tuning is required). High-fidelity, low-noise rules should go live immediately. Rules that require tuning should be deployed in a monitoring-only mode first, with thresholds tuned against a 30-day baseline before being set to alert.
Q: The CISO wants to share the full PIR with the board. What do you include?
The board gets the executive summary only — Section 1. The board needs: what happened (3 sentences), business impact (financial, operational, reputational), current status (is this over), the top 3 action items being taken, and the cost. The board does not need and should not receive the full technical timeline, the complete IOC list, or the detailed attack chain. Two reasons: (1) board members are not security technical practitioners and detailed technical sections obscure the business message; (2) a full technical PIR sent to 12 board members is a significant information security risk in itself — if any board member's email is compromised, the attacker now has the full technical playbook for your environment. The detailed technical sections should have restricted distribution: IR team, CISO, legal counsel, and external IR firm if applicable.
Q: Six months after the incident, how do you know whether the PIR recommendations actually improved security?
Three measurements: (1) Completion audit — have all P1 and P2 recommendations been implemented by their due dates? Assign an owner to check completion criteria (not just "task closed"). (2) Detection coverage — take the MITRE ATT&CK techniques from the incident and run a purple team or detection gap analysis against them. Do the new rules fire on simulated attacker behavior? (3) Incident comparison — when the next incident occurs, compare the MTTD for stages that had detection gaps in the previous incident. If the new rules are deployed and working, the MTTD for those stages should be shorter. This is the ultimate test: not "did we implement the recommendation" but "did the recommendation make us faster at detecting the same technique."
Q: The same attacker group is targeting your industry. Other organizations are sharing IOCs from their incidents. How do you use this intelligence in your PIR and detection uplift?
Shared intelligence from sector peers is extremely valuable for detection uplift and should be integrated into the PIR in two ways. First, add the shared IOCs to your detection rules immediately — C2 domains, malware hashes, infrastructure IPs, and TTPs. These are validated indicators from real incidents against similar organizations; their fidelity is high. Second, compare your incident timeline with the shared TTP descriptions. If the same actor used WDigest registry modification, BloodHound domain enumeration, and GPO-based ransomware deployment against another organization, and you only had one of those three techniques covered by detection rules, you now have a specific detection gap list for the other two — even if they weren't used in your specific incident. Proactively covering techniques this actor uses builds detection ahead of the next attack, not just in response to the current one.