Lessons Learned and Detection Uplift
Structuring the post-incident review, writing detection rules from incident IOCs and behaviors, updating the runbook based on gaps found, and turning each incident into a measurable security improvement.
A year after a serious ransomware incident, the CISO asks: "What did we change after that incident to make sure it doesn't happen again?" The IR lead says "we patched the vulnerability and reset all the passwords." The CISO asks "did we add any new detections?" Silence. The incident generated months of work but zero new detections in the SIEM. No new runbook updates. No improvement to the tooling. The next time the same attacker technique is used — even by a different threat actor — it will take just as long to detect and just as long to respond. Every incident is an investment opportunity: you learn exactly what the attacker's technique looks like in your specific environment. That learning has to turn into code.
The Post-Incident Review
The PIR is a structured blameless review held 5-7 days after recovery. Its goal is to identify systemic failures, not to assign fault.
Post-Incident Review Agenda (90 minutes)
═══════════════════════════════════════════════════════════════════
Participants: IR team, SOC lead, infrastructure lead affected,
CISO, (optional) external IR firm lead if engaged
1. Timeline review (20 min):
Walk the full incident timeline from initial access to recovery
Identify: detection time, response time, containment time
Document: dwell time, mean time to detect (MTTD), MTTR
2. What went well (15 min):
Specific things that worked — be concrete
"The EDR bulk isolation took under 2 minutes" is useful
"The team did great" is not useful
3. What didn't go well (25 min):
Without blame: what process, tooling, or gap failed?
Examples: "We didn't have C2 blocked at the proxy pre-defined"
"The network team needed a change window we didn't have"
"The backup predating the compromise was 60 days old"
4. Action items (30 min):
Each gap → specific action item with owner and due date
No "review our processes" — must be specific and measurable
Examples: "Deploy DNS RPZ blocking for malware categories by [date]"
"Add emergency change authorization to incident runbook by [date]"
"Increase backup frequency to 24h for Tier 1 systems by [date]"
PIR output: written report, action item tracker, detection gap list
Writing Detections From the Incident
Every technique the attacker used that wasn't detected before — or was detected too slowly — should become a new detection rule. This is the highest-value output of any incident.
# Example new detections written from a real incident
# DETECTION 1: Cobalt Strike default named pipe pattern
# (from incident forensics — attacker used default CS pipe names)
index=windows_sysmon EventCode=17 OR EventCode=18
| where PipeName IN ("\postex_*", "\msagent_*", "\mojo.*", "\wkssvc*")
| stats count by host, PipeName, Image, ProcessId
| where count > 0
| eval alert="Cobalt Strike named pipe pattern"
# DETECTION 2: LSASS memory access from non-standard process
# (attacker used Mimikatz to dump LSASS — this would have caught it)
index=windows_sysmon EventCode=10
| where TargetImage LIKE "%lsass.exe"
AND NOT (SourceImage LIKE "%\\Windows\\System32\\%"
OR SourceImage LIKE "%\\Windows\\SysWOW64\\%"
OR SourceImage LIKE "%\\MsSense.exe"
OR SourceImage LIKE "%\\SecurityHealthHost.exe")
| stats count by host, SourceImage, TargetImage, GrantedAccess
| where count > 0
# DETECTION 3: WMI event subscription created
# (attacker used WMI persistence — this event was in the logs but not alerted)
index=windows_security EventCode=5861
| table _time, host, Message
| eval alert="WMI event subscription created"
# DETECTION 4: Scheduled task created with encoded command
index=windows_security EventCode=4698
| rex field=TaskContent "(?i)(?:powershell|cmd).+?(?P<encoded>(?:frombase64|encodedcommand)[^\"]*)"
| where isnotnull(encoded)
| table _time, host, TaskName, encodedDetection Gap Analysis
The gap analysis documents which phases of the attack were not detected, and why. This is the architecture document for the detection improvement roadmap.
| Attack phase | What attacker did | Why not detected | Uplift action |
|---|---|---|---|
| Initial access | Exploited ProxyShell (Exchange CVE-2021-34473) | No IDS signature for ProxyShell; Exchange not in SIEM scope | Add Exchange IIS logs to SIEM; deploy ProxyShell YARA rule to network IDS |
| Execution | PowerShell download cradle via encoded command | Script Block Logging not enabled; only command-line events collected | Enable PowerShell Script Block Logging (Event 4104) on all systems |
| Persistence | WMI subscription created | Event 5861 not alerted — in SIEM but no rule on it | Create detection rule for Event 5861 (already done above) |
| Lateral movement | Pass-the-hash from LSASS dump | Sysmon Event 10 (LSASS access) not deployed as detection | Deploy Sysmon Event 10 detection (already done above) |
| C2 | HTTPS Cobalt Strike beacon | No TLS inspection; proxy logs existed but no pattern match for CS JA3 | Deploy JA3/JA3S Cobalt Strike signature to proxy/NDR; enable TLS inspection for workstations |
| Exfiltration | Large ZIP archive created then uploaded | DLP not deployed; no alert on large outbound file transfer | DLP for sensitive data patterns; alert on outbound transfers >100MB over 24h from workstations |
Updating the Runbook
Every gap in the IR response — decisions that had to be made without documented guidance, steps the team wasn't sure about, authorization that took too long — should result in a runbook update.
| Gap identified | Runbook addition |
|---|---|
| Network team required change window; delayed VLAN isolation by 45 minutes | Add section: "Emergency change authorization for IR containment actions — IR lead or CISO can authorize emergency changes; post-hoc change record filed within 24 hours" |
| IR team didn't know which external IR retainer to call at T+0 | Add section: "External IR retainer contact list — firm name, 24/7 hotline, account number, named contact." Put it on the first page. |
| No pre-approved SITREP template; IR lead spent 40 minutes writing the first status report | Add pre-filled SITREP template to the runbook. Attach as standalone document that can be filled in under 5 minutes. |
| krbtgt double-reset timing wasn't tracked; second reset happened before 10 hours elapsed | Add krbtgt reset procedure with explicit instruction: "Record first reset timestamp. Schedule second reset for exactly 12 hours later. Both must be documented." |
A blameless post-incident review means the focus is on systemic failures rather than individual errors — "the runbook didn't cover this scenario" rather than "X analyst made the wrong call." It doesn't mean ignoring deliberate negligence or repeated failure to follow existing procedures. If an IR action failed because someone didn't follow a documented procedure they were trained on, that's a personnel management issue separate from the PIR process. The PIR identifies process and tooling gaps. Management accountability handles deliberate or repeated individual failures. Both can be true at the same time.
External Reporting Obligations
Some incidents trigger regulatory notification requirements. These deadlines run from the moment of discovery, not from containment or recovery.
| Regulation | Trigger | Notification deadline | To whom |
|---|---|---|---|
| GDPR (EU) | Personal data breach affecting EU residents | 72 hours from discovery to supervisory authority; "without undue delay" to affected individuals if high risk | Lead supervisory authority (DPA); affected individuals if high risk |
| HIPAA (US) | Breach of unsecured protected health information | Without unreasonable delay, max 60 days from discovery; 10 business days if law enforcement delay | HHS OCR; affected individuals; media (if >500 affected in a state) |
| SEC Rule 10b-5 / 8-K (US public companies) | Material cybersecurity incident | 4 business days after determining materiality | SEC Form 8-K filing; investors |
| FTC Safeguards (US financial/education) | Security breach affecting 500+ customers | 30 days from discovery | FTC |
| State breach notification laws (US) | Varies by state — generally PII exposure | Varies: 30-90 days from discovery depending on state | State AG and/or affected residents |
Q & A
Q: The PIR reveals that the IR team made a decision during the incident that, in retrospect, was clearly wrong. How do you document this without it becoming a legal liability?
Document the decision, the information available at the time, and the process followed. "The IR team decided to delay isolation for 2 hours to extend scope observation. At the time, the assessment was that no active destruction was occurring. The decision was made by [title], not [name]. In retrospect, the attacker used this window to establish additional persistence." This framing documents the decision accurately without assigning blame. If outside counsel has been engaged, share the draft PIR with them before finalizing — they may advise on specific language that protects attorney-client privilege or limits litigation exposure while still being accurate. The goal is accuracy that a court would find credible, not a document that hides what happened.
Q: You identified 12 detection gaps. How do you prioritize which ones to address first?
Prioritize by: (1) which gaps would catch the same threat actor if they returned — initial access and C2 detection first, since these are the phases that most directly lead to the same incident happening again. (2) Which gaps are achievable quickly — enabling PowerShell Script Block Logging takes minutes and has immediate value. (3) Which gaps align with your most likely threat scenarios — if ransomware is your primary risk, prioritize lateral movement detection over supply chain detection. Build a detection roadmap with quarterly deliverables rather than trying to close all 12 gaps in the week after recovery. Each detection that ships is a permanent improvement; a detection backlog that overwhelms the team ships nothing.