Chapter 33

Building an IR Program

Most organizations have IR capability — some staff who can respond to incidents. Very few have an IR program — a structured, measured, continuously improving system that converts incidents into security improvements. The gap between the two is the gap between reactive defense and systematic risk reduction.

Scenario

You've been hired to build an IR program at a 1,200-person company. There is currently one "security analyst" (actually a generalist IT person with some security responsibilities), no playbooks, no case management, no retainer with an external IR firm, no incident classification policy, and no post-incident review process. They have Defender for Endpoint (EDR) and a Splunk instance with 90-day retention. The CISO has given you a year to build something real and a $500k budget. Where do you start, what do you do in what order, and what does "done" look like?

IR Program Maturity Model

Before deciding what to build, know where you are. Most IR maturity models have 5 levels. Understanding your current level determines the priority order of investments. Skipping levels — buying SOAR before you have stable detection — creates expensive tools that don't get used.

  IR Program Maturity Levels
  ══════════════════════════════════════════════════════════════════════

  LEVEL 1: Ad-hoc — "We respond when things break"
  ──────────────────────────────────────────────────
  Characteristics:
    • No documented IR process
    • No dedicated IR staff; response is whoever is available
    • Detection is primarily user reports and help desk tickets
    • No case management; incidents tracked in email threads or spreadsheets
    • No retainer with external IR firm
    • Post-incident review = a brief conversation, if any

  LEVEL 2: Defined — "We have a plan"
  ────────────────────────────────────
  Characteristics:
    • IR policy and procedures documented
    • IR team identified (may be part-time)
    • Basic detection: EDR + SIEM in place
    • Case management tool (even if basic)
    • External IR firm retainer signed
    • Post-incident reviews written for P1 incidents
    • Incident classification (P1/P2/P3/P4) defined

  LEVEL 3: Managed — "We measure what we do"
  ─────────────────────────────────────────────
  Characteristics:
    • IR playbooks for top-5 incident types (ransomware, BEC, etc.)
    • MTTD/MTTR measured and reported
    • SOAR automation for enrichment
    • Tabletop exercises quarterly
    • Detection coverage mapped to MITRE ATT&CK
    • 24/7 alert monitoring (MDR or internal SOC with on-call)

  LEVEL 4: Optimized — "We continuously improve"
  ─────────────────────────────────────────────────
  Characteristics:
    • Purple team exercises driving detection improvements
    • Threat intelligence feeding detection rules
    • SOAR automation beyond enrichment (soft containment)
    • Post-incident reviews drive measurable program changes
    • IR feeds back to risk management and architectural decisions
    • Detection gap rate tracked; coverage target defined and measured

  LEVEL 5: Adaptive — "Our program adapts to the threat landscape"
  ─────────────────────────────────────────────────────────────────
  Characteristics:
    • Threat-intel-driven detection engineering cycle
    • Automated containment for highest-confidence indicators
    • Continuous validation via BAS (Breach and Attack Simulation)
    • IR team actively contributes to industry threat intelligence sharing
    • Incident data drives security architecture decisions with board visibility

Technology Stack Priority Order

Technology decisions in IR programs are commonly made in the wrong order — buying expensive automation before basic visibility, or SIEM before EDR. The priority order below reflects dependencies: each layer depends on the layer below it being stable.

  IR Technology Investment Order
  ══════════════════════════════════════════════════════════════════════

  LAYER 1: VISIBILITY (must have before anything else)
  ──────────────────────────────────────────────────────
  Without this, everything else is guessing.
    • EDR on all endpoints (100% coverage — a 95% coverage EDR is
      a 5% blind spot attackers will use)
    • Log collection: Windows Event Logs, DNS, proxy/web, auth
      (AD/Okta), VPN, firewall, email
    • SIEM or log aggregation with minimum 90-day retention
      (12-month preferred for investigation reach)
  Budget: $50–$150k/yr depending on org size and EDR choice

  LAYER 2: DETECTION AND CASE MANAGEMENT
  ────────────────────────────────────────
  Without this, visibility is data without action.
    • SIEM detection rules for your top-10 threat scenarios
      (prioritized by threat intel for your industry)
    • Case management (TheHive if budget-constrained, or ServiceNow
      Security Operations / Jira Service Management for integration)
    • Incident classification policy and escalation procedures
    • External IR firm retainer
  Budget: $30–$80k/yr (retainer + tool costs)

  LAYER 3: PROCESS AND PEOPLE
  ────────────────────────────
  Without this, detection creates alerts nobody handles well.
    • Documented playbooks: ransomware, BEC, insider threat,
      credential compromise, web application attack (minimum 5)
    • IR team: at minimum one dedicated senior IR analyst;
      on-call rotation or MDR coverage for after-hours
    • Tabletop exercise program: quarterly minimum
    • Post-incident review process: written PIRs for all P1/P2
  Budget: $150–$300k/yr (primarily personnel)

  LAYER 4: AUTOMATION AND IMPROVEMENT
  ──────────────────────────────────────
  Build only after layers 1–3 are stable.
    • SOAR for enrichment automation
    • Purple team / threat simulation program
    • Threat intelligence feeds integrated into detection
    • Detection coverage mapping and gap analysis
  Budget: $50–$150k/yr additional

  ORDER MATTERS: organizations that buy SOAR before they have
  stable detection, or buy threat intel before they have a
  way to operationalize it, waste money on unused capability.

Team Structure

The right team structure depends on organization size, budget, and risk profile. The fundamental question is how many dedicated IR staff you need versus how much you rely on an MDR provider for coverage and an external IR firm for surge capacity.

Org sizeMinimum viable IR teamMDR vs internal SOCExternal retainer
Small (<500 employees)1 security analyst (IR + detection engineering + other responsibilities), CISO who can make IR decisionsMDR provider for 24/7 alert monitoring — cost-effective alternative to internal 24/7 SOCEssential — no internal surge capacity; $30–$60k/yr for a reputable firm's retainer
Mid-size (500–2000 employees)2–3 dedicated security analysts, 1 IR lead, part-time SIEM/detection engineeringMDR for nights and weekends, or 3-shift internal coverage (expensive); hybrid commonRequired for P1 incidents beyond team capacity; $50–$100k/yr
Large (2000–10000 employees)Dedicated SOC team (5–10 analysts), IR team (2–4 investigators), detection engineering (1–2), CISO oversightInternal SOC plus MDR for overflow or specialized capabilityStill valuable for large-scale P1 and regulatory support; may negotiate $150k+ with significant discount on hourly rates
Enterprise (10000+ employees)Full SOC with tier 1/2/3 structure, dedicated IR team, threat intel, detection engineering, threat huntingInternal — at this scale, outsourcing SOC loses institutional knowledgeSpecialized capability: forensics labs, malware analysis, attribution support
Mental model: the IR program flywheel

A mature IR program operates as a flywheel: each incident generates post-incident findings, findings drive detection improvements, detection improvements catch the next incident faster, faster detection enables faster response, faster response generates cleaner forensic data, cleaner data generates better findings. Organizations that skip the post-incident review step or don't act on the findings break the flywheel — they respond to incidents at the same speed forever, never improving. The discipline that matters most for building a good IR program is not the technology purchases or the playbook writing — it is the institutional commitment to converting every incident into a program improvement, even a small one. After 12 months of consistent post-incident reviews that drive changes, the program looks fundamentally different than after 12 months of incident response with no improvement cycle.

Testing the Program

An IR program that has never been tested hasn't been validated — it's an untested theory. Testing IR capability before a real incident reveals the gaps in plans, processes, and technology that you cannot find in any other way.

Test typeWhat it testsCadenceWho participates
Tabletop exerciseDecision-making and communication under pressure; playbook gaps; organizational roles and escalation pathsQuarterlyIR team, CISO, legal, communications, relevant business units. No technical execution — discussion only.
Functional exerciseEnd-to-end IR process including technical response; tests that tools work and people can use themSemi-annuallyFull IR team; may involve IT, HR, communications. Technical actions executed in test environment.
Red team / purple teamDetection coverage — whether specific attack techniques fire alerts; response effectiveness to live attacker TTPsAnnually minimum; quarterly for mature programsRed team (internal or external) + SOC/IR team coordination. Purple team format = both teams learn.
BAS (Breach and Attack Simulation)Continuous automated detection coverage validation against a library of attack techniquesContinuous (automated)Security engineering team monitors results; no real-time human involvement in execution
Communication testThat out-of-band communication channels (Signal group, phone tree, war room access) actually workQuarterlyIR team; CISO; executive stakeholders. Verify phone numbers are current, Signal group is populated.

Year One: What Done Looks Like

For the scenario in this chapter's opening — building an IR program from near-zero with a $500k budget — this is what a realistic year-one achievement looks like, in priority order.

  Year One IR Program Build Plan
  ══════════════════════════════════════════════════════════════════════

  MONTHS 1–3: Foundation
  ───────────────────────
  ✓ Inventory current state: what logs are collected? what coverage
    does the EDR have? what incidents happened in the last 12 months?
  ✓ Sign external IR firm retainer
  ✓ Establish incident classification policy (P1/P2/P3/P4 with definitions)
  ✓ Document escalation paths and communication plan (who to call for P1)
  ✓ Set up TheHive or equivalent case management
  ✓ Close EDR coverage gaps (get to 95%+)
  ✓ Ensure critical log sources are flowing to SIEM with appropriate retention
  Budget impact: $30k retainer + $10k tooling + analyst time

  MONTHS 3–6: Detection and Process
  ────────────────────────────────────
  ✓ Write 5 core playbooks: ransomware, BEC, phishing, insider threat,
    credential compromise
  ✓ Tune top-20 SIEM alert rules (reduce false positives; improve signal)
  ✓ Establish post-incident review template and process
  ✓ Run first tabletop exercise
  ✓ Establish out-of-band communication channel (Signal group with IR team
    and leadership)
  ✓ Map current detection coverage to MITRE ATT&CK (know your gaps)
  Budget impact: analyst time (people are the main cost here)

  MONTHS 6–9: Automation and Measurement
  ────────────────────────────────────────
  ✓ SOAR enrichment automation for top-3 alert types
  ✓ Establish baseline MTTD/MTTR measurement from case management data
  ✓ Run first functional exercise (test the playbooks end-to-end)
  ✓ Implement basic threat intelligence feeds (free tier sources minimum)
  ✓ MDR coverage for after-hours if no internal 24/7 capability
  Budget impact: $50k SOAR tooling or MDR; $30–50k threat intel

  MONTHS 9–12: Validation and Improvement
  ──────────────────────────────────────────
  ✓ First external red team / purple team exercise
  ✓ MTTD/MTTR first annual report: here is where we started,
    here is where we are, here is the trend
  ✓ Detection coverage report: what percentage of MITRE techniques
    covered; what are the priority gaps
  ✓ Year-two roadmap based on what the testing revealed
  Budget impact: $50–100k red team exercise

  END STATE at year one:
    • 95%+ endpoint coverage
    • Case management in place and being used consistently
    • 5 documented, tested playbooks
    • Baseline MTTD/MTTR established
    • External IR retainer active
    • Tabletop exercises running quarterly
    • SOAR enrichment working for primary alert types
    • First purple team results informing detection roadmap
    This is Level 2 solidly, beginning of Level 3 in some areas.

Q & A

Q: The CISO asks you to prioritize getting to "24/7 detection coverage" in year one. But you have no staff for a full 24/7 SOC. What are the realistic options?

Three realistic options for organizations without budget for an internal 24/7 SOC: (1) MDR (Managed Detection and Response): outsource overnight and weekend monitoring to an MDR provider. MDR providers monitor your environment from their own SOC 24/7, alert you to confirmed threats, and provide initial triage. Cost: $50–$150k/yr depending on environment size and service level. Tradeoff: the MDR SOC doesn't have deep knowledge of your environment; initial triage may be generic; escalation takes time. Good MDR providers tune their detection to your environment over time. (2) On-call rotation for internal team: your 2–3 analysts share a 24/7 on-call rotation. SIEM/EDR alerts are configured with intelligent suppression so only P1/P2 alerts page out of hours. Analyst is not actively monitoring — they are alerted to respond. Tradeoff: analyst fatigue with frequent after-hours pages; P3 events may not be handled until next business day. Requires clear alert threshold policy. (3) Hybrid: MDR handles after-hours monitoring and initial triage for P3/P4 events; P1/P2 escalation goes to internal analyst on-call for decision-making. This is the most common model for mid-size organizations. The MDR filters noise and the internal team handles decisions. Cost is lower than full MDR with faster response on critical events than MDR-only. Recommendation for year one: sign an MDR contract alongside the IR firm retainer. MDR covers 24/7 monitoring; the IR firm retainer covers surge for P1 events. This gives you year-one coverage while you build out the internal team in year two.

Q: The board asks: "What's our risk if we have a major incident tomorrow — before you've built out the program?" How do you answer honestly without alarming everyone?

Answer it in three parts: current capability, current gap, and timeline for gap closure: (1) Current capability: "Today, we have [EDR coverage, SIEM, one analyst]. We can detect and respond to commodity attacks — ransomware that our EDR signatures catch, phishing that our email security flags, account compromise if the user reports it. We have an external IR firm retainer signed, so if something major happens tomorrow, we can have experts on-site within 4 hours." (2) Current gap: "For a sophisticated, targeted attack — the kind that takes weeks to unfold — our detection capability today is limited. We would likely not detect the early stages. We'd catch it eventually, but probably at a later stage than is ideal, and our response would be slower and less coordinated than it will be in 12 months." (3) Timeline and residual risk: "By month 6, we'll have five tested playbooks and 24/7 alert monitoring coverage. By month 12, we'll have validated detection coverage and a practiced response team. Between now and month 6, our residual risk for sophisticated attacks is elevated. The mitigation is the external IR firm retainer — we can respond well even if we don't detect early." (4) What the board should decide: whether the residual risk level is acceptable given the cost of accelerating the program. If the board decides to accelerate, here are the additional investments that would help most: [two or three specific items]. The honest answer acknowledges the gap without overstating it and gives the board a decision to make rather than just a status report to absorb.

Q: 12 months in, you present the program results and the MTTD improved from "unknown" (no measurement) to a measured 6 days. The CISO says "6 days is terrible, industry average is much better." How do you respond?

Context and trajectory matter more than the absolute number: (1) Measurement baseline: 12 months ago, there was no measurement at all. The 6-day figure represents incidents that were detected — it doesn't capture any incidents that weren't detected at all (for which MTTD would be infinite). Moving from no measurement to measured 6 days is program progress, not a failure. (2) Industry context: the 6-day figure likely compares favorably to the actual industry reality once measurement methodology is matched. Many "4-hour MTTD" industry claims use alert-fire-to-acknowledgment time, not forensically-determined initial access to detection. If your 6 days is measured from forensically-determined initial access, it may represent a more honest number than many published metrics. (3) What's in the 6 days: decompose the number. If you have 6-day MTTD because the initial access was undetected for 5 days and the escalation from alert to response took 20 hours, those are two different problems. The detection gap (5 days) requires better detection coverage. The escalation gap (20 hours) requires process improvement. Year two investments should address the larger of the two. (4) Trajectory: is the 6 days improving? If the first quarter was 9 days and the most recent quarter was 4 days, the trend is what matters — the program is working. Stable bad numbers mean the improvement investments haven't hit yet. Decreasing numbers mean the program is improving. (5) Year two plan: present the specific investments (purple team findings, new detection rules, SOAR enrichment expansion) that will target the largest contributors to the 6-day figure. The number will improve; here's why and by how much.

Q: You're hiring your first dedicated IR analyst. What do you look for beyond technical skills?

Technical skill is table stakes but rarely the differentiator: (1) Communication under pressure: IR requires clear communication to non-technical stakeholders during the highest-stress moments. An analyst who can write a coherent SITREP at 3am during an active breach and explain it to a VP in plain language is worth more than a technically superior analyst who can't. Assess this in the interview with a scenario: "Explain this phishing attack to me as if I'm the CEO. Now explain it to me as if I'm a board member." (2) Documentation discipline: IR investigators who write down what they find and why they made decisions while they're making them (not after, from memory) are much more effective. Ask for a writing sample or example from a past investigation. (3) Comfort with ambiguity: real incidents don't have clear answers. The analyst who can take action on incomplete information, document the uncertainty, and adjust as new information emerges is more effective than one who needs certainty before acting. Assess with a scenario where key information is missing. (4) Learning mindset: the threat landscape changes faster than any training program. An analyst who actively follows threat intelligence (reads CTI reports, knows current threat actors) and translates it into their own work is worth 3x one who only follows assigned training. Ask what they've read recently and whether they can name three current threat actors active in your industry. (5) Collaboration: IR is never solo work. Ability to work in a high-pressure team environment, accept direction during an incident, and give clear updates to teammates are as important as technical depth.