Chapter 32

IR Metrics and Program Measurement

Metrics are the only way to know if your IR program is improving. MTTD and MTTR are the headline numbers, but they hide as much as they reveal — understanding what those numbers actually measure, how they can be gamed, and what the supporting metrics are changes how you run the program.

Scenario

The CISO presents at the board meeting: "Our mean time to detect is 4 hours and our mean time to respond is 18 hours — both improved significantly from last year." A board member asks: "How does that compare to industry?" The CISO answers: "It's very good." The board member asks: "How do you know detection at 4 hours is good? What incidents are you missing entirely that you have no detection time for?" The CISO pauses. The most important question about MTTD is not how fast detection happens for incidents that were detected — it's what percentage of attacker activity was never detected at all. MTTD only measures what you know about. This chapter explores what IR metrics actually measure, what they miss, and how to build a measurement system that drives program improvement.

MTTD and MTTR: What They Actually Measure

Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR) are the two primary IR metrics reported to leadership. Both measure something real, but both have significant blind spots that matter for a security leader interpreting the numbers.

  MTTD: Mean Time to Detect
  ══════════════════════════════════════════════════════════════════════

  Definition:
    Mean time from when attacker activity BEGAN to when the SOC DETECTED it
    For each incident: Detection Time = (Detection timestamp) - (Initial access timestamp)
    MTTD = Average(Detection Time) across all incidents in the period

  What it measures:
    For incidents that were detected: how fast detection happened

  What it DOES NOT measure:
    ─────────────────────────
    Incidents where the attacker was NEVER detected
    (These have MTTD = ∞. They are not in the denominator.)

    If your environment has 10 detected incidents (MTTD: 4 hours)
    and 3 undetected intrusions (MTTD: unmeasurable), you don't have
    a 4-hour MTTD. You have a 4-hour MTTD for detected incidents,
    which is a different statement.

  The dark matter problem:
    ─────────────────────────
    Industry data suggests organizations are breached an average of
    200+ days before detection. This average includes organizations
    where detection happened in hours AND organizations where the
    attacker was present for years (or was never detected).
    Your MTTD doesn't tell you where you are on this distribution.

  Measurement methodology:
    ─────────────────────────
    Initial access timestamp: the earliest evidence of attacker
    activity found during forensic investigation — NOT the time
    of the detection alert. These are often weeks or months apart.
    MTTD must be recalculated after investigation completes
    with the forensically-determined initial access time.
  MTTR: Mean Time to Respond/Remediate
  ══════════════════════════════════════════════════════════════════════

  Two different definitions (choose one and stick with it):

  1. Mean Time to Contain:
     Time from detection to full containment of the threat
     (attacker no longer has network access, no active malware running)
     Measures speed of stopping active damage

  2. Mean Time to Remediate:
     Time from detection to full remediation
     (systems restored, root cause addressed, new controls deployed)
     Measures time to full recovery

  These are not the same number. Containment might happen in 4 hours;
  full remediation (reimaging, patching, tuning detections) may take weeks.
  Be explicit about which you're measuring when reporting to leadership.

  What MTTR measures:
    Efficiency of the IR process for incidents that were detected
    and that were fully remediated within the measurement period

  What MTTR does NOT measure:
    Quality of remediation: an incident with MTTR of 4 hours that
    was incomplete (root cause not addressed, persistence not found)
    followed by a recurrence is not a 4-hour MTTR success story.
    It's a failure that happened to close a ticket quickly.

  Measurement methodology:
    Contain/remediate timestamp: the time when the final remediation
    action was verified (not when the ticket was closed — tickets are
    closed administratively, not forensically).

Supporting Metrics That Drive Improvement

MTTD and MTTR are outcome metrics — they tell you what happened. To improve them, you need process metrics — measurements of where in the IR process things are slow or failing. Without process metrics, improving MTTD/MTTR requires guessing at root causes.

MetricWhat it measuresHow to measureWhat it tells you
Detection coverage ratePercentage of attack techniques that would be detected by your current detection stackThreat simulation (purple team) or MITRE ATT&CK coverage mapping — run the technique, confirm whether detection firesWhere your blind spots are before attackers find them
Alert-to-case conversion ratePercentage of alerts that result in a real case (true positive)Count true positive alerts / total alerts per period from case managementQuality of your detection rules — high false positive rate means analyst time wasted on noise
Time to escalate (P1 alert)Time from P1 alert firing to IR lead being engagedTimestamp in case management: alert creation vs first analyst action vs IR lead assignmentWhether your on-call and escalation process is working; measures the hand-off gap
Time to enrichTime from alert creation to full enrichment context being available to analystCase management timestamps — alert created vs enrichment data available (SOAR automation should reduce this to minutes)Efficiency of enrichment automation; measures whether SOAR is delivering value
Recurrence ratePercentage of incidents where the same attacker, technique, or vector recurred after remediationThreat intelligence correlation across cases — same IOC, same TTPs appearing in new case after previous case closedQuality of remediation — recurrence means root cause wasn't addressed
Detection gap ratePercentage of incidents where forensics determined the attack had been active significantly longer than detection indicated (e.g., 2-week dwell before detection)For each closed incident: days between initial access and detection — report the distribution, not just the meanWhether your detections are catching early-stage attacks or only late-stage indicators

Perverse Incentives: How IR Metrics Get Gamed

Metrics drive behavior — including behavior that optimizes for the metric without improving the underlying outcome. Understanding how IR metrics are commonly gamed helps you design metrics that are harder to game and interpret reported numbers more critically.

  Common IR Metric Gaming Patterns
  ══════════════════════════════════════════════════════════════════════

  Gaming MTTD:
  ─────────────────────────────────────────────────────────────────────
  Pattern: Setting the "initial access" timestamp to the SIEM
           alert time rather than forensically-determined initial
           access time. This makes MTTD look much better.

  Example: Attacker entered on Day 1. Alert fired on Day 14.
           Forensically-determined MTTD: 14 days.
           Gaming: record "incident start" as the Day 14 alert.
           Reported MTTD: 0 days (or however long from alert to response).

  How to prevent: require forensic determination of initial access
  time for all P1 incidents, with mandatory post-incident review.
  MTTD should be recalculated after investigation — not locked
  at the time of the initial alert.

  Gaming MTTR:
  ─────────────────────────────────────────────────────────────────────
  Pattern: Closing cases quickly (good MTTR) without completing
           full remediation. Ticket closed ≠ remediation complete.

  Example: Ransomware incident. Systems restored from backup.
           Ticket closed (MTTR: 5 days). But initial access vector
           (vulnerable VPN) not patched. Attacker re-enters in 3 weeks.

  How to prevent: require a remediation verification checklist before
  case close. Define "remediated" as: (1) attacker access revoked,
  (2) persistence removed, (3) initial access vector addressed,
  (4) detection rule for the technique added or confirmed exists.
  Cases should not be closeable until this checklist is signed off.

  Gaming alert-to-case rate:
  ─────────────────────────────────────────────────────────────────────
  Pattern: Creating cases for obvious false positives to "prove" high
           conversion rate, or ignoring genuine alerts to avoid
           creating cases for complex investigations.

  Pattern 2: Suppressing alert types that produce many false positives
             rather than tuning the detections — this improves the rate
             but reduces coverage.

  How to prevent: track suppressed alert types and require justification
  for each suppression. Audit sample of "ignored" alerts monthly.
Why MTTD alone doesn't measure detection program quality

The most important IR metric that most organizations don't measure is detection coverage: what percentage of real-world attacker techniques would fire a detection in your environment? MTTD tells you how fast you detect things you detect. Coverage tells you how much you're missing entirely. An organization with MTTD of 2 hours but 40% MITRE ATT&CK coverage has a significant blind spot — 60% of common attacker techniques would go undetected regardless of how fast the response team moves. Purple team exercises — where a red team performs specific MITRE-mapped techniques while the detection team monitors for alerts — are the only way to measure coverage with confidence. Organizations that invest in quarterly purple team exercises consistently have better security outcomes than those that optimize only for MTTD/MTTR speed on the incidents they already detect.

Metrics Reporting Cadence and Audience

Different metrics belong at different reporting levels. Presenting process metrics to the board or MTTD/MTTR to line analysts confuses both audiences and wastes reporting effort.

Reporting levelCadenceMetrics to include
Board / CEOQuarterlyMTTD/MTTR trend (are we improving?), incident volume by severity, cost of incidents, detection coverage percentage, comparison to industry (Verizon DBIR, Ponemon data)
CISO / VP SecurityMonthlyAll board metrics + alert-to-case conversion rate, recurrence rate, detection gap distribution, coverage by MITRE technique, SOAR automation success rate, analyst capacity utilization
IR team leadWeeklyAll monthly metrics + time-to-escalate per incident, time-to-enrich, playbook execution success rate, specific alerts/techniques showing up repeatedly (signals a tuning opportunity), analyst open caseload
SOC analystReal-time / daily shift reportPersonal alert handling time, current open cases, playbook next steps, escalation queue status

Q & A

Q: Your CISO wants to benchmark your IR metrics against industry. Where do you get credible comparison data?

Several reliable sources with different perspectives: (1) Verizon Data Breach Investigations Report (DBIR) — published annually, covers thousands of real incidents. The most widely cited industry data source. Reports detection timelines, breach patterns, and attack vectors by industry vertical. The DBIR's finding that most breaches have days-to-months MTTD is the baseline reality check. (2) IBM Security / Ponemon Cost of a Data Breach Report — annual survey covering response costs, MTTD/MTTR, and breach impact by industry. Useful for cost benchmarking. (3) Mandiant M-Trends — annual report from Mandiant's incident response engagements, covering global MTTD, dwell time, attack vectors. Gives you a picture of what IR teams see in the most serious incidents. (4) SANS IR Survey — annual survey of security practitioners self-reporting their program metrics. More practitioner-level than vendor-driven. (5) ISACs (Information Sharing and Analysis Centers) for your industry sector — if you're in finance, healthcare, energy, etc., your sector ISAC may publish sector-specific benchmarks. (6) Important caveat on benchmarking: industry averages include organizations at all maturity levels and of all sizes. Comparing against a number that includes organizations with no security program is a low bar. A better benchmark is the upper quartile for your industry and size, not the mean.

Q: Your MTTD has been 8 hours for 6 months and isn't improving despite tuning work. What's your diagnostic approach?

If the headline metric isn't moving, the diagnostic approach is to decompose it into its constituent time periods and find where the time is going: (1) Break MTTD into stages: initial access to first malicious event logged → first event logged to alert fired → alert fired to analyst acknowledged → analyst acknowledged to MTTD. If you don't have timestamps for each stage, add instrumentation to your case management workflow to capture them. (2) The "event logged to alert fired" gap: are events reaching your SIEM in near-real-time, or with multi-hour lag? Some log sources (endpoints in remote offices, certain SIEM collectors) have collection latency. A 4-hour collection lag means an 8-hour MTTD can never be better than 4 hours regardless of how good your detection rules are. (3) The detection coverage gap: if 60% of attacker techniques don't fire any detection, the 40% that do may already be fast. The actual MTTD problem is that you're detecting the late-stage techniques (impact phase) rather than early-stage (initial access, persistence). Run a purple team exercise and map your detections to MITRE ATT&CK stages. (4) The alert quality gap: if analysts are handling 500 alerts per day, they're likely triaging quickly rather than thoroughly. A fast triage that misses the real threat keeps MTTD high by delaying escalation on the incidents that matter. Review a sample of escalated P1 incidents: were there earlier alerts that should have triggered investigation sooner?

Q: An analyst asks why you measure MTTD/MTTR in calendar time when a lot of that time is nights and weekends when nobody is working the case. Should you use business hours instead?

This is a legitimate measurement question with a real tradeoff: (1) Attackers work in calendar time, not business hours. An attacker who enters your environment Friday at 5pm has all weekend to establish persistence, escalate privileges, and exfiltrate data before you return Monday at 9am. The business-hours-adjusted MTTD doesn't capture that reality — it makes your program look better than it is from a risk perspective. (2) Calendar MTTD is the right metric for risk posture. It tells you: "For any given breach that starts, how long does the attacker have uncontested access?" That is the risk-relevant question, and the answer doesn't pause for weekends. (3) Business hours MTTR is a valid supplementary metric for operational planning. If you want to measure analyst efficiency — how many hours of analyst effort does it take to contain and remediate once an incident is engaged — business hours is more useful for capacity planning and analyst performance. (4) Recommended approach: report MTTD in calendar time (this is the risk metric), report analyst response time in business hours (this is the operational metric), and separately track after-hours detection and escalation as a coverage metric. An organization that has good MTTD during business hours but 3x worse MTTD on nights and weekends has a staffing coverage gap that calendar MTTD reveals and business-hours MTTD hides.