Chapter 13

Recovery Planning

Recovery is not just putting systems back — it is putting them back differently than they were when the attacker found them. The pressure to restore operations quickly is real and legitimate. The tendency to skip security steps under that pressure is how organizations get hit again within weeks.

Scenario

A ransomware event has encrypted 200 workstations and 14 servers. The company's payroll system runs on one of those servers. Payroll processes in 31 hours. The CEO is asking for a restoration timeline every 30 minutes. The IR lead is being told in no uncertain terms that the payroll server must be online in 18 hours or the company will miss payroll for 1,400 employees — a legal obligation in most jurisdictions.

The IR team has a backup from 72 hours ago. It has never been tested. The initial access vector was an unpatched vulnerability in the VPN appliance — the same vulnerability that still exists on the appliance that is still connected to the network. The backup was made using a service account that the attacker compromised.

This is a recovery planning scenario. The urgency is real. The risks are also real. Recovery planning is about navigating this tension with the best possible outcome — not the fastest possible outcome.

Recovery Sequencing: What Comes Back First

The instinct in recovery is to bring back the most critical systems first. The security reality is the opposite: the most critical systems require the most thorough validation and should come back last in the sequence. Restoring a compromised system to serve critical functions without full validation creates a critical dependency on a potentially unclean host.

  Recovery Sequence Logic
  ═══════════════════════════════════════════════════════════════════

  STEP 1: Foundation Layer (must come back first)
  ─────────────────────────────────────────────────────────────────
  → Domain controllers (clean validated)
  → DNS/DHCP infrastructure
  → Identity management (AD, RADIUS)
  → Core networking infrastructure
  Reason: nothing else can be securely restored without clean
  identity infrastructure. Restoring application servers against
  a compromised DC means the DC can re-compromise the apps.

  STEP 2: Security Infrastructure
  ─────────────────────────────────────────────────────────────────
  → EDR management server (so you can monitor what comes back)
  → Log collection infrastructure (SIEM, syslog)
  → Patch management (WSUS, SCCM, Intune)
  → Vulnerability scanner
  Reason: you need visibility before restoring anything critical.
  Restoring business systems without monitoring in place means
  you won't detect if the attacker returns during recovery.

  STEP 3: Core Business Infrastructure
  ─────────────────────────────────────────────────────────────────
  → Email (after identity layer is clean)
  → File servers (after verifying backup integrity)
  → Core application servers (one at a time, with monitoring)

  STEP 4: Crown Jewels Last
  ─────────────────────────────────────────────────────────────────
  → Financial systems
  → ERP / core business applications
  → Customer data systems
  → IP repositories (source code, design files, R&D)
  Reason: these are the highest-value targets. They receive
  the most thorough validation and the last restoration slot
  ensures the rest of the environment is clean when they reconnect.

  Business pressure reverses this order. Your job is to explain
  why the order is what it is and get explicit authorization
  if the business chooses to override it.
Why the payroll server cannot go first

The payroll server being mission-critical doesn't make it safe to restore first — it makes the cost of a second compromise higher. If you restore payroll against a compromised DC and the attacker re-establishes access through that DC and re-deploys ransomware, the cost is the original incident cost plus a second incident that now hits a running payroll system. The IR lead's job is to make this argument to the CEO explicitly, get a decision, and document it. If the business chooses to accept the risk and restore payroll early anyway, that decision should be in writing from someone with the authority to accept operational risk.

Clean Backup Verification

The most dangerous assumption in recovery planning is "we have backups." Having backups and having usable, clean backups are different things. Verification must happen before the restoration decision, not during it.

Verification stepWhat it catchesHow to perform it
Restore test to isolated environment Backup media corruption, backup software failure, configuration issues that prevent actual restore Restore the backup to a clean, isolated VM with no network connectivity and verify the system boots and functions correctly
Backup timestamp verification Backup predates initial compromise, or backup postdates it and may contain malware Compare backup timestamp against confirmed first-attacker-activity timestamp from the investigation
Backup integrity scan Malware present in the backup image (attacker was in the environment when backup ran) Mount the backup read-only in an isolated environment and run AV/EDR scan before restoring to production
Backup service account audit Backup service account was compromised — backup system itself may be compromised Review all access to backup infrastructure during incident window; verify backup service account credentials and reset if any doubt
Backup media verification Offline/tape backups may have been physically tampered with (insider threat scenario) Chain of custody records for physical media; verify hash of media image matches creation record
The untested backup disaster

Organizations discover their backups don't work during recovery — not before. Common failures: the backup job appeared to complete but the data is corrupt; the backup includes the OS but not the application data; the backup requires the original server hardware to restore (bare-metal restore to different hardware fails); the backup software license has expired and the restore agent won't run; the backup requires a decryption key that nobody can find. Any of these, discovered when you're 18 hours from a payroll deadline, extends the incident significantly. The cost of a monthly restore test in a test environment is trivial compared to discovering backup failure during a real incident.

Patch Before Reconnect

Every system that returns to the network during recovery must be patched against the vulnerability that was exploited in the incident — at minimum. Restoring a system with the same vulnerability that enabled the initial compromise is rebuilding the exact attack surface the attacker used.

The attacker knows how they got in. If you don't close the door before reconnecting, they will use the same door again — often immediately, since they know the environment is in recovery and less likely to have full detection coverage active.

  Patch Requirements for Recovery
  ═══════════════════════════════════════════════════════════════════

  Minimum (must have before any system reconnects):
  ─────────────────────────────────────────────────────────────────
  → Patch the specific CVE that was the initial access vector
  → Confirm patch on every restored system before reconnection
  → Verify patching on any system that could provide the same
    initial access path (same vulnerability class, same software)

  Better (should have before critical systems reconnect):
  ─────────────────────────────────────────────────────────────────
  → All critical and high severity patches applied (not just
    the specific CVE — the attacker may have noted other vulns
    while in the environment)
  → EDR reinstalled and verified reporting to management console
    before system goes back online
  → Local firewall rules reviewed and tightened

  Best (should happen during the recovery window):
  ─────────────────────────────────────────────────────────────────
  → Close every gap the attacker exploited for lateral movement,
    not just initial access
  → Apply security baseline hardening to every restored system
  → Implement new detective controls that would have caught this
    incident earlier

  The recovery window is the one time you have legitimate reason
  to make sweeping security changes with business buy-in.
  Use it.

Hardening Changes Applied During Recovery

The recovery window is uniquely valuable for security improvement. The business is already accepting downtime and disruption. Security changes that would normally require months of change management can be applied during recovery with the same authorization. The attacker just proved these gaps exist — the argument for closing them has never been stronger.

Gap the attacker exploitedHardening applied during recovery
LLMNR/NBT-NS poisoning (Responder) for credential capture Disable LLMNR and NBT-NS via Group Policy on all restored systems
Unsigned SMB allowed NTLM relay Enable SMB signing required on all restored servers and workstations
Same local admin password on all workstations Deploy LAPS to all restored workstations before they reconnect
PowerShell without Script Block Logging Enable Script Block Logging (Event 4104) via GPO during restoration
Sysmon not deployed; EDR had gaps in coverage Deploy Sysmon and verify EDR agent on every restored host before reconnection
No network segmentation between workstations and servers Implement VLAN separation at recovery; workstations cannot reach servers directly
Wdigest enabled; plaintext credentials in LSASS Disable Wdigest via registry on all restored systems

Communicating the Recovery Timeline

One of the hardest skills in IR is communicating recovery timelines to business leadership under pressure. Three principles:

  Recovery Timeline Communication Principles
  ═══════════════════════════════════════════════════════════════════

  Principle 1: Give ranges, not point estimates
  ─────────────────────────────────────────────────────────────────
  Wrong: "The payroll server will be online at 6 PM."
  Right: "The payroll server will be online between 4 PM and 8 PM,
         depending on how long backup verification takes and whether
         we find issues during the restore test."

  The business can plan around a range. A missed point estimate
  destroys trust and adds pressure that leads to corners being cut.

  Principle 2: Explain what determines the timeline
  ─────────────────────────────────────────────────────────────────
  The business doesn't understand why recovery takes longer than
  expected unless you tell them. "We're waiting for backup
  verification to complete, and we found that the backup may
  contain malware so we're scanning it in isolation" is a
  legitimate explanation that most business leaders will accept.
  "It's taking longer than expected" is not.

  Principle 3: Communicate decision points, not just status
  ─────────────────────────────────────────────────────────────────
  When the business can affect the timeline with a decision
  (accept risk to reconnect earlier, authorize additional
  resources, defer non-critical recovery), tell them. Don't
  make decisions for them. "We can restore the payroll server
  in 6 hours if you authorize us to skip the backup integrity
  scan and accept the risk that it may contain malware. The
  alternative is the 18-hour timeline I quoted earlier. Which
  do you want?" is the right framing.
Mental model: recovery as a security project, not a restoration project

The framing of recovery as "getting back to normal" is wrong. You should never get back to exactly the state you were in before the incident — that state was compromised. Recovery is the process of building a better-secured version of your environment on top of the clean validated foundation established by eradication. Every system that comes back online should be more secure than it was when the attacker found it. Recovery is the only incident phase that creates lasting security improvement — all the others are about stopping the bleeding.

Q & A

Q: The backup we have is from before the incident but is 45 days old. The business says the data loss is unacceptable. What are the options?

Four options exist. First, restore from the 45-day backup and accept the data loss — appropriate if the data from the past 45 days can be reconstructed from other sources (paper records, partner systems, email archives). Second, restore the 45-day backup for the system and attempt to recover recent data from other sources (transaction logs, journal files, incremental data from other systems). Third, pay the ransom and use the attacker's decryptor to recover the encrypted recent data — expensive, unreliable, potentially illegal depending on the group. Fourth, accept that the business must operate without some data and determine which data is truly irreplaceable vs inconvenient to lose. The IR team presents the options with associated timelines, costs, and risks. The business makes the decision. The IR team does not make business continuity decisions unilaterally.

Q: We want to restore the VPN appliance but it's the initial access vector. Can we just patch it and bring it back?

Patching and re-enabling the same appliance carries significant risk — the appliance may have been compromised at a level that patching doesn't address (persistent configuration backdoors, modified firmware). The safer approach: restore the VPN appliance from a known-good configuration backup (not a backup taken after initial compromise), apply the patch, verify the configuration against a known-good baseline, and confirm the patch is applied and active before re-enabling. If no known-good configuration backup exists, consider a clean factory reset of the appliance followed by fresh configuration. Additionally, enable enhanced logging on the VPN appliance during monitored recovery so any attempt to reuse the same CVE or configuration would be visible. If the appliance is a commodity product (ASA, Pulse, Fortinet), threat intelligence may have indicators of appliance-level persistence specific to that CVE.

Q: The CISO is saying we need to communicate a clear recovery date to the board. How do you handle this?

Give the CISO what they need to communicate to the board while being honest about what you know and don't know. Draft the communication together: "Based on current progress, we expect [priority systems] to be restored by [date range] and full recovery complete by [date range]. This assumes [list key assumptions: backup integrity check completes successfully, no additional compromised hosts are found, patch deployment proceeds as planned]. If these assumptions do not hold, we will update the timeline immediately." The board communication should include the assumptions explicitly — this protects both the IR team and the CISO if the timeline slips, and gives the board an accurate picture of the uncertainty involved. Never let a CISO communicate a timeline to the board without the assumptions attached. The assumptions are what turn a promise into a forecast.

Q: How do you prioritize recovery when you have 200 systems down and limited staff?

Triage by three criteria in order: (1) business impact of continued downtime — use actual business input, not IT assumptions, about which systems are most critical to revenue, legal obligations, or safety; (2) dependency structure — systems that other systems depend on come before the dependent systems (you can't restore an application server before you restore the database it connects to); (3) validation complexity — simpler systems that can be validated and restored quickly may be worth doing early to restore partial capacity and free staff for more complex systems. Build a recovery schedule before starting, assign ownership for each system, and track actual completion against the schedule. Ad hoc recovery without a schedule results in the most vocal stakeholders getting their systems first, not the most critical ones.

Q: Should we restore from backup or rebuild from scratch? Which is faster and which is safer?

Depends on the system. For workstations with a current golden image: rebuild from golden image is almost always faster than restoring from backup and produces a cleaner, known-good system. For servers with complex configurations: restore from backup (if clean) is faster because you don't have to reconstruct the configuration. For systems where the backup may be compromised: rebuild from scratch with configuration documented from memory or a clean configuration backup. The safety ranking is: rebuild from scratch with current golden image > restore from known-clean backup > restore from backup of uncertain cleanliness. The speed ranking is almost the opposite for complex servers. The practical answer: rebuild workstations, evaluate servers case-by-case based on backup cleanliness confidence and configuration complexity.

Q: What if the business wants to run in parallel — some users on clean systems, some still waiting? Is there a security risk to mixed operation?

Yes, significant risk. When clean and potentially-compromised systems coexist on the same network, the attacker can potentially pivot from a still-compromised host to re-compromise a restored one. The common mitigation is network segmentation: clean systems go on a clean VLAN with firewall rules preventing any traffic from the unrestored segment. If that level of segmentation isn't feasible, weigh the business benefit of partial recovery against the risk of re-compromise before full restoration. The threshold question is: have you done eradication on the remaining-to-be-restored systems? If not, they should be treated as potentially compromised and isolated from clean systems.