Supply Chain Compromise Playbook
Supply chain attacks weaponize the trust relationships between an organization and its software vendors, IT providers, or managed service providers. The defender's problem is unique: the malicious code arrives through a legitimate, trusted update channel, signed and verified. The attacker's access looks indistinguishable from normal vendor operations until you know what to look for.
SolarWinds, December 2020. 18,000 organizations updated their SolarWinds Orion network monitoring software to version 2019.4 or 2020.2. The update was delivered through SolarWinds' official update channel, cryptographically signed with SolarWinds' certificate. It passed every technical verification check. Buried in the update was SUNBURST — a backdoor installed by Russia's SVR (Foreign Intelligence Service) that had been dormant in the SolarWinds build system for months.
The SUNBURST backdoor would lay dormant for two weeks after installation, then beacon to a C2 domain (avsvmcloud.com). Before beaconing, it would check if the host was a security researcher's sandbox or honeypot — if so, it would remain dormant. It communicated via DNS subdomains that encoded victim hostnames, creating a low-and-slow exfiltration channel that looked like normal DNS traffic.
For organizations that ran SolarWinds: they had no warning, no patch, and no way to know they were compromised until FireEye publicly disclosed the campaign on December 13, 2020. The response playbook starts from that disclosure moment — when you learn your organization is potentially affected by a disclosed supply chain compromise.
Two Categories of Supply Chain Compromise
Supply chain compromises divide into two major categories with different IR implications. Software supply chain compromises (like SUNBURST) involve malicious code delivered through a software vendor's update or distribution mechanism. Service supply chain compromises involve a managed service provider (MSP) or IT vendor's privileged access being used by an attacker to access the MSP's customers.
| Software supply chain | Service / MSP supply chain | |
|---|---|---|
| Attack mechanism | Malicious code embedded in vendor software update, package, library, or build pipeline | Attacker compromises MSP's management tools or credentials, then pivots to MSP's customers through the management access the MSP already has |
| Entry point | Software update from trusted vendor (SUNBURST, 3CX, Kaseya VSA, NotPetya/M.E.Doc) | MSP's RMM tool (ConnectWise, Kaseya, N-central), credential compromise of MSP's admin accounts |
| Scope problem | Any organization running the affected software version is potentially compromised — the population is the vendor's entire customer base | Any organization the MSP manages with privileged access is potentially compromised — MSP's client list |
| Initial indication | Typically: public disclosure by security researcher, vendor, or government agency. Rarely detected by affected organizations first. | Unusual admin activity from MSP accounts, MSP notifies clients, or a client's security team detects unusual RMM tool activity |
| Response priority | Scope assessment (are we affected?), isolation of affected systems, IOC-based hunt | Immediately revoke or restrict MSP access, then assess scope of MSP's actions in your environment |
SUNBURST Deep Dive: Mechanics of a Nation-State Supply Chain Attack
SUNBURST's technical design reflects a sophisticated actor optimizing for three goals: wide initial access, long dwell time, and selective exploitation of high-value targets. Understanding these mechanics helps build detection logic for similar future attacks.
SUNBURST Technical Execution Flow
═══════════════════════════════════════════════════════════════════
Stage 1: Compromise of Build Pipeline (months before distribution)
─────────────────────────────────────────────────────────────────
SVR compromised SolarWinds' build environment (SUNSPOT implant).
SUNSPOT monitored the build process for the Orion product.
When Orion's source code was compiled, SUNSPOT injected SUNBURST
source code into the build — the resulting binary was functionally
identical to the legitimate build, plus the backdoor.
The injected code passed code review because it was never committed
to the source repository — it was injected at compile time.
Stage 2: Distribution via Legitimate Update Channel
─────────────────────────────────────────────────────────────────
Affected versions: Orion 2019.4.5200.9083 and 2020.2.5.0
Distributed: via SolarWinds' update mechanism, signed with
SolarWinds' legitimate code signing certificate
Passed: all technical verification (signature check, hash check)
Downloaded by: ~18,000 organizations
Stage 3: Dormancy Period (14 days minimum)
─────────────────────────────────────────────────────────────────
SUNBURST does nothing for 2 weeks after installation.
Purpose: evade sandbox analysis (sandboxes time-limit analysis),
and create distance between installation and malicious activity
so incident timeline analysis is harder.
Stage 4: Anti-Analysis Checks
─────────────────────────────────────────────────────────────────
Before beaconing, SUNBURST checks:
→ Is this a security research machine? (specific process names, registry keys)
→ Is the SolarWinds Orion process running? (if not, don't beacon)
→ Is the domain part of a blocklist of known security company domains?
If any check indicates a non-target environment: remain dormant forever.
Stage 5: C2 Communication via DNS
─────────────────────────────────────────────────────────────────
C2 domain: avsvmcloud.com (attacker-controlled)
Protocol: DNS A record lookups (subdomain encodes victim data)
Subdomain format: [encoded-victim-data].[guid].appsync-api.
[region].avsvmcloud.com
The victim's hostname, domain, and identifier are encoded in the
subdomain. Attacker reads victim identity from their DNS server logs
without sending any data to the victim.
Low detection: DNS queries look like normal Orion telemetry traffic.
Stage 6: Selective Activation
─────────────────────────────────────────────────────────────────
SVR reviewed the 18,000 beaconing victims and selected ~40-50
for active exploitation (US government agencies, defense contractors,
tech companies, think tanks).
For selected victims: TEARDROP/RAINDROP second-stage malware deployed.
For others: never activated — remained dormant in their environments.
SUNBURST was not designed to exploit all 18,000 organizations that installed the backdoor. It was designed to provide access to a large pool so the real targets could be identified. For the 17,950+ organizations that installed SUNBURST but were not actively exploited, the "compromise" was technical but the real-world impact was limited. From an IR perspective, this creates a triage problem: every organization with SUNBURST installed had to conduct a full investigation to determine whether they were in the actively-exploited group. The cost of that investigation across 18,000 organizations vastly exceeded the attacker's operational cost of deploying SUNBURST to all of them.
Scope Assessment: Are We One of the Real Targets?
When a supply chain compromise is publicly disclosed, every potentially affected organization faces the same immediate question: are we just an incidental installation, or did the attacker actively use our environment? The answer determines the urgency and depth of the response.
Supply Chain Scope Assessment Process ═══════════════════════════════════════════════════════════════════ STEP 1: Am I affected? (Did we run the affected software version?) ───────────────────────────────────────────────────────────────── Check: inventory of installed software against affected version range. For SUNBURST: any SolarWinds Orion 2019.4 or 2020.2 installation. Result: YES / NO / UNKNOWN If NO: document your basis for saying NO. Do not assume. If UNKNOWN: treat as YES until proven otherwise. If YES: proceed to Step 2. STEP 2: Did the backdoor execute? (Technical presence check) ───────────────────────────────────────────────────────────────── For SUNBURST: check for the specific DLL (SolarWinds.Orion.Core. BusinessLayer.dll) with the malicious hash. Check EDR for process execution of the affected binary. Check: was the service running? Was SolarWinds Orion operational? (SUNBURST only beacons if Orion is running) STEP 3: Did the backdoor communicate? (Active compromise check) ───────────────────────────────────────────────────────────────── Check DNS/proxy logs for queries to avsvmcloud.com. Check firewall/proxy logs for DNS queries to *.avsvmcloud.com subdomains. Check: any connections to known SUNBURST C2 IPs (see published IOC lists). If YES to Step 3: you are in the actively-beaconing group. Proceed to full investigation. STEP 4: Were you actively exploited? (Second-stage check) ───────────────────────────────────────────────────────────────── Not all beaconing victims were actively exploited. Check for second-stage implants: TEARDROP, RAINDROP, Cobalt Strike. Check for unusual privileged access from SolarWinds service accounts. Check for SAML token forgery activity (Golden SAML attack). Check for unusual M365 access from SolarWinds service account. If YES to Step 4: you are in the active exploitation group. Full nation-state IR response; external IR firm engagement likely warranted.
Limiting and Revoking Vendor Access
The first tactical action in a supply chain response is restricting the vendor's access to your environment. This is the equivalent of the "isolate the compromised host" action in a normal endpoint incident — except the "host" is the vendor's management access, which may be deeply integrated into your infrastructure.
| Vendor access type | Containment action | Operational impact |
|---|---|---|
| Software agent (SolarWinds Orion monitoring) | Isolate systems running the agent from the network; disable the agent service; block C2 communications at firewall | Network monitoring capability offline until clean software version deployed |
| MSP remote management tool (RMM) | Immediately disable or revoke all MSP administrative accounts and API credentials; block RMM agent communication | MSP can no longer manage your systems remotely — evaluate if any critical operations need alternative coverage |
| Service account with AD privileges | Disable the account; reset the password; remove from privileged groups | Any automated processes using this account will fail — identify and address |
| VPN or direct network access | Terminate VPN sessions; revoke certificates; add firewall block rule for vendor's IP ranges | Vendor cannot connect remotely for support — accept and manage until investigation is complete |
SUNBURST-Derived Indicators and Detection Logic
The SUNBURST campaign produced a rich set of detection indicators and techniques that have become part of the standard supply chain compromise detection toolkit. These indicators are useful beyond SUNBURST itself — the detection logic applies to similar DNS-based C2 communication patterns.
SUNBURST-Derived Detection Rules ═══════════════════════════════════════════════════════════════════ 1. Software-specific file hash detection ───────────────────────────────────────────────────────────────── Alert on: presence of known-bad hash of the affected DLL. Published hashes allow immediate detection on any host. Detection type: IOC-based (low ongoing value after vendor patches) 2. DNS C2 detection for SUNBURST pattern ───────────────────────────────────────────────────────────────── Alert on: DNS queries to avsvmcloud.com or subdomains. More broadly: DNS queries to domains that CISA/NCSC flagged. Detection type: IOC-based for this specific campaign. 3. Unusual DNS query volume from monitoring/management processes ───────────────────────────────────────────────────────────────── Alert on: monitoring tools making DNS queries to domains outside their expected infrastructure (their SaaS vendor, update servers). A network monitoring tool querying random-looking subdomains of an external domain is anomalous. Detection type: behavioral (applies to future supply chain attacks) 4. Service account accessing cloud resources it hasn't before ───────────────────────────────────────────────────────────────── In SUNBURST: the SolarWinds service account was used to access Microsoft 365 resources and forge SAML tokens. Alert on: service accounts associated with monitoring tools accessing M365 management APIs, Exchange Online, or admin portals. Detection type: behavioral (high-value for detecting lateral movement from any supply chain compromise) 5. SAML token anomalies ───────────────────────────────────────────────────────────────── The "Golden SAML" attack forges SAML authentication tokens. Forged tokens lack some attributes present in legitimately-issued tokens. Alert on: SAML tokens with unusual issuer, missing standard claims, or from a signing certificate not in your known certificate list. Detection type: sophisticated; requires mature identity security tooling.
The Disclosure Decision
Supply chain compromises create a disclosure question unique to this incident type: not just "do we have regulatory notification obligations" but also "do we have an obligation to notify the vendor and other potentially affected parties." This is legally complex and involves considerations beyond standard breach notification.
Supply Chain Disclosure Considerations
═══════════════════════════════════════════════════════════════════
REGULATORY NOTIFICATION (standard breach notification rules apply):
─────────────────────────────────────────────────────────────────
→ Was customer PII accessed? → State breach notification obligations
→ Was PHI accessed? → HIPAA notification
→ Are you a public company? → SEC disclosure obligations (material breach)
→ Are you an FCEB (Federal Civilian Executive Branch) agency? → CISA
notification required
VENDOR NOTIFICATION:
─────────────────────────────────────────────────────────────────
In a responsible disclosure context: if you discover new technical
details about the compromise that the vendor is not aware of, sharing
them (through CERT/CC, CISA, or directly with the vendor's PSIRT)
helps protect other affected organizations.
The vendor may also be a required notification target under your
contracts with them (review incident notification clauses).
PEER ORGANIZATION NOTIFICATION:
─────────────────────────────────────────────────────────────────
Through ISACs and trusted information sharing organizations,
sharing IOCs from your investigation (C2 infrastructure you found
that hasn't been publicly disclosed, attacker TTPs in your environment)
helps other victims investigate more effectively.
This sharing is typically protected by legal safe harbor provisions
(CISA's AIS program in the US).
LAW ENFORCEMENT:
─────────────────────────────────────────────────────────────────
For nation-state supply chain attacks: report to CISA and FBI.
These agencies coordinate government response and may have
intelligence about the campaign that would help your investigation.
Government reporting may be required for certain critical
infrastructure sectors.
Q & A
Q: We installed the affected SolarWinds version but our DNS logs don't show any queries to avsvmcloud.com. Does that mean we weren't compromised?
It significantly reduces the probability of active exploitation but does not conclusively rule it out. Several explanations for absent DNS queries: (1) The dormancy check failed — SUNBURST may have determined your environment matched a security research sandbox (specific processes, domain names, or other checks) and remained dormant without ever beaconing. (2) DNS logging gaps — if your DNS logs don't have full coverage for all hosts during the affected period, queries may have occurred without being logged. (3) The affected software was not actively running during the dormancy period — SUNBURST only beacons when the Orion service is running. If the service was stopped or the host was offline for the 2-week dormancy period, no beacon. Document your basis for the "not affected" conclusion: software version confirmed present, DNS logs confirmed complete for the period, no evidence of active exploitation indicators. Accept residual uncertainty and apply enhanced monitoring for the indicators for 90 days.
Q: The investigation concludes we were in the beaconing group but not actively exploited. What's the remediation?
Three tracks: (1) Technical remediation — rebuild or restore all systems running the affected software version from known-clean backups predating the compromise window, or update to the patched version and verify no secondary compromise occurred. Rotate all credentials that the affected software had access to (database passwords, API keys, service account credentials). (2) Detection — implement the behavioral detection rules for software monitoring tools making unusual DNS queries and service accounts accessing unusual cloud resources. These detections apply to future supply chain campaigns, not just SUNBURST. (3) Vendor risk management — this incident provides the evidence for a formal vendor risk assessment of SolarWinds and any other vendors with similarly privileged monitoring access to your environment. The question to assess: what would an attacker have access to if any of our monitoring vendors were compromised the same way? Apply least-privilege principles to monitoring agent permissions.
Q: Our MSP was compromised and used their access to our systems. Where do we start the investigation?
Immediately revoke all MSP access — admin accounts, API credentials, VPN access, RMM tool agents. This is containment; do it before investigation. Then: (1) Identify what access the MSP had — every account, every system they managed, every tool they used. This is the potential scope of compromise. (2) Pull audit logs for all MSP-associated accounts for the past 90 days — look for activity that doesn't match the MSP's expected operational patterns (unusual hours, unusual systems accessed, unusual data accessed, privilege escalation, new account creation). (3) The MSP's RMM tool (ConnectWise, Kaseya, etc.) may have logs of what commands were executed via the tool on your systems — request these from the MSP or pull from your own endpoint agents. (4) Check for persistence mechanisms the attacker may have installed via the MSP's access: new accounts, modified scheduled tasks, new registry keys, installed software on managed systems. Treat this as a full compromise of every system the MSP managed with admin access.
Q: How do you build defenses against future supply chain attacks when you can't verify every software update?
You cannot verify every update, but you can reduce the blast radius of a successful supply chain compromise. Three principles: (1) Least privilege for vendor software — network monitoring agents don't need domain admin. Security tools don't need write access to your AD. Limit what an attacker can do if they compromise a vendor's software by limiting the permissions that software runs with. Map every vendor agent's permissions and reduce them to what's functionally required. (2) Network segmentation for monitoring infrastructure — SolarWinds should not have been able to make arbitrary DNS queries to external domains; a firewall rule allowing only Orion's expected update and telemetry destinations would have prevented SUNBURST's C2 communication. (3) Behavioral monitoring of software that has high system privilege — a network monitoring tool making DNS queries to random external subdomains is anomalous. Monitoring the behavior of trusted software and alerting on deviations from its expected behavior is a powerful detection layer for supply chain compromises.
Q: SUNBURST was first discovered by FireEye when it found the attack in its own environment. Should all organizations have a similar capability?
FireEye detected SUNBURST because they noticed a new device registering for MFA from an unusual location associated with an employee's account — a detection that required specific monitoring of device registration events in their identity platform. This detection is replicable by any organization that monitors identity events at the right level of fidelity. The broader capability FireEye had is best described as: sophisticated detection focused on their own high-value targets (they are a security company and are themselves a high-value target for nation-state actors) and detection engineering that looks for subtle anomalies in expected behavior rather than relying solely on known-bad indicators. Not every organization can replicate a tier-1 threat intelligence company's detection program. But every organization can: monitor identity events (device registration, token issuance anomalies), monitor privileged account activity, and restrict network egress for monitoring tools to expected destinations. The supply chain attack that most organizations will face is not a nation-state SUNBURST equivalent — it's a compromised MSP or a malicious npm package. Those are detectable with the controls described above.