TCP/IP Deep Dive for Forensics
Every field in the IP and TCP headers is a potential forensic indicator. Understanding the protocol mechanics — handshakes, teardowns, retransmissions, flags — lets you distinguish attacker-crafted packets from legitimate traffic, identify scanning and evasion techniques, and correctly interpret what Wireshark and Zeek tell you about a session.
You're reviewing a PCAP from a suspected C2 session. You see a sequence of TCP SYN packets to port 443 with no SYN-ACK responses, followed by a RST from the server, followed by a successful connection 2 seconds later. Then the connection stays open for 47 hours with 16-byte data transfers every 60 seconds. Understanding what each of those behaviors means at the protocol level — why there was a RST, what 47-hour duration implies, what 16-byte payloads at 60-second intervals tells you — is the foundation of correct analysis.
TCP Header Fields as Forensic Indicators
TCP Header (20 bytes minimum) ═══════════════════════════════════════════════════════════════════ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Destination Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Acknowledgment Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data | |U|A|P|R|S|F| | | Offset| Reserved |R|C|S|S|Y|I| Window | | | |G|K|H|T|N|N| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Checksum | Urgent Pointer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options (if Data Offset > 5) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Forensic significance of each field: ├── Source Port: ephemeral ports (49152-65535 = Windows OS range) │ well-known source port = craft packet or misconfig ├── Seq/Ack Num: ISN (Initial Sequence Number) — some OS stacks use │ predictable ISN generation; retransmits reuse seq num ├── Flags: SYN/ACK/RST/FIN combos tell connection lifecycle story ├── Window Size: OS fingerprinting — Windows: 65535, Linux: 29200, │ macOS: 65535; Zeek tracks this for passive OS detection └── Options: MSS, SACK, Timestamps, Window Scale — all OS-specific
Three-Way Handshake in Forensics
TCP Handshake — What Each Packet Tells You
═══════════════════════════════════════════════════════════════════
Client Server
│ │
│──── SYN (seq=x, win=65535, MSS=1460, TS opt) ─────►│ ← OS fingerprint here
│ │ (window size + options)
│◄─── SYN-ACK (seq=y, ack=x+1, win=29200) ──────────│ ← Server OS fingerprint
│ │
│──── ACK (seq=x+1, ack=y+1) ───────────────────────►│ ← Connection established
│ │
│◄════ DATA FLOWS ═══════════════════════════════════►│
Forensic anomalies in the handshake:
├── SYN with no SYN-ACK response → port is closed or filtered
├── SYN-ACK with no ACK → half-open scan (nmap -sS) or resource exhaustion
├── SYN to many ports in rapid succession → port scan
├── RST immediately after SYN-ACK → port scanner with -sT and close
├── SYN with unusual options → crafted packet, evasion, or nmap OS scan (-O)
└── Multiple SYN retransmits → target unreachable or SYN flood in progress
Connection states in flow records:
├── S0 (Zeek) → SYN only, no response = port closed/filtered
├── S1 → SYN + SYN-ACK, no ACK = half-open
├── SF → Normal open + FIN = legitimate completed session
├── REJ → RST in response to SYN = port closed
└── RSTO/RSTR → RST after connection established = abrupt close
TCP Flags and Session Teardown
| Flag combination | Meaning | Forensic indicator |
|---|---|---|
| SYN only | Connection initiation | Normal; SYN flood if from many sources |
| SYN+ACK | Server accepts connection | Port is open; missing = filtered or closed |
| RST | Immediate connection abort | From server: port closed; mid-session RST: firewall or IDS reset injection |
| RST+ACK | Reset with acknowledgment | Stateful firewall/IDS injecting resets to terminate connections |
| FIN+ACK → FIN+ACK → ACK | Graceful 4-way teardown | Normal session close; both sides cooperated |
| PSH+ACK | Push data, acknowledge | Normal data transfer; PSH tells receiver to deliver immediately |
| URG | Urgent pointer active | Rare in normal traffic; used by some C2 to bypass stateless inspection |
| NULL (no flags) | No flags set | Nmap NULL scan (-sN); triggers response from closed ports on some OS |
| FIN only (no ACK) | No flags except FIN | Nmap FIN scan (-sF); evasion — some stateless firewalls pass this |
| SYN+FIN | Illegal combination | Crafted packet; evasion or OS fingerprinting probe |
When you see a mid-session RST that appears to come from one endpoint but that endpoint's other packets look normal, you may be seeing RST injection — a technique used by stateful firewalls, IDS systems (Snort/Suricata in IPS mode), or occasionally by attackers to terminate sessions. RST-injecting devices typically send a RST with a sequence number in the current receive window, which is accepted by the TCP stack. If you see a connection abruptly terminate with a RST but the application-layer exchange was incomplete (e.g., HTTP request sent, no response, then RST), suspect that a security control killed the connection. The RST's TTL may differ from other packets from the "same" endpoint — a dead giveaway that it was injected from a third party on the network path.
Retransmissions and Out-of-Order Segments
TCP Retransmission Types — Forensic Significance
═══════════════════════════════════════════════════════════════════
1. TCP Retransmission (exact duplicate of lost segment)
Cause: ACK not received within RTO (Retransmit Timeout)
Forensic meaning: network loss, rate limiting, or IDS dropping packets
Wireshark label: [TCP Retransmission]
2. Fast Retransmit (3 duplicate ACKs triggers immediate retransmit)
Cause: receiver signals loss via duplicate ACKs before timeout
Forensic meaning: isolated packet drop (common in busy networks)
Wireshark label: [TCP Fast Retransmission]
3. Out-of-Order segment
Cause: packets arrive in wrong order (normal in load-balanced paths)
Forensic meaning: multiple network paths, or PCAP capture at wrong point
Wireshark label: [TCP Out-of-Order]
4. Duplicate ACK
Cause: receiver got unexpected sequence number, asking for missing segment
Forensic meaning: normal congestion signal; high dup-ACK rate = packet loss
Forensic alert: consistent retransmit pattern during a specific session
can indicate that a security device is selectively dropping packets from
that flow — e.g., a DLP appliance dropping SMTP attachments will create
retransmit loops for the dropped segment.
To check in tshark:
tshark -r capture.pcap -Y "tcp.analysis.retransmission" -T fields \
-e frame.time -e ip.src -e ip.dst -e tcp.dstport
TCP Options as OS Fingerprints
| TCP option | Code | Purpose | Forensic use |
|---|---|---|---|
| MSS (Maximum Segment Size) | 2 | Largest segment sender will accept | MSS=1460 = Ethernet LAN; MSS=1380 = VPN tunnel (lower MTU reveals tunneling) |
| SACK (Selective Acknowledgment) | 4/5 | Acknowledge non-contiguous ranges | SACK Permitted in SYN = modern OS; absence suggests old OS or custom stack |
| Timestamps | 8 | RTT measurement, PAWS (Protection Against Wrapped Seqs) | Timestamp value reveals uptime estimate; unusual increment rate = VM with suspended state |
| Window Scale | 3 | Scale window field by 2^n for high-bandwidth links | Different scales per OS; custom C2 implants sometimes omit this (non-standard stack) |
| NOP (No Operation) | 1 | Padding for option alignment | Option ordering (NOP+NOP+TS vs TS+NOP+NOP) is OS-specific; used in p0f and nmap -O |
A legitimate browser opening an HTTPS connection to a website sends a SYN with predictable TCP options: MSS=1460, SACK Permitted, Timestamps, Window Scale — in a specific order determined by the OS TCP stack. A custom-written C2 implant that builds raw sockets or uses a non-standard TLS library may send a SYN with different options, different ordering, or no options at all. Zeek's conn.log includes the TCP option set, and tools like p0f and Zeek's HASSH-equivalent passive OS fingerprinting detect these anomalies. A host claiming to be a Windows 10 workstation sending a SYN with a Linux-style TCP option ordering is a red flag worth investigating.
Half-Open Connections and Scanning Patterns
Scanning Signatures in pcap
═══════════════════════════════════════════════════════════════════
SYN scan (nmap -sS):
→ Send SYN to each port
→ SYN-ACK received? Port open → immediately send RST (no full handshake)
→ RST received? Port closed
→ No response? Filtered
Sign in PCAP: SYN → SYN-ACK → RST (RST comes from scanner, not target)
Full connect scan (nmap -sT):
→ Complete 3-way handshake, then RST or FIN
→ Leaves full handshake in PCAP; more detectable
UDP scan (nmap -sU):
→ Send UDP datagram to each port
→ ICMP Port Unreachable? Port closed
→ No response? Open or filtered
Sign in PCAP: burst of UDP datagrams to many ports, ICMP unreachable responses
Host sweep (nmap -sn):
→ ICMP Echo Request to each IP in range
→ Plus ARP requests on local network
Sign in PCAP: sequential ICMP echo from single source to adjacent IPs
Detection thresholds:
> 10 ports/second from one source = port scan
> 5 new hosts/minute from one source = host sweep
> 30% of connections in S0 state (SYN no response) = scanning
Q & A
Q: I see a RST packet mid-connection with a different TTL than surrounding packets from the same IP. What does that mean?
A RST with a different TTL than other packets from the same IP almost certainly indicates RST injection — a third-party device on the network path is inserting a forged RST to terminate the connection. This is typically a firewall, IDS/IPS, or DLP appliance. The injecting device needs to craft a RST with a sequence number in the receiver's current window, but it doesn't know the exact TTL the real endpoint would use. A few TTL values off (e.g., surrounding packets have TTL=64, RST has TTL=128) is the telltale sign. In a forensic context, this means a security control terminated the connection — document this as "connection terminated by inline security device at [timestamp]" rather than treating it as normal session teardown. If the RST is injected TO the victim (not from the victim), it may be an attacker using RST injection as a disruption technique.