Zeek at Scale
A single Zeek process can saturate 1–2 Gbps. Production environments at 10 Gbps or higher require Zeek cluster mode, load balancing across multiple worker processes, and an optimized log pipeline that ships structured logs to a SIEM or search cluster. This chapter covers the architecture and operational configuration of Zeek at enterprise scale.
A financial services firm needs to monitor 40 Gbps of traffic at their internet perimeter. A single Zeek process can handle about 2 Gbps. They deploy a dedicated 40-Gbps network visibility appliance that load-balances traffic across 20 Zeek worker processes using AF_PACKET fan-out. All workers write logs to Kafka, which feeds Elasticsearch. The result: real-time searchable network metadata at 40 Gbps with 90-day retention.
Scale Architecture
Zeek at Scale — Data Flow
═══════════════════════════════════════════════════════════════════
Network tap (passive, full-duplex)
│
▼
Load balancer (PF_RING DNA / AF_PACKET TPACKET_V3 fan-out)
├── RSS (Receive Side Scaling) — NIC hardware distributes by flow hash
├── PF_RING cluster — round-robin or hash-based to multiple Zeek workers
└── Zero-copy path: kernel copies packet to shared ring buffer once
Workers (N processes, each reads 1/N of traffic)
├── worker-1: processes flows hashed to it → generates logs
├── worker-2: processes flows hashed to it → generates logs
├── ...
└── worker-N: each worker is independent (no shared state per worker)
CRITICAL: Hash by flow 5-tuple, not round-robin!
├── All packets of a single TCP connection must go to ONE worker
├── Wrong: round-robin → half the packets of each connection go to each worker
│ → neither can reassemble streams → empty logs
└── Right: hash(src_ip, src_port, dst_ip, dst_port, proto) % N → worker selection
Log aggregation:
Workers → local log files
↓
Log shipper (Filebeat / Logstash / Vector)
↓
Kafka (durable, ordered, high-throughput queue)
↓
Elasticsearch / Splunk / OpenSearch
(30-90 day retention, full-text search, dashboards)
Manager process:
├── Health monitoring of workers
├── zeekctl deploy/status commands
└── Manages cluster configuration
Logger process:
└── Receives log streams from workers and writes to disk
(separates log I/O from packet processing)
Log Shipping to Elasticsearch
#!/bin/bash
echo "=== Option 1: Filebeat for Zeek logs ==="
# filebeat.yml configuration for Zeek logs to Elasticsearch
cat > /etc/filebeat/filebeat.yml << 'FBCONF'
filebeat.modules:
- module: zeek
connection:
enabled: true
var.paths: ["/opt/zeek/logs/current/conn.log"]
dns:
enabled: true
var.paths: ["/opt/zeek/logs/current/dns.log"]
http:
enabled: true
var.paths: ["/opt/zeek/logs/current/http.log"]
ssl:
enabled: true
var.paths: ["/opt/zeek/logs/current/ssl.log"]
files:
enabled: true
var.paths: ["/opt/zeek/logs/current/files.log"]
notice:
enabled: true
var.paths: ["/opt/zeek/logs/current/notice.log"]
output.elasticsearch:
hosts: ["https://elastic:9200"]
username: "zeek_shipper"
password: "${ELASTIC_PASSWORD}"
index: "zeek-%{[event.dataset]}-%{+yyyy.MM.dd}"
processors:
- add_host_metadata: {}
- add_tags:
tags: ["zeek", "network-monitor"]
FBCONF
echo ""
echo "=== Option 2: Vector for high-throughput log shipping ==="
cat > /etc/vector/vector.toml << 'VCONF'
[sources.zeek_conn]
type = "file"
include = ["/opt/zeek/logs/current/conn.log"]
ignore_checkpoints = false
read_from = "beginning"
[transforms.parse_zeek_conn]
type = "remap"
inputs = ["zeek_conn"]
source = '''
# Parse Zeek TSV format
. = parse_csv!(string!(.message), delimiter: "\t")
.ts = to_float!(.ts)
.event.dataset = "zeek.conn"
'''
[sinks.elasticsearch]
type = "elasticsearch"
inputs = ["parse_zeek_conn"]
endpoint = "https://elastic:9200"
index = "zeek-conn-%Y.%m.%d"
[sinks.elasticsearch.auth]
strategy = "basic"
user = "zeek_shipper"
password = "${ELASTIC_PASSWORD}"
VCONF
echo ""
echo "=== Option 3: Zeek JSON output (easier parsing) ==="
# Add to local.zeek to output JSON instead of TSV
cat >> /opt/zeek/share/zeek/site/local.zeek << 'ZEEK'
@load tuning/json-logs
redef LogAscii::use_json = T;
ZEEK
echo "After this change, Zeek logs are newline-delimited JSON:"
echo '{"ts":1700000000.123,"uid":"Abc123","id.orig_h":"10.0.0.1",...}'Performance Tuning Reference
| Parameter | What It Controls | Tuning |
|---|---|---|
| lb_procs per worker | Number of parallel processes per NIC queue | 1 process per 2 CPU cores, max 8 per worker |
| tcp_SYN_timeout | How long to wait for SYN-ACK before expiring | Reduce to 10s in high-traffic environments |
| tcp_session_timer | How long to keep idle established connections | Reduce to 30s if memory is limited |
| LogAscii::output_to_stdout | Write logs to stdout instead of files | Use with Docker/Kubernetes log collection |
| LogRotationInterval | How often to rotate log files (seconds) | 3600 (hourly) for log shipping pipeline |
| Site::local_nets | Define internal networks | Set to your RFC1918 ranges for proper classification |
| BPF filter on interface | Pre-filter packets before Zeek sees them | Filter out broadcast/multicast if not needed |
The most common Zeek cluster deployment mistake is configuring load balancing as round-robin across workers. In round-robin, packet 1 of a TCP connection goes to worker-1, packet 2 goes to worker-2, and so on. Neither worker can reassemble the TCP stream: they each have half the packets. The result: conn.log shows connections with nearly zero bytes, http.log is empty, ssl.log shows no handshakes. The correct approach is always flow-based hashing: all packets with the same 5-tuple (src_ip, src_port, dst_ip, dst_port, proto) are guaranteed to go to the same worker. AF_PACKET fan-out with PACKET_FANOUT_HASH implements this correctly. PF_RING with cluster_id does as well. Always verify your load balancing is flow-based, not round-robin, before deploying Zeek in cluster mode. The symptom is silent: the cluster appears to run fine but produces garbage logs.
Q & A
Q: How do I know if Zeek is keeping up and not dropping packets?
Zeek reports packet drop statistics in its stats.log (if loaded). Key metrics: (1) zeekctl status shows per-worker status; if a worker shows "crashed" or "stopped" it has restarted due to overload. (2) In stats.log: watch the pkt_drop_rate field — anything above 1% is a problem worth addressing. (3) Check the weird.log: a high rate of "above_hole_data_without_any_acks" or similar TCP reassembly weirdness indicates Zeek is receiving packets it can't reassemble due to drops or out-of-order delivery. (4) At the NIC level: ethtool -S eth0 | grep drop shows hardware ring buffer drops before Zeek even sees packets — if the NIC is dropping, Zeek can't help. (5) Cross-check conn.log against known traffic: if you know a connection transferred 100 MB but conn.log shows 50 MB, there are drops. The fix hierarchy is: hardware (NIC ring buffers → ethtool -G), kernel (socket buffers → sysctl), load balancing (AF_PACKET fan-out), then Zeek configuration.