Every other page here is an anatomy: a header, a field list, the bytes. This one is a path. That is why it never fit the shape - a firewall is not a protocol, it is the thing that happens to every protocol, in between the pages. A packet arrives and something decides its fate, and the decision is invisible in the packet itself. So the recurring question is not "what do these bytes mean" but "which box did this, and why did it not tell me". Start with the one everybody has lost an afternoon to: connection reset by peer, where the peer swears it sent nothing. Every packet below is real, with every checksum computed. Hover a field to light its bytes, click a field marked + for its lookup table.
The error names the peer. The peer is frequently innocent. A RST is four bytes of flag in a header anyone on the path can write, the source address is trivially forged, and nothing in the packet records who really produced it. Six things send resets, and they are distinguishable on the wire.
| source | when | what it looks like |
|---|---|---|
| the peer's kernel | Nothing is listening on the port. | RST+ACK, one RTT after your SYN, TTL consistent with the peer's other packets. The honest case. |
| the peer's application | SO_LINGER with l_linger=0, or close() called with unread data still in the receive queue. | RST mid-conversation instead of FIN. TTL still matches the peer. Common in proxies avoiding TIME_WAIT. |
| a REJECT rule | -j REJECT --reject-with tcp-reset on the peer host or a firewall in front of it. | Immediate, and it answers the SYN. TTL matches whichever box holds the rule, which may be several hops short of the peer. |
| an injected RST | A DPI or censorship middlebox that wants the connection dead. | Source address forged as the peer. TTL is wrong. Often arrives before the peer's real reply, and the peer keeps talking afterwards. |
| conntrack or NAT | The flow sat idle past a middlebox timeout and the state entry was evicted. | Dies after a consistent idle period. The next packet either draws a RST or vanishes entirely. |
| a load balancer | Idle timeout, backend health check failure, or connection draining during a deploy. | RST rather than a graceful FIN, often in bursts correlated with a rollout. |
A middlebox three hops from the client, forging 203.0.113.5:443. Every field is what the client actually sees. Byte 8 is the one that gives it away.
tcpdump -nnvvi any 'host 203.0.113.5' and read the ttl field on each. A gap of several is the tell.RCV.NXT no longer kills the connection, it draws a challenge ACK instead. An off-path attacker now has to guess the exact sequence number rather than land anywhere in the window. None of that helps against a box that can see the flow, which is what a DPI middlebox is - it reads the real sequence numbers off the wire and writes a perfectly acceptable reset.0x12345679 against a SYN of 0x12345678.This is the difference between "my app hung for two minutes" and "my app failed instantly with a clear error", and it is one word in one firewall rule. DROP discards the packet and sends nothing, so the client learns only when its own retransmit timer gives up. REJECT sends an explicit refusal, so the client fails on the next round trip.
There is no packet to inspect, which is the entire problem. The capture shows only your own SYNs leaving, on a doubling timer.
net.ipv4.tcp_syn_retries, default 6. Backoff doubles: 1, 2, 4, 8, 16, 32, 64 seconds, so the total is 127 seconds before connect() returns ETIMEDOUT. If your hang is always roughly two minutes, this is why, and something is dropping your SYN.net.ipv4.tcp_retries2, default 15, on an already established connection. Roughly 924 seconds, about 15 minutes, before the kernel gives up. If a connection wedges for a quarter of an hour and then errors, packets stopped getting through and nobody said so.The default for -j REJECT is icmp-port-unreachable: ICMP type 3, code 3. It quotes the packet that caused it, and the quote is more useful than the error.
connect(). It is also why an ICMP error cannot be validated deeply: 8 bytes is not enough to prove the sender saw the whole segment, which is what made blind ICMP attacks against TCP practical enough for RFC 5927 to exist.The client's SYN, shown without options for clarity - a real one carries MSS, SACK-permitted, timestamps and window scale. Bytes 0 through 19 here are what get quoted back inside the ICMP above, with only the TTL and checksum differing.
| --reject-with | on the wire | use |
|---|---|---|
| icmp-port-unreachable | type 3 code 3 | The default. Indistinguishable from a host with nothing listening, which is usually what you want. |
| icmp-host-unreachable | type 3 code 1 | Claims the host is gone. Clients may cache this and back off harder. |
| icmp-net-unreachable | type 3 code 0 | Claims the whole network is unreachable. Rarely appropriate from a host firewall. |
| icmp-proto-unreachable | type 3 code 2 | The protocol number is not supported at all. |
| icmp-admin-prohibited | type 3 code 13 | Says out loud that a policy blocked this. Honest, and tells a scanner a firewall exists. |
| tcp-reset | a TCP RST | TCP only. Looks exactly like a closed port - and is suspect number three on the previous tab. |
The ip6tables spelling differs: ICMPv6 type 1 code 4 is icmp6-port-unreachable, and the default there is the same idea with different numbers. nftables writes all of this as reject with icmp type port-unreachable or reject with tcp reset, and a bare reject picks the family-appropriate default.
--reject-with tcp-reset is rejected outright unless the rule already matched -p tcp.Netfilter is five points in the IP stack where registered functions get to look at a packet and return a verdict. Everything else - tables, chains, conntrack, NAT - is code registered at one of those five points with a priority number that decides who runs first. Read the path downward; a packet enters at the top and takes one branch at the routing decision.
sk_buff. Nothing filterable yet.NOTRACK and skip conntrack entirely.net.ipv4.ip_forward=1. filter at priority 0.MASQUERADE. Source rewritten after routing, so the rule can know which interface was chosen.Locally generated traffic skips the first half: the socket routes first, then hits OUTPUT (NF_INET_LOCAL_OUT, with raw, conntrack, mangle and DNAT all registered there too), then joins the same POSTROUTING as everything else.
From include/uapi/linux/netfilter_ipv4.h. Lower runs first. nftables lets you write these names directly in a base chain declaration.
| constant | value | nft name | what registers there |
|---|---|---|---|
| NF_IP_PRI_CONNTRACK_DEFRAG | -400 | - | IPv4 defragmentation for conntrack |
| NF_IP_PRI_RAW | -300 | raw | the raw table; NOTRACK |
| NF_IP_PRI_CONNTRACK | -200 | - | state lookup and assignment |
| NF_IP_PRI_MANGLE | -150 | mangle | the mangle table |
| NF_IP_PRI_NAT_DST | -100 | dstnat | DNAT, at PREROUTING and OUTPUT |
| NF_IP_PRI_FILTER | 0 | filter | the filter table; almost every rule you write |
| NF_IP_PRI_SECURITY | 50 | security | SELinux and friends |
| NF_IP_PRI_NAT_SRC | 100 | srcnat | SNAT and MASQUERADE, at POSTROUTING |
| NF_IP_PRI_CONNTRACK_CONFIRM | INT_MAX | - | commit the conntrack entry, last of all |
iptables is a compatibility front end writing nftables rules.NF_INET_LOCAL_IN and NF_INET_LOCAL_OUT; iptables calls the same two hooks INPUT and OUTPUT. Kernel traces, nft monitor trace output and the tunables reference all use the LOCAL_ spelling, so the two vocabularies collide exactly when you are debugging. They are the same hook.Conntrack is what turns a packet filter into a stateful firewall: it remembers flows so a rule can say "replies to things we started" without enumerating them. It also introduces a table that can fill, entries that can expire, and a vocabulary that looks like TCP's but is not.
| ctstate | means | gotcha |
|---|---|---|
| NEW | A packet conntrack has no entry for, that looks like the start of a flow. | Not the same as "has the SYN bit". A stray mid-stream ACK for a flow conntrack forgot is also NEW, which is why loose rulesets accept things they did not mean to. |
| ESTABLISHED | Traffic has been seen in both directions on this tuple. | Nothing to do with TCP ESTABLISHED. A UDP flow becomes conntrack-ESTABLISHED the moment one reply comes back, and TCP is ESTABLISHED here well before the three-way handshake finishes. |
| RELATED | A new flow that an existing entry expects: an ICMP error quoting a tracked flow, or an FTP data channel opened by the helper. | Helpers parse application payloads in the kernel to predict these. That is exactly as dangerous as it sounds, and why nf_conntrack_helper now defaults to off. |
| INVALID | Cannot be associated with anything: out-of-window TCP, malformed headers, ICMP errors for flows conntrack never saw. | Drop these explicitly. Left unhandled they can slip through a rule that only tests for NEW, and they pollute captures during debugging. |
| UNTRACKED | Deliberately exempted with -j NOTRACK in the raw table. | Costs you all stateful matching for that traffic. Used on very high rate flows to keep the table from filling. |
The entry stores two tuples, not one. The reply tuple is what NAT rewrites, and it is how a translated packet finds its way back: the return traffic is matched against the reply tuple and un-translated automatically. That is why NAT only has to run once per flow.
| sysctl | default | note |
|---|---|---|
| nf_conntrack_tcp_timeout_established | 432000 | Five days. Generous on purpose, and almost never what the middleboxes between you and the peer use. |
| nf_conntrack_tcp_timeout_syn_sent | 120 | Matches the client's own 127-second connect timeout closely enough to be confusing. |
| nf_conntrack_tcp_timeout_time_wait | 120 | |
| nf_conntrack_tcp_timeout_close_wait | 60 | |
| nf_conntrack_udp_timeout | 30 | Rises to 120 once the flow is assured. This is why long-lived UDP needs keepalives. |
| nf_conntrack_max | RAM-sized | Often around 256K. When it fills, new flows are dropped and dmesg says nf_conntrack: table full, dropping packet. |
tcp_retries2 expires 15 minutes later; or a RST, and you get suspect number five from the first tab. The fix is not a firewall rule, it is TCPKeepAlive or an application-level ping under the shortest timeout on the path. The signature is that it always dies after the same idle interval.conntrack -S reports per-CPU insert_failed and drop; those counters moving is the difference between "the network is slow" and "we are out of table". See Attack & Mitigation for the flood side of this, and the Linux network stack tunables for nf_conntrack_max and its hashsize.The two positions are not arbitrary and are worth memorising, because every confusing NAT bug follows from them. Destination translation has to happen before the routing decision, or the packet would be routed to the address you are trying to change. Source translation has to happen after it, or the rule could not know which interface, and therefore which source address, was selected.
REDIRECT. Rewrites the destination. Port forwarding and load balancing live here.MASQUERADE. Rewrites the source, now that the outgoing interface is known.REDIRECT is the special case that means "to this machine".-A FORWARD -d 203.0.113.5 -p tcp --dport 443 -j ACCEPT, and nothing matches: by the time FORWARD runs, the packet is addressed to 10.0.0.7:8443. The rule has to be written against the translated address. The only chain that sees the original is PREROUTING itself.One rule, both spellings, token by token. It rejects new inbound connections to 443 with a reset - which is to say, it is the third suspect from the first tab, written out.
| token | kind | does |
|---|---|---|
| -A INPUT | chain | Append to the INPUT chain. The filter table is implied by omission, and filter+INPUT means NF_INET_LOCAL_IN at priority 0. The hook is never written down. |
| -p tcp | match | Protocol number 6. Also unlocks the tcp match extension, without which --dport is a syntax error. |
| --dport 443 | match ext | Belongs to the implicit tcp extension loaded by -p tcp, not to iptables itself. |
| -m conntrack | match module | Loads the conntrack matcher. The older -m state is a deprecated alias. |
| --ctstate NEW | match ext | Only flows conntrack has no entry for. Established traffic is unaffected, so this rule cannot break connections already open. |
| -j REJECT | target | Terminal. Generates a reply, then drops. |
| --reject-with tcp-reset | target opt | Send a RST instead of the default ICMP port-unreachable. Legal only because -p tcp matched. |
| token | kind | does |
|---|---|---|
| inet | family | One ruleset for IPv4 and IPv6 at once. iptables needs two separate binaries and two rulesets to do this. |
| type filter hook input | base chain | The hook, stated. type can also be nat or route. |
| priority filter | base chain | The integer from the previous tab, by name: filter is 0. Write -150 or mangle and the chain slots in there instead. |
| policy drop | base chain | The verdict when no rule matches. Only base chains have one. |
| tcp dport 443 | expression | No -p tcp needed: the protocol is implied by the header being tested. |
| ct state new | expression | Same conntrack state, lowercase. Sets can be inline: ct state { established, related } accept. |
| reject with tcp reset | verdict | Same generated packet as above. |
-I (insert at the top) and -A (append at the bottom) are the two most consequential characters in the command. If a rule seems to do nothing, check what is above it before you check the rule.iptables command is a translation layer that writes nftables rules into the kernel. iptables -L shows what iptables wrote; nft list ruleset shows everything, including rules from other tools. On a modern box with Docker, Kubernetes or firewalld installed, those two commands routinely disagree - and nft list ruleset is the one telling the truth.Guessing at rulesets is a waste of an afternoon. Every one of these answers a specific question, roughly in order of how much they cost to run.
A rule with a moving counter is doing something. A DROP rule whose counter matches your attempts exactly is your answer, and you did not need a packet capture to find it.
This is the only tool that shows the packet arriving at each hook in order and names the rule that finally decided. Narrow the match first - tracing everything on a busy host produces more output than you can read.
A DESTROY event arriving well before your connection failed is the idle-timeout story from the conntrack tab, confirmed rather than guessed.
nstat and netstat -s expose drops that no firewall rule accounts for: ListenOverflows means the accept queue filled and the kernel silently discarded a completed handshake, which looks exactly like a firewall drop from the client. net.ipv4.conf.*.rp_filter in strict mode drops packets whose source would not route back out the arriving interface, which looks like a firewall drop and appears in no ruleset at all - the classic asymmetric-routing false alarm.What the failure looks like from the application, and where to go next.
| symptom | most likely | check |
|---|---|---|
| Hangs about 127 seconds, then ETIMEDOUT | Your SYN is being DROPped, somewhere. | tcp_syn_retries arithmetic on the Drop tab. Capture both ends: did the SYN arrive? |
| Instant "connection refused" | Nothing listening, or a REJECT rule. Both look identical to the client. | ss -ltn on the server. If a listener exists, a rule sent that. |
| "Connection reset by peer", peer denies it | Injected RST, or a REJECT rule short of the peer. | TTL of the RST versus the peer's other packets. Capture both ends. |
| Dies after a fixed idle period, every time | Conntrack or NAT state expiry in the middle. | The interval is the middlebox's timeout. Add keepalives below it. |
| Handshake fine, large transfers hang | PMTU black hole: ICMP type 3 code 4 is being filtered. | Stop dropping ICMP. Test with ping -M do -s 1472. |
| Wedges roughly 15 minutes, then errors | An established flow stopped getting through mid-stream. | tcp_retries2. Something changed on the path after the connection opened. |
| Works to the host, not through it | Rules written in INPUT on a box that forwards. | The routing-decision branch on the Hook Path tab. Transit never visits INPUT. |
| Port forward reaches nothing | FORWARD rule written against the pre-DNAT address. | filter sees post-DNAT addresses. See the NAT tab. |
| Changed a NAT rule, nothing changed | Open flows still use the mapping recorded at their first packet. | Flush conntrack, or test with a brand new connection. |
| Intermittent drops under load, no rule matches | Conntrack table full, or accept-queue overflow. | conntrack -S, dmesg | grep conntrack, nstat | grep -i overflow. |
| One direction works, the other silently fails | Strict rp_filter with asymmetric routing. | Appears in no ruleset. sysctl net.ipv4.conf.all.rp_filter. |
| iptables looks empty, traffic still blocked | Rules written by another tool through nftables. | nft list ruleset, not iptables -L. |