MTU, fragmentation and path MTU discovery

What it is

The maximum transmission unit is the largest packet a link accepts. RFC 894 fixes 1500 octets for IP over Ethernet; PPPoE reduces it to 1492 (RFC 2516), and every tunnel header (VPN, VXLAN, GRE) takes more. The smallest MTU along a route is the path MTU. RFC 1191 describes IPv4 path MTU discovery: send with the Don't Fragment bit set, and lower the packet size when a router returns an ICMP "Datagram Too Big" message. RFC 8201 describes the IPv6 version, where routers never fragment, the minimum link MTU is 1280 octets (RFC 8200) and the signal is an ICMPv6 Packet Too Big message. RFC 4821 adds packetization-layer discovery (PLPMTUD): TCP probes with larger segments and treats repeated loss of large segments as evidence of a smaller path MTU, without depending on ICMP; RFC 8899 extends the idea to UDP-based transports such as QUIC.

Why it matters

When a firewall drops all ICMP, discovery breaks and the failure looks bizarre: the TCP handshake succeeds, small requests work, and the first large response hangs until a timeout. Nothing is logged because the packets are simply discarded. The pattern appears whenever a tunnel or VPN sits in the path and the endpoints assume 1500.

How to apply

• Allow ICMP type 3 code 4 (IPv4) and ICMPv6 Packet Too Big through every firewall; blocking ICMP wholesale disables path MTU discovery.
• On routers and VPN gateways, clamp the TCP MSS to the path MTU (nftables tcp option maxseg size set rt mtu); the peers then never send segments too large for the tunnel.
• Set the interface MTU correctly on tunnel interfaces instead of leaving the default.
• On Linux hosts behind uncertain paths, enable tcp_mtu_probing (tcp(7)) so that TCP recovers by probing.
• Test with ping -M do -s 1472 host (1472 bytes of payload plus 28 bytes of headers equals 1500) and decrease until the reply comes back.

Pitfalls

Clamping the MSS helps only TCP; UDP applications (DNS with large answers, VPN payloads, QUIC) must keep their own datagrams below the path MTU or rely on RFC 8899-style probing. IPv4 fragmentation still exists when DF is clear, but RFC 8900 explains why it is fragile: only the first fragment carries port numbers, so a stateless firewall must either pass or block all later fragments, and the loss of one fragment loses the whole packet.