Featured image

Cheatsheet: network debugging on Linux

Some useful commands for network debugging on Linux.

Wireshark

… on remote host

ssh server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -

… on remote host over a jump server

ssh -J jumpserver server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -

… on remote host over a jump server (if the private key/ssh config for the target host lays only on the jump host)

on the jump server:

ssh server 'tcpdump -ni any -s0 -U -w- udp port 53' > /tmp/packets.pcap

on our local machine:

ssh jump "tail -f -c +0 /tmp/packets.pcap" | wireshark -k -i -

Find your traffic in tcpdump

Use source ports (NAT does not change the port) for the tcpdump filter.
Explanation: Linux uses a port out of ip_local_port_range as source port for outgoing connections.

kmille@linbox /proc% cat sys/net/ipv4/ip_local_port_range
32768   60999

This means: if we use a lower port for outgoing traffic we can easily find it.

tcpdump -ni any portrange 2000-3000
nc -p 2000 -v localhost 8000
nc -s 192.168.10.70 -p 2000 -v localhost 8000
curl --local-port 2001 -v localhost:8000
curl --interface 192.168.10.70 --local-port 2001 -v localhost:8000

Find the dropping firewall

Use mtr for a simple traceroute because it can add a TCP_SYN with destination port 443 as layer 4. It will show you where the packet is dropped when there are multiple firewalls and you don’t know which is dropping your SYNs.

mtr --tcp -P 443 server

Useful tcpdump filter

SYN only (someone tries to connect but the firewall drops => you will see the retransmissions)

tcpdump -ni any 'tcp[tcpflags] == tcp-syn'

Another way to find retransmissions: sar -n ETCP 1 and check the retrans/s column. To find out which connection/application use.

watch -n 0.1 'ss -tpn state syn-sent'

SYN and SYN-ACK (shows only new established tcp connections)

tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13]=18'

SYN and RST (port is closed)

tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13] & 4!=0'

SYN and ICMP port unreachable (firewall rejects packet)

tcpdump -ni any 'tcp[tcpflags] == tcp-syn or icmp[0] = 3'

SYN AND SYN-ACK AND RST AND ICMP port unreachable

tcpdump -ni any '(tcp[tcpflags] == tcp-syn or tcp[13]=18) or tcp[13] & 4!=0 or icmp[0] = 3'

For outgoing connections use tcpconnect to get the process which is sending the packets. Or ss -tanp or netstat -tanp.

Debugging iptables

Clear and check the counter (pkts/bytes)

iptables -Z INPUT
iptables -Z INPUT 1 # one is the number you see with iptables -vnL INPUT --line-numbers
watch -n 0.1 iptables -vnL INPUT

Log traffic to dmesg

iptables -I INPUT --match multiport --sports 2000:3000 -j LOG --log-prefix "our debug traffic"

Use the nflog interface

iptables -A FORWARD -p icmp -j NFLOG --nflog-group 5
wireshark -ni nflog:5

How to trace the iptables rules Linux applies
on older systems

modprobe ipt_LOG
echo ipt_LOG >/proc/sys/net/netfilter/nf_log/2

on newer systems

modprobe nf_log_ipv4
sysctl net.netfilter.nf_log.2=nf_log_ipv4

For filtering, use the raw table.
Use the OUTPUT chain for outgoing traffic.
Use the PREROUTING chain for incoming traffic.

As an example

iptables -t raw -I OUTPUT -p icmp -j TRACE
iptables -t raw -I PREROUTING -p icmp -j TRACE

Check dmesg for some output. If there is no output use nft monitor trace instead of dmesg (on systems using netfilter and not nftables). You can also use /usr/sbin/iptables-legacy instead of the new default /usr/sbin/iptables-nft.

Example output of the tracing

kmille@linbox ~% dmesg -wHT
...
[Thu Apr 23 21:55:11 2020] TRACE: raw:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 	CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: nat:OUTPUT:policy:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: filter:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: nat:POSTROUTING:rule:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: raw:PREROUTING:policy:2 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1
[Thu Apr 23 21:55:11 2020] TRACE: filter:INPUT:rule:3 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1

This is what it looks like in theory:
flow is great

For prettier output

kmille@linbox ~% dmesg  | grep TRACE: | egrep -v 'security|raw' | cut -d ' ' -f 3-8,14-17,21-22 | column -t
nat:OUTPUT:policy:1     IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
filter:OUTPUT:policy:2  IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
nat:POSTROUTING:rule:1  IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
filter:INPUT:rule:3     IN=wlp3s0  OUT=        MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00  SRC=193.99.144.80  DST=192.168.10.70  PROTO=ICMP  TYPE=0  CODE=0  ID=13

My mac addresses is redacted on purpose. For production you can use the following rules:

iptables -t raw -I OUTPUT -p tcp -m multiport --sports 2000:3000 -j TRACE
iptables -t raw -I OUTPUT -p tcp -m multiport --dports 2000:3000 -j TRACE
iptables -t raw -I PREROUTING -p tcp -m multiport --dports 2000:3000 -j TRACE
iptables -t raw -I PREROUTING -p tcp -m multiport --sports 2000:3000 -j TRACE

To clear the tracing rules, use

iptables -t raw -F