Some useful commands for network debugging on Linux.
Wireshark
… on remote host
ssh server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -
… on remote host over a jump server
ssh -J jumpserver server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i -
… on remote host over a jump server (if the private key/ssh config for the target host lays only on the jump host)
on the jump server:
ssh server 'tcpdump -ni any -s0 -U -w- udp port 53' > /tmp/packets.pcap
on our local machine:
ssh jump "tail -f -c +0 /tmp/packets.pcap" | wireshark -k -i -
Find your traffic in tcpdump
Use source ports (NAT does not change the port) for the tcpdump filter.
Explanation: Linux uses a port out of ip_local_port_range as source port for outgoing connections.
kmille@linbox /proc% cat sys/net/ipv4/ip_local_port_range
32768 60999
This means: if we use a lower port for outgoing traffic we can easily find it.
tcpdump -ni any portrange 2000-3000
nc -p 2000 -v localhost 8000
nc -s 192.168.10.70 -p 2000 -v localhost 8000
curl --local-port 2001 -v localhost:8000
curl --interface 192.168.10.70 --local-port 2001 -v localhost:8000
Find the dropping firewall
Use mtr for a simple traceroute because it can add a TCP_SYN with destination port 443 as layer 4. It will show you where the packet is dropped when there are multiple firewalls and you don’t know which is dropping your SYNs.
mtr --tcp -P 443 server
Useful tcpdump filter
SYN only (someone tries to connect but the firewall drops => you will see the retransmissions)
tcpdump -ni any 'tcp[tcpflags] == tcp-syn'
Another way to find retransmissions: sar -n ETCP 1 and check the retrans/s column. To find out which connection/application use.
watch -n 0.1 'ss -tpn state syn-sent'
SYN and SYN-ACK (shows only new established tcp connections)
tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13]=18'
SYN and RST (port is closed)
tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13] & 4!=0'
SYN and ICMP port unreachable (firewall rejects packet)
tcpdump -ni any 'tcp[tcpflags] == tcp-syn or icmp[0] = 3'
SYN AND SYN-ACK AND RST AND ICMP port unreachable
tcpdump -ni any '(tcp[tcpflags] == tcp-syn or tcp[13]=18) or tcp[13] & 4!=0 or icmp[0] = 3'
For outgoing connections use tcpconnect to get the process which is sending the packets. Or ss -tanp or netstat -tanp.
Debugging iptables
Clear and check the counter (pkts/bytes)
iptables -Z INPUT
iptables -Z INPUT 1 # one is the number you see with iptables -vnL INPUT --line-numbers
watch -n 0.1 iptables -vnL INPUT
Log traffic to dmesg
iptables -I INPUT --match multiport --sports 2000:3000 -j LOG --log-prefix "our debug traffic"
Use the nflog interface
iptables -A FORWARD -p icmp -j NFLOG --nflog-group 5
wireshark -ni nflog:5
How to trace the iptables rules Linux applies
on older systems
modprobe ipt_LOG
echo ipt_LOG >/proc/sys/net/netfilter/nf_log/2
on newer systems
modprobe nf_log_ipv4
sysctl net.netfilter.nf_log.2=nf_log_ipv4
For filtering, use the raw table.
Use the OUTPUT chain for outgoing traffic.
Use the PREROUTING chain for incoming traffic.
As an example
iptables -t raw -I OUTPUT -p icmp -j TRACE
iptables -t raw -I PREROUTING -p icmp -j TRACE
Check dmesg for some output. If there is no output use nft monitor trace instead of dmesg (on systems using netfilter and not nftables). You can also use /usr/sbin/iptables-legacy instead of the new default /usr/sbin/iptables-nft.
Example output of the tracing
kmille@linbox ~% dmesg -wHT
...
[Thu Apr 23 21:55:11 2020] TRACE: raw:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: nat:OUTPUT:policy:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: filter:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: nat:POSTROUTING:rule:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100
[Thu Apr 23 21:55:11 2020] TRACE: raw:PREROUTING:policy:2 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1
[Thu Apr 23 21:55:11 2020] TRACE: filter:INPUT:rule:3 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1
This is what it looks like in theory:
For prettier output
kmille@linbox ~% dmesg | grep TRACE: | egrep -v 'security|raw' | cut -d ' ' -f 3-8,14-17,21-22 | column -t
nat:OUTPUT:policy:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=193.99.144.80 LEN=84 PROTO=ICMP TYPE=8 CODE=0 ID=13
filter:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=193.99.144.80 LEN=84 PROTO=ICMP TYPE=8 CODE=0 ID=13
nat:POSTROUTING:rule:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=193.99.144.80 LEN=84 PROTO=ICMP TYPE=8 CODE=0 ID=13
filter:INPUT:rule:3 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=193.99.144.80 DST=192.168.10.70 PROTO=ICMP TYPE=0 CODE=0 ID=13
My mac addresses is redacted on purpose. For production you can use the following rules:
iptables -t raw -I OUTPUT -p tcp -m multiport --sports 2000:3000 -j TRACE
iptables -t raw -I OUTPUT -p tcp -m multiport --dports 2000:3000 -j TRACE
iptables -t raw -I PREROUTING -p tcp -m multiport --dports 2000:3000 -j TRACE
iptables -t raw -I PREROUTING -p tcp -m multiport --sports 2000:3000 -j TRACE
To clear the tracing rules, use
iptables -t raw -F