Simulating Lost Packets
I don't know about you, but if I can't fully explain why something is going wrong, I'm never sure that I really got it fixed. Gremlins that magically "disappear" are just in hiding, waiting to strike again, as far as I'm concerned. Case in point, a busy network and allegedly lost SIP packets. The customer reports a particular random occurrence, and there's just no way to know what caused it. But, investigation into the network yields a surprising number of dropped packets by various pieces of network equipment. Hmmm. How would things respond if certain packets are dropped. In particular, how would things respond if the phone's SIP 200 OK packet didn't make it back to the PBX? Would the phone think that it was on the call, but the PBX not? How to study this in a lab environment? Linux to the rescue!
A simple install of Centos 6 in an everyday PC with 2 network cards can yield a router with Wireshark built in and the ability to control and drop packets with iptables!
By default, Linux won't route packets, so we need to edit /etc/sysctl.conf so that net.ipv4.ip_forward has a value of 1, rather than the default value of 0. A reboot is in order to make sure the kernel learns that it can route packets now.
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
In our lab environment, eth0 will be connected to the PBX, and eth1 will be connected to the phones, and each interface gets static addresses as appropriate. The phones will be set with the address of eth1 as their default gateway, and the PBX will be set with the address of eth0 as its default gateway. Once configured and connected, the Linux router should be able to ping the PBX and all the phones.
Let's make sure everything works by disabling iptables on the Linux router. This is done by entering the following.
/etc/init.d/iptables stop
Now test and make sure the phones can call do everything they need to do. Once that is done, we can create a new /etc/sysconfig/iptables that looks like the following. Be aware that this is not any type of firewall or security set of iptables controls; it is simply for simulating dropped/lost packets!
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:SIPOK - [0:0]
-A FORWARD -o eth0 -s 10.0.0.40 -p udp -m udp --dport 5060 -m string --string "200 OK" --algo bm --to 65535 -j SIPOK
-A FORWARD -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A SIPOK -m statistic --mode random --probability 0.8 -j DROP
COMMIT
Recall that eth0 is the PBX side. We want Wireshark to be able to capture packets from the phone on eth1, but we don't want some of those packets to make it to the PBX. The achieve this, we filter packets as they are heading out of eth0, rather than when they are coming in eth1 (which is where Wireshark will be sniffing). In our scenario, we were wanting to filter 80% of the SIP 200 OK packets from a particular phone at IP address 10.0.0.40. We used -s 10.0.0.40 to specify that source IP address, and -p udp -m udp --dport 5060 to specify SIP packets, and -m string --string "200 OK" --algo bm --to 65535 to look for packets that have the 200 OK response. If a packet meets all these requirements, we jump to the SIPOK chain. In this chain, we simply use random probability to drop 80% of the packets that make it to this chain with -m statistic --mode random --probability 0.8. Packets that don't get dropped here, get an implied return to the calling chain (FORWARD) which will accept the packet and send it on its merry way.
Once we restart iptables with the following, we can do our testing and sniffing, and verify our theories. We will have plenty of data thanks to Wireshark, and we can adjust our environment quite easily if we need to alter which or how many packets get dropped.
/etc/init.d/iptables start
It's simple, but techniques like this can be used to simulate all kinds of packet loss scenarios in a very simple lab environment. And, having Wireshark running on the router provides all the evidence to support conclusions and theories!