I recently found out that drbd link between two HP DL385 machines tends to stall under high io load. Investigations turned out that tcp segmentation performed by the nic is faulty.

Problem is easy to reproduce. You need some traffic generator (I used xinetd chargen), some traffic sink (I used nc > /dev/null) and iptraf or tcpdump. Set up traffic generator on the machine you want to test and connect to it from outside. You can observe with “iptraf -d ethX” or “tcpdump -vi ethX | grep bad” that ammount of tcp packets with incorrect checksum is unusualy high (I observed around 10%). You can also see these bad packets on the sink machine with tcpdump.

After playing with ethtool -K settings I found out that tcp segmentation offload is to blame. You can turn it off simply with ethtool -K ethX tso off and bad packets magically disappear.

I verified this to be true with bnx2 driver version 1.4.43, 1.4.52d and 1.5.11-rh. I'll repeat tests with 1.6.7 (directly from broadcom site) in about two weeks. There are some reports that it fixes the problem.

Update: the fix went in just after 1.5.11, so all versions after it work properly, including 1.6.7.