Last week I spent far too much time trying to get my Windows 11 machine to talk to an antique Tyan SMDC (Server Management Daughter Card) IPMI module over the network.
At first, I tried Tyan’s own old TSO (Tyan System Operator) software in a Windows XP VM using NAT networking. The software installed without incident but was not able to discover any IPMI-enabled servers. All my attempts to manually set up communications to the server also failed. I also tried native Windows software (ipmiutil) with effectively identical result — the SMDC would not respond.
Next I tried my old PC running Windows 10. It behaved the same — ipmiutil refused to talk to the SMDC. At this point, I strongly suspected that the SMDC was not set up right, but just to be sure, I rebooted the old system to Linux and tried ipmitool. Lo and behold, ipmitool on Linux could talk to the SMDC! I set up a similar Windows XP virtual machine on the Linux host and guess what, the TSO software worked fine as well.
All my attempts to disable Windows firewalls and such on the Windows hosts made zero difference. In desperation, I ran Wireshark on the Windows 11 system and established that yes, there is outgoing UDP traffic (to port 623 on the SMDC), and what’s more, the machine is in fact receiving replies from the SMDC! But somehow Windows was eating up the incoming UDP packets and software running on the machine never saw anything. That was, to put it mildly, suspicious.
In an attempt to further narrow down the problem, I tested an old laptop running the latest Windows 10. To my surprise, ipmiutil worked just fine. That shot down my working theory that something in recent Windows 10/11 ate the IPMI UDP packets.
On the Windows 11 machine, I ran PktMon in the hope that it would tell me something that Wireshark did not. Bingo, it did. Although PktMon is generally not that different from Wireshark, it tracks how the network traffic flows through the various Windows networking software layers.
In a text log file produced by PktMon, I found an important clue:
DropReason INET: checksum is invalid
The UDP packet was received and moved up from the network interface through a few filter layers, but the Windows TCP/IP stack dropped it due to an invalid checksum. I turned on checksum validation in Wireshark, but Wireshark thought that the IP and UDP checksums in the received packet were just fine. That was even more suspicious.
The only thing left was to start flipping various options in the driver (e1r68x64.sys) for the Windows 11 machine’s NIC, an Intel I211. It didn’t take me long to establish that turning off UDP receive checksum offloading for IPv4 fixed the problem. Suddenly ipmiutil worked, and VMs running on the system started talking to the SMDC as well.
Needless to say, the Windows 10 machine which I tried second also has an Intel NIC (82579LM), and disabling IPv4 UDP Rx checksum offload in the e1i65x64.sys driver fixed the problem as well.
The working Windows 10 laptop has completely different networking hardware (Killer Wireless 1525 adapter with no offloading, or at least nothing configurable) and wasn’t affected.
What Is Going On Here?
Needless to say, UPD Rx checksum offloading is not generally broken on these systems. If it were, DHCP wouldn’t work, DNS wouldn’t work, and I would have noticed problems far sooner.
The UDP packets sent by the SMDC look fairly innocent and there is nothing obviously wrong about them. Linux, even with the same Intel networking hardware, thinks they’re fine. The Windows TCP/IP stack, when it has to validate the checksums on its own, thinks they’re fine.
Unlike Tx checksum offloading which fills in the checksums in the outgoing packets, Rx checksum offloading does not modify any data. The hardware validates the checksums of incoming packets and sets a valid/invalid flag in a receive descriptor. The driver reports the results to the higher software layers. When the TCP/IP stack verifies the checksums, it does not perform the calculations itself and only looks at the flags set by the hardware/driver. The only possible conclusion is that the Intel hardware and/or driver validates the UDP checksum incorrectly, causing valid packets to be dropped.
I am mystified as to how the Intel NICs/drivers get this wrong. In the straightforward case of a small (76 bytes) UDP packet with no VLAN tags, no tunneling wrappers, and generally nothing special about it, calculating the UDP checksum is not exactly rocket science.
That said, the checksum validation is not done entirely in hardware and (at least on the Intel hardware) needs help from the driver. So there is room for error.
Am I the Only One?
Diagnosing this problem was quite time consuming. Initially I assumed that the remote end (SMDC) either wasn’t receiving or sending. I have had all kinds of trouble with similar old IPMI devices in the past, and I have no reason to think that the Tyan IPMI implementation is particularly solid. But that was not the problem in this case.
Wireshark showed me that the remote end was sending reasonable replies, but provided zero hints as to why software running on my Windows machine wasn’t receiving anything (that’s not a criticism of Wireshark, an incredibly useful tool). PktMon was more helpful and clearly showed that the packets were being dropped by the Windows TCP/IP stack due to a supposedly invalid checksum. That was a strong incentive to start messing with the NIC driver’s checksum offload settings.
I tried searching for a solution but only found vague hints that yes, checksum offloading (both send and receive) can cause trouble, and when it does, the problems may not be terribly obvious. Some even go so far as to recommend turning the offloading entirely, with the argument that if the offloading ever causes any trouble, diagnosing it will be far costlier than anything the offloading ever saved. Which is an argument that has some merit.
Possible Cause
Without disassembling and tracing the Intel Windows drivers (something I don’t feel like doing), I can only speculate about the cause. I suspect it has something to do with the “Packet ID” field in the IP header. I even learned that there is a 2013 RFC which redefines how the ID field should be used and interpreted, apparently drafted after someone realized that it was used in a manner that actually wasn’t compliant with the older RFCs (in short, the ID was getting recycled far too quickly).
The TCP/IP stack on the Tyan SMDC is very simplistic and seems to use the incoming packet as a template for the outgoing one. As a result, the incoming and outgoing packets (from the SMDC’s point of view) have the same the ID. But the outgoing packet is not marked as Do Not Fragment, which could potentially cause trouble. Potentially because in practice, it’s effecrively guaranteed not to get fragmented. But that may be what’s confusing the UDP checksum offloading in the Intel Windows drivers, which is sensitive to fragmentation.
Since I can’t fix the Windows drivers anyway, turning off IPv4 UDP Rx checksum offloading is a perfectly workable solution for me. Except I really shouldn’t need to do that…