A few days ago I spent far too much time debugging a largish piece of 16-bit Windows code written in assembler. I found a scenario where (fortunately fairly reproducibly) Windows crashed because the internal state of a DLL got corrupted.
I could see that the state made no sense and a supposedly small rectangle was suddenly tens of thousands of pixels large, causing segment overruns when it was copied. The internal state of the DLL was corrupted, and it very much looked like a buffer overflow.
I added size checks to make sure nearby buffers weren’t being overwritten, but the checks never fired. Or rather they only fired when the state was already corrupted.
Then I tried reshuffling the data so that the buffer which I suspected of overflowing was at the very end of the data segment, hoping that it would cause a protection fault when the buffer overrun happened. But the fault never happened, and the state was still getting corrupted.
And then it finally hit me. The DLL’s internal state was getting corrupted not because a data copy ran past the end of a buffer, but because it was copying data in the wrong direction. In some situations, a REP MOVSB instruction could be executed with the correct source and destination address and the correct count, but with the CPU direction flag set (rather of clear), causing the copy to decrement addresses instead of incrementing.
This is a situation programmers nowadays just don’t think of. Backward copying is generally avoided because CPUs are not good at it and it’s therefore slow. In a language like C, buffer overflows can happen, but only running past the end of a buffer, not overwriting data located in memory before the start of a buffer (it’s possible to decrement an array index so that it points below the first element, but that is rare). The cause of an overflow is invariably a size problem, with the amount of data copied being larger than the size of the destination buffer. That’s what I was looking for, but the bug was something different.
ABIs generally assume that the CPU direction flag must be clear, and if some code sets it, then it must clear the flag again before returning to the caller or before calling other functions. String move instructions thus always work in the “up” direction unless someone explicitly sets the direction flag (which implies assembler code), and such buffer “underflows” normally can’t happen. But the CPU is still capable of executing them and with old and sketchy code, it is something to be aware of.