There’s an interesting (and quite annoying) bug specific to Windows 3.x running in 386 enhanced mode (also known as Windows/386) and using a PS/2 mouse. In some situations, the mouse may jump to the top or the bottom of the screen, especially when the system is somewhat unresponsive. The cause of the jumpiness turns out to be a subtle bug in VKD (Virtual Keyboard Device), a virtual device (VxD) which is part of Windows/386.
The way PS/2 mouse input is handled in Windows/386 is extraordinarily complex. The mouse driver (MOUSE.DRV) simply uses the BIOS INT 15h, sub-function C2h interface to register a mouse callback routine. The BIOS uses IRQ 12 and I/O ports 60h/64h to read data from the mouse. However, the VKD VxD inserts a whole new layer between the BIOS and the mouse hardware.
In Windows/386, VKD emulates a PS/2 style keyboard controller, including the mouse (auxiliary device) support. Microsoft implemented that in order to better separate VMs from hardware (and prevent VMs from globally controlling the A20 gate etc.) and provide hotkey support in a way which is transparent to the BIOS and DOS. Another benefit of emulating the keyboard controller is the ability to paste input into VMs in a way which is effectively indistinguishable from keyboard input. Emulation of the PS/2 auxiliary interface does not appear to bring any real benefit to the user, but was probably inevitable since it shares the same controller as keyboard input.
Now a small digression about the PS/2 mouse protocol. Whenever the mouse is moved or a button is pressed/released, the mouse sends a “packet” of data consisting of three bytes (there are extensions to that protocol, irrelevant for a discussion of standard Windows 3.x). A status byte is followed by two bytes with X and Y direction movement data. The movement information is 9 bits for each direction, with the top bit of each stored in the status byte. There is no guaranteed way to recognize where the packet starts and ends, therefore driver code must take great care to not lose any byte of data; if it does, the PS/2 mouse protocol will lose synchronization and the data will be misinterpreted.
For the mouse support, VKD (more specifically the VAD, or Virtual Auxiliary Device component of VKD) implements a 32-byte queue (circular buffer). The VAD reads incoming bytes from the real keyboard/mouse controller and stores them internally, until at some later point the BIOS fetches them from the emulated controller. The queue size is a power of two for fast access. If the queue is full, the VAD takes care to always throw away three bytes of incoming data so that the PS/2 protocol stays synchronized. Unfortunately, there’s a bug in the queue overflow handling logic.
Because the VAD queue is 32 bytes in size, there’s room for 10 complete PS/2 packets and one incomplete, containing only the status byte and X direction movement data. And that’s where the bug is. The VAD will realize that the queue is full only after it has stored two out of three bytes of an incoming PS/2 packet. It will then discard the next three bytes, i.e. the last byte of the current packet plus the first two bytes of the next.
If the queue has emptied somewhat by the time the last byte (Y movement information) of the next mouse packet arrives, that byte will be stored together with the first two bytes of the earlier packet. The mouse will jump if the Y movement direction has changed between the two packets; it should be kept in mind that multiple packets may be discarded if the queue is not emptied quickly enough. If the direction changes, the top bit of a 9-bit quantity in the status byte won’t match the low 8 bits in the Y movement byte. Typically either a small positive number (e.g. 1) will be transformed to a large negative number (e.g. -255) or vice versa.
That explains why the mouse only ever jumps in the Y direction, either to the top or the bottom of the screen. Because the status byte is always consistent with the X direction movement byte, no jumps can occur in the horizontal direction. Under normal circumstances, the system should be emptying the VAD queue quickly enough that it will never overflow. But if it for any reason doesn’t and the queue fills up, the mouse jumps are quite likely to occur.
The above information was obtained by reading the VKD sources in the Windows 3.1 DDK (especially DDK\386\VDD\VAD.ASM) and Microsoft’s WDEB386 debugger in conjunction with a debug build of WIN386.EXE (also supplied with the DDK). The WDEB386 debugger needed patching to run on a post-486 CPU. The .VKD command in WDEB386 was used to enable VAD trace-outs, later read with the .LQ command.