Several years ago, I noticed that in some virtual and physical environments, the DOS Shell in both IBM’s and Microsoft’s DOS 4.0x has a very odd glitch: the mouse pointer moves, but does not register any clicks… except on scroll bars, and even then only partially.
On and off, I tried to find the source of the problem. I suspected the keyboard controller (with PS/2 mice), keyboard BIOS, PS/2 mouse BIOS… but never found any bug. Not long ago I finally found the cause of the problem, and it was not at all what I had expected.
The problem didn’t seem to be in the mouse itself, or the mouse driver. In fact the same problem happened with different mouse drivers, and whether nor not the shell’s built-in PS/2 mouse driver (PCIBMDRV.MOS) was used. I tried disassembling the built-in driver but found it to be quite simple and highly unlikely to cause trouble.
Later I made another interesting observation: On one system where the DOS Shell didn’t work right with a mouse, running the shell in an OS/2 VDM magically fixed the problem. On the other hand, running the shell in a Windows 3.x DOS box did not!
That made me wonder about the emulated keyboard controller hardware and how it might differ between Windows and OS/2. Fortunately the virtual keyboard device source code for both systems is available in their respective DDKs.
Examining the code made me realize that Windows 3.x essentially passes through most keyboard controller commands, but OS/2 does not and emulates much of the controller state. Perhaps by luck I happened onto the culprit: The keyboard enable bit (or keyboard inhibit bit, depending on which reference one reads), or bit 4 in the keyboard controller status register which may be obtained by using keyboard controller command C0h. What does that have to do with mouse input? Well…
Some readers of this site may remember that IBM PC/AT machines came with a curious gadget: a key lock (or key switch) mechanism. Most users considered it useless, but it was no doubt required by some corporate customers.
What the key lock did was exactly what the name suggests: it locked the keyboard. When the keyboard was locked, the system wouldn’t boot. A true blue PC/AT would first say “302-System Unit Keylock is Locked”, then “-Unlock System Unit Keylock”, and finally “(RESUME = “F1″ KEY)”. And obviously until the keyboard was unlocked, pressing the F1 key didn’t do much.
When the keyboard was locked in a running system, this simply inhibited any keyboard input but the system remained running. This was perhaps intended for leaving systems unattended and prevent accidental or intentional interference (janitors, untrained personnel).
The writers of the DOS Shell at IBM clearly thought about this scenario and realized that if the mouse can control the shell, a malicious user can still do a lot of damage even when the keyboard is locked—for example by deleting system files (which can be done by just using the mouse). That’s a problem, at least if one cares about the key lock.
So the DOS Shell writers decided to be clever. The shell appears to employ a fairly standard event-driven user interface. At some point mouse clicks are converted to internal events, and at that time the shell checks the keyboard inhibit state (i.e. the key switch locked/unlocked position). If the keyboard is found to be locked, mouse clicks are simply discarded.
Ironically, by the time the DOS Shell came out (1988), the key lock mechanism was obsolete. The XT Model 286 did not come with a key lock at all. PS/2 machines replaced the mechanical switch with a password mechanism which acted like a keyboard lock but could not be easily engaged in a running system, at least to my knowledge.
The DOS Shell writers were well aware of that and checked the machine model and sub-model before reading the keyboard lock state. And that’s where things get very messy.
On an IBM XT Model 286, the keyboard inhibit status bit always indicates an unlocked keyboard. Clone PCs presumably behaved like either a PC/AT (real keyboard lock) or an XT Model 286 (permanently unlocked). This was probably the situation when DOS 4.01 was current.
DOS 5.0 (1991) used a different shell written by Microsoft and the input handling was different, presumably ignoring such archaisms as a keyboard lock. Clone makers, but even IBM (for example ThinkPad T23) forgot that the keyboard lock had ever existed, but pretended that their systems were PC/ATs.
The upshot is that on at least some (if not most) newer systems, keyboard controller command C0h will indicate that the keyboard lock is engaged. On such a system, the DOS shell in IBM/MS DOS 4.0x will ignore mouse clicks because it will think it’s a PC/AT with a key lock that’s in the locked position. Ironically, keyboard input will work just fine because the Shell doesn’t bother checking the inhibit switch and instead expects keyboard input to be prevented on the keyboard controller level (as it would be on a system with a real keyboard lock).
OS/2 2.0 was written long enough ago that the keyboard inhibit switch had not yet been forgotten, and OS/2 MVDM faithfully emulates PC/AT behavior (see PKBDCMD_QUERYINHIBIT and VDMQueryInhibit in the OS/2 DDK source code). Windows 3.x simply passes through the real hardware state, therefore DOS Shell on a given system behaves the same whether running under Enhanced mode Windows 3.x or not (at least with regard to the keyboard inhibit state bit).
All in all, quite a puzzle.