Maximum Overkill

A reader pointed at the 1992 game Comanche: Maximum Overkill by NovaLogic as a possible unreal mode user—since the original game release was known to be problematic.

I examined the game and found no unreal mode usage, just a rather unusual homebrew DOS extender (which, among other things, uses expand-down segments for data, presumably in order to catch null pointer accesses).

The original 1992 Comanche release is extremely picky about memory managers. The game refuses to work with EMM emulators and crashes when EMM386 is active. The game may also hang or reboot the system when HIMEM.SYS is not loaded (the problem seems to be system configuration dependent).

These issues are clearly noted in the Comanche documentations, but at the same time they’re also all signs of a substandard DOS extender.

On top of that, the game authors accomplished a remarkable feat: By only loosely following Intel’s instructions on how to enter protected mode, NovaLogic managed to write a game which worked on the then-existing 386 and 486 processors, but under some circumstances failed to run on Pentium and later processors.

To NovaLogic’s credit, these problems were fixed, and an updated CD-ROM release from 1994 works significantly better.

Here is the original 1992 Comanche code sequence which switches from real to protected mode:

1318:0b96 fa                      cli
1318:0b97 2e 0f 01 0e 62 00       sidt [cs:00062h]
1318:0b9d 2e 0f 01 06 5a 00       sgdt [cs:0005ah]
1318:0ba3 66 9c                   pushfd
1318:0ba5 66 2e 8f 06 68 00       pop dword [cs:00068h]
1318:0bab e8 cb 08                call 01479h
1318:0bae 0f 20 c0                mov eax, cr0
1318:0bb1 66 83 c8 01             or eax, byte 000000001h
1318:0bb5 0f 22 c0                mov cr0, eax
1318:0bb8 b8 18 13                mov ax, 01318h
1318:0bbb 8a d4                   mov dl, ah
1318:0bbd c1 e0 04                sal ax, 004h
1318:0bc0 c0 ea 04                shr dl, 004h
1318:0bc3 2a f6                   sub dh, dh
1318:0bc5 05 00 00                add ax, 00000h
1318:0bc8 80 d2 00                adc dl, 000h
1318:0bcb 2e a3 44 00             mov word [cs:00044h], ax
1318:0bcf 2e 89 16 46 00          mov word [cs:00046h], dx
1318:0bd4 2e 0f 01 16 42 00       lgdt [cs:00042h]
1318:0bda b8 18 13                mov ax, 01318h

At address 1318:0BB5, the game enables protected mode by writing to control register CR0. The following instructions then execute in protected mode.

A key detail is that contrary to Intel’s recommendations, the game does not reload segment registers, and does not perform any jumps prescribed by Intel. As a consequence, the game continues running with real-mode CS after switching to protected mode.

The trouble hits at address 1318:0BCB. The game tries to write to memory using a CS segment override. Ironically, the game could just as easily have used an ES segment override because at that point, ES and CS (but not DS!) point to the same segment.

This code worked just fine on 386 and 486 processors, which is obviously how it made it into a commercial game release (because that’s all that was available when the game was released). But the same code often failed on the Pentium, released just a few months later, in March 1993 (the game files are dated November 1992).

How is that possible? In the Pentium, Intel made a subtle but crucial architectural change.

Why CS Is Weird

As discussed elsewhere, real mode works very much like Ring-0 protected mode, with a few relatively minor differences that relate primarily to segment register loading and interrupt dispatching.

However, Intel needed to solve an important problem for which there was no obvious clean solution. In real mode, the CS segment must be writable, yet in protected mode, a writable code segment simply does not exist. This is not such a problem after CPU reset, because the initial CS value can be more or less anything. But it is a serious problem after returning to real mode from protected mode—because by definition, protected mode only loads code selectors into CS, and those can’t be writable. Yet after returning to real mode, CS somehow must be made writable, otherwise real-mode code cannot function.

According to Robert Collins (usually a very reliable source), Intel’s original solution was to modify the CS access rights on any far control transfer (i.e. every time CS got reloaded) in real mode to 93h, which indicates a read-write data segment. This is how the Intel 80386 and 80486 worked.

In the Pentium, the processor does not change the CS access rights in real mode but rather actively ignores them, allowing CS to be writable.

The difference, obviously, is what happens immediately after a transition to protected mode. On the 386/486, CS will be writable because the access rights were (practically guaranteed to have been) modified to read/write. On the Pentium and later processors, the processor will enforce the existing access rights, usually set to 9Bh (an executable and readable but not writable code segment) after a previous transition from protected to real mode. The CS segment will not be writable.

Once again, this is an example of a bug that could not have been caught by any amount of testing, unless perhaps the authors had access to pre-production Pentium processors… but that’s not something game developers usually need.

NovaLogic also released the game at about the worst possible time. If it had been a few months later, the problem with Pentium processors would have been likely noticed and fixed before the release. If it had been a few months earlier, there is some chance that Intel could have found the problem and either changed the Pentium behavior or notified NovaLogic that their code is about to stop working—obviously having a popular game break on a new processor wasn’t in Intel’s interest either.

Documentation

At first, I was about to put the blame entirely on the NovaLogic developers for ignoring Intel’s documentation, which says the following (Pentium Pro Family Developer’s Manual Volume 3: Operating System Writer’s Guide, December 1995, section 8.8.1, Switching to Protected Mode): Immediately following the MOV CR0 instruction, execute a far JMP or far CALL instruction. (This operation is typically a far jump or call to the next instruction in the instruction stream.)

Note that a far jump executed in protected mode is guaranteed to load the CS attributes from the GDT, fully the under control of the software performing the mode switch.

But the Comanche developers presumably did not possess a time machine, hence expecting them to read a 1995 manual back in 1992 could be considered unfair. What, then, did the existing 386 and 486 documentation available in 1992 say? Surely it said the same thing?

Well… not really. This is section 10.4.3, PE Bit, from the 1992 Intel486 Microprocessor Family Programmer’s Reference Manual:

Protected mode is entered by setting the PE bit in the CR0 register. Either an LMSW or
MOV CR0 instruction may be used to set this bit (the MSW register is part of the CR0
register). Because the processor overlaps the interpretation of several instructions, it is
necessary to discard the instructions which already have been read into the processor. A
JMP instruction immediately after the LMSW instruction changes the flow of execution,
so it has the effect of emptying the processor of instructions which have been fetched or
decoded.
After entering protected mode, the segment registers continue to hold the contents they
had in real address mode. Software should reload all the segment registers. Execution in
protected mode begins with a CPL of 0.

Note that the 1992 manual required a JMP instruction, but the purpose was to flush the prefetch queue and ensure that all following instructions were decoded and executed in protected mode; for this purpose, a near jump was sufficient. On newer processors (starting with the Pentium), branch prediction avoids prefetch flushes, so the microarchitecture had to solve the problem differently (e.g. by the MOV CR0 instruction causing a prefetch queue flush).

The 1992 Intel manual states that “software should reload all the segment registers”, but does not indicate that this must be done. The 1995 manual is much less ambiguous. Note that a far jump serves two purposes—it flushes the prefetch queue and it reloads the CS register, ensuring that the CS attributes are set to a known value.

Although Intel does not spell it out, CS must be reloaded in protected mode for interrupts and exceptions to function. The reason is that if an exception or interrupt happens while the real-mode CS is still in effect, it is extremely unlikely that execution could continue after an interrupt—because the real-mode CS saved on the stack is more or less guaranteed to be invalid in protected mode. The same goes for other segment registers—the real-mode DS/ES/FS/GS/SS values can be saved in protected mode, but not restored.

Switching Back to Real Mode

There is also a question of how Intel documented the process of switching from protected mode back to real mode. The 1986 Intel 80386 Programmer’s Reference Manual in section 14.5, Switching Back to Real-Address Mode, recommends the following: Jump to the real mode code to be executed using a far JMP. This action flushes the instruction queue and puts appropriate values in the access rights of the CS register.

Although this is perhaps not entirely explicit, the documentation clearly says that a far control transfer in real mode will modify CS access rights.

The documentation changed for the Pentium. The 1993 Pentium Processor User’s Manual Volume 3: Architecture and Programming Manual in section 16.6.2, Switching Back to Real-Address Mode, states the following: Jump to the real mode program using a far JMP instruction. This flushes the instruction queue (of the Intel386 and Intel486 processors) and puts appropriate values in the access rights of the CS register. This step is not required on the Pentium processor, however, for downwards compatibility, a far JMP should be included as part of the switching back to real-address mode process.

Once again the language is not entirely clear, but implies that a far jump is not required because the Pentium needs neither the explicit prefetch queue flush nor the CS attribute fix.

Testing

My initial theory was that the 1992 Comanche release should work on 386 and 486 systems but fail on all Pentium and newer CPUs. Unfortunately, testing quickly destroyed that theory.

I found that the original Comanche release works on an Alaris Cougar board with the built-in IBM 486BL3 as well as Intel 486 DX4 OverDrive (as expected), but also works with an Intel Pentium OverDrive, which was much less expected.

Next I found that the game also works on an Intel Advanced/ATX (Thor) board with Pentium 133 MHz, Pentium OverDrive 166 MHz (PODP3V166), as well as Pentium MMX OverDrive 200 MHz (PODPMT66X200) CPUs.

At the same time, the original game fails on an IBM ThinkPad 760XL (Pentium MMX 166 MHz), and it also fails on a somewhat newer Intel AN430TX board with a Pentium MMX 166 MHz processor.

To be clear, I verified that the problem is exactly the one described above—trying to write to memory using a CS segment override. Patching the original game executable to use ES segment overrides instead of CS avoids the problem.

At any rate, it’s not the case that the game works or fails on Pentium CPUs in general. It works on some Pentium-based systems, yet fails on others. How is that possible?

Initial CPU State and BIOS

I strongly suspect the difference is caused by the system’s BIOS.

On Intel CPUs, the documented initial state of CS attributes after reset or INIT is “Present, R/W, Accessed”—exactly the same as DS/ES/SS/FS/GS attributes. The catch is that (as explained above) once a Pentium or newer CPU reloads CS in protected mode, it is no longer possible to return to the initial CS attributes.

That is, “transferring execution to a segment which is not executable” is one of the causes of a #GP exception. Because it is impossible to load a non-code segment into CS in protected mode, it is also not possible to make CS writable (since writable code segments don’t exist — the attribute bit which means “writable” for data segments means “readable” for code segments, since code segments may be executable but not readable).

386+ BIOSes typically need to briefly switch to protected mode. It’s the only way to access extended memory, which may also include accessing the APIC registers. However, switching to protected mode does not necessarily require loading CS in protected mode. In fact many older BIOSes run short (or short-ish) code sequences in protected mode while only using short jumps. Newer BIOSes tend to run some 32-bit code during initialization, and that does require loading CS in protected mode.

The upshot is that on some systems with a Pentium or newer CPU, when an operating system boots, CS still contains the initial reset attributes and therefore the code segment will be writable immediately after switching to protected mode. On other systems, the BIOS reloads CS in protected mode, inevitably making CS read-only. After switching to protected mode, the code segment will not be writable.

Note again that with a 386/486 CPU, the behavior is different. Regardless of what the BIOS does in protected mode, CS attributes get changed to read/write on real-mode far control transfers, hence the code segment is effectively guaranteed to be writable after transition to protected mode.

Walking on Eggshells

On 386/486 systems, the original 1992 Comanche release works reliably. On Pentium and newer systems, it may work if the BIOS is “cooperative” and leaves the CS attributes in their reset state.

However, the state of such systems is somewhat unstable. Running more or less any protected-mode program will change the CS attributes, and Comanche will then immediately crash. Needless to say, this includes running Comanche itself! Running the game once works, but running it a second time does not. I imagine this may have caused some head scratching.

To verify the behavior I wrote a small test program which replicates what Comanche does—attempts to write through a CS override immediately after switching to protected mode. Unlike the game, my program is able to handle a #GP fault which occurs on the write and exits cleanly.

The program allowed me to verify that yes, 386/486 CPUs can reliably write through CS immediately after switching to protected mode. Pentium systems may be able to do so, depending on the BIOS. But even on such systems, running almost any protected-mode software irreversibly changes the CS attributes; writing through CS then no longer works in protected mode, and Comanche crashes on startup.

Summary

The Comanche: Maximum Overkill game released in late 1992 uses an unsafe code sequence which reliably works on 386/486 systems, but sometimes or always fails on Pentium processors released in early 1993, as well as on later Intel CPU models.

The problem is caused by a subtle change in handling real-mode CS attributes. While 386 and 486 CPUs change the CS attributes to read/write when reloading CS in real mode, Pentium and later systems ignore the attributes in real mode but do not change them.

As a consequence, immediately after entering protected mode, CS is reliably writable on 386/486 CPUs, but may or may not be writable on Pentium and later systems, depending on whether CS had been reloaded in protected mode since reset or not.

In practice, the game ran on many older Pentium systems whose system BIOS never changed CS in protected mode… but only once. An attempt to start Comanche second time is guaranteed to fail on Pentium and later CPUs.

The old game executable can be easily patched to use ES instead of CS segment overrides, but the original release is still extremely picky about the memory managers used. An updated Comanche executable works significantly better.

Addendum

The above article describes the behavior of Intel CPUs. I suspect that modern AMD CPUs behave slightly differently, in a manner that is not identical to either the old 386/486 behavior or the newer Pentium behavior.