A reader pointed at the 1992 game Comanche: Maximum Overkill by NovaLogic as a possible unreal mode user—since the original game release was known to be problematic.
I examined the game and found no unreal mode usage, just a rather unusual homebrew DOS extender (which, among other things, uses expand-down segments for data, presumably in order to catch null pointer accesses).
The original 1992 Comanche release is extremely picky about memory managers. The game refuses to work with EMM emulators and crashes when EMM386 is active. The game may also hang or reboot the system when HIMEM.SYS is not loaded (the problem seems to be system configuration dependent).
These issues are clearly noted in the Comanche documentations, but at the same time they’re also all signs of a substandard DOS extender.
On top of that, the game authors accomplished a remarkable feat: By only loosely following Intel’s instructions on how to enter protected mode, NovaLogic managed to write a game which worked on the then-existing 386 and 486 processors, but under some circumstances failed to run on Pentium and later processors.
To NovaLogic’s credit, these problems were fixed, and an updated CD-ROM release from 1994 works significantly better.
Here is the original 1992 Comanche code sequence which switches from real to protected mode:
1318:0b96 fa cli
1318:0b97 2e 0f 01 0e 62 00 sidt [cs:00062h]
1318:0b9d 2e 0f 01 06 5a 00 sgdt [cs:0005ah]
1318:0ba3 66 9c pushfd
1318:0ba5 66 2e 8f 06 68 00 pop dword [cs:00068h]
1318:0bab e8 cb 08 call 01479h
1318:0bae 0f 20 c0 mov eax, cr0
1318:0bb1 66 83 c8 01 or eax, byte 000000001h
1318:0bb5 0f 22 c0 mov cr0, eax
1318:0bb8 b8 18 13 mov ax, 01318h
1318:0bbb 8a d4 mov dl, ah
1318:0bbd c1 e0 04 sal ax, 004h
1318:0bc0 c0 ea 04 shr dl, 004h
1318:0bc3 2a f6 sub dh, dh
1318:0bc5 05 00 00 add ax, 00000h
1318:0bc8 80 d2 00 adc dl, 000h
1318:0bcb 2e a3 44 00 mov word [cs:00044h], ax
1318:0bcf 2e 89 16 46 00 mov word [cs:00046h], dx
1318:0bd4 2e 0f 01 16 42 00 lgdt [cs:00042h]
1318:0bda b8 18 13 mov ax, 01318h
At address 1318:0BB5, the game enables protected mode by writing to control register CR0. The following instructions then execute in protected mode.
A key detail is that contrary to Intel’s recommendations, the game does not reload segment registers, and does not perform any jumps prescribed by Intel. As a consequence, the game continues running with real-mode CS after switching to protected mode.
The trouble hits at address 1318:0BCB. The game tries to write to memory using a CS segment override. Ironically, the game could just as easily have used an ES segment override because at that point, ES and CS (but not DS!) point to the same segment.
This code worked just fine on 386 and 486 processors, which is obviously how it made it into a commercial game release (because that’s all that was available when the game was released). But the same code often failed on the Pentium, released just a few months later, in March 1993 (the game files are dated November 1992).
How is that possible? In the Pentium, Intel made a subtle but crucial architectural change.
Why CS Is Weird
As discussed elsewhere, real mode works very much like Ring-0 protected mode, with a few relatively minor differences that relate primarily to segment register loading and interrupt dispatching.
However, Intel needed to solve an important problem for which there was no obvious clean solution. In real mode, the CS segment must be writable, yet in protected mode, a writable code segment simply does not exist. This is not such a problem after CPU reset, because the initial CS value can be more or less anything. But it is a serious problem after returning to real mode from protected mode—because by definition, protected mode only loads code selectors into CS, and those can’t be writable. Yet after returning to real mode, CS somehow must be made writable, otherwise real-mode code cannot function.
According to Robert Collins (usually a very reliable source), Intel’s original solution was to modify the CS access rights on any far control transfer (i.e. every time CS got reloaded) in real mode to 93h, which indicates a read-write data segment. This is how the Intel 80386 and 80486 worked.
In the Pentium, the processor does not change the CS access rights in real mode but rather actively ignores them, allowing CS to be writable.
The difference, obviously, is what happens immediately after a transition to protected mode. On the 386/486, CS will be writable because the access rights were (practically guaranteed to have been) modified to read/write. On the Pentium and later processors, the processor will enforce the existing access rights, usually set to 9Bh (an executable and readable but not writable code segment) after a previous transition from protected to real mode. The CS segment will not be writable.
Once again, this is an example of a bug that could not have been caught by any amount of testing, unless perhaps the authors had access to pre-production Pentium processors… but that’s not something game developers usually need.
NovaLogic also released the game at about the worst possible time. If it had been a few months later, the problem with Pentium processors would have been likely noticed and fixed before the release. If it had been a few months earlier, there is some chance that Intel could have found the problem and either changed the Pentium behavior or notified NovaLogic that their code is about to stop working—obviously having a popular game break on a new processor wasn’t in Intel’s interest either.
Documentation
At first, I was about to put the blame entirely on the NovaLogic developers for ignoring Intel’s documentation, which says the following (Pentium Pro Family Developer’s Manual Volume 3: Operating System Writer’s Guide, December 1995, section 8.8.1, Switching to Protected Mode): Immediately following the MOV CR0 instruction, execute a far JMP or far CALL instruction. (This operation is typically a far jump or call to the next instruction in the instruction stream.)
Note that a far jump executed in protected mode is guaranteed to load the CS attributes from the GDT, fully the under control of the software performing the mode switch.
But the Comanche developers presumably did not possess a time machine, hence expecting them to read a 1995 manual back in 1992 could be considered unfair. What, then, did the existing 386 and 486 documentation available in 1992 say? Surely it said the same thing?
Well… not really. This is section 10.4.3, PE Bit, from the 1992 Intel486 Microprocessor Family Programmer’s Reference Manual:
Protected mode is entered by setting the PE bit in the CR0 register. Either an LMSW or
MOV CR0 instruction may be used to set this bit (the MSW register is part of the CR0
register). Because the processor overlaps the interpretation of several instructions, it is
necessary to discard the instructions which already have been read into the processor. A
JMP instruction immediately after the LMSW instruction changes the flow of execution,
so it has the effect of emptying the processor of instructions which have been fetched or
decoded.
After entering protected mode, the segment registers continue to hold the contents they
had in real address mode. Software should reload all the segment registers. Execution in
protected mode begins with a CPL of 0.
Note that the 1992 manual required a JMP instruction, but the purpose was to flush the prefetch queue and ensure that all following instructions were decoded and executed in protected mode; for this purpose, a near jump was sufficient. On newer processors (starting with the Pentium), branch prediction avoids prefetch flushes, so the microarchitecture had to solve the problem differently (e.g. by the MOV CR0 instruction causing a prefetch queue flush).
The 1992 Intel manual states that “software should reload all the segment registers”, but does not indicate that this must be done. The 1995 manual is much less ambiguous. Note that a far jump serves two purposes—it flushes the prefetch queue and it reloads the CS register, ensuring that the CS attributes are set to a known value.
Although Intel does not spell it out, CS must be reloaded in protected mode for interrupts and exceptions to function. The reason is that if an exception or interrupt happens while the real-mode CS is still in effect, it is extremely unlikely that execution could continue after an interrupt—because the real-mode CS saved on the stack is more or less guaranteed to be invalid in protected mode. The same goes for other segment registers—the real-mode DS/ES/FS/GS/SS values can be saved in protected mode, but not restored.
Switching Back to Real Mode
There is also a question of how Intel documented the process of switching from protected mode back to real mode. The 1986 Intel 80386 Programmer’s Reference Manual in section 14.5, Switching Back to Real-Address Mode, recommends the following: Jump to the real mode code to be executed using a far JMP. This action flushes the instruction queue and puts appropriate values in the access rights of the CS register.
Although this is perhaps not entirely explicit, the documentation clearly says that a far control transfer in real mode will modify CS access rights.
The documentation changed for the Pentium. The 1993 Pentium Processor User’s Manual Volume 3: Architecture and Programming Manual in section 16.6.2, Switching Back to Real-Address Mode, states the following: Jump to the real mode program using a far JMP instruction. This flushes the instruction queue (of the Intel386 and Intel486 processors) and puts appropriate values in the access rights of the CS register. This step is not required on the Pentium processor, however, for downwards compatibility, a far JMP should be included as part of the switching back to real-address mode process.
Once again the language is not entirely clear, but implies that a far jump is not required because the Pentium needs neither the explicit prefetch queue flush nor the CS attribute fix.
Testing
My initial theory was that the 1992 Comanche release should work on 386 and 486 systems but fail on all Pentium and newer CPUs. Unfortunately, testing quickly destroyed that theory.
I found that the original Comanche release works on an Alaris Cougar board with the built-in IBM 486BL3 as well as Intel 486 DX4 OverDrive (as expected), but also works with an Intel Pentium OverDrive, which was much less expected.
Next I found that the game also works on an Intel Advanced/ATX (Thor) board with Pentium 133 MHz, Pentium OverDrive 166 MHz (PODP3V166), as well as Pentium MMX OverDrive 200 MHz (PODPMT66X200) CPUs.
At the same time, the original game fails on an IBM ThinkPad 760XL (Pentium MMX 166 MHz), and it also fails on a somewhat newer Intel AN430TX board with a Pentium MMX 166 MHz processor.
To be clear, I verified that the problem is exactly the one described above—trying to write to memory using a CS segment override. Patching the original game executable to use ES segment overrides instead of CS avoids the problem.
At any rate, it’s not the case that the game works or fails on Pentium CPUs in general. It works on some Pentium-based systems, yet fails on others. How is that possible?
Initial CPU State and BIOS
I strongly suspect the difference is caused by the system’s BIOS.
On Intel CPUs, the documented initial state of CS attributes after reset or INIT is “Present, R/W, Accessed”—exactly the same as DS/ES/SS/FS/GS attributes. The catch is that (as explained above) once a Pentium or newer CPU reloads CS in protected mode, it is no longer possible to return to the initial CS attributes.
That is, “transferring execution to a segment which is not executable” is one of the causes of a #GP exception. Because it is impossible to load a non-code segment into CS in protected mode, it is also not possible to make CS writable (since writable code segments don’t exist — the attribute bit which means “writable” for data segments means “readable” for code segments, since code segments may be executable but not readable).
386+ BIOSes typically need to briefly switch to protected mode. It’s the only way to access extended memory, which may also include accessing the APIC registers. However, switching to protected mode does not necessarily require loading CS in protected mode. In fact many older BIOSes run short (or short-ish) code sequences in protected mode while only using short jumps. Newer BIOSes tend to run some 32-bit code during initialization, and that does require loading CS in protected mode.
The upshot is that on some systems with a Pentium or newer CPU, when an operating system boots, CS still contains the initial reset attributes and therefore the code segment will be writable immediately after switching to protected mode. On other systems, the BIOS reloads CS in protected mode, inevitably making CS read-only. After switching to protected mode, the code segment will not be writable.
Note again that with a 386/486 CPU, the behavior is different. Regardless of what the BIOS does in protected mode, CS attributes get changed to read/write on real-mode far control transfers, hence the code segment is effectively guaranteed to be writable after transition to protected mode.
Walking on Eggshells
On 386/486 systems, the original 1992 Comanche release works reliably. On Pentium and newer systems, it may work if the BIOS is “cooperative” and leaves the CS attributes in their reset state.
However, the state of such systems is somewhat unstable. Running more or less any protected-mode program will change the CS attributes, and Comanche will then immediately crash. Needless to say, this includes running Comanche itself! Running the game once works, but running it a second time does not. I imagine this may have caused some head scratching.
To verify the behavior I wrote a small test program which replicates what Comanche does—attempts to write through a CS override immediately after switching to protected mode. Unlike the game, my program is able to handle a #GP fault which occurs on the write and exits cleanly.
The program allowed me to verify that yes, 386/486 CPUs can reliably write through CS immediately after switching to protected mode. Pentium systems may be able to do so, depending on the BIOS. But even on such systems, running almost any protected-mode software irreversibly changes the CS attributes; writing through CS then no longer works in protected mode, and Comanche crashes on startup.
Summary
The Comanche: Maximum Overkill game released in late 1992 uses an unsafe code sequence which reliably works on 386/486 systems, but sometimes or always fails on Pentium processors released in early 1993, as well as on later Intel CPU models.
The problem is caused by a subtle change in handling real-mode CS attributes. While 386 and 486 CPUs change the CS attributes to read/write when reloading CS in real mode, Pentium and later systems ignore the attributes in real mode but do not change them.
As a consequence, immediately after entering protected mode, CS is reliably writable on 386/486 CPUs, but may or may not be writable on Pentium and later systems, depending on whether CS had been reloaded in protected mode since reset or not.
In practice, the game ran on many older Pentium systems whose system BIOS never changed CS in protected mode… but only once. An attempt to start Comanche second time is guaranteed to fail on Pentium and later CPUs.
The old game executable can be easily patched to use ES instead of CS segment overrides, but the original release is still extremely picky about the memory managers used. An updated Comanche executable works significantly better.
Addendum One
The behavior of the 80386 is well explained in a confidential January 1986 Intel memo titled Returning to real mode on the 80386. The memo says: The CS is a special case: since it is not possible since it is not possible to make CS writeable while in protected mode, an architectural feature reloads real mode attributes into the CS descriptor during real mode far jumps.
Addendum Two
The above article describes the behavior of Intel CPUs. I suspect that modern AMD CPUs behave slightly differently, in a manner that is not identical to either the old 386/486 behavior or the newer Pentium behavior.
Really cool read, thank you! 🙂
NovaLogic later released a patch for the game that makes it work with EMM managers, and most likely fixes this Pentium issue too. I think it fixes is by just patching in the DOS/4GW DOS extender that most other protected mode DOS games used, but I’m not sure.
The patch file is called Comanche-Maximum-Overkill_Patch_DOS_EN.zip. The site that hosts the patch is an Abandonware site, so I don’t know if I’m allowed to link to it directly, but the patch itself can also be found as an attachment in this GitHub thread post: https://github.com/dosemu2/dosemu2/issues/1703#issuecomment-1180907421
SHA-256 checksum and VirusTotal scan result (still, use at your own risk!): https://www.virustotal.com/gui/file/053919a97dad64b8134c4f2b6765f671a98ff52a977e9775686225c2fa6875ad
Two common implementations:
1) Treat CR0.PE=0 as W=1 for CS.
2) Have an actual internal CS.ar.W bit, which the CS descriptor from memory lacks.
Both have their pros and cons.
At sandpile it’s visible in two places: RESET/INIT state, and SMM entry state – look for “writeable”.
And yes, clarification for 1) above: EFLAGS.VM=1 – same W=1 effect, of course.
At least on my Ryzen machine, V86 mode does not seem to need any kind of special handling because the CPU puts 0xF3 into all segment register attributes when entering V86 mode (the AMD IRET documentation lies about this from what I can tell). But maybe some CPUs work differently?
There is one other thing that bugs me about the AMD documentation which is that AMD (ARM Vol 2 section 14.1.3, Processor Initialization State) suggests that after reset, CS attributes are set to “Executable/Readable Code Segment”. I don’t have any easy way to check if the docs lie or if the behavior is really different from Intel CPUs.
Just recently in a discussion on HN about fantastic nand2mario 386 clone progress rep_lodsb mentioned https://news.ycombinator.com/item?id=47818225 observing on his 286
>executing three pre-decoded instructions without a jump after setting PE causes a triple fault for some reason
and
> Intel documented that the very first instruction after enabling protected mode had to be an “intra-segment” (not inter-segment) jump, to flush the prefetch queue.
contemporary 286-Pentium bioses I looked at all do this, but I just so happened to be disassembling Atom N270 945GSE Mini-ITX industrial board Phoenix bios from 2010 that does not
seg000:FD56 Unreal_FFD56 proc near
seg000:FD56 lgdt fword ptr cs:[bx]
seg000:FD5A mov eax, cr0
seg000:FD5D or al, 1
seg000:FD5F mov cr0, eax
seg000:FD62 jmp short $+2
seg000:FD64
seg000:FD64 loc_FFD64:
seg000:FD64 mov ax, 8
seg000:FD67 mov ds, ax
seg000:FD69 mov es, ax
seg000:FD6B mov eax, cr0
seg000:FD6E and al, 0FEh
seg000:FD70 mov cr0, eax
seg000:FD73 jmp short $+2
seg000:FD75
seg000:FD75 loc_FFD75:
seg000:FD75 xor ax, ax
seg000:FD77 mov ds, ax
seg000:FD79 mov es, ax
seg000:FD7B retn
just short jumps working fine on Atom. This surprised me quite a bit. Now after reading your post am I understanding this right that this works because no memory (especially cs indexed) is being touched?
My understanding is that on the Pentium and later, an intra-segment jump is pointless because the mov to CR0 already flushed things (and a jump might not cause a flush thanks to the branch predictor).
But yes, I am 99% certain that on the 386/486, the point of the jump was to flush the pipeline and make sure that there aren’t any partially executed instructions that access memory, although there could be other instructions that behave differently in real vs protected mode (except real mode and Ring-0 protected mode is not that different). I believe this is really much more of an issue when enabling/disabling paging. If the instructions immediately following the enabling of protected mode just do some arithmetic on registers, then flushing or not probably won’t make a difference because the results will be identical in real and protected mode anyway.
Part of it was no doubt future proofing — when the 386 was done, no one knew how deep the pipeline on some upcoming compatible CPU might be. The flush induced by a JMP was safe and avoided potential future problems. Intel knew that some instructions needed to be re-decoded/re-executed in protected mode and it was far simpler and safer to tell developers “always do the jump” than providing some super detailed documentation about exactly what’s safe and what’s not. It’s also a case where Intel would not want to document the internals in too much details because that would tie their hands in the future.
The Comanche game does not do any jumps and it works on 386/486 machines just fine.
For reference, the MR. BIOS on my Alaris Cougar board always follows toggling the PE bit by either ‘jmp short $+2’ or another short jump, in one case a near return. This BIOS works with 386/486 class CPUs as well as Pentium OverDrive.
Whether inter-segment (far) jumps are necessary in protected mode or not is a whole another kettle of fish. It is pretty clear that for short sequences, including enabling unreal mode, they’re not. But for establishing a protected-mode environment which either uses 32-bit code and/or can handle interrupts/exceptions, a far jump is an absolute requirement.
The Pentium situation is interesting, because it means the state of the CPU immediately after a RESET isn’t the same as after doing a MOV CR0,CR0||-2 to leave protected mode… a little bit reminiscent of how an 80286 can’t return to real mode without a reset.
So a Pentium really has two kinds of real mode: two kinds of real mode which function identically, except only one can get you into the next thing; and then it has two kinds of protected mode, one where the current value of CS: is writeable, and one where it never can be.
(And yes, my own protected mode code I wrote tended to use the CS: that was already setup since all it was doing was moving a bit of data around, and doing a far jump made the operation that was already slow even slower. I never bothered thinking about writing with CS).
Yes, the Pentium behavior is indeed interesting because the reset state of CS attributes persists until it’s changed, but once it is changed, there’s no way to get back without resetting the CPU again (either a hard reset or INIT). Not really typical for x86 CPUs.
Actually I have not explored this but there could be a way to recover the original CS attributes using SMM.
A lot of this goes back to the fact that returning from protected mode was never properly architected on the 386. It was clearly a late change made with very minimal microcode modifications, and it didn’t take people long to figure out unreal mode and then Intel was stuck with it. Old (1985) 386 docs said that CR0.PE can be reset but clearing it “will have unpredictable effects”. It is possible that the only hardware change Intel made was reloading the CS attributes on real mode far control transfers, because that was the one thing software could not do.
But wait… there is more! (With x86, there almost always is, isn’t there?)
Although sreg.{sel,base,lim,ar} are often referred to as the segment descriptor cache, that nomenclature is… well, keep reading. 🙂
The P5 actually had what was called the SDC – yep, you guessed it… a Segment Descriptor Cache. An actual 16-entry 2-way associative cache with LRU replacement policy, for recently loaded segment descriptors, operating in Protected Mode. The result was a hit latency of 2 cycles, and a miss latency of 11 cycles. The primary benefactor was 16-bit PM, though the SDC also operated in 32-bit PM.
The P6 shipped without this sort of thing… which contributed to its not-so-great performance in the presence of frequent segment descriptor loads… until that got addressed later on, in the P2 proliferation (“sreg renaming”).
The P5 SDC was visible via data breakpoint #DBs (not triggering for cached entries) and LGDT (invalidated it – the a recommendation that debuggers issue LGDT) and performance counter events (accesses vs hits), but in general it tried to maintain consistency with the L1 data cache.
So yeah… x86 segmentation… what’s not to like, eh? 🙂
And yes, this also impacts CPL – i.e. using SS.ar.PL versus using a dedicated CPL “register”. Both of those implementations have been used in practice.
And hey, while we’re at it… TR.ar.V is another interesting case.
It should act like a valid bit, and cause a #TS(0) exception on all implicit TSS accesses (stack switch, task switch, TSS32.IOPB, or TSS32.IRB).
Most processors don’t implement that behavior though.
Probably a stupid question, but I noticed that all the pentiums that succeeded were Overdrives, and the two that failed were “real” Pentiums. I know they had to do some design changes to let a pentium-class processor work in a 486 mobo…could that possibly account for any of the differences (vs the BIOS theory)?
That was my first thought too, but no — check the Intel Thor board results. A regular 133 MHz Pentium (P54C) worked too.
Also note that yes, the PODP5V (P24T) aka Socket 2/3 Pentium OverDrive is an odd duck, the Socket 5/7 OverDrives were just regular P54C/P55C Pentiums with a voltage regulator added.
I’m fairly convinced that newer BIOSes tend to run 32-bit code during POST and that forces them to reload CS in protected mode. Once that’s done, then as the article describes, it is no longer possible to return to the initial CS attribute state as it was after reset, and CS will never be writable in protected mode.
Yes, I know about the Pentium SDC… probably a Windows 3.x optimization? I can imagine that although it worked in 32-bit mode too, it didn’t do that much because there’s no need for frequent segment register reloading in 32-bit code. But in 16-bit code definitely yes. Far calls everywhere, DS/ES reloaded all the time.
With the PPro, Intel’s big miscalculation was that by ’96 everyone would ditch the horrible old 16-bit code in order to preserve their own sanity. But then Windows 9x said NOPE, you have to wait another five years.
I’m aware that these two implementations exist, but do you know how they can be distinguished in software?
Is there anywhere in the SMM structure this flag bit would be hiding? I can’t find anything.
Dear Michal, thank you for taking your time to investigate good ol’ Comanche game based on my poor memory and single comment 🙂
So it wasn’t unreal mode usage just strange entry into protected mode an issue with first release. Good to know after all those years. As legacy BIOSes grew in size, exploiting protected mode before any operating system code has been run become a must. At the end we’ve got over complicated UEFI firmware but that is a completely different story. Once again, thanks for your investigation.
Right, not unreal mode, although the original Comanche was very picky about the DOS configuration (HIMEM/EMM386 etc) in a way very similar to unreal-mode software. On top of that their code might not run on Pentium systems at all. My guess is that the game worked reliably on a 386/486 with the right DOS config. On a Pentium machine not so much, because if it worked once then it crashed the second time.
Indeed I have been running the game originally on 486 (DX2 probably or so). Anyway it is always great to debunk another myth from DOS times.
An explicit W bit for CS might appear in the SMM SSM… or… it might not.
If dumped by SMI, then RSM would still need to reconcile it with PE/VM.
So it’s not actually worth dumping it in the first place.
In the long run the proper approach is of course to migrate the legacy complexity from HW/microcode to SW/emulators – you know… x86-S done right, and brought to its natural conclusion, rather than resisted by the cargo cult crowd… 😉
A rather brief check on a machine that seems to have the “CS is still writeable after the BIOS boots” shows no difference in my attempt to dump the SMM SSM versus after entering protected mode, doing a far jump, and then exiting to real mode.
So this must be some bit that does not get covered by SMM. The real question is why it exists at all. After all, when in real mode, it’s not necessary, and in protected mode it’s not necessary either.
Intel/AMD could really help that by opening up their validation suites. From my own experience with x86 emulator work I know that there are very, very many dark corners in the x86 architecture and nailing down the exact behavior is real difficult. There are gigantic holes in the official documentation, like flag state after instruction execution (yes, I know in practice there are several different ways microarchitectures do it). Every time the documentation says “undefined” there’s a big fat red question mark, because in practice undefined does not even remotely mean unpredictable, and if it’s predictable then software may well intentionally or unintentionally rely on it.
So what are the CS attributes saved in the SSM? Can you tell? And do you have code to test SMM on regular hardware?
Also, is CS actually writable in SMM?
The hypothetical CS.ar.W bit would exist precisely so that it could be set upon entering real (and V86) mode and cleared when exiting. Because the CPU somehow needs to know that CS is writable. The 386/486 simply checks the actual attributes. The Pentium and later does not, which means some other mechanism must be used. The CPU could check the current mode when writing through CS, or it could have a CS.ar.W bit that is set when entering real/V86 mode. I imagine this would be faster and simpler because all segment registers could be treated symmetrically.
As Christian says, the CS.ar.W bit would not need to be saved because it should precisely reflect the current PE/VM state (something like ‘NOT (PE XOR VM)’, assuming that VM can’t be set in real mode). So mode transitions can always derive CS.ar.W based on other state.
> Also, is CS actually writable in SMM?
Historically… yes… by virtue of PE=VM=0 after entry.
Nowadays… no… because of Intel’s SMM Protected Mode feature, which – as its name suggests – causes SMM entry to go straight to Protected Mode, rather than Real Mode. (This got added in the HSW/BDW generation, iirc.)
Recall that classic SMM has a MCE gap, because SMM entry clears CR4.MCE and does not establish a SMM-based #MC handler. As a result, any #MC that occurs after SMM entry but before the SMM handler can establish its own #MC handler and enable CR4.MCE again, leads to shutdown – a rather undesirable outcome for RAS.
SMM_MCA_CAP.SMM_PROT_MODE (MSR 0x17D bit 54) => it’s supported
SMM_PROT_MODE_BASE.ENABLE (MSR 0x9D bit 0) => it’s enabled
SMM_PROT_MODE_BASE.SMMSEG_PA (bits 31:8) => SMMSEG phys addr
SMMSEG structure:
dword +0 = reserved
dword +4 = enabled features (bit 0 = enabled (must be 1), bit 1 = CR4.MCE)
dword +8 = GDTR.limit (using lower 16 bits; upper 16 bits are reserved)
dword +12 = GDTR.base offset from SMBASE
dword +16 = CS.selector (if 0x0000, then 0x0008 will be used – i.e. no NULL)
dword +20 = reserved
dword +24 = ESP offset from SMBASE
dword +28 = reserved
dword +32 = IDTR.limit (using lower 16 bits; upper 16 bits are reserved)
dword +36 = IDTR.base offset from SMBASE
It’s fairly obvious where in memory SMMSEG should vs shouldn’t go. 😎
Now, I don’t know why the above information is hard to find in public docs. Perhaps I should add it to sandpile’s SMM page. Including a look at what AMD does, of course.
Tested it on a Pentium MMX class system (old Compaq laptop), definitely pre-bdw/hsw. The only way to trigger it is to use real mode APM to invoke SMM (and then immediately exit it), and inspect the memory range where it stores the SMM segment.
It is possible that performing a suspend and resume is actually resetting the CPU, but supposedly it doesn’t (it just slows the clock rate down a lot). Beyond that I can’t say.
What model is that? I might have the technical reference for it, I’d just be curious if it has any detail.
I suspect laptop OEMs were much more eager to use SMM as soon as they could because they had all those goodies like brightness/volume control and power management, which almost certainly needed SMM in order to be able to work in an OS-independent way. Especially in the era before ACPI. The 386SL/486SL processors had SMM for a loooong time.
Fwiw, SMM-based power management does not actually work for STD/STR, at all, period.
https://www.sandpile.org/x86/legacy.htm
“[T]he processor is subject to the limitations described in SDM Vol 1 pre-075, section D.3.5.”
https://kib.kiev.ua/x86docs/Intel/SDMs/253665-074.pdf
It’s too bad Intel decided to remove D.3.5. altogether in SDM 075, instead of moving it to a legacy section of the manual. Oh well.
PS: Interestingly enough… this exact problem did come up in my interview with Google back in early 2007… and my response may have helped in actually getting hired… LOL
The only reason I used that laptop is because it’s easy to get SMM on it and it leaves the SSM accessible from main memory. I’ll check the model next time I have access to it.
A rather glaring problem is I have no idea if the CPU goes through a RESET or not. Is there a way to tell?
Slightly remember this game anouncement from in an computer broschure back then. But missing the point here, why bother with this dinosaur at all when you can run it in an emulator (or a virtual box/machine running windows 95!?) or it tun in DOS back time?
Yes, there is. See my previous response. Read section D.3.5.
The point is not to play the game, the point is to understand why it had trouble running on real PCs even back in the 1990s.
On the topic of the Pentium Pro and it not being optimized for frequent segment register reloading. Unauthorized Windows 95 suggests there was a lot of pre-release genuine confusion about whether “Chicago” would just be an evolution of WfW 3.11 with more code in VxDs and fewer calls to DOS/BIOS in V86 mode (as the author ultimately shows), whether it would just build on top of Win32s or if it would have a 100% 32-bit kernel completely displacing krnl386/user/gdi, whether even better, it would be a true 32-bit OS that “just uses DOS as a bootloader,” a la loadlin, or whether it was something else.
I don’t know how legitimate these “misunderstandings” were and if Intel felt misled about the future direction of Windows and the extent to which that fed into PPro architectural decisions or not?
Microsoft’s entire direction circa 1993 (when the Pentium was released, and the Pentium Pro was being designed) was that NT was the future – and would be used on not just Intel, but PowerPC, MIPS, Alpha, SPARC, i860, Clipper, Motorola 88000, PA-RISC, and who knows what else. The intention was a pure 32-bit design.
Windows 3.1 was supposed to be the last release of that generation of Windows… but NT just took up way too much memory, and Windows for Workgroups 3.11 came out also in 1993 as a stopgap, with a large part of it being an intention to be a thinner client for NT servers in order to promote NT sales (which worked) and get people to migrate off of LAN Manager Server.
Of course, making WFW3.11 usable meant dealing with the endless headaches LAN Manager tended to introduce like running out of conventional memory or being unpleasantly slow inside Windows 3.1, which had spawned numerous third-party solutions like Netroom. So they just cranked our WFW3.11 with the obvious fixes (32-bit file access via VxD, cacheing via VxD, and the entire networking stack moved into VxDs). Problem solved, stopgap done.
It became pretty obvious that 16MB machines were not going to be the norm by 1995. Nobody had an operating system that worked well in 4MB with a GUI and a network client other than Microsoft. What is impressive is that Windows 95 had as much pure 32-bit code as it did – it was far less mixed than OS/2 3.0 or even 4.0 were, which were still both sporting 16-bit storage, HPFS, TCP/IP, and SMB clients. From looking at the betas, some of this came fairly late in the game to Win95.
It’s also painfully obvious the incremental approach was a better one for long-term operating system development (real mode Windows -> Win/386 VM -> Win 3.0 protected mode Windows + VM -> Win 3.1’s 32-bit disk access -> WFW 3.11’s nearly all-protected mode runtime -> Windows 95). NT and OS/2 both were a lot of broken promises that shipped late, although the question always remains in my mind of why the Microsoft OS/2 2.0 beta was solid and pretty much complete, and then IBM spent 2 years to actually release it.
I don’t know how much Intel was misled about Win9x vs how much Intel and the entire industry were misled about Windows NT.
I mean, Windows NT was actually out in 1993. So was OS/2 2.x. But Microsoft never said, “hey guys, let’s postpone a real 32-bit OS by five years or so”. Question is, did Microsoft even want that to happen? I don’t think it was Microsoft’s plan to delay NT by so many years. More likely OEMs were pushing hard for Win9x and Microsoft followed the money.
So in the end I’m not at all sure how much Intel was misled vs how much Microsoft lost control of the OS landscape and things didn’t go according to plan.
I completely agree with the first part, but not the last. Win9x was an evolutionary cul-de-sac. It was a pile of unstable garbage that barely worked. I used Win9x back in the day and it was not a good experience.
Microsoft knew that Win9x was never going to do SMP and it was never going to run on a non-x86 CPU. It was always a stopgap product, unable to evolve further. Microsoft simply could not afford to not have NT.
Also if IBM spent 2 years to release MS OS/2 betas, that would imply the OS was done in March 1990. It absolutely was not. Most of the delay was caused by the simple fact that if you have a complex product and replace one team of developers with another, it’s going to take time.
I think the main reason that NT was near-useless in 1993 (and in 1995) was because of the steep memory requirements; the current era is somewhat similar, but from 1993-1995 we got to experience memory prices simply not budging for two years… an 8MB machine was still “standard” in 1995, with budget being 4MB. Either kind of machine was not going to run NT in any kind of useful way.
OS/2 2.x sat somewhere between Windows 95 and NT in terms of memory use (or waste); if IBM had cut the Workplace Shell in 1992, OS/2 2.x would have probably had a lot more market penetration. The MS OS/2 2.0 beta works pretty well in 4MB; 2.0 GA is unusable.
But I feel like going to the trouble to develop and release Win32s shows that Microsoft had a transition period planned all along, to have a slimmed down Windows built on the existing foundation but that is ABI-compatible with most NT user binaries. But probably not for it to still be a flagship product for nine more years.
More perplexing to me is the decision to release Windows ME at such a late date.
The funny thing is that I have a feeling that WINMEM32 (this was before Win32s) and OS/2 2.0 was designed to target different markets.
@Josh Rodd. The thing is that even if Windows NT have run well on standard hardware (which didn’t). You have to look at the market, 1995 and I think still 1998 There was the need to run DOS software which NT didn’t do well. My first NT was 3.51 so I don’t know how well Windows NT 3.1 run Windows 3.x software. And then it’s price, how many companies will pay for a new machine with Windows NT with new software versions (if they had it), for a sizeable part of their employees.
In my experience Windows NT was considered an operating system for Servers and Workstations till Windows 2000, after all not a lot of games (or software that needed special drivers like video capture devices, CD burners, radio tuners, radio modems, negative photo scanners, etc.) run in Windows NT before XP.
Also Windows 95, 98, ME was the operating system for OEMs, to be preinstalled, I don’t think that NT was preinstalled (at least not in non server PCs) till XP.
About OS/2 I wonder how much time IBM spent in QA. IBM had the fame at the time of testing well their software (even Bill Gates said in one of the first PC Magazines (I think) that was IBM who tested the first version of DOS and made Microsoft make a better product.
After all that Betas that Michal comments have to had feedback for IBM.
You’ll be pleased to know that the p4 depot that holds sandpile
still lives in a W2K VM which turns 19 years old next week. And
yes, that installation saw another 7 years of service on a variety
of physical machines before then. Don’t fix what ain’t broken…
Alas, a change of hypervisor is coming my way, sooner or later.
Oh, and as for getting back on-topic… mode switches… but wait,
there is more. Stay tuned… I’ll need to write things up a bit first.
Yes, OS/2 Warp really did lower the memory requirements and was barely usable with 4MB and quite OK with 8MB. The WPS was in retrospect a very, very questionable decision.
Another factor was that in the mid-1990s laptops were getting more widespread. While OS/2 supported laptops quite well (especially IBM’s own), NT barely even tried before Windows 2000, so Win9x was the only practical option. I am not sure what drove that strategy, presumably RAM sizes had something to do with it but there was probably more to it.
Yep, there definitely was a strategy. Win32s was part of it, and there was even Phar Lap TNT for console applications. But I’m pretty sure the transition period was meant to be way shorter.
I recall that there were all kinds of issues caused by the fact that Win9x and NT were built on different code bases, so not only were the capabilities different but the API behavior was also not always the same. Games were one big area where Win9x was significantly more capable than NT.
Having to develop separate display drivers for Win9x and NT was no fun for OEMs either I’m sure.
Speaking of inconsistent APIs across the two platforms, Kermit 95 (open sourced in 2011) is one of the few Win32 terminal emulators I know of that used the Win32 console rather than just reimplementing the display using a GUI (like PuTTY). They had to deal with all kinds of workarounds for Win9x, since the Win32 console there still tied into the MS-DOS support. Eventually they threw in the towel and did their own terminal in a GUI as well.
PCMCIA was probably a big thing keeping NT off laptops, right? Since the driver architecture wasn’t designed for devices to come and go, and the possibility to fall back to running the BIOS in V86 mode as an escape hatch when needed was probably useful for that.