A Brief History of Unreal Mode

After a run-in with a particularly crazy manifestation of unreal mode (Flat Assembler, or fasm), I decided to dig deeper into the history of this undocumented yet very widely used feature of 32-bit x86 processors.

For the purposes of this discussion, unreal mode is a variant of the x86 real mode with non-standard segment limits and/or attributes, different from the processor state at reset. To recap, real mode on the 286 and later CPUs has much more in common with protected mode than with the real (and only) mode of the 8086. Notably, undefined opcodes raise exceptions, segment limit overruns cause general protection or stack faults, and (on the 386 and later) 32-bit registers and 32-bit addressing can be used—subject to limit checks.

The origins of unreal mode are shrouded in the mists of time. But enough is known that certain outlines are quite clearly defined. Let’s present a rough timeline of unreal mode.

• 1985—the Intel 386 became available in silicon. Like the 286, the 386 was not designed to switch from protected back to real mode. Intel’s idea was presumably that users should either leave the 386 in real mode, reset the CPU to get back to it (like a 286), or use the 386’s new V86 mode. This may sound like a crazy claim, but it’s not. In the October 15, 1991 issue of PC Magazine, page 436 (“Stepping Up”), Jeff Prosise wrote quite clearly: “The very first 80386 chips that rolled off the line (A-step chips) could not be switched from protected to real mode.” Indeed the confidential Intel iAPX 386 Architecture Specification, revision 1.8 from June 1985, had only this to say on the subject of switching from protected back to real mode: “Disabling protection by loading a 0 into the PE bit in CR0 (assuming the current value is 1) will have unpredictable effects.” In other words, the original 386 design tracked the 286: in real mode, segment limits and attributes retain their initial reset values and cannot be changed.
• 1985?—someone or someones (Microsoft? Compaq?) convinced Intel to relent and allow switching from protected to real mode by clearing the PE bit in control register CR0; this was no doubt inspired by the headaches the 286 was causing. It is unclear if there was actual silicon change or only a documentation change. While a transition to V86 mode always carefully loads all segment registers, making any funny business impossible, the act of clearing the PE bit in CR0 does almost nothing. It affects future segment register loads in real mode, it affects interrupt and exception dispatching, but it has near zero impact on the immediate CPU state. Crucially, it does not change the current segment limits or attributes, which allows the currently executing code to continue running. Note that the I/O Permission Bitmap (applicable in protected mode) was an even later addition to the 386 design.
• October 1985—The 386 datasheet (80386 High Performance Microprocessor with Integrated Memory Management, Intel order no. 231630-001), says in section 2.3.5: “In Real Address Mode, only the base address is updated directly (by shifting the selector value four bits to the left) [when loading segment registers], since the segment maximum limit and attributes are fixed in Real Mode.” In section 2.3.4, the document says: “In Real Address Mode, the maximum segment size is fixed at 64 Kbytes.” In other words, this document both explains why unreal mode works and claims that it does not exist. The datasheet also declares in section 2.3.6 that switching back to real mode is possible, but no additional detail is given: “If PE [bit in CR0 register] is reset, the processor operates again in Real Mode. PE may be set by loading MSW or CR0. PE can be reset only by a load into CR0.”
• January 8, 1986—Intel writes a confidential memo titled Returning to real mode on the 80386. This memo explains not only how to set up a canonical real-mode environment on return from real mode, but also why it is necessary: “While operating in REAL mode, the 80386 uses exactly the same memory management functions as in protected mode. However, when the part resets into real mode the values loaded into the descriptors appear as if they were 8086 style segments. In real mode, when a segment register is loaded, only the base field is changed, in particular the value placed into the base is selector*16. Since only the base is changed, it is necessary to set the access rights while still in protected mode.” The memo proceeds to explain the required segment attribute setup, information which appeared in the official 1986 PRM (Programmer’s Reference Manual) for the 80386. What the memo also says, and the PRM does not, is that the code segment (CS) cannot be made writable in protected mode, but “an architectural feature reloads real mode attributes into the CS descriptor during real mode far jumps”. This memo explains real-mode operation of the 386 far better than the official documentation, and serves unreal mode on a silver platter to anyone willing to experiment even just a little.
• April 1986—The updated 386 datasheet, Intel order no. 231630-002, offers a tantalizing hint of unreal mode in section 2.3.6: “Resetting the PE bit [in CR0] is typically part of a longer instruction sequence needed for proper transition from Protected Mode to Real Mode.” The quoted text was not present in the original October 1985 edition of the datasheet, and presumably refers to the information first published in the Jan ’86 memo.
• 1986—The 1986 Intel 80386 Programmer’s Reference Manual says that to switch from protected to real mode, the programmer must among other things transfer control to a code segment with 64K limit, and load SS, DS, ES, FS, and GS segment registers with selectors that have a 64K limit, are byte granular, expand-up, writable, and present. Only then can CR0.PE be cleared. In other words, Intel was practically begging programmers to see what happens when they don’t do that. The PRM publishes the how-to information from the Jan ’86 memo, but none of the background explanation.
• 1987—In the 1987 edition of the 80386 System Software Writer’s Guide (Intel order no. 231499-001), in section 9.2 on page 9-3 (4th para), Intel hints: “Because, except for base address, descriptor register values cannot be loaded in real mode, 8086-compatible  attributes must be loaded into the data segment descriptor registers before switching to real mode.”
• January 15, 1988—Phoenix 386 BIOS with this date employs unreal mode to emulate some aspects of the 286 LOADALL instruction, by returning to real mode with segment bases not corresponding to the segment register value. The date is tentative; this was the oldest 386 BIOS using this technique available for analysis. More below.
• August 4, 1988—Microsoft adds 386 “Big Mode” or “Real Big Mode” code to implement 386 extended memory moves in HIMEM.SYS 2.04. Microsoft’s method is quite clever, using an exception handler to recover from anyone resetting the segment limits behind HIMEM’s back. Extended memory copying can thus be run with interrupts enabled, with no impact on interrupt latency (that was a significant problem on 286 CPUs). The term “unreal mode” is not used. This change can be dated exactly, thanks to comments in HIMEM.ASM. At the same time, LOADALL support was added to the HIMEM 286 extended memory move.
• August 15, 1988—Microsoft makes an archive of the HIMEM.SYS 2.04 source code, with LOADALL usage removed but unreal mode code left in place. It is unclear how widely this code was distributed at the time. In 1992, the source archive from 1988 was published on Softlib as part of XMS.EXE, which contains the new XMS 3.0 specification. In the late 1980s and early 1990s, Microsoft published the HIMEM.SYS source code together with the XMS 2.0 specification (the first published version was 2.01, before the unreal mode code was added).
• February 7, 1989—Microsoft publishes HIMEM.SYS version 2.06 source code as XMS20.ARC; unlike the previous release, this one produces an exact match of the official binary, because the source code includes 386 Real Big Mode and 286 LOADALL (yes, really!) support.
• March 21, 1989—Microsoft publishes an updated HIMEM.SYS version 2.06 source archive as S12023.EXE, formerly XMS20.ARC. This one survived to the present. Although the 1989 source code releases were public, they seem to have gone more or less completely unnoticed.
• 1989—In the 386 SX Microprocessor Programmer’s Reference Manual (order no. 140331-001), section 14.5 (Switching Back to Real-Address Mode), Intel drops another big hint. To the paragraph describing how to load segment registers when transitioning from protected back to real mode, the following sentence is added: “Note that if the segment registers are not reloaded, execution continues using the descriptors loaded during protected mode.” The same text is also added to the updated 1990 edition of the 386 DX PRM, and appears in the 1990 i486 PRM and all subsequent Intel programming manuals.
• November 1989—In the November/December issue of Programmer’s Journal, Thomas Roden (a software engineer at AST Research, one of the big PC OEMs) publishes an article titled Four Gigabytes in Real Mode (page 89). This is the oldest known public description of unreal mode, although the term “unreal mode” is never used. Unreal mode is described, including the exception handler technique to handle other code resetting the segment limits. Thanks to his position at AST Research, Mr. Roden was already able to confirm that unreal mode works not only on the 386DX and SX, but also on the then-brand-new Intel 80486. The article mentions that it may be possible to set the D-bit of the code segment to run in real mode with 32-bit CS. There is no explicit mention or even a hint that Mr. Roden was aware that HIMEM.SYS was already using unreal mode.
• January 1990—In the German c’t magazine, Harald Albrecht publishes an article titled Grenzenlos: Vier Gigabyte im Real Mode des 80386 adressieren (Boundless: Addressing four gigabytes in 80386’s real mode). Mr. Albrecht suggests that the 386 documentation directly challenges programmers to not follow the prescribed return-to-real-mode method exactly. The article is notable for documenting that in real mode, a far jump partially changes CS attributes, and also that after a switch to/from protected mode, a near jump is sufficient to flush the prefetch queue (a far jump is not necessary). Mr. Albrecht does not name the non-standard real mode but presents a TSR which implements a fast INT 15h/87h block move routine which uses 4GB selector limits, runs with interrupts enabled, and does not disable the A20 gate when done. The author incorrectly claims that (paraphrasing) no one needs the A20 gate disabled, something that others learned the hard way not to be true. Mr. Albrecht also notes that the segment limit extension technique is unusable in V86 mode, and that 32-bit code or stack segments are not practical due to corruption of the high word of EIP/ESP. The article was probably written at about the same time as Mr. Roden’s PJ article; it may be an independent discovery, although Mr. Albrecht’s earlier article in the November 1989 issue of c’t (Odyssee im Adressraum) explicitly mentions the HIMEM.SYS source code provided in Microsoft’s XMS Developer’s Kit.
• July 1990—In the July 1990 issue of Dr. Dobb’s Journal, Al Williams publishes an article titled DOS + 386 = 4 Gigabytes! (page 62), again describing unreal mode. The article is by all appearances another independent discovery of unreal mode, as there are no hints Mr. Williams was aware of either HIMEM.SYS or the Nov/Dec ’89 PJ article. Again, the term unreal mode is not used.
• October 1990—In the October 1990 issue of DDJ (page 12), a reader letter from Thomas Roden appears, pointing out minor problems with the July ’90 DDJ article about unreal mode. It is clear from the letter that Mr. Roden had access to an ICE (In-Circuit Emulator), which no doubt greatly eased unreal mode experimentation. An editor’s note identifies Mr. Roden as the author of the Nov/Dec ’89 PJ article.
• January 1991—On page 176 of the 1/91 issue of German DOS International magazine, an article titled Vier GByte im Real Mode unter MS-DOS (Four Gigabytes in Real Mode under MS-DOS) by Martin Althaus explains in detail how to change segment limits to enable full 4GB addressing in real mode. The necessity of enabling the A20 gate is discussed, and complete example code is presented; the article also notes that the technique does not work when EMM386 (or V86 mode in general) is in use. The article does not explore what might happen if segment attributes in real mode are changed in other ways, beyond setting the G bit. The author does not claim to have invented the unnamed technique, but also does not give any references to earlier publications.
• March 1991—In Chapter 18 of Assembly Language Programming for the Intel 80XXX Family, William B. Giles describes in detail (page 676 and following) how to program 386 real-mode selectors such that they cover the entire 4GB address space. Example code is also presented. The code samples are dated September 4, 1990. The ’89 PJ article is referenced as the source of information on expanding real-mode selector limits.
  
• March 1991—In Chapter 18 of DOS 5: A Developer’s Guide, Al Williams again describes unreal mode, with code examples. Note that the exact publication date is unclear; March, August, September, and October 1991 are given. A 1992 review of the book discusses Chapter 18 in some detail, together with referencing the ’90 DDJ and ’89 PJ articles.
• 1991—In 80×86 Architecture & Programming Volume II: Architecture Reference (page 72), Rakesh K. Agarwal (a former member of the 80386 design team) clearly explains what all the official Intel documentation carefully doesn’t: “When [switching from protected to real mode] is done, the only action taken by the 80×86 is to clear the PE flag. In particular, the current state of the descriptor registers is left undisturbed.” And further: “Such a pseudo REAL, or UNREAL, execution mode can cause a REAL mode system to crash, unless used very carefully” (capitalization in original). This may be the first published use of the term “unreal mode”.
• April 16, 1992—Origin releases Ultima VII: The Black Gate,  a game using a custom DOS extender appropriately named “Voodoo”. Ultima VII is a proof that unreal mode is a terrible idea for DOS applications; the game requires a significant amount of free conventional memory, but is incompatible with 386-based DOS extenders, as well as with any DOS-compatible advanced operating systems.
• 1992—German DOS Extra Nr. 20 (supplement of the DOS International magazine) publishes the HugeRealMode driver, enabling larger than 64K code segments in unreal mode, with restrictions (more below).
• June 29, 1993—In his Tutor column starting on page 302, Jeff Prosise wrote a section on “accessing 4GB from real mode” without using LOADALL. “Recently, another method of accessing 4GB […] was brought to my attention by a reader on PC MagNet.” The text gives a brief overview of unreal mode and refers to Chapter 18 of DOS 5: A Developer’s Guide by Al Williams (also the author of the 1990 DDJ article).
• January 1994—Chapter 8 (Extended Memory Access from Real Mode) of Geoff Chappell’s DOS Internals provides a very detailed description of unreal mode and HIMEM’s use of it. There is talk of “unreal” segment properties (quotes in original, page 356), but the chapter is about “real-mode Flat Memory Model”, and “Big Real Mode” is also gets a mention. The chapter also provides a detailed treatment of 286 and 386 LOADALL. It is probably the best description of the inner workings of HIMEM.SYS. Curiously, in Chapter 12 (page 443) Mr. Chappell grumbles that only the outdated HIMEM.SYS 2.01 source code is available on CompuServe (apparently uploaded by Steve Gibson of InfoWorld). Clearly, Microsoft’s 1989 and 1992 HIMEM.SYS source code releases did a very good job of flying under the radar when they escaped Mr. Chappell’s laser-like attention.
• September 21, 1994—IBM files patent application 309,862 titled Method for expanding addressable memory range in real-mode processing to facilitate loading of large programs into high memory. U.S. patent 5,642,491 was granted on June 24, 1997. The patent describes unreal mode, and mentions Chappell’s DOS Internals, pages 355-385. The patent makes strange references to a “64-kbyte wrapping feature associated with true 8086 real mode”; that might refer to stack wrapping when PUSH and POP instruction are used, but no such wrapping is used for data segments. The patent suggests using unreal mode during operating system initialization only, not at run-time.
• 1993-1995—Unreal mode becomes a thing and every up-and-coming programmer writes a utility or library to use it. It is popular among demo coders, who are dismayed when EMM386 use is mandated for Assembly ’95, interfering with unreal mode.
• November 11, 1994—Italian programmer Daniele Paccaloni publishes UNREAL.EXE, a utility to enable unreal mode on a 386 or later CPU, and UNREAL.DOC, a document explaining how unreal mode works. In a continuation of the recurring theme, Mr. Paccaloni appears to have been completely unaware of the fact that unreal mode had been in use for years, or the growing body of literature about it. It appears that Mr. Paccaloni also independently re-invented the term “unreal mode”.
• September 1995—Robert R. Collins writes an article (NB: The exact date is uncertain, Sep ’95 is a guess based on source file timestamps) titled Descriptor Cache Register Anomalies; this is probably the definitive description of unreal mode. Mr. Collins thoroughly explored the CPU behavior using LOADALL for 386 CPUs, and also using SMM together with an ICE on 486, Pentium, and Pentium Pro CPUs. Mr. Collins showed that real mode is much closer to protected mode than one would be led to believe, and most protections are in fact in place. Mr. Collins also found that CS register attributes are handled differently on different CPUs generations. On older processors, far jumps change the CS attributes as described in the Jan ’86 memo, while newer CPUs preserve the CS attributes but ignore them in real mode.
• February 1996—In the book Protected Mode Software Architecture (chapter 5, page 61), Tom Shanley purportedly describes “Big Real Mode”, yet makes the ridiculous claim that segment offsets are still restricted to 64K (while the segment base can be set to any 32-bit address before returning to real mode). Perhaps this is just shows that unreal mode is not well understood, despite numerous authors’ attempts to explain it.
• September 20, 1996—Phoenix and Intel publish the POST Memory Manager Specification version 1.0 (PMM specification), which mandates the use of “big real mode” (aka unreal mode). Big Real Mode is not explained in detail, presumably because readers already know how it works. The PMM specification is later adopted by the PXE and PCI firmware 3.0 standards. Intel thus manages the astonishing feat of specifying and mandating the use of unreal mode, while simultaneously denying that it exists. Although the original 1.0 PMM specification may be lost, the touched-up PMM 1.01 is available.
• August 1998—Dr. Dobb’s Journal does not publish an article titled The Segment Descriptor Cache by Robert R. Collins. The article says that “[u]nreal mode has been used commonly since it was discovered on the 80386. Unreal mode is so commonly used, in fact, that Intel has been forced to support this mode as part of legacy 80×86 behavior, though it’s never been documented.”
• August 2004—(This entry is only included for completeness.) In The Unabridged Pentium 4, Tom Shanley correctly describes unreal mode, saying that it “is sometimes referred to as Big Real Mode, Flat Real Mode, Real Big Mode and UnReal Mode” (italics in original). Unfortunately the author makes new, although minor, incorrect claims, namely that CS:IP and SS:SP are always used in real mode, precluding larger than 64K code and stack segments. Larger than 64K code and stack segments are indeed in practice unusable in unreal mode, but for different reasons.

But… Is Unreal Mode Any Good?

Yes and no. As a general-purpose programming technique it is unusable, because it absolutely cannot function in V86 mode. Transitions to V86 mode always force real-mode compatible segment limits and attributes. That means unreal mode cannot be used together with EMM386 or other DOS memory managers utilizing V86 mode. Unreal mode also cannot be used in the DOS boxes of 386 Enhanced Mode Windows 3.x, in the DOS boxes of OS/2 2.x, Windows NT, or Windows 9x. That is an extremely serious drawback.

The reason why the Voodoo memory manager in Origin’s Ultima VII games is unique is that no one else wanted to repeat the same mistake. The Ultima VII: The Black Isle credits capture the situation rather well:

On the other hand, when unreal mode can be used, it is very useful. HIMEM.SYS uses unreal mode to speed up extended memory access, and perhaps more importantly, preserve normal interrupt latency. Firmware can and does use unreal mode for accessing memory beyond 1 MB during initialization; it avoids switching between real and protected mode, and in firmware there is no danger of segment limits being reset.

Basic unreal mode (data segments with up to 4G limits, as well as data segments with non-standard base) has become a standard part of the x86 architecture.

Unreal LOADALL

At least as far back as January 1988, Phoenix 386 BIOS combined two undocumented favorites, using unreal mode to emulate the 286 LOADALL instruction. When handling an invalid opcode fault, the BIOS checks whether the faulting instruction is a 286 LOADALL (opcode 0Fh 05h); if so, it examines the provided LOADALL state buffer at address 800h and checks whether the segment bases for ES and/or DS are 1MB or higher. If so, the BIOS builds descriptors with the appropriate bases, loads them in protected mode, and returns to real mode with these “unreal” values in the segment descriptor cache (segment base other than 16 times segment value). The segment bases will get overwritten as soon as ES or DS gets reloaded in real mode, but that is entirely consistent with LOADALL behavior.

The LOADALL emulation is very limited, but it is sufficient to support the use of LOADALL in OS/2, and possibly others.

The detailed history of this method is unclear. What’s known is that the first 386 BIOS, shipped with the Compaq DeskPro 386 in 1986, did not use unreal mode. Compaq was clearly on good terms with Intel and got the secret memo describing the 386 LOADALL instruction. The Compaq 386 BIOS emulated 286 LOADALL rather accurately using the 386 LOADALL instruction, going at least as far back as September 1986 and continuing to use the same method at least until 1989 (it would no longer work on a 486 CPU).

IBM’s PS/2 Model 80 BIOS did not emulate LOADALL at all, and is therefore uninteresting.

There are also known to have been more or less no-name 386 systems sold around 1987 which used effectively a 286 BIOS and had no provisions to emulate LOADALL, nor did they use 386 instructions at all.

It is unknown when Phoenix first introduced unreal mode usage to emulate LOADALL, or if other vendors did that before Phoenix. Clone 386 BIOS images from before 1988 are rather difficult to find.

Unreal Mode Compatibility

Basic unreal mode with extended DS, ES, FS, and/or GS segment limits is widely compatible across all 32-bit x86 CPU implementations. After all that’s what is mandated by PMM and by extension, PXE and PCI 3.0 Firmware specifications, all standards created with a significant involvement of Intel.

Extending data segment limits in fact causes shockingly few problems, as evidenced by the fact that HIMEM.SYS does just that. On an 8086, referencing a word at offset 0FFFFh causes a wrap-around and the second byte is fetched from offset zero. The 286 is already incompatible with this behavior, causing a #GP fault. On a 386, any attempt to address beyond 64K likewise causes #GP faults, which are typically fatal. Therefore, functioning software does not do it and extending the segment limit has no visible impact.

Anything else gets more complicated, and interrupts are the number one enemy. For example, Intel CPUs typically support non-standard data segment attributes; it’s possible to make DS read-only, but that won’t be terribly useful when running real-mode software. It is likewise possible to set the segment limit smaller than 64K, but that won’t make 16-bit software happy.

It is possible to extend the CS limit up to 4GB and use 32-bit EIP in unreal mode. In practice that is not very usable because interrupts in real mode only save 16-bit IP and restore it on return. The German DOS Extra magazine presented a hair-raising scheme to solve the problem, using the CR3 register to store the high word of EIP so that it could be restored on return from interrupts, but this requires software to manually keep track of the high word. In practice, larger than 64K code segments are unusable in unreal mode because the complications very quickly outweigh any benefits unreal mode might have.

It is also possible to set the D bit in CS, changing the default operand size to 32-bit. This may reduce the size of unreal code if it is largely 32-bit (by obviating the need for overrides). Unfortunately, this again causes serious trouble with interrupts, because existing 16-bit code cannot run correctly with the D bit set. It is possible to switch to protected mode, clear the D bit, and execute the 16-bit handler every time an interrupt occurs, and in fact it’s exactly what fasm does (or at least did in some versions), but with so much mode switching and complexity, the advantages of unreal mode are rapidly lost.

Similar trouble strikes when attempting to use larger than 64K stack segments. It is possible, but again destroys compatibility with existing 16-bit real-mode code. The complications are such that one might as well use a regular DOS extender and skip all the unreal mode incompatibilities.

A larger problem is that the “advanced” aspects of unreal mode may not be implemented identically across CPU generations and vendors, precisely because they have never been documented. It is not specified anywhere exactly which segment attributes are ignored and which are honored in real mode. Protected mode, on the other hand, works consistently.

What Does It All Mean?

Unreal mode is almost certainly an accident of history, a side effect of the fact that the initial 386 design had no architected way of switching from protected mode back to real mode. Once the technique started being used, instead of clearly documenting how it works, Intel in its typical fashion documented only certain aspects of it, such that only programmers who already know about unreal mode find traces of it in the official documentation.

On the other hand, Intel did leave enough hints in the public documentation that unreal mode appears to have been independently discovered a number of times. What is fascinating is that Microsoft apparently implemented unreal mode support in HIMEM.SYS and made its source code available before any literature on unreal mode was published. Starting in late 1989, articles describing unreal mode were written without knowing of each other, and likely without knowing of Microsoft’s prior published work. Reinventing wheels may be satisfying, but in the end it’s just not very productive.

PS: Images of 386 clone BIOSes (other than Compaq and IBM) from 1987 and earlier would be worth checking for LOADALL emulation and possible other unreal mode use.

Feb 2021 Update: The original 231630-001 datasheet has now been recovered and this post updated accordingly.