It has only recently been brought to my attention that Intel’s SYSENTER/SYSEXIT instructions have rather unusual past, and their origin is shrouded in mystery and confusion. One facet of the usage of these instructions is also a little unorthodox.
Depending on who you ask, the SYSENTER/SYSEXIT instruction pair was introduced either in the Pentium Pro or in the Pentium II. But that makes absolutely no sense on the face of it, right? Either the Pentium Pro supported these instructions or it didn’t. Well…
The Pentium Pro (aka P6) was introduced in November 1995 and it was the first member of a tremendously successful family of processors which destroyed all of x86’s RISC rivals in the late 1990s, and quite possibly saved Intel after the Itanium and NetBurst debacles. The Pentium Pro and its immediate successors, Pentium II and Pentium III, turned out to be immensely scalable and between 1995 and 2001, raised the clock speed from 150 MHz (original Pentium Pro) all the way to 1.4 GHz (Pentium III-S and other Tualatin PIII variants).
The Pentium Pro also introduced a few new instructions, such as CMOV/FCMOV (conditional moves) or RDPMC, and a number of architectural features (global pages, page address extensions, machine check exceptions, MTRRs). The Pentium Pro documentation from December 1995 (Pentium Pro Family Developer’s Manual, Volume 2: Programmer’s Reference Manual, Intel order no. 242691-001) certainly made no mention of SYSENTER or associated MSRs.
On the other hand, Intel’s U.S. Patent no. 5948097 (Method and apparatus for changing privilege levels in a computer system without use of a call gate, Glew et al.) filed in August 1996 describes SYSENTER in detail and says: “For one embodiment, SYSENTER and SYSEXIT are assembly-language instructions that may be executed on an Intel architecture processor, such as the Pentium Pro processor.”
The book Pentium Pro and Pentium II System Architecture (Shanley) claims that SYSENTER/SYSEXIT were introduced in the Pentium II but notes that “these instructions are not documented in the currently-released (as of 8/7/97) Intel manuals”. The book was published in December 1997. The book also says that SYSENTER/SYSEXIT capability is indicated by the SEP bit in the CPUID feature mask, but only if the processor is family 6, model 3, stepping 3 or greater. That is quite unusual and goes against the logic of CPUID.
The Pentium II Processor Developer’s Manual from October 1997 (Intel order no. 243502-001) contains absolutely no mention of SYSENTER or SYEXIT, and neither does the Intel Architecture Software Developer’s Manual from December 1997 (Intel order numbers 243190-001, 243191-001, 243192-001).
On the other hand Intel’s Application Note AP-485, Intel Processor Identification and the CPUID Instruction from June 1997 (Intel order no. 241618-007) is slightly older and does mention SYSENTER and SYSEXIT, but only explains how to detect the presence of the instructions, not how to use them. It is likely that AP-485 is the source of information in Pentium Pro and Pentium II System Architecture.
The second edition of the Intel Architecture Software Developer’s Manual is from April 1999 (Intel order numbers 243190-002, 243191-002, 243192-002) also covers the Pentium III and does properly document SYSENTER and SYSEXIT—as instructions introduced in the Pentium II. The instruction set reference gives the following pseudocode for detecting SYSENTER presence (already seen in the AP-485 CPUID note):
IF (CPUID SEP bit is set) IF (Family == 6) AND (Model < 3) AND (Stepping < 3) THEN Fast System Call NOT supported FI ELSE Fast System Call is supported FI
It also says: “The Pentium Pro processor (Model = 1) returns a set SEP CPUID feature bit, but does not support the SYSENTER/SYSEXIT instructions.” Curiously, the SYSENTER MSRs are not listed in the MSR reference (Appendix B of Volume 3: System Programming) at all.
It is notable that the current (July 2017) Intel SDM lists the SYSENTER MSRs as architectural MSRs supported on the Pentium Pro and later (not Pentium II and later as one might think).
In October 1998, the Pentium II Specification Update (Intel order no. 243337-019) added erratum A62, SYSENTER/SYSEXIT instructions can implicitly load “null segment selector” to SS and CS registers. The erratum does not affect a typical operating system and wouldn’t be very noteworthy on its own.
What is quite noteworthy is that the same erratum (SYSENTER/SYSEXIT instructions can implicitly load “null segment selector” to SS and CS registers) was also added as erratum 82 to the October 1998 edition of the Pentium Pro Processor Specification Update (Intel order no. 242689-032). The obvious question is how that is possible, if the Pentium Pro supposedly does not support SYSENTER/SYSEXIT?
Some have claimed that erratum 82 is in fact the reason why SYSENTER support should be disabled on Pentium Pros. That is extremely difficult to believe when the Pentium II has the exact same erratum.
Out in the Wild
It wasn’t until the early 2000s when SYSENTER/SYSEXIT started being used; given that public documentation may have only surfaced in 1999, that should not be surprising. Windows XP (2001) added optional support for SYSENTER and SYSEXIT. In 2003, Linux introduced SYSENTER support in the 2.6.0 kernel.
And immediately there was trouble. The initial Linux code caused system crashes on Pentium Pro CPUs. The reason? Linux implemented the model/stepping version check documented by Intel in the SDM instruction reference.
To this day (July 2017), the Intel SDM instruction reference claims that the check for CPUs which have the SEP bit but do not support SYSENTER is ‘IF (Family = 6) and (Model < 3) and (Stepping < 3)’.
But sometime in 2000 or so, the CPUID application note was changed to list the SYSENTER support condition to ‘IF (Family == 6) AND (ModelStepping < 0x33)’ which is certainly not the same! Notably for later-model Pentium Pros with model 1, stepping 9, the two conditions give different results.
As Linux users discovered, the condition given in the instruction reference is incorrect and SYSENTER (or is it SYSEXIT?) does not behave as expected even on later Pentium Pros.
An interesting question is the Pentium Pro OverDrive, in reality a Deschutes Pentium II. The PPro OverDrive identifies itself as model 3, stepping 2 (in reality it is a close relative of model 5 Pentium IIs). According to the CPUID test, it does not support SYSENTER. According to common sense, it really should.
So What Happened?
The unanswered question is what really happened (and even Andy Glew, the author of the SYSENTER patent and designer of the instruction doesn’t know). SYSENTER certainly was part of the Pentium Pro design. It seems likely that Intel discovered some problem with the SYSENTER instruction very shortly before the chip was released. And instead of documenting the problems and fixing them, Intel decided not to document the instruction at all.
Given that existing software had no expectations about SYSENTER, that didn’t cause trouble. Only after the Pentium II came out and SYSENTER was documented, it became apparent that it had been present on the Pentium Pro all along, but was either buggy or functioned differently. It is not obvious why Intel chose not to fix it in later Pentium Pro steppings.
So… what does SYSENTER and SYSEXIT actually do on Pentium Pro CPUs? How is it different from the later models? And what about the Pentium Pro OverDrive? So many questions.