Some time ago I wrote about IBM PC XENIX 1.0 and why it won’t work on 386 and later processors. Thanks to a kind reader, I’ve been able to analyze the object files used to link the kernel, and I believe I now understand exactly how XENIX uses the reserved space in LDT descriptors and why my patched kernel doesn’t fully work. This problem is likely common to all early editions of Microsoft XENIX 3.0, not just the IBM-branded release.
What XENIX stores in the reserved word is the size of the corresponding segment on disk; the value for a given descriptor is returned by a kernel-internal function called
dscrsw(). This is used when an executable file gets loaded and also when swapped out. The latter is likely the reason why the segment size on file is stored in the descriptor table at all.
Incidentally, why is the troublemaking routine called
dscrsw anyway? The answer to that riddle can be found in
/usr/include/sys/relsym86.h: The last word of the descriptor table structure is defined as
d_sw, or “software defined word, unused”. If only.
My patch made
dscrsw() return the segment limit rather the physical (on-disk) size. That happens to work well for executables which have a single segment not followed by any other data… which happens to be the vast majority of XENIX 3.0 executables. In that case, the OS ends up reading exactly as much as it should (typically less than what it was asked for).
For files with symbols, XENIX ends up reading more than it should and possibly causes some corruption. Similar trouble occurs with multi-segment executables, and those may not work at all. There are remarkably few of those shipped with IBM XENIX. In the base package, it’s only
vsh (Visual Shell). While
vsh is easy to live without,
vi is much more critical.
On the whole, the patching was about a 90% success but in the meantime, a much better solution turned up; more about that later.
The descriptor mess made me wonder again why XENIX was written that way. The oldest relevant documentation available to me, the Intel iAPX 286 Operating Systems Writer’s Guide (order no. 121960-001) from 1983, speaks quite clearly: The last word of any descriptor type is “reserved for iAPX 386, must be zero”.
That documentation was available well before IBM XENIX 1.0 was released (late 1984), so there was at least some sloppiness involved. But there is another, more charitable explanation.
It is possible that in earlier versions of 286 documentation, the last descriptor word was merely marked as reserved or unused, and the implication was that it would be used on a 386 if some not-yet-defined bit marked the segment as a “386” segment. But in the end, that’s not how it worked, and IBM XENIX 1.0 is living proof—the 386 uses the last word always, regardless of descriptor type. In other words, there was a silent change in behavior where existing 286 binaries simply behave differently on a 386.
And to be fair, that difference is spelled out quite clearly in the Intel 80386 Programmer’s Reference Manual (order no. 230985-001): “Because the 80386 uses the contents of the reserved word (last word) of every descriptor, 80286 programs that place values in this word may not execute correctly on the 80386.” That indicates the problem was known to Intel… in 1986.
But Microsoft’s XENIX 3.0 (aka IBM XENIX 1.0) for the 286 was unlucky enough that it was released before any 386 hardware whatsoever was available, so no one could notice that it doesn’t work at all on the newer CPU. Fortunately, CPU vendors learned from such mistakes and consistently enforce reserved bits in newer designs (i.e. attempts to set reserved bits cause faults).