IBM XENIX: Two Steps Forward

There are reasons to revisit an old topic. Very old, considering that IBM Xenix 1.0 was released in 1984, well over 30 years ago. To recap, this version of Xenix is unique in that it runs only on 286 processors. It requires a 286 but is incompatible with 386 and later processors due to the use of a reserved word in GDT/LDT descriptors. Good times. And there is a story related to the Xenix 386 incompatibility, but that’s something for another post.

Starting the IBM XENIX installer

Last time the OS/2 Museum attacked IBM Xenix, the incompatibility with any 32-bit x86 processors was a big hurdle. In the meantime, VirtualBox learned to emulate a 286 processor (version 5.1.16 or later). The emulation is far from perfect but it is good enough to run IBM Xenix 1.0 with only slight patching. The required modifications concern incompatibility of IBM Xenix 1.0 with post-PC/AT hardware, namely EGA/VGA graphics and hard disks bigger than 20 or 30 MB. Both issues had been explored 30+ years ago by contemporary Xenix users.

The EGA/VGA fix involves patching a single location in the kernel to prevent it from destroying the display. Fixing the disk access is more complicated due to the very idiosyncratic disk handling in IBM Xenix 1.0. To recap, the OS reads the drive type from the CMOS, but uses its own built-in drive tables—several slightly different ones for different parts of the OS. The fix is forcing the drive type to 2 to make Xenix happy, and make sure the attached hard disk uses 615/4/17 CHS geometry or bigger. That needs to be done for the kernel, fdisk, and the boot loader.

With these patches in place, it’s possible to install IBM Xenix 1.0 in a VM (not a straightforward  process). Note that both the install disk and disk 3 of the set need to be patched, otherwise installing the extended OS undoes some of the patches from the installation disk.

Completed IBM XENIX 1.0 install

At the end of the installation process, one ends up with a nice little System III  Unix bootable from hard disk.

Configuring a VM

But wait, the installation disk won’t boot in a VM, it just hangs. A good starting point is a default DOS VM with a 500 MB virtual disk (which will be largely wasted, but who cares). Several adjustments need to be made to the VM:

  • Restrict the RAM size to 4 MB. This old Xenix can’t handle much memory, and it probably wasn’t even possible to put more into the original PC/AT.
  • Set the floppy drive type to 1.2 MB (5¼″) because Xenix can’t handle a 5¼″ disk image in a 3½″ floppy drive:
    VBoxManage setextradata <vm> VBoxInternal/Devices/i82078/0/LUN#0/Config/Type "Floppy 1.20"
  • Set the VM’s CPU type to 286 because this Xenix can’t deal with anything else:
    VBoxManage modifyvm <vm> --cpu-profile "Intel 80286"
  • Don’t forget to patch the installation disk and disk 3 of the OS.

Without tweaking the VM, the installation disk will not load. Without patching the disks, it will be hard to see anything and it won’t be possible to install Xenix unless the drive type reported in CMOS is 2. Let’s try booting the installation disk:

286 XENIX boot prompt

If all the prep went well, after a few seconds the system should end up here:

Install/maintenance disk shell

Don’t forget to shut down the system with haltsys or the file system will be dirty and fsck will need to run on next boot.


The other news item is that IBM’s SDS (Software Development System) for Xenix 1.0 (and separately, for Xenix 2.0) has surfaced. The SDS is a set of three high-density 5¼″ floppies with assembler (a MASM variant!), C compiler, linker, debugger (adb, shudder), lint, headers, libraries, and miscellaneous utilities. Basically state of the art in 1984, with support for the 286 architecture. The compiler appears to be rather similar to the DOS-based Microsoft C 3.0, which should not be surprising.

IBM XENIX booted from hard drive

The SDS can be easily enough installed with xinstall and using it should be no problem for anyone familiar with traditional Unix development tools.

Playing around with IBM XENIX SDS

The SDS for Xenix 2.0 additionally supports cross-development of DOS executables. It is believed that around 1985, Microsoft used these cross-development tools internally for DOS development. Later, development shifted to OS/2 hosts.

This entry was posted in 286, IBM, VirtualBox, Xenix. Bookmark the permalink.

22 Responses to IBM XENIX: Two Steps Forward

  1. Yuhong Bao says:

    I think something like is more suitable for the job anyway.

  2. Michal Necasek says:

    Curious. How do you attach to an emulated serial port from outside the browser? With putty or another VM, for example.

  3. Yuhong Bao says:

    I don’t think this is supproted.

  4. zeurkous says:

    The horrible things people do w/ WWW browsers these days… :r

  5. Michal Necasek says:

    It’s amazing what PCjs can do. But it’s just different from what you can do with a desktop virtualizer (which again won’t run on, say, an iPad, like PCjs does). As usual, pick the right tool for the job.

  6. zeurkous says:

    I’m not sure I’d call it ‘amazing’. ‘Insane’ springs more to my little mind :X

  7. zeurkous says:

    Alright, I suppose I’ll elaborate, just in case: I think that, like a Disney movie, on the surface, it’s indeed amazing and wonderful and magical and…

    …but right under that surface lurks crap design, horrible implementation (feel free to swap the ‘crap’ and ‘horrible’ w/ each other as appropriate), and… well, need me say more?

    Computers, while wonderful machines, are very complex devices that are better constructed and handled quite carefully. “But it runs on my fondleslab!” makes a decent argument when said fondleslab isn’t a piece of superficial junk. Unfortunately, the ones I’ve encountered until now all have been.

    I guess I’m one of those weird types who grumble about complexity and black boxes and the ease of which people accept them. What’s certain is that I’d rather be considered weird than ignore my own good senses.

    Hm, that turned out to be my ROTD. Sorry!

  8. NTfinity says:

    use PCjs or PCem to emulate (it actually software-emulates a 286 pc and is way easier to make work) and not vbox/vmware, which is usually used for modern OSes. (By that I mean WinXP+)

  9. Michal Necasek says:

    If you can’t get 286 Xenix or Windows 1.0 or OS/2 1.0 or whatever going in VirtualBox, that’s your problem 🙂

  10. Rugxulo says:

    “The SDS … with assembler (a MASM variant!)”

    I’m not sure why this is a surprise. There were many, many MASM variants over the years (and I’m far from an expert).

    “Basically state of the art in 1984, with support for the 286 architecture.”

    Ah, so MASM v4 then, right?

    Did you know that the famous ArrowSoft Assembler 2.00d was actually just MASM 4.0 in disguise? Yeah, apparently one guy privately found out (2011?), and I emailed another guy to compare the binaries. Turns out that it was compressed and hacked but only like 600 bytes difference (i.e. not much). Very strange, and this was even widely mirrored to various places (e.g. Simtel) for decades!

    Obviously the 386 (and MASM v5) was much more popular. Sounds almost silly to have to say, but it’s true.

  11. Michal Necasek says:

    Why it’s a surprise… how many Unixes do you know that use MASM as their system assembler?

    I don’t think the Xenix tools exactly correspond to specific DOS-based versions. The assembler and compiler should be something like MASM 4.0 and MS C 3.0 but not precisely. What I know is that the 286 Xenix is where Microsoft’s own C compiler first appeared (MS C 3.0 and later on DOS).

    And sure, the 386 and MASM 5.x were a lot more popular… but not in 1984.

  12. Rugxulo says:

    How many Unixes? How many did MS and/or IBM work on? Come on, it’s probably portable C code (although v1 was allegedly in Pascal). If they can port stuff between DOS, OS/2, and NT, then sure Xenix isn’t out of the question either. Certainly MASM ran on all of those, too. It got around!

    Not in 1984 … obviously! But my point is that most people left behind 286s when the 386 gained traction. You won’t find many references to anything before MASM v5, and even v6 is way more popular. So we’re in the dark about why things were the way they were back then.

  13. Michal Necasek says:

    You seem to be arguing my points for me — yes, a Unix with MASM was a very rare thing.

    It’s really difficult to judge relative popularity of products between 1985/1990/1995. There were far fewer PCs in the old days, and because there was no Internet to speak of, a lot less information survived. So not finding many references to MASM 4 is entirely expected but it doesn’t prove much. Note that I’m not saying MASM 4 was wildly popular, just pointing out that it’s very hard to make an informed assessment.

  14. For Old Hack says:

    From someone who was there, and a user at the time, this is *amazing*.

    I started with covalient unix on a NCR box, but wanted something that ran it at home on my XT. So, I loaded Xenix-86. All fine and dandy on a box that with 640K of ram, and two 20MB hard disks, was as fast as a 286, but with no virtual memory. ( the swap disk being on the second hard disk, improved performance significantly ).

    Everything worked like a charm, until I tried to compile something from another user in the serial port. ( Nethack sources… ). Blew both file systems out of the water in a non-recoverable way. I would wait for a more powerful box, and five years later, and a 386 that would run Microport. Much better experience, and … again with two hard disks, and the swap file on the second hard disk, it was snappy. Got a lot of work out of that box.

    As far as MASM was concerned, in 1985/86 running on a two year old box, there was never a problem with compatibility or assembly or linking as long as your projects were small. I was able to go through K&R as well as some AT&T tutorials.
    I had used MASM from 1.0, and MASM 3.0 did its job. A very different experience from the asm on the covalient box, but it was a welcome tool.

    The Xenix-86 we used was a run time for an application ( SABRE ) trainer, and ran with little problem as an OS, and development was small, but it was exactly what I would expect. Its a miracle that the thing booted, ( like DOS ), and from there it worked.

  15. ForOldHack says:

    I was digging around looking for a very old FPU benchmark and came upon a few facts: IBM XENIX was not released in 1984. It was announced in December of 1984, for 1985.

    PC Tech Journal, Dec 1984, pg 115: IBM Personal Computer XENIX, IBM, Availability: First quarter 1985.

    IBM knew damn well from the i286 developer notes that the upper word for the descriptor tables was reserved for the i386, and it was documented by Intel. ( Thanks, for having both the times and dates for these documents ). ( pg 122, PC Tech Journal, Fig 2, Segment Descriptor Access Rights word, 1st byte “INTEL RESERVED*. Note: Must be set to 0 for compatibility with iAPX 386.” Pg 130, iAPX 286 DESCRIPTORS
    “Two of those bytes are reserved for iAPX 386 compatibility…”

    References: Intel iAPX 286 Hardware Reference Manual. #210760-001 (1983! ) Intel iAPX 286 Operating Systems Writer’s Guide #121960-001. (1983!) ( Both are on bitsavers )

    Dont bother, they say the same thing, in fact, it looks like the text was lifted right out of the developer manual.

    IBM read the documentation, ignored the note, and made XENIX 1.0 286 only.

  16. Michal Necasek says:

    But I’m telling you, IBM didn’t write IBM XENIX. Microsoft did, maybe with Intel’s help. What’s more, Intel also used XENIX for their own 286-based machines (they weren’t PC compatibles), and Intel had 286 XENIX no later than November 1984.

    It is true that the text about the descriptor word being ‘reserved for iAPX 386 compatibility’ appears as early as 1983 in Intel documentation. We don’t know if it also appeared in older 286 documents. We also don’t know what documentation the people who ported XENIX to the 286 worked from.

    It’s possible that the 286 XENIX authors did not have documentation warning about 386 compatibility. It is also entirely possible that they did, and said “but who the heck cares about a nonexistent CPU”. They may well have worked in an environment where there was absolutely zero expectation that existing software written for one CPU model would be used unchanged on a different CPU model.

  17. ForOldHack says:

    Yes, XENIX was all Microsoft. I am realizing that at the time, of course they did not issue a fix to get XENIX ON the 386, they had PS/2s with A/IX to sell. Microsoft had XENIX 386 to sell… sell sell sell, not fix fix fix.

    “It’s possible that the 286 XENIX authors did not have documentation warning about 386 compatibility. ” Its not possible, required reading would have been designing OSs for the 286, which had the diagram, that the hardware manual had, and the PC Tech journal had. Why fix a $500 piece of software when you can sell another $500 piece of software when you sell them a new machine.

    I never tried XENIX on the Above board accelerator, or the Above board AT, because they ran dos/windows just fine, and XENIX was a file system destroyer.

  18. ForOldHack says:

    But, you are absolutely right, I need to track TWO time lines: IBM XENIX which was a rebranding of Microsoft XENIX and Intel XENIX.

    I understand that Microsoft was using their DEC machines for XENIX development, and Intel must have been using compile engines.

    “We don’t know if it also appeared in older 286 documents.” Now I have to find engineering documentation…

  19. ForOldHack says:

    More damning documentation:
    Intel iAPX 286 Development builder, (c) 1981. ( I remember reading this when it came out, and remember this too: )
    “The Evaluation Builder reserves the first 17 entries ( 0-16 ) in each of the description tables for Intel Use. If any of these entries are used, a warning is ussued but the specific entry is used.”
    What I did not remember was that Intel reserved MORE than the first byte, they also reserved the 1st bit of the 2nd byte.

  20. Michal Necasek says:

    Actually, I have to completely disagree, but thank you for pointing out this document that I was previously unaware of!

    Looking at Intel document 121711-001, the text clearly talks about entries in the GDT/LDT, meaning that entries with index 0 to 16 in those tables are reserved for Intel use. What you quoted does not apply to the actual descriptors.

    The diagrams on pages 5-20 and 5-26 are very interesting because they show the last word of gate descriptors as “S/W reserved”. Page 5-27 also shows “the software reserved word” as part of a general descriptor entry.

    I think you in fact found the explanation why Microsoft (or Intel or whoever designed that actual bit) used the last word of a descriptor entry in XENIX as generic storage! That word is called SW_WORD in the Evaluation Builder language, which sure sounds like “a word that software can freely use” rather than “a word strictly reserved for hardware”.

    Now the ‘short d_sw; /* software defined word, unused */’ entry in the descriptor struct in the XENIX header files makes a lot more sense.

    For a definitive answer, we’d need the corresponding iAPX 286 User’s Manual, Intel order no. 121749 from 1981. I’ve never seen a copy of it. But it sure looks like the weird 286 XENIX descriptor management was written based on early 286 documentation which made absolutely no mention of 386 compatibility.

  21. ForOldHack says:

    Ok. More data:
    The IBM AT Tech Ref had bios listings:
    Those BIOS listings, had a GDT initializer in it in file SYSDATA.INC,
    containing the following line:
    IBM PC AT BIOS dated 01/10/84 based on the BIOS listings in the IBM PC AT Technical Reference dated March 1984.
    The BIOS was originally built using IBM MASM 1.0.

    My point is this:
    Some person sat down to port XENIX, and must have had one or two of these three manuals, and chose to ignore the reserved address 7h. I would say it was not ignorance, but to both make it so it would not run on 386s ( another $500 sale), and would have a undisturbed place in memory to do status things with.

  22. Michal Necasek says:

    Yeah, we know what the 286 manuals said in 1983. The thing is that XENIX is older than the PC/AT, because it ran on Intel’s 286 machines before the PC/AT even existed. And we don’t know what the 286 manuals said in late 1981 or early 82, though we can guess it wasn’t “reserved for 386 compatibility”.

    Whoever wrote the 286 XENIX code did not care about the nonexistent 386. It was just a convenient place to store a word of data. Also, in the porting to the 286, the descriptor usage would have had to be sorted out very early.

    From the Intel XENIX 286 overview: “Intel moved the previous release of XENIX from the 8086 microprocessor to the iAPX 286 microprocessor, and Microsoft used the new product as the basis for its Release 3 of the operating system.” So whoever was responsible very likely was at Intel.

    You will notice in the timeline shown here that 286 XENIX is placed firmly into 1983, perhaps early 1983. It’s hard to say when development on 286 XENIX started. Keep in mind that Intel had XENIX for the 8086 probably as early as 1981, and it is possible that Intel was working on porting to the 286 before the 286 was even released.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.