IBM XENIX: Two Steps Forward

There are reasons to revisit an old topic. Very old, considering that IBM Xenix 1.0 was released in 1984, well over 30 years ago. To recap, this version of Xenix is unique in that it runs only on 286 processors. It requires a 286 but is incompatible with 386 and later processors due to the use of a reserved word in GDT/LDT descriptors. Good times. And there is a story related to the Xenix 386 incompatibility, but that’s something for another post.

Starting the IBM XENIX installer

Last time the OS/2 Museum attacked IBM Xenix, the incompatibility with any 32-bit x86 processors was a big hurdle. In the meantime, VirtualBox learned to emulate a 286 processor (version 5.1.16 or later). The emulation is far from perfect but it is good enough to run IBM Xenix 1.0 with only slight patching. The required modifications concern incompatibility of IBM Xenix 1.0 with post-PC/AT hardware, namely EGA/VGA graphics and hard disks bigger than 20 or 30 MB. Both issues had been explored 30+ years ago by contemporary Xenix users.

The EGA/VGA fix involves patching a single location in the kernel to prevent it from destroying the display. Fixing the disk access is more complicated due to the very idiosyncratic disk handling in IBM Xenix 1.0. To recap, the OS reads the drive type from the CMOS, but uses its own built-in drive tables—several slightly different ones for different parts of the OS. The fix is forcing the drive type to 2 to make Xenix happy, and make sure the attached hard disk uses 615/4/17 CHS geometry or bigger. That needs to be done for the kernel, fdisk, and the boot loader.

With these patches in place, it’s possible to install IBM Xenix 1.0 in a VM (not a straightforward  process). Note that both the install disk and disk 3 of the set need to be patched, otherwise installing the extended OS undoes some of the patches from the installation disk.

Completed IBM XENIX 1.0 install

At the end of the installation process, one ends up with a nice little System III  Unix bootable from hard disk.

Configuring a VM

But wait, the installation disk won’t boot in a VM, it just hangs. A good starting point is a default DOS VM with a 500 MB virtual disk (which will be largely wasted, but who cares). Several adjustments need to be made to the VM:

  • Restrict the RAM size to 4 MB. This old Xenix can’t handle much memory, and it probably wasn’t even possible to put more into the original PC/AT.
  • Set the floppy drive type to 1.2 MB (5¼″) because Xenix can’t handle a 5¼″ disk image in a 3½″ floppy drive:
    VBoxManage setextradata <vm> VBoxInternal/Devices/i82078/0/LUN#0/Config/Type "Floppy 1.20"
  • Set the VM’s CPU type to 286 because this Xenix can’t deal with anything else:
    VBoxManage modifyvm <vm> --cpu-profile "Intel 80286"
  • Don’t forget to patch the installation disk and disk 3 of the OS.

Without tweaking the VM, the installation disk will not load. Without patching the disks, it will be hard to see anything and it won’t be possible to install Xenix unless the drive type reported in CMOS is 2. Let’s try booting the installation disk:

286 XENIX boot prompt

If all the prep went well, after a few seconds the system should end up here:

Install/maintenance disk shell

Don’t forget to shut down the system with haltsys or the file system will be dirty and fsck will need to run on next boot.


The other news item is that IBM’s SDS (Software Development System) for Xenix 1.0 (and separately, for Xenix 2.0) has surfaced. The SDS is a set of three high-density 5¼″ floppies with assembler (a MASM variant!), C compiler, linker, debugger (adb, shudder), lint, headers, libraries, and miscellaneous utilities. Basically state of the art in 1984, with support for the 286 architecture. The compiler appears to be rather similar to the DOS-based Microsoft C 3.0, which should not be surprising.

IBM XENIX booted from hard drive

The SDS can be easily enough installed with xinstall and using it should be no problem for anyone familiar with traditional Unix development tools.

Playing around with IBM XENIX SDS

The SDS for Xenix 2.0 additionally supports cross-development of DOS executables. It is believed that around 1985, Microsoft used these cross-development tools internally for DOS development. Later, development shifted to OS/2 hosts.

This entry was posted in 286, IBM, VirtualBox, Xenix. Bookmark the permalink.

13 Responses to IBM XENIX: Two Steps Forward

  1. Yuhong Bao says:

    I think something like is more suitable for the job anyway.

  2. Michal Necasek says:

    Curious. How do you attach to an emulated serial port from outside the browser? With putty or another VM, for example.

  3. Yuhong Bao says:

    I don’t think this is supproted.

  4. zeurkous says:

    The horrible things people do w/ WWW browsers these days… :r

  5. Michal Necasek says:

    It’s amazing what PCjs can do. But it’s just different from what you can do with a desktop virtualizer (which again won’t run on, say, an iPad, like PCjs does). As usual, pick the right tool for the job.

  6. zeurkous says:

    I’m not sure I’d call it ‘amazing’. ‘Insane’ springs more to my little mind :X

  7. zeurkous says:

    Alright, I suppose I’ll elaborate, just in case: I think that, like a Disney movie, on the surface, it’s indeed amazing and wonderful and magical and…

    …but right under that surface lurks crap design, horrible implementation (feel free to swap the ‘crap’ and ‘horrible’ w/ each other as appropriate), and… well, need me say more?

    Computers, while wonderful machines, are very complex devices that are better constructed and handled quite carefully. “But it runs on my fondleslab!” makes a decent argument when said fondleslab isn’t a piece of superficial junk. Unfortunately, the ones I’ve encountered until now all have been.

    I guess I’m one of those weird types who grumble about complexity and black boxes and the ease of which people accept them. What’s certain is that I’d rather be considered weird than ignore my own good senses.

    Hm, that turned out to be my ROTD. Sorry!

  8. NTfinity says:

    use PCjs or PCem to emulate (it actually software-emulates a 286 pc and is way easier to make work) and not vbox/vmware, which is usually used for modern OSes. (By that I mean WinXP+)

  9. Michal Necasek says:

    If you can’t get 286 Xenix or Windows 1.0 or OS/2 1.0 or whatever going in VirtualBox, that’s your problem 🙂

  10. Rugxulo says:

    “The SDS … with assembler (a MASM variant!)”

    I’m not sure why this is a surprise. There were many, many MASM variants over the years (and I’m far from an expert).

    “Basically state of the art in 1984, with support for the 286 architecture.”

    Ah, so MASM v4 then, right?

    Did you know that the famous ArrowSoft Assembler 2.00d was actually just MASM 4.0 in disguise? Yeah, apparently one guy privately found out (2011?), and I emailed another guy to compare the binaries. Turns out that it was compressed and hacked but only like 600 bytes difference (i.e. not much). Very strange, and this was even widely mirrored to various places (e.g. Simtel) for decades!

    Obviously the 386 (and MASM v5) was much more popular. Sounds almost silly to have to say, but it’s true.

  11. Michal Necasek says:

    Why it’s a surprise… how many Unixes do you know that use MASM as their system assembler?

    I don’t think the Xenix tools exactly correspond to specific DOS-based versions. The assembler and compiler should be something like MASM 4.0 and MS C 3.0 but not precisely. What I know is that the 286 Xenix is where Microsoft’s own C compiler first appeared (MS C 3.0 and later on DOS).

    And sure, the 386 and MASM 5.x were a lot more popular… but not in 1984.

  12. Rugxulo says:

    How many Unixes? How many did MS and/or IBM work on? Come on, it’s probably portable C code (although v1 was allegedly in Pascal). If they can port stuff between DOS, OS/2, and NT, then sure Xenix isn’t out of the question either. Certainly MASM ran on all of those, too. It got around!

    Not in 1984 … obviously! But my point is that most people left behind 286s when the 386 gained traction. You won’t find many references to anything before MASM v5, and even v6 is way more popular. So we’re in the dark about why things were the way they were back then.

  13. Michal Necasek says:

    You seem to be arguing my points for me — yes, a Unix with MASM was a very rare thing.

    It’s really difficult to judge relative popularity of products between 1985/1990/1995. There were far fewer PCs in the old days, and because there was no Internet to speak of, a lot less information survived. So not finding many references to MASM 4 is entirely expected but it doesn’t prove much. Note that I’m not saying MASM 4 was wildly popular, just pointing out that it’s very hard to make an informed assessment.

Leave a Reply

Your email address will not be published. Required fields are marked *