IBM XENIX: Two Steps Forward

There are reasons to revisit an old topic. Very old, considering that IBM Xenix 1.0 was released in 1984, well over 30 years ago. To recap, this version of Xenix is unique in that it runs only on 286 processors. It requires a 286 but is incompatible with 386 and later processors due to the use of a reserved word in GDT/LDT descriptors. Good times. And there is a story related to the Xenix 386 incompatibility, but that’s something for another post.

Starting the IBM XENIX installer

Last time the OS/2 Museum attacked IBM Xenix, the incompatibility with any 32-bit x86 processors was a big hurdle. In the meantime, VirtualBox learned to emulate a 286 processor (version 5.1.16 or later). The emulation is far from perfect but it is good enough to run IBM Xenix 1.0 with only slight patching. The required modifications concern incompatibility of IBM Xenix 1.0 with post-PC/AT hardware, namely EGA/VGA graphics and hard disks bigger than 20 or 30 MB. Both issues had been explored 30+ years ago by contemporary Xenix users.

The EGA/VGA fix involves patching a single location in the kernel to prevent it from destroying the display. Fixing the disk access is more complicated due to the very idiosyncratic disk handling in IBM Xenix 1.0. To recap, the OS reads the drive type from the CMOS, but uses its own built-in drive tables—several slightly different ones for different parts of the OS. The fix is forcing the drive type to 2 to make Xenix happy, and make sure the attached hard disk uses 615/4/17 CHS geometry or bigger. That needs to be done for the kernel, fdisk, and the boot loader.

With these patches in place, it’s possible to install IBM Xenix 1.0 in a VM (not a straightforward  process). Note that both the install disk and disk 3 of the set need to be patched, otherwise installing the extended OS undoes some of the patches from the installation disk.

Completed IBM XENIX 1.0 install

At the end of the installation process, one ends up with a nice little System III  Unix bootable from hard disk.

Configuring a VM

But wait, the installation disk won’t boot in a VM, it just hangs. A good starting point is a default DOS VM with a 500 MB virtual disk (which will be largely wasted, but who cares). Several adjustments need to be made to the VM:

  • Restrict the RAM size to 4 MB. This old Xenix can’t handle much memory, and it probably wasn’t even possible to put more into the original PC/AT.
  • Set the floppy drive type to 1.2 MB (5¼″) because Xenix can’t handle a 5¼″ disk image in a 3½″ floppy drive:
    VBoxManage setextradata <vm> VBoxInternal/Devices/i82078/0/LUN#0/Config/Type "Floppy 1.20"
  • Set the VM’s CPU type to 286 because this Xenix can’t deal with anything else:
    VBoxManage modifyvm <vm> --cpu-profile "Intel 80286"
  • Don’t forget to patch the installation disk and disk 3 of the OS.

Without tweaking the VM, the installation disk will not load. Without patching the disks, it will be hard to see anything and it won’t be possible to install Xenix unless the drive type reported in CMOS is 2. Let’s try booting the installation disk:

286 XENIX boot prompt

If all the prep went well, after a few seconds the system should end up here:

Install/maintenance disk shell

Don’t forget to shut down the system with haltsys or the file system will be dirty and fsck will need to run on next boot.


The other news item is that IBM’s SDS (Software Development System) for Xenix 1.0 (and separately, for Xenix 2.0) has surfaced. The SDS is a set of three high-density 5¼″ floppies with assembler (a MASM variant!), C compiler, linker, debugger (adb, shudder), lint, headers, libraries, and miscellaneous utilities. Basically state of the art in 1984, with support for the 286 architecture. The compiler appears to be rather similar to the DOS-based Microsoft C 3.0, which should not be surprising.

IBM XENIX booted from hard drive

The SDS can be easily enough installed with xinstall and using it should be no problem for anyone familiar with traditional Unix development tools.

Playing around with IBM XENIX SDS

The SDS for Xenix 2.0 additionally supports cross-development of DOS executables. It is believed that around 1985, Microsoft used these cross-development tools internally for DOS development. Later, development shifted to OS/2 hosts.

This entry was posted in 286, IBM, VirtualBox, Xenix. Bookmark the permalink.

7 Responses to IBM XENIX: Two Steps Forward

  1. Yuhong Bao says:

    I think something like is more suitable for the job anyway.

  2. Michal Necasek says:

    Curious. How do you attach to an emulated serial port from outside the browser? With putty or another VM, for example.

  3. Yuhong Bao says:

    I don’t think this is supproted.

  4. zeurkous says:

    The horrible things people do w/ WWW browsers these days… :r

  5. Michal Necasek says:

    It’s amazing what PCjs can do. But it’s just different from what you can do with a desktop virtualizer (which again won’t run on, say, an iPad, like PCjs does). As usual, pick the right tool for the job.

  6. zeurkous says:

    I’m not sure I’d call it ‘amazing’. ‘Insane’ springs more to my little mind :X

  7. zeurkous says:

    Alright, I suppose I’ll elaborate, just in case: I think that, like a Disney movie, on the surface, it’s indeed amazing and wonderful and magical and…

    …but right under that surface lurks crap design, horrible implementation (feel free to swap the ‘crap’ and ‘horrible’ w/ each other as appropriate), and… well, need me say more?

    Computers, while wonderful machines, are very complex devices that are better constructed and handled quite carefully. “But it runs on my fondleslab!” makes a decent argument when said fondleslab isn’t a piece of superficial junk. Unfortunately, the ones I’ve encountered until now all have been.

    I guess I’m one of those weird types who grumble about complexity and black boxes and the ease of which people accept them. What’s certain is that I’d rather be considered weird than ignore my own good senses.

    Hm, that turned out to be my ROTD. Sorry!

Leave a Reply

Your email address will not be published. Required fields are marked *