In 1994, IBM started shipping software with support for XDF, or eXtended Density Format, first in OS/2 Warp and a few months later in PC DOS 7.0. Some of IBM’s software packages were also distributed on XDF diskettes. XDF allowed the user to format a 3½” floppy which normally holds 1,440KB of data to a special format with 1,840KB capacity (well, almost—see below), or almost 28% improvement. This came at a time when CD-ROMs were not yet ubiquitous and software was distributed on rapidly growing piles of floppies. Since floppies were relatively expensive to manufacture and duplicate, shipping software on, say, 16 diskettes instead of 20 was an attractive proposition.
The XDF technology was not developed by IBM. The inventor was Roger D. Ivey (as can be seen in a XDF disk’s boot sector) and was first licensed to IBM by Backup Technologies, Inc. of Tampa, FL. Roger Ivey had previously written the Fastback PC backup software. Later releases of XDF software list Ametron Technologies, Inc. as the licensor. Ametron is a company specializing in selling intellectual property.
The XDF technology used several interesting techniques to achieve both higher storage capacity and speed. The XDF utilities mentioned “Patent(s) Pending” but there is no obvious record of a relevant patent ever being granted by the USPTO. Let’s take a look at some of XDF’s tricks.
The basic problem with higher density floppies when using 512-byte sectors is that once more than 10 or so sectors per track are used, significant capacity is lost to inter-sector gaps, address marks, CRCs, an other “useless” overhead. With 3½” HD floppies, it is possible to use more than the standard 18 sectors per track; the hard limit appears to be 21 sectors which translates to 1,680KB total with 80 tracks. XDF squeezes another 160KB into the same space, using standard hardware and media.
Note that the rest of the article talks about the 1,840KB XDF format unless otherwise noted. XDF also supported a 1,520KB format for 5¼” HD floppies (normally 1.2MB) and a 3,680KB format for 3½” ED media (normally 2.88MB).
The basic XDF approach involved the use of larger sectors, which reduced the sector overhead. On a 3½” HD floppy (DD media were not supported by XDF), where the standard format used 18 sectors per track (9KB), a XDF track contained the equivalent of 23 sectors (11.5KB). Physically, the track was formatted with only four sectors in 8KB, 2KB, 1KB, and 512B sizes (for the 11.5KB total). With an unformatted capacity of about 12,500 bytes, there was just not quite enough space for one 8KB and one 4KB sector (let alone a single 16K sector), while the 8/2/1/0.5 breakdown was reliable.
From reading typical floppy controller documentation, it is not obvious whether a standard FDC would be able to create XDF diskettes. The FORMAT TRACK command takes the sector size as a parameter for the entire track, yet the data supplied to the command for each sector also includes the sector size code. Does the FDC observe the per-track sector size or the per-sector one? FDC documentation is typically annoyingly quiet on this.
The Hitachi HD63265 FDC user’s manual is one of the rare documents which addresses this. Hitachi unambiguously says (on page 56 of the manual) that the actual sector size will be as specified for the entire track, but the size code written to the sector ID fields will be as supplied for each sector. In other words, the sectors will always be the same size on a track. If that’s true, how did XDF format tracks with non-uniform sector sizes?
XDF—of course—cheated. When formatting a XDF track, the FDC was told to create 39 128-byte sectors. However, the sector sizes written to the sector IDs were entirely different. When the sector would be later written, the FDC would only pay attention to the sector size code in the ID field (as there’s no other sector size information on the medium) and happily overwrite the gaps, sector IDs, sync fields, and CRCs of the “fake” 128-byte sectors.
This way XDF circumvented the limited FDC capabilities and created a very unusual but valid format with standard floppy hardware.
The special sector layout was key for achieving high storage capacity. In addition to varying sector sizes, XDF used interleaving and sector slipping to speed up XDF media reading and writing. As a result, XDF was faster than alternative high-capacity formats.
DOS Implementation and Compatibility
On DOS, a XDF.COM utility (TSR) provided standard DOS access for XDF-formatted floppies. The XDFCOPY.EXE tool was used to copy XDF disks, create images from disks and write images to disks. As a side functionality , XDFCOPY also handled most standard DOS formats.
Given XDF’s highly non-standard format, one might think that the DOS-based XDF utilities needed to use direct FDC access. They did not—instead, they showed just how far the standard floppy BIOS services can be convinced to go.
For formatting, the BIOS user supplies a buffer containing per-sector parameters that the FDC then transfers during execution of the FORMAT TRACK command. For reads and writes, XDF dynamically changed the DPT (Diskette Parameter Table) and used the standard BIOS read/write services to access one sector at a time.
It is also apparent that without the XDF TSR loaded, BIOS/DOS won’t be able to do much with XDF media. To this end, XDF implemented a clever backwards compatibility scheme.
When the XDF TSR was loaded, a 3½” HD XDF floppy looked like a medium with 80 cylinders and 23 sectors per track. When the XDF TSR wasn’t loaded, the same medium looked like a teensy disk with just a few sectors total, usually containing some text file showing the XDF disk’s contents or a readme file.
The first cylinder of a XDF floppy used standard 512-byte sectors, only with 19 sectors per track rather than the standard 18. The format used 11 sectors per FAT (for the 3½” HD XDF variant, by far the one most widely used). It abused the fact that the second FAT is normally never read by DOS (as long as the first FAT copy is readable). The second FAT copy on a XDF disk therefore did not exist at all, but instead there was a small 8-sector backwards compatible image. The XDF sectors used high sector IDs and were thus not seen by standard DOS/BIOS. Track 0 (cylinder 0, head 0) was laid out as follows: 1 boot sector, 10 FAT sectors, 8 sectors of a backward compatible micro-disk. That is 19 sectors total.
The second track (cylinder 0, head 1) started with the one last FAT sector (for a total of 11), followed by the root directory (14 sectors) and finally the data area (4 sectors). Now there’s an obvious problem: The XDF disk claimed to have 23 sectors per track, therefore there should have been 9 data sectors (23 – 14) on the second track, but there were only 4. What happened with the missing 5 sectors? Easy: XDF marked the corresponding clusters in the FAT as bad.
To recap: XDF pretended that the first track (cylinder 0, head 0) contained the boot sector plus two FATs with 11 sector each (23 total). The next track (cylinder 0, head 1) then logically contained 14 sectors of root directory and 9 data sectors (23 total), out of which 5 were marked bad. In reality, the first track held 8 sectors of a compatible micro-disk and 11 sectors of XDF data (19 total); those were the XDF boot sector and the first 10 sectors of one FAT. The second track actually contained the last sector of FAT, 14 sectors of root directory, and 4 data sectors (19 total). Starting with cylinder 1, there was really the equivalent of 23 sectors (512 byte size) per track.
It is important to note that XDF images always contained full XDF-sized tracks, even for cylinder 0. Some sectors in a XDF image were thus not written to a disk, but the XDF image was uniform.
One pleasant side effect of the XDF scheme is that a typical XDF image looks like a proper DOS-formatted disk with 23 sectors per track and 3,680 512-byte sectors. Since the FAT filesystem uses logical sector numbers rather than C/H/S addressing, tools capable of processing FAT disk images can use XDF images without modification.
A further side effect is that when XDF images are used with virtualization software, DOS VMs may be able to access such images without the XDF TSR loaded.
Around the same time, Microsoft developed its own high-capacity format called DMF (Distribution Media Format). DMF was still a standard format on the BIOS level, with 80 cylinders and 21 sectors (interleaved) per track. Since DMF was optimized for distribution, it used larger clusters (and hence smaller FATs) and a minimal root directory with only 16 entries (one sector). This made DMF unsuitable as a general-purpose floppy format. At 1,680KB per disk, DMF also had noticeably lower storage capacity than XDF.
XDF’s closest cousin was the 2M format used by the eponymous DOS utility. The 2M format could store up to 1,972KB (or just over 2 million bytes) on a 3½” HD floppy. 2M used a very similar strategy utilizing varying size sectors and thus minimizing sector overhead. Additionally, 2M used 82 tracks rather than the standard 80. It is questionable whether the use of 82 tracks was fully compatible with all drives. At any rate, the 2M format does not appear to have been used for commercial software distribution.
It is not entirely clear whether 2M or XDF was developed first, and whether their development was independent. Both were most likely initially written in 1993. It is apparent that XDF was considerably more successful, having been adopted by IBM for use in DOS and OS/2 and for distributing IBM software.