The CP/M compatible interface in DOS was initially documented, later forgotten, and then re-discovered every once in a while.
In 1989, John Switzer described parts of the CALL 5 system call interface mechanism in a slightly hysterical article as a “back door” into DOS and called it a security risk, despite the fact that it was a compatibility interface very deliberately maintained in every version of DOS. However, the article correctly pointed out that the CALL 5 interface bypassed INT 21h hooks. In theory, that could have been used for nefarious purposes; then again, worrying about that on the DOS platform was like being gravely concerned that cold wind could get in through a small crack in the window in a house without a roof. DOS simply wasn’t a secure foundation and patching a tiny hole couldn’t fix the fact that any program running on DOS effectively owned the entire system.
On page 5 of Undocumented DOS (first edition, 1990), Andrew Schulman wrote: “For example, in DOS 1.0 Microsoft documented the fact that, in addition to using INT 21h, applications could call operating system functions with a CALL 5 instruction. This DOS holdover from CP/M was used by several then important programs, including WordStar.” That suggests the CALL 5 interface had at one point been officially documented for Microsoft’s DOS, not just SCP’s 86-DOS. It is unknown whether IBM’s DOS Technical References documented the CALL 5 interface or not.
Incidentally, the CALL 5 interface does not appear to be mentioned in the second edition of Undocumented DOS. However, the second edition is very significantly different from the first, to the point where it’s almost misleading to call it the second edition.
At any rate, WordStar is mentioned as one of the applications which actually used the CALL 5 interface. That is not so surprising, because WordStar had been originally written as a CP/M application.
In 2000, the CALL 5 interface and WordStar made another appearance, although it’s not clear whether Joel Spolsky’s article on chicken and egg problems supports the link or not. It does say that WordStar was ported to DOS with almost no changes; unfortunately, the article is so riddled with wildly inaccurate and just plain wrong claims that it’s difficult to take it very seriously. Calling XENIX an 8-bit version of UNIX is either a joke gone wrong or amazingly ignorant—when was XENIX 8-bit, and what would be the point of running an 8-bit OS on a 16-bit CPU? The article also says: “In fact, WordStar was ported [from CP/M] to DOS by changing one single byte in the code.” It’s unclear how that would have worked when DOS was designed for easy porting of CP/M applications running on the 8-bit 8080 CPU—when DOS was written, 16-bit CP/M didn’t even exist, and in fact that was the whole reason why DOS (née 86-DOS, née QDOS) had been written in the first place!
As a side note, Spolsky’s claim that at the release of the IBM PC, DOS competed against XENIX and UCSD p-System is also very wide off the mark. The three operating systems initially announced for the IBM PC were DOS, UCSD p-System, and CP/M-86 (rather than XENIX, which came much later). But more importantly, neither p-System nor CP/M-86 were available at the launch, and for more than half a year DOS had no competition whatsoever.
Other curiously inaccurate and misleading statements have been made in relation to the CALL 5 system call interface. For example, the otherwise excellent Undocumented PC by Frank van Gilluwe mentions that interrupt vectors 30h and 31h in fact contain a 5-byte far jump instruction, but then goes on to say that “the new DPMI service handler in protected mode uses interrupt 31h, which destroys the last byte of this 5 byte far jump.” That is a self-contradictory statement, since the protected mode interrupt vector table (IVT) is completely different and separate from the real-mode IVT!
Reading the DPMI specification, it is clear that INT 31h is protected-mode only; determining DPMI presence and switching to protected mode is done without any use of INT 31h. Indeed, even when a DPMI server like Quarterdeck’s QDPMI or the 386MAX DPMI server from Qualitas is installed (or for that matter, Windows 3.x in Enhanced mode), the far jump at 0:C0h is undisturbed.
But back to WordStar. According to the unofficial WordStar history, the DOS version of WordStar 3.0 (released in April 1982) was converted from CP/M with minimal modifications. It is plausible that it used the CP/M style CALL 5 system call interface. Unfortunately, without a copy of the DOS version of WordStar 3.0, this cannot be confirmed.
By version 3.3, the DOS version of WordStar had already been converted to use the INT 21h system call interface, although it was still limited to CP/M functionality—most importantly, no support for directories.
It is plausible that WordStar 3.0 may have been the one important application which forced the development of the A20 gate and the associated nonsense, but it is unlikely that it was the only reason. If anyone knows of other significant software which provably required 8086-style address wraparound, either directly or (as was likely the case for WordStar 3.0) indirectly, please leave a comment.