As part of a hobby project, I set out to reconstruct assembly source code that should be built with an old version of MASM and exactly match an existing old binary. In the process I learned how old MASM versions worked, and why programmers hated MASM. Note that “old versions” in this context means MASM 5.x and older, i.e. older than MASM 6.0.
The way old MASM works is relatively straightforward but its documentation often explains it very poorly or not at all. MASM is a two-pass assembler, and that indirectly explains almost everything about its quirks. This is different from more modern N-pass assemblers which automatically run multiple passes to resolve ambiguities.
The core of the problem is that MASM tries to be clever, but it’s not nearly clever enough. It is very questionable whether MASM’s cleverness is a solution or a problem; other assemblers are stricter, relying on programmers to resolve ambiguities. This perhaps puts slightly more of a burden on the programmer but results in more readable, consistent source code.
Most ambiguities result from the fact that like most assemblers, MASM does not require symbols to be declared before they’re referenced. In the first pass, MASM generates “provisional” code, making guesses about what unknown symbols are. At the end of the first pass, all symbols are known (if they’re not, the assembly will fail).
In the second pass, MASM applies what it learned in the first pass and generates the final object code. If the guesses made in the first pass turn out to be incompatible with the second pass, MASM will report the dreaded “phase error”. More about that later.
The crucial thing to understand is that in the first pass, MASM generates enough object code to resolve all offsets, i.e. at the end of the first pass, MASM will know for each symbol defined in the source code at which offset it will be located, because it will have determined how big all generated code and data is.
Now comes the “cleverness”. For example if MASM sees a JMP to an unknown label, it will assume a 16-bit near jump, i.e. a three-byte instruction. In the second pass, MASM may find out that the jump target is within +127/-128 bytes and generates a short jump, a two-byte instruction. Crucially, the third byte will be replaced by a NOP so that the instruction still effectively takes up three bytes.
MASM might also find out that the label is in a different segment, requiring a far jump. In that case, the jump instruction will not fit within three bytes and a phase error will result.
The programmer also has the option of writing ‘JMP SHORT xxx’ rather than ‘JMP xxx’. In that case, MASM will always generate a two-byte short jump, and possibly fail with an error if the target is not within the range of a short jump.
This is where those ‘NOP after JMP’ instructions come from. It is MASM (or perhaps some other assembler/compiler) turning a near jump into a short jump but not truly reducing the instruction size.
If the jump target is in another segment, the programmer may also write ‘JMP FAR PTR xxx’, telling MASM to generate a far jump and avoiding a phase error if the target label is in another segment but not yet known in the first pass.
Interestingly, there is at least one situation where the NOPs can be useful, especially because there does not appear to be any way (short of manually emitting opcodes) of telling MASM to generate a near jump when a short jump is possible. The BIOS component of DOS 1.x uses a CP/M inspired jump table where the “exported” interface is accessed by calling into some known base address plus an offset which is the function number times three (that being the JMP instruction size). The dispatch table looks like this:
DISPATCH: JMP FUNC0 JMP FUNC1 JMP FUNC3
This would be conceptually invoked as ‘CALL FAR PTR DISPATCH+(FUNC*3)’ because the dispatch table is assumed to consist of a sequence of near jumps. If MASM turns one or more of those jumps into short jumps but pads them with a NOP, the dispatch table will still work. If an assembler ends up producing only 2-byte jumps without padding, the dispatch table will go up in flames.
There are other situations where NOPs can be generated. For example ‘MOV DATA, 5’ will be byte or word sized, depending on the type of ‘DATA’. If ‘DATA’ has not yet been seen in pass 1, MASM will generate a 6-byte MOV instruction, big enough for a word-sized move. In pass 2, MASM may know that ‘DATA’ is a byte variable; in that case, the instruction will be reduced to 5 bytes, but again followed by a NOP.
This situation is exactly what ‘BYTE PTR’ can be used for. When ‘DATA’ ends up being a variable with a known size (byte or word), MASM will set the MOV instruction size based on that and not complain. The programmer can write ‘MOV BYTE PTR DATA, 5’ to prevent MASM from guessing the instruction size, or to override what MASM would do.
There are other situations where MASM can be unpleasantly clever. Remember those ASSUME directives? They are quite important.
Consider a situation where everything (code and data) is in a single segment named CODE, and the source file contains an ‘ASSUME CS:CODE’ directive but not more. If you write ‘MOV BYTE PTR VAR,1’, you may get a phase error depending on whether ‘VAR’ has been seen or not. Why is that?
MASM is clever and if it knows that VAR is in the code segment, it will automatically generate a CS segment override. But if it has not yet seen ‘VAR’ in the first phase, it won’t leave room for the prefix, and in the second phase it’ll report a phase error when it figures out that a segment prefix is needed but there’s no room for it.
Again, an explicitly coded segment prefix (e.g. ‘MOV BYTE PTR CS:VAR,1’) avoids this situation. Programmers need to keep this cleverness in mind because if they forget to say ‘ASSUME DS:CODE’ (assuming the DS segment register does in fact point to the CODE segment containing the data items), MASM will helpfully generate unnecessary CS segment overrides.
Perhaps the most questionable MASM feature is guessing that when possible, a label refers to the value at the label’s address. Thus ‘MOV AX,WORD PTR [VAR]’ can be shortened to ‘MOV AX,[VAR]’, because MASM reasonably assumes that moving to AX means a word-sized operation, but the same result can also be achieved with just ‘MOV AX,VAR’. This leads to a confusing syntax where brackets sometimes must be used as a dereferencing operator and sometimes they’re optional.
I’m not sure what problem Microsoft was trying to solve by making the syntax so vague. It is clearly inconsistent because ‘MOV AX,BX’ and ‘MOV AX,[BX]’ are two different things, yet ‘MOV AX,VAR’ and ‘MOV AX,[VAR]’ is (often) the same. It’s the kind of syntactic sugar that’s bad for you.
It’s even worse because there are differences between MASM versions in this area. For example, MASM 1.10 will assemble ‘MOV AX,VAR’ the same way regardless of how VAR is defined. But IBM MASM 2.0 accept it if only we have ‘VAR DW 0’ and report an error (“Operand types must match”) if ‘VAR DB 0’ is seen instead. MASM 5.10A flags the situation as a warning (again “Operand type must match”) and produces the same code as old MASM 1.10. Microsoft appears to have gone back and forth on this, probably because the original MASM behavior was unhelpfully vague but too much existing code relied on it.
Some other assemblers (e.g. SCP’s ASM) have unambiguous syntax and ‘MOV AX,VAR’ will correspond to MASM’s ‘MOV AX,OFFSET VAR’; if dereferencing is desired, it must be made explicit with brackets.
Much of this used to be documented in old MASM manuals, like the one here. For whatever reason, newer MASM documentation (e.g. MASM 5.0 User’s Guide) does not bother explaining these seemingly small but very important details which are tied to MASM’s two-pass processing. The behavior is not difficult to grasp once the basics of MASM operation are understood, but without that, MASM may appear to behave in a very arbitrary and capricious manner.