Disabling Quick Edit Mode

Last week I decided to finally solve a minor annoyance that’s been pestering me since I switched my main development machine from Windows 7 to Windows 10 over a year ago. As it is with these things, a certain threshold of pain needs to be reached before delving into the unknown, because one never knows how many rabbit holes it will lead to.

The problem: Every now and then, I use the Open Watcom tools, often for cross-development to DOS or OS/2. When I do that, I tend to use the console-based tools, including the vi editor and wd debugger, in part because they work the same across platforms. Both tools can work with a mouse, and especially the debugger is just much productive when used with a mouse. On Windows 7, that works fine. Not so much on Windows 10, because instead of sending mouse events to the application, Windows tries to select text in the console window. Not helpful:

Quick Edit Mode makes the mouse unusable for a console application

Anyone who has fought with this problem probably knows the answer: Quick Edit Mode. When Quick Edit is enabled, mouse selects text from the console window, Enter copies, right-click pastes. Great for a shell, much less so in a mouse-driven text mode UI. On Windows 10, Quick Edit mode is on by default, and that is what caused me grief.

So, is Quick Edit Mode some kind of Windows 10 novelty? Far from it! Quick Edit has been with us for quite some time, the only change in Windows 10 was that it’s on by default. In other words, the Open Watcom tools have always had the same problem, but no one really noticed.

Programmatically Turning Quick Edit Mode Off

Turning off Quick Edit mode is slightly non-obvious. It can be done through the obvious enough SetConsoleMode API, but as Microsoft documents, it is accomplished through not passing the ENABLE_QUICK_EDIT_MODE flag and passing ENABLE_EXTENDED_FLAGS.

If that sounds a little fishy, it should—and it gets worse.

The Quick Edit Mode has been present in Windows NT ever since the original version 3.1 (1993):

Quick Edit Mode can be enabled in Windows NT 3.1

The ENABLE_QUICK_EDIT_MODE flag did not turn up in the NT platform SDKs until many, many years later, in the SDK for Windows Vista (2007). However, since Windows NT 3.51, the ENABLE_QUICK_EDIT_MODE was supported, but undocumented and hidden.

It’s confusing enough that a quick timeline may be in order:

  • 1993: Windows NT 3.1 released, supports Quick Edit mode. Applications cannot control Quick Edit mode through GetConsoleMode/SetConsoleMode.
  • 1995: Windows NT 3.51 adds ENABLE_QUICK_EDIT_MODE flag, allowing applications to disable it through SetConsoleMode. However, together with Insert Mode, the Quick Edit functionality is undocumented and partially hidden through a flag eventually called ENABLE_EXTENDED_FLAGS.
  • 2007: The Windows Vista SDK defines ENABLE_QUICK_EDIT_MODE, ENABLE_INSERT_MODE, and ENABLE_EXTENDED_FLAGS; the functionality is documented but in a woefully misleading way.
  • 2015: Windows 10 enables Quick Edit Mode by default, forcing innocent console applications to deal with the complication.

So how does this work in detail? Let’s go through the changes.

First of all, note that when changing the console mode, SetConsoleMode will fail with ERROR_INVALID_PARAMETER if it is passed any flags it does not understand. That complicates writing Win32 applications designed to work across NT versions (we will ignore Windows 9x here, as that has its own set of complexities in the Win32 console department).

In the times of Windows NT 3.1 and 3.5, things were simple. Applications could neither query nor set the Quick Edit Mode state. Starting with NT 3.51, things became much more interesting.

The ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE flags were present, but considered private, and guarded by ENABLE_EXTENDED_FLAGS. The semantics were so strange that people writing Microsoft’s documentation apparently didn’t understand them either.

It would perhaps be logical if ENABLE_EXTENDED_FLAGS were required to set ENABLE_QUICK_EDIT_MODE and/or ENABLE_INSERT_MODE through SetConsoleMode, and GetConsoleMode would return ENABLE_EXTENDED_FLAGS if either ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE is set.

But that’s not how it works at all. SetConsoleMode does not need ENABLE_EXTENDED_FLAGS to set ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE, but does require ENABLE_EXTENDED_FLAGS in order to clear both of the other flags.

GetConsoleMode is even stranger. It may return ENABLE_EXTENDED_FLAGS when neither ENABLE_QUICK_EDIT_MODE nor ENABLE_INSERT_MODE is set, but it may also return none of the three flags even when ENABLE_QUICK_EDIT_MODE and/or ENABLE_INSERT_MODE is in fact set.

In the Windows Vista (NT 6.0) SDK, Microsoft defined ENABLE_QUICK_EDIT_MODE, ENABLE_INSERT_MODE, and ENABLE_EXTENDED_FLAGS in wincon.h. The “new” flags were also documented, but the documentation is misleading. It claims that ENABLE_EXTENDED_FLAGS must be set when setting either ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE (wrong), and makes no mention that attempting to set these flags on old NT versions or on Windows 9x will cause SetConsoleMode to fail. It is quite possible that people writing the documentation did not understand just how tricky the API is.

As an aside, the Vista SDK also defined the ENABLE_AUTO_POSITION flag but did not document it at all. Testing shows that ENABLE_AUTO_POSITION behaves like ENABLE_QUICK_MODE and ENABLE_INSERT_MODE in that it is tied to ENABLE_EXTENDED_FLAG.

For the sake of completion, it should be mentioned that Windows 10 added and even documented a new ENABLE_VIRTUAL_TERMINAL_INPUT flag. To complete the confusion, this new flag does not depend on ENABLE_EXTENDED_FLAG.

How The Heck…

The behavior described above is probably straight from the How Not To Design an API book. How did Microsoft achieve that?

The ENABLE_EXTENDED_FLAGS bit is actually stored separately in the console state. To demonstrate what havoc that wreaks, let’s consider the following sequence of events:

  1. Call SetConsoleMode with ENABLE_QUICK_EDIT_MODE. That enables the Quick Edit Mode and internally sets ENABLE_EXTENDED_FLAGS whether or not the caller specified ENABLE_EXTENDED_FLAGS.
  2. Call GetConsoleMode. It will return ENABLE_EXTENDED_FLAGS and ENABLE_QUICK_EDIT_MODE flags both set.
  3. Call SetConsoleMode with none of ENABLE_EXTENDED_FLAGS, ENABLE_QUICK_EDIT_MODE, or ENABLE_INSERT_MODE flags set. That could be any application written using SetConsoleMode with flags publicly documented before 2007. This will not turn off ENABLE_QUICK_EDIT_MODE, but it will clear ENABLE_EXTENDED_FLAGS in the console state.
  4. Call GetConsoleMode. It will return neither ENABLE_EXTENDED_FLAGS nor ENABLE_QUICK_EDIT_MODE, even though Quick Edit Mode is in fact still enabled.
  5. Call SetConsoleMode with ENABLE_INSERT_MODE but none of ENABLE_EXTENDED_FLAGS or ENABLE_QUICK_EDIT_MODE. Now Insert Mode will be enabled, and Quick Edit Mode will be disabled.

Why would Microsoft do such a thing? Most likely there was some reason. The best guess is that some existing applications did not restore the original console state on exit and would turn off both Quick Edit Mode and Insert Mode when run. That could explain why ENABLE_EXTENDED_FLAGS was required to turn off Quick Edit Mode and Insert Mode.

But it does not explain why ENABLE_EXTENDED_FLAGS itself can be turned off. Especially when e.g. with NT 3.51, the default console state is ENABLE_QUICK_EDIT_MODE off, ENABLE_INSERT_MODE off, but ENABLE_EXTENDED_FLAGS on!

Using GetConsoleMode and SetConsoleMode safely

Given the above, it should be apparent that using GetConsoleMode and SetConsoleMode 100% safely is not possible. The reason for that is simple: An application cannot be certain that it queried the true console state on startup, and therefore cannot necessarily restore the state on termination.

Fortunately, in most cases ENABLE_EXTENDED_FLAGS will be set on NT 3.51 and later. When GetConsoleMode sees that flag set, it can be certain that the ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE bits are also correct. However, if ENABLE_EXTENDED_FLAGS is clear, the status of ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE is indeterminate.

As long as the application never sets any of ENABLE_EXTENDED_FLAGS, ENABLE_QUICK_EDIT_MODE, and ENABLE_INSERT_MODE, all will be well—and that may be the use case Microsoft was trying to cover. But once any of those bits are set, the state will be modified and cannot be safely restored.

In general, applications should strive to preserve the ENABLE_EXTENDED_FLAGS bit whenever it’s set because that is the only way to make sure GetConsoleMode can fully read the console state.

This mess was directly caused by Microsoft first hiding the undocumented functionality (why?) and later documenting it (because it’s too useful and necessary!). And needless to say, people have been wondering about it before, although no one appears to have tracked how the functionality evolved over time.

Wait, There’s Even More In Windows 10!

Windows 10 now makes things worse for console applications by intercepting the Ctrl+V key combination by default.

Much like the mess caused by Microsoft’s use of the forward slash, Ctrl+V was an idea stolen elsewhere, causing clashes down the road. As we all know, the Ctrl+C and Ctrl+V shortcuts were stolen from the Mac, except in the original they were Command+C and Command+V, and modern Macs have a separate Ctrl key so there is no clash. (To be fair, the shortcuts may have originally been Ctrl+C and Ctrl+V on Xerox PARC machines, but that’s likely not where Microsoft got them from.)

The PC had no Command key and Microsoft used the Control key in Windows. The trouble is that Ctrl+C is a very longstanding key combination used with terminals, going back to the 1960s, and it is in fact something that DOS supported since before Windows and NT supported from the beginning as well.

Windows users have been trained to use Ctrl+C and Ctrl+V for copy and paste, instead of the old IBM CUA Ctrl+Ins and Shift+Ins shortcuts, which perhaps require more finger-twisting but have the major advantage that they don’t clash with anything. (Note that generally the CUA shortcuts work just fine in Windows 10, including in console windows.)

And so we now have Microsoft repeating the stupidity that Linux users have long been cursing, with GUI terminal applications randomly deciding to intercept certain keys and combinations, leaving them unavailable to the software running inside the terminal. If a console application uses Ctrl+V, it may be out of luck on Windows 10 and never see that key combination. This is clearly a clash of expectations where users who never left the Windows island are convinced that Ctrl+V must always mean “paste”, and users who come from different backgrounds are upset that they can’t use Ctrl+V for what they need to.

Apparently hand-editing JSON files now considered to be the solution.

Happy Mousing

Whether Ctrl+V is working or not, I can now use the mouse again with the console wd debugger. Quick Edit Mode gets automatically turned off:

Quick Edit Mode off, Insert Mode preserved

The mouse works, left clicking triggers menus and selects symbols, right clicking brings up context menus:

Right-click brings up a context menu

I have managed to scratch this particular itch, but the rabbit hole that the fixing led to turned out to be really deep.

This entry was posted in NT, Undocumented, Watcom. Bookmark the permalink.

22 Responses to Disabling Quick Edit Mode

  1. Fernando says:

    I was ready to show my age and talk about IBM CUA Ctrl-Insert, Shift-Insert, Shift-Delete. Of course you know about that…

  2. Michal Necasek says:

    Sounds like Raymond Chen didn’t realize just how screwed up Get/SetConsoleMode was, either 🙂 If Windows didn’t do such a good job hiding Quick Edit Mode, those editors would have disabled it when they turned on mouse input. But no…

  3. Yuhong Bao says:

    Not all of them. For example, old DOS applications.

  4. starfrost says:

    I know this is unrelated, but I would like to say that Windows NT 3.1 build 196.1 (September 12, 1991) was just found and leaked online. It’s the earliest version of NT so far! https://archive.org/details/windows-nt-3.1-pdk-1.196-sept.-1991

  5. Michal Necasek says:

    That’s excellent! But wait, there’s a scan of the CD but no ISO image, just a bunch of files? That’s… weird.

  6. I had to check but didn’t Windows/386 also have quick select enabled by default?

    Speaking of things that have been hiding there but toggled the other way.

  7. Jistuce says:

    I have always used shift-del for cut and shift-ins for paste. They MAKE SENSE and wedged themselves immediately in my brain when I learned them. Ctrl-c and -v may be easier, but I don’t actually REMEMBER them.

    Congratulations on solving a weird friggin’ behavioral change.

  8. Michal Necasek says:

    I use the CUA shortcuts too, precisely because they don’t conflict with anything. And Ctrl-C is something that I often enough need as a terminal break. The only issue is that in a couple of cases, especially in browsers, Ctrl-Ins/Shift-Ins does not work. But it does work in the vast majority of contexts in current Windows 10.

  9. Richard Wells says:

    Rather amazing how Powershell has moved into the driver of expected behavior with the planned usage of frequent pasting of snippets into terminal and copying of results back out leading to the Windows 10 defaults. No matter what default is chosen, some user will be unhappy. I would have set up different command prompts so powershell gets QEM while applications ported from other OSes and console applications with clipboard support don’t have QEM.

    I think the smart context sensitive hot keys like having CTRL-C be copy when text is selected but break when no text is selected will cause frequent unwanted actions when the user thinks the system is in a different mode than the system thinks it is.

  10. Jeff says:

    The biggest issue I have with Ctrl-Ins/Shift-Ins is that I can never find the Ins key on my MacBook keyboard! 😉

  11. Michal Necasek says:

    Yeah that can be a problem, especially when running stuff in a VM on a MacBook. External USB keyboards can fix that depending on the usage. I guess Ctrl-Ins/Shift-Ins still won’t work in macOS though 🙂

  12. Richard Wells says:

    I have often thought that CUA shortcuts were often chosen to adversely impact competing products. The CUA exit keys (F3 and Alt-F4) were Multimate’s Center and Vertical Addition commands. So either Multimate for OS/2 would have to break with the CUA guidlines or break with the earlier UI creating a Wordstar 2000 problem which will provide a substantial advantage to IBM products.

    CTRL-C was a fairly widespread copy command even before the Mac. It could probably be traced to Wordstar’s CTRL-K block sequence with CTRL-KC being the copy. Simpler word processors like Sierra’s HomeWord for 1983 switched that to CTRL-C for the PC Jr or OpenApple-C for the Apple II (with some combo for the early keyboards lacking an OpenApple key). DTP word processors preferred the CTRL-C for center.

    At least the truly strange command sequences of Electric Pencil didn’t take hold.

  13. Michal Necasek says:

    I doubt IBM could have found key combinations that didn’t clash with any existing software…

  14. Stu says:

    “And so we now have Microsoft repeating the stupidity that Linux users have long been cursing, with GUI terminal applications randomly deciding to intercept certain keys and combinations, leaving them unavailable to the software running inside the terminal.”

    Linux terminals have de-facto standardised on using Ctrl+Shift+C/V for copy/paste. Sure, that just moves the potential clash to another combination, but at least it’s lesser used and doesn’t interfere with the longstanding Ctrl+C (although I’m sure I’m not the only person who forgets to press shift when intending to copy from time-to-time, thankfully usually harmlessly).

    Then there are those of us who sometimes use web-based terminals, where not only is Ctrl+C a problem, but also Ctrl+W, Ctrl+Q, etc.

    (Also, are you aware that your anti-spam measures don’t work in recent versions of Chrome unless the entire page is loaded over HTTPS? Might be worth moving the entire site to HTTPS by default…)

  15. zeurkous says:

    Me uses xterm(1) on a real UNIX, so it’s {copy,paste} w/ the pointer for
    me 🙂 Mebarely has a need for it though (w/ UNIX there are cleaner ways
    to exchange data between programs, and for shorter strings the 10-digit
    interface is just faster).

    On messy stuff: meprefers the {Ctrl,Shift}+Ins convention. The
    Ctrl+{C,V} convention in particular does not make any sense to me;
    indeed, as pointed out above, it conflicts w/ m$’ own idea of a
    terminal! “Interface consistency”? What’s that? 😡

  16. Michal Necasek says:

    Can you elaborate on what exactly the problem is? Also, the entire site should now be working over HTTPS.

    Yeah Ctrl+Shift+C/V is a sensible solution for consoles, but the Linux people just come up with something else crazy instead, like taking over the F10 key.

  17. Stu says:

    “Can you elaborate on what exactly the problem is? Also, the entire site should now be working over HTTPS.”

    It appears that the links in the RSS feed are still HTTP; when the “Really, I’m not a spammer.” tickbox is clicked on an HTTP page Chrome (and presumably other browsers) disallows the request and an “There was a problem with the request. Please try reloading the page.” alert is shown.

  18. Stu says:

    Also (sorry to double-post), after posting a comment, I’m returned an HTTP, not HTTPS url.

  19. Michal Necasek says:

    Thanks for the information. Sadly I’m running against the limitations of WordPress. I would not like to disable HTTP entirely because, well, try using HTTPS from an old OS. Nothing works. But HTTP still does. There are ways to deal with it on the WordPress side but none very attractive.

  20. wenson chen says:

    sorry but where is the code? Thank you

  21. Michal Necasek says:

    Which code exactly, the Windows NT code for the console? Or something else?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.