AVX support disrupts WoW64 debugging

Sometimes, the old and the new intersect in unexpected ways. After upgrading to a Sandy Bridge based system (Core i7) and Windows 7 SP1 64-bit some time ago, I noticed that debugging exceptions in 32-bit user programs didn’t quite work right. Recently the issue has been brought to my attention again and I had to do some digging.

The symptom of the problem is that when a debugger stops on a second chance exception, the context of the (32-bit) process being debugged is damaged and the debugger stops in a bogus location, namely ZwRaiseException or NtRaiseException (same thing with two different names). That is a huge problem when the debugged process handles certain exceptions. The debugger can’t stop on every first chance exception, because those occur in the normal flow of execution. Yet if a real bug pops up and causes an unhandled exception, the debugger will not show where the problem is!

The issue is 100% reproducible… but only on some systems. The ingredients are: 64-bit Windows 7 SP1, a 32-bit process being debugged, and a recent CPU. The process doesn’t matter, the type of exception doesn’t matter, the debugger doesn’t matter (both 64-bit and 32-bit debuggers are affected). Software installed on the system doesn’t matter beyond the base OS.

Problem

A log of a short debugging session with WinDbg/NTSD illustrates the problem:

(7b8.4d4): WOW64 breakpoint - code 4000001f (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
ntdll32!LdrpDoDebuggerBreak+0x2c:
77000fab cc              int     3
0:000:x86> g (7b8.4d4): Integer divide-by-zero - code c0000094 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for crashm.exe
crashm!main+0x10:
00401020 f7f9            idiv    eax,ecx                           <----- correct
0:000:x86> k
ChildEBP RetAddr
0018ff48 004010e5 crashm!main+0x10 [crash.c @ 7]
0018ff88 75dd339a crashm!mainCRTStartup+0xb4
0018ff94 76f99ef2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 76f99ec5 ntdll32!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll32!_RtlUserThreadStart+0x1b
0:000:x86> g
(7b8.4d4): Integer divide-by-zero - code c0000094 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4                             <----- wrong
0:000:x86> k
ChildEBP RetAddr
0018fa7c 76f7014d ntdll32!ZwRaiseException+0x12
0018fa8c c0000094 ntdll32!KiUserExceptionDispatcher+0x29
WARNING: Frame IP not in any known module. Following frames may be wrong.
0018ff48 004010e5 0xc0000094
0018ff88 75dd339a crashm!mainCRTStartup+0xb4
0018ff94 76f99ef2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 76f99ec5 ntdll32!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll32!_RtlUserThreadStart+0x1b
0:000:x86> g
(7b8.4d4): Unknown exception - code 00000000 (first chance)
(7b8.4d4): Unknown exception - code 00000000 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4
0:000:x86> g
WARNING: Continuing a non-continuable exception
(7b8.4d4): Unknown exception - code 00000000 (first chance)
(7b8.4d4): Unknown exception - code 00000000 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4

The first chance exception is reported correctly, but as mentioned earlier, stopping on first chance exceptions may not be feasible. The second chance exception is erroneously reported in ZwRaiseException, which is completely bogus. The stack is damaged enough that execution cannot be continued at all, which compounds the problem.

It’s worth pointing out that on the damaged stack, the first bogus frame (going from bottom to top as printed) shows the location as 0xC0000094. That certainly does not look like a valid address, but it looks exactly like the exception code of a division by zero. If ever something looked like a coincidence… this is not it.

Yet that only shows Windows is doing something wrong, not what or why.

For reference, this was the test program:

#include <stdio.h>

int main( int argc, char **argv )
{
    int i;

    i = 33 / (argc - 1);
    return( i );
}

The testcase was written in a slightly convoluted way to defeat compiler optimizations and ensure a division error will happen at run-time.

The compiler and linker used doesn’t matter, and the bug can be reproduced with a tiny assembler program as well.

Clues

While investigating the problem, I naturally tried to reproduce it in a VM, as it would have been easier to analyze the issue in a virtual environment. But… the problem didn’t happen in a VM. The Windows 7 version was exactly the same, the host CPU was the same, the software used was the same. But what failed on the host stubbornly worked in a VM, no matter how the host and guest were configured (memory size, number of CPUs, etc.).

I knew that was a clue, but couldn’t interpret it. One possibility was that virtualization masked some new-fangled CPU feature and that hid the problem. That could explain why the bug showed up on Intel Core i7 and i5 systems, but not Intel Core 2 or AMD Phenom machines.

A few days later, a coworker mentioned in passing that the AVX instruction set requires a larger memory area to save the state. That got me wondering, because in earlier investigation I noticed that on the broken systems, there seemed to be more data on the exception stack.

Microsoft documents that AVX is indeed supported and enabled by default on Windows 7 SP1. AVX state is indeed saved when exceptions are dispatched. Hmm, could that be it?

To confirm or disprove the theory, I’d need a way to disable AVX. That cannot be done in the system’s BIOS. But hey… Microsoft documents how to disable AVX in Windows as a workaround for a completely unrelated problem. The xsavedisable switch is not properly documented, but maybe it works…

Solution

I executed

bcdedit /set xsavedisable 1

from an elevated command prompt and rebooted the system. Re-ran the debugger testcase and lo and behold, now exception handling works properly!

The problem is definitely a bug in Windows 7 SP1 on AVX-enabled systems; however, it seems to be specific to the debug support in the WoW64 component. A 32-bit version of Windows 7 SP1 behaves correctly, and so does debugging of 64-bit processes. A 32-bit process can handle its own exceptions correctly when running under WoW64, it’s only when a 32-bit process is run under a debugger within WoW64 that trouble strikes.

I can only guess that with AVX on, the kernel saves AVX context on the exception stack in the user process (after continuing a first chance exception), but WoW64 does not expect the additional data when a debugger is attached and messes up the stack. That of course does no good and the user process is toast.

For now, disabling AVX is an acceptable workaround for me. Hopefully the bug will be eventually fixed and it will be possible to both use AVX and debug 32-bit processes properly.

This entry was posted in Windows, x86. Bookmark the permalink.

18 Responses to AVX support disrupts WoW64 debugging

  1. Jiayi Lee says:

    Hi,
    Not related to this post, but when do you continue DOS history?

  2. michaln says:

    When I’m happy with what I’ve researched and written :)

  3. EdL says:

    Thank you so much, you’ve just saved my work a lot (more) trouble trying to figure out why we can’t debug crashes on our new PC’s.

    Have you submitted this as a bug report to Microsoft?

  4. michaln says:

    I submitted an informal report which did get to the right people, but I don’t know if and when a fix might be forthcoming.

  5. Peter says:

    Thank you for this timely information! It has saved me a lot of frustration.

  6. Dave says:

    I have a similar setup to you (Win7 SP1 64-bit on Core i7) but I’m not able to reproduce this issue. I do, however often get minidumps from clients in which the excepting thread’s callstack unwinds to ZwRaiseException. Any word on if MS has patched in a fix for this (just wondering since you said you submitted an informal report). Working on the assumption that perhaps I have the patch but some users do not…

  7. michaln says:

    I don’t think there’s a patch available and I haven’t heard from Microsoft. Maybe I need to retry and see if the problem is still there…

    I believe the problem should be reproducible on any AVX system, assuming AVX is enabled. And of course only for 32-bit processes.

    From what I can tell, there are many other reasons for getting a crash in ZwRaiseException though.

  8. Anonymous says:

    I wonder if that’s the same thing as described here: GetThreadContext() may return stale contents

  9. michaln says:

    No, definitely a different problem. What the link describes has nothing to do with AVX, while what I ran into is specific to systems with AVX support.

  10. Pingback: When Even Crashing Doesn’t Work | Random ASCII

  11. Pingback: Are you still there? | Windows Live space

  12. bik says:

    michaln you are cool Cool COOL!

    This issue has been driving me insane on my new FX machine…
    This post should sort to the top on Google for “zwraiseexception” but it doesn’t and a month ago I ignored it anyway due to the “OS/2 Museum”.

    Confoundingly VS2010 & 12 didn’t exhibit this sitting on the zwraiseexception, which hastened my port from Open watcom to VC++ but it kept bugging me!

  13. michaln says:

    I believe the Visual Studio debuggers tend to intercept first chance exceptions or don’t intercept the exception at all. In fact this is probably the #1 reason why the bug even crept in. I didn’t analyze Visual Studio (since I don’t use it), but WinDbg is definitely affected.

  14. Martin Ba says:

    Do you have done any explicit tests wrt. to the behaviour of the MiniDumpWriteDump function? Does it work properly? You seem to imply so: “… A 32-bit process can handle its own exceptions correctly when running under WoW64, it’s only when a 32-bit process is run under a debugger within WoW64 that trouble strikes …”

    I have found the following, but am unsure whether that holds always:
    * MiniDumpWriteDump works from within the global UHEF (installed by SetUnhandledExceptionFilter) in the same process
    * MiniDumpWriteDump works from another process also (say when I spawn another process from my UHEF and invoke it on the crashed process)
    * I tried with SysInternals’ procdump.exe tool and the -e switch – and the dump generated this way has a messed-up stack — I guess this would be due to the fact that procdump.exe attaches as a debugger to the [-e]-monitored process.

  15. michaln says:

    Sorry, I didn’t do any experiments with MiniDumpWriteDump. But what you said would be consistent with my observations.

  16. Pingback: Should This Windows 7 Bug be Fixed? | Random ASCII

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>