Windows - using the debugger

[Paul Adams]

Debugging Windows 101

Foreword
This is a brief tutorial on initial analysis of "crash dumps" which the enthusiast may find of interest if they encounter situations with hanging/crashing apps, bluescreens or a hung OS.

It is NOT intended to be an in-depth complete solution to every problem scenario, but a "point in the right direction" to maybe give some clues as to what might be the cause, for those that want to know more.

The following procedures are "one-off" actions which can prepare the system to create data that is useful to help pinpoint a problem - they don't "fix" anything in themselves.


To make any sense of the dumps, or in some cases to even create them, the Debugging Tools for Windows needs to be downloaded and installed first:
http://www.microsoft.com/whdc/devtoo...g/default.mspx


Common Procedures

1. Configure page file settings to ensure a kernel memory dump can be produced

- Right-click (My) Computer, click Properties
- (Vista only) Click 'Advanced System Settings' and OK the UAC prompt
- Select the 'Advanced' tab and click the 'Settings' button in the 'Performance' section
- Select the 'Advanced' tab and click the 'Change' button in the 'Virtual Memory' section

Recommended option is to select 'Automatically manage paging file size for all drives'

If you do manually set the size and location of the page file, ensure that the size of page file on the system volume (where Windows resides) is "big enough to hold a copy of all kernel memory" (this is why it's easiest to let Windows determine what it needs, this is something that can change from system to system - and also why it's not necessarily a good idea to run without a page file, or with a tiny one on a separate volume)

----

2. Configure memory dump option to request a kernel memory dump on a bugcheck

- Right-click (My) Computer, click Properties
- (Vista only) Click 'Advanced System Settings' and OK the UAC prompt
- Select the 'Advanced' tab and click the 'Settings' button in the 'Startup and Recovery' section
- From the 'Write debugging information' drop-down list, select 'Kernel memory dump'
- Check the path for the dump file is "%SystemRoot%\MEMORY.DMP"

NOTE:
In order to produce a valid dump of kernel memory when a bugcheck occurs, the following must be true:
> the page file on the system drive (where Windows resides) must be large enough to hold it
> there must be enough free space on the system drive to make a copy of the dump file from the page file to %SystemRoot%\MEMORY.DMP
> the disk holding the system volume must be accessible at the time of the bugcheck (i.e. 'drive disappears' = no dump)
> any ASR (Automated System Recovery) feature must not hard-reboot the system during the dump (only really applicable to servers)

----

3. Configure "Crash on CTRL-scroll" to allow manual bugchecks

** THIS FEATURE REQUIRES A PS/2 KEYBOARD (ON EVERY OS OTHER THAN WINDOWS SERVER 2003) **

- Launch regedit.exe
- Drill down to the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters
- Add a registry value with the following details:
Name: CrashOnCtrlScroll
Type: DWORD
Data: 1
- Reboot

This allows you to manually cause a bugcheck by holding down the right CTRL key and hitting Scroll Lock twice
It is useful for creating a memory dump from systems that appear to be hung

For more details see http://support.microsoft.com/kb/244139

----

4. Configure debugging symbols path environment variable

- Right-click (My) Computer, click Properties
- (Vista only) Click 'Advanced System Settings' and OK the UAC prompt
- Select the 'Advanced' tab and click the 'Environment Variables' button at the bottom
- Click the 'New' button in the 'System variables' section, enter the following values:
Variable name: _NT_SYMBOL_PATH
Variable value: srv*c:\symbols*http://msdl.microsoft.com/download/symbols

This allows tools that use debugging symbols to be able to locate them - if they are locally stored then these are used, otherwise the symbol server is queried and they are downloaded and stored locally for later reference

What are 'symbols'?
These are (often large) files which are associated with files containing executable code (drivers, DLLs, EXEs, CPLs, etc.) and provide information such as function names, variable names, locations and sizes of data structures, so that the debugger isn't just trying to deal with a collection of 0's and 1's

The above symbol path can be changed if c:\symbols is not a useful or valid place for a particular system, but bear in mind this folder will accumulate symbols over time if a debugger is used frequently for dump analysis and it will not get cleared down

Every build of a particular file, if a change is made, will create a unique symbols file, so you will see multiple .PDB files for the same binary as hotfixes or service packs are applied to an OS, or if you look at dumps from other users' systems

[Paul Adams]

Debugging Scenarios

Assuming you have installed the Debugging Tools for Windows and have a scenario you want to take a look at in more detail, here are some quick definitions and and overview of how we get to the point where the debugger can start to be used...

---

Process Crash

This is where you typically get the message along the lines of "This program has performed an illegal operation and will be shut down" at random points or when performing a specific action

This is an unhandled exception in user-mode - not critical to the system as a whole

- Start a command prompt, navigate to the folder where you installed the Debugging Tools
- Start the program which crashes (for the sake of the example I will call it CRASHAPP.EXE)
- Switch to the command prompt window and run the following command:
ADPlus -crash -ctcf -pn CRASHAPP.EXE -o C:\Dumps

ADPlus switches explained:
> -crash = the debugger is to watch the process until it throws an exception, then attach and create a dump before exiting it
> -ctcf = create a complete memory dump of the process with as much information as possible
> -pn = the name of the process to which the debugger should attach
> -o = the output folder where the user-mode dumps are to be generated, in unique sub-folders (under C:\Dumps in the above example)

If the program crashes immediately on startup, however, then you can use the debugger to launch it and perform a debug directly on the program rather than rely on a "post mortem" dump file

If there are multiple instances of the same process name, then the process ID (PID) can be used to identify it and the switch '-p' used instead of '-pn' and the process name - PIDs can be seen through Task Manager, TASKLIST.EXE, Process Explorer, a kernel debug, and no doubt hundreds of other methods

---

Process Hang

A lot of people report a hang as a crash and vice versa, but the symptoms and root causes are very different.
A HUNG application is waiting on an event that is either never going to happen, or is taking a long time, and in the meantime it appears unrespsonsive, maybe without the window updating ("repainting").

- Start a command prompt, navigate to the folder where you installed the Debugging Tools
- Start the program which crashes (for the sake of the example I will call it HANGAPP.EXE)
- When the process HANGAPP.EXE hangs, switch to the command prompt window and run the following command:
ADPlus -hang -pn HANGAPP.EXE -o C:\Dumps

ADPlus switches explained:
> -hang = the debugger is to passively attach to the process and take a snapshot of its memory space without causing it to exit
> -pn = the name of the process to which the debugger should attach
> -o = the output folder where the user-mode dumps are to be generated, in unique sub-folders (under C:\Dumps in the above example)

As with a process crash, if there are multiple instances of the same process name, then identify the correct PID and use it instead, with '-p'

Often it can be useful to take 2 or 3 dumps of a hung process at 10-second intervals, to see if there are differences between them to indicate if it is just very, very slow to update or is "deadlocked" - just run the same command again as the sub-folder name for the dump contains the time you ran the command

---

System Crash

Bluescreen, bluescreen of death (BSOD), bugcheck, crash - doubtless there are many other names given to it too

This is an unhandled exception in kernel-mode - as every process on the system shares kernel space, if something bad happens here then it can be bad news for everyone and to prevent further problems it shuts up shop
And it is FURTHER problems, as we have detected something unexpected but that is not to say the kernel isn't a bullet-ridden corpse by this point

Follow common procedures 1 & 2 to make Windows produce something that can be reviewed to provide some clues

When the OS crashes (bugchecks), the kernel memory is dumped into the page file and the system restarts
On starting Windows, the page file is found to contain a crash dump and it is copied to %systemroot%\MEMORY.DMP

---

System Hang

If Windows becomes unresponsive but does not bugcheck to produce a dump file, and common procedures 1, 2 & 3 have been followed beforehand, you can create a "STOP 0xE2" memory dump by holding down the right CTRL key and hitting SCROLL LOCK twice - this then goes through the same process as the "System Crash" description above

In the event of a hung system, instead of creating a crash dump and then analysing it offline, it may be possible to "live debug" the computer through a serial, USB or firewire cable from a separate computer (clearly the computer cannot suspend itself in order to debug itself - the debugger process would be suspended at the same time, and there would be no routines available to copy symbols around)

Live debugging can alsobe used for scenarios where the dump file is not being created on a bugcheck - possibly due to a hardware problem or the disk controller decides the disk is no longer present

[Paul Adams]

Analysing Dumps

Okay, so you have your recurring problem, you prepared the system to generate the useful data and now you have something to look at... now what?

In order to make sense of the raw data in the dumps, you need the relevant symbols for (ideally all) the modules containing code in it - see common procedure 4

Now we're ready to actually load the dump file for analysis...

- Launch WinDbg
- Click File
- Click 'Open Crash Dump', browse to the .DMP file and double-click it

The header of the dump file will let the debugger know what type of dump it is and what symbols are essential to even make a start - i.e. for a dump of EXPLORER.EXE, at the very least we will need symbols for this file

WinDbg may therefore look like it isn't doing a great deal initially, whereas it is in fact checking for (and downloading) symbols it needs

There is a bar at the bottom for you to enter commands into the debugger once it has done the necessary downloading

Commands I typically enter when debugging, so I know what the debugger is doing when it appears to be idle:

!sym noisy
- This turns on "noisy" symbol loading so you can see where it is getting symbol files from (the symbols files can be large as take a while to download sometimes)

0:007> !sym noisy
noisy mode - symbol prompts on

.reload /f
- This forces a reload of symbols for all modules in the dump file, even those which have not yet been downloaded but might trigger a download in the middle of your debugging session

0:007> .reload /f
Reloading current modules
.
SYMSRV:  Q:\Symbols\Script Checker Interceptor.pdb\527CFF7666C84738B5BDD72E225ADC321\Script Checker Interceptor.pdb not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/Script Checker Interceptor.pdb/527CFF7666C84738B5BDD72E225ADC321/Script Checker Interceptor.pdb not found
DBGHELP: Script Checker Interceptor.pdb - file not found
DBGHELP: O:\out_Win32\Release\Script Checker Interceptor.pdb - file not found
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for scrchpg.dll - 
DBGHELP: scrchpg - export symbols
.
SYMSRV:  iexplore.pdb from http://msdl.microsoft.com/download/symbols: 86673 bytes - copied         
DBGHELP: iexplore - public symbols  
         Q:\Symbols\iexplore.pdb\3544BAF610664EC3B420AF05F04F589B2\iexplore.pdb
.
SYMSRV:  ieframe.pdb from http://msdl.microsoft.com/download/symbols: 2160324 bytes - copied         
DBGHELP: IEFRAME_4df0000 - public symbols  
         Q:\Symbols\ieframe.pdb\4A4E76B2DB544787AD0633C6BA8271CE2\ieframe.pdb
.
SYMSRV:  Q:\Symbols\r3hook64.pdb\993432011C36406A93CDDA0CADB66DB41\r3hook64.pdb not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/r3hook64.pdb/993432011C36406A93CDDA0CADB66DB41/r3hook64.pdb not found
DBGHELP: r3hook64.pdb - file not found
DBGHELP: O:\out_win32\Release\r3hook64.pdb - file not found
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for r3hook.dll - 
DBGHELP: r3hook - export symbols
.
DBGHELP: IEFRAME - public symbols  
         Q:\Symbols\ieframe.pdb\4A4E76B2DB544787AD0633C6BA8271CE2\ieframe.pdb

!analyze -v
- Most people start here, as it gives you a quick overview of an exception (assuming it wasn't a manual dump) and in some cases a "probably caused by" guess (and it IS a guess, in some cases very accurate and in others completely wrong)

!vm
- Presents a summary of virtual memory along with a list of the processes running at the time of the crash

lm ft
- "List Modules" (with file location & timestamp fields) to get a complete list of module information to look for old drivers or those known to be versions which are not stable

0:007> lmft
start             end                 module name
00000000`003e0000 00000000`00405000   scrchpg  scrchpg.dll  Fri Mar 09 17:46:53 2007 (45F18F7D)
00000000`00c40000 00000000`00cdb000   iexplore C:\Program Files (x86)\Internet Explorer\iexplore.exe Tue Jun 26 03:50:59 2007 (46807103)
00000000`04df0000 00000000`053bb000   IEFRAME_4df0000 IEFRAME.dll  Tue Jun 26 04:50:31 2007 (46807EF7)
00000000`10000000 00000000`10010000   r3hook   r3hook.dll   Fri Mar 09 17:51:14 2007 (45F19082)
00000000`724c0000 00000000`72a8b000   IEFRAME  IEFRAME.dll  Tue Jun 26 04:53:52 2007 (46807FC0)
00000000`72e60000 00000000`72ea5000   SCHANNEL SCHANNEL.dll Tue Jun 19 04:07:52 2007 (46773A78)
00000000`72eb0000 00000000`72ed1000   NTMARTA  NTMARTA.dll  Thu Nov 02 10:43:55 2006 (4549BDDB)
00000000`72fe0000 00000000`73010000   MLANG    MLANG.dll    Thu Nov 02 10:40:07 2006 (4549BCF7)
00000000`73060000 00000000`7320a000   gdiplus  gdiplus.dll  Thu Nov 02 10:38:55 2006 (4549BCAF)...

lm vm MODULENAME
- Display verbose information on module MODULENAME, often the author of the module and the version strings are in here

0:007> lmvm r3hook
start             end                 module name
00000000`10000000 00000000`10010000   r3hook     (export symbols)       r3hook.dll
    Loaded symbol image file: r3hook.dll
    Image path: r3hook.dll
    Image name: r3hook.dll
    Timestamp:        Fri Mar 09 17:51:14 2007 (45F19082)
    CheckSum:         0001316A
    ImageSize:        00010000
    File version:     6.0.2.621
    Product version:  6.0.2.621
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Kaspersky Lab
    ProductName:      Kaspersky Anti-Virus
    InternalName:     R3HOOK
    OriginalFilename: R3HOOK.DLL
    ProductVersion:   6.0.2.621
    FileVersion:      6.0.2.621
    FileDescription:  Kaspersky Anti-Virus Ring 3 Hooker
    LegalCopyright:   Copyright © Kaspersky Lab 1996-2007.
    LegalTrademarks:  Kaspersky™ Anti-Virus ®  is registered trademark of Kaspersky Lab.

kv 50
- show the last (up to) 50 stack entries for the current thread ('kv' by itself might show only the top portion of a stack if it's large)

0:000> kv 50
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0017dca8 00000000`76f6ed73 : 00000000`00000001 00000000`7708ead9 00000000`00100246 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0xa
00000000`0017dcb0 00000000`7708e96d : 00000000`00000001 00000000`0017ded0 00000000`00000000 00000000`00000000 : kernel32!WaitForMultipleObjectsEx+0x10b
00000000`0017ddc0 00000000`7708e85e : 00000000`00000001 00000000`0024c260 00000000`00279f50 00000000`00000000 : USER32!RealMsgWaitForMultipleObjectsEx+0x129
00000000`0017de60 000007fe`f92b8fdf : 00000000`002899f0 00000000`0028bb00 00000000`ffffffff 000007fe`f92c3ad9 : USER32!MsgWaitForMultipleObjectsEx+0x46
00000000`0017dea0 000007fe`f92ad845 : 00000000`00000001 00000000`0024c260 00000000`00279f50 00000000`ffffffff : IEUI!CoreSC::Wait+0x4f
00000000`0017def0 000007fe`f5d41b7d : 00000000`00275260 00000000`00000000 00000000`00000000 00000000`00000000 : IEUI!WaitMessageEx+0x75
00000000`0017df30 000007fe`f5d7ccf7 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0024c260 : IEFRAME!CBrowserFrame::FrameMessagePump+0x1d0
00000000`0017dfa0 000007fe`f5d7dfa4 : 00000000`00000001 00000000`00275260 00000000`00000000 00000000`00272300 : IEFRAME!BrowserThreadProc+0x47
00000000`0017dfd0 000007fe`f5d7debf : 10e1f12f`0000000a 00000000`0024c260 00000000`001fc7c0 00000000`00000001 : IEFRAME!BrowserNewThreadProc+0x92
00000000`0017e010 000007fe`f5d7d6e8 : 00000000`0024c260 00000000`0024c260 00000000`00000001 00000000`00000000 : IEFRAME!SHOpenFolderWindow+0x202
00000000`0017f0c0 00000000`00a8d3d2 : 00000000`001fc7c0 00000000`00000001 00000000`00000001 00720074`00620027 : IEFRAME!IEWinMain+0x369
00000000`0017f370 00000000`00a91b6e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : iexplore!wWinMain+0x35a
00000000`0017f810 00000000`76f6cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : iexplore!StringVPrintfWorkerW+0x272
00000000`0017f8d0 00000000`7718c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0017f900 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

The analysis of a dump depends very much on the type of dump (user mode or kernel mode), whether it was a crash or a hang, and if it was a crash what the exception code was - there isn't a magic "!what_went_wrong" instruction to solve everything for you.

Without symbols, debugging is almost impossible - so if MyFancyApp.exe by MadeUpCompany keeps crashing randomly then chances are only the developers at MadeUpCompany could provide any useful diagnosis


Crash dumps I find generally easier to diagnose than hang dumps, as there is a point from which we can work backwards to work out how we might have arrived at the exception

That said, hang dumps are sometimes simply "deadlocks"
e.g. thread A owns resource X, holding it exclusively and wants resource Y, while thread B owns resource Y exclusively and wants resource X - the 2 threads are now deadlocked waiting for something that will never occur

For a crash dump, the stack of the running thread at the time of the crash is the first clue as it must have caused the exception - but it does NOT mean that it is guaranteed to be the bad guy (though this is very often the case)

Consider a case where driver P is naughty and overruns a buffer it allocated, extending into a memory area allocated by driver Q - driver P is able to work with its data quite happily, complete its work and hand control back to whoever wants it - then along comes driver Q at some future time to work on its data and finds it to be garbage
BOOM - unhandled exception in kernel mode = bugcheck - stack trace evdience says "probably caused by driver Q"


In order to do any decent kind of analysis, the more information we have, the better - in the case of dump analysis, this means more dumps
What is interesting it what the dumps have in common - because of how memory is used dynamically, in the "driver P/driver Q" scenario above it might present as different STOP codes in different drivers and so appear to have no particular pattern


A particularly useful tool if you suspect drivers is "Driver Verifier" - verifier.exe - which happens to be built into Windows
This is a troubleshooting tool and should only be used for identifying problems with system stability, as enabling debugging options here will have an impact on performance - don't go blindly enabling everything this tool can do or you may render Windows unbootable (though "Last Known Good" should get you back)

If, for example, you have a machine which keeps bugchecking with messages relating to "pool corruption", then the "speical pool" option on all 3rd party drivers could help - in the "driver P/driver Q" scenario it would actually bugcheck the system earlier, with a different STOP code, and point the finger at driver P when it overran its pool allocation

"Drivers" can include video, audio, network, USB, antivirus, virtual device, filters - they can be related to specific hardware devices or used by software such as services or system applications
Sometimes assumptions made by drivers can lead to problems that only occur when used in combination with each other
- if 2 filter drivers assume they are at the very top or bottom of the filter stack, for example - and this is why running 2 separate AV products at the same time might not be wise

[kidzer]

Another interesting article for us geeky hexites to read with interest - thank you Mr. Adams

[TheAnimus]

Very good work

[Paul Adams]

User-Mode Process Debugging
When taking a look at a dump from a single process, it is both simpler and trickier than kernel dumps.

Simpler because there are less threads to be concerned with than in a kernel dump, and smaller virtual memory space.

Trickier because you can't see the kernel stack of any thread, so threads may appear to end with calls into ntdll.dll - and also because a lot of the time you are dealing with 3rd party binaries to which you don't have symbols.

A large number of features in Windows are extensible by 3rd parties by loading their DLLs into their process virtual memory spaces - Internet Explorer, Explorer, Print Spooler, etc.
If any of these DLLs contains code which is buggy, the process can crash, often pointing to the wrong module as the root cause - in the same way that bugs in drivers can cause Windows to crash and point to "the kernel".

Spotting 3rd party modules in Microsoft processes is pretty simple - there are no symbols on the symbol server for them.

A bit of "hands-on" - apologies if it's not very readable, the output is wide due to the OS being 64-bit.

Here is me launching Notepad and attaching WinDbg to the process (thus suspending it):

Microsoft (R) Windows Debugger Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.

*** wait with pending attach
Symbol search path is: srv*Q:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00000000`ff380000 00000000`ff3af000 C:\Windows\system32\notepad.exe
ModLoad: 00000000`77900000 00000000`77a7a000 C:\Windows\system32\ntdll.dll
ModLoad: 00000000`776f0000 00000000`77821000 C:\Windows\system32\kernel32.dll
ModLoad: 000007fe`fe260000 000007fe`fe35f000 C:\Windows\system32\ADVAPI32.dll
ModLoad: 000007fe`ff780000 000007fe`ff8b9000 C:\Windows\system32\RPCRT4.dll
ModLoad: 000007fe`fe770000 000007fe`fe7d1000 C:\Windows\system32\GDI32.dll
ModLoad: 00000000`77830000 00000000`778fa000 C:\Windows\system32\USER32.dll
ModLoad: 000007fe`ff9c0000 000007fe`ffa61000 C:\Windows\system32\msvcrt.dll
ModLoad: 000007fe`fe590000 000007fe`fe61e000 C:\Windows\system32\COMDLG32.dll
ModLoad: 000007fe`fe1e0000 000007fe`fe251000 C:\Windows\system32\SHLWAPI.dll
ModLoad: 000007fe`fd9c0000 000007fe`fdbb0000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_1559f1c6f365a7fa\COMCTL32.dll
ModLoad: 000007fe`feb30000 000007fe`ff764000 C:\Windows\system32\SHELL32.dll
ModLoad: 000007fe`f8650000 000007fe`f86a7000 C:\Windows\system32\WINSPOOL.DRV
ModLoad: 000007fe`fe7e0000 000007fe`fe9c0000 C:\Windows\system32\ole32.dll
ModLoad: 000007fe`ff8e0000 000007fe`ff9b8000 C:\Windows\system32\OLEAUT32.dll
ModLoad: 000007fe`ffb70000 000007fe`ffb98000 C:\Windows\system32\IMM32.DLL
ModLoad: 000007fe`fea20000 000007fe`feb24000 C:\Windows\system32\MSCTF.dll
ModLoad: 000007fe`ff770000 000007fe`ff77c000 C:\Windows\system32\LPK.DLL
ModLoad: 000007fe`fe630000 000007fe`fe6ca000 C:\Windows\system32\USP10.dll
ModLoad: 00000000`10000000 00000000`10016000 C:\PROGRA~2\KASPER~1\KASPER~1.0\x64\r3hook.dll
ModLoad: 000007fe`fe060000 000007fe`fe068000 C:\Windows\system32\PSAPI.DLL
ModLoad: 000007fe`fca40000 000007fe`fca92000 C:\Windows\system32\uxtheme.dll
(1c0.f04): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00000000`7794fdf0 cc int 3

Let's turn on noisy symbol loading so we get some feedback...

0:001> !sym noisy
noisy mode - symbol prompts on

Now let's force a reload of all the modules in the process...

0:001> .reload /f
Reloading current modules

Here's what you get when a module has no corresponding symbol (Kaspersky AV in this case):

SYMSRV: Q:\Symbols\r3hook64.pdb\9462E15B735F4198BB03E8D310E6C6171\r3hook64.pdb not found
SYMSRV: http://msdl.microsoft.com/download/s...1/r3hook64.pdb not found
DBGHELP: C:\PROGRA~2\KASPER~1\KASPER~1.0\x64\r3hook64.pdb - file not found
DBGHELP: O:\out_x64\Release\r3hook64.pdb - file not found
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\PROGRA~2\KASPER~1\KASPER~1.0\x64\r3hook.dll -
DBGHELP: r3hook - export symbols

Here is what you get if you have the symbols in the local store:

DBGHELP: kernel32 - public symbols
Q:\Symbols\kernel32.pdb\0F0F039AC5E3459088FF0DA50AC145582\kernel32.pdb

Here is what you get if the symbols weren't available locally, but they were successfully located on the upstream symbole server and copied to the local store:

SYMSRV: notepad.pdb from http://msdl.microsoft.com/download/symbols: 31532 bytes - copied
DBGHELP: notepad - public symbols
Q:\Symbols\notepad.pdb\C672449E319649709326A522288FBA281\notepad.pdb

The loaded modules can be listed with path & timestamp details by using the command lmft:

0:001> lmft
start end module name
00000000`10000000 00000000`10016000 r3hook C:\PROGRA~2\KASPER~1\KASPER~1.0\x64\r3hook.dll Fri Mar 09 17:56:33 2007 (45F191C1)
00000000`776f0000 00000000`77821000 kernel32 C:\Windows\system32\kernel32.dll Thu Nov 02 12:14:48 2006 (4549D328)
00000000`77830000 00000000`778fa000 USER32 C:\Windows\system32\USER32.dll Thu Feb 15 06:22:33 2007 (45D3EE19)
00000000`77900000 00000000`77a7a000 ntdll C:\Windows\system32\ntdll.dll Thu Nov 02 12:16:02 2006 (4549D372)
00000000`ff380000 00000000`ff3af000 notepad C:\Windows\system32\notepad.exe Thu Nov 02 10:32:09 2006 (4549BB19)
000007fe`f8650000 000007fe`f86a7000 WINSPOOL C:\Windows\system32\WINSPOOL.DRV Thu Nov 02 12:16:12 2006 (4549D37C)
000007fe`fca40000 000007fe`fca92000 uxtheme C:\Windows\system32\uxtheme.dll Thu Nov 02 12:15:07 2006 (4549D33B)
000007fe`fd9c0000 000007fe`fdbb0000 COMCTL32 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_1559f1c6f365a7fa\COMCTL32.dll Thu Nov 02 12:14:51 2006 (4549D32B)
000007fe`fe060000 000007fe`fe068000 PSAPI C:\Windows\system32\PSAPI.DLL Thu Nov 02 12:14:24 2006 (4549D310)
000007fe`fe1e0000 000007fe`fe251000 SHLWAPI C:\Windows\system32\SHLWAPI.dll Thu Nov 02 12:14:39 2006 (4549D31F)
000007fe`fe260000 000007fe`fe35f000 ADVAPI32 C:\Windows\system32\ADVAPI32.dll Thu Nov 02 12:11:35 2006 (4549D267)
000007fe`fe590000 000007fe`fe61e000 COMDLG32 C:\Windows\system32\COMDLG32.dll Thu Nov 02 12:14:51 2006 (4549D32B)
000007fe`fe630000 000007fe`fe6ca000 USP10 C:\Windows\system32\USP10.dll Thu Nov 02 12:15:03 2006 (4549D337)
000007fe`fe770000 000007fe`fe7d1000 GDI32 C:\Windows\system32\GDI32.dll Thu Nov 02 12:11:47 2006 (4549D273)
000007fe`fe7e0000 000007fe`fe9c0000 ole32 C:\Windows\system32\ole32.dll Thu Nov 02 12:14:31 2006 (4549D317)
000007fe`fea20000 000007fe`feb24000 MSCTF C:\Windows\system32\MSCTF.dll Thu Nov 02 12:13:42 2006 (4549D2E6)
000007fe`feb30000 000007fe`ff764000 SHELL32 C:\Windows\system32\SHELL32.dll Wed Jun 27 05:27:50 2007 (4681D936)
000007fe`ff770000 000007fe`ff77c000 LPK C:\Windows\system32\LPK.DLL Thu Nov 02 12:12:33 2006 (4549D2A1)
000007fe`ff780000 000007fe`ff8b9000 RPCRT4 C:\Windows\system32\RPCRT4.dll Tue Jul 17 06:21:15 2007 (469C43BB)
000007fe`ff8e0000 000007fe`ff9b8000 OLEAUT32 C:\Windows\system32\OLEAUT32.dll Thu Nov 02 12:14:34 2006 (4549D31A)
000007fe`ff9c0000 000007fe`ffa61000 msvcrt C:\Windows\system32\msvcrt.dll Thu Nov 02 12:13:37 2006 (4549D2E1)
000007fe`ffb70000 000007fe`ffb98000 IMM32 C:\Windows\system32\IMM32.DLL Thu Nov 02 12:13:15 2006 (4549D2CB)

The tilde (~) command is used for focusing on a specific thread in a process.
~6s means switch the implicit thread to 6
~* kv 50 means "for every thread, run the kv 50 command" (display stack up to 50 deep)

Call stacks are read from the bottom to the top, so assuming no stack corruption you would expect them all to start with an obvious function.

If a thread is hung then you may find references to EnterCriticalSection, WaitForSingleObject, WaitForMultipleObjects, AcquirePushlockExclusive, etc.

This is the output from the thread in my Notepad.exe process when I broke in:

0:001> ~* kv 50
0 Id: 1c0.af4 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`0024fda8 00000000`7784e6ea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`ff381728 : USER32!ZwUserGetMessage+0xa
00000000`0024fdb0 00000000`ff386eca : 00000000`002f0048 00000000`006f02ff 000007fe`fea23b90 00000000`00000001 : USER32!GetMessageW+0x34
00000000`0024fde0 00000000`ff38cf8b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : notepad!WinMain+0x176
00000000`0024fe60 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : notepad!IsTextUTF8+0x24f
00000000`0024ff20 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0024ff50 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

# 1 Id: 1c0.f04 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`033efcb8 00000000`779e33e8 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgBreakPoint
00000000`033efcc0 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgUiRemoteBreakin+0x38
00000000`033efcf0 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`033efd20 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

There are only 2 threads in the process at present - the second is my debugger, and the first is the "main" thread (see the "WinMain" reference?) which is doing what most processes do for their whole lives... waiting for user input so they can get some work to do.

[Paul Adams]

Remember, while the debugger is attached to a running process, it has all its threads suspended so the window will not repaint itself and the OS will report it to be "not responding", which is correct.
To allow the process to continue running (maybe to go through some process to cause a crash), use the g command.

Here I used g to resume Notepad, then click File and Open - the ModLoad entries appeared in the debugger window as it is still attached...

0:000> g
ModLoad: 000007fe`fe6d0000 000007fe`fe76c000 C:\Windows\system32\CLBCatQ.DLL
ModLoad: 000007fe`f7170000 000007fe`f730c000 C:\Windows\system32\browseui.dll
ModLoad: 000007fe`fc080000 000007fe`fc162000 C:\Windows\system32\PROPSYS.dll
ModLoad: 000007fe`fc650000 000007fe`fc691000 C:\Windows\system32\DUser.dll
ModLoad: 000007fe`f8a80000 000007fe`f8b08000 C:\Program Files\Common Files\microsoft shared\ink\tiptsf.dll
ModLoad: 000007fe`f83b0000 000007fe`f8486000 C:\Windows\system32\WindowsCodecs.dll
ModLoad: 000007fe`fde80000 000007fe`fdeb5000 C:\Windows\system32\apphelp.dll
ModLoad: 000007fe`f7690000 000007fe`f7737000 C:\Windows\System32\cscui.dll
ModLoad: 000007fe`f79a0000 000007fe`f79ab000 C:\Windows\System32\CSCDLL.dll
ModLoad: 000007fe`f8700000 000007fe`f870c000 C:\Windows\System32\CSCAPI.dll
ModLoad: 000007fe`fe3b0000 000007fe`fe58e000 C:\Windows\system32\SETUPAPI.dll
ModLoad: 000007fe`fdee0000 000007fe`fdf06000 C:\Windows\system32\USERENV.dll
ModLoad: 000007fe`fdec0000 000007fe`fdedb000 C:\Windows\system32\Secur32.dll
ModLoad: 000007fe`f86b0000 000007fe`f86c1000 C:\Windows\system32\dwmapi.dll
ModLoad: 000007fe`fc620000 000007fe`fc650000 C:\Windows\system32\xmllite.dll
ModLoad: 000007fe`f7310000 000007fe`f743a000 C:\Windows\System32\shdocvw.dll
ModLoad: 00000000`02ce0000 00000000`02d0c000 C:\Program Files (x86)\Kaspersky Lab\Kaspersky Anti-Virus 6.0\x64\scrchpg.dll
ModLoad: 000007fe`fcff0000 000007fe`fd033000 C:\Windows\system32\rsaenh.dll
ModLoad: 000007fe`f6ed0000 000007fe`f6f22000 C:\Windows\System32\msshsq.dll
ModLoad: 000007fe`f36f0000 000007fe`f3708000 C:\Windows\system32\thumbcache.dll
ModLoad: 000007fe`f7070000 000007fe`f7168000 C:\Windows\system32\actxprxy.dll
ModLoad: 000007fe`f5e30000 000007fe`f64f1000 C:\Windows\system32\ieframe.dll
ModLoad: 000007fe`fe9c0000 000007fe`fea1a000 C:\Windows\system32\iertutil.dll
ModLoad: 000007fe`f7010000 000007fe`f706b000 C:\Windows\system32\ntshrui.dll
ModLoad: 000007fe`fdc90000 000007fe`fdd29000 C:\Windows\system32\NETAPI32.dll
ModLoad: 000007fe`fd5a0000 000007fe`fd5c8000 C:\Windows\system32\slc.dll
ModLoad: 000007fe`fd7a0000 000007fe`fd7bb000 C:\Windows\system32\SAMLIB.dll
ModLoad: 000007fe`fcf40000 000007fe`fcf6c000 C:\Windows\system32\NTMARTA.DLL
ModLoad: 000007fe`ffba0000 000007fe`ffbf5000 C:\Windows\system32\WLDAP32.dll
ModLoad: 000007fe`fe360000 000007fe`fe3a4000 C:\Windows\system32\WS2_32.dll
ModLoad: 000007fe`fe620000 000007fe`fe627000 C:\Windows\system32\NSI.dll
ModLoad: 000007fe`f7640000 000007fe`f764b000 C:\Windows\system32\LINKINFO.dll

...then I used CTRL-Break to break in with the debugger again, suspending Notepad once more:

(1c0.ce4): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00000000`7794fdf0 cc int 3

What this shows is that Windows loads DLLs when needed (on demand), not by default - now see what the loaded module list looks like:

0:011> lmft
start end module name
00000000`02ce0000 00000000`02d0c000 scrchpg C:\Program Files (x86)\Kaspersky Lab\Kaspersky Anti-Virus 6.0\x64\scrchpg.dll Fri Mar 09 17:56:36 2007 (45F191C4)
00000000`10000000 00000000`10016000 r3hook C:\PROGRA~2\KASPER~1\KASPER~1.0\x64\r3hook.dll Fri Mar 09 17:56:33 2007 (45F191C1)
00000000`776f0000 00000000`77821000 kernel32 C:\Windows\system32\kernel32.dll Thu Nov 02 12:14:48 2006 (4549D328)
00000000`77830000 00000000`778fa000 USER32 C:\Windows\system32\USER32.dll Thu Feb 15 06:22:33 2007 (45D3EE19)
00000000`77900000 00000000`77a7a000 ntdll C:\Windows\system32\ntdll.dll Thu Nov 02 12:16:02 2006 (4549D372)
00000000`ff380000 00000000`ff3af000 notepad C:\Windows\system32\notepad.exe Thu Nov 02 10:32:09 2006 (4549BB19)
000007fe`f36f0000 000007fe`f3708000 thumbcache C:\Windows\system32\thumbcache.dll Thu Nov 02 12:15:18 2006 (4549D346)
000007fe`f5e30000 000007fe`f64f1000 ieframe C:\Windows\system32\ieframe.dll Sat Aug 18 05:38:46 2007 (46C669C6)
000007fe`f6ed0000 000007fe`f6f22000 msshsq C:\Windows\System32\msshsq.dll Wed Dec 20 07:45:34 2006 (4588DC0E)
000007fe`f7010000 000007fe`f706b000 ntshrui C:\Windows\system32\ntshrui.dll Thu Nov 02 12:16:27 2006 (4549D38B)
000007fe`f7070000 000007fe`f7168000 actxprxy C:\Windows\system32\actxprxy.dll Thu Nov 02 12:11:18 2006 (4549D256)
000007fe`f7170000 000007fe`f730c000 browseui C:\Windows\system32\browseui.dll Thu Nov 02 12:11:55 2006 (4549D27B)
000007fe`f7310000 000007fe`f743a000 shdocvw C:\Windows\System32\shdocvw.dll Thu Nov 02 12:14:33 2006 (4549D319)
000007fe`f7640000 000007fe`f764b000 LINKINFO C:\Windows\system32\LINKINFO.dll Thu Nov 02 12:12:17 2006 (4549D291)
000007fe`f7690000 000007fe`f7737000 cscui C:\Windows\System32\cscui.dll Thu Nov 02 12:15:29 2006 (4549D351)
000007fe`f79a0000 000007fe`f79ab000 CSCDLL C:\Windows\System32\CSCDLL.dll Thu Nov 02 12:15:25 2006 (4549D34D)
000007fe`f83b0000 000007fe`f8486000 WindowsCodecs C:\Windows\system32\WindowsCodecs.dll Thu May 24 05:35:18 2007 (465507F6)
000007fe`f8650000 000007fe`f86a7000 WINSPOOL C:\Windows\system32\WINSPOOL.DRV Thu Nov 02 12:16:12 2006 (4549D37C)
000007fe`f86b0000 000007fe`f86c1000 dwmapi C:\Windows\system32\dwmapi.dll Thu Nov 02 12:12:55 2006 (4549D2B7)
000007fe`f8700000 000007fe`f870c000 CSCAPI C:\Windows\System32\CSCAPI.dll Thu Nov 02 12:15:24 2006 (4549D34C)
000007fe`f8a80000 000007fe`f8b08000 tiptsf C:\Program Files\Common Files\microsoft shared\ink\tiptsf.dll Thu Nov 02 12:15:25 2006 (4549D34D)
000007fe`fc080000 000007fe`fc162000 PROPSYS C:\Windows\system32\PROPSYS.dll Thu Nov 02 12:14:21 2006 (4549D30D)
000007fe`fc620000 000007fe`fc650000 xmllite C:\Windows\system32\xmllite.dll Thu Nov 02 12:15:28 2006 (4549D350)
000007fe`fc650000 000007fe`fc691000 DUser C:\Windows\system32\DUser.dll Thu Nov 02 12:12:54 2006 (4549D2B6)
000007fe`fca40000 000007fe`fca92000 uxtheme C:\Windows\system32\uxtheme.dll Thu Nov 02 12:15:07 2006 (4549D33B)
000007fe`fcf40000 000007fe`fcf6c000 NTMARTA C:\Windows\system32\NTMARTA.DLL Thu Nov 02 12:16:20 2006 (4549D384)
000007fe`fcff0000 000007fe`fd033000 rsaenh C:\Windows\system32\rsaenh.dll Thu Nov 02 12:14:37 2006 (4549D31D)
000007fe`fd5a0000 000007fe`fd5c8000 slc C:\Windows\system32\slc.dll Thu Jun 21 05:18:03 2007 (4679EDEB)
000007fe`fd7a0000 000007fe`fd7bb000 SAMLIB C:\Windows\system32\SAMLIB.dll Thu Nov 02 12:14:29 2006 (4549D315)
000007fe`fd9c0000 000007fe`fdbb0000 COMCTL32 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_1559f1c6f365a7fa\COMCTL32.dll Thu Nov 02 12:14:51 2006 (4549D32B)
000007fe`fdc90000 000007fe`fdd29000 NETAPI32 C:\Windows\system32\NETAPI32.dll Thu Nov 02 12:14:04 2006 (4549D2FC)
000007fe`fde80000 000007fe`fdeb5000 apphelp C:\Windows\system32\apphelp.dll Thu Nov 02 12:11:03 2006 (4549D247)
000007fe`fdec0000 000007fe`fdedb000 Secur32 C:\Windows\system32\Secur32.dll Thu Nov 02 12:15:11 2006 (4549D33F)
000007fe`fdee0000 000007fe`fdf06000 USERENV C:\Windows\system32\USERENV.dll Thu Nov 02 12:15:02 2006 (4549D336)
000007fe`fe060000 000007fe`fe068000 PSAPI C:\Windows\system32\PSAPI.DLL Thu Nov 02 12:14:24 2006 (4549D310)
000007fe`fe1e0000 000007fe`fe251000 SHLWAPI C:\Windows\system32\SHLWAPI.dll Thu Nov 02 12:14:39 2006 (4549D31F)
000007fe`fe260000 000007fe`fe35f000 ADVAPI32 C:\Windows\system32\ADVAPI32.dll Thu Nov 02 12:11:35 2006 (4549D267)
000007fe`fe360000 000007fe`fe3a4000 WS2_32 C:\Windows\system32\WS2_32.dll Thu Nov 02 12:16:26 2006 (4549D38A)
000007fe`fe3b0000 000007fe`fe58e000 SETUPAPI C:\Windows\system32\SETUPAPI.dll Thu Nov 02 12:14:32 2006 (4549D318)
000007fe`fe590000 000007fe`fe61e000 COMDLG32 C:\Windows\system32\COMDLG32.dll Thu Nov 02 12:14:51 2006 (4549D32B)
000007fe`fe620000 000007fe`fe627000 NSI C:\Windows\system32\NSI.dll Thu Nov 02 12:16:00 2006 (4549D370)
000007fe`fe630000 000007fe`fe6ca000 USP10 C:\Windows\system32\USP10.dll Thu Nov 02 12:15:03 2006 (4549D337)
000007fe`fe6d0000 000007fe`fe76c000 CLBCatQ C:\Windows\system32\CLBCatQ.DLL Thu Nov 02 12:12:02 2006 (4549D282)
000007fe`fe770000 000007fe`fe7d1000 GDI32 C:\Windows\system32\GDI32.dll Thu Nov 02 12:11:47 2006 (4549D273)
000007fe`fe7e0000 000007fe`fe9c0000 ole32 C:\Windows\system32\ole32.dll Thu Nov 02 12:14:31 2006 (4549D317)
000007fe`fe9c0000 000007fe`fea1a000 iertutil C:\Windows\system32\iertutil.dll Thu Nov 02 12:12:28 2006 (4549D29C)
000007fe`fea20000 000007fe`feb24000 MSCTF C:\Windows\system32\MSCTF.dll Thu Nov 02 12:13:42 2006 (4549D2E6)
000007fe`feb30000 000007fe`ff764000 SHELL32 C:\Windows\system32\SHELL32.dll Wed Jun 27 05:27:50 2007 (4681D936)
000007fe`ff770000 000007fe`ff77c000 LPK C:\Windows\system32\LPK.DLL Thu Nov 02 12:12:33 2006 (4549D2A1)
000007fe`ff780000 000007fe`ff8b9000 RPCRT4 C:\Windows\system32\RPCRT4.dll Tue Jul 17 06:21:15 2007 (469C43BB)
000007fe`ff8e0000 000007fe`ff9b8000 OLEAUT32 C:\Windows\system32\OLEAUT32.dll Thu Nov 02 12:14:34 2006 (4549D31A)
000007fe`ff9c0000 000007fe`ffa61000 msvcrt C:\Windows\system32\msvcrt.dll Thu Nov 02 12:13:37 2006 (4549D2E1)
000007fe`ffb70000 000007fe`ffb98000 IMM32 C:\Windows\system32\IMM32.DLL Thu Nov 02 12:13:15 2006 (4549D2CB)
000007fe`ffba0000 000007fe`ffbf5000 WLDAP32 C:\Windows\system32\WLDAP32.dll Thu Nov 02 12:16:38 2006 (4549D396)

[Paul Adams]

Also, see what has happened to our simple 2-thread process, plus how !sym noisy shows you the symbols being downloaded for the new modules:

0:011> ~* kv 50
0 Id: 1c0.af4 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`0024f548 00000000`7772ed73 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0xa
00000000`0024f550 00000000`7784e96d : 00000000`00000001 00000000`0024f730 00000000`00000000 00000000`ffffffff : kernel32!WaitForMultipleObjectsEx+0x10b
SYMSRV: DUser.pdb from http://msdl.microsoft.com/download/symbols: 152774 bytes - copied
DBGHELP: DUser - public symbols
Q:\Symbols\DUser.pdb\12140286AA73426D80EA4D2ACC25DEFC2\DUser.pdb
00000000`0024f660 000007fe`fc651ab6 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`003643e0 : USER32!RealMsgWaitForMultipleObjectsEx+0x129
00000000`0024f700 000007fe`fc65371f : 00000000`ffffffff 00000000`00000000 00000000`0000049f 00000000`003643e0 : DUser!CoreSC::Wait+0x62
00000000`0024f750 000007fe`fc653696 : 00000000`00000000 00000000`00bdd660 fffff980`1d308c20 00000000`000024ff : DUser!CoreSC::WaitMessage+0x6f
00000000`0024f790 00000000`7783bd1a : fffffa80`04786060 00000000`00220318 00000000`00bf4ef8 00000000`00000001 : DUser!MphWaitMessageEx+0x36
00000000`0024f7c0 00000000`77952016 : 00000000`000d0040 00000000`00bdd660 00000000`00000001 00000000`7784b85d : USER32!_ClientWaitMessageExMPH+0x1a
00000000`0024f810 00000000`7784df2a : 00000000`778373e9 00000000`000d0040 00000000`00bdd660 00000000`00000001 : ntdll!KiUserCallbackDispatcherContinue (TrapFrame @ 00000000`0024f6d8)
00000000`0024f878 00000000`778373e9 : 00000000`000d0040 00000000`00bdd660 00000000`00000001 00000000`00000001 : USER32!ZwUserWaitMessage+0xa
00000000`0024f880 00000000`7783760a : 00000000`000d0040 000007fe`fe592e10 00000000`00000001 00000000`00000000 : USER32!DialogBox2+0x261
00000000`0024f900 00000000`778374c6 : 00000000`00346e80 00000000`000d0040 000007fe`fe592e10 000007fe`fe83cdd8 : USER32!InternalDialogBox+0x134
00000000`0024f960 00000000`77837918 : 00000000`00000000 00000000`00000001 00000000`000d0040 00000000`00323830 : USER32!DialogBoxIndirectParamAorW+0x58
00000000`0024f9a0 000007fe`fe593d2e : 00000000`00000000 00000000`00000001 00000000`000d0040 00000000`00000000 : USER32!DialogBoxIndirectParamW+0x18
00000000`0024f9e0 00000000`ff385146 : 00000600`00000008 00000000`00346e80 10892178`00000008 00000000`00000000 : COMDLG32!CFileOpenSave::Show+0x143
00000000`0024fa50 00000000`ff38547e : 00000000`00000000 00000000`00001808 00000000`000d0040 00000000`0024fb60 : notepad!ShowOpenSaveDialog+0xde
00000000`0024fac0 00000000`ff385b98 : 00000000`00000000 000007fe`fca62590 00000000`00000001 00000000`0034a9f8 : notepad!InvokeOpenDialog+0x136
00000000`0024fb20 00000000`ff386d33 : 00000000`00000001 00000000`00000005 00000000`00000000 00000000`00000000 : notepad!NPCommand+0x380
00000000`0024fc50 00000000`7784e25a : 00000000`00000000 00000000`ff380000 00000000`00000001 00000000`00000000 : notepad!NPWndProc+0x55b
00000000`0024fca0 00000000`7784ec8f : 00000000`0024fe20 00000000`ff3867d8 00000000`00000000 00000000`00bf4ed0 : USER32!UserCallWinProcCheckWow+0x1ad
00000000`0024fd60 00000000`ff386eb7 : 00000000`002f0048 00000000`006f02ff 00000000`ff3867d8 00000000`00000001 : USER32!DispatchMessageWorker+0x389
00000000`0024fde0 00000000`ff38cf8b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : notepad!WinMain+0x163
00000000`0024fe60 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : notepad!IsTextUTF8+0x24f
00000000`0024ff20 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0024ff50 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

1 Id: 1c0.1c4 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`02c7fab8 00000000`7772ed73 : 00000002`00000000 00000000`02003d3f 00000000`00000048 00000000`00000001 : ntdll!NtWaitForMultipleObjects+0xa
00000000`02c7fac0 00000000`7784e96d : 00000000`00000001 00000000`02c7fca0 00000000`00000000 00000000`ffffffff : kernel32!WaitForMultipleObjectsEx+0x10b
00000000`02c7fbd0 000007fe`fc651ab6 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000003 : USER32!RealMsgWaitForMultipleObjectsEx+0x129
00000000`02c7fc70 000007fe`fc651aef : 00000000`0035fff0 00000000`00000000 00000000`00000003 00000000`00000003 : DUser!CoreSC::Wait+0x62
00000000`02c7fcc0 000007fe`fc65e4ad : 00000000`00000000 00000000`02c7fdc8 00000000`00000000 00000000`7772ec1c : DUser!CoreSC::xwProcessNL+0xd5
00000000`02c7fd30 000007fe`fc65e3cc : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : DUser!GetMessageExA+0x7b
00000000`02c7fd80 000007fe`ff9d94e7 : 00000000`001266c0 00000000`00344760 00000000`00000000 00000000`00000000 : DUser!ResourceManager::SharedThreadProc+0xe8
00000000`02c7fe10 000007fe`ff9d967d : 000007fe`ffa52740 00000000`001266c0 00000000`00000000 00000000`00000000 : msvcrt!endthreadex+0x47
00000000`02c7fe40 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msvcrt!endthreadex+0x100
00000000`02c7fe70 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`02c7fea0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

2 Id: 1c0.c44 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`05a5f768 00000000`77728c5c : 00000000`05a5f7c8 00000000`00000000 00000000`05a5f7c8 00000000`00000000 : ntdll!ZwRemoveIoCompletion+0xa
00000000`05a5f770 000007fe`ff7be7a3 : 000007fe`ff898e30 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!GetQueuedCompletionStatus+0x48
00000000`05a5f7d0 000007fe`ff7be66a : 00000000`00000000 000007fe`ff81b1c0 000007fe`ff898e30 00000000`00000000 : RPCRT4!COMMON_ProcessCalls+0x101
00000000`05a5f860 000007fe`ff7be8e9 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!LOADABLE_TRANSPORT:: ProcessIOEvents+0x148
00000000`05a5f910 000007fe`ff7be89d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!ProcessIOEventsWrapper+0x9
00000000`05a5f940 000007fe`ff7be924 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!BaseCachedThreadRoutine+0x9b
00000000`05a5f980 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!ThreadStartRoutine+0x24
00000000`05a5f9b0 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`05a5f9e0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

3 Id: 1c0.1f8 Suspend: 1 Teb: 000007ff`fffd8000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`05affc48 00000000`7772d908 : 00000000`00000000 00000000`00ffffff 00000000`0000021c 000007fe`fe816ca9 : ntdll!NtDelayExecution+0xa
00000000`05affc50 000007fe`fe82cdc0 : 00000000`003e7e00 000007fe`00000000 ffffffff`dc3cba00 00000000`00000000 : kernel32!SleepEx+0x84
00000000`05affcd0 000007fe`fe823b7e : 000007fe`fe7e0000 000007fe`fe823c6a 00000000`00000000 00000000`03dd6a80 : ole32!CROIDTable::WorkerThreadLoop+0x10
00000000`05affd00 000007fe`fe823c0a : 00000000`003e7e00 00000000`003e7e00 00000000`00000000 00000000`00000000 : ole32!CRpcThread::WorkerLoop+0x1e
00000000`05affd40 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x2a
00000000`05affd70 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`05affda0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

...list of threads truncated to fit in 1 post - you get the idea...

9 Id: 1c0.eb4 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`048af338 000007fe`fe59507c : 00000000`00000000 00000000`048af3d8 00000000`048af3d0 000007fe`fe594c6f : ntdll!NtFsControlFile+0xa
00000000`048af340 000007fe`fe594bef : 00000000`00000000 00000000`00000000 00000000`05c37240 000007fe`fe1f3857 : COMDLG32!PathIsVolumeGUIDWorker+0x1a4
00000000`048af3b0 000007fe`fe594eda : 00000000`00000000 ffffffff`fffffffe 00000000`048af4d0 00000000`00000000 : COMDLG32!_GetNetFileName+0x3ef
00000000`048af400 000007fe`fe594e5e : 00000000`00000000 00000000`00000001 00000000`00000000 00000000`05c37240 : COMDLG32!SHCheckSnapshotAvailability+0x40
00000000`048af430 000007fe`fe594da1 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`05c37240 : COMDLG32!SHAreSnapshotsAvailable+0x8a
00000000`048af470 000007fe`fe594d12 : 00000000`05c4d0c0 00000000`05c37290 00000000`00000000 00000000`03e17870 : COMDLG32!HavePreviousVersionsAt+0x51
00000000`048af4a0 000007fe`fe5ac30c : 00000000`00000000 00000000`003ea9e8 00000000`03e175a0 00000000`0000000c : COMDLG32!CGetCommandStateTask::InternalResumeRT+0x52
SYMSRV: browseui.pdb from http://msdl.microsoft.com/download/symbols: 776495 bytes - copied
DBGHELP: browseui - public symbols
Q:\Symbols\browseui.pdb\F4628AE06CAD42F78AA4C43BFCB8D4D52\browseui.pdb
00000000`048af4d0 000007fe`f717dd93 : 00000000`03df6910 00000000`05c33070 00000000`00000000 00000000`00000000 : COMDLG32!CRunnableTask::Run+0xd0
00000000`048af500 000007fe`f717dfc3 : 00000000`03de3e10 00000000`03de3e00 00000000`00000000 00000000`00000000 : browseui!CShellTaskThread::ThreadProc+0x2c7
00000000`048af5e0 000007fe`fe1f411f : 00000000`03de3e10 00000000`00000000 00000000`03debc10 00000000`03debc10 : browseui!CShellTaskThread::s_ThreadProc+0x33
00000000`048af610 00000000`77926500 : 00000000`05c37420 00000000`05c37420 00000000`00000000 00000000`77927091 : SHLWAPI!ExecuteWorkItemThreadProc+0xf
00000000`048af640 00000000`77947b59 : 00000000`00383370 00000000`03e176f8 00000000`00321bb8 00000000`03debc10 : ntdll!RtlpTpWorkCallback+0xf0
00000000`048af6f0 00000000`7772cdcd : 00000000`00383370 00000001`0001fffd 00000000`00383370 00000000`05c37420 : ntdll!TppWorkerThread+0x3ad
00000000`048af960 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`048af990 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

10 Id: 1c0.9b0 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`046ffbc8 00000000`77947a20 : 00000000`00383300 00000000`00000000 00000000`77926410 00000000`05c65930 : ntdll!NtWaitForWorkViaWorkerFactory+0xa
00000000`046ffbd0 00000000`7772cdcd : 00000000`00383370 00000000`00000000 00000000`00383370 00000000`05c65930 : ntdll!TppWorkerThread+0x23e
00000000`046ffe40 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`046ffe70 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

# 11 Id: 1c0.ce4 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`049ff9f8 00000000`779e33e8 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgBreakPoint
00000000`049ffa00 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!DbgUiRemoteBreakin+0x38
00000000`049ffa30 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`049ffa60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

[Paul Adams]

If you try to debug a process without symbols, the best you can hope for is the module in which the instructions reside, but not a clue what they are - this is a thread in 7-Zip's File Manager:

0 Id: aec.d78 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
Child-SP RetAddr : Args to Child : Call Site
00000000`0012fd88 00000000`7784e6ea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000001 : USER32!ZwUserGetMessage+0xa
00000000`0012fd90 00000000`0040ac3b : 00000000`00000000 00000000`00000000 00000000`00440c7c 00000000`00000022 : USER32!GetMessageW+0x34
00000000`0012fdc0 00000000`0044b821 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 7zFM+0xac3b
00000000`0012fe90 00000000`7772cdcd : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 7zFM+0x4b821
00000000`0012ff60 00000000`7794c6e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0012ff90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

The symbols provide a useful description of the functions and the layout of the data structures - so the Windows API functions are visible, but the stuff that jumps into 3rd party-land is just hexadecimal offsets.

Be aware that attaching WinDbg to a running process is "invasive" by default - when you detach the debugger the process will get killed too.

User-mode debugging goes a lot further than I have touched on here - you can set breakpoints when a particular function is called (or exits), or if a particular memory location is accessed (read or write), so you can "run until breakpoint is hit" if you want to capture a specific suspected problem.

It's not a silver bullet, and it is a bit of an art, often identifying modules that are unexpected (malware? software you thought was uninstalled?) or very, very old (I sometimes see ones dated in the 1990's, pre-dating even Windows 2000).

Explorer is commonly blamed for problems which are actually caused by addons, either buggy or conflicting ones.
The first instance of Explorer.exe is your desktop, task bar, Start menu and system tray - and by default any folder window you open is just another thread inside the same process.
Any utility you install that adds options to the context menu, for example - they are modules now loaded into every Explorer.exe process you create.