In
any event, judging from my email, I'm not alone in my predicament. I
receive a number of requests from people who want to know about the
inner workings of 16-bit applications running under Windows NT. With
this in mind, I'll devote this month's column to describing the Windows
NT support for monitoring 16-bit programs. I'll start with a brief
overview of how Windows NT runs programs for 16-bit Windows and MS-DOS®.
Under
Windows NT 3.5 and earlier, every MS-DOS and 16-bit Windows-based
program ran in its own virtual machine (VM). To these programs, each VM
appeared as a relatively complete Windows 3.1-based system, down to
having its own Local Descriptor Table (LDT). The activities of each
16-bit program were completely separate from those of other 16-bit
programs. As a result, a 16-bit program could blow up with no ill effect
on any other program.
Starting
with version 3.51, Windows NT acquired the ability to run multiple
16-bit programs in the same VM. These programs shared the same address
space and LDT, much like Windows 3.1. The good thing about this
capability is that once a 16-bit program loaded, subsequent 16-bit
programs would load faster and use less memory by using the already
created VM. The downside to sharing VMs is that if one 16-bit program
choked, the entire VM would go away (again, much like Windows 3.1).
Luckily, the capability of running a badly behaving application in its
own VM is still present. In the Run dialog, the Run in Separate Memory
Space checkbox lets you specify that a separate VM should be used for
the selected program, rather than running it in the communal system VM.
The
focal point of VM capability in Windows NT is NTVDM.EXE from the
SYSTEM32 directory. Each running instance of NTVDM.EXE constitutes a
separate VM. NTVDM.EXE is a Win32®
program. However, it uses its separate address space to create a VM
that runs alongside regular Win32 processes. In other words, programs
for MS-DOS and 16-bit Windows are like children belonging to the adult
NTVDM process. NTVDM is just another adult Win32 process, with
essentially the same rights and privileges as any other Win32-based
program. This architecture even has its own name, Windows On Windows
(WOW).
To
briefly digress from Windows NT, Windows 95 has quite a different method
for mixing MS-DOS, 16-bit Windows, and Win32 apps. Each MS-DOS-based
program has its own VM. However, 16-bit Windows-based applications share
the same VM and LDT, and have no address space protection from one
another.
Under
Windows 95, when a Win32 process has the CPU, that process can see the
memory of the VM containing all the 16-bit programs. Likewise, 16-bit
code running in the context of a 32-bit process (via a thunk) can see
the 32-bit code and data of that process. At least the memory of other
32-bit programs isn't visible. This looseness of the Windows 95 address
space is required for the flexibility of Windows 95 thunking, as well as
to reduce excess thread switching and memory copying.
This
concludes the five-cent tour of MS-DOS and 16-bit Windows-based
programs running under Windows NT and Windows 95. Now, it's on to the
facilities that Windows NT has for snooping around in this environment.
The word you want to remember here is VDMDBG (that is, VDM Debug).
Nearly all of the VDM debugging support is in a DLL named VDMDBG.DLL,
which resides in the SYSTEM32 directory.
If
you've never heard of VDMDBG.DLL, don't feel bad. Microsoft doesn't
mention it in any of the online documentation that I searched.
Nonetheless, VDMDBG.H and VDMDBG.LIB come with Visual C++®
and the Platform SDK (at least as of the January 1998 edition). In the
Platform SDK, there's a .HLP file from September 1994 in the
\MSSDK\DOC\MISC directory. Alas, I couldn't find VDMDBG.HLP on my Visual
Studio™ 97 CDs.
Using
VDMDBG.DLL, you could write a 32-bit debugger program that runs on
Windows NT and debugs 16-bit programs. However, VDMDBG.DLL isn't solely
for debugger writers. Programmers often wonder how the Windows NT Task
Manager (TASKMGR.EXE) is able to display the running 16-bit Windows
tasks in the Processes tab. The answer is VDMDBG.DLL, which TASKMGR.EXE
links to implicitly. If you look closely, the 16-bit Windows tasks you
see in the Task Manager are always subordinate to an NTVDM process
instance.
The VDMDBG API
As
an experienced debugger writer, I felt like a party to a mugging when I
first read VDMDBG.HLP. The introductory text is not for the faint of
heart. It took many rereadings before the document made sense. Having
undergone the steep learning curve, I can save you some time.
The
VDMDBG API can be broken up into two sections. The first part is the
simple stuff. The much larger remaining set of APIs is for hardcore
programmers. If you're just interested in enumerating through 16-bit
Windows tasks, skip to sections 5.8 and 5.9 of VDMDBG.HLP. There, you'll
find an understandable description of VDMEnumProcessWOW and
VDMEnumTaskWOW.
VDMEnumProcessWOW
enumerates all of the VDMs in the system (that is, each instance of
NTVDM.EXE). For each VDM, the API invokes a callback function that you
define. The primary parameter to the callback function is the process ID
for the particular instance of NTVDM.EXE. In addition, the callback
receives an attribute DWORD and whatever user-supplied value was given
to VDMEnumProcessWOW. The attribute parameter has only one flag value
defined, WOW_SYSTEM. This flag is set for the instance of NTVDM.EXE that
future 16-bit Windows-based programs will run in if a separate address
space wasn't requested. Unlike most Windows enumerations, you return
FALSE from the callback function to continue the enumeration.
Digging
down a level, you come to VDMEnumTaskWOW and VDMEnumTaskWOWEx.
VDMEnumTaskWOWEx isn't explained in VDMDBG.HLP, but VDMDBG.H provides a
comment that describes the additional capabilities. Both APIs take a
process ID as input and call another user-defined callback function. For
VDMEnumTaskWOW, the callback function receives the thread ID of a
16-bit task running in the VDM, plus the 16-bit HMODULE, the 16-bit
HTASK, and the user-supplied DWORD. The VDMEnumTaskWOWEx callback
receives all of these parameters and adds a pointer to the file name and
module name of the EXE for the 16-bit task. If the relationship between
a Win32 thread ID and a 16-bit Windows task isn't clear yet, don't
fret. I'll come back to this topic later.
With
these simple enumerations, you can easily get at the same information
that TASKMGR.EXE uses. From here on out, though, the remainder of
VDMDBG's API is a tougher climb. Sure, there are functions that look
like they might be used easily at first glance. For instance,
VDMGlobalFirst and VDMGlobalNext are nearly dead ringers for the 16-bit
GlobalFirst and GlobalNext APIs in TOOLHELP.DLL. Ditto for
VDMModuleFirst and VDMModuleNext, which are like TOOLHELP's ModuleFirst
and ModuleNext APIs.
The
distinctions between these 32-bit APIs and their
16-bit cousins are threefold. For starters, the 32-bit APIs require that
the WOWDEB.EXE task runs in the target debugee's VM. (You will find
WOWDEB.EXE in your SYSTEM32 directory.) How do you load a task in a
particular NTVDM session? Refer to the VDMStartTaskInWOW API, which does
not seem to be documented except in a comment from VDMDBG.H.
The
second issue is that these APIs require a thread handle. Where can you
get a thread handle? There's an OpenProcess API that returns process
handles, but there's no corresponding OpenThread API. To my knowledge,
there are only two ways to get an original thread handle: to be the
creator of the thread, or to be passed a thread handle as part of a
debugger process. Since NTVDM.EXE creates the threads for each 16-bit
Windows task, the first option is out.
The
second option for getting a thread handle is the debugger approach. When
acting as a debugger process, the debugger receives notification of
each new thread, along with a new thread handle. You could call
DebugActiveProcess and attach yourself as a debugger to a running
instance of NTVDM.EXE. This isn't necessarily a good idea, though. Once
you begin acting as a debugger for another process, you can't detach
yourself. If your process terminates, so will the debuggee. In the case
of NTVDM.EXE, that could mean any number of 16-bit Windows tasks simply
vanishing because your process has terminated.
Speaking
of debuggers, this segues nicely into the third hassle with APIs like
VDMModuleFirst and VDMGlobalNext. Because they will cause 16-bit system
code to execute in KRNL386.EXE, events that a debugger would want to see
may be triggered. For this reason, these APIs take a callback address
that will be invoked if a debug-related event occurs. Just to enumerate
modules or selectors, you're suddenly expected to handle Win32 debug
events. Perhaps it's called VDMDBG.DLL for a reason!
I'm not
going to describe all the remaining APIs that VDMDBG.DLL provides. It's
enough to say that the APIs provide the basic functionality required for
a 32-bit program to perform surgery in a 16-bit Windows environment.
However, something that's worth spending more time on is how 16-bit
system events percolate up to a 32-bit process.
If
you've used NotifyRegister and InterruptRegister from the 16-bit
TOOLHELP.DLL, you may recall events such as GP faults, module loads,
segment frees, tasks starting, and so on. Using VDMDBG.DLL, a Win32
process can see those same events, albeit in a different manner. Figure 1
shows the VDMDBG.DLL events and their 16-bit TOOLHELP.DLL equivalents
(where applicable). Alas, I haven't been able to generate all of the
listed events in my informal testing.
Let's
say you wanted to monitor 16-bit Windows events from a Win32 program.
How would you go about doing it? The answer is (not surprisingly) to act
as a debugger for the NTVDM process. Before reading ahead, you should
be at least somewhat familiar with the Win32 debug architecture and
debug loops. See the Win32 documentation and John Robbins' article, "Multiple Threads in Visual Basic 5.0, Part II: Writing a Win32 Debugger" (MSJ, September 1997) if you need assistance with this topic.
VDMDBG.DLL
dedicates a special exception code (STATUS_VDM_EVENT) to communicate
16-bit events to a 32-bit debugger. When the 32-bit debugger sees an
exception code of STATUS_VDM_EVENT, it can determine the 16-bit event
type and other information by reading the exception arguments. Section 4
of VDMDBG.HLP describes most of the 16-bit event types. VDMDBG.H
includes a set of macros (W1, W2, and so on) that help crack apart the
exception arguments into the desired fields.
Let's
look at an example to clarify this point. The situation is as follows:
your process is acting as a Win32 debugger on an instance of NTVDM.EXE.
The program is dutifully processing debug events with a
WaitForDebugEvent/ContinueDebugEvent loop. The user starts a new 16-bit
program, which the system runs in the VDM you're monitoring. As a
result, a DBG_TASKSTART is generated. To the 32-bit debugger, this
appears as an exception with a code of STATUS_VDM_EVENT. The W1 field of
the ExceptionInformation is 10 (DBG_TASKSTART).
Digging
deeper into the example, the VDMDBG documentation states that the W3
field contains a pointer to an IMAGE_NOTE structure. The IMAGE_NOTE
structure contains a module name (for instance, NUKEDLL), the associated
file name (for instance, C:\COLUMNS\COL8\
NUKEDLL.EXE), the HMODULE, and the HTASK.
When I
first wrote code to try this, I blindly cast the DW3 field to an
IMAGE_NOTE pointer, and was baffled when my code faulted. It turns out
that the IMAGE_NOTE structure is located in the debuggee process. No
problem; just use ReadProcessMemory to suck the information over to the
debugger process. The same goes for the SEGMENT_NOTE structure, which is
used by several other 16-bit events.
Experimenting with VDMDBG.DLL
To show some of the capabilities of VDMDBG.DLL, I wrote the VDMDBGDemo program, shown in Figure 2.
When you first start VDMDBGDemo, the action is in the bottom tree view
control. If you have any NTVDM sessions running, they'll show up in the
tree view. Otherwise, you can start a 16-bit Windows-based app and hit
the refresh button, which updates the tree view contents with current
information. If you do nothing at all, the tree view refreshes every 10
seconds via a WM_TIMER message.
The
top-level contents of the tree view are filled in by the results of
using VDMEnumProcessWOW. In VDMDBGDemo.CPP, this occurs in the
PopulateTree and VDMProcessEnumProc functions (see Figure 3).
As the callback encounters each NTVDM session, it in turn calls
VDMEnumTaskWOWEx. This callback function (VDMTaskEnumProc) adds the
second-level tree view nodes that describe each 16-bit Windows process
running in the VDM.
When you first run VDMDBGDemo, the upper listbox will remain conspicuously empty, unlike the one shown in Figure 2.
This listbox shows the 16-bit events that have occurred in one of the
NTVDM sessions. How do you see these events? As I mentioned, a Win32
debug loop is necessary to receive the exceptions that convey 16-bit
event information. How are event strings added to the listbox? In the
dialog procedure (VDMDBGDemoDlgProc), I use a custom-defined window
message (WM_LB_ADDITEM). Why didn't I just use LB_ADDSTRING directly?
The reasons will become clear shortly.
Figure 4
contains all of the code for a Win32 debug loop that monitors an NTVDM
process. To start a thread that monitors one of the NTVDM sessions,
highlight the session (as shown in Figure 2), then click the
Attach button. This causes VDMDBGDemo to start a new thread to monitor
the selected session. The entry point to this thread's code is the
VDMDebugThreadFunc function. Near the very beginning of the function, it
calls DebugActiveProcess on the selected NTVDM process. If the call
succeeds, the code enters into a WaitForDebugEvent/ContinueDebugEvent
loop. This loop will continue indefinitely or until the NTVDM session
terminates.
