In
the Mem Usage column of the Windows NT® Task Manager, you can see that
the memory used by a process may rise or fall drastically. A common
developer problem is that an application's memory usage as shown by
TaskMan goes up, but your typical memory-tracking tools don't show
corresponding memory or resource leaks. The underlying reason for this
discrepancy is that most tools focus on heap-allocated memory. This is a
very narrow view of the world!
It's
easy for people to forget that memory consumed by a process is much
more than just its calls to malloc or new. Generally speaking, almost
all the memory used by a process falls into one of these categories:
- Executable code in a loaded module
- Read-only data in a loaded module (including resources)
- Writable memory in a module (for example, the .data section)
- Win32® heaps (including the default heap)
- Suballocated heaps (for example, from the Visual C++® runtime library)
- VirtualAlloced memory
- Memory-mapped files
- Thread stacks
- Environment
- System data structures (including the Thread Information Block and page tables)
In its
normal course of execution, Windows® pages memory in and out of the
process address space. For instance, pages making up the code and data
areas of a loaded DLL don't use any physical memory until something
references them. When a reference occurs, only the touched page is
mapped in. Likewise, when system memory gets tight, Windows can swap out
pages of code and data. In this very dynamic situation, limiting
yourself to watching heap allocations quickly leads to frustration.
MemDiff: A First Stab
Pondering
this problem, it came to me that viewing the process memory space at
the page level is a more logical approach than monitoring heaps. Thus
was born the MemDiff library. MemDiff is a crude yet effective way to
see how your process's use of memory changes between two points in your
code. Although I would have liked to have made MemDiff work on Windows
95 and Windows 98, too many APIs I needed are only available in Windows
NT.
MemDiff
consists of three simple functions. The first is MDTakeSnapshot, which
takes a process handle and returns a snapshot handle. A snapshot is a
relatively compact representation of your process's address space at the
time you call it. In the simplest usage scenario, you'll make two calls
to MDTakeSnapshot, once before the target section of code, and again
after the target code has executed.
The
second function, MDCompareSnapshot, compares the two snapshots to
report how much and where your address space has changed. In the report,
logically related pages (for instance, adjacent pages in a heap) are
lumped together. In addition, it attempts to provide a meaningful
description of the pages.
MDCompareSnapshot
writes its output to a standard Win32 file handle, which you provide.
This gives you flexibility for where the report goes. MDCompareSnapshot
has an optional verbose parameter. If it's not specified, the default is
a nonverbose report. This mode is usually easier to work with. More on
this later.
The
third function is MDFreeSnapshot. After calling MDCompareSnapshot,
you'll want to pass the snapshot handle to MDFreeSnapshot. As you'd
expect, it frees the associated memory, which can be a nontrivial
amount. A snapshot is good for only one comparison. As I'll show later,
the act of comparing two snapshots destroys some of the data. I could
have worked around this at the expense of additional memory and code
complexity. I chose the easy way out, and put safeguards in the code to
prevent erroneous results from using a snapshot more than once.
To
have the least side effects on your code, the downloadable sources
build MemDiff to a static .LIB file that you link into your application
or DLL. The .LIB file is built with the multithreaded C++ runtime
LIBCMT(D).DLL. If you want to use MemDiff from a language that doesn't
support static .LIBs (such as Visual Basic®), feel free to play with the
project settings to make the MemDiff project compile as a DLL. If you
do this, be sure to export the three functions mentioned previously.
Interpreting MemDiff Results
To create results for discussing here, I wrote a small sample program that exercises the MemDiff code. Figure 1
shows the code for MemDiffDemo.cpp, which uses all three MemDiff APIs.
Intermixed with the two MDTakeSnapshot calls, the program plays around
with heap memory, loads and unloads DLLs, VirtualAllocs memory, and
opens a memory-mapped file. The goal is to create an address space
scenario where MDCompareSnapshot has a variety of things to display.
For
output, MemDiffDemo writes to stdout. This lets me see the results in a
console window or redirect the output to a file with the >
redirection operator. In your own code, you'll probably want to use
CreateFile or some other API that returns a handle WriteFile can work
with. Heck, go nuts and use named pipes to see MemDiff results from a
process running on another machine. Who says I haven't embraced n-tier computing?
Figure 2
shows the normal, nonverbose output from running MemDiffDemo. The
first line shows the net difference in memory between the first and
second snapshots. A negative value means less memory was being used when
Snapshot2 was taken. The next line, Memory allocated, is how much newly
paged-in memory was in Snapshot2. Line three, Memory freed, tells how
much memory was present during Snapshot1, but not during Snapshot2.
The
fourth line breaks down the new memory in Snapshot2 into private and
shared amounts. Private memory is memory used solely by your process.
This includes heap memory and normal (non-shared) data sections in EXEs
or DLLs. Shared memory is potentially usable by more than just your
process. When looking for what's leaking in your application, the
private memory is usually more likely to be of concern.
Prime
examples of shared memory are code pages and read-only data sections.
Because these pages of memory don't change, the operating system can use
the CPU hardware to map the physical pages of RAM into multiple
address spaces. For example, every process uses code in NTDLL.DLL, but
the same physical pages of RAM holding NTDLL.DLL's code are shared
between all processes. Remember though, just because the memory can be
shared doesn't mean that your process isn't the only one using it.
After
the initial four lines, the remainder of the MemDiff output is an
accounting of memory that's unique to each snapshot. Although internally
MemDiff is working with a raw list of physically present pages, it
tries to coalesce related blocks. Under Snapshot2 in Figure 2,
you'll see a handful of memory blocks of different sizes. For example,
starting at address 0x00910000 are four pages of memory (16KB) that
belong to a Win32 heap. A bit later, you'll see 8KB used by USER32.DLL,
20KB used by GDI32.DLL, and so on. Stop and think about this. Various
amounts of memory are being reported for DLLs. The DLLs listed were all
loaded before Snapshot1 took place, yet Snapshot2 shows that they're
using additional memory. How can this be the case?
What
you're seeing is the demand page loading of Windows NT in action. The
code and data of an EXE or DLL aren't assigned to physical RAM until the
EXE or DLL is read from, written to, or executed. Whatever I did
between the two MemDiffDemo snapshots caused additional pages from
USER32, KERNEL32, GDI32, and NTDLL to be mapped in.
In
this case, the additional DLL pages to were mapped in because
MemDiffDemo loaded WININET.DLL, and then unloaded it. Note that no
pages from WININET.DLL show up in the report, since it's completely gone
from memory. However, whatever WININET.DLL did in its DllMain caused
pages from other system DLLs to be mapped in.
This is a great example of how your reported memory usage can go up
without you doing anything wrong.
Figure 3
shows a snippet of the verbose variation of MemDiff's results. To get
this version of the output, set the final parameter of MDCompareSnapshot
to true. The primary difference in the report is that all pages in the
coalesced blocks must be contiguous and have exactly the same page
attributes.
Since
coalesced blocks in verbose mode must have the same attribute, you'll
often see multiple blocks for a given DLL (although not in the case of Figure 3).
For instance, in a single DLL you may see one block reported for the
code pages, another block for the resources, and a third block for the
writable data section. Since all pages in a verbose report block have
the same attributes, I included the attributes in the output. Refer to
the QueryWorkingSet documentation for a description of the possible page
attributes.The MemDiff Source
Figure 4
shows the primary guts of MemDiff. MDTakeSnapshot uses QueryWorkingSet
from PSAPI.DLL, which I've included in the download files (Nov99Hood.exe
(31KB)) in case it's not on your system. QueryWorkingSet returns an
array of DWORDs, with each DWORD representing the address and page
attributes of a page mapped into the address space. Since I don't know
in advance how much memory QueryWorkingSet needs for the entire address
space, I call VirtualAlloc in a loop until I get enough memory to
hold all the page DWORDs.
The
snapshot handle that MDTakeSnapshot returns is just a pointer to the
VirtualAlloced memory. At the beginning of the snapshot memory is a
MEMDIFF_SNAPSHOT structure. Immediately following the MEMDIFF_SNAPSHOT
is the array of DWORDs that QueryWorkingSet fills in. The
MEMDIFF_SNAPSHOT members let me verify the validity of snapshots passed
to the other MemDiff functions, determine how many pages are in the
snapshot, and perform other housekeeping duties.
The
meat of the code is called from the MDCompareSnapshot function. The
code first verifies that both snapshot handle parameters are valid.
Next, the FilterOutCommonPages function throws out all pages that
shouldn't be in the reports that follow. It does this by sorting and
comparing both snapshots. A page that's in both snapshots is
automatically thrown out. In addition, the pages holding the snapshot
data itself are thrown out. Throwing out a page means setting its value
in the snapshot array to 0, which is why a snapshot is only good for one
comparison. Feel free to improve on this quick-and-dirty algorithm.
After
filtering down the snapshot data, MDCompareSnapshot then calls the
SummaryReport and DetailedReport functions. Both functions write their
output to the file handle you designate. The summary report is the
simple stuff at the beginning of the output. It simply spins through
both snapshots, counting the different types of pages as it goes. A
little math and voilà, your summary results!
The
DetailedReportHelper function is much more complicated since it has the
onerous task of identifying where a block of pages comes from. First,
though, the function has to coalesce related ranges of pages. This is
where the fVerbose parameter to MDCompareSnapshot comes in. With
fVerbose set to false, the code lumps together all pages that have the
same allocation base as reported by VirtualQuery. With fVerbose set
to true, all pages in a coalesced block must be contiguous, have the
same attributes as reported in the QueryWorkingSet bits, and must
share the same allocation base.
Once
coalesced, DetailedReportHelper first tries to identify the block by
calling GetModuleFileName. This easily finds pages that belong to the
memory image of a loaded EXE or DLL. If the page isn't in that category,
I next check to see if the block is in one of the process Win32 heaps.
This includes the default heap (GetProcessHeap) and any heaps created by
HeapCreate. Suballocator-style heaps, which use VirtualAlloc and
partition the memory themselves, won't be detected. The current Visual
C++ new and malloc fall into this category.
The
heap identification code is written as a class (CProcessHeaps), and
implemented in ProcessHeaps.cpp and ProcessHeaps.h. This code, which can
be found at the link at the top of this article, was written to be fast
rather than excruciatingly accurate. It's possible for some blocks to
slip by and not be identified. Feel free to fix my implementation to use
the HeapWalk API at the expense of CPU time. While you're at it, you
could add code to search whatever suballocator-style heaps are
present. Happy hunting!
Meanwhile
back at the block type identification code, Marlin has a few words on
insurance. Sorry, wrong storyline! If the block isn't identified by
GetModuleFileName, or by the heap code, the options are dwindling.
PSAPI.DLL has the GetMappedFileNameA API, which tells you if the page is
from a memory-mapped file. If it's not a memory-mapped file, you're out
of luck (at least in this episode). The memory could be plain old
VirtualAlloced memory, additional stack pages, or who knows what. If you
have a good algorithm for identifying what an arbitrary page of memory
is, by all means try it out. If it works well, let me know.MemDiff Caveats
I've
already mentioned some of the restrictions on the MemDiff library as
currently written. It's not easy to use from Visual Basic or other
languages that don't support static .LIBs. The solution is to rebuild
the code as a DLL and be aware of the side effects. And of course,
snapshots are only valid for one comparison.
Beyond
these usage restrictions, darker things lie. The big one is the
potential for misidentifying a block of memory, particularly in the
detailed report for the first snapshot. When a snapshot is taken,
MemDiff doesn't attempt to identify each block in the snapshot. To do so
would take extra time and memory. When the snapshots are compared, the
identification code only has the process state at the time of the
comparison to lean on. Since the process state can change dramatically
between snapshot and comparison time, there's plenty of opportunity for
error.
You
can easily construct scenarios where a page of memory used for one
thing at the time of a snapshot is used for something else entirely at
the time of identification (inside MDCompareSnapshot). The simplest
example is a DLL in place during Snapshot1 that unloads, then another
DLL loads in its place at the time of Snapshot2. The block
identification code is likely to identify Snapshot1 pages as belonging
to the wrong DLL. Alternatively, pages may go away entirely between
snapshots. In this case, the identification code fails completely and
reports <unknown>.
A
final note on strange results from MemDiff. You may occasionally see
pages with addresses above 2GB. Usually these addresses are something
like 0xC017F000. These are pages that Windows NT is using for storing
the process memory map. If the process memory space grows enough, the
kernel-mode code that manages the process memory space needs to allocate
another page of memory to hold page table entries. Note that the
discussion here assumes a "normal" Windows NT-based system with a
maximum user-mode address of 2GB. It's possible to boot Windows NT in
the /3GB mode where specially marked processes can use addresses up to
0xC0000000.Some Final Words
The
MemDiff library has the potential to help in many circumstances.
However, the code is still pretty rough. Given the restrictions of time
and space, I leaned toward simplicity. I'd expect MemDiff to be tweaked
and modified by others to better report on their specific scenarios.
Among
ideas for improvement, you could identify and tag memory pages at the
time of a snapshot, look for and include compiler runtime heaps, and
identify thread stack memory. (Remember, there might be multiple threads
in the process.) If your code VirtualAllocs memory and has identifiable
patterns to the data, its easy enough to add code to look for your
specific data.
Finally,
a big thank you to Osiris Pedroso at Autodesk for helping me test
MemDiff. Aside from just making suggestions, Osiris ran MemDiff on
AutoCad itself. The output files he sent back made me knuckle down to
come up with better algorithms than I originally wrote. MemDiff is
significantly better for his help.