The unfortunate interaction between LOAD_LIBRARY_AS_DATAFILE and DialogBox

Date:October 6, 2005 / year-entry #294
Tags:code;history
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20051006-09/?p=33883
Comments:    6
Summary:Some people have noticed that if you load a DLL with the LOAD_LIBRARY_AS_DATAFILE flag, you sometimes get strange behavior if you then pass that HINSTANCE to a dialog box function. The problem here is that since the bottom 16 bits of a proper HINSTANCE are always zero, different components have "borrowed" those bits for different...

Some people have noticed that if you load a DLL with the LOAD_LIBRARY_AS_DATAFILE flag, you sometimes get strange behavior if you then pass that HINSTANCE to a dialog box function.

The problem here is that since the bottom 16 bits of a proper HINSTANCE are always zero, different components have "borrowed" those bits for different purposes. The kernel uses the bottom bit to distinguish modules loaded by having been mapped into memory as sections (i.e., loaded normally) from those who have been mapped as one giant block (loaded as a datafile). It needs to know this so that the various resource-management functions such as the FindResource function know how to interpret the data in order to locate the resource in question. Although everybody now knows that the HINSTANCE is the base address of the DLL, in principle, it is an opaque value (and in the 16-bit world, the value was indeed opaque).

Meanwhile, the window manager has its own problems. In order to support 16-bit applications running seamlessly on the desktop (not in a virtual machine, as discussed earlier), as well as thunking between 16-bit and 32-bit code, it needs to accept both 32-bit HINSTANCE values as well as 16-bit HINSTANCE values. Since memory allocation granularity is 64KB, the window manager knows that valid 32-bit HINSTANCEs have zero in the bottom 16 bits, whereas 16-bit HINSTANCE values are nonzero there.

Perhaps you see the conflict now.

If you pass the instance handle of a DLL loaded as a datafile, the kernel will set the bottom bit as a signal to itself to locate its resources in the flat datafile manner rather than in the mapped DLL manner. But the window manager sees that the bottom 16 bits are not all zero and assumes that it has been given a 16-bit HINSTANCE value.

Amazingly, this doesn't cause a problem most of the time because the things that need to be handled differently between 32-bit and 16-bit HINSTANCEs are relatively minor. The one that is most likely to bite you is the dialog instance data segment.

In 16-bit Windows, a dialog box came with its own data segment, which was used as the local data segment for controls hosted by that dialog box. Most controls didn't need a lot of storage in the local data segment, so the issue of where it came from wasn't really important. The big exception was edit controls, since they can contain multiple kilobytes of text. a dozen kilobytes of text may very well not fit in the application's data segment. Therefore, creating a new data segment gave the edit controls on the dialog a new 64KB block of memory to store their data in. Programs were expected to extract the data from the edit control via mechanisms such as the GetWindowText function and store the result someplace that had the capacity to handle it (outside the cramped local data segment).

In order to maintain compatibility with 16-bit programs who are expecting this behavior to continue, the window manager, when it sees a 16-bit HINSTANCE, dutifully creates a 16-bit data segment in which to store the data for the edit controls, using a helper function provided by the 16-bit emulation layer. But if you aren't really a 16-bit program, then the 16-bit emulation layer is not active, and consequently it never got a chance to tell the window manager how to create one of these compatibility segments. Result: Crash.

The solution is to add the DS_LOCALEDIT style to your dialog box styles. This flag means "Do not create a dialog box data segment; just keep using the data segment of the caller." Therefore, when your LOAD_LIBRARY_AS_DATAFILE HINSTANCE is mistaken for a 16-bit dialog template, the dialog manager won't try to create a dialog box data segment and therefore won't call that function that doesn't exist.

I believe this issue has been resolved in Windows XP SP 2. The window manager uses a different mechanism to detect that it is being asked to create a dialog box on behalf of a 16-bit program and is no longer faked out by the faux-HINSTANCEs produced by the LoadLibraryEx function.

Comments (6)
  1. Heston Holtmann says:
    October 6, 2005 at 2:12 pm
    1. When did this bug come into existence; i.e; When Win95 was developed?

      2. When did Microsoft become aware of this bug?

      3. Why did it take so long to finaly ship the fix?

      Just curious?

  2. Daev says:
    October 7, 2005 at 4:00 am

    Craig:

    I’d strongly recommend "Windows System Programming" (Third Edition) by Johnson M. Hart. It’s a clear, professional study of Win32 for experienced Unix and VMS programmers.

    The issue with malloc & free between different DLLs is this: if you choose to build each DLL statically linked and bound to its own copy of the C run-time library, then each DLL will maintain its own heap, its own streams and file handles, its own errno, etc.

    Instead, link your C programs and DLLs against the "multithreaded DLL" version of the library (using the /MD compiler flag), which makes them all share a single copy of the MSVCRT.DLL library.

  3. Craig Ringer says:
    October 6, 2005 at 11:40 pm

    This sort of thing is pretty amazing. I don’t envy the Windows team their job of maintaining source and binary compatibility for 15-year-old applications while trying to produce a secure, reliable OS for users who don’t want to sacrifice anything for security or reliability.

    Eek.

    In fairness, this sort of scary quirk is why I avoid win32 programming like the plague. On the other hand, if I had a large body of existing win32 code to maintain I imagine I’d be very thankful indeed for this sort of compatibility hack.

    On a side note, is there any MSDN coverage of safe memory handling in DLLs and safe pointer use between DLLs? I found very little, and most information I’ve seen has been from other (perhaps outdated) sources. As a UNIX guy who’s dabbling with Windows, this is immensely frustrating – on UNIX, shared libraries seem *WAY* simpler. It’s very hard to find out what you can and can not safely do with/in DLLs on win32. I mean things like "allocate memory in DLL 1 and free it in DLL 2". It’s even harder to find out what limitations are due to the inherent architecture of DLLs on win32, and what is due to binary compatibility problems caused by the need to support runtime linking DLLs built by three different compilers into an executable built by a fourth.

    Maybe MSDN could use a section with references for people coming from UNIX/Linux platforms – they’ll understand C and C++, but not the win32 API, process model, threading model, shared library handling, etc. A section with links, and where possible explanations of the win32 systems in terms of the UNIX/Linux approach to each problem, would be really handy.

  4. Andreas Haeber says:
    October 7, 2005 at 8:06 am

    Maybe a bit late to post a comment here now.

    Craig, if you ever read this page again: Look at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnucmg/html/UCMGch09.asp for a comparison of Win32 and UNIX.

  5. josh says:
    October 7, 2005 at 2:01 am

    If you’re going straight to Win32 API for your memory management, it doesn’t care whether you do stuff in a DLL, the EXE, separate DLLs, whatever. It’s all a single process.

    If you link two pieces of code against different *instances* of some runtime library and use it to manage memory, you could have problems. The C and C++ runtime libraries have a significant amount of internal state, including their own private heaps (apparently…). The rule is, any time state is created in the runtime in one DLL, don’t expect that state to exist in another.

    However, if you link all DLLs against the same version of the multithreaded dll version of the runtime, they’ll share a single instance and the problem goes away. That’s the proper solution.

    Or at least, that’s how I remember it…

  6. Neil says:
    October 7, 2005 at 7:16 am

    > the window manager knows that valid 32-bit HINSTANCEs have zero in the bottom 16 bits, whereas 16-bit HINSTANCE values are nonzero there.

    Presumably 16-bit HINSTANCE values are zero in the *top* 16 bits, whereas 32-bit HINSTANCEs are nonzero there.

    Or am I overlooking the obvious?

Comments are closed.


*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index