A history of GlobalLock, part 3: Transitioning to Win32

Date:November 8, 2004 / year-entry #387
Tags:history
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20041108-00/?p=37363
Comments:    14
Summary:Now that you know how the 16-bit memory manager handled the global heap, it's time to see how this got transitioned to the new 32-bit world. The GlobalAlloc function continued to emulate all its previous moveability rules, but the return value of GlobalAlloc was no longer a selector since Win32 used the processor in "flat...

Now that you know how the 16-bit memory manager handled the global heap, it's time to see how this got transitioned to the new 32-bit world.

The GlobalAlloc function continued to emulate all its previous moveability rules, but the return value of GlobalAlloc was no longer a selector since Win32 used the processor in "flat mode".

This means that the old trick of caching a selector and reallocating the memory out from under it no longer worked.

Moveability semantics were preserved. Memory blocks still had a lock count, even though it didn't really accomplish anything since Win32 never compacted memory. (Recall that the purpose of the lock count was to prevent memory from moving during a compaction.)

Moveable memory and locking could have been eliminated completely, if it weren't for the GlobalFlags function. This function returns several strange bits of information—now entirely irrelevant—the most troubling of which is the lock count. Consequently, the charade of locking must be maintained just in case there's some application that actually snoops at the lock count, or a program that expected the GlobalReAlloc function to fail on a locked block.

Aside from that, moveable memory gets you nothing aside from overhead.

The LocalAlloc function also carries the moveability overhead, but since local memory was never passed between DLLs in Win16, the local heap functions don't carry as much 16-bit compatibility overhead as the global heap functions. LocalAlloc is preferred over GlobalAlloc in Win32 for that reason. (Of course, many functions require a specific type of memory allocation, in which case you don't have any choice. The clipboard, for example, requires moveable global handles, and COM requires use of the task allocator.)

Next time, an insight into how locking is implemented (even though it doesn't do anything).

Comments (14)
  1. Tom says:
    November 8, 2004 at 7:33 am

    QUOTE

    Consequently, the charade of locking must be maintained just in case there’s some application that actually snoops at the lock count, or a program that expected the GlobalReAlloc function to fail on a locked block.

    /QUOTE

    Does that mean that the Win32 team tried to remove locks and found that it broke some applications, or did they decide to emulate them from the start.

    How much of this stuff is determined in the design phase, and how much gets found in testing? Just curious really.

  2. Ivo says:
    November 8, 2004 at 9:30 am

    Why doesn’t Win32 compact memory? The application would be able to allocate and free big GMEM_MOVEABLE blocks and not worry about fragmentation of the address space.

  3. Cooney says:
    November 8, 2004 at 9:39 am

    I would imagine that, with rare exception, fragmentation isn’t an issue anymore. Each process has 4G, after all.

  4. Mike Dunn says:
    November 8, 2004 at 9:59 am

    If moveable memory is a thing of the past, what about the clipboard still requires the GMEM_MOVEABLE flag?

  5. Raymond Chen says:
    November 8, 2004 at 10:21 am

    Because there are apps that crash if the memory isn’t moveable.

  6. KJK::Hyperion says:
    November 8, 2004 at 11:06 am

    The clipboard doesn’t use the global heap anymore – nowadays, it’s based on the much more mondane shared memory. I guess that, since shared memory requires a nearly identical allocate/lock sequence, there would have been no value in devising an alternate API

  7. Andrei says:
    November 8, 2004 at 12:11 pm

    Cooney,

    each Win32 user process has 4Gb only on AMD64 boxes when they are run in WOW64. Normally, they have 2Gb or 3Gb of addressable virtual memory depending on whether /3Gb switch was used or not.

    Aside from that, fragmentation is an issue, you can have problems with fragmentation of the Win32 heap and fragmentation of virtual memory, depending on what memory manager you’re using.

  8. Nate says:
    November 8, 2004 at 1:32 pm

    Dare I ask what about non movable memory crashes certain programs? I cannot fathom what sort of practice would result in that, unless someone did something absurd like checking for the flags and deliberately crashing if they were not there.

    Then again, I bet Raymond has seen it all by now.

  9. Raymond Chen says:
    November 8, 2004 at 1:49 pm

    Some programs make undocumented assumptions about how moveable memory happens to be implemented. Here’s an MSDN article that describes the internals of the heap manager:

    http://msdn.microsoft.com/library/en-us/dngenlib/html/msdn_heapmm.asp

    Instead of using the documented GlobalLock function, those bad apps reach in and dereference the global handle table directly.

    You can see what goes wrong if the memory is not actually moveable.

  10. Raymond Chen says:
    November 8, 2004 at 9:54 am

    Ivo: You’d have to do more than that. You would have to make sure *every* allocation (especially the small ones) is allocated moveable so that a compaction will move it. Do you really want to go back to the world of locking all memory before using it (and remembering to unlock it when done)?

    For one thing, it means no more "new" operator (since "new" allocates non-moveable memory). I doubt many programs would be willing to give up "new".

  11. Darren Winsper says:
    November 9, 2004 at 3:06 am

    Andrei: isn’t the 4GB addressable area going to cause backwards compatibility problems for Win32 processes? The reason they’re not given 3GB by default in 32-bit is because programmers were always assuming a 2GB stack i.e. the highest bit being zero.

  12. Tom says:
    November 9, 2004 at 3:28 am

    Darren

    IIRC, only applications which are marked "large address aware" are give 4GB on AMD64.

    http://www.amd.com/us-en/assets/content_type/DownloadableAssets/Expand_Memory_of_32-bit_App_-_Microsoft_4GT-_6204.pdf

  13. Chep says:
    November 9, 2004 at 6:54 am

    Cooney: let me tell you that CAD/CAM applications are certainly starting to feel some pressure from 2GB/4GB heap fragmentation issues.

  14. foxyshadis says:
    November 10, 2004 at 9:42 pm

    Specialized apps like that should build their own memory manager. There’s no reason to mess with one that does the right thing in most circumstances unless there’s a major bug or it’s being redesigned from scratch. MS might work with them, but generally won’t rewrite its OS for one or two small segments of the population.

Comments are closed.


*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index