Using DuplicateHandle to help manage the ownership of kernel handles

Date:October 20, 2016 / year-entry #221
Tags:code
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20161020-00/?p=94555
Comments:    12
Summary:That one's mine.

A customer was using a third party I/O library that also gave you access to the underlying HANDLE, in case you needed it. The customer needed that underlying HANDLE so it could pass it to _open_osfhandle() and get a C runtime file descriptor, which could then be used by other C runtime functions to operate on the I/O object.

Everything was great until it came time to close the I/O object, because both the I/O library and the C runtime tried to close the handle. and that resulted in assertion failures due to invalid handles.

The problem here is that both the I/O library and the C runtime think that they are responsible for closing the handle. The I/O library wants to close the handle because it created the handle in the first place, and the special method to obtain that underlying HANDLE wasn't transferring ownership of the handle to you; it merely gave you a handle that you could borrow. On the other hand, the _open_osfhandle() function will close the handle when the file descriptor is closed, because the function assumes that you're giving it not only the handle, but also the responsibility to close the handle.

Neither library has a way to change the handle semantics. There isn't a way to tell the I/O library or the C runtime, "Hey, don't close that handle."

The solution here is to use the Duplicate­Handle function to create a brand new handle that refers to the same underlying kernel object. You can then pass the duplicate to _open_osfhandle(). Both the I/O library and the C runtime library will close their respective handles. Since each handle is closed exactly once, balance is restored to the universe.

Exercise (easy): Suppose you have a C runtime file descriptor, and you want to take the underlying kernel handle and give it to another library, which will close the handle you give it. How do you manage this without running into a double-close bug?

Exercise (slightly harder): Suppose your program needs more than 2048 C runtime file descriptors, which is more than _setmaxstdio accepts. Fortunately, your program doesn't actively use all of the descriptors at the same time, so you're thinking that you can virtualize the file descriptors by "paging them in" and "paging them out" from underlying kernel handles. How would you do this?

Comments (12)
  1. ZLB says:
    October 20, 2016 at 7:38 am

    Easy Excercise: use _get_osfhandle(…) to get a HANDLE from a file descriptor, the DuplicateHandle() it and pass the duplicate to the library.

  2. SimonRev says:
    October 20, 2016 at 8:12 am

    To the harder question — This feels a lot like when someone asks why they cannot have more than 10000 GDI objects open at the same time. To which the answer is almost always: You are doing it wrong. Sure you could use DuplicateHandle with some of the machinery in the stdio library to get the underlying OS handle, but it still feels very wrong. If you don’t need 2048 simultaneous file descriptors, then close them when you don’t need them. If you do need 2048 simultaneous file descriptors, redesign your program.

    1. MNGoldenEagle says:
      October 20, 2016 at 11:50 am

      What if you’re designing a file server or CDN?

      1. Dmitry says:
        October 20, 2016 at 2:47 pm

        If the main point of your design is doing heavy lifting in some specific area, then you probably will not get away with one-liner. Standard libraries are designed for convenience in the first place, not for some extreme edge cases.

    2. Antonio Rodríguez says:
      October 20, 2016 at 12:34 pm

      I first though “if you have to asks the limit, you are doing it wrong”. Then I remembered cases like file servers, database servers, etc. Even with caching, it’s reasonable to have hundreds or thousands of files opened at the same time in those cases. It’s not the 80s anymore, and Windows is used as a server OS, so its API and C runtime have to support those scenarios.

  3. Darran Rowe says:
    October 20, 2016 at 8:22 am

    For the harder one, I would say that you keep a table of your Windows handles and permissions and ask for a C runtime file handle based on that Windows handle. You can then duplicate the handle and use _open_osfhandle to get the file handle.
    Once the handle has been used, you can then close the C runtime file handle. This allows reuse of the C runtime while keeping everything open.

    1. Klimax says:
      October 20, 2016 at 11:22 am

      I don’t think there’s better answer. (Well, I though I might get my answer before other… :D )

  4. Florian S. says:
    October 20, 2016 at 9:37 am

    Correct me if I’m wrong, but wouldn’t it be better if the third-party library created a duplicate of the handle before it actually returns it to the caller? I mean, if you were to implement it… :)

    1. Joshua says:
      October 20, 2016 at 11:52 am

      Nope. Almost never does the callee need to hang on to the handle. Usually stuff like this exists for calling DeviceIoControl.

    2. Antonio Rodríguez says:
      October 20, 2016 at 12:29 pm

      As Joshua says, it’s reasonable that the library provides the original handle: if you need it for short-term use, you use it directly. And if you need to make longer use, it’s your responsibility to duplicate and close it. As long as it is documented, it’s perfectly logical and valid.

      In any case, it’s futile discussing whether the library should duplicate the handle or not. It’s a third party library, and you can’t modify it. All you can do is to use it or to work around its bugs or design problems.

  5. Josh B says:
    October 21, 2016 at 11:50 pm

    I’m hoping Raymond comes in to give his answer to the hard one before comments close, rather than sticking it on the queue for 2020. That sounds like something that might come in handy someday.

    1. Klimax says:
      October 22, 2016 at 3:14 pm

      It’s not really hard and most of tools are already present. The only missing piece is _get_osfhandle.

Comments are closed.


*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index