Some questions about unflushed data and calling FlushFileBuffers on a new handle to a file

Date:May 24, 2017 / year-entry #122
Tags:code
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20170524-00/?p=96215
Comments:    11
Summary:Will it get flushed out eventually?

Consider the following sequence of events:

  1. Process A opens a file with Create­File with attributes that include neither FILE_FLAG_NO_BUFFERING nor FILE_FLAG_WRITE_THROUGH.
  2. Process A writes to the file with Write­File. These writes are internally buffered since we didn't disable buffering.
  3. Process A crashes without calling Close­Handle, and before the operating system's internal buffers are flushed to disk.

First question: Under these conditions, will the data written in step 2 be lazy-written to disk? Or is it at risk of being lost forever because the handle wasn't closed?

Let's look at the last part first. Whether the process closed the handle before crashing doesn't affect the story, because the kernel will close all the handles as part of process cleanup. The handle does get closed eventually. Whether the handle closure was done explicitly by the app or implicitly by the kernel doesn't affect the answer.

Okay, now let's look at the first part: Yes, the data written in step 2 will eventually be lazy-written to disk, assuming your system doesn't crash before then.

And that's the middle part of the question: The data is at risk of being lost forever, but not because the handle wasn't closed. It's at risk of being lost forever because the system might crash before it gets flushed out.

Okay, let's extend the scenario:

  1. Before the operating system flushes its internal buffers naturally, Process B opens the same file, with the same attributes.
  2. Process B calls Flush­File­Buffers.

Does this call to Flush­File­Buffers cause the data written by Process A to be flushed to disk?

Yes. A call to Flush­File­Buffers will flush data for that file, even if the data was written by a different handle.

If Flush­File­Buffers is never called, then the operating system will flush the buffer at its convenience.

Note: In step 2, the relevant call is Write­File. If you write the data to the file using a runtime-provided function like fwrite, then that data might be sitting in the runtime's buffer without ever triggering a Write­File. Only when the data is written with Write­File does the data actually reach the operating system's buffers.

Comments (11)
  1. henke37 says:
    May 24, 2017 at 8:19 am

    Different handle to the same open file object or different handle to the same open file object? You should be clearer, even if only one is what you’ve possibly meant.

    Also note that WriteFile has to complete before the system will ensure that the data will not be lost unless the system crashes.

    1. Sergei Vorobiev says:
      May 24, 2017 at 1:42 pm

      Those buffers belong to stream control block, not to file object.

    2. Wombat says:
      June 4, 2017 at 10:50 pm

      “Different handle to the same open file object or different handle to the same open file object?”

      What’s the difference? All I can see is one has D for different, and the other has d. Otherwise you have said the same thing twice.

      I’m sure you meant something, but I cannot tell what.

  2. StefanH says:
    May 24, 2017 at 9:51 am

    I must have been doing to much Win32 programming, because I usually gain a lot of insight from this blog. Today, however, I already knew the answers and couldn’t imagine it working any other way.

  3. cheong00 says:
    May 25, 2017 at 1:55 am

    Need some clarification.

    Does it mean “the handle to Process A and B points to the same file buffer”, or is it just that “the call to FlushFileBuffers() will flush all file buffers attached to the underlying file record regardless of which file handle you’re calling the function from”?

    1. StefanH says:
      May 25, 2017 at 9:55 am

      There is only one file object and hence one file buffer even when multiple processes have multiple handles to it. Hence, if you call FlushFileBuffers(), that’s the buffer that’s flushed.

      1. Sergei Vorobiev says:
        May 26, 2017 at 10:19 pm

        As long as you don’t mind I put on my kernel-colored glasses.

        FILE_OBJECT is what HANDLE is in kernel. there are multiple streams in a file. And cache is fully coherent. Therefore there is one Stream Control Block. The File Objects point to it. Your handles are um nicknames for File Objects.

        Yes, there is one SCB. You are correct in the assumption that multiple users of the same stream see the same bytes.

  4. alegr1 says:
    May 25, 2017 at 8:13 am

    Does WriteFile on a file opened with FILE_FLAG_NO_BUFFERING also updates/invalidates any copies of the data in cache? I would expect that for data coherency reasons.

  5. Some Handles Don't Flush? says:
    May 25, 2017 at 4:42 pm

    –“Process B opens the same file, with the same attributes”
    But if Process B uses different attributes for the same file, this behavior is not guaranteed? I could not find anything in FlushFileBuffers, CreateFile, SetFileAttributes, “Creating and Opening Files”, that said this. It seems sensible that Flush”File”Buffers would flush all buffers associated with the “file object”, no matter which handle was used to invoke the action. But, if some handles to the file don’t invoke the same action, because different attributes were specified when the handle was created, that seems like something one should know.

    1. Sergei Vorobiev says:
      May 26, 2017 at 10:21 pm

      It is guaranteed. There is one Stream Control Block.

  6. Sergei Vorobiev says:
    May 26, 2017 at 10:26 pm

    Raymond, can we have the next installment discussing FlushFileBuffers() and network shares?

Comments are closed.


*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index