| Date: | March 1, 2013 / year-entry #59 |
| Tags: | code |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20130301-00/?p=5093 |
| Comments: | 42 |
| Summary: | Perhaps one of the most misunderstood sentences in the Win32 documentation is this little bit in the documentation for CreateFileMapping: If hFile is INVALID_HANDLE_VALUE, the calling process must also specify a size for the file mapping object in the dwMaximumSizeHigh and dwMaximumSizeLow parameters. In this scenario, CreateFileMapping creates a file mapping object of a specified... |
Perhaps one of the most misunderstood sentences in the Win32 documentation is this little bit in the documentation for
When people read the underlined portion, they interpret this to mean "The data in the file mapping object will be written to the system paging file." But that's not what it says. It says that it is backed by the system paging file. In other words, "If I need to page this memory out, I will store it in the system paging file." Note the word "if". Usually, people get all worked up about the description because "I don't want this data to be written to disk by the creator, and then read from the disk by the consumer. I want this to be stored in RAM, just like the memory I allocate with In other words, "backed by the system paging file" just means "handled like regular virtual memory." If the memory is freed before it ever gets paged out, then it will never get written to the system paging file. Just like you wanted. The documentation was written with kernel-colored glasses. They figured that you knew that paging file-backed memory was just a way of saying "normal pageable memory." Exercise: What happens if paging is disabled? Where is the memory backed if there is no paging file? |
Comments (42)
Comments are closed. |
If the paging file is switched off (or too small) then the CreateFileMapping would fail if there was not enough free ram to accommodate the request?
If there is no paging file, then the memory isn't backed up by anything, and you have to keep the pages resident in memory because there's nowhere to page them out to.
I'm with Skrobov: disabling paging means that all pages remain in physical RAM, of course with the exception of the ones corresponding to actual memory mapped files.
[Exercise: What happens if paging is disabled? Where is the memory backed if there is no paging file?]
If there is no pagefile for some reason or another, then RAM is backed by itself. If there's not enough RAM than allocations fail. I believe that Windows is strict in that regard that all there will be enough RAM + page to always meet allocated regions except for PAGE_*_WRITECOPY (which faults if it can't allocate). You can't really turn off paging as all loaded images are pagefiles (as much as I'd like it to be the case that a specific EXE is not locked while running there's no way to do that).
"(as much as I'd like it to be the case that a specific EXE is not locked while running there's no way to do that)."
You mean there is no way currently to do that in Windows — there are most certainly ways that it could be done, as other OSes have proved it possible. One way to do it would be to use reference counting to keep the mapped image backed by the same contents, even though the file system now points at new contents. And then as soon as the EXE is closed, the reference count would go to 0 and the old contents would be deallocated.
I assume it works the way it does in Windows because of limitations in older filesystems — though it would be nice if at least for EXEs on reasonably modern filesystems (like NTFS), we could overwrite EXE files even while open.
You can't overwrite an EXE file while it's open, but you can do the next best thing: if you rename the file (yes, you can rename am EXE or DLL while it's running) then you can write out a new one.
I don't think that images are just mapped to memory like memory mapped files, because you can write to images in memory without consecutively altering the file on disk (or in any other processes where the same image is loaded).
Where is the memory backed if there is no paging file?
In the cloud!
@CherryDT: It could use copy on write.
@dennis:
IIRC exe being locked is side effect of old memory saving measures, so executable can be paged out from memory in case or low free memory.
@Ray: "Usually, people get all worked up about the description because…". What people? The mechanism of paging is a long established one. This is so ancient and behaves the same way that none I know would be worked up about that sentences. And as others already said, if user defines "no paging" then you either create the memory mapped file and keep using it or you fail at CreateFileMapping.
@MarkV: "You can't overwrite an EXE file while it's open…". Are you sure? Are you betting money on that sentence? Use Google for "unlocker", a little free utility then talk again. Given enough rights you can do anything. Log in as Admin and you can overwrite entire memory, including kernel and if you want to have fun, with current systems of plenty of RAM, you can replace Windows with a Linux that has everything it needs in memory. Heck, you can even read entire memory, write it to disk (hibernate style so far), load the Linux, show you have Linux to your friends (little bragging), and when they left reload Windows from disk to continue your work.
@CherryDT: They are in fact mapped into memory EXACTLY like memory mapped files, in that both are mapped into your address space via NtMapViewOfSection. There's a minor difference in that executable sections get magic extra parsing in the kernel to map their sections and ASLR the image, but in terms of how memory gets back to disk, it's the same.
All of the pages in an EXE are copy-on-write unless you map it as PAGE_READWRITE (although WTF you'd do that is beyond me), so when you write to that page, you copy the page and that page is now backed by the pagefile instead of the image on disk.
@dennis, klimax
You can't write to a running executable on Unix, either, because they do the same thing and have done since the 70s. The relevant errno is ETXTBSY. Being able to delete it doesn't really delete the file, and I suspect that the reason that executables aren't opened with SHARE_DELETE is the same reason nothing else is: backwards compatibility.
@Danny: "what people?"
Probably the same people who think "virtual memory" means "disk" and who think that user-mode programs "allocate RAM".
"What happens if paging is disabled? Where is the memory backed if there is no paging file?"
It's backed by the full faith and credit of the United States Government.
Zing!
@Danny: "You can't overwrite an EXE file while it's open…". Are you sure? Are you betting money on that sentence? <tangent>
Obviously this was meant in terms of operations logically supported by the system. Modifying internal state of other processes or the kernel is generally not advisable.
@John: Obviously, but one is what the system suppose to do and else what you can do once inside having full power. Think malware – do you really think the people behind those care what the system is suppose to do? They will close your opened handles, read your data and generally piss in every corner of your computer because they CAN.
It's interesting that Windows NT has a distinction between VirtualAlloc and CreateFileMappingW(INVALID_HANDLE_VALUE) at all. You could build an entire memory management system on anonymous pagefile-backed section objects rather than VirtualAlloc. In fact, that's what Linux does.
I've been told that the reason is that NT's section objects have more kernel overhead than private memory allocations.
There're also programmers myth caused by that MSDN's pharse. I saw some people that believe that disabling pagefile causes CreateFIleMapping(INVALID_HANDLE_VALUE..) to fail.
BTW my observation: if system has lot of RAM. Really lot of RAM so you think that it doesnt need even be bothered by managing swap – its beter to disable it completely than set it size "minimal". Somewhy settings size to just small causes strange bugs and failures to allocate VM while there'is lot of RAM. It seems that if VMM works with swap enabled – i assumes that it always can allocate a swap space, an even if it really has a space in RAM, ut anyway if it decided o put something to swap and it FAILED – then it goes into some kind of panic frustraion. If swap is disabled – then it really uses RAM while there'is enough space to to all commited pages that not backed FS (file mappings and images).
Killer{R} – "MSDN's pharse" Haha (farce)
It gets backed to msdn man and blog pages. This has a huge performance and potential security issues, but little impact on overall readablity. /joking
@Killer{R}: Ever create a ramdrive and a pagefile on it? The system will page to it. It turns out with the right drivers the ramdrive can be created above the 4GB barrier on 32 bit Windows, which allows consumer grade Windows to make some use (turns out to be quite a bit) of the extra RAM.
Just don't turn on paging to ramdrive and hibernate at the same time. That combination is incompatible.
@Myria: If Linux truly uses sections underneath its heaps then that sounds like such a waste, especially when sections are essentially a fat layer over a VirtualAlloc-like primitive. But then again we ARE talking about the Model T equivalent of desktop OS kernels which used to hardcode the process limit and had a global kernel mode lock. Linux = eww gross
@Joshua: If you're running a 32-bit OS on a device that has more than 4GB of RAM, and then installing unsupported drivers such as a RAM disk to copy memory between < 4GB to > 4GB, then You're Doing It Wrong(TM).
Use x64 like a normal person. In fact, Breaching The 4GB RAM Limit Is The Primary Reason To Run 64-bit OSes.
Nitpicky comment: You can't turn off paging, unless you want every process to run in the same address space.
Nitpicky comment to nitpicky comment: Unless you're using segmentation, which no modern OS uses, and which doesn't exist on x86-64.
"[You'd be surprised how many people ask …"
I doubt it, actually. One of the hazards of listening to customers is that your dataset is skewed towards customers who don't understand, don't like, or don't think before pushing "Send". Nevertheless, I suspect that the *proportion* of programmers who are confused by the wording is rather small.
Nitpicky comment x 3: Paging != MMU. You can have separate MMU tables for each process without having a page file.
@immibis:
How would that work? Leaving aside the fact that protected mode means seperate virtual address spaces for each process, just think about the fact that the standard executable base address is 0x400000 for 32 bit processes, I think it is 0x140000000 for 64 bit processes. So if you think about it, all .exe files will try to load in at the same address (or close to each other if you take ASLR into account) and will overwrite each other. So no, the system relies on the concept that each user mode process has its own address space.
@Matt: Would you like to try your hand at obtaining 64 bit fixes for multiple accessibility issues. I have been abusing the pachability of 32 bit linkage on my workstation to fix accessibility bugs. (None of these get shipped anywhere). If I could get a 32 bit EE build of Windows Server 2008 R2 (not shipped), I'd use that (32 bit EE uses all the RAM on the system).
Long-time readers of this blog will know this isn't the first time a MSDN article is deemed by Raymond as "inaccurate", "prone to confusion", etc. My question is… why don't you fix it? And by "you" I don't mean Raymond, but Microsoft…
@Ken Hagan> Nevertheless, I suspect that the *proportion* of programmers who are confused by the wording is rather small.
I've even had someone tell me not to disable page file because file mapping will not work.
It's easy to overwrite the content of a running exe, just start it from (virtual) removable media or a (virtual) file share.
The file system can't guarantee anything.
@Matt: x86 CPUs supported >4gb years (decades?) before x86-64 was avalable.
"4gb is enough" – sure, we've had PAE since the Pentium Pro (1995)… but that's still an address space of 4GB, which is a lot less convenient than x64 Long Mode (introduced with Opteron in 2003 – so, yeah, almost a decade).
One thing is convenience, though, but I wonder if anybody has looked at performance of Long Mode address space versus having to re-map a 32bit window into the PAE address space. The required ring3<>ring0 switch and TLB flush has non-zero cost :)
People start to misunderatand when you are talking about FILES, without meaning "things you can click on in explorer and open with an app". Don't use the word FILE if it's not a regular file in the filesystem. Call it something else, because it IS something else.
To allocate shared memory with an API called CreateFileMapping is stupid. You should have called it AllocateSharedMemory instead.
@f0dder: For almost a decade, before x86-64, there was no choice of convenience, there was only the choice of using all memory or not. And the only way windows supported using >4gb was through pae or similar non-conveniece.
@4gig: I rather they called it CreateSection. You're forgetting that there's a companion function MapViewOfFile which breathes life into CreateFileMapping, and has no necessary link to sharing memory.
@Raymond: So what is the exact answer for you exercise? I always wanted to know that.
Though, even in user-mode, we could alloc the memory at non-paged pool to prevent page-out by VirtualLock. Couldn't we?
@Jeho Kim
The exact answer has been given already. It is backed by nothing (or still the system page file which has a size of 0).
The line "In other words, "backed by the system paging file" just means "handled like regular virtual memory."" basically says that the pages you get from CreateFileMapping with the file handle set to INVALID_HANDLE_VALUE work exactly like the pages that you get from VirtualAlloc, and the Windows memory manager uses this as the basis for the heaps, meaning that this is how the memory you normally use in a program that you write acts.
The CreateFileMapping/MapViewOfFile function names imply something here that isn't true. Windows only guarantees that the contents of memory is written to disk when you use a valid file handle in CreateFileMapping. So this is one of those situations where the function name wasn't the best choice.
#Klimax
You said "IIRC exe being locked is side effect of old memory saving measures, so executable can be paged out from memory in case or low free memory."
That's not quite true. The executable doesn't get paged out – its pages get discarded, and reloaded from the original file when referenced again. That's why the code has to be read-only.
"That's not quite true. The executable doesn't get paged out – its pages get discarded, and reloaded from the original file when referenced again. That's why the code has to be read-only."
Paegs that were modified – are paged out to page file. Pages that were not modified – are not. The actual protection of pages doesnt matter, its only mean if page can be modified and for pages of image view can be easily changed by VirtualProtect from readonly to copy-on-write and back.
@Random832
Linux, altrough it blocks writing to executable files with ETXTBUSY, does not block writing to libraries: e.g.
$ sudo dd if=/dev/zero of=/lib/i686/cmov/libc.so.6
[ 311.941962] Kernel panic – not syncing: Attempted to kill init!
@Blahblah: As a long-time reader, I get the impression that Raymond generally *does* ask that the MSDN pages be updated, when he happens to find those pages that are written incorrectly or confusingly.
The hibernate file?