| Date: | August 14, 2006 / year-entry #273 |
| Tags: | other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20060814-06/?p=30133 |
| Comments: | 34 |
| Summary: | Here's a question that came from a customer: I have set my boot options to /3GB /NOPAE. Even though the server has 4GB of physical memory, we see only 3.5GB of physical memory on the server after it boots up. However, when we change the boot options to /PAE, we can observe 4GB of memory.... |
Here's a question that came from a customer:
Remember that in the absence of the /PAE switch, the Windows memory manager is limited to a 4GB physical address space. Most of that address space is filled with RAM, but not all of it. Memory-mapped devices (such as your video card) will use some of that physical address space, as will the BIOS ROMs. After all the non-memory devices have had their say, there will be less than 4GB of address space available for RAM below the 4GB physical address boundary. The rest of it went above the boundary. Here's a picture:
On this imaginary system, the motherboard assigned the ROMs and the hardware devices to the physical address space between 3.5GB and 4GB (occupying 0.5GB of address space). When you start plugging in your memory chips, then, they are assigned physical addresses starting at the bottom, and then skip over the address space that has already been assigned to the hardware and ROM, then resume. On this imaginary system, then, the 0.5GB of address space used for hardware and ROMs causes that much memory to get shoved upwards, and it ends up above the 4GB boundary. Without PAE, the processor is capable only of addressing memory below the 4GB boundary, which means that the memory above that boundary is inaccessible. It's consuming electricity but isn't doing anything. The solution is to enable /PAE so that the processor can access the physical address space above the 4GB boundary. Because that's what PAE stands for: Physical Address Extension. |
Comments (34)
Comments are closed. |
Memory limits again. It seems not that long ago the questions where: why is not my newly bought 1MB memory seen by Windows? and the answer was to check the HIMEM.SYS line in CONFIG.SYS, or the EMM386.EXE etc..
But the good news are we do learn from history, thanks to the wonders of virtual memory /PAE requires no app modifications (as compared to AWE or HIMEM.SYS).
Rgrds Henry
No application modifications – but if you have any devices doing DMA, and you almost certainly do these days, you’ll hit other problems: even if the OS can put data above the 4GB barrier, that doesn’t mean the hard drive controller, video card etc can access it!
Result: just as back in the old days of the 1 and 16MB barriers, you end up needing to allocate bounce buffers in "low" memory and copy data to/from those buffers. (Unless your CPU is civilized enough to have an IOMMU, which can map a window of virtual memory for hardware in the same way the OS does for applications – unfortunately, it seems the Intel x86-64 chips lack this feature.)
Worse, you can run out of space in this bounce buffer area (whichever approach you used) – at which point, I/O to non-64-bit devices stops. Not nice when this is your shiny new fileserver, as happened to me earlier this year.
My understanding was that the hardware memory decoding was unchanged, but rather that booting /PAE enabled the OS to configure the memory-mapped I/O devices to live above 4GB. The I/O devices’ use of resources can be remapped, while the memory decoding can be simple. You still get small holes for any devices that can’t be moved.
However, there are still many drivers out there that can’t handle addresses over 4GB. This caused an interesting design issue for XP SP2: to enable the No Execute behaviour on recent AMD and Intel processors, the processor must be put in PAE mode. But drivers can’t handle addresses over 4GB. What to do for compatibility? Microsoft decided that, if /NoExecute was specified, hardware support was available, but /PAE was not specified, it would still map memory-mapped I/O devices below 4GB.
The /3GB switch is a complete red herring as it applies to user-mode process virtual address space, and has nothing to do with physical memory.
As far as I know, enabling PAE will cause a performance hit (in the order of a few %) because of the extra translations needed in the CPU to address memory.
I have a desktop in the office that was recently co-opted for use as a VM host, and given 4 1GB DIMMs to help in it’s new job.
However, Windows (2003 Standard) is only reporting 3.5GB of memory, even with the /PAE switch.
This is a recent Dell box (not positive on the chipset, but it’s a Pentium D CPU). Is this common on desktops to not be able to use PAE to access the last bit of memory?
I would imagine it’s typically off by default as most boxes don’t use that much memory.
Was this thing ordered originally to be a workstation or a server? If it originally came with, say, 1GB of memory, it’d be pointless and possibly somewhat wasteful of CPU to set PAE.
kinkie: The processor has a Translation Lookaside Buffer which contains recent virtual to physical mappings. This is a hardware circuit that searches all entries in parallel, rather than sequentially, so in most cases there is no delay to physical memory access (from paging support, at least). I don’t think PAE affects the efficiency of the TLB at all – all TLB entries are the same width regardless of whether the processor is in PAE mode or not.
Without a TLB any paging system would be horribly, probably prohibitively, expensive. x64 has a four-level page directory/table architecture, IIRC, which is one more than 32-bit PAE.
The x86 architecture manages its TLB automatically – any access to a page that isn’t in the TLB will cause the processor to automatically read the page tables and, if the page table entry corresponding to the page is valid, it will place the entry in the TLB (potentially discarding another entry already present). If the PTE isn’t valid a page fault exception occurs. Some other architectures require the TLB to be managed by software – a TLB cache miss causes an exception, IIRC, and the OS then has to load the TLB with the correct data. Windows CE used to do this on all architectures, not building a complete set of page tables (showing its SHx and MIPS heritage), but as of CE 4.0 now does keep page tables in RAM for x86 and ARM architectures. This gave a significant speed-up.
When the OS changes any kind of page mapping, it must remove any affected entries from the TLB, otherwise the processors’ mappings could be stale.
So, this is memory on the devices that is being mapped into the virtual address space ?
i.e, on a 2GB RAM system, since the devices are still in the 3.5-4.0GB address space, we seem to lose no RAM ?
Would I be correct in phrasing this as: “virtual addresses outside the 3.5-4GB bracket are backed by RAM, while those in this range are backed by on-device/ROM memory” ?
On a slight tangent… we had a program that was compiled with /LARGEADDRESSAWARE that worked fine with /3GB but occasionally crashed on Wow64. Turns out that pointers are signed datatypes! (I suppose it kind of makes sense to keep programs that use -1 as a special pointer value working).
So guess what happens when you assign an address ReadFileScatter’s FILE_SEGMENT_ELEMENT PVOID64? It gets sign extended if your memory address is above the 2GB boundary. This isn’t a problem in 32bit mode since the top DWORD gets ignored, but on Wow64… kaboom!
Hmmm… makes me wonder… is the PVOID64 actually useful for a 32bit app? Is there some magic way to perhaps use it to read data directly into AWE allocated pages without actually mapping them?
Jonathan: I’m in almost the exact same situation. Seeing your comment, I thought I wrote it!
Anyway, my story is: I have a Dell Precision 370. Originally it had 1GB RAM, and Windows Server 2003 Standard on it. As soon as I installed Windows, System properties said "Physical Address Extension", so I guess it was on.
Later, I added more memory to 4GB. BIOS shows 4GB correctly, but Windows shows only 3.25GB.
Answer: http://www.interact-sw.co.uk/iangblog/2005/08/05/is3gbenough
/PAE doesn’t always help. There are quite a few motherboards out there that only support a 32-bit physical address range. The top 4 address pins on the CPU that PAE uses are not wired to anything. (Or at least, not wired to anything useful.)
These systems can’t do the trick of shifting the last half gig of RAM up into the 4-4.5GB address range. So if you plug 4GB of RAM into such a system, you simply won’t be able to use the top half gig or so because the hardware is incapable of making it visible to the CPU.
/PAE enables the OS to use the top 4 bits of the CPU’s address bus, but it doesn’t enable windows to add new wires to your motherboard’s PCB, or add new transistors to your motherboard’s chipset…
Raymond Chen explains why newer servers cannot see the full 4GB
in the machine until you add the /PAE…
kristof, that sounds like a bug in wow64.dll.
64 and 32 bit code can (and does) coexist within a process *. When a system call is done to the 32 bit ntdll.dll, ntdll does a user-mode far call to wow64cpu.dll. That DLL provides CPU emulation (on x86-64, it runs code directly). wow64cpu.dll calls wow64.dll.
wow64.dll’s job is to translate NT system calls from 32 bit to 64 bit. It handles structure member translation. Normally, it makes a point to zero-extend all parameters. It sounds like its handling of NtReadFileScatter is incorrect and does a sign extension instead.
* Microsoft has implemented "features" to deliberately hide this, such as NtMapViewOfSection not working with SEC_IMAGE on 32 bit images in a 64 bit process. There is no reason for this restriction other than to discourage developers from doing it.
I’ve wondered about this on my XP box but when I check my Server 2003 Enterprise box it already seems to be doing this and adding the /3GB /PAE switches made no difference. Is that a Server 2003 Enterprise thing?
It is time to move on to 64-bit machines, because we are slowly and steadily reaching the 4 GB physical address limit with our memory intensive applications. I can’t seem to work properly these days without having a 1GB RAM on 32-bit systems. More the memory on the system, life gets easier/happier.
I seem to remember reading in MSDN that Windows XP no longer accesses any memory that gets remapped above the 4GB point (in real addresses) because too many drivers couldn’t handle it. If I understood correctly, Windows XP RTM honoured the /PAE switch the same way as Windows 2003 does, but XP after either SP1 or SP2 stopped doing so. The /PAE switch still helps stop malware though.
Also I’ve been told that there are a lot of motherboards which can’t really handle 4GB even though their specs say they can. I only returned 3 that didn’t match their specs this way but I’ve been told that it’s a lot more common than that. Some motherboards have hardware and BIOSes that solve some parts of the remapping above the 4GB point but don’t solve enough to really work, and the makers didn’t really test running an OS with 4GB of memory installed.
Also in some motherboards that really do work properly, it can be hard to find the BIOS options necessary to enable the remapping.
Mike Dimmick said that remapping applies to memory-mapped I/O devices. I’m pretty sure that this isn’t usually true. Motherboards (working ones) remap part of the memory above the 4GB point because I/O devices stay below the 4GB point.
@kristof:
PVOID64 is generally not useful for 32 bit apps. In the case of FILE_SEGMENT_ELEMENT you’re supposed to use PtrToPtr64() macro to avoid problems with sign extension on 32 bit (this should probably be mentioned in the docs – I’ll forward it to the doc team).
Yup, we used PtrToPtr64 once we knew about it :)
Why a 32bit app should be required to know about 64bit pointers, I’m sure there’s some good reason for it. Right?!
I have the same problem of not being able to see (or use) the 4G memory. I am using the PowerEdge SC420 and Windows XP professional SP2. I see the "Physical Address Extension" (PAE is on)in the "Properties", but it still shows 3G, although I can see 4G in BIOS and 4G using Sandra 2007. In the specifications of the SC420, it should be able to support up to 4G memory and the 64bit system. I guess I am going to 1. try removing the SP2. 2. try a 64bit Linux to see if i can see the full 4G.
If anybody is using PowerEdge as well, please share your case with me, thanks.
I have the same problem of not being able to see (or use) the 4G memory. I am using the PowerEdge SC420 and Windows XP professional SP2. I see the "Physical Address Extension" (PAE is on)in the "Properties", but it still shows 3G, although I can see 4G in BIOS and 4G using Sandra 2007. In the specifications of the SC420, it should be able to support up to 4G memory and the 64bit system. I guess I am going to 1. try removing the SP2. 2. try a 64bit Linux to see if i can see the full 4G.
If anybody is using PowerEdge as well, please share your case with me, thanks.
XP SP2 changed things so that it will not access physical addresses above 4GB anymore, even in /PAE mode
(and most system now run in PAE if they have DEP enabled)
This is mentioned in
http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/sp2mempr.mspx
"To constrain compatibility issues, Windows XP Service Pack 2 includes hardware abstraction layer (HAL) changes that mimic the 32-bit HAL DMA behavior. The altered HAL grants unlimited map registers when the system is running in PAE mode. In addition, the kernel memory manager ignores any physical address above 4 GB. Any system RAM beyond the 4 GB barrier would be made unaddressable by Windows and be unusable in the system. By limiting the address space to 4 GB, devices with 32-bit DMA bus master capability will not see a transaction with an address above the 4 GB barrier. Because these changes remove the need to double-buffer the transactions, they avoid a class of bugs in some drivers related to proper implementation of double buffering support."
Hi guys,
I just got the following PDF file from Intel addressing the chipset issue of not being able to use the full amount of 4GB. I am wondering if the mapping problem still applies in a 64bit system.
ftp://download.intel.com/support/motherboards/server/sb/ta71902.pdf
Thanks.
Anyway, it wasn’t too long ago that I got that dream laptop, a Dell Precision M90. They kind of screwed
It caps the physical address.
PingBack from http://www.DailyCupOfTech.com/2007/03/11/mia-ram-in-32-bit-vista/
The consumer versions of 32-bit Windows XP and Vista have a stated limit of 4 GB RAM, but a practical
I think this (http://blogs.msdn.com/oldnewthing/archive/2006/08/14/699521.aspx) is the problem. There are definitely 4 1GB sticks of RAM in my machine, so I doubt it’s hardware. Plus, when I boot the computer, the BIOS sees the full 4 GB. So it’s som
PingBack from http://www.simonandjun.com/wordpress/index.php/2008/01/18/windows-2003-with-4gb-memory/
An experiment — I am wondering the spam filters will pick up comments with the above title as spam,
PingBack from http://www.tomshw.it/forum/schede-madri-ram/85948-perche-non-mi-vengono-riconosciute-tutte-le-ram-3.html#post885060
PingBack from http://www.zimbra.com/forums/administrators/20683-solved-if-my-server-has-4gb-memory-how-many-users-can-support-2.html#post102655
PingBack from http://castalie.comoj.com/?p=248
PingBack from http://agix.wordpress.com/2009/04/11/4-gb-di-ram-su-windows-vista-co/