32-bit Windows memory management: Can it see more than 3ish GB?

[Thorin]

Okay, so the question is:

When there is 4GB of physical RAM available in a PC, can 32-bit Windows access all of it?
Meaning can it map virtual user address space and/or virtual kernel address space to every single bit of every single byte of that 4GB?

Assumptions and Rules for this Discussion:
- When we say 4GB we mean 2^32 bytes
- Specify whether you mean XP or Vista
- Please use the correct terms
- If someone uses a term incorrectly, politely inform them
- If someone's pointed out your incorrect term don't edit your post, or their post won't make sense
- Don't post in a holier-than-thou tone of voice - proofread and imagine what tone it might seem like to someone else
- Provide credible links to back up statements, especially if people ask for them
- Read links provided before posting a rebuttal
- Only post when you have something to add to the discussion - don't just nitpick

[Lazybones]

I thought I had put this one to rest with URLs to back it up

Limits
http://msdn2.microsoft.com/en-us/library/aa366778.aspx#physical_memory_limits_windows_xp

This link talks about tuning a system with the /3GB flag and application support.
http://msdn2.microsoft.com/en-us/library/bb613473(VS.85).aspx

More on /3GB and PAE memory options also covers how the memory is split up.. (applies to 32bit 2k,xp,2k3)
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

- When we say 4GB we mean 2^32 bytes
MS just lists it as 4GB, they state the OS can see, and address up to this amount of physical memory

- Specify whether you mean XP or Vista
The first link covers the limits in 32bit and 64bit vista as well

On a system equipped with 4GB of physical memory on windows XP 32Bit the default behavior is for 2GB to be mapped to user/application address space and 2GB to the kernal address space. Using the /3GB flag shifts it to 3GB for application memory and 1 GB for kernel memory space.

[Tom]

In computer terms GB is GiB, in units of 1024 bytes, not Si's 1000. So 2^32 is 4GB, at least for anything not related to disk storage (they like Si units, so they can say its 4GB but only be 3.8 GiB or whatever).

Just thought I'd point that out.

[Thorin]

I'll start this off with an informational post.  I found that Rolf Kristensen did a good job of summing up how WinNT (and thus WinXP) breaks up virtual user address space and virtual kernel address space.

http://smallvoid.com/article/winnt-kernel-memory.html

Key Points

Virtual address space is the address space given to an application.  4GB on 32-bit Windows XP.

Virtual user address space is the virtual address space available to an application.  Typically 2GB on 32-bit Windows XP.

Virtual kernel address space is the virtual address space available to the kernel.  Typically 2GB on 32-bit Windows XP.

Every application is given 4GB of virtual address space, but 2GB of it is dedicated to the kernel, and according to the linked article the 2GB of virtual kernel address space from each application is shared.  So, for instance, three applications have 4GB of virtual address space, but are only using 8GB of virtual address space all combined (2GB of virtual user address space each, plus 2GB of virtual kernel address space shared between all three applications).

Virtual address space is mapped through address tables to physical memory (also known as RAM) or virtual memory (also known as Page File).

32-bit Windows XP can be made to access more than 4GB of physical memory through the use of PAE (Physical Address Extensions), although doing so will cause more of the virtual kernel address space to be consumed by the address tables that map virtual address space to physical memory and physical memory to virtual memory.

Applications running on 32-bit Windows XP can access physical memory above 4GB made available through PAE by using AWE (Address Windowing Extensions).

BIOSes reserve the top portion of the 4GB address range for other devices such as PCI cards.  If 4GB of physical memory is available on the motherboard, then some BIOSes will re-map the other devices to higher address spaces.  When the BIOS does not re-map the other devices, then the BIOS will only make 4GB of physical memory minus the address size of the other devices available to Windows.

Even if the BIOS can re-map the other devices, if Windows has one or more drivers that it believes cannot handle the remapping, then Windows XP disables the remapping.  This means that even if the BIOS can provide all 4GB of physical memory address space to Windows XP, Windows XP still won't use that portion that it considers reserved for other devices.

Rolf Kristensen has another article, short but sweet, that indicates that the virtual kernel address space may in fact get swapped out from physical memory to virtual memory:

http://smallvoid.com/article/winnt-core-paging.html

Here he describes how to stop that from happening, which clearly implies that virtual kernel address space swapping from physical to virtual memory is a common concern for Windows XP users.

I believe that 32-bit Windows XP *can* access all 4GB of physical RAM, if it believes all of the device drivers it loads are okay with other devices being re-mapped as per the BIOS.

Apologies to those who already posted while I was typing this up.  SMF wouldn't show me the other posts while I was previewing.

[Thorin]

Lazy, your links point out that:

- Applications running under 32-bit WinXP are given 2GB of virtual user address space that they can store stuff in
- 32-bit WinXP can be altered to give applications 3GB of virtual user address space

However, an application will always be given 2GB of virtual user address space (or 3GB with the /3GB switch) regardless of how much physical memory (RAM) is in the machine.  If there is 1GB of physical memory, only part of the virtual user address space will be loaded in that physical memory - the rest will be loaded in the virtual memory (pagefile).

This means that if there are two applications running on a system with 4GB of physical memory, it is theoretically possible that the first application's 2GB of virtual user address space is stored in the first 2GB of physical memory, the second application's 2GB of virtual user address space is stored in the second 2GB of physical memory, and the virtual kernel address space is stored entirely in virtual memory.

But still, we keep talking about virtual address space here, instead of whether Windows XP can actually use the upper limit of the 4GB of physical memory in a machine.  Is Windows XP able to store virtual user address space or virtual kernel address space in the top GB of that physical memory?

As per my previous post, I believe it can under the right circumstances.  I think Windows XP just frequently errs on the side of caution when it comes to the device drivers it thinks might not handle a re-mapping of address space for PCI devices, so most people's Windows XP doesn't actually use that physical memory.  So for most people, the extra stick of RAM is wasted money as it's never getting accessed.

After doing quite a bit of googling, I've noticed that there are a lot of screenshots showing Windows XP not registering all of the physical memory and only a couple of screenshots showing XP properly registering all 4GB.  However, the fact that those screenshots exist suggest to me that WinXP is actually capable of accessing all 4GB of physical memory in near-perfect situations.

How it maps that physical memory to various virtual address spaces is interesting, but not the correct topic for this question, I believe.

I've tried to make sure that I use the correct terms at all times, but I want to especially point out that the virtual user address space and its limit is not the same as the physical memory.  There exists a mapping between them, but it's not one-to-one.

[Cova]

I think we're all pretty much agreeing on most points now - I just wanna toss a couple more things in...

First - I don't believe that the kernel space can be completely stored in virtual memory.  Yes - some of it can be paged, and the second link in Thorin's second post in this thread includes instructions on how to prevent the pagable parts of the kernel from being  swapped to disk.  That article links to a MS KB article http://support.microsoft.com/kb/Q184419 which includes this block of text:

User-mode and kernel-mode drivers and kernel-mode system code is usually written to be either pageable or non-pageable. In cases where drivers or system code is pageable, you can use the following registry entry to keep this pageable code in RAM, but this is only advisable on systems with extremely large amounts of RAM.

I don't have a link saying so - but I'm fairly certain that there will always be some kernel code/data that is non-pagable and therefore must always be kept in physical RAM.  At a bare minimum, the memory-manger part of the kernel should always have to be in RAM, as when an interrupt is tripped requiring something to be paged from disk into RAM, it has to have a memory location for the CPU to jump to to start reading instructions from on how to do that paging.  That doesn't change the conclusion though - that XP can, in the right circumstances, use a full 4GB of RAM - but that usage must allocate at least a little bit permanently to the kernel.


Also - while not directly related, we've touched on PAE a few times.  Anyone considering PAE should really do a LOT of reading up on it, it is NOT trivial to make work, and causes a lot of other issues.  Unless you're running a SQL server or something that REALLY needs like 8+GB of RAM, stay FAR FAR away from PAE.  There is a performance hit with turning on PAE, as the OS now has to maintain a second set of paging tables.  Applications also need to be PAE aware in order to make use of that extra RAM (and VERY few are - far far fewer than support the /3GB switch).  And the biggest problem IMHO - if you do have a PAE aware application using >4GB of RAM, all of that memory HAS to be permanently allocated to that application and then falls outside of the windows memory manager.  eg. if you have a box with 10GB of RAM (32-bit OS with PAE), and install SQL server, and configure it all to use PAE, then you a) cannot use SQL's dynamic memory allocation, you must tell it exactly how much of that 10GB to take.  and b) that memory effectively dissapears once you do that. If you configured SQL to take 8GB of RAM, and start SQL and open windows task manager, you'd be suprized by what you see - your memory usage will look like SQL isn't running, SQL will only report using a very small amount of RAM - the slight overhead it uses on top of its PAE allocation.  That 8GB cannot be used by any other application anymore - windows cannot page SQL out of it to allow another app to use it.

We played with PAE here on our PeopleSoft SQL cluster - the boxes are all running 2003 x64, and multiple SQL 2000 SP4 instances (32-bit), which allows us some interesting options.  Notably, 32-bit PAE-aware app's can run on a x64 OS, and get >=4GB of RAM if they want it, while the OS doesn't have to deal with the PAE overheads so much.  Our boxes happen to have 12GB of physical RAM, and typically have 2 SQL instances running (can increase to 4-5 if the test instance is running and one cluster node fails).  With PAE enabled, cluster-failovers were a serious problem as the SQL instances already running couldn't release any RAM for the instances that are failing over.  And there was still a small performance hit.  We've now disabled PAE on all of the SQL instances, limiting each to 4GB RAM max (so a server with 2 instances has wasted RAM) - the cluster is much more stable, and performance is better.  In the future we'll move to SQL 2005 which has native x64 support.

[Tom]

The only "problem" I have with PAE is it slows the kernel down a little. Linux supports PAE natively. Nyah.

Then again IIRC the amd64 arch's spec for the page tables looks a HECK of a lot like PAE, so well, its kind of something that needed done

[Thorin]

Cova, you're correct that some parts of the kernel must be stored in the "nonpaged pool".  This is a portion of the physical memory that the memory management subsystem of the Windows kernel absolutely will not move from physical memory to virtual memory (aka "page out").  The following page has a bit of a write-up on the paged and nonpaged pools, including a description of how to start up poolmon to see how your pools are being used:

http://book.itzero.com/read/microsoft/0507/Microsoft.Press.Microsoft.Windows.Internals.Fourth.Edition.Dec.2004.internal.Fixed.eBook-DDU_html/0735619174/ch07lev1sec3.html

It doesn't list what processes are non-pageable, though, and I haven't been able to find any specific examples.

I need to correct your statements about PAE, however.  PAE gives the 32-bit Windows kernel the ability to address physical memory greater than 4GB.  If there were 8GB of physical memory in a machine and you're running three applications, for instance, it could load the first application's virtual user address space into the first 2GB of physical memory, the second application's virtual address space into the second 2GB of physical memory, and the third application's virtual address space into the third 2GB of physical memory, while loading the virtual kernel address space into the last 2GB of physical memory.  At this point, none of the applications need to do anything special to access that extra RAM - they're all constrained to their 2GB of virtual user address space each.

What you're talking about is AWE (Address Windows Extensions), which lets applications directly access physical memory higher than 4GB without worrying about getting paged out by other applications.

This is nicely summed up in Wikipedia:

http://en.wikipedia.org/wiki/Physical_Address_Extension

Wikipedia also points out that AWE doesn't necessarily require PAE enabled, and that AWE could be used on systems with less than 4GB.  I assume that means that physical memory below 4GB will simply get blocked off for the application (this is something I didn't know until this post):

http://en.wikipedia.org/wiki/Address_Windowing_Extensions

In the situation that you describe your servers to be in now, with a 64-bit processor running a 64-bit OS running a 32-bit application, you would have turned off AWE to limit the SQL Server instances to 4GB.

Beyond correcting the terms that you're using, though, I have to agree that the message you intend is absolutely correct.  Configuring 32-bit applications to use more than 4GB of RAM is difficult, potentially unstable, and smacks of hack.

[Cova]

Sorry - ya, I tend to use just PAE to describe everything involved in using >4GB on 32-bit systems.  We're getting down into layers of the machine that I don't often get to discuss - not many people understand it, and it's usually better if I simplify for most people somewhat without introducing too many new TLAs.

[Thorin]

I had a discussion at work today about whether it's worth it to install 4GB of RAM in a standard version of Windows Server 2003 (not Server 2003 R2).  I pointed him to this thread as a thorough discussion of how much physical memory (RAM) gets used by Win2k3.

And then I turned to the Internet and started some light reading...

I found a long and detailed article about memory management and how it apparently is limited by a license value rather than an architectural limit of any kind.  Basically, the writer indicates that the kernel for Vista 32-bit and Server 2008 32-bit are the same, and then wonders why Server 2008 uses more than 4GB of physical memory if it's installed.

I warn you, it's a long article, but the author very definitely knows what he's talking about and makes sure to support his arguments.  So here it is:

http://www.geoffchappell.com/viewer.htm?doc=notes/windows/license/memory.htm

If you're a techie, I DARE YOU TO READ IT!

[Tom]

I had a discussion at work today about whether it's worth it to install 4GB of RAM in a standard version of Windows Server 2003 (not Server 2003 R2).  I pointed him to this thread as a thorough discussion of how much physical memory (RAM) gets used by Win2k3.

And then I turned to the Internet and started some light reading...

I found a long and detailed article about memory management and how it apparently is limited by a license value rather than an architectural limit of any kind.  Basically, the writer indicates that the kernel for Vista 32-bit and Server 2008 32-bit are the same, and then wonders why Server 2008 uses more than 4GB of physical memory if it's installed.

I warn you, it's a long article, but the author very definitely knows what he's talking about and makes sure to support his arguments.  So here it is:

http://www.geoffchappell.com/viewer.htm?doc=notes/windows/license/memory.htm

If you're a techie, I DARE YOU TO READ IT!

It is technically possible for a 32bit os to see more than 4G of ram, but using PAE changes the memory addressing mode, and will break drivers that haven't been updated. All sorts of @%&# broke in linux in "bigmem" mode when they added it. Some older drivers probably still won't work with it.

But yeah, Windows has been capable of running with PAE for some time, for a while it was that compatibility issue that kept it back, but I'd have to agree that its mostly just used for product differentiation now.

But who gives a rats ass about 32bit OSs anymore. Upgrade to 64bit and all your memory woes go away (unless your windows license limits memory in 64bit too ).

[Lazybones]

As of 2008 R2 ms is nolonger releasing 32bit server os releases. Almost all r2 server products such as OCS, SCOM and Exchange 2007 are 64bit native only installs.

It is expected that windows 7 will be the last 32bit desktop os from ms as well.

[Thorin]

It is technically possible for a 32bit os to see more than 4G of ram, but using PAE changes the memory addressing mode, and will break drivers that haven't been updated. All sorts of @%&# broke in linux in "bigmem" mode when they added it. Some older drivers probably still won't work with it.

But yeah, Windows has been capable of running with PAE for some time, for a while it was that compatibility issue that kept it back, but I'd have to agree that its mostly just used for product differentiation now.

But who gives a rats ass about 32bit OSs anymore. Upgrade to 64bit and all your memory woes go away (unless your windows license limits memory in 64bit too ).

Therein lies the complaint of the guy who thoroughly investigated it - Microsoft charges money for changing to a new version of Windows, even if just from 32 bit to 64 bit.  And it's not this that the guy has a problem with, it is the fact that Microsoft is persisting the myth that it is somehow a hardware or memory addressing problem, rather than simply Microsoft not willing to give you access.

Keep in mind, for a while you could get your XP to access more than 4GB of RAM.  Then Service Pack 3 came down, which instituted a new limit and bam!  Windows users suddenly have to pay to unlock access to said RAM.  And it's not explained why, it's left for the user to misunderstand and think that he has to buy a new version of Windows because of a hardware limitation...

The other interesting part I got out of that article was that I could run Windows 2000 Professional on a machine with 8GB of RAM and all of the RAM would be accessible...

[Tom]

You should see what Intel is doing with its CPUs. Any function that has been added recently isn't enabled on even half of its cpus. I hear it makes it really really hard to manage and select hardware. You have to pay extra close attention to the specs of each model or you could be missing something important like HALF the cache, VT-x, VT-d, NX, or other things like that.

Where as AMD has just been adding all of the extra features to all of its cpus.

[Lazybones]

You get both the 32bit and 64bit editions when you purchase the retail versions of Vista, and likely Will with windows 7. It is only the cheaper OEM editions that are limited to one or the other per licence.

Yes, through odd paging methods 2000 and XP COULD address more memory however it doesn't do you any good if your drivers can't and your apps can't due to API limitations 32bit apps that want to use more than 2GB need to call a special 32bit API to SEE that RAM this was covered in one of the past links I made. Some server software and some high end workstation software where written for this but not many.

If you want to get more performance your really need to go 64bit with 64bit memory aware apps. Complaining of limitations of 2000 and XP two long old unsupported OS on new cheap high memory systems is rather silly.