[LWN Logo]

Date:   Wed, 3 Nov 1999 17:25:34 +0100 (CET)
From:   Ingo Molnar <mingo@chiara.csoma.elte.hu>
To:     Neil Conway <njcmail@fusrs5a.culham.ukaea.org.uk>
Subject: The 64GB memory thing

On Wed, 3 Nov 1999, Neil Conway wrote:

> The recent thread about >4GB surprised me, as I didn't even think >2GB
> was very stable yet.  Am I wrong?  Are people out there using 4GB
> boxes with decent stability?  I presume it's a 2.3 feature, yes?

the 64GB stuff got included recently. It's a significant rewrite of the
lowlevel x86 MM and generic MM layer, here is a short description about

my 'HIGHMEM patch' went into the 2.3 kernel starting at pre4-2.3.23. This
means a heavily rewritten VM subsystem (to deal with pte's bigger than
machine word size), and a much rewritten x86 memory and boot architecture.
In fact there is no bigmem anymore, it has been replaced by (the i think
more correct term) 'high memory'. It utilizes 3-level page tables on PPro+
CPUs called 'Physical Address Extension' (PAE) mode. In PAE mode the CPU
uses a completely different and incompatible page table structure, which
is 3-level and has 64-bit page table entries and cover up to 64GB physical
RAM. Virtual space is still unchanged, 4GB. Highmem is completely
transparent to user-space.

There is a new 'High Memory Support' option under 'Processor type and
                      High Memory Support 
                            ( ) off                             
                            ( ) 4GB                             
                            (X) 64GB                            
'off' is up to 1GB RAM, utilizing 2-level page tables and no highmem
support. '4GB' is utilizing 2-level page tables and high memory support
for any <4GB physical RAM that cannot be permanently mapped by the kernel.
'64GB' mode utilizes 3-level page tables (for everything). The theoretical
limit of high memory on IA32 boxes is 16 TB - there is lots of space in
64-bit PAE pte's, although current CPUs only support up to 64GB RAM. (the
biggest current chipsets supports up to 32GB RAM)

about the structure of the patch/feature itself, kernel internals:

pgtable.h got split up into pgtable-2level.h and pgtable-3level.h, which
should be the only 'global #ifdef' distincting 3-level from 2-level page
tables on x86. There were lots of assumptions throughout the arch/i386
tree that assumed 2-level page tables, these places all had to be fixed
and converted to 'generic 3-level page table code'. There are only a few
CONFIG_X86_PAE #ifdefs left, i intend to cut down the number of these even
more, to keep the x86 lowlevel MM/boot code easy to maintain.

the generic kernel was almost safe wrt. 3-level page tables, but
nevertheless it had bugs which only triggered in PAE mode. For example,
one pgd entry in PAE mode covers 1GB of virtual memory, and some loops
which iterated through virtual memory had buggy exit conditions and broke
in subtle ways when they were running in the upper-most 1GB of virtual
memory. (ie. kernel space) There were about 20 of such buggy loops
throughout the MM code.

the much bigger generic change was that pte's got 64-bit, although the
architecture itself was still 32-bit. Lots of VM-internal code had to be
reworked to never assume that 'sizeof(pte_t) == sizeof(unsigned long)'.
Examples are the swapping code, IPC shared memory. Bigger than
machine-word ptes were not supported by Linux previously.

also i guess many of you have noticed the new mm/bootmem.c allocator -
this was necessery because on my 8GB box mem_map is more than 100 MB (!),
and the 'naive' boot-time allocation we did in earlier kernels simply did
not work on 'slightly noncontinous' physical maps like my box has. (at
64MB there is an ACPI area which caused problems)

[this short description should give you a scope of the changes, and i/we
are still fixing some of the impact in 2.3.25. (Christoph just posted his
problems with IPC shared memory)]

Backporting to 2.2: while the bigmem patch was small and simple and got
backported to 2.2, the highmem patch is basically impossible to be
backported in a maintainable way as it touches some 60 files all over the

64 GB PAE mode works just fine on my 8GB RAM, 8-way Xeon box:

 11:25pm  up 5 min,  2 users,  load average: 7.78, 4.30, 1.77
30 processes: 21 sleeping, 9 running, 0 zombie, 0 stopped
CPU states:  0.0% user,  7.2% system, 92.8% nice, 0.0% idle
Mem: 8241152K av,7720960K used, 520192K free,      0K shrd,   2168K buff
Swap:      0K av,      0K used,      0K free                  9756K cached

  215 root      19  19 1000M 1.0G   232 R N  1.0G 11.6 12.4   0:11 db_serv
  180 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:31 db_serv
  182 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:30 db_serv
  183 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:30 db_serv
  184 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:30 db_serv
  185 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:30 db_serv
  186 root      19  19 1000M 1.0G   232 R N  1.0G 11.5 12.4   0:29 db_serv
  247 root      19  19  500M 500M   232 R N     0 11.5  6.2   0:04 db_serv2
  181 root       1   0   996  996   828 R       0  7.2  0.0   0:16 top
  177 root       0   0   984  984   768 S       0  0.1  0.0   0:00 bash
    1 root       0   0   476  476   408 S       0  0.0  0.0   0:00 init

(these are 8x ~1G-RSS processes using up all 8GB physical RAM, one running
on each CPU)

future plans:

right now high memory is seriously underused on typical servers due to the
page cache still being in low memory. On 2.2 with bigmem the
lowmem:highmem ratio is around 5:1, this means that the 'effective' size
of my 8GB box in 2.2 is only ~2.4GB. The exception are workloads where
most memory is allocated as shared memory or user memory, but this is not
the case for a typical web or fileserver. On my box the pagecache is
already in high memory (and we are ready to add 64-bit DMA to device
drivers), and the lowmem:highmem ratio is up to 1:10. This means that 8GB
RAM is already fully utilized on a typical server workload.


ps. to have correct memory statistics (top, vmstat, free) with >4GB RAM
you need the newest procps package (or i can send the patches).

To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/