PURPOSE
-------
This article will assist you to clarify which is the
max SGA size value according the differents kernels and methodology to increase it.
SCOPE & APPLICATION
-------------------
This is intended for Linux system administrators and for IA-32 architecture.
On Itanium processor there are not particular limits because
bypassed in a 64 bit executable.
INTRODUCTION
------------
~ Very Large Memory (VLM) on systems with up to 64GB of RAM
is the ability to use up to 64GB pagecache on 32-bit system.
The Advanced Server 2.1 and 3.0 kernel allows Oracle to allocate and use more than
4GB of memory for the database buffer cache on a 32-bit Intel platform. This
feature is also called VLM (Very Large Memory) in Oracle documents. With
Oracle9iR2 on Red Hat Linux Advanced Server, the SGA can be increased
to a theoretical limit of about 62GB depending on the available RAM memory
on the system. Current hardware limitations and practical considerations further
limit the actual size of SGA that can be configured, but it is still several times
larger than the size of the SGA without VLM. Running in VLM mode requires
some setup changes to Linux and imposes some limitations on what features and
init.ora parameters can be used.
~ PAE (Page Address Extensions)
In order to get above 4GB virtual memory on IA-32 architecture a technique
known as PAE (Page Address Extensions) is used.
It is a method that translates 32-bit (2**32 = 4GB) linear addresses
to 36-bit (2**36 = 64GB) physical addresses. In the linux kernel, the support
is provided through a compile time option that produces two
separate kernels the SMP kernel which supports only upto 4GB VM
and the enterprise kernel which can go up to 64GB VM
(also called Very Large Memory 'VLM' capable).
The enterprise kernel is able to use up to 64GB pagecache
without any modifications. This means applications like oracle
can make use of the large memory and scale up
to a large number of users without loss of performance or reliability.
(shmfs/tmpfs,ramfs is implemented using PAE)
See also
http://en.wikipedia.org/wiki/Physical_Address_Extension
- Red Hat Linux Advanced Server 2.1
~ Large SGA size capability through changing of mapped_base (Ref. [1], [2])
The default address where oracle is loaded from is 0x50000000 and the
default mapped base is 0x40000000 (decimal 1073741824). The space between
0x40000000 to 0x5000000 is reserved for loading oracle libraries. If the
mapped_base is lowered to 0x10000000 (decimal 268435456), then the space
between 0x10000000 to 0x15000000 is used for loading the oracle libraries.
Oracle executable will start getting loaded from 0x15000000 thus allowing a
bigger SGA and a larger window size. The lowered values of mapped base
specified above are examples, and the user should set these to values
appropriate for their systems. In order to use this method
It will be necessary relocating the SGA attach address.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| oracle libraries
| 0x50000000 |
|
|
+------------+
|
|
|
|
|
|Original Base
| 0x40000000 |decimal 1073741824
|
+------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
oracle libraries
| 0x15000000 |
|
+------------+
|
|
|
|Lowered Base
| 0x10000000 |
decimal 268435456
+------------+
|
|
+------------+
Note:
Lowering mapped_base is a method available only
on RedHat Advanced Server (RHAS) 2.1
this is automatically done in Red Hat Enterprise Linux 3.0
~ Shared memory file-system(shmfs) support (Ref. [1], [2])
Memory-based file system optimized for shared memory operations
and for larger SGA size.
- Red Hat Enterprise Linux 3.0 - 4.0
~ hugemem kernel (Ref. [3], [4], [5])
Red Hat Enterprise Linux 3.0/4.0 includes a new kernel known as the
hugemem kernel. This kernel supports a 4GB per process user space
(versus 3GB for the other kernels), and a 4GB direct kernel space.
Using this kernel allows Red Hat Enterprise Linux to run on systems
with up to 64GB of main memory. The hugemem kernel is required in
order to use all the memory in system configurations containing more
than 16GB of memory. The hugemem kernel can also benefit
configurations running with less memory (if running an application
that could benefit from the larger per process user space, for example.)
The hugemem kernel feature is also called 4GB-4GB Split Kernel
A classic 32-bit 4GB virtual address space is split 3GB
for user processes and 1GB for the kernel
The new scheme (4gb/4gb) permits 4GB of virtual address space
for the kernel and almost 4GB for each user process
3GB:1GB Split
0Gb 1Gb 2Gb 3Gb 4Gb
|-------------|-------------|-------------|-------------|
|-->
Per-User Process
Kernel
Per-User Process
Kernel
>>SGA MAX Size 1.7 GB 4 – 1 – 1 – 0.3 = 1.7GB left for SGA
A picture of Memory Layout:
---------------------------
4GB +------------------+ 0xFFFFFFFF
|
|
|Kernel stuff|
|
| 0xE0000000 SSKGMTOP
|
|(from sskgm.h may17 label)
|
|
3GB +------------------+ 0xC0000000 __PAGE_OFFSET
||Stack grows| (include/asm-i386/page.h)
|vdown...|
2.98GB|------------------| 0xBF000000
|
|
|
|
|Oracle SGA|
|max 1776MB|
|max 1.75GB|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1.25GB+------------------| 0x50000000GENKSMS_SGA_ADDR
| Shared libraries |
| lib*.so|
|
|
|
|
|
|
|
|
1GB +------------------+ 0x40000000TASK_UNMAPPED_BASE
| applicaton code|
|(Oracle .text)|
^
|
|
|
|
| 0x20000000SSKGMBOTTOM|
|
| (from sskgm.h may17 label) |
|
|
|
|
|
|
128MB |------------------| 0x08000000
|
|
|
|
|
|
|
0 +------------------+ 0x00000000
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ RedHat Advanced Server (RHAS) 2.1
~
~ Configuration : 3+1 & lower mapped Base Address
~ ->> 3Gb user process + 1gb Kernel + lower mapped Base Address
~
~ >>>SGA MAX Size 2.7 GB /proc/sys/kernel/shmmax
- Relocating the SGA (low SGA attach address):
% cd $ORACLE_HOME/rdbms/lib
% cp ksms.s ksms.s_orig
% genksms -s 0x15000000> ksms.s
% make -f ins_rdbms.mk ksms.o
% make -f ins_rdbms.mk ioracle
- lower the mapped base for a single bash terminal session
1.Open a terminal session (Oracle session).
2.Open a second terminal session and su to root (root session).
3.Find out the process id for the Oracle session.
For example: do "echo $$" in the Oracle session.
4.Now lower the mapped base for the Oracle session to 0x10000000.
From the root session, echo 268435456 >/proc//mapped_base,
whereis the process id determined in step 4.
5.From the Oracle terminal session, startup the Oracle instance.
The SGA now begins at a lower address, so more of the address
space can be used by Oracle.
Now you can increase the init.ora values of db_cache_size or db_block_buffers
to increase the size of the database buffer cache.
In this case the max SGA size will be 2.7 GB
Linux memory layout after lowering lower mapped Base Address:
-------------------------------------------------------------
4GB+------------------+ 0xFFFFFFFF
|
|
|Kernel stuff|
|
|
|
|
|
|
3GB+------------------+ 0xC0000000__PAGE_OFFSET
||Stack grows|(include/asm-i386/page.h)
|vdown...|
2.98GB |------------------| 0xBF000000SSKGMTOP
|
|
|
|
|Oracle SGA|
|max 2720MB|
|max 2.65GB|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
336MB+------------------+ 0x15000000SSKGMBOTTOM
| Shared libraries |
(GENKSMS_SGA_ADDR)
| lib*.so|
256MB|------------------| 0x10000000TASK_UNMAPPED_BASE
| applicaton code|
I(Oracle .text)I
128MB+------------------+ 0x08000000
|
|
|
|
0|------------------| 0x00000000
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ RedHat Enterprise Linux (RHEL) 3.0 - 4.0
~
~ Default Configuration : 4gb/4gb split (hugemem kernel)
~
~ >>>SGA MAX Size 2.7 GB>>SGA MAX Size 3.42 GB 4GB – 0.33GB – 0.25GB = 3.42
How to lower the SGA attach address in short, for details see [1]Note 211424.1:
- Modify shmmax:
% echo 3000000000 > /proc/sys/kernel/shmmax
- Relocating the SGA:
% cd $ORACLE_HOME/rdbms/lib
% cp ksms.s ksms.s_orig
% genksms -s 0x15000000> ksms.s
% make -f ins_rdbms.mk ksms.o
% make -f ins_rdbms.mk ioracle
- Note:
Lowering 'mapped_base' for a single bash terminal session is a method
available only on RedHat Advanced Server (RHAS) 2.1
This is not need on Red Hat Enterprise Linux 3.0/4.0 because automatically done.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ RedHat Advanced Server(RHAS) 2.1(shmfs/tmpfs)
~ RedHat Enterprise Linux (RHEL) 3.0(shmfs/tmpfs, ramfs)
~
~ Configuration : VLM mode + in-memory filesystem (shmfs/tmpfs, ramfs)
~
~ >>>SGA MAX Size 62GB/proc/sys/kernel/shmmax
- Set the init.ora parameter use_indirect_data_buffers=true
- Startup oracle.
. How to use the memory file system ramfs in short,
for details see [4]Note 262004.1:
- Mount the shmfs file system as root using command:
% umount /dev/shm
% mount -t ramfs ramfs /dev/shm
% chown oracle:dba /dev/shm
- Increase the "max locked memory" ulimit (ulimit -l)
Add the following to /etc/security/limits.conf:
oracle softmemlock 3145728
oracle hardmemlock 3145728
(in case of ssh see details on [4]Note 262004.1)
- Set the init.ora parameter use_indirect_data_buffers=true
- Startup oracle.
CONCLUSION
----------
A process on a 32 bit platform can not address more than 4Gb directly.
This means we can only get 1Gb more if we use RHEL 3.0 with 4gb/4gb split kernel.
If we need larger shared pool (large Buffer Cache) We have to use VLM,
this means shmfs and indirect buffers.
With a 4Gb| lower mapped Base Address | 2.7 GB|[1,2]|
|--------|------------------|-----|---------------------------|-------------|-----|
| AS 2.1 | all| 4Gb|low SGA attach address(***)| 3.42 GB | |
|--------|------------------|-----|---------------------------|-------------|-----|
| AS 3/4 | hugemem—SMP| 64Gb| in-memory filesystem|62 GB (*)|[4,9]|
|---------------------------------------------------------------------------------|
(*) theoretic depending on phisical RAM and on block size
(**) depending on block size
(***) +4gb/4gb split, with ahttp://otn.oracle.com/tech/linux ... x-Tech-WP-Final.PDF
- Linux Virtual Memory in Red Hat Linux Advanced Server 2.f/linuxVM-WP-022 REFERENCE
. |
