Enabling Linux HugePages

 

Enabling Linux HugePages can improve the performance of the Oracle JRockit JVM and have several advantages over the default page size of 4 KB in Linux.

In this recipe, the HugePages will be enabled on all machines in the PRD_DOMAIN domain. A new JVM parameter will be added to the Managed Servers so the JVM makes use of the HugePages.

PRD01 hosts the PRD_SVR1 and PRD_SVR2 Managed Servers. Since each instanceis configured with a heap size of 2 GB, reserving 4 GB for HugePages in server1 should be enough.

Before configuring the Linux HugePages, shut down all WebLogic Servers in the domain.Also stop the Administration Console instance and the Node Manager.

To enable the Linux HugePages, you must log in as root user:

1. Log in as root user to the shell and execute the following commands to check the HugePages configuration:

[root@server1]$ cat /proc/meminfo | grep Huge

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

Hugepagesize: 2048 kB

2. Create a mount point of type hugetlbfs:

[root@server1]$ mkdir /mnt/hugepages

[root@server1]$ chmod -R 777 /mnt/hugepages

[root@server1]$ mount -t hugetlbfs nodev /mnt/hugepages

3. Add the following mount point to the /etc/fstab:

[root@server1]$ vi /etc/fstab

4. Add the following line:

hugetlbfs /mnt/hugepages hugetlbfs rw,mode=0777 0 0

5. Enter :ws! to save the file and exit.

6. Edit the sysctl.conf file under /etc/:

[root@server1]$ vi /etc/sysctl.conf

7. Reserve 2048 HugePages by adding the following line:

vm.nr_hugepages = 2048

8. Enter :ws! to save the file and exit.

9. Execute the following command to make the changes effective:

[root@server1]# sysctl -p

10. Verify the change:

[root@server1]$ cat /proc/meminfo | grep Huge

HugePages_Total: 2048

HugePages_Free: 2048

HugePages_Rsvd: 0

Hugepagesize: 2048 kB

11. Restart the host if the HugePages could not be allocated.

12. Repeat the preceding steps on all machines in the PROD_DOMAIN.

To enable the Managed Servers to make use of the HugePages, follow the ensuing steps:

1. Access the Administration Console by pointing your web browser to

http://adminhost:7001/console.

2. Click on the Lock & Edit button to start a new edit session.

3. Expand the Environment tree to the left and then click on Servers.

4. Click on the PROD_Server01 link and then click on the Server Start tab.

5. Add the following to the Arguments field and click on the Save button:

-XX:+UseLargePagesForHeap

6. Click on the Activate Changes button.

7. Restart PRD_SVR1.

8. Repeat the preceding steps for PRD_SVR2, PRD_SVR3, and PRD_SVR4.

The first thing to do is to estimate how much memory should be reserved as HugePages.

Sum all heap sizes of the JVMs in the machine. In our case, we have 2 JVMs with 2 GB each so a total of 4 GB should be reserved in HugePages.

To calculate how many pages, get the memory needed and divide by the page size. The formula is sum of JVM heaps / page size.

The memory needed is 4 GB (4096 MB) and the page size is 2 MB (Hugepagesize is 2048 KB). The number of HugePages is 4096/2 = 2048. This 4 GB of reserved HugePages will be pre-allocated and cannot be used by other applications, even when the JVMs are not running, so be sure to do the sizing properly.

It is a common mistake to reserve a large amount of memory in HugePages and the host starts to swap. Remember to leave enough memory for the other applications and the operating system. Avoid the swapping at all costs. If you have the -Xverbose:gc,memory enabled, you can check if the JVM is using the HugePages properly in the STDOUT logfile.

[INFO ][memory ] Using 2MB pages for Java heap.

With the PROD_Server01 started, checking the /proc/meminfo should also reveal that HugePages are in use.

[user@server1]$ cat /proc/meminfo | grep Huge

HugePages_Total: 2048

HugePages_Free: 1024

HugePages_Rsvd: 0

Hugepagesize: 2048 kB

Using HugePages provides some significant advantages:

Because of the larger page size, the page table will be smaller in size. With HugePages, a 10 GB heap size should use only 5120 page entries despite the 2621440 page entries present when using the default 4 KB page size. This minimizes the CPU cost and the page table memory usage.

A performance boost in memory operations because the TLB cache works more efficiently, with more cache hits.

The memory reserved for HugePages will never swap to disk.

It forces a more controlled memory usage of the heap sizes.