关于numa、transparent_hugepage、hugepage

EL7 OS层面禁用NUMA和关闭transparent_hugepage：
1、修改 /etc/default/grub
vi /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never"

2、grub2-mkconfig使修改生效
###(BIOS based systems)
# cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.$(date +%F)
# grub2-mkconfig -o /boot/grub2/grub.cfg

###(UEFI based systems)
# cp /boot/efi/EFI/redhat/grub.cfg /boot/efi/EFI/redhat/grub.cfg.$(date +%F)
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

3、重启Linux操作系统
# /sbin/reboot
[root@ebsdbprd1 ~]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.14.35-1902.300.11.el7uek.x86_64 root=/dev/mapper/ol-root ro crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never

在数据库层开启Hugepage：
根据官方建议启动Hugepage。参考文档如下：HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
1）修改文件 /etc/security/limits.conf
内存128G，这个值以KB单位，比物理内存稍小点。比SGA值要大没有关系。
* soft memlock 64000000
* hard memlock 64000000
2 ）修改 /etc/sysctl.conf
SGA设置80G，页大小为2M，页数为 80*1024/2=40960 页数时以进行计算，要稍大一点。
...
vm.nr_hugepages = 40962 #sga=80G
...

3） 检查
# grep -i page /proc/meminfo
AnonHugePages: 0 kB <<<0表示已禁用transparent hugepages

HugePages_Total: 40962 <<<Hugepage大小
HugePages_Free: 40962
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 0 kB

常用文件：
# cat /boot/grub2/grub.cfg
# cat /boot/efi/EFI/redhat/grub.cfg
# cat /etc/default/grub
# cat /boot/grub/grub.conf
# cat /etc/grub.conf
# cat /proc/cmdline
ls -l /etc/grub.conf
ls -l /boot/grub/grub.conf

/etc/grub.conf是/boot/grub/grub.conf的链接文件
# ls -l /etc/grub.conf
lrwxrwxrwx. 1 root root 22 Aug 30  2017 /etc/grub.conf -> ../boot/grub/grub.conf
# ls -l /boot/grub/grub.conf
-rw------- 1 root root 1584 Jul 22  2020 /boot/grub/grub.conf

cp -pr /boot/grub/grub.conf /boot/grub/grub.conf.$(date +%F)
sed -i 's#rhgb quiet#rhgb quiet numa=off transparent_hugepage=never#g' /boot/grub/grub.conf
cat /boot/grub/grub.conf

numa常用命令：
# numactl --show
# numactl --hardware
# grep -i numa /var/log/dmesg
# grep acpi=off /proc/cmdline
# grep numa=off /proc/cmdline
# egrep 'acpi=off|numa=off' /proc/cmdline
# numactl -H | grep available
If the number of available nodes is "2 nodes (0-1)" or "8 nodes (0-7)" then NUMA is enabled.  If the number of available nodes is "1 nodes (0)" then NUMA is not enabled.

大页常用命令：
# cat /proc/meminfo|grep Huge
# cat /proc/meminfo|grep Page
# cat /proc/meminfo|egrep 'Huge|Page'
# grep AnonHugePages /proc/meminfo
# egrep 'trans|thp' /proc/vmstat
# grep -e AnonHugePages  /proc/*/smaps|awk '{ if($2>4) print $0}'|awk -F "/" '{print $0; system("ps -fp " $3)}'
# grep -i HugePages_Total /proc/meminfo

根据MOS文档“Oracle Linux - Transparent Huge Pages (THP) and Memory Compaction Causing Processes to Unresponsive on UEK2 (文档 ID 1606759.1)”，需要关闭操作系统的Transparent Huge Page特性。
关闭这一特性的优先方法是修改/boot/grub/grub.conf文件，在kernel行的后面加上"transparent_hugepage=never"：
title Red Hat Enterprise Linux (2.6.32-358.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_xty64-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_xty64/lv_root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_DM  KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_xty64/lv_swap  transparent_hugepage=never
        initrd /initramfs-2.6.32-358.el6.x86_64.img

其他可选择的方法有：
To disable THP at run time
Run the following commands to disable THP on-the-fly:
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag

或在/etc/rc.local中加入下面的代码行：
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
在重启后，确认如下结果：
cat /sys/kernel/mm/transparent_hugepage/defrag
always [never]
cat /sys/kernel/mm/transparent_hugepage/enabled 
always [never]

如果结果非预期，可能是由于Linux的ktune和tuned服务所导致。在此种情况下，建议关闭这两项服务：
service tuned stop
chkconfig tuned off
service ktune stop
chkconfig ktune off
或者：
tuned-adm off

https://access.redhat.com/solutions/46111

https://access.redhat.com/solutions/48756

https://access.redhat.com/solutions/1320153

NOTE:361468.1 –HugePages on Oracle Linux 64-bit
NOTE:2279458.1 –Transparent Hugepage is not getting disabled
NOTE:2567141.1 –Oracle Linux 6 – How to disable NUMA on EFI Boot Loaders using GRUB2?
NOTE:2336923.1 –Oracle Linux: How to Disable Transparent HugePages on Oracle Linux 7
NOTE:1557478.1 –Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and UEK2 and above
NOTE:2066217.1 –Oracle Linux 7 – How to disable Transparent HugePages for RHCK kernel?
NOTE:1606759.1 –Oracle Linux – Transparent Huge Pages (THP) and Memory Compaction Causing Processes to Get Unresponsive on UEK2

Please ensure to take a backup of the existing /boot/grub2/grub.cfg before rebuilding.
On BIOS-based machines: ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
On UEFI-based machines: ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

On BIOS-based machines:
sed -i 's#rhgb quiet#rhgb quiet numa=off transparent_hugepage=never#g' /etc/default/grub
sed -i 's#numa=off transparent_hugepage=never numa=off transparent_hugepage=never#numa=off transparent_hugepage=never#g' /etc/default/grub
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

On UEFI-based machines:
sed -i 's#rhgb quiet#rhgb quiet numa=off transparent_hugepage=never#g' /etc/default/grub
sed -i 's#numa=off transparent_hugepage=never numa=off transparent_hugepage=never#numa=off transparent_hugepage=never#g' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg

cat /etc/grub.conf
cat /boot/grub/grub.conf
cat /proc/cmdline
cat /proc/meminfo|egrep -i 'huge|page'

transparent_hugepage更改前查看是开启状态
cat /sys/kernel/mm/transparent_hugepage/enabledcat
cat /sys/kernel/mm/transparent_hugepage/defrag

方式一：实时生效，不需要重启机器（重启后失效）
执行命令
echo "never" >> /sys/kernel/mm/transparent_hugepage/enabled
echo "never" >> /sys/kernel/mm/transparent_hugepage/defrag

执行成功后再看状态
把这两个命令加入开机执行，防止重启后失效

vi /etc/rc.d/rc.local
方式二：需要重启后生效（该方法未验证）
编辑 /etc/default/grub

在GRUB_CMDLINE_LINUX加入选项 transparent_hugepage=never

重新生成grub配置文件
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

重启系统后验证结果
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/enabled