oerr ORA-2730*node
[oracle@p19c01 ~]$ oerr ORA 27300 27300, 00000, "OS system dependent operation:%s failed with status: %s" // *Cause: OS system call error // *Action: contact Oracle Support [oracle@p19c01 ~]$ oerr ORA 27301 27301, 00000, "OS failure message: %s" // *Cause: OS system call error // *Action: contact Oracle Support // [oracle@p19c01 ~]$ oerr ORA 27302 27302, 00000, "failure occurred at: %s" // *Cause: OS system call error // *Action: contact Oracle Support //
*Cause: OS system call errorshell
The errors are usually caused by OS system call error or OS configuration issue 。數據庫
能夠發現,若是數據庫trace日誌中出現ORA-2730*的錯誤,很大多是因爲OS system的錯誤致使。bash
Tue Sep 01 04:05:33 2020 skgxpvfynet: mtype: 61 process 417356 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4) Errors in file /u01/app/oracle/diag/rdbms/*****/******/trace/******1_w001_417356.trc (incident=96021): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:sendmsg failed with status: 105 ORA-27301: OS failure message: No buffer space available ORA-27302: failure occurred at: sskgxpsnd2 Incident details in: /u01/app/oracle/diag/rdbms/*****/******/incident/incdir_96021/******1_w001_417356_i96021.trc opidrv aborting process W001 ospid (417356) as a result of ORA-603
以上爲某一套數據庫trace日誌中報錯的信息,能夠發現幾個關鍵點:session
一、status: 105 oracle
二、 failure occurred at: sskgxpsnd2app
三、No buffer space available。ide
Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors (Doc ID 579365.1) 測試
經過MOS文章能夠查詢到與上述報錯相符的信息:ui
See: Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (Doc ID 2041723.1)
關於解決方案中提到的vm.min_free_kbytes參數,單位是KB,Oracle建議的取值範圍是0.4%到5%之間,若是內存<=32G,能夠忽略不設置。
查看系統vm.min_free_kbytes默認設置大小:
[root@p19c01 ~]# sysctl -a | grep vm.min_free_kbytes vm.min_free_kbytes = 11433
查看系統開啓NUMA的node數量:
numactl --hardware | grep available: | awk '{print $2}'
Notes:該參數表示Linux VM最低保留多少的空閒內存空間,當可用的內存低於配置參數時,系統會進行cache內存的回收,來進行內存的釋放。
★ 參數簡述 該Linux系統參數的功能是用來設置一個最小內存空間給系統內核使用 該值設置過大會浪費空間,保留太小會形成系統壓力 ★ 知識點 ※ 用途:合理設置該值有助於Linux系統更有效地回收內存 ※ 注意:對於內存較小(≤32GB)的環境不建議設置,保持默認便可 ※ 危險:不要對正在運行的環境進行設置 ※ 有修改操做,非原版 ※ vm.min_free_kbytes參數的單位是:KB ※ 內存水位 min水位:下的內存是保留給內核使用的;當到達min,會觸發內存的direct reclaim low水位:比min高一些,當內存可用量小於low的時候,會觸發 kswapd回收內存 high水位:繼續睡眠 ※ 內存回收方式 direct reclaim : 觸發min水位線時執行 kswapd reclaim : 觸發low水位線時執行
這裏摘自其餘博主的一個計算腳本能夠用來計算vm.min_free_kbytes,直接複製到shell控制檯執行便可:
MIN_FREE_KBYTES_SYSCTL=$(egrep ^vm.min_free_kbytes /etc/sysctl.conf | awk '{print $3}') MIN_FREE_KBYTES_MEMORY=$(cat /proc/sys/vm/min_free_kbytes) NUMA_NODE_COUNT=$(numactl --hardware | grep available: | awk '{print $2}') TOTAL_MEMORY_KBYTES=$(free -k | awk '/Mem:/ {print $2}') NUMA_BASED=$(( $NUMA_NODE_COUNT * 1048576 )) MEMORY_BASED=$(( $TOTAL_MEMORY_KBYTES / 200 )) if [[ $NUMA_BASED -ge $MEMORY_BASED ]] then RECOMMEND_VALUE=$NUMA_BASED else RECOMMEND_VALUE=$MEMORY_BASED fi OFFSET=$(echo $RECOMMEND_VALUE*.05 | bc | cut -d"." -f1) LOWER_BOUND=$(echo $RECOMMEND_VALUE-$OFFSET | bc) UPPER_BOUND=$(echo $RECOMMEND_VALUE+$OFFSET | bc) if [[ $MIN_FREE_KBYTES_SYSCTL -ge LOWER_BOUND && $MIN_FREE_KBYTES_SYSCTL -le UPPER_BOUND ]] then SYSCTL_IN_RANGE=YES else SYSCTL_IN_RANGE=NO fi #sysctl in range? if [[ $MIN_FREE_KBYTES_MEMORY -ge LOWER_BOUND && $MIN_FREE_KBYTES_MEMORY -le UPPER_BOUND ]] then MEMORY_IN_RANGE=YES else MEMORY_IN_RANGE=NO fi DETAIL=$( echo -e "Total Memory: $TOTAL_MEMORY_KBYTES"; echo -e "NUMA node count: $NUMA_NODE_COUNT"; echo -e "NUMA calculated: $NUMA_BASED"; echo -e "memory calculated: $MEMORY_BASED"; echo -e "recommended value: $RECOMMEND_VALUE"; echo -e "permitted range: $LOWER_BOUND to $UPPER_BOUND"; echo -e "in sysctl.conf: $MIN_FREE_KBYTES_SYSCTL"; echo -e "sysctl in range?: $SYSCTL_IN_RANGE"; echo -e "in active memory: $MIN_FREE_KBYTES_MEMORY"; echo -e "memory in range?: $MEMORY_IN_RANGE"; ) if [[ $SYSCTL_IN_RANGE = YES && $MEMORY_IN_RANGE = YES ]] then echo -e "SUCCESS: vm.min_free_kbytes is configured as recommended. Details:\n\n$DETAIL" elif [[ $MIN_FREE_KBYTES_SYSCTL -lt $LOWER_BOUND || $MIN_FREE_KBYTES_MEMORY -lt $LOWER_BOUND ]] then echo -e ":: Result : 【FAILURE】: vm.min_free_kbytes is not configured as recommended" ZZT_V1=$(( $NUMA_NODE_COUNT * 1 * 1024 * 1024 )) ZZT_V2=$(( $TOTAL_MEMORY_KBYTES * 5 /10 / 100 )) if [ $ZZT_V1 -gt $ZZT_V2 ] then ZZT_MAX=$ZZT_V1 else ZZT_MAX=$ZZT_V2 fi ZZT_MAX_GB=$(($ZZT_MAX/1024/1024)) echo ">>> if output is <FAILURE>,please vi sysctl.conf and edit param's value for:vm.min_free_kbytes and reboot" echo ">>> [formula_oracle]vm.min_free_kbytes value (Kb) =MAX(1GB * number_numa_nodes, 0.5% * total_memory) " echo ">>> [calculated_zzt]vm.min_free_kbytes = $ZZT_MAX (About: $ZZT_MAX_GB GB)" ZZT_V3=32 ZZT_V4=$(($TOTAL_MEMORY_KBYTES/1024/1024)) if [ $ZZT_V4 -gt $ZZT_V3 ] then echo ">>> [PASS]The current memory is suitable for setting system parameters." else echo ">>> [WARN]Your memory is too small to set this parameter." fi echo -e "Details:\n\n$DETAIL" elif [[ $MIN_FREE_KBYTES_SYSCTL -gt $UPPER_BOUND && $MIN_FREE_KBYTES_MEMORY -gt $UPPER_BOUND ]] then echo -e "WARNING: vm.min_free_kbytes is not configured as recommended. Details:\n\n$DETAIL" else echo -e "ERROR: Inconsistent results. Details:\n\n$DETAILS" fi
測試結果以下: