在高併發的大數據 場景下Linux服務器報錯fork: retry:資源暫時不可用的解決辦法

 

在高併發的大數據 場景下Linux服務器報錯fork: retry:資源暫時不可用

jvm會產生一個hs_err_pid74299.log相似這樣的日誌文件

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (gcTaskThread.cpp:46), pid=74299, tid=140016377341760
#
# JRE version: (7.0_71-b14) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode linux-amd64 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

默認狀況下Linux服務起的core core file size設置爲0，須要調整該參數，可是這個參數並不能 解決問題；

問題的根本緣由在於服務器的運行應用程序的打開文件的最大數及最大進程數設置的相對較小默認爲4096

須要修改以下配置：

/etc/security/limits.conf

* soft nofile 327680
* hard nofile 327680
hdfs soft nproc 131072
hdfs hard nproc 131072
mapred soft nproc 131072
mapred hard nproc 131072
hbase soft nproc 131072
hbase hard nproc 131072
zookeeper soft nproc 131072
zookeeper hard nproc 131072
hive soft nproc 131072
hive hard nproc 131072
root soft nproc 131072
root hard nproc 131072

 

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

ll /etc/security/limits.d/
總用量 4
-rw-r--r-- 1 root root 228 7月 20 16:08 20-nproc.conf_bak_20190720

 

問題獲得解決。