在Centos下hadoop native-library錯誤

Hadoop-2.7.3 + Centos 6.7 64位環境下使用弄一套Hadoop環境過程的一些小事情。

（1）開始檢查

[hadoop@xxx6 ~]$ hadoop checknative -a
16/03/21 23:12:46 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/03/21 23:12:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/hadoop-2.7.2/lib/native/libhadoop.so
zlib:    true /lib64/libz.so.1
snappy:  false 
lz4:     true revision:99
bzip2:   false 
openssl: false Cannot load libcrypto.so (libcrypto.so: 沒法打開共享對象文件: 沒有那個文件或目錄)!
16/03/21 23:12:46 INFO util.ExitUtil: Exiting with status 1

 

（2）問題查找

https://issues.apache.org/jira/browse/HADOOP-12845

Google一通下來，才知道這個是hadoop-2.7.2這個版本的BUG，還有待解決，目前只能去迴避問題。

 

On RedHat system (Fedora/Centos/...) the /usr/lib64/libcrypto.so is a link provided by
openssl-devel RPM package which is fine on a build/development host,  but devel packages are
not supposed to be installed on Production servers (Hadoop Cluster) and the openssl RPM package
don't include that link which is a problem.

（3）問題解決

能夠創建以下符號連接便可：

[hadoop@xxx6 ~]$ cd /usr/lib64/
[hadoop@xxx lib64]$ ln -s libcrypto.so.1.0.1e libcrypto.so

 

創建符號連接後再檢查hadoop native，則現象改變了！

[hadoop@xxx6 ~]$ hadoop checknative -a
16/03/21 23:13:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/03/21 23:13:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/hadoop-2.7.2/lib/native/libhadoop.so
zlib:    true /lib64/libz.so.1
snappy:  false 
lz4:     true revision:99
bzip2:   false 
openssl: true /usr/lib64/libcrypto.so 16/03/21 23:13:51 INFO util.ExitUtil: Exiting with status 1