Hadoop生態圈-HUE環境部署

時間 2020-06-02

原文原文鏈接

　　　　　　　　　　　　　　　　　　Hadoop生態圈-HUE環境部署html

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　做者：尹正傑java

一.HUE簡介python

1>.HUE的由來mysql

　　HUE全稱是HadoopUser Experience，看這名字就知道怎麼回事了吧。沒錯，直白來講就是Hadoop用戶體驗，是一個開源的Apache Hadoop UI系統，由Cloudera Desktop演化而來，最後Cloudera公司將其貢獻給Apache基金會的Hadoop社區，它是基於Python Web框架Django實現的。經過使用HUE咱們能夠在瀏覽器端的Web控制檯上與Hadoop集羣進行交互來分析處理數據。linux

2>.官網地址c++

　　HUE官網地址可參考：http://gethue.com/overview/。git

二.HUE安裝github

1>.部署環境介紹（我使用的是CentOS 7.2的操做系統）web

2>.安裝開發JDK版本

　　如今你生成環境用的Java開發版本，關於Java的下載地址可參考官網：http://www.oracle.com/technetwork/java/javase/downloads，下載後經過上傳工具（如winscp等工具，或者你也可使用linux自帶的rz命令進行上傳）

[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package lrzsz.x86_64 0:0.12.20-36.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================ Package Arch Version Repository Size ============================================================================================================ Installing: lrzsz x86_64 0.12.20-36.el7                  base                   78 k Transaction Summary ============================================================================================================ Install 1 Package Total download size: 78 k Installed size: 181 k Downloading packages: lrzsz-0.12.20-36.el7.x86_64.rpm                                                      |  78 kB  00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : lrzsz-0.12.20-36.el7.x86_64                                                              1/1 Verifying : lrzsz-0.12.20-36.el7.x86_64                                                              1/1 Installed: lrzsz.x86_64 0:0.12.20-36.el7 Complete! [yinzhengjie@s100 ~]$

安裝linux自帶的上傳工具命令（[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz）

[yinzhengjie@s100 ~]$ ll total 0 [yinzhengjie@s100 ~]$ rz #會彈出對話框，選擇你要上傳的文件便可 [yinzhengjie@s100 ~]$ ll total 181296
-rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo mkdir /soft [yinzhengjie@s100 ~]$ sudo chown yinzhengjie:yinzhengjie /soft/ [yinzhengjie@s100 ~]$ tar zxf jdk-8u181-linux-x64.tar.gz -C /soft/ [yinzhengjie@s100 ~]$ ln -s /soft/jdk1.8.0_181/ /soft/jdk [yinzhengjie@s100 ~]$ ll /soft/ total 4 lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo vi /etc/profile [yinzhengjie@s100 ~]$ more /etc/profile | tail -3 #添加java的環境變量 #ADD JAVA_PATH JAVA_HOME=/soft/jdk PATH=$PATH:$JAVA_HOME/bin [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ source /etc/profile #從新加載系統配置文件，使得配置文件在當前shell下生效 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ java -version #查看java版本 openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) [yinzhengjie@s100 ~]$

3>.準備環境依賴

　　查看官網安裝文檔（http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.0/manual.html），咱們就能夠看到相關的依賴環境以下：（CentOS具體依賴包參考：http://cloudera.github.io/hue/latest/admin-manual/manual.html#installation）

　　Centos安裝操做系統環境以下:

[yinzhengjie@s100 ~]$sudo yum -y install apache-maven ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com ..........　　　　　　接下來的信息我就省略了，安裝過程以及成功後會有如下輸出：（見下圖）

　　安裝過程以下：

　　安裝成功以下：

4>.下載HUE安裝包

[yinzhengjie@s100 ~]$ sudo yum -y install wget [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package wget.x86_64 0:1.14-15.el7_4.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================ Package Arch Version Repository Size ============================================================================================================ Installing: wget x86_64 1.14-15.el7_4.1                  base                  547 k Transaction Summary ============================================================================================================ Install 1 Package Total download size: 547 k Installed size: 2.0 M Downloading packages: wget-1.14-15.el7_4.1.x86_64.rpm                                                      | 547 kB  00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wget-1.14-15.el7_4.1.x86_64                                                              1/1 Verifying : wget-1.14-15.el7_4.1.x86_64                                                              1/1 Installed: wget.x86_64 0:1.14-15.el7_4.1 Complete! [yinzhengjie@s100 ~]$

安裝下載命令wget（[yinzhengjie@s100 ~]$ sudo yum -y install wget）

5>.刪除安裝後的openJDK

　　舒適提示：使用yum安裝這些包的同時，也會自動安裝openJDK的依賴，因此，請自行刪除安裝後的openJDK

[yinzhengjie@s100 ~]$ rpm -qa | grep java javapackages-tools-3.4.1-11.el7.noarch java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 python-javapackages-3.4.1-11.el7.noarch tzdata-java-2018e-3.el7.noarch java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps python-javapackages-3.4.1-11.el7.noarch [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps javapackages-tools-3.4.1-11.el7.noarch [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ rpm -qa | grep java tzdata-java-2018e-3.el7.noarch [yinzhengjie@s100 ~]$

6>.解壓HUE安裝包

[yinzhengjie@s100 ~]$ ll total 216276
-rw-r--r--. 1 yinzhengjie yinzhengjie  35818977 Sep 19  2016 hue-3.7.0-cdh5.3.6.tar.gz -rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ tar -zxf hue-3.7.0-cdh5.3.6.tar.gz -C /soft/ [yinzhengjie@s100 ~]$ ll /soft/ total 8 drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28  2015 hue-3.7.0-cdh5.3.6 lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$

7>.編譯HUE

[yinzhengjie@s100 ~]$ ll /soft/ total 8 drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28  2015 hue-3.7.0-cdh5.3.6 lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$ cd /soft/hue-3.7.0-cdh5.3.6/ [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ ll total 64 drwxr-xr-x. 21 yinzhengjie yinzhengjie  4096 Jul 28  2015 apps drwxr-xr-x.  3 yinzhengjie yinzhengjie  4096 Jul 28  2015 cloudera drwxr-xr-x.  5 yinzhengjie yinzhengjie    54 Jul 28  2015 desktop drwxr-xr-x.  6 yinzhengjie yinzhengjie  4096 Jul 28  2015 docs drwxr-xr-x.  3 yinzhengjie yinzhengjie    23 Jul 28  2015 ext -rw-r--r--.  1 yinzhengjie yinzhengjie 11358 Jul 28  2015 LICENSE.txt -rw-r--r--.  1 yinzhengjie yinzhengjie  4715 Jul 28  2015 Makefile -rw-r--r--.  1 yinzhengjie yinzhengjie  8505 Jul 28  2015 Makefile.sdk -rw-r--r--.  1 yinzhengjie yinzhengjie  3498 Jul 28  2015 Makefile.vars -rw-r--r--.  1 yinzhengjie yinzhengjie  2192 Jul 28  2015 Makefile.vars.priv drwxr-xr-x.  2 yinzhengjie yinzhengjie    20 Jul 28  2015 maven -rw-r--r--.  1 yinzhengjie yinzhengjie   801 Jul 28  2015 NOTICE.txt -rw-r--r--.  1 yinzhengjie yinzhengjie  1517 Jul 28  2015 README drwxr-xr-x.  4 yinzhengjie yinzhengjie    65 Jul 28  2015 tools -rw-r--r--.  1 yinzhengjie yinzhengjie   932 Jul 28  2015 VERSION [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ make apps　　　　　　　　　　　　#開始編譯HUE --- Creating virtual environment at /soft/hue-3.7.0-cdh5.3.6/build/env python2.7 /soft/hue-3.7.0-cdh5.3.6/tools/virtual-bootstrap/virtual-bootstrap.py \ -qq --no-site-packages /soft/hue-3.7.0-cdh5.3.6/build/env --- /soft/hue-3.7.0-cdh5.3.6/build/env ready make[1]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop' make -C core env-install make[2]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop/core' --- Building egg for markdown running bdist_egg running egg_info ......... #編譯時會出現下圖的報錯信息

　　解決方案：

解決方案： [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ sudo vi /usr/include/openssl/x509.h [sudo] password for yinzhengjie: #刪除如下兩行： 751 X509_REVOKED *X509_REVOKED_dup(X509_REVOKED *rev); 　 752 X509_REQ *X509_REQ_dup(X509_REQ *req); [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$

　　使用root權限刪除圖中的751和752行代碼，而後在從新編譯便可。編譯成功截圖以下：

8>.配置HUE

[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ vi desktop/conf/hue.ini [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep secret_key= desktop/conf/hue.ini secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_host= desktop/conf/hue.ini http_host=172.30.1.100 [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_port= desktop/conf/hue.ini http_port=8888 [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep time_zone= desktop/conf/hue.ini time_zone=Asia/Shanghai [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$

　　具體配置以下：

9>.啓動HUE

10>.關閉服務器防火牆以及selinux服務。

[yinzhengjie@s100 ~]$ systemctl stop firewalld #臨時關閉防火牆 [yinzhengjie@s100 ~]$ systemctl disable firewalld #禁用防火牆的服務開機自啓 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ su root Password: [root@s100 yinzhengjie]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# getenforce Enforcing [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# cat /etc/selinux/config  | grep SELINUX= | grep -v ^# SELINUX=disabled [root@s100 yinzhengjie]#

11>.訪問HUE的webUI並設置管理員帳號

　　　Duang~若是你出現如下界面說明你HUE啓動成功啦！

三.經過git方式安裝HUE

　　經查閱資料，發現github上也有教程教你如何安裝HUE，須要你本地服務器安裝git命令，詳情請參考：https://github.com/cloudera/hue#development-prerequisites

[yinzhengjie@s100 ~]$ sudo yum -y install git [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package git.x86_64 0:1.8.3.1-14.el7_5 will be installed --> Processing Dependency: perl-Git = 1.8.3.1-14.el7_5 for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: rsync for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Git) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Error) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: libgnome-keyring.so.0()(64bit) for package: git-1.8.3.1-14.el7_5.x86_64 --> Running transaction check ---> Package libgnome-keyring.x86_64 0:3.12.0-1.el7 will be installed ---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed ---> Package perl-Git.noarch 0:1.8.3.1-14.el7_5 will be installed ---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================== Installing: git x86_64 1.8.3.1-14.el7_5                           updates                         4.4 M Installing for dependencies: libgnome-keyring                         x86_64                         3.12.0-1.el7                               base                            109 k perl-Error                               noarch                         1:0.17020-2.el7                            base                             32 k perl-Git                                 noarch                         1.8.3.1-14.el7_5                           updates                          54 k perl-TermReadKey                         x86_64                         2.30-20.el7                                base                             31 k rsync x86_64 3.1.2-4.el7                                base                            403 k Transaction Summary ========================================================================================================================================================== Install 1 Package (+5 Dependent packages) Total download size: 5.0 M Installed size: 23 M Downloading packages: (1/6): perl-TermReadKey-2.30-20.el7.x86_64.rpm                                                                                     |  31 kB  00:00:00 (2/6): libgnome-keyring-3.12.0-1.el7.x86_64.rpm                                                                                    | 109 kB  00:00:00 (3/6): perl-Error-0.17020-2.el7.noarch.rpm                                                                                         |  32 kB  00:00:00 (4/6): perl-Git-1.8.3.1-14.el7_5.noarch.rpm                                                                                        |  54 kB  00:00:00 (5/6): rsync-3.1.2-4.el7.x86_64.rpm                                                                                                | 403 kB  00:00:00 (6/6): git-1.8.3.1-14.el7_5.x86_64.rpm                                                                                             | 4.4 MB  00:00:02     
---------------------------------------------------------------------------------------------------------------------------------------------------------- Total 1.7 MB/s | 5.0 MB  00:00:02 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. ** Found 12 pre-existing rpmdb problem(s), 'yum check' output follows: ant-1.9.2-9.el7.noarch has missing requires of java-devel >= ('0', '1.5.0', None) ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils >= ('0', '1.7.5', None) xalan-j2-2.7.1-23.el7.noarch has missing requires of jpackage-utils xerces-j2-2.11.0-17.el7_0.noarch has missing requires of java xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils xml-commons-apis-1.4.01-16.el7.noarch has missing requires of java xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils Installing : 1:perl-Error-0.17020-2.el7.noarch                                                                                                      1/6 Installing : rsync-3.1.2-4.el7.x86_64                                                                                                               2/6 Installing : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                    3/6 Installing : libgnome-keyring-3.12.0-1.el7.x86_64                                                                                                   4/6 Installing : perl-Git-1.8.3.1-14.el7_5.noarch                                                                                                       5/6 Installing : git-1.8.3.1-14.el7_5.x86_64                                                                                                            6/6 Verifying : git-1.8.3.1-14.el7_5.x86_64                                                                                                            1/6 Verifying : libgnome-keyring-3.12.0-1.el7.x86_64                                                                                                   2/6 Verifying : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                    3/6 Verifying : 1:perl-Error-0.17020-2.el7.noarch                                                                                                      4/6 Verifying : rsync-3.1.2-4.el7.x86_64                                                                                                               5/6 Verifying : perl-Git-1.8.3.1-14.el7_5.noarch                                                                                                       6/6 Installed: git.x86_64 0:1.8.3.1-14.el7_5 Dependency Installed: libgnome-keyring.x86_64 0:3.12.0-1.el7  perl-Error.noarch 1:0.17020-2.el7  perl-Git.noarch 0:1.8.3.1-14.el7_5  perl-TermReadKey.x86_64 0:2.30-20.el7 rsync.x86_64 0:3.1.2-4.el7 Complete! [yinzhengjie@s100 ~]$