手動安裝 Cloudera Manager Tarballs

1、安裝前準備

一、安裝配置數據庫

查看 Cloudera Manager and Managed Service Data Stores. 安裝配置數據庫，文檔點擊這裏MySQL Database,Oracle Database, or External PostgreSQL Database 

二、(僅CDH 5) 在RHEL 5 和 CentOS 5, 安裝Python 2.6 or 2.7

2、安裝Cloudera Manager和Agents

下載tarball包，地址： Cloudera Manager Version and Download Information.  拷貝tarball包到全部機器並解壓：

$ sudo mkdir /opt/cloudera-manager

$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

文件被解壓到一個名字與Cloudera Manager 版本相符的子目錄。好比： /opt/cloudera-manager/cm-5.0/. 這個全路徑後面會用到，tarball_root 目錄.

一、Perform Configuration Required by Single User Mode單用戶模式所需的配置

單用戶模式的配置步驟請點擊這裏  Single User Mode Requirements.

二、建立用戶

在全部主機上手動建立cloudera-scm帳戶:

$ sudo useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

確保 --home 路徑與我的環境匹配. 這個參數應與tarball的解壓目錄相符。好比你的--home 多是/opt/cm-5.0/run/cloudera-scm-server.

三、建立Clodera Manager Server 本地數據存儲目錄

1. 建立此目錄: /var/lib/cloudera-scm-server.
2. 修改此目錄的權限，以便cloudera-scm 用戶及其組有操做權限.

   $ sudo mkdir /var/log/cloudera-scm-server $ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server

四、配置Cloudera Manager Agents

• 修改每一個Cloudera Manager Agent 主機上的 tarball_root/etc/cloudera-scm-agent/config.ini文件，配置 Cloudera Manager Agent 所指向的 Cloudera Manager Server :

  Property	Description
  server_host	Name of the host where Cloudera Manager Server is running.
  server_port	Port on the host where Cloudera Manager Server is running.

• By default, a tarball installation has a var subdirectory where state is stored.  默認狀況，tarball包裏有一個var子目錄用來存儲狀態。
• In a non-tarball installation, state is stored in /var. 非源碼安裝時狀態存儲在/var目錄。
• Cloudera recommends that you reconfigure the tarball installation to use an external directory as the /var equivalent (/var or any other directory outside the tarball) 。Cloudera 推薦源碼安裝從新配置爲一個外部目錄，像/var同樣（/var 或者其餘tarball安裝目錄之外的目錄）
• so that when you upgrade Cloudera Manager, the new tarball installation can access this state。以便升級Cloudera Manager 時，新tarball包能夠訪問這個狀態. 
• Configure the installation to use an external directory for storing state by editing tarball_root/etc/default/cloudera-scm-agent and setting the CMF_VAR variable to the location of the /var equivalent. 經過修改tarball_root/etc/default/cloudera-scm-agent 文件的CMF_VAR 變量的值來配置了一個外部的目錄
• If you do not reuse the state directory between different tarball installations, duplicate Cloudera Manager Agent entries can occur in the Cloudera Manager database.若是你在不一樣的源碼安裝中不會再使用這個狀態目錄，

五、自定義Cloudera Manager User 及目錄

若是不打算修改默認配置，請跳過此步。

默認Cloudera Manager 會在/var/log and /var/lib下建立以下目錄：

• /var/log/cloudera-scm-headlamp
• /var/log/cloudera-scm-firehose
• /var/log/cloudera-scm-alertpublisher
• /var/log/cloudera-scm-eventserver
• /var/lib/cloudera-scm-headlamp
• /var/lib/cloudera-scm-firehose
• /var/lib/cloudera-scm-alertpublisher
• /var/lib/cloudera-scm-eventserver
• /var/lib/cloudera-scm-server

If you are using a custom username and custom directories for Cloudera Manager, you must create these directories on the Cloudera Manager Server host and assign ownership of these directories to the custom username. Cloudera Manager installer makes no changes to any directories that already exist. Cloudera Manager cannot write to any existing directories for which it does not have proper permissions, and if you do not change ownership, Cloudera Management Service roles may not perform as expected. To resolve these issues, do one of the following:

• Change ownership of existing directories:
  1. Use the chown command to change ownership of all existing directories to the Cloudera Manager user. If the Cloudera Manager username and group arecloudera-scm, to change the ownership of the headlamp log directory, you issue a command similar to the following:

     $ sudo chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp

• Use alternate directories:
  1. If the directories you plan to use do not exist, create them. For example, to create /var/cm_logs/cloudera-scm-headlamp for use by the cloudera-scm user, you can use the following commands:

     mkdir /var/cm_logs/cloudera-scm-headlamp
     chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp

  2. Connect to the Cloudera Manager Admin Console.
  3. Select Clusters > Cloudera Management Service
  4. Select Scope > role name.
  5. Click the Configuration tab.
  6. Enter a term in the Search field to find the settings to be changed. For example, you might enter /var or directory.
  7. Update each value with the new locations for Cloudera Manager to use.
        Note: The configuration property for the  Cloudera Manager Server Local Data Storage Directory (default value is: /var/lib/cloudera-scm-server) is located on a different page:

     1. Select Administration > Settings.
     2. Type directory in the Search box.
     3. Enter the directory path in the Cloudera Manager Server Local Data Storage Directory property.
  8. Click Save Changes to commit the changes.

3、爲Cloudera Manager Server 配置數據庫

      在下面選擇一種方式

• 外部數據庫-  Preparing a Cloudera Manager Server External Database.
• 內部數據庫-  Installing and Starting the Cloudera Manager Server Embedded Database.

4、建立 Parcel 目錄

1. 在 Cloudera Manager Server 主機, 建立一個 parcel 庫目錄:

   $ sudo mkdir -p /opt/cloudera/parcel-repo

2. 修改目錄的權限:

   $ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

   自定義用戶名請修改成相應的用戶名及組名
   

3. 在集羣的每臺機器上建立一個parcels 目錄:

   $ sudo mkdir -p /opt/cloudera/parcels

4. 修改權限:

   $ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

5、啓動Cloudera Manager Server

   注意: 這裏假定你沒有啓動HDFS及MapReduce。若是有這些服務在運行請看下面:

1. 關閉 HDFS 及MapReduce. See Stopping Services (CDH 4) or Stopping Services (CDH 5) for the commands to stop these services.
2. 配置初始化腳本爲開機不啓動. Use commands similar to those shown in Configuring init to Start Core Hadoop System Services (CDH 4) orConfiguring init to Start Hadoop System Services (CDH 5), but disable the start on boot (for example, $ sudo chkconfig hadoop-hdfs-namenode off).

Contact Cloudera Support for help converting your existing Hadoop configurations for use with Cloudera Manager.

The way in which you start the Cloudera Manager Server varies according to what account you want the Server to run under:

在須要啓動Cloudera Manager Server 的主機上運行以下命令：

• root用戶:

  $ sudotarball_root/etc/init.d/cloudera-scm-server start

• 其餘用戶. If you run as another user, ensure the user you created for Cloudera Manager owns the location to which you extracted the tarball including the newly created database files. If you followed the earlier examples and created the directory /opt/cloudera-manager and the user cloudera-scm, you could use the following command to change ownership of the directory:

  $ sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

  Once you have established ownership of directory locations, you can start Cloudera Manager Server using the user account you chose. For example, you might run the Cloudera Manager Server as cloudera-service. In this case, you have the following options:

  • Run the following command:

    $ sudo -u cloudera-servicetarball_root/etc/init.d/cloudera-scm-server start

  • Edit the configuration files so the script internally changes the user. Then run the script as root:
    1. Remove the following line from tarball_root/etc/default/cloudera-scm-server:

       export CMF_SUDO_CMD=" "

    2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-server to the user you want the server to run as. For example, to run ascloudera-service, change the user and group as follows:

       USER=cloudera-service
       GROUP=cloudera-service

    3. Run the server script as root:

       $ sudotarball_root/etc/init.d/cloudera-scm-server start

• Cloudera Manager Server 開機自啓動:
  1. 在 Cloudera Manager Server 主機運行以下命令:
     • RHEL-compatible and SLES

       $ cptarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
       $ chkconfig cloudera-scm-server on

     • Debian/Ubuntu

       $ cptarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
       $ update-rc.d cloudera-scm-server defaults

  2. 在 Cloudera Manager Server 主機, 打開the /etc/init.d/cloudera-scm-server 文件並把 CMF_DEFAULTS 的值，從${CMF_DEFAULTS:-/etc/default} 改成 tarball_root/etc/default.

若是 Cloudera Manager Server 運行失敗, 請看  Troubleshooting Installation and Upgrade Problems.

6、啓動Cloudera Manager Agents

Start the Cloudera Manager Agent according to the account you want the Agent to run under:

• 在每一個Agent 主機上運行以下命令啓動Cloudera Manager Agent：

  $ sudotarball_root/etc/init.d/cloudera-scm-agent start

  Agent啓動後會與Cloudera Manager Server通訊。
• If you are running single user mode, start Cloudera Manager Agent using the user account you chose. For example, to run the Cloudera Manager Agent as cloudera-scm, you have the following options:
  • Run the following command:

    $ sudo -u cloudera-scmtarball_root/etc/init.d/cloudera-scm-agent start

  • Edit the configuration files so the script internally changes the user, and then run the script as root:
    1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:

       export CMF_SUDO_CMD=" "

    2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run ascloudera-scm, change the user and group as follows:

       USER=cloudera-scm
       GROUP=cloudera-scm

    3. Run the Agent script as root:

       $ sudotarball_root/etc/init.d/cloudera-scm-agent start

• Cloudera Manager Agents開機自啓動:
  1. Run the following commands on each Agent host:
     • RHEL-compatible and SLES

       $ cptarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
       $ chkconfig cloudera-scm-agent on

     • Debian/Ubuntu

       $ cptarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
       $ update-rc.d cloudera-scm-agent defaults

  2. On each Agent, open the tarball_root/etc/init.d/cloudera-scm-agent file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} totarball_root/etc/default.

7、安裝依賴

When you install with tarballs and parcels, some services may require additional dependencies that are not provided by Cloudera. On each host, install the required packages:

• Red-hat compatible
  • chkconfig
  • python (2.6 required for CDH 5)
  • bind-utils
  • psmisc
  • libxslt
  • zlib
  • sqlite
  • cyrus-sasl-plain
  • cyrus-sasl-gssapi
  • fuse
  • portmap
  • fuse-libs
  • redhat-lsb
• SLES
  • chkconfig
  • python (2.6 required for CDH 5)
  • bind-utils
  • psmisc
  • libxslt
  • zlib
  • sqlite
  • cyrus-sasl-plain
  • cyrus-sasl-gssapi
  • fuse
  • portmap
  • python-xml
  • libfuse2
• Debian/Ubuntu
  • lsb-base
  • psmisc
  • bash
  • libsasl2-modules
  • libsasl2-modules-gssapi-mit
  • zlib1g
  • libxslt1.1
  • libsqlite3-0
  • libfuse2
  • fuse-utils or fuse
  • rpcbind

8、啓動及登錄Cloudera Manager 管理控制檯

The Cloudera Manager Server URL takes the following form http://Server host:port, where Server host is the fully qualified domain name or IP address of the host where the Cloudera Manager Server is installed, and port is the port configured for the Cloudera Manager Server. The default port is 7180.

1. Wait several minutes for the Cloudera Manager Server to complete its startup. To observe the startup process, run tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log on the Cloudera Manager Server host. If the Cloudera Manager Server does not start, see Troubleshooting Installation and Upgrade Problems.
2. In a web browser, enter http://Server host:7180, where Server host is the fully qualified domain name or IP address of the host where the Cloudera Manager Server is running. The login screen for Cloudera Manager Admin Console displays.
3. Log into Cloudera Manager Admin Console. The default credentials are: Username: admin Password: admin. Cloudera Manager does not support changing the adminusername for the installed account. You can change the password using Cloudera Manager after you run the installation wizard. Although you cannot change theadmin username, you can add a new user, assign administrative privileges to the new user, and then delete the default admin account.

9、選擇Cloudera Manager 版本及主機

1. When you start the Cloudera Manager Admin Console, the install wizard starts up. Click Continue to get started.
2. Choose which edition to install:
   • Cloudera Express, which does not require a license, but provides a limited set of features.
   • Cloudera Enterprise Data Hub Edition Trial, which does not require a license, but expires after 60 days and cannot be renewed.
   • Cloudera Enterprise with one of the following license types:
     • Basic Edition
     • Flex Edition
     • Data Hub Edition
   If you choose Cloudera Express or Cloudera Enterprise Data Hub Edition Trial, you can upgrade the license at a later time. See Managing Licenses.
3. If you elect Cloudera Enterprise, install a license:
   1. Click Upload License.
   2. Click the document icon to the left of the Select a License File text field.
   3. Navigate to the location of your license file, click the file, and click Open.
   4. Click Upload.
   Click Continue to proceed with the installation.
4. Information is displayed indicating what the CDH installation includes. At this point, you can access online Help or the Support Portal. Click Continue to proceed with the installation.
5. Click the Currently Managed Hosts tab.
6. Choose the hosts to add to the cluster.
7. Click Continue.

10、選擇安裝軟件的方法並安裝軟件

1. Install CDH and managed services using parcels:
   1. Use Parcels
      1. Choose the parcels to install. The choices depend on the repositories you have chosen; a repository can contain multiple parcels. Only the parcels for the latest supported service versions are configured by default.
         You can add additional parcels for previous versions by specifying custom repositories. For example, you can find the locations of the previous CDH 4 parcels at http://archive.cloudera.com/cdh4/parcels/. Or, if you are installing CDH 4.3 and want to use  policy-file authorization, you can add the Sentry parcel using this mechanism.

         1. To specify the parcel directory, specify the local parcel repository, add a parcel repository, or specify the properties of a proxy server through which parcels are downloaded, click the More Options button and do one or more of the following:
            • Parcel Directory and Local Parcel Repository Path - Specify the location of parcels on cluster hosts and the Cloudera Manager Server host. If you change the default value for Parcel Directory and have already installed and started Cloudera Manager Agents, restart the Agents:

              $ sudo service cloudera-scm-agent restart

            • Parcel Repository - In the Remote Parcel Repository URLs field, click the  button and enter the URL of the repository. The URL you specify is added to the list of repositories listed in the Configuring Cloudera Manager Server Parcel Settings page and a parcel is added to the list of parcels on the Select Repository page. If you have multiple repositories configured, you see all the unique parcels contained in all your repositories.
            • Proxy Server - Specify the properties of a proxy server.
         2. Click OK.
      2. Select the release of Cloudera Manager Agent. You can choose either the version that matches the Cloudera Manager Server you are currently using or specify a version in a custom repository. If you opted to use custom repositories for installation files, you can provide a GPG key URL that applies for all repositories. Click Continue.
   2. Click Continue. Cloudera Manager installs the CDH and managed service parcels. During parcel installation, progress is indicated for the phases of the parcel installation process in separate progress bars. If you are installing multiple parcels, you see progress bars for each parcel. When the Continue button at the bottom of the screen turns blue, the installation process is completed. Click Continue.
2. Click Continue. The Host Inspector runs to validate the installation and provides a summary of what it finds, including all the versions of the installed components. If the validation is successful, click Finish.

11、添加服務

使用 Cloudera Manager 嚮導配置並啓動CDH及其餘所管理的服務.

1. In the first page of the Add Services wizard, choose the combination of services to install and whether to install Cloudera Navigator:
   • Click the radio button next to the combination of services to install:

     CDH 4	CDH 5
     Core Hadoop - HDFS, MapReduce, ZooKeeper, Oozie, Hive, and Hue Core with HBase Core with Impala All Services - HDFS, MapReduce, ZooKeeper, HBase, Impala, Oozie, Hive, Hue, and Sqoop Custom Services - Any combination of services.	Core Hadoop - HDFS, YARN (includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, and Sqoop Core with HBase Core with Impala Core with Search Core with Spark All Services - HDFS, YARN (includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, Sqoop, HBase, Impala, Solr, Spark, and Key-Value Store Indexer Custom Services - Any combination of services.

     As you select services, keep the following in mind:
     • Some services depend on other services; for example, HBase requires HDFS and ZooKeeper. Cloudera Manager tracks dependencies and installs the correct combination of services.
     • In a Cloudera Manager deployment of a CDH 4 cluster, the MapReduce service is the default MapReduce computation framework. Choose Custom Services to install YARN, or use the Add Service functionality to add YARN after installation completes.
          Note: You can create a YARN service in a CDH 4 cluster, but it is not considered production ready.
     • In a Cloudera Manager deployment of a CDH 5 cluster, the YARN service is the default MapReduce computation framework. Choose Custom Servicesto install MapReduce, or use the Add Service functionality to add MapReduce after installation completes.
          Note: In CDH 5, the MapReduce service has been deprecated. However, the MapReduce service is fully supported for backward compatibility through the CDH 5 lifecycle.
     • The Flume service can be added only after your cluster has been set up.
   • If you have chosen Data Hub Edition Trial or Cloudera Enterprise, optionally select the Include Cloudera Navigator checkbox to enable Cloudera Navigator. See the Cloudera Navigator Documentation.
   Click Continue.
2. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. You can reassign role instances if necessary.

   Click a field below a role to display a dialog containing a list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts, or Custom to display the pageable hosts dialog.

   The following shortcuts for specifying hostname patterns are supported:

   • Range of hostnames (without the domain portion)

     Range Definition	Matching Hosts
     10.1.1.[1-4]	10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
     host[1-3].company.com	host1.company.com, host2.company.com, host3.company.com
     host[07-10].company.com	host07.company.com, host08.company.com, host09.company.com, host10.company.com

   • IP addresses
   • Rack name

   Click the View By Host button for an overview of the role assignment by hostname ranges.

3. When you are satisfied with the assignments, click Continue.
4. On the Database Setup page, configure settings for required databases:
   1. Enter the database host, database type, database name, username, and password for the database that you created when you set up the database.
   2. Click Test Connection to confirm that Cloudera Manager can communicate with the database using the information you have supplied. If the test succeeds in all cases, click Continue; otherwise, check and correct the information you have provided for the database and then try the test again. (For some servers, if you are using the embedded database, you will see a message saying the database will be created at a later step in the installation process.) The Review Changes screen displays.
5. Review the configuration changes to be applied. Confirm the settings entered for file system paths. The file paths required vary based on the services to be installed.If you chose to add the Sqoop service, indicate whether to use the default Derby database or the embedded PostgreSQL database. If the latter, type the database name, host, and user credentials that you specified when you created the database.
      Warning: Do not place DataNode data directories on NAS devices. When resizing an NAS, block replicas can be deleted, which will result in reports of missing blocks.
   Click Continue. The wizard starts the services.
6. When all of the services are started, click Continue. You see a success message indicating that your cluster has been successfully started.
7. Click Finish to proceed to the Home Page.

(可選) Change the Cloudera Manager User

After configuring your services, the installation wizard automatically starts the Cloudera Management Service, assuming that it runs using cloudera-scm. If you configured this service to run using a user other than cloudera-scm, the Cloudera Management Service roles do not start automatically. To change the service configuration to use the user account that you selected:

1. Connect to the Cloudera Manager Admin Console.
2. Do one of the following:
   • Select Clusters > Cloudera Management Service > Cloudera Management Service.
   • On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link.
3. Click the Configuration tab.
4. Use the search box to find the property to change. For example, you might enter "system" to find the System User and System Group properties.
5. Make any changes required to the System User and System Group to ensure Cloudera Manager uses the proper user accounts.
6. Click Save Changes.
7. Start the Cloudera Management Service roles.

Change the Default Administrator Password

As soon as possible, change the default administrator password:

1. Right-click the logged-in username at the far right of the top navigation bar and select Change Password.
2. Enter the current password and a new password twice, and then click Update.

Configure Oozie Data Purge Settings

If you added an Oozie service, you can change your Oozie configuration to control when data is purged in order to improve performance, cut down on database disk usage, or to keep the history for a longer period of time. Limiting the size of the Oozie database can also improve performance during upgrades. See Configuring Oozie Data Purge Settings Using Cloudera Manager.

Test the Installation

You can test the installation following the instructions in Testing the Installation.