【原創】大數據基礎之Ambari（2）經過Ambari部署ElasticSearch（ELK）

ambari2.7.3（hdp3.1） 安裝 elasticsearch6.3.2

ambari的hdp中原生不支持elasticsearch安裝，下面介紹如何經過mpack方式使ambari支持elasticsearch安裝：

一 安裝Service

1 下載

Mpack include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

# wget https://community.hortonworks.com/storage/attachments/87416-elasticsearch-mpack-2600-9.tar.gz

2 安裝mpack

# ambari-server install-mpack --mpack=/path/to/87416-elasticsearch-mpack-2600-9.tar.gz --verbose

3 重啓ambari-server

# ambari-server restart

訪問ambari頁面發現services中並無elasticsearch，檢查mpack.json發現問題：

/var/lib/ambari-server/resources/mpacks/elasticsearch-ambari.mpack-6.3.2/mpack.json

        {

          "service_name" : "ELASTICSEARCH",

          "service_version" : "6.3.2",

          "applicable_stacks" : [

            {

              "stack_name" : "HDP",

              "stack_version" : "2.3"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.4"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.5"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.6"

            }

          ]

        }

applicable_stacks代表service只應用到hdp2.3-2.6版本，沒有3.1，手工加上；

mpack詳細結構詳見：https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs

要注意直接修改mpack.json而後重啓ambari-server無論用，須要從新install mpack：

1 卸載

ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack

2 修改mpack.json並從新打包

elasticsearch_mpack-2.6.0.0-9.tar.gz

3 安裝mpack（同上）

4 重啓ambari-server（同上）

這時就能夠在頁面上看到ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

 

二 部署ElasticSearch

下面開始部署ElasticSearch，發現報錯：

Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>

    BeforeAnyHook().execute()

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute

    method(env)

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook

    setup_users()

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users

    groups = params.user_to_groups_dict[user],

KeyError: u'elasticsearch'

Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-303.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-303.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']

跟進報錯的代碼：

/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py

     35   should_create_users_and_groups = False
     36   if params.host_sys_prepped:
     37     should_create_users_and_groups = not params.sysprep_skip_create_users_and_groups
     38   else:
     39     should_create_users_and_groups = not params.ignore_groupsusers_create
     40 
     41   if should_create_users_and_groups:
     42     for group in params.group_list:
     43       Group(group,
     44       )
     45 
     46     for user in params.user_list:
     47       User(user,
     48            uid = get_uid(user) if params.override_uid == "true" else None,
     49            gid = params.user_to_gid_dict[user],
     50            groups = params.user_to_groups_dict[user],
     51            fetch_nonlocal_groups = params.fetch_nonlocal_groups,
     52            )

這時須要將配置ignore_groupsusers_create打開，查看和修改配置有兩種方式：

1 命令行

# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

詳細參數詳見

# python configs.py --help

2 rest api

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"

從返回的items中找到tag，而後再請求（增長tag參數）：

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"

返回結果中包含全部的property

"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",

修改須要先將上面的配置保存到一個文件，好比/tmp/configurations.tmp，而後修改，而後再調用接口:

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name

而後再部署便可；

三 啓動ElasticSearch

啓動elasticsearch，報錯：

1）

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'java_home' was not found in configurations dictionary!

須要返回ambari的elasticsearch config頁面，將java_home修改成實際的java home路徑：

# JAVA_HOME must be provided here for OS that use systemd service launch
JAVA_HOME={{java64_home}}

2）

File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2", line 85, in top-level template code
gateway.expected_data_nodes: {{ expected_data_nodes }}
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!

查看報錯的文件

/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2
/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2

發現報錯的這行，是由於hostname沒有賦值致使，直接去掉這行

node.name: {{ hostname }}

在ambari-server端對應的模板文件爲：

# grep hostname /var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch*
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2.5.6.2: name: {{hostname}}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2.5.6.2: name: {{hostname}}

而後能夠啓動成功：

 

 

參考：

https://community.hortonworks.com/articles/215078/how-to-install-elk-stack-632-in-ambari.htmlhttps://community.hortonworks.com/questions/212871/add-custom-service-via-ambari-error.htmlhttps://community.hortonworks.com/articles/63557/how-to-edit-the-cluster-envxml-entries-using-ambar.html