Twitter開源的Heron快速安裝部署教程

什麼是Heron？

Twitter使用Storm實時分析海量數據已經有好幾年了，並在2011年將其開源。該項目稍後開始在Apache基金會孵化，並在2015年秋天成爲頂級項目。Storm以季度爲發佈週期，而且向着人們指望的穩定版前進。但一直以來，Twitter都在致力於開發替代方案Heron，由於Storm沒法知足他們的實時處理需求。Twitter如今已經用Heron徹底替換了Storm。前者如今天天處理「數10TB的數據，生成數10億輸出元組」，在一個標準的單詞計數測試中，「吞吐量提高了6到14倍，元組延遲下降到了原來的五到十分之一」，硬件減小了2/3。當被問到Twitter是否會開源Heron時，Ramasamy說「在短期內不會，但長期來看可能。」然而就在2016年5月25日，Twitter正式宣佈Heron開源。Twitter工程經理Karthik Ramasamy在博客上宣佈了這一消息。

快速安裝Heron

1、使用安裝腳本下載heron的二進制文件

安裝腳本的下載地址是https://github.com/twitter/heron/releases。

博主用的是centos7，因此下載了centos版本的安裝文件。

 

[heron@spark ~]$ ls heron-client-install-0.14.4-centos.sh heron-tools-install-0.14.4-centos.sh [heron@spark ~]$ 

下載完後開始安裝

[root@spark heron]# chmod +x heron-client-install-0.14.4-centos.sh [root@spark heron]# ./heron-client-install-0.14.4-centos.sh --warning=no-timestamp Heron client installer ---------------------- Uncompressing..tar xfz /usr/local/heron/heron-client.tar.gz -C /usr/local/heron --warning=no-timestamp .... Heron is now installed! Make sure you have "/usr/local/bin" in your path. See http://heronstreaming.io/docs/getting-started for how to use Heron. heron.build.version : '0.14.4-1-g8de8003' heron.build.time : Wed Oct 5 17:55:30 PDT 2016 heron.build.timestamp : 1475715348000 heron.build.host : tw-mbp-kramasamy heron.build.user : kramasamy heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f heron.build.git.status : Clean [root@spark heron]# 

 默認安裝的路徑是/usr/local/bin。

也能夠指定安裝路徑  ./heron-client-install-VERSION-PLATFORM.sh --user

安裝完後，配置一下 /etc/profile

vi /etc/profile

export PATH=$PATH:/usr/local/bin

而後source一下

source /etc/profile

安裝另外一個文件

[root@spark heron]# chmod +x heron-tools-install-0.14.4-centos.sh [root@spark heron]# ./heron-tools-install-0.14.4-centos.sh --warning=no-timestamp Heron tools installer --------------------- Uncompressing..tar xfz /usr/local/herontools/heron-tools.tar.gz -C /usr/local/herontools --warning=no-timestamp .... Heron Tools is now installed! Make sure you have "/usr/local/bin" in your path. See http://heronstreaming.io/docs/getting-started for how to use Heron. heron.build.version : '0.14.4-1-g8de8003' heron.build.time : Wed Oct 5 17:55:30 PDT 2016 heron.build.timestamp : 1475715348000 heron.build.host : tw-mbp-kramasamy heron.build.user : kramasamy heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f heron.build.git.status : Clean [root@spark heron]# 

最後確認一下是否安裝完成

[root@spark local]# heron version heron.build.version : '0.14.4-1-g8de8003' heron.build.time : Wed Oct 5 17:55:30 PDT 2016 heron.build.timestamp : 1475715348000 heron.build.host : tw-mbp-kramasamy heron.build.user : kramasamy heron.build.git.revision : 8de80037791dfde3d493a6ec255429908716f03f heron.build.git.status : Clean [root@spark local]# 

2、提交一個topology

[root@spark local]# heron submit local /usr/local/heron/examples/heron-examples.jar com.twitter.heron.examples.ExclamationTopology ExclamationTopology --deploy-deactivated INFO: Using config file under /usr/local/heron/conf/local INFO: Launching topology 'ExclamationTopology' INFO: Topology 'ExclamationTopology' launched successfully INFO: Elapsed time: 4.417s. [root@spark local]# 

提交了topology後，heron並不會激活它。

咱們查看一下topology是否成功啓動和工做目錄

[root@spark local]# ls -al ~/.herondata/topologies/local/root/ExclamationTopology container_1_exclaim1_1.pid heron-executor-0.stdout log-files metrics.json.metricsmgr-2.1 container_1_exclaim1_3.pid heron-executor-1.pid metrics.json.metricsmgr-0.0 metrics.json.metricsmgr-2.2 container_1_word_5.pid heron-executor-1.stderr metrics.json.metricsmgr-0.1 metrics.json.metricsmgr-2.3 container_2_exclaim1_2.pid heron-executor-1.stdout metrics.json.metricsmgr-0.2 metrics.json.metricsmgr-2.4 container_2_exclaim1_4.pid heron-executor-2.pid metrics.json.metricsmgr-0.3 metricsmgr-0.pid container_2_word_6.pid heron-executor-2.stderr metrics.json.metricsmgr-0.4 metricsmgr-1.pid ExcalamationTopology.defn heron-executor-2.stdout metrics.json.metricsmgr-1.0 metricsmgr-2.pid heron-conf heron-shell-0.pid metrics.json.metricsmgr-1.1 release.yaml heron-core heron-shell-1.pid metrics.json.metricsmgr-1.2 stmgr-1.pid heron-examples.jar heron-shell-2.pid metrics.json.metricsmgr-1.3 stmgr-2.pid heron-executor-0.pid heron-tmaster.pid metrics.json.metricsmgr-1.4 heron-executor-0.stderr java-f6c01842-started.stderr metrics.json.metricsmgr-2.0 [root@spark local]# 

全部實例的日誌文件均可以在工做目錄下的日誌文件中找到。

[root@spark local]# ls -al ~/.herondata/topologies/local/root/ExclamationTopology/log-files/ container_1_exclaim1_1.log.0 container_2_word_6.log.0 heron-ExclamationTopology-scheduler.log.0.lck container_1_exclaim1_1.log.0.lck container_2_word_6.log.0.lck heron-shell.log container_1_exclaim1_3.log.0 gc.container_1_exclaim1_1.log metricsmgr-0.log.0 container_1_exclaim1_3.log.0.lck gc.container_1_exclaim1_3.log metricsmgr-0.log.0.lck container_1_word_5.log.0 gc.container_1_word_5.log metricsmgr-1.log.0 container_1_word_5.log.0.lck gc.container_2_exclaim1_2.log metricsmgr-1.log.0.lck container_2_exclaim1_2.log.0 gc.container_2_exclaim1_4.log metricsmgr-2.log.0 container_2_exclaim1_2.log.0.lck gc.container_2_word_6.log metricsmgr-2.log.0.lck container_2_exclaim1_4.log.0 gc.metricsmgr.log container_2_exclaim1_4.log.0.lck heron-ExclamationTopology-scheduler.log.0 

3、啓動 Heron Tracker

[root@spark local]# heron-tracker INFO: 11/09/2016 09:55:17 +0000 Connecting to file state with rootpath: /root/.herondata/repository/state/local INFO: 11/09/2016 09:55:17 +0000 State watch triggered for topologies. INFO: 11/09/2016 09:55:17 +0000 Adding new topology: ExcalamationTopology, state_manager: local INFO: 11/09/2016 09:55:17 +0000 Registering a watch with uid: 6d218788-1c8c-4f43-806a-a217c59ab4b5 INFO: 11/09/2016 09:55:17 +0000 No execution state found for: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology execution state: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Tracker has started INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Running on port: 8888 INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology pplan: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Using config file: /usr/local/herontools/conf/heron_tracker.yaml INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology tmaster: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Watch triggered for topology scheduler location: ExcalamationTopology INFO: 11/09/2016 09:55:17 +0000 Setting topology info for topology: ExcalamationTopology 

啓動後能夠經過瀏覽器來查看剛剛提交的topology是否成功，端口號是8888

瞭解更多的heron tracker，請參考Heron Tracker Rest API

4、啓動 Heron UI

[root@spark ~]# heron-ui INFO: 11/09/2016 10:02:25 +0000 Listening at http://0.0.0.0:8889 INFO: 11/09/2016 10:02:25 +0000 Using tracker url: http://localhost:8888 INFO: 11/09/2016 10:02:29 +0000 200 GET /topologies (192.168.25.57) 52.86ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/bootstrap.js (192.168.25.57) 37.41ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/jquery.min.js (192.168.25.57) 24.41ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/jquery-ui-1.8.23.custom.min.js (192.168.25.57) 16.54ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/moment.min.2.8.3.js (192.168.25.57) 15.74ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/underscore-min.1.6.0.js (192.168.25.57) 9.57ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/css/visstyle.css (192.168.25.57) 6.50ms INFO: 11/09/2016 10:02:29 +0000 200 GET /static/js/common.js?v=aea0e809de29e66079cc8ee4c7ea16f7 (192.168.25.57) 2.54ms INFO: 11/09/2016 10:02:30 +0000 200 GET /static/js/alltopologies.js?v=f6ba00d23089084d4cb32f8bfcf351a0 (192.168.25.57) 3.48ms INFO: 11/09/2016 10:02:30 +0000 200 GET /static/img/logo54x54.png (192.168.25.57) 9.42ms INFO: 11/09/2016 10:02:31 +0000 200 GET /static/icomoon/fonts/icomoon.woff (192.168.25.57) 21.88ms INFO: 11/09/2016 10:02:31 +0000 200 GET /topologies/list.json?format=json (192.168.25.57) 6.98ms 

啓動後能夠經過瀏覽器查看，端口號8889。（heron-tracker必須啓動，否則報錯）

 

5、topology的管理命令

在第二步中，咱們已經提交了topology，如今咱們能夠用命令激活、中止、殺死等命令

heron activate local ExclamationTopology
heron deactivate local ExclamationTopology
heron kill local ExclamationTopology

 

博文地址：http://julyme.com/20161108/21.html

我的博客：http://julyme.com