jstorm單機部署和實時gps點統計實踐

1 安裝zookeeper

 1.1 下載

  wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

 1.2 配置

  配置zookeeper參數zoo.cfg，步驟以下：

   a cp opt/zookeeper-3.4.6/conf/zoo_sample.cfg opt/zookeeper-3.4.6/conf/zoo.cfg

   b 確認參數clientPort=2181（ 因爲jstorm依賴zk服務,其storm.yaml默認zk的ip和port是：localhost:2181） 

 1.3 啓動和檢驗

   運行zookeeper-3.4.6/bin/zkServer.sh，如看到以下相似日誌，表示單點zk服務啓動成功： 

2015-02-05 19:39:38,849 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg   2015-02-05 19:39:38,855 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3   2015-02-05 19:39:38,855 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0   2015-02-05 19:39:38,855 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.   2015-02-05 19:39:38,856 [myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode   2015-02-05 19:39:38,876 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg   2015-02-05 19:39:38,876 [myid:] - INFO  [main:ZooKeeperServerMain@95] - Starting server

2 安裝jstorm

 2.1 下載 

  wget http://42.121.19.155/jstorm/jstorm-0.9.6.2.zip

 2.2 運行

  opt/jstorm-0.9.6.2/bin/start.sh

 2.3 查看

  在/opt/jstorm-0.9.6.2/logs/能夠分別看到nimbus.log和supervisor.log，其內容以下:

2015-02-06 00:23:19 JStormServerUtils:83 main] Successfully touch pid  /opt/storm-test/jstorm-0.9.6.2/data/supervisor/pids/10520

3 安裝jstorm-ui

 3.1 下載tomcat

   wget http://mirrors.cnnic.cn/apache/tomcat/tomcat-8/v8.0.15/bin/apache-tomcat-8.0.15.tar.gz

 3.2 拷貝jstorm-ui

   a cp jstorm-ui-0.9.6.2.war opt/apache-tomcat-8.0.17/webapps

   b ln -s jstorm-ui-0.9.6.2 ROOT 

 3.3 啓動

   運行opt/apache-tomcat-8.0.17/bin/startup.sh

 3.4 查看

   訪問 http://10.17.133.52:8080/cluster.jsf?clusterName=，能夠看到jstorm以下監控頁面:  

 4 jstorm實踐

  4.1. 背景

     a. 功能

        實時接收gps數據採集點，按指定的時間維度(如1分鐘)統計打印各個源頭的gps個數

     b. 編碼

        實現一個Spout:fpSubscriber和兩個Bolt:fpCombiner和fpWriter，代碼以下：             

        TopologyBuilder builder = new TopologyBuilder();

        builder.setSpout("fpSubscriber", new FpSubscriber(), 1);

        builder.setBolt("fpCombiner", new FpStat(), 1).shuffleGrouping("fpSubscriber");

        builder.setBolt("fpWriter", new FpWriter(), 1).shuffleGrouping("fpCombiner");

       

        Config conf = new Config();

        conf.put(Config.NIMBUS_HOST, "localhost");

        conf.put(Config.NIMBUS_THRIFT_PORT, 6627);

        conf.setDebug(true);

        conf.setNumWorkers(3);

       

        if(args!=null && args.length > 0) {

            StormSubmitter.submitTopology(args[0], conf, builder.createTopology());

        } else {

          StormSubmitter.submitTopology("fp-stat", conf, builder.createTopology());

        }

    說明：     

      fpSubscriber：實時接收線上fp發佈數據，推給下游fpCombiner的節點;

      fpCombiner:進行gps按dscode的分組和統計，並定時分發下游fpWriter節點;

      fpWriter：打印或持久化每一個dscode的按期gpscount個數的統計結果.

4.2 部署         

  a. 打包並上傳應用程序fp-stat.jar

  b. 啓動 ./bin/jstorm jar fp-stat.jar storm.test.FpStatTopology

4.3 展現驗證

  a. 界面展現storm-ui以下： 

      能夠看到一個fp-stat的topo任務,其狀態爲active；點擊進入fp-stat，發現有三個worker,分別是一個Spout:fpSubscriber，兩個Bolt:fpCombiner和fpWriter。

  b. 結果顯示

      在fpWriter輸出日誌目錄opt/ jstorm-0.9.6.2/logs/ fp-stat-worker-6801.log.out能夠發現以下統計結果：