storm+kestrel+zookeeper

storm+kestrel+zookeeper

 

環境：

2臺服務器：192.168.1.166

           192.168.1.167

系統：centos 5.6

 

部署：192.168.1.166：ui,nimbus,supervisor

      192.168.1.167:supervisor,kestrel,zookeeper

 

    由於公司網站對數據的實時性要求比較高，因此最近一直在研究storm，由於9月份剛開源，因此一些資料相對來講比較說，只能從官方wiki上去查，地址： https://github.com/nathanmarz/storm/wiki

    storm 是一個分佈式的、容錯的實時計算系統，它被託管在 GitHub 上，遵循 Eclipse Public License 1.0。Storm是由BackType開發的實時處理系統，BackType如今已在Twitter麾下。GitHub上的最新版本是Storm 0.5.2，基本是用 Clojure 寫的。

 

一、安裝zookeeper

 

zookeeper集羣部署方式：

tar xvzf zookeeper-3.3.3.tar.gz

cd zookeeper-3.3.3

mv conf/zoo-sample.cfg conf/zoo.cfg

vim conf/zoo.cfg

 

tickTime=2000

initLimit=5

syncLimit=2

dataDir=/var/lib/storm/zookeeper

dataLogDir=/var/log/zookeeper

clientPort=2181

server.1=192.168.1.166:2888:3888

server.2=192.168.1.167:2888:3888

 

保存退出。

mkdir /var/lib/storm/zookeeper

vim /var/lib/storm/zookeeper/myid

本機是166，因此這裏輸入1

保存退出。

 

另外2臺機器一樣的步驟，不過myid改成2,3便可。

注意：zookeeper的集羣必須是單數的機器，也就是說要麼3臺作，要麼單臺作僞集羣，2臺作出來的話我我的測試是有問題的，這裏也不是很肯定，若是有哪位知道，麻煩告知下。

 

啓動2臺上面的服務：

/usr/local/zookeeper-3.3.3/bin/zkServer.sh start

 

#zookeeper單機部署：

同集羣相同：只是配置文件略有不一樣：

#vim zoo.cfg

tickTime=2000

minSessionTimeout=2000

maxSessionTimeout=20000

dataDir=/var/lib/storm/zookeeper

dataLogDir=/var/log/zookeeper

clientPort=2181

保存退出

而後直接啓動服務便可：/usr/local/zookeeper-3.3.3/bin/zkServer.sh start

 

 

一、安裝zeromq

 安裝依賴包

 yum -y install gcc-c++ e2fsprogs.x86_64 e2fsprogs-devel.x86_64

  

tar xvzf zeromq-2.1.7.tar.gz

cd zeromq-2.1.7

./configure

make 

make install

 

二、安裝jzmq

 

tar xvzf nathanmarz-jzmq-dd3327d.tar.gz

cd tar xvzf nathanmarz-jzmq-dd3327d

yum install pkgconfig libtool.x86_64

./autogen.sh

./configure

make

make install

 

 

三、安裝python 2.6.6

 

tar jxvf Python-2.6.6.tar.bz2

./configure --bindir=/usr/bin --libdir=/usr/lib

make 

make install

 

四、安裝kestrel

安裝kestrel須要安裝daemon

tar xvzf daemon-0.6.4.tar.gz

cd daemon-0.6.4

./config

make

make test

make install

make install-daemon-conf

make install-slack

 

tar xvzf kestrel-2.1.3.zip

mv kestrel-2.1.3 /usr/local/kestrel

vim /usr/local/kestrel/scripts/kestrel.sh

修改APP_HOME="/usr/local/$APP_NAME/current"爲

APP_HOME="/usr/local/$APP_NAME"

保存

cp -rp /usr/local/kestrel/scripts/kestrel.sh /etc/init.d/kestrel

service kestrel start

 

五、安裝storm

unzip storm-0.6.0.zip

cd storm-0.6.0

cd conf

vim strom.yaml

內容以下：

============================================================================

java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"

 

### storm.* configs are general configurations

# the local dir is where jars are kept

 storm.local.dir: "/var/lib/storm/data"

 storm.zookeeper.servers:

     - "192.168.1.166"

 storm.zookeeper.port: 2181

 storm.zookeeper.root: "/var/lib/storm/storm"

 storm.zookeeper.session.timeout: 20000

 storm.cluster.mode: "distributed" # can be distributed or local

 storm.local.mode.zmq: false

 

### nimbus.* configs are for the master

 nimbus.host: "192.168.1.166"

 nimbus.thrift.port: 6627

 nimbus.childopts: "-Xmx2048m"

 nimbus.task.timeout.secs: 30

 nimbus.supervisor.timeout.secs: 60

 nimbus.monitor.freq.secs: 10

 nimbus.task.launch.secs: 240

 nimbus.reassign: true

 nimbus.file.copy.expiration.secs: 600

 

 ui.port: 8080

 

 drpc.port: 3772

 

 

 supervisor.slots.ports:

    - 6700

    - 6701

    - 6702

    - 6703

 supervisor.childopts: "-Xmx2048m"

#how long supervisor will wait to ensure that a worker process is started

 supervisor.worker.start.timeout.secs: 240

#how long between heartbeats until supervisor considers that worker dead and tries to restart it

 supervisor.worker.timeout.secs: 30

#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary

 supervisor.monitor.frequency.secs: 3

#how frequently the supervisor heartbeats to the cluster state (for nimbus)

 supervisor.heartbeat.frequency.secs: 5

 supervisor.enable: true

 

### worker.* configs are for task workers

 worker.childopts: "-Xmx768m"

 worker.heartbeat.frequency.secs: 1

 

 task.heartbeat.frequency.secs: 3

 task.refresh.poll.secs: 10

 

 zmq.threads: 1

 zmq.linger.millis: 5000

 

 =======================================================================================================

日誌路徑修改：vim log4j/storm.log.properties

 修改：log4j.appender.A1.File = logs/${logfile.name}   爲log4j.appender.A1.File = /var/log/${logfile.name}

 

 保存退出。啓動服務。

 nimbus：nohup /opt/storm/storm-0.6.0/bin/storm nimbus &

         nohup /opt/storm/storm-0.6.0/bin/storm ui &

nohup /opt/storm/storm-0.6.0/bin/storm supervisor &

 supervisor:nohup /opt/storm/storm-0.6.0/bin/storm supervisor &

 

 ui訪問：http://192.168.1.166:8080

 

說明：storm0.5.4的集羣部署，我這邊測試老是有問題，具體緣由未知，也不知道是否爲bug，不過6.0的總算成功，有興趣的能夠試試。