Tachyon 0.7.1僞分佈式集羣安裝與測試

Tachyon是一個高容錯的分佈式文件系統，容許文件之內存的速度在集羣框架中進行可靠的共享，就像Spark和 MapReduce那樣。經過利用信息繼承，內存侵入，Tachyon得到了高性能。Tachyon工做集文件緩存在內存中，而且讓不一樣的 Jobs/Queries以及框架都能內存的速度來訪問緩存文件。所以，Tachyon能夠減小那些須要常常使用的數據集經過訪問磁盤來得到的次數。

源碼下載

源碼地址:https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git

編譯

進入 TACHYON_HOME

git tag

v0.1.0
v0.2.0
v0.2.1
v0.3.0
v0.4.0
v0.4.1
v0.4.1-thrift
v0.5.0
v0.6.0
v0.6.1
v0.6.2
v0.6.3
v0.6.4
v0.7.0
v0.7.0-rc1
v0.7.1

切換到v0.7.1版本

git checkout v0.7.1 
Note: checking out 'v0.7.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD 目前位於 a2b69c9... [maven-release-plugin] prepare release v0.7.1

執行以下命令:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn clean package  -Dhadoop.version=2.3.0-cdh5.1.0 -Dspark.version=1.4.1 -DskipTests

配置

1. 建立journal目錄
   mkdir -p /home/cluster/apps/tachyon/journal
2. 建立ramdisk
   mkdir -p /home/cluster/apps/tachyon/ramdisk
3. 將tachyon_home/conf/tachyon-env.sh.template 修改成tachyon_home/conf/tachyon-env.sh
   mv conf/tachyon-env.sh.template conf/tachyon-env.sh
4. 編輯tachyon_home/conf/tachyon-env.sh 增長以下內容

export JAVA_HOME=/home/cluster/share/java1.7
export HADOOP_HOME=/home/cluster/apps/hadoop
export TACHYON_HOME=/home/cluster/apps/tachyon
export TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdisk
export TACHYON_MASTER_ADDRESS=master

1. 修改tachyon日誌存放路徑:
   tachyon_home/下建立logs目錄
   mkdir logs
   vim conf/log4j.properties 增長以下內容:
   tachyon.logs.dir=/home/cluster/apps/tachyon/logs
2. 將tachyon_home/conf/core-site.xml.template 修改成tachyon_home/conf/core-site.xml
   mv conf/core-site.xml.template conf/core-site.xml
   編輯 conf/core-site.xml,增長以下內容:
   
<configuration>
<property>
　 <name>fs.tachyon.impl</name>
　　<value>tachyon.hadoop.TFS</value>
　</property>
</configuration>

3. 格式化Tachyon

bin/tachyon format
Connecting to localhost as wangyue...
Formatting Tachyon Worker @ wangyue-um
Connection to localhost closed.
Formatting Tachyon Master @ master

啓動

bin/tachyon-start.sh local
Killed 0 processes on wangyue-um
Killed 0 processes on wangyue-um
Connecting to localhost as wangyue...
Killed 0 processes on wangyue-um
Connection to localhost closed.
[sudo] password for wangyue: 
Formatting RamFS: /home/cluster/apps/tachyon/ramdisk (1gb)
Starting master @ master
Starting worker @ wangyue-um

jps下:
21025 ZeppelinServer
22214 Worker
21987 Master
21360 TachyonWorker
32212 RemoteInterpreterServer
23361 NameNode
29603 RemoteInterpreterServer
21324 TachyonMaster
23493 DataNode
21412 Jps
27005 RemoteInterpreterServer

已經啓動TachyonMaster & TachyonWorker

tachyon-start.sh local命令將同時在本地啓動Master和Worker進程。須要注意的是，運行tachyon-start.sh local命令必定要擁有切換root的密碼，否者會沒法啓動。這是由於RamFS的格式化須要root權限。

測試

webui:http://master:19999

/home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH
/default_tests_files/BasicFile_CACHE_THROUGH has been removed
2015-09-09 20:00:17,911 INFO   (MasterClient.java:connect) - Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:19998
2015-09-09 20:00:17,938 INFO   (MasterClient.java:connect) - User registered with the master @ master/127.0.0.1:19998; got UserId 5
2015-09-09 20:00:17,964 INFO   (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 57 ms.
2015-09-09 20:00:17,999 INFO   (WorkerClient.java:connect) - Trying to get local worker host : wangyue-um.local
2015-09-09 20:00:18,013 INFO   (WorkerClient.java:connect) - Connecting local worker @ wangyue-um.local/10.32.24.78:29998
2015-09-09 20:00:18,098 INFO   (BlockOutStream.java:get) - Writing with local stream. tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH
2015-09-09 20:00:18,136 INFO   (CommonUtils.java:createBlockPath) - Folder /home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was created!
2015-09-09 20:00:18,140 INFO   (LocalBlockOutStream.java:<init>) - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 536870912
2015-09-09 20:00:18,203 INFO   (CommonUtils.java:printTimeTakenMs) - writeFile to file /default_tests_files/BasicFile_CACHE_THROUGH took 238 ms.
2015-09-09 20:00:18,246 INFO   (CommonUtils.java:printTimeTakenMs) - readFile file /default_tests_files/BasicFile_CACHE_THROUGH took 43 ms.
Passed the test!

執行  bin/tachyon runTests 循環測試

中止

wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh 
Killed 1 processes on wangyue-um
Killed 1 processes on wangyue-um
Connecting to localhost as wangyue...
Killed 0 processes on wangyue-um
Connection to localhost closed.

執行jps:
21025 ZeppelinServer
22214 Worker
21987 Master
32212 RemoteInterpreterServer
23361 NameNode
29603 RemoteInterpreterServer
27469 Jps
23493 DataNode
27005 RemoteInterpreterServer
沒有tachyon相關進程

尊重原創,拒絕轉載,http://blog.csdn.net/stark_summer/article/details/48321605

版權聲明：本文爲博主原創文章，未經博主容許不得轉載。