Tachyon是一個高容錯的分布式文件系統(tǒng)，允許文件以內(nèi)存的速度在集群框架中進(jìn)行可靠的共享，就像Spark和 MapReduce那樣。通過利用信息繼承，內(nèi)存侵入，Tachyon獲得了高性能。Tachyon工作集文件緩存在內(nèi)存中，并且讓不同的 Jobs/Queries以及框架都能內(nèi)存的速度來訪問緩存文件。因此，Tachyon可以減少那些需要經(jīng)常使用的數(shù)據(jù)集通過訪問磁盤來獲得的次數(shù)。

源碼下載

源碼地址:https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git

編譯

進(jìn)入 TACHYON_HOME

git tagv0.1.0 v0.2.0 v0.2.1 v0.3.0 v0.4.0 v0.4.1 v0.4.1-thrift v0.5.0 v0.6.0 v0.6.1 v0.6.2 v0.6.3 v0.6.4 v0.7.0 v0.7.0-rc1 v0.7.1

切換到v0.7.1版本

git checkout v0.7.1 Note: checking out 'v0.7.1'.You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:git checkout -b new_branch_nameHEAD 目前位于 a2b69c9... [maven-release-plugin] prepare release v0.7.1

執(zhí)行如下命令:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn clean package -Dhadoop.version=2.3.0-cdh5.1.0 -Dspark.version=1.4.1 -DskipTests

配置

創(chuàng)建journal目錄
mkdir -p /home/cluster/apps/tachyon/journal

創(chuàng)建ramdisk
mkdir -p /home/cluster/apps/tachyon/ramdisk

將tachyon_home/conf/tachyon-env.sh.template 修改為tachyon_home/conf/tachyon-env.sh
mv conf/tachyon-env.sh.template conf/tachyon-env.sh

編輯tachyon_home/conf/tachyon-env.sh 增加如下內(nèi)容

export JAVA_HOME=/home/cluster/share/java1.7 export HADOOP_HOME=/home/cluster/apps/hadoop export TACHYON_HOME=/home/cluster/apps/tachyon export TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdisk export TACHYON_MASTER_ADDRESS=master

修改tachyon日志存放路徑:
tachyon_home/下創(chuàng)建logs目錄
mkdir logs
vim conf/log4j.properties 增加如下內(nèi)容:
tachyon.logs.dir=/home/cluster/apps/tachyon/logs

將tachyon_home/conf/core-site.xml.template 修改為tachyon_home/conf/core-site.xml
mv conf/core-site.xml.template conf/core-site.xml
編輯 conf/core-site.xml,增加如下內(nèi)容:

<configuration>
<property>
　 <name>fs.tachyon.impl</name>
　　<value>tachyon.hadoop.TFS</value>
　</property>
</configuration>

格式化Tachyon

bin/tachyon format Connecting to localhost as wangyue... Formatting Tachyon Worker @ wangyue-um Connection to localhost closed. Formatting Tachyon Master @ master

啟動

bin/tachyon-start.sh local Killed 0 processes on wangyue-um Killed 0 processes on wangyue-um Connecting to localhost as wangyue... Killed 0 processes on wangyue-um Connection to localhost closed. [sudo] password for wangyue: Formatting RamFS: /home/cluster/apps/tachyon/ramdisk (1gb) Starting master @ master Starting worker @ wangyue-umjps下: 21025 ZeppelinServer 22214 Worker 21987 Master 21360 TachyonWorker 32212 RemoteInterpreterServer 23361 NameNode 29603 RemoteInterpreterServer 21324 TachyonMaster 23493 DataNode 21412 Jps 27005 RemoteInterpreterServer已經(jīng)啟動TachyonMaster & TachyonWorker

tachyon-start.sh local命令將同時在本地啟動Master和Worker進(jìn)程。需要注意的是，運(yùn)行tachyon-start.sh local命令一定要擁有切換root的密碼，否者會無法啟動。這是因?yàn)镽amFS的格式化需要root權(quán)限。

測試

webui:http://master:19999

/home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH /default_tests_files/BasicFile_CACHE_THROUGH has been removed 2015-09-09 20:00:17,911 INFO (MasterClient.java:connect) - Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:19998 2015-09-09 20:00:17,938 INFO (MasterClient.java:connect) - User registered with the master @ master/127.0.0.1:19998; got UserId 5 2015-09-09 20:00:17,964 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 57 ms. 2015-09-09 20:00:17,999 INFO (WorkerClient.java:connect) - Trying to get local worker host : wangyue-um.local 2015-09-09 20:00:18,013 INFO (WorkerClient.java:connect) - Connecting local worker @ wangyue-um.local/10.32.24.78:29998 2015-09-09 20:00:18,098 INFO (BlockOutStream.java:get) - Writing with local stream. tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH 2015-09-09 20:00:18,136 INFO (CommonUtils.java:createBlockPath) - Folder /home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was created! 2015-09-09 20:00:18,140 INFO (LocalBlockOutStream.java:<init>) - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 536870912 2015-09-09 20:00:18,203 INFO (CommonUtils.java:printTimeTakenMs) - writeFile to file /default_tests_files/BasicFile_CACHE_THROUGH took 238 ms. 2015-09-09 20:00:18,246 INFO (CommonUtils.java:printTimeTakenMs) - readFile file /default_tests_files/BasicFile_CACHE_THROUGH took 43 ms. Passed the test!執(zhí)行 bin/tachyon runTests 循環(huán)測試

停止

wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh Killed 1 processes on wangyue-um Killed 1 processes on wangyue-um Connecting to localhost as wangyue... Killed 0 processes on wangyue-um Connection to localhost closed.執(zhí)行jps: 21025 ZeppelinServer 22214 Worker 21987 Master 32212 RemoteInterpreterServer 23361 NameNode 29603 RemoteInterpreterServer 27469 Jps 23493 DataNode 27005 RemoteInterpreterServer 沒有tachyon相關(guān)進(jìn)程

尊重原創(chuàng),拒絕轉(zhuǎn)載,http://blog.csdn.net/stark_summer/article/details/48321605

版權(quán)聲明：本文為博主原創(chuàng)文章，未經(jīng)博主允許不得轉(zhuǎn)載。

轉(zhuǎn)載于:https://www.cnblogs.com/stark-summer/p/4829743.html

總結(jié)

以上是生活随笔為你收集整理的Tachyon 0.7.1伪分布式集群安装与测试的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。