Hadoop+HBase平臺上安裝snappy

在測試Apache Phoenix過程當中，遇到了許多問題，首當其衝的就是缺乏snappy壓縮包。沒辦法，hbase-0.98.6.1沒有自帶，就只能本身安裝了。首先固然仍是上網搜一搜，看有沒有寫的比較好的安裝步驟。結果發現，千篇一概，都是要下載編譯一個貌似是hadoop-snappy的包。仔細看了看過程，還挺嚇人的，中間出現了各類錯誤，甚至還對gcc的版本有要求。一氣之下，就不想裝了（實際上是比較懶了嘿嘿）。

而後某一天忽然看到了這篇博客：How to install snappy with HBase 0.94.x，做者說他摸索了3天才找出來這個不走彎路的辦法，且僅需5分鐘就能搞定(方法很簡單，但我仍是花了大概5個小時的時間，在Ubuntu上安裝東西，你懂的。。。)

文章中介紹了一種安裝在HBase上安裝snappy的方法，很簡單，因而決定一試，現將這篇文中講到的方法翻譯過來與你們分享：

一、檢查snappy壓縮包是否安裝

命令爲：bin/hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy

若是顯示信息爲：

12/12/03 10:30:02 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:30:02 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
     at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
     at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
     at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
     at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
     at org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)

 則說明snappy壓縮包沒有安裝；

二、下載snappy-*.tar.gz壓縮包(只要和hbase版本兼容就能夠，個人是snappy-1.1.1.tar.gz)，解壓；

三、進入snappy目錄，進行編譯，兩條命令：

      ./configure

       make

四、make完以後會產生一個libsnappy.so文件(這就是咱們所須要的庫！！！)，正常狀況下出如今當前目錄./libs/libsnappy.so，可是不少時候不按套路出牌，跑到別的文件夾下了，若是make沒有出錯，能夠在根目錄search一下，確定能找到這個文件；

五、將生成的這個libsnappy.so拷貝到HBase的lib/native/Linux-ARCH目錄下，ARCH表明 amd64 或 i386-32，注意，對於amd64的HBase可能沒有這個目錄，此時，須要手動建立：

     mkdir /opt/hbase-0.98.6.1/lib/native/Linux-amd64-64

六、若是仍是不肯定HBase在哪裏查找lib，那麼能夠修改log4j文件中的日誌級別(log level)進行調試；

七、從新運行第1步中的命令，如今看到的信息應該爲：

12/12/03 10:34:35 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
12/12/03 10:34:35 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32C not available. 
12/12/03 10:34:35 DEBUG util.FSUtils: Creating file:file:/tmp/test.txtwith permission:rwxrwxrwx
12/12/03 10:34:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/12/03 10:34:35 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:34:35 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 10:34:35 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
    at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
    at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
    at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
    at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
    at org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
    at org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)

八、能夠看到，snappy已經能夠找到了，可是尚未加載(not loaded)。想加載的話，還須要拷貝hadoop的本地庫到與libsnappy.so同一個路徑下，hadoop的本地庫路徑爲：

      hadoop-1.2.1/lib/native/Linux-ARCH/libhadoop.so；

       若是這個路徑下沒有，能夠根據所使用的hadoop版本到 https://archive.apache.org/dist/hadoop/core/ 下載相應的tar.gz包，解壓以後就能找到所須要的文件了；

九、再次運行測試命令(第1步中的命令)，能夠獲得：

12/12/03 10:37:48 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32 not available.
12/12/03 10:37:48 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
12/12/03 10:37:48 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32C not available. 
12/12/03 10:37:48 DEBUG util.FSUtils: Creating file:file:/tmp/test.txtwith permission:rwxrwxrwx
12/12/03 10:37:48 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/12/03 10:37:48 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:37:48 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 10:37:48 INFO snappy.LoadSnappy: Snappy native library loaded
12/12/03 10:37:48 INFO compress.CodecPool: Got brand-new compressor
12/12/03 10:37:48 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:disabled
12/12/03 10:37:49 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:37:49 INFO compress.CodecPool: Got brand-new decompressor
SUCCESS

      看到SUCCESS，說明安裝成功，snappy壓縮包可使用，搞定。

ps：建議你們能夠去看下這篇文章

網址：http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with#.VREJ__Sl8tM

 

注意：若是在make過程當中出錯，在從新make時必定要把以前的snappy文件夾刪除，從新解壓出一個新的文件夾，切記