HBase實戰（1）：使用Python鏈接Hbase數據庫【未測試】

時間 2019-11-17

標籤 hbase 實戰使用 python 鏈接數據庫未測試欄目 Hadoop 简体版

原文原文鏈接

使用Python鏈接Hbase數據庫html

1，Hbase下載。html5

下載地址：http://hbase.apache.org/downloads.htmlpython

2，本地Hbase安裝linux

root@master:/usr/local/setup_tools# tar -zxvf hbase-2.0.0-bin.tar.gzshell

    root@master:/usr/local/setup_tools# mv hbase-2.0.0 /usr/local/
    root@master:/usr/local/setup_tools# cd /usr/local
    root@master:/usr/local# ls | grep hbase
    hbase-2.0.0
    root@master:/usr/local#數據庫

    root@master:/usr/local/hbase-2.0.0# vi /etc/profile
    export HBASE_HOME=/usr/local/hbase-2.0.0

    export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$HIVE_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin:$IDEA_HOME/bin:$eclipse_HOME:$MAVEN_HOME/bin:$ALLUXIO_HOME/bin:$HBASE_HOME/binapache

root@master:/usr/local/hbase-2.0.0# source /etc/profileubuntu

3，配置ruby

修改hbase-site.xml，設置存儲數據的根目錄。app

root@master:/usr/local/hbase-2.0.0/conf# vi hbase-site.xml

    <configuration>
        <property>
            <name>hbase.rootdir</name>
            <value>file:///usr/local/hbase-2.0.0/data</value>
        </property>

    </configuration>

4，啓動hbase

    root@master:/usr/local/hbase-2.0.0# cd bin
    root@master:/usr/local/hbase-2.0.0/bin# ls
    considerAsDead.sh     hbase             hbase-config.cmd hbase-jruby             master-backup.sh replication               start-hbase.sh zookeepers.sh
    draining_servers.rb   hbase-cleanup.sh hbase-config.sh   hirb.rb                 region_mover.rb   rolling-restart.sh        stop-hbase.cmd
    get-active-master.rb hbase.cmd         hbase-daemon.sh   local-master-backup.sh regionservers.sh shutdown_regionserver.rb stop-hbase.sh
    graceful_stop.sh      hbase-common.sh   hbase-daemons.sh local-regionservers.sh region_status.rb start-hbase.cmd           test
    root@master:/usr/local/hbase-2.0.0/bin# start-hbase.sh
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    running master, logging to /usr/local/hbase-2.0.0/logs/hbase-root-master-master.out
    root@master:/usr/local/hbase-2.0.0/bin# jps
    2757 Jps
    2685 HMaster

5，使用hbase shell

    root@master:/usr/local/hbase-2.0.0/bin# hbase shell
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    HBase Shell
    Use "help" to get list of supported commands.
    Use "exit" to quit this interactive shell.
    Version 2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018
    Took 0.0044 seconds
    hbase(main):001:0>

    hbase(main):003:0> version
    2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018
    Took 0.0054 seconds
    hbase(main):004:0>

啓動hbase thrift服務。

    root@master:/usr/local/hbase-2.0.0/bin# hbase-daemon.sh start thrift
    running thrift, logging to /usr/local/hbase-2.0.0/logs/hbase-root-thrift-master.out
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/local/hbase-2.0.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/local/alluxio-1.7.0-hadoop-2.6/client/alluxio-1.7.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    root@master:/usr/local/hbase-2.0.0/bin# jps
    3332 Jps
    3254 ThriftServer
    2685 HMaster
    root@master:/usr/local/hbase-2.0.0/bin#

二，使用Python鏈接Hbase。

1，創建Python開發環境 ---虛擬環境。

先進行檢查。

    root@master:/usr/local/hbase-2.0.0/bin# pip
    The program 'pip' is currently not installed. You can install it by typing:
    apt-get install python-pip
    root@master:/usr/local/hbase-2.0.0/bin# python
    Python 2.7.6 (default, Mar 22 2014, 22:59:56)
    [GCC 4.8.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> exit()
    root@master:/usr/local/hbase-2.0.0/bin#

安裝pip。

    root@master:/usr/local/hbase-2.0.0/bin# apt-get install python-pip
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      diffstat hardening-includes libapt-pkg-perl libarchive-zip-perl
      libautodie-perl libclass-accessor-perl libemail-valid-perl libio-pty-perl
      libio-string-perl libipc-run-perl libipc-system-simple-perl
      liblist-moreutils-perl libnet-dns-perl libnet-domain-tld-perl libnet-ip-perl
      libparse-debianchangelog-perl libperlio-gzip-perl libsub-identify-perl
      libsub-name-perl libtext-levenshtein-perl patchutils t1utils
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      python-chardet-whl python-colorama python-colorama-whl python-distlib
      python-distlib-whl python-html5lib python-html5lib-whl python-pip-whl
      python-pkg-resources python-requests-whl python-setuptools
      python-setuptools-whl python-six-whl python-urllib3-whl python-wheel
    Suggested packages:
      python-genshi python-distribute python-distribute-doc
    Recommended packages:
      python-dev-all
    The following NEW packages will be installed:
      python-chardet-whl python-colorama python-colorama-whl python-distlib
      python-distlib-whl python-html5lib python-html5lib-whl python-pip
      python-pip-whl python-requests-whl python-setuptools python-setuptools-whl
      python-six-whl python-urllib3-whl python-wheel
    The following packages will be upgraded:
      python-pkg-resources
    1 upgraded, 15 newly installed, 0 to remove and 637 not upgraded.
    Need to get 1,744 kB of archives.
    After this operation, 4,184 kB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]
    1% [2 python-chardet-whl 14.0 kB/170 kB 8%]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]
    Get:4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]
    2% [4 python-chardet-whl 38.1 kB/170 kB 22%]
    6% [4 python-chardet-whl 99.9 kB/170 kB 59%]
    Get:5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama all 0.2.5-0.1ubuntu2 [18.4 kB]
    Get:6 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama-whl all 0.2.5-0.1ubuntu2 [18.2 kB]
    Get:7 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib all 0.1.8-1ubuntu1 [113 kB]
    Get:8 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib-whl all 0.1.8-1ubuntu1 [140 kB]
    Get:9 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib all 0.999-3~ubuntu1 [83.5 kB]
    Get:10 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib-whl all 0.999-3~ubuntu1 [109 kB]
    Get:11 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-six-whl all 1.5.2-1ubuntu1 [10.5 kB]
    Get:12 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-urllib3-whl all 1.7.1-1ubuntu4 [64.0 kB]
    Get:13 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-requests-whl all 2.2.1-1ubuntu0.3 [227 kB]
    Get:14 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools-whl all 3.3-1ubuntu2 [244 kB]
    Get:15 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip-whl all 1.5.4-1ubuntu4 [111 kB]
    Get:16 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-pkg-resources all 3.3-1ubuntu2 [61.9 kB]
    Get:17 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools all 3.3-1ubuntu2 [230 kB]
    Get:18 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip all 1.5.4-1ubuntu4 [97.3 kB]
    Get:19 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python-wheel all 0.24.0-1~ubuntu1 [44.7 kB]
    Fetched 1,723 kB in 9min 30s (3,019 B/s)
    Selecting previously unselected package python-chardet-whl.
    (Reading database ... 156494 files and directories currently installed.)
    Preparing to unpack .../python-chardet-whl_2.2.1-2~ubuntu1_all.deb ...
    Unpacking python-chardet-whl (2.2.1-2~ubuntu1) ...
    Selecting previously unselected package python-colorama.
    Preparing to unpack .../python-colorama_0.2.5-0.1ubuntu2_all.deb ...
    Unpacking python-colorama (0.2.5-0.1ubuntu2) ...
    Selecting previously unselected package python-colorama-whl.
    Preparing to unpack .../python-colorama-whl_0.2.5-0.1ubuntu2_all.deb ...
    Unpacking python-colorama-whl (0.2.5-0.1ubuntu2) ...
    Selecting previously unselected package python-distlib.
    Preparing to unpack .../python-distlib_0.1.8-1ubuntu1_all.deb ...
    Unpacking python-distlib (0.1.8-1ubuntu1) ...
    Selecting previously unselected package python-distlib-whl.
    Preparing to unpack .../python-distlib-whl_0.1.8-1ubuntu1_all.deb ...
    Unpacking python-distlib-whl (0.1.8-1ubuntu1) ...
    Selecting previously unselected package python-html5lib.
    Preparing to unpack .../python-html5lib_0.999-3~ubuntu1_all.deb ...
    Unpacking python-html5lib (0.999-3~ubuntu1) ...
    Selecting previously unselected package python-html5lib-whl.
    Preparing to unpack .../python-html5lib-whl_0.999-3~ubuntu1_all.deb ...
    Unpacking python-html5lib-whl (0.999-3~ubuntu1) ...
    Selecting previously unselected package python-six-whl.
    Preparing to unpack .../python-six-whl_1.5.2-1ubuntu1_all.deb ...
    Unpacking python-six-whl (1.5.2-1ubuntu1) ...
    Selecting previously unselected package python-urllib3-whl.
    Preparing to unpack .../python-urllib3-whl_1.7.1-1ubuntu4_all.deb ...
    Unpacking python-urllib3-whl (1.7.1-1ubuntu4) ...
    Selecting previously unselected package python-requests-whl.
    Preparing to unpack .../python-requests-whl_2.2.1-1ubuntu0.3_all.deb ...
    Unpacking python-requests-whl (2.2.1-1ubuntu0.3) ...
    Selecting previously unselected package python-setuptools-whl.
    Preparing to unpack .../python-setuptools-whl_3.3-1ubuntu2_all.deb ...
    Unpacking python-setuptools-whl (3.3-1ubuntu2) ...
    Selecting previously unselected package python-pip-whl.
    Preparing to unpack .../python-pip-whl_1.5.4-1ubuntu4_all.deb ...
    Unpacking python-pip-whl (1.5.4-1ubuntu4) ...
    Preparing to unpack .../python-pkg-resources_3.3-1ubuntu2_all.deb ...
    Unpacking python-pkg-resources (3.3-1ubuntu2) over (3.3-1ubuntu1) ...
    Selecting previously unselected package python-setuptools.
    Preparing to unpack .../python-setuptools_3.3-1ubuntu2_all.deb ...
    Unpacking python-setuptools (3.3-1ubuntu2) ...
    Selecting previously unselected package python-pip.
    Preparing to unpack .../python-pip_1.5.4-1ubuntu4_all.deb ...
    Unpacking python-pip (1.5.4-1ubuntu4) ...
    Selecting previously unselected package python-wheel.
    Preparing to unpack .../python-wheel_0.24.0-1~ubuntu1_all.deb ...
    Unpacking python-wheel (0.24.0-1~ubuntu1) ...
    Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
    Setting up python-chardet-whl (2.2.1-2~ubuntu1) ...
    Setting up python-colorama (0.2.5-0.1ubuntu2) ...
    Setting up python-colorama-whl (0.2.5-0.1ubuntu2) ...
    Setting up python-distlib (0.1.8-1ubuntu1) ...
    Setting up python-distlib-whl (0.1.8-1ubuntu1) ...
    Setting up python-html5lib (0.999-3~ubuntu1) ...
    Setting up python-html5lib-whl (0.999-3~ubuntu1) ...
    Setting up python-six-whl (1.5.2-1ubuntu1) ...
    Setting up python-urllib3-whl (1.7.1-1ubuntu4) ...
    Setting up python-requests-whl (2.2.1-1ubuntu0.3) ...
    Setting up python-setuptools-whl (3.3-1ubuntu2) ...
    Setting up python-pip-whl (1.5.4-1ubuntu4) ...
    Setting up python-pkg-resources (3.3-1ubuntu2) ...
    Setting up python-setuptools (3.3-1ubuntu2) ...
    Setting up python-pip (1.5.4-1ubuntu4) ...
    Setting up python-wheel (0.24.0-1~ubuntu1) ...
    root@master:/usr/local/hbase-2.0.0/bin#

Pip安裝好以後，安裝virtualenv軟件，讓操做系統支持虛擬環境，能夠獲得命令virtualenv或者pyvenv命令。

    root@master:/usr/local/hbase-2.0.0/bin# pip install virtualenv
    Downloading/unpacking virtualenv
      Downloading virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB): 1.9MB downloaded
    Installing collected packages: virtualenv
    Successfully installed virtualenv
    Cleaning up...
    root@master:/usr/local/hbase-2.0.0/bin#

建立虛擬環境

    root@master:/usr/local# mkdir myPythonEnv
    root@master:/usr/local# cd myPythonEnv
    root@master:/usr/local/myPythonEnv# pwd
    /usr/local/myPythonEnv
    root@master:/usr/local/myPythonEnv# Pyvenv project-env
    Pyvenv: command not found
    root@master:/usr/local/myPythonEnv# virtualenv project-env
    New python executable in /usr/local/myPythonEnv/project-env/bin/python
    Installing setuptools, pip, wheel...done.
    root@master:/usr/local/myPythonEnv#

進入虛擬環境：

root@master:/usr/local/myPythonEnv/project-env/bin# source /usr/local/myPythonEnv/project-env/bin/activate
(project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

安裝Thrift依賴包

    (project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install thrift
    Collecting thrift
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      SNIMissingWarning
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecurePlatformWarning
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecurePlatformWarning
      Downloading https://files.pythonhosted.org/packages/c6/b4/510617906f8e0c5660e7d96fbc5585113f83ad547a3989b80297ac72a74c/thrift-0.11.0.tar.gz (52kB)
        100% |████████████████████████████████| 61kB 22kB/s
    Collecting six>=1.7.2 (from thrift)
      Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
    Building wheels for collected packages: thrift
      Running setup.py bdist_wheel for thrift ... done
      Stored in directory: /root/.cache/pip/wheels/be/36/81/0f93ba89a1cb7887c91937948519840a72c0ffdd57cac0ae8f
    Successfully built thrift
    Installing collected packages: six, thrift
    Successfully installed six-1.11.0 thrift-0.11.0

安裝hbase-thrift依賴包

    (project-env) root@master:/usr/local/myPythonEnv/project-env/bin# pip install hbase-thrift
    Collecting hbase-thrift
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      SNIMissingWarning
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecurePlatformWarning
    /usr/local/myPythonEnv/project-env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecurePlatformWarning
      Downloading https://files.pythonhosted.org/packages/89/f7/dbb6c764bb909ed361c255828701228d8c9867d541cfef84127e6f3704cc/hbase-thrift-0.20.4.tar.gz
    Requirement already satisfied: Thrift in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from hbase-thrift) (0.11.0)
    Requirement already satisfied: six>=1.7.2 in /usr/local/myPythonEnv/project-env/lib/python2.7/site-packages (from Thrift->hbase-thrift) (1.11.0)
    Building wheels for collected packages: hbase-thrift
      Running setup.py bdist_wheel for hbase-thrift ... done
      Stored in directory: /root/.cache/pip/wheels/fe/51/f2/afb7b010cd97910aa0b651d492735a38ed69a93a817444904e
    Successfully built hbase-thrift
    Installing collected packages: hbase-thrift
    Successfully installed hbase-thrift-0.20.4
    (project-env) root@master:/usr/local/myPythonEnv/project-env/bin#

Python鏈接HBASE的測試程序

    (project-env) root@master:/usr/local/noc_python_201806# vi queryTest.py

    from thrift import Thrift
    from thrift.transport import TSocket
    from thrift.transport import TTransport
    from thrift.protocol import TBinaryProtocol

    from hbase import Hbase
    from hbase.ttypes import *

    transport = TSocket.TSocket('localhost', 9090)

    transport = TTransport.TBufferedTransport(transport)
    protocol = TBinaryProtocol.TBinaryProtocol(transport)

    client = Hbase.Client(protocol)
    transport.open()
    print client.getTableNames()

運行結果爲，查詢出表：

    (project-env) root@master:/usr/local/noc_python_201806# python queryTest.py
    ['test', 'testHbase']
    (project-env) root@master:/usr/local/noc_python_201806#

Hbase中的表

hbase(main):025:0> list TABLE test testHbase 2 row(s) Took 0.0188 seconds => ["test", "testHbase"] hbase(main):026:0>