大數據之superset

一、概述

superset大數據可視化的利器，深度集成durid，結合kylin、presto完成強大的大數據可視化功能，曾用名Panoramix、caravel。相比caravel它有個比較搶眼的功能SQL  lab。具體可參考官方文檔

 

二、安裝

提早在10.0.2.245服務器上面部署好redis。參考個人另外一文章：http://www.cnblogs.com/cuishuai/p/8033672.html

使用docker進行安裝，首先要先安裝docker，採用的是centos7直接使用yum安裝便可。docker安裝完成後還須要安裝docker-compose

#yum -y install docker docker-compose

#yum -y install git

找到最新的superset的docker：https://github.com/amancevice/superset，git clone到服務器上。

#cd /data

#git clone https://github.com/amancevice/superset

修改docker-compose.yml文件

#cat docker-compose.yml

version: '2'
services:
    image: amancevice/superset
    container_name: superset
    volumes:
      - /data/superset/conf/superset_config.py:/etc/superset/superset_config.py
    ports:
      - 8088:8088

 

注：必需要作hosts映射，由於要使用hive或presto填寫地址必須使用主機名不能使用ip地址，因爲pyhive0.5不支持ip，新版本已修復。utils.py很重要，主要是爲了消除sql lab的timeout，

文件路徑：/usr/local/lib/python3.5/dist-packages/superset/utils.py

將上述的signal註釋掉，新加兩個pass。這種方法把控制超時發送信號的代碼注掉了，這樣查詢超過30s的時候就不會把進程kill掉。

默認docker是沒有安裝vim的修改很不方便，能夠按以下步驟安裝vim：

1）首先cd到docker-compose.yml的目錄下啓動容器：docker-compose  up -d 

2)進入容器：docker exec -it -u 0  superset  /bin/bash

3)修改debian的源：

#cd /etc/apt/

# cat >>sources.list<< EOF

deb http://mirrors.aliyun.com/debian wheezy main contrib non-free
deb-src http://mirrors.aliyun.com/debian wheezy main contrib non-free
deb http://mirrors.aliyun.com/debian wheezy-updates main contrib non-free
deb-src http://mirrors.aliyun.com/debian wheezy-updates main contrib non-free
deb http://mirrors.aliyun.com/debian-security wheezy/updates main contrib non-free
deb-src http://mirrors.aliyun.com/debian-security wheezy/updates main contrib non-free

EOF

#apt-get update && apt-get install vim

 

#vim  /usr/local/lib/python3.5/dist-packages/superset/utils.py

def __enter__(self):
try:
#signal.signal(signal.SIGALRM, self.handle_timeout)
#signal.alarm(self.seconds)
pass
except ValueError as e:
logging.warning("timeout can't be used in the current context")
logging.exception(e)

def __exit__(self, type, value, traceback):
try:
#signal.alarm(0)
pass

添加如上兩個註釋，並使用pass代替。

配置文件：

#cat superset_config.py

#---------------------------------------------------------
# Superset specific config
#---------------------------------------------------------
ROW_LIMIT = 5000
SUPERSET_WORKERS = 4
SUPERSET_WEBSERVER_TIMEOUT = 3000
SUPERSET_WEBSERVER_PORT = 8088
#---------------------------------------------------------

#---------------------------------------------------------
# Flask App Builder configuration
#---------------------------------------------------------
# Your App secret key
SECRET_KEY = '\2\1thisismyscretkey\1\2\e\y\y\h'

# The SQLAlchemy connection string to your database backend
# This connection defines the path to the database that stores your
# superset metadata (slices, connections, tables, dashboards, ...).
# Note that the connection information to connect to the datasources
# you want to explore are managed directly in the web UI
#SQLALCHEMY_DATABASE_URI = 'sqlite:////data/superset.db'
SQLALCHEMY_DATABASE_URI = 'sqlite:////home/superset/superset.db'

# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
# Add endpoints that need to be exempt from CSRF protection
WTF_CSRF_EXEMPT_LIST = []

# Set this API key to enable Mapbox visualizations
MAPBOX_API_KEY = ''

CACHE_DEFAULT_TIMEOUT = 60*60*6
CACHE_CONFIG = {
'CACHE_TYPE': 'redis',
'CACHE_REDIS_HOST': '10.0.2.245',
'CACHE_REDIS_PORT': '6379',
'CACHE_REDIS_URL': 'redis://10.0.2.245:6379'
}

class CeleryConfig(object):
    BROKER_URL = 'redis://10.0.2.245:6379/0'
    CELERY_IMPORTS = ('superset.sql_lab',)
    CELERY_RESULT_BACKEND = 'redis://10.0.2.245:6379/0'
   # CELERY_ANNOTATIONS = {'tasks.add':{'rate_limit':'10/s'}}

CELERY_CONFIG = CeleryConfig

from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(
    host='10.0.2.245', port=6379, key_prefix='superset_results')

進入到存放docker-compose.yml的目錄下面：

#docker-compose up -d 

#docker exec -it superset  superset db upgrade

#docker exec -it superset  superset  load_examples

#docker exec -it superset superset-init

啓動superset worker

#docker exec -it superset /bin/bash

$nohup  superset worker &

 

默認是superset用戶，若是想獲取root用戶權限

#docker exec -it  -u 0 superset /bin/bash

 

三、打開瀏覽器訪問

宿主機ip:port

http://192.168.1.100:8088

輸入上面初始化設置的用戶名密碼

 

四、鏈接presto,spark是10.0.2.245，因爲以前的老版本不支持ip，如今新版本已經修改。能夠直接是用ip