DW(Columnstore)

mysql/mariadb<oltp> as olap:
      缺乏hash join和parallel
      複雜的SQL查詢,只會有一個CPU正在滿負荷地運轉
      方案選型了之後，只能作更多的優化
        儘可能使用主鍵查詢
        選擇索引的字段，必定要符合差別率高的字段
        在group by後面使用order by null，禁用默認的group by排序
        手動拆分SQL，再合併結果集
        my.cnf配置優化，增長臨時表空間和key_buffer_size

mariadb columnstore (InfiniDB olap store engine)
    大數據離線分析
    大規模分佈式並行（MPP架構）
    高性能
    壓縮
    列式
  (對比非開源mysql數據倉庫系統Infobright)

columnstore:三層架構
      用戶模塊 User Module：Execution Manager／DMLProc／DDLProc
      性能模塊 Performance Module：n*Performance Module
      數據存儲層 Data Storage: 本地磁盤(local disks)/共享存儲(SAN)/分佈式存儲(EBS/HDFS/GlusterFS)

columnstore docker install(基於centos的docker):
      git clone https://github.com/mariadb-corporation/mariadb-columnstore-docker.git
      cd mariadb-columnstore-docker
      docker build -t david:mcs .
      docker run -d -p 4306:3306 -v /home/david/dev/dbt3:/mnt/dbt3 --name mcs david:mcs

      execute mcsadmin or mcsimport:docker exec -it mcs bash
      mcsadmin getsystemstatus
      ...

      docker stop mcs
      docker start mcs
      docker rm -v mcs
      docker rmi david:mcs

columnstore vagrant install(基於centos的vm):
　　　 ＃安裝virtualbox
      note:disable SecureBoot when boot linux
      sudo apt-get install linux-headers-`uname -r`
      sudo apt-get install virtualbox
      sudo apt-get install dkms
      sudo dpkg-reconfigure virtualbox-dkms
      sudo dpkg-reconfigure virtualbox
      (sudo modprobe vboxdrv)
      (sudo /etc/init.d/vboxdrv setup)
      (sudo cp vboxdrv.ko /lib/modules/)
      (make sure SecureBoot is disabled:sudo mokutil --sb-state)

      vagrant plugin install vagrant-libvirt
      vagrant plugin install vagrant-vbguest

      ＃下載vagrant配置
      git clone https://github.com/mariadb-corporation/mariadb-columnstore-vagrant.git
      cd mariadb-columnstore-vagrant
      vagrant status
      download mariadb-cs-centos-7 and unzip to columnstore

      #啓動虛擬機
      (box urls:http://www.vagrantbox.es/)
      vagrant box list

 vagrant box add centos/7 https://github.com/holms/vagrant-centos7-box/releases/download/7.1.1503.001/CentOS-7.1.1503-x86_64-netboot.box --provider=virtualbox
      vagrant up --provider=virtualbox
      (vagrant halt)
      (vagrant suspend)
      (vagrant resume)
      (vagrant reload)
      (vagrant ssh)
      (vagrant destroy)

      #鏈接mcs
      vagrant ssh
      sudo su -
      mcsadmin
 

columnstore:os準備和dw安裝（ubuntu14不支持,未驗證）
      vi /etc/hosts
      公私鑰認證，打通SSH無密碼
      關閉IPTABLES防火牆/關閉SELINUX:vi /etc/selinux/config
      關閉文件系統訪問時間和更改磁盤IO算法:vi /etc/fstab; echo "deadline" >   /sys/block/sdb/queue/scheduler
      優化Linux系統內核: vi /etc/sysctl.conf
      調整文件描述符ulimit爲65535:vi /etc/security/limits.conf;
      關閉NUMA:vi /etc/grub.conf
      安裝jemalloc內存管理器(相似malloc,tcmalloc):jemalloc*
      重啓服務器:reboot

      安裝boost軟件包:
        wget   http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz
        tar zxvf boost_1_55_0.tar.gz
        cd boost_1_55_0
        ./bootstrap.sh   --with-libraries=atomic,date_time,exception,filesystem,iostreams,locale,program_options,regex,signals,system,test,thread,timer,log   --prefix=/usr
        ./b2 install

      安裝Perl依賴包:
        expect perl perl-DBI openssl zlib perl-DBD-MySQL

      安裝mariadb:
        cd /root/
        wget　https://downloads.mariadb.com/enterprise/dapw-ktc5/mariadb-columnstore/1.0.6/centos/x86_64/6/mariadb-columnstore-1.0.6-1-centos6.x86_64.bin.tar.gz
        tar zxvf   mariadb-columnstore-1.0.6-1-centos6.x86_64.bin.tar.gz -C /usr/local/
        /usr/local/mariadb/columnstore/bin/postConfigure

      Report Tool:
       /usr/local/mariadb/columnstore/bin/columnstoreSupport -a
       /usr/local/mariadb/columnstore/bin/columnstoreSupport -a -p 'password'

      Troubleshooting工具：
      　　/usr/local/mariadb/columnstore/bin/editem
          /usr/local/mariadb/columnstore/bin/clearshm
          /usr/local/mariadb/columnstore/bin/configxml.sh
          /usr/local/mariadb/columnstore/bin/dbrmctl
          /usr/local/mariadb/columnstore/bin/viewtablelock
          /usr/local/mariadb/columnstore/bin/cleartablelock

      管理mariadb：
        mcsadmin getSystemStatus
        (cat /root/.bashrc
            alias mcsmysql='/usr/local/mariadb/columnstore/mysql/bin/mysql   --defaults-file=/usr/local/mariadb/columnstore/mysql/my.cnf -u root'
            alias ma=/usr/local/mariadb/columnstore/bin/mcsadmin
            alias mcsadmin=/usr/local/mariadb/columnstore/bin/mcsadmin
            alias home='cd /usr/local/mariadb/columnstore'
            alias log='cd /var/log/mariadb/columnstore/'
            alias core='cd /var/log/mariadb/columnstore/corefiles'
            alias tmsg='tail -f /var/log/messages'
            alias tdebug='tail -f   /var/log/mariadb/columnstore/debug.log'
            alias tinfo='tail -f   /var/log/mariadb/columnstore/info.log'
            alias dbrm='cd   /usr/local/mariadb/columnstore/data1/systemFiles/dbrm'
            alias module='cat   /usr/local/mariadb/columnstore/local/module'
        or
        /usr/local/mariadb/columnstore/bin/columnstoreAlias
        )

        mcsadmin assignPmDBRootConfig 3 pm

        mcsadmin addModule pm2 host1 mypwd
        mcsadmin alterSystem-enableModule pm2
        mcsadmin adddbroot 1
        mcsadmin assigndbrootpmconfig 2 pm2
        mcsadmin startSystem

        mcsadmin addModule um2 host1
        mcsadmin alterSystem-enableModule um2

        mcsadmin removeModule pm2
        mcsadmin removeModule um2

        mcsadmin stopSystem
        mcsadmin startSystem
        mcsadmin restartSystem
        mcsadmin shutdownSystem

        mcsadmin alterSystem-disablemodule pm2,pm3
        mcsadmin alterSystem-enablemodule pm2,pm3
        switchParentOAMModule pm3

        mcsadmin getActiveSQLStatements
        mcsadmin getModuleConfig
        mcsadmin getSoftwareInfo
        mcsadmin getStorageConfig
        mcsadmin getSystemInfo
        mcsadmin getSystemNetworkConfig
        mcsadmin getSystemResourceUsage
        mcsadmin getProcessStatus
        setSystemConfig ExternalMajorThreshold 80

    操做數據庫：
      mcsmysql -p'password' db_name

      數據導入mariadb：
        從其餘mysql導出：
          select * from sbtest into outfile '/tmp/sbtest.txt'   FIELDS TERMINATED BY ','
          OPTIONALLY ENCLOSED BY '"'  LINES TERMINATED BY '\n';
          #FIELDS TERMINATED BY ',' --字段的結束符
          #OPTIONALLY ENCLOSED BY '"' --字符串的分割符
          #LINES TERMINATED BY '\n' --行的結束符
        導入mcs:
          load data infile '/tmp/sbtest.txt' into table sbtest   FIELDS TERMINATED BY ','  OPTIONALLY   ENCLOSED BY '"'  LINES TERMINATED   BY '\n';

          /usr/local/mariadb/columnstore/bin/cpimport  test    sbtest    /root/tmp/sbtest.txt  -E   '"'  -s ','
          #test是數據庫
          #sbtest是表
          #-E是字符串的分割符
          #-s是字段的結束符

      mariadb數據分析：
          mcsmysql test

      mariadb參數調優：
        /usr/local/mariadb/columnstore/etc/Columnstore.xml
        (columnstore不支持主鍵、索引、timestamp、collate用法、char\varchar類型的sum/average用法)

      mariadb基準測試：
        InfiniDB比InnoDB建錶速度慢，語法定製。
        InfiniDB比InnoDB增長列快100倍
        InfiniDB比InnoDB百列字段千萬級數據插入快4倍
        InfiniDB比InnoDB百列字段千萬級空間少20%
        InfiniDB比InnoDB百列字段千萬級查詢快100倍
        InfiniDB比InnoDB億級插入快3倍
        InfiniDB比InnoDB億級數據空間少50%
        InfiniDB比InnoDB億級數據查詢快150倍
        基於成熟MySQL協議，Tableau能夠使用

      mariadb日誌：
        /var/log/mariadb/columnstore／*.log

      mariadb擴從節點：
        未知

      其餘暫時未知