clickhouse單機部署及實時同步mysql數據

clickhouse單機部署及實時同步mysql數據

clickhouse20.8.3部分

示例版本：20.8.3

環境檢查

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
顯示：SSE 4.2 supported，則環境支持

下載安裝（單機模式）

安裝依賴

yum install -y curl
yum install -y libtool

添加Clickhouse 的yum鏡像

curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo bash

檢查鏡像狀況

yum list | grep clickhouse

安裝 server client

yum install -y clickhouse-server 
yum install -y clickhouse-client
安裝完後
vim /etc/clickhouse-server/users.xml
            <password>alexNB</password>
給default用戶設置密碼

啓動server
/etc/init.d/clickhouse-server start|restart|stop
啓動client CLI
clickhouse-client
clickhouse-client -m #能夠在命令窗口輸入多行命令
中止
systemctl stop clickhouse-server

外網訪問(可選)

vim /etc/clickhouse-server/config.xml
把 <listen_host>::</listen_host> 的註釋打開，這樣的話才能讓ClickHouse被除本機之外的服務器訪問
若是禁用了ipv6，使用下面配置
<listen_host>0.0.0.0</listen_host>

• 修改默認目錄
• 服務端的配置目錄：/etc/clickhouse-server
• 數據文件路徑：var/lib/clickhouse /var/lib/clickhouse/
• 日誌文件路徑： /var/log/clickhouse-server/ 如/var/log/clickhouse-server/clickhouse-server.log
• 存儲和日誌須要建議修改成非系統盤，其餘默認便可

卸載

查看安裝
yum list installed | grep clickhouse
clickhouse-client.noarch               21.3.3.14-2                     @repo.clickhouse.tech_rpm_stable_x86_64
clickhouse-common-static.x86_64        21.3.3.14-2                     @repo.clickhouse.tech_rpm_stable_x86_64
clickhouse-server.noarch               21.3.3.14-2                     @repo.clickhouse.tech_rpm_stable_x86_64
卸載並刪除目錄
yum remove -y clickhouse-common-static
yum remove -y clickhouse-server-commonrm -rf /var/lib/clickhouserm -rf /etc/clickhouse*rm -rf /var/log/clickhouse*

mysql_8.13

安裝完master以後,執行sql查看是否知足條件

mysql> show variables like '%gtid_mode%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| gtid_mode     | ON    |
+---------------+-------+
1 row in set (0.00 sec)
mysql> show variables like '%enforce_gtid_consistency%';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| enforce_gtid_consistency | ON    |
+--------------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like '%binlog_format%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW   |
+---------------+-------+
1 row in set (0.00 sec)

若不知足修改配置文件並重啓mysql

cat /etc/my.cnf

gtid_mode=ON
enforce_gtid_consistency=1
binlog_format=ROW

MySQL8建立用戶並受權的語句

mysql>create user brady@'%' identified  by 'brady';

受權示例

mysql>grant all privileges on *.* to brady@'%' with grant option;

刷新權限

mysql>flush privileges;

修改遠程鏈接用戶的加密規則

mysql>ALTER USER 'brady'@'%' IDENTIFIED WITH mysql_native_password BY 'brady';

數據同步

進入clickhouse數據庫

clickhouse-client -u default -h <IP> --password alexNB

開啓複製通道

select * from system.settings where name ='allow_experimental_database_materialize_mysql';

SET allow_experimental_database_materialize_mysql=1;

同步數據

CREATE DATABASE DBNAME ENGINE = MaterializeMySQL('MYSQLIP:PORT', 'SOURCEDB', 'MYSQL_DB', 'PASSWORD');

CREATE DATABASE data_dataTransfer_ch_update ENGINE = MaterializeMySQL('172.22.254.217:3306', 'data_dataTransfer_ch', 'brady', 'brady');

k8s-master :) show databases;
SHOW DATABASES
Query id: 7469cc9b-da21-44ad-9eb5-f32cddcadbe7
┌─name────────────────────────┐
│ audit                       │
│ data_dataTransfer_ch        │
│ data_dataTransfer_ch_update │
│ default                     │
│ system                      │
│ tstu                        │
└─────────────────────────────┘
6 rows in set. Elapsed: 0.002 sec. 

k8s-master :) use data_dataTransfer_ch_update;
USE data_dataTransfer_ch_update
Query id: 552cecae-df70-49e1-a4c4-8bae3e7f4f7d
Ok.
0 rows in set. Elapsed: 0.001 sec. 

k8s-master :) select task_id,data_val from data_dataTransfer_ch_update.t_task_check_field_data where task_id = '796D82C8-CB64-41AC-84EE-9D292880FEBC';

SELECT
    task_id,
    data_val
FROM data_dataTransfer_ch_update.t_task_check_field_data
WHERE task_id = '796D82C8-CB64-41AC-84EE-9D292880FEBC'

Query id: 676b491b-df7a-4010-84ed-acb8dec5a393

┌─task_id──────────────────────────────┬─data_val──────────────────────────────────────────────────────────────┐
│ 796D82C8-CB64-41AC-84EE-9D292880FEBC │ http://110.110.110.110:8085/vul/xss/xssblind/admin_login.php            │
│ 796D82C8-CB64-41AC-84EE-9D292880FEBC │ http://110.110.110.110:8085/vul/xss/xssblind/admin_login.php            │
│ 796D82C8-CB64-41AC-84EE-9D292880FEBC │ http://110.110.110.110:8085//vul/xss/xsspost/post_login.php             │
│ 796D82C8-CB64-41AC-84EE-9D292880FEBC │ http://110.110.110.110:8085/vul/sqli/sqli_iu/sqli_login.php             │
│ 796D82C8-CB64-41AC-84EE-9D292880FEBC │ http://110.110.110.110:8085/vul/infoleak/findabc.php                    │

至此實時同步結束
能夠在mysql對數據表進行增刪改操做，在clickhouse能夠看到效果

clickhouse版本更新迭代較快，老版本bug較多不穩定。
20.8這個版本bug也挺多，若是有遇到能夠參考下篇文章
大佬移步這裏
或者這裏