pg-xl 的基本方式安裝與使用

os: centos 7.4
pgxl:pg.version ‘10.3 (Postgres-XL 10alpha2)node

pgxl 是一款很是實用的橫向擴展的開源軟件，繼承了不少pgxc的功能，在replication 和sharding 方面有着很是棒的用處。
pgxl 不嚴格的說是 pgxc的升級增強版。是對官方 postgresql 的版本的修改提高，爲大牛點贊。python

Global Transaction Monitor (GTM)
全局事務管理器，確保羣集範圍內的事務一致性。 GTM負責發放事務ID和快照做爲其多版本併發控制的一部分。
集羣可選地配置一個備用GTM，以改進可用性。此外，能夠在協調器間配置代理GTM，可用於改善可擴展性，減小GTM的通訊量。linux

GTM Standby
GTM的備節點，在pgxc,pgxl中，GTM控制全部的全局事務分配，若是出現問題，就會致使整個集羣不可用，爲了增長可用性，增長該備用節點。當GTM出現問題時，GTM Standby能夠升級爲GTM，保證集羣正常工做。c++

GTM-Proxy
GTM須要與全部的Coordinators通訊重點內容，爲了下降壓力，能夠在每一個Coordinator機器上部署一個GTM-Proxy。git

Coordinator
協調員管理用戶會話，並與GTM和數據節點進行交互。協調員解析，並計劃查詢，並給語句中的每個組件發送下一個序列化的全局性計劃。
爲節省機器，一般此服務和數據節點部署在一塊兒。web

Data Node
數據節點是數據實際存儲的地方。數據的分佈能夠由DBA來配置。爲了提升可用性，能夠配置數據節點的熱備以便進行故障轉移準備。sql

總結：
gtm是負責ACID的，保證分佈式數據庫全局事務一致性。得益於此，就算數據節點是分佈的，可是你在主節點操做增刪改查事務時，就如同只操做一個數據庫同樣簡單。
Coordinator是調度的，將操做指令發送到各個數據節點。
datanodes是數據節點，分佈式存儲數據。數據庫

規劃以下：
node1 192.168.56.101 gtm,gtm-proxy,coordinator1,coordinator2vim

node2 192.168.56.102 datanode
node3 192.168.56.103 datanodecentos

在網上了解是都是把 coordinator 和 datanode放在一塊兒，本次實驗沒有放在一塊兒，運行結果徹底ok。

下載

https://www.postgres-xl.org/download/

https://git.postgresql.org/gitweb/?p=postgres-xl.git;a=summary
git://git.postgresql.org/git/postgres-xl.git

安裝

node1 須要安裝依賴包

# yum install -y bison flex perl-ExtUtils-Embed readline-devel zlib-devel pam-devel libxml2-devel libxslt-devel openldap-devel python-devel gcc gcc-c++ openssl-devel cmake openjade docbook-style-dsssl uuid uuid-devel

關閉防火牆，selinux，其他linux性能設置請參考postgresql的設置

# systemctl stop firewalld.service 
# systemctl disable firewalld.service 
# vim /etc/selinux/config
disabled

node1 節點上建立用戶

# groupadd postgres
# useradd postgres -g postgres 
# passwd postgres

# mkdir -p /usr/pgxl-10
# chown -R postgres:postgres /usr/pgxl-10

# mkdir -p /var/lib/pgxl
# mkdir -p /var/lib/pgxl/coordinator1
# mkdir -p /var/lib/pgxl/coordinator2
# mkdir -p /var/lib/pgxl/gtm
# mkdir -p /var/lib/pgxl/gtm_standby
# mkdir -p /var/lib/pgxl/gtm_proxy1
# mkdir -p /var/lib/pgxl/gtm_proxy2
# chown -R postgres:postgres /var/lib/pgxl

node1 節點 postgres 用戶的環境變量

# su - postgres
$ vi ~/.bash_profile
export PGUSER=postgres
export PGHOME=/usr/pgxl-10

export LD_LIBRARY_PATH=$PGHOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export PATH=$PGHOME/bin:$PATH

export TEMP=/tmp
export TMPDIR=/tmp

node1 上編譯安裝,注意不須要安裝社區版

$ cd /tmp
$ git clone git://git.postgresql.org/git/postgres-xl.git
$ cd postgres-xl
$ git branch -r
  origin/HEAD -> origin/master
  origin/XL9_5_STABLE
  origin/XL_10_STABLE
  origin/master
  origin/xl_dbt3_expt
  origin/xl_doc_update
  origin/xl_test
$ git checkout XL_10_STABLE
Branch XL_10_STABLE set up to track remote branch XL_10_STABLE from origin.
Switched to a new branch 'XL_10_STABLE'
$ git status
# On branch XL_10_STABLE
nothing to commit, working directory clean  
$ ./configure --prefix=/usr/pgxl-10 --with-perl --with-python --with-openssl --with-pam --with-ldap --with-libxml --with-libxslt
$ make world
$ make install-world

$ cd contrib
$ make 
$ make install

node一、node二、node3配置ssh相互免密登陸

過程略

node一、node二、node3同步下時間

# ntpdate asia.pool.ntp.org

node1節點上 gtm

$ which initgtm
/usr/pgxl-10/bin/initgtm

初始化 gtm

$ initgtm -Z gtm -D /var/lib/pgxl/gtm

The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.


fixing permissions on existing directory /var/lib/pgxl/gtm ... ok
creating configuration files ... ok
creating control file ... ok

Success.
You can now start the GTM server using:

    gtm -D /var/lib/pgxl/gtm
or
    gtm_ctl -Z gtm -D /var/lib/pgxl/gtm -l logfile start

gtm.conf 的詳細內容

$ vi /var/lib/pgxl/gtm/gtm.conf

nodename = 'node1_gtm'
listen_addresses = '*'
port = 6666
startup = ACT
log_file = 'gtm.log'
log_min_messages = NOTICE

node1節點上 gtm_proxy

因爲node1上計劃安裝兩個 coordinator，因此也須要安裝兩個 gtm_proxy

$ which initgtm
/usr/pgxl-10/bin/initgtm

初始化第一個 gtm_proxy

$ initgtm -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy1
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.


fixing permissions on existing directory /var/lib/pgxl/gtm_proxy1 ... ok
creating configuration files ... ok

Success.
You can now start the GTM proxy server using:

    gtm_proxy -D /var/lib/pgxl/gtm_proxy1
or
    gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy1 -l logfile start

第一個 gtm_proxy 參數文件

$ vi /var/lib/pgxl/gtm_proxy1/gtm_proxy.conf

nodename = 'node1_gtm_proxy1'
listen_addresses = '*'
port = 6668
gtm_host = 'node1'
gtm_port = 6666
log_file = 'gtm_proxy.log'
log_min_messages = NOTICE

初始化第二個 gtm_proxy

$ initgtm -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy2
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.


fixing permissions on existing directory /var/lib/pgxl/gtm_proxy2 ... ok
creating configuration files ... ok

Success.
You can now start the GTM proxy server using:

    gtm_proxy -D /var/lib/pgxl/gtm_proxy2
or
    gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy2 -l logfile start

第二個 gtm_proxy 參數文件

$ vi /var/lib/pgxl/gtm_proxy2/gtm_proxy.conf

nodename = 'node1_gtm_proxy2'
listen_addresses = '*'
port = 6669
gtm_host = 'node1'
gtm_port = 6666
log_file = 'gtm_proxy.log'
log_min_messages = NOTICE

node1節點上 coordinator

兩個 coordinator

初始化第一個 coordinator

$ initdb -D /var/lib/pgxl/coordinator1 --nodename coordinator1 -E UTF8 --locale=C -U postgres -W

Success.

Success. You can now start the database server of the Postgres-XL coordinator using:

    pg_ctl -D /var/lib/pgxl/coordinator1 -l logfile start -Z coordinator

or
 You can now start the database server of the Postgres-XL datanode using:

    pg_ctl -D /var/lib/pgxl/coordinator1 -l logfile start -Z datanode

第一個 coordinator 參數文件

$ vi /var/lib/pgxl/coordinator1/pg_hba.conf

host    all             all             192.168.56.101/32         trust
host    all             all             192.168.56.102/32         trust
host    all             all             192.168.56.103/32         trust

$ vi /var/lib/pgxl/coordinator1/postgresql.conf

listen_addresses = '*'
port = 5432
max_connections = 100

pooler_port = 6670
max_pool_size = 100

gtm_host = 'node1'
gtm_port = 6668
pgxc_node_name = 'coordinator1'

初始化第二個 coordinator

$ initdb -D /var/lib/pgxl/coordinator2 --nodename coordinator2 -E UTF8 --locale=C -U postgres -W

Success.

Success. You can now start the database server of the Postgres-XL coordinator using:

    pg_ctl -D /var/lib/pgxl/coordinator2 -l logfile start -Z coordinator

or
 You can now start the database server of the Postgres-XL datanode using:

    pg_ctl -D /var/lib/pgxl/coordinator2 -l logfile start -Z datanode

第二個 coordinator 參數文件

$ vi /var/lib/pgxl/coordinator2/pg_hba.conf

host    all             all             192.168.56.101/32         trust
host    all             all             192.168.56.102/32         trust
host    all             all             192.168.56.103/32         trust

$ vi /var/lib/pgxl/coordinator2/postgresql.conf

listen_addresses = '*'
port = 5433
max_connections = 100

pooler_port = 6671
max_pool_size = 100

gtm_host = 'node1'
gtm_port = 6669
pgxc_node_name = 'coordinator2'

node2節點上 datanode

安裝 pgxl，可參考node1節點上的安裝過程

$ initdb -D /var/lib/pgxl/data --nodename datanode1 -E UTF8 --locale=C -U postgres -W
Success. You can now start the database server of the Postgres-XL coordinator using:

    pg_ctl -D /var/lib/pgxl/data -l logfile start -Z coordinator

or
 You can now start the database server of the Postgres-XL datanode using:

    pg_ctl -D /var/lib/pgxl/data -l logfile start -Z datanode

$ vi /var/lib/pgxl/data/pg_hba.conf

host    all             all             192.168.56.101/32         trust
host    all             all             192.168.56.102/32         trust
host    all             all             192.168.56.103/32         trust


$ vi /var/lib/pgxl/data/postgresql.conf

listen_addresses = '*'
port = 5432
max_connections = 100

pooler_port = 6667
max_pool_size = 100

gtm_host = 'node1'
gtm_port = 6668
pgxc_node_name = 'datanode1'

node3節點上 datanode

安裝 pgxl，可參考node1節點上的安裝過程

$ initdb -D /var/lib/pgxl/data --nodename datanode2 -E UTF8 --locale=C -U postgres -W
Success. You can now start the database server of the Postgres-XL coordinator using:

    pg_ctl -D /var/lib/pgxl/data -l logfile start -Z coordinator

or
 You can now start the database server of the Postgres-XL datanode using:

    pg_ctl -D /var/lib/pgxl/data -l logfile start -Z datanode

$ vi /var/lib/pgxl/data/pg_hba.conf

host    all             all             192.168.56.101/32         trust
host    all             all             192.168.56.102/32         trust
host    all             all             192.168.56.103/32         trust

$ vi /var/lib/pgxl/data/postgresql.conf    

listen_addresses = '*'
port = 5432
max_connections = 100

pooler_port = 6667
max_pool_size = 100

gtm_host = 'node1'
gtm_port = 6669
pgxc_node_name = 'datanode2'

pg-xl集羣啓動

啓動順序是 GTM=>GTM Standby=>GTM-Proxy=>Datanodes=>Coordinators

node1 節點啓動 gtm

$ gtm_ctl -Z gtm -D /var/lib/pgxl/gtm start

對應的關閉

$ gtm_ctl -Z gtm -D /var/lib/pgxl/gtm stop

node1 節點啓動 gtm-proxy

$ gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy1 start
$ gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy2 start

對應的關閉

$ gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy1 stop
$ gtm_ctl -Z gtm_proxy -D /var/lib/pgxl/gtm_proxy2 stop

node2 節點啓動 datanode

$ pg_ctl -D /var/lib/pgxl/data start -Z datanode

對應的關閉

$ pg_ctl stop -m fast -D /var/lib/pgxl/data -Z datanode

node3 節點啓動 datanode

$ pg_ctl -D /var/lib/pgxl/data start -Z datanode

對應的關閉

$ pg_ctl stop -m fast -D /var/lib/pgxl/data -Z datanode

node1 節點啓動 coordinator

$ pg_ctl -D /var/lib/pgxl/coordinator1 start -Z coordinator
$ pg_ctl -D /var/lib/pgxl/coordinator2 start -Z coordinator

對應的關閉

$ pg_ctl stop -m fast -D /var/lib/pgxl/coordinator1 -Z coordinator
$ pg_ctl stop -m fast -D /var/lib/pgxl/coordinator2 -Z coordinator

pg-xl配置

node1 節點上登陸 coordinator1

$ psql -p 5432
psql (PGXL 10alpha2, based on PG 10.3 (Postgres-XL 10alpha2))
Type "help" for help.

postgres=# 
postgres=# select * from pgxc_node;
 node_name    | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |  node_id   
--------------+-----------+-----------+-----------+----------------+------------------+------------
 coordinator1 | C         |      5432 | localhost | f              | f                | 1148549230
(1 row)

postgres=# alter node coordinator1 with (type=coordinator,host='node1', port=5432);
postgres=# create node coordinator2 with (type=coordinator,host='node1', port=5433);

postgres=# create node datanode1 with (type=datanode, host='node2',port=5432,primary,preferred);
postgres=# create node datanode2 with (type=datanode, host='node3',port=5432);

postgres=# select pgxc_pool_reload();
postgres=# select * from pgxc_node;
  node_name   | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id   
--------------+-----------+-----------+-----------+----------------+------------------+-------------
 coordinator1 | C         |      5432 | node1     | f              | f                |  1148549230
 coordinator2 | C         |      5433 | node1     | f              | f                | -2089598990
 datanode1    | D         |      5432 | node2     | t              | t                |   888802358
 datanode2    | D         |      5432 | node3     | f              | f                |  -905831925
(4 rows)

依次操做 node1 節點的 coordinator2，node2 節點的 datanode，node3 節點的datanode

驗證

只能經過node1 節點的 coordinator一、coordinator1 去操做。node2，node3節點的數據都是隻讀的。

目前的coordinator配置以下，也能夠往前翻看看記錄。
node1 coordinator1 5432
node1 coordinator2 5433

$ psql -p 5432
psql (PGXL 10alpha2, based on PG 10.3 (Postgres-XL 10alpha2))
Type "help" for help.

postgres=#
postgres=# create table tmp_t0(c0 varchar(100),c1 varchar(100));
peiybdb=# insert into tmp_t0(c0,c1) SELECT id::varchar,md5(id::varchar) FROM generate_series(1,10000) as id;
INSERT 0 10000 peiybdb=# \d+ tmp_t0 Table "public.tmp_t0" Column | Type | Collation | Nullable | Default | Storage | Stats target | Description --------+------------------------+-----------+----------+---------+----------+--------------+------------- c0 | character varying(100) | | | | extended | | c1 | character varying(100) | | | | extended | | Distribute By: HASH(c0) Location Nodes: ALL DATANODES

到 node二、node3的數據節點查看對應的數據

node2 查詢結果

peiybdb=# select count(1) from tmp_t0;
 count 
-------
  5081
(1 row)

node3 查詢結果

peiybdb=# select count(1) from tmp_t0;
 count 
-------
  4919
(1 row)

若是在node2，node3上插入數據，會收到報錯提示。

peiybdb=# insert into tmp_t0(c0,c1) SELECT id::varchar,md5(id::varchar) FROM generate_series(1,10000) as id;
ERROR:  cannot execute INSERT in a read-only transaction

下面是簡單的建表分析

CREATE TABLE table_name(...)
DISTRIBUTE BY 
HASH(col)|MODULO(col)|ROUNDROBIN|REPLICATION
TO NODE(nodename1,nodename2...)

能夠看到，若是DISTRIBUTE BY 後面有以下選項：
REPLICATION，則是複製模式，其他則是分片模式，
HASH 指的是按照指定列的哈希值分佈數據，
MODULO 指的是按照指定列的取摩運算分佈數據，
ROUNDROBIN 指的是按照輪詢的方式分佈數據

TO NODE指定了數據分佈的節點範圍，若是沒有指定則默認全部數據節點參與數據分佈。若是沒有指定分佈模式，即便用普通的CREATE TABLE語句，PGXL會默認採用分片模式將數據分佈到全部數據節點。

參考：
https://www.postgres-xl.org/
https://www.postgres-xl.org/overview/
https://www.postgres-xl.org/download/
https://www.postgres-xl.org/documentation/

https://git.postgresql.org/gitweb/?p=postgres-xl.git;a=summary

https://www.2ndquadrant.com/en/resources/postgres-xl/

https://www.postgresql.org/download