pgloader 學習（四）一些簡單操做例子

上邊已經說明了pgloader 的基本使用（篇理論），可是對於實際操做偏少，如下是一個簡單的操做

不像官方文檔那樣，我爲了方便，直接使用docker-compose 運行，同時這個環境，會在後邊大部分場景使用,同時
對於pgloader 的dsl暫時不會仔細說明，後邊會有介紹

環境準備

• docker-compose 文件

 

version: "3"

services:

  pgloader-csv:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader 

  pgloader-sqlite:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader 

  pgloader-pg:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader 

  pgloader-mysql:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader 

  mysql:

    image: mysql:5.7.16

    ports:

      - 3306:3306

    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci

    environment:

      MYSQL_ROOT_PASSWORD: dalongrong

      MYSQL_DATABASE: gogs

      MYSQL_USER: gogs

      MYSQL_PASSWORD: dalongrong

      TZ: Asia/Shanghai

  postgres:

    image: postgres:9.6.11

    ports:

    - "5432:5432"

    environment:

    - "POSTGRES_PASSWORD:dalong"

  postgres2:

    image: postgres:9.6.11

    ports:

    - "5433:5432"

    environment:

    - "POSTGRES_PASSWORD:dalong"

• 說明
  項目代碼我已經提交github了，後續哥更新都會在裏邊，地址 pgloader-learning

加載csv 文件

參考命令行參數

 

LOAD CSV

     FROM '/opt/db/csv/app.csv' (x, y, a, b, c, d)

     INTO postgresql://postgres:dalong@postgres:5432/postgres?csv (a, b, d, c)

     WITH truncate,

          skip header = 1,

          fields optionally enclosed by '"',

          fields escaped by double-quote,

          fields terminated by ','

​

      SET client_encoding to 'latin1',

          work_mem to '12MB',

          standard_conforming_strings to 'on'

​

   BEFORE LOAD DO

    $$ drop table if exists csv; $$,

    $$ create table csv (

        a bigint,

        b bigint,

        c char(2),

        d text

       );

  $$;

docker-compose csv service 修改

  pgloader-csv:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader /loader/csv/csv.load

執行

docker-compose up pgloader-csv

執行效果

pgloader-csv_1 | WARNING:

pgloader-csv_1 | Couldn't re-execute SBCL with proper personality flags (/proc isn't mounted? setuid?)

pgloader-csv_1 | Trying to continue anyway.

pgloader-csv_1 | 2019-06-13T02:04:22.020000Z LOG pgloader version "3.6.2~devel"

pgloader-csv_1 | 2019-06-13T02:04:22.063000Z WARNING pgloader always talk to PostgreSQL in utf-8, client_encoding has been forced to 'utf8'.

pgloader-csv_1 | 2019-06-13T02:04:22.318000Z LOG report summary reset

pgloader-csv_1 | table name errors rows bytes total time

pgloader-csv_1 | ----------------------- --------- --------- --------- --------------

pgloader-csv_1 | fetch 0 0 0.006s

pgloader-csv_1 | before load 0 2 0.013s

pgloader-csv_1 | ----------------------- --------- --------- --------- --------------

pgloader-csv_1 | "public"."csv" 0 6 0.2 kB 0.050s

pgloader-csv_1 | ----------------------- --------- --------- --------- --------------

pgloader-csv_1 | Files Processed 0 1 0.015s

pgloader-csv_1 | COPY Threads Completion 0 2 0.052s

pgloader-csv_1 | ----------------------- --------- --------- --------- --------------

pgloader-csv_1 | Total import time ? 6 0.2 kB 0.067s

pgloader-project_pgloader-csv_1 exited with code 0

pg 數據庫數據

 

加載固定寬度數據

由於官方文檔加載數據源的緣由（美國)。。。因此我改成使用代碼倉庫中demo 的數據

數據內容地址

代碼倉庫 https://github.com/dimitri/pgloader/blob/master/test/fixed-guess.load && https://github.com/dimitri/pgloader/blob/master/test/fixed-guess.dat

命令行參數

由於我使用docker 因此官方demo 稍有修改

LOAD FIXED

     FROM /opt/db/csv/fixed-guess.dat

     INTO postgresql://postgres:dalong@postgres:5432/postgres

   TARGET TABLE fixed.guess

​

      WITH fixed header

​

      SET work_mem to '14MB',

          standard_conforming_strings to 'on'

​

  before load do

  $$ create schema if not exists fixed; $$,

  $$ drop table if exists fixed.guess; $$,

  $$

    create table fixed.guess

    (

          QECDPO character varying(3),

          QECSEM character varying(3),

          QELSEM character varying(30),

          QERSEM character varying(15),

          QENCOM integer,

          QESCRE smallint,

          QEACRE smallint,

          QEMCRE smallint,

          QEJCRE smallint,

          QEHCRE integer,

          QECUCR character varying(10),

          QESMAJ smallint,

          QEAMAJ smallint,

          QEMMAJ smallint,

          QEJMAJ smallint,

          QEHMAJ integer,

          QECUMJ character varying(10),

          QETOPD character varying(1)

    );

  $$ ;

docker-compose 修改

  pgloader-fixed:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader /loader/csv/fixed.load

執行

docker-compose up pgloader-fixed

執行效果

Creating pgloader-project_pgloader-fixed_1 ... done

Attaching to pgloader-project_pgloader-fixed_1

pgloader-fixed_1 | WARNING:

pgloader-fixed_1 | Couldn't re-execute SBCL with proper personality flags (/proc isn't mounted? setuid?)

pgloader-fixed_1 | Trying to continue anyway.

pgloader-fixed_1 | 2019-06-13T02:28:41.032000Z LOG pgloader version "3.6.2~devel"

pgloader-fixed_1 | 2019-06-13T02:28:41.261000Z LOG Parsed 18 columns specs from header:

pgloader-fixed_1 | (

pgloader-fixed_1 | QECDPO from 0 for 7 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QECSEM from 7 for 7 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QELSEM from 14 for 31 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QERSEM from 45 for 16 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QENCOM from 61 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QESCRE from 101 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEACRE from 141 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEMCRE from 181 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEJCRE from 221 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEHCRE from 261 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QECUCR from 301 for 11 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QESMAJ from 312 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEAMAJ from 352 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEMMAJ from 392 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEJMAJ from 432 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QEHMAJ from 472 for 40 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QECUMJ from 512 for 11 [null if blanks, trim right whitespace],

pgloader-fixed_1 | QETOPD from 523 for 5 [null if blanks, trim right whitespace]

pgloader-fixed_1 | )

pgloader-fixed_1 | 2019-06-13T02:28:41.321000Z LOG report summary reset

pgloader-fixed_1 | table name errors rows bytes total time

pgloader-fixed_1 | ----------------------- --------- --------- --------- --------------

pgloader-fixed_1 | fetch 0 0 0.006s

pgloader-fixed_1 | before load 0 3 0.023s

pgloader-fixed_1 | ----------------------- --------- --------- --------- --------------

pgloader-fixed_1 | "fixed"."guess" 0 6 0.5 kB 0.044s

pgloader-fixed_1 | ----------------------- --------- --------- --------- --------------

pgloader-fixed_1 | Files Processed 0 1 0.031s

pgloader-fixed_1 | COPY Threads Completion 0 2 0.052s

pgloader-fixed_1 | ----------------------- --------- --------- --------- --------------

pgloader-fixed_1 | Total import time ? 6 0.5 kB 0.083s

pgloader-project_pgloader-fixed_1 exited with code 0

pg 數據庫數據

 

加載maxmind 的demo

這個由於擴展緣由，只貼demo，就不運行了

命令行參數

/*

 * Loading from a ZIP archive containing CSV files. The full test can be

 * done with using the archive found at

 * http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip

 *

 * And a very light version of this data set is found at

 * http://pgsql.tapoueh.org/temp/foo.zip for quick testing.

 */

​

LOAD ARCHIVE

   FROM http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip

   INTO postgresql:///ip4r

​

   BEFORE LOAD DO

     $$ create extension if not exists ip4r; $$,

     $$ create schema if not exists geolite; $$,

     $$ create table if not exists geolite.location

       (

          locid integer primary key,

          country text,

          region text,

          city text,

          postalcode text,

          location point,

          metrocode text,

          areacode text

       );

     $$,

     $$ create table if not exists geolite.blocks

       (

          iprange ip4r,

          locid integer

       );

     $$,

     $$ drop index if exists geolite.blocks_ip4r_idx; $$,

     $$ truncate table geolite.blocks, geolite.location cascade; $$

​

   LOAD CSV

        FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/

             WITH ENCODING iso-8859-1

             (

                locId,

                country,

                region null if blanks,

                city null if blanks,

                postalCode null if blanks,

                latitude,

                longitude,

                metroCode null if blanks,

                areaCode null if blanks

             )

        INTO postgresql:///ip4r?geolite.location

             (

                locid,country,region,city,postalCode,

                location point using (format nil "(~a,~a)" longitude latitude),

                metroCode,areaCode

             )

        WITH skip header = 2,

             fields optionally enclosed by '"',

             fields escaped by double-quote,

             fields terminated by ','

​

  AND LOAD CSV

        FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/

             WITH ENCODING iso-8859-1

             (

                startIpNum, endIpNum, locId

             )

        INTO postgresql:///ip4r?geolite.blocks

             (

                iprange ip4r using (ip-range startIpNum endIpNum),

                locId

             )

        WITH skip header = 2,

             fields optionally enclosed by '"',

             fields escaped by double-quote,

             fields terminated by ','

​

   FINALLY DO

     $$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;

sqlite 數據加載

命令行參數

這個demo，直接使用命令名運行，因此就直接貼docker-compose 運行的配置了

  pgloader-sqlite:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite postgresql://postgres:dalong@postgres:5432/postgres

運行效果

pgloader-sqlite_1 | 2019-06-13T02:38:24.315000Z LOG Migrating from #<SQLITE-CONNECTION sqlite:///tmp/pgloader/Chinook_Sqlite_AutoIncrementPKs.sqlite {1006666AC3}>

pgloader-sqlite_1 | 2019-06-13T02:38:24.316000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://postgres@postgres:5432/postgres {10067C7A23}>

pgloader-sqlite_1 | 2019-06-13T02:38:24.927000Z LOG report summary reset

pgloader-sqlite_1 | table name errors rows bytes total time

pgloader-sqlite_1 | ----------------------- --------- --------- --------- --------------

pgloader-sqlite_1 | fetch 0 0 1m35.196s

pgloader-sqlite_1 | fetch meta data 0 55 0.091s

pgloader-sqlite_1 | Create Schemas 0 0 0.001s

pgloader-sqlite_1 | Create SQL Types 0 0 0.010s

pgloader-sqlite_1 | Create tables 0 22 0.080s

pgloader-sqlite_1 | Set Table OIDs 0 11 0.008s

pgloader-sqlite_1 | ----------------------- --------- --------- --------- --------------

pgloader-sqlite_1 | album 0 347 10.5 kB 0.026s

pgloader-sqlite_1 | customer 0 59 6.7 kB 0.024s

pgloader-sqlite_1 | artist 0 275 6.8 kB 0.022s

pgloader-sqlite_1 | employee 0 8 1.4 kB 0.006s

pgloader-sqlite_1 | invoice 0 412 31.0 kB 0.074s

pgloader-sqlite_1 | genre 0 25 0.3 kB 0.034s

pgloader-sqlite_1 | invoiceline 0 2240 43.6 kB 0.086s

pgloader-sqlite_1 | mediatype 0 5 0.1 kB 0.113s

pgloader-sqlite_1 | playlisttrack 0 8715 57.3 kB 0.154s

pgloader-sqlite_1 | playlist 0 18 0.3 kB 0.032s

pgloader-sqlite_1 | track 0 3503 236.6 kB 0.131s

pgloader-sqlite_1 | ----------------------- --------- --------- --------- --------------

pgloader-sqlite_1 | COPY Threads Completion 0 4 0.177s

pgloader-sqlite_1 | Create Indexes 0 33 0.363s

pgloader-sqlite_1 | Index Build Completion 0 33 0.069s

pgloader-sqlite_1 | Reset Sequences 0 10 0.018s

pgloader-sqlite_1 | Primary Keys 0 11 0.021s

pgloader-sqlite_1 | Create Foreign Keys 0 11 0.031s

pgloader-sqlite_1 | Create Triggers 0 0 0.001s

pgloader-sqlite_1 | Install Comments 0 0 0.000s

pgloader-sqlite_1 | ----------------------- --------- --------- --------- --------------

pgloader-sqlite_1 | Total import time ? 15607 394.5 kB 0.680s

pgloader-project_pgloader-sqlite_1 exited with code 0

pg 數據庫

 

mysql 數據加載

mysql 數據準備

mysql -uroot -h127.0.0.1 -pdalongrong

create database f1db;

use f1db

source init-db/mysql/f1db.sql

pg 數據庫準備

psql -h localhost -U postgres

create database f1db

docker-compose 修改

  pgloader-mysql:

    image: dimitri/pgloader

    volumes: 

    - "./init-db:/opt/db"

    - "./pgloader-config:/loader"

    command: pgloader mysql://root:dalongrong@mysql/f1db postgresql://postgres:dalong@postgres:5432/f1db

執行

docker-compose up pgloader-mysql

執行效果

pgloader-mysql_1 | WARNING:

pgloader-mysql_1 | Couldn't re-execute SBCL with proper personality flags (/proc isn't mounted? setuid?)

pgloader-mysql_1 | Trying to continue anyway.

pgloader-mysql_1 | 2019-06-13T02:58:31.034000Z LOG pgloader version "3.6.2~devel"

pgloader-mysql_1 | 2019-06-13T02:58:31.118000Z LOG Migrating from #<MYSQL-CONNECTION mysql://root@mysql:3306/f1db {10064966F3}>

pgloader-mysql_1 | 2019-06-13T02:58:31.119000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://postgres@postgres:5432/f1db {10066DFD33}>

pgloader-mysql_1 | 2019-06-13T02:58:35.542000Z LOG report summary reset

pgloader-mysql_1 | table name errors rows bytes total time

pgloader-mysql_1 | ------------------------- --------- --------- --------- --------------

pgloader-mysql_1 | fetch meta data 0 33 0.091s

pgloader-mysql_1 | Create Schemas 0 0 0.001s

pgloader-mysql_1 | Create SQL Types 0 0 0.006s

pgloader-mysql_1 | Create tables 0 26 0.092s

pgloader-mysql_1 | Set Table OIDs 0 13 0.005s

pgloader-mysql_1 | ------------------------- --------- --------- --------- --------------

pgloader-mysql_1 | f1db.circuits 0 73 8.5 kB 0.038s

pgloader-mysql_1 | f1db.constructors 0 209 15.1 kB 0.030s

pgloader-mysql_1 | f1db.drivers 0 847 80.4 kB 0.104s

pgloader-mysql_1 | f1db.laptimes 0 457064 12.0 MB 3.246s

pgloader-mysql_1 | f1db.constructorresults 0 11420 191.3 kB 0.189s

pgloader-mysql_1 | f1db.constructorstandings 0 12176 255.4 kB 0.171s

pgloader-mysql_1 | f1db.driverstandings 0 32286 732.3 kB 0.315s

pgloader-mysql_1 | f1db.pitstops 0 6994 235.0 kB 0.093s

pgloader-mysql_1 | f1db.races 0 1018 102.9 kB 0.054s

pgloader-mysql_1 | f1db.seasons 0 70 4.1 kB 0.039s

pgloader-mysql_1 | f1db.qualifying 0 8074 309.6 kB 0.183s

pgloader-mysql_1 | f1db.results 0 24340 1.4 MB 0.476s

pgloader-mysql_1 | f1db.status 0 135 1.7 kB 0.009s

pgloader-mysql_1 | ------------------------- --------- --------- --------- --------------

pgloader-mysql_1 | COPY Threads Completion 0 4 3.893s

pgloader-mysql_1 | Create Indexes 0 20 1.535s

pgloader-mysql_1 | Index Build Completion 0 20 0.026s

pgloader-mysql_1 | Reset Sequences 0 10 0.019s

pgloader-mysql_1 | Primary Keys 0 13 0.018s

pgloader-mysql_1 | Create Foreign Keys 0 0 0.000s

pgloader-mysql_1 | Create Triggers 0 0 0.001s

pgloader-mysql_1 | Install Comments 0 0 0.000s

pgloader-mysql_1 | ------------------------- --------- --------- --------- --------------

pgloader-mysql_1 | Total import time ? 554706 15.3 MB 5.492s

pg 效果

 

說明

以上只是簡單的官方文檔的搬運以及集成docker-compose 的運行demo，部分遷移沒有寫，完整的能夠參考官方文檔

參考資料

https://pgloader.readthedocs.io/en/latest/tutorial/tutorial.html
https://github.com/rongfengliang/pgloader-learning