咱們在數據庫運維過程當中不免會涉及到須要對文本數據進行處理,並導入到數據庫中,本文整理了一些導入導出時常見的場景進行示例演示。html
提示:
演示環境MySQL版本的
mysql Ver 14.14 Distrib 5.7.32, for linux-glibc2.12 (x86_64) using EditLine wrappermysql
文章後續示例均使用如下命令導出的 csv 格式樣例數據(以 , 逗號作分隔符,以 " 雙引號做爲界定符)linux
測試數據表結構以下:sql
Create Table: CREATE TABLE `t_menu` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單名稱', `parent_id` int(11) DEFAULT '0' COMMENT '父菜單id', `level` int(11) DEFAULT '1' COMMENT '菜單等級,從1開始', `url` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單連接', `icon` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單圖標', `order` int(11) DEFAULT NULL COMMENT '同級菜單順序', `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `menu_type` int(3) DEFAULT '2' COMMENT '菜單類型:0:目錄,1:頁面,2:不區分(兼容老數據)', PRIMARY KEY (`id`), UNIQUE KEY `unique_menu_name_level_parent_id` (`name`,`level`,`parent_id`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=202 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci 1 row in set (0.00 sec)
-- 導出基礎參數數據庫
load data infile '/data/mysql/tmp/b_menu.txt' replace into table `menu.tmp` character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n'; localhost "mgr01" 10:52:02 test01>select * into outfile '/data/mysql/tmp/b_menu.txt' character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' from test01.b_menu limit 10; Query OK, 10 rows affected (0.00 sec)
[root@test ~]# cat /data/mysql/tmp/b_menu.txt "1","核心數據指標","30","2","/index",\N,"1","2019-06-19 19:58:10","2019-10-31 20:27:37","1" "2","拍機數據","29","2","/auction-dashboard",\N,"1","2019-06-19 19:58:24","2019-10-24 20:21:36","1" "3","產品滯留數據","31","2","/product-dashboard",\N,"1","2019-06-19 19:58:42","2019-10-24 20:21:36","1" "4","發貨數據","42","3","/product-data",\N,"1","2019-08-29 17:44:35","2019-11-18 17:22:29","1" "6","退租數據","14","2","/tuizushuju","","3","2019-09-25 19:05:47","2019-11-18 17:23:40","1" "7","呆滯數據","14","2","/daizhishuju","","2","2019-09-25 19:12:29","2019-11-18 17:23:40","1" "10","發貨數據明細","14","2","/shujumingxi","","4","2019-09-25 19:15:37","2019-11-18 17:23:40","1" "12","增率統計","32","3","/branch-dashboard",\N,"1","2019-09-26 21:23:16","2020-01-15 21:03:38","1" "13","增率詳細","32","3","/customer-dashboard",\N,"2","2019-09-26 21:23:46","2020-01-15 21:03:38","1" "14","產品部數據","0","1","/svn7kezaqe9","","5","2019-09-29 21:58:09","2020-07-28 21:18:50","0"
建立測試臨時表 menu.tmp:app
CREATE TABLE `menu.tmp` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單名稱', `parent_id` int(11) DEFAULT '0' COMMENT '父菜單id', `level` int(11) DEFAULT '1' COMMENT '菜單等級,從1開始', `url` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單連接', `icon` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單圖標', `order` int(11) DEFAULT NULL COMMENT '同級菜單順序', `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `menu_type` int(3) DEFAULT '2' COMMENT '菜單類型:0:目錄,1:頁面,2:不區分(兼容老數據)', PRIMARY KEY (`id`), UNIQUE KEY `unique_menu_name_level_parent_id` (`name`,`level`,`parent_id`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; localhost "mgr01" 10:59:07 test01>load data infile '/data/mysql/tmp/b_menu.txt' replace into table `menu.tmp` character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n'; Query OK, 10 rows affected (0.03 sec) Records: 10 Deleted: 0 Skipped: 0 Warnings: 0 localhost "mgr01" 10:59:17 test01> localhost "mgr01" 11:00:28 test01>select * from `menu.tmp`; +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ | id | name | parent_id | level | url | icon | order | create_time | update_time | menu_type | +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ | 1 | 核心數指標 | 30 | 2 | /index | NULL | 1 | 2019-06-19 19:58:10 | 2019-10-31 20:27:37 | 1 | | 2 | 易機數據 | 29 | 2 | /auction-dashboard | NULL | 1 | 2019-06-19 19:58:24 | 2019-10-24 20:21:36 | 1 | | 3 | 產品滯留數據 | 31 | 2 | /product-dashboard | NULL | 1 | 2019-06-19 19:58:42 | 2019-10-24 20:21:36 | 1 | | 4 | 發貨數據 | 42 | 3 | /product-data | NULL | 1 | 2019-08-29 17:44:35 | 2019-11-18 17:22:29 | 1 | | 6 | 退數據 | 14 | 2 | /tuizushuju | | 3 | 2019-09-25 19:05:47 | 2019-11-18 17:23:40 | 1 | | 7 | 數據 | 14 | 2 | /daizhishuju | | 2 | 2019-09-25 19:12:29 | 2019-11-18 17:23:40 | 1 | | 10 | 數據明細 | 14 | 2 | /shujumingxi | | 4 | 2019-09-25 19:15:37 | 2019-11-18 17:23:40 | 1 | | 12 | 租率統計 | 32 | 3 | /branch-dashboard | NULL | 1 | 2019-09-26 21:23:16 | 2020-01-15 21:03:38 | 1 | | 13 | 租率詳細 | 32 | 3 | /customer-dashboard | NULL | 2 | 2019-09-26 21:23:46 | 2020-01-15 21:03:38 | 1 | | 14 | 產品部數據 | 0 | 1 | /svn7kezaqe9 | | 5 | 2019-09-29 21:58:09 | 2020-07-28 21:18:50 | 0 | +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ 10 rows in set (0.00 sec)
只須要文本文件中部分數據導入到數據表中運維
臨時建立2個字段的表結構:ide
localhost "mgr01" 11:09:48 test01>create table menu_tmp01 as select id,name,level,url from `menu.tmp`; ERROR 1786 (HY000): Statement violates GTID consistency: CREATE TABLE ... SELECT. localhost "mgr01" 11:00:38 test01>create table `menu.tmp01` select id,name,level,url from `menu.tmp`; ERROR 1786 (HY000): Statement violates GTID consistency: CREATE TABLE ... SELECT.
緣由是MySQL開啓了Gtid,致使的:svn
通常mysql5.7之前版本是支持create table XXX as select * from XXX; 這種建立表的語法,可是MySQL5.7.x版本里面gtid是開啓的,會報錯
ERROR 1786 (HY000):Statement violates GTID consistency: CREATE TABLE ... SELECT.
官方說明:https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html函數
有2種方式關閉MySQL的開啓的Gtid:
第一種 直接修改MySQL的my.cnf的配置文件,重啓MySQL服務:
gtid_mode = off
enforce_gtid_consistency = 0
第二種方式就是在線滾動修改參數:
嘗試在線動態修改時的報錯:
localhost "mgr01" 11:15:36 test01>SET @@GLOBAL.ENFORCE_GTID_CONSISTENCY = off; ERROR 1779 (HY000): GTID_MODE = ON requires ENFORCE_GTID_CONSISTENCY = ON. localhost "mgr01" 11:16:49 test01> set global GTID_MODE = off; ERROR 1788 (HY000): The value of @@GLOBAL.GTID_MODE can only be changed one step at a time: OFF <-> OFF_PERMISSIVE <-> ON_PERMISSIVE <-> ON. Also note that this value must be stepped up or down simultaneously on all servers. See the Manual for instructions.
上面提示若是當前值爲ON,要設置爲OFF,則先設置爲GTID_MODE=ON_PERMISSIVE,再設置GTID_MODE=OFF_PERMISSIVE,再設置GTID_MODE = off,若是將OFF設置爲ON,則反過來設置便可。
繼續設置:
localhost "mgr01" 11:25:51 test01>set @@GLOBAL.GTID_MODE=ON_PERMISSIVE; Query OK, 0 rows affected (0.03 sec) localhost "mgr01" 11:25:52 test01>set @@GLOBAL.GTID_MODE=OFF_PERMISSIVE; Query OK, 0 rows affected (0.01 sec)
若是set @@GLOBAL.GTID_MODE=OFF_PERMISSIVE;報錯時,通常是以下報錯:
mysql> set @@GLOBAL.GTID_MODE=OFF_PERMISSIVE; ERROR 1766 (HY000): The system variable gtid_mode cannot be set when there is an ongoing transaction.
上面報錯,當有正在進行的事務時,不能設置,因此就COMMIT一下:
localhost "mgr01" 11:26:00 test01>commit; Query OK, 0 rows affected (0.00 sec) localhost "mgr01" 11:27:48 test01>set @@GLOBAL.GTID_MODE=OFF_PERMISSIVE; Query OK, 0 rows affected (0.00 sec) localhost "mgr01" 11:28:01 test01>set @@GLOBAL.GTID_MODE=OFF; Query OK, 0 rows affected (0.02 sec) localhost "mgr01" 11:28:19 test01> show variables like 'GTID_MODE'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | gtid_mode | OFF | +---------------+-------+ 1 row in set (0.00 sec)
而後再設置SET GLOBAL ENFORCE_GTID_CONSISTENCY = off:
localhost "mgr01" 11:29:03 test01>show variables like 'ENFORCE_GTID_CONSISTENCY'; +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | enforce_gtid_consistency | OFF | +--------------------------+-------+ **到此時在線把Gtid關閉掉了。**
再次執行
create table menu_tmp01 as select id,name,level,url from menu.tmp
;
create table menu_tmp02 select id,name,level,url from menu.tmp
;
localhost "mgr01" 11:29:17 test01>create table menu_tmp01 as select id,name,level,url from `menu.tmp`; Query OK, 10 rows affected (0.04 sec) Records: 10 Duplicates: 0 Warnings: 0 localhost "mgr01" 11:30:10 test01>desc menu_tmp01; +-------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-----+---------+-------+ | id | int(11) | NO | | 0 | | | name | varchar(255) | YES | | NULL | | | level | int(11) | YES | | 1 | | | url | varchar(255) | YES | | NULL | | +-------+--------------+------+-----+---------+-------+ 4 rows in set (0.00 sec) localhost "mgr01" 11:30:20 test01>create table menu_tmp02 select id,name,level,url from `menu.tmp`; Query OK, 10 rows affected (0.04 sec) Records: 10 Duplicates: 0 Warnings: 0 localhost "mgr01" 11:30:45 test01>desc menu_tmp02; +-------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-----+---------+-------+ | id | int(11) | NO | | 0 | | | name | varchar(255) | YES | | NULL | | | level | int(11) | YES | | 1 | | | url | varchar(255) | YES | | NULL | | +-------+--------------+------+-----+---------+-------+ 4 rows in set (0.00 sec) localhost "mgr01" 11:30:50 test01>
接着 場景1.LOAD 文件中的字段比數據表中的字段多 ,把只須要文本文件中部分數據導入到數據表中演示
-- 導入數據語句
load data infile '/data/mysql/tmp/b_menu.txt' replace into table test01.menu_tmp01 character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' (@C1,@C2,@C3,@C4,@C5,@C6,@7,@8,@9,@10) set id=@C1, name=@C2, level=@C4, url=@C5;
導入數據:
load data infile '/data/mysql/tmp/b_menu.txt' replace into table test01.menu_tmp01 character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' (@C1,@C2,@C3,@C4,@C5,@C6,@7,@8,@9,@10) -- 該部分對應b_menu.txt件中10列數據 -- 只對導出數據中指定的2個列與表中字段作匹配,mapping關係指定的順序不影響導入結果 set id=@C1, name=@C2, level=@C4, url=@C5; localhost "mgr01" 11:46:19 test01>load data infile '/data/mysql/tmp/b_menu.txt' -> replace into table test01.menu_tmp01 -> character set utf8mb4 -> fields terminated by ',' -> enclosed by '"' -> lines terminated by '\n' -> (@C1,@C2,@C3,@C4,@C5,@C6,@7,@8,@9,@10) -- 該部分對應b_menu.txt件中10列數據 -> -- 只對導出數據中指定的2個列與表中字段作匹配,mapping關係指定的順序不影響導入結果 -> set id=@C1, -> name=@C2, -> level=@C4, -> url=@C5; Query OK, 10 rows affected (0.01 sec) Records: 10 Deleted: 0 Skipped: 0 Warnings: 0 localhost "mgr01" 11:46:26 test01>select * from menu_tmp01; +----+--------------------+-------+---------------------+ | id | name | level | url | +----+--------------------+-------+---------------------+ | 1 | 核心數據指標 | 2 | /index | | 2 | 易機數據 | 2 | /auction-dashboard | | 3 | 產品滯留數據 | 2 | /product-dashboard | | 4 | 發貨數據 | 3 | /product-data | | 6 | 退租數據 | 2 | /tuizushuju | | 7 | 呆滯數據 | 2 | /daizhishuju | | 10 | 發貨數據明細 | 2 | /shujumingxi | | 12 | 增率統計 | 3 | /branch-dashboard | | 13 | 增率詳細 | 3 | /customer-dashboard | | 14 | 產品部數據 | 1 | /svn7kezaqe9 | +----+--------------------+-------+---------------------+ 10 rows in set (0.00 sec)
說明:表字段不只包含文本文件中全部數據,還包含了額外的字段
導出部分MySQL表test01.b_menu部分字段的數據到文本文件:
select id,name,url,create_time into outfile '/data/mysql/tmp/c_menu.txt' character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' from test01.b_menu limit 10; [root@test tmp]# cat /data/mysql/tmp/c_menu.txt "1","核心數據指標","/index","2019-06-19 19:58:10" "2","易機數據","/auction-dashboard","2019-06-19 19:58:24" "3","產品滯留數據","/product-dashboard","2019-06-19 19:58:42" "4","發貨數據","/product-data","2019-08-29 17:44:35" "6","退租數據","/tuizushuju","2019-09-25 19:05:47" "7","呆滯數據","/daizhishuju","2019-09-25 19:12:29" "10","發貨數據明細","/shujumingxi","2019-09-25 19:15:37" "12","增率統計","/branch-dashboard","2019-09-26 21:23:16" "13","增率詳細","/customer-dashboard","2019-09-26 21:23:46" "14","產品部數據","/svn7kezaqe9","2019-09-29 21:58:09"
建立測試表a_menu:
CREATE TABLE `a_menu` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單名稱', `parent_id` int(11) DEFAULT '0' COMMENT '父菜單id', `level` int(11) DEFAULT '1' COMMENT '菜單等級,從1開始', `url` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單連接', `icon` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '菜單圖標', `order` int(11) DEFAULT NULL COMMENT '同級菜單順序', `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `menu_type` int(3) DEFAULT '2' COMMENT '菜單類型:0:目錄,1:頁面,2:不區分(兼容老數據)', PRIMARY KEY (`id`), UNIQUE KEY `unique_menu_name_level_parent_id` (`name`,`level`,`parent_id`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
load data infile '/data/mysql/tmp/c_menu.txt' replace into table test01.a_menu character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' (@C1,@C2,@C3,@C4,@C5,@C6,@7,@8,@9,@10) -- 該部分對應test01.a_menu表中10列字段 -- 只對導出數據中指定的4個列與表中字段作匹配,mapping關係指定的順序不影響導入結果. a_menu表中多出的字段不作處理,這些表字段以設定的默認值和null來處理 set id=@C1, name=@C2, url=@C3, create_time=@C4; -- 此行set後面的這些@C1 @C2 @C3 @C4 指的是導出文件/data/mysql/tmp/c_menu.txt中的先後順序的4列數值。
下面的sql纔是正確的姿式:
localhost "mgr01" 12:50:02 (none)>load data infile '/data/mysql/tmp/c_menu.txt' replace into table test01.a_menu character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' (@C1,@C2,@C3,@C4,@C5,@C6,@7,@8,@9,@10) set id=@C1, name=@C2, url=@C3, create_time=@C4; Query OK, 10 rows affected (0.02 sec) Records: 10 Deleted: 0 Skipped: 0 Warnings: 0 localhost "mgr01" 12:50:23 (none)>select * from test01.a_menu; +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ | id | name | parent_id | level | url | icon | order | create_time | update_time | menu_type | +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ | 1 | 核心數據指標 | 0 | 1 | /index | NULL | NULL | 2019-06-19 19:58:10 | 2021-03-27 12:50:23 | 2 | | 2 | 易機數據 | 0 | 1 | /auction-dashboard | NULL | NULL | 2019-06-19 19:58:24 | 2021-03-27 12:50:23 | 2 | | 3 | 產品滯留數據 | 0 | 1 | /product-dashboard | NULL | NULL | 2019-06-19 19:58:42 | 2021-03-27 12:50:23 | 2 | | 4 | 發貨數據 | 0 | 1 | /product-data | NULL | NULL | 2019-08-29 17:44:35 | 2021-03-27 12:50:23 | 2 | | 6 | 退租數據 | 0 | 1 | /tuizushuju | NULL | NULL | 2019-09-25 19:05:47 | 2021-03-27 12:50:23 | 2 | | 7 | 呆滯數據 | 0 | 1 | /daizhishuju | NULL | NULL | 2019-09-25 19:12:29 | 2021-03-27 12:50:23 | 2 | | 10 | 發貨數據明細 | 0 | 1 | /shujumingxi | NULL | NULL | 2019-09-25 19:15:37 | 2021-03-27 12:50:23 | 2 | | 12 | 增率統計 | 0 | 1 | /branch-dashboard | NULL | NULL | 2019-09-26 21:23:16 | 2021-03-27 12:50:23 | 2 | | 13 | 增率詳細 | 0 | 1 | /customer-dashboard | NULL | NULL | 2019-09-26 21:23:46 | 2021-03-27 12:50:23 | 2 | | 14 | 產品部數據 | 0 | 1 | /svn7kezaqe9 | NULL | NULL | 2019-09-29 21:58:09 | 2021-03-27 12:50:23 | 2 | +----+--------------------+-----------+-------+---------------------+------+-------+---------------------+---------------------+-----------+ 10 rows in set (0.00 sec)
從場景 2 的驗證能夠看到,emp 表中新增的字段 fullname,modify_date,delete_flag 字段在導入時並未作處理,被置爲了 NULL 值,若是須要對其進行處理,可在 LOAD 時經過 MySQL支持的函數 或給定 固定值 自行定義數據,對於文件中存在的字段也可作函數處理,結合導入導出,實現簡單的 ETL 功能,以下所示:
-- 導入數據語句
load data infile '/data/mysql/3306/tmp/employees.txt' replace into table demo.emp character set utf8mb4 fields terminated by ',' enclosed by '"' lines terminated by '\n' (@C1,@C2,@C3,@C4,@C5,@C6) -- 該部分對應employees.txt文件中6列數據
-- 如下部分明確對錶中字段與數據文件中的字段作Mapping關係,不存在的數據經過函數處理生成(也可設置爲固定值)
set emp_no=@C1, birth_date=@C2, first_name=upper(@C3), -- 將導入的數據轉爲大寫 last_name=lower(@C4), -- 將導入的數據轉爲小寫 fullname=concat(first_name,' ',last_name), -- 對first_name和last_name作拼接 gender=@C5, hire_date=@C6 , modify_date=now(), -- 生成當前時間數據 delete_flag=if(hire_date<'1988-01-01','Y','N'); -- 對須要生成的值基於某一列作條件運算
參考文檔:
https://mp.weixin.qq.com/s/WNXRshkvC3bFcc5NDaWlrw