PostgreSQL的pg_xlog文件數計算和在線清理

postgresql的pg_xlog是記錄數據庫事務信息用的，相似oracle的redo信息，也叫wal日誌(write ahead log)，就是在寫數據到磁盤裏成爲固定數據以前，先寫入到日誌裏，而後必定條件下觸發調用fsync()將此數據刷到磁盤。好處很少說，下面主要看看postgresql一般會保留多少個pg_xlog文件，以及如何在線清理。

1、環境及參數

環境:
postgresql 9.2.4(單臺，未作流複製)
vmware 6.0
centos 6.2

參數:
postgres=# show checkpoint_segments ;   --設置相對大一點,默認過小，防止頻繁checkpoint
checkpoint_segments
---------------------
30
(1 row)

postgres=# show checkpoint_timeout ;
checkpoint_timeout
--------------------
5min
(1 row)

postgres=# show checkpoint_completion_target ;
checkpoint_completion_target
------------------------------
0.5
(1 row)

postgres=# show wal_keep_segments ;     --未開啓流複製預留xlog參數
 wal_keep_segments 
-------------------
 0
(1 row)

postgres=# show archive_mode ;             --關閉歸檔
archive_mode 
--------------
off
(1 row)

理論上合理的pg_xlog通常在(2+checkpoint_completion_target)*checkpoint_segment+1左右浮動，可是高併發環境下若是checkpoint_segment設置比較大，且checkpoint較少被觸發的時候，pg_xlog下會存有不少的事務日誌，嚴重會耗掉磁盤存儲，因此設置checkpoint_segment的大小要視磁盤容量和每一個pg_xlog文件的大小來看，有些時候須要手工調整參數來收縮。

2、測試過程:

1.模擬pg_xlog文件增加
模擬大數據量載入,開啓多個窗口插入數據,每一個窗口插入1000W數據,能夠看到pg_xlog增加不少，原來只有8個文件

postgres=# create table tbl_kenyon(id int,cname varchar(50),remark text);
CREATE TABLE

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

此時檢查pg_xlog下面的文件數量

[postgres@localhost pg_xlog]$ ll
total 4063240
-rw-------  1 postgres postgres 67108864 Jul 14 23:05 00000001000000020000002F
-rw-------  1 postgres postgres 67108864 Jul 14 23:13 000000010000000200000030
-rw-------  1 postgres postgres 67108864 Jul 14 23:18 000000010000000200000031
-rw-------  1 postgres postgres 67108864 Jul 14 23:25 000000010000000200000032
-rw-------  1 postgres postgres 67108864 Jul 14 22:42 000000010000000200000033
-rw-------  1 postgres postgres 67108864 Jul 14 22:42 000000010000000200000034
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000035
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000036
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000200000037
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000038
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000039
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003A
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000020000003B
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003C
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000020000003D
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003E
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000000
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000001
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000002
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000003
-rw-------. 1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000004
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000005
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000006
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000007
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000008
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000009
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000000A
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000030000000B
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000C
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000000D
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000E
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000F
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000010
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000011
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000012
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000013
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000014
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000015
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000016
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000017
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000018
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000019
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001A
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001B
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001C
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001D
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001E
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001F
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000020
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000021
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000022
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000023
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000024
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000025
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000026
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000027
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000028
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000029
-rw-------. 1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002A
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002B
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002C
-rw-------  1 postgres postgres 67108864 Jul 14 22:56 00000001000000030000002D
drwx------. 2 postgres postgres     4096 Jun 12 23:19 archive_status
[postgres@localhost pg_xlog]$ ll|wc -l
64

2.模擬pg_xlog文件清理
這類文件對DB是很重要的，不能手工直接清理。每一個文件64MB，是源碼編譯時--with-wal-segsize時指定的，或者登錄查看

postgres=# show wal_segment_size ;  
wal_segment_size 
------------------
 64MB
(1 row)

要收縮這個事務日誌空間，減小事務日誌數，能夠調整參數 checkpoint_segments，本次調爲3

postgres=# show checkpoint_segments ;
 checkpoint_segments 
---------------------
 3
(1 row)


postgres=# checkpoint;
CHECKPOINT

 xlog文件有可能不是當即減小的，若是想模擬當即減小，能夠模擬數據庫頻繁checkpoint，如從新作上述的大數據庫加載或者更新操做，此時去看pg_xlog的文件數，能看到在減小了，開啓兩個窗口操做:

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,2000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 2000000

[postgres@localhost pg_xlog]$ ll|wc -l
62
[postgres@localhost pg_xlog]$ ll|wc -l
49
[postgres@localhost pg_xlog]$ ll|wc -l
46
[postgres@localhost pg_xlog]$ ll|wc -l
46
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
40
[postgres@localhost pg_xlog]$ ll|wc -l
40
[postgres@localhost pg_xlog]$ ll|wc -l
31
[postgres@localhost pg_xlog]$ ll|wc -l
25

後續發生的checkpoint，將會逐步收縮pg_xlog的文件數量，直到知足條件爲止。以前寫過另一種較爲激進的清理pg_xlog的方式，須要停機清理，參考 http://my.oschina.net/Kenyon/blog/101432

3.其餘
若是是流複製，還有wal_keep_segments這個參數也要考慮一下。後續。

4.說明

checkpoint_segments (integer)
Maximum number of log file segments between automatic WAL checkpoints (each segment
is normally 16 megabytes). The default is three segments. Increasing this parameter can increase
the amount of time needed for crash recovery. This parameter can only be set in the
postgresql.conf file or on the server command line.

5.參考:
http://my.oschina.net/Kenyon/blog/101432
http://francs3.blog.163.com/blog/static/4057672720112182158386/