Postgres的TOAST技術

1、介紹
首先，Toast是一個名字縮寫，全寫是The OverSized Attribute Storage Technique，即超尺寸字段存儲技術，顧名思義，是說超長字段在Postgres的一個存儲方式。Postgres採用的存儲默認是每一個頁面存儲固定8Kb大小的數據，而且元組不容許跨頁面存儲，因此並不能直接存儲大字段數據。Toast就是爲此應運而生，它會將大字段值壓縮或者分散爲多個物理行來存儲。對於用戶來講徹底不用關注這一技術實現，徹底是透明的。

2、TOAST的存儲方式
      Postgres的部分類型數據支持toast，不是所有類型是由於有些字段類型是不會產生大字段數據的，徹底不必用到Toast技術(好比date,time,boolean等)。支持Toast的數據類型應當時變長的(variable-length)，變長字段最多可選擇32bit的列頭(header),Toast佔用兩個變長的bit位來做爲FLAG，故Toast的邏輯尺寸限制是(2^30-1)~1GB，當兩個bit都是0是，這個數據類型的值就是非Toast的(untoasted)。當表中字段任何一個有Toast，那這個表都會有這一個相關聯的Toast表，OID被存儲在pg_class.reltoastrelid裏面。超出的的數值將會被分割成chunks，並最多toast_max_chunk_size 個byte(缺省是2Kb)，當存儲的行數據超過toast_tuple_threshold值(一般是2kB)，就會觸發toast存儲，這時toast將會壓縮或者移動字段值直到超出部分比toast_tuple_targer值小(這個值一般也是2KB)。

相比較普通表(MAIN TABLE),TOAST有額外的三個字段

chunk_id   :標識TOAST表的OID字段
chunk_seq  :chunk的序列號，與chunk_id的組合惟一索引能夠加速訪問
chunk_data :存儲TOAST表的實際數據

Toast有識別4種不一樣可存儲toast的策略：

--plain避免壓縮或行外存儲
PLAIN prevents either compression or out-of-line storage; furthermore it disables use of single-byte headers for varlena types. This is the only possible strategy for columns of non-TOAST-able data types

--extended容許壓縮和行外存儲(默認toast存儲)
EXTENDED allows both compression and out-of-line storage. This is the default for most TOASTable data types. Compression will be attempted first, then out-of-line storage if the row is still too big

--external容許行外但不容許壓縮
EXTERNAL allows out-of-line storage but not compression. Use of EXTERNAL will make substring operations on wide text and bytea columns faster(at the penalty of increased storage space) because these operations are optimized to fetch only the required parts of the out-of-line value when it is not compressed

--main容許壓縮但不容許行外存儲
MAIN allows compression but not out-of-line storage. (Actually, out-of-line storage will still be performed for such columns, but only as a last resort when there is no other way to make the row small enough to fit on a page

上述壓縮採用的是LZ compression技術，
源碼見： postgresql-9.2.3/src/backend/utils/adt/pg_lzcompress.c

TOAST能夠經過SQL來更改存儲方式，示例：

ALTER TABLE table_name ALTER COLUMN  column_name SET STORAGE {PLAIN | EXTENDED | MAIN | EXTERNAL};

postgres=# \d+ t_kenyon
                            Table "public.t_kenyon"
Column |       Type        | Modifiers | Storage  | Stats target | Description
--------+-------------------+-----------+----------+--------------+-------------
dd     | character varying |           | extended |              |
Has OIDs: no

postgres=# alter table t_kenyon alter column dd set storage main;
ALTER TABLE
postgres=# \d+ t_kenyon
                            Table "public.t_kenyon"
Column |       Type        | Modifiers | Storage | Stats target | Description
--------+-------------------+-----------+---------+--------------+-------------
dd     | character varying |           | main    |              |
Has OIDs: no

3、TOAST表的計算
計算一個表的大小時要注意統計Toast的大小，由於對超長字段存儲時，在基礎表上可能只存了20%，另外的數據都存到了Toast裏面去了，計算大小時要結合起來看，索引也是同樣，對於表裏有main或者extended類型的會建立Toast表，二者的關聯是經過pg_class裏的OID去關聯的。下面舉例子：
1.TOAST表關聯查詢
Example a:

[postgres@localhost ~]$ psql
psql (9.2.3)
Type "help" for help.

postgres=# create table t_kenyon(id int);
CREATE TABLE

postgres=# select relname,reltoastrelid from pg_class where relname = 't_kenyon';
relname  | reltoastrelid
----------+---------------
t_kenyon |             0
(1 row)

postgres=# \d+ t_kenyon
                       Table "public.t_kenyon"
Column |  Type   | Modifiers | Storage | Stats target | Description
--------+---------+-----------+---------+--------------+-------------
id     | integer |           | plain   |              |
Has OIDs: no

上面的字段沒有toast表，由於字段int是定長的。
Example b:

postgres=# select relname,reltoastrelid from pg_class where relname = 't_kenyon';
relname  | reltoastrelid
----------+---------------
t_kenyon |         16411
(1 row)

postgres=# select relname from pg_class where oid = 16411;
    relname    
----------------
pg_toast_16408
(1 row)

2.TOAST表計算大小

postgres=# drop table t_kenyon;
DROP TABLE
postgres=# create table t_kenyon(id int,vname varchar(48),remark text);
CREATE TABLE
postgres=# \d+ t_kenyon
                              Table "public.t_kenyon"
Column |         Type          | Modifiers | Storage  | Stats target | Description
--------+-----------------------+-----------+----------+--------------+-------------
id     | integer               |           | plain    |              |
vname  | character varying(48) |           | extended |              |
remark | text                  |           | extended |              |
Has OIDs: no

postgres=# select oid,relname,reltoastrelid from pg_class where relname = 't_kenyon';
  oid   | relname  | reltoastrelid
--------+----------+---------------
121174 | t_kenyon |        121177
(1 row)

postgres=# insert into t_kenyon select generate_series(1,2000),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God',500);
INSERT 0 2000

postgres=# insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',2000);
INSERT 0 2
postgres=# select pg_column_size(id),pg_column_size(vname),pg_column_size(remark) from t_kenyon limit 2;
pg_column_size | pg_column_size | pg_column_size
----------------+----------------+----------------
              4 |             29 |            851
              4 |             29 |            851
(2 rows)

--查看基礎表和Toast的大小
postgres=# select pg_relation_size(121174);
pg_relation_size
------------------
             8192
(1 row)

postgres=# select pg_relation_size(121177);
pg_relation_size
------------------
                0
(1 row)

--文本數據量增多，這時能夠看到toast表字段大小在2kb左右時有大小了
postgres=# insert into t_kenyon select generate_series(3,4),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',4000);
INSERT 0 2
postgres=# select pg_relation_size(121174);
pg_relation_size
------------------
             8192
(1 row)

postgres=# select pg_relation_size(121177);
pg_relation_size
------------------
                0
(1 row)


postgres=# insert into t_kenyon select generate_series(5,6),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',5500);
INSERT 0 2
postgres=# select pg_column_size(id),pg_column_size(vname),pg_column_size(remark) from t_kenyon;
pg_column_size | pg_column_size | pg_column_size
----------------+----------------+----------------
              4 |             29 |            851
              4 |             29 |            851
              4 |             29 |           1651
              4 |             29 |           1651
              4 |             29 |           2247
              4 |             29 |           2247
(6 rows)

postgres=# insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',10000);
INSERT 0 2
postgres=# select pg_relation_size(121174);
pg_relation_size
------------------
             8192
(1 row)

postgres=# select pg_relation_size(121177);
pg_relation_size
------------------
            16384
(1 row)

postgres=# insert into t_kenyon select generate_series(7,8),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',20000);
INSERT 0 2
postgres=# select id,pg_column_size(id),pg_column_size(vname),pg_column_size(remark) from t_kenyon;
id | pg_column_size | pg_column_size | pg_column_size
----+----------------+----------------+----------------
  1 |              4 |             29 |            851
  2 |              4 |             29 |            851
  3 |              4 |             29 |           1651
  4 |              4 |             29 |           1651
  5 |              4 |             29 |           2247
  6 |              4 |             29 |           2247
  7 |              4 |             29 |           8056
  8 |              4 |             29 |           8056
(8 rows)

postgres=# select pg_relation_size(121174);
pg_relation_size
------------------
             8192
(1 row)

postgres=# select pg_relation_size(121177);
pg_relation_size
------------------
            24576
(1 row)

能夠看到後插入的數據隨着字段內容的增多，toast段一直在變大。這個和Oracle存儲的大字段內容比較像，Oracle存儲Blob類的數據時也是指定另外的segment來存儲，而不是在原表中存儲，固然能夠設置enable storage in row來指定表中存儲，因此Oracle裏的表異常大時通常不是水位線太高就是表字段裏存了大數據類型了。

4、TOAST的優缺點
1.能夠存儲超長超大字段，避免以前不能直接存儲的限制
2.物理上與普通表是分離的，檢索查詢時不檢索到該字段會極大地加快速度
3.更新普通表時，該表的Toast數據沒有被更新時，不用去更新Toast表

Toast的劣勢：
1.對大字段的索引建立是一個問題，有可能會失敗，其實一般也不建議在大字段上建立，全文檢索卻是一個解決方案。
2.大字段的更新會有點慢，其它DB也存在，通病

5、其餘 Toast的源碼: postgresql-9.2.3/src/backend/access/heap/tuptoaster.c postgresql-9.2.3/src/backend/catalog/toasting.c