Collectd & InfluxDb & Grafana 之二: Postgresql 統計

2016-09-27 更新: 數據庫統計指標說明

Postgresql 統計圖表配置

下載配置文件並導入, 而後根據本身的Collectd配置進行調整

https://raw.githubusercontent...

最後的效果以下圖

Collectd 統計項

查看types.db文件中定義的Postgresql統計規範

root@ubuntu:~# cat /usr/share/collectd/types.db |grep pg_
pg_blks           value:DERIVE:0:U
pg_db_size        value:GAUGE:0:U
pg_n_tup_c        value:DERIVE:0:U
pg_n_tup_g        value:GAUGE:0:U
pg_numbackends    value:GAUGE:0:U
pg_scan           value:DERIVE:0:U
pg_xact           value:DERIVE:0:U

第二個字段爲數據源類型, types.db規範參考types.db.5.shtml

關於 Collectd 的 types.db 數據規範定義文件

types.db - 系通通計收集守護進程collectd的數據集說明

大綱

bitrate value:GAUGE:0:4294967295
counter value:COUNTER:U:U
if_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295

描述

• 對每一個數據集說明, type.db文件都包含了一行. 每行由兩個字段組成, 由空格或者tab分隔.

• 第一個字段定義了數據集的名稱, 第二個字段定義了數據源說明的列表, 以空格分隔, 對每個列表單元都以逗號分隔.

• 數據源說明的格式受到RRDtool的數據源說明格式影響. 每一個數據源由4部分組成, 分別爲數據源名, 類型, 最小值和最大值, 之間由:分隔. ds-name:ds-type:min:max.

• 其中ds-type包含4中類型, ABSOULUTE, COUNTER, DERIVE或者GAUSE.

• mix和max定義了固定值範圍. 若是U在min或者max中指定, 則意味着不知道範圍.

文件

types.db 的文件配置在 collectd.conf 中. 在Ubuntu中, 該文件的默認位置爲 /usr/share/collectd/types.db.

定製types.db

若是你想指定一個定製的類型, 你能夠在默認的 types.db 裏添加, 或者能夠另起一行在下面添加一個新的文件.

For example:
　　TypesDB "/opt/collectd/share/collectd/types.db"
　　TypesDB "/opt/collectd/etc/types.db.custom"

注意: 若是你想使用這種方式, 必須在網絡中全部系統中都添加該文件.

Collectd 的 Postgresql 配置

postgresql 插件從PostgreSQL數據庫中查詢統計信息. 它保持一個到全部配置的數據庫的鏈接, 而且當鏈接中斷時重連. 數據庫是由一個 <Database> 配置塊進行配置. 默認統計是從PostgreSQL的統計收集器統計的. 要使這個插件可以正常的工做, 須要啓用數據庫的統計蒐集功能. 參考 [Statistics Collector]()文檔

經過使用 <Query> 塊指定自定義的數據庫查詢, 能夠蒐集任何數據.

<Plugin postgresql>
  <Query locks>
    Statement "
    SELECT COUNT(mode) AS count, mode
      FROM pg_locks GROUP BY mode
    UNION SELECT COUNT(*) AS count, 'waiting' AS mode
      FROM pg_locks
      WHERE granted is false;
    "
    <Result>
      Type "gauge"
      InstancePrefix "pg_locks"
      InstancesFrom "mode"
      ValuesFrom "count"
    </Result>
  </Query>
  <Query seq_scans>
    Statement "
      SELECT CASE WHEN status='OK' THEN 0 ELSE 1 END AS status FROM (
        SELECT get_seq_scan_on_large_tables AS status FROM collectd.get_seq_scan_on_large_tables
      ) AS foo;
    "
    <Result>
      Type "gauge"
      InstancePrefix "pg_seq_scans"
      ValuesFrom "status"
    </Result>
  </Query>
  <Query connections>
    Statement "
      SELECT COUNT(state) AS count, state FROM (SELECT CASE
      WHEN state = 'idle' THEN 'idle'
      WHEN state = 'idle in transaction' THEN 'idle_in_transaction'
      WHEN state = 'active' THEN 'active'
      ELSE 'unknown' END AS state FROM collectd.pg_stat_activity) state
      GROUP BY state
      UNION SELECT COUNT(*) AS count, 'waiting' AS state FROM collectd.pg_stat_activity WHERE waiting;
    "
    <Result>
      Type "pg_numbackends"
      InstancePrefix "state"
      InstancesFrom "state"
      ValuesFrom "count"
    </Result>
  </Query>
  <Query slow_queries>
    Statement "
    SELECT COUNT(*) AS count FROM collectd.pg_stat_activity
    WHERE state='active' and now()-query_start > '300 seconds'::interval 
    AND query ~* '^(insert|update|delete|select)';
    "
    <Result>
      Type "counter"
      InstancePrefix "pg_slow_queries"
      ValuesFrom "count"
    </Result>
  </Query>
  <Query txn_wraparound>
    Statement "
    SELECT age(datfrozenxid) as txn_wrap_age FROM pg_database ;
    "
    <Result>
      Type "counter"
      InstancePrefix "txn_wraparound"
      ValuesFrom "txn_wrap_age"
    </Result>
  </Query>
  <Query wal_files>
    Statement "
    SELECT archived_count AS count, failed_count AS failed FROM pg_stat_archiver;
    "
    <Result>
      Type "gauge"
      InstancePrefix "pg_wal_count"
      ValuesFrom "count"
    </Result>
    <Result>
      Type "gauge"
      InstancePrefix "pg_wal_failed"
      ValuesFrom "failed"
    </Result>
  </Query>
  <Query avg_querytime>
    Statement "
    SELECT sum(total_time)/sum(calls) AS avg_querytime FROM collectd.get_stat_statements() ;
    "
    <Result>
      Type "gauge"
      InstancePrefix "pg_avg_querytime"
      ValuesFrom "avg_querytime"
    </Result>
  </Query>
  <Query scans>
    Statement "
    SELECT 
        sum(idx_scan) as index_scans, 
        sum(seq_scan) as seq_scans, 
        sum(idx_tup_fetch) as index_tup_fetch, 
        sum(seq_tup_read) as seq_tup_read
    FROM pg_stat_all_tables ;
    "
    <Result>
      Type "pg_scan"
      InstancePrefix "index"
      ValuesFrom "index_scans"
    </Result>
    <Result>
      Type "pg_scan"
      InstancePrefix "seq"
      ValuesFrom "seq_scans"
    </Result>
    <Result>
      Type "pg_scan"
      InstancePrefix "index_tup"
      ValuesFrom "index_tup_fetch"
    </Result>
    <Result>
      Type "pg_scan"
      InstancePrefix "seq_tup"
      ValuesFrom "seq_tup_read"
    </Result>
  </Query>
  <Query checkpoints>
    Statement "
    SELECT (checkpoints_timed + checkpoints_req) AS total_checkpoints 
    FROM pg_stat_bgwriter ;
    "
    <Result>
      Type "counter"
      InstancePrefix "pg_checkpoints"
      ValuesFrom "total_checkpoints"
    </Result>
  </Query>
  <Query slave_lag>
    Statement "
    SELECT CASE
      WHEN pg_is_in_recovery = 'false'
      THEN 0
      ELSE COALESCE(ROUND(EXTRACT(epoch FROM now() - pg_last_xact_replay_timestamp())),0)
    END AS seconds
    FROM pg_is_in_recovery();
    "
    <Result>
      Type "counter"
      InstancePrefix "slave_lag"
      ValuesFrom "seconds"
    </Result>
  </Query>
  <Database "test">
    Host "localhost"
    Port "5432"
    User "collectd"
    Password "XXX"
    Query "backends"
    Query "transactions"
    Query "queries"
    Query "table_states"
    Query "disk_io"
    Query "disk_usage"
    Query "query_plans"
    Query "connections"
    Query "slow_queries"
    Query "txn_wraparound"
    Query "locks"
    Query "slave_lag"
    Query "scans"
    Query "checkpoints"
    Query "avg_querytime"
    Query "wal_files"
    Query "seq_scans"
  </Database>
</Plugin>

自定義查詢

緩存命中率

<Query cache_hit_ratio>
  Statement "
  SELECT sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as cache_hit_ratio 
  FROM pg_statio_user_tables;
  "
  <Result>
    Type "gauge"
    InstancePrefix "cache_hit_ratio"
    ValuesFrom "cache_hit_ratio"
  </Result>
</Query>

索引命中率

<Query cache_idx_hit_ratio>
  Statement "
  SELECT (sum(idx_blks_hit) - sum(idx_blks_read)) / sum(idx_blks_hit) as cache_idx_hit_ratio 
  FROM pg_statio_user_indexes;
  "
  <Result>
    Type "gauge"
    InstancePrefix "cache_idx_hit_ratio"
    ValuesFrom "cache_idx_hit_ratio"
  </Result>
</Query>

TPS

<Query tps>
  Statement "
  SELECT datname, xact_commit + xact_rollback AS tps 
  FROM pg_catalog.pg_stat_database;
  "
  <Result>
    Type "derive"
    InstancePrefix "tps"
    InstancesFrom "datname"
    ValuesFrom "tps"
  </Result>
</Query>