Prometheus 簡單使用及 exporter 開發

安裝路徑 Prometheus /usr/local/devops/prometheus

• 添加用戶組groupadd prometheus
• 添加用戶 useradd -g prometheus -m -d /var/lib/prometheus -s /sbin/nologin prometheus
• 建立服務 vim /etc/systemd/system/prometheus.service

[Unit]
Description=prometheus
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/devops/prometheus/prometheus --config.file=/usr/local/devops/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus
Restart=on-failure
[Install]
WantedBy=multi-user.target

• 下載 node_exporter 並解壓 vim /etc/systemd/system/node_exporter.service

[Unit]
Description=node_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/devops/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target

Node Exporter默認的抓取地址爲http://IP:9100/metrics, 在 prometheus.yml 文件中添加

- job_name: 'node1-metrics'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: node1

重啓 Prometheus 服務

prometheus 採用 pull 發方式採集數據，因此 exporter 須要按照 Prometheus 採集數據的格式暴露出須要採集的數據, 這個數據經過 curl ip:port/metrics 能夠看獲得，如：
curl 127.0.0.1:9100/metrics

# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.55175897068e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.16535296e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter

以第一行來舉例：# 號表示註釋說明 annotation，process_start_time_seconds 爲 metric name，後邊的就是對應的值

Prometheus的Client Library提供度量的四種基本類型包括:

• Counter 計數器
• Gauge 儀表盤
• Histogram 直方圖
• Summary 概要

當訪問Exporter的/metrics API地址時咱們能夠看到相似於一下返回值，其中HELP用於說明度量類型，TYPE用於數據類型說明。

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 255.477922

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 312.0

Counter

Counter類型比如計數器，用於統計相似於：CPU時間，API訪問總次數，異常發生次數等等場景。這些指標的特色就是增長不減小。
所以當咱們須要統計CPU的使用率時，咱們須要使用rate()函數計算該Counter在過去一段時間內在每個時間序列上的每秒的平均增加率

Gauge

Gauge類型，英文直譯的話叫「計量器」，可是和Counter的翻譯太相似了，所以我我的更喜歡使用」儀表盤「這個稱呼。儀表盤的特色就是數值是能夠增長或者減小的。所以Gauge適合用於如：當前內存使用率，當前CPU使用率，當前溫度，當前速度等等一系列的監控指標。

Histogram

Histogram 柱狀圖這個比較直接,更多的是用於統計一些數據分佈的狀況，用於計算在必定範圍內的分佈狀況，同時還提供了度量指標值的總和。

Summary

Summary摘要和Histogram柱狀圖比較相似，主要用於計算在必定時間窗口範圍內度量指標對象的總數以及全部對量指標值的總和。

用 php 寫一個 psr-15 middleware 的 prometheus exporter

安裝prometheus php client

composer require jimdo/prometheus_client_php

class Prometheus implements MiddlewareInterface
{

    /**
     * @param ServerRequestInterface $request
     * @param RequestHandlerInterface $handler
     * @return ResponseInterface
     * @throws \Prometheus\Exception\MetricsRegistrationException
     */
    public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
    {

        $start = microtime( true);
        $uri = $request->getUri()->getPath();
        $registry = new CollectorRegistry(new Redis());
        // export
        if ($uri === '/metrics') {
            $render = new RenderTextFormat();
            $metrics = $render->render($registry->getMetricFamilySamples());
            $response = new TextResponse($metrics, 200);
//            $response->withHeader('Content-type',  RenderTextFormat::MIME_TYPE);
            return $response;
        }

        $response = $handler->handle($request);
        $end = microtime(true);
        $duration = ($end - $start);
        $statusCode = $response->getStatusCode();
        $context = $request->getAttribute('context');
        $routes = $context->getRoutes();
        $method = $request->getMethod();
        $labels = ['status_code', 'method', 'route'];
        foreach ($routes as $route) {
            $labelValues = [$method, $statusCode, $route];
            $counter = $registry->registerCounter(
                'knight',
                'knight_request_total', 'Total number of HTTP requests',
                $labels
            );

            $counter->inc($labelValues);
            $histogram = $registry->registerHistogram(
                'knight',
                'knight_request_duration_seconds',
                'duration histogram of http responses',
                $labels,
                [0.005, 0.05, 0.1, 0.5, 1.5, 10]
            );
            
            $histogram->observe($duration, $labelValues);
        }


        return $response;
    }
}

curl https://blog.sangsay.com/api/metrics

response:

# HELP knight_request_duration_seconds duration histogram of http responses
# TYPE knight_knight_request_duration_seconds histogram
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="0.005"} 0
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="0.05"} 4
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="0.1"} 4
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="0.5"} 4
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="1.5"} 4
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="10"} 4
knight_knight_request_duration_seconds_bucket{status_code="GET",method="200",route="/posts",le="+Inf"} 4
knight_knight_request_duration_seconds_count{status_code="GET",method="200",route="/posts"} 4
knight_knight_request_duration_seconds_sum{status_code="GET",method="200",route="/posts"} 0.0942027568817144
# HELP knight_knight_request_total Total number of HTTP requests
# TYPE knight_knight_request_total counter
knight_knight_request_total{status_code="GET",method="200",route="/posts"} 4