Kubernetes系列之監控Metres-server實戰篇

時間 2019-11-14

標籤 kubernetes 系列監控 metres server 實戰简体版

原文原文鏈接

本次系列使用的所需部署包版本都使用的目前最新的或最新穩定版，安裝包地址請到公衆號內回覆【K8s實戰】獲取node

介紹nginx

從 Kubernetes 1.8 開始，Kubernetes 經過 Metrics API 獲取資源使用指標，例如容器 CPU 和內存使用狀況。這些度量指標能夠由用戶直接訪問，例如經過使用kubectl top 命令，或者使用集羣中的控制器。git

Metrics API: 經過 Metrics API，您能夠得到 node 或 pod 當前的資源使用狀況（可是不存儲）。github

大體是說它符合 kubernetes 的監控架構設計，受 heapster 項目啓發，而且比 heapster 優點在於：web

訪問不須要 apiserver 的代理機制，提供認證和受權等;json

不少集羣內組件依賴它（HPA,scheduler,kubectl top），所以它應該在集羣中默認運行;api

下載編排瀏覽器

[root@master-01 opt]# git clone https://github.com/kubernetes-incubator/metrics-server[root@master-01 opt]# cd metrics-server/deploy/1.8+/

建立metrics-server證書架構

建立簽名請求app

[root@master-01 1.8+]# cd /etc/kubernetes/ssl/[root@master-01 ssl]# cat > metrics-server-csr.json <<EOF{"CN": "aggregator","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hangzhou","L": "Hangzhou","O": "k8s","OU": "4Paradigm"}]}EOF

建立證書和私鑰

[root@master-01 ssl]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes metrics-server-csr.json|cfssljson -bare metrics-server2019/03/13 15:23:01 [INFO] generate received request2019/03/13 15:23:01 [INFO] received CSR2019/03/13 15:23:01 [INFO] generating key: rsa-20482019/03/13 15:23:01 [INFO] encoded CSR2019/03/13 15:23:01 [INFO] signed certificate with serial number 1026675139058810263099374133507485748972230132012019/03/13 15:23:01 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 ("Information Requirements").

同步證書

同步證書到master-二、master-03

[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.131:/etc/kubernetes/ssl/[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.132:/etc/kubernetes/ssl/

開啓聚合配置

修改kube-apiserver配置文件來支持metres-server，加入以下啓動參數來啓用aggregation layer：

--proxy-client-cert-file=/etc/kubernetes/ssl/metrics-server.pem \--proxy-client-key-file=/etc/kubernetes/ssl/metrics-server-key.pem \--runtime-config=api/all=true \--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \--requestheader-allowed-names=aggregator \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-group-headers=X-Remote-Group \--requestheader-username-headers=X-Remote-User

--requestheader-XXX、--proxy-client-XXX 是 kube-apiserver 的 aggregator layer 相關的配置參數，metrics-server & HPA 須要使用。

--requestheader-client-ca-file：用於簽名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的證書；在啓用了 metric aggregator 時使用。

若是 --requestheader-allowed-names 不爲空，則--proxy-client-cert-file 證書的 CN 必須位於 allowed-names 中，默認爲 aggregator。

注意：須要重啓三臺主控的kube-apiserver

若是不開啓聚合配置可能會報以下錯誤

這是由於沒用開啓聚合層

I0313 05:18:36.447202 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)Error: cluster doesn't provide requestheader-client-ca-file

修改編排文件

在metrics-server-deployment.yaml文件中containers字段下添加以下

command:- /metrics-server- --kubelet-insecure-tls- --kubelet-preferred-address-types=InternalIP

若是不添加稍後可能會報以下錯誤

E0313 08:23:41.193222 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.209.130: unable to fetch metrics from Kubelet 192.168.209.130 (192.168.209.130): Get https://192.168.209.130:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.131: unable to fetch metrics from Kubelet 192.168.209.131 (192.168.209.131): Get https://192.168.209.131:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.132: unable to fetch metrics from Kubelet 192.168.209.132 (192.168.209.132): Get https://192.168.209.132:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.133: unable to fetch metrics from Kubelet 192.168.209.133 (192.168.209.133): Get https://192.168.209.133:10250/stats/summary/: x509: certificate signed by unknown authority]

資料

https://github.com/kubernetes-incubator/metrics-server/issues/67

https://github.com/mattkelly/metrics-server/commit/bfddc174c783290cb86d6da2fe1182d53a3b9bd5

gcr.io的鏡像訪問不到的話須要將metrics-server-deployment.yaml中的鏡像替換爲：registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1

建立metrics-server

[root@master-01 1.8+]# kubectl apply -f ./

查看服務狀態

[root@master-01 1.8+]# kubectl get pod -nkube-systemNAME READY STATUS RESTARTS AGEmetrics-server-7c499cd69d-499js 1/1 Running 0 14s

測試功能

能夠看到資源使用信息採集到了

[root@master-01 1.8+]# kubectl top pods --all-namespacesNAMESPACE NAME CPU(cores) MEMORY(bytes) default dnstools-6b77cc4988-b5smz 0m 2Mi default nginx-7899755b7-rgdch 0m 2Mi default tests-mychart-7d84ff968f-76d2l 1m 3Mi default wordpress-test-mariadb-59cfd7c475-27chl 5m 116Mi default wordpress-test-wordpress-6fc9b7cc7f-b2nfq 4m 149Mi ingress-nginx grafana-69549786b6-d78nv 1m 30Mi ingress-nginx prometheus-server-8658d8cdbb-4qps2 1m 20Mi kube-system coredns-5d668bd598-4xxwn 3m 13Mi kube-system coredns-5d668bd598-f5g96 2m 9Mi kube-system kubernetes-dashboard-cb55bd5bd-gc84g 1m 19Mi kube-system metrics-server-84f9775b88-gh7x7 2m 16Mi kube-system tiller-deploy-87d7c6dfb-kxj7p 1m 9Mi monitoring kube-state-metrics-6f8967c6c5-nzkxp 2m 30Mi monitoring node-exporter-4n9wj 1m 8Mi monitoring node-exporter-5wtgw 0m 8Mi monitoring node-exporter-gdj8f 1m 11Mi monitoring node-exporter-p96zj 1m 9Mi monitoring prometheus-operator-795895d784-v569s 1m 10Mi [root@master-01 1.8+]# kubectl top nodesNAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.209.130 158m 7% 1965Mi 53% 192.168.209.131 103m 5% 1859Mi 50% 192.168.209.132 123m 6% 2152Mi 58% 192.168.209.133 38m 1% 1022Mi 27%

經過 kube-apiserver接口訪問

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes/

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/namespace/pods/

[root@master-01 1.8+]# curl -k https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods {"kind": "PodMetricsList","apiVersion": "metrics.k8s.io/v1beta1","metadata": {"selfLink": "/apis/metrics.k8s.io/v1beta1/pods"},"items": [{"metadata": {"name": "coredns-5d668bd598-f5g96","namespace": "kube-system","selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5d668bd598-f5g96","creationTimestamp": "2019-03-13T09:23:58Z"},"timestamp": "2019-03-13T09:23:17Z","window": "30s","containers": [{"name": "coredns","usage": {"cpu": "2073503n","memory": "9628Ki"}...............

瀏覽器訪問

這裏的30000端口是我把metrics-server端口改爲NodePort了。

好了，進行到這，metrics就部署完了，敬請期待後續分享，謝謝！

END

若是您以爲不錯，請別忘了轉發、分享、點贊讓更多的人去學習，您的舉手之勞，就是對小編最好的支持，很是感謝！