Kebernetes 學習總結(13) K8s 資源監控
kubernetes新一代的監控模型由：核心指標流水線和第三方非核心監控流水線組成。核心指標流水線由kubelet、metric-server 以及由API-server提供的API組成；負責CPU累積使用率、內存實時使用率、POD資源佔用率、Container磁盤佔用率等。而第三方非核心監控流水線 負責從OS收集各類指標數據並提供給終端用戶、存儲系統、以及HPA等。
監控系統收集兩種指標： 資源指標與自定義指標。node
Metrics-server 是資源指標API 。它提供核心指標，包括CPU累積使用率、內存實時使用率、Pod 的資源佔用率及容器的磁盤佔用率。這些指標由kubelet、metrics-server以及由API server提供的。
Prometheus是自定義指標 的提供者。它收集的數據還須要通過kube-state-metrics轉換處理，再由 k8s-prometheus-adapter 輸出爲metrics-api 才能被 kubernetes cluster 所讀取。用於從系統收集各類指標數據，並通過處理提供給 終端用戶、存儲系統以及HPA，這些數據包括核心指標和許多非核心指標。
資源指標API 負責收集各類資源指標，但它須要擴展APIServer 。能夠利用 aggregator 將 metrics-server 與 APIServer進行聚合，達到擴展功能的效果。這樣 就能夠利用擴展的 API Server 功能(即資源指標API)進行收集 各類資源指標(1.8+支持)。kubectl top 、HPA等功能組件 必須依賴資源指標API （早期版本它們依賴heapster）。
HPA 根據CPU、Memory、IO、net connections等指標進行擴展或收縮(早期的heapster只能提供CPU、Memory指標)
1、metrics-server
是託管在kubernetes cluster上的一個Pod ，再由 kube-aggregator 將它和原API Server 進行聚合，達到擴展API 的效果。它是如今 kubectl top 、HPA的前提依賴。
部署metrics-server 以下：
參考 ：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-servernginx
[root@k8s-master-dev metric-v0.3]# cat metrics-server.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- deployments
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
group: metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.0
imagePullPolicy: IfNotPresent
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
volumeMounts:
- name: tmp-dir
mountPath: /tmp
---
apiVersion: v1
kind: Service
metadata:
name: metrics-server
namespace: kube-system
labels:
kubernetes.io/name: "Metrics-server"
spec:
selector:
k8s-app: metrics-server
ports:
- port: 443
protocol: TCP
targetPort: 443
[root@k8s-master-dev metric-v0.3]# kubectl apply -f metrics-server.yaml
[root@k8s-master-dev metric-v0.3]# cd
[root@k8s-master-dev ~]# kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
*metrics.k8s.io/v1beta1*
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
[root@k8s-master-dev ~]#
[root@k8s-master-dev ~]# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master-dev 299m 3% 1884Mi 11%
k8s-node1-dev 125m 1% 4181Mi 26%
k8s-node2-dev 66m 3% 2736Mi 17%
k8s-node3-dev 145m 1% 2686Mi 34%
[root@k8s-master-dev metric-v0.3]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
mongo-0 12m 275Mi
mongo-1 11m 251Mi
mongo-2 8m 271Mi
[root@k8s-master-dev metric-v0.3]#
當metrics-server部署完畢後，如上所示能夠查看到 metrics相關的API，而且可使用kubectl top 命令查看node或pod的資源佔用狀況 。
若是須要安裝最新版本能夠 git clone https://github.com/kubernetes-incubator/metrics-server.git
cd metrics-server/deploy/1.8+/
kubectl apply -f ./
若是發現metrics-server 的pod能夠正常啓動，但在執行kubectl top node時提示metrics-server 不可用，在執行 kubectl log metrics-server-* -n kube-system 時有錯誤提示，極可能是由於：resource-reader.yaml 文件中 ClusterRole 的rules中缺乏 namespaces 權限，以及 metrics-server-deployment.yaml文件中container下缺乏如下語句，以忽略tls認證。git
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
2、Prometheus
架構圖以下：
Prometheus 經過node_exporter獲取各Nodes的信息。 node_exporter它只負責節點級別的信息彙總，若是須要採集其它指標數據，就須要部署專用的exporter 。
Prometheus 經過 metrics-url 地址到各Pods獲取數據 。
prometheus 提供了一個Restful 風格的PromQL接口，可讓用戶輸入查詢表達式。但K8s的 API Server 沒法查詢其值 ，由於它們默認的數據格式不統一。數據須要kube-state-metrics組件將其處理、轉換，而後由k8s-prometheus-adapter組件讀取並聚合到API上，最後 kubernetes cluster 的API server 才能識別。
因此各節點須要部署node_exporter 組件，而後Prometheus從各節點的node_exporter上獲取infomation，而後就能夠經過 PromQL 查詢各類數據。這些數據的格式再由kube-state-metrics組件進行轉換，而後再由kube-prometheus-adapter組件將轉換後的數據輸出爲Custom metrics API ，並聚合到API上，以便用戶使用。
示意圖以下
部署Prometheus, 以下：
參考 ：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus
1) 定義名稱空間github
[root@k8s-master-dev prometheus]# cd k8s-prom/ [root@k8s-master-dev k8s-prom]# [root@k8s-master-dev k8s-prom]# ls k8s-prometheus-adapter namespace.yaml podinfo README.md kube-state-metrics node_exporter prometheus [root@k8s-master-dev k8s-prom]# cat namespace.yaml --- apiVersion: v1 kind: Namespace metadata: name: prom [root@k8s-master-dev k8s-prom]# kubectl apply -f namespace.yaml namespace/prom created
2) 部署node_exporterweb
[root@k8s-master-dev k8s-prom]# cd node_exporter/ [root@k8s-master-dev node_exporter]# ls node-exporter-ds.yaml node-exporter-svc.yaml [root@k8s-master-dev node_exporter]# vim node-exporter-ds.yaml [root@k8s-master-dev node_exporter]# kubectl apply -f ./ daemonset.apps/prometheus-node-exporter created service/prometheus-node-exporter created [root@k8s-master-dev node_exporter]# kubectl get pods -n prom NAME READY STATUS RESTARTS AGE prometheus-node-exporter-7729r 1/1 Running 0 17s prometheus-node-exporter-hhc7f 1/1 Running 0 17s prometheus-node-exporter-jxjcq 1/1 Running 0 17s prometheus-node-exporter-pswbb 1/1 Running 0 17s [root@k8s-master-dev node_exporter]# cd ..
3) 部署prometheusshell
[root@k8s-master-dev k8s-prom]# cd prometheus/ [root@k8s-master-dev prometheus]# ls prometheus-cfg.yaml prometheus-deploy.yaml prometheus-rbac.yaml prometheus-svc.yaml [root@k8s-master-dev prometheus]# kubectl apply -f ./ configmap/prometheus-config created deployment.apps/prometheus-server created clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/prometheus created [root@k8s-master-dev prometheus]# kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/prometheus-node-exporter-7729r 1/1 Running 0 1m pod/prometheus-node-exporter-hhc7f 1/1 Running 0 1m pod/prometheus-node-exporter-jxjcq 1/1 Running 0 1m pod/prometheus-node-exporter-pswbb 1/1 Running 0 1m pod/prometheus-server-65f5d59585-5fj6n 1/1 Running 0 33s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/prometheus NodePort 10.98.96.66 <none> 9090:30090/TCP 34s service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 1m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 4 4 4 4 4 <none> 1m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/prometheus-server 1 1 1 1 34s NAME DESIRED CURRENT READY AGE replicaset.apps/prometheus-server-65f5d59585 1 1 1 34s [root@k8s-master-dev prometheus]#
而後就能夠以PromQL的方式查詢數據了，以下所示：
4) 部署kube-state-metricsvim
[root@k8s-master-dev prometheus]# cd .. [root@k8s-master-dev k8s-prom]# cd kube-state-metrics/ [root@k8s-master-dev kube-state-metrics]# ls kube-state-metrics-deploy.yaml kube-state-metrics-rbac.yaml kube-state-metrics-svc.yaml [root@k8s-master-dev kube-state-metrics]# kubectl apply -f ./ deployment.apps/kube-state-metrics created serviceaccount/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created service/kube-state-metrics created [root@k8s-master-dev kube-state-metrics]# [root@k8s-master-dev kube-state-metrics]# kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/kube-state-metrics-58dffdf67d-j4jdv 0/1 Running 0 34s pod/prometheus-node-exporter-7729r 1/1 Running 0 3m pod/prometheus-node-exporter-hhc7f 1/1 Running 0 3m pod/prometheus-node-exporter-jxjcq 1/1 Running 0 3m pod/prometheus-node-exporter-pswbb 1/1 Running 0 3m pod/prometheus-server-65f5d59585-5fj6n 1/1 Running 0 2m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-state-metrics ClusterIP 10.108.165.171 <none> 8080/TCP 35s service/prometheus NodePort 10.98.96.66 <none> 9090:30090/TCP 2m service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 3m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 4 4 4 4 4 <none> 3m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/kube-state-metrics 1 1 1 0 35s deployment.apps/prometheus-server 1 1 1 1 2m NAME DESIRED CURRENT READY AGE replicaset.apps/kube-state-metrics-58dffdf67d 1 1 0 35s replicaset.apps/prometheus-server-65f5d59585 1 1 1 2m [root@k8s-master-dev kube-state-metrics]# cd ..
5) 部署prometheus-adapter
參考 ：https://github.com/DirectXMan12/k8s-prometheus-adapter/tree/master/deployapi
[root@k8s-master-dev k8s-prom]# cd k8s-prometheus-adapter/
[root@k8s-master-dev k8s-prometheus-adapter]# ls
custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml custom-metrics-apiserver-service.yaml
custom-metrics-apiserver-auth-reader-role-binding.yaml custom-metrics-apiservice.yaml
custom-metrics-apiserver-deployment.yaml custom-metrics-cluster-role.yaml
custom-metrics-apiserver-deployment.yaml.bak custom-metrics-config-map.yaml
custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml custom-metrics-resource-reader-cluster-role.yaml
custom-metrics-apiserver-service-account.yaml hpa-custom-metrics-cluster-role-binding.yaml
[root@k8s-master-dev k8s-prometheus-adapter]# grep secretName custom-metrics-apiserver-deployment.yaml
secretName: cm-adapter-serving-certs
[root@k8s-master-dev k8s-prometheus-adapter]# cd /etc/kubernetes/pki/
[root@k8s-master-dev pki]# (umask 077; openssl genrsa -out serving.key 2048)
Generating RSA private key, 2048 bit long modulus
.....................+++
..........+++
e is 65537 (0x10001)
[root@k8s-master-dev pki]#
[root@k8s-master-dev pki]# openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
[root@k8s-master-dev pki]# openssl x509 -req -in serving.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out serving.crt -days 3650
Signature ok
subject=/CN=serving
Getting CA Private Key
[root@k8s-master-dev pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key -n prom
secret/cm-adapter-serving-certs created
[root@k8s-master-dev pki]# kubectl get secret -n prom
NAME TYPE DATA AGE
cm-adapter-serving-certs Opaque 2 9s
default-token-w4f44 kubernetes.io/service-account-token 3 8m
kube-state-metrics-token-dfcmf kubernetes.io/service-account-token 3 4m
prometheus-token-4lb78 kubernetes.io/service-account-token 3 6m
[root@k8s-master-dev pki]#
[root@k8s-master-dev pki]# cd -
/root/manifests/prometheus/k8s-prom/k8s-prometheus-adapter
[root@k8s-master-dev k8s-prometheus-adapter]# ls custom-metrics-config-map.yaml
custom-metrics-config-map.yaml
[root@k8s-master-dev k8s-prometheus-adapter]# cat custom-metrics-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: prom
data:
config.yaml: |
rules:
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters: []
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[5m]))
by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_seconds_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[5m]))
by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)$
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_total$
resources:
template: <<.Resource>>
name:
matches: ""
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_seconds_total
resources:
template: <<.Resource>>
name:
matches: ^(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters: []
resources:
template: <<.Resource>>
name:
matches: ^(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
[root@k8s-master-dev k8s-prometheus-adapter]# grep namespace custom-metrics-apiserver-deployment.yaml
namespace: prom
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl apply -f ./
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
deployment.apps/custom-metrics-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
serviceaccount/custom-metrics-apiserver created
service/custom-metrics-apiserver created
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
configmap/adapter-config created
clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl get cm -n prom
NAME DATA AGE
adapter-config 1 21s
prometheus-config 1 21m
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl get all -n prom
NAME READY STATUS RESTARTS AGE
pod/custom-metrics-apiserver-65f545496-2hfvb 1/1 Running 0 40s
pod/kube-state-metrics-58dffdf67d-j4jdv 0/1 Running 0 20m
pod/prometheus-node-exporter-7729r 1/1 Running 0 23m
pod/prometheus-node-exporter-hhc7f 1/1 Running 0 23m
pod/prometheus-node-exporter-jxjcq 1/1 Running 0 23m
pod/prometheus-node-exporter-pswbb 1/1 Running 0 23m
pod/prometheus-server-65f5d59585-5fj6n 1/1 Running 0 22m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/custom-metrics-apiserver ClusterIP 10.100.7.28 <none> 443/TCP 41s
service/kube-state-metrics ClusterIP 10.108.165.171 <none> 8080/TCP 20m
service/prometheus NodePort 10.98.96.66 <none> 9090:30090/TCP 22m
service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 23m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-node-exporter 4 4 4 4 4 <none> 23m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/custom-metrics-apiserver 1 1 1 1 42s
deployment.apps/kube-state-metrics 1 1 1 0 20m
deployment.apps/prometheus-server 1 1 1 1 22m
NAME DESIRED CURRENT READY AGE
replicaset.apps/custom-metrics-apiserver-65f545496 1 1 1 42s
replicaset.apps/kube-state-metrics-58dffdf67d 1 1 0 20m
replicaset.apps/prometheus-server-65f5d59585 1 1 1 22m
[root@k8s-master-dev k8s-prometheus-adapter]#
[root@k8s-master-dev k8s-prometheus-adapter]# kubectl api-versions | grep custom
custom.metrics.k8s.io/v1beta1
[root@k8s-master-dev k8s-prometheus-adapter]#
3、Grafana
grafana 是一個可視化面板，有着很是漂亮的圖表和佈局展現，功能齊全的度量儀表盤和圖形編輯器，支持 Graphite、zabbix、InfluxDB、Prometheus、OpenTSDB、Elasticsearch 等做爲數據源，比 Prometheus 自帶的圖表展現功能強大太多，更加靈活，有豐富的插件，功能更增強大。(使用promQL語句查詢出了一些數據，而且在 Prometheus 的 Dashboard 中進行了展現，可是明顯能夠感受到 Prometheus 的圖表功能相對較弱，因此通常會使用第三方的工具展現這些數據，例Grafana)
部署Grafana
參考 ：https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb架構
[root@k8s-master-dev prometheus]# ls
grafana k8s-prom
[root@k8s-master-dev prometheus]# cd grafana/
[root@k8s-master-dev grafana]# head -11 grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: monitoring-grafana
namespace: prom
spec:
replicas: 1
selector:
matchLabels:
task: monitoring
k8s-app: grafana
[root@k8s-master-dev grafana]# tail -2 grafana.yaml
k8s-app: grafana
type: NodePort
[root@k8s-master-dev grafana]# kubectl apply -f grafana.yaml
deployment.apps/monitoring-grafana created
service/monitoring-grafana created
[root@k8s-master-dev grafana]# kubectl get pods -n prom
NAME READY STATUS RESTARTS AGE
custom-metrics-apiserver-65f545496-2hfvb 1/1 Running 0 13m
kube-state-metrics-58dffdf67d-j4jdv 1/1 Running 0 32m
monitoring-grafana-ffb4d59bd-w9lg9 0/1 Running 0 8s
prometheus-node-exporter-7729r 1/1 Running 0 35m
prometheus-node-exporter-hhc7f 1/1 Running 0 35m
prometheus-node-exporter-jxjcq 1/1 Running 0 35m
prometheus-node-exporter-pswbb 1/1 Running 0 35m
prometheus-server-65f5d59585-5fj6n 1/1 Running 0 34m
[root@k8s-master-dev grafana]# kubectl get svc -n prom
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
custom-metrics-apiserver ClusterIP 10.100.7.28 <none> 443/TCP 13m
kube-state-metrics ClusterIP 10.108.165.171 <none> 8080/TCP 32m
monitoring-grafana NodePort 10.100.131.108 <none> 80:42690/TCP 22s
prometheus NodePort 10.98.96.66 <none> 9090:30090/TCP 34m
prometheus-node-exporter ClusterIP None <none> 9100/TCP 35m
[root@k8s-master-dev grafana]#
Grafana的使用，默認用戶名密碼都是admin，登陸後首先添加數據源 (若是登陸grafana web 時不用輸入用戶名、密碼便可操做，說明在grafana.yml 文件中的GF_AUTH_ANONYMOUS_ENABLED 項設置了true，致使匿名用戶以admin的角色登陸；將其更改成 false，而後再次kubectl apply -f grafana.yml 便可解決 )
指定Prometheus 的數據源從哪一個PromQL URL獲取：
而後導入Dashboard (Dashboard能夠在https://grafana.com/dashboards下載)
(補充) 筆者在grafana官網下載了k8s 相關的dashboard 以下所示：
而後將下載的k8s cluster summary 再導入到本身環境的grafana中，效果以下所示：
若是對dashboard不滿意，能夠自行建立或修改Dashboard. app
(補充)使用ingress 代理promethemus和grafana :
[root@k8s-master1-dev ~]# cat ingress-rule-monitor-svc.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-rule-monitor
namespace: prom
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8, 192.168.0.0/16"
spec:
rules:
- host: grafana-devel.domain.cn
http:
paths:
- path:
backend:
serviceName: monitoring-grafana
servicePort: 80
- host: prometheus-devel.domain.cn
http:
paths:
- path:
backend:
serviceName: prometheus
servicePort: 9090
# kubectl apply -f ingress-rule-monitor-svc.yaml
相關文章
- 1. Kebernetes 學習總結(14) K8s Horizontal Pod Autoscaling
- 2. Kebernetes 學習總結(6) Volumes
- 3. Kebernetes 學習總結(8) statefulset
- 4. Kebernetes 學習總結(15) K8s多master
- 5. Kebernetes 學習總結(10) Dashboard
- 6. Kebernetes 學習總結(5) Ingress
- 7. Kebernetes 學習總結(7) configmap 和 secret
- 8. Prometheus 監控K8S集羣資源監控
- 9. Kebernetes 學習總結(11) K8s 網絡模型分析(2)
- 10. Kebernetes 學習總結(11) K8s 網絡模型-canal策略控制(3)
- 更多相關文章...
- • Docker 資源彙總 - Docker教程
- • XML 總結 下一步學習什麼呢？ - XML 教程
- • Tomcat學習筆記（史上最全tomcat學習筆記）
- • Kotlin學習（一）基本語法
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. CVPR 2020 論文大盤點-光流篇
- 2. Photoshop教程_ps中怎麼載入圖案？PS圖案如何導入？
- 3. org.pentaho.di.core.exception.KettleDatabaseException:Error occurred while trying to connect to the
- 4. SonarQube Scanner execution execution Error --- Failed to upload report - 500: An error has occurred
- 5. idea 導入源碼包
- 6. python學習 day2——基礎學習
- 7. 3D將是頁遊市場新賽道？
- 8. osg--交互
- 9. OSG-交互
- 10. Idea、spring boot 圖片(pgn顯示、jpg不顯示)解決方案
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. Kebernetes 學習總結(14) K8s Horizontal Pod Autoscaling
- 2. Kebernetes 學習總結(6) Volumes
- 3. Kebernetes 學習總結(8) statefulset
- 4. Kebernetes 學習總結(15) K8s多master
- 5. Kebernetes 學習總結(10) Dashboard
- 6. Kebernetes 學習總結(5) Ingress
- 7. Kebernetes 學習總結(7) configmap 和 secret
- 8. Prometheus 監控K8S集羣資源監控
- 9. Kebernetes 學習總結(11) K8s 網絡模型分析(2)
- 10. Kebernetes 學習總結(11) K8s 網絡模型-canal策略控制(3)