kubernetes 1.14安裝部署EFK日誌收集系統

時間 2020-07-21

標籤 kubernetes 1.14 安裝部署 efk 日誌收集系統简体版

原文原文鏈接

簡單介紹：html

EFK 組合插件是k8s項目的一個日誌解決方案，它包括三個組件：Elasticsearch, Fluentd, Kibana。相對於ELK這樣的架構，k8s官方推行了EFK，可能Fluentd相對於Logstash更加輕量級吧。node

Elasticsearch 是日誌存儲和日誌搜索引擎，Fluentd 負責把k8s集羣的日誌發送給 Elasticsearch, Kibana 則是可視化界面查看和檢索存儲在 Elasticsearch 的數據。linux

GitHub官網：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearchgit

特別補充：

Kubernetes官方推薦的EFK組合，其實對於咱們大多數公司來講並不合適，其中一點緣由就是這個組合須要應用日誌打印到標準輸出。github

推薦作法：仍是使用傳統的ELK組合，ELK能夠部署在K8S集羣以外，日誌寫入指定存儲目錄文件中，並採用filebeat方式採集。docker

測試環境：後端

System	Hostname	IP
CentOS 7.6	k8s-master	138.138.82.14
CentOS 7.6	k8s-node1	138.138.82.15
CentOS 7.6	k8s-node2	138.138.82.16

注：如下是簡單的安裝部署操做，實際用於生產環境的話還要根據不一樣的後端存儲進而修改yaml中的相應存儲路徑。api

操做步驟：瀏覽器

1. 獲取yaml文件架構

從github官網下載：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

或者下載整個tar包：https://storage.googleapis.com/kubernetes-release/release/v1.14.1/kubernetes-server-linux-amd64.tar.gz

解壓後yanl文件位置： kubernetes/cluster/addons/fluentd-elasticsearch/

複製yaml文件到另外一處進行修改操做：

[root@k8s-master ~]# mkdir efk [root@k8s-master ~]# cp kubernetes/cluster/addons/fluentd-elasticsearch/*.yaml efk/ [root@k8s-master ~]# cd efk/ [root@k8s-master efk]# ll 總用量 36 -rw-r--r-- 1 root root 382 5月 5 13:55 es-service.yaml -rw-r--r-- 1 root root 2906 5月 5 14:44 es-statefulset.yaml -rw-r--r-- 1 root root 16124 5月 5 13:55 fluentd-es-configmap.yaml -rw-r--r-- 1 root root 2702 5月 5 13:55 fluentd-es-ds.yaml -rw-r--r-- 1 root root 1052 5月 5 13:55 kibana-deployment.yaml -rw-r--r-- 1 root root 354 5月 5 13:55 kibana-service.yaml

2. 準備鏡像（在全部主機上）

注：由於被牆的緣故，加上鏡像文件比較大，啓動會費時，能夠提早單獨下載好所需鏡像。

①修改 es-statefulset.yaml （https://www.docker.elastic.co/#）

②修改 fluentd-es-ds.yaml

③無需修改 kibana-deployment.yaml

說明：關於kibana鏡像以-oss結尾的版本好奇（default release that contains the commercially licensed code. The release that contains only open source will have the -oss appended to it.）：

https://www.elastic.co/guide/en/kibana/6.0/docker.html

https://discuss.elastic.co/t/what-are-the-differences-between-the-kibana-oss-and-non-oss-build/152364/3

附上鏡像拉取文本：

~]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.2
~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch:v2.4.0
~]# docker pull docker.elastic.co/kibana/kibana-oss:6.6.1

View Code

3. 發佈yaml文件到k8s中

[root@k8s-master efk]# kubectl apply -f .

4. 查看各pod狀態

[root@k8s-master efk]# kubectl get po -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-node-b78m4                       1/1     Running   0          7d1h    138.138.82.16   k8s-node2    <none>           <none> calico-node-r5mlj                       1/1     Running   0          7d1h    138.138.82.14   k8s-master   <none>           <none> calico-node-z5tdh                       1/1     Running   0          7d1h    138.138.82.15   k8s-node1    <none>           <none> coredns-fb8b8dccf-6mgks                 1/1     Running   0          7d1h    192.168.0.7     k8s-master   <none>           <none> coredns-fb8b8dccf-cbtlx                 1/1     Running   0          7d1h    192.168.0.6     k8s-master   <none>           <none> elasticsearch-logging-0                 1/1     Running   0          10m     192.168.2.11    k8s-node2    <none>           <none> elasticsearch-logging-1                 1/1     Running   0          10m     192.168.1.17    k8s-node1    <none>           <none> etcd-k8s-master                         1/1     Running   0          7d1h    138.138.82.14   k8s-master   <none>           <none> fluentd-es-v2.4.0-fm9n2                 1/1     Running   0          10m     192.168.1.16    k8s-node1    <none>           <none> fluentd-es-v2.4.0-xjx6h                 1/1     Running   0          10m     192.168.2.10    k8s-node2    <none>           <none> kibana-logging-f4d99b69f-hc4h9          1/1     Running   0          10m     192.168.1.15    k8s-node1    <none>           <none> kube-apiserver-k8s-master               1/1     Running   0          7d1h    138.138.82.14   k8s-master   <none>           <none> kube-controller-manager-k8s-master      1/1     Running   1          7d1h    138.138.82.14   k8s-master   <none>           <none> kube-proxy-c9xd2                        1/1     Running   0          7d1h    138.138.82.14   k8s-master   <none>           <none> kube-proxy-fp2r2                        1/1     Running   0          7d1h    138.138.82.15   k8s-node1    <none>           <none> kube-proxy-lrsw7                        1/1     Running   0          7d1h    138.138.82.16   k8s-node2    <none>           <none> kube-scheduler-k8s-master               1/1     Running   1          7d1h    138.138.82.14   k8s-master   <none>           <none> kubernetes-dashboard-5f7b999d65-45vlk   1/1     Running   0          6d7h    192.168.1.14    k8s-node1    <none>           <none> metrics-server-7579f696d8-pgcc4         1/1     Running   0          6d22h   192.168.1.12    k8s-node1    <none>           <none>

順利的話，稍等一小會就成功running狀態了。若是有問題，就查看容器日誌，基本上能夠定位到全部錯誤緣由。

5. 查看kibana

留意kibana的日誌輸出，這裏給出個人實際一段：

[root@k8s-master efk]# kubectl logs -n kube-system kibana-logging-f4d99b69f-hc4h9 -f {"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["warning","config","deprecation"],"pid":1,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."} {"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["plugin","warning"],"pid":1,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"} {"type":"log","@timestamp":"2019-05-05T08:02:59Z","tags":["warning","elasticsearch","config","deprecation"],"pid":1,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:kibana@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:console@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:interpreter@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:metrics@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:timelion@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"} {"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"} {"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["warning"],"pid":1,"kibanaVersion":"6.6.1","nodes":[{"version":"6.6.2","http":{"publish_address":"192.168.2.11:9200"},"ip":"192.168.2.11"}],"message":"You're running Kibana 6.6.1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.6.2 @ 192.168.2.11:9200 (192.168.2.11)"} {"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."} {"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_1."} {"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_1."} {"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":1,"message":"Finished in 994ms."} {"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}

View Code

以上顯示了啓動成功的一段過程。

查看kibana地址

[root@k8s-master ~]# kubectl cluster-info |grep kibana Kibana is running at https://138.138.82.14:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy

在瀏覽器中輸入以上連接便可，效果以下：

EFK插件簡單部署到此結束，實際生產應用還需修改yaml文件相關參數。

結束.