在k8s上部署日誌系統elfk
日誌系統elfk
前言
通過上週的技術預研，在本週一經過開會研究，根據公司的現有業務流量和技術棧，決定選擇的日誌系統方案爲：elasticsearch(es)+logstash(lo)+filebeat(fi)+kibana(ki)組合。es選擇使用aliyun提供的es，lo&fi選擇本身部署，ki是阿里雲送的。由於申請ecs須要必定的時間，暫時選擇部署在測試&生產環境(吐槽一下，我司測試和生產公用一套k8s而且託管與aliyun......)。用時一天(前期有部署的差很少過)完成在kubernetes上部署完成elfk(先部署起來再說，優化什麼的後期根據須要再搞)。node
組件簡介
es 是一個實時的、分佈式的可擴展的搜索引擎，容許進行全文、結構化搜索，它一般用於索引和搜索大量日誌數據，也可用於搜索許多不一樣類型的文。linux
lo 主要的有點就是它的靈活性，主要由於它有不少插件，詳細的文檔以及直白的配置格式讓它能夠在多種場景下應用。咱們基本上能夠在網上找到不少資源，幾乎能夠處理任何問題。nginx
做爲 Beats 家族的一員，fi 是一個輕量級的日誌傳輸工具，它的存在正彌補了 lo 的缺點fi做爲一個輕量級的日誌傳輸工具能夠將日誌推送到中心lo。web
ki是一個分析和可視化平臺，它能夠瀏覽、可視化存儲在es集羣上排名靠前的日誌數據，並構建儀表盤。ki結合es操做簡單集成了絕大多數es的API，是專業的日誌展現應用。docker
數據採集流程圖
日誌流向：logs_data---> fi ---> lo ---> es---> ki。shell
logs_data經過fi收集日誌，輸出到lo，經過lo作一些過濾和修改以後傳送到es數據庫，ki讀取es數據庫作分析。數據庫
部署
根據我司的實際集羣情況，此文檔部署將徹底還原日誌系統的部署狀況。
api
在本地MAC安裝kubectl鏈接aliyun託管k8s
在客戶端(隨便本地一臺虛機上)安裝和託管的k8s同樣版本的kubectlapp
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.8/bin/linux/amd64/kubectl chmod +x ./kubectl mv ./kubectl /usr/local/bin/kubectl 將阿里雲託管的k8s的kubeconfig 複製到$HOME/.kube/config 目錄下，注意用戶權限的問題
部署ELFK
申請一個名稱空間(通常一個項目一個名稱空間)。curl
# cat kube-logging.yaml apiVersion: v1 kind: Namespace metadata: name: loging
部署es。網上找個差很少的資源清單，根據本身的需求進行適當的修改，運行，出錯就根據日誌進行再修改。
# cat elasticsearch.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-class
namespace: loging
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
# Supported policies: Delete, Retain
reclaimPolicy: Delete
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: datadir1
namespace: logging
labels:
type: local
spec:
storageClassName: local-class
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/data1"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: loging
spec:
serviceName: elasticsearch
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:7.3.1
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: "discovery.type"
value: "single-node"
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-class"
resources:
requests:
storage: 5Gi
---
kind: Service
apiVersion: v1
metadata:
name: elasticsearch
namespace: loging
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
clusterIP: None
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
部署ki。由於根據數據採集流程圖，ki是和es結合的，配置相對簡單。
# cat kibana.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kibana
namespace: loging
labels:
k8s-app: kibana
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kibana
template:
metadata:
labels:
k8s-app: kibana
spec:
containers:
- name: kibana
image: kibana:7.3.1
resources:
limits:
cpu: 1
memory: 500Mi
requests:
cpu: 0.5
memory: 200Mi
env:
- name: ELASTICSEARCH_HOSTS
#注意value是es的services,由於es是有狀態，用的無頭服務，因此鏈接的就不只僅是pod的名字了
value: http://elasticsearch:9200
ports:
- containerPort: 5601
name: ui
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: loging
spec:
ports:
- port: 5601
protocol: TCP
targetPort: ui
selector:
k8s-app: kibana
配置ingress-controller。由於我司用的是阿里雲託管的k8s自帶的nginx-ingress，而且配置了強制轉換https。因此kibana-ingress也要配成https。
# openssl genrsa -out tls.key 2048 # openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=kibana.test.realibox.com # kubectl create secret tls kibana-ingress-secret --cert=tls.crt --key=tls.key
kibana-ingress配置以下。提供兩種，一種是https，一種是http。
https:
# cat kibana-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana
namespace: loging
spec:
tls:
- hosts:
- kibana.test.realibox.com
secretName: kibana-ingress-secret
rules:
- host: kibana.test.realibox.com
http:
paths:
- path: /
backend:
serviceName: kibana
servicePort: 5601
http:
# cat kibana-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana
namespace: logging
spec:
rules:
- host: kibana.test.realibox.com
http:
paths:
- path: /
backend:
serviceName: kibana
servicePort: 5601
部署lo。由於lo的做用是對fi收集到的日誌進行過濾，須要根據不一樣的日誌作不一樣的處理，因此可能要常常性的進行改動，要進行解耦。因此選擇以configmap的形式進行掛載。
# cat logstash.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: logstash
namespace: loging
spec:
replicas: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: elastic/logstash:7.3.1
volumeMounts:
- name: config
mountPath: /opt/logstash/config/containers.conf
subPath: containers.conf
command:
- "/bin/sh"
- "-c"
- "/opt/logstash/bin/logstash -f /opt/logstash/config/containers.conf"
volumes:
- name: config
configMap:
name: logstash-k8s-config
---
apiVersion: v1
kind: Service
metadata:
labels:
app: logstash
name: logstash
namespace: loging
spec:
ports:
- port: 8080
targetPort: 8080
selector:
app: logstash
type: ClusterIP
# cat logstash-config.yaml
---
apiVersion: v1
kind: Service
metadata:
labels:
app: logstash
name: logstash
namespace: loging
spec:
ports:
- port: 8080
targetPort: 8080
selector:
app: logstash
type: ClusterIP
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-k8s-config
namespace: loging
data:
containers.conf: |
input {
beats {
port => 8080 #filebeat鏈接端口
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"] #es的service
index => "logstash-%{+YYYY.MM.dd}"
}
}
注意：修改configmap 至關於修改鏡像。必須從新apply 應用資源清單才能生效。根據數據採集流程圖，lo的數據由fi流入，流向es。
部署fi。fi的主要做用是進行日誌的採集，而後將數據交給lo。
# cat filebeat.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: loging
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# hints.enabled: true
output.logstash:
hosts: ['${LOGSTASH_HOST:logstash}:${LOGSTASH_PORT:8080}'] #流向lo
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: loging
labels:
app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: loging
labels:
app: filebeat
spec:
selector:
matchLabels:
app: filebeat
template:
metadata:
labels:
app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: elastic/filebeat:7.3.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env: #注入變量
- name: LOGSTASH_HOST
value: logstash
- name: LOGSTASH_PORT
value: "8080"
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: loging
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: loging
labels:
app: filebeat
---
至此完成在k8s上部署es+lo+fi+ki ，進行簡單驗證。
驗證
查看svc、pod、ingress信息
# kubectl get svc,pods,ingress -n loging NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 151m service/kibana ClusterIP xxx.168.239.2xx <none> 5601/TCP 20h service/logstash ClusterIP xxx.168.38.1xx <none> 8080/TCP 122m NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 151m pod/filebeat-24zl7 1/1 Running 0 118m pod/filebeat-4w7b6 1/1 Running 0 118m pod/filebeat-m5kv4 1/1 Running 0 118m pod/filebeat-t6x4t 1/1 Running 0 118m pod/kibana-689f4bd647-7jrqd 1/1 Running 0 20h pod/logstash-76bc9b5f95-qtngp 1/1 Running 0 122m NAME HOSTS ADDRESS PORTS AGE ingress.extensions/kibana kibana.test.realibox.com xxx.xx.xx.xxx 80, 443 19h
web配置
配置索引
發現
至此算是簡單完成。後續須要不斷優化，不過那是後事了。
問題總結
這應該算是第一次親自在測試&生產環境部署應用了，並且是本身很不熟悉的日子系統，遇到了不少問題，須要總結。
- 如何調研一項技術棧；
- 如何選定方案；
- 由於網上幾乎沒有找到相似的方案(也不曉得別的公司是怎麼搞的，反正網上找不到有效的可能借鑑的)。須要本身根據不一樣的文檔總結嘗試；
- 一個組件的標籤儘量一致；
- 如何查看公司是否作了端口限制和https強制轉換；
- 遇到IT的事必定要看日誌，這點很重要，日誌能夠解決絕大多數問題；
- 一我的再怎麼整也會忽略一些點，本身先嚐試而後請教朋友，共同進步。
- 項目先上線再說別的，目前是這樣，一件事又百分之20的把握就能夠去作了。百分之80再去作就沒啥意思了。
- 自學重點學的是理論，公司才能學到操做。
相關文章
- 1. ELK日誌系統部署
- 2. 在 k8s 上部署 mysql operator
- 3. Jenkins在k8s上部署
- 4. elk日誌系統安裝部署
- 5. 部署elk日誌分析系統
- 6. centos7 部署 ELK 日誌系統
- 7. CentOS7.4部署ELK日誌分析系統
- 8. ELK日誌分析系統部署
- 9. ELK日誌系統淺析與部署
- 10. k8s系列---EFK日誌系統
- 更多相關文章...
- • Maven 自動化部署 - Maven教程
- • 操作系統（OS）平臺 統計 - 瀏覽器信息
- • Docker容器實戰(七) - 容器眼光下的文件系統
- • 使用阿里雲OSS+CDN部署前端頁面與加速靜態資源
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. ELK日誌系統部署
- 2. 在 k8s 上部署 mysql operator
- 3. Jenkins在k8s上部署
- 4. elk日誌系統安裝部署
- 5. 部署elk日誌分析系統
- 6. centos7 部署 ELK 日誌系統
- 7. CentOS7.4部署ELK日誌分析系統
- 8. ELK日誌分析系統部署
- 9. ELK日誌系統淺析與部署
- 10. k8s系列---EFK日誌系統