上篇:html
Ingress 基本概念:node
部署 httpbin 服務,一樣,官方demo已經提供了該配置文件,執行以下命令應用便可:docker
[root@m1 ~]# kubectl apply -f /usr/local/istio-1.8.1/samples/httpbin/httpbin.yaml serviceaccount/httpbin created service/httpbin created deployment.apps/httpbin created [root@m1 ~]#
爲 httpbin 服務配置 Ingress 網關,定義 Ingress gateway,以下所示:json
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: httpbin-gateway spec: selector: istio: ingressgateway # use Istio default gateway implementation servers: - port: number: 80 name: http protocol: HTTP hosts: # 暴露給客戶端訪問的host,也就是訪問該host時纔會進入這個Gateway - "httpbin.example.com" EOF
而後定義 Virtual Service 配置路由規則並關聯該 Gateway:後端
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin spec: hosts: # host要與Gateway中的定義對應上 - "httpbin.example.com" gateways: # 經過名稱關聯指定的Gateway - httpbin-gateway http: - match: # 請求匹配規則,也能夠說是暴露哪些接口 - uri: prefix: /status - uri: prefix: /delay route: - destination: port: number: 8000 host: httpbin EOF
使用以下命令獲取 istio-ingressgateway 服務的實際請求地址和端口號(但服務的 EXTERNAL-IP 爲 pending 或 none 時採用此方式,詳見:官方文檔):api
[root@m1 ~]# kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}' 32482 [root@m1 ~]# kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}' 192.168.243.140 [root@m1 ~]#
接下來使用 curl
命令測試一下 httpbin 的接口可否正常訪問:安全
[root@m1 ~]# curl -s -I -HHost:httpbin.example.com "http://192.168.243.140:32482/status/200" HTTP/1.1 200 OK server: istio-envoy date: Tue, 22 Dec 2020 03:45:17 GMT content-type: text/html; charset=utf-8 access-control-allow-origin: * access-control-allow-credentials: true content-length: 0 x-envoy-upstream-service-time: 16 [root@m1 ~]#
訪問沒有被暴露的接口,此時返回404:bash
[root@m1 ~]# curl -s -I -HHost:httpbin.example.com "http://192.168.243.140:32482/headers" HTTP/1.1 404 Not Found date: Tue, 22 Dec 2020 03:47:15 GMT server: istio-envoy transfer-encoding: chunked [root@m1 ~]#
有進入網格的流量也就有從網格出去的流量,這種入口流量與出口流量也就是咱們常說的南北流量,在Istio中咱們能夠對網格的入口和出口流量進行管控。網絡
Istio中訪問外部服務的方法:架構
global.outboundTrafficPolicy.mode = ALLOW_ANY
(默認)Egress 概念:
在本小節,咱們將實踐建立一個 Egress 網關,讓內部服務(sleep)經過它訪問外部服務(httpbin.org),這兩個服務在前面的章節示例中都已經演示過了:
查看 istio-egressgateway 組件是否存在:
[root@m1 ~]# kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE istio-egressgateway-d84f95b69-dmpzf 1/1 Running 1 27h ...
確認 sleep 服務已處於正常運行狀態:
[root@m1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE ... sleep-854565cb79-gm6hj 2/2 Running 2 15h
爲 httpbin.org 這個外部服務定義 ServiceEntry:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: httpbin spec: hosts: - httpbin.org ports: - number: 80 name: http-port protocol: HTTP resolution: DNS EOF
確認建立成功:
[root@m1 ~]# kubectl get se NAME HOSTS LOCATION RESOLUTION AGE httpbin ["httpbin.org"] DNS 2s
定義 Egress gateway:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: istio-egressgateway spec: selector: istio: egressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - httpbin.org EOF
定義路由,將流量引導到 istio-egressgateway:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: vs-for-egressgateway spec: hosts: - httpbin.org gateways: - istio-egressgateway - mesh http: - match: # 針對內部服務的路由規則,會把全部內部的請求都指向egress網關這個節點 - gateways: - mesh port: 80 route: # 將請求路由到egress網關 - destination: host: istio-egressgateway.istio-system.svc.cluster.local # egress組件的dns名稱 subset: httpbin port: number: 80 weight: 100 - match: # 針對Egress網關的路由規則 - gateways: - istio-egressgateway port: 80 route: # 將Egress網關的請求指向最終的外部服務地址,即httpbin.org - destination: host: httpbin.org port: number: 80 weight: 100 --- apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: dr-for-egressgateway spec: host: istio-egressgateway.istio-system.svc.cluster.local # egress組件的dns名稱 subsets: - name: httpbin EOF
測試訪問 httpbin.org 服務的接口:
[root@m1 ~]# kubectl exec -it sleep-854565cb79-gm6hj -c sleep -- curl http://httpbin.org/ip { "origin": "172.22.152.252, 223.74.101.7" } [root@m1 ~]#
查看日誌驗證出口流量是否通過了Egress網關,輸出了以下日誌信息表明Egress網關配置成功,出口流量通過了該Egress網關:
[root@m1 ~]# kubectl logs -f istio-egressgateway-d84f95b69-dmpzf -n istio-system ... [2020-12-22T06:32:22.057Z] "GET /ip HTTP/2" 200 - "-" 0 47 835 834 "172.22.152.252" "curl/7.69.1" "88e2392e-9b43-9e5f-8007-82873cdd9701" "httpbin.org" "54.164.234.192:80" outbound|80||httpbin.org 172.22.78.138:55966 172.22.78.138:8080 172.22.152.252:35658 - -
此時 sleep 服務訪問外部服務的流程以下圖:
對於一個分佈式系統來講,出現網絡故障是在所不免的,所以如何提高系統彈性,提高系統在面對故障時的處理能力是分佈式架構很是重要的一個主題。其中,超時和重試是很是重要且經常使用的,用於提高系統彈性的機制。
基本概念
接下來咱們仍是經過Bookinfo這個應用來做爲演示,對其中的一些服務添加超時策略和重試策略。咱們會將請求指向 reviews 服務的 v2 版本,並在 ratings 服務中添加延遲設置,模擬一個故障出現的狀況,以此來驗證咱們設置的超時和重試策略是否生效:
首先,建立一個Virtual Service將請求路由到 reviews 服務的 v2 版本:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews spec: hosts: - reviews http: - route: - destination: host: reviews subset: v2 EOF
給 ratings 服務注入延遲,模擬故障:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: delay: percent: 100 fixedDelay: 2s route: - destination: host: ratings subset: v1 EOF
給 reviews 服務添加超時策略:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews spec: hosts: - reviews http: - route: - destination: host: reviews subset: v2 timeout: 1s # 添加超時策略 EOF
此時刷新應用頁面,能夠看到返回了錯誤信息:
將 reviews 服務的超時策略取消掉,而後給 ratings 服務添加劇試策略:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - fault: delay: percent: 100 fixedDelay: 5s route: - destination: host: ratings subset: v1 retries: attempts: 2 # 重試次數 perTryTimeout: 1s # 每次重試的間隔時間 EOF
查看 ratings 服務的 Sidecar 日誌,而後刷新應用頁面,正常狀況下從日誌輸出能夠看到重試了兩次請求:
[root@m1 ~]# kubectl logs -f ratings-v1-7d99676f7f-jhcv6 -c istio-proxy ... [2020-12-22T07:57:28.104Z] "GET /ratings/0 HTTP/1.1" 200 - "-" 0 48 0 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "c07ce12d-cddf-950d-93ae-f8fee93aeca6" "ratings:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:54592 172.22.152.250:9080 172.22.78.132:46554 outbound_.9080_.v1_.ratings.default.svc.cluster.local default [2020-12-22T07:57:31.108Z] "GET /ratings/0 HTTP/1.1" 200 - "-" 0 48 0 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "c07ce12d-cddf-950d-93ae-f8fee93aeca6" "ratings:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:54628 172.22.152.250:9080 172.22.78.132:42092 outbound_.9080_.v1_.ratings.default.svc.cluster.local default
配置選項:
本小節咱們實踐一下爲 httpbin 服務添加斷路器配置,而後經過負載測試工具來觸發熔斷。斷路器相關配置是在服務的 DestinationRule 中裏進行配置的,以下所示:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: httpbin spec: host: httpbin trafficPolicy: # 流量策略 connectionPool: # 定義鏈接池,艙壁模式,利用鏈接池來隔離資源 tcp: maxConnections: 1 # tcp請求容許的最大鏈接數爲1 http: # http每一個鏈接的最大請求設置爲1 http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: # 異常檢測 consecutiveErrors: 1 # 失敗的次數,達到該次數就觸發熔斷 interval: 1s # 熔斷的間隔時間 baseEjectionTime: 3m # 最小驅逐時間,實現指數級的退避策略 maxEjectionPercent: 100 # 最大可被驅逐的比例 EOF
安裝fortio,使用該負載測試工具來觸發熔斷:
[root@m1 ~]# kubectl apply -f /usr/local/istio-1.8.1/samples/httpbin/sample-client/fortio-deploy.yaml service/fortio created deployment.apps/fortio-deploy created [root@m1 ~]#
先嚐試發送單個請求,確認該工具可以正常工做:
[root@m1 ~]# export FORTIO_POD=$(kubectl get pods -lapp=fortio -o 'jsonpath={.items[0].metadata.name}') [root@m1 ~]# kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio curl -quiet http://httpbin:8000/get HTTP/1.1 200 OK server: envoy date: Tue, 22 Dec 2020 08:32:18 GMT content-type: application/json content-length: 622 access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 22 { "args": {}, "headers": { "Content-Length": "0", "Host": "httpbin:8000", "User-Agent": "fortio.org/fortio-1.11.3", "X-B3-Parentspanid": "918687527bfa9f4c", "X-B3-Sampled": "1", "X-B3-Spanid": "7c652cb91e0e3ad0", "X-B3-Traceid": "7c29c42837142e88918687527bfa9f4c", "X-Envoy-Attempt-Count": "1", "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=8452370e5dc510c7fd99321b642b4af3bd83bc832636112365a4a45e344d0875;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default" }, "origin": "127.0.0.1", "url": "http://httpbin:8000/get" } [root@m1 ~]#
沒問題後,經過以下命令進行併發壓測,併發數是3,執行30次:
[root@m1 ~]# kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get 08:36:00 I logger.go:127> Log level is now 3 Warning (was 2 Info) Fortio 1.11.3 running at 0 queries per second, 4->4 procs, for 30 calls: http://httpbin:8000/get Starting at max qps with 3 thread(s) [gomax 4] for exactly 30 calls (10 per thread + 0) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) 08:36:00 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503) Ended after 31.42646ms : 30 calls. qps=954.61 Aggregated Function Time : count 30 avg 0.0020163533 +/- 0.001477 min 0.000302125 max 0.005454188 sum 0.060490598 # range, mid point, percentile, count >= 0.000302125 <= 0.001 , 0.000651062 , 36.67, 11 > 0.001 <= 0.002 , 0.0015 , 43.33, 2 > 0.002 <= 0.003 , 0.0025 , 83.33, 12 > 0.003 <= 0.004 , 0.0035 , 86.67, 1 > 0.004 <= 0.005 , 0.0045 , 93.33, 2 > 0.005 <= 0.00545419 , 0.00522709 , 100.00, 2 # target 50% 0.00216667 # target 75% 0.00279167 # target 90% 0.0045 # target 99% 0.00538606 # target 99.9% 0.00544738 Sockets used: 15 (for perfect keepalive, would be 3) Jitter: false Code 200 : 17 (56.7 %) # 56.7 %的請求返回成功,狀態碼爲200 Code 503 : 13 (43.3 %) # 43.3 %的請求返回失敗,狀態碼爲503,說明觸發了熔斷 Response Header Sizes : count 30 avg 130.33333 +/- 114 min 0 max 230 sum 3910 Response Body/Total Sizes : count 30 avg 587.23333 +/- 302.8 min 241 max 852 sum 17617 All done 30 calls (plus 0 warmup) 2.016 ms avg, 954.6 qps [root@m1 ~]#
若是但願查看具體指標可使用以下命令:
$ kubectl exec "$FORTIO_POD" -c istio-proxy -- pilot-agent request GET stats | grep httpbin | grep pending
配置選項:
在配置好網絡(包括故障恢復策略)後,咱們可使用Istio的故障注入機制來測試應用程序的總體故障恢復能力。故障注入是一種將錯誤引入系統以確保系統可以承受並從錯誤條件中恢復的測試方法。
因此故障注入機制特別有用,能夠提早暴露一些故障恢復策略不兼容或限制性太強,從而可能致使的關鍵服務不可用的問題。故障注入在業界的發展和應用例子:
其實咱們在以前的小節中早已演示過了Istio的故障注入配置,在超時與重試的小節中,咱們就爲 ratings 服務注入過一個延遲故障:
使用以下命令將Bookinfo應用各個服務的路由信息設置到各自的 v1 版本:
[root@m1 ~]# kubectl apply -f /usr/local/istio-1.8.1/samples/bookinfo/networking/virtual-service-all-v1.yaml
而後將 reviews 服務的流量指向它的 v2 版本,由於只有 v2 和 v3 版本纔會調用 ratings 服務:
[root@m1 ~]# kubectl apply -f /usr/local/istio-1.8.1/samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
給 ratings 服務注入延遲故障:
[root@m1 ~]# kubectl apply -f /usr/local/istio-1.8.1/samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml
virtual-service-ratings-test-delay.yaml
文件的內容:
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: ratings spec: hosts: - ratings http: - match: - headers: # 經過header匹配指定的用戶名,目的是針對已登陸的指定用戶所產生的流量進行故障模擬 end-user: exact: jason fault: # 配置故障注入 delay: # 故障類型爲延遲故障 percentage: # 指定會受故障影響的流量比例 value: 100.0 fixedDelay: 7s # 延遲多長時間 route: - destination: host: ratings subset: v1 - route: - destination: host: ratings subset: v1
配置選項:
相信不少開發人員都遇到過這樣的問題,就是在開發/測試環境中運行良好的功能,一上線就出問題。即使作足了單元測試、集成測試,測試覆蓋率也很高,也依然會有這種問題存在,而且這類問題在開發/測試環境難以復現。
出現這種問題的一個主要緣由,是由於線上環境,特別是數據環境,好比說數據量、請求的併發量以及用戶使用數據的方式都與開發/測試環境很是的不同。因爲這種不一致性致使咱們很難在開發/測試環境中發現線上的問題。
那麼一個很是好的解決辦法,就是使用流量鏡像機制,將線上流量復刻一份到開發/測試環境中進行測試。
流量鏡像(Traffic mirroring,也稱爲Shadowing)是一個強大的概念,它容許開發團隊以儘量小的風險爲生產環境帶來更改。流量鏡像機制能夠將實時流量的副本發送到一個鏡像服務。對流量的鏡像發生在主服務的關鍵請求路徑以外。
接下來咱們實踐一下如何配置Istio中的流量鏡像機制,需求是將發送到 v1 版本的流量鏡像到 v2 版本。所以,咱們首先部署 httpbin 服務的 v1 和 v2 版本。v1 版本的配置以下:
apiVersion: apps/v1 kind: Deployment metadata: name: httpbin-v1 spec: replicas: 1 selector: matchLabels: app: httpbin version: v1 template: metadata: labels: app: httpbin version: v1 spec: containers: - image: docker.io/kennethreitz/httpbin imagePullPolicy: IfNotPresent name: httpbin command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"] ports: - containerPort: 80
部署 httpbin 服務的 v2 版本:
apiVersion: apps/v1 kind: Deployment metadata: name: httpbin-v2 spec: replicas: 1 selector: matchLabels: app: httpbin version: v2 template: metadata: labels: app: httpbin version: v2 spec: containers: - image: docker.io/kennethreitz/httpbin imagePullPolicy: IfNotPresent name: httpbin command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"] ports: - containerPort: 80
爲 httpbin 建立一個 Service 資源,讓 Pod 可以經過服務的方式暴露出來:
apiVersion: v1 kind: Service metadata: name: httpbin labels: app: httpbin spec: ports: - name: http port: 8000 targetPort: 80 selector: app: httpbin
爲 httpbin 服務建立默認的虛擬服務與目標規則:
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin spec: hosts: - httpbin http: - route: - destination: host: httpbin subset: v1 # 將請求指向v1版本 weight: 100 --- apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: httpbin spec: host: httpbin subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2
測試一下可否正常訪問到 httpbin 服務的接口:
[root@m1 ~]# export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) [root@m1 ~]# kubectl exec "${SLEEP_POD}" -c sleep -- curl -s http://httpbin:8000/headers { "headers": { "Accept": "*/*", "Content-Length": "0", "Host": "httpbin:8000", "User-Agent": "curl/7.69.1", "X-B3-Parentspanid": "86bf83bdc5d1be76", "X-B3-Sampled": "1", "X-B3-Spanid": "07b81e555626fe41", "X-B3-Traceid": "72d7ebc62e57ec7386bf83bdc5d1be76", "X-Envoy-Attempt-Count": "1", "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/default;Hash=a9722ea6836d43d677f56f8b39cddbec522311a27ea0baf25ea08b265303d4f3;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/sleep" } } [root@m1 ~]#
完成以上的準備工做後,咱們就能夠在虛擬服務中爲 httpbin 服務的 v2 版本配置對 v1 版本的流量鏡像:
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin spec: hosts: - httpbin http: - route: - destination: host: httpbin subset: v1 weight: 100 mirror: # 配置流量鏡像 host: httpbin # 指定鏡像服務,即複製到哪一個服務上 subset: v2 # 限定服務的版本,實現了將v1的流量複製到v2上的效果 mirror_percent: 100 # 流量複製的比例
嘗試請求 httpbin 服務的接口,因爲咱們配置了路由規則,該請求必定是被路由到 v1 版本上的:
[root@m1 ~]# kubectl exec "${SLEEP_POD}" -c sleep -- curl -s http://httpbin:8000/headers
此時觀察 v2 版本的日誌,從日誌輸出中能夠發現 v2 版本也接收到了相同的請求,表明咱們的流量鏡像配置生效了:
[root@m1 ~]# kubectl logs -f httpbin-v2-75d9447d79-rtc9x -c httpbin ... 127.0.0.1 - - [22/Dec/2020:10:05:56 +0000] "GET /headers HTTP/1.1" 200 591 "-" "curl/7.69.1"
配置選項: