k8s集羣性能測試-運用kubemark

1、測試目標
一、集羣容量上限：Node總數和Pod總數兩個維度
二、性能瓶頸
三、服務延遲

k8s集羣性能測試-運用kubemark

[root@10-2-19-17 ~]# cd $GOPATH/src/k8s.io/kubernetes

[root@10-2-19-17 kubernetes]# ./hack/generate-bindata.sh
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes
Generated bindata file : test/e2e/generated/bindata.go has 13126 test/e2e/generated/bindata.go lines of lovely automated artifacts
No changes in generated bindata file: pkg/generated/bindata.go
~/gocode/src/k8s.io/kubernetes

[root@10-2-19-17 kubernetes]# make WHAT="test/e2e/e2e.test"
+++ [1127 17:07:39] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1127 17:07:39] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes/test/e2e/generated
~/gocode/src/k8s.io/kubernetes/test/e2e/generated
+++ [1127 17:07:40] Building go targets for linux/amd64:
    test/e2e/e2e.test

[root@10-2-19-17 kubernetes]# make ginkgo
+++ [1127 17:09:31] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1127 17:09:31] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes/test/e2e/generated
~/gocode/src/k8s.io/kubernetes/test/e2e/generated
+++ [1127 17:09:32] Building go targets for linux/amd64:
    vendor/github.com/onsi/ginkgo/ginkgo

[root@10-2-19-17 kubernetes]# export KUBECTL_PATH=/usr/bin/kubectl

建立kubeconfig

cp /etc/kubernetes/admin.conf  /root/.kube/
mv admin.conf config

執行performance test

[root@10-2-20-7 kubernetes]#  go run hack/e2e.go -v --test  --test_args="--host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]" >> logs/log.txt 
flag provided but not defined: -v
Usage of /tmp/go-build327486475/command-line-arguments/_obj/exe/e2e:
  -get
    	go get -u kubetest if old or not installed (default true)
  -old duration
    	Consider kubetest old if it exceeds this (default 24h0m0s)
2017/11/30 10:13:10 e2e.go:54: NOTICE: go run hack/e2e.go is now a shim for test-infra/kubetest
2017/11/30 10:13:10 e2e.go:55:   Usage: go run hack/e2e.go [--get=true] [--old=24h0m0s] -- [KUBETEST_ARGS]
2017/11/30 10:13:10 e2e.go:56:   The separator is required to use --get or --old flags
2017/11/30 10:13:10 e2e.go:57:   The -- flag separator also suppresses this message
2017/11/30 10:13:10 e2e.go:76: Calling kubetest -v --test --test_args=--host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]...
2017/11/30 10:13:10 util.go:422: Please use kubetest --provider=local (instead of deprecated KUBERNETES_PROVIDER=local)
2017/11/30 10:13:10 util.go:155: Running: ./cluster/kubectl.sh --match-server-version=false version
2017/11/30 10:13:10 util.go:157: Step './cluster/kubectl.sh --match-server-version=false version' finished in 235.717856ms
2017/11/30 10:13:10 util.go:155: Running: ./hack/e2e-internal/e2e-status.sh
Local doesn't need special preparations for e2e tests
2017/11/30 10:13:10 util.go:157: Step './hack/e2e-internal/e2e-status.sh' finished in 279.000773ms
2017/11/30 10:13:10 util.go:155: Running: ./hack/ginkgo-e2e.sh --host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]
Local doesn't need special preparations for e2e tests
2017/11/30 10:18:34 util.go:157: Step './hack/ginkgo-e2e.sh --host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]' finished in 5m23.832173249s
2017/11/30 10:18:34 e2e.go:80: Done
[root@10-2-20-7 kubernetes]#

輸出日誌：

...
[Finish:Performance]
[AfterEach] [k8s.io] Load capacity
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120
Nov 30 10:18:04.052: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-load-phvcx" for this suite.
Nov 30 10:18:14.161: INFO: namespace: e2e-tests-load-phvcx, resource: bindings, ignored listing per whitelist
STEP: Destroying namespace "e2e-tests-load-30-nodepods-1-4h4pc" for this suite.
Nov 30 10:18:34.339: INFO: namespace: e2e-tests-load-30-nodepods-1-4h4pc, resource: bindings, ignored listing per whitelist

??[SLOW TEST:181.733 seconds]
[k8s.io] Load capacity
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:650
  [Feature:Performance] should be able to handle 30 pods per node { ReplicationController} with 0 secrets and 0 daemons
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:265
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSNov 30 10:18:34.455: INFO: Running AfterSuite actions on all node
Nov 30 10:18:34.455: INFO: Running AfterSuite actions on node 1

Ran 2 of 591 Specs in 323.273 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 589 Skipped PASS

Ginkgo ran 1 suite in 5m23.72874212s
Test Suite Passed

測試過程當中會產生兩個namespace: 完成後會刪除

[root@10-2-20-7 e2e]# kubectl get namespace
NAME                           STATUS        AGE
default                        Active        36d
e2e-tests-density-30-1-vmjnw   Active        1m
e2e-tests-density-cbzmb        Active        1m

默認狀況下，在每一個測試結束時，它將刪除Kubemark Master上的命名空間和它下面的全部東西（例如事件，複製控制器），這會花費不少時間。大多數狀況下不須要這樣的工做：若是您在運行後刪除了Kubemark集羣run-e2e-tests.sh，你不關心命名空間的刪除性能，具體涉及到etcd; 有一個標誌，使您能夠避免名稱空間刪除：--delete-namespace=false。添加標誌應該讓你看到日誌：Found DeleteNamespace=false, skipping namespace deletion!

中途出現錯誤按ctrl+c 結束，用下面命令清理：

go run hack/e2e.go -v --down

注意：若是您在本地羣集上重複運行測試，則可能須要按期執行一些手動清理：

rm -rf /var/run/kubernetes #清除kube生成的憑據，有時陳舊的權限可能會致使問題。
sudo iptables -F #清除kube-proxy所留下的IP表規則。

參考：

 

https://supereagle.github.io/2017/03/09/kubemark/ 
https://www.tuicool.com/articles/BBbIBzq 

https://www.cnblogs.com/opama/p/5931515.html