Golang微服務實踐

時間 2019-11-29

標籤 golang 微服實踐欄目 Go 简体版

原文原文鏈接

本文經過一個完整的項目的示例，從Golang項目的結構、分層思想、依賴注入、錯誤處理、單元測試、服務治理、框架選擇等方面介紹Go語言項目的開發經驗.html

目的

提供一個完整的go語言項目編程示例
介紹Go開發應該遵照的規範
介紹編程思想在go語言中的應用
服務治理相關的框架、中間件介紹

示例項目

Github源碼go-project-samplelinux

快速開始

git clone https://github.com/sdgmf/go-project-sample.git
    cd go-project-sample
    make docker-compose
複製代碼

訪問接口： http://localhost:8080/product/1
consul: http://localhost:8500/
grafana: http://localhost:3000/
jaeger: http://localhost:16686/search
Prometheus: http://localhost:9090/graph

包結構

關於golang項目的包結構，Dave Chaney博客《Five suggestions for setting up a Go project》裏討論了package和command的包設計建議，還有一個社區廣泛承認的包結構規範project-layout。在這兩個兩篇文章的知道下，結合常見的互聯網微服務項目，我又細化了以下的項目結構。git

.
├── api
│   └── proto
├── build
│   ├── details
│   ├── products
│   ├── ratings
│   └── reviews
├── cmd
│   ├── details
│   ├── products
│   ├── ratings
│   └── reviews
├── configs
├── deployments
├── dist
├── internal
│   ├── app
│   │   ├── details
│   │   │   ├── controllers
│   │   │   ├── grpcservers
│   │   │   ├── repositorys
│   │   │   └── services
│   │   ├── products
│   │   │   ├── controllers
│   │   │   ├── grpcclients
│   │   │   └── services
│   │   ├── ratings
│   │   │   ├── controllers
│   │   │   ├── grpcservers
│   │   │   ├── repositorys
│   │   │   └── services
│   │   └── reviews
│   │       ├── controllers
│   │       ├── grpcservers
│   │       ├── repositorys
│   │       └── services
│   └── pkg
│       ├── app
│       ├── config
│       ├── consul
│       ├── database
│       ├── jaeger
│       ├── log
│       ├── models
│       ├── transports
│       │   ├── grpc
│       │   └── http
│       │       └── middlewares
│       │           └── ginprom
│       └── utils
│           └── netutil
├── mocks
└── scripts

複製代碼

/cmd

同project-layout程序員

"該項目的main方法。每一個應用程序的目錄名稱應與您要擁有的可執行文件的名稱相匹配（例如，/cmd/myapp）。不要在應用程序目錄中放入大量代碼。若是您認爲代碼能夠導入並在其餘項目中使用，那麼它應該存在於/ pkg目錄中。若是代碼不可重用或者您不但願其餘人重用它，請將該代碼放在/ internal目錄中。你會驚訝於別人會作什麼，因此要明確你的意圖！一般有一個小的main函數能夠從/ internal和/ pkg目錄中導入和調用代碼，而不是其餘任何東西。"github

/internal/pkg

同project-layoutgolang

"私有應用程序和庫代碼。這是您不但願其餘人在其應用程序或庫中導入的代碼。將您的實際應用程序代碼放在/internal/app目錄（例如/internal/app/myapp）和/internal/ pkg目錄中這些應用程序共享的代碼（例如/internal/pkg/myprivlib）。"sql

內部的包採用平鋪的方式。docker

/internal/pkg/config

加載配置文件,或者從配置中心獲取配置和監聽配置變更。數據庫

/internal/pkg/database

數據庫鏈接初始化和ORM框架初始化配置。編程

/internal/pkg/models

結構體定義。

/internal/pkg/transport

http/gpc 傳輸層

/internal/app/products

應用內部代碼

/internal/app/products/controllers

MVC控制層

/internal/app/products/services

領域邏輯層

/internal/app/products/repositorys

存儲層

/internal/app/products/grpcclients

grpc client

/internal/app/details/grpcservers

grpc servers

/mocks

mockery 生成的mock實現

/api

OpenAPI/Swagger規範，JSON模式文件，協議定義文件等。

/grafana

生成grafana dashboard 用到的腳本

/scripts

sql、部署腳本等

/build

Dockerfile、docker-compose

/deployment

docker-compose/kubernetes等配置

分層

MVC、領域模型、ORM 這些都是經過把特定職責的代碼拆分到不一樣的層次對象裏，在Java裏這些分層概念在各類框架裏都有體現(如SSH,SSM等經常使用框架組合)，而且早已造成了默認的規約，是否還適用go語言嗎？答案是確定的。Martin Fowler在《企業應用架構模式》就闡述過度層帶來的各類好處。

便代碼複用,提升代碼可維護性.如service的代碼可被http協議和grpc協議複用，若是增長thrift協議的接口也很方便。
層次清晰，代碼可讀性更高。
方便單元測試,單元測試每每由於依賴持久的存儲而沒法進行,若是持久化代碼抽取到單獨的對象裏，這就變的很簡單了.

依賴注入

Java 程序員都很熟悉依賴注入和控制翻轉這種思想，Spring正式基於依賴注入的思想開發。依賴注入的好處是解耦，對象的組裝交給容器來控制(選擇須要的實現類、是否單例和初始化).基於依賴注入能夠很方便的實現單元測試和提升代碼可維護性。

關於Golang依賴注入的討論《Dependency Injection in Go》,Golang依賴注入的Package有 Uber的dig,fx,facebook 的 inject,google的wire。dig、fx和inject都是基於反射實現，wire是經過代碼生成實現，代碼生成的方式是顯式的。

本示例經過wire來完成依賴注入. 編寫wire.go,wire會根據wire.go生成代碼。

// +build wireinject

package main

import (
    "github.com/google/wire"
    "github.com/zlgwzy/go-project-sample/cmd/app"
    "github.com/zlgwzy/go-project-sample/internal/pkg/config"
    "github.com/zlgwzy/go-project-sample/internal/pkg/database"
    "github.com/zlgwzy/go-project-sample/internal/pkg/log"
    "github.com/zlgwzy/go-project-sample/internal/pkg/services"
    "github.com/zlgwzy/go-project-sample/internal/pkg/repositorys"
    "github.com/zlgwzy/go-project-sample/internal/pkg/transport/http"
    "github.com/zlgwzy/go-project-sample/internal/pkg/transport/grpc"
)

var providerSet = wire.NewSet(
    log.ProviderSet,
    config.ProviderSet,
    database.ProviderSet,
    services.ProviderSet,
    repositorys.ProviderSet,
    http.ProviderSet,
    grpc.ProviderSet,
    app.ProviderSet,
)

func CreateApp(cf string) (*app.App, error) {
    panic(wire.Build(providerSet))
}

複製代碼

生成代碼

go get github.com/google/wire/cmd/wire

wire ./...
複製代碼

生成後的代碼在wire_gen.go

// Code generated by Wire. DO NOT EDIT.

//go:generate wire
//+build !wireinject

package main

import (
    "github.com/google/wire"
    "github.com/zlgwzy/go-project-sample/cmd/app"
    "github.com/zlgwzy/go-project-sample/internal/pkg/config"
    "github.com/zlgwzy/go-project-sample/internal/pkg/database"
    "github.com/zlgwzy/go-project-sample/internal/pkg/log"
    "github.com/zlgwzy/go-project-sample/internal/pkg/services"
    "github.com/zlgwzy/go-project-sample/internal/pkg/repositorys"
    "github.com/zlgwzy/go-project-sample/internal/pkg/transport/grpc"
    "github.com/zlgwzy/go-project-sample/internal/app/proxy/grpcservers"
    "github.com/zlgwzy/go-project-sample/internal/pkg/transport/http"
    "github.com/zlgwzy/go-project-sample/internal/app/proxy/controllers"
)

// Injectors from wire.go:

func CreateApp(cf string) (*app.App, error) {
    viper, err := config.New(cf)
    if err != nil {
        return nil, err
    }
    options, err := log.NewOptions(viper)
    if err != nil {
        return nil, err
    }
    logger, err := log.New(options)
    if err != nil {
        return nil, err
    }
    httpOptions, err := http.NewOptions(viper)
    if err != nil {
        return nil, err
    }
    databaseOptions, err := database.NewOptions(viper, logger)
    if err != nil {
        return nil, err
    }
    db, err := database.New(databaseOptions)
    if err != nil {
        return nil, err
    }
    productsRepository := repositorys.NewMysqlProductsRepository(logger, db)
    productsService := services.NewProductService(logger, productsRepository)
    productsController := controllers.NewProductsController(logger, productsService)
    initControllers := controllers.CreateInitControllersFn(productsController)
    engine := http.NewRouter(httpOptions, initControllers)
    server, err := http.New(httpOptions, logger, engine)
    if err != nil {
        return nil, err
    }
    grpcOptions, err := grpc.NewOptions(viper)
    if err != nil {
        return nil, err
    }
    productsServer, err := grpcservers.NewProductsServer(logger, productsService)
    if err != nil {
        return nil, err
    }
    initServers := grpcservers.CreateInitServersFn(productsServer)
    grpcServer, err := grpc.New(grpcOptions, logger, initServers)
    if err != nil {
        return nil, err
    }
    appApp, err := app.New(logger, server, grpcServer)
    if err != nil {
        return nil, err
    }
    return appApp, nil
}

// wire.go:

var providerSet = wire.NewSet(log.ProviderSet, config.ProviderSet, database.ProviderSet, services.ProviderSet, repositorys.ProviderSet, http.ProviderSet, grpc.ProviderSet, app.ProviderSet)

複製代碼

面向接口編程

多態和單元測試必須,比較好理解再也不解釋。

顯式編程

Golang的開發推崇這樣一種顯式編程的思想，顯式的初始化、方法調用和錯誤處理.

儘量不要使用包級別的全局變量.
儘可能不要使用init函數，初始化操做能夠在main函數中調用，這樣方便閱讀代碼和控制初始化順序。
函數都要返回錯誤，用if err != nil 顯式的處理錯誤.
依賴的參數讓調用者去控制（控制翻轉的思想),能夠看下節依賴注入。

幾個大佬都討論過這個問題，博士Peter的《A theory of modern Go》認爲魔法代碼的核心是」no package level vars; no func init「.單這也不是絕對。 Dave Cheny在《go-without-package-scoped-variables》作了更詳細的說明.

打印日誌

使用比較多的兩個日誌庫，logrush和zap,我的更喜歡zap。

初始化logger,經過viper加載日誌相關配置,lumberjack負責日誌切割。

// Options is log configration struct
type Options struct {
     Filename   string
     MaxSize    int
     MaxBackups int
     MaxAge     int
     Level      string
     Stdout     bool
}

func NewOptions(v *viper.Viper) (*Options, error) {
     var (
          err error
          o   = new(Options)
     )
     if err = v.UnmarshalKey("log", o); err != nil {
          return nil, err
     }

     return o, err
}

// New for init zap log library
func New(o *Options) (*zap.Logger, error) {
     var (
          err    error
          level  = zap.NewAtomicLevel()
          logger *zap.Logger
     )

     err = level.UnmarshalText([]byte(o.Level))
     if err != nil {
          return nil, err
     }

     fw := zapcore.AddSync(&lumberjack.Logger{
          Filename:   o.Filename,
          MaxSize:    o.MaxSize, // megabytes
          MaxBackups: o.MaxBackups,
          MaxAge:     o.MaxAge, // days
     })

     cw := zapcore.Lock(os.Stdout)

     // file core 採用jsonEncoder
     cores := make([]zapcore.Core, 0, 2)
     je := zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig())
     cores = append(cores, zapcore.NewCore(je, fw, level))

     // stdout core 採用 ConsoleEncoder
     if o.Stdout {
          ce := zapcore.NewConsoleEncoder(zap.NewDevelopmentEncoderConfig())
          cores = append(cores, zapcore.NewCore(ce, cw, level))
     }

     core := zapcore.NewTee(cores...)
     logger = zap.New(core)

     zap.ReplaceGlobals(logger)

     return logger, err
}


複製代碼

logger應該做爲私有變量，這樣能夠統一添加對象的標示。

type Object struct {
    logger *zap.Logger
}

// 統一添加標示
func NewObject(logger *zap.Logger){
    return &Object{
        logger:  logger.With(zap.String("type","Object"))
    }

}
複製代碼

錯誤處理

錯誤處理仍是看Dave Cheny的博客《Stack traces and the errors package》,《Don’t just check errors, handle them gracefully》。

使用類型判斷錯誤。
包裝錯誤，記錄錯誤的上下文。
使用 pakcage errors
只處理一次錯誤，處理錯誤意味着檢查錯誤值並作出決定。

錯誤日誌

logger.Error("get product by id error", zap.Error(err))
複製代碼

{
    "level":"error",
    "ts":1564056905.4602501,
    "msg":"get product by id error",
    "error":"product service get product error: get product error[id=2]: record not found",
    "errorVerbose":"record not found get product error[id=2]
github.com/zlgwzy/go-project-sample/internal/pkg/repositorys.(*MysqlProductsRepository).Get
/Users/xxx/code/go/go-project-sample/internal/pkg/repositorys/products.go:29
github.com/zlgwzy/go-project-sample/internal/pkg/services.(*DefaultProductsService).Get
/Users/xxx/code/go/go-project-sample/internal/pkg/services/products.go:27
github.com/zlgwzy/go-project-sample/internal/app/proxy/controllers.(*ProductsController).Get
/Users/xxx/code/go/go-project-sample/internal/app/proxy/controllers/products.go:30
github.com/gin-gonic/gin.(*Context).Next
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.RecoveryWithWriter.func1
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/recovery.go:83
github.com/gin-gonic/gin.(*Context).Next
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:389
github.com/gin-gonic/gin.(*Engine).ServeHTTP
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:351
net/http.serverHandler.ServeHTTP
/usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:2774
net/http.(*conn).serve
/usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:1878
runtime.goexit
/usr/local/Cellar/go/1.12.6/libexec/src/runtime/asm_amd64.s:1337
product service get product error
github.com/zlgwzy/go-project-sample/internal/pkg/services.(*DefaultProductsService).Get
/Users/xxx/code/go/go-project-sample/internal/pkg/services/products.go:28
github.com/zlgwzy/go-project-sample/internal/app/proxy/controllers.(*ProductsController).Get
/Users/xxx/code/go/go-project-sample/internal/app/proxy/controllers/products.go:30
github.com/gin-gonic/gin.(*Context).Next
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.RecoveryWithWriter.func1
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/recovery.go:83
github.com/gin-gonic/gin.(*Context).Next
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:389
github.com/gin-gonic/gin.(*Engine).ServeHTTP
/Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:351
net/http.serverHandler.ServeHTTP
/usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:2774
net/http.(*conn).serve
/usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:1878
runtime.goexit
/usr/local/Cellar/go/1.12.6/libexec/src/runtime/asm_amd64.s:1337"
}

複製代碼

接口中返回錯誤

gin 的使用方式:

func Handler(c *gin.Context) {
    err := //
    c.String(http.StatusInternalServerError, "%+v", err)
}


複製代碼

curl http://localhost:8080/product/5 輸出:

rpc error: code = Unknown desc = details grpc service get detail error: detail service get detail error: get product error[id=5]: record not found
get rating error
github.com/sdgmf/go-project-sample/internal/app/products/services.(*DefaultProductsService).Get
     /Users/xxx/code/go/go-project-sample/internal/app/products/services/products.go:50
github.com/sdgmf/go-project-sample/internal/app/products/controllers.(*ProductsController).Get
     /Users/xxx/code/go/go-project-sample/internal/app/products/controllers/products.go:30
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/opentracing-contrib/go-gin/ginhttp.Middleware.func4
     /Users/xxx/go/pkg/mod/github.com/opentracing-contrib/go-gin@v0.0.0-20190301172248-2e18f8b9c7d4/ginhttp/server.go:99
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/sdgmf/go-project-sample/internal/pkg/transports/http/middlewares/ginprom.(*GinPrometheus).Middleware.func1
     /Users/xxx/code/go/go-project-sample/internal/pkg/transports/http/middlewares/ginprom/ginprom.go:105
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-contrib/zap.RecoveryWithZap.func1
     /Users/xxx/go/pkg/mod/github.com/gin-contrib/zap@v0.0.0-20190528085758-3cc18cd8fce3/zap.go:109
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-contrib/zap.Ginzap.func1
     /Users/xxx/go/pkg/mod/github.com/gin-contrib/zap@v0.0.0-20190528085758-3cc18cd8fce3/zap.go:32
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.RecoveryWithWriter.func1
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/recovery.go:83
github.com/gin-gonic/gin.(*Context).Next
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:389
github.com/gin-gonic/gin.(*Engine).ServeHTTP
     /Users/xxx/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:351
net/http.serverHandler.ServeHTTP
     /usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:2774
net/http.(*conn).serve
     /usr/local/Cellar/go/1.12.6/libexec/src/net/http/server.go:1878
runtime.goexit
     /usr/local/Cellar/go/1.12.6/libexec/src/runtime/asm_amd64.s:1337
複製代碼

添加監控

Prometheus

做爲新一代的監控框架，Prometheus 具備如下特色：

強大的多維度數據模型：
時間序列數據經過 metric 名和鍵值對來區分。
全部的 metrics 均可以設置任意的多維標籤。
數據模型更隨意，不須要刻意設置爲以點分隔的字符串。
能夠對數據模型進行聚合，切割和切片操做。
支持雙精度浮點類型，標籤能夠設爲全 unicode。
靈活而強大的查詢語句（PromQL）：在同一個查詢語句，能夠對多個 metrics 進行乘法、加法、鏈接、取分數位等操做。
易於管理： Prometheus server 是一個單獨的二進制文件，可直接在本地工做，不依賴於分佈式存儲。
高效：平均每一個採樣點僅佔 3.5 bytes，且一個 Prometheus server 能夠處理數百萬的 metrics。
使用 pull 模式採集時間序列數據，這樣不只有利於本機測試並且能夠避免有問題的服務器推送壞的 metrics。
能夠採用 push gateway 的方式把時間序列數據推送至 Prometheus server 端。
能夠經過服務發現或者靜態配置去獲取監控的 targets。
有多種可視化圖形界面。
易於伸縮

Go基礎監控

import (
    "github.com/opentracing-contrib/go-gin/ginhttp"
    "github.com/gin-gonic/gin"
)
r := gin.New()

r.GET("/metrics", gin.WrapH(promhttp.Handler()))

複製代碼

http監控

建立internal/pkg/transports/http/middlewares/ginprom/ginprom.go

package ginprom

import (
     "strconv"
     "sync"
     "time"

     "github.com/gin-gonic/gin"
     "github.com/prometheus/client_golang/prometheus"
)

const (
     metricsPath = "/metrics"
     faviconPath = "/favicon.ico"
)

var (
     httpHistogram = prometheus.NewHistogramVec(prometheus.HistogramOpts{
          Namespace: "http_server",
          Name:      "requests_seconds",
          Help:      "Histogram of response latency (seconds) of http handlers.",
     }, []string{"method", "code", "uri"})
)

func init() {
     prometheus.MustRegister(httpHistogram)
}

type handlerPath struct {
     sync.Map
}

func (hp *handlerPath) get(handler string) string {
     v, ok := hp.Load(handler)
     if !ok {
          return ""
     }
     return v.(string)
}

func (hp *handlerPath) set(ri gin.RouteInfo) {
     hp.Store(ri.Handler, ri.Path)
}

// GinPrometheus struct
type GinPrometheus struct {
     engine   *gin.Engine
     ignored map[string]bool
     pathMap *handlerPath
     updated bool
}

// Option 可配置參數
type Option func(*GinPrometheus) // Ignore 添加忽略的路徑 func Ignore(path ...string) Option {
     return func(gp *GinPrometheus) {
          for _, p := range path {
               gp.ignored[p] = true
          }
     }
}

// New 構造器
func New(e *gin.Engine, options ...Option) *GinPrometheus {
     // 參數驗證
     if e == nil {
          return nil
     }
     gp := &GinPrometheus{
          engine: e,
          ignored: map[string]bool{
               metricsPath: true,
               faviconPath: true,
          },
          pathMap: &handlerPath{},
     }
     for _, o := range options {
          o(gp)
     }
     return gp
}

func (gp *GinPrometheus) updatePath() {
     gp.updated = true
     for _, ri := range gp.engine.Routes() {
          gp.pathMap.set(ri)
     }
}

// Middleware 返回中間件
func (gp *GinPrometheus) Middleware() gin.HandlerFunc {
     return func(c *gin.Context) {
          if !gp.updated {
               gp.updatePath()
          }
          // 把不須要的過濾掉
          if gp.ignored[c.Request.URL.String()] == true {
               c.Next()
               return
          }
          start := time.Now()

          c.Next()

          httpHistogram.WithLabelValues(
               c.Request.Method,
               strconv.Itoa(c.Writer.Status()),
               gp.pathMap.get(c.HandlerName()),
          ).Observe(time.Since(start).Seconds())
     }
}

複製代碼

在internal/pkg/transports/http/http.go添加:

r.Use(ginprom.New(r).Middleware()) // 添加prometheus 監控
複製代碼

grpc監控

在server添加：

gs = grpc.NewServer(
               grpc.StreamInterceptor(grpc_middleware.ChainStreamServer(
                    grpc_prometheus.StreamServerInterceptor,
               )),
               grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(
                    grpc_prometheus.UnaryServerInterceptor,
               )),
          )
複製代碼

在client添加：

grpc_prometheus.EnableClientHandlingTimeHistogram()
     o.GrpcDialOptions = append(o.GrpcDialOptions,
          grpc.WithInsecure(),
          grpc.WithUnaryInterceptor(grpc_middleware.ChainUnaryClient(
               grpc_prometheus.UnaryClientInterceptor,
          ),
          grpc.WithStreamInterceptor(grpc_middleware.ChainStreamClient(
               grpc_prometheus.StreamClientInterceptor,
          ),
     )
複製代碼

添加dashbord

能夠經過自動生成dashboard，能夠集成到本身公司的CICD系統中，上線後dashboard就有了,下面介紹如何經過jsonnet生成dashboard

安裝jsonnet
下載graonnet-lib

建立 grafana/dashboard.jsonnet

local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local row = grafana.row;
local singlestat = grafana.singlestat;
local prometheus = grafana.prometheus;
local graphPanel = grafana.graphPanel;
local template = grafana.template;
local row = grafana.row;

local app = std.extVar('app');

local baseUp() = singlestat.new(
  'Number of instances',
  datasource='Prometheus',
  span=2,
  valueName='current',
  transparent=true,
).addTarget(
  prometheus.target(
    'sum(up{app="' + app + '"})', instant=true
  )
);

local baseGrpcQPS() = singlestat.new(
  'Number of grpc request  per seconds',
  datasource='Prometheus',
  span=2,
  valueName='current',
  transparent=true,
).addTarget(
  prometheus.target(
    'sum(rate(grpc_server_handled_total{app="' + app + '",grpc_type="unary"}[1m]))',
     instant=true
  )
);

local baseGrpcError() = singlestat.new(
  'Percentage of grpc error request',
  format='percent',
  datasource='Prometheus',
  span=2,
  valueName='current',
  transparent=true,
).addTarget(
  prometheus.target(
    'sum(rate(grpc_server_handled_total{app="' + app + '",grpc_type="unary",grpc_code!="OK"}[1m])) /sum(rate(grpc_server_started_total{app="' + app + '",grpc_type="unary"}[1m])) * 100.0',
    instant=true
  )
);

local baseHttpQPS() = singlestat.new(
  'Number of http request  per seconds',
  datasource='Prometheus',
  span=2,
  valueName='current',
  transparent=true,
).addTarget(
  prometheus.target(
    'sum(rate(http_server_requests_seconds_count{app="' + app + '"}[1m]))',
     instant=true
  )
);

local baseHttpError() = singlestat.new(
  'Percentage of http error request',
  datasource='Prometheus',
  format='percent',
  span=2,
  valueName='current',
  transparent=true,
).addTarget(
  prometheus.target(
    'sum(rate(http_server_requests_seconds_count{app="' + app + '",code!="200"}[1m])) /sum(rate(http_server_requests_seconds_count{app="' + app + '"}[1m])) * 100.0',
    instant=true
  )
);

local goState(metric, description=null, format='none') = graphPanel.new(
  metric,
  span=6,
  fill=0,
  min=0,
  legend_values=true,
  legend_min=false,
  legend_max=true,
  legend_current=true,
  legend_total=false,
  legend_avg=false,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  description=description,
).addTarget(
  prometheus.target(
    metric + '{app="' + app + '"}',
    datasource='Prometheus',
    legendFormat='{{instance}}'
  )
);




local grpcQPS(kind='server', groups=['grpc_code']) = graphPanel.new(
  //title='grpc_' + kind + '_qps_' + std.join(',', groups),
  title='Number of grpc ' + kind + ' request  per seconds group by (' + std.join(',', groups) + ')',
  description='Number of grpc ' + kind + ' request per seconds  group by (' + std.join(',', groups) + ')',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    'sum(rate(grpc_' + kind + '_handled_total{app="' + app + '",grpc_type="unary"}[1m])) by (' + std.join(',', groups) + ')',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);


local grpcErrorPercentage(kind='server', groups=['instance']) = graphPanel.new(
  //title='grpc_' + kind + '_error_percentage_' + std.join(',', groups),
  title='Percentage of grpc ' + kind + ' error request group by (' + std.join(',', groups) + ')',
  description='Percentage of grpc ' + kind + ' error request group by (' + std.join(',', groups) + ')',
  format='percent',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    'sum(rate(grpc_'+kind+'_handled_total{app="' + app + '",grpc_type="unary",grpc_code!="OK"}[1m])) by (' + std.join(',', groups) + ')/sum(rate(grpc_'+kind+'_started_total{app="' + app + '",grpc_type="unary"}[1m])) by (' + std.join(',', groups) + ')* 100.0',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);


local grpcLatency(kind='server', groups=['instance'], quantile='0.99') = graphPanel.new(
  title='Latency of grpc ' + kind + ' request group by (' + std.join(',', groups) + ')',
  description='Latency of grpc ' + kind + ' request group by (' + std.join(',', groups) + ')',
  format='ms',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    '1000 * histogram_quantile(' + quantile + ',sum(rate(grpc_' + kind + '_handling_seconds_bucket{app="' + app + '",grpc_type="unary"}[1m])) by (' + std.join(',', groups) + ',le))',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);




local httpQPS(kind='server', groups=['grpc_code']) = graphPanel.new(
  title='Number of http' + kind + ' request group by (' + std.join(',', groups) + ') per seconds',
  description='Number of http' + kind + ' request group by (' + std.join(',', groups) + ') per seconds',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    'sum(rate(http_server_requests_seconds_count{app="' + app + '"}[1m])) by (' + std.join(',', groups) + ')',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);


local httpErrorPercentage(groups=['instance']) = graphPanel.new(
  //title='grpc_' + kind + '_error_percentage_' + std.join(',', groups),
  title='Percentage of http error request group by (' + std.join(',', groups) + ') ',
  description='Percentage of http error request group by (' + std.join(',', groups) + ')',
  format='percent',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    'sum(rate(http_server_requests_seconds_count{app="' + app + '",status!="200"}[1m])) by (' + std.join(',', groups) + ')/sum(rate(http_server_requests_seconds_count{app="' + app + '"}[1m])) by (' + std.join(',', groups) + ')* 100.0',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);

local httpLatency(groups=['instance'], quantile='0.99') = graphPanel.new(
  title='Latency of http request group by (' + std.join(',', groups) + ')',
  description='Latency of http request group by (' + std.join(',', groups) + ')',
  format='ms',
  legend_values=true,
  legend_max=true,
  legend_current=true,
  legend_alignAsTable=true,
  legend_rightSide=true,
  transparent=true,
  span=6,
  fill=0,
  min=0,
).addTarget(
  prometheus.target(
    '1000 * histogram_quantile(' + quantile + ',sum(rate(http_server_requests_seconds_bucket{app="' + app + '"}[1m])) by (' + std.join(',', groups) + ',le))',
    datasource='Prometheus',
    legendFormat='{{' + std.join('}}.{{', groups) + '}}'
  )
);


dashboard.new(app, schemaVersion=16, tags=['go'], editable=true, uid=app)
.addPanel(row.new(title='Base', collapse=true)
          .addPanel(baseUp(), gridPos={ x: 0, y: 0, w: 4, h: 10 })
          .addPanel(baseGrpcQPS(), gridPos={x: 4, y: 0, w: 4, h: 10 })
          .addPanel(baseGrpcError(), gridPos={x: 8, y: 0, w: 4, h: 10 })
          .addPanel(baseHttpQPS(), gridPos={x: 12, y: 0, w: 4, h: 10 })
          .addPanel(baseHttpError(), gridPos={x: 16, y: 0, w: 4, h: 10 })
          ,{  })
.addPanel(row.new(title='Go', collapse=true)
          .addPanel(goState('go_goroutines', 'Number of goroutines that currently exist'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_alloc_bytes', 'Number of bytes allocated and still in use'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_alloc_bytes_total', 'Total number of bytes allocated, even if freed'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_buck_hash_sys_bytes', 'Number of bytes used by the profiling bucket hash table'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_frees_total', 'Total number of frees'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_gc_cpu_fraction', "The fraction of this program's available CPU time used by the GC since the program started."), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_gc_sys_bytes', 'Number of bytes used for garbage collection system metadata'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_alloc_bytes', 'Number of heap bytes allocated and still in use'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_idle_bytes', 'Number of heap bytes waiting to be used'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_inuse_bytes', 'Number of heap bytes that are in use'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_objects', 'Number of allocated objects'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_released_bytes', 'Number of heap bytes released to OS'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_heap_sys_bytes', 'Number of heap bytes obtained from system'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_last_gc_time_seconds', 'Number of seconds since 1970 of last garbage collection'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_lookups_total', 'Total number of pointer lookups'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_mallocs_total', 'Total number of mallocs'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_mcache_inuse_bytes', 'Number of bytes in use by mcache structures'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_mcache_sys_bytes', 'Number of bytes used for mcache structures obtained from system'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_mspan_inuse_bytes', 'Number of bytes in use by mspan structures'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_mspan_sys_bytes', 'Number of bytes used for mspan structures obtained from system'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_next_gc_bytes', 'Number of heap bytes when next garbage collection will take place'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_other_sys_bytes', 'Number of bytes used for other system allocations'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_stack_inuse_bytes', 'Number of bytes in use by the stack allocator'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_stack_sys_bytes', 'Number of bytes obtained from system for stack allocator'), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(goState('go_memstats_sys_bytes', 'Number of bytes obtained from system'), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})

.addPanel(row.new(title='Grpc Server request rate', collapse=true)
          .addPanel(grpcQPS('server', ['grpc_code']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('server', ['instance']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('server', ['grpc_service']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('server', ['grpc_service', 'grpc_method']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc Server request error percentage', collapse=true)
          .addPanel(grpcErrorPercentage('server', ['grpc_service']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcErrorPercentage('server', ['grpc_service', 'grpc_method']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcErrorPercentage('server', ['instance']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc server 99%-tile Latency of requests', collapse=true)
          .addPanel(grpcLatency('server', ['grpc_code'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('server', ['instance'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('server', ['grpc_service'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('server', ['grpc_service', 'grpc_method'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc server 90%-tile Latency of requests', collapse=true)
        .addPanel(grpcLatency('server', ['grpc_code'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
        .addPanel(grpcLatency('server', ['instance'], 0.90), gridPos={ x: 12, y: 0, w: 12, h: 10 })
        .addPanel(grpcLatency('server', ['grpc_service'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
        .addPanel(grpcLatency('server', ['grpc_service', 'grpc_method'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
        , {})
.addPanel(row.new(title='Grpc client request rate', collapse=true)
          .addPanel(grpcQPS('client', ['grpc_code']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('client', ['instance']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('client', ['grpc_service']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcQPS('client', ['grpc_service', 'grpc_method']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc client request error percentage', collapse=true)
          .addPanel(grpcErrorPercentage('client', ['grpc_service']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcErrorPercentage('client', ['grpc_service', 'grpc_method']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcErrorPercentage('client', ['instance']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc client 99%-tile Latency of requests', collapse=true)
          .addPanel(grpcLatency('client', ['grpc_code'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['instance'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['grpc_service'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['grpc_service', 'grpc_method'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Grpc client 90%-tile Latency of requests', collapse=true)
          .addPanel(grpcLatency('client', ['grpc_code'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['instance'], 0.90), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['grpc_service'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(grpcLatency('client', ['grpc_service', 'grpc_method'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Http server request rate', collapse=true)
          .addPanel(httpQPS( ['grpc_code']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpQPS( ['instance']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(httpQPS( ['grpc_service']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpQPS( ['grpc_service', 'grpc_method']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Http server request error percentage', collapse=true)
          .addPanel(httpErrorPercentage( ['instance']), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpErrorPercentage( ['method','uri']), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Http server 99%-tile Latency of requests', collapse=true)
          .addPanel(httpLatency( ['grpc_code'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['instance'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['grpc_service'], 0.99), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['grpc_service', 'grpc_method'], 0.99), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
.addPanel(row.new(title='Http server 90%-tile Latency of requests', collapse=true)
          .addPanel(httpLatency( ['grpc_code'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['instance'], 0.90), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['grpc_service'], 0.90), gridPos={ x: 0, y: 0, w: 12, h: 10 })
          .addPanel(httpLatency( ['grpc_service', 'grpc_method'], 0.90), gridPos={ x: 12, y: 0, w: 12, h: 10 })
          , {})
複製代碼

建立grafana/dashboard-api.jsonnet

local dash = import './dashboard.jsonnet';

{
  dashboard: dash,
  folderId: 0,
  overwrite: false,
}
複製代碼

生成jsonnet配置

jsonnet -J ./grafana/grafonnet-lib  -o ./grafana/dashboards-api/$$app-api.json  --ext-str app=$$app  ./grafana/dashboard-api.jsonnet ;

複製代碼

調研grafana api

curl -X DELETE --user admin:admin  -H "Content-Type: application/json" 'http://localhost:3000/api/dashboards/db/$$app'
curl -x POST --user admin:admin  -H "Content-Type: application/json" --data-binary "@./grafana/dashboards-api/$$app-api.json" http://localhost:3000/api/dashboards/db 
複製代碼

dashboard預覽

生成alermanager 告警

TODO

調用鏈跟蹤

Jaeger

Jaeger 是Uber開源的基於Opentracing 的一個實現，相似於zipkin。

建立internal/pkg/jaeger/jaeger.go

package jaeger

import (
     "github.com/google/wire"
     "github.com/opentracing/opentracing-go"
     "github.com/pkg/errors"
     "github.com/spf13/viper"
     "github.com/uber/jaeger-client-go/config"
     "github.com/uber/jaeger-lib/metrics/prometheus"
     "go.uber.org/zap"
)

func NewConfiguration(v *viper.Viper, logger *zap.Logger) (*config.Configuration, error) {
     var (
          err error
          c   = new(config.Configuration)
     )

     if err = v.UnmarshalKey("jaeger", c); err != nil {
          return nil, errors.Wrap(err, "unmarshal jaeger configuration error")
     }

     logger.Info("load jaeger configuration success")

     return c, nil
}

func New(c *config.Configuration) (opentracing.Tracer, error) {

     metricsFactory := prometheus.New()
     tracer, _, err := c.NewTracer(config.Metrics(metricsFactory))

     if err != nil {
          return nil, errors.Wrap(err, "create jaeger tracer error")
     }

     return tracer, nil
}

var ProviderSet = wire.NewSet(New, NewConfiguration)

複製代碼

Grpc

修改internal/pkg/transports/grpc/server.go

gs = grpc.NewServer(
               grpc.StreamInterceptor(grpc_middleware.ChainStreamServer(
                    otgrpc.OpenTracingStreamServerInterceptor(tracer),
               )),
               grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(
                    otgrpc.OpenTracingServerInterceptor(tracer),
               )),
          )
複製代碼

修改internal/pkg/transports/grpc/client.go

conn, err := grpc.DialContext(ctx, target, grpc.WithInsecure(),
          grpc.WithUnaryInterceptor(grpc_middleware.ChainUnaryClient(
               otgrpc.OpenTracingClientInterceptor(tracer)),
          ),
          grpc.WithStreamInterceptor(grpc_middleware.ChainStreamClient(
               otgrpc.OpenTracingStreamClientInterceptor(tracer)),
          ),)

複製代碼

Gin

修改internal/pkg/transports/http/http.go

import "github.com/opentracing-contrib/go-gin/ginhttp"

r.Use(ginhttp.Middleware(tracer))
複製代碼

單元測試

存儲層測試

添加repositorys/wire.go 建立要測試的對象，會根據ProviderSet注入合適的依賴。

// +build wireinject

package repositorys

import (
     "github.com/google/wire"
     "github.com/sdgmf/go-project-sample/internal/pkg/config"
     "github.com/sdgmf/go-project-sample/internal/pkg/database"
     "github.com/sdgmf/go-project-sample/internal/pkg/log"
)



var testProviderSet = wire.NewSet(
     log.ProviderSet,
     config.ProviderSet,
     database.ProviderSet,
     ProviderSet,
)

func CreateDetailRepository(f string) (DetailsRepository, error) {
     panic(wire.Build(testProviderSet))
}


複製代碼

添加repositorys/products_test.go,這裏採用表格驅動的方法進行測試,存儲層測試會依賴數據庫。

package repositorys

import (
     "flag"
     "github.com/stretchr/testify/assert"
     "testing"
)

var configFile = flag.String("f", "details.yml", "set config file which viper will loading.")

func TestDetailsRepository_Get(t *testing.T) {
     flag.Parse()

     sto, err := CreateDetailRepository(*configFile)
     if err != nil {
          t.Fatalf("create product Repository error,%+v", err)
     }

     tests := []struct {
          name     string
          id       uint64
          expected bool
     }{
          {"id=1", 1, true},
          {"id=2", 2, true},
          {"id=3", 3, true},
     }

     for _, test := range tests {
          t.Run(test.name, func(t *testing.T) {
               _, err := sto.Get(test.id)

               if test.expected {
                    assert.NoError(t, err )
               }else {
                    assert.Error(t, err)
               }
          })
     }
}

複製代碼

運行測試

go test -v ./internal/app/details/repositorys -f $(pwd)/configs/details.yml

=== RUN   TestDetailsRepository_Get
use config file -> /Users/xxx/code/go/go-project-sample/configs/details.yml
=== RUN   TestDetailsRepository_Get/id=1
=== RUN   TestDetailsRepository_Get/id=2
=== RUN   TestDetailsRepository_Get/id=3
--- PASS: TestDetailsRepository_Get (0.11s)
    --- PASS: TestDetailsRepository_Get/id=1 (0.00s)
    --- PASS: TestDetailsRepository_Get/id=2 (0.00s)
    --- PASS: TestDetailsRepository_Get/id=3 (0.00s)
PASS
ok      github.com/sdgmf/go-project-sample/internal/app/details/repositorys     0.128s

複製代碼

邏輯層測試

經過mockery自動生成mock對象.

mockery --all
複製代碼

添加internal/app/details/services/wire.go

// +build wireinject

package services

import (
     "github.com/google/wire"
     "github.com/sdgmf/go-project-sample/internal/pkg/config"
     "github.com/sdgmf/go-project-sample/internal/pkg/database"
     "github.com/sdgmf/go-project-sample/internal/pkg/log"
     "github.com/sdgmf/go-project-sample/internal/app/details/repositorys"
)

var testProviderSet = wire.NewSet(
     log.ProviderSet,
     config.ProviderSet,
     database.ProviderSet,
     ProviderSet,
)

func CreateDetailsService(cf string, sto repositorys.DetailsRepository) (DetailsService, error) {
     panic(wire.Build(testProviderSet))
}


複製代碼

存儲層使用生成的MockProductsRepository，能夠直接在用例中定義Mock方法的返回值。

建立 services/details_test.go

package services

import (
     "flag"
     "github.com/sdgmf/go-project-sample/internal/pkg/models"
     "github.com/sdgmf/go-project-sample/mocks"
     "github.com/stretchr/testify/assert"
     "github.com/stretchr/testify/mock"
     "testing"
)

var configFile = flag.String("f", "details.yml", "set config file which viper will loading.")

func TestDetailsRepository_Get(t *testing.T) {
     flag.Parse()

     sto := new(mocks.DetailsRepository)

     sto.On("Get", mock.AnythingOfType("uint64")).Return(func(ID uint64) (p *models.Detail) {
          return &models.Detail{
               ID: ID,
          }
     }, func(ID uint64) error {
          return nil
     })

     svc, err := CreateDetailsService(*configFile, sto)
     if err != nil {
          t.Fatalf("create product serviceerror,%+v", err)
     }

     // 表格驅動測試
     tests := []struct {
          name     string
          id       uint64
          expected uint64
     }{
          {"1+1", 1, 1},
          {"2+3", 2, 2},
          {"4+5", 3, 3},
     }

     for _, test := range tests {
          t.Run(test.name, func(t *testing.T) {
               p, err := svc.Get(test.id)
               if err != nil {
                    t.Fatalf("product service get proudct error,%+v", err)
               }

               assert.Equal(t, test.expected, p.ID)
          })
     }
}

複製代碼

控制層測試

添加controllers/details_test.go,利用httptest進行測試

package controllers

import (
     "encoding/json"
     "flag"
     "fmt"
     "github.com/gin-gonic/gin"
     "github.com/sdgmf/go-project-sample/internal/pkg/models"
     "github.com/sdgmf/go-project-sample/mocks"
     "github.com/stretchr/testify/assert"
     "github.com/stretchr/testify/mock"
     "io/ioutil"
     "net/http/httptest"
     "testing"
)

var r *gin.Engine
var configFile = flag.String("f", "details.yml", "set config file which viper will loading.")

func setup() {
     r = gin.New()
}

func TestDetailsController_Get(t *testing.T) {
     flag.Parse()
     setup()

     sto := new(mocks.DetailsRepository)

     sto.On("Get", mock.AnythingOfType("uint64")).Return(func(ID uint64) (p *models.Detail) {
          return &models.Detail{
               ID: ID,
          }
     }, func(ID uint64) error {
          return nil
     })

     c, err := CreateDetailsController(*configFile, sto)
     if err != nil {
          t.Fatalf("create product serviceerror,%+v", err)
     }

     r.GET("/proto/:id", c.Get)

     tests := []struct {
          name     string
          id       uint64
          expected uint64
     }{
          {"1", 1, 1},
          {"2", 2, 2},
          {"3", 3, 3},
     }

     for _, test := range tests {
          t.Run(test.name, func(t *testing.T) {
               uri := fmt.Sprintf("/proto/%d", test.id)
               // 構造get請求
               req := httptest.NewRequest("GET", uri, nil)
               // 初始化響應
               w := httptest.NewRecorder()

               // 調用相應的controller接口
               r.ServeHTTP(w, req)

               // 提取響應
               rs := w.Result()
               defer func() {
                    _ = rs.Body.Close()
               }()

               // 讀取響應body
               body, _ := ioutil.ReadAll(rs.Body)
               p := new(models.Detail)
               err := json.Unmarshal(body, p)
               if err != nil {
                    t.Errorf("unmarshal response body error:%v", err)
               }

               assert.Equal(t, test.expected, p.ID)
          })
     }

}


複製代碼

grpc測試

測試Server

添加grpcservers/details_test.go

package grpcservers

import (
     "context"
     "flag"
     "github.com/sdgmf/go-project-sample/api/proto"
     "github.com/sdgmf/go-project-sample/internal/pkg/models"
     "github.com/sdgmf/go-project-sample/mocks"
     "github.com/stretchr/testify/assert"
     "github.com/stretchr/testify/mock"
     "testing"
)

var configFile = flag.String("f", "details.yml", "set config file which viper will loading.")

func TestDetailsService_Get(t *testing.T) {
     flag.Parse()

     service := new(mocks.DetailsService)

     service.On("Get", mock.AnythingOfType("uint64")).Return(func(ID uint64) (p *models.Detail) {
          return &models.Detail{
               ID: ID,
          }
     }, func(ID uint64) error {
          return nil
     })

     server, err := CreateDetailsServer(*configFile, service)
     if err != nil {
          t.Fatalf("create product server error,%+v", err)
     }

     // 表格驅動測試
     tests := []struct {
          name     string
          id       uint64
          expected uint64
     }{
          {"1+1", 1, 1},
          {"2+3", 2, 2},
          {"4+5", 3, 3},
     }

     for _, test := range tests {
          t.Run(test.name, func(t *testing.T) {
               req := &proto.GetDetailRequest{
                    Id: test.id,
               }
               p, err := server.Get(context.Background(), req)
               if err != nil {
                    t.Fatalf("product service get proudct error,%+v", err)
               }

               assert.Equal(t, test.expected, p.Id)
          })
     }

}

複製代碼

mock grpc client

/internal/app/products/services/products_test.go:

package services

import (
     "context"
     "flag"
     "github.com/golang/protobuf/ptypes"
     "github.com/sdgmf/go-project-sample/api/proto"
     "github.com/sdgmf/go-project-sample/mocks"
     "github.com/stretchr/testify/assert"
     "github.com/stretchr/testify/mock"
     "google.golang.org/grpc"
     "testing"
)

var configFile = flag.String("f", "products.yml", "set config file which viper will loading.")

func TestDefaultProductsService_Get(t *testing.T) {
     flag.Parse()

     detailsCli := new(mocks.DetailsClient)
     detailsCli.On("Get", mock.Anything, mock.Anything).
          Return(func(ctx context.Context, req *proto.GetDetailRequest, cos ...grpc.CallOption) *proto.Detail {
               return &proto.Detail{
                    Id:          req.Id,
                    CreatedTime: ptypes.TimestampNow(),
               }
          }, func(ctx context.Context, req *proto.GetDetailRequest, cos ...grpc.CallOption) error {
               return nil
          })

     ratingsCli := new(mocks.RatingsClient)

     ratingsCli.On("Get", context.Background(), mock.AnythingOfType("*proto.GetRatingRequest")).
          Return(func(ctx context.Context, req *proto.GetRatingRequest, cos ...grpc.CallOption) *proto.Rating {
               return &proto.Rating{
                    Id:          req.ProductID,
                    UpdatedTime: ptypes.TimestampNow(),
               }
          }, func(ctx context.Context, req *proto.GetRatingRequest, cos ...grpc.CallOption) error {
               return nil
          })

     reviewsCli := new(mocks.ReviewsClient)

     reviewsCli.On("Query", context.Background(), mock.AnythingOfType("*proto.QueryReviewsRequest")).
          Return(func(ctx context.Context, req *proto.QueryReviewsRequest, cos ...grpc.CallOption) *proto.QueryReviewsResponse {
               return &proto.QueryReviewsResponse{
                    Reviews: []*proto.Review{
                         &proto.Review{
                              Id:          req.ProductID,
                              CreatedTime: ptypes.TimestampNow(),
                         },
                    },
               }
          }, func(ctx context.Context, req *proto.QueryReviewsRequest, cos ...grpc.CallOption) error {
               return nil
          })

     svc, err := CreateProductsService(*configFile, detailsCli, ratingsCli, reviewsCli)
     if err != nil {
          t.Fatalf("create product service error,%+v", err)
     }

     // 表格驅動測試
     tests := []struct {
          name     string
          id       uint64
          expected bool
     }{
          {"id=1", 1, true},
     }

     for _, test := range tests {
          t.Run(test.name, func(t *testing.T) {
               _, err := svc.Get(context.Background(), test.id)

               if test.expected {
                    assert.NoError(t, err)
               } else {
                    assert.Error(t, err)
               }
          })
     }
}

複製代碼

Makefile

編寫Makefile

.PHONY: run
run:
     go run ./cmd -f cmd/app.yml
.PHONY: wire
wire:
    wire ./...
.PHONY: test
test:
    go test -v ./... -f `pwd`/cmd/app.yml -covermode=count -coverprofile=dist/test/cover.out

.PHONY: build
build:
    GOOS=linux GOARCH="amd64" go build ./cmd -o dist/sample5-linux-amd64
    GOOS=darwin GOARCH="amd64" go build ./cmd -o dist/sample5-darwin-amd64
.PHONY: cover
cover:
    go tool cover -html=dist/test/cover.out
.PHONY: mock
mock:
    mockery --all --inpkg
.PHONY: lint
lint:
    golint ./...
.PHONY: proto
proto:
    protoc -I api/proto ./api/proto/products.proto --go_out=plugins=grpc:api/proto
docker: build
    docker-compose -f docker/sample/docker-compose.yml up
複製代碼