kubernets controller 和 CRD的擴展

 sample git repo

各個組件開發指導

operator 介紹   

 此圖來自谷歌員工的實踐介紹

 

client-go的使用和源碼分析 

(dlv) p pods
*k8s.io/api/core/v1.PodList {
        TypeMeta: k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "", APIVersion: ""},
        ListMeta: k8s.io/apimachinery/pkg/apis/meta/v1.ListMeta {
                SelfLink: "/api/v1/pods",
                ResourceVersion: "145208",
                Continue: "",},
        Items: []k8s.io/api/core/v1.Pod len: 8, cap: 8, [
                (*k8s.io/api/core/v1.Pod)(0xc00033a000),
                (*k8s.io/api/core/v1.Pod)(0xc00033a360),
                (*k8s.io/api/core/v1.Pod)(0xc00033a6c0),
                (*k8s.io/api/core/v1.Pod)(0xc00033aa20),
                (*k8s.io/api/core/v1.Pod)(0xc00033ad80),
                (*k8s.io/api/core/v1.Pod)(0xc00033b0e0),
                (*k8s.io/api/core/v1.Pod)(0xc00033b440),
                (*k8s.io/api/core/v1.Pod)(0xc00033b7a0),
        ],}

　　

從運行流程和list-watch看kubernetes系統的設計理念

 開發operator擴展kubernetes 調研整理

如何基於Kubernetes開發自定義的Controller

Kubernetes CRD (CustomResourceDefinition) 自定義資源類型

kubernetes系列之十四：Kubernetes CRD(CustomResourceDefinition)概覽

kubernetes系列之十六：Kubernetes CRD sample-controller的編譯和測試

This Picture from this blog and official doc

 Analyzing value of Operator Framework for Kubernetes community

 

Install GO:

wget -O- https://raw.githubusercontent.com/udhos/update-golang/master/update-golang.sh  | sudo bash

export PATH=$PATH:/usr/local/go/bin

go env |grep GOPATH >> ~/.profile
go env |grep GOROOT >> ~/.profile
echo "export GOPATH" >> ~/.profile
echo "export GOROOT " >> ~/.profile
echo "export PATH=\$PATH:/usr/local/go/bin:$GOPATH/bin:$GOROOT/bin" >> ~/.profile

source ~/.profile

 

install sample:

go get github.com/tools/godep
cd $GOPATH/src
git clone https://github.com/kubernetes/sample-controller.git
cd sample-controller
godep restore
cd $GOPATH/src/k8s.io/sample-controller
go build -o sample-controller .

 

debug:

go get github.com/derekparker/delve/cmd/dlv
dlv exec ./sample-controller -- -kubeconfig=$HOME/.kube/config

 

分析:

main.go

1.  獲取配置信息（構建client使用）

 clientcmd.BuildConfigFromFlags(masterURL, kubeconfig)

 從"/home/ubuntu/.kube/config" 獲取登錄的證書, key和CA

2. 生成一個client，來登錄api server

// "k8s.io/client-go/kubernetes"

kubeClient, err := kubernetes.NewForConfig(cfg)

3. 生成一個sample的client，用戶來登錄controller

// clientset "k8s.io/sample-controller/pkg/client/clientset/versioned"

clientset.NewForConfig(cfg)

4.  分別生成 kubeInformerFactory 和 exampleInformerFactory

// kubeinformers "k8s.io/client-go/informers"

 

// informers "k8s.io/sample-controller/pkg/client/informers/externalversions"

 

// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
type SharedInformerFactory interface {
        Start(stopCh <-chan struct{})
        InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
}

 

// SharedInformerOption defines the functional option type for SharedInformerFactory.
type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory

type sharedInformerFactory struct {
        client           versioned.Interface
        namespace        string
        tweakListOptions internalinterfaces.TweakListOptionsFunc
        lock             sync.Mutex
        defaultResync    time.Duration
        customResync     map[reflect.Type]time.Duration

        informers map[reflect.Type]cache.SharedIndexInformer
        // startedInformers is used for tracking which informers have been started.
        // This allows Start() to be called multiple times safely.
        startedInformers map[reflect.Type]bool
}

　reflect.Type  Go Reflect 

 

SharedInformer具備共享數據緩存，而且可以將對緩存更改的通知分發給經過AddEventHandler註冊的多個listeners。 若是使用此方法，則與標準Informer相比，有一個行爲不一樣。 當您收到通知時，緩存將至少與通知同樣的fresh，甚至更fresh。 您不該該依賴緩存的內容與處理函數中收到的通知徹底匹配。 若是建立後緊接着是刪除，則緩存可能沒有這個條目了。 這比廣播者有優點，由於它容許咱們在多個控制器之間共享公共緩存。 擴展廣播者須要咱們爲每一個wach保留重複的緩存。

// SharedInformer has a shared data cache and is capable of distributing notifications for changes
// to the cache to multiple listeners who registered via AddEventHandler. If you use this, there is
// one behavior change compared to a standard Informer.  When you receive a notification, the cache
// will be AT LEAST as fresh as the notification, but it MAY be more fresh.  You should NOT depend
// on the contents of the cache exactly matching the notification you've received in handler
// functions.  If there was a create, followed by a delete, the cache may NOT have your item.  This
// has advantages over the broadcaster since it allows us to share a common cache across many
// controllers. Extending the broadcaster would have required us keep duplicate caches for each
// watch.
type SharedInformer interface {
        // AddEventHandler adds an event handler to the shared informer using the shared informer's resync
        // period.  Events to a single handler are delivered sequentially, but there is no coordination
        // between different handlers.
        AddEventHandler(handler ResourceEventHandler)
        // AddEventHandlerWithResyncPeriod adds an event handler to the shared informer using the
        // specified resync period.  Events to a single handler are delivered sequentially, but there is
        // no coordination between different handlers.
        AddEventHandlerWithResyncPeriod(handler ResourceEventHandler, resyncPeriod time.Duration)
        // GetStore returns the Store.
        GetStore() Store
        // GetController gives back a synthetic interface that "votes" to start the informer
        GetController() Controller
        // Run starts the shared informer, which will be stopped when stopCh is closed.
        Run(stopCh <-chan struct{})
        // HasSynced returns true if the shared informer's store has synced.
        HasSynced() bool
        // LastSyncResourceVersion is the resource version observed when last synced with the underlying
        // store. The value returned is not synchronized with access to the underlying store and is not
        // thread-safe.
        LastSyncResourceVersion() string
}

type SharedIndexInformer interface {
        SharedInformer
        // AddIndexers add indexers to the informer before it starts.
        AddIndexers(indexers Indexers) error
        GetIndexer() Indexer
}

　　

懷疑是代碼自動生成的。

git grep "func NewSharedInformerFactory"
pkg/client/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
pkg/client/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/client-go/informers/factory.go:func NewSharedInformerFactory(client kubernetes.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/client-go/informers/factory.go:func NewSharedInformerFactoryWithOptions(client kubernetes.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/MixedCase/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/MixedCase/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/internalversion/factory.go:func NewSharedInformerFactory(client internalversion.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/internalversion/factory.go:func NewSharedInformerFactoryWithOptions(client internalversion.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/crd/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/crd/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/cmd/informer-gen/generators/factory.go:func NewSharedInformerFactory(client {{.clientSetInterface|raw}}, defaultResync {{.timeDuration|raw}}) SharedInformerFactory {
vendor/k8s.io/code-generator/cmd/informer-gen/generators/factory.go:func NewSharedInformerFactoryWithOptions(client {{.clientSetInterface|raw}}, defaultResync {{.timeDuration|raw}}, options ...SharedInformerOption) SharedInformerFactory {

　　

5.  生成 controller 

"controller.go" 定義了Controller

低層定義見 pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go

6.  運行 kubeInformerFactory， exampleInformerFactory 和 controller

(dlv) b k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start
Breakpoint 5 set at 0xf42f08 for k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start() ./pkg/client/informers/externalversions/factory.go:111
(dlv) b k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start
Breakpoint 6 set at 0xefb888 for k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start() ./vendor/k8s.io/client-go/informers/factory.go:126

// https://github.com/kubernetes/sample-controller/blob/master/pkg/client/informers/externalversions/factory.go

// Start initializes all requested informers.
func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
	f.lock.Lock()
	defer f.lock.Unlock()

	for informerType, informer := range f.informers {
		if !f.startedInformers[informerType] {
			go informer.Run(stopCh)
			f.startedInformers[informerType] = true
		}
	}
}

　　

 0  0x0000000000f46733 in main.(*Controller).enqueueFoo
    at ./controller.go:338
 1  0x0000000000f4785e in main.NewController.func1
    at ./controller.go:120
 2  0x0000000000e6f96d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:202
 3  0x0000000000e81066 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate
    at <autogenerated>:1
 4  0x0000000000e7ee3b in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:552
 5  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 6  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
11  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
12  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
13  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

 

0  0x0000000000f469f0 in main.(*Controller).handleObject
    at ./controller.go:378
 1  0x0000000000f47f5e in main.(*Controller).handleObject-fm
    at ./controller.go:130
 2  0x0000000000e6f8f9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:195
 3  0x0000000000e80f32 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd
    at <autogenerated>:1
 4  0x0000000000e7eecd in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
 5  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 6  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
11  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
12  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
13  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

(dlv) print ownerRef
*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference {
        APIVersion: "samplecontroller.k8s.io/v1alpha1",
        Kind: "Foo",
        Name: "example-foo",
        UID: "118da85e-00fa-11e9-96e2-fa163e199d30",
        Controller: *true,
        BlockOwnerDeletion: *true,}

　　

goroutines                                                                                                                                  [41/326]
  Goroutine 1 - User: ./controller.go:171 main.(*Controller).Run (0xf45a11)
  Goroutine 2 - User: /usr/local/go/src/runtime/asm_amd64.s:311 runtime.systemstack_switch (0x457400)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 4 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 5 - User: ./vendor/k8s.io/klog/klog.go:941 k8s.io/sample-controller/vendor/k8s.io/klog.(*loggingT).flushDaemon (0x5f185b)
  Goroutine 6 - User: /usr/local/go/src/runtime/sigqueue.go:139 os/signal.signal_recv (0x44243c)
  Goroutine 7 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 8 - User: ./pkg/signals/signal.go:36 k8s.io/sample-controller/pkg/signals.SetupSignalHandler.func1 (0xf449f4)
  Goroutine 9 - User: ./vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop
(0x772ab6)
  Goroutine 10 - User: ./vendor/k8s.io/client-go/tools/record/event.go:231 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/record.(*eventBroadcast$
rImpl).StartEventWatcher.func1 (0xe9d054)
  Goroutine 11 - User: ./vendor/k8s.io/client-go/tools/record/event.go:231 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/record.(*eventBroadcast$
rImpl).StartEventWatcher.func1 (0xe9d054)
  Goroutine 12 - User: ./vendor/k8s.io/client-go/util/workqueue/queue.go:198 k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*Type).upda$
eUnfinishedWorkLoop (0xea1212)
  Goroutine 13 - User: /usr/local/go/src/runtime/lock_futex.go:228 runtime.notetsleepg (0x40c4f7)
  Goroutine 14 - User: ./vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:206 k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*d$
layingType).waitingLoop (0xe9f2cf)
  Goroutine 15 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 16 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 18 - User: /usr/local/go/src/runtime/proc.go:3063 runtime.exitsyscall (0x4353a9)
  Goroutine 20 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 21 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 22 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 34 - User: /usr/local/go/src/runtime/proc.go:3063 runtime.exitsyscall (0x4353a9)
  Goroutine 36 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedP
rocessor).run (0xe79b46)
  Goroutine 37 - User: ./vendor/k8s.io/client-go/tools/cache/controller.go:103 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller).
Run.func1 (0xe7e0d4)
  Goroutine 38 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:302 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).w$
tchHandler (0xe764e5)
  Goroutine 41 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 42 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 43 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 44 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 45 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 50 - User: ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:374 k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.poller.fu$
c1.1 (0x9b4d93)
  Goroutine 52 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*shared$
rocessor).run (0xe79b46)
  Goroutine 53 - User: ./vendor/k8s.io/client-go/tools/cache/controller.go:103 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller)$
Run.func1 (0xe7e0d4)
  Goroutine 54 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:302 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).w$
tchHandler (0xe764e5)
  Goroutine 58 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 59 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 60 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 61 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 62 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 63 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 65 - User: /usr/local/go/src/runtime/netpoll.go:173 internal/poll.runtime_pollWait (0x428dc6)
  Goroutine 66 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:549 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*proces$
orListener).run.func1.1 (0xe7ed04)
  Goroutine 67 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:517 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*proces$
orListener).pop (0xe7a356)
  Goroutine 68 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 69 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 71 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:207 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).L$
stAndWatch.func1 (0xe7e56d)
  Goroutine 72 - User: /usr/local/go/src/runtime/lock_futex.go:228 runtime.notetsleepg (0x40c4f7)
  Goroutine 74 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
* Goroutine 82 - User: ./controller.go:341 main.(*Controller).enqueueFoo (0xf46741) (thread 12081)
  Goroutine 83 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:517 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*process
orListener).pop (0xe7a356)
  Goroutine 99 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:207 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).Li
stAndWatch.func1 (0xe7e56d)
  Goroutine 102 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 103 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 114 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 146 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 179 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 187 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)

　　

Other:

Understanding kubernetes networking: ingress

Load Balance East-West Traffic in Kubernetes Environment Using NetScaler CPX

Kubernetes網絡分析之Flannel

How we run Kubernetes in Kubernetes aka Kubeception

Istio Handbook——Istio中文指南/服務網格實踐手冊

Kubernetes與雲原生應用概覽

 

比較典型CRD的例子是：

基於Helm和Operator的K8S應用管理

帶你入門operator-framework

CoreOS有狀態應用管理的框架--Operator詳解

如何在GO語言中使用Kubernetes API？

視頻： 沃趣：袁琳峯-使用 Operator 來擴展 Kubernetes