CRD开发

步骤

编写 CRD 并将其部署到 K8s 集群里。让 K8s 知道有这个资源及其结构属性，在用户提交该自定义资源的定义时（通常是 YAML 文件定义），K8s 能够成功校验该资源并创建出对应的 Go struct 进行持久化，同时触发控制器的调谐逻辑
编写 Controller 并将其部署到 K8s 集群里。负责资源的调谐逻辑（生命周期/状态的维护），通过expectStatus和actualStatus的差异对比，进行Reconcile

开发框架

kubebuilder/operator-sdk 为主流

Kubebuilder

介绍

Kubebuilder is a framework for building Kubernetes APIs using custom resource definitions (CRDs). like go-zere, springboot

Kubebuilder is developed on top of the controller-runtime and controller-tools libraries.

架构和概念

架构： https://book.kubebuilder.io/architecture.html

GVK GVR。一般资源的yaml定义中apiVersion==GV（group version）， kind==K
schema。 kind 和 go types的映射（后面提及）, 其实就是一个json串
manager。管理所有的controller、webhook等
Controller。脚手架文件，实现Reconcile即可

安装 Kubebuilder

https://github.com/kubernetes-sigs/kubebuilder/releases

版本依赖

go version v1.15+ (kubebuilder v3.0 < v3.1).

go version v1.16+ (kubebuilder v3.1 < v3.3).

go version v1.17+ (kubebuilder v3.3+).

docker version 17.03+.

kubectl version v1.11.3+.

Access to a Kubernetes v1.11.3+ cluster.

创建项目

初始化和API创建

初始化

// 需要先init目录
[root@master kubebuilder]# ./kubebuilder init --domain huangzhipeng.cn 
Error: failed to initialize project: unable to inject the configuration to "base.go.kubebuilder.io/v3": error finding current repository: could not determine repository path from module data, package data, or by initializing a module: go: cannot determine module path for source directory /root/kubebuilder (outside GOPATH, module path must be specified)

[root@master demo]# go mod init demo
go: creating new go.mod: module demo

--------
// 国内需要设置代理
[root@master demo]# ../kubebuilder_linux_amd64 init --domain huangzhipeng.cn
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/[email protected]
go: sigs.k8s.io/[email protected]: reading sigs.k8s.io/controller-runtime/go.mod at revision v0.11.0: unknown revision v0.11.0
Error: failed to initialize project: unable to scaffold with "base.go.kubebuilder.io/v3": exit status 1

export GOPROXY=https://goproxy.io
export GO111MODULE=on
[root@master demo]# cat /etc/profile

创建api

../kubebuilder create api --group webapp --version v1 --kind Guestbook 
make
[root@master demo]# tree
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── guestbook_types.go    //api的定义
│       └── zz_generated.deepcopy.go
├── bin
│   ├── controller-gen
│   └── manager
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_guestbooks.yaml   
│   │       └── webhook_in_guestbooks.yaml
│   ├── default
│   │   ├── kustomization.yaml            //patch注入的操作
│   │   ├── manager_auth_proxy_patch.yaml   
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── guestbook_editor_role.yaml
│   │   ├── guestbook_viewer_role.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   └── service_account.yaml
│   └── samples
│       └── webapp_v1_guestbook.yaml    //cr的yaml样例，默认为空
├── controllers
│   ├── guestbook_controller.go   # controller的调谐逻辑
│   └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

安装CRD

make install
[root@master demo]# kubectl get crd |grep huangzhipeng
guestbooks.webapp.huangzhipeng.cn                     2022-02-10T05:57:51Z

部署 controller(部署到k8s)

访问问题修改

Dockerfile 中的 FROM gcr.io/distroless/static:nonroot
Dockerfile 中增加ENV GOPROXY https://goproxy.cn,direct ENV GO111MODULE on
/config/default/manager_auth_proxy_patch.yaml 文件中的 gcr.io/kubebuilder/kube-rbac-proxy

镜像构建和部署

这里把镜像push到了私有镜像仓库

make docker-build docker-push IMG=ccr.deepwisdomai.com/infra/go-base-image/kubebuilder-demo:v0.0.1
make deploy IMG=ccr.deepwisdomai.com/infra/go-base-image/kubebuilder-demo:v0.0.1


  Warning  Failed     7s (x3 over 47s)   kubelet            Failed to pull image "ccr.deepwisdomai.com/infra/go-base-image/kubebuilder-demo:latest": rpc error: code = Unknown desc = Error response from daemon: Head "https://ccr.deepwisdomai.com/v2/infra/go-base-image/kubebuilder-demo/manifests/latest": denied: access forbidden
  Warning  Failed     7s (x3 over 47s)   kubelet            Error: ErrImagePull
解决：
1. kubectl create secret docker-registry docker-hub-inner --docker-server=http://ccr.deepwisdomai.com --docker-username=huangzhipeng --docker-password=xxxx -n demo-system
2. imagePullSecrets 配置为 docker-hub-inner  （这里可通过patch的方式进行配置，否则更新controller时，又会被还原）

patch

# This patch inject a sidecar container which is a HTTP proxy for the
# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: kube-rbac-proxy
        image: kubesphere/kube-rbac-proxy:v0.8.0
        args:
        - "--secure-listen-address=0.0.0.0:8443"
        - "--upstream=http://127.0.0.1:8080/"
        - "--logtostderr=true"
        - "--v=10"
        ports:
        - containerPort: 8443
          protocol: TCP
          name: https
      - name: manager
        args:
        - "--health-probe-bind-address=:8081"
        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"
      imagePullSecrets:
      - name: docker-hub-inner   # 镜像拉取的secret
[root@master default]# cat manager_auth_proxy_patch.yaml 

//部署成功
[root@master demo]# kubectl get pods -n demo-system
NAME                                       READY   STATUS    RESTARTS   AGE
demo-controller-manager-5f94f8c64c-74x48   2/2     Running   0          2m17s

创建 CR

创建示例的CR

kubectl apply -f webapp_v1_guestbook.yaml

// 啥都没有
[root@master samples]# kubectl get guestbooks.webapp.huangzhipeng.cn guestbook-sample -o yaml
apiVersion: webapp.huangzhipeng.cn/v1
kind: Guestbook
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"webapp.huangzhipeng.cn/v1","kind":"Guestbook","metadata":{"annotations":{},"name":"guestbook-sample","namespace":"default"},"spec":null}
  creationTimestamp: "2022-02-10T07:51:58Z"
  generation: 1
  managedFields:
  - apiVersion: webapp.huangzhipeng.cn/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-02-10T07:51:58Z"
  name: guestbook-sample
  namespace: default
  resourceVersion: "9606144"
  uid: ede17786-d8b9-4221-9cfe-9aba8683b970

添加自定义逻辑

修改api/v1/guestbook_types.go

增加几个字段， FirstName， LastName， Status

// GuestbookSpec defines the desired state of Guestbook
type GuestbookSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Guestbook. Edit guestbook_types.go to remove/update
        Foo string `json:"foo,omitempty"`
        FirstName string `json:"firstname"`
        LastName  string `json:"lastname"`
}

// GuestbookStatus defines the observed state of Guestbook
type GuestbookStatus struct {
        // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
        // Important: Run "make" to regenerate code after modifying this file
        Status string `json:"Status"`
}

修改controllers/guestbook_controller.go

这里只是获取Guestbook对象，然后打印对象的值和更新Status字段的值

func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // _ = log.FromContext(ctx)

        // TODO(user): your logic here

        //r.Log.WithValues("apiexamplea", req.NamespacedName)
        obj := &webappv1.Guestbook{}
        if err := r.Get(context.Background(), req.NamespacedName, obj); err != nil {
                log.Println(err, "Unable to fetch object")
        } else {
                log.Println("Geeting from Kubebuilder to", obj.Spec.FirstName, obj.Spec.LastName)
        }

        obj.Status.Status = "Running"
        if err := r.Status().Update(context.Background(), obj); err != nil {
                log.Println(err, "unable to update status")
        }

        return ctrl.Result{}, nil
}

部署更新

// 1. CRD
...
          spec:
            description: GuestbookSpec defines the desired state of Guestbook
            properties:
              firstname:
                type: string
              foo:
                description: Foo is an example field of Guestbook. Edit guestbook_types.go
                  to remove/update
                type: string
              lastname:
                type: string
            required:
            - firstname   # guestbook_types.go 新增的字段
            - lastname
            type: object
          status:
            description: GuestbookStatus defines the observed state of Guestbook
            properties:
              Status:
                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
                  of cluster Important: Run "make" to regenerate code after modifying
                  this file'
                type: string
            required:
            - Status   # guestbook_types.go 新增的字段
            type: object
        type: object
...


// 2. controller
make docker-build docker-push IMG=ccr.deepwisdomai.com/infra/go-base-image/kubebuilder-demo:v0.0.4
make deploy IMG=ccr.deepwisdomai.com/infra/go-base-image/kubebuilder-demo:v0.0.4

// 3. 更新cr 

apiVersion: webapp.huangzhipeng.cn/v1
kind: Guestbook
metadata:
  name: guestbook-sample
spec:
  # TODO(user): Add fields here
  firstname: zhipeng  # 指定增加俩字段的值
  lastname: huang

验证更新

// 查看controller的调谐测试输出
2022-02-10T10:07:20.954Z	INFO	controller.guestbook	Starting workers	{"reconciler group": "webapp.huangzhipeng.cn", "reconciler kind": "Guestbook", "worker count": 1}
2022/02/10 10:07:20 Geeting from Kubebuilder to zhipeng huang
[root@master controllers]# kubectl logs demo-controller-manager-7488d45bc4-t6ndv -n demo-system -c manager

// 查看cr(guestbook-sample)的的status
...
spec:
  firstname: zhipeng
  lastname: huang
status:
  Status: Running
[root@master ~]# kubectl get guestbooks.webapp.huangzhipeng.cn guestbook-sample -o yaml

KB源码阅读

本质是把Reconcile函数绑定到manager，同时维护一个queue, 当有event发生时，入queue，另外一头由mange调用对应的Reconcile函数

informers、listers、clientsets生成

因为有些时候需要吧CR暴露给其它服务调用，所以借助code-generator需要生成相关的调用代码, 参考https://zhuanlan.zhihu.com/p/499752398

api 改目录
加doc.go\register.go
**_type.go 加//+genclient注释
./hack加tools.go、update-codegen.sh、verify-codegen.sh，改对应的配置项
code-generator 放到项目的上级目录，无需vendor
Makefile 增加指令
最终生成的代码在GOPATH 路径下的src

refer

https://mp.weixin.qq.com/s/Gzpq71nCfSBc1uJw3dR7xA
https://github.com/kubernetes-sigs/kubebuilder
https://book.kubebuilder.io/architecture.html
https://jimmysong.io/kubernetes-handbook/develop/kubebuilder-example.html
https://xuejipeng.github.io/kubebuilder-doc-cn/cronjob-tutorial/running.html
https://lailin.xyz/post/operator-09-kubebuilder-code.html

简单的CRD开发