项目快速交付至k8s集群

一、背景

记录一个Flask项目的快速交付的具体实现。所涉及的步骤为：

• Dockerfile脚本编写
• Helm Chart编写
• Jenkins接入

参考地址：https://gitee.com/xiangys0134/deploy/tree/master/flask-demo

二、构建基础Docker镜像

参考地址：https://gitee.com/xiangys0134/deploy/blob/master/flask-demo/base_dockerfile/Dockerfile

# vim Dockerfile    # 添加如下内容
FROM python:3.8.17-alpine3.18
COPY requirements.txt /opt/requirements.txt
RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r /opt/requirements.txt

# cat requirements.txt 
Flask==2.2.2

# docker build -t registry.cn-hangzhou.aliyuncs.com/xiangys0134/flask:2.2.2 .

三、测试Flask

# docker run -it registry.cn-hangzhou.aliyuncs.com/xiangys0134/flask:2.2.2 sh
# docker cp flaskweb1 gracious_wiles:/tmp        #另一个终端拷贝代码
/tmp/flaskweb1 # python -m flask run --host=0.0.0.0        #启动容器测试

# 备注：通过在容器中启动对应的python命令运行flask

四、编写应用Dockerfile

4.1 Dockerfile文件

# vim Dockerfile
FROM registry.cn-hangzhou.aliyuncs.com/xiangys0134/flask:2.2.2
# 镜像的作者 
LABEL maintainer="<250919938@qq.com>"

COPY . /app

ARG REGISTRY=https://pypi.org/simple

RUN pip install -i ${REGISTRY} -r /app/requirements.txt

WORKDIR /app

ENTRYPOINT ["python -m flask run --host=0.0.0.0"]

4.2 Build镜像

# docker build --build-arg REGISTRY="https://pypi.tuna.tsinghua.edu.cn/simple" -t registry.cn-hangzhou.aliyuncs.com/xiangys0134/app:frontend-api-v202307201044 .

# docker push registry.cn-hangzhou.aliyuncs.com/xiangys0134/app:frontend-api-v202307201045

# 网站项目参考地址：https://gitee.com/xiangys0134/deploy/tree/master/flask-demo/git

五、编写Helm

# helm create flask
# ll    # 查看内容
total 8
drwxr-xr-x 2 k8s-dev-test k8s-dev-test    6 Jul 20 10:11 charts
-rw-r--r-- 1 k8s-dev-test k8s-dev-test 1141 Jul 20 10:11 Chart.yaml
drwxr-xr-x 3 k8s-dev-test k8s-dev-test  193 Jul 20 13:56 templates
-rw-r--r-- 1 k8s-dev-test k8s-dev-test 3056 Jul 20 10:53 values.yaml

# 内容地址：https://gitee.com/xiangys0134/deploy/tree/master/flask-demo/flask

六、服务部署至k8s集群

6.1 手动部署

$ helm upgrade --install --namespace default --set image.tag=frontend-api-v202307201045 --set image.repository=registry.cn-hangzhou.aliyuncs.com/xiangys0134/app  --set replicaCount=1 frontend-api-flask .

# helm具体使用参考地址：https://helm.sh/zh/docs/

6.2 测试

# curl 192.168.7.45:31185/  # curl访问网站
Hello World!

七、Jenkins接入

7.1 构建镜像job

jenkins任务：test-demo-build，git分支：master

@Library('ops-pipline-library')_

def map = [:]
    // 自定义构建运行node节点环境,根据实际情况进行调整
    map.put('JENKINS_NODE','node1')
    // 应用的名称，最好与helm定义的应用一致
    map.put('APP_NAME','frontendapi')
    //认证仓库地址
    map.put('DOCKER_REGISTRY','https://registry.cn-shenzhen.aliyuncs.com')
    //镜像仓库认证key，需在jenkins中预先配置认证key
    map.put('DOCKER_REGISTRY_KEY','fb805ed9-7e7e-4c79-a9e4-325d5afacc7a')
    //应用仓库镜像，与镜像仓库地址一一对应
    map.put('DOCKER_IMG','registry.cn-shenzhen.aliyuncs.com/xiangys0134-dev/demo1-dev')
    //项目仓库地址，与GIT_KEY一一对应，如果要选择http/Https模式则GIT_KEY也同样需要调整
    map.put('GIT_URL','ssh://git@gitee.com/xiangys0134/deploy.git')
    //jenkins仓库认证key，对应GIT_URL
    map.put('GIT_KEY','e4bf623a-00e5-4169-68db-35be9ddc283e')
    // 编译目录及dockerfile build目录，根目录下则不填写。如果存在二级目录则写相对路径示例：abc/edb
    map.put('BUILD_DIR','flask-demo/git')
    // Dockerfile文件
    map.put('DOCKERFILE_NAME',"Dockerfile")
    // docker_build参数,示例：docker build --build-arg env=test -f Dockerfile
    map.put('DOCKER_BUILD_ARGS','--build-arg REGISTRY="https://pypi.tuna.tsinghua.edu.cn/simple" -f ')
    //编译命令，没有可以不填写
    map.put('BUILD_CMD',"")
   //触发多个job任务信息，release_dev|release_dev 必须与choices列表一一对应，否则报错
   //示例 map.put('DEPLOY_MAP','release_dev:yunwei_update_test,release_test:yunwei_update_test2')
    map.put('DEPLOY_MAP','release_dev:test-demo-update')
docker_img_build_base_v1(map)

7.2 部署至k8s集群

jenkins任务：test-demo-update，镜像tag：frontendapi-test-202307200536

...
# deploy
helm upgrade --install --atomic --reset-values --cleanup-on-fail --timeout 4m0s --namespace {namespace} \
--set image.repository='registry.cn-shenzhen.aliyuncs.com/xiangys0134-dev/demo1-dev' \
--set dockerbase64=dockerbase64 \
--set service.port=5000 \
--set service.targetport=5000 \
--set service.nodeport=31186 \
--set image.tag=release_tag \
--set resources.requests.cpu=200m \
--set resources.requests.memory=128Mi \
--set resources.limits.cpu=500m \
--set resources.limits.memory=512Mi \
--set livenessProbe.enabled=true \
--set readinessProbe.enabled=true \
--set livenessProbe.initialDelaySeconds=60 \
--set readinessProbe.initialDelaySeconds=60 \
--set livenessProbe.type=httpGet \
--set readinessProbe.type=httpGet \
--set livenessProbe.path=/ \
--set readinessProbe.path=/ \
--set replicaCount=1 \
--set lifecycle.preStop.exec.command[0]="/bin/sh" \
--set lifecycle.preStop.exec.command[1]="-c" \
--set lifecycle.preStop.exec.command[2]="sleep 10;" \
--set command="command" \
flask .
...

/* 备注：以上仅包含所用到的一些参数--set 只传递变量值，以下简单叙述部分功能
command         涉及程序的启动命令（可覆盖dockerfile的启动命令）
dockerbase64    涉及镜像仓库secret
service         涉及服务相关配置，默认为NodePort类型。可以指定对应service
resources       涉及分配的pod的cpu、内存资源
livenessProbe   涉及存活探测，默认为tcp探测
readinessProbe  同上
lifecycle       涉及pod的生命周期调用
其他：
nodeSelector    节点选择
tolerations     容忍度
pdb             干扰预算
ingress         涉及ingress7层负载
autoscaling     涉及pod水平自动扩缩
skywalking      涉及APM监控
*/

7.3 yaml信息

[k8s-dev-test@rancher-k8s-conn flask]$ kubectl get pods flask-6655749767-p49qb -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-07-20T05:57:50Z"
  generateName: flask-6655749767-
  labels:
    app: flask
    pod-template-hash: "6655749767"
...
spec:
  containers:
  - command:
    - sh
    - -c
    - python -m flask run --host=0.0.0.0
    envFrom:
    - configMapRef:
        name: flask
    image: registry.cn-shenzhen.aliyuncs.com/xiangys0134-dev/demo1-dev:frontendapi-test-202307200536
    imagePullPolicy: Always
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - sleep 10;
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /
        port: 5000
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 3
    name: flask
    ports:
    - containerPort: 5000
      name: http
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 5000
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 200m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xqwvs
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: flask-aws-image-secret
  nodeName: node-10
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-xqwvs
    secret:
      defaultMode: 420
      secretName: default-token-xqwvs
...