k8s配置elk抓取pod容器日志

一、背景

k8s集群推荐使用efk日志抓取，当然也有使用filebeat抓取的。不管elk和efk都有两种模式进行抓取，分别是在每个节点部署agent抓取容器标准输出。第二种则是在每个pod下挂载filebeat进行日志抓取。

二、filebeat挂载示例

[root@xiangys0134-k8s-master filebeat]# cat filebeat.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      enabled: true
      tags: k8s-osp-monitor-service-logs
      fields:
        log_source: k8s-osp-monitor-service-logs
      paths:
        - /logs/xc-osp/monitor-service/*/*.log
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

    output.redis:
      hosts: ["192.168.10.105:6380"]
      db: 0
      timeout: 5
      password: "intel.com"
      key: "default_list"
      keys:
        - key: "k8s-osp-monitor-service-logs"
          when.equals:
            fields.log_source: "k8s-osp-monitor-service-logs"
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: test-filebeat
  namespace: default
  labels:
    app: test-filebeat
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-filebeat
  template:
    metadata:
      labels:
        app: test-filebeat
    spec:
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.6.1
          args: [
            "-c", "/etc/filebeat/filebeat.yml",
            "-e",
          ]
          volumeMounts:
            - name: monitor-log
              mountPath: /etc/filebeat/
      volumes:
        - configMap: 
            name: filebeat-config
          name: monitor-log

三、helm版本

kind: Deployment
apiVersion: apps/v1
metadata:
  name: xc-ops-deploy
  labels:
    app: xc-ops-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xc-ops-test
  template:
    metadata:
      labels:
        app: xc-ops-test
    spec:
      imagePullSecrets:
        - name: aliyun-image-secret
      containers:
        - name: xc-ops-registry
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8761
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                    ./registry-service/bin/docker-entrypoint.sh start
                    "
                    ]

        - name: xc-ops-monitor
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi        
          ports:
          - containerPort: 8765
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                     ./monitor-service/bin/docker-entrypoint.sh start
                    "
                    ]
          volumeMounts:
            - name: monitor-data
              mountPath: /logs/
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.6.1
          args: [
            "-c", "/etc/filebeat/filebeat.yml",
            "-e",
          ]
          volumeMounts:
            - name: monitor-log
              mountPath: /etc/filebeat/
            - name: monitor-data
              mountPath: /logs/

        - name: xc-ops-config
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8763
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                    ./config-service/bin/docker-entrypoint.sh start
                    "
                    ]
        - name: xc-ops-gateway
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8762
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                    sleep 5 && \
                    ./gateway-service/bin/docker-entrypoint.sh start
                    "
                    ]

        - name: xc-ops-oauth
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8766
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                     sleep 7 && \
                    ./oauth-service/bin/docker-entrypoint.sh start
                    "
                    ]
        - name: xc-ops-user
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8764
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                     sleep 8 && \
                     ./user-service/bin/docker-entrypoint.sh start
                    "
                    ]
        - name: xc-ops-quartz
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            requests:
              cpu: 100m
              memory: 48Mi
          ports:
          - containerPort: 8769
            protocol: TCP
          envFrom:
          - configMapRef:
              name: xc-ops-env
          command: [
                    "sh",
                    "-c",
                    "
                     sleep 9 && \
                     ./quartz-service/bin/docker-entrypoint.sh start
                    "
                    ]
      volumes:
        - configMap: 
            name: filebeat-config
          name: monitor-log
        - emptyDir: {}
          name: monitor-data

备注：这里的话就需要每个应用容器绑定一个filebeat容器进行日志收集，在资源使用率上不太友好，但胜在灵活。我查过相关资料，一个filebeat最大内存资源使用率大概13M

四、监测

[root@elk-redis ~]# redis-cli -p 6380 -a intel.com keys \*

备注：已经将日志推送至redis中了

五、kibana