APM分布式监控Skywalking

一、背景介绍

分布式链路追踪有Cat、Zipkin、Skywalking等工具，其中cat和zipkin都需要开发人员在代码层面埋点。而skywalking提供了对k8s的pod进行分布式链路监控。只需要对pod使用Sidecar注入代理即可，无需改动程序代码。

官方文档：https://github.com/apache/skywalking

chart安装文档：https://github.com/apache/skywalking-kubernetes

二、部署

[k8s-prod-de@ip-172-20-21-242 apm]git clone https://github.com/apache/skywalking-kubernetes
[k8s-prod-de@ip-172-20-21-242 apm] cd skywalking-kubernetes/chart
[k8s-prod-de@ip-172-20-21-242 chart]helm repo add elastic https://helm.elastic.co
[k8s-prod-de@ip-172-20-21-242 chart] helm dep up skywalking
[k8s-prod-de@ip-172-20-21-242 chart]export SKYWALKING_RELEASE_NAME=skywalking
[k8s-prod-de@ip-172-20-21-242 chart] export SKYWALKING_RELEASE_NAMESPACE=monitoring
[k8s-prod-de@ip-172-20-21-242 chart]$ vim ./skywalking/values-my-es.yaml
host: es-elasticsearch-client.logging

三、helm安装

helm upgrade --install "{SKYWALKING_RELEASE_NAME}" skywalking -n "{SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.image.tag=8.5.0-es6 \
  --set oap.storageType=elasticsearch \
  --set ui.image.tag=8.5.0 \
  -f ./skywalking/values-my-es.yaml

四、应用apm监控测试

#参考文档
apiVersion: v1
kind: Pod
metadata:
  name: agent-as-sidecar
spec:
  restartPolicy: Never

  volumes:
    - name: skywalking-agent
      emptyDir: { }

  containers:
    - name: agent-container
      image: apache/skywalking-java-agent:8.5.0-alpine
      volumeMounts:
        - name: skywalking-agent
          mountPath: /agent
      command: [ "/bin/sh" ]
      args: [ "-c", "cp -R /skywalking/agent /agent/" ]

    - name: app-container
      image: springio/gs-spring-boot-docker
      volumeMounts:
        - name: skywalking-agent
          mountPath: /skywalking
      env:
        - name: JAVA_TOOL_OPTIONS
          value: "-javaagent:/skywalking/agent/skywalking-agent.jar"

#yaml文档
[k8s-prod-de@ip-172-20-21-242 fastbull-universal-service]grep -v "^#|^" templates/deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: {{ template "my-demo-app-universal-service.fullname" . }}
  name: {{ include "my-demo-app-universal-service.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "my-demo-app-universal-service.fullname" . }}
  template:
    metadata:
      labels:
        app: {{ template "my-demo-app-universal-service.fullname" . }}
    spec:
      volumes:
      - name: skywalking-agent
        emptyDir: { }
      initContainers:
        - name: agent-container
          image: apache/skywalking-java-agent:8.5.0-alpine
          volumeMounts:
            - name: skywalking-agent
              mountPath: /agent
          command: [ "/bin/sh" ]
          args: [ "-c", "cp -R /skywalking/agent /agent/" ]
      containers:
        - command:
            - sh
            - '-c'
            - java -javaagent:/skywalking/agent/skywalking-agent.jar -jar /app/app.jar
          env:
            - name: LANG
              value: C.UTF-8
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: TZ
              value: Asia/Shanghai
            - name: SW_AGENT_NAME
              value: demo-application
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              value: skywalking-oap.monitoring:11800
            - name: SW_AGENT_SPAN_LIMIT
              value: '2000'
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /checkHealth
              port: {{ .Values.service.targetport }}
              scheme: HTTP
            initialDelaySeconds: 120
            periodSeconds: 15
            successThreshold: 1
            timeoutSeconds: 1
          name: {{ .Chart.Name }}
          volumeMounts:
            - name: skywalking-agent
              mountPath: /skywalking
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /checkHealth
              port: {{ .Values.service.targetport }}
              scheme: HTTP
            initialDelaySeconds: 120
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: {{ .Values.resources.limits.cpu }}
              memory: {{ .Values.resources.limits.memory }}
            requests:
              cpu: {{ .Values.resources.requests.cpu }}
              memory: {{ .Values.resources.requests.memory }}
      imagePullSecrets:
        - name: {{ template "my-demo-app-universal-service.fullname" . }}-aws-image-secret
      restartPolicy: Always

五、app分布式链路追踪

5.1查看api调用成功率及响应时间

5.2查看jvm堆栈信息

5.3查看rds调用成功率及db响应时间

5.4同理查看mongodb状态