K8S监控-Prometheus部署

StatefulSet

由于要存储收集的监控历史数据，故创建有状态服务(使用nfs存储类)：

vim prometheus.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
    component: server
    release: v2.26.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: server
  template:
    metadata:
      labels:
        app: prometheus
        component: server
        release: v2.26.0
    spec:
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
            type: ''
        - name: config-volume
          configMap:
            name: prometheus
            items:
              - key: prometheus.yml
                path: prometheus.yml
            defaultMode: 420
      containers:
        - name: prometheus-server
          image: 'prom/prometheus:v2.26.0'
          command:
            - /bin/sh
            - '-c'
            - 'prometheus --storage.tsdb.retention=30d --config.file=/etc/config/prometheus.yml --storage.tsdb.path=/data/${HOSTNAME} --web.enable-lifecycle'
          ports:
            - containerPort: 9090
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 8Gi
            requests:
              cpu: 500m
              memory: 2Gi
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: pvc
              mountPath: /data
            - name: localtime
              readOnly: true
              mountPath: /etc/localtime
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 30
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 30
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
      restartPolicy: Always
      serviceAccountName: prometheus
      serviceAccount: prometheus
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: pvc
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: managed-nfs-storage
  serviceName: prometheus-headless

kubectl create -f prometheus.yml
kubectl get statefulsets.apps -n monitoring

这里通过serviceAccount指定了Pod运行的用户为上面建立的访问用户。

可以查看Pod实例所挂载指定的ServiceAccount访问令牌文件：

kubectl exec -n monitoring -it prometheus-0 -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/

Tips: 当启动时带有–web.enable-lifecycle参数，可通过发送 HTTP POST请求到 /-/reload 地址使Prometheus在运行时重新加载配置文件。且如果修改过的配置文件有错误将不会应用。

curl -X POST http://172.16.220.143:30090/-/reload

 

Service

建立服务：

vim prometheus-svc.yml

kind: Service
apiVersion: v1
metadata:
  name: prometheus-svc
  namespace: monitoring
  labels:
    app: prometheus
    component: server
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9090
      nodePort: 30090
  selector:
    app: prometheus
    component: server
  type: NodePort

kubectl create -f prometheus-svc.yml
kubectl get svc -n monitoring

Tips：定义成NodePort类型是为了方便K8S群集外访问，而不用创建Ingress。

访问其中一个节点查看部署情况：http://172.16.220.143:30090/