K8S监控-Prometheus告警配置

作者

告警Alertmanger

上面设置了规则，也可以正常告警，但并不会将告警主动推送给用户。要想主动推送给用户可以配置将告警信息推送至Alertmanger。

创建配置文件：

vim prometheus-alertmanager-config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
  labels:
    app: prometheus-alertmanager
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
      smtp_smarthost: 'mail.xxx.com:465'
      smtp_from: 'Prometheus<[email protected]>'
      smtp_auth_username: '[email protected]'
      smtp_auth_password: 'YourPass'
      smtp_require_tls: false
    templates:
      - /etc/config/template/*.tmpl
    route:
      group_by: ['alertname']   # 报警分组依据
      group_wait: 30s   # 进行分组,等待时间后发送警报
      group_interval: 5m   # 同一Alert Group告警,等待间隔后再发送新警报
      repeat_interval: 30m  # Alert Group警报没变化且发送成功,重发警报的周期
      receiver: 'email'  # 发送警报的接收者的名称,以下receivers name的名称
    receivers:
      - name: 'email'
        email_configs:
        - to: 'Chris<[email protected]>'
          html: '{{ template "email.html" . }}'
          headers: { Subject: "[Prometheus] 报警邮件"}
          send_resolved: true
    inhibit_rules:
      - source_match:
          level: 'Critical'
        target_match:
          level: 'Warning'
        equal: ['alertname', 'dev', 'instance']

kubectl create -f prometheus-alertmanager-config.yml
kubectl -n monitoring get configmaps

Tips：alertname就是在上面告警规则中rules – alert 中定义的规则名称。

创建告警模板：

vim prometheus-alertmanager-tmpl.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-tmpl
  namespace: monitoring
  labels:
    app: prometheus-alertmanager
data:
  email.tmpl: |
    {{ define "email.html" }}
    <html><head>
      <link rel="shortcut icon" href="data:image/ico;base64,aWNv">
      <meta content="text/html; charset=utf-8">
    </head>
    <body style="font-family:msyh,Verdana,sans-serif;">
    <table style="width:95%"; class="gridtable"; border="1">
    
    {{ if gt (len .Alerts.Firing) 0 -}}
      <tr bgcolor="#99ccff"; style="color:red"><th colspan=6>Alerts Status Report.</th></tr>
      <tr>
        <th bgcolor="#99ccff">>告警类型</th>
        <th bgcolor="#99ccff">>实例</th>
        <th bgcolor="#99ccff">>当前值</th>
        <th bgcolor="#99ccff">>告警级别</th>
        <th bgcolor="#99ccff">>开始时间</th>
        <th bgcolor="#99ccff">>告警摘要</th>
      </tr>
    {{- end }}
    {{ range .Alerts.Firing }}
      <tr>
        <td>{{ index .Labels "alertname" }}</td>
        <td>{{ index .Labels "instance" }}</td>
        <td>{{ index .Annotations "value" }}</td>
        <td>{{ index .Labels "level" }}</td>
        <td>{{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}</td>
        <td>{{ index .Annotations "summary" }}</td>
      </tr>
    {{ end }}
    
    {{ if gt (len .Alerts.Resolved) 0 -}}
      <tr bgcolor="#99ccff"; style="color:green"><th colspan=7>Resolved Status Report.</th></tr>
      <tr>
        <th bgcolor="#99ccff">>告警类型</th>
        <th bgcolor="#99ccff">>实例</th>
        <th bgcolor="#99ccff">>当前值</th>
        <th bgcolor="#99ccff">>告警级别</th>
        <th bgcolor="#99ccff">>开始时间</th>
        <th bgcolor="#99ccff">>恢复时间</th>
        <th bgcolor="#99ccff">>告警摘要</th>
      </tr>
    {{- end }}
    {{ range .Alerts.Resolved }}
      <tr>
        <td>{{ index .Labels "alertname" }}</td>
        <td>{{ index .Labels "instance" }}</td>
        <td>{{ index .Annotations "value" }}</td>
        <td>{{ index .Labels "level" }}</td>
        <td>{{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}</td>
        <td>{{ .EndsAt.Format "2006-01-02 15:04:05 MST" }}</td>
        <td>{{ index .Annotations "summary" }}</td>
      </tr>
    {{ end }}
    </table>
    </body>
    {{ end }}
kubectl create -f prometheus-alertmanager-tmpl.yml
kubectl -n monitoring get configmaps

部署AlertMananger

用Deployment的就可以了，当然也可以用StatefulSet来存放信息：

vim prometheus-alertmanager.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: prometheus
    component: alertmanager
    release: v0.21.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: alertmanager
  template:
    metadata:
      labels:
        app: prometheus
        component: alertmanager
        release: v0.21.0
    spec:
      priorityClassName: system-cluster-critical
      containers:
        - name: prometheus-alertmanager
          image: "prom/alertmanager:v0.21.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --config.file=/etc/config/alertmanager.yml
            - --storage.path=/data
          ports:
            - containerPort: 9093
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9093
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: tmpl-volume
              mountPath: /etc/config/template
            - name: data-volume
              mountPath: /data
          resources:
            limits:
              cpu: 10m
              memory: 50Mi
            requests:
              cpu: 10m
              memory: 50Mi
      volumes:
        - name: config-volume
          configMap:
            name: alertmanager-config
        - name: tmpl-volume
          configMap:
            name: alertmanager-tmpl
        - name: data-volume
          emptyDir: {}

kubectl create -f prometheus-alertmanager.yml
kubectl -n monitoring get deployments.apps

也可查看挂载：

kubectl -n monitoring exec -it alertmanager-786cfb86c4-jntn7 -- ls /etc/config/
kubectl -n monitoring exec -it alertmanager-786cfb86c4-jntn7 -- ls /etc/config/template

创建服务：

vim prometheus-alertmanager-svc.yml
kind: Service
apiVersion: v1
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: prometheus
    component: alertmanager
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9093
      nodePort: 30093
  selector:
    app: prometheus
    component: alertmanager
  type: NodePort

kubectl create -f prometheus-alertmanager-svc.yml
kubectl -n monitoring get svc

通过节点IP:30093即可访问到其：

修改Prometheus配置，使其在触发告警后可将信息传递给Alertmanger。

vim prometheus-config.yml
...
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - alertmanager:80
      alert_relabel_configs:
      - source_labels: [kubernetes_pod]
        action: replace
        target_label: pod
        regex: (.+)
      - source_labels: [pod_name]
        action: replace
        target_label: pod
        regex: (.+)
...

kubectl replace -f prometheus-config.yml
curl -X POST http://172.16.220.143:3009{0,3}/-/reload

 

打赏微海报分享
页码： 1 2 3 4

《K8S监控-Prometheus告警配置》有1条评论

发表评论

error: Content is protected !!