Alertmanager

Alertmanager config

When adding the first Teams Channel specific to the cluster, you need to copy the default routes: and recipients: section from the common values.yaml and add this to the cluster specific yaml file.

The example below takes the default ‘null’ and ‘enabling_alerts’ configuration and adds a configuration for team ‘leefomgeving’ using a webhook secret stored in the external secret backend with the key ‘leefomgeving_alerts’:

alertmanager:
  config:
    receivers:
      - name: "null"
      - name: enabling_alerts
        msteams_configs:
          - send_resolved: true
            webhook_url_file: /var/webhooks/enabling_alerts
            title: '{{ if eq .Status "firing" }}🚨 **Enabling team FIRING** 🔥{{- else -}}🙌 **Enabling team RESOLVED** 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'
      - name: leefomgeving_alerts
        msteams_configs:
          - send_resolved: true
            webhook_url_file: /var/webhooks/leefomgeving_alerts
            title: '{{ if eq .Status "firing" }}🚨 **Leefomgeving FIRING** 🔥{{- else -}}🙌 **Leefomgeving RESOLVED ** 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'
    route:
      routes:
        - receiver: "null"
          matchers:
            - severity = none
          continue: false
        - receiver: enabling_alerts
          matchers:
            - team = enabling
            - severity =~ "error|critical"
          continue: true
        - receiver: leefomgeving_alerts
          matchers:
            - team = leefomgeving
            - severity =~ "error|critical"
          continue: true

Adding custom alert rules

Custom alerts for your application can be defined using the Prometheus rules custom resource. Take care to add the label mapping the alert to the recipient defined in the earlier step. Also add a label to have the FLOW instance of alertmanager pick this custom resource up. You can add the custom resource to your own namespace and manage them in an helm chart if you like.

Required labels for our FLOW specific alertmanager:

For the prometheusRule resource:

    flow.rws.nl/observe: "true"

For any alert you define in the custom rule:

  team: [team name used in the routes section in the step above]

Example

Below an example of a Prometheus rules custom resource with the required label to be picked up by the FLOW observability stack. The rule uses Prometheus data to check on Argo CD application health.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    flow.rws.nl/observe: "true"
  name: argocd-application-controller
spec:
  groups:
  - name: argocd
    rules:
    - alert: ArgoAppUnhealthy
      annotations:
        description: |
          The application {{ $labels.name }} has been unhealthy for over 10 minutes which means that the rollout of this application has failed.
        summary: Application not healthy
      expr: |
        argocd_app_info{health_status!="Healthy"} == 1
      for: 10m
      labels:
        team: enabling
        severity: critical
    - alert: ArgoAppNotSynced
      annotations:
        description: |
          The application {{ $labels.name }} has not been synchronized for over 2 hours which means that the state of this application has drifted away from the state inside Git.
        summary: Application not synchronized
      expr: |
        argocd_app_info{sync_status!="Synced"} == 1
      for: 2h
      labels:
        team: enabling
        severity: error

More information about defining alerts can be found in the upstream documentation linked below.

Alertmanager

Alertmanager config

Adding custom alert rules

Example

External Documentation