Skip to content

Wire Alertmanager

AKKO v2026.04 ships Alertmanager with a log-only default receiver so no notification leaks during a dry-install. To satisfy DORA Art. 10 (detection) and NIS2 Art. 21(2)(b) (incident handling), wire real destinations before opening production traffic.

Overlay

Use helm/examples/values-alertmanager-receivers.yaml as a starting point:

helm upgrade akko oci://harbor.akko-ai.com/akko-charts/akko \
  --version 2026.04 \
  -f values-harbor.yaml \
  -f values-domain.yaml \
  -f values-dev-secrets.yaml \
  -f values-alertmanager-receivers.yaml \
  --set alertmanager.slackApiUrl=<webhook> \
  --set alertmanager.pagerDutyKey=<service-key> \
  --set alertmanager.emailTo=ops@customer.example

Routing topology

critical → PagerDuty  +  #akko-incidents (Slack)
warning  → #akko-alerts (Slack)
SLO breach → email (ops@)
everything → akko-default (log-only, for audit)

Critical alerts double-notify (PagerDuty + Slack) to survive a single channel outage.

Secrets

Inject creds via values-dev-secrets.yaml (gitignored) or SealedSecrets / ExternalSecrets. Never commit webhook URLs or SMTP passwords.

alertmanager:
  slackApiUrl: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX"
  pagerDutyKey: "your-integration-service-key"
  emailTo: "ops@customer.example"
  smtpHost: "smtp.office365.com:587"
  smtpUser: "alerts@customer.example"
  smtpPassword: "<masked>"

Inhibit rules

Two rules reduce noise:

  1. critical → inhibit warning for the same alertname + namespace
  2. NodeNotReady → inhibit every Pod* alert on that node

Validation

# Trigger a synthetic alert
kubectl -n akko exec svc/akko-alertmanager -- \
  amtool alert add TestAlert severity=critical namespace=akko
# Expect a PagerDuty page + #akko-incidents Slack post

# List active alerts
kubectl -n akko exec svc/akko-alertmanager -- amtool alert

Observability

  • Prometheus counter alertmanager_notifications_total{integration}
  • Alertmanager UI at https://alertmanager.<domain>/ (admin-gated)
  • logs layer contains the akko-default log-only receiver JSON for forensic queries

Troubleshooting

Symptom Cause Fix
Alert fires but no PagerDuty page Key wrong or routing_key empty amtool config routes test
Slack returns 403 Webhook revoked Regenerate in Slack admin, bump Secret
Email stuck pending SMTP host/port unreachable from the cluster kubectl exec ... -- nc -zv smtp.office365.com 587
Flood of warnings Inhibit rules missing Confirm inhibit_rules block is rendered in rendered manifest