Skip to content

Monitoring Stack

Metrics, logs, dashboards, and alerting.
Dashboards URL https://metrics.akko.local (Perses)
Metrics layer (Prometheus) URL https://prometheus.akko.local
Authentication Identity (Keycloak) SSO (Dashboards)
Helm sub-charts vmstack (victoria-metrics-k8s-stack), vlogs (victoria-logs-single), akko-observability (Perses + Fluent Bit)

Overview

AKKO includes a full observability stack that provides metrics collection, log aggregation, visualization, and alerting. All monitoring services start with the core profile, no extra flags needed.

The logs and dashboards backends live in the akko-observability Lego layer, which ships permissive Apache 2.0 / CNCF defaults (VictoriaLogs, Perses, Fluent Bit) in place of the AGPL Loki + Grafana + Promtail bundle.

                    metrics.akko.local
                          |
                     Traefik (TLS)
                          |
                      Perses (:8080)
                   /       |       \
          Prometheus  VictoriaLogs  Alertmanager
           (:9090)    (:9428)    (:9093)
              |          |
         scrape      Fluent Bit
         targets     (DaemonSet)
              |          |
      +-------+---+   pod stdout
      |       |   |   + node logs
    object storage  JHub  ...

Metrics vs Logs — Prometheus vs VictoriaLogs

The two pillars of the observability layer answer different questions. They are complementary, not competitors: Prometheus tells you how much / how fast, VictoriaLogs tells you what happened.

Prometheus VictoriaLogs
Data type Metrics, numeric time series (CPU %, memory, latency, req/s, pod count) Logs, timestamped text lines (app messages, errors, audit trail)
Example http_requests_total{pod="x"} 4213 at time T 2026-06-14 01:12 ERROR trino: query failed for user carol_analyst
Answers "Is the system healthy? What are the trends?" "Why did it break? What was user X doing?"
Ingestion Pull, periodically scrapes /metrics endpoints Push, logs are shipped to it (by Fluent Bit)
Query language PromQL (rates, aggregations, percentiles) LogsQL (full-text search, filters, patterns)
Volume profile small and regular (one number every 15s) large and bursty (floods of text)
Replaces (de-facto standard) Loki (AGPL). VictoriaLogs is Apache 2.0

How they fit together in AKKO

  • Prometheus scrapes metrics, visualized in Perses (Platform Dashboards, metrics.<domain>), then Alertmanager fires when a metric crosses a threshold.
  • VictoriaLogs stores logs (fed by Fluent Bit, which collects them from every pod), explored in the cockpit Logs page (logs.<domain>).

Typical diagnosis flow: a Prometheus metric alerts you that something is wrong (latency spiking), then you dive into VictoriaLogs logs to understand why.

Analogy

Prometheus is the car's dashboard (speed, temperature, fuel, continuous numbers). VictoriaLogs is the flight recorder / logbook (the detailed, line-by-line story of what happened). Both live in the akko-observability Lego layer alongside Perses (dashboards), Alertmanager (alert routing) and Fluent Bit (log collection).

Service exposure (layer-first)

Neither raw engine is exposed at the edge. The metrics UI is Perses at metrics.<domain> ("Platform Dashboards"); logs are read from the cockpit Logs page. prometheus.<domain> and victorialogs.<domain> are intentionally not published, they are backend components, surfaced only in the cockpit Architecture page and health checks.

Components

Metrics layer (Prometheus)

Metrics collection engine. Scrapes targets every 15 seconds and evaluates alert rules every 15 seconds. Deployed via the victoria-metrics-k8s-stack chart (Helm key vmstack).

Setting Value
URL https://prometheus.akko.local
Internal port 9090
Config monitoring values in helm/akko/values.yaml
Scrape interval 15s

Active scrape targets:

Job Target Metrics Path
prometheus localhost:9090 /metrics
alertmanager akko-alertmanager:9093 /metrics
storage akko-storage:9000 /storage/v2/metrics/cluster
jupyterhub akko-jupyterhub:8000 /hub/metrics

Query engine (Trino) and Orchestration engine (Airflow) metrics

Query engine and Orchestration engine scrape targets are currently disabled because their health endpoints return JSON, not Metrics layer format. To enable them:

  • Query engine: Deploy JMX Exporter as a Java agent in the Query engine image (port 9483)
  • Orchestration engine: Deploy a statsd_exporter sidecar or install apache-airflow-providers-statsd

Dashboards (Perses)

Visualization and dashboard platform (Perses, Apache 2.0, CNCF Sandbox). Deployed in the akko-observability layer. Pre-configured with the Prometheus and VictoriaLogs datasources.

Setting Value
URL https://metrics.akko.local
Internal service akko-perses
Internal port 8080
Image persesdev/perses:v0.50.0
Authentication Identity SSO
Dashboards helm/akko/charts/akko-observability/dashboards/*

Pre-configured datasources:

Datasource Type URL Default
Prometheus prometheus http://akko-prometheus-server:9090 Yes
VictoriaLogs loki (Loki-API compatible) http://akko-victorialogs:9428 No
Alertmanager alertmanager http://akko-alertmanager:9093 No

Logs layer (VictoriaLogs)

Log aggregation engine (VictoriaLogs, Apache 2.0). Receives logs from Fluent Bit and makes them queryable through Perses and the cockpit Logs page using LogsQL. It also exposes a Loki-compatible query API on the same port, so LogQL clients keep working during migration. Deployed via the victoria-logs-single chart (Helm key vlogs).

Setting Value
Internal service akko-victorialogs
Internal port 9428 (LogsQL push + query, Loki-compat)
Image victoriametrics/victoria-logs:v1.6.0-victorialogs
Retention 14d (default)

Log shipper (Fluent Bit)

Log collection agent (Fluent Bit, CNCF Graduated). Runs as a cluster DaemonSet, collects pod stdout and node logs, and forwards them to VictoriaLogs with labels for service name, container, and other metadata.

Setting Value
DaemonSet akko-fluent-bit
ConfigMap akko-fluent-bit-config
Image cr.fluentbit.io/fluent/fluent-bit:3.1.10
Source Pod stdout + node filesystem logs

Alertmanager

Alert routing and notification engine. Receives alerts from Metrics layer and routes them based on severity.

Setting Value
Internal port 9093
Default receiver Logging only (stdout)

Alert Rules

AKKO ships with pre-configured alert rules (rendered by the chart into a PrometheusRule):

Alert Severity Condition For
ServiceDown Critical up == 0 2 min
HighLatency Warning Response time > 2s 5 min
HighMemoryUsage Warning Container memory > 90% of limit 2 min
HighCPUUsage Warning Container CPU > 80% 5 min
DiskSpaceRunningLow Warning Filesystem > 85% used 5 min
PostgresConnectionsHigh Warning Connections > 80% of max 5 min
StorageBucketEmpty Info Bucket has 0 objects 10 min

Alert routing is configured with severity-based grouping:

  • Critical alerts: 5-second group wait, 15-minute repeat interval
  • All other alerts: 10-second group wait, 1-hour repeat interval
  • Inhibition: A critical alert suppresses warnings for the same service

Accessing Dashboards

  1. Open https://metrics.akko.local
  2. Log in with Identity SSO (e.g., alice with the admin role)
  3. Browse the pre-built Perses dashboards
  4. Query Prometheus metrics or VictoriaLogs logs directly

Querying Logs

To view logs from a specific AKKO service, use a LogsQL query against VictoriaLogs (via the cockpit Logs page or the Perses logs datasource):

{container_name="akko-trino"}

Filter by log level:

{container_name="akko-airflow"} |= "ERROR"

Querying Metrics

Query Prometheus with PromQL:

# Service availability (1 = up, 0 = down)
up

# JupyterHub active users
jupyterhub_running_servers

# object storage requests per second
rate(storage_http_requests_total[5m])

Adding Custom Alerts

Add rules to the chart's PrometheusRule via the monitoring values, for example:

groups:
  - name: my-custom-alerts
    rules:
      - alert: SlowTrinoQueries
        expr: trino_query_execution_time_seconds > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Slow Trino queries detected"
          description: "Queries taking longer than 30 seconds for 5+ minutes."

Prometheus automatically loads the rules provisioned by the chart.

Adding Custom Dashboards

Perses dashboards are managed as code. Place dashboard definitions under helm/akko/charts/akko-observability/dashboards/ and they are provisioned on deploy by the perses-dashboards-init Job.

Configuring Notifications

The default Alertmanager configuration uses a logging-only receiver (alerts appear in stdout). To enable real notifications, edit the monitoring Alertmanager config in your values overlay:

receivers:
  - name: default
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#akko-alerts'
        send_resolved: true

receivers:
  - name: default
    webhook_configs:
      - url: 'http://your-webhook-endpoint:5001/'
        send_resolved: true

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'akko-alerts@example.com'
  smtp_auth_username: 'user'
  smtp_auth_password: 'pass'

receivers:
  - name: default
    email_configs:
      - to: 'team@example.com'
        send_resolved: true

After editing, restart Alertmanager:

kubectl rollout restart deploy/akko-alertmanager -n akko

Configuration Reference

Location Purpose
helm/akko/values.yaml (monitoring) Prometheus scrape targets, alert rules, Alertmanager routing
helm/akko/values.yaml (vlogs) VictoriaLogs storage and retention config
helm/akko/charts/akko-observability/values.yaml logs / dashboards / shipper slot backends
helm/akko/charts/akko-observability/dashboards/* Perses dashboard definitions
helm/akko/charts/akko-observability/templates/fluentbit-config.yaml Fluent Bit log scraping config

Kubernetes Deployment

Prometheus and Alertmanager are deployed via the victoria-metrics-k8s-stack chart (key monitoring in values.yaml). VictoriaLogs is deployed via the victoria-logs-single chart (key vlogs); Perses and Fluent Bit ship in the akko-observability layer. Fluent Bit collects pod logs from the Kubernetes node filesystem, not the Docker socket.