Skip to content

Runbook: LokiIngestionSlow

Alerte : LokiIngestionSlow (PrometheusRule, severity warning)

Symptôme :

Log ingestion rate < 80% of target, or queue size > 5000 entries for more than 10 min. Risk: log loss (log shipper buffer saturated).

Severity : 🟡 warning, notif Slack #akko-observability

Legacy alert, opt-in only

Read this first. The default AKKO logs path is Fluent Bit -> VictoriaLogs. There is no Loki and no Promtail in a stock install.

The alert name LokiIngestionSlow is kept for backward compatibility. It is gated behind loki.enabled in prometheusrule-akko.yaml, and loki.enabled defaults to false. It only fires when a client explicitly re-enables a Loki backend with loki.enabled: true. On a default install this alert never fires.

If you are on the default path, jump straight to Default path: Fluent Bit -> VictoriaLogs.

Default path: Fluent Bit -> VictoriaLogs

Slow or lost logs on a stock install come from one of three places: VictoriaLogs ingestion health, Fluent Bit backpressure, or VictoriaLogs storage/retention.

1. VictoriaLogs ingestion health

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl port-forward -n akko svc/akko-victorialogs 9428:9428 &
curl -sS http://localhost:9428/metrics | grep -E "vl_rows_ingested_total|vl_http_requests_total"

Key metrics: - vl_rows_ingested_total (should keep growing) - vl_http_requests_total{path="/insert/loki/api/v1/push"} (Loki-compat push endpoint) - vl_free_disk_space_bytes (should stay well above zero)

Also check the VictoriaLogs pod is healthy:

kubectl get pod -n akko -l app.kubernetes.io/name=victoria-logs-single
kubectl logs -n akko -l app.kubernetes.io/name=victoria-logs-single --tail=200 | grep -iE "error|reject|cannot"

2. Fluent Bit backpressure (the log shipper)

kubectl get ds akko-fluent-bit -n akko
kubectl logs ds/akko-fluent-bit -n akko --tail=200 | grep -iE "error|retry|backpressure|drop|chunk"

Symptoms of backpressure: - [warn] [engine] failed to flush chunk (VictoriaLogs not accepting fast enough) - [error] [output] ... connection refused (VictoriaLogs service down or wrong endpoint) - growing retry counts

Check the Fluent Bit config (endpoint, mem_buf_limit, retry policy):

kubectl get configmap akko-fluent-bit-config -n akko -o yaml | grep -A30 '\[OUTPUT\]'

3. VictoriaLogs storage and retention

VictoriaLogs default retention is 14d on a 50Gi PVC. If the PVC is full, ingestion slows or stalls.

kubectl get pvc -n akko -l app.kubernetes.io/name=victoria-logs-single
kubectl exec -n akko -it <victorialogs-pod> -- df -h /storage

Causes fréquentes + fix (default path)

Cause Symptôme Fix
VictoriaLogs PVC full vl_free_disk_space_bytes near 0, ingestion stalls Reduce retention (14d default) or bump storageSize in akko-observability values, then helm upgrade
Fluent Bit flush failing failed to flush chunk in akko-fluent-bit logs Check VictoriaLogs service reachable; raise mem_buf_limit in akko-fluent-bit-config
VictoriaLogs service down connection refused from Fluent Bit Restart / check akko-victorialogs pod, verify service on port 9428
Endpoint misconfigured 404 on push, 0 rows ingested Fix OUTPUT endpoint in akko-fluent-bit-config (Loki-compat /insert/loki/api/v1/push or native VictoriaLogs push)
High log volume spike ingestion lag, growing retries Reduce noisy sources at the Fluent Bit INPUT/FILTER level

Reduce log volume (fix #1 for spikes)

Trim noisy sources at the Fluent Bit level rather than dropping data at storage. Fields to avoid promoting to high-cardinality stream fields: - pod_name (changes on every restart) - container_id - trace_id - request_id - user_id without bucketing

Prefer stable low-cardinality fields: - namespace, app, component - level (info/warn/error, a handful of values)

Fix pérenne (R02)

Adjust VictoriaLogs retention and size

# helm/akko/charts/akko-observability/values.yaml (logs slot)
logs:
  backend: victorialogs
  victorialogs:
    retention: "14d"        # default; lower it to free space, or raise storageSize
    storageSize: 50Gi

Then commit + push + helm upgrade. Never kubectl edit configmap.

Free VictoriaLogs storage

VictoriaLogs enforces retention automatically. If the PVC is full before retention kicks in, either lower retention or raise storageSize in the values above, then helm upgrade. Do not edit data on the PVC by hand.

Prévention

  • Perses dashboard "Logs Ingestion" (cockpit Logs page) with panels:
  • Rows/s ingested (vl_rows_ingested_total)
  • Fluent Bit flush retries / dropped chunks
  • VictoriaLogs free disk space
  • PrometheusRule précoce : warning when free disk space drops below 20%
  • Review mensuel of noisy log sources in akko-fluent-bit-config

Lessons learned

  • L10 : VictoriaLogs retention is 14d by default. VictoriaLogs is not a data warehouse: logs older than the retention window should go to cold storage (object storage direct).

Liens utiles