Init Jobs — Platform Bootstrap¶

Overview¶

AKKO uses a set of Kubernetes Jobs to bootstrap the platform on every helm install or helm upgrade. These jobs create PostgreSQL schemas, S3 buckets, Iceberg catalog (Polaris) Iceberg catalogs, Identity (Keycloak) OAuth clients, BI engine (Superset) dashboards, and AI functions. All jobs are idempotent — they can run repeatedly without side effects.

The init jobs are packaged in the akko-init sub-chart and replace the need for manual setup steps after deployment.

Architecture¶

helm install / upgrade
        |
        v
  +------------------+
  | akko-init chart  |
  +--------+---------+
           |
  +--------+--------+--------+--------+--------+--------+--------+--------+
  |        |        |        |        |        |        |        |        |
  v        v        v        v        v        v        v        v        v
postgres postgres storage   polaris keycloak keycloak superset  ai     bootstrap
 -init   -data    -init   -init  -clients  -sync  -bootstrap func   -admin
         -init

Jobs¶

Job	Purpose	Depends On
postgres-init	Creates infrastructure databases (keycloak, airflow, superset, polaris, openmetadata, mlflow, jupyterhub) and extensions	PostgreSQL (infra)
postgres-data-init	Creates functional databases (banking, healthcare, geospatial, rag) with PostGIS + pgvector extensions and seed data	PostgreSQL (data)
storage-init	Creates S3 buckets (akko-warehouse, akko-production, akko-projects, akko-shared, akko-users) and access policies	object storage
polaris-init	Creates the `akko-warehouse` Iceberg catalog and namespaces (banking, healthcare, retail, telecom, manufacturing, raw, staging, analytics)	Apache Polaris
keycloak-clients	Configures OAuth2 redirect URIs for all services behind oauth2-proxy, using wildcard domain patterns	Identity
keycloak-sync	Synchronizes LDAP users from directory service into Identity (when directory service is enabled)	Identity, directory service
superset-bootstrap	Creates datasets, charts, and the demo dashboard in BI engine via the API	BI engine, Query engine (Trino)
ai-functions	Installs PL/Python AI functions in PostgreSQL Data (LLM-powered SQL functions)	PostgreSQL (data), AI gateway (LiteLLM)
bootstrap-admin	Creates the initial admin user and assigns RBAC roles	Identity

Idempotency¶

Every job uses CREATE ... IF NOT EXISTS, PUT (not POST), or equivalent patterns. This means:

First install: everything is created from scratch
Subsequent upgrades: existing resources are left untouched, only missing resources are created
No data loss: jobs never DROP or DELETE existing data

-- Example: postgres-init ensure.sql
CREATE DATABASE keycloak WITH OWNER = akko_admin;
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE SCHEMA IF NOT EXISTS banking;

Configuration¶

Enabling / Disabling Jobs¶

Each job can be individually toggled:

akko-init:
  polaris:
    enabled: true
  storage:
    enabled: true
  ollama:
    enabled: true
  superset:
    enabled: true
  keycloak:
    enabled: true
  aiFunctions:
    enabled: true
  keycloakClients:
    enabled: true

Iceberg catalog Namespaces¶

The Iceberg catalog init job creates namespaces for data organization:

akko-init:
  polaris:
    catalog:
      name: akko-warehouse
      namespaces:
        - banking
        - healthcare
        - retail
        - telecom
        - manufacturing
        - raw
        - staging
        - analytics

object storage Buckets¶

akko-init:
  storage:
    buckets:
      - akko-warehouse
      - akko-production
      - akko-projects
      - akko-shared
      - akko-users

Backup CronJobs¶

The init chart also includes optional backup CronJobs for PostgreSQL and object storage:

akko-init:
  backup:
    postgres:
      enabled: false        # Enable in production
      schedule: "0 2 * * *" # 2 AM daily
      retention: 7          # Keep 7 backups
    storage:
      enabled: false
      schedule: "0 4 * * *"
      retention: 7

Helm Chart¶

helm/akko/charts/akko-init/
├── Chart.yaml
├── values.yaml
├── files/                          # SQL scripts, config files
└── templates/
    ├── _helpers.tpl
    ├── postgres-init-job.yaml
    ├── postgres-ensure-configmap.yaml
    ├── postgres-data-init-job.yaml
    ├── postgres-data-ensure-configmap.yaml
    ├── storage-init-job.yaml
    ├── polaris-init-job.yaml
    ├── keycloak-clients-job.yaml
    ├── keycloak-sync-job.yaml
    ├── superset-bootstrap-job.yaml
    ├── ai-functions-job.yaml
    ├── bootstrap-admin-job.yaml
    ├── backup-postgres-cronjob.yaml
    ├── backup-storage-cronjob.yaml
    ├── backup-pvc.yaml
    └── serviceaccount.yaml

Resource Usage¶

Init jobs are ephemeral and lightweight:

resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 128Mi

Jobs run to completion and are cleaned up by Kubernetes after success (ttlSecondsAfterFinished).

Troubleshooting¶

Job Failed — Pod in Error State¶

Symptoms: A job pod shows Error or BackoffLimitExceeded. The platform is partially initialized.

Cause: The target service (PostgreSQL, object storage, Identity, etc.) was not ready when the job ran. Network policies or secrets may be misconfigured.

Solution:

# Check which jobs failed
kubectl get jobs -n akko -l app.kubernetes.io/instance=akko | grep -v "1/1"

# View logs of the failed job
kubectl logs -n akko job/<job-name> --tail=50

# Re-run a specific job by deleting it (Helm will recreate on next upgrade)
kubectl delete job -n akko <job-name>
helm upgrade akko helm/akko/ -n akko -f helm/examples/values-dev.yaml

PostgreSQL Init: Permission Denied¶

Symptoms: The postgres-init job fails with permission denied or role does not exist.

Cause: The PostgreSQL superuser credentials in the Helm secret do not match what is configured in the PostgreSQL pod.

Solution:

# Check the init job logs
kubectl logs -n akko job/akko-postgres-init --tail=30

# Verify the secret matches the PostgreSQL config
kubectl get secret -n akko akko-postgresql -o jsonpath='{.data.postgres-password}' | base64 -d

# If credentials changed, the PostgreSQL volume retains the old password
# You may need to ALTER USER inside the running PostgreSQL pod
kubectl exec -n akko deploy/akko-postgresql -- psql -U postgres -c "ALTER USER akko_admin WITH PASSWORD 'newpassword';"

object storage Init: Bucket Already Exists¶

This is normal and expected. The object storage init job uses mc mb --ignore-existing, so pre-existing buckets are silently skipped.

Iceberg catalog Init: Catalog Creation Failed¶

Symptoms: The polaris-init job fails with HTTP 500 or connection refused.

Cause: Iceberg catalog may still be initializing its database schema. The job needs Iceberg catalog to be fully ready.

Solution:

# Check Polaris readiness
kubectl get pods -n akko -l app.kubernetes.io/name=akko-polaris

# View polaris-init logs
kubectl logs -n akko job/akko-polaris-init --tail=30

# Re-run the job
kubectl delete job -n akko akko-polaris-init
helm upgrade akko helm/akko/ -n akko -f helm/examples/values-dev.yaml

License¶

All init job code is part of the AKKO platform: Apache 2.0.