Harbor Container Registry¶

Harbor is the container registry AKKO uses to store its custom Docker images and Helm charts (OCI format). It is deployed independently from the AKKO release, with its own PostgreSQL and its own storage, so that AKKO can be uninstalled, reinstalled, or moved without ever rebuilding images.

Overview¶

Harbor is a CNCF-graduated open-source container registry. AKKO uses it to host:

The 11 AKKO custom images (akko-postgres, akko-spark, akko-notebook, akko-cockpit, akko-trino, akko-ai-service, akko-mlflow, akko-airflow, akko-dbt, akko-mcp-trino, akko-mcp-openmetadata).
The AKKO umbrella Helm chart, packaged and pushed as an OCI artifact.

Harbor also ships with built-in Trivy vulnerability scanning, image signing, replication, robot accounts, and project-level RBAC.

Architecture¶

Harbor is independent from AKKO by design:

Concern	Harbor	AKKO
Namespace	`harbor`	`akko`
PostgreSQL	Bundled (in `harbor` namespace)	`akko-postgresql` (in `akko` namespace)
Object storage	Local-path PVC	object storage
Helm release	`harbor`	`akko`
Lifecycle	Long-lived infra	Application release

Consequences:

helm uninstall akko leaves Harbor (and all images) intact.
kubectl delete namespace akko leaves Harbor untouched.
Harbor can be upgraded without impacting AKKO workloads.
AKKO can be redeployed from scratch at any time, pulling the exact same pinned image tags from Harbor.

This separation is mandatory for any production topology where the build server, the registry, and the runtime cluster have different lifecycles.

Installation¶

A single script deploys Harbor with sane defaults for AKKO:

bash helm/scripts/deploy-harbor.sh

The script:

Creates the harbor namespace.
Adds the Harbor Helm repository.
Installs the official Harbor chart with bundled PostgreSQL and local-path PVCs.
Creates the akko project where all AKKO images will be pushed.

The full install takes ~3 minutes on a laptop-class machine.

Deploy Harbor before AKKO

Always deploy Harbor before the first AKKO install in production. This way helm install akko can pull images from Harbor from day one, and the cluster never depends on the build server being reachable.

Accessing Harbor¶

Surface	Endpoint
Web UI	`http://harbor.<domain>:30500`
Docker / Helm	`localhost:30500` (on the cluster node)
Default admin	`admin` / `akko-harbor-admin-2026`

Change the default password in production

The default akko-harbor-admin-2026 password is meant for local k3d / dev. In production, rotate the admin password before any image is pushed. See Security hardening below.

docker login localhost:30500
# Username: admin
# Password: akko-harbor-admin-2026

helm registry login localhost:30500 \
  --username admin \
  --password akko-harbor-admin-2026

Pushing Images¶

After building AKKO images locally with helm/scripts/build-images.sh, tag and push them to Harbor:

docker tag akko-cockpit:2026.04 localhost:30500/akko/cockpit:2026.04
docker push localhost:30500/akko/cockpit:2026.04

For all 12 images at once, rely on the build script's AKKO_REGISTRY flag:

AKKO_REGISTRY=localhost:30500/akko \
AKKO_PARALLEL=4 \
  bash helm/scripts/build-images.sh

This builds every custom image in parallel and pushes the full set under the akko project. Tags follow the YYYY.MM convention (e.g. 2026.04). The registry never contains a latest tag.

Pulling Images (k3s / k3d)¶

The Kubernetes nodes must authenticate with Harbor to pull AKKO images.

k3s:

# /etc/rancher/k3s/registries.yaml
mirrors:
  "localhost:30500":
    endpoint:
      - "http://localhost:30500"
configs:
  "localhost:30500":
    auth:
      username: admin
      password: akko-harbor-admin-2026

Then restart the k3s agent so it picks up the new config:

systemctl restart k3s

EKS / GKE / AKS / OpenShift: use imagePullSecrets instead. Create a docker-registry secret in the akko namespace and reference it through global.imagePullSecrets in the Helm values. A robot account (least privilege, pull-only) should be used instead of the admin user.

Helm Chart OCI Push¶

AKKO's Helm chart is itself pushed to Harbor as an OCI artifact, so the runtime cluster can helm install without needing access to the git repo:

helm package helm/akko/
helm push akko-2026.04.tgz oci://localhost:30500/akko

Install from Harbor:

helm install akko oci://localhost:30500/akko/akko --version 2026.04 \
  -n akko --create-namespace \
  -f values-dev.yaml \
  -f values-domain.yaml \
  -f values-dev-secrets.yaml \
  --set-file akko-keycloak.realm.data=realm-domain.json

Backup¶

Harbor owns two pieces of stateful data:

PVC	Content	Backup strategy
`harbor-database`	PostgreSQL (projects, users, scan results, policies)	`pg_dump` nightly via CronJob
`harbor-registry`	Actual image layers (blobs)	`rsync` to external storage nightly

Example nightly pg_dump CronJob:

kubectl exec -n harbor deploy/harbor-database -- \
  pg_dumpall -U postgres > /backups/harbor-$(date +%F).sql

Example registry sync (blob store):

kubectl -n harbor cp harbor-registry-0:/storage /backups/harbor-registry/
# or rsync from the local-path node directory
rsync -a /var/lib/rancher/k3s/storage/harbor-registry/ \
  backup-host:/backups/harbor-registry/

Restore is the reverse: psql < dump.sql + rsync of the blob store, then helm upgrade Harbor to re-wire the config.

Troubleshooting¶

`unauthorized: authentication required` on `docker push`¶

The Docker credential cache is stale.

docker logout localhost:30500
docker login localhost:30500

`connection refused` on port 30500¶

Check that Harbor core and its nginx ingress are running.

kubectl get pods -n harbor
kubectl get svc -n harbor

If the harbor service is not of type NodePort on 30500, the deploy script needs to be re-run.

`project not found` when pushing¶

The akko project was not created (or was deleted). Recreate it via the Harbor API:

curl -X POST \
  -u admin:akko-harbor-admin-2026 \
  -H "Content-Type: application/json" \
  -d '{"project_name":"akko","public":false}' \
  http://localhost:30500/api/v2.0/projects

Pod `CrashLoopBackOff`¶

Inspect the logs of the failing component:

kubectl logs -n harbor deploy/harbor-core
kubectl logs -n harbor deploy/harbor-jobservice
kubectl logs -n harbor sts/harbor-database

The most common cause in dev is the database PVC running out of disk — check kubectl describe pvc -n harbor.

Security Hardening (production)¶

Rotate the admin password. Change harborAdminPassword both in the Helm values and via the Harbor API (the password is stored in the DB, the Helm value is only a seed).
Enable TLS. Set expose.tls.enabled: true and wire a real certificate, ideally via cert-manager with a ClusterIssuer.
Use robot accounts. Never let CI or clusters authenticate as admin. Create one robot account for push (CI) and one for pull (each target cluster), scoped to the akko project only.
Enable Trivy. Vulnerability scanning is on by default; enforce a severity threshold at the project level to block critical CVEs from being pulled.
Enable content signing. Notary / Cosign signing is off by default — enable it in production so clusters only pull signed images.
Enable audit logs. Ship Harbor audit logs to logs layer (or any SIEM) via log shipper, side by side with the rest of the AKKO audit trail.

Capacity Planning¶

PVC	Default size	Rule of thumb
`harbor-registry`	100 Gi	One full AKKO image set ≈ 15 GB → 100 Gi holds ~6 versions
`harbor-database`	20 Gi	Comfortable for millions of metadata rows
`harbor-chartmuseum` (if enabled)	5 Gi	Legacy; OCI push does not require it
`harbor-jobservice`	1 Gi	Scan jobs log directory

For long-lived registries, enable tag retention policies on the akko project (Harbor UI → Projects → akko → Policy → Tag Retention) to keep, for example, the last 5 tags of each repository, and garbage-collect unreferenced blobs weekly.

Next Steps¶

Installation — deploy AKKO pulling from Harbor.
Air-Gapped Deployment — use Harbor as the single source of truth for a fully offline cluster.
Troubleshooting — general AKKO troubleshooting.