Skip to content

ADR-034: Helm chart distribution — OCI artifact in Harbor

Status

Accepted — 2026-04-25

Context

Until Sprint 46 the AKKO umbrella chart helm/akko/ was deployed to Netcup directly from a filesystem path:

helm install akko helm/akko/ ...

This pattern has three production-grade gaps:

  1. Not reproducible. A "release" is whatever was committed to main when the deploy script ran. Two operators on two days can install different chart bytes from the same main tag.
  2. Not versionable. Customers cannot pin to a specific chart release. Chart.yaml's version field is descriptive but not enforceable — nobody fails a deploy because the version was bumped.
  3. Not signable / not auditable. Filesystem paths cannot carry cosign attestations, SBOMs, or provenance. They are also invisible to the cluster: the cluster has no record of what was installed.

The 2026-04-25 CTO audit (Sprint 46 kick-off) flagged this as the single biggest distribution gap on the path to commercial parity with Databricks / Snowflake / Cloudera, who all expose versioned chart artifacts.

The reference deployment doc (docs/docs/admin/deploy-from-harbor.md) already describes an OCI install from oci://harbor.akko-ai.com/akko-charts/akko. The producer side — actually publishing the chart to that path — was the missing piece.

Considered Options

Option A — Keep filesystem-only

Continue installing from helm/akko/. Customers Git-clone the repo.

Verdict: rejected. Fails reproducibility, versioning and provenance requirements simultaneously. Already documented as a gap.

Option B — ChartMuseum or HTTP repository

Stand up a ChartMuseum (or simple HTTP file server) and publish index.yaml on every release.

Verdict: rejected. Adds a second registry surface to operate (ChartMuseum + Harbor), with its own auth model, its own backups, and no content addressing. Cosign on a tarball-by-URL is awkward; signature verification requires a side-channel.

Option C — OCI artifact in Harbor (chosen)

Publish the chart as an OCI artifact at oci://harbor.akko-ai.com/akko-charts/akko, the same registry that already hosts the AKKO container images.

Verdict: accepted. Single registry surface, content-addressed, native cosign support, Helm 3.8+ first-class, Harbor v2 RBAC reused for both images and charts.

Option D — Public Artifact Hub / OCI on ghcr.io

Publish to a public catalog (Artifact Hub) backed by GitHub Container Registry.

Verdict: rejected. AKKO's distribution model is sovereign and self-hosted (cf. ADR-022 rejecting GitHub Actions on the same grounds). Public catalogs introduce a US-vendor dependency and bypass Harbor's audit trail.

Decision

Publish the AKKO umbrella chart as an OCI artifact in the AKKO Harbor registry.

  • Coordinates: oci://${AKKO_HARBOR_URL}/${AKKO_HARBOR_CHART_PROJECT}/akko
  • Defaults: harbor.akko-ai.com / akko-charts
  • Producer: helm/scripts/push-chart-harbor.sh (manual + Woodpecker pipeline .woodpecker/40-helm-chart-push.yml)
  • Consumer: every cluster, including Netcup itself via bash helm/scripts/deploy-netcup-full.sh --from-oci

Sub-charts under helm/akko/charts/<sub-chart>/ are not pushed individually — they are bundled in the umbrella tarball and version-locked by Chart.yaml's dependencies. Pushing them separately would multiply the artifact surface for no operator benefit (we never reuse a sub-chart in isolation).

Rationale

  • Single source of truth. Images and charts live in the same Harbor project namespace, share auth (the same robot account pulls both), share backup, share lifecycle policies (retention rules apply uniformly).
  • Helm 3.8+ native. No external tooling. helm push, helm pull, helm install oci://… are all first-class. Helm 3.14+ is already the AKKO baseline (matches deploy-from-harbor.sh prerequisites).
  • Cosign-ready. OCI artifacts are signed by digest, identical to container images. Sprint 46 stream A1 will wire cosign sign into the publish pipeline; the artifact does not need to change.
  • Multi-channel-ready. Harbor accepts arbitrary tags on the same artifact. Future channels (stable, beta, nightly) become tag conventions, not separate registries.
  • Self-hosted, on-prem, audit-friendly. Harbor runs inside the AKKO cluster. Every pull is logged. No third-party CDN sees the customer's request pattern.

Consequences

Positive

  • Customers can pin to an exact chart version: --version 2026.4.1.
  • Multi-cluster portability — the same artifact installs on k3s, EKS, GKE, AKS, OpenShift, Outscale OKS, OVHcloud Managed Kubernetes from one command.
  • Provenance: every install can be traced back to the Harbor publish event and (post stream A1) the cosign signature.
  • The --from-oci flag in deploy-netcup-full.sh lets us validate the publish path end-to-end on Netcup itself before any customer.
  • Idempotent publish — push-chart-harbor.sh refuses to overwrite an existing version unless --force is set, preventing silent republishing.

Negative

  • Helm 3.7 is now unsupported on the consumer side. We already require 3.14+ but the OCI dependency makes this stricter (3.8 minimum).
  • One additional Woodpecker step (40-helm-chart-push.yml) per chart-touching push to main. Mitigated by path filters — doc-only commits skip it.
  • Bumping the chart version on every helm/akko/** change becomes a hard rule, not a soft convention. The publish script enforces this with the idempotency probe.

Neutral

  • Filesystem installs (helm install akko helm/akko/) keep working unchanged. This is required for offline / air-gapped builds and for local development. --from-oci is opt-in.

Implementation

File Role
helm/akko/Chart.yaml Chart metadata: home, sources, kubeVersion >= 1.28, annotations.artifacthub.io/* (consumed by Harbor UI). Version bumped to 2026.4.1.
helm/scripts/push-chart-harbor.sh Producer: lint → package → idempotency probe → Harbor project bootstrap → registry login → helm push. Honours AKKO_HARBOR_URL, AKKO_HARBOR_CHART_PROJECT, AKKO_HARBOR_PASSWORD.
.woodpecker/40-helm-chart-push.yml CI: dry-run gate + publish + cosign placeholder (stream A1 follow-up). Triggers on push to main + tag v* + manual. Path-filtered.
helm/scripts/deploy-netcup-full.sh Consumer: --from-oci flag pulls the chart from Harbor instead of the filesystem. Default unchanged (filesystem).
docs/docs/admin/deploy-from-harbor.md (+ .fr.md) User-facing install instructions referencing the OCI coordinates.
docs/docs/getting-started/installation.md "Install from OCI Helm registry" section — multi-cluster portability example.

Multi-channel plan (post-Sprint 46)

Tag Source trigger Audience
2026.4.1 (immutable) tag v2026.4.1 on the repo customers, pinned installs
stable re-tag of latest passing release enterprise auto-upgrade
beta push to release/* branches pilot customers
nightly push to main (today) internal Netcup, CI smoke

The producer script supports this via --version overrides; the channel re-tags will be added in a future ADR once the v* tag flow has shipped at least one release.

Validation

  • Local dry-run: bash helm/scripts/push-chart-harbor.sh --dry-run packages the chart into /tmp/akko-charts/akko-<version>.tgz and exits 0. No Harbor reachability required.
  • CI publish gate: .woodpecker/40-helm-chart-push.yml runs the dry-run as a separate step before the publish step. If lint or package fail, no Harbor write happens.
  • Idempotency: re-running the producer at the same version exits non-zero with an explicit message. --force is required to overwrite, and only CI release jobs are expected to set it.
  • End-to-end: bash helm/scripts/deploy-netcup-full.sh --from-oci pulls the published chart and re-installs Netcup from it. A green smoke-test run after this is the gate that closes stream A4.

References

  • ADR-022 — CI strategy (rejected GitHub Actions, chose Woodpecker)
  • ADR-027 — Storage backend (Harbor is the registry layer for both images and charts)
  • ADR-028 — Observability backend (same Harbor pull pattern)
  • ADR-029 — Governance over license (Harbor is CNCF Graduated, Helm is CNCF Graduated)
  • Sprint 46 master plan — /Users/ab2dridi/newera/akko-technical-map/sprints/sprint-46-master-plan.md
  • Stream A1 (cosign signing) — to be wired into the placeholder step in 40-helm-chart-push.yml
  • Helm OCI documentation — https://helm.sh/docs/topics/registries/
  • Harbor v2.0 API — https://goharbor.io/docs/2.0.0/build-customize-contribute/configure-swagger/