Skip to content

ADR-027: Storage Backend — SeaweedFS as Default, BYOS for External

Status

Accepted — 2026-04-25

Context

AKKO ships an S3-compatible object store as a load-bearing layer: Iceberg tables (Polaris), MLflow artifacts, Quarto published reports, Spark intermediate data, RAG document blobs and the catalog manager all depend on it. Until Sprint 42 the default backend was MinIO.

Two events forced a re-evaluation in Q1 2026:

  1. License flip. MinIO Inc. relicensed the community server under AGPL v3 in 2024. AGPL is incompatible with AKKO's commercial SaaS posture (any network use of the binary forces source disclosure of the calling stack). See feedback_no_confidential_in_public.md and the AGPL alert in MEMORY.md.
  2. Maintenance mode. On 2025-12-03 MinIO Inc. publicly moved the OSS edition to maintenance mode, removing features and pushing customers toward the commercial AIStor product. Single-vendor + maintenance mode = a load-bearing dependency we no longer control.

We need a storage layer that is (a) S3-compatible so the rest of the platform stays untouched, (b) under a commercial-friendly license, (c) deployable in k3d / k3s / EKS / GKE / AKS / OpenShift / bare-metal / air-gapped, and (d) swappable so customers can BYOS their own object store.

Considered Options

Option License Governance Vitality (commits/mo) Maturity Cloud-native Score
SeaweedFS Apache 2.0 Single-vendor (chrislusf) High (200+) Production Yes (Helm OK) 8.0
Ceph via Rook LGPL 2.1 / Apache 2.0 Linux Foundation / CNCF Very high Battle-tested Yes (Operator) 7.5
CubeFS Apache 2.0 CNCF Incubating Medium Production Yes 7.0
Garage AGPL v3 Deuxfleurs (assoc.) Medium Production Yes 3.0 (eliminated — AGPL)
RustFS Apache 2.0 Single-vendor Medium Alpha Yes 4.0 (eliminated — alpha)
JuiceFS CE Apache 2.0 Single-vendor (Juicedata) High Production POSIX-first 5.5 (poor S3 fit)
MinIO (status quo) AGPL v3 Single-vendor (maintenance mode) Frozen Production Yes 2.0 (eliminated)

Decision

SeaweedFS is the packaged default. AKKO ships it as a sub-chart with sensible defaults (3 volume servers, EC 4+2 erasure coding, S3 gateway on :8333).

External / BYOS mode is a first-class option. A storage.mode: external value bypasses the bundled chart and points every consumer (Polaris, MLflow, Spark, Trino) at a customer-provided endpoint: AWS S3, GCS via interop, Azure Blob via the S3 proxy, Ceph RGW, MinIO Enterprise (AIStor) or any other S3-compatible target.

We follow the Lego Pattern documented in ADR-029: one packaged default, one BYOS mode, never two parallel packaged implementations. This keeps the test matrix bounded.

Rationale

  • Apache 2.0 removes the AGPL blocker.
  • S3 API preserves zero-cost migration: Iceberg, MLflow, Spark, Quarto and the catalog already speak S3, no code change.
  • Single-vendor risk is mitigated by the BYOS escape hatch — any AKKO customer who distrusts the upstream can flip to Ceph/AWS/GCS in one Helm value.
  • Ceph via Rook scored close but adds a heavy operator footprint (CRDs, MON/OSD/MGR pods, ~3 GB RAM minimum) that hurts our 16 GB MacBook dev target and air-gapped edge ARM scenarios.
  • CubeFS is promising but its CNCF status is Incubating, the S3 gateway is younger and the community is mostly Chinese, increasing onboarding friction for our EU pilots.

Consequences

Positive

  • License clean — SaaS commercialisation unblocked.
  • Single Helm value storage.mode={packaged|external} covers every customer profile.
  • SeaweedFS resource footprint (~300 MB RAM idle) fits k3d / 16 GB dev.
  • akko-minio:9000 compat Service preserves zero-touch migration for legacy consumers (Spark history server, old notebooks).

Negative

  • SeaweedFS governance is single-vendor. We accept the risk because the BYOS mode protects every customer from a future relicensing.
  • Erasure coding semantics differ slightly from MinIO; one Spark-on-S3 multipart upload edge case required an extra retry config (handled in helm/akko/charts/akko-spark/values.yaml).
  • Bucket policy syntax is a subset of AWS/MinIO IAM; the Polaris init job had to be reworked (commit 90cda21).

Neutral

  • Customers running MinIO Enterprise simply set storage.mode=external — no feature regression for them.

Implementation

  • Sub-chart: /Users/ab2dridi/newera/akko/helm/akko/charts/akko-storage/
  • templates/seaweedfs-master.yaml, seaweedfs-volume.yaml, seaweedfs-filer.yaml, seaweedfs-s3.yaml
  • templates/external-secret.yaml (BYOS credentials)
  • templates/compat-service.yaml (akko-minio:9000 -> SeaweedFS S3 gateway)
  • Values: helm/akko/values.yaml -> storage.mode, storage.packaged.replicas, storage.external.endpoint
  • Polaris config patched to use the S3 gateway: polaris/application.properties
  • Smoke test: tests/integration/storage-swap-smoke.py (round-trip PUT / HEAD / GET / DELETE on a 32 MB object, verify byte-equality)

Validation

  • Sprint 43 deploy (Netcup): SeaweedFS handled 1.2 TB of Iceberg parquet without a single 5xx over 72 h.
  • BYOS mode validated against AWS S3 (eu-west-3) on a customer dry-run.
  • Audit cadence: every 6 months we re-score the comparison table; any drop below 7.0 for SeaweedFS triggers a P1 ticket to evaluate Ceph / CubeFS as the new default.
  • CI: tests/run-all.sh exercises both packaged and external modes via Kind matrix.

References

  • Sprint 43 commits: 4bc1a12 (chart skeleton), d4e4a46 (Polaris repointing), 90cda21 (compat shim + smoke test)
  • MinIO 2024 license change: https://github.com/minio/minio/pull/19771
  • MinIO maintenance mode announcement, 2025-12-03
  • SeaweedFS upstream: https://github.com/seaweedfs/seaweedfs
  • Related: ADR-020 (monitoring relicensing), ADR-029 (governance > license)