Woodpecker CI — self-hosted CI/CD¶

AKKO ships a self-hosted Woodpecker server (Apache-2.0, OSS) that replaces GitHub Actions. Pipelines run as pods on the same k3s cluster that hosts the rest of the platform — no external runners, no billing limits, no "job not started" failures when the GitHub free tier exhausts.

	Woodpecker (current)	GitHub Actions (legacy)
Cost	0 €	$0.008 / Linux minute past free tier
Concurrency	k3s scheduler	GitHub-imposed
Storage	local-path PVC	GHA cache
Visibility	`ci.<domain>` (oauth2-proxy gated)	github.com
Sovereignty	100 % on-prem	externally hosted

Architecture¶

GitHub repo  ──webhook──►  Woodpecker server (akko-woodpecker-server, port 8000)
                                 │
                                 │  gRPC :9000 (shared secret)
                                 ▼
                       Woodpecker agent (2 replicas)
                                 │
                                 │  k8s API (namespaced Role)
                                 ▼
                       Pipeline pods (one per step)
                                 │
                                 │  PVC /woodpecker (per-build)
                                 ▼
                       Logs streamed back to the server UI

Server: web UI, REST API, GitHub OAuth, scheduler. Backed by the shared akko-postgresql cluster (database woodpecker).
Agent: pulls work over gRPC, creates pipeline pods in the same namespace via a namespaced Role (no cluster-wide privilege).
Pipeline pods: ephemeral, one per step. Mount a per-build PVC (workspace.pvcSize: 10Gi default) so all steps in a pipeline share the same /woodpecker/<repo>/<branch> workspace.

File layout in this repo¶

.woodpecker/                              # auto-discovered by the server
├── 00-validate.yml                       # ACTIVE  — helm lint + template --strict + grep changeme
├── 01-helm-lint.yml                      # ACTIVE  — helm lint (dev + rendered)
├── 02-helm-template.yml                  # ACTIVE  — helm template + kubeconform
├── 03-no-hardcoding.yml                  # ACTIVE  — no hardcoded IPs/domains (R01)
├── 04-secrets.yml                        # ACTIVE  — gitleaks + no plaintext passwords
├── 05-doc-consistency.yml                # ACTIVE  — mkdocs nav, bilingual, service coverage
├── 06-trivy.yml                          # ACTIVE  — Trivy CRITICAL CVE scan (14 images)
├── 07-trino-ai-plugin.yml               # ACTIVE  — Maven build + unit tests
├── 08-deploy-netcup.yml.disabled         # DISABLED — deploy to Netcup (enable manually)
├── 09-tests.yml                          # ACTIVE  — pytest + Playwright smoke
├── 10-aden-tests.yml                     # ACTIVE  — ADEN unit tests + coverage gate
└── 12-post-deploy-tests.yml.disabled     # DISABLED — post-deploy E2E (depends on 08)

helm/akko/charts/akko-woodpecker/         # Helm sub-chart that runs server + agent

The numeric prefix gives a stable order in the Woodpecker dashboard; all pipelines run in parallel by default — the prefix is purely cosmetic.

Activating a disabled pipeline¶

Rename the file to remove .disabled:

cd .woodpecker/
mv 08-deploy-netcup.yml.disabled 08-deploy-netcup.yml

Commit and push. Woodpecker auto-discovers all .yml files in .woodpecker/.

Pipeline matrix¶

Pipeline	Trigger	Gate	Runs on PR?
`00-validate`	`helm/**` changes	lint + template + changeme	Yes
`01-helm-lint`	`helm/**` changes	lint dev + rendered	Yes
`02-helm-template`	`helm/**` changes	template + kubeconform	Yes
`03-no-hardcoding`	`helm/**` changes	grep forbidden patterns	Yes
`04-secrets`	any push / PR	gitleaks + plaintext scan	Yes
`05-doc-consistency`	any push / PR	mkdocs build, bilingual	Yes
`06-trivy`	`docker/**` changes	CRITICAL CVE scan	Yes
`07-trino-ai-plugin`	`docker/trino-ai-functions/**`	Maven test + package	Yes
`09-tests`	`tests/**` changes	pytest + Playwright	Yes
`10-aden-tests`	`docker/aden/**`	unit tests + 70% coverage	Yes
`08-deploy-netcup`	push to main only	helm upgrade (needs secrets)	No (disabled)
`12-post-deploy-tests`	after 08 succeeds	E2E health + UI	No (disabled)

Enable on a cluster¶

1. Create a GitHub OAuth App¶

GitHub → Settings → Developer settings → OAuth Apps → New OAuth App

Field	Value
Application name	`AKKO Woodpecker (<env>)`
Homepage URL	`https://ci.<domain>/`
Authorization callback URL	`https://ci.<domain>/authorize`

Note the Client ID + Client Secret.

2. Generate the agent shared secret¶

openssl rand -hex 32

3. Add to `helm/examples/values-dev-secrets.yaml` (gitignored)¶

akko-woodpecker:
  agent:
    secret: <agent-shared-secret>
  github:
    client_id: <oauth-app-client-id>
    client_secret: <oauth-app-client-secret>

4. Enable the sub-chart¶

# helm/examples/values-dev.yaml (or your overlay)
akko-woodpecker:
  enabled: true

5. Helm upgrade¶

helm upgrade akko helm/akko/ -n akko \
  -f helm/examples/values-dev.yaml \
  -f helm/examples/values-domain.yaml \
  -f helm/examples/values-dev-secrets.yaml \
  --set-file akko-keycloak.realm.data=helm/examples/realm-domain.json

6. Add the GitHub webhook¶

After the first OAuth login, Woodpecker prints the webhook URL — paste it into your GitHub repo (Settings → Webhooks → Add webhook). Content type application/json, secret = the agent shared secret.

7. Add deploy secrets in the Woodpecker UI¶

The 08-deploy-netcup.yml pipeline reads two secrets from the Woodpecker secret store (Repo Settings → Secrets):

Secret	Value
`kubeconfig`	content of `/etc/rancher/k3s/k3s.yaml` (the in-cluster kubeconfig the agent will use)
`akko_domain`	bare hostname, e.g. `159.195.77.208.nip.io`

Mark them available to push events only (not pull requests) so a malicious PR cannot deploy to prod.

Migrating a workflow from GitHub Actions¶

The Woodpecker schema is close enough to GHA that a 1:1 port is usually trivial. The two main differences:

GitHub Actions	Woodpecker
`jobs.<id>.steps.[].uses: actions/checkout@v4`	implicit — every pipeline gets a fresh git clone
`runs-on: ubuntu-latest`	N/A — every step picks its own `image:` (Docker)
`env:` (job-level)	`environment:` (step-level)
`secrets.MY_SECRET`	`secrets: [my_secret]` + `from_secret:` env wiring
`if: github.ref == 'refs/heads/main'`	`when:` block at the top of the pipeline file

Troubleshooting¶

Pipeline stuck "Pending" — agent has reached WOODPECKER_MAX_WORKFLOWS=4. Either wait, or bump replicas in helm/akko/charts/akko-woodpecker/values.yaml.
agent secret mismatch — the agent and server have different WOODPECKER_AGENT_SECRET. Re-roll: edit the Helm secret + restart both deployments.
github oauth: invalid client — double-check the callback URL in the GitHub OAuth App matches https://ci.<domain>/authorize exactly.
postgres connection refused — the server is starting before akko-postgresql is ready. Bump readinessProbe initialDelaySeconds, or rely on the post-install hook to wait.

Roadmap¶

[ ] OPA-style approval gates for 08-deploy-netcup (manual click from akko-admin in the UI before helm upgrade runs).
[ ] Slack notifications on failure (Woodpecker has a built-in notify plugin).
[ ] Metrics endpoint — Woodpecker exposes Prometheus metrics on /metrics; add a ServiceMonitor.