Deploying without GitOps¶

The intended path onto a cluster is an ArgoCD Application that consumes the OCI artefact ghcr.io/stuttgart-things/schmetterpause-kustomize and patches it per environment — that is issue #81. This document describes the other path: the same manifests straight onto a cluster with task and kubectl, no Argo in between.

It is meant for a test cluster, for the first bring-up of a new environment, and for fault-finding when it is unclear whether a problem comes from the application or from Argo. It is not meant for continuous operation: there is no drift detection and no prune.

Without a cluster at all, the same image runs on Azure Container Apps with a managed Postgres. That path is Terraform and is described in terraform/README.md. It mirrors the kcl module setting for setting, and a change to either lands in both.

The examples use cicd-test2 throughout. Replace the cluster domain, the gateway name and the store name with the target environment's.

What the cluster has to bring¶

Three things, plus one that depends on the secrets variant you pick. This is the quick check before a bring-up; what each of them has to be -- versions, listeners, storage, and what backups need on top -- is in Platform requirements.

# 1. CloudNativePG operator
kubectl -n postgres get deploy cloudnative-pg

# 2. A programmed gateway with a TLS listener
kubectl -n default get gateway cilium-gateway
kubectl -n default get gateway cilium-gateway \
  -o jsonpath='{range .spec.listeners[*]}{.name}  {.hostname}  {.tls.certificateRefs[*].name}{"\n"}{end}'

# 3. Does the gateway accept routes from other namespaces?
kubectl -n default get gateway cilium-gateway \
  -o jsonpath='{range .spec.listeners[*]}{.name}  {.allowedRoutes.namespaces.from}{"\n"}{end}'

# 4. A StorageClass for Postgres
kubectl get storageclass

On 3: the Gateway API default is Same. Unless that says All or carries a matching selector, the gateway will not accept the HTTPRoute from the application's namespace — visible only on the route, not on the gateway.

The External Secrets Operator is not a prerequisite. It is the more comfortable of two paths, not the only one; the next section puts both side by side.

Secrets: two paths¶

The application needs two Secrets, and they are named the same under either variant, because the Deployment reads them by name:

Secret	Keys	Read by
`schmetterpause-app`	`SP_SESSION_KEY`, optionally `SP_KIOSK_TOKEN`	the application
`schmetterpause-db`	`username`, `password`, `SP_DATABASE_URL`	CloudNativePG and the application

schmetterpause-db is of type kubernetes.io/basic-auth and does double duty: CloudNativePG reads the owner credentials from it at initdb, the application reads SP_DATABASE_URL. Both coming from the same Secret is why the role and the DSN cannot drift apart.

Two Secrets rather than one is least privilege: the migration initContainer gets only the database Secret and never sees the session key.

Changing SP_SESSION_KEY is not a restart, it is logging every player out. The test TestARestartWithADifferentKeyForgetsEverybody records that.

Variant A — External Secrets¶

The path for a cluster that has a ClusterSecretStore anyway. Both Secrets are then produced from one Vault entry and refreshed on a schedule; the repository and the command line hold only a path, never a value.

kubectl get crd | grep externalsecrets.external-secrets.io
kubectl get clustersecretstore

The store is named vault-<cluster>, never plain vault. kubectl get clustersecretstore is the reliable source. A store reporting Ready=True proves the login works — not that the policy may read the entry. That only shows on the ExternalSecret.

The entry sits under the mount the store itself carries, and is named schmetterpause:

Key	Contents	Form
`session-key`	key for the session cookie	32 bytes, base64
`username`	the database's owner role	`schmetterpause`
`password`	that role's password	hex, no special characters
`kiosk-token`	only when the kiosk is on	hex

password and kiosk-token are deliberately hex rather than base64: the password is interpolated into a DSN unescaped, and an @, /, : or ? in it makes the URL parse differently. The session key passes through no URL and may be base64.

remoteRef.key holds the entry name and nothing else. The store already carries the mount and version: v2, and ESO composes <mount>/data/<key> from them. Both ways of getting this wrong point at something that exists nowhere, and report nothing at apply time:

schmetterpause                    correct
schmetterpause/data/cicd-test2    -> cicd-test2/data/schmetterpause/data/…
schmetterpause-cicd-test2         -> the cluster twice; the mount IS the cluster

Variant B — ordinary Kubernetes Secrets¶

The path for a cluster without ESO. One command creates both Secrets, generates the values, and assembles the DSN the way the ESO template would:

kubectl create ns schmetterpause
task kcl:secrets NAMESPACE=schmetterpause

The task creates and does not update. A second call aborts rather than setting a new session key — that would not be a failure anyone sees, it would be one where everybody is logged out the next morning.

With the kiosk: task kcl:secrets NAMESPACE=schmetterpause KIOSK=true.

Doing it by hand works the same way when the values come from somewhere else. All that matters is that the password inside SP_DATABASE_URL is the same one as under password, and that it contains no character that takes a URL apart.

From here on only the profile differs: variant A renders with PROFILE=base, variant B with PROFILE=existing-secrets.

Everything at once¶

With the operator in place and the secrets settled, the rest is one command:

task kcl:up PROFILE=~/environments/cicd-test2.yaml

The path in that example is a home directory on purpose. A profile is flat key: value with no apiVersion and no kind, so a GitOps engine that reconciles the directory it sits in will fail to decode it — and a failing root Kustomization applies nothing at all, not just the file it choked on. See "Where a profile must not live" in kcl/README.md; it happened, and it stalled a whole cluster.

It creates the namespace, applies the application, waits for schmetterpause-db to appear, and only then applies the CloudNativePG Cluster.

The wait keeps the single command independent of timing. CNPG reads the owner credentials from that Secret at initdb; under variant A the Secret does not exist at apply time but a few seconds later, once the ExternalSecret has synced. Applying the Cluster into that gap turns out to be harmless as well: measured on CloudNativePG 1.30, a Cluster without its Secret waits in Setting up primary and bootstraps about 30 seconds after the Secret appears. The order is kept because it costs nothing and reads better in a log, not because anything depends on it.

Everything else settles on its own, which is a property of Kubernetes rather than of this command: the pod reports secret not found, gets the Secret seconds later and is content; the migration initContainer crashloops against the missing database and gets through by itself once it is there.

The rest of this document is the same thing step by step — worth following on a first bring-up, and the place to look when kcl:up stops somewhere.

1. Postgres operator¶

Once per cluster, not per application:

helmfile apply \
  --file 'git::https://github.com/stuttgart-things/helm.git@database/postgres.yaml.gotmpl' \
  --state-values-set namespace=postgres \
  --state-values-set version=0.29.0

2. Namespace¶

The base deliberately renders none. Several applications can share a namespace, and when more than one ships it as a resource, ArgoCD reports a SharedResource — and then one application's prune cycle deletes the other's namespace.

kubectl create ns schmetterpause

3. Applying the manifests¶

task kcl:apply -- \
  -D config.image=ghcr.io/stuttgart-things/schmetterpause:3382ad1 \
  -D config.clusterDomain=cicd-test2.4sthings.tiab.ssc.sva.de \
  -D config.gatewayName=cilium-gateway \
  -D config.secretStoreName=vault-cicd-test2 \
  -D config.vaultPath=schmetterpause

For variant B, without the last two lines and with the other profile:

task kcl:apply PROFILE=existing-secrets -- \
  -D config.image=ghcr.io/stuttgart-things/schmetterpause:3382ad1 \
  -D config.clusterDomain=cicd-test2.4sthings.tiab.ssc.sva.de \
  -D config.gatewayName=cilium-gateway

Everything after -- is passed to kcl, behind the profile's own values, and a later -D wins. The profile supplies the ground, the command line the handful of values that make this environment.

Rendering without a profile is the trap. kcl run kcl/main.k -D … loads no profile; every field not passed then falls back to its default in schema.k rather than the profile's. httpRouteEnabled is false there and secretsMode is external — both deliberately reticent, and neither what a cluster wants. Hence task kcl:apply rather than kcl run.

task kcl:render shows the same thing without applying it. For more values than fit comfortably on a command line, PROFILE also takes a path to a file outside the repository — see examples/, which puts this against the other two ways of adapting an environment.

For variant A, check the secrets before anything else:

kubectl -n schmetterpause get externalsecret

Both have to report SecretSynced. The pod is not running at this point — that is correct, the database is still missing.

4. Postgres cluster¶

Only now, because CloudNativePG reads the owner password from schmetterpause-db at initdb, and that has to be in place first — in variant A as soon as the ExternalSecret has synced, in variant B since task kcl:secrets.

task kcl:apply ENTRY=kcl/database.k PROFILE=~/environments/cicd-test2.yaml

kcl/database.k is a separate entry point, not part of main.k. kcl:publish renders main.k, so no value anyone sets can put a Cluster into the published artefact — the base stays a base whose removal cannot take a database with it.

Rendering it from the same module is what stops four values drifting apart. initdb.owner, initdb.database and the Service the DSN dials all come from config.dbOwner, config.dbName and config.dbClusterName; the Secret named in initdb.secret is the one the DSN is built from. Previously these lived in a helmfile invocation, a Vault entry and this module, and keeping them equal was something a person did between two windows.

The one value the module cannot check is username in the secret store. It has to equal dbOwner, because that Secret is where CNPG reads the credentials.

Defaults: one instance, ghcr.io/cloudnative-pg/postgresql:18, 8Gi, the cluster's default StorageClass, no superuser access. Why one instance: replicas and backups protect against different things, and the realistic danger on a test cluster is a deliberate rebuild, against which a replica does nothing. The trigger for changing this is backups, not replicas.

Through the catalogue instead¶

Under ArgoCD the Cluster is its own Application from infra/cloudnative-pg/cluster, and the same thing by helmfile is:

helmfile apply \
  --file 'git::https://github.com/stuttgart-things/helm.git@database/postgres-cluster.yaml.gotmpl' \
  --state-values-set namespace=schmetterpause \
  --state-values-set version=0.8.1 \
  --state-values-set clusterName=schmetterpause-db \
  --state-values-set database=schmetterpause \
  --state-values-set owner=schmetterpause \
  --state-values-set appSecretName=schmetterpause-db

That is the route for a cluster where the database is managed with everything else's databases rather than with this application. It also adds backups, which database.k does not render: an object-store target is environment configuration, and this module holds none.

The PVC appears only with the pod when the StorageClass is WaitForFirstConsumer.

Once the cluster is up, the migration initContainer gets through on its next backoff by itself. No intervention is needed; anyone impatient deletes the pod.

5. Checking¶

kubectl -n schmetterpause get cluster
kubectl -n schmetterpause get po
kubectl -n schmetterpause get httproute schmetterpause -o yaml | sed -n '/^status:/,$p'

curl -sSI https://schmetterpause.cicd-test2.4sthings.tiab.ssc.sva.de/healthz   # 200
curl -sSI http://schmetterpause.cicd-test2.4sthings.tiab.ssc.sva.de/           # 301

When something does not work¶

What the failures on this path have in common is that none of them look like failures.

An ExternalSecret reports SecretSyncedError and everything else stays green. Variant A only. Usually a property name the Vault entry does not have. The missing name is in .status.conditions[].message. It is visible only on the resource itself — pods, store and gateway say nothing about it.

The HTTPRoute has no status at all and the host answers 404. Not Accepted=False, but an empty .status.parents. Then the parentRef points at a Gateway that does not exist — typically because namespace is missing, so the route's own namespace is meant. schema.k now rejects an empty gatewayNamespace; if the route is still statusless, the name or the sectionName is wrong.

The initContainer fails with no such host. The database is missing or named differently. The message is useful anyway: if it says user=schmetterpause database=schmetterpause, the DSN parsed cleanly, so the password contains no character that takes the URL apart and only the name is really missing.

The pod will not start: secret "schmetterpause-db" not found. The order is wrong — the Secret has to exist before the Deployment. In variant A that is a matter of seconds and resolves itself; in variant B it means task kcl:secrets was skipped.

Tearing down¶

helmfile destroy --file 'git::https://github.com/stuttgart-things/helm.git@database/postgres-cluster.yaml.gotmpl' \
  --state-values-set namespace=schmetterpause --state-values-set clusterName=schmetterpause-db
kubectl delete ns schmetterpause

The Postgres cluster's PVC hangs off the namespace and goes with it. With a StorageClass whose reclaimPolicy is Delete — openebs-hostpath, for instance — the data is then gone. On a test cluster that is intended; anywhere else it is not.