feat(buildsecrets): sync tenant build-secrets from AWS Secrets Manager (no ESO)

[nicacioliveira]

Summary

Operator reconciles a K8s Secret named build-secrets in each opted-in site namespace, sourcing data from AWS Secrets Manager via the cluster-local account. Builder Jobs already consume the Secret opportunistically via envFrom: { optional: true } (admin #3201), so the chain is end-to-end with no further admin changes.

Previous approach (delegated to ESO via ExternalSecret) was scrapped — see "Why direct, not ESO" below.

How it fits in the bigger picture

flowchart LR
    subgraph p3 ["Phase 3 — NOT DONE"]
        UI["Admin UI / API"]
    end

    subgraph aws ["AWS Secrets Manager"]
        SM["sites/&lt;site&gt;/build<br/>JSON: { TENANT_TOKEN: ..., FOO: ... }"]
    end

    subgraph p2 ["Phase 2 — this PR"]
        OP["Operator<br/>BuildSecretsReconciler"]
    end

    subgraph cluster ["Site namespace (sites-&lt;site&gt;)"]
        NS["Namespace<br/>annotation: deco.sites/build-secrets-managed=enabled"]
        SEC["K8s Secret<br/>build-secrets<br/>(labels: managed-by=operator)"]
        JOB["Build Job"]
    end

    subgraph p1 ["Phase 1 — already merged (admin #3201)"]
        ENVFROM["envFrom: { secretRef, optional: true }"]
    end

    UI -.->|"writes (not yet)"| SM
    NS -->|"watch"| OP
    OP -->|"GetSecretValue (lazy, on reconcile)"| SM
    OP -->|"create / update / delete"| SEC
    ENVFROM -.->|"declared in admin Job spec"| JOB
    SEC -->|"env vars at pod start"| JOB

Loading

Phase status

Phase	Repo	What	Status
1	deco-sites/admin	Builder Job envFrom opportunistic mount of build-secrets	✅ merged (admin #3201)
2	decocms/operator (this PR)	Reconciler syncs SM ↔ K8s Secret based on namespace annotation	🟡 this PR
3	deco-sites/admin (future)	UI/API for tenant to write into AWS SM + set annotation	❌ not started

Why direct, not ESO

ESO is great for shared secrets that always exist (cloudflare-token, github-token). Per-tenant secrets are the opposite shape — "no upstream data yet" is normal, not an error to surface in kubectl get externalsecret. Direct-source gives us:

• Lazy creation: operator only materialises the K8s Secret when the upstream key actually exists. No SecretSyncedError noise for un-provisioned tenants.
• Single component in the critical path: easier to debug, fewer failure modes.
• Cleaner abstraction: Source interface lives in this repo. Swapping backend (GCP SM, Vault, even back to ESO via an ESOSource) is a new file, not a CRD migration.
• Phase 3 simpler: admin writes to SM, sets the annotation. Operator handles the rest. No second sync layer to orchestrate.

The interface:

type Source interface {
    Get(ctx, key) (data map[string]string, exists bool, err error)
}

AWSSource implements it via aws-sdk-go-v2/service/secretsmanager. Selected via env var:

BUILD_SECRETS_BACKEND=aws        # default
BUILD_SECRETS_BACKEND=disabled   # bypass the reconciler entirely

Reconciliation rules

Annotation	Upstream SM	Local K8s Secret	Reconcile action
off	—	—	no-op
off	—	operator-managed	delete
off	—	unowned	leave alone
on	missing	—	no-op (status: upstream-missing)
on	missing	operator-managed	delete
on	exists	—	create with operator labels
on	exists	operator-managed, data drifted	update
on	exists	operator-managed, data matches	skip write (idempotent)
on	exists	unowned	log warning, no-op

Adoption safety: A K8s Secret named build-secrets without labels deco.sites/managed-by=operator + deco.sites/feature=build-secrets is treated as user-managed. The operator refuses to touch it (returns ErrNotOwned). This protects the Secrets currently created by hand (frigidaire-pr, frigidaire-es, etc.) — they keep working unchanged after this lands.

Force-sync (no HTTP endpoint, annotation only — see controller doc comment)

Re-fetch ONE site from AWS immediately, without waiting for ResyncPeriod (15min default):

kubectl annotate ns sites-<site> \
  deco.sites/build-secrets-sync=$(date +%s) --overwrite

Re-fetch ALL managed sites at once:

kubectl annotate ns -l deco.sites/build-secrets-managed=enabled \
  deco.sites/build-secrets-sync=$(date +%s) --overwrite

Files

• internal/buildsecrets/source.go — Source interface + shared constants
• internal/buildsecrets/aws_source.go — AWSSource (AWS Secrets Manager)
• internal/buildsecrets/secret.go — Sync / Remove of the K8s Secret
• internal/buildsecrets/*_test.go — fake Source + mock SM client, 81% coverage
• internal/controller/buildsecrets_controller.go — Reconciler, watches Namespace + Secret
• cmd/main.go — wires Source via env var, registers reconciler conditionally
• config/rbac/role.yaml + chart/templates/clusterrole-operator-manager-role.yaml — drops ESO verbs, adds Secret CRUD (regenerated)
• go.mod / go.sum — adds aws-sdk-go-v2/service/secretsmanager, bumps SDK to v1.41.7

Companion PR (deploy first)

Operator needs secretsmanager:GetSecretValue on the cluster-local account. To be added in terraform-eks-cluster/eks-setup on the deco-operator Pod Identity, scoped to arn:aws:secretsmanager:us-west-2:<account>:secret:sites/*/build*. Will open separately.

Test plan

After IAM PR merges + this operator chart bump lands in infra_applications/provisioning/deco-operator/main/values.yaml:

• Create AWS SM key (account 610552715391, us-west-2 — eks-envs first): sites/build-igorteste/build = {"TENANT_TEST_KEY":"from-sm","TENANT_FROM_AWS":"yes"}
• kubectl --context eks-envs annotate ns sites-build-igorteste deco.sites/build-secrets-managed=enabled
• Confirm Secret/build-secrets materialises with labels deco.sites/managed-by=operator, deco.sites/feature=build-secrets, data matches SM
• Trigger a build of build-igorteste — confirm env vars present in builder container
• Update SM with new JSON, force-sync via annotation bump, confirm Secret updated within seconds
• Remove namespace annotation — confirm Secret deleted; next build runs without the env vars
• Re-annotate — Secret recreated from SM
• On a namespace with a manually-created build-secrets (e.g. sites-frigidaire-pr), annotate it. Confirm operator logs Skipping unowned Secret and the manual Secret stays intact (existing builds keep working)
• Repeat on eks-serverless (account 520819256705)

Out of scope

• Admin UI for tenants to manage these secrets (Phase 3)
• Migration of existing manually-created Secrets into operator-managed (intentional — they keep working as-is until someone deletes them and lets the operator recreate from SM)

[cubic-dev-ai]

1 issue found across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 13 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>