Skip to content

Run edge e2e in CI against the real federated architecture#248

Draft
scotwells wants to merge 17 commits into
mainfrom
ci/federated-e2e-environment
Draft

Run edge e2e in CI against the real federated architecture#248
scotwells wants to merge 17 commits into
mainfrom
ci/federated-e2e-environment

Conversation

@scotwells

@scotwells scotwells commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What this does

Runs the edge end-to-end tests in CI against the real thing customers depend on. Until now CI only exercised a single-stack stand-in — vanilla gateway, no extension server, no federation — so an entire class of problems could never show up until production. This adds a CI job that stands up the production-fidelity edge and drives real traffic through it:

  • Two clusters — control plane and edge — wired the way production is.
  • Real Karmada federation carrying configuration from the control plane out to the edge, the same delivery path production uses.
  • The production gateway version, the Coraza WAF data plane customer traffic actually passes through, and the NSO extension server.
  • The full edge scenario set: the firewall enforces, an offline origin fails cleanly, the branded error page shows, and one bad certificate can't break its neighbors.

It also makes the environment reproducible from a clean checkout, which it wasn't before — bringing it up previously relied on files a developer already had locally. Those fixes live in Taskfile.test-infra.yml:

  • A tools task that installs the pinned binaries the environment needs (kind v0.32.0, karmadactl, kustomize, cmctl, chainsaw) instead of expecting them hand-placed.
  • An eg-chart task that pulls the Envoy Gateway helm chart (gitignored under config/**/charts, so absent on CI) before its bundled CRDs are applied.
  • An upstream-gateway-crds task that installs the Gateway-API CRDs on the control-plane cluster before cert-manager, which otherwise crashloops (it runs with enableGatewayAPI: true) and stalls bring-up.

Why it matters

A test is only as trustworthy as how closely it resembles the real thing, and CI is where we get that assurance on every change. Wiring the federated environment into CI means the edge's core guarantees are proven automatically, in a setup that mirrors production, before anything reaches customers.

Proven green

The federated job passed end to end on the runner: two-cluster bring-up, real Karmada federation, and the full scenario set — WAF blocking real SQL-injection and path-traversal requests with 403s, the config-dump parity gate confirming the proxy runs exactly the intended set, the branded error page, connector-offline handling, and single-cert isolation. Total ~16 minutes.

How to read this PR

The federated environment lives across the stack in #247 (#224#228), and none of it is on main yet — so this branch merges that stack in order to have a complete environment for CI to run against. The reviewable change is two files: the new workflow (.github/workflows/test-e2e-federated.yml) and the environment tasks in Taskfile.test-infra.yml. Once #224#228 land, this branch rebases onto main and the diff collapses to just those.

The three Taskfile.test-infra.yml reproducibility fixes (tools, eg-chart, upstream-gateway-crds) are really properties of the environment itself and would sit equally well in #226 — happy to move them there if you'd prefer the environment PR own them.

Merge order: the #247 stack first, then this.

Notes

  • The job is heavier than the existing single-stack e2e (two clusters + a full Karmada control plane), so it's path-filtered to edge-relevant changes and carries a 60-minute ceiling. It runs alongside the current test-e2e.yml; once this proves stable, the older single-stack job can be retired.
  • Karmada's cross-cluster networking is written for macOS dev hosts but works unchanged on the Linux runner (the apiserver is reached through a host-port mapping); the workflow only adds the inotify-limit and disk-space headroom a Linux runner needs.

Part of #247.

scotwells and others added 17 commits June 25, 2026 13:30
Describes how firewall and connector policy authored centrally is delivered
to each edge, and how each edge reads it on arrival. Named for what it does
rather than the tool that moves it, so the intent stays legible if the
mechanism changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds a read-only endpoint that reports the set of firewall and connector
protections the platform handed to the edge. Read-only and off the traffic
path; it observes configuration and never touches live requests. This is the
reference point for later confirming the edge runs exactly what it was given.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds a check that compares what the platform intended for the edge against
what the edge is actually serving, and names any gap in plain terms (missing,
wrong count, wrong place, rejected, or dropped at a limit). Turns a healthy
-looking response into real evidence the configuration took hold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds four end-to-end scenarios that send real traffic through the edge and
confirm the promises customers depend on: the firewall enforces, an offline
origin fails cleanly, the branded error page shows, and one bad certificate
can't break its neighbors. Each asserts on a real response, not just that the
control plane wrote the config. Includes a plain-language guide to how we
test the edge and what we can't yet guarantee.

These scenarios need the two-cluster production-fidelity environment, so they
live under test/e2e-edge (separate from the single-stack test/e2e suite the
default CI runs) and execute via `task test-infra:e2e` against that environment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Brings up a two-cluster environment that mirrors how the edge really runs:
the production gateway version, the firewall data plane real traffic passes
through, and the actual path configuration travels to the edge. Lets tests
exercise the route customer traffic takes instead of a simplified stand-in.
Test scaffolding only; no change to how the edge runs for customers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The endpoint that reports what the last build intended to change exists only
so a test can confirm the proxy is running exactly that set. Serve it (and do
the per-build recording that backs it) only when the --enable-programmed-set
flag is passed, so production exposes nothing and does no extra work on the
build path.

The flag joins the extension-server flag surface and is wired through the
deployment the same way as the others: an arg that reads an env var
(ENABLE_PROGRAMMED_SET), defaulting off, so an overlay can turn it on with a
strategic-merge patch on env instead of rewriting the args list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The endpoint that lets the parity test confirm the proxy is running exactly the
set the build intended is off by default and served only when asked for. The
base deployment reads --enable-programmed-set from the ENABLE_PROGRAMMED_SET env
var; flip it to "true" in the downstream edge overlay with a strategic-merge
patch on env so the test environment keeps that visibility while production
exposes nothing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
…-set-endpoint' into ci/federated-e2e-environment
Adds a CI job that stands up the production-fidelity edge — two clusters
wired with real Karmada federation, the production gateway version, the
Coraza WAF data plane, and the NSO extension server — and runs the edge
e2e scenarios against real traffic. Until now CI only exercised a
single-stack stand-in that couldn't reach the federation delivery path.

Also adds a `test-infra:tools` task that installs the pinned binaries the
environment needs (kind v0.32.0, karmadactl, kustomize, cmctl, chainsaw),
so CI — and a fresh checkout — can provision them instead of hand-placing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The gateway CRDs the downstream cluster needs live inside the Envoy
Gateway helm chart, which is gitignored (config/**/charts). On a dev
machine the chart is already vendored, so eg-crds finds it; on a fresh CI
checkout it is absent and bring-up fails at the first CRD apply. Add an
eg-chart task that pulls the chart to the expected path if missing, and
run it ahead of eg-crds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The upstream manager hangs in ContainerCreating waiting on a secret; the
prior diagnostics only dumped pod lists and container logs, which are
empty for a container that never started. Add describe output for
not-ready pods and cert-manager restart/issuance state so the mount
failure is visible from the run log.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
cert-manager runs with enableGatewayAPI:true, so it crashloops at startup
when the Gateway-API CRDs are absent. The downstream cluster gets them
from eg-crds first; the upstream only got them later via prepare-upstream
(config/e2e). The resulting crashloop delayed issuance of the manager's
webhook serving cert by ~2.5min, so the manager sat in ContainerCreating
waiting on the secret and bring-up timed out.

Add an upstream-gateway-crds task (same chart-bundled CRDs as eg-crds) and
run it before cert-manager-upstream. Also bump the manager readiness wait
to 300s and fix the diagnostics describe loop so pod events are captured.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The issue #243 regression (a TrafficProtectionPolicy in Enforce mode
must not take the gateway offline) needs the real WAF data plane —
extension server + Coraza on the datum-downstream-gateway class. It lived
in test/e2e, which the single-stack workflow auto-runs and cannot exclude,
so it failed there against an environment with no WAF (the Gateway never
reaches Programmed). Relocate it to test/e2e-edge, adapted to the
two-cluster topology and the shared fixtures/templates, and add it to the
federated scenario set. The single-stack suite keeps the fast operator-only
tests; edge/WAF/federation tests belong here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant