Run edge e2e in CI against the real federated architecture#248
Draft
scotwells wants to merge 17 commits into
Draft
Run edge e2e in CI against the real federated architecture#248scotwells wants to merge 17 commits into
scotwells wants to merge 17 commits into
Conversation
Describes how firewall and connector policy authored centrally is delivered to each edge, and how each edge reads it on arrival. Named for what it does rather than the tool that moves it, so the intent stays legible if the mechanism changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds a read-only endpoint that reports the set of firewall and connector protections the platform handed to the edge. Read-only and off the traffic path; it observes configuration and never touches live requests. This is the reference point for later confirming the edge runs exactly what it was given. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds a check that compares what the platform intended for the edge against what the edge is actually serving, and names any gap in plain terms (missing, wrong count, wrong place, rejected, or dropped at a limit). Turns a healthy -looking response into real evidence the configuration took hold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Adds four end-to-end scenarios that send real traffic through the edge and confirm the promises customers depend on: the firewall enforces, an offline origin fails cleanly, the branded error page shows, and one bad certificate can't break its neighbors. Each asserts on a real response, not just that the control plane wrote the config. Includes a plain-language guide to how we test the edge and what we can't yet guarantee. These scenarios need the two-cluster production-fidelity environment, so they live under test/e2e-edge (separate from the single-stack test/e2e suite the default CI runs) and execute via `task test-infra:e2e` against that environment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
Brings up a two-cluster environment that mirrors how the edge really runs: the production gateway version, the firewall data plane real traffic passes through, and the actual path configuration travels to the edge. Lets tests exercise the route customer traffic takes instead of a simplified stand-in. Test scaffolding only; no change to how the edge runs for customers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The endpoint that reports what the last build intended to change exists only so a test can confirm the proxy is running exactly that set. Serve it (and do the per-build recording that backs it) only when the --enable-programmed-set flag is passed, so production exposes nothing and does no extra work on the build path. The flag joins the extension-server flag surface and is wired through the deployment the same way as the others: an arg that reads an env var (ENABLE_PROGRAMMED_SET), defaulting off, so an overlay can turn it on with a strategic-merge patch on env instead of rewriting the args list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The endpoint that lets the parity test confirm the proxy is running exactly the set the build intended is off by default and served only when asked for. The base deployment reads --enable-programmed-set from the ENABLE_PROGRAMMED_SET env var; flip it to "true" in the downstream edge overlay with a strategic-merge patch on env so the test environment keeps that visibility while production exposes nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
…-set-endpoint' into ci/federated-e2e-environment
…nto ci/federated-e2e-environment
…ent' into ci/federated-e2e-environment
… into ci/federated-e2e-environment
…into ci/federated-e2e-environment
Adds a CI job that stands up the production-fidelity edge — two clusters wired with real Karmada federation, the production gateway version, the Coraza WAF data plane, and the NSO extension server — and runs the edge e2e scenarios against real traffic. Until now CI only exercised a single-stack stand-in that couldn't reach the federation delivery path. Also adds a `test-infra:tools` task that installs the pinned binaries the environment needs (kind v0.32.0, karmadactl, kustomize, cmctl, chainsaw), so CI — and a fresh checkout — can provision them instead of hand-placing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The gateway CRDs the downstream cluster needs live inside the Envoy Gateway helm chart, which is gitignored (config/**/charts). On a dev machine the chart is already vendored, so eg-crds finds it; on a fresh CI checkout it is absent and bring-up fails at the first CRD apply. Add an eg-chart task that pulls the chart to the expected path if missing, and run it ahead of eg-crds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The upstream manager hangs in ContainerCreating waiting on a secret; the prior diagnostics only dumped pod lists and container logs, which are empty for a container that never started. Add describe output for not-ready pods and cert-manager restart/issuance state so the mount failure is visible from the run log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
cert-manager runs with enableGatewayAPI:true, so it crashloops at startup when the Gateway-API CRDs are absent. The downstream cluster gets them from eg-crds first; the upstream only got them later via prepare-upstream (config/e2e). The resulting crashloop delayed issuance of the manager's webhook serving cert by ~2.5min, so the manager sat in ContainerCreating waiting on the secret and bring-up timed out. Add an upstream-gateway-crds task (same chart-bundled CRDs as eg-crds) and run it before cert-manager-upstream. Also bump the manager readiness wait to 300s and fix the diagnostics describe loop so pod events are captured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
The issue #243 regression (a TrafficProtectionPolicy in Enforce mode must not take the gateway offline) needs the real WAF data plane — extension server + Coraza on the datum-downstream-gateway class. It lived in test/e2e, which the single-stack workflow auto-runs and cannot exclude, so it failed there against an environment with no WAF (the Gateway never reaches Programmed). Relocate it to test/e2e-edge, adapted to the two-cluster topology and the shared fixtures/templates, and add it to the federated scenario set. The single-stack suite keeps the fast operator-only tests; edge/WAF/federation tests belong here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Runs the edge end-to-end tests in CI against the real thing customers depend on. Until now CI only exercised a single-stack stand-in — vanilla gateway, no extension server, no federation — so an entire class of problems could never show up until production. This adds a CI job that stands up the production-fidelity edge and drives real traffic through it:
It also makes the environment reproducible from a clean checkout, which it wasn't before — bringing it up previously relied on files a developer already had locally. Those fixes live in
Taskfile.test-infra.yml:toolstask that installs the pinned binaries the environment needs (kind v0.32.0, karmadactl, kustomize, cmctl, chainsaw) instead of expecting them hand-placed.eg-charttask that pulls the Envoy Gateway helm chart (gitignored underconfig/**/charts, so absent on CI) before its bundled CRDs are applied.upstream-gateway-crdstask that installs the Gateway-API CRDs on the control-plane cluster before cert-manager, which otherwise crashloops (it runs withenableGatewayAPI: true) and stalls bring-up.Why it matters
A test is only as trustworthy as how closely it resembles the real thing, and CI is where we get that assurance on every change. Wiring the federated environment into CI means the edge's core guarantees are proven automatically, in a setup that mirrors production, before anything reaches customers.
Proven green
The federated job passed end to end on the runner: two-cluster bring-up, real Karmada federation, and the full scenario set — WAF blocking real SQL-injection and path-traversal requests with 403s, the config-dump parity gate confirming the proxy runs exactly the intended set, the branded error page, connector-offline handling, and single-cert isolation. Total ~16 minutes.
How to read this PR
The federated environment lives across the stack in #247 (#224–#228), and none of it is on
mainyet — so this branch merges that stack in order to have a complete environment for CI to run against. The reviewable change is two files: the new workflow (.github/workflows/test-e2e-federated.yml) and the environment tasks inTaskfile.test-infra.yml. Once #224–#228 land, this branch rebases ontomainand the diff collapses to just those.The three
Taskfile.test-infra.ymlreproducibility fixes (tools,eg-chart,upstream-gateway-crds) are really properties of the environment itself and would sit equally well in #226 — happy to move them there if you'd prefer the environment PR own them.Merge order: the #247 stack first, then this.
Notes
test-e2e.yml; once this proves stable, the older single-stack job can be retired.Part of #247.