feat(telemetry): implement Envoy access log billing pipeline and E2E test environment

[JoseSzycho]

Why

To establish a robust, file-based HTTP metering pipeline that collects and structures billing telemetry from Envoy proxy instances. Measuring customer usage (request count, bandwidth, and connection duration) requires an architecture that is reliable, decoupled from security filters, and easy to validate. This PR introduces the production log-based billing collector along with a complete multi-cluster local development environment and automated E2E testing suite to ensure correctness.

How it Works

1. Metadata Injection (Extension Server)

• Project Name Mapping: The extension server reads the meta.datumapis.com/upstream-cluster-name namespace label to map downstream namespaces to their human-readable project names.
• Unconditional Metadata Stamping: The extension server stamps the project_name directly into the datum-gateway Envoy route metadata namespace during the XDS translation phase. This happens before any WAF/Coraza filter validation, ensuring that the project identifier is always written and captured in the JSON access log even if the security filter is disabled.

2. Log Collection & Processing (Vector)

• Log Ingestion & Access: Envoy writes access logs directly to stdout. This allows a node-local Vector instance to read and tail the proxy's log files from the host node.
• CloudEvent Generation: Using a Vector Remap Language (VRL) transform, Vector tails the log stream, parses the JSON-formatted access logs, maps namespaces, and generates four standardized CloudEvent metrics (requests, ingress-bytes, egress-bytes, and connection-seconds) which are batched and pushed to the billing gateway service.

3. Dev & E2E Testing Infrastructure

• Multi-Cluster Local Setup: The bootstrap task (task dev:bootstrap) provisions a local Kind multi-cluster environment (nso-standard and nso-infra) linked via internal kubeconfig secrets.
• Extension Server Overlay (extension-server-e2e): Adapts the server for local testing by configuring cert-manager issued server/client certificates for mutual TLS (mTLS), swapping CSI driver-backed storage mounts for standard Secrets, and adjusting replicas and PDB limits to support single-node Kind nodes.
• Envoy Gateway overlay: Overrides the default Helm configuration to connect the proxy to the local extension server and register XDS listener, route, and cluster translation hooks.
• Downstream Reference Alignment: Updates the development downstream gateway configurations to output access logs directly to /dev/stdout (matching the new file-based log ingestion flow).
• Automated Integration Tests: Adds a comprehensive Chainsaw test suite (test/e2e/billing/chainsaw-test.yaml) that simulates HTTP traffic and runs verification scripts to confirm that all four CloudEvent categories are correctly generated and received by a mock billing gateway.

[scotwells]

@0xmc added you as a reviewer to this so you have visibility into how we're getting access logs flowing with project IDs embedded.

[JoseSzycho]

@scotwells @0xmc seems that there is a bug here:

network-services-operator/internal/extensionserver/cache/index.go

Lines 73 to 79 in f42c6d7

	// Derive project name from the upstream cluster name label. The label
	// value is "cluster-<projectName>" (written by mappednamespace.go). In
	// single-cluster deployments where no cluster name is configured the
	// label is absent and ProjectNames[ns.Name] is left empty.
	if clusterLabel := ns.Labels[downstreamclient.UpstreamOwnerClusterNameLabel]; clusterLabel != "" {
	idx.ProjectNames[ns.Name] = strings.TrimPrefix(clusterLabel, "cluster-")
	}

This is deployed into staging, and instead of resolving the project name, it is resolving the namespace UID in which the project lives.

Fixing this.

[scotwells]

@JoseSzycho dod you confirm what that label is set to on the edge namespace?

[JoseSzycho]

@scotwells the labels seems to be correct. Seems that the issue is related into a bug in Vector VRL transform. I'm continuing with the investigation

[JoseSzycho]

@scotwells @0xmc I confirm that this PR:

• https://github.com/datum-cloud/infra/commit/18650aa2e0ab2d71069a00a731e76886aa7898aa
  Fixed the bug.

All seems to be good.

[scotwells]

Great work on this Jose!