contract testing adr

Makefile

@@ -144,17 +144,17 @@ test_integration_restart_server: test_setup_restart_server
 test_integration_single: test_setup
 	@export KOSLI_TESTS=true $(FAKE_CI_ENV) && $(GOTESTSUM) -- -p=1 ./... -run "${TARGET}"
 
-test_smoke_aws: ensure_gotestsum ## Run AWS contract and smoke tests against real AWS (requires AWS creds)
-	@echo "Running AWS contract and smoke tests against real AWS..."
+test_contract_aws: ensure_gotestsum ## Run AWS contract tests against real AWS (requires AWS creds)
+	@echo "Running AWS contract tests against real AWS..."
 	@echo "Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set"
 	@$(GOTESTSUM) -- -v -p=1 -run "LambdaContract_RealAWS|AWSTestSuite/TestGetLambdaPackageData|AWSTestSuite/TestGetEcsTasksData|AWSTestSuite/TestGetS3Data" ./internal/aws/
 
-test_smoke_github: ensure_gotestsum ## Run GitHub contract tests against real GitHub API (requires KOSLI_GITHUB_TOKEN)
+test_contract_github: ensure_gotestsum ## Run GitHub contract tests against real GitHub API (requires KOSLI_GITHUB_TOKEN)
 	@echo "Running GitHub contract tests against real GitHub API..."
 	@echo "Requires KOSLI_GITHUB_TOKEN to be set"
 	@$(GOTESTSUM) -- -v -p=1 -run "GitHubContract_RealGitHub" ./internal/github/
 
-test_contract: test_smoke_aws test_smoke_github ## Run all contract tests against real external services
+test_contract: test_contract_aws test_contract_github ## Run all contract tests against real external services
 
 
 test_docs: deps vet ensure_network test_setup ## Test docs

TODO.md

@@ -100,7 +100,7 @@
 - [ ] Fake-backed unit tests for orchestration (concurrent cluster/service/task fetching, error propagation)
 - [ ] `NewECSClientFunc` factory + inject fake into `snapshotECS_test.go` command tests
 - [ ] Trim existing ECS integration tests to smoke tests
-- [ ] Add ECS to `make test_smoke_aws`
+- [ ] Add ECS to `make test_contract_aws`
 
 ### S3
 
@@ -110,7 +110,7 @@
 - [ ] Fake-backed unit tests for path include/exclude filtering and digest computation
 - [ ] `NewS3ClientFunc` factory + inject fake into `snapshotS3_test.go` command tests
 - [ ] Trim existing S3 integration tests to smoke tests
-- [ ] Add S3 to `make test_smoke_aws`
+- [ ] Add S3 to `make test_contract_aws`
 
 ### Azure Apps
 

docs/adr/20260421-fakes-and-contract-tests.md

@@ -0,0 +1,62 @@
+---
+title: "20260421 - Fakes and contract tests for external service integrations"
+description: "Use in-memory fakes and contract tests to decouple integration test suites from live external services"
+status: "Accepted"
+date: "2026-04-21"
+---
+
+# 20260421 - Fakes and contract tests for external service integrations
+
+## Overview
+
+Use in-memory fakes and a shared contract test suite to decouple the main integration test suite from live external services (AWS Lambda, GitHub API, etc.), while ensuring the fakes remain faithful to the real APIs.
+
+## Context
+
+Several CLI commands interact with external services — AWS Lambda, GitHub, GitLab, Bitbucket, Azure DevOps — to gather evidence for attestations and snapshots. The integration tests for these commands previously required live credentials and network access to real external services.
+
+This created three problems:
+
+1. **Test suite reliability**: Any test that calls a live external service can fail due to network issues, rate limits, credential expiry, or upstream changes — causes entirely unrelated to our code.
+2. **Test suite speed and scope**: Tests that depend on real external state can only verify a narrow set of scenarios. It is impractical to test error paths or edge cases against live services.
+3. **Who can run the tests**: Developers without credentials for a given external service could not run the tests for commands that depend on it. This limited who could contribute to those areas of the codebase.
+
+The pattern was first introduced for AWS Lambda (#763) and then extended to GitHub (#807).
+
+## Decision
+
+For each external service integration, we:
+
+1. **Define an interface** at the operation level that the command layer depends on, expressing what the service does in domain terms rather than SDK terms.
+
+2. **Write a shared contract test function** that asserts the key behavioural properties of the interface — what fields are present, what error behaviour to expect, and how edge cases are handled.
+
+3. **Run the contract tests against both implementations**: the shared contract test function is invoked twice — once against the fake (in the regular test suite, no credentials required) and once against the real service (env-gated, called from `make test_contract`). Running the same suite against both is what guarantees they behave identically.
+
+4. **Build an in-memory fake** that satisfies the same interface and passes the same contract tests. The fake is the only implementation used in the main integration test suite.
+
+5. **Inject the fake** into the command layer via a package-level factory variable. Tests swap the factory in `SetupTest` and restore it in `TearDownTest`.
+
+## The interface abstraction level
+
+We chose to fake at the **operation level** rather than the **SDK client level**. This means the interface speaks in domain terms (e.g. "get PR evidence for this commit") rather than SDK terms (individual HTTP, GraphQL, or SDK calls).
+
+This keeps fakes simple and free of SDK types, and the interface boundary is stable even if the underlying SDK or transport changes. In some cases (e.g. GraphQL clients that use Go reflection internally) SDK-level faking is impractical without reimplementing SDK machinery, which makes operation-level faking the only viable option.
+
+## Contract tests vs the main integration suite
+
+The shared contract test function is run against **both** implementations:
+
+- **Fake**: runs as part of the regular test suite (`make test_integration`), requires no external credentials. This is what keeps the fake honest on every PR.
+- **Real service**: runs via `make test_contract` in CI on a schedule, gated on service credentials. This is the authoritative run that verifies the contract against the actual API.
+
+Running the same test function against both implementations is the mechanism that guarantees they behave the same way. If the real API changes and the real-service run starts failing, it signals that the fake must be updated to match.
+
+The main integration test suite runs against the local Kosli server only and uses fakes for all external service calls. This keeps the main suite fast and deterministic.
+
+## Consequences
+
+- The main integration suite no longer requires live AWS or GitHub credentials to run. Any developer can run the full suite and contribute to commands that depend on external services.
+- Behavioural contracts between fakes and real APIs are made explicit and machine-checked.
+- Adding a new external service integration requires writing a contract test before (or alongside) the fake, which documents the real API behaviour.
+- The fake must be kept honest: if the real API changes in a way that breaks the contract tests, the fake must be updated to match.