Skip to content

fix(sui,aptos): skip tarring cwd when ContractsDir is empty#2519

Merged
Tofel merged 3 commits intomainfrom
fix-sui-aptos-ctf-cwd-tar-race
Apr 14, 2026
Merged

fix(sui,aptos): skip tarring cwd when ContractsDir is empty#2519
Tofel merged 3 commits intomainfrom
fix-sui-aptos-ctf-cwd-tar-race

Conversation

@Fletch153
Copy link
Copy Markdown
Contributor

@Fletch153 Fletch153 commented Apr 14, 2026

Summary

newSui and newAptos in framework/components/blockchain/ unconditionally call filepath.Abs(in.ContractsDir) and attach the result as a testcontainers.ContainerFile. When the caller leaves ContractsDir empty (the common case for chains that do not need host contracts copied in), filepath.Abs("") resolves to the process working directory. testcontainers-go then tars that whole cwd via archive/tar to copy into the container.

In integration-tests the cwd is a live test working directory — log files, db dumps, artifact collectors — actively being appended to while the container starts. If a file grows between tar.FileInfoHeader (header size from os.Stat) and the subsequent io.Copy, stdlib's tar.Writer returns ErrWriteTooLong and testcontainers surfaces:

create container: created hook: can't copy <cwd> to container:
error compressing file: archive/tar: write too long

Evidence — real CI failure

chainlink run 24372896623, job 71183740665, test Test_CCIPTokenTransfer_Sui2EVM_BurnMintTokenPool:

ctf_provider.go:230: Attempt 1/10: Failed to start CTF Sui container:
  create container: created hook: can't copy
  /home/runner/_work/chainlink/chainlink/integration-tests/smoke/ccip to container:
  error compressing file: archive/tar: write too long
ctf_provider.go:230: Attempt 2/10: Failed to start CTF Sui container: ...
ctf_provider.go:230: Attempt 3/10: Failed to start CTF Sui container: ...

Cwd printed (/home/runner/.../integration-tests/smoke/ccip) matches the predicted race target exactly — that directory is actively written by deployment/logger.SingleFileLogger (relative logs/ dir, autoFlushWriter syncs on every write) while the container starts.

Cause chain (verified link-by-link)

  1. Caller passes empty ContractsDirchainlink-deployments-framework/chain/sui/provider/ctf_provider.go:199-206 builds blockchain.Input{Type: TypeSui, ...} with no ContractsDir field. CTFChainProviderConfig exposes no such knob.
  2. filepath.Abs("") = cwd — Go stdlib contract. sui.go:98 on main: absPath, err := filepath.Abs(in.ContractsDir).
  3. cwd attached as ContainerFilesui.go:153-158: Files: []ContainerFile{{HostFilePath: absPath, ContainerFilePath: "/"}}, unconditional.
  4. testcontainers-go tars the host dirlifecycle.go:146-172 defaultCopyFileToContainerHook walks HostFilePath; Go's archive/tar.Writer.Write returns ErrWriteTooLong when body bytes exceed header Size.
  5. Retry absorbs most occurrenceschainlink-deployments-framework/chain/sui/provider/ctf_provider.go:152,226 wraps NewBlockchainNetwork in retry.Do with retry.Attempts(10), retry.Delay(1s). Most runs self-heal by attempt 2-3; occasional hard failures exhaust the retry budget.

Fix

Gate the ContainerFile entry on in.ContractsDir != "":

  • Callers that explicitly pass a directory keep identical copy-to-container behaviour.
  • Callers that need no host files (the current default for e.g. CCIP Sui smoke tests) no longer tar a busy cwd, eliminating the stat→stream race.

Applied to both sui.go and aptos.go — they share the identical shape and bug. Solana (solana.go) intentionally untouched: every Solana caller sets ContractsDir explicitly (Solana validator requires .so artifacts mounted at boot), so the race path is never exercised there.

Safety analysis

  • Non-empty ContractsDir callers: identical behaviour, same filepath.Abs + ContainerFile entry.
  • Empty ContractsDir callers: req.Files = nil. testcontainers-go hook loops for _, f := range files — zero iterations on nil, no container-side state change. Verified in testcontainers-go@v0.41.0/lifecycle.go:151.
  • Org-wide audit of TypeSui/TypeAptos callers returned 3: chainlink-deployments-framework Sui provider, chainlink-deployments-framework Aptos provider, chainlink-aptos/integration-tests/ccip/ccip_deployment_test.go. None pass a ContractsDir. The empty path isn't a documented default — the struct comment `toml:"contracts_dir" comment:"Solana's contracts directory"` explicitly scopes the field to Solana. The cwd-tarring was a latent side-effect of reusing the Solana-shaped field for Sui/Aptos, not a feature.
  • Aptos/Sui containers don't consume tarred cwd contents regardless: newSui runs sui start --force-regenesis --with-faucet (stock validator + faucet), newAptos runs aptos node run-local-testnet. Neither reads files from / inside the container. Contract publishing happens host-side via sui client publish / aptos move publish over RPC against the running node.

Test Plan

  • go build ./components/blockchain/... clean
  • go vet ./components/blockchain/... clean
  • CI green on this PR (TestAptosSmoke, TestSolanaSmoke, TestSmoke, TestChaos, lint, vuln, ~30 checks total all passing)
  • Downstream verification: chainlink PR #22011 bumps CTF to this branch's HEAD. Local go build ./... passes in chainlink/deployment and chainlink/integration-tests. Full chainlink CI pending post-merge bump.

Notes

No behaviour change for anyone who was already setting ContractsDir. Draft until downstream chainlink verification CI confirms zero archive/tar: write too long retry lines in Sui CCIP smoke tests.

newSui and newAptos unconditionally called filepath.Abs(in.ContractsDir)
and added the result as a testcontainers ContainerFile. When the caller
leaves ContractsDir empty, filepath.Abs("") resolves to the process
working directory, causing testcontainers-go to tar the entire cwd via
archive/tar and copy it into the container.

In integration-tests the cwd is a live test working directory (logs,
db dumps, artifact collectors). Any file that grows between
tar.FileInfoHeader's stat-derived size and the subsequent io.Copy trips
the stdlib tar writer's ErrWriteTooLong guard, which testcontainers
surfaces as:

  create container: created hook: can't copy <cwd> to container:
  error compressing file: archive/tar: write too long

This manifests as a Sui CCIP smoke-test flake in downstream consumers
(chainlink CCIP in-memory tests), typically hidden by the provider's
retry loop but occasionally exhausting retries and always adding noise.

Gate the ContainerFile entry on a non-empty ContractsDir. Callers that
explicitly pass a directory continue to get the prior copy-to-container
behaviour; callers that need no host files no longer tar a busy cwd.
Fletch153 added a commit to smartcontractkit/chainlink that referenced this pull request Apr 14, 2026
Bumps github.com/smartcontractkit/chainlink-testing-framework/framework
across all repo modules to pull in the upstream fix that gates the
testcontainers ContainerFile entry on a non-empty ContractsDir.

Previously, Sui and Aptos blockchain providers in the framework called
filepath.Abs(in.ContractsDir) and unconditionally added a Files entry
to the container request. Because filepath.Abs("") resolves to cwd,
testcontainers-go tarred the entire test working directory. Any file
that grew between os.Stat (header size) and io.Copy (body) tripped
archive/tar's ErrWriteTooLong guard, surfacing in CCIP Sui smoke tests
as:

  ctf_provider.go:230: Attempt 1/10: Failed to start CTF Sui container:
    create container: created hook: can't copy ... to container:
    error compressing file: archive/tar: write too long

The retry loop (10 attempts, 1s) usually masked the flake but
intermittently exhausted, hard-failing tests like
Test_CCIPTokenTransfer_Sui2EVM_BurnMintTokenPool_WithAllowlist.

Upstream fix: smartcontractkit/chainlink-testing-framework#2519
Same logical change (gate Files on ContractsDir != ""), restructured to
reduce diff noise: compute files slice before req literal, assign via
Files: files — req struct keeps original field order, trailing commas,
and WaitingFor placement untouched.
Fletch153 added a commit to smartcontractkit/chainlink that referenced this pull request Apr 14, 2026
Picks up smartcontractkit/chainlink-testing-framework#2519 refactor
commit (14664092839c) — same behavioural fix, minimized diff.
@Fletch153 Fletch153 marked this pull request as ready for review April 14, 2026 17:29
@Fletch153 Fletch153 requested a review from a team as a code owner April 14, 2026 17:29
@Tofel Tofel enabled auto-merge (squash) April 14, 2026 18:31
Fletch153 added a commit to smartcontractkit/chainlink-aptos that referenced this pull request Apr 14, 2026
… race

Picks up smartcontractkit/chainlink-testing-framework#2519 — the Aptos
CTF provider in chainlink-deployments-framework builds blockchain.Input
with no ContractsDir, causing CTF's newAptos to filepath.Abs("") = cwd
and tar-stream the live test working directory via testcontainers-go.
Growing log files trip archive/tar: write too long intermittently.

This bump lands the upstream guard for the Aptos path (identical shape
to the Sui fix). No behaviour change for callers passing an explicit
ContractsDir.

Unsigned — will sign before ready-for-review.
Fletch153 added a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 14, 2026
Pulls in smartcontractkit/chainlink-testing-framework#2519. The Sui and
Aptos CTF providers in chain/{sui,aptos}/provider/ctf_provider.go build
blockchain.Input with no ContractsDir, which made upstream newSui/newAptos
call filepath.Abs("") = cwd and tar-stream the live test working
directory. Growing files trip archive/tar: write too long intermittently,
surfacing as retries (mostly masked by the retry.Attempts(10) loop) in
Test_CTFChainProvider_Initialize.

Module bump only — no source changes. Patch-level release.

Unsigned — will sign before ready-for-review.
@Tofel Tofel merged commit 2d5d34f into main Apr 14, 2026
85 of 88 checks passed
@Tofel Tofel deleted the fix-sui-aptos-ctf-cwd-tar-race branch April 14, 2026 18:41
Fletch153 added a commit to smartcontractkit/chainlink that referenced this pull request Apr 14, 2026
Pulls in smartcontractkit/chainlink-testing-framework#2519 — the Sui/Aptos
CTF provider cwd tar-race fix. Eliminates intermittent 'archive/tar:
write too long' flakes in Sui CCIP smoke tests.

Tidied go.sum across all 8 modules.
Fletch153 added a commit to smartcontractkit/chainlink-aptos that referenced this pull request Apr 14, 2026
Pulls in smartcontractkit/chainlink-testing-framework#2519 — the
Sui/Aptos CTF provider cwd tar-race fix. The Aptos provider in
chainlink-deployments-framework builds blockchain.Input with no
ContractsDir, causing upstream newAptos to filepath.Abs("") = cwd
and tar-stream the live test working directory. Growing log files
trip archive/tar: write too long intermittently.

Module bump only — no source changes.
Fletch153 added a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 14, 2026
Pulls in smartcontractkit/chainlink-testing-framework#2519. The Sui
and Aptos CTF providers in chain/{sui,aptos}/provider/ctf_provider.go
build blockchain.Input with no ContractsDir, which made upstream
newSui/newAptos call filepath.Abs("") = cwd and tar-stream the live
test working directory. Growing files trip archive/tar: write too long
intermittently, surfacing as retries in Test_CTFChainProvider_Initialize.

Module bump only — no source changes. Patch-level release.

Changeset: .changeset/fix-sui-aptos-cwd-tar-race.md
github-merge-queue Bot pushed a commit to smartcontractkit/chainlink that referenced this pull request Apr 15, 2026
…ace (#22011)

* fix(ctf): bump chainlink-testing-framework to fix Sui/Aptos cwd tar race

Bumps github.com/smartcontractkit/chainlink-testing-framework/framework
across all repo modules to pull in the upstream fix that gates the
testcontainers ContainerFile entry on a non-empty ContractsDir.

Previously, Sui and Aptos blockchain providers in the framework called
filepath.Abs(in.ContractsDir) and unconditionally added a Files entry
to the container request. Because filepath.Abs("") resolves to cwd,
testcontainers-go tarred the entire test working directory. Any file
that grew between os.Stat (header size) and io.Copy (body) tripped
archive/tar's ErrWriteTooLong guard, surfacing in CCIP Sui smoke tests
as:

  ctf_provider.go:230: Attempt 1/10: Failed to start CTF Sui container:
    create container: created hook: can't copy ... to container:
    error compressing file: archive/tar: write too long

The retry loop (10 attempts, 1s) usually masked the flake but
intermittently exhausted, hard-failing tests like
Test_CCIPTokenTransfer_Sui2EVM_BurnMintTokenPool_WithAllowlist.

Upstream fix: smartcontractkit/chainlink-testing-framework#2519

* chore: bump CTF framework to minimal-diff fix version

Picks up smartcontractkit/chainlink-testing-framework#2519 refactor
commit (14664092839c) — same behavioural fix, minimized diff.

* chore: bump CTF framework to v0.15.16

Pulls in smartcontractkit/chainlink-testing-framework#2519 — the Sui/Aptos
CTF provider cwd tar-race fix. Eliminates intermittent 'archive/tar:
write too long' flakes in Sui CCIP smoke tests.

Tidied go.sum across all 8 modules.
github-merge-queue Bot pushed a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 15, 2026
## Summary

Bumps
`github.com/smartcontractkit/chainlink-testing-framework/framework` to
`v0.15.16` to pick up the upstream Sui/Aptos CTF provider cwd-tar-race
fix.

Upstream PR: smartcontractkit/chainlink-testing-framework#2519 (merged
as `v0.15.16`).

## Root Cause

`chain/sui/provider/ctf_provider.go` and
`chain/aptos/provider/ctf_provider.go` both build `blockchain.Input`
with no `ContractsDir`. Upstream CTF's `newSui`/`newAptos` called
`filepath.Abs(in.ContractsDir)` unconditionally — Go stdlib resolves
`filepath.Abs("")` to the process cwd. That cwd was attached as a
`testcontainers.ContainerFile`, tarring the host test working directory.
Files being written concurrently (logs, db dumps) trip
`archive/tar.ErrWriteTooLong` between `os.Stat` (header size) and
`io.Copy` (body).

The `retry.Attempts(10)` wrapper in `startContainer` masks most
occurrences. `Test_CTFChainProvider_Initialize` (which actually starts a
container) hits this path in this repo's own CI.

## Fix

Module bump only — no source changes. Pins framework to `v0.15.16`
(stable tag), which gates the `Files:` entry on `ContractsDir != ""`.

Changeset included: `.changeset/fix-sui-aptos-cwd-tar-race.md` (patch).

## Safety

- Callers passing a non-empty `ContractsDir`: identical behaviour.
- Callers passing empty (both Sui and Aptos providers in this repo):
`req.Files` now nil. testcontainers-go's hook iterates `range files` —
zero iterations on nil, no-op (verified upstream in `lifecycle.go:151`).
- Delta in `framework/components/blockchain/` from previous pin
(`v0.15.13`) to `v0.15.16`: 1 unrelated commit (`#2517 update sui
version`) + the 2 fix commits. Minimal transitive risk.

## Test plan

- [x] `go build ./...` clean
- [x] `go mod tidy` clean
- [x] CI green — 22 checks pass, 2 skipped, 0 failing
Fletch153 added a commit to smartcontractkit/chainlink-aptos that referenced this pull request Apr 15, 2026
Pulls in smartcontractkit/chainlink-testing-framework#2519 — the
Sui/Aptos CTF provider cwd tar-race fix. The Aptos provider in
chainlink-deployments-framework builds blockchain.Input with no
ContractsDir, causing upstream newAptos to filepath.Abs("") = cwd
and tar-stream the live test working directory. Growing log files
trip archive/tar: write too long intermittently.

Module bump only — no source changes.
graham-chainlink pushed a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 16, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to ggoh/test-ci, this
PR will be updated.


# Releases
## chainlink-deployments-framework@0.95.1

### Patch Changes

-
[#929](#929)
[`9d78329`](9d78329)
Thanks [@Fletch153](https://github.com/Fletch153)! - Bump
chainlink-testing-framework/framework to pick up the Sui/Aptos CTF
provider cwd-tar-race fix (upstream
smartcontractkit/chainlink-testing-framework#2519). Resolves
intermittent `archive/tar: write too long` flakes in
`Test_CTFChainProvider_Initialize` for the Sui and Aptos providers.

## operations-gen@0.0.2

### Patch Changes

-
[`80d3475`](80d3475)
Thanks [@graham-chainlink](https://github.com/graham-chainlink)! - test

---------

Co-authored-by: app-token-issuer-engops[bot] <144731339+app-token-issuer-engops[bot]@users.noreply.github.com>
graham-chainlink pushed a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 16, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to ggoh/test-ci, this
PR will be updated.


# Releases
## chainlink-deployments-framework@0.95.1

### Patch Changes

-
[#929](#929)
[`9d78329`](9d78329)
Thanks [@Fletch153](https://github.com/Fletch153)! - Bump
chainlink-testing-framework/framework to pick up the Sui/Aptos CTF
provider cwd-tar-race fix (upstream
smartcontractkit/chainlink-testing-framework#2519). Resolves
intermittent `archive/tar: write too long` flakes in
`Test_CTFChainProvider_Initialize` for the Sui and Aptos providers.

## operations-gen@0.0.2

### Patch Changes

-
[`dbf0e57`](dbf0e57)
Thanks [@graham-chainlink](https://github.com/graham-chainlink)! - test

---------

Co-authored-by: app-token-issuer-engops[bot] <144731339+app-token-issuer-engops[bot]@users.noreply.github.com>
graham-chainlink pushed a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 16, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to ggoh/test-ci, this
PR will be updated.


# Releases
## chainlink-deployments-framework@0.95.1

### Patch Changes

-
[#929](#929)
[`9d78329`](9d78329)
Thanks [@Fletch153](https://github.com/Fletch153)! - Bump
chainlink-testing-framework/framework to pick up the Sui/Aptos CTF
provider cwd-tar-race fix (upstream
smartcontractkit/chainlink-testing-framework#2519). Resolves
intermittent `archive/tar: write too long` flakes in
`Test_CTFChainProvider_Initialize` for the Sui and Aptos providers.

## tools/operations-gen@0.0.2

### Patch Changes

-
[`dbf0e57`](dbf0e57)
Thanks [@graham-chainlink](https://github.com/graham-chainlink)! - test

---------

Co-authored-by: app-token-issuer-engops[bot] <144731339+app-token-issuer-engops[bot]@users.noreply.github.com>
github-merge-queue Bot pushed a commit to smartcontractkit/chainlink-deployments-framework that referenced this pull request Apr 16, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and the packages will
be published to npm automatically. If you're not ready to do a release
yet, that's fine, whenever you add more changesets to main, this PR will
be updated.


# Releases
## chainlink-deployments-framework@0.95.1

### Patch Changes

-
[#929](#929)
[`9d78329`](9d78329)
Thanks [@Fletch153](https://github.com/Fletch153)! - Bump
chainlink-testing-framework/framework to pick up the Sui/Aptos CTF
provider cwd-tar-race fix (upstream
smartcontractkit/chainlink-testing-framework#2519). Resolves
intermittent `archive/tar: write too long` flakes in
`Test_CTFChainProvider_Initialize` for the Sui and Aptos providers.

-
[#938](#938)
[`9565eff`](9565eff)
Thanks [@AnieeG](https://github.com/AnieeG)! - Fix/update anvil startup
to use offical anvil image, adding --no-storage-caching option to start
up

---------

Co-authored-by: app-token-issuer-engops[bot] <144731339+app-token-issuer-engops[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants