mode: development — multiple developers' terraform states collide on non-prefixable shared resources

[alexlatchford]

Problem

When multiple developers deploy the same bundle to a shared workspace using mode: development, each gets an independent terraform state under their personal workspace path (/Workspace/Users/<email>/.bundle/<bundle>/lab/state/). DAB handles this for most resources by prefixing names ([dev username] for jobs/dashboards, dev_username_ for schemas), so each developer gets isolated copies.

However, several resource types do not get dev-prefixed, causing N terraform states to compete over the same singleton resources in a shared workspace:

Resource type	Dev prefixed?	Result
Jobs, pipelines, dashboards	Yes ([dev username])	Works — each developer gets their own copy
UC schemas (auto-created)	Yes (dev_username_)	Works
Secret scopes	No — workspace-level, flat namespace	Collision — first deploy creates, all others fail with "already exists"
UC volumes (explicit catalog/schema)	No — names are literal when catalog_name/schema_name set via variables	Collision — same
Database instances	No — workspace-level	Collision — same
UC grants	N/A	Can also collide if targeting shared resources

The first developer to deploy creates these resources successfully. Every subsequent developer gets:

Error: cannot create secret scope: Scope my-scope already exists!
Error: cannot create volume: Volume 'catalog.schema.my_volume' already exists
Error: failed to create database_instance ... already exists.

Terraform applies other resources successfully, so the deploy is functionally fine for jobs/dashboards/etc., but exits non-zero which breaks CI-like workflows and creates confusing noise.

Not resolved by the direct engine

We tested with DATABRICKS_BUNDLE_ENGINE=direct on CLI v0.295.0. The direct engine still delegates secret scopes, volumes, and database instances to terraform — the errors are identical, with Terraform error during deployment and bundle.tf.json references in the output. So this is not currently addressed by the engine migration path.

Why this matters

• bundle destroy is dangerous — Any developer who runs destroy deletes shared resources out from under everyone else. There's no ownership distinction between "I created this" and "this existed before me."
• bundle deployment bind doesn't help — Binding adopts the resource into the developer's state, but then destroy will try to delete it. No way to say "use this but don't own it."
• No idempotent create — Terraform's create fails if the resource exists. There's no create_if_not_exists or adopt_if_exists semantic.
• Workaround is manual — Teams end up with comments like # Remove in local development for bundle deployment and manually commenting out resources before deploying locally, which is error-prone.

Reproduction

1. Create a bundle with a secret_scope, volume, or database_instance resource
2. Deploy with mode: development as Developer A — succeeds
3. Deploy with mode: development as Developer B to the same workspace — fails with "already exists"
4. Repeat step 3 with DATABRICKS_BUNDLE_ENGINE=direct — same result

Potential routes forward

We're not sure what the right solution looks like here, but a few directions that could help:

1. create_if_not_exists / adopt semantics — A resource-level lifecycle option that treats "already exists" as success rather than error, adopting the existing resource into state without taking destructive ownership. This feels like the most general solution and would benefit both the terraform and direct engines.

2. Broader dev prefixing — Extending the [dev username] / dev_username_ prefixing to cover secret scopes, volumes, and database instances (or at least volumes where catalog/schema are set via variables) would preserve full developer isolation. We understand this may be intentionally scoped today, but the current boundary creates a gap.

3. Shared resource annotation — Something like lifecycle: { shared: true } that tells the bundle "create if missing, but never destroy and don't error if it already exists." This would distinguish between resources a developer owns vs. resources they depend on.

4. Per-target include/exclude — Already requested in [Feature Request] DABs: allow target specific includes #2878. Would let teams exclude shared resources from dev targets. Not a full solution (doesn't handle first-developer-creates) but would reduce noise for subsequent developers.

5. Direct engine native support — As the direct engine takes over from terraform, ensuring these resource types are handled natively with idempotent create/update semantics would resolve the issue going forward.

Open to other ideas — happy to discuss trade-offs or provide more context from our setup.

Related issues

• [Feature Request] DABs: allow target specific includes #2878 — Target-specific includes (open, Enhancement)
• DAB deploy different resources to different targets #2364 — Deploy different resources to different targets (closed, not planned)
• Create UC schema from DAB without development mode name prefix #1779 — UC schema without dev prefix (closed)
• Add secret scopes support in assets bundling #2744 — Secret scope support added (merged, no idempotency discussed)
• Support secret scopes with direct engine #4098 — Secret scope support in direct engine (closed, completed — but still delegates to terraform)

Environment

• Databricks CLI: v0.295.0 (also reproduced on v0.260.0)
• DATABRICKS_BUNDLE_ENGINE=direct — same behavior
• Bundle with mode: development target
• Shared workspace with multiple developers

---

[andrewnester]

Hi @alexlatchford ! Thanks for the report.
Indeed, resources where the name is the addressable identifier (e.g. apps, secret scopes, volumes), are not auto-prefixed. Various reasons include:

1. a hard limit on name length, which is easy to exceed when auto-prefixing
2. strict regexes for character sets or patterns.
3. Unintentional resource recreates when the name / prefix changes.

We suggest to manually prefix them with the username information in dev target if that's the behaviour you'd like to have