Skip to content

feat(samples): refactor and simplify core samples#91

Open
rederik76 wants to merge 3 commits into
mainfrom
refactor/samples-restructure-bundles
Open

feat(samples): refactor and simplify core samples#91
rederik76 wants to merge 3 commits into
mainfrom
refactor/samples-restructure-bundles

Conversation

@rederik76
Copy link
Copy Markdown
Collaborator

Consolidate the existing bronze_sample, silver_sample, gold_sample, and test_data_and_orchestrator bundles into two purpose-built Databricks Asset Bundles:

  • feature-samples: single-schema bundle demonstrating all framework features (CDC, snapshots, DQ, table migration, materialized views, Kafka, etc.)
  • pattern-samples: multi-schema medallion bundle (bronze → silver → gold) demonstrating end-to-end integration patterns

Key changes:

  • Remove test_data_and_orchestrator; fold orchestration and staging notebooks into each bundle directly
  • Add per-layer schema variables (bronze_schema, silver_schema, gold_schema) to pattern-samples bundle and deploy/destroy scripts
  • Add explicit database token to all targetDetails in both bundles and qualify flow-type targetTable references accordingly to align with framework's fully-qualified table name construction
  • Remove DPM sample; rename pipeline resource YAMLs and job tasks for clarity
  • Update dataFlowGroup values to be unique per directory to prevent cross-pipeline spec loading within a shared bundle.sourcePath
  • Update all pipeline root_path values to ${workspace.file_path}/src
  • Move src_notebooks to src/notebooks across both bundles
  • Rewrite samples/README.md; add deploy_feature_samples.sh and deploy_pattern_samples.sh; update Sphinx documentation

…amples bundles

Consolidate the existing bronze_sample, silver_sample, gold_sample, and
test_data_and_orchestrator bundles into two purpose-built Databricks Asset
Bundles:

- feature-samples: single-schema bundle demonstrating all framework features
  (CDC, snapshots, DQ, table migration, materialized views, Kafka, etc.)
- pattern-samples: multi-schema medallion bundle (bronze → silver → gold)
  demonstrating end-to-end integration patterns

Key changes:
- Remove test_data_and_orchestrator; fold orchestration and staging notebooks
  into each bundle directly
- Add per-layer schema variables (bronze_schema, silver_schema, gold_schema)
  to pattern-samples bundle and deploy/destroy scripts
- Add explicit database token to all targetDetails in both bundles and
  qualify flow-type targetTable references accordingly to align with
  framework's fully-qualified table name construction
- Remove DPM sample; rename pipeline resource YAMLs and job tasks for clarity
- Update dataFlowGroup values to be unique per directory to prevent
  cross-pipeline spec loading within a shared bundle.sourcePath
- Update all pipeline root_path values to ${workspace.file_path}/src
- Move src_notebooks to src/notebooks across both bundles
- Rewrite samples/README.md; add deploy_feature_samples.sh and
  deploy_pattern_samples.sh; update Sphinx documentation
@rederik76 rederik76 self-assigned this May 23, 2026
@rederik76 rederik76 linked an issue May 23, 2026 that may be closed by this pull request
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Refactor and simplify samples

1 participant