feat: additive schema evolution via add_managed_table#24
Conversation
Replace the destructive snapshot -> delete-database -> recreate -> reload dance in ensure_managed_database with an additive add_managed_table call per newly-required table. New tables are declared in place on the existing managed database; existing tables and their data (including dlt bookkeeping) are left untouched. - hotdata_client.py: declare missing tables in place; drop _reload_table and the parquet/tempfile snapshot machinery. - job_client.py: refresh the now-stale drop_storage comment. - pyproject: require hotdata-framework>=0.6.0 (adds add_managed_table). - tests: assert additive add with no recreate (unit + in-memory e2e). - docs: architecture/runbook/README updated for in-place table declaration.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additive add_managed_table cleanly replaces the destructive recreate. Verified: no broken imports (write_table_parquet still used by job_client.py), unit + e2e tests meaningfully assert no recreate/data movement and a stable db id, and docs are consistent. LGTM.
super nit (not blocking): tests/test_e2e_inmemory.py module docstring (line 7) still says "the data-preserving schema-evolution recreate" — that path is gone now; consider "in-place schema evolution".
Resolve conflicts after the v0.5.0 release (#23) landed on main: - pyproject: keep version 0.5.0 (main) with hotdata-framework>=0.6.0 (this branch) - CHANGELOG: keep the [Unreleased] additive-evolution note above the [0.5.0] section - uv.lock: regenerate for hotdata-framework 0.6.0
There was a problem hiding this comment.
Additive schema evolution via add_managed_table is correct and a clear improvement over the destructive recreate — no data-loss window, no re-upload, and partial-failure recovery is idempotent (already-added tables are filtered by existing). Stale parquet/snapshot machinery removed cleanly, and the parquet helper is still correctly used by the load path. Tests, docs, CHANGELOG, and the 0.6.0 dependency bump are all consistent.
1 participant
Summary
Replaces the destination's destructive schema-evolution workaround with an additive table declaration now that the SDK/framework support it.
Previously, when a later run required a table the managed database lacked,
ensure_managed_databasewould: snapshot every existing table into memory → delete the whole database → recreate it with the union of tables → reload all snapshots. This had a data-loss window (non-atomic delete), re-uploaded all data on every new-table run, and round-tripped everything through parquet.Now it calls
HotdataClient.add_managed_tableper newly-required table: the table is declared in place on the existing database (added empty, populated by the subsequent load). Existing tables and their data — including dlt's_dlt_version/_dlt_loads/_dlt_pipeline_statebookkeeping — are left untouched.Changes
hotdata_client.py: additive add; removed_reload_tableand the snapshot/parquet/tempfile machinery (99 → 70 lines).job_client.py: refreshed the now-staledrop_storagecomment.pyproject.toml:hotdata-framework>=0.6.0(addsadd_managed_table).test_ensure_adds_missing_table_without_recreate(unit) asserts no delete/recreate/reload; in-memory e2etest_schema_evolution_preserves_datanow asserts the database id is unchanged across runs. 51 passed.Depends on hotdata-framework 0.6.0 (adds
add_managed_table) — see hotdata-dev/sdk-python-framework#32. CI will fail until 0.6.0 is on PyPI anduv.lockis regenerated. Draft until then.Verification (local, against the framework branch)
ruff check— cleanpytest— 51 passedmypy src/hotdata_dlt_destination/hotdata_client.py— cleanNot changed
Merge/append stay read-modify-write — the SDK load mode is still replace-only. The large-table read-modify-write cost is tracked separately.