Skip to content

feat: native append/replace/upsert load modes#38

Draft
shefeek-jinnah wants to merge 1 commit into
mainfrom
feat/native-mutations
Draft

feat: native append/replace/upsert load modes#38
shefeek-jinnah wants to merge 1 commit into
mainfrom
feat/native-mutations

Conversation

@shefeek-jinnah

Copy link
Copy Markdown

🚧 Draft — depends on the SDK's mode/key support (hotdata-framework, separate PR) and a regenerated hotdata client whose managed-table decls carry key. Dependency floors are bumped with TODO(release) markers; un-draft once those are released and the exact pins are set.

What

Push loads down as the server's native load modes instead of a client-side read-modify-write.

  • append / replace upload the batch directly — no full-table read + rewrite.
  • merge → native upsert by the resource's primary_key (updates matches, inserts the rest); the table's key is declared up front (initialize_storage + reconcile-add).
  • insert-only, and a merge with no resolvable primary key, still combine client-side and replace (the _dlt_id fallback can't match a logical row across runs).

Also fixes two combine-path bugs

  • resolve_primary_key now reads dlt's per-column primary_key hints — it read a top-level table key dlt never sets, so merge silently deduped by the row-unique _dlt_id and produced duplicates across runs instead of updating in place.
  • Arrow string_view/large reads are normalized to plain string/binary before combining (fixes ArrowTypeError: string_view vs string).

Testing

  • 86 passed, 2 skipped offline; the in-memory harness now models the native modes, so test_merge_dedup validates the native upsert path.
  • Server side proven on a live deploy (native append/replace/delete/update/upsert via the HTTP API).
  • Pending: native upsert/delete through the live connector — needs the regenerated client.

Before un-drafting

  • Set exact dep pins (hotdata, hotdata-framework).
  • Run ruff + mypy.

Push loads down as the server's native load modes instead of a
client-side read-modify-write:

- append / replace upload the batch directly (no full-table read+rewrite)
- merge -> native upsert by the table's declared key (updates matches,
  inserts the rest); the key is declared up front via the SDK's new
  create_managed_database(keys=) / add_managed_table(key=)
- insert-only, and a merge without a resolvable primary key, still combine
  client-side and replace

Also fixes two long-standing bugs in the combine path:
- resolve_primary_key now reads dlt's per-column primary_key hints (it read
  a top-level table key dlt never sets, so merge deduped by the row-unique
  _dlt_id -> duplicates across runs)
- normalise Arrow string_view/large reads to plain string/binary before
  combining (ArrowTypeError: string_view vs string)

Needs the SDK's mode/key support and the key-capable hotdata client;
dependency floors bumped (exact tags TBD at release).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant