Skip to content

Commit 28e42ce

Browse files
feat(uploads): support streaming uploads with on-demand part URLs (#132)
Co-authored-by: hotdata-automation[bot] <267177015+hotdata-automation[bot]@users.noreply.github.com>
1 parent c71e2c0 commit 28e42ce

21 files changed

Lines changed: 996 additions & 15 deletions

.openapi-generator/FILES

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,9 @@ docs/LoadManagedTableRequest.md
9292
docs/LoadManagedTableResponse.md
9393
docs/ManagedSchemaResponse.md
9494
docs/ManagedTableResponse.md
95+
docs/MintUploadPartsRequest.md
96+
docs/MintUploadPartsResponse.md
97+
docs/MintedUploadPartResponse.md
9598
docs/NumericProfileDetail.md
9699
docs/QueryApi.md
97100
docs/QueryRequest.md
@@ -242,6 +245,9 @@ hotdata/models/load_managed_table_request.py
242245
hotdata/models/load_managed_table_response.py
243246
hotdata/models/managed_schema_response.py
244247
hotdata/models/managed_table_response.py
248+
hotdata/models/mint_upload_parts_request.py
249+
hotdata/models/mint_upload_parts_response.py
250+
hotdata/models/minted_upload_part_response.py
245251
hotdata/models/numeric_profile_detail.py
246252
hotdata/models/query_request.py
247253
hotdata/models/query_response.py
@@ -371,6 +377,9 @@ test/test_load_managed_table_request.py
371377
test/test_load_managed_table_response.py
372378
test/test_managed_schema_response.py
373379
test/test_managed_table_response.py
380+
test/test_mint_upload_parts_request.py
381+
test/test_mint_upload_parts_response.py
382+
test/test_minted_upload_part_response.py
374383
test/test_numeric_profile_detail.py
375384
test/test_query_api.py
376385
test/test_query_request.py

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Changed
11+
12+
- feat(uploads): support streaming uploads with on-demand part URLs
13+
1014
## [0.5.0] - 2026-06-28
1115

1216
### Added

docs/CreateUploadRequest.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Name | Type | Description | Notes
1010
**checksum_value** | **str** | Integrity checksum value, paired with &#x60;checksum_algo&#x60;. Optional. | [optional]
1111
**content_encoding** | **str** | Content encoding to record for the uploaded file (for example &#x60;gzip&#x60;). Optional. | [optional]
1212
**content_type** | **str** | Content type to record for the uploaded file (for example the Parquet, CSV, or JSON MIME type). Optional. | [optional]
13-
**declared_size_bytes** | **int** | The exact size, in bytes, of the file you will upload. Validated at create time against the maximum allowed size, and again at finalize against the bytes actually stored — a mismatch fails the finalize. |
13+
**declared_size_bytes** | **int** | The exact size, in bytes, of the file you will upload. Optional. When provided, it is validated at create time against the maximum allowed size, and again at finalize against the bytes actually stored — a mismatch fails the finalize. Omit it to create a streaming (unknown-size) upload: the session is always multi-part and returns no part URLs up front; instead you mint part URLs on demand from &#x60;POST /v1/uploads/{upload_id}/parts&#x60; as you upload, and finalize validates only that the file is non-empty. | [optional]
1414
**filename** | **str** | Original file name, recorded with the upload for your own bookkeeping. Optional and advisory — it does not affect where the bytes are stored or how they are loaded. | [optional]
1515
**part_size** | **int** | Preferred size, in bytes, of each part for a large (multi-part) upload. Optional hint — the service clamps it to the allowed part-size range and to the maximum number of parts, and ignores it for small files uploaded with a single &#x60;PUT&#x60;. Omit to let the service choose. | [optional]
1616

docs/MintUploadPartsRequest.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# MintUploadPartsRequest
2+
3+
Request body for `POST /v1/uploads/{upload_id}/parts`: mint presigned upload URLs for specific parts of a streaming (unknown-size) multi-part upload. Provide the 1-based part numbers you want URLs for. Mint parts as you upload, and re-request a part number if its URL expires before you finish — the parts you have already uploaded are unaffected.
4+
5+
## Properties
6+
7+
Name | Type | Description | Notes
8+
------------ | ------------- | ------------- | -------------
9+
**part_numbers** | **List[int]** | The 1-based part numbers to mint URLs for. Must be non-empty; each number must be between 1 and the maximum number of parts allowed. |
10+
11+
## Example
12+
13+
```python
14+
from hotdata.models.mint_upload_parts_request import MintUploadPartsRequest
15+
16+
# TODO update the JSON string below
17+
json = "{}"
18+
# create an instance of MintUploadPartsRequest from a JSON string
19+
mint_upload_parts_request_instance = MintUploadPartsRequest.from_json(json)
20+
# print the JSON string representation of the object
21+
print(MintUploadPartsRequest.to_json())
22+
23+
# convert the object into a dict
24+
mint_upload_parts_request_dict = mint_upload_parts_request_instance.to_dict()
25+
# create an instance of MintUploadPartsRequest from a dict
26+
mint_upload_parts_request_from_dict = MintUploadPartsRequest.from_dict(mint_upload_parts_request_dict)
27+
```
28+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
29+
30+

docs/MintUploadPartsResponse.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# MintUploadPartsResponse
2+
3+
Response body for `POST /v1/uploads/{upload_id}/parts`: the minted part URLs, in ascending part-number order.
4+
5+
## Properties
6+
7+
Name | Type | Description | Notes
8+
------------ | ------------- | ------------- | -------------
9+
**parts** | [**List[MintedUploadPartResponse]**](MintedUploadPartResponse.md) | The minted part URLs, in ascending part-number order. &#x60;PUT&#x60; each part&#39;s bytes to its URL and keep the response&#39;s &#x60;ETag&#x60; to pass to finalize. |
10+
11+
## Example
12+
13+
```python
14+
from hotdata.models.mint_upload_parts_response import MintUploadPartsResponse
15+
16+
# TODO update the JSON string below
17+
json = "{}"
18+
# create an instance of MintUploadPartsResponse from a JSON string
19+
mint_upload_parts_response_instance = MintUploadPartsResponse.from_json(json)
20+
# print the JSON string representation of the object
21+
print(MintUploadPartsResponse.to_json())
22+
23+
# convert the object into a dict
24+
mint_upload_parts_response_dict = mint_upload_parts_response_instance.to_dict()
25+
# create an instance of MintUploadPartsResponse from a dict
26+
mint_upload_parts_response_from_dict = MintUploadPartsResponse.from_dict(mint_upload_parts_response_dict)
27+
```
28+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
29+
30+

docs/MintedUploadPartResponse.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# MintedUploadPartResponse
2+
3+
One minted part URL.
4+
5+
## Properties
6+
7+
Name | Type | Description | Notes
8+
------------ | ------------- | ------------- | -------------
9+
**part_number** | **int** | The 1-based part number this URL is for. |
10+
**url** | **str** | Short-lived URL to &#x60;PUT&#x60; this part&#39;s bytes to. Keep the response&#39;s &#x60;ETag&#x60; and pass the &#x60;{part_number, e_tag}&#x60; pair to finalize. |
11+
12+
## Example
13+
14+
```python
15+
from hotdata.models.minted_upload_part_response import MintedUploadPartResponse
16+
17+
# TODO update the JSON string below
18+
json = "{}"
19+
# create an instance of MintedUploadPartResponse from a JSON string
20+
minted_upload_part_response_instance = MintedUploadPartResponse.from_json(json)
21+
# print the JSON string representation of the object
22+
print(MintedUploadPartResponse.to_json())
23+
24+
# convert the object into a dict
25+
minted_upload_part_response_dict = minted_upload_part_response_instance.to_dict()
26+
# create an instance of MintedUploadPartResponse from a dict
27+
minted_upload_part_response_from_dict = MintedUploadPartResponse.from_dict(minted_upload_part_response_dict)
28+
```
29+
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
30+
31+

docs/UploadSessionResponse.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@ Name | Type | Description | Notes
99
**finalize_token** | **str** | One-time token that authorizes finalizing this upload. Returned exactly once at create time — store it; it cannot be retrieved again. |
1010
**headers** | **Dict[str, str]** | Headers you must send verbatim with each &#x60;PUT&#x60;. Currently always empty; present so a future mode can require signed headers without changing the response shape. |
1111
**mode** | **str** | Upload mode: &#x60;single&#x60; (upload the whole file with one &#x60;PUT&#x60; to &#x60;url&#x60;) or &#x60;multipart&#x60; (upload each part with one &#x60;PUT&#x60; to the matching entry in &#x60;part_urls&#x60;). Modeled as a string so additional modes can be added later without breaking clients. |
12-
**part_size** | **int** | For a &#x60;multipart&#x60; upload, the size in bytes to split the file into: send bytes &#x60;[(i-1) * part_size, i * part_size)&#x60; to &#x60;part_urls[i - 1]&#x60;, with the last part carrying the remainder. Slice by this value — do **not** divide the file evenly by &#x60;part_urls.len()&#x60;, which can make a non-final part smaller than the 5 MiB minimum that storage requires (the upload then fails at finalize). Absent for &#x60;single&#x60; uploads. | [optional]
13-
**part_urls** | **List[str]** | For a &#x60;multipart&#x60; upload, the per-part URLs in ascending part order: &#x60;PUT&#x60; your file&#39;s part *i* (1-based) to &#x60;part_urls[i - 1]&#x60; and keep each response&#39;s &#x60;ETag&#x60;, then pass the &#x60;{part_number, e_tag}&#x60; list to finalize. Absent for &#x60;single&#x60; uploads. | [optional]
12+
**part_size** | **int** | For a &#x60;multipart&#x60; upload (both known-size and streaming), the size in bytes to split the file into: send bytes &#x60;[(i-1) * part_size, i * part_size)&#x60; as part *i*, with the last part carrying the remainder. Slice by this value — do **not** divide the file evenly by the number of parts, which can make a non-final part smaller than the 5 MiB minimum that storage requires (the upload then fails at finalize). Absent for &#x60;single&#x60; uploads. | [optional]
13+
**part_urls** | **List[str]** | For a known-size &#x60;multipart&#x60; upload, the per-part URLs in ascending part order: &#x60;PUT&#x60; your file&#39;s part *i* (1-based) to &#x60;part_urls[i - 1]&#x60; and keep each response&#39;s &#x60;ETag&#x60;, then pass the &#x60;{part_number, e_tag}&#x60; list to finalize. Absent for &#x60;single&#x60; uploads, and also absent for a streaming (unknown-size) &#x60;multipart&#x60; upload — there, mint part URLs on demand via &#x60;POST /v1/uploads/{upload_id}/parts&#x60;. | [optional]
1414
**upload_id** | **str** | Identifier for this upload. Pass it to the finalize endpoint and to the managed-table load endpoint once finalized. |
1515
**url** | **str** | The URL to &#x60;PUT&#x60; the raw file bytes to, for a &#x60;single&#x60; upload. Short-lived — upload promptly and finalize. Absent for &#x60;multipart&#x60; uploads (use &#x60;part_urls&#x60;). | [optional]
1616

docs/UploadsApi.md

Lines changed: 96 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Method | HTTP request | Description
88
[**create_upload_sessions_batch_handler**](UploadsApi.md#create_upload_sessions_batch_handler) | **POST** /v1/uploads/batch | Create upload sessions in bulk
99
[**finalize_upload_handler**](UploadsApi.md#finalize_upload_handler) | **POST** /v1/uploads/{upload_id}/finalize | Finalize upload
1010
[**list_uploads**](UploadsApi.md#list_uploads) | **GET** /v1/files | List uploads
11+
[**mint_upload_parts_handler**](UploadsApi.md#mint_upload_parts_handler) | **POST** /v1/uploads/{upload_id}/parts | Mint upload part URLs
1112
[**upload_file**](UploadsApi.md#upload_file) | **POST** /v1/files | Upload file
1213

1314

@@ -16,7 +17,7 @@ Method | HTTP request | Description
1617
1718
Create upload session
1819

19-
Create an upload session for a file you will send directly to storage. Based on the declared size, the response is one of two shapes. For a small file (`mode: single`) it contains a short-lived `url` to `PUT` the whole file to. For a large file (`mode: multipart`) it contains `part_urls` and `part_size`: split the file into `part_size`-byte chunks (the last is the remainder) and `PUT` chunk *i* (1-based) to `part_urls[i - 1]`, keeping each response's `ETag`. Slice by `part_size`, not by an even division across `part_urls.len()` (which can make a non-final part too small). In both cases the response also includes a one-time `finalize_token`. After uploading, call the finalize endpoint with the token (and, for multipart, the `{part_number, e_tag}` list) to make the upload usable as managed-table contents. The returned upload ID can then be passed to the managed-table load endpoint.
20+
Create an upload session for a file you will send directly to storage. The response is one of three shapes. For a small file (`mode: single`) it contains a short-lived `url` to `PUT` the whole file to. For a large file with a known size (`mode: multipart`) it contains `part_urls` and `part_size`: split the file into `part_size`-byte chunks (the last is the remainder) and `PUT` chunk *i* (1-based) to `part_urls[i - 1]`, keeping each response's `ETag`. Slice by `part_size`, not by an even division across `part_urls.len()` (which can make a non-final part too small). For a file whose size you do not know up front, omit `declared_size_bytes`: the response is `mode: multipart` with a `part_size` but NO `part_urls`. As you stream, call `POST /v1/uploads/{upload_id}/parts` with a batch of `part_numbers` to mint per-part `PUT` URLs, `PUT` each part and keep its `ETag`, then finalize as for any multipart upload. In all cases the response also includes a one-time `finalize_token`. After uploading, call the finalize endpoint with the token (and, for multipart, the `{part_number, e_tag}` list) to make the upload usable as managed-table contents. The returned upload ID can then be passed to the managed-table load endpoint.
2021

2122
You may hint a preferred part size with `part_size`; the service clamps it to the allowed range and ignores it for single-`PUT` uploads.
2223

@@ -198,7 +199,7 @@ Name | Type | Description | Notes
198199
199200
Finalize upload
200201

201-
Confirm that a file has been uploaded to storage and make it usable as managed-table contents. Supply the `finalize_token` returned when the session was created, in the `X-Upload-Finalize-Token` header. The uploaded file's size is validated against the size declared at create time; a mismatch is rejected. Finalize is exactly-once: a second finalize of the same upload is rejected.
202+
Confirm that a file has been uploaded to storage and make it usable as managed-table contents. Supply the `finalize_token` returned when the session was created, in the `X-Upload-Finalize-Token` header. When you declared a size at create time, the uploaded file's size is validated against it and a mismatch is rejected. An upload created without a declared size is finalized from its uploaded parts; it must be non-empty and is rejected if it exceeds the server's maximum upload size. Finalize is exactly-once: a second finalize of the same upload is rejected.
202203

203204
### Example
204205

@@ -368,6 +369,99 @@ Name | Type | Description | Notes
368369

369370
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
370371

372+
# **mint_upload_parts_handler**
373+
> MintUploadPartsResponse mint_upload_parts_handler(upload_id, x_upload_finalize_token, mint_upload_parts_request)
374+
375+
Mint upload part URLs
376+
377+
Mint short-lived presigned URLs for specific parts of a multi-part upload. This is required for a streaming (unknown-size) upload — created by omitting the declared size — which mints no part URLs up front. It also works for a known-size multi-part upload: use it to re-mint a part whose URL expired before you uploaded that part. Supply the `finalize_token` returned when the session was created, in the `X-Upload-Finalize-Token` header, and the 1-based `part_numbers` you want URLs for. `PUT` each part's bytes to its URL, keep each response's `ETag`, then pass the `{part_number, e_tag}` list to finalize. You may mint parts in batches as you upload, and re-mint a part number whose URL expired before you finished uploading it.
378+
379+
### Example
380+
381+
* Api Key Authentication (WorkspaceId):
382+
* Bearer Authentication (BearerAuth):
383+
384+
```python
385+
import hotdata
386+
from hotdata.models.mint_upload_parts_request import MintUploadPartsRequest
387+
from hotdata.models.mint_upload_parts_response import MintUploadPartsResponse
388+
from hotdata.rest import ApiException
389+
from pprint import pprint
390+
391+
# Defining the host is optional and defaults to https://api.hotdata.dev
392+
# See configuration.py for a list of all supported configuration parameters.
393+
configuration = hotdata.Configuration(
394+
host = "https://api.hotdata.dev"
395+
)
396+
397+
# The client must configure the authentication and authorization parameters
398+
# in accordance with the API server security policy.
399+
# Examples for each auth method are provided below, use the example that
400+
# satisfies your auth use case.
401+
402+
# Configure API key authorization: WorkspaceId
403+
configuration.api_key['WorkspaceId'] = os.environ["API_KEY"]
404+
405+
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
406+
# configuration.api_key_prefix['WorkspaceId'] = 'Bearer'
407+
408+
# Configure Bearer authorization: BearerAuth
409+
configuration = hotdata.Configuration(
410+
access_token = os.environ["BEARER_TOKEN"]
411+
)
412+
413+
# Enter a context with an instance of the API client
414+
with hotdata.ApiClient(configuration) as api_client:
415+
# Create an instance of the API class
416+
api_instance = hotdata.UploadsApi(api_client)
417+
upload_id = 'upload_id_example' # str | Upload session ID returned at create time
418+
x_upload_finalize_token = 'x_upload_finalize_token_example' # str | One-time finalize token returned when the session was created
419+
mint_upload_parts_request = hotdata.MintUploadPartsRequest() # MintUploadPartsRequest |
420+
421+
try:
422+
# Mint upload part URLs
423+
api_response = api_instance.mint_upload_parts_handler(upload_id, x_upload_finalize_token, mint_upload_parts_request)
424+
print("The response of UploadsApi->mint_upload_parts_handler:\n")
425+
pprint(api_response)
426+
except Exception as e:
427+
print("Exception when calling UploadsApi->mint_upload_parts_handler: %s\n" % e)
428+
```
429+
430+
431+
432+
### Parameters
433+
434+
435+
Name | Type | Description | Notes
436+
------------- | ------------- | ------------- | -------------
437+
**upload_id** | **str**| Upload session ID returned at create time |
438+
**x_upload_finalize_token** | **str**| One-time finalize token returned when the session was created |
439+
**mint_upload_parts_request** | [**MintUploadPartsRequest**](MintUploadPartsRequest.md)| |
440+
441+
### Return type
442+
443+
[**MintUploadPartsResponse**](MintUploadPartsResponse.md)
444+
445+
### Authorization
446+
447+
[WorkspaceId](../README.md#WorkspaceId), [BearerAuth](../README.md#BearerAuth)
448+
449+
### HTTP request headers
450+
451+
- **Content-Type**: application/json
452+
- **Accept**: application/json
453+
454+
### HTTP response details
455+
456+
| Status code | Description | Response headers |
457+
|-------------|-------------|------------------|
458+
**200** | Minted part URLs | - |
459+
**400** | Invalid finalize token, invalid part numbers, batch too large, or the upload is not a multi-part upload | - |
460+
**404** | Upload session not found | - |
461+
**501** | Storage backend cannot issue upload URLs | - |
462+
463+
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
464+
371465
# **upload_file**
372466
> UploadResponse upload_file(body)
373467

hotdata/__init__.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,9 @@
133133
"LoadManagedTableResponse",
134134
"ManagedSchemaResponse",
135135
"ManagedTableResponse",
136+
"MintUploadPartsRequest",
137+
"MintUploadPartsResponse",
138+
"MintedUploadPartResponse",
136139
"NumericProfileDetail",
137140
"QueryRequest",
138141
"QueryResponse",
@@ -283,6 +286,9 @@
283286
from hotdata.models.load_managed_table_response import LoadManagedTableResponse as LoadManagedTableResponse
284287
from hotdata.models.managed_schema_response import ManagedSchemaResponse as ManagedSchemaResponse
285288
from hotdata.models.managed_table_response import ManagedTableResponse as ManagedTableResponse
289+
from hotdata.models.mint_upload_parts_request import MintUploadPartsRequest as MintUploadPartsRequest
290+
from hotdata.models.mint_upload_parts_response import MintUploadPartsResponse as MintUploadPartsResponse
291+
from hotdata.models.minted_upload_part_response import MintedUploadPartResponse as MintedUploadPartResponse
286292
from hotdata.models.numeric_profile_detail import NumericProfileDetail as NumericProfileDetail
287293
from hotdata.models.query_request import QueryRequest as QueryRequest
288294
from hotdata.models.query_response import QueryResponse as QueryResponse

0 commit comments

Comments
 (0)