SingleUploadThreshold config paramter is introduced.#66
Merged
johha merged 3 commits intocloudfoundry:mainfrom Mar 12, 2026
Merged
Conversation
- This parameter controls the upload strategy. File's with size smaller or equel then this value will be uploaded with single put, bigger file will be uploaded with multipart. However this parameter can't be bigger than 5GB since AWS S3 has hard limit on single put. But there is a exception with GCS, it don't have such hard limit and it don't support multipart upload. So we set this parameter to a big number to not block file larger than 5GB to upload into GCS. - CreateDefaultClient replaced with CreateKeepAliveDefaultClient for boshttp client. After conduction some benchmark tests, we observe that using keep alive with concurrenct upload/download decreases response time.
johha
reviewed
Mar 12, 2026
Comment on lines
98
to
108
| uploadInput := &s3.PutObjectInput{ | ||
| Body: src, | ||
| Bucket: aws.String(cfg.BucketName), | ||
| Key: b.key(dest), | ||
| } | ||
| if cfg.ServerSideEncryption != "" { | ||
| uploadInput.ServerSideEncryption = types.ServerSideEncryption(cfg.ServerSideEncryption) | ||
| } | ||
| if cfg.SSEKMSKeyID != "" { | ||
| uploadInput.SSEKMSKeyId = aws.String(cfg.SSEKMSKeyID) | ||
| } |
Contributor
There was a problem hiding this comment.
Same as line 132++ -> could be moved to a helper
cf-foundation-community-automation
bot
moved this to Inbox
in Foundational Infrastructure Working Group
aramprice
requested review from
a team,
evanfarrar and
peanball
and removed request for
a team
aramprice
moved this from Inbox
to Pending Review | Discussion
in Foundational Infrastructure Working Group
johha
approved these changes
Mar 12, 2026
github-project-automation
bot
moved this from Pending Review | Discussion
to Pending Merge | Prioritized
in Foundational Infrastructure Working Group
github-project-automation
bot
moved this from Pending Merge | Prioritized
to Done
in Foundational Infrastructure Working Group
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The S3 client's multipart_upload boolean flag did not give users control over when to use single vs. multipart upload. Additionally, the BOSH HTTP client was using CreateDefaultClient, which doesn't reuse TCP connections, resulting in higher latency under concurrent upload/download workloads.
Solution
Single upload threshold:
Replaced the multipart_upload flag with a new single_upload_threshold config parameter (bytes). Files at or below this size use a single PutObject call; larger files use multipart upload. The value is capped at 5GB to respect the AWS S3 hard limit. For GCS, which requires single put for all uploads but has no size limit, the threshold is automatically set to [math.MaxInt64]
Keep-alive HTTP client:
Replaced CreateDefaultClient with CreateKeepAliveDefaultClient. Benchmark tests showed this reduces response time for concurrent uploads/downloads by reusing existing TCP connections.