[server] Retry rev-mismatched code-sync inserts server-side#21319
Open
[server] Retry rev-mismatched code-sync inserts server-side#21319
Conversation
When multiple workspaces sync globalState concurrently, the If-Match rev becomes stale between the client's GET and POST. The VS Code client retries 412 responses with no recursion limit, exhausting its 100-request/5-min budget and triggering 'Settings sync is suspended'. Retry the insert up to 3 times server-side with the current latest rev, absorbing transient concurrency conflicts before they reach the client. Co-authored-by: Ona <no-reply@ona.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes CLC-2223
Problem
Customers see "Settings sync is suspended temporarily because the current device is making too many requests" banners. Logs show repeated 412 responses from
/code-sync/v1/resource/globalStateuntil the client hits its rate limit.Root cause: When the VS Code client POSTs to update a resource, it sends
If-Match: <last-known-rev>. If the server's latest rev has changed (e.g. another workspace wrote to the same resource), the server returns 412. The VS Code client handles 412 by re-fetching remote data and retryingperformSync— with no recursion limit. In a multi-workspace scenario (or with rapidly-changingglobalState), the rev keeps changing between the client's GET and POST, creating a cascading retry loop. Each retry cycle costs 2+ HTTP requests (GET + POST), and the client'sRequestsSessionbudget is 100 requests per 5 minutes. The loop exhausts this budget, triggering the suspension banner.Fix
Retry the
db.insertcall up to 3 times server-side when a rev mismatch occurs, using the server's current latest rev for each attempt. This absorbs transient concurrency conflicts before they reach the client, breaking the unbounded retry loop.The retry is scoped: it only applies when the initial insert fails due to a rev mismatch (
latestRevwas provided but didn't match), and is skipped foreditSessionsresources which use a different error code (400).Changes
components/server/src/code-sync/code-sync-service.ts: Server-side retry loop inpostResourcefor rev-mismatched insertscomponents/gitpod-db/src/typeorm/code-sync-resource-db.ts: NewgetLatestRevision()method to fetch the current rev without a full resource listcomponents/gitpod-db/src/typeorm/code-sync-resource-db.spec.db.ts: Tests forgetLatestRevision()