Preload default Ollama model before suggestions#781
Merged
Conversation
Comment on lines
+135
to
+138
| let didPreload = try await client.preloadDefaultOllamaModel( | ||
| configuration: configuration, | ||
| apiKey: try apiKeyProvider() | ||
| ) |
Contributor
There was a problem hiding this comment.
When focus moves across editable fields while the default Ollama model is still loading, each prewarm call starts another 120-second /api/generate request for the same model. These tasks are not tracked or cancelled by the prediction work controller, so rapid focus changes can queue duplicated local model-load work before the first autocomplete request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preload Cotabby's default local Ollama model through its native API before the first autocomplete request, so cancellable typing work cannot abort the model's cold start. The preload has a 120-second timeout and keeps the model resident, while non-Ollama OpenAI-compatible endpoints remain untouched.
Validation
swiftlint lint --quiet— exited 0.xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build-for-testing -derivedDataPath build/DerivedData— TEST BUILD SUCCEEDED.xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build -derivedDataPath build/DerivedData— BUILD SUCCEEDED.OpenAICompatibleAPIClientTestsexecution could not load the app-hosted test bundle because the host and bundle have different Team IDs, the repository's documented local signing limitation. Compilation and linking of the test bundle succeeded.Linked issues
None.
Risk / rollout notes
The Ollama-only preload is deliberately restricted to Cotabby's exact default endpoint (
http://127.0.0.1:11434/v1). Custom OpenAI-compatible endpoints continue using only the standard API, and preload failures remain best-effort warnings rather than user-facing generation failures.Greptile Summary
This PR adds native Ollama preloading for Cotabby's default local endpoint. The main changes are:
/api/generateURL derived from the OpenAI-compatible base URL.Confidence Score: 5/5
This looks safe to merge after a small cleanup to avoid repeated local preloads.
Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift
Important Files Changed
Reviews (1): Last reviewed commit: "Preload default Ollama model before sugg..." | Re-trigger Greptile