Skip to content

Preload default Ollama model before suggestions#781

Merged
FuJacob merged 1 commit into
mainfrom
fix/ollama-cold-start
Jul 11, 2026
Merged

Preload default Ollama model before suggestions#781
FuJacob merged 1 commit into
mainfrom
fix/ollama-cold-start

Conversation

@FuJacob

@FuJacob FuJacob commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Preload Cotabby's default local Ollama model through its native API before the first autocomplete request, so cancellable typing work cannot abort the model's cold start. The preload has a 120-second timeout and keeps the model resident, while non-Ollama OpenAI-compatible endpoints remain untouched.

Validation

  • swiftlint lint --quiet — exited 0.
  • xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build-for-testing -derivedDataPath build/DerivedDataTEST BUILD SUCCEEDED.
  • xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build -derivedDataPath build/DerivedDataBUILD SUCCEEDED.
  • Focused OpenAICompatibleAPIClientTests execution could not load the app-hosted test bundle because the host and bundle have different Team IDs, the repository's documented local signing limitation. Compilation and linking of the test bundle succeeded.

Linked issues

None.

Risk / rollout notes

The Ollama-only preload is deliberately restricted to Cotabby's exact default endpoint (http://127.0.0.1:11434/v1). Custom OpenAI-compatible endpoints continue using only the standard API, and preload failures remain best-effort warnings rather than user-facing generation failures.

Greptile Summary

This PR adds native Ollama preloading for Cotabby's default local endpoint. The main changes are:

  • A default-only /api/generate URL derived from the OpenAI-compatible base URL.
  • A preload request that keeps the selected Ollama model resident.
  • OpenAI-compatible engine prewarm logic that runs the preload best-effort.
  • Tests for the default Ollama route and non-default endpoint no-op behavior.

Confidence Score: 5/5

This looks safe to merge after a small cleanup to avoid repeated local preloads.

  • The native Ollama request is limited to the exact default loopback endpoint.
  • The payload and timeout are covered by focused tests.
  • Repeated focus changes can start duplicate preload requests for the same model.

Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift

Important Files Changed

Filename Overview
Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift Updates the prewarm comment to include the OpenAI-compatible Ollama preload path.
Cotabby/Models/OpenAICompatibleEndpointModels.swift Adds a default-only native Ollama generate URL derived from the existing base URL.
Cotabby/Services/Runtime/OpenAICompatibleAPIClient.swift Adds the native Ollama preload request and status validation.
Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift Adds best-effort preload behavior, with a duplicate-request cleanup noted for repeated focus changes.
CotabbyTests/OpenAICompatibleAPIClientTests.swift Adds coverage for Ollama URL derivation, preload payload fields, timeout, and non-default endpoint behavior.

Fix All in Codex Fix All in Claude Code

Reviews (1): Last reviewed commit: "Preload default Ollama model before sugg..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

@FuJacob FuJacob merged commit 5cdaa7f into main Jul 11, 2026
5 checks passed
@FuJacob FuJacob deleted the fix/ollama-cold-start branch July 11, 2026 00:31
Comment on lines +135 to +138
let didPreload = try await client.preloadDefaultOllamaModel(
configuration: configuration,
apiKey: try apiKeyProvider()
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Duplicate Ollama Preloads

When focus moves across editable fields while the default Ollama model is still loading, each prewarm call starts another 120-second /api/generate request for the same model. These tasks are not tracked or cancelled by the prediction work controller, so rapid focus changes can queue duplicated local model-load work before the first autocomplete request.

Fix in Codex Fix in Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant