feat: Add support for OpenAI-compatible LLM endpoints (Groq, Ollama, Azure OpenAI, etc.)

[Shaamam]

🔴 Required Information

Is your feature request related to a specific problem?

Yes. Currently, ADK-Java only supports Google's Gemini and Claude models out of the box. Developers cannot easily use other LLM
providers that implement the OpenAI Chat Completions API format, such as:

• Groq (fast inference with open models)
• Ollama (local models for offline development)
• Azure OpenAI (enterprise deployments)
• Perplexity (search-augmented responses)
• Any custom OpenAI-compatible endpoint

Python ADK supports this via LiteLLM integration, but Java ADK lacks a native solution. This creates friction for developers who
want to:

• Experiment with different model providers
• Reduce inference costs by using alternative providers
• Develop/test locally without external API calls
• Support enterprise requirements (Azure OpenAI)

Describe the Solution You'd Like

Add a new OpenAiCompatibleLlm class that wraps the existing ChatCompletionsHttpClient to support any OpenAI-compatible endpoint
with a simple builder pattern.

Key capabilities:

1. Builder pattern for easy configuration (baseUrl, headers, timeout)
2. Pattern-based registry integration (e.g., groq-.*, ollama-.*)
3. Reuses existing ChatCompletionsHttpClient infrastructure
4. Non-streaming requests (matches current ChatCompletionsHttpClient capabilities)
5. Comprehensive tests (unit + integration + manual verification)

Impact on your work

Impact Level: High

This feature is critical for:

• My current project: Building a multi-model agent system that needs to switch between Gemini (complex reasoning) and Groq
  (fast responses) based on task type
• Cost optimization: Groq is 10x cheaper than Gemini for simple tasks
• Local development: Need Ollama support for offline testing without API costs

Timeline: Would like this in the next release if possible. Currently using a workaround with direct HTTP calls, but it's not
maintainable.

Willingness to contribute

Yes - I have already implemented this feature with:

• ✅ Core OpenAiCompatibleLlm class with builder pattern
• ✅ 13 unit tests (builder validation, registry integration, error handling)
• ✅ Integration tests with Ollama
• ✅ Manual verification with Groq API (tested successfully)
• ✅ README documentation with examples
• ✅ Follows Google Java Style Guide (google-java-format applied)
• ✅ Aligned with CONTRIBUTING.md requirements

Can submit PR immediately if maintainers approve this approach.

---

🟡 Recommended Information

Describe Alternatives You've Considered

1. Direct HTTP calls: Manually building requests to OpenAI-compatible endpoints

   • ❌ Doesn't integrate with LlmRegistry
   • ❌ No pattern matching for model resolution
   • ❌ Duplicates HTTP client logic
   • ❌ Not maintainable

2. Separate library wrapper: Create external library for each provider

   • ❌ Fragments ecosystem
   • ❌ Each provider needs separate maintenance
   • ❌ Doesn't leverage existing ADK infrastructure

3. Use Python ADK instead: Switch to Python for LiteLLM support

   • ❌ Not viable for Java-based projects
   • ❌ Team expertise is in Java

4. Wait for official provider support: Request Google add each provider individually

   • ❌ Slow (requires coordination with each provider)
   • ❌ Doesn't scale to custom/internal endpoints

Why the proposed solution is better: Single implementation supports all OpenAI-compatible providers, reuses existing
infrastructure, and allows custom endpoints.

Proposed API / Implementation

// Example 1: Groq (fast inference)
OpenAiCompatibleLlm groq = OpenAiCompatibleLlm.builder()
    .baseUrl("https://api.groq.com/openai/v1/")
    .headers(ImmutableMap.of("Authorization", "Bearer " + apiKey))
    .modelName("llama-3.3-70b-versatile")
    .timeoutMillis(30_000)
    .build();

// Register pattern for model resolution
groq.registerWithPattern("groq-.*");

// Use with LlmAgent
LlmAgent agent = LlmAgent.builder()
    .model("groq-llama-3.3-70b-versatile")
    .instruction("You are a helpful assistant.")
    .build();

String response = agent.runAsync(invocationContext).blockingFirst();

// Example 2: Ollama (local models)
OpenAiCompatibleLlm ollama = OpenAiCompatibleLlm.builder()
    .baseUrl("http://localhost:11434/v1/")
    .headers(ImmutableMap.of())  // No auth for local
    .modelName("ollama-llama2")
    .build();

ollama.registerWithPattern("ollama-.*");

// Example 3: Azure OpenAI (enterprise)
OpenAiCompatibleLlm azure = OpenAiCompatibleLlm.builder()
    .baseUrl("https://<resource>.openai.azure.com/openai/deployments/<deployment>/")
    .headers(ImmutableMap.of("api-key", azureApiKey))
    .modelName("azure-gpt-4")
    .build();

azure.registerWithPattern("azure-.*");

Implementation Architecture:

• OpenAiCompatibleLlm extends BaseLlm
• Wraps existing ChatCompletionsHttpClient (no code duplication)
• Uses LlmRegistry.registerLlm(pattern, factory) for pattern matching
• Throws UnsupportedOperationException for live connections (OpenAI API limitation)

Files to be added/modified:
M README.md
A core/src/main/java/com/google/adk/models/OpenAiCompatibleLlm.java
A core/src/test/java/com/google/adk/models/OpenAiCompatibleLlmTest.java
A core/src/test/java/com/google/adk/models/OpenAiCompatibleLlmIntegrationTest.java
A core/src/test/java/com/google/adk/models/ManualGroqTest.java

Additional Context

Testing completed:

• ✅ Unit tests: 13 tests, all passing
• ✅ Integration tests: Ollama-based tests for real endpoints
• ✅ Manual verification: Successfully tested with Groq API (llama-3.3-70b-versatile)
  • Simple completion test
  • Registry integration test
  • Multi-turn conversation with context retention

Design decisions:

• Non-streaming only: Matches ChatCompletionsHttpClient capabilities (streaming can be added in future PR)
• Pattern-based registration: Allows multiple providers with different prefixes (e.g., groq-, ollama-, azure-*)
• No live connections: OpenAI Chat Completions API doesn't support bidirectional live connections

Questions for maintainers:

1. Should we add streaming support in a follow-up PR, or include it now?
2. Any concerns with the pattern-based registration approach?
3. Should integration tests be tagged/skipped in CI (currently requires manual Ollama setup)?

Related: Addresses similar functionality to Python ADK's LiteLLM integration, but using native Java implementation that reuses
existing ADK infrastructure.

---

Implementation ready: I have a fully tested, working implementation and can submit a PR immediately upon approval of this approach.

[hemasekhar-p]

Hi @Shaamam, Thank you for submitting this feature request. It is currently under review by our team, we will keep you updated if any additional information is required. We appreciate your input.

[hemasekhar-p]

@anFatum, Please take a look into it.

[Shaamam]

Hi @hemasekhar-p and @anFatum,

Thank you for reviewing this! I've submitted PR #1202 with a complete implementation to facilitate the code review.

The PR includes:

• Core OpenAiCompatibleLlm implementation
• 13 unit tests (all passing)
• Integration tests with Ollama
• Manual verification with Groq API (tested successfully)
• Documentation updates

All tests pass locally and the implementation has been verified with real Groq API. Happy to make any adjustments based on your feedback!

[svetanis]

Hello Team! I saw this issue and wanted to flag that I've been working on a related solution.
I submitted a PR (feat:contrib/model-prism: pluggable LLM backend SPI for ADK Java via ServiceLoader #1199) recently that covers similar ground — OpenAiCompatibleLlm as a base class, plus provider discovery via ServiceLoader SPI as a contrib module. It also includes streaming support and 11 demo classes.

Happy to collaborate, align approaches, or break the PR into smaller pieces if that's easier to review. Just wanted to make sure the efforts are visible to each other.

[FuturMix]

Great feature request and solid PR work by @Shaamam and @svetanis. The multi-provider pattern is essential for production agent systems — here are a few architectural considerations from running similar setups:

Model routing by task type is the biggest cost win

The issue author mentioned switching between Gemini (complex reasoning) and Groq (fast responses) based on task type — this is exactly the right pattern. In practice, a well-configured routing layer can reduce API costs by 40-70% without quality loss by sending:

• Planning/reasoning → expensive model (Gemini, Claude Opus)
• Code generation → mid-tier (Sonnet, GPT-5.5)
• Summarization/formatting → cheap/fast model (Haiku, DeepSeek, Groq)

Consider the provider abstraction carefully

Both PRs take slightly different approaches. A few things worth considering for the final design:

1. Credential isolation: Each provider may need different auth headers (Authorization: Bearer vs x-api-key vs custom). The builder should support a generic header map rather than assuming Authorization only.

2. Model name normalization: Some providers accept gpt-5.5 while others need openai/gpt-5.5. Having an optional model alias map in the config avoids hardcoding provider-specific naming.

3. Streaming support: The initial non-streaming implementation is fine for an MVP, but agent workflows with long outputs really benefit from streaming. The SPI approach in PR feat:contrib/model-prism: pluggable LLM backend SPI for ADK Java via ServiceLoader #1199 seems better positioned for this.

4. Fallback chain: For production reliability, being able to define primary → fallback provider chains (e.g., Groq → Ollama if Groq is rate-limited) is very valuable.

The Python ADK's LiteLLM integration handles some of this by delegating to LiteLLM's provider registry — the Java equivalent could follow a similar pattern where an OpenAI-compatible gateway handles provider abstraction externally, and ADK-Java only needs to talk to one endpoint.