Skip to content

feat: Add OpenAiCompatibleLlm for custom OpenAI-compatible endpoints#1202

Open
Shaamam wants to merge 1 commit into
google:mainfrom
Shaamam:main
Open

feat: Add OpenAiCompatibleLlm for custom OpenAI-compatible endpoints#1202
Shaamam wants to merge 1 commit into
google:mainfrom
Shaamam:main

Conversation

@Shaamam
Copy link
Copy Markdown

@Shaamam Shaamam commented May 18, 2026

Link to Issue

1. Link to an existing issue (if applicable):

Testing Plan

This PR adds support for OpenAI-compatible LLM endpoints (Groq, Ollama, Azure OpenAI, Perplexity, etc.) via a new
OpenAiCompatibleLlm class.

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Summary of passed java test results:

[INFO] Running com.google.adk.models.OpenAiCompatibleLlmTest
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] Running com.google.adk.models.OpenAiCompatibleLlmIntegrationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 (requires Ollama)
[INFO]
[INFO] BUILD SUCCESS

Test coverage:

  • Builder validation (baseUrl and modelName required)
  • URL validation (rejects invalid URLs)
  • Header configuration (ImmutableMap and regular Map)
  • Custom timeout configuration
  • Registry pattern matching
  • Error handling for unsupported operations
  • Integration tests with Ollama (simple completion, multi-turn, timeout, registry)

Manual End-to-End (E2E) Tests:

Instructions to manually test the changes:

Setup:

# Set Groq API key
export GROQ_API_KEY="your-api-key-here"

# Compile test sources
cd /path/to/adk-java
./mvnw -f core/pom.xml test-compile

Run manual test:
./mvnw -f core/pom.xml exec:java \
  -Dexec.mainClass="com.google.adk.models.ManualGroqTest" \
  -Dexec.classpathScope=test

Test output (logs):

=== OpenAiCompatibleLlm Manual Test with Groq ===

[Test 1] Direct OpenAiCompatibleLlm usage
-----------------------------------------
✓ Created OpenAiCompatibleLlm for Groq
  Model: llama-3.3-70b-versatile

Sending request to Groq...
✓ Got response: Hello from Groq!

[Test 2] Registry integration
-----------------------------
✓ Registered pattern 'groq-.*'
✓ Successfully resolved model from registry

[Test 3] Multi-turn conversation
---------------------------------
Sending multi-turn conversation...
✓ Got response: Your name is Shaamam.
✓ Correctly remembered name from conversation!

==================================================
✓ ALL TESTS PASSED
==================================================

All three test scenarios passed successfully with real Groq API endpoint (llama-3.3-70b-versatile).

Checklist:

  • I have read the ./CONTRIBUTING.md document.
  • My pull request contains a single commit.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context:

Implementation approach:

  • Wraps existing ChatCompletionsHttpClient (reuses infrastructure, no code duplication)
  • Builder pattern for easy configuration (baseUrl, headers, timeout)
  • Pattern-based registry integration (e.g., groq-., ollama-.)
  • Non-streaming only (matches current ChatCompletionsHttpClient capabilities)
  • No live connections (OpenAI Chat Completions API limitation)

Files changed:
M README.md
A core/src/main/java/com/google/adk/models/OpenAiCompatibleLlm.java
A core/src/test/java/com/google/adk/models/OpenAiCompatibleLlmTest.java
A core/src/test/java/com/google/adk/models/OpenAiCompatibleLlmIntegrationTest.java
A core/src/test/java/com/google/adk/models/ManualGroqTest.java

Example usage:

  Groq (fast inference):
  OpenAiCompatibleLlm groq = OpenAiCompatibleLlm.builder()
      .baseUrl("https://api.groq.com/openai/v1/")
      .headers(ImmutableMap.of("Authorization", "Bearer " + apiKey))
      .modelName("llama-3.3-70b-versatile")
      .build();
  groq.registerWithPattern("groq-.*");

  LlmAgent agent = LlmAgent.builder()
      .model("groq-llama-3.3-70b-versatile")
      .instruction("You are a helpful assistant.")
      .build();

  Ollama (local models):
  OpenAiCompatibleLlm ollama = OpenAiCompatibleLlm.builder()
      .baseUrl("http://localhost:11434/v1/")
      .modelName("ollama-llama2")
      .build();
  ollama.registerWithPattern("ollama-.*");

Ready for review. Happy to make adjustments based on feedback from @hemasekhar-p and @anFatum.

  - Implement OpenAiCompatibleLlm class wrapping ChatCompletionsHttpClient
  - Support for Groq, Ollama, Azure OpenAI, Perplexity, and any OpenAI-compatible endpoint
  - Add comprehensive unit tests and integration tests
  - Update README with usage examples
  - Tested and verified with Groq API
@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 18, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Add support for OpenAI-compatible LLM endpoints (Groq, Ollama, Azure OpenAI, etc.)

1 participant