Extend v1/messages by introducing token-in/out and returning routed experts

[lvhan028]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[Copilot]

Pull request overview

This PR extends the Anthropic-compatible /v1/messages API to accept additional /generate-style inputs and to optionally return richer generation metadata (token IDs and routed experts), with corresponding updates to streaming and tests.

Changes:

• Extend MessagesRequest/MessagesResponse to include input_ids, image_data, return_token_ids, return_routed_experts, plus response fields output_ids and routed_experts.
• Update /v1/messages endpoint to validate/route between messages vs input_ids(+image_data) and to populate output_ids/routed_experts in responses.
• Update Anthropic SSE streaming to optionally emit output_ids and routed_experts; add/extend tests for the new behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_lmdeploy/serve/anthropic/test_endpoints.py	Adds tests for new request fields and response metadata in both streaming and non-stream modes.
lmdeploy/serve/anthropic/streaming.py	Adds optional emission of output_ids in content_block_delta and routed_experts in message_delta.
lmdeploy/serve/anthropic/protocol.py	Extends Anthropic request/response models with new optional fields.
lmdeploy/serve/anthropic/endpoints/messages.py	Adds validation and routing for input_ids/image_data, wires new flags into streaming and non-stream responses.
lmdeploy/serve/anthropic/adapter.py	Passes return_routed_experts into GenerationConfig.

Comments suppressed due to low confidence (2)

lmdeploy/serve/anthropic/endpoints/messages.py:109

• When input_ids mode is used (messages empty and resolved_input_ids set), tool-calling inputs like tools/tool_choice are silently ignored because no response_parser is built and tools=None is passed into async_engine.generate. It would be clearer to reject tool-calling parameters in input_ids mode (400) or explicitly support them.

        parser_cls = getattr(server_context, 'response_parser_cls', None)
        if request.tools and (parser_cls is None or parser_cls.tool_parser_cls is None):
            return create_error_response(
                HTTPStatus.BAD_REQUEST,
                'Please launch the api_server with --tool-call-parser if you want to use tool calling.')

tests/test_lmdeploy/serve/anthropic/test_endpoints.py:608

• Despite the docstring, this test never sets return_routed_experts=True and never asserts on routed_experts. As written it only verifies output_ids behavior. Add a request with return_routed_experts=True and assert the response contains the expected routed_experts value (or adjust the docstring/name).

def test_messages_non_stream_with_output_ids_and_routed_experts():
    """When return_token_ids is True, output_ids must be populated with the
    generated token IDs.

    When return_routed_experts is True, routed_experts must be populated from the engine result.
    """
    client = _make_client()
    # Test output_ids when return_token_ids is True
    response = client.post(
        '/v1/messages',
        headers={'anthropic-version': '2023-06-01'},
        json={
            'model': 'fake-model',
            'max_tokens': 16,
            'messages': [{'role': 'user', 'content': 'Hi'}],
            'return_token_ids': True,
        },
    )
    assert response.status_code == 200
    data = response.json()
    # output_ids should be populated from token IDs generated by the fake engine
    assert data.get('output_ids') == [101, 102]

    # Test that output_ids is None when return_token_ids is False (default)
    response2 = client.post(
        '/v1/messages',
        headers={'anthropic-version': '2023-06-01'},
        json={
            'model': 'fake-model',
            'max_tokens': 16,
            'messages': [{'role': 'user', 'content': 'Hi'}],
        },
    )
    assert response2.status_code == 200
    data2 = response2.json()
    assert data2.get('output_ids') is None

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.