Skip to content

[ci] add k4v2 testcase and fix some fail cases#4601

Open
zhulinJulia24 wants to merge 24 commits into
InternLM:mainfrom
zhulinJulia24:add_k4v2
Open

[ci] add k4v2 testcase and fix some fail cases#4601
zhulinJulia24 wants to merge 24 commits into
InternLM:mainfrom
zhulinJulia24:add_k4v2

Conversation

@zhulinJulia24
Copy link
Copy Markdown
Collaborator

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
  3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

Copilot AI review requested due to automatic review settings May 20, 2026 05:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the autotest/CI matrix and related test utilities to add coverage for KV cache quantization policy 42 (TurboQuant / K4V2), refreshes some model selections, and tightens a few flaky/failure-prone test behaviors in long-context and RESTful interface tests.

Changes:

  • Add quant_policy=42 to functional config generation and introduce a kvint42 allowlist (plus a prefix-caching exclusion for linear-attention Qwen3.5 models).
  • Improve long-context pipeline tests by asserting stream output structure/content and failing fast on spawn worker crashes.
  • Update CI/model configs: expand daily E2E models, adjust evaluation batch size, and trim the OpenCompass eval dataset/summarizer scope.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
autotest/utils/evaluate_utils.py Removes in-config worker overrides for OpenCompass runs.
autotest/utils/config_utils.py Adds quant policy 42 path + kvint42 allowlist and excludes prefix caching for Qwen3.5.
autotest/tools/common_case_config.py Switches fallback MLLM model to THUDM/cogvlm-chat-hf.
autotest/interface/restful/test_restful_completions_v1.py Adds additional debug prints for RESTful completions tests.
autotest/interface/restful/test_restful_chat_completions_v1.py Makes “empty content” assertions more permissive (not ...).
autotest/interface/pipeline/test_pipeline_longtext_func.py Strengthens streaming/passkey assertions and improves spawn failure reporting; updates model coverage.
autotest/evaluate/eval_config_chat.py Reduces eval batch size (1024 → 256).
autotest/config.yml Updates model lists and introduces pytorch_quantization.kvint42 allowlist.
.github/workflows/daily_ete_test.yml Expands daily RESTful model matrix and adds --trust-remote-code when starting API server.
.github/scripts/eval_base_config.py Narrows imported datasets and summarizer entries for evaluation config.
Comments suppressed due to low confidence (1)

.github/workflows/daily_ete_test.yml:744

  • --trust-remote-code enables executing arbitrary model repository code inside the CI runner/container. If only certain models require it, consider gating this flag via the matrix (e.g., add it to matrix.extra only for those models) or otherwise documenting/justifying why it must be enabled globally for this job.
          ln -s ${{env.REPORT_DIR}}/.pytest_cache autotest
      - name: Start restful api
        run: |
          lmdeploy serve api_server /nvme/qa_test_models/${{matrix.model_path}} --tp ${{matrix.tp}} --backend ${{matrix.backend}} ${{matrix.extra}} --allow-terminate-by-client --trust-remote-code > ${{env.REPORT_DIR}}/${{matrix.backend}}_${{matrix.model}}_${{matrix.generate_type}}_start_restful.log 2>&1 &
          echo "restful_pid=$!"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread autotest/utils/config_utils.py
Comment thread .github/workflows/daily_ete_test.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants