fix(server): flatten OpenAI multimodal content before chat template#1387
fix(server): flatten OpenAI multimodal content before chat template#1387fuheaven wants to merge 2 commits into
Conversation
New neo model is trained at 4K, so lift the resolution ceiling from 2K: - add 4K presets to ImageConfig aspect-ratio table and allowed size set - bump smart_resize max_pixels from 2048^2 to 4096^2 in ImageConfig.get_resolution (t2i / custom height-width) and X2IParams.update_hw (it2i output = input size) Verified end-to-end: custom 4096x4096 and 4K 16:9 (4096x2304) t2i generation.
OpenAI-style it2i/vqa requests send message.content as a list of parts (image_url/text). Qwen-style HF chat templates call string methods like startswith on content, which raises UndefinedError and 500s the request. Flatten list content to a string with <image>/<audio> placeholders so the template can render; image bytes are already extracted via MultimodalParams.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cafd5825ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| elif t == "image_url": | ||
| parts.append("<image>") | ||
| elif t == "audio_url": | ||
| parts.append("<audio>") |
There was a problem hiding this comment.
When an OpenAI-style image_url/audio_url request is served by Qwen2/3-VL or Qwen3-Omni, this replaces the content with generic <image>/<audio> text while _get_images_and_audios still attaches the media. Those tokenizers do not count these generic strings: Qwen2/3-VL looks for vision_start_id immediately followed by vision_end_id and then raises invalid image tag num when the attached image count is nonzero (lightllm/models/qwen2_vl/model.py:66-85), and Omni does the same for audio start/end ids (lightllm/models/qwen3_omni_moe_thinker/model.py:126-144). This means multimodal chat completions for those models still fail after prompt building; preserve the list for templates that know how to render it or emit the tokenizer-specific sentinels instead.
Useful? React with 👍 / 👎.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for 4K image resolutions by updating the aspect ratio mapping, size set, and maximum pixel limits in smart_resize across api_models.py and x2i_params.py. It also introduces a helper function _flatten_multimodal_content in build_prompt.py to flatten OpenAI-style multimodal content lists into plain strings with placeholder tokens, preventing template rendering crashes. The feedback suggests simplifying the retrieval of text content and improving compatibility with alternative multimodal type keys (e.g., "image" and "audio").
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| t = item.get("type") | ||
| if t == "text": | ||
| parts.append(item.get("text", "") or "") | ||
| elif t == "image_url": | ||
| parts.append("<image>") | ||
| elif t == "audio_url": | ||
| parts.append("<audio>") |
There was a problem hiding this comment.
改进建议
- 简化
.get()的冗余默认值：在 Python 中，dict.get(key, default)仅在键不存在时返回默认值。如果键存在但其值为None，它仍然会返回None。因此，如果"text"被显式设置为None，item.get("text", "")仍会返回None。使用item.get("text") or ""更加简洁、地道，并且能完美处理键不存在以及值为None的情况。 - 增强多模态类型的兼容性：部分客户端或自定义集成可能会使用
"image"或"audio"作为类型，而不是"image_url"或"audio_url"。同时支持这两种形式可以使展平逻辑更加健壮和具有前瞻性。
| t = item.get("type") | |
| if t == "text": | |
| parts.append(item.get("text", "") or "") | |
| elif t == "image_url": | |
| parts.append("<image>") | |
| elif t == "audio_url": | |
| parts.append("<audio>") | |
| t = item.get("type") | |
| if t == "text": | |
| parts.append(item.get("text") or "") | |
| elif t in ("image_url", "image"): | |
| parts.append("<image>") | |
| elif t in ("audio_url", "audio"): | |
| parts.append("<audio>") |
1 participant
OpenAI 风格的 it2i/vqa 请求会把 message.content 发成 list（image_url + text）。Qwen 风格的 HF chat template 会直接对 content 调字符串方法（如 startswith），于是 Jinja 报 'list object' has no attribute 'startswith'，接口 500。
修复：在 apply_chat_template 前把 list content 展平成字符串，图片用![]()
、音频用 占位；图片数据本身仍由上游 _get_images_and_audios → MultimodalParams 处理。