Skip to content

fix(server): flatten OpenAI multimodal content before chat template #1387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fuheaven wants to merge 2 commits into
base: neo_plus_clean
Choose a base branch
from
+40 −14
Open

fix(server): flatten OpenAI multimodal content before chat template #1387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
068ea3d
feat(x2i): raise image generation resolution cap to 4K
fuheaven Jul 6, 2026
cafd582
fix(server): flatten OpenAI multimodal content before chat template
fuheaven Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions lightllm/server/api_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,19 +447,19 @@ class ImageConfig(BaseModel):
cfg_norm: Optional[Literal["none", "cfg_zero_star", "global", "text_channel", "channel"]] = None
dynamic_resolution: Optional[bool] = True
_aspect_ratio_to_resolution: ClassVar[dict] = {
"1:1": {"1K": (1024, 1024), "1.5K": (1536, 1536), "2K": (2048, 2048)},
"16:9": {"1.5K": (2048, 1152), "2K": (2720, 1536)},
"9:16": {"1.5K": (1152, 2048), "2K": (1536, 2720)},
"3:2": {"1.5K": (1888, 1248), "2K": (2496, 1664)},
"2:3": {"1.5K": (1248, 1888), "2K": (1664, 2496)},
"4:3": {"1.5K": (1760, 1312), "2K": (2368, 1760)},
"3:4": {"1.5K": (1312, 1760), "2K": (1760, 2368)},
"1:2": {"1.5K": (1088, 2144), "2K": (1440, 2880)},
"2:1": {"1.5K": (2144, 1088), "2K": (2880, 1440)},
"1:3": {"1.5K": (864, 2592), "2K": (1152, 3456)},
"3:1": {"1.5K": (2592, 864), "2K": (3456, 1152)},
"1:1": {"1K": (1024, 1024), "1.5K": (1536, 1536), "2K": (2048, 2048), "4K": (4096, 4096)},
"16:9": {"1.5K": (2048, 1152), "2K": (2720, 1536), "4K": (4096, 2304)},
"9:16": {"1.5K": (1152, 2048), "2K": (1536, 2720), "4K": (2304, 4096)},
"3:2": {"1.5K": (1888, 1248), "2K": (2496, 1664), "4K": (4096, 2720)},
"2:3": {"1.5K": (1248, 1888), "2K": (1664, 2496), "4K": (2720, 4096)},
"4:3": {"1.5K": (1760, 1312), "2K": (2368, 1760), "4K": (4096, 3072)},
"3:4": {"1.5K": (1312, 1760), "2K": (1760, 2368), "4K": (3072, 4096)},
"1:2": {"1.5K": (1088, 2144), "2K": (1440, 2880), "4K": (2048, 4096)},
"2:1": {"1.5K": (2144, 1088), "2K": (2880, 1440), "4K": (4096, 2048)},
"1:3": {"1.5K": (864, 2592), "2K": (1152, 3456), "4K": (1376, 4096)},
"3:1": {"1.5K": (2592, 864), "2K": (3456, 1152), "4K": (4096, 1376)},
}
_size_set: ClassVar[set[str]] = {"1.5K", "2K"}
_size_set: ClassVar[set[str]] = {"1.5K", "2K", "4K"}

@field_validator("image_size", mode="before")
@classmethod
Expand Down Expand Up @@ -508,7 +508,7 @@ def get_resolution(self):
base = self._aspect_ratio_to_resolution[self.aspect_ratio][self.image_size]
w, h = base

h, w = smart_resize(h, w, factor=32, min_pixels=1024 * 1024, max_pixels=2048 * 2048)
h, w = smart_resize(h, w, factor=32, min_pixels=1024 * 1024, max_pixels=4096 * 4096)
return w, h


Expand Down
26 changes: 26 additions & 0 deletions lightllm/server/build_prompt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,37 @@ def _alias_reasoning_to_reasoning_content(messages: list) -> None:
msg["reasoning_content"] = reasoning


def _flatten_multimodal_content(messages: list) -> None:
# OpenAI-style chat requests may send content as a list of parts
# ({"type": "text", ...}, {"type": "image_url", ...}). Most HF chat
# templates (Qwen, LLaMA, ...) call string methods on `content` directly
# and crash on lists. Flatten into a plain string with placeholder tokens
# so the template can render; the image data itself is already extracted
# upstream via `_get_images_and_audios` and fed through MultimodalParams.
for msg in messages:
content = msg.get("content")
if not isinstance(content, list):
continue
parts: list = []
for item in content:
if not isinstance(item, dict):
continue
t = item.get("type")
if t == "text":
parts.append(item.get("text", "") or "")
elif t == "image_url":
parts.append("<image>")
elif t == "audio_url":
parts.append("<audio>")
Comment on lines +100 to +103

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep model-specific multimodal sentinels

When an OpenAI-style image_url/audio_url request is served by Qwen2/3-VL or Qwen3-Omni, this replaces the content with generic <image>/<audio> text while _get_images_and_audios still attaches the media. Those tokenizers do not count these generic strings: Qwen2/3-VL looks for vision_start_id immediately followed by vision_end_id and then raises invalid image tag num when the attached image count is nonzero (lightllm/models/qwen2_vl/model.py:66-85), and Omni does the same for audio start/end ids (lightllm/models/qwen3_omni_moe_thinker/model.py:126-144). This means multimodal chat completions for those models still fail after prompt building; preserve the list for templates that know how to render it or emit the tokenizer-specific sentinels instead.

Useful? React with 👍 / 👎.

Comment on lines +97 to +103

@gemini-code-assist gemini-code-assist Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

改进建议

  1. 简化 .get() 的冗余默认值：在 Python 中，dict.get(key, default) 仅在键不存在时返回默认值。如果键存在但其值为 None，它仍然会返回 None。因此，如果 "text" 被显式设置为 None，item.get("text", "") 仍会返回 None。使用 item.get("text") or "" 更加简洁、地道，并且能完美处理键不存在以及值为 None 的情况。
  2. 增强多模态类型的兼容性：部分客户端或自定义集成可能会使用 "image" 或 "audio" 作为类型，而不是 "image_url" 或 "audio_url"。同时支持这两种形式可以使展平逻辑更加健壮和具有前瞻性。
Suggested change
t = item.get("type")
if t == "text":
parts.append(item.get("text", "") or "")
elif t == "image_url":
parts.append("<image>")
elif t == "audio_url":
parts.append("<audio>")
t = item.get("type")
if t == "text":
parts.append(item.get("text") or "")
elif t in ("image_url", "image"):
parts.append("<image>")
elif t in ("audio_url", "audio"):
parts.append("<audio>")

msg["content"] = "\n".join(p for p in parts if p != "")


async def build_prompt(request, tools) -> str:
# pydantic格式转成dict， 否则，当根据tokenizer_config.json拼template时，Jinja判断无法识别
messages = [m.model_dump(by_alias=True, exclude_none=True) for m in request.messages]
_normalize_tool_call_arguments(messages)
_alias_reasoning_to_reasoning_content(messages)
_flatten_multimodal_content(messages)

kwargs = {"conversation": messages}
if request.character_settings:
Expand Down
2 changes: 1 addition & 1 deletion lightllm/server/core/objs/x2i_params.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def update_hw(self, width: int, height: int):
return
from lightllm.models.neo_chat_moe.vision_process import smart_resize

h, w = smart_resize(height, width, factor=32, min_pixels=512 * 512, max_pixels=2048 * 2048)
h, w = smart_resize(height, width, factor=32, min_pixels=512 * 512, max_pixels=4096 * 4096)
self.width = w
self.height = h
self.has_updated_hw = True
Expand Down