[BugFix] Seperate prometheus multiproc dir for single-server multi-dp services

[liyonghua0910]

Motivation

Fix metric interference when multiple data-parallel services run on one server by isolating Prometheus multiprocess files per DP rank.

Modifications

• Track the original PROMETHEUS_MULTIPROC_DIR set during metrics initialization.
• Add setup_dp_prometheus_dir() to create per-DP dp{i} subdirectories and switch the target environment.
• Apply DP-specific Prometheus dirs when launching internal-adapter DP services from LLMEngine / EngineService and when multi_api_server starts per-DP API server processes.
• Update unit tests for multi API server Prometheus dirs and Prometheus setup behavior.

Usage or Command

N/A

Accuracy Tests

N/A

Checklist

• Add at least a tag in the PR title.
  • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
  • You can add new tags based on the PR content, but the semantics must be clear.
• Format your code, run pre-commit before commit.
• Add unit tests. Please write the reason in this PR if no unit tests.
• Provide accuracy results.
• If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 59.45946% with 15 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@e58f31c). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...astdeploy/metrics/prometheus_multiprocess_setup.py	73.91%	5 Missing and 1 partial ⚠️
fastdeploy/engine/engine.py	16.66%	5 Missing ⚠️
fastdeploy/engine/common_engine.py	33.33%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #8059   +/-   ##
==========================================
  Coverage           ?   67.48%           
==========================================
  Files              ?      475           
  Lines              ?    66959           
  Branches           ?    10332           
==========================================
  Hits               ?    45184           
  Misses             ?    18892           
  Partials           ?     2883           

Flag	Coverage Δ
GPU	77.50% <59.45%> (?)
XPU	6.95% <35.13%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[PaddlePaddle-bot]

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-07-07 12:04:35 UTC+08:00

CI报告基于以下代码生成（30分钟更新一次）:
PR commit: 6dd6b4b | Merge base: e58f31c (branch: develop)

---

1 Required任务 : 8/10 通过

总执行（rerun次数）	总任务	✅ 通过	❌ 失败	⏳ 运行中	⏸️ 等待中	跳过
42(0)	42	38	4	0	0	0

任务	错误类型	置信度	日志
Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage	PR问题	高	Job
Approval	需要 Approval	高	Job

2 失败详情

🔴 Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — PR问题（置信度: 高）

错误类型: PR问题 | 置信度: 高
分析器: ci_analyze_unittest_fastdeploy
失败用例: 单测通过，覆盖率校验失败

用例	错误摘要
diff coverage	PR diff 覆盖率 67%，低于 80%，12 行未覆盖

关键日志:

Failure. Coverage is below 80%.
fastdeploy/engine/common_engine.py (66.7%): Missing lines 2777,2783
fastdeploy/engine/engine.py (16.7%): Missing lines 867,905-906,909-910
fastdeploy/metrics/prometheus_multiprocess_setup.py (78.3%): Missing lines 68-72
total_num_violations: 12, total_percent_covered: 67

• 根因摘要: 新增 DP Prometheus 分支未覆盖
  测试均已通过，但 Verify Code Coverage Threshold (80%) 步骤失败。PR 新增的 internal-adapter DP Prometheus 目录切换分支，以及 setup_dp_prometheus_dir(..., move_existing=True) 中迁移 .db 文件的分支没有被单测覆盖，导致 diff coverage 只有 67%。

修复建议:

1. 在 tests/metrics/test_prometheus_multiprocess_setup.py 增加 setup_dp_prometheus_dir(dp_id=0, move_existing=True) 测试，创建 .db 与非 .db 文件，断言 .db 被移动到 dp0/ 且环境变量指向 dp0。
2. 为 fastdeploy/engine/engine.py 和 fastdeploy/engine/common_engine.py 的 FD_ENABLE_INTERNAL_ADAPTER=True、data_parallel_size > 1 分支补充或扩展 mock 测试，断言 get_original_prom_dir() 与 setup_dp_prometheus_dir(i/0, base_prom_dir, move_existing=True) 被调用。

关联变更: fastdeploy/engine/common_engine.py:2777,2783; fastdeploy/engine/engine.py:867,905-906,909-910; fastdeploy/metrics/prometheus_multiprocess_setup.py:68-72

🔴 Approval — 需要 Approval（置信度: 高）

该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。

[PaddlePaddle-bot]

🤖 Paddle-CI-Agent | pr_review | 2026-07-06 16:21:53

📋 Review 摘要

PR 概述：为单机多 DP 服务拆分 Prometheus multiprocess 目录，避免多个 DP rank 共享 .db 文件。
变更范围：Engine DP 子进程启动、OpenAI multi API server、Prometheus multiprocess 初始化与测试。
影响面 Tag：[Engine] [APIServer]

问题

级别	文件	概述
🔴 Bug	fastdeploy/engine/engine.py:910	单 server 多 DP 最终只从 dp0 目录采集 metrics，DP1+ 指标会被遗漏
🔴 Bug	fastdeploy/engine/common_engine.py:2783	EngineService 同类启动路径也只暴露 dp0 metrics

📝 PR 规范检查

符合规范。

总体评价

目录隔离方向合理，但采集端没有同步支持多 DP 子目录聚合。当前实现会让单 server 多 DP 场景的 /metrics 丢失非 DP0 rank 指标，需要修复后再合入。