Skip to content

Commit bb8ea8a

Browse files
authored
feat(deps): multi-version mangling — Level 1 fallback for cross-major coexistence (#19)
When SemVer merge fails (typically two non-overlapping pins like =0.0.1 ⨯ =0.0.2 across two consumers of the same package), instead of hard-erroring the resolver now stages the secondary version's source under `target/.mangled/<consumer>/...` and rewrites its module declarations to a mangled name (`<X>__v<M>_<m>_<p>__mcpp`). The direct consumer's source is staged + rewritten too so its `import X;` points at the mangled secondary BMI. Both BMIs coexist in the build graph at distinct module names; C++23 module attachment isolates their internal symbols, so no namespace mangling is needed. Touches: - `src/pm/mangle.cppm` (new) — pure helper: name format + a regex-free source rewriter that handles `(export )?module X;`, `(export )?module X:Y;`, `(export )?import X;`, `import X:Y;`. Bare partition imports (`import :Y;`) and the global module fragment (`module ;`) are left intact. 11 new unit tests in `tests/unit/test_mangle.cpp`. - `src/cli.cppm` — resolver conflict branch: when `try_merge_semver` fails, run mangling instead of returning. WorkItem gains a `consumerDepIndex` field so the fallback can locate the consumer manifest for staging. Resolved record uses the mangled module name as `[package].name` so the modgraph validator's "module must be prefixed by package name" check passes. - `src/build/plan.cppm` + `ninja_backend.cppm` — when two compile units share a basename across packages (e.g. cmdline 0.0.1 and staged-mangled 0.0.2 both have `parse.cppm`), the colliding object paths get a per-package subdirectory (`obj/<sanitized-pkg>/<file>`). `.ddi` placement follows. Single-package builds keep the original `obj/<file>` layout so cache hashes don't churn. - `tests/e2e/33_multi_version_mangling.sh` (new) — Y-shape graph: libA pinned to cmdline 0.0.1, libB pinned to cmdline 0.0.2; the build succeeds with both BMIs (`mcpplibs.cmdline.gcm` + `mcpplibs.cmdline__v0_0_2__mcpp.gcm`) on disk and the runtime output matches. - `tests/e2e/32_semver_merge.sh` — drop the previously-irreconcilable case; that's now Level 1's territory and is covered by test 33. MVP scope (clear errors out of bounds): - Conflicting consumer must be a dep, not the main package. - Secondary version must be a leaf (no own transitive deps). Both surfaces are documented in the error message so users know which limit they hit and have a workaround (pin one version). CHANGELOG 0.0.3 entry: covers all three resolution levels (transitive walker / SemVer merge / mangling) and the obj-path-namespacing side-fix to the build backend.
1 parent 32d6847 commit bb8ea8a

8 files changed

Lines changed: 699 additions & 81 deletions

File tree

CHANGELOG.md

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
66
## [Unreleased] — 0.0.3
77

8-
依赖解析体系的两步演进:0.0.2 release tag 之后合入的 transitive walker
9-
之上,这一版叠加 SemVer 合并;后续 PR 还会补上多版本 mangling 兜底。
8+
依赖解析体系的三步演进:0.0.2 release tag 之后合入 transitive walker,
9+
这一版补齐 SemVer 合并(Level 2)+ 多版本 mangling 兜底(Level 1)
1010

1111
### 新增
1212

@@ -20,10 +20,28 @@
2020
以不同版本约束声明时,resolver 会把两条原始约束 AND 合并(裸版本号视作
2121
`=X.Y.Z`),向 index 重新查询,选出同时满足两侧的具体版本。若该版本与
2222
此前已 pin 的不一致,旧的 manifest 与 `[build].include_dirs` 会被原地
23-
替换为新版本的内容,孩子依赖也按新 manifest 重新入队。完全无重叠
24-
(典型如 `=0.0.1``=0.0.2`)仍硬报错并提示后续 PR 会用多版本
25-
mangling 兜底。新增 e2e `32_semver_merge.sh` 覆盖兼容合并 + 不可调和
26-
两条主链路。
23+
替换为新版本的内容,孩子依赖也按新 manifest 重新入队。新增 e2e
24+
`32_semver_merge.sh` 覆盖兼容合并 + 不可调和两条主链路。
25+
26+
-**多版本 mangling 兜底(Level 1)** —— SemVer 合并失败时(典型如
27+
`=0.0.1``=0.0.2` 这种无重叠的 pin),resolver 不再硬报错,而是把次要
28+
版本的源码 stage 到 `target/.mangled/<consumer>/...` 下,通过正则改写
29+
`(export )?module X;` / `(export )?module X:Y;` / `(export )?import X;`
30+
把模块名替换成 `<X>__v<M>_<m>_<p>__mcpp` 形式,让两个 BMI 在同一构建图
31+
里以不同模块名共存(C++23 module attachment 帮我们做 ABI 隔离,无需额外
32+
namespace mangle)。直接 consumer 的源码也一并 stage + 改写,让它的
33+
`import` 指向 mangled 副本。MVP 范围:仅处理 dep-as-consumer + 叶子
34+
secondary 两种情形,主包做 consumer 或 secondary 还有自己的 transitive
35+
deps 时报清晰错误并建议显式 pin。新增 `src/pm/mangle.cppm`(纯改写
36+
helper + 11 个单元测试)和 e2e `33_multi_version_mangling.sh`
37+
38+
### 改进
39+
40+
- 🔧 **构建后端按需为多包做 obj 路径命名空间** —— `plan.cppm` 检测到
41+
跨包同名源文件(多版本 mangling 后两个 `parse.cppm` 同时存在的常见情形)
42+
时,自动把 `obj/<file>.o` 改为 `obj/<sanitized-pkg>/<file>.o`,`.ddi`
43+
扫描产物随之放在 object 同目录下。无碰撞时仍是原始 `obj/<file>.o`
44+
布局,不影响现有缓存命中。
2745

2846
第二个公开版本。新增 C 语言一等公民支持、xpkg 风格依赖命名空间、包管理子系统骨架重构,以及 lib-root 约定。
2947

src/build/ninja_backend.cppm

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -357,15 +357,19 @@ std::string emit_ninja_string(const BuildPlan& plan) {
357357

358358
if (dyndep) {
359359
// ── Phase 1: scan edges (one .ddi per TU). ──────────────────────
360-
// .ddi is placed beside the object: obj/<src>.ddi.
360+
// .ddi is placed beside the object so multi-version mangling can
361+
// namespace by package without producing two `build` rules with
362+
// the same `.ddi` output (plan.cppm switches `cu.object` from
363+
// `obj/<file>.o` to `obj/<pkg>/<file>.o` whenever a basename
364+
// collides across packages — `.ddi` follows that placement).
361365
// Skip .c files: they have no `import`s and don't need P1689 scan;
362366
// running them through cxx_scan would route them through g++ /
363367
// -fmodules which is exactly what C support is here to avoid.
364368
std::vector<std::string> ddi_paths;
365369
ddi_paths.reserve(plan.compileUnits.size());
366370
for (auto& cu : plan.compileUnits) {
367371
if (is_c_source(cu.source)) continue;
368-
auto ddi = (std::filesystem::path("obj")
372+
auto ddi = (cu.object.parent_path()
369373
/ cu.source.filename()).string() + ".ddi";
370374
ddi_paths.push_back(ddi);
371375
append(std::format("build {} : cxx_scan {}\n",

src/build/plan.cppm

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,12 +96,33 @@ BuildPlan make_plan(const mcpp::manifest::Manifest& manifest,
9696
plan.stdBmiPath = stdBmiPath;
9797
plan.stdObjectPath = stdObjectPath;
9898

99+
// 1a. Detect basename collisions across packages (multi-version mangling
100+
// stages a second copy of the same dep, so `parse.cppm` and friends
101+
// can show up twice). For colliding files we namespace the object
102+
// path by the unit's owning package so `obj/<file>.o` doesn't get
103+
// two `build` rules.
104+
std::map<std::string, int> basenameCount;
105+
for (auto idx : topoOrder) {
106+
basenameCount[object_filename_for(graph.units[idx].path)]++;
107+
}
108+
auto sanitize_pkg = [](const std::string& s) {
109+
std::string out; out.reserve(s.size());
110+
for (char c : s) out += (c == '.' ? '_' : c);
111+
return out;
112+
};
113+
99114
// 1. Compile units in topological order
100115
for (auto idx : topoOrder) {
101116
auto& u = graph.units[idx];
102117
CompileUnit cu;
103118
cu.source = u.path;
104-
cu.object = std::filesystem::path("obj") / object_filename_for(u.path);
119+
const auto fname = object_filename_for(u.path);
120+
if (basenameCount[fname] > 1 && !u.packageName.empty()) {
121+
cu.object = std::filesystem::path("obj")
122+
/ sanitize_pkg(u.packageName) / fname;
123+
} else {
124+
cu.object = std::filesystem::path("obj") / fname;
125+
}
105126
if (u.provides) {
106127
cu.providesModule = u.provides->logicalName;
107128
}

src/cli.cppm

Lines changed: 190 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ import mcpp.config;
3232
import mcpp.fetcher;
3333
import mcpp.pm.resolver; // PR-R4: extracted from cli.cppm
3434
import mcpp.pm.commands; // PR-R5: cmd_add / cmd_remove / cmd_update live here now
35+
import mcpp.pm.mangle; // Level 1 multi-version fallback (cross-major coexistence)
3536
import mcpp.ui;
3637
import mcpp.bmi_cache;
3738
import mcpp.dyndep;
@@ -1062,11 +1063,15 @@ prepare_build(bool print_fingerprint,
10621063
};
10631064
std::map<ResolvedKey, ResolvedRecord> resolved;
10641065

1066+
// Sentinel for "the consumer is the main package" (no dep_manifests entry).
1067+
constexpr std::size_t kMainConsumer = static_cast<std::size_t>(-1);
1068+
10651069
struct WorkItem {
10661070
std::string name; // dep map key as written
10671071
mcpp::manifest::DependencySpec spec; // copy (we may mutate version)
10681072
std::string requestedBy; // who asked for it
10691073
std::string originalConstraint; // spec.version BEFORE pinning (for SemVer merge)
1074+
std::size_t consumerDepIndex; // dep_manifests slot of who pushed this child; kMainConsumer for main
10701075
};
10711076
std::deque<WorkItem> worklist;
10721077

@@ -1212,15 +1217,76 @@ prepare_build(bool print_fingerprint,
12121217
}
12131218
};
12141219

1220+
// Stage a dep's source files into a fresh directory, rewriting their
1221+
// module / import declarations against `rename`. Used by the multi-
1222+
// version mangling fallback (Level 1) so two cross-major copies of
1223+
// the same package can coexist with distinct module names.
1224+
//
1225+
// Headers (referenced via `[build].include_dirs`) are NOT staged —
1226+
// those keep pointing at the original install dir via absolutized
1227+
// include paths.
1228+
auto stage_with_rewrite = [](const std::filesystem::path& srcRoot,
1229+
const std::filesystem::path& dstRoot,
1230+
const mcpp::manifest::Manifest& depManifest,
1231+
const std::map<std::string, std::string>& rename)
1232+
-> std::expected<void, std::string>
1233+
{
1234+
std::error_code ec;
1235+
std::filesystem::create_directories(dstRoot, ec);
1236+
if (ec) return std::unexpected(std::format(
1237+
"stage: cannot create '{}': {}", dstRoot.string(), ec.message()));
1238+
1239+
// Resolve the source globs against the original root, falling
1240+
// back to the convention default if the manifest didn't set any.
1241+
std::vector<std::string> globs = depManifest.modules.sources;
1242+
if (globs.empty()) {
1243+
globs = { "src/**/*.cppm", "src/**/*.cpp",
1244+
"src/**/*.cc", "src/**/*.c" };
1245+
}
1246+
std::set<std::filesystem::path> sourceFiles;
1247+
for (auto const& g : globs) {
1248+
for (auto& p : mcpp::modgraph::expand_glob(srcRoot, g)) {
1249+
sourceFiles.insert(p);
1250+
}
1251+
}
1252+
if (sourceFiles.empty()) {
1253+
return std::unexpected(std::format(
1254+
"stage: no source files found under '{}' (globs={})",
1255+
srcRoot.string(), globs.size()));
1256+
}
1257+
1258+
for (auto const& f : sourceFiles) {
1259+
auto rel = std::filesystem::relative(f, srcRoot, ec);
1260+
if (ec) return std::unexpected(std::format(
1261+
"stage: cannot relativize '{}': {}", f.string(), ec.message()));
1262+
auto dst = dstRoot / rel;
1263+
std::filesystem::create_directories(dst.parent_path(), ec);
1264+
1265+
std::ifstream is(f);
1266+
if (!is) return std::unexpected(std::format(
1267+
"stage: cannot read '{}'", f.string()));
1268+
std::stringstream buf; buf << is.rdbuf();
1269+
std::string content = buf.str();
1270+
1271+
std::string out = mcpp::pm::rewrite_module_decls(content, rename);
1272+
std::ofstream os(dst);
1273+
if (!os) return std::unexpected(std::format(
1274+
"stage: cannot write '{}'", dst.string()));
1275+
os << out;
1276+
}
1277+
return {};
1278+
};
1279+
12151280
// Seed the worklist from the main manifest. Dev-deps only when the
12161281
// caller wants them; they're never propagated transitively.
12171282
const std::string mainPkgLabel = m->package.name;
12181283
for (auto& [n, s] : m->dependencies) {
1219-
worklist.push_back({n, s, mainPkgLabel, s.version});
1284+
worklist.push_back({n, s, mainPkgLabel, s.version, kMainConsumer});
12201285
}
12211286
if (includeDevDeps) {
12221287
for (auto& [n, s] : m->devDependencies) {
1223-
worklist.push_back({n, s, mainPkgLabel + " (dev-dep)", s.version});
1288+
worklist.push_back({n, s, mainPkgLabel + " (dev-dep)",
1289+
s.version, kMainConsumer});
12241290
}
12251291
}
12261292

@@ -1274,21 +1340,124 @@ prepare_build(bool print_fingerprint,
12741340
item.originalConstraint,
12751341
fetcher);
12761342
if (!merged) {
1277-
return std::unexpected(std::format(
1278-
"dependency '{}{}{}' has irreconcilable versions in "
1279-
"the transitive graph:\n"
1280-
" '{}' (constraint '{}') requested by '{}'\n"
1281-
" '{}' (constraint '{}') requested by '{}'\n"
1282-
"SemVer merge: {}\n"
1283-
"C++ modules require a single global version of each "
1284-
"package; pick a version compatible with both "
1285-
"consumers, or ask one upstream to widen its dep "
1286-
"range. (cross-major fallback via multi-version "
1287-
"mangling is planned in a follow-up PR)",
1288-
key.ns, key.ns.empty() ? "" : ".", key.shortName,
1289-
it->second.version, it->second.constraint, it->second.requestedBy,
1290-
spec.version, item.originalConstraint, item.requestedBy,
1291-
merged.error()));
1343+
// Level 1 fallback: multi-version mangling. Two
1344+
// versions can't be reconciled by SemVer, but they
1345+
// can coexist in the same build if we mangle the
1346+
// secondary copy's module name and rewrite the one
1347+
// consumer that asked for it. The primary keeps its
1348+
// authored module name so consumers that don't care
1349+
// about the secondary see no churn.
1350+
//
1351+
// MVP scope (these limits surface as clear errors):
1352+
// * The conflicting consumer must be a dep, not
1353+
// the main package — main-package mangling
1354+
// would mean rewriting user-authored sources,
1355+
// which is too surprising for a fallback path.
1356+
// * The secondary version must be a leaf (no own
1357+
// transitive deps) — recursive mangling is
1358+
// deferred to a follow-up.
1359+
if (item.consumerDepIndex == kMainConsumer) {
1360+
return std::unexpected(std::format(
1361+
"dependency '{}{}{}' has irreconcilable versions:\n"
1362+
" '{}' (constraint '{}') requested by '{}'\n"
1363+
" '{}' (constraint '{}') requested by '{}'\n"
1364+
"SemVer merge: {}\n"
1365+
"Multi-version mangling can't help here — the conflict "
1366+
"involves the main package directly. Pin one version "
1367+
"explicitly in your mcpp.toml.",
1368+
key.ns, key.ns.empty() ? "" : ".", key.shortName,
1369+
it->second.version, it->second.constraint, it->second.requestedBy,
1370+
spec.version, item.originalConstraint, item.requestedBy,
1371+
merged.error()));
1372+
}
1373+
1374+
auto loaded = loadVersionDep(name, spec.version);
1375+
if (!loaded) return std::unexpected(loaded.error());
1376+
auto& [secondaryRoot, secondaryManifest] = *loaded;
1377+
1378+
if (!secondaryManifest.dependencies.empty()) {
1379+
return std::unexpected(std::format(
1380+
"dependency '{}{}{}' has irreconcilable versions:\n"
1381+
" '{}' requested by '{}'\n"
1382+
" '{}' requested by '{}'\n"
1383+
"Multi-version mangling fallback only handles leaf "
1384+
"secondaries in 0.0.3 — but the secondary v{} declares "
1385+
"its own dependencies, which would need recursive "
1386+
"mangling. Pin one version explicitly, or wait for "
1387+
"the recursive-mangling extension.",
1388+
key.ns, key.ns.empty() ? "" : ".", key.shortName,
1389+
it->second.version, it->second.requestedBy,
1390+
spec.version, item.requestedBy,
1391+
spec.version));
1392+
}
1393+
1394+
// Module names in the source files use the dep's full
1395+
// [package].name (e.g. "mcpplibs.cmdline"), not the
1396+
// namespaced-subtable shortName. Use that for the
1397+
// rename key so the rewriter actually matches what the
1398+
// .cppm sources declare.
1399+
const std::string moduleName = secondaryManifest.package.name;
1400+
std::string mangled =
1401+
mcpp::pm::mangle_name(moduleName, spec.version);
1402+
1403+
// Stage layout:
1404+
// <root>/target/.mangled/<consumerPkg>/<dep>__<version>/ ← rewritten secondary source
1405+
// <root>/target/.mangled/<consumerPkg>/__self__/ ← rewritten consumer source
1406+
auto& consumerManifest = *dep_manifests[item.consumerDepIndex];
1407+
auto consumerRoot = packages[item.consumerDepIndex + 1].root;
1408+
auto stageBase = *root / "target" / ".mangled"
1409+
/ consumerManifest.package.name;
1410+
auto secStage = stageBase
1411+
/ std::format("{}__{}", moduleName, spec.version);
1412+
auto consumerStage = stageBase / "__self__";
1413+
1414+
std::map<std::string, std::string> rename{ {moduleName, mangled} };
1415+
if (auto r = stage_with_rewrite(secondaryRoot, secStage,
1416+
secondaryManifest, rename); !r)
1417+
return std::unexpected(r.error());
1418+
if (auto r = stage_with_rewrite(consumerRoot, consumerStage,
1419+
consumerManifest, rename); !r)
1420+
return std::unexpected(r.error());
1421+
1422+
// Re-anchor the consumer's PackageRoot at its staged copy
1423+
// so the modgraph scanner picks up the rewritten imports.
1424+
packages[item.consumerDepIndex + 1].root = consumerStage;
1425+
1426+
// Record the staged secondary as a brand-new dep entry
1427+
// under its mangled name, so future encounters of this
1428+
// exact (ns, mangled) pair dedup cleanly. The original
1429+
// primary entry (it->second) is untouched.
1430+
auto stagedManifest = secondaryManifest;
1431+
// Update [package].name to the mangled module name so
1432+
// the modgraph validator (which checks "exported module
1433+
// must be prefixed by package name") accepts the
1434+
// rewritten sources.
1435+
stagedManifest.package.name = mangled;
1436+
// Absolutize secondary's include_dirs against its original
1437+
// install root so the staged copy still finds headers.
1438+
for (auto& inc : stagedManifest.buildConfig.includeDirs) {
1439+
if (inc.is_relative()) inc = secondaryRoot / inc;
1440+
}
1441+
1442+
dep_manifests.push_back(
1443+
std::make_unique<mcpp::manifest::Manifest>(std::move(stagedManifest)));
1444+
packages.push_back({secStage, *dep_manifests.back()});
1445+
auto added = propagateIncludeDirs(secStage, *dep_manifests.back());
1446+
1447+
ResolvedKey mangledKey{key.ns, mangled};
1448+
resolved[mangledKey] = ResolvedRecord{
1449+
.version = spec.version,
1450+
.constraint = item.originalConstraint,
1451+
.requestedBy = item.requestedBy,
1452+
.source = "version",
1453+
.depIndex = dep_manifests.size() - 1,
1454+
.includeDirsAdded = std::move(added),
1455+
};
1456+
1457+
mcpp::ui::info("Mangled",
1458+
std::format("{} v{} ↔ v{} → {} (cross-major fallback)",
1459+
moduleName, it->second.version, spec.version, mangled));
1460+
continue;
12921461
}
12931462

12941463
// Combine the constraint strings so future merges AND with
@@ -1365,7 +1534,8 @@ prepare_build(bool print_fingerprint,
13651534
for (auto& [child_name, child_spec] :
13661535
dep_manifests[it->second.depIndex]->dependencies) {
13671536
worklist.push_back({child_name, child_spec, newLabel,
1368-
child_spec.version});
1537+
child_spec.version,
1538+
it->second.depIndex});
13691539
}
13701540
continue;
13711541
}
@@ -1514,9 +1684,10 @@ prepare_build(bool print_fingerprint,
15141684
key.ns.empty() ? "" : ".",
15151685
key.shortName,
15161686
sourceKind == "version" ? spec.version : sourceKind);
1687+
const std::size_t selfIdx = dep_manifests.size() - 1;
15171688
for (auto& [child_name, child_spec] : dep_manifests.back()->dependencies) {
15181689
worklist.push_back({child_name, child_spec, thisDepLabel,
1519-
child_spec.version});
1690+
child_spec.version, selfIdx});
15201691
}
15211692
}
15221693

0 commit comments

Comments
 (0)