[None][perf] Add selectable SafeTensors weight-load policies

[chienchunhung]

What changed

Adds an experimental native TRT-LLM WeightLoadPlan for selecting the SafeTensors checkpoint-loading policy. The default ordered plan is:

direct_rank_read
shared_host_producer
gpu_broadcast
legacy_fallback

Configure it with TRTLLM_HF_WEIGHT_LOAD_PLAN or HfWeightLoader(weight_load_plan=...). A single policy is strict; a comma-separated string or tuple is an ordered fallback plan.

Policy	Status	Behavior
direct_rank_read	Implemented, primary	Local ranks cooperatively stripe checkpoint byte ranges into the OS page cache, then each rank uses the existing mmap/tensor materialization path.
shared_host_producer	Implemented baseline	One local producer stages the checkpoint into the page cache and all local ranks consume it after synchronization. This is a benchmarkable baseline, not yet Yijin's pinned shared-memory producer/consumer pipeline.
gpu_broadcast	Reserved	Recognized and preflighted, but unavailable until TRT-LLM has a rank-aware final-weight materialization interface and topology-aware CUDA/NCCL fan-out.
legacy_fallback	Implemented	Existing per-rank loader behavior. Required for unsupported checkpoint formats and used when earlier policies do not qualify.

For eligible dense HF/AUTO SafeTensors checkpoints, the default chooses direct_rank_read. If the existing raw-weight cache is enabled and no plan is explicitly supplied, legacy behavior is preserved. Explicit cooperative policies take precedence and warn that the cache is bypassed.

Why

Cold starts on large checkpoints can be storage-bound. The new plan makes the I/O policy explicit and lets us compare:

• rank-aware parallel reads,
• a shared local producer,
• future GPU-topology-aware fan-out, and
• the existing loader,

without coupling checkpoint source selection (local/HF cache/MX/GMS) to transport and placement policy.

Distributed safety

The cooperative paths add:

• exact checkpoint manifest and format consensus across participating ranks,
• active communicator size and plan consensus,
• node-local communicators derived from the active MPI communicator,
• local backing-file identity checks before cooperative reads,
• coordinated prefetch and mmap failures before barriers, and
• collective-free fallback for rank-local MX/GMS/Mistral paths.

The implementation uses regular buffered pread to warm the OS page cache. It does not claim O_DIRECT, GDS, or direct final GPU placement.

What gpu_broadcast is intended to mean

A producer per node or NVLink/NVSwitch island reads bounded chunks, copies them to a producer GPU once, and distributes rank-ready weights over CUDA P2P/NCCL. Replicated tensors can use broadcast; TP/EP-sharded tensors require scatter or grouped send/receive, and PP ranks receive only their owned layers. Whether this beats parallel per-rank H2D depends on topology and shard layout, so this PR reserves the policy without silently routing it through legacy I/O.

Validation

• Focused checkpoint-loader tests: 58 passed
• py_compile: passed
• git diff --check: passed
• Selected pre-commit suite (format, lint, spelling, test-list validation, DCO): passed

Real multi-rank MPI validation and cold-cache storage benchmarks remain required before this leaves draft status.