Skip to content

Fix closest-CPU-NUMA detection regression for GPU executors#304

Closed
nileshnegi wants to merge 1 commit into
candidatefrom
users/nileshnegi/fix/numa-mapping
Closed

Fix closest-CPU-NUMA detection regression for GPU executors#304
nileshnegi wants to merge 1 commit into
candidatefrom
users/nileshnegi/fix/numa-mapping

Conversation

@nileshnegi
Copy link
Copy Markdown
Collaborator

Motivation

Restore the Linux-NUMA-indexed cpuAgents[] table that the GPU-to-closest-CPU lookup in CollectTopology() depends on.

Technical Details

After #303, CPU agents were enumerated via hsa_iterate_agents + HSA_AGENT_INFO_NODE and stored at the HSA/KFD topology node id, which is not guaranteed to equal the Linux NUMA index (e.g., on systems where GPU agents are enumerated before CPU agents in the KFD topology). When the two diverged, cpuAgents[i] stayed default-initialized, the closestCpuAgent lookup never matched, and GetClosestCpuNumaToGpu() returned -1 for every GPU. That broke the topology display (all GPUs reported "no closest NUMA") and produced malformed transfer descriptors like "R0C-1" in any preset that consumes the result.

Switch back to the allocation-based enumeration that preserves cpuAgents[i] == "agent owning Linux NUMA i" while still tolerating NUMA nodes restricted by the process's cpuset / mempolicy: AllocateMemory(MEM_CPU) already rejects such nodes since #303, so we skip them on ERR_FATAL and leave the slot zero-initialized. Restricted NUMAs cannot service transfers anyway, so a zero-handle entry is the correct sentinel and harmless to the handle comparison performed by the closest-CPU lookup.

Also, drop the explicit hsa_init() / hsa_shut_down() around the iteration: HIP already manages HSA lifetime for this process, and tearing it down between topology collection and the rest of the run is unnecessary.

Test Plan

Test Result

Verified on 8 x gfx950, 2 NUMA nodes:

Submission Checklist

Restore the Linux-NUMA-indexed cpuAgents[] table that the GPU-to-closest-CPU
lookup in CollectTopology() depends on. After #303, CPU agents were
enumerated via hsa_iterate_agents + HSA_AGENT_INFO_NODE and stored at the
HSA/KFD topology node id, which is not guaranteed to equal the Linux NUMA
index (e.g. on systems where GPU agents are enumerated before CPU agents in
the KFD topology). When the two diverged, cpuAgents[i] stayed
default-initialised, the closestCpuAgent lookup never matched, and
GetClosestCpuNumaToGpu() returned -1 for every GPU. That broke the topology
display (all GPUs reported with no closest NUMA) and produced malformed
transfer descriptors like "R0C-1" in any preset that consumes this info.

Switch back to the allocation-based enumeration that preserves
cpuAgents[i] == "agent owning Linux NUMA i" while still tolerating NUMA
nodes restricted by the process's cpuset / mempolicy: AllocateMemory(MEM_CPU)
already rejects such nodes since #303, so we skip them on ERR_FATAL and
leave the slot zero-initialised. Restricted NUMAs cannot service transfers
anyway, so a zero-handle entry is the correct sentinel and harmless to the
handle comparison performed by the closest-CPU lookup.

Also drop the explicit hsa_init() / hsa_shut_down() around the iteration:
HIP already manages HSA lifetime for this process and tearing it down
between topology collection and the rest of the run is unnecessary.

Verified on 8 x gfx950, 2 NUMA nodes:
- ./TransferBench now prints the expected "NUMA 00 -> GPUs 0 1 2 3" /
  "NUMA 01 -> GPUs 4 5 6 7" mapping that pre-#303 (e8edacf) produced.

Co-authored-by: Claude <claude@anthropic.com>
Copilot AI review requested due to automatic review settings May 19, 2026 06:26
@nileshnegi nileshnegi requested a review from a team as a code owner May 19, 2026 06:26
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Restores the pre-#303 behavior where cpuAgents[i] is indexed by Linux NUMA node, fixing a regression where GPU-to-closest-CPU lookups returned -1 because HSA/KFD node ids did not always match Linux NUMA indices.

Changes:

  • Replace hsa_iterate_agents + HSA_AGENT_INFO_NODE enumeration with allocation-based discovery using AllocateMemory({MEM_CPU, i}, ...) and hsa_amd_pointer_info::agentOwner.
  • Skip NUMA nodes restricted by cpuset/mempolicy (leaving zero-initialized sentinel) instead of aborting.
  • Drop explicit hsa_init()/hsa_shut_down() around the iteration since HIP manages HSA lifetime.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@nileshnegi
Copy link
Copy Markdown
Collaborator Author

Fixed in #305

@nileshnegi nileshnegi closed this May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants