[ET Device Support] CudaAllocator: device memory allocator for CUDA backend

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
• [ET Device Support] Extract shared device test utilities to reduce redundancy #18762
• [ET Device Support] Add device tensor helper functions to TensorPtr API #18761
• [ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
• [ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
• [ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
• [ET Device Support] Define AOT device copy ops registry #18728
• -> [ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
• [ET Device Support] Module: allocate device memory for planned buffers #18476
• [ET Device Support] MemoryManager: add per-buffer device metadata #18475
• [ET Device Support] MethodMeta: expose per-buffer device placement API #18474
• [ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
• [ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
• [ET Device Support] Device-aware memory planning: separate buffers per device type #18375
• [ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
• [ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
• [ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
• [ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
• [ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
• [executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Implement CudaAllocator, a concrete DeviceAllocator using cudaMalloc/ cudaFree/cudaMemcpy for CUDA device memory management. The allocator is automatically registered with the DeviceAllocatorRegistry when the CUDA backend library is linked, ensuring DeviceMemoryBuffer::create(CUDA) works transparently.

Differential Revision: D98014184

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18477

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 87 New Failures, 1 Unrelated Failure

As of commit da3d587 with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

• Build Presets / apple (pybind) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
• Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Build Presets / windows (pybind) / build (gh)
  Process completed with exit code 1.
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t 357bb210ee8ac79f688412064d12b843611bbfad4b77b6ee0c9e840536bb11ec /exec failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c3237b16daef035bdafd4fa0ed1b2b704dc34cfdf1cb872805d6b974d80fb49 /exec failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
  RuntimeError: Command docker exec -t 444ecfda2629c33dade863c5d61674d4b74d5a4a610ae751390bc05ae67f3aa0 /exec failed with exit code 1
• pull / android / build-android (gh)
  Process completed with exit code 2.
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command docker exec -t 2a82e81a13631a7c71321af6c22c2e28e58785717855e362fb259637c40d0a23 /exec failed with exit code 2
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-lora-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-lora-multimethod-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t bac68423aa798cdec19a4f7df564adc2f9cd510bc2ab526acd296099c3e3a74b /exec failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t ed50660cc29de6de244fecbc71db6012c9efb8666d8c582e3d4191aaaee73ff9 /exec failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t b0cd06ec52abaa94d29b26e04acee85b2fe9191431f40cc30519f0cc652e6397 /exec failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 3beaf88e2b39390b914640a5bc520eef1eb93b5f025ab2c382cac90c48658a3a /exec failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t e8dda1b5b54d50796eb261fd50588848adef38bb923a42b83faaac82453312aa /exec failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t bbc642d6ecedef97e44d7ab41418c197bbf00eb56bf25475ec11149e01358a3b /exec failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t a4223ebcfffb89b7b8b44265c8027410e8a51968f4ff409ed98b364cb20591d2 /exec failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 41ca8b8ee6cb219cad91f66628ded7d47c8a5df3e0d1d88f173deac27ca41cfa /exec failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t e4df11bcde6049b2bba84153a9fb7f392fbf12d617859ff2a7663af11c7a9aaf /exec failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 9ff83f94817ed836cfa1e72b9f6f7a4c171b4270dfccb28aaccba812bb190c9d /exec failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t e8d854d0c17a857e976b51a327026a0eca96321a748480870651d60e301ddb3e /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t b3ca179adf1951b26069ee698815531456f06fd21a89627bb00b7adc3a6ec5eb /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t bc78df34ec8f9e8b7120388fdd645171c0c4c24778831c6f62cf6aaabbc8b54f /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t 8d5bc94ea26be5d1e1037d768eed2b0f72f012a8badd9e9d40406dc5cf7a6aa1 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 0fc6717d4c999ca1d0d612ee7b1cdbe2400d9e51d2c25ad855b911fd258e867a /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 3d764e3f7a70de69db159855672c6a0e257946b6996509147877d084750367f1 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t cdbcd3000778e0fc78da7681ddd218ce0f5c6102d21eb8304cb9c6cbd105d517 /exec failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  test_mimi
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command docker exec -t 555411d69dafebf9474d11f18a7acd9ee4d799cd304fdd65e9692f83c167b921 /exec failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 67b3ff1e40fdb285014652e05e6db6788c71cba169aa6ae6293b1a125c084cfa /exec failed with exit code 1
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-vulkan-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 906c7ef76b81a6247d53312f57aaebeffd9abbb1b10cff0dddaa9e3beba7a38b /exec failed with exit code 1
• pull / test-vulkan-operators-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• pull / unittest / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 583a7241e96726e74fbafb6704d823f5108b13ab7215c79ba246b1fa225ab806 /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
  RuntimeError: Command docker exec -t 8a26d471d84537beaeb01988369fa6462ee6688037c862bbff59b8dbcde72cd3 /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 056602c0c675609fc5691cc97f02b30c2575095671b16e1e50aac1724e5895fa /exec failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 1863a94e63a6864673f39c31cc07d4fad751435abcb92286445f01c9bddec29c /exec failed with exit code 2
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  RuntimeError: Command docker exec -t 2570462ec5479823724d4e6ca232906199698208d46b24d33f8f7484673e1b5f /exec failed with exit code 2
• pull / unittest-wasm-bindings / linux-job (gh)
  RuntimeError: Command docker exec -t 4ff132ee071f6551aa4871f4835f3c3f19d3d42e2e1e9c5fb92aaae325212833 /exec failed with exit code 2
• Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, non-quantized) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, quantized-int4-tile-packed) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-tile-packed) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-tile-packed) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-small, non-quantized) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, non-quantized) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, quantized-int4-tile-packed) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / test-models-cuda / linux-job (gh)
  undefined reference to executorch::backends::cuda::CudaAllocator::allocate_async(unsigned long, signed char, CUstream_st*)'`
• Test CUDA Builds / unittest-cuda / linux-job (gh)
  undefined reference to executorch::backends::cuda::CudaAllocator::allocate_async(unsigned long, signed char, CUstream_st*)'`
• Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-metal) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-metal) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, quantized-int4-metal) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (openai, whisper-small, non-quantized) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test Metal Backend / export-model-metal-artifact (openai, whisper-small, quantized-int4-metal) / macos-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / test-models-linux (ic3, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
  The process '/usr/bin/git' failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[digantdesai]

Review automatically exported from Phabricator review in Meta.