torchvulkan 0.2.0a2__tar.gz → 0.2.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (138) hide show
  1. torchvulkan-0.2.1/PKG-INFO +223 -0
  2. torchvulkan-0.2.1/README.md +206 -0
  3. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/benchmark/benchmark.py +111 -64
  4. torchvulkan-0.2.1/examples/conv.py +90 -0
  5. torchvulkan-0.2.1/examples/llm.py +142 -0
  6. torchvulkan-0.2.1/examples/mlp.py +65 -0
  7. torchvulkan-0.2.1/media/benchmark-cuda.png +0 -0
  8. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/backend.cpp +5 -0
  9. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/decompositions.cpp +399 -0
  10. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/binary.cpp +112 -0
  11. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/helpers.h +1 -1
  12. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/matmul.cpp +24 -0
  13. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/nllloss.cpp +30 -0
  14. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/nn.cpp +12 -0
  15. torchvulkan-0.2.1/src/api/ops/optim.cpp +277 -0
  16. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/pool.cpp +18 -0
  17. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/reduce.cpp +24 -0
  18. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/tensor/transfer.cpp +5 -5
  19. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/allocator.cpp +6 -1
  20. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/allocator.h +2 -1
  21. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/device_context.cpp +8 -0
  22. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/device_context.h +1 -0
  23. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/dispatch.cpp +10 -3
  24. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/memory.h +7 -1
  25. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/shader.cpp +1 -1
  26. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/torchvulkan/__init__.py +3 -0
  27. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/torchvulkan/_version.py +3 -3
  28. torchvulkan-0.2.0a2/PKG-INFO +0 -204
  29. torchvulkan-0.2.0a2/README.md +0 -187
  30. torchvulkan-0.2.0a2/examples/llm.py +0 -136
  31. torchvulkan-0.2.0a2/examples/mnist.py +0 -62
  32. torchvulkan-0.2.0a2/media/benchmark-cuda.png +0 -0
  33. torchvulkan-0.2.0a2/media/benchmark-mps.png +0 -0
  34. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.github/actions/setup-toolchain/action.yml +0 -0
  35. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.github/scripts/fetch-moltenvk-macos.sh +0 -0
  36. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.github/scripts/install-toolchain-linux.sh +0 -0
  37. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.github/workflows/release.yaml +0 -0
  38. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.github/workflows/test_torchvulkan.yml +0 -0
  39. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/.gitignore +0 -0
  40. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/CMakeLists.txt +0 -0
  41. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/LICENSE +0 -0
  42. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/extern/vma/vk_mem_alloc.h +0 -0
  43. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/extern/volk/volk.c +0 -0
  44. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/extern/volk/volk.h +0 -0
  45. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/licenses/MoltenVK-LICENSE +0 -0
  46. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/pyproject.toml +0 -0
  47. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/requirements.txt +0 -0
  48. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/scripts/build_shaders.py +0 -0
  49. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/scripts/config.py +0 -0
  50. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/scripts/pack_shaders.py +0 -0
  51. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/guard.cpp +0 -0
  52. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/hook.cpp +0 -0
  53. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/bitwise.cpp +0 -0
  54. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/compare.cpp +0 -0
  55. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/complex.cpp +0 -0
  56. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/conv.cpp +0 -0
  57. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/ctc.cpp +0 -0
  58. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/distributions.cpp +0 -0
  59. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/dropout.cpp +0 -0
  60. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/fft.cpp +0 -0
  61. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/fill.cpp +0 -0
  62. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/grid_sample.cpp +0 -0
  63. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/indexing.cpp +0 -0
  64. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/internal.h +0 -0
  65. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/linalg.cpp +0 -0
  66. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/random.cpp +0 -0
  67. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/sampling.cpp +0 -0
  68. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/scan.cpp +0 -0
  69. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/scatter.cpp +0 -0
  70. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/search.cpp +0 -0
  71. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/shape.cpp +0 -0
  72. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/sort.cpp +0 -0
  73. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/special.cpp +0 -0
  74. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/unary.cpp +0 -0
  75. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/ops/upsample.cpp +0 -0
  76. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/registration.cpp +0 -0
  77. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/tensor/allocation.cpp +0 -0
  78. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/tensor/cast.cpp +0 -0
  79. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/api/tensor/views.cpp +0 -0
  80. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/CMakeLists.txt +0 -0
  81. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/core/cast.slang +0 -0
  82. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/core/dtype.slang +0 -0
  83. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/core/indexing.slang +0 -0
  84. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/core/philox.slang +0 -0
  85. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/core/special_functions.slang +0 -0
  86. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/arg_reduce.slang +0 -0
  87. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/binary.slang +0 -0
  88. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/cast.slang +0 -0
  89. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/cholesky.slang +0 -0
  90. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/compare.slang +0 -0
  91. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/conv.slang +0 -0
  92. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/copy.slang +0 -0
  93. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/ctc.slang +0 -0
  94. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/distribution.slang +0 -0
  95. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/dropout.slang +0 -0
  96. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/eig.slang +0 -0
  97. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/embedding_backward.slang +0 -0
  98. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/fft.slang +0 -0
  99. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/fill.slang +0 -0
  100. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/grid_sample.slang +0 -0
  101. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/householder.slang +0 -0
  102. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/index_gather.slang +0 -0
  103. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/integer_binary.slang +0 -0
  104. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/jacobi.slang +0 -0
  105. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/ldl.slang +0 -0
  106. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/lu.slang +0 -0
  107. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/matmul_coop.slang +0 -0
  108. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/matmul_simd.slang +0 -0
  109. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/multinomial.slang +0 -0
  110. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/nllloss.slang +0 -0
  111. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/pool.slang +0 -0
  112. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/random.slang +0 -0
  113. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/reduce.slang +0 -0
  114. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/reduce_subgroup.slang +0 -0
  115. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/resample.slang +0 -0
  116. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/resample_u8.slang +0 -0
  117. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/scan.slang +0 -0
  118. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/scan_arg.slang +0 -0
  119. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/scatter.slang +0 -0
  120. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/searchsorted.slang +0 -0
  121. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/sort.slang +0 -0
  122. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/special.slang +0 -0
  123. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/triangular_solve.slang +0 -0
  124. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/unary.slang +0 -0
  125. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/unary_ops.slang +0 -0
  126. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/shaders/ops/where.slang +0 -0
  127. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/builders.h +0 -0
  128. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/cache.cpp +0 -0
  129. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/cache.h +0 -0
  130. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/dispatch.h +0 -0
  131. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/generator.cpp +0 -0
  132. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/generator.h +0 -0
  133. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/memory.cpp +0 -0
  134. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/shader.h +0 -0
  135. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/vulkan_context.cpp +0 -0
  136. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/src/vulkan/vulkan_context.h +0 -0
  137. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/tests/test_pytorch_ops.py +0 -0
  138. {torchvulkan-0.2.0a2 → torchvulkan-0.2.1}/tox.ini +0 -0
@@ -0,0 +1,223 @@
1
+ Metadata-Version: 2.4
2
+ Name: torchvulkan
3
+ Version: 0.2.1
4
+ Summary: Cross-Platform Vulkan Backend for PyTorch
5
+ Author: Zain Sharief
6
+ License-Expression: MIT
7
+ License-File: LICENSE
8
+ License-File: licenses/MoltenVK-LICENSE
9
+ Project-URL: Homepage, https://github.com/zainsharief/torchvulkan
10
+ Project-URL: Repository, https://github.com/zainsharief/torchvulkan
11
+ Project-URL: Issues, https://github.com/zainsharief/torchvulkan/issues
12
+ Requires-Dist: torch>=2.10.0
13
+ Provides-Extra: test
14
+ Requires-Dist: pytest; extra == "test"
15
+ Requires-Dist: tox; extra == "test"
16
+ Description-Content-Type: text/markdown
17
+
18
+ <h1 align="center">torchvulkan</h1>
19
+ <p align="center">
20
+ <i>A Cross-Platform Vulkan Backend for PyTorch</i>
21
+ </p>
22
+
23
+ <p align="center">
24
+ <a href="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml"><img alt="CI" src="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml/badge.svg"></a>
25
+ <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
26
+ <img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-blue">
27
+ <img alt="Status: experimental" src="https://img.shields.io/badge/status-experimental-orange">
28
+ </p>
29
+
30
+ `torchvulkan` registers a `vulkan` device with PyTorch and runs tensor operations as [Vulkan](https://www.vulkan.org/) compute shaders. Vulkan runs on GPUs from AMD, Intel, NVIDIA, and Apple (via MoltenVK), plus software rasterizers, so you get one vendor-neutral GPU backend across Linux, macOS, and Windows.
31
+
32
+ You use it like any other PyTorch device:
33
+
34
+ ```python
35
+ import torch
36
+ import torchvulkan # registers the 'vulkan' device on import
37
+
38
+ x = torch.randn(1024, 1024).to("vulkan")
39
+ y = torch.randn(1024, 1024).to("vulkan")
40
+
41
+ z = (x @ y).relu() # runs on the GPU via Vulkan compute shaders
42
+ print(z.sum().item()) # move a scalar back to the CPU
43
+ ```
44
+
45
+ > [!WARNING]
46
+ > **Experimental.** `torchvulkan` is an early-stage project. Most of PyTorch's operators run natively on the GPU; the few that don't fall back to the CPU. Expect breaking changes before `1.0`.
47
+
48
+ ## Table of contents
49
+
50
+ - [Installation](#installation)
51
+ - [Quickstart](#quickstart)
52
+ - [Examples](#examples)
53
+ - [Benchmarks](#benchmarks)
54
+ - [Python API](#python-api)
55
+ - [CPU fallback](#cpu-fallback)
56
+ - [Environment variables](#environment-variables)
57
+ - [Limitations](#limitations)
58
+ - [How it works](#how-it-works)
59
+
60
+ ## Installation
61
+
62
+ ```bash
63
+ pip install torchvulkan
64
+ ```
65
+
66
+ At runtime you need a **Vulkan 1.3 (or newer) driver**: your GPU's driver on Linux and Windows. On macOS the wheel bundles [MoltenVK](https://github.com/KhronosGroup/MoltenVK), so nothing else is needed; set `TORCHVULKAN_SYSTEM_VULKAN=1` to use the system Vulkan loader instead. Verify it worked:
67
+
68
+ ```python
69
+ import torchvulkan
70
+ print(torchvulkan.is_available(), torchvulkan.device_count())
71
+ ```
72
+
73
+ ## Quickstart
74
+
75
+ ```python
76
+ import torch
77
+ import torchvulkan
78
+
79
+ # Move tensors to the Vulkan device
80
+ a = torch.tensor([[1.0, 2.0], [3.0, 4.0]], device="vulkan")
81
+ b = torch.ones(2, 2, device="vulkan")
82
+
83
+ c = a @ b + 1.0 # matmul + broadcasted add
84
+ d = c.exp().sum(dim=0) # elementwise + reduction
85
+
86
+ print(d.cpu()) # bring the result back to the CPU
87
+ ```
88
+
89
+ `nn.Module`s move to the device the usual way:
90
+
91
+ ```python
92
+ model = torch.nn.Linear(128, 10).to("vulkan")
93
+ out = model(torch.randn(32, 128).to("vulkan"))
94
+ ```
95
+
96
+ ## Examples
97
+
98
+ Each example trains a model end to end on the Vulkan device: forward pass, loss, backward pass and optimizer step all run through the backend.
99
+
100
+ | Example | Model | Data |
101
+ | --- | --- | --- |
102
+ | [`examples/mlp.py`](examples/mlp.py) | A multi-layer perceptron with GELU and dropout, trained with AdamW | MNIST |
103
+ | [`examples/llm.py`](examples/llm.py) | A small modern decoder-only transformer: RMSNorm, rotary embeddings, grouped-query attention, a SwiGLU MLP and tied embeddings. Samples text when it's done | Its own source code, character by character |
104
+ | [`examples/resnet.py`](examples/resnet.py) | A ResNet with residual blocks, batch norm and global average pooling, trained with SGD and a one-cycle schedule | CIFAR-10 |
105
+
106
+ ```bash
107
+ pip install torchvision # for the MNIST and CIFAR-10 datasets
108
+ python examples/mlp.py
109
+ python examples/llm.py
110
+ python examples/resnet.py
111
+ ```
112
+
113
+ Training looks the same as on any other device:
114
+
115
+ ```python
116
+ device = torch.device("vulkan" if torchvulkan.is_available() else "cpu")
117
+ model = ResNet().to(device)
118
+ optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
119
+
120
+ for data, targets in train_loader:
121
+ data, targets = data.to(device), targets.to(device)
122
+ loss = criterion(model(data), targets)
123
+ optimizer.zero_grad()
124
+ loss.backward()
125
+ optimizer.step()
126
+ ```
127
+
128
+ ## Benchmarks
129
+
130
+ [`benchmark/benchmark.py`](benchmark/benchmark.py) times a combined forward + backward pass (`float16` on accelerators, `float32` on CPU) on every available backend, sweeping each model's size. Each backend runs in its own subprocess. The models are the ones from the [examples](#examples):
131
+
132
+ - `mlp`: sweeps the hidden width, up to ~141M parameters.
133
+ - `llm`: sweeps the model width (`n_embd`), up to ~26M parameters.
134
+ - `resnet`: sweeps the base channel width, up to ~6M parameters.
135
+
136
+ <p align="center">
137
+ <img alt="Forward + backward pass time vs. model size for the MLP, LLM and ResNet on an NVIDIA GPU: vulkan vs. CUDA and CPU" src="media/benchmark-cuda.png" width="100%">
138
+ </p>
139
+
140
+ <p align="center">
141
+ <i>Vulkan vs. CUDA vs. CPU.</i>
142
+ </p>
143
+
144
+ ```bash
145
+ pip install matplotlib
146
+ python benchmark/benchmark.py all # every model
147
+ ```
148
+
149
+ ## Python API
150
+
151
+ `torchvulkan` mirrors the `torch.cuda` module, and is also reachable as `torch.vulkan`:
152
+
153
+ ```python
154
+ import torchvulkan
155
+
156
+ # devices
157
+ torchvulkan.is_available() # -> bool: a usable Vulkan device was found
158
+ torchvulkan.device_count() # -> int: number of Vulkan devices (discrete GPUs first)
159
+ torchvulkan.current_device() # -> int: index of the current device
160
+ torchvulkan.set_device(1) # make "vulkan" mean "vulkan:1"
161
+ torchvulkan.get_device_name() # -> str
162
+ torchvulkan.get_device_properties() # -> name, total_memory, vendor_id, api_version, subgroup_size, ...
163
+ torchvulkan.synchronize() # block until queued GPU work has finished
164
+
165
+ # memory
166
+ torchvulkan.memory_allocated() # bytes held by tensors
167
+ torchvulkan.max_memory_allocated() # peak since the last reset_peak_memory_stats()
168
+ torchvulkan.memory_reserved() # bytes held by the allocator, including its cache
169
+ torchvulkan.memory_stats() # -> dict
170
+ torchvulkan.mem_get_info() # -> (free, total): what can be allocated given the budget
171
+ torchvulkan.empty_cache() # release cached device memory back to the driver
172
+
173
+ # random numbers (per-device)
174
+ torchvulkan.manual_seed(0) # also: manual_seed_all, seed, seed_all, initial_seed
175
+ torchvulkan.get_rng_state() # also: set_rng_state, get_rng_state_all, set_rng_state_all
176
+
177
+ torchvulkan.is_bf16_supported() # -> bool: bfloat16 runs on the GPU
178
+ torchvulkan.__version__ # package version
179
+ ```
180
+
181
+ Importing `torchvulkan` registers the backend, after which the device is addressed as `"vulkan"` / `torch.device("vulkan")` throughout PyTorch.
182
+
183
+ ## CPU fallback
184
+
185
+ By default, any operation, dtype, or tensor shape the backend doesn't implement natively runs on the **CPU**, so models keep working despite partial coverage. The warning is printed once, for the first operation that falls back; call `torch.set_warn_always(True)` to see every one.
186
+
187
+ Set `TORCHVULKAN_STRICT=1` to turn those silent fallbacks into errors. This is useful for finding which operators a workload actually needs:
188
+
189
+ ```bash
190
+ TORCHVULKAN_STRICT=1 python your_script.py
191
+ ```
192
+
193
+ ## Environment variables
194
+
195
+ | Variable | Effect |
196
+ | --- | --- |
197
+ | `TORCHVULKAN_STRICT=1` | Raise an error instead of falling back to the CPU. |
198
+ | `TORCHVULKAN_SYSTEM_VULKAN=1` | macOS: use the system Vulkan loader instead of the bundled MoltenVK. |
199
+ | `TORCHVULKAN_VALIDATION=1` | Enable the Khronos validation layer, if it's installed. `=sync` also enables synchronization validation. On macOS this needs `TORCHVULKAN_SYSTEM_VULKAN=1`. |
200
+ | `TORCHVULKAN_PROFILE=1` | Print each compute pipeline's driver statistics (such as register use) when it's compiled, on drivers that support `VK_KHR_pipeline_executable_properties`. |
201
+ | `TORCHVULKAN_PIPELINE_CACHE=0` | Don't cache compiled pipelines between runs. By default they're cached in `%LOCALAPPDATA%\torchvulkan` on Windows and `$XDG_CACHE_HOME/torchvulkan` (or `~/.cache/torchvulkan`) elsewhere. |
202
+ | `TORCHVULKAN_CACHE_DIR` | Directory for the pipeline cache. |
203
+
204
+ ## Limitations
205
+
206
+ - `torch.compile` with the default Inductor backend doesn't support the `vulkan` device. Run eagerly, or use `torch.compile(model, backend="aot_eager")`.
207
+ - Optimizers created with `fused=True` (such as `torch.optim.AdamW(..., fused=True)`) run their update on the CPU. The default (foreach) implementation runs on the GPU.
208
+ - `float64` runs on the GPU only on devices with `shaderFloat64`; elsewhere it falls back to the CPU. Transcendental functions such as `exp`, `sin` and `tanh` are computed in `float32` precision for `float64` tensors.
209
+ - `bfloat16` runs on the GPU only on devices with `VK_KHR_shader_bfloat16`; elsewhere it falls back to the CPU.
210
+ - Complex tensors can live on the device and FFTs run on the GPU, but most complex arithmetic falls back to the CPU.
211
+ - `torch>=2.10.0` only. The extension is compiled against 2.13.0 but has testing support for >=2.10.0.
212
+ - APIs and coverage may change before `1.0.0`.
213
+
214
+ ## How it works
215
+
216
+ 1. Importing the package registers a PyTorch `PrivateUse1` backend and renames it to `vulkan`.
217
+ 2. Operations dispatched to a `vulkan` tensor are routed to native implementations that record and submit **Vulkan compute shaders**. The shaders are written in [Slang](https://github.com/shader-slang/slang), compiled to SPIR-V at build time, and embedded into the extension.
218
+ 3. Device memory is managed with [VMA](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator); Vulkan entry points are loaded with [volk](https://github.com/zeux/volk).
219
+ 4. Anything without a native implementation is handled by [CPU fallback](#cpu-fallback).
220
+
221
+
222
+ > [!NOTE]
223
+ > **Disclaimer:** `torchvulkan` is an independent open-source project and is not affiliated with or endorsed by PyTorch or the Linux Foundation.
@@ -0,0 +1,206 @@
1
+ <h1 align="center">torchvulkan</h1>
2
+ <p align="center">
3
+ <i>A Cross-Platform Vulkan Backend for PyTorch</i>
4
+ </p>
5
+
6
+ <p align="center">
7
+ <a href="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml"><img alt="CI" src="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml/badge.svg"></a>
8
+ <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
9
+ <img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-blue">
10
+ <img alt="Status: experimental" src="https://img.shields.io/badge/status-experimental-orange">
11
+ </p>
12
+
13
+ `torchvulkan` registers a `vulkan` device with PyTorch and runs tensor operations as [Vulkan](https://www.vulkan.org/) compute shaders. Vulkan runs on GPUs from AMD, Intel, NVIDIA, and Apple (via MoltenVK), plus software rasterizers, so you get one vendor-neutral GPU backend across Linux, macOS, and Windows.
14
+
15
+ You use it like any other PyTorch device:
16
+
17
+ ```python
18
+ import torch
19
+ import torchvulkan # registers the 'vulkan' device on import
20
+
21
+ x = torch.randn(1024, 1024).to("vulkan")
22
+ y = torch.randn(1024, 1024).to("vulkan")
23
+
24
+ z = (x @ y).relu() # runs on the GPU via Vulkan compute shaders
25
+ print(z.sum().item()) # move a scalar back to the CPU
26
+ ```
27
+
28
+ > [!WARNING]
29
+ > **Experimental.** `torchvulkan` is an early-stage project. Most of PyTorch's operators run natively on the GPU; the few that don't fall back to the CPU. Expect breaking changes before `1.0`.
30
+
31
+ ## Table of contents
32
+
33
+ - [Installation](#installation)
34
+ - [Quickstart](#quickstart)
35
+ - [Examples](#examples)
36
+ - [Benchmarks](#benchmarks)
37
+ - [Python API](#python-api)
38
+ - [CPU fallback](#cpu-fallback)
39
+ - [Environment variables](#environment-variables)
40
+ - [Limitations](#limitations)
41
+ - [How it works](#how-it-works)
42
+
43
+ ## Installation
44
+
45
+ ```bash
46
+ pip install torchvulkan
47
+ ```
48
+
49
+ At runtime you need a **Vulkan 1.3 (or newer) driver**: your GPU's driver on Linux and Windows. On macOS the wheel bundles [MoltenVK](https://github.com/KhronosGroup/MoltenVK), so nothing else is needed; set `TORCHVULKAN_SYSTEM_VULKAN=1` to use the system Vulkan loader instead. Verify it worked:
50
+
51
+ ```python
52
+ import torchvulkan
53
+ print(torchvulkan.is_available(), torchvulkan.device_count())
54
+ ```
55
+
56
+ ## Quickstart
57
+
58
+ ```python
59
+ import torch
60
+ import torchvulkan
61
+
62
+ # Move tensors to the Vulkan device
63
+ a = torch.tensor([[1.0, 2.0], [3.0, 4.0]], device="vulkan")
64
+ b = torch.ones(2, 2, device="vulkan")
65
+
66
+ c = a @ b + 1.0 # matmul + broadcasted add
67
+ d = c.exp().sum(dim=0) # elementwise + reduction
68
+
69
+ print(d.cpu()) # bring the result back to the CPU
70
+ ```
71
+
72
+ `nn.Module`s move to the device the usual way:
73
+
74
+ ```python
75
+ model = torch.nn.Linear(128, 10).to("vulkan")
76
+ out = model(torch.randn(32, 128).to("vulkan"))
77
+ ```
78
+
79
+ ## Examples
80
+
81
+ Each example trains a model end to end on the Vulkan device: forward pass, loss, backward pass and optimizer step all run through the backend.
82
+
83
+ | Example | Model | Data |
84
+ | --- | --- | --- |
85
+ | [`examples/mlp.py`](examples/mlp.py) | A multi-layer perceptron with GELU and dropout, trained with AdamW | MNIST |
86
+ | [`examples/llm.py`](examples/llm.py) | A small modern decoder-only transformer: RMSNorm, rotary embeddings, grouped-query attention, a SwiGLU MLP and tied embeddings. Samples text when it's done | Its own source code, character by character |
87
+ | [`examples/resnet.py`](examples/resnet.py) | A ResNet with residual blocks, batch norm and global average pooling, trained with SGD and a one-cycle schedule | CIFAR-10 |
88
+
89
+ ```bash
90
+ pip install torchvision # for the MNIST and CIFAR-10 datasets
91
+ python examples/mlp.py
92
+ python examples/llm.py
93
+ python examples/resnet.py
94
+ ```
95
+
96
+ Training looks the same as on any other device:
97
+
98
+ ```python
99
+ device = torch.device("vulkan" if torchvulkan.is_available() else "cpu")
100
+ model = ResNet().to(device)
101
+ optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
102
+
103
+ for data, targets in train_loader:
104
+ data, targets = data.to(device), targets.to(device)
105
+ loss = criterion(model(data), targets)
106
+ optimizer.zero_grad()
107
+ loss.backward()
108
+ optimizer.step()
109
+ ```
110
+
111
+ ## Benchmarks
112
+
113
+ [`benchmark/benchmark.py`](benchmark/benchmark.py) times a combined forward + backward pass (`float16` on accelerators, `float32` on CPU) on every available backend, sweeping each model's size. Each backend runs in its own subprocess. The models are the ones from the [examples](#examples):
114
+
115
+ - `mlp`: sweeps the hidden width, up to ~141M parameters.
116
+ - `llm`: sweeps the model width (`n_embd`), up to ~26M parameters.
117
+ - `resnet`: sweeps the base channel width, up to ~6M parameters.
118
+
119
+ <p align="center">
120
+ <img alt="Forward + backward pass time vs. model size for the MLP, LLM and ResNet on an NVIDIA GPU: vulkan vs. CUDA and CPU" src="media/benchmark-cuda.png" width="100%">
121
+ </p>
122
+
123
+ <p align="center">
124
+ <i>Vulkan vs. CUDA vs. CPU.</i>
125
+ </p>
126
+
127
+ ```bash
128
+ pip install matplotlib
129
+ python benchmark/benchmark.py all # every model
130
+ ```
131
+
132
+ ## Python API
133
+
134
+ `torchvulkan` mirrors the `torch.cuda` module, and is also reachable as `torch.vulkan`:
135
+
136
+ ```python
137
+ import torchvulkan
138
+
139
+ # devices
140
+ torchvulkan.is_available() # -> bool: a usable Vulkan device was found
141
+ torchvulkan.device_count() # -> int: number of Vulkan devices (discrete GPUs first)
142
+ torchvulkan.current_device() # -> int: index of the current device
143
+ torchvulkan.set_device(1) # make "vulkan" mean "vulkan:1"
144
+ torchvulkan.get_device_name() # -> str
145
+ torchvulkan.get_device_properties() # -> name, total_memory, vendor_id, api_version, subgroup_size, ...
146
+ torchvulkan.synchronize() # block until queued GPU work has finished
147
+
148
+ # memory
149
+ torchvulkan.memory_allocated() # bytes held by tensors
150
+ torchvulkan.max_memory_allocated() # peak since the last reset_peak_memory_stats()
151
+ torchvulkan.memory_reserved() # bytes held by the allocator, including its cache
152
+ torchvulkan.memory_stats() # -> dict
153
+ torchvulkan.mem_get_info() # -> (free, total): what can be allocated given the budget
154
+ torchvulkan.empty_cache() # release cached device memory back to the driver
155
+
156
+ # random numbers (per-device)
157
+ torchvulkan.manual_seed(0) # also: manual_seed_all, seed, seed_all, initial_seed
158
+ torchvulkan.get_rng_state() # also: set_rng_state, get_rng_state_all, set_rng_state_all
159
+
160
+ torchvulkan.is_bf16_supported() # -> bool: bfloat16 runs on the GPU
161
+ torchvulkan.__version__ # package version
162
+ ```
163
+
164
+ Importing `torchvulkan` registers the backend, after which the device is addressed as `"vulkan"` / `torch.device("vulkan")` throughout PyTorch.
165
+
166
+ ## CPU fallback
167
+
168
+ By default, any operation, dtype, or tensor shape the backend doesn't implement natively runs on the **CPU**, so models keep working despite partial coverage. The warning is printed once, for the first operation that falls back; call `torch.set_warn_always(True)` to see every one.
169
+
170
+ Set `TORCHVULKAN_STRICT=1` to turn those silent fallbacks into errors. This is useful for finding which operators a workload actually needs:
171
+
172
+ ```bash
173
+ TORCHVULKAN_STRICT=1 python your_script.py
174
+ ```
175
+
176
+ ## Environment variables
177
+
178
+ | Variable | Effect |
179
+ | --- | --- |
180
+ | `TORCHVULKAN_STRICT=1` | Raise an error instead of falling back to the CPU. |
181
+ | `TORCHVULKAN_SYSTEM_VULKAN=1` | macOS: use the system Vulkan loader instead of the bundled MoltenVK. |
182
+ | `TORCHVULKAN_VALIDATION=1` | Enable the Khronos validation layer, if it's installed. `=sync` also enables synchronization validation. On macOS this needs `TORCHVULKAN_SYSTEM_VULKAN=1`. |
183
+ | `TORCHVULKAN_PROFILE=1` | Print each compute pipeline's driver statistics (such as register use) when it's compiled, on drivers that support `VK_KHR_pipeline_executable_properties`. |
184
+ | `TORCHVULKAN_PIPELINE_CACHE=0` | Don't cache compiled pipelines between runs. By default they're cached in `%LOCALAPPDATA%\torchvulkan` on Windows and `$XDG_CACHE_HOME/torchvulkan` (or `~/.cache/torchvulkan`) elsewhere. |
185
+ | `TORCHVULKAN_CACHE_DIR` | Directory for the pipeline cache. |
186
+
187
+ ## Limitations
188
+
189
+ - `torch.compile` with the default Inductor backend doesn't support the `vulkan` device. Run eagerly, or use `torch.compile(model, backend="aot_eager")`.
190
+ - Optimizers created with `fused=True` (such as `torch.optim.AdamW(..., fused=True)`) run their update on the CPU. The default (foreach) implementation runs on the GPU.
191
+ - `float64` runs on the GPU only on devices with `shaderFloat64`; elsewhere it falls back to the CPU. Transcendental functions such as `exp`, `sin` and `tanh` are computed in `float32` precision for `float64` tensors.
192
+ - `bfloat16` runs on the GPU only on devices with `VK_KHR_shader_bfloat16`; elsewhere it falls back to the CPU.
193
+ - Complex tensors can live on the device and FFTs run on the GPU, but most complex arithmetic falls back to the CPU.
194
+ - `torch>=2.10.0` only. The extension is compiled against 2.13.0 but has testing support for >=2.10.0.
195
+ - APIs and coverage may change before `1.0.0`.
196
+
197
+ ## How it works
198
+
199
+ 1. Importing the package registers a PyTorch `PrivateUse1` backend and renames it to `vulkan`.
200
+ 2. Operations dispatched to a `vulkan` tensor are routed to native implementations that record and submit **Vulkan compute shaders**. The shaders are written in [Slang](https://github.com/shader-slang/slang), compiled to SPIR-V at build time, and embedded into the extension.
201
+ 3. Device memory is managed with [VMA](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator); Vulkan entry points are loaded with [volk](https://github.com/zeux/volk).
202
+ 4. Anything without a native implementation is handled by [CPU fallback](#cpu-fallback).
203
+
204
+
205
+ > [!NOTE]
206
+ > **Disclaimer:** `torchvulkan` is an independent open-source project and is not affiliated with or endorsed by PyTorch or the Linux Foundation.