torchvulkan 0.2.0a2__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.github/workflows/release.yaml +13 -10
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.github/workflows/test_torchvulkan.yml +6 -7
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/CMakeLists.txt +3 -0
- torchvulkan-0.2.2/PKG-INFO +224 -0
- torchvulkan-0.2.2/README.md +206 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/benchmark/benchmark.py +111 -64
- torchvulkan-0.2.2/examples/conv.py +90 -0
- torchvulkan-0.2.2/examples/llm.py +142 -0
- torchvulkan-0.2.2/examples/mlp.py +65 -0
- torchvulkan-0.2.2/media/benchmark-cuda.png +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/pyproject.toml +4 -4
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/backend.cpp +6 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/decompositions.cpp +405 -1
- torchvulkan-0.2.2/src/api/fallback.cpp +13 -0
- torchvulkan-0.2.2/src/api/fallback.h +11 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/binary.cpp +115 -2
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/bitwise.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/compare.cpp +4 -3
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/fill.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/grid_sample.cpp +1 -2
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/helpers.h +1 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/indexing.cpp +6 -5
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/linalg.cpp +5 -11
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/matmul.cpp +29 -4
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/nllloss.cpp +33 -2
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/nn.cpp +12 -0
- torchvulkan-0.2.2/src/api/ops/optim.cpp +277 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/pool.cpp +18 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/reduce.cpp +27 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/scan.cpp +3 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/scatter.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/search.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/sort.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/special.cpp +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/unary.cpp +3 -2
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/registration.cpp +0 -14
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/tensor/transfer.cpp +7 -6
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/core/dtype.slang +3 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/eig.slang +1 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/grid_sample.slang +120 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/householder.slang +2 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/allocator.cpp +50 -12
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/allocator.h +1 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/cache.cpp +34 -11
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/cache.h +11 -5
- torchvulkan-0.2.2/src/vulkan/device_context.cpp +96 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/device_context.h +6 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/dispatch.cpp +10 -3
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/memory.h +7 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/shader.cpp +1 -8
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/shader.h +0 -1
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/vulkan_context.cpp +1 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/vulkan_context.h +2 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/tests/test_pytorch_ops.py +12 -7
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/torchvulkan/__init__.py +18 -5
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/torchvulkan/_version.py +3 -3
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/tox.ini +1 -3
- torchvulkan-0.2.0a2/PKG-INFO +0 -204
- torchvulkan-0.2.0a2/README.md +0 -187
- torchvulkan-0.2.0a2/examples/llm.py +0 -136
- torchvulkan-0.2.0a2/examples/mnist.py +0 -62
- torchvulkan-0.2.0a2/media/benchmark-cuda.png +0 -0
- torchvulkan-0.2.0a2/media/benchmark-mps.png +0 -0
- torchvulkan-0.2.0a2/src/vulkan/device_context.cpp +0 -54
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.github/actions/setup-toolchain/action.yml +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.github/scripts/fetch-moltenvk-macos.sh +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.github/scripts/install-toolchain-linux.sh +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/.gitignore +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/LICENSE +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/extern/vma/vk_mem_alloc.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/extern/volk/volk.c +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/extern/volk/volk.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/licenses/MoltenVK-LICENSE +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/requirements.txt +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/scripts/build_shaders.py +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/scripts/config.py +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/scripts/pack_shaders.py +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/guard.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/hook.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/complex.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/conv.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/ctc.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/distributions.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/dropout.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/fft.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/internal.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/random.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/sampling.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/shape.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/ops/upsample.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/tensor/allocation.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/tensor/cast.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/api/tensor/views.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/CMakeLists.txt +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/core/cast.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/core/indexing.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/core/philox.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/core/special_functions.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/arg_reduce.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/binary.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/cast.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/cholesky.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/compare.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/conv.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/copy.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/ctc.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/distribution.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/dropout.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/embedding_backward.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/fft.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/fill.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/index_gather.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/integer_binary.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/jacobi.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/ldl.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/lu.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/matmul_coop.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/matmul_simd.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/multinomial.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/nllloss.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/pool.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/random.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/reduce.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/reduce_subgroup.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/resample.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/resample_u8.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/scan.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/scan_arg.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/scatter.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/searchsorted.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/sort.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/special.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/triangular_solve.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/unary.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/unary_ops.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/shaders/ops/where.slang +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/builders.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/dispatch.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/generator.cpp +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/generator.h +0 -0
- {torchvulkan-0.2.0a2 → torchvulkan-0.2.2}/src/vulkan/memory.cpp +0 -0
|
@@ -21,7 +21,7 @@ jobs:
|
|
|
21
21
|
|
|
22
22
|
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
|
|
23
23
|
with:
|
|
24
|
-
python-version: "3.
|
|
24
|
+
python-version: "3.15"
|
|
25
25
|
|
|
26
26
|
- name: Build sdist
|
|
27
27
|
run: |
|
|
@@ -34,18 +34,20 @@ jobs:
|
|
|
34
34
|
path: dist/*.tar.gz
|
|
35
35
|
|
|
36
36
|
build_wheels:
|
|
37
|
-
name: Wheels (${{ matrix.name }})
|
|
37
|
+
name: Wheels (${{ matrix.name }} / ${{ matrix.python }})
|
|
38
38
|
runs-on: ${{ matrix.os }}
|
|
39
39
|
strategy:
|
|
40
40
|
fail-fast: false
|
|
41
41
|
matrix:
|
|
42
|
+
os: [ubuntu-latest, macos-15, windows-latest]
|
|
43
|
+
python: [cp311, cp312, cp313, cp314, cp315]
|
|
42
44
|
include:
|
|
43
|
-
-
|
|
44
|
-
|
|
45
|
-
-
|
|
46
|
-
|
|
47
|
-
-
|
|
48
|
-
|
|
45
|
+
- os: ubuntu-latest
|
|
46
|
+
name: Linux x86_64
|
|
47
|
+
- os: macos-15
|
|
48
|
+
name: macOS Apple Silicon
|
|
49
|
+
- os: windows-latest
|
|
50
|
+
name: Windows x86_64
|
|
49
51
|
steps:
|
|
50
52
|
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
|
51
53
|
with:
|
|
@@ -67,13 +69,14 @@ jobs:
|
|
|
67
69
|
uses: ./.github/actions/setup-toolchain
|
|
68
70
|
|
|
69
71
|
- name: Build wheels
|
|
70
|
-
uses: pypa/cibuildwheel@
|
|
72
|
+
uses: pypa/cibuildwheel@f9b966cfadf4109556cadc5bb39ddb72d0ce664e # v4.3.5
|
|
71
73
|
env:
|
|
72
74
|
CIBW_BUILD_VERBOSITY: "1"
|
|
75
|
+
CIBW_BUILD: ${{ matrix.python }}-*
|
|
73
76
|
|
|
74
77
|
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
|
75
78
|
with:
|
|
76
|
-
name: dist-${{ matrix.os }}
|
|
79
|
+
name: dist-${{ matrix.os }}-${{ matrix.python }}
|
|
77
80
|
path: wheelhouse/*.whl
|
|
78
81
|
|
|
79
82
|
pypi-publish:
|
|
@@ -8,7 +8,7 @@ on:
|
|
|
8
8
|
|
|
9
9
|
jobs:
|
|
10
10
|
build-and-test:
|
|
11
|
-
name: ${{ matrix.os }} / Py${{ matrix.python-version }}
|
|
11
|
+
name: ${{ matrix.os }} / Py${{ matrix.python-version }}
|
|
12
12
|
runs-on: ${{ matrix.os }}
|
|
13
13
|
env:
|
|
14
14
|
SKBUILD_CMAKE_GENERATOR: Ninja
|
|
@@ -16,8 +16,7 @@ jobs:
|
|
|
16
16
|
fail-fast: false
|
|
17
17
|
matrix:
|
|
18
18
|
os: [ubuntu-latest, macos-15, windows-latest]
|
|
19
|
-
python-version: ["3.
|
|
20
|
-
torch-factor: ["torch210", "torch213"]
|
|
19
|
+
python-version: ["3.11", "3.13", "3.15"]
|
|
21
20
|
|
|
22
21
|
steps:
|
|
23
22
|
- name: Checkout Code
|
|
@@ -50,7 +49,7 @@ jobs:
|
|
|
50
49
|
DNNL_MAX_CPU_ISA: AVX2 # legacy name for the same oneDNN ISA cap (older bundled oneDNN)
|
|
51
50
|
LP_NATIVE_VECTOR_WIDTH: "256" # cap lavapipe's LLVM JIT to AVX2; AVX-512 codegen faults (0xc000001d) on GHA runners
|
|
52
51
|
run: |
|
|
53
|
-
tox -e py
|
|
52
|
+
tox -e py
|
|
54
53
|
|
|
55
54
|
validation:
|
|
56
55
|
name: validation layers (lavapipe, debug)
|
|
@@ -83,7 +82,7 @@ jobs:
|
|
|
83
82
|
run: |
|
|
84
83
|
python -m pip install --upgrade pip
|
|
85
84
|
pip install tox
|
|
86
|
-
tox -e py
|
|
85
|
+
tox -e py -- -k "test_correctness_add or test_correctness_mm or test_correctness_sum or test_correctness_sort or test_correctness_index or test_correctness_cumsum or test_gradients_nn_functional_linear"
|
|
87
86
|
|
|
88
87
|
sanitizers:
|
|
89
88
|
name: ASan + UBSan (lavapipe)
|
|
@@ -108,7 +107,7 @@ jobs:
|
|
|
108
107
|
SKBUILD_CMAKE_ARGS: "-DCMAKE_CXX_FLAGS=-fsanitize=address,undefined -fno-omit-frame-pointer;-DCMAKE_MODULE_LINKER_FLAGS=-fsanitize=address,undefined"
|
|
109
108
|
run: |
|
|
110
109
|
python -m pip install --upgrade pip
|
|
111
|
-
pip install "
|
|
110
|
+
pip install $(python -c 'import tomllib; print(" ".join(tomllib.load(open("pyproject.toml", "rb"))["build-system"]["requires"]))') pytest expecttest numpy ninja
|
|
112
111
|
pip install . --no-build-isolation -v
|
|
113
112
|
|
|
114
113
|
- name: Run tests with the sanitizer runtimes preloaded
|
|
@@ -123,7 +122,7 @@ jobs:
|
|
|
123
122
|
run: |
|
|
124
123
|
cd "$RUNNER_TEMP"
|
|
125
124
|
export LD_PRELOAD="$(gcc -print-file-name=libasan.so):$(gcc -print-file-name=libubsan.so)"
|
|
126
|
-
python -m pytest "$GITHUB_WORKSPACE/tests/test_pytorch_ops.py" -q -k "
|
|
125
|
+
python -m pytest "$GITHUB_WORKSPACE/tests/test_pytorch_ops.py" -q -k "test_correctness_add or test_correctness_mm or test_correctness_sort or test_correctness_index or test_correctness_sum"
|
|
127
126
|
|
|
128
127
|
testing-success:
|
|
129
128
|
name: testing success
|
|
@@ -30,6 +30,9 @@ target_compile_features(torchvulkan_lib PRIVATE cxx_std_20)
|
|
|
30
30
|
|
|
31
31
|
install(TARGETS torchvulkan_lib DESTINATION torchvulkan)
|
|
32
32
|
|
|
33
|
+
file(WRITE "${TORCHVULKAN_GENERATED_DIR}/_build_info.py" "torch_version = \"${Torch_VERSION}\"\n")
|
|
34
|
+
install(FILES "${TORCHVULKAN_GENERATED_DIR}/_build_info.py" DESTINATION torchvulkan)
|
|
35
|
+
|
|
33
36
|
if(APPLE)
|
|
34
37
|
find_library(TORCHVULKAN_MOLTENVK NAMES libMoltenVK.dylib)
|
|
35
38
|
if(TORCHVULKAN_MOLTENVK)
|
|
@@ -0,0 +1,224 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: torchvulkan
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: Cross-Platform Vulkan Backend for PyTorch
|
|
5
|
+
Author: Zain Sharief
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
License-File: licenses/MoltenVK-LICENSE
|
|
9
|
+
Project-URL: Homepage, https://github.com/zainsharief/torchvulkan
|
|
10
|
+
Project-URL: Repository, https://github.com/zainsharief/torchvulkan
|
|
11
|
+
Project-URL: Issues, https://github.com/zainsharief/torchvulkan/issues
|
|
12
|
+
Requires-Python: >=3.11
|
|
13
|
+
Requires-Dist: torch<2.16,>=2.15.0
|
|
14
|
+
Provides-Extra: test
|
|
15
|
+
Requires-Dist: pytest; extra == "test"
|
|
16
|
+
Requires-Dist: tox; extra == "test"
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
<h1 align="center">torchvulkan</h1>
|
|
20
|
+
<p align="center">
|
|
21
|
+
<i>A Cross-Platform Vulkan Backend for PyTorch</i>
|
|
22
|
+
</p>
|
|
23
|
+
|
|
24
|
+
<p align="center">
|
|
25
|
+
<a href="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml"><img alt="CI" src="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml/badge.svg"></a>
|
|
26
|
+
<a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
|
|
27
|
+
<img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-blue">
|
|
28
|
+
<img alt="Status: experimental" src="https://img.shields.io/badge/status-experimental-orange">
|
|
29
|
+
</p>
|
|
30
|
+
|
|
31
|
+
`torchvulkan` registers a `vulkan` device with PyTorch and runs tensor operations as [Vulkan](https://www.vulkan.org/) compute shaders. Vulkan runs on GPUs from AMD, Intel, NVIDIA, and Apple (via MoltenVK), plus software rasterizers, so you get one vendor-neutral GPU backend across Linux, macOS, and Windows.
|
|
32
|
+
|
|
33
|
+
You use it like any other PyTorch device:
|
|
34
|
+
|
|
35
|
+
```python
|
|
36
|
+
import torch
|
|
37
|
+
import torchvulkan # registers the 'vulkan' device on import
|
|
38
|
+
|
|
39
|
+
x = torch.randn(1024, 1024).to("vulkan")
|
|
40
|
+
y = torch.randn(1024, 1024).to("vulkan")
|
|
41
|
+
|
|
42
|
+
z = (x @ y).relu() # runs on the GPU via Vulkan compute shaders
|
|
43
|
+
print(z.sum().item()) # move a scalar back to the CPU
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
> [!WARNING]
|
|
47
|
+
> **Experimental.** `torchvulkan` is an early-stage project. Most of PyTorch's operators run natively on the GPU; the few that don't fall back to the CPU. Expect breaking changes before `1.0`.
|
|
48
|
+
|
|
49
|
+
## Table of contents
|
|
50
|
+
|
|
51
|
+
- [Installation](#installation)
|
|
52
|
+
- [Quickstart](#quickstart)
|
|
53
|
+
- [Examples](#examples)
|
|
54
|
+
- [Benchmarks](#benchmarks)
|
|
55
|
+
- [Python API](#python-api)
|
|
56
|
+
- [CPU fallback](#cpu-fallback)
|
|
57
|
+
- [Environment variables](#environment-variables)
|
|
58
|
+
- [Limitations](#limitations)
|
|
59
|
+
- [How it works](#how-it-works)
|
|
60
|
+
|
|
61
|
+
## Installation
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
pip install torchvulkan
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
At runtime you need a **Vulkan 1.3 (or newer) driver**: your GPU's driver on Linux and Windows. On macOS the wheel bundles [MoltenVK](https://github.com/KhronosGroup/MoltenVK), so nothing else is needed; set `TORCHVULKAN_SYSTEM_VULKAN=1` to use the system Vulkan loader instead. Verify it worked:
|
|
68
|
+
|
|
69
|
+
```python
|
|
70
|
+
import torchvulkan
|
|
71
|
+
print(torchvulkan.is_available(), torchvulkan.device_count())
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
## Quickstart
|
|
75
|
+
|
|
76
|
+
```python
|
|
77
|
+
import torch
|
|
78
|
+
import torchvulkan
|
|
79
|
+
|
|
80
|
+
# Move tensors to the Vulkan device
|
|
81
|
+
a = torch.tensor([[1.0, 2.0], [3.0, 4.0]], device="vulkan")
|
|
82
|
+
b = torch.ones(2, 2, device="vulkan")
|
|
83
|
+
|
|
84
|
+
c = a @ b + 1.0 # matmul + broadcasted add
|
|
85
|
+
d = c.exp().sum(dim=0) # elementwise + reduction
|
|
86
|
+
|
|
87
|
+
print(d.cpu()) # bring the result back to the CPU
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
`nn.Module`s move to the device the usual way:
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
model = torch.nn.Linear(128, 10).to("vulkan")
|
|
94
|
+
out = model(torch.randn(32, 128).to("vulkan"))
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Examples
|
|
98
|
+
|
|
99
|
+
Each example trains a model end to end on the Vulkan device: forward pass, loss, backward pass and optimizer step all run through the backend.
|
|
100
|
+
|
|
101
|
+
| Example | Model | Data |
|
|
102
|
+
| --- | --- | --- |
|
|
103
|
+
| [`examples/mlp.py`](examples/mlp.py) | A multi-layer perceptron with GELU and dropout, trained with AdamW | MNIST |
|
|
104
|
+
| [`examples/llm.py`](examples/llm.py) | A small modern decoder-only transformer: RMSNorm, rotary embeddings, grouped-query attention, a SwiGLU MLP and tied embeddings. Samples text when it's done | Its own source code, character by character |
|
|
105
|
+
| [`examples/resnet.py`](examples/resnet.py) | A ResNet with residual blocks, batch norm and global average pooling, trained with SGD and a one-cycle schedule | CIFAR-10 |
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
pip install torchvision # for the MNIST and CIFAR-10 datasets
|
|
109
|
+
python examples/mlp.py
|
|
110
|
+
python examples/llm.py
|
|
111
|
+
python examples/resnet.py
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Training looks the same as on any other device:
|
|
115
|
+
|
|
116
|
+
```python
|
|
117
|
+
device = torch.device("vulkan" if torchvulkan.is_available() else "cpu")
|
|
118
|
+
model = ResNet().to(device)
|
|
119
|
+
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
|
|
120
|
+
|
|
121
|
+
for data, targets in train_loader:
|
|
122
|
+
data, targets = data.to(device), targets.to(device)
|
|
123
|
+
loss = criterion(model(data), targets)
|
|
124
|
+
optimizer.zero_grad()
|
|
125
|
+
loss.backward()
|
|
126
|
+
optimizer.step()
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
## Benchmarks
|
|
130
|
+
|
|
131
|
+
[`benchmark/benchmark.py`](benchmark/benchmark.py) times a combined forward + backward pass (`float16` on accelerators, `float32` on CPU) on every available backend, sweeping each model's size. Each backend runs in its own subprocess. The models are the ones from the [examples](#examples):
|
|
132
|
+
|
|
133
|
+
- `mlp`: sweeps the hidden width, up to ~141M parameters.
|
|
134
|
+
- `llm`: sweeps the model width (`n_embd`), up to ~26M parameters.
|
|
135
|
+
- `resnet`: sweeps the base channel width, up to ~6M parameters.
|
|
136
|
+
|
|
137
|
+
<p align="center">
|
|
138
|
+
<img alt="Forward + backward pass time vs. model size for the MLP, LLM and ResNet on an NVIDIA GPU: vulkan vs. CUDA and CPU" src="media/benchmark-cuda.png" width="100%">
|
|
139
|
+
</p>
|
|
140
|
+
|
|
141
|
+
<p align="center">
|
|
142
|
+
<i>Vulkan vs. CUDA vs. CPU.</i>
|
|
143
|
+
</p>
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
pip install matplotlib
|
|
147
|
+
python benchmark/benchmark.py all # every model
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
## Python API
|
|
151
|
+
|
|
152
|
+
`torchvulkan` mirrors the `torch.cuda` module, and is also reachable as `torch.vulkan`:
|
|
153
|
+
|
|
154
|
+
```python
|
|
155
|
+
import torchvulkan
|
|
156
|
+
|
|
157
|
+
# devices
|
|
158
|
+
torchvulkan.is_available() # -> bool: a usable Vulkan device was found
|
|
159
|
+
torchvulkan.device_count() # -> int: number of Vulkan devices (discrete GPUs first)
|
|
160
|
+
torchvulkan.current_device() # -> int: index of the current device
|
|
161
|
+
torchvulkan.set_device(1) # make "vulkan" mean "vulkan:1"
|
|
162
|
+
torchvulkan.get_device_name() # -> str
|
|
163
|
+
torchvulkan.get_device_properties() # -> name, total_memory, vendor_id, api_version, subgroup_size, ...
|
|
164
|
+
torchvulkan.synchronize() # block until queued GPU work has finished
|
|
165
|
+
|
|
166
|
+
# memory
|
|
167
|
+
torchvulkan.memory_allocated() # bytes held by tensors
|
|
168
|
+
torchvulkan.max_memory_allocated() # peak since the last reset_peak_memory_stats()
|
|
169
|
+
torchvulkan.memory_reserved() # bytes held by the allocator, including its cache
|
|
170
|
+
torchvulkan.memory_stats() # -> dict
|
|
171
|
+
torchvulkan.mem_get_info() # -> (free, total): what can be allocated given the budget
|
|
172
|
+
torchvulkan.empty_cache() # release cached device memory back to the driver
|
|
173
|
+
|
|
174
|
+
# random numbers (per-device)
|
|
175
|
+
torchvulkan.manual_seed(0) # also: manual_seed_all, seed, seed_all, initial_seed
|
|
176
|
+
torchvulkan.get_rng_state() # also: set_rng_state, get_rng_state_all, set_rng_state_all
|
|
177
|
+
|
|
178
|
+
torchvulkan.is_bf16_supported() # -> bool: bfloat16 runs on the GPU
|
|
179
|
+
torchvulkan.__version__ # package version
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Importing `torchvulkan` registers the backend, after which the device is addressed as `"vulkan"` / `torch.device("vulkan")` throughout PyTorch.
|
|
183
|
+
|
|
184
|
+
## CPU fallback
|
|
185
|
+
|
|
186
|
+
By default, any operation, dtype, or tensor shape the backend doesn't implement natively runs on the **CPU**, so models keep working despite partial coverage. The warning is printed once, for the first operation that falls back; call `torch.set_warn_always(True)` to see every one.
|
|
187
|
+
|
|
188
|
+
Set `TORCHVULKAN_STRICT=1` to turn those silent fallbacks into errors. This is useful for finding which operators a workload actually needs:
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
TORCHVULKAN_STRICT=1 python your_script.py
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
## Environment variables
|
|
195
|
+
|
|
196
|
+
| Variable | Effect |
|
|
197
|
+
| --- | --- |
|
|
198
|
+
| `TORCHVULKAN_STRICT=1` | Raise an error instead of falling back to the CPU. |
|
|
199
|
+
| `TORCHVULKAN_STRICT_ALLOCATION=1` | Raise an out-of-memory error instead of allocating past the VRAM budget. By default torchvulkan allocates anyway (with a one-time warning), and the driver may put the memory in system RAM, which is much slower. |
|
|
200
|
+
| `TORCHVULKAN_SYSTEM_VULKAN=1` | macOS: use the system Vulkan loader instead of the bundled MoltenVK. |
|
|
201
|
+
| `TORCHVULKAN_VALIDATION=1` | Enable the Khronos validation layer, if it's installed. `=sync` also enables synchronization validation. On macOS this needs `TORCHVULKAN_SYSTEM_VULKAN=1`. |
|
|
202
|
+
| `TORCHVULKAN_PROFILE=1` | Print each compute pipeline's driver statistics (such as register use) when it's compiled, on drivers that support `VK_KHR_pipeline_executable_properties`. |
|
|
203
|
+
| `TORCHVULKAN_PIPELINE_CACHE=0` | Don't cache compiled pipelines between runs. By default they're cached in `%LOCALAPPDATA%\torchvulkan` on Windows and `$XDG_CACHE_HOME/torchvulkan` (or `~/.cache/torchvulkan`) elsewhere. |
|
|
204
|
+
| `TORCHVULKAN_CACHE_DIR` | Directory for the pipeline cache. |
|
|
205
|
+
|
|
206
|
+
## Limitations
|
|
207
|
+
|
|
208
|
+
- `torch.compile` with the default Inductor backend doesn't support the `vulkan` device. Run eagerly, or use `torch.compile(model, backend="aot_eager")`.
|
|
209
|
+
- Optimizers created with `fused=True` (such as `torch.optim.AdamW(..., fused=True)`) run their update on the CPU. The default (foreach) implementation runs on the GPU.
|
|
210
|
+
- `float64` runs on the GPU only on devices with `shaderFloat64`; elsewhere it falls back to the CPU. Transcendental functions such as `exp`, `sin` and `tanh` are computed in `float32` precision for `float64` tensors.
|
|
211
|
+
- `bfloat16` runs on the GPU only on devices with `VK_KHR_shader_bfloat16`; elsewhere it falls back to the CPU.
|
|
212
|
+
- Complex tensors can live on the device and FFTs run on the GPU, but most complex arithmetic falls back to the CPU.
|
|
213
|
+
- APIs and coverage may change before `1.0.0`.
|
|
214
|
+
|
|
215
|
+
## How it works
|
|
216
|
+
|
|
217
|
+
1. Importing the package registers a PyTorch `PrivateUse1` backend and renames it to `vulkan`.
|
|
218
|
+
2. Operations dispatched to a `vulkan` tensor are routed to native implementations that record and submit **Vulkan compute shaders**. The shaders are written in [Slang](https://github.com/shader-slang/slang), compiled to SPIR-V at build time, and embedded into the extension.
|
|
219
|
+
3. Device memory is managed with [VMA](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator); Vulkan entry points are loaded with [volk](https://github.com/zeux/volk).
|
|
220
|
+
4. Anything without a native implementation is handled by [CPU fallback](#cpu-fallback).
|
|
221
|
+
|
|
222
|
+
|
|
223
|
+
> [!NOTE]
|
|
224
|
+
> **Disclaimer:** `torchvulkan` is an independent open-source project and is not affiliated with or endorsed by PyTorch or the Linux Foundation.
|
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
<h1 align="center">torchvulkan</h1>
|
|
2
|
+
<p align="center">
|
|
3
|
+
<i>A Cross-Platform Vulkan Backend for PyTorch</i>
|
|
4
|
+
</p>
|
|
5
|
+
|
|
6
|
+
<p align="center">
|
|
7
|
+
<a href="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml"><img alt="CI" src="https://github.com/zainsharief/torchvulkan/actions/workflows/test_torchvulkan.yml/badge.svg"></a>
|
|
8
|
+
<a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
|
|
9
|
+
<img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-blue">
|
|
10
|
+
<img alt="Status: experimental" src="https://img.shields.io/badge/status-experimental-orange">
|
|
11
|
+
</p>
|
|
12
|
+
|
|
13
|
+
`torchvulkan` registers a `vulkan` device with PyTorch and runs tensor operations as [Vulkan](https://www.vulkan.org/) compute shaders. Vulkan runs on GPUs from AMD, Intel, NVIDIA, and Apple (via MoltenVK), plus software rasterizers, so you get one vendor-neutral GPU backend across Linux, macOS, and Windows.
|
|
14
|
+
|
|
15
|
+
You use it like any other PyTorch device:
|
|
16
|
+
|
|
17
|
+
```python
|
|
18
|
+
import torch
|
|
19
|
+
import torchvulkan # registers the 'vulkan' device on import
|
|
20
|
+
|
|
21
|
+
x = torch.randn(1024, 1024).to("vulkan")
|
|
22
|
+
y = torch.randn(1024, 1024).to("vulkan")
|
|
23
|
+
|
|
24
|
+
z = (x @ y).relu() # runs on the GPU via Vulkan compute shaders
|
|
25
|
+
print(z.sum().item()) # move a scalar back to the CPU
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
> [!WARNING]
|
|
29
|
+
> **Experimental.** `torchvulkan` is an early-stage project. Most of PyTorch's operators run natively on the GPU; the few that don't fall back to the CPU. Expect breaking changes before `1.0`.
|
|
30
|
+
|
|
31
|
+
## Table of contents
|
|
32
|
+
|
|
33
|
+
- [Installation](#installation)
|
|
34
|
+
- [Quickstart](#quickstart)
|
|
35
|
+
- [Examples](#examples)
|
|
36
|
+
- [Benchmarks](#benchmarks)
|
|
37
|
+
- [Python API](#python-api)
|
|
38
|
+
- [CPU fallback](#cpu-fallback)
|
|
39
|
+
- [Environment variables](#environment-variables)
|
|
40
|
+
- [Limitations](#limitations)
|
|
41
|
+
- [How it works](#how-it-works)
|
|
42
|
+
|
|
43
|
+
## Installation
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
pip install torchvulkan
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
At runtime you need a **Vulkan 1.3 (or newer) driver**: your GPU's driver on Linux and Windows. On macOS the wheel bundles [MoltenVK](https://github.com/KhronosGroup/MoltenVK), so nothing else is needed; set `TORCHVULKAN_SYSTEM_VULKAN=1` to use the system Vulkan loader instead. Verify it worked:
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
import torchvulkan
|
|
53
|
+
print(torchvulkan.is_available(), torchvulkan.device_count())
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Quickstart
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
import torch
|
|
60
|
+
import torchvulkan
|
|
61
|
+
|
|
62
|
+
# Move tensors to the Vulkan device
|
|
63
|
+
a = torch.tensor([[1.0, 2.0], [3.0, 4.0]], device="vulkan")
|
|
64
|
+
b = torch.ones(2, 2, device="vulkan")
|
|
65
|
+
|
|
66
|
+
c = a @ b + 1.0 # matmul + broadcasted add
|
|
67
|
+
d = c.exp().sum(dim=0) # elementwise + reduction
|
|
68
|
+
|
|
69
|
+
print(d.cpu()) # bring the result back to the CPU
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
`nn.Module`s move to the device the usual way:
|
|
73
|
+
|
|
74
|
+
```python
|
|
75
|
+
model = torch.nn.Linear(128, 10).to("vulkan")
|
|
76
|
+
out = model(torch.randn(32, 128).to("vulkan"))
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Examples
|
|
80
|
+
|
|
81
|
+
Each example trains a model end to end on the Vulkan device: forward pass, loss, backward pass and optimizer step all run through the backend.
|
|
82
|
+
|
|
83
|
+
| Example | Model | Data |
|
|
84
|
+
| --- | --- | --- |
|
|
85
|
+
| [`examples/mlp.py`](examples/mlp.py) | A multi-layer perceptron with GELU and dropout, trained with AdamW | MNIST |
|
|
86
|
+
| [`examples/llm.py`](examples/llm.py) | A small modern decoder-only transformer: RMSNorm, rotary embeddings, grouped-query attention, a SwiGLU MLP and tied embeddings. Samples text when it's done | Its own source code, character by character |
|
|
87
|
+
| [`examples/resnet.py`](examples/resnet.py) | A ResNet with residual blocks, batch norm and global average pooling, trained with SGD and a one-cycle schedule | CIFAR-10 |
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
pip install torchvision # for the MNIST and CIFAR-10 datasets
|
|
91
|
+
python examples/mlp.py
|
|
92
|
+
python examples/llm.py
|
|
93
|
+
python examples/resnet.py
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Training looks the same as on any other device:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
device = torch.device("vulkan" if torchvulkan.is_available() else "cpu")
|
|
100
|
+
model = ResNet().to(device)
|
|
101
|
+
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
|
|
102
|
+
|
|
103
|
+
for data, targets in train_loader:
|
|
104
|
+
data, targets = data.to(device), targets.to(device)
|
|
105
|
+
loss = criterion(model(data), targets)
|
|
106
|
+
optimizer.zero_grad()
|
|
107
|
+
loss.backward()
|
|
108
|
+
optimizer.step()
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Benchmarks
|
|
112
|
+
|
|
113
|
+
[`benchmark/benchmark.py`](benchmark/benchmark.py) times a combined forward + backward pass (`float16` on accelerators, `float32` on CPU) on every available backend, sweeping each model's size. Each backend runs in its own subprocess. The models are the ones from the [examples](#examples):
|
|
114
|
+
|
|
115
|
+
- `mlp`: sweeps the hidden width, up to ~141M parameters.
|
|
116
|
+
- `llm`: sweeps the model width (`n_embd`), up to ~26M parameters.
|
|
117
|
+
- `resnet`: sweeps the base channel width, up to ~6M parameters.
|
|
118
|
+
|
|
119
|
+
<p align="center">
|
|
120
|
+
<img alt="Forward + backward pass time vs. model size for the MLP, LLM and ResNet on an NVIDIA GPU: vulkan vs. CUDA and CPU" src="media/benchmark-cuda.png" width="100%">
|
|
121
|
+
</p>
|
|
122
|
+
|
|
123
|
+
<p align="center">
|
|
124
|
+
<i>Vulkan vs. CUDA vs. CPU.</i>
|
|
125
|
+
</p>
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
pip install matplotlib
|
|
129
|
+
python benchmark/benchmark.py all # every model
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
## Python API
|
|
133
|
+
|
|
134
|
+
`torchvulkan` mirrors the `torch.cuda` module, and is also reachable as `torch.vulkan`:
|
|
135
|
+
|
|
136
|
+
```python
|
|
137
|
+
import torchvulkan
|
|
138
|
+
|
|
139
|
+
# devices
|
|
140
|
+
torchvulkan.is_available() # -> bool: a usable Vulkan device was found
|
|
141
|
+
torchvulkan.device_count() # -> int: number of Vulkan devices (discrete GPUs first)
|
|
142
|
+
torchvulkan.current_device() # -> int: index of the current device
|
|
143
|
+
torchvulkan.set_device(1) # make "vulkan" mean "vulkan:1"
|
|
144
|
+
torchvulkan.get_device_name() # -> str
|
|
145
|
+
torchvulkan.get_device_properties() # -> name, total_memory, vendor_id, api_version, subgroup_size, ...
|
|
146
|
+
torchvulkan.synchronize() # block until queued GPU work has finished
|
|
147
|
+
|
|
148
|
+
# memory
|
|
149
|
+
torchvulkan.memory_allocated() # bytes held by tensors
|
|
150
|
+
torchvulkan.max_memory_allocated() # peak since the last reset_peak_memory_stats()
|
|
151
|
+
torchvulkan.memory_reserved() # bytes held by the allocator, including its cache
|
|
152
|
+
torchvulkan.memory_stats() # -> dict
|
|
153
|
+
torchvulkan.mem_get_info() # -> (free, total): what can be allocated given the budget
|
|
154
|
+
torchvulkan.empty_cache() # release cached device memory back to the driver
|
|
155
|
+
|
|
156
|
+
# random numbers (per-device)
|
|
157
|
+
torchvulkan.manual_seed(0) # also: manual_seed_all, seed, seed_all, initial_seed
|
|
158
|
+
torchvulkan.get_rng_state() # also: set_rng_state, get_rng_state_all, set_rng_state_all
|
|
159
|
+
|
|
160
|
+
torchvulkan.is_bf16_supported() # -> bool: bfloat16 runs on the GPU
|
|
161
|
+
torchvulkan.__version__ # package version
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Importing `torchvulkan` registers the backend, after which the device is addressed as `"vulkan"` / `torch.device("vulkan")` throughout PyTorch.
|
|
165
|
+
|
|
166
|
+
## CPU fallback
|
|
167
|
+
|
|
168
|
+
By default, any operation, dtype, or tensor shape the backend doesn't implement natively runs on the **CPU**, so models keep working despite partial coverage. The warning is printed once, for the first operation that falls back; call `torch.set_warn_always(True)` to see every one.
|
|
169
|
+
|
|
170
|
+
Set `TORCHVULKAN_STRICT=1` to turn those silent fallbacks into errors. This is useful for finding which operators a workload actually needs:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
TORCHVULKAN_STRICT=1 python your_script.py
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
## Environment variables
|
|
177
|
+
|
|
178
|
+
| Variable | Effect |
|
|
179
|
+
| --- | --- |
|
|
180
|
+
| `TORCHVULKAN_STRICT=1` | Raise an error instead of falling back to the CPU. |
|
|
181
|
+
| `TORCHVULKAN_STRICT_ALLOCATION=1` | Raise an out-of-memory error instead of allocating past the VRAM budget. By default torchvulkan allocates anyway (with a one-time warning), and the driver may put the memory in system RAM, which is much slower. |
|
|
182
|
+
| `TORCHVULKAN_SYSTEM_VULKAN=1` | macOS: use the system Vulkan loader instead of the bundled MoltenVK. |
|
|
183
|
+
| `TORCHVULKAN_VALIDATION=1` | Enable the Khronos validation layer, if it's installed. `=sync` also enables synchronization validation. On macOS this needs `TORCHVULKAN_SYSTEM_VULKAN=1`. |
|
|
184
|
+
| `TORCHVULKAN_PROFILE=1` | Print each compute pipeline's driver statistics (such as register use) when it's compiled, on drivers that support `VK_KHR_pipeline_executable_properties`. |
|
|
185
|
+
| `TORCHVULKAN_PIPELINE_CACHE=0` | Don't cache compiled pipelines between runs. By default they're cached in `%LOCALAPPDATA%\torchvulkan` on Windows and `$XDG_CACHE_HOME/torchvulkan` (or `~/.cache/torchvulkan`) elsewhere. |
|
|
186
|
+
| `TORCHVULKAN_CACHE_DIR` | Directory for the pipeline cache. |
|
|
187
|
+
|
|
188
|
+
## Limitations
|
|
189
|
+
|
|
190
|
+
- `torch.compile` with the default Inductor backend doesn't support the `vulkan` device. Run eagerly, or use `torch.compile(model, backend="aot_eager")`.
|
|
191
|
+
- Optimizers created with `fused=True` (such as `torch.optim.AdamW(..., fused=True)`) run their update on the CPU. The default (foreach) implementation runs on the GPU.
|
|
192
|
+
- `float64` runs on the GPU only on devices with `shaderFloat64`; elsewhere it falls back to the CPU. Transcendental functions such as `exp`, `sin` and `tanh` are computed in `float32` precision for `float64` tensors.
|
|
193
|
+
- `bfloat16` runs on the GPU only on devices with `VK_KHR_shader_bfloat16`; elsewhere it falls back to the CPU.
|
|
194
|
+
- Complex tensors can live on the device and FFTs run on the GPU, but most complex arithmetic falls back to the CPU.
|
|
195
|
+
- APIs and coverage may change before `1.0.0`.
|
|
196
|
+
|
|
197
|
+
## How it works
|
|
198
|
+
|
|
199
|
+
1. Importing the package registers a PyTorch `PrivateUse1` backend and renames it to `vulkan`.
|
|
200
|
+
2. Operations dispatched to a `vulkan` tensor are routed to native implementations that record and submit **Vulkan compute shaders**. The shaders are written in [Slang](https://github.com/shader-slang/slang), compiled to SPIR-V at build time, and embedded into the extension.
|
|
201
|
+
3. Device memory is managed with [VMA](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator); Vulkan entry points are loaded with [volk](https://github.com/zeux/volk).
|
|
202
|
+
4. Anything without a native implementation is handled by [CPU fallback](#cpu-fallback).
|
|
203
|
+
|
|
204
|
+
|
|
205
|
+
> [!NOTE]
|
|
206
|
+
> **Disclaimer:** `torchvulkan` is an independent open-source project and is not affiliated with or endorsed by PyTorch or the Linux Foundation.
|