qnnx-backend 0.1.0__tar.gz

qnnx_backend-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thomas Le Gall
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

qnnx_backend-0.1.0/PKG-INFO

@@ -0,0 +1,309 @@
+Metadata-Version: 2.4
+Name: qnnx-backend
+Version: 0.1.0
+Summary: A PyTorch torch.compile backend for Qualcomm QNN.
+Author: Thomas Le Gall
+License-Expression: MIT
+Project-URL: Repository, https://gitlab.com/legall.thomas.ll/qnnx-backend
+Project-URL: ORCID, https://orcid.org/0009-0007-4387-0435
+Requires-Python: <3.14,>=3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch<2.14,>=2.13
+Requires-Dist: onnxruntime-qnn>=2.4
+Requires-Dist: onnx
+Requires-Dist: onnxscript
+Provides-Extra: dev
+Requires-Dist: numpy; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Dynamic: license-file
+
+# qnnx-backend
+
+`qnnx-backend` is an experimental PyTorch `torch.compile` backend for Qualcomm QNN.
+
+It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.
+
+> [!WARNING]
+> `qnnx-backend` is currently experimental and intended for inference only.
+
+## Features
+
+- PyTorch integration through `torch.compile(..., backend="qnn")`
+- Qualcomm NPU/HTP acceleration
+- Qualcomm GPU acceleration
+- Automatic NPU → GPU → CPU fallback
+- Explicit device selection
+- Operator compatibility validation
+- Static model parameter and buffer freezing
+- QNN availability detection
+
+## Requirements
+
+- Windows ARM64
+- Qualcomm Snapdragon platform with QNN support
+- Python 3.13
+
+### Tested configuration
+
+- PyTorch 2.13
+- ONNX Runtime QNN 2.4
+- Snapdragon X Elite
+
+## Installation
+
+`qnnx-backend` currently requires **Python 3.13 ARM64** on Windows ARM64.
+
+> **Important:** make sure you are using a native ARM64 Python installation.  
+> A Python x64 installation will not use the required Windows ARM64 wheels.
+
+Python.org provides a dedicated **Windows ARM64 installer** for Python 3.13.
+
+You can verify your Python architecture with:
+
+```bash
+python -c "import platform; print(platform.machine())"
+```
+
+The output should be:
+
+```text
+ARM64
+```
+
+Then install PyTorch:
+
+```bash
+python -m pip install torch==2.13.0+cpu --index-url https://download.pytorch.org/whl/cpu
+```
+
+Finally, install `qnnx_backend`:
+
+```bash
+python -m pip install qnnx-backend
+```
+
+## Usage
+
+### Basic usage
+
+`qnnx_backend` integrates with PyTorch through `torch.compile`.
+
+```python
+import torch
+import qnnx_backend
+
+model = MyModel().eval()
+x = torch.randn(1, 32)
+
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+)
+
+with torch.inference_mode():
+    output = compiled_model(x)
+```
+
+By default, `qnnx_backend` automatically selects an available QNN accelerator.
+
+### Checking QNN availability
+
+You can check whether at least one QNN accelerator is available:
+
+```python
+qnnx_backend.is_available()
+```
+
+You can also check a specific device:
+
+```python
+qnnx_backend.is_available("npu")
+qnnx_backend.is_available("gpu")
+```
+
+### Device selection
+
+The QNN backend supports the following device options:
+
+- `"auto"` — automatically selects an available accelerator
+- `"npu"` — uses the Qualcomm NPU/HTP backend
+- `"gpu"` — uses the Qualcomm GPU backend
+- `"cpu"` — uses the original PyTorch graph without QNN acceleration
+
+The default is `"auto"`.
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+    },
+)
+```
+
+### Backend options
+
+| Option | Values | Default | Description |
+| --- | --- | --- | --- |
+| `device` | `auto`, `npu`, `gpu`, `cpu` | `auto` | Selects the execution backend. |
+| `strict` | `True`, `False` | `False` | Controls whether compilation failures can fall back. |
+| `fp16` | `True`, `False` | `False` | Enables HTP FP16 precision for NPU execution. |
+
+### Automatic fallback
+
+With the default configuration, `qnnx_backend` falls back when QNN compilation is not possible.
+
+In `"auto"` mode, devices are attempted in the following order:
+
+```text
+NPU -> GPU -> CPU
+```
+
+For example:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+)
+```
+
+If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.
+
+### Strict mode
+
+Use `strict=True` when execution on the selected QNN device is required:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+        "strict": True,
+    },
+)
+```
+
+In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.
+
+This is useful when testing whether a graph is actually supported by a specific QNN backend.
+
+### NPU FP16 execution
+
+For NPU execution, FP16 precision can be enabled with:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+        "fp16": True,
+    },
+)
+```
+
+This option controls the QNN HTP FP16 precision setting.
+
+## Inference
+
+`qnnx_backend` currently targets inference workloads.
+
+Models should be switched to evaluation mode before compilation:
+
+```python
+model.eval()
+```
+
+Inference should normally be performed using:
+
+```python
+with torch.inference_mode():
+    output = compiled_model(x)
+```
+
+Training and backward execution are not currently supported.
+
+## Model parameters and buffers
+
+Model parameters and buffers are captured when the QNN graph is compiled.
+
+For example:
+
+```python
+compiled_model = torch.compile(
+    model.eval(),
+    backend="qnn",
+)
+
+compiled_model(x)
+```
+
+After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.
+
+Recompile the model after changing its parameters:
+
+```python
+compiled_model = torch.compile(
+    model.eval(),
+    backend="qnn",
+)
+```
+
+## Static shapes
+
+Version 0.1.0 currently targets static input shapes.
+
+Compile the model using the input shapes that will be used during inference.
+
+Dynamic-shape support is not currently provided.
+
+## Operator support
+
+`qnnx_backend` validates exported ONNX operators against the currently supported QNN backends.
+
+Operator support may differ between NPU/HTP and GPU.
+
+See the [operator compatibility list](docs/operators.md) for the current support matrix.
+
+## Benchmarks
+
+Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.
+
+The results below show steady-state inference latency after warm-up.
+
+### Small models
+
+Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).
+
+| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
+| --- | ---: | ---: | ---: | ---: |
+| Small MLP | 16 | 0.042 ms | 0.201 ms | 0.203 ms |
+| Small CNN | 16 | 0.299 ms | 0.953 ms | **0.212 ms** |
+| Small Transformer | 16 | 0.750 ms | 1.459 ms | **0.306 ms** |
+
+### Medium models
+
+Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).
+
+| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
+| --- | ---: | ---: | ---: | ---: |
+| Medium MLP | 16 | 2.523 ms | 0.505 ms | **0.333 ms** |
+| Medium CNN | 16 | 134.117 ms | 11.292 ms | **2.143 ms** |
+| Medium Transformer | 16 | 47.343 ms | 9.176 ms | **2.036 ms** |
+
+See [detailed benchmarks](docs/benchmarks.md) for batch-size scaling, first-call latency, median, p95, standard deviation, and benchmark methodology.
+
+## Author
+
+Developed by **Thomas Le Gall** — [ORCID](https://orcid.org/0009-0007-4387-0435) · [GitLab](https://gitlab.com/legall.thomas.ll).
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.
+
+Copyright © 2026 Thomas Le Gall.

qnnx_backend-0.1.0/README.md

@@ -0,0 +1,288 @@
+# qnnx-backend
+
+`qnnx-backend` is an experimental PyTorch `torch.compile` backend for Qualcomm QNN.
+
+It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.
+
+> [!WARNING]
+> `qnnx-backend` is currently experimental and intended for inference only.
+
+## Features
+
+- PyTorch integration through `torch.compile(..., backend="qnn")`
+- Qualcomm NPU/HTP acceleration
+- Qualcomm GPU acceleration
+- Automatic NPU → GPU → CPU fallback
+- Explicit device selection
+- Operator compatibility validation
+- Static model parameter and buffer freezing
+- QNN availability detection
+
+## Requirements
+
+- Windows ARM64
+- Qualcomm Snapdragon platform with QNN support
+- Python 3.13
+
+### Tested configuration
+
+- PyTorch 2.13
+- ONNX Runtime QNN 2.4
+- Snapdragon X Elite
+
+## Installation
+
+`qnnx-backend` currently requires **Python 3.13 ARM64** on Windows ARM64.
+
+> **Important:** make sure you are using a native ARM64 Python installation.  
+> A Python x64 installation will not use the required Windows ARM64 wheels.
+
+Python.org provides a dedicated **Windows ARM64 installer** for Python 3.13.
+
+You can verify your Python architecture with:
+
+```bash
+python -c "import platform; print(platform.machine())"
+```
+
+The output should be:
+
+```text
+ARM64
+```
+
+Then install PyTorch:
+
+```bash
+python -m pip install torch==2.13.0+cpu --index-url https://download.pytorch.org/whl/cpu
+```
+
+Finally, install `qnnx_backend`:
+
+```bash
+python -m pip install qnnx-backend
+```
+
+## Usage
+
+### Basic usage
+
+`qnnx_backend` integrates with PyTorch through `torch.compile`.
+
+```python
+import torch
+import qnnx_backend
+
+model = MyModel().eval()
+x = torch.randn(1, 32)
+
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+)
+
+with torch.inference_mode():
+    output = compiled_model(x)
+```
+
+By default, `qnnx_backend` automatically selects an available QNN accelerator.
+
+### Checking QNN availability
+
+You can check whether at least one QNN accelerator is available:
+
+```python
+qnnx_backend.is_available()
+```
+
+You can also check a specific device:
+
+```python
+qnnx_backend.is_available("npu")
+qnnx_backend.is_available("gpu")
+```
+
+### Device selection
+
+The QNN backend supports the following device options:
+
+- `"auto"` — automatically selects an available accelerator
+- `"npu"` — uses the Qualcomm NPU/HTP backend
+- `"gpu"` — uses the Qualcomm GPU backend
+- `"cpu"` — uses the original PyTorch graph without QNN acceleration
+
+The default is `"auto"`.
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+    },
+)
+```
+
+### Backend options
+
+| Option | Values | Default | Description |
+| --- | --- | --- | --- |
+| `device` | `auto`, `npu`, `gpu`, `cpu` | `auto` | Selects the execution backend. |
+| `strict` | `True`, `False` | `False` | Controls whether compilation failures can fall back. |
+| `fp16` | `True`, `False` | `False` | Enables HTP FP16 precision for NPU execution. |
+
+### Automatic fallback
+
+With the default configuration, `qnnx_backend` falls back when QNN compilation is not possible.
+
+In `"auto"` mode, devices are attempted in the following order:
+
+```text
+NPU -> GPU -> CPU
+```
+
+For example:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+)
+```
+
+If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.
+
+### Strict mode
+
+Use `strict=True` when execution on the selected QNN device is required:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+        "strict": True,
+    },
+)
+```
+
+In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.
+
+This is useful when testing whether a graph is actually supported by a specific QNN backend.
+
+### NPU FP16 execution
+
+For NPU execution, FP16 precision can be enabled with:
+
+```python
+compiled_model = torch.compile(
+    model,
+    backend="qnn",
+    options={
+        "device": "npu",
+        "fp16": True,
+    },
+)
+```
+
+This option controls the QNN HTP FP16 precision setting.
+
+## Inference
+
+`qnnx_backend` currently targets inference workloads.
+
+Models should be switched to evaluation mode before compilation:
+
+```python
+model.eval()
+```
+
+Inference should normally be performed using:
+
+```python
+with torch.inference_mode():
+    output = compiled_model(x)
+```
+
+Training and backward execution are not currently supported.
+
+## Model parameters and buffers
+
+Model parameters and buffers are captured when the QNN graph is compiled.
+
+For example:
+
+```python
+compiled_model = torch.compile(
+    model.eval(),
+    backend="qnn",
+)
+
+compiled_model(x)
+```
+
+After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.
+
+Recompile the model after changing its parameters:
+
+```python
+compiled_model = torch.compile(
+    model.eval(),
+    backend="qnn",
+)
+```
+
+## Static shapes
+
+Version 0.1.0 currently targets static input shapes.
+
+Compile the model using the input shapes that will be used during inference.
+
+Dynamic-shape support is not currently provided.
+
+## Operator support
+
+`qnnx_backend` validates exported ONNX operators against the currently supported QNN backends.
+
+Operator support may differ between NPU/HTP and GPU.
+
+See the [operator compatibility list](docs/operators.md) for the current support matrix.
+
+## Benchmarks
+
+Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.
+
+The results below show steady-state inference latency after warm-up.
+
+### Small models
+
+Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).
+
+| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
+| --- | ---: | ---: | ---: | ---: |
+| Small MLP | 16 | 0.042 ms | 0.201 ms | 0.203 ms |
+| Small CNN | 16 | 0.299 ms | 0.953 ms | **0.212 ms** |
+| Small Transformer | 16 | 0.750 ms | 1.459 ms | **0.306 ms** |
+
+### Medium models
+
+Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).
+
+| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
+| --- | ---: | ---: | ---: | ---: |
+| Medium MLP | 16 | 2.523 ms | 0.505 ms | **0.333 ms** |
+| Medium CNN | 16 | 134.117 ms | 11.292 ms | **2.143 ms** |
+| Medium Transformer | 16 | 47.343 ms | 9.176 ms | **2.036 ms** |
+
+See [detailed benchmarks](docs/benchmarks.md) for batch-size scaling, first-call latency, median, p95, standard deviation, and benchmark methodology.
+
+## Author
+
+Developed by **Thomas Le Gall** — [ORCID](https://orcid.org/0009-0007-4387-0435) · [GitLab](https://gitlab.com/legall.thomas.ll).
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.
+
+Copyright © 2026 Thomas Le Gall.

qnnx_backend-0.1.0/pyproject.toml

@@ -0,0 +1,75 @@
+[build-system]
+requires = ["setuptools>=70"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "qnnx-backend"
+version = "0.1.0"
+description = "A PyTorch torch.compile backend for Qualcomm QNN."
+readme = "README.md"
+license = "MIT"
+authors = [
+    { name = "Thomas Le Gall" }
+]
+
+
+requires-python = ">=3.13,<3.14"
+dependencies = [
+    "torch>=2.13,<2.14",
+    "onnxruntime-qnn>=2.4",
+    "onnx",
+    "onnxscript",
+]
+
+[project.urls]
+Repository = "https://gitlab.com/legall.thomas.ll/qnnx-backend"
+ORCID = "https://orcid.org/0009-0007-4387-0435"
+
+[project.optional-dependencies]
+dev = [
+    "numpy",
+    "pytest",
+    "pytest-cov",
+]
+
+[project.entry-points."torch_dynamo_backends"]
+qnn = "qnnx_backend.backend:qnn_backend"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py313"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+[tool.hatch.envs.default]
+detached = true
+dependencies = [
+    "ruff",
+    "pyright"
+]
+
+[tool.hatch.envs.default.scripts]
+lint = "ruff check ."
+lint-fix = "ruff check . --fix"
+format = "ruff format ."
+format-check = "ruff format --check ."
+typecheck = "pyright"
+
+check = [
+    "ruff check .",
+    "ruff format --check .",
+    "pyright"
+]
+
+fix = [
+    "ruff format .",
+    "ruff check . --fix",
+]

qnnx_backend-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

qnnx_backend-0.1.0/src/qnnx_backend/__init__.py

@@ -0,0 +1,11 @@
+from qnnx_backend.compatibility import QNNDevice
+from qnnx_backend.runtime.qnn import QNNRuntime
+
+
+def is_available(
+    device: QNNDevice | None = None,
+) -> bool:
+    if device is not None:
+        return QNNRuntime.is_device_available(device)
+
+    return QNNRuntime.is_device_available("npu") or QNNRuntime.is_device_available("gpu")