tensoracle 0.4.1__tar.gz → 0.5.1__tar.gz

{tensoracle-0.4.1 → tensoracle-0.5.1}/.gitignore

@@ -2,5 +2,5 @@
 .nf/logs/
 .venv/
 __pycache__/
-src/tensoracle/_version.py
+python/src/tensoracle/_version.py
 site/

{tensoracle-0.4.1 → tensoracle-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tensoracle
-Version: 0.4.1
+Version: 0.5.1
 Summary: Build, save, reuse, and execute TensorRT engines from safetensors weights
 Project-URL: documentation, https://sync-dev-org.github.io/tensoracle/
 Project-URL: repository, https://github.com/sync-dev-org/tensoracle
@@ -44,7 +44,7 @@ Choose the path that matches your work:
 | --- | --- | --- |
 | Inference application | [Reopen a saved engine](#use-a-saved-engine-for-inference) and enqueue it with `handle.create_execution()`. | The application supplies device buffers and CUDA streams, for example through CuPy or `cuda-python`. |
 | Standard graph developer | [Define and build a graph](#define-and-build-a-standard-graph), [save and reopen](#save-and-reopen-an-engine), and [compare outputs](#compare-output-arrays). | None for standard TensorRT layers. |
-| Cubin plugin developer | [Build a cubin plugin](#build-a-cubin-plugin) and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake, and a C++17 compiler. |
+| Cubin plugin developer | [Build a cubin plugin](#build-a-cubin-plugin) and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake 3.20 or later, and a C++17 compiler. |
 
 | Requirement | Inference application | Standard graph developer | Cubin plugin developer |
 | --- | --- | --- | --- |
@@ -55,11 +55,11 @@ Choose the path that matches your work:
 | Triton 3.8.0, `ptxas` 12.9.86, `cuobjdump` 13.3 | Not needed | Not needed | Installed with `tensoracle[producer]` for cubin production |
 | CUDA toolkit | Not needed | Not needed | Native plugin build |
 | TensorRT 11 SDK | Not needed | Not needed | Native plugin build |
-| CMake and C++17 compiler | Not needed | Not needed | Native plugin build |
+| CMake 3.20 or later and C++17 compiler | Not needed | Not needed | Native plugin build |
 
 The supported targets are Linux x86_64 (including WSL2), TensorRT 11, and GPU generations sm86, sm89, and sm120. The producer and native build tools are separate from the `tensoracle` runtime dependencies. Engine building, engine reuse, output comparison, and cubin production do not require torch. Converting `.pth` checkpoints requires the `convert` extra described below.
 
-Follow the [four-stage optimization workflow](https://github.com/sync-dev-org/tensoracle/blob/main/docs/optimization.md) to establish output quality, evaluate fp16 and plugins, and measure speed.
+Follow the [four-stage optimization workflow](https://github.com/sync-dev-org/tensoracle/blob/main/docs/optimization.md) to establish output quality, evaluate lower precision and plugins, and measure speed.
 
 ## Convert a PyTorch checkpoint
 
@@ -104,7 +104,6 @@ def graph(network: trt.INetworkDefinition) -> None:
 
 
 declaration = EngineBuild(
-    precision="fp32",
     inputs={"features": InputShape.ranged((1, 1, 2), (1, 64, 2), (1, 448, 2))},
     settings=BuilderSettings(workspace_limit=1 << 30, optimization_level=3),
 )
@@ -113,7 +112,9 @@ plan: bytes = build_serialized_engine(graph, declaration)
 
 Use `InputShape.fixed((1, 64, 2))` for one input size; the graph may declare that size directly or use `-1` for a dimension fixed by the profile. For a range, put `-1` in each varying dimension of the graph input and declare its minimum, optimum, and maximum sizes. Every graph input needs a matching entry in `EngineBuild.inputs`.
 
-Set `precision="fp16"` to declare an fp16 engine. The graph function chooses each tensor's actual type, including any fp32 tensors inside an fp16 graph. TF32 is always disabled; `BuilderSettings(tf32=True)` raises `ValueError`. The function returns serialized plan bytes. Invalid declarations raise `ValueError`, and a failed TensorRT build raises `RuntimeError`.
+The engine's precision is determined by the tensor types that the graph function places in the network. For an fp16 graph, use `trt.float16` tensors and `np.float16` weights where needed; the graph can also contain fp32 tensors. `EngineBuild` declares input shapes and builder settings. TF32 is always disabled; `BuilderSettings(tf32=True)` raises `ValueError`. The function returns serialized plan bytes. Invalid declarations raise `ValueError`, and a failed TensorRT build raises `RuntimeError`.
+
+The same API builds bf16, int8, and fp8 graphs. Cast tensors to `trt.bfloat16` for bf16; for int8/fp8, place `add_quantize` / `add_dequantize` layers and load their scales from safetensors. The application determines scales with an external quantization library and chooses output quality limits with `compare_outputs`. fp8 targets sm89 and sm120; bf16 and int8 also target sm86. Follow [Build bf16, int8, and fp8 engines](https://github.com/sync-dev-org/tensoracle/blob/main/docs/precision.md) for a complete example, NumPy dtype constraints, and comparison against an fp32 engine. Use different engine names or graph versions for different precisions.
 
 ## Save and reopen an engine
 
@@ -138,7 +139,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -265,6 +266,18 @@ Install the producer extra when you need to compile cubins from your own kernels
 pip install 'tensoracle[producer]'
 ```
 
+Every repository that adopts tensoracle must use this top-level layout:
+
+```text
+cpp/
+  CMakeLists.txt
+python/
+  pyproject.toml
+  src/
+```
+
+Keep C++ sources in `cpp/`, with a CMake project that builds and runs its tests through CTest without Python. Keep the Python project, its src-layout package, and its uv environment in `python/`; run the project's uv commands from that directory. Tensoracle itself follows this layout, and the installed C++ helper paths described below remain inside its Python package.
+
 `tensoracle.producer.compile_bundle` uses the extra's **Triton 3.8.0**, **ptxas 12.9.86**, and **cuobjdump 13.3**. It finds the NVIDIA wheel executables by default; explicit `ptxas` and `cuobjdump` paths can override them. The normal import, engine build, and engine reuse paths do not import the producer or Triton. Cubin production does not need a GPU or import torch; run it without importing torch first. The caller supplies Triton functions and `KernelSpec` records with their public argument types, compile time constants, and any divisibility assumptions. The producer writes sm86, sm89, and sm120 cubins, `bundle.json`, and a C++ translation unit that embeds exactly those cubins. The generated C++ entry records carry each public argument's order, type, count, and divisibility rule into the common launcher. It rejects an unexpected parameter ABI or nonzero scratch requirement. The included [example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md#rebuild-the-cubins) has the command to rebuild its cubins from a repository clone.
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.

{tensoracle-0.4.1 → tensoracle-0.5.1}/README.md

@@ -14,7 +14,7 @@ Choose the path that matches your work:
 | --- | --- | --- |
 | Inference application | [Reopen a saved engine](#use-a-saved-engine-for-inference) and enqueue it with `handle.create_execution()`. | The application supplies device buffers and CUDA streams, for example through CuPy or `cuda-python`. |
 | Standard graph developer | [Define and build a graph](#define-and-build-a-standard-graph), [save and reopen](#save-and-reopen-an-engine), and [compare outputs](#compare-output-arrays). | None for standard TensorRT layers. |
-| Cubin plugin developer | [Build a cubin plugin](#build-a-cubin-plugin) and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake, and a C++17 compiler. |
+| Cubin plugin developer | [Build a cubin plugin](#build-a-cubin-plugin) and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake 3.20 or later, and a C++17 compiler. |
 
 | Requirement | Inference application | Standard graph developer | Cubin plugin developer |
 | --- | --- | --- | --- |
@@ -25,11 +25,11 @@ Choose the path that matches your work:
 | Triton 3.8.0, `ptxas` 12.9.86, `cuobjdump` 13.3 | Not needed | Not needed | Installed with `tensoracle[producer]` for cubin production |
 | CUDA toolkit | Not needed | Not needed | Native plugin build |
 | TensorRT 11 SDK | Not needed | Not needed | Native plugin build |
-| CMake and C++17 compiler | Not needed | Not needed | Native plugin build |
+| CMake 3.20 or later and C++17 compiler | Not needed | Not needed | Native plugin build |
 
 The supported targets are Linux x86_64 (including WSL2), TensorRT 11, and GPU generations sm86, sm89, and sm120. The producer and native build tools are separate from the `tensoracle` runtime dependencies. Engine building, engine reuse, output comparison, and cubin production do not require torch. Converting `.pth` checkpoints requires the `convert` extra described below.
 
-Follow the [four-stage optimization workflow](https://github.com/sync-dev-org/tensoracle/blob/main/docs/optimization.md) to establish output quality, evaluate fp16 and plugins, and measure speed.
+Follow the [four-stage optimization workflow](https://github.com/sync-dev-org/tensoracle/blob/main/docs/optimization.md) to establish output quality, evaluate lower precision and plugins, and measure speed.
 
 ## Convert a PyTorch checkpoint
 
@@ -74,7 +74,6 @@ def graph(network: trt.INetworkDefinition) -> None:
 
 
 declaration = EngineBuild(
-    precision="fp32",
     inputs={"features": InputShape.ranged((1, 1, 2), (1, 64, 2), (1, 448, 2))},
     settings=BuilderSettings(workspace_limit=1 << 30, optimization_level=3),
 )
@@ -83,7 +82,9 @@ plan: bytes = build_serialized_engine(graph, declaration)
 
 Use `InputShape.fixed((1, 64, 2))` for one input size; the graph may declare that size directly or use `-1` for a dimension fixed by the profile. For a range, put `-1` in each varying dimension of the graph input and declare its minimum, optimum, and maximum sizes. Every graph input needs a matching entry in `EngineBuild.inputs`.
 
-Set `precision="fp16"` to declare an fp16 engine. The graph function chooses each tensor's actual type, including any fp32 tensors inside an fp16 graph. TF32 is always disabled; `BuilderSettings(tf32=True)` raises `ValueError`. The function returns serialized plan bytes. Invalid declarations raise `ValueError`, and a failed TensorRT build raises `RuntimeError`.
+The engine's precision is determined by the tensor types that the graph function places in the network. For an fp16 graph, use `trt.float16` tensors and `np.float16` weights where needed; the graph can also contain fp32 tensors. `EngineBuild` declares input shapes and builder settings. TF32 is always disabled; `BuilderSettings(tf32=True)` raises `ValueError`. The function returns serialized plan bytes. Invalid declarations raise `ValueError`, and a failed TensorRT build raises `RuntimeError`.
+
+The same API builds bf16, int8, and fp8 graphs. Cast tensors to `trt.bfloat16` for bf16; for int8/fp8, place `add_quantize` / `add_dequantize` layers and load their scales from safetensors. The application determines scales with an external quantization library and chooses output quality limits with `compare_outputs`. fp8 targets sm89 and sm120; bf16 and int8 also target sm86. Follow [Build bf16, int8, and fp8 engines](https://github.com/sync-dev-org/tensoracle/blob/main/docs/precision.md) for a complete example, NumPy dtype constraints, and comparison against an fp32 engine. Use different engine names or graph versions for different precisions.
 
 ## Save and reopen an engine
 
@@ -108,7 +109,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -235,6 +236,18 @@ Install the producer extra when you need to compile cubins from your own kernels
 pip install 'tensoracle[producer]'
 ```
 
+Every repository that adopts tensoracle must use this top-level layout:
+
+```text
+cpp/
+  CMakeLists.txt
+python/
+  pyproject.toml
+  src/
+```
+
+Keep C++ sources in `cpp/`, with a CMake project that builds and runs its tests through CTest without Python. Keep the Python project, its src-layout package, and its uv environment in `python/`; run the project's uv commands from that directory. Tensoracle itself follows this layout, and the installed C++ helper paths described below remain inside its Python package.
+
 `tensoracle.producer.compile_bundle` uses the extra's **Triton 3.8.0**, **ptxas 12.9.86**, and **cuobjdump 13.3**. It finds the NVIDIA wheel executables by default; explicit `ptxas` and `cuobjdump` paths can override them. The normal import, engine build, and engine reuse paths do not import the producer or Triton. Cubin production does not need a GPU or import torch; run it without importing torch first. The caller supplies Triton functions and `KernelSpec` records with their public argument types, compile time constants, and any divisibility assumptions. The producer writes sm86, sm89, and sm120 cubins, `bundle.json`, and a C++ translation unit that embeds exactly those cubins. The generated C++ entry records carry each public argument's order, type, count, and divisibility rule into the common launcher. It rejects an unexpected parameter ABI or nonzero scratch requirement. The included [example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md#rebuild-the-cubins) has the command to rebuild its cubins from a repository clone.
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.

tensoracle-0.5.1/build_backend.py

@@ -0,0 +1,94 @@
+"""Build from the Python project using the repository's canonical shared sources."""
+
+import shutil
+import tempfile
+import tomllib
+from contextlib import contextmanager
+from pathlib import Path
+
+from hatchling import build as hatchling_build
+from hatchling.builders.sdist import SdistBuilder
+from hatchling.builders.wheel import WheelBuilder
+
+get_requires_for_build_editable = hatchling_build.get_requires_for_build_editable
+get_requires_for_build_sdist = hatchling_build.get_requires_for_build_sdist
+get_requires_for_build_wheel = hatchling_build.get_requires_for_build_wheel
+
+
+@contextmanager
+def _builder(builder_class):
+    project = Path(__file__).resolve().parent
+    config = tomllib.loads((project / "pyproject.toml").read_text(encoding="utf-8"))
+    # Source archives already contain the shared files next to pyproject.toml.
+    source_archive = (project / "PKG-INFO").is_file()
+    repository = project if source_archive else project.parent
+    shared = ("README.md", "LICENSE", "cpp", "docs", "examples", "mkdocs.yml", ".gitignore")
+    for name in shared:
+        if not (repository / name).exists():
+            raise FileNotFoundError(f"Required build source is missing: {repository / name}")
+    if source_archive:
+        yield builder_class(str(project), config=config)
+        return
+
+    with tempfile.TemporaryDirectory(prefix="tensoracle-build-") as directory:
+        staged = Path(directory)
+        for base, names in (
+            (project, ("pyproject.toml", "uv.lock", "build_backend.py", "src", "tests")),
+            (repository, shared),
+        ):
+            for name in names:
+                source, destination = base / name, staged / name
+                if source.is_dir():
+                    shutil.copytree(
+                        source,
+                        destination,
+                        ignore=shutil.ignore_patterns("__pycache__", "*.pyc", ".pytest_cache"),
+                    )
+                else:
+                    shutil.copyfile(source, destination)
+        hatch = config["tool"]["hatch"]
+        hatch["version"]["raw-options"]["root"] = str(repository)
+        hatch["build"]["dev-mode-dirs"] = [str(project / "src")]
+        yield builder_class(str(staged), config=config)
+
+
+def _build(builder_class, directory, version):
+    output = str(Path(directory).resolve())
+    with _builder(builder_class) as builder:
+        artifact = next(builder.build(directory=output, versions=[version]))
+        if version == "editable":
+            generated = Path(builder.root) / "src/tensoracle/_version.py"
+            target = Path(__file__).resolve().parent / "src/tensoracle/_version.py"
+            if generated != target:
+                shutil.copyfile(generated, target)
+        return Path(artifact).name
+
+
+def build_sdist(sdist_directory, config_settings=None):
+    return _build(SdistBuilder, sdist_directory, "standard")
+
+
+def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
+    return _build(WheelBuilder, wheel_directory, "standard")
+
+
+def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
+    return _build(WheelBuilder, wheel_directory, "editable")
+
+
+def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
+    with _builder(WheelBuilder) as builder:
+        builder.metadata.validate_fields()
+        directory = Path(metadata_directory) / f"{builder.artifact_project_id}.dist-info"
+        directory.mkdir(parents=True, exist_ok=True)
+        metadata = builder.config.core_metadata_constructor(builder.metadata)
+        (directory / "METADATA").write_text(metadata, encoding="utf-8")
+        for relative in builder.metadata.core.license_files:
+            target = directory / "licenses" / relative
+            target.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copyfile(Path(builder.root) / relative, target)
+        return directory.name
+
+
+def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
+    return prepare_metadata_for_build_wheel(metadata_directory, config_settings)

tensoracle-0.5.1/cpp/CMakeLists.txt

@@ -0,0 +1,23 @@
+cmake_minimum_required(VERSION 3.20)
+project(tensoracle_cubin LANGUAGES CXX)
+
+set(TENSORRT_ROOT "" CACHE PATH "TensorRT 11 SDK root containing include/NvInfer.h")
+if(NOT EXISTS "${TENSORRT_ROOT}/include/NvInfer.h")
+    message(FATAL_ERROR "TensorRT SDK headers are missing; set TENSORRT_ROOT to the TensorRT 11 SDK")
+endif()
+find_package(CUDAToolkit REQUIRED)
+
+add_library(tensoracle_cubin STATIC src/cubin_plugin.cpp)
+target_compile_features(tensoracle_cubin PUBLIC cxx_std_17)
+target_include_directories(tensoracle_cubin PUBLIC
+    "${CMAKE_CURRENT_SOURCE_DIR}/include" "${TENSORRT_ROOT}/include")
+target_link_libraries(tensoracle_cubin PUBLIC CUDA::cuda_driver)
+target_compile_options(tensoracle_cubin PRIVATE
+    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wno-deprecated-declarations>)
+
+include(CTest)
+if(BUILD_TESTING)
+    add_executable(tensoracle_core_test tests/core.cpp)
+    target_link_libraries(tensoracle_core_test PRIVATE tensoracle_cubin)
+    add_test(NAME tensoracle_fields_and_sha256 COMMAND tensoracle_core_test)
+endif()

tensoracle-0.5.1/cpp/tests/core.cpp

@@ -0,0 +1,31 @@
+// SP-TENSORACLE-004: retain typed fields in order and compute SHA-256.
+#include "tensoracle/cubin_plugin.hpp"
+#include <cstring>
+using namespace tensoracle::cubin;
+using namespace nvinfer1;
+int main() {
+    if (sha256("abc", 3) !=
+        "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad") return 1;
+    float weight[2] = {1.25f, 2.5f};
+    int version = 7;
+    PluginField input[] = {
+        {"weight", weight, PluginFieldType::kFLOAT32, 2},
+        {"version", &version, PluginFieldType::kINT32, 1},
+    };
+    PluginFieldCollection source{2, input};
+    Fields fields;
+    if (!Fields::parse(&source, fields)) return 2;
+    auto const* saved = fields.collection();
+    if (saved->nbFields != 2 ||
+        std::strcmp(saved->fields[0].name, "weight") != 0 ||
+        std::strcmp(saved->fields[1].name, "version") != 0 ||
+        saved->fields[0].type != PluginFieldType::kFLOAT32 ||
+        saved->fields[0].length != 2 ||
+        std::memcmp(saved->fields[0].data, weight, sizeof(weight)) != 0) return 3;
+    weight[0] = 9.0f;
+    if (std::memcmp(saved->fields[0].data, weight, sizeof(weight)) == 0) return 4;
+    Fields copy = fields;
+    if (std::memcmp(copy.collection()->fields[0].data, saved->fields[0].data,
+                    sizeof(weight)) != 0) return 5;
+    return 0;
+}

{tensoracle-0.4.1 → tensoracle-0.5.1}/docs/index.md

@@ -2,7 +2,7 @@
 
 `tensoracle` builds TensorRT engines from model graphs supplied through TensorRT's Python network API. A model package owns its graph, safetensors weights, and output quality limits. `tensoracle` supplies engine building, saved engine reuse, execution, output comparison, and tools for cubin plugins. These workflows do not require torch. Only the optional `.pth` checkpoint converter requires torch, installed with `tensoracle[convert]`.
 
-Follow the [four-stage optimization workflow](optimization.md) to establish output quality, evaluate fp16 and plugins, and measure speed.
+Follow the [four-stage optimization workflow](optimization.md) to establish output quality, evaluate lower precision and plugins, and measure speed.
 
 ## Terms
 
@@ -16,7 +16,7 @@ Follow the [four-stage optimization workflow](optimization.md) to establish outp
 | --- | --- | --- |
 | Inference application | Use `open_engine` to reopen a matching saved engine, then enqueue it with `handle.create_execution()`. | The application supplies buffers and streams, for example through CuPy or `cuda-python`. |
 | Standard graph developer | Define a TensorRT graph, build it with `build_serialized_engine` or `open_engine`, and measure outputs with `compare_outputs`. | None for standard TensorRT layers. |
-| Cubin plugin developer | Produce cubins, build a native TensorRT plugin, and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake, and a C++17 compiler. |
+| Cubin plugin developer | Produce cubins, build a native TensorRT plugin, and package it with the model. | `tensoracle[producer]`, CUDA toolkit, TensorRT SDK, CMake 3.20 or later, and a C++17 compiler. |
 
 | Requirement | Inference application | Standard graph developer | Cubin plugin developer |
 | --- | --- | --- | --- |
@@ -27,7 +27,7 @@ Follow the [four-stage optimization workflow](optimization.md) to establish outp
 | Triton 3.8.0, `ptxas` 12.9.86, `cuobjdump` 13.3 | Not needed | Not needed | Installed with `tensoracle[producer]` for cubin production |
 | CUDA toolkit | Not needed | Not needed | Native plugin build |
 | TensorRT 11 SDK | Not needed | Not needed | Native plugin build |
-| CMake and C++17 compiler | Not needed | Not needed | Native plugin build |
+| CMake 3.20 or later and C++17 compiler | Not needed | Not needed | Native plugin build |
 
 The supported platform is Linux x86_64 (including WSL2) with sm86, sm89, or sm120 GPUs and TensorRT 11. The producer and native build tools are separate from the `tensoracle` runtime dependencies. Engine building, engine reuse, output comparison, and cubin production do not require torch. Converting `.pth` checkpoints requires the `convert` extra.
 
@@ -161,12 +161,14 @@ Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutput
 
 ### Build a standard graph
 
-Define `graph(network)` with TensorRT's Python network API and load model weights from safetensors. Describe each graph input with `InputShape.fixed` or `InputShape.ranged`, and pass an `EngineBuild` declaration to `build_serialized_engine`. `open_engine` uses the same declaration plus a name, graph version, weight files, and `EngineIO` to save and reuse a checked engine. The supported precisions are fp32 (with TF32 disabled) and fp16. The [graph example](https://github.com/sync-dev-org/tensoracle#define-and-build-a-standard-graph) shows the exact calls.
+Define `graph(network)` with TensorRT's Python network API and load model weights from safetensors. Describe each graph input with `InputShape.fixed` or `InputShape.ranged`, and pass an `EngineBuild` declaration to `build_serialized_engine`. `EngineBuild` declares input shapes and builder settings; the graph function determines precision through the tensor types it places in the network. The supported precisions are fp32 (with TF32 disabled), fp16, bf16, int8, and fp8. The [bf16, int8, and fp8 guide](precision.md) shows tensor casts, explicit Q/DQ layers, and scales supplied through safetensors; fp8 targets sm89 and sm120. `open_engine` uses the same declaration plus a name, graph version, weight files, and `EngineIO` to save and reuse a checked engine. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. The [graph example](https://github.com/sync-dev-org/tensoracle#define-and-build-a-standard-graph) shows the exact calls.
 
 The application runs the engine and passes a CPU NumPy candidate output and a corresponding reference output to `compare_outputs`. The application chooses the acceptance limits. See [Compare output arrays](https://github.com/sync-dev-org/tensoracle#compare-output-arrays).
 
 ### Build a cubin plugin
 
+Repositories that adopt tensoracle must follow the [repository layout in the plugin guide](https://github.com/sync-dev-org/tensoracle#build-a-cubin-plugin).
+
 Install `tensoracle[producer]` when you need to compile cubins from your own kernels:
 
 ```bash

{tensoracle-0.4.1 → tensoracle-0.5.1}/docs/optimization.md

@@ -4,15 +4,15 @@ Use the same representative inputs, shapes, and corresponding outputs throughout
 
 ## 1. Establish the fp32 reference
 
-- **tensoracle tools:** Build an fp32 TensorRT engine with `EngineBuild(precision="fp32")` and `build_serialized_engine` or `open_engine`. TF32 is disabled for the engine. Run the engine in the application, then use `compare_outputs(candidate, reference, atol=..., rtol=...)` on corresponding CPU NumPy output arrays.
+- **tensoracle tools:** Build an fp32 TensorRT engine from a graph whose tensor types are fp32, using an `EngineBuild` declaration for input shapes and builder settings with `build_serialized_engine` or `open_engine`. TF32 is disabled for the engine. Run the engine in the application, then use `compare_outputs(candidate, reference, atol=..., rtol=...)` on corresponding CPU NumPy output arrays.
 - **You provide:** Run the official PyTorch implementation in fp32 outside `tensoracle` and fix its outputs as the reference. Supply the TensorRT graph, weights, representative inputs, and an acceptance limit for each output. Keep the reference and candidate inputs and preprocessing identical.
 - **Gate:** Reject non-finite outputs and any output that misses your limits. Advance only when every fp32 TensorRT output passes against its fixed PyTorch reference. Correct the graph or inputs before trying lower precision if this gate fails.
 
 ## 2. Accept lower precision by quality
 
-- **tensoracle tools:** Build an fp16 TensorRT engine with `EngineBuild(precision="fp16")`. Use `compare_outputs` with each fp16 output as the candidate and its accepted fp32 TensorRT output as the reference. It reports non-finite counts and, for finite floating point arrays, maximum absolute error, relative L2 error, cosine similarity, and, when both tolerances are supplied, counts and fractions outside tolerance.
+- **tensoracle tools:** Build an fp16, bf16, int8, or fp8 TensorRT engine using the same `EngineBuild` declaration for input shapes and builder settings. The graph function determines precision through its tensor types. For int8 and fp8, the graph places explicit quantize/dequantize layers and reads their scales from safetensors; the application prepares those scales with an external quantization library. Follow [Build bf16, int8, and fp8 engines](precision.md) for a complete graph example and the GPU targets (fp8 requires sm89 or sm120). When saving with `open_engine`, use a different `name` or `graph_version` from the fp32 engine. Use `compare_outputs` with each lower precision output as the candidate and its accepted fp32 TensorRT output as the reference. It reports non-finite counts and, for finite floating point arrays, maximum absolute error, relative L2 error, cosine similarity, and, when both tolerances are supplied, counts and fractions outside tolerance.
 - **You provide:** Choose limits for those model-independent measurements. Compute any model-specific measurements separately, such as PSNR for images or application-specific time-series quality measures, and choose their limits. Run both engines on the same inputs.
-- **Gate:** Accept the fp16 engine only when its outputs meet all applicable limits, including the model-specific ones. Otherwise keep the accepted fp32 engine or revise the candidate and repeat this stage.
+- **Gate:** Accept the lower precision engine only when its outputs meet all applicable limits, including the model-specific ones. Otherwise keep the accepted fp32 engine or revise the candidate and repeat this stage.
 
 ## 3. Replace a remaining bottleneck with a plugin
 

tensoracle-0.5.1/docs/precision.md

@@ -0,0 +1,154 @@
+# Build bf16, int8, and fp8 engines
+
+The graph function selects tensor types. Use the same `EngineBuild`, `open_engine`, and execution API for fp32, fp16, bf16, int8, and fp8. For bf16, cast tensors to `trt.bfloat16`. For int8 and fp8, place explicit quantize/dequantize (Q/DQ) layers and their scales in the graph. TensorRT 11's [`add_quantize` and `add_dequantize` operators](https://docs.nvidia.com/deeplearning/tensorrt/11.4.0/_static/operators/Quantize.html) accept the quantized type and dequantized type explicitly.
+
+| Graph precision | Tensor placement in this example | Supported tensoracle GPU targets |
+| --- | --- | --- |
+| fp32 | fp32 matrix multiplication and bias, with TF32 disabled | sm86, sm89, sm120 |
+| bf16 | Cast features, matrix, and bias to bf16; cast the result to fp32 | sm86, sm89, sm120 |
+| int8 | Q/DQ the features and matrix with `trt.int8` and explicit scales | sm86, sm89, sm120 |
+| fp8 | Q/DQ the features and matrix with `trt.fp8` (E4M3) and explicit scales | sm89, sm120 |
+
+Do not build the fp8 graph on sm86. The graph keeps fp32 input and output buffers for every precision; the internal tensors determine its arithmetic. TensorRT chooses implementations for the typed graph, so an int8 or fp8 graph can retain higher precision operations such as the bias addition. This example does not establish a speed improvement.
+
+## Prepare weights and scales outside tensoracle
+
+For a real model, use an external quantization library, such as [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer), to determine scales from representative data or quantization-aware training. The model package translates those results into its safetensors file and graph. tensoracle does not calculate scales, calibrate a model, or translate a quantizer's export format.
+
+Store the fp32 weights and positive fp32 scales together in safetensors. In this example, `matrix` has shape `(32, 16)`, `bias` has shape `(1, 16)`, and each scale has shape `(1,)`. The keys `int8.input_scale`, `int8.weight_scale`, `fp8.input_scale`, and `fp8.weight_scale` distinguish the two quantization recipes. Both Q and DQ use the same scale for each tensor.
+
+The following small fixture uses maximum absolute values to choose scales. Its inputs and weights match the GPU test; this scale calculation is only a demonstration for that fixture, not a calibration recipe for other models:
+
+```python
+from pathlib import Path
+
+import numpy as np
+from safetensors.numpy import save_file
+
+path = Path("linear.safetensors")
+features = np.random.default_rng(11).uniform(-1, 1, (16, 32)).astype(np.float32)
+rng = np.random.default_rng(29)
+matrix = rng.uniform(-0.25, 0.25, (32, 16)).astype(np.float32)
+bias = rng.uniform(-0.1, 0.1, (1, 16)).astype(np.float32)
+weights = {"matrix": matrix, "bias": bias}
+for precision, maximum in (("int8", 127), ("fp8", 448)):
+    weights[f"{precision}.input_scale"] = np.array(
+        [np.max(np.abs(features)) / maximum], dtype=np.float32
+    )
+    weights[f"{precision}.weight_scale"] = np.array(
+        [np.max(np.abs(matrix)) / maximum], dtype=np.float32
+    )
+save_file(weights, path)
+```
+
+[`safetensors.numpy` 0.8.0](https://github.com/huggingface/safetensors/blob/v0.8.0/bindings/python/py_src/safetensors/numpy.py) with NumPy 2.5 does not directly load bf16 or fp8 tensors into NumPy arrays: bf16 loading raises `TypeError`, and FP8 E4M3 loading raises `AttributeError`. A safetensors file can store those types, but its NumPy reader cannot represent them in this environment. Keep this example's weights and scales in fp32, then cast or quantize in TensorRT. If an existing checkpoint stores bf16 or fp8 weights, convert them with a reader that supports that dtype outside this NumPy graph path. Installing torch is not required for this example.
+
+## Define the typed graph
+
+This graph computes `features @ matrix + bias`. The bf16 branch casts all three operands before arithmetic and returns fp32. The int8/fp8 branch applies per-tensor Q/DQ to the features and matrix; its bias stays fp32. Keeping the stored matrix in fp32 lets TensorRT quantize it with the supplied scale during building.
+
+```python
+import tensorrt as trt
+from safetensors.numpy import load_file
+
+
+def graph(path, precision):
+    def define(network):
+        weights = load_file(path)
+
+        def constant(array):
+            array = np.ascontiguousarray(array, dtype=np.float32)
+            return network.add_constant(array.shape, array).get_output(0)
+
+        features = network.add_input("features", trt.float32, (16, 32))
+        matrix = constant(weights["matrix"])
+        bias = constant(weights["bias"])
+        if precision == "bf16":
+            features = network.add_cast(features, trt.bfloat16).get_output(0)
+            matrix = network.add_cast(matrix, trt.bfloat16).get_output(0)
+            bias = network.add_cast(bias, trt.bfloat16).get_output(0)
+        elif precision in ("int8", "fp8"):
+            quantized_type = trt.int8 if precision == "int8" else trt.fp8
+
+            def qdq(tensor, key):
+                scale = constant(weights[f"{precision}.{key}_scale"])
+                quantized = network.add_quantize(tensor, scale, quantized_type).get_output(0)
+                return network.add_dequantize(quantized, scale, trt.float32).get_output(0)
+
+            features = qdq(features, "input")
+            matrix = qdq(matrix, "weight")
+        value = network.add_matrix_multiply(
+            features, trt.MatrixOperation.NONE, matrix, trt.MatrixOperation.NONE
+        ).get_output(0)
+        value = network.add_elementwise(value, bias, trt.ElementWiseOperation.SUM).get_output(0)
+        if precision == "bf16":
+            value = network.add_cast(value, trt.float32).get_output(0)
+        value.name = "result"
+        network.mark_output(value)
+
+    return define
+```
+
+## Build, save, and reuse each precision
+
+Use a distinct engine name or graph version whenever tensor types or the graph recipe change. The same weight file serves all four graphs here, so its digest cannot distinguish their precision. Scales are part of that file and therefore part of the saved engine identity.
+
+```python
+from tensoracle import (
+    BuilderSettings, EngineBuild, EngineIO, InputShape, TensorSpec, open_engine,
+)
+
+declaration = EngineBuild(
+    {"features": InputShape.fixed((16, 32))},
+    BuilderSettings(optimization_level=0),
+)
+io = EngineIO(
+    {"features": TensorSpec(trt.float32, (16, 32))},
+    {"result": TensorSpec(trt.float32, (16, 16))},
+)
+
+
+def open_linear(precision):
+    return open_engine(
+        graph(path, precision),
+        declaration,
+        name=f"linear-{precision}",
+        graph_version="1",
+        weights=(path,),
+        io=io,
+    )
+
+
+reference_handle = open_linear("fp32")
+candidate_handle = open_linear("bf16")  # Or "int8"; "fp8" on sm89/sm120.
+```
+
+TF32 remains disabled by tensoracle, including for the fp32 reference. Engines are saved under `XDG_CACHE_HOME/tensoracle`, or `~/.cache/tensoracle`. Repeat the same call with unchanged build conditions in another process to reuse the plan without invoking the graph function. Keep the weight file and the engine names/versions stable across the restart. Follow [Execute an engine](index.md#execute-an-engine) to allocate fp32 input/output buffers and enqueue each handle with the same inputs.
+
+## Compare outputs and choose limits
+
+First establish the fp32 engine against fixed fp32 outputs from the official model implementation, as described in [stage 1](optimization.md#1-establish-the-fp32-reference). Then run the candidate and accepted fp32 engine with identical inputs. Pass their CPU NumPy outputs to `compare_outputs` and choose quality limits in the application:
+
+```python
+from tensoracle import compare_outputs
+
+# candidate_output and fp32_output are the application's engine outputs.
+comparison = compare_outputs(candidate_output, fp32_output, atol=0.01, rtol=0)
+accepted = (
+    comparison.candidate_nonfinite_count == 0
+    and comparison.reference_nonfinite_count == 0
+    and comparison.outside_tolerance_count == 0
+    and comparison.relative_l2_error is not None
+    and comparison.relative_l2_error < 0.01
+)
+```
+
+These example limits belong to the small bf16 fixture. For the fixture above, a TensorRT 11.4 run on sm120 measured:
+
+| Candidate | Maximum absolute error | Relative L2 error | Test `atol` (`rtol=0`) | Test relative L2 limit |
+| --- | --- | --- | --- | --- |
+| bf16 | 0.00357 | 0.00294 | 0.01 | 0.01 |
+| int8 | 0.00710 | 0.00572 | 0.02 | 0.02 |
+| fp8 | 0.0501 | 0.0382 | 0.12 | 0.08 |
+
+The test allows at least twice the observed errors for differences in GPU tactics. These are not tensoracle defaults or quality guarantees for other models. Choose representative data, per-output limits, and model-specific quality measurements for the real application. The GPU test also checks that the two engines occupy separate cache entries and reproduce their outputs in another process without calling either graph.

{tensoracle-0.4.1 → tensoracle-0.5.1}/docs/tensorrt-sdk.md

@@ -102,7 +102,7 @@ Keep `TENSORRT_ROOT` set to the absolute path of the verified `sdk` directory. F
 
 ```bash
 CMAKE_ARGS="-DTENSORRT_ROOT=$TENSORRT_ROOT" uv build examples/cubin_plugin --wheel
-TENSORACLE_SDK_ROOT="$TENSORRT_ROOT" uv run pytest -q tests/test_example_plugin.py
+TENSORACLE_SDK_ROOT="$TENSORRT_ROOT" uv run --project python pytest -q python/tests/test_example_plugin.py
 ```
 
 The tests also need the CUDA toolkit; tests marked `gpu` require an NVIDIA GPU and driver. The SDK is a build input. The application still selects the TensorRT runtime and verifies its own plugin with that runtime.

{tensoracle-0.4.1 → tensoracle-0.5.1}/examples/cubin_plugin/README.md

@@ -9,7 +9,7 @@ The first plugin computes an fp32 32 by 16 dot product and then scales it in a s
 From the tensoracle repository root, run:
 
 ```bash
-uv run --extra producer python -c 'from pathlib import Path; from examples.cubin_plugin.kernels import kernel_specs; from tensoracle.producer import compile_bundle; compile_bundle(kernel_specs(), Path("examples/cubin_plugin/bundle"))'
+uv run --project python --extra producer python -c 'from pathlib import Path; from examples.cubin_plugin.kernels import kernel_specs; from tensoracle.producer import compile_bundle; compile_bundle(kernel_specs(), Path("examples/cubin_plugin/bundle"))'
 ```
 
 This installs `tensoracle[producer]` with Triton 3.8.0, ptxas 12.9.86, and cuobjdump 13.3, and recreates `bundle/` without using a GPU. `compile_bundle` finds ptxas and cuobjdump in their installed NVIDIA wheels. The producer checks the cubin ELF SM, public and hidden argument layout, register count, and zero scratch sizes before writing the manifest. Run the tests below with the producer extra alone so torch is absent; `compile_bundle` rejects an already imported torch.
@@ -17,7 +17,7 @@ This installs `tensoracle[producer]` with Triton 3.8.0, ptxas 12.9.86, and cuobj
 Check that the regenerated files match the committed example without using a GPU or setting producer environment variables:
 
 ```bash
-uv run --extra producer pytest -m "not gpu" tests/test_producer.py
+uv run --project python --extra producer pytest -m "not gpu" python/tests/test_producer.py
 ```
 
 Without the producer extra, the compilation tests skip with an explanation of the missing tools.
@@ -39,7 +39,7 @@ Install the tensoracle wheel and this example wheel into a runtime Python enviro
 Run from the tensoracle repository with the SDK and CUDA toolkit paths set. The GPU tests build the example wheel, install it in a temporary runtime environment, and run dynamic shapes, direct builds, saved engine reuse, simultaneous execution in two distinct CUDA contexts, and an fp32 versus TF32 control. Each GPU subprocess has a 50 second limit.
 
 ```bash
-TENSORACLE_SDK_ROOT="$TENSORRT_ROOT" uv run pytest -q tests/test_example_plugin.py
+TENSORACLE_SDK_ROOT="$TENSORRT_ROOT" uv run --project python pytest -q python/tests/test_example_plugin.py
 ```
 
 Record the GPU name, SM, TensorRT runtime version, test result, and wheel digest on sm86, sm89, and sm120 separately. The code and wheel contain all three cubin generations; successful execution on one generation does not establish the other two.

{tensoracle-0.4.1 → tensoracle-0.5.1}/examples/cubin_plugin/example_cubin_plugin/__init__.py

@@ -27,7 +27,6 @@ def weights_path() -> Path:
 
 def declaration() -> EngineBuild:
     return EngineBuild(
-        "fp32",
         {"features": InputShape.ranged((1, 32), (16, 32), (32, 32))},
         BuilderSettings(optimization_level=0),
     )

{tensoracle-0.4.1 → tensoracle-0.5.1}/mkdocs.yml

@@ -17,6 +17,8 @@ plugins:
       handlers:
         python:
           paths:
+            - python/src
+            # Standalone source archives keep src next to pyproject.toml.
             - src
           options:
             docstring_style: google
@@ -31,6 +33,7 @@ plugins:
 nav:
   - Home: index.md
   - Optimization workflow: optimization.md
+  - bf16, int8, and fp8 engines: precision.md
   - TensorRT SDK setup: tensorrt-sdk.md
   - API reference:
       - Engine and comparison: api/index.md