tensoracle 0.3.2__tar.gz → 0.4.1__tar.gz

{tensoracle-0.3.2 → tensoracle-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tensoracle
-Version: 0.3.2
+Version: 0.4.1
 Summary: Build, save, reuse, and execute TensorRT engines from safetensors weights
 Project-URL: documentation, https://sync-dev-org.github.io/tensoracle/
 Project-URL: repository, https://github.com/sync-dev-org/tensoracle
@@ -138,7 +138,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -269,6 +269,23 @@ pip install 'tensoracle[producer]'
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.
 
+`tensoracle::cubin::PluginBase` accepts a `PluginIO` declaration as its fourth constructor argument. `inputs` lists one `TensorSpec` per input in network order; each spec declares a TensorRT `type` and `format`. `output` declares the single output's type and format, and `shape_input_index` selects the input whose shape the output copies. For example, declare two fp16 / HWC8 inputs and one fp16 / HWC8 output with the shape of input 0:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor. For one fp16 / HWC8 input, use `PluginIO{{half}, half, 0}`. The base class checks tensor counts, types, and formats during format negotiation, configuration, and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check the dimensions of every input in declaration order and the output; configuration may include wildcard dimensions (`-1`). For a residual operation, this callback can reject different shapes for its two inputs. The caller's kernel must handle the declared storage format. Omitting the fourth argument keeps one fp32 / LINEAR input and output with input 0's shape, so `PluginBase(name, plugin_namespace, fields)` remains valid. An empty input list or an invalid `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 `tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments

{tensoracle-0.3.2 → tensoracle-0.4.1}/README.md

@@ -108,7 +108,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -239,6 +239,23 @@ pip install 'tensoracle[producer]'
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.
 
+`tensoracle::cubin::PluginBase` accepts a `PluginIO` declaration as its fourth constructor argument. `inputs` lists one `TensorSpec` per input in network order; each spec declares a TensorRT `type` and `format`. `output` declares the single output's type and format, and `shape_input_index` selects the input whose shape the output copies. For example, declare two fp16 / HWC8 inputs and one fp16 / HWC8 output with the shape of input 0:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor. For one fp16 / HWC8 input, use `PluginIO{{half}, half, 0}`. The base class checks tensor counts, types, and formats during format negotiation, configuration, and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check the dimensions of every input in declaration order and the output; configuration may include wildcard dimensions (`-1`). For a residual operation, this callback can reject different shapes for its two inputs. The caller's kernel must handle the declared storage format. Omitting the fourth argument keeps one fp32 / LINEAR input and output with input 0's shape, so `PluginBase(name, plugin_namespace, fields)` remains valid. An empty input list or an invalid `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 `tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments

{tensoracle-0.3.2 → tensoracle-0.4.1}/cpp/include/tensoracle/cubin_plugin.hpp

@@ -41,13 +41,31 @@ private:
     nvinfer1::PluginFieldCollection collection_{};
 };
 
-// Implements TensorRT V3 lifecycle and the format/serialization boilerplate.
+struct TensorSpec {
+    nvinfer1::DataType type{nvinfer1::DataType::kFLOAT};
+    nvinfer1::TensorFormat format{nvinfer1::TensorFormat::kLINEAR};
+};
+
+// Declares the input count, each tensor's type and format, and the input whose
+// shape is copied to the single output. For configurations this cannot express
+// (multiple outputs or computed output shapes), developers can override virtual
+// V3 methods or ask tensoracle's maintainers to extend the declaration.
+struct PluginIO {
+    std::vector<TensorSpec> inputs{TensorSpec{}};
+    TensorSpec output{};
+    std::int32_t shape_input_index{0};
+};
+
+// Implements TensorRT V3 lifecycle and format/serialization boilerplate using
+// a PluginIO declaration. For other configurations, developers can override
+// virtual V3 methods or ask tensoracle's maintainers to extend the declaration.
 class PluginBase : public nvinfer1::IPluginV3,
                    public nvinfer1::IPluginV3OneCore,
                    public nvinfer1::IPluginV3OneBuild,
                    public nvinfer1::IPluginV3OneRuntime {
 public:
-    PluginBase(char const* name, char const* plugin_namespace, Fields fields);
+    // Throws std::invalid_argument for no inputs or an out-of-range shape input.
+    PluginBase(char const* name, char const* plugin_namespace, Fields fields, PluginIO io = {});
     ~PluginBase() noexcept override = default;
 
     nvinfer1::IPluginCapability* getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept override;
@@ -79,13 +97,16 @@ public:
 
 protected:
     Fields const& fields() const noexcept { return fields_; }
-    virtual std::int32_t validate_shape(nvinfer1::Dims const& input,
+    // Inputs are in declaration order; the array is valid only during this call.
+    // configurePlugin may provide wildcard (-1) dimensions.
+    virtual std::int32_t validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count,
         nvinfer1::Dims const& output) const noexcept = 0;
 
 private:
     char const* name_;
     char const* namespace_;
     Fields fields_;
+    PluginIO io_;
 };
 
 enum class ArgumentKind : std::uint8_t { kPointer, kInt32, kFloat32 };

{tensoracle-0.3.2 → tensoracle-0.4.1}/cpp/src/cubin_plugin.cpp

@@ -4,11 +4,48 @@
 #include <array>
 #include <cstring>
 #include <limits>
+#include <stdexcept>
 #include <utility>
 
 namespace tensoracle::cubin {
 namespace {
 
+nvinfer1::PluginTensorDesc const& tensor_desc(nvinfer1::PluginTensorDesc const& value) noexcept
+{
+    return value;
+}
+
+nvinfer1::PluginTensorDesc const& tensor_desc(nvinfer1::DynamicPluginTensorDesc const& value) noexcept
+{
+    return value.desc;
+}
+
+bool matches_tensor(nvinfer1::PluginTensorDesc const& descriptor, TensorSpec const& spec) noexcept
+{
+    return descriptor.type == spec.type && descriptor.format == spec.format;
+}
+
+template <typename Descriptor, typename Validate>
+std::int32_t validate_tensors(Descriptor const* inputs, std::int32_t input_count,
+    Descriptor const* outputs, std::int32_t output_count, PluginIO const& io,
+    Validate validate) noexcept
+{
+    if (inputs == nullptr || outputs == nullptr || input_count < 0 || output_count != 1 ||
+        static_cast<std::size_t>(input_count) != io.inputs.size()) return -1;
+    for (std::int32_t index = 0; index < input_count; ++index)
+        if (!matches_tensor(tensor_desc(inputs[index]), io.inputs[index])) return -1;
+    if (!matches_tensor(tensor_desc(outputs[0]), io.output)) return -1;
+    try {
+        std::vector<nvinfer1::Dims> dims;
+        dims.reserve(io.inputs.size());
+        for (std::int32_t index = 0; index < input_count; ++index)
+            dims.push_back(tensor_desc(inputs[index]).dims);
+        return validate(dims.data(), input_count, tensor_desc(outputs[0]).dims);
+    } catch (...) {
+        return -1;
+    }
+}
+
 std::size_t field_size(nvinfer1::PluginFieldType type) noexcept
 {
     using nvinfer1::PluginFieldType;
@@ -102,8 +139,14 @@ void Fields::rebuild()
 
 nvinfer1::PluginFieldCollection const* Fields::collection() noexcept { return &collection_; }
 
-PluginBase::PluginBase(char const* name, char const* plugin_namespace, Fields fields)
-    : name_(name), namespace_(plugin_namespace), fields_(std::move(fields)) {}
+PluginBase::PluginBase(char const* name, char const* plugin_namespace, Fields fields, PluginIO io)
+    : name_(name), namespace_(plugin_namespace), fields_(std::move(fields)), io_(std::move(io))
+{
+    if (io_.inputs.empty() || io_.inputs.size() > static_cast<std::size_t>(
+            std::numeric_limits<std::int32_t>::max()) || io_.shape_input_index < 0 ||
+        static_cast<std::size_t>(io_.shape_input_index) >= io_.inputs.size())
+        throw std::invalid_argument("PluginIO requires inputs and a valid shape_input_index");
+}
 
 nvinfer1::IPluginCapability* PluginBase::getCapabilityInterface(
     nvinfer1::PluginCapabilityType type) noexcept
@@ -124,20 +167,23 @@ bool PluginBase::supportsFormatCombination(std::int32_t position,
     nvinfer1::DynamicPluginTensorDesc const* in_out,
     std::int32_t input_count, std::int32_t output_count) noexcept
 {
-    if (in_out == nullptr || input_count != 1 || output_count != 1 || position < 0 || position > 1)
+    if (in_out == nullptr || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size() ||
+        output_count != 1 || position < 0 || position > input_count)
         return false;
-    auto const& descriptor = in_out[position].desc;
-    return descriptor.type == nvinfer1::DataType::kFLOAT &&
-        descriptor.format == nvinfer1::TensorFormat::kLINEAR;
+    auto const& spec = position < input_count ? io_.inputs[position] : io_.output;
+    return matches_tensor(in_out[position].desc, spec);
 }
 
 std::int32_t PluginBase::getOutputDataTypes(nvinfer1::DataType* output_types,
     std::int32_t output_count, nvinfer1::DataType const* input_types,
     std::int32_t input_count) const noexcept
 {
-    if (output_types == nullptr || input_types == nullptr || output_count != 1 || input_count != 1 ||
-        input_types[0] != nvinfer1::DataType::kFLOAT) return -1;
-    output_types[0] = nvinfer1::DataType::kFLOAT;
+    if (output_types == nullptr || input_types == nullptr || output_count != 1 || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size()) return -1;
+    for (std::int32_t index = 0; index < input_count; ++index)
+        if (input_types[index] != io_.inputs[index].type) return -1;
+    output_types[0] = io_.output.type;
     return 0;
 }
 
@@ -146,9 +192,10 @@ std::int32_t PluginBase::getOutputShapes(nvinfer1::DimsExprs const* inputs,
     nvinfer1::DimsExprs* outputs, std::int32_t output_count,
     nvinfer1::IExprBuilder&) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 ||
+    if (inputs == nullptr || outputs == nullptr || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size() ||
         shape_input_count != 0 || output_count != 1) return -1;
-    outputs[0] = inputs[0];
+    outputs[0] = inputs[io_.shape_input_index];
     return 0;
 }
 
@@ -156,8 +203,10 @@ std::int32_t PluginBase::configurePlugin(nvinfer1::DynamicPluginTensorDesc const
     std::int32_t input_count, nvinfer1::DynamicPluginTensorDesc const* outputs,
     std::int32_t output_count) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 || output_count != 1) return -1;
-    return validate_shape(inputs[0].desc.dims, outputs[0].desc.dims);
+    return validate_tensors(inputs, input_count, outputs, output_count, io_,
+        [this](nvinfer1::Dims const* dims, std::int32_t count, nvinfer1::Dims const& output) {
+            return validate_shape(dims, count, output);
+        });
 }
 
 std::size_t PluginBase::getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const*, std::int32_t,
@@ -169,8 +218,10 @@ std::int32_t PluginBase::onShapeChange(nvinfer1::PluginTensorDesc const* inputs,
     std::int32_t input_count, nvinfer1::PluginTensorDesc const* outputs,
     std::int32_t output_count) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 || output_count != 1) return -1;
-    return validate_shape(inputs[0].dims, outputs[0].dims);
+    return validate_tensors(inputs, input_count, outputs, output_count, io_,
+        [this](nvinfer1::Dims const* dims, std::int32_t count, nvinfer1::Dims const& output) {
+            return validate_shape(dims, count, output);
+        });
 }
 
 nvinfer1::IPluginV3* PluginBase::attachToContext(nvinfer1::IPluginResourceContext*) noexcept

{tensoracle-0.3.2 → tensoracle-0.4.1}/docs/index.md

@@ -70,7 +70,7 @@ handle = open_engine(
 )
 ```
 
-`my_model` represents the application's model package. On a valid cache hit, `open_engine` does not call `graph`; on a miss, it builds and saves a new engine. `open_engine` itself does not execute the engine or manage GPU buffers. Create an execution unit from its returned handle to enqueue inference. Saved engines live under `XDG_CACHE_HOME/tensoracle`, or `~/.cache/tensoracle` when `XDG_CACHE_HOME` is unset. See the [complete cache example](https://github.com/sync-dev-org/tensoracle#save-and-reopen-an-engine) for the declaration and expected input and output types.
+`my_model` represents the application's model package. On a valid cache hit, `open_engine` does not call `graph`; on a miss, it builds and saves a new engine. Raise `graph_version` when the graph code changes or when any other caller value that changes the engine contents but is not part of the cache identity changes, such as a plugin manifest or a build recipe. `open_engine` itself does not execute the engine or manage GPU buffers. Create an execution unit from its returned handle to enqueue inference. Saved engines live under `XDG_CACHE_HOME/tensoracle`, or `~/.cache/tensoracle` when `XDG_CACHE_HOME` is unset. See the [complete cache example](https://github.com/sync-dev-org/tensoracle#save-and-reopen-an-engine) for the declaration and expected input and output types.
 
 ### Execute an engine
 
@@ -175,6 +175,23 @@ pip install 'tensoracle[producer]'
 
 `tensoracle.producer.compile_bundle` uses the extra's Triton 3.8.0, `ptxas` 12.9.86, and `cuobjdump` 13.3 to produce cubins for sm86, sm89, and sm120. It finds the NVIDIA wheel executables by default and accepts explicit paths as overrides. Cubin production does not need a GPU or import torch; run it without importing torch first. Build the native plugin with a TensorRT 11 SDK, CUDA toolkit, CMake, and a C++17 compiler. Inspect the resulting bundle or wheel with `tensoracle.bundle`. The [cubin plugin example](https://github.com/sync-dev-org/tensoracle/tree/main/examples/cubin_plugin) covers rebuilding from a repository clone, packaging, and runtime use. Normal import, engine building, and engine reuse do not import the producer or Triton, and do not need the producer extra or native build tools.
 
+Pass a `tensoracle::cubin::PluginIO` declaration as the fourth argument to `PluginBase(name, plugin_namespace, fields, io)`. The `inputs` vector determines the input count and declares each input's TensorRT `type` and `format` through `TensorSpec`; `output` declares the single output's type and format. `shape_input_index` selects the input whose shape the output copies. This example declares two fp16 / HWC8 inputs and an fp16 / HWC8 output with input 0's shape:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor; use `PluginIO{{half}, half, 0}` for one fp16 / HWC8 input. The common implementation negotiates the declared types and formats, sets the output type and shape, and checks counts, types, and formats during configuration and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check all input dimensions in declaration order and the output dimensions, including wildcard dimensions (`-1`) during configuration. A residual plugin can reject mismatched input shapes in this callback. The kernel must handle the declared storage format. Omitting `io` keeps one fp32 / LINEAR input and output with input 0's shape; the original three-argument constructor call remains valid. An empty input list or an out-of-range `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 ## API reference
 
 - [Engine building, cache reuse, execution, and output comparison](api/index.md)

{tensoracle-0.3.2 → tensoracle-0.4.1}/docs/optimization.md

@@ -18,7 +18,7 @@ Use the same representative inputs, shapes, and corresponding outputs throughout
 
 First check whether a combination of standard TensorRT layers can preserve the operation's meaning, shapes, control flow, and output quality. For example, a deformable convolution may be expressible through grid sampling, mask multiplication, and convolution. If a plugin bundled with TensorRT matches the required behavior, compare that option too. Use a custom cubin plugin only when the suitable alternatives still leave a bottleneck.
 
-- **tensoracle tools:** `tensoracle.producer.compile_bundle` generates cubins and a manifest for supported GPU generations. The shared C++ plugin helpers support a caller-built native plugin, and `tensoracle.bundle.inspect_bundle` or `inspect_wheel` checks the cubins and their match with the plugin. Pass the plugin library through `plugins=` to `build_serialized_engine` or `open_engine`. Follow [Build a cubin plugin](https://github.com/sync-dev-org/tensoracle#build-a-cubin-plugin) and the [cubin plugin example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md) for the generation, build, and inspection operations.
+- **tensoracle tools:** `tensoracle.producer.compile_bundle` generates cubins and a manifest for supported GPU generations. The shared C++ plugin helpers support a caller-built native plugin with declared tensor types, formats, and an input whose shape the output copies; `tensoracle.bundle.inspect_bundle` or `inspect_wheel` checks the cubins and their match with the plugin. Pass the plugin library through `plugins=` to `build_serialized_engine` or `open_engine`. Follow [Build a cubin plugin](index.md#build-a-cubin-plugin) for the `PluginIO` declaration, including fp16 / HWC8 and two-input examples, and the [cubin plugin example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md) for the generation, build, and inspection operations.
 - **You provide:** Identify the bottleneck, write the kernel and model-specific plugin behavior, choose its supported shapes and types, build and package its native library, and run the resulting engines. An fp32 kernel must not use TF32.
 - **Gate:** For a custom cubin plugin, the bundle and plugin inspections must pass. Reapply stage 1 to the replacement's fp32 engine against the same fixed PyTorch reference. If using fp16, reapply stage 2 against an accepted fp32 engine with the same limits and model-specific measurements. Advance only when the replacement meets those quality gates; otherwise retain the accepted implementation.
 

{tensoracle-0.3.2 → tensoracle-0.4.1}/examples/cubin_plugin/plugin.cpp

@@ -117,7 +117,7 @@ public:
     {
         if (!valid_ || input_descriptors == nullptr || output_descriptors == nullptr ||
             inputs == nullptr || outputs == nullptr || inputs[0] == nullptr || outputs[0] == nullptr ||
-            validate_shape(input_descriptors[0].dims, output_descriptors[0].dims) != 0) return -1;
+            validate_shape(&input_descriptors[0].dims, 1, output_descriptors[0].dims) != 0) return -1;
         int rows = input_descriptors[0].dims.d[0];
         if (rows < 1 || rows > INT_MAX / 16) return -1;
         int const index = sm_index();
@@ -141,9 +141,10 @@ public:
     }
 
 protected:
-    std::int32_t validate_shape(Dims const& input, Dims const& output) const noexcept override
+    std::int32_t validate_shape(Dims const* inputs, std::int32_t input_count,
+        Dims const& output) const noexcept override
     {
-        return shape_matches(input, output, 32) ? 0 : -1;
+        return input_count == 1 && shape_matches(inputs[0], output, 32) ? 0 : -1;
     }
 
 private:
@@ -178,7 +179,7 @@ public:
     {
         if (!valid_ || input_descriptors == nullptr || output_descriptors == nullptr ||
             inputs == nullptr || outputs == nullptr || inputs[0] == nullptr || outputs[0] == nullptr ||
-            validate_shape(input_descriptors[0].dims, output_descriptors[0].dims) != 0) return -1;
+            validate_shape(&input_descriptors[0].dims, 1, output_descriptors[0].dims) != 0) return -1;
         int const rows = input_descriptors[0].dims.d[0];
         if (rows < 1 || rows > INT_MAX / 16) return -1;
         int const index = sm_index();
@@ -194,9 +195,10 @@ public:
     }
 
 protected:
-    std::int32_t validate_shape(Dims const& input, Dims const& output) const noexcept override
+    std::int32_t validate_shape(Dims const* inputs, std::int32_t input_count,
+        Dims const& output) const noexcept override
     {
-        return shape_matches(input, output, 16) ? 0 : -1;
+        return input_count == 1 && shape_matches(inputs[0], output, 16) ? 0 : -1;
     }
 
 private:

{tensoracle-0.3.2 → tensoracle-0.4.1}/pyproject.toml

@@ -52,7 +52,15 @@ local_scheme = "no-local-version"
 version-file = "src/tensoracle/_version.py"
 
 [tool.hatch.build.targets.sdist]
-exclude = ["/.nf/", "/.github/", "/AGENTS.md", "/CLAUDE.md", "/tools/"]
+exclude = [
+    "/.nf/",
+    "/.github/",
+    "/AGENTS.md",
+    "/CLAUDE.md",
+    "/tools/",
+    "/noren.yaml",
+    "/noren-words.txt",
+]
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/tensoracle"]
@@ -66,6 +74,7 @@ dev = [
     "cupy-cuda13x>=14.2.0",
     "mkdocs-material>=9.7.7",
     "mkdocstrings[python]>=1.0.6",
+    "noren==0.2.0",
     "pytest>=8.0",
     "ruff>=0.8",
 ]
@@ -75,7 +84,7 @@ testpaths = ["tests"]
 addopts = ["--strict-markers"]
 markers = [
     "gpu: tests that build or load a TensorRT engine on a CUDA GPU",
-    "req(id): parent requirement ID in .nf/docs/requirements.md",
+    "req(id): ID of the adopted spec item in .nf/docs/spec.md that the test verifies",
 ]
 
 [tool.ruff]

{tensoracle-0.3.2 → tensoracle-0.4.1}/src/tensoracle/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.3.2'
-__version_tuple__ = version_tuple = (0, 3, 2)
+__version__ = version = '0.4.1'
+__version_tuple__ = version_tuple = (0, 4, 1)
 
 __commit_id__ = commit_id = None

tensoracle-0.4.1/tests/cubin_io/check.cpp

@@ -0,0 +1,158 @@
+#include "tensoracle/cubin_plugin.hpp"
+
+#include <cassert>
+#include <cstring>
+#include <stdexcept>
+#include <utility>
+
+using namespace nvinfer1;
+using namespace tensoracle::cubin;
+
+class Probe final : public PluginBase {
+public:
+    explicit Probe(PluginIO io = {}) : PluginBase("probe", "tests", {}, std::move(io)) {}
+    IPluginV3* clone() noexcept override { return nullptr; }
+    int32_t enqueue(PluginTensorDesc const*, PluginTensorDesc const*, void const* const*,
+        void* const*, void*, cudaStream_t) noexcept override { return -1; }
+
+protected:
+    int32_t validate_shape(Dims const* inputs, int32_t count,
+        Dims const& output) const noexcept override
+    {
+        for (int32_t i = 0; i < count; ++i) {
+            if (inputs[i].nbDims != output.nbDims) return -1;
+            for (int32_t j = 0; j < output.nbDims; ++j)
+                if (inputs[i].d[j] != output.d[j]) return -1;
+        }
+        return 0;
+    }
+};
+
+PluginIO mixed_io()
+{
+    return {{{DataType::kHALF, TensorFormat::kHWC8},
+             {DataType::kFLOAT, TensorFormat::kLINEAR}},
+            {DataType::kINT32, TensorFormat::kLINEAR}, 1};
+}
+
+int main(int argc, char** argv)
+{
+    assert(argc == 2);
+    Probe plugin(mixed_io());
+    DynamicPluginTensorDesc descriptors[3]{};
+    DataType types[] = {DataType::kHALF, DataType::kFLOAT, DataType::kINT32};
+    TensorFormat formats[] = {TensorFormat::kHWC8, TensorFormat::kLINEAR,
+                             TensorFormat::kLINEAR};
+    for (int i = 0; i < 3; ++i) {
+        descriptors[i].desc.type = types[i];
+        descriptors[i].desc.format = formats[i];
+        descriptors[i].desc.dims = Dims4(1, 48, 3, 5);
+    }
+    if (std::strcmp(argv[1], "formats") == 0) {
+        for (int i = 0; i < 3; ++i) {
+            assert(plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.type = DataType::kINT8;
+            assert(!plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.type = types[i];
+            descriptors[i].desc.format = TensorFormat::kCHW32;
+            assert(!plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.format = formats[i];
+        }
+        // TensorRT has not initialized descriptors beyond the requested position.
+        descriptors[1].desc.type = DataType::kINT8;
+        descriptors[2].desc.format = TensorFormat::kCHW32;
+        assert(plugin.supportsFormatCombination(0, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(-1, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(3, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(0, nullptr, 2, 1));
+    } else if (std::strcmp(argv[1], "counts") == 0) {
+        DataType output{};
+        PluginTensorDesc runtime[] = {descriptors[0].desc, descriptors[1].desc};
+        assert(plugin.getNbOutputs() == 1);
+        for (int count : {-1, 0, 1, 3}) {
+            assert(!plugin.supportsFormatCombination(0, descriptors, count, 1));
+            assert(plugin.getOutputDataTypes(&output, 1, types, count) != 0);
+            assert(plugin.configurePlugin(descriptors, count, descriptors + 2, 1) != 0);
+            assert(plugin.onShapeChange(runtime, count, &descriptors[2].desc, 1) != 0);
+        }
+        for (int count : {-1, 0, 2}) {
+            assert(!plugin.supportsFormatCombination(0, descriptors, 2, count));
+            assert(plugin.getOutputDataTypes(&output, count, types, 2) != 0);
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, count) != 0);
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, count) != 0);
+        }
+        assert(plugin.getOutputDataTypes(nullptr, 1, types, 2) != 0);
+        assert(plugin.getOutputDataTypes(&output, 1, nullptr, 2) != 0);
+        assert(plugin.configurePlugin(nullptr, 2, descriptors + 2, 1) != 0);
+        assert(plugin.configurePlugin(descriptors, 2, nullptr, 1) != 0);
+        assert(plugin.onShapeChange(nullptr, 2, &descriptors[2].desc, 1) != 0);
+        assert(plugin.onShapeChange(runtime, 2, nullptr, 1) != 0);
+    } else if (std::strcmp(argv[1], "types") == 0) {
+        DataType output{};
+        assert(plugin.getOutputDataTypes(&output, 1, types, 2) == 0);
+        assert(output == DataType::kINT32);
+        for (int i = 0; i < 2; ++i) {
+            auto previous = types[i];
+            types[i] = DataType::kINT8;
+            assert(plugin.getOutputDataTypes(&output, 1, types, 2) != 0);
+            types[i] = previous;
+        }
+    } else if (std::strcmp(argv[1], "shapes") == 0) {
+        assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) == 0);
+        PluginTensorDesc runtime[] = {descriptors[0].desc, descriptors[1].desc};
+        assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) == 0);
+        descriptors[1].desc.dims.d[2] = 7;
+        runtime[1] = descriptors[1].desc;
+        assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+        assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+        descriptors[1].desc.dims.d[2] = 3;
+        runtime[1] = descriptors[1].desc;
+        for (int i = 0; i < 3; ++i) {
+            descriptors[i].desc.type = DataType::kINT8;
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+            descriptors[i].desc.type = types[i];
+            descriptors[i].desc.format = TensorFormat::kCHW32;
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+            descriptors[i].desc.format = formats[i];
+            auto saved = i < 2 ? runtime[i] : descriptors[2].desc;
+            auto& changed = i < 2 ? runtime[i] : descriptors[2].desc;
+            changed.type = DataType::kINT8;
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+            changed = saved;
+            changed.format = TensorFormat::kCHW32;
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+            changed = saved;
+        }
+    } else if (std::strcmp(argv[1], "defaults") == 0) {
+        Probe default_plugin;
+        descriptors[0].desc.type = descriptors[1].desc.type = DataType::kFLOAT;
+        descriptors[0].desc.format = descriptors[1].desc.format = TensorFormat::kLINEAR;
+        assert(default_plugin.getNbOutputs() == 1);
+        assert(default_plugin.supportsFormatCombination(0, descriptors, 1, 1));
+        assert(default_plugin.supportsFormatCombination(1, descriptors, 1, 1));
+        assert(!default_plugin.supportsFormatCombination(0, descriptors, 2, 1));
+        DataType input = DataType::kFLOAT, output{};
+        assert(default_plugin.getOutputDataTypes(&output, 1, &input, 1) == 0);
+        assert(output == DataType::kFLOAT);
+        assert(default_plugin.configurePlugin(descriptors, 1, descriptors + 1, 1) == 0);
+        assert(default_plugin.onShapeChange(&descriptors[0].desc, 1,
+            &descriptors[1].desc, 1) == 0);
+        descriptors[0].desc.type = DataType::kHALF;
+        assert(!default_plugin.supportsFormatCombination(0, descriptors, 1, 1));
+    } else if (std::strcmp(argv[1], "invalid") == 0) {
+        for (int shape_input : {-1, 2}) {
+            auto io = mixed_io();
+            io.shape_input_index = shape_input;
+            bool rejected = false;
+            try { Probe invalid(io); }
+            catch (std::invalid_argument const&) { rejected = true; }
+            assert(rejected);
+        }
+        auto io = mixed_io();
+        io.inputs.clear();
+        bool rejected = false;
+        try { Probe invalid(io); }
+        catch (std::invalid_argument const&) { rejected = true; }
+        assert(rejected);
+    } else { return 1; }
+}

tensoracle-0.4.1/tests/cubin_io/kernels.cu

@@ -0,0 +1,11 @@
+#include <cuda_fp16.h>
+
+// All tensors use the same packed HWC8 storage. Channels are padded by the caller.
+extern "C" __global__ void half_io(__half const* x, __half const* residual,
+    __half* output, int count, int mode)
+{
+    int index = blockIdx.x * blockDim.x + threadIdx.x;
+    if (index >= count) return;
+    output[index] = mode == 0 ? __hadd(x[index], x[index])
+        : mode == 1 ? __hadd(x[index], residual[index]) : residual[index];
+}