tensoracle 0.3.2__tar.gz → 0.4.0__tar.gz

{tensoracle-0.3.2 → tensoracle-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tensoracle
-Version: 0.3.2
+Version: 0.4.0
 Summary: Build, save, reuse, and execute TensorRT engines from safetensors weights
 Project-URL: documentation, https://sync-dev-org.github.io/tensoracle/
 Project-URL: repository, https://github.com/sync-dev-org/tensoracle
@@ -138,7 +138,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -269,6 +269,23 @@ pip install 'tensoracle[producer]'
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.
 
+`tensoracle::cubin::PluginBase` accepts a `PluginIO` declaration as its fourth constructor argument. `inputs` lists one `TensorSpec` per input in network order; each spec declares a TensorRT `type` and `format`. `output` declares the single output's type and format, and `shape_input_index` selects the input whose shape the output copies. For example, declare two fp16 / HWC8 inputs and one fp16 / HWC8 output with the shape of input 0:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor. For one fp16 / HWC8 input, use `PluginIO{{half}, half, 0}`. The base class checks tensor counts, types, and formats during format negotiation, configuration, and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check the dimensions of every input in declaration order and the output; configuration may include wildcard dimensions (`-1`). For a residual operation, this callback can reject different shapes for its two inputs. The caller's kernel must handle the declared storage format. Omitting the fourth argument keeps one fp32 / LINEAR input and output with input 0's shape, so `PluginBase(name, plugin_namespace, fields)` remains valid. An empty input list or an invalid `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 `tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments

{tensoracle-0.3.2 → tensoracle-0.4.0}/README.md

@@ -108,7 +108,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under `~/.cache` when that variable is unset. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, precision, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -239,6 +239,23 @@ pip install 'tensoracle[producer]'
 
 The wheel also contains `tensoracle/cpp/include`, `tensoracle/cpp/src`, and `tensoracle/cpp/cmake`. Pass the installed `tensoracle/cpp` directory to CMake as `TENSORACLE_CPP_ROOT` and include `TensoracleCubinPlugin.cmake`. Its `tensoracle_add_cubin_plugin` function compiles the common C++ source, the caller's plugin source, and the generated `bundle.cpp` into the caller's library. The caller implements the graph's meaning, field meanings, weight placement, launch sequence, and variant selection. The common source owns typed fields in their original order, handles `IPluginV3` lifecycle methods, checks cubin digests and resources, loads one module per CUDA context, checks the generated argument count and specialization rules, appends the two supported zero scratch pointers, and launches kernels. The caller selects a variant using `Kernel::accepts` and derives the count passed to `Kernel::launch` from its argument array. The common source releases modules and device allocations in their owning contexts. The library must export TensorRT's `getCreators`; a static registration constructor is not supported.
 
+`tensoracle::cubin::PluginBase` accepts a `PluginIO` declaration as its fourth constructor argument. `inputs` lists one `TensorSpec` per input in network order; each spec declares a TensorRT `type` and `format`. `output` declares the single output's type and format, and `shape_input_index` selects the input whose shape the output copies. For example, declare two fp16 / HWC8 inputs and one fp16 / HWC8 output with the shape of input 0:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor. For one fp16 / HWC8 input, use `PluginIO{{half}, half, 0}`. The base class checks tensor counts, types, and formats during format negotiation, configuration, and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check the dimensions of every input in declaration order and the output; configuration may include wildcard dimensions (`-1`). For a residual operation, this callback can reject different shapes for its two inputs. The caller's kernel must handle the declared storage format. Omitting the fourth argument keeps one fp32 / LINEAR input and output with input 0's shape, so `PluginBase(name, plugin_namespace, fields)` remains valid. An empty input list or an invalid `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 `tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments

{tensoracle-0.3.2 → tensoracle-0.4.0}/cpp/include/tensoracle/cubin_plugin.hpp

@@ -41,13 +41,31 @@ private:
     nvinfer1::PluginFieldCollection collection_{};
 };
 
-// Implements TensorRT V3 lifecycle and the format/serialization boilerplate.
+struct TensorSpec {
+    nvinfer1::DataType type{nvinfer1::DataType::kFLOAT};
+    nvinfer1::TensorFormat format{nvinfer1::TensorFormat::kLINEAR};
+};
+
+// Declares the input count, each tensor's type and format, and the input whose
+// shape is copied to the single output. For configurations this cannot express
+// (multiple outputs or computed output shapes), developers can override virtual
+// V3 methods or ask tensoracle's maintainers to extend the declaration.
+struct PluginIO {
+    std::vector<TensorSpec> inputs{TensorSpec{}};
+    TensorSpec output{};
+    std::int32_t shape_input_index{0};
+};
+
+// Implements TensorRT V3 lifecycle and format/serialization boilerplate using
+// a PluginIO declaration. For other configurations, developers can override
+// virtual V3 methods or ask tensoracle's maintainers to extend the declaration.
 class PluginBase : public nvinfer1::IPluginV3,
                    public nvinfer1::IPluginV3OneCore,
                    public nvinfer1::IPluginV3OneBuild,
                    public nvinfer1::IPluginV3OneRuntime {
 public:
-    PluginBase(char const* name, char const* plugin_namespace, Fields fields);
+    // Throws std::invalid_argument for no inputs or an out-of-range shape input.
+    PluginBase(char const* name, char const* plugin_namespace, Fields fields, PluginIO io = {});
     ~PluginBase() noexcept override = default;
 
     nvinfer1::IPluginCapability* getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept override;
@@ -79,13 +97,16 @@ public:
 
 protected:
     Fields const& fields() const noexcept { return fields_; }
-    virtual std::int32_t validate_shape(nvinfer1::Dims const& input,
+    // Inputs are in declaration order; the array is valid only during this call.
+    // configurePlugin may provide wildcard (-1) dimensions.
+    virtual std::int32_t validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count,
         nvinfer1::Dims const& output) const noexcept = 0;
 
 private:
     char const* name_;
     char const* namespace_;
     Fields fields_;
+    PluginIO io_;
 };
 
 enum class ArgumentKind : std::uint8_t { kPointer, kInt32, kFloat32 };

{tensoracle-0.3.2 → tensoracle-0.4.0}/cpp/src/cubin_plugin.cpp

@@ -4,11 +4,48 @@
 #include <array>
 #include <cstring>
 #include <limits>
+#include <stdexcept>
 #include <utility>
 
 namespace tensoracle::cubin {
 namespace {
 
+nvinfer1::PluginTensorDesc const& tensor_desc(nvinfer1::PluginTensorDesc const& value) noexcept
+{
+    return value;
+}
+
+nvinfer1::PluginTensorDesc const& tensor_desc(nvinfer1::DynamicPluginTensorDesc const& value) noexcept
+{
+    return value.desc;
+}
+
+bool matches_tensor(nvinfer1::PluginTensorDesc const& descriptor, TensorSpec const& spec) noexcept
+{
+    return descriptor.type == spec.type && descriptor.format == spec.format;
+}
+
+template <typename Descriptor, typename Validate>
+std::int32_t validate_tensors(Descriptor const* inputs, std::int32_t input_count,
+    Descriptor const* outputs, std::int32_t output_count, PluginIO const& io,
+    Validate validate) noexcept
+{
+    if (inputs == nullptr || outputs == nullptr || input_count < 0 || output_count != 1 ||
+        static_cast<std::size_t>(input_count) != io.inputs.size()) return -1;
+    for (std::int32_t index = 0; index < input_count; ++index)
+        if (!matches_tensor(tensor_desc(inputs[index]), io.inputs[index])) return -1;
+    if (!matches_tensor(tensor_desc(outputs[0]), io.output)) return -1;
+    try {
+        std::vector<nvinfer1::Dims> dims;
+        dims.reserve(io.inputs.size());
+        for (std::int32_t index = 0; index < input_count; ++index)
+            dims.push_back(tensor_desc(inputs[index]).dims);
+        return validate(dims.data(), input_count, tensor_desc(outputs[0]).dims);
+    } catch (...) {
+        return -1;
+    }
+}
+
 std::size_t field_size(nvinfer1::PluginFieldType type) noexcept
 {
     using nvinfer1::PluginFieldType;
@@ -102,8 +139,14 @@ void Fields::rebuild()
 
 nvinfer1::PluginFieldCollection const* Fields::collection() noexcept { return &collection_; }
 
-PluginBase::PluginBase(char const* name, char const* plugin_namespace, Fields fields)
-    : name_(name), namespace_(plugin_namespace), fields_(std::move(fields)) {}
+PluginBase::PluginBase(char const* name, char const* plugin_namespace, Fields fields, PluginIO io)
+    : name_(name), namespace_(plugin_namespace), fields_(std::move(fields)), io_(std::move(io))
+{
+    if (io_.inputs.empty() || io_.inputs.size() > static_cast<std::size_t>(
+            std::numeric_limits<std::int32_t>::max()) || io_.shape_input_index < 0 ||
+        static_cast<std::size_t>(io_.shape_input_index) >= io_.inputs.size())
+        throw std::invalid_argument("PluginIO requires inputs and a valid shape_input_index");
+}
 
 nvinfer1::IPluginCapability* PluginBase::getCapabilityInterface(
     nvinfer1::PluginCapabilityType type) noexcept
@@ -124,20 +167,23 @@ bool PluginBase::supportsFormatCombination(std::int32_t position,
     nvinfer1::DynamicPluginTensorDesc const* in_out,
     std::int32_t input_count, std::int32_t output_count) noexcept
 {
-    if (in_out == nullptr || input_count != 1 || output_count != 1 || position < 0 || position > 1)
+    if (in_out == nullptr || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size() ||
+        output_count != 1 || position < 0 || position > input_count)
         return false;
-    auto const& descriptor = in_out[position].desc;
-    return descriptor.type == nvinfer1::DataType::kFLOAT &&
-        descriptor.format == nvinfer1::TensorFormat::kLINEAR;
+    auto const& spec = position < input_count ? io_.inputs[position] : io_.output;
+    return matches_tensor(in_out[position].desc, spec);
 }
 
 std::int32_t PluginBase::getOutputDataTypes(nvinfer1::DataType* output_types,
     std::int32_t output_count, nvinfer1::DataType const* input_types,
     std::int32_t input_count) const noexcept
 {
-    if (output_types == nullptr || input_types == nullptr || output_count != 1 || input_count != 1 ||
-        input_types[0] != nvinfer1::DataType::kFLOAT) return -1;
-    output_types[0] = nvinfer1::DataType::kFLOAT;
+    if (output_types == nullptr || input_types == nullptr || output_count != 1 || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size()) return -1;
+    for (std::int32_t index = 0; index < input_count; ++index)
+        if (input_types[index] != io_.inputs[index].type) return -1;
+    output_types[0] = io_.output.type;
     return 0;
 }
 
@@ -146,9 +192,10 @@ std::int32_t PluginBase::getOutputShapes(nvinfer1::DimsExprs const* inputs,
     nvinfer1::DimsExprs* outputs, std::int32_t output_count,
     nvinfer1::IExprBuilder&) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 ||
+    if (inputs == nullptr || outputs == nullptr || input_count < 0 ||
+        static_cast<std::size_t>(input_count) != io_.inputs.size() ||
         shape_input_count != 0 || output_count != 1) return -1;
-    outputs[0] = inputs[0];
+    outputs[0] = inputs[io_.shape_input_index];
     return 0;
 }
 
@@ -156,8 +203,10 @@ std::int32_t PluginBase::configurePlugin(nvinfer1::DynamicPluginTensorDesc const
     std::int32_t input_count, nvinfer1::DynamicPluginTensorDesc const* outputs,
     std::int32_t output_count) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 || output_count != 1) return -1;
-    return validate_shape(inputs[0].desc.dims, outputs[0].desc.dims);
+    return validate_tensors(inputs, input_count, outputs, output_count, io_,
+        [this](nvinfer1::Dims const* dims, std::int32_t count, nvinfer1::Dims const& output) {
+            return validate_shape(dims, count, output);
+        });
 }
 
 std::size_t PluginBase::getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const*, std::int32_t,
@@ -169,8 +218,10 @@ std::int32_t PluginBase::onShapeChange(nvinfer1::PluginTensorDesc const* inputs,
     std::int32_t input_count, nvinfer1::PluginTensorDesc const* outputs,
     std::int32_t output_count) noexcept
 {
-    if (inputs == nullptr || outputs == nullptr || input_count != 1 || output_count != 1) return -1;
-    return validate_shape(inputs[0].dims, outputs[0].dims);
+    return validate_tensors(inputs, input_count, outputs, output_count, io_,
+        [this](nvinfer1::Dims const* dims, std::int32_t count, nvinfer1::Dims const& output) {
+            return validate_shape(dims, count, output);
+        });
 }
 
 nvinfer1::IPluginV3* PluginBase::attachToContext(nvinfer1::IPluginResourceContext*) noexcept

{tensoracle-0.3.2 → tensoracle-0.4.0}/docs/index.md

@@ -70,7 +70,7 @@ handle = open_engine(
 )
 ```
 
-`my_model` represents the application's model package. On a valid cache hit, `open_engine` does not call `graph`; on a miss, it builds and saves a new engine. `open_engine` itself does not execute the engine or manage GPU buffers. Create an execution unit from its returned handle to enqueue inference. Saved engines live under `XDG_CACHE_HOME/tensoracle`, or `~/.cache/tensoracle` when `XDG_CACHE_HOME` is unset. See the [complete cache example](https://github.com/sync-dev-org/tensoracle#save-and-reopen-an-engine) for the declaration and expected input and output types.
+`my_model` represents the application's model package. On a valid cache hit, `open_engine` does not call `graph`; on a miss, it builds and saves a new engine. Raise `graph_version` when the graph code changes or when any other caller value that changes the engine contents but is not part of the cache identity changes, such as a plugin manifest or a build recipe. `open_engine` itself does not execute the engine or manage GPU buffers. Create an execution unit from its returned handle to enqueue inference. Saved engines live under `XDG_CACHE_HOME/tensoracle`, or `~/.cache/tensoracle` when `XDG_CACHE_HOME` is unset. See the [complete cache example](https://github.com/sync-dev-org/tensoracle#save-and-reopen-an-engine) for the declaration and expected input and output types.
 
 ### Execute an engine
 
@@ -175,6 +175,23 @@ pip install 'tensoracle[producer]'
 
 `tensoracle.producer.compile_bundle` uses the extra's Triton 3.8.0, `ptxas` 12.9.86, and `cuobjdump` 13.3 to produce cubins for sm86, sm89, and sm120. It finds the NVIDIA wheel executables by default and accepts explicit paths as overrides. Cubin production does not need a GPU or import torch; run it without importing torch first. Build the native plugin with a TensorRT 11 SDK, CUDA toolkit, CMake, and a C++17 compiler. Inspect the resulting bundle or wheel with `tensoracle.bundle`. The [cubin plugin example](https://github.com/sync-dev-org/tensoracle/tree/main/examples/cubin_plugin) covers rebuilding from a repository clone, packaging, and runtime use. Normal import, engine building, and engine reuse do not import the producer or Triton, and do not need the producer extra or native build tools.
 
+Pass a `tensoracle::cubin::PluginIO` declaration as the fourth argument to `PluginBase(name, plugin_namespace, fields, io)`. The `inputs` vector determines the input count and declares each input's TensorRT `type` and `format` through `TensorSpec`; `output` declares the single output's type and format. `shape_input_index` selects the input whose shape the output copies. This example declares two fp16 / HWC8 inputs and an fp16 / HWC8 output with input 0's shape:
+
+```cpp
+#include "tensoracle/cubin_plugin.hpp"
+
+tensoracle::cubin::PluginIO residual_io()
+{
+    using namespace tensoracle::cubin;
+    TensorSpec half{nvinfer1::DataType::kHALF, nvinfer1::TensorFormat::kHWC8};
+    return PluginIO{{half, half}, half, 0};
+}
+```
+
+Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in the derived plugin's constructor; use `PluginIO{{half}, half, 0}` for one fp16 / HWC8 input. The common implementation negotiates the declared types and formats, sets the output type and shape, and checks counts, types, and formats during configuration and shape changes. Implement `validate_shape(nvinfer1::Dims const* inputs, std::int32_t input_count, nvinfer1::Dims const& output) const noexcept` to check all input dimensions in declaration order and the output dimensions, including wildcard dimensions (`-1`) during configuration. A residual plugin can reject mismatched input shapes in this callback. The kernel must handle the declared storage format. Omitting `io` keeps one fp32 / LINEAR input and output with input 0's shape; the original three-argument constructor call remains valid. An empty input list or an out-of-range `shape_input_index` throws `std::invalid_argument` at construction.
+
+For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
+
 ## API reference
 
 - [Engine building, cache reuse, execution, and output comparison](api/index.md)

{tensoracle-0.3.2 → tensoracle-0.4.0}/docs/optimization.md

@@ -18,7 +18,7 @@ Use the same representative inputs, shapes, and corresponding outputs throughout
 
 First check whether a combination of standard TensorRT layers can preserve the operation's meaning, shapes, control flow, and output quality. For example, a deformable convolution may be expressible through grid sampling, mask multiplication, and convolution. If a plugin bundled with TensorRT matches the required behavior, compare that option too. Use a custom cubin plugin only when the suitable alternatives still leave a bottleneck.
 
-- **tensoracle tools:** `tensoracle.producer.compile_bundle` generates cubins and a manifest for supported GPU generations. The shared C++ plugin helpers support a caller-built native plugin, and `tensoracle.bundle.inspect_bundle` or `inspect_wheel` checks the cubins and their match with the plugin. Pass the plugin library through `plugins=` to `build_serialized_engine` or `open_engine`. Follow [Build a cubin plugin](https://github.com/sync-dev-org/tensoracle#build-a-cubin-plugin) and the [cubin plugin example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md) for the generation, build, and inspection operations.
+- **tensoracle tools:** `tensoracle.producer.compile_bundle` generates cubins and a manifest for supported GPU generations. The shared C++ plugin helpers support a caller-built native plugin with declared tensor types, formats, and an input whose shape the output copies; `tensoracle.bundle.inspect_bundle` or `inspect_wheel` checks the cubins and their match with the plugin. Pass the plugin library through `plugins=` to `build_serialized_engine` or `open_engine`. Follow [Build a cubin plugin](index.md#build-a-cubin-plugin) for the `PluginIO` declaration, including fp16 / HWC8 and two-input examples, and the [cubin plugin example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md) for the generation, build, and inspection operations.
 - **You provide:** Identify the bottleneck, write the kernel and model-specific plugin behavior, choose its supported shapes and types, build and package its native library, and run the resulting engines. An fp32 kernel must not use TF32.
 - **Gate:** For a custom cubin plugin, the bundle and plugin inspections must pass. Reapply stage 1 to the replacement's fp32 engine against the same fixed PyTorch reference. If using fp16, reapply stage 2 against an accepted fp32 engine with the same limits and model-specific measurements. Advance only when the replacement meets those quality gates; otherwise retain the accepted implementation.
 

{tensoracle-0.3.2 → tensoracle-0.4.0}/examples/cubin_plugin/plugin.cpp

@@ -117,7 +117,7 @@ public:
     {
         if (!valid_ || input_descriptors == nullptr || output_descriptors == nullptr ||
             inputs == nullptr || outputs == nullptr || inputs[0] == nullptr || outputs[0] == nullptr ||
-            validate_shape(input_descriptors[0].dims, output_descriptors[0].dims) != 0) return -1;
+            validate_shape(&input_descriptors[0].dims, 1, output_descriptors[0].dims) != 0) return -1;
         int rows = input_descriptors[0].dims.d[0];
         if (rows < 1 || rows > INT_MAX / 16) return -1;
         int const index = sm_index();
@@ -141,9 +141,10 @@ public:
     }
 
 protected:
-    std::int32_t validate_shape(Dims const& input, Dims const& output) const noexcept override
+    std::int32_t validate_shape(Dims const* inputs, std::int32_t input_count,
+        Dims const& output) const noexcept override
     {
-        return shape_matches(input, output, 32) ? 0 : -1;
+        return input_count == 1 && shape_matches(inputs[0], output, 32) ? 0 : -1;
     }
 
 private:
@@ -178,7 +179,7 @@ public:
     {
         if (!valid_ || input_descriptors == nullptr || output_descriptors == nullptr ||
             inputs == nullptr || outputs == nullptr || inputs[0] == nullptr || outputs[0] == nullptr ||
-            validate_shape(input_descriptors[0].dims, output_descriptors[0].dims) != 0) return -1;
+            validate_shape(&input_descriptors[0].dims, 1, output_descriptors[0].dims) != 0) return -1;
         int const rows = input_descriptors[0].dims.d[0];
         if (rows < 1 || rows > INT_MAX / 16) return -1;
         int const index = sm_index();
@@ -194,9 +195,10 @@ public:
     }
 
 protected:
-    std::int32_t validate_shape(Dims const& input, Dims const& output) const noexcept override
+    std::int32_t validate_shape(Dims const* inputs, std::int32_t input_count,
+        Dims const& output) const noexcept override
     {
-        return shape_matches(input, output, 16) ? 0 : -1;
+        return input_count == 1 && shape_matches(inputs[0], output, 16) ? 0 : -1;
     }
 
 private:

{tensoracle-0.3.2 → tensoracle-0.4.0}/src/tensoracle/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.3.2'
-__version_tuple__ = version_tuple = (0, 3, 2)
+__version__ = version = '0.4.0'
+__version_tuple__ = version_tuple = (0, 4, 0)
 
 __commit_id__ = commit_id = None

tensoracle-0.4.0/tests/cubin_io/check.cpp

@@ -0,0 +1,158 @@
+#include "tensoracle/cubin_plugin.hpp"
+
+#include <cassert>
+#include <cstring>
+#include <stdexcept>
+#include <utility>
+
+using namespace nvinfer1;
+using namespace tensoracle::cubin;
+
+class Probe final : public PluginBase {
+public:
+    explicit Probe(PluginIO io = {}) : PluginBase("probe", "tests", {}, std::move(io)) {}
+    IPluginV3* clone() noexcept override { return nullptr; }
+    int32_t enqueue(PluginTensorDesc const*, PluginTensorDesc const*, void const* const*,
+        void* const*, void*, cudaStream_t) noexcept override { return -1; }
+
+protected:
+    int32_t validate_shape(Dims const* inputs, int32_t count,
+        Dims const& output) const noexcept override
+    {
+        for (int32_t i = 0; i < count; ++i) {
+            if (inputs[i].nbDims != output.nbDims) return -1;
+            for (int32_t j = 0; j < output.nbDims; ++j)
+                if (inputs[i].d[j] != output.d[j]) return -1;
+        }
+        return 0;
+    }
+};
+
+PluginIO mixed_io()
+{
+    return {{{DataType::kHALF, TensorFormat::kHWC8},
+             {DataType::kFLOAT, TensorFormat::kLINEAR}},
+            {DataType::kINT32, TensorFormat::kLINEAR}, 1};
+}
+
+int main(int argc, char** argv)
+{
+    assert(argc == 2);
+    Probe plugin(mixed_io());
+    DynamicPluginTensorDesc descriptors[3]{};
+    DataType types[] = {DataType::kHALF, DataType::kFLOAT, DataType::kINT32};
+    TensorFormat formats[] = {TensorFormat::kHWC8, TensorFormat::kLINEAR,
+                             TensorFormat::kLINEAR};
+    for (int i = 0; i < 3; ++i) {
+        descriptors[i].desc.type = types[i];
+        descriptors[i].desc.format = formats[i];
+        descriptors[i].desc.dims = Dims4(1, 48, 3, 5);
+    }
+    if (std::strcmp(argv[1], "formats") == 0) {
+        for (int i = 0; i < 3; ++i) {
+            assert(plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.type = DataType::kINT8;
+            assert(!plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.type = types[i];
+            descriptors[i].desc.format = TensorFormat::kCHW32;
+            assert(!plugin.supportsFormatCombination(i, descriptors, 2, 1));
+            descriptors[i].desc.format = formats[i];
+        }
+        // TensorRT has not initialized descriptors beyond the requested position.
+        descriptors[1].desc.type = DataType::kINT8;
+        descriptors[2].desc.format = TensorFormat::kCHW32;
+        assert(plugin.supportsFormatCombination(0, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(-1, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(3, descriptors, 2, 1));
+        assert(!plugin.supportsFormatCombination(0, nullptr, 2, 1));
+    } else if (std::strcmp(argv[1], "counts") == 0) {
+        DataType output{};
+        PluginTensorDesc runtime[] = {descriptors[0].desc, descriptors[1].desc};
+        assert(plugin.getNbOutputs() == 1);
+        for (int count : {-1, 0, 1, 3}) {
+            assert(!plugin.supportsFormatCombination(0, descriptors, count, 1));
+            assert(plugin.getOutputDataTypes(&output, 1, types, count) != 0);
+            assert(plugin.configurePlugin(descriptors, count, descriptors + 2, 1) != 0);
+            assert(plugin.onShapeChange(runtime, count, &descriptors[2].desc, 1) != 0);
+        }
+        for (int count : {-1, 0, 2}) {
+            assert(!plugin.supportsFormatCombination(0, descriptors, 2, count));
+            assert(plugin.getOutputDataTypes(&output, count, types, 2) != 0);
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, count) != 0);
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, count) != 0);
+        }
+        assert(plugin.getOutputDataTypes(nullptr, 1, types, 2) != 0);
+        assert(plugin.getOutputDataTypes(&output, 1, nullptr, 2) != 0);
+        assert(plugin.configurePlugin(nullptr, 2, descriptors + 2, 1) != 0);
+        assert(plugin.configurePlugin(descriptors, 2, nullptr, 1) != 0);
+        assert(plugin.onShapeChange(nullptr, 2, &descriptors[2].desc, 1) != 0);
+        assert(plugin.onShapeChange(runtime, 2, nullptr, 1) != 0);
+    } else if (std::strcmp(argv[1], "types") == 0) {
+        DataType output{};
+        assert(plugin.getOutputDataTypes(&output, 1, types, 2) == 0);
+        assert(output == DataType::kINT32);
+        for (int i = 0; i < 2; ++i) {
+            auto previous = types[i];
+            types[i] = DataType::kINT8;
+            assert(plugin.getOutputDataTypes(&output, 1, types, 2) != 0);
+            types[i] = previous;
+        }
+    } else if (std::strcmp(argv[1], "shapes") == 0) {
+        assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) == 0);
+        PluginTensorDesc runtime[] = {descriptors[0].desc, descriptors[1].desc};
+        assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) == 0);
+        descriptors[1].desc.dims.d[2] = 7;
+        runtime[1] = descriptors[1].desc;
+        assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+        assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+        descriptors[1].desc.dims.d[2] = 3;
+        runtime[1] = descriptors[1].desc;
+        for (int i = 0; i < 3; ++i) {
+            descriptors[i].desc.type = DataType::kINT8;
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+            descriptors[i].desc.type = types[i];
+            descriptors[i].desc.format = TensorFormat::kCHW32;
+            assert(plugin.configurePlugin(descriptors, 2, descriptors + 2, 1) != 0);
+            descriptors[i].desc.format = formats[i];
+            auto saved = i < 2 ? runtime[i] : descriptors[2].desc;
+            auto& changed = i < 2 ? runtime[i] : descriptors[2].desc;
+            changed.type = DataType::kINT8;
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+            changed = saved;
+            changed.format = TensorFormat::kCHW32;
+            assert(plugin.onShapeChange(runtime, 2, &descriptors[2].desc, 1) != 0);
+            changed = saved;
+        }
+    } else if (std::strcmp(argv[1], "defaults") == 0) {
+        Probe default_plugin;
+        descriptors[0].desc.type = descriptors[1].desc.type = DataType::kFLOAT;
+        descriptors[0].desc.format = descriptors[1].desc.format = TensorFormat::kLINEAR;
+        assert(default_plugin.getNbOutputs() == 1);
+        assert(default_plugin.supportsFormatCombination(0, descriptors, 1, 1));
+        assert(default_plugin.supportsFormatCombination(1, descriptors, 1, 1));
+        assert(!default_plugin.supportsFormatCombination(0, descriptors, 2, 1));
+        DataType input = DataType::kFLOAT, output{};
+        assert(default_plugin.getOutputDataTypes(&output, 1, &input, 1) == 0);
+        assert(output == DataType::kFLOAT);
+        assert(default_plugin.configurePlugin(descriptors, 1, descriptors + 1, 1) == 0);
+        assert(default_plugin.onShapeChange(&descriptors[0].desc, 1,
+            &descriptors[1].desc, 1) == 0);
+        descriptors[0].desc.type = DataType::kHALF;
+        assert(!default_plugin.supportsFormatCombination(0, descriptors, 1, 1));
+    } else if (std::strcmp(argv[1], "invalid") == 0) {
+        for (int shape_input : {-1, 2}) {
+            auto io = mixed_io();
+            io.shape_input_index = shape_input;
+            bool rejected = false;
+            try { Probe invalid(io); }
+            catch (std::invalid_argument const&) { rejected = true; }
+            assert(rejected);
+        }
+        auto io = mixed_io();
+        io.inputs.clear();
+        bool rejected = false;
+        try { Probe invalid(io); }
+        catch (std::invalid_argument const&) { rejected = true; }
+        assert(rejected);
+    } else { return 1; }
+}

tensoracle-0.4.0/tests/cubin_io/kernels.cu

@@ -0,0 +1,11 @@
+#include <cuda_fp16.h>
+
+// All tensors use the same packed HWC8 storage. Channels are padded by the caller.
+extern "C" __global__ void half_io(__half const* x, __half const* residual,
+    __half* output, int count, int mode)
+{
+    int index = blockIdx.x * blockDim.x + threadIdx.x;
+    if (index >= count) return;
+    output[index] = mode == 0 ? __hadd(x[index], x[index])
+        : mode == 1 ? __hadd(x[index], residual[index]) : residual[index];
+}

tensoracle-0.4.0/tests/cubin_io/plugin.cpp

@@ -0,0 +1,106 @@
+#include "tensoracle/cubin_plugin.hpp"
+#include "kernel_bytes.hpp"
+
+#include <climits>
+#include <new>
+#include <vector>
+
+using namespace nvinfer1;
+using namespace tensoracle::cubin;
+
+namespace {
+char const* const kNames[] = {"TensoracleHalfUnary", "TensoracleHalfResidual", "TensoracleHalfSelect"};
+constexpr char kNamespace[] = "tensoracle.tests.io";
+
+PluginIO declaration(int mode)
+{
+    TensorSpec half{DataType::kHALF, TensorFormat::kHWC8};
+    return {std::vector<TensorSpec>(mode == 0 ? 1 : 2, half), half, mode == 2 ? 1 : 0};
+}
+
+class HalfPlugin final : public PluginBase {
+public:
+    explicit HalfPlugin(int mode) : PluginBase(kNames[mode], kNamespace, {}, declaration(mode)), mode_(mode) {}
+    IPluginV3* clone() noexcept override
+    {
+        try { return new HalfPlugin(mode_); }
+        catch (...) { return nullptr; }
+    }
+    int32_t enqueue(PluginTensorDesc const* input_desc, PluginTensorDesc const* output_desc,
+        void const* const* inputs, void* const* outputs, void*, cudaStream_t stream) noexcept override
+    {
+        int input_count = mode_ == 0 ? 1 : 2;
+        Dims dims[2] = {input_desc[0].dims, input_desc[input_count - 1].dims};
+        if (validate_shape(dims, input_count, output_desc[0].dims) != 0) return -1;
+        auto const& shape = output_desc[0].dims;
+        int64_t count = shape.d[0] * ((shape.d[1] + 7) / 8 * 8) * shape.d[2] * shape.d[3];
+        if (count <= 0 || count > INT_MAX) return -1;
+        int size = static_cast<int>(count);
+        void const* x = inputs[0];
+        void const* residual = inputs[input_count - 1];
+        void* output = outputs[0];
+        void* arguments[] = {&x, &residual, &output, &size, &mode_};
+        CUmodule module{};
+        CUfunction function{};
+        auto status = cuModuleLoadData(&module, kKernelBytes);
+        if (status == CUDA_SUCCESS) status = cuModuleGetFunction(&function, module, "half_io");
+        auto native_stream = reinterpret_cast<CUstream>(stream);
+        if (status == CUDA_SUCCESS)
+            status = cuLaunchKernel(function, (size + 127) / 128, 1, 1, 128, 1, 1, 0,
+                native_stream, arguments, nullptr);
+        if (status == CUDA_SUCCESS) status = cuStreamSynchronize(native_stream);
+        if (module) static_cast<void>(cuModuleUnload(module));
+        return status == CUDA_SUCCESS ? 0 : -1;
+    }
+
+protected:
+    int32_t validate_shape(Dims const* inputs, int32_t input_count,
+        Dims const& output) const noexcept override
+    {
+        if (input_count != (mode_ == 0 ? 1 : 2) || output.nbDims != 4) return -1;
+        for (int i = mode_ == 2 ? 1 : 0; i < input_count; ++i) {
+            if (inputs[i].nbDims != 4) return -1;
+            for (int j = 0; j < 4; ++j)
+                if (inputs[i].d[j] != -1 && output.d[j] != -1 &&
+                    inputs[i].d[j] != output.d[j]) return -1;
+        }
+        return 0;
+    }
+
+private:
+    int mode_;
+};
+
+class Creator final : public IPluginCreatorV3One {
+public:
+    explicit Creator(int mode) : mode_(mode) {}
+    char const* getPluginName() const noexcept override { return kNames[mode_]; }
+    char const* getPluginVersion() const noexcept override { return "1"; }
+    char const* getPluginNamespace() const noexcept override { return kNamespace; }
+    PluginFieldCollection const* getFieldNames() noexcept override { return &fields_; }
+    IPluginV3* createPlugin(char const*, PluginFieldCollection const* fields,
+        TensorRTPhase) noexcept override
+    {
+        if (fields == nullptr || fields->nbFields != 0) return nullptr;
+        try { return new HalfPlugin(mode_); }
+        catch (...) { return nullptr; }
+    }
+private:
+    int mode_;
+    PluginFieldCollection fields_{};
+};
+Creator gUnary(0), gResidual(1), gSelect(2);
+}
+
+#if defined(_WIN32)
+#define IO_EXPORT __declspec(dllexport)
+#else
+#define IO_EXPORT __attribute__((visibility("default")))
+#endif
+extern "C" IO_EXPORT void setLoggerFinder(ILoggerFinder*) noexcept {}
+extern "C" IO_EXPORT IPluginCreatorInterface* const* getCreators(int32_t& count) noexcept
+{
+    static IPluginCreatorInterface* creators[] = {&gUnary, &gResidual, &gSelect};
+    count = 3;
+    return creators;
+}

tensoracle-0.4.0/tests/plugin_io_runtime.py

@@ -0,0 +1,35 @@
+"""Build one invalid plugin graph in a separate TensorRT process."""
+
+import sys
+from pathlib import Path
+
+import tensorrt as trt
+
+from test_plugin_io import _plan
+
+
+def main():
+    plugin, fault = sys.argv[1:]
+    shapes = {"x": (1, 48, 3, 5), "residual": (1, 48, 3, 5)}
+    if fault == "missing-input":
+        shapes.pop("residual")
+    elif fault == "extra-input":
+        shapes["extra"] = (1, 48, 3, 5)
+    elif fault == "shape":
+        shapes["residual"] = (1, 48, 4, 5)
+    try:
+        _plan(
+            Path(plugin), "Residual", shapes, dtype=trt.float32 if fault == "type" else trt.float16
+        )
+    except RuntimeError as error:
+        assert str(error) in {
+            "TensorRT rejected the plugin IO declaration",
+            "TensorRT failed to build the serialized engine",
+        }, str(error)
+        print("declaration rejected")
+    else:
+        raise AssertionError("TensorRT built a network outside the plugin declaration")
+
+
+if __name__ == "__main__":
+    main()

tensoracle-0.4.0/tests/test_plugin_io.py

@@ -0,0 +1,230 @@
+"""Native cubin plugins declare tensor types, formats, and their output shape source."""
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import numpy as np
+import pytest
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+from tensoracle import BuilderSettings, EngineBuild, InputShape, build_serialized_engine
+
+ROOT = Path(__file__).resolve().parents[1]
+FIXTURES = ROOT / "tests/cubin_io"
+
+
+@pytest.fixture(scope="module")
+def native_tools():
+    sdk = os.environ.get("TENSORACLE_SDK_ROOT")
+    cuda = os.environ.get("TENSORACLE_CUDA_ROOT")
+    if not sdk or not cuda:
+        pytest.skip("TensorRT SDK and CUDA toolkit were not supplied")
+    return Path(sdk), Path(cuda)
+
+
+def _compile(source, output, native_tools, *options):
+    sdk, cuda = native_tools
+    result = subprocess.run(
+        [
+            "c++",
+            "-std=c++17",
+            "-Wno-deprecated-declarations",
+            f"-I{ROOT / 'cpp/include'}",
+            f"-I{sdk / 'include'}",
+            f"-I{cuda / 'include'}",
+            str(source),
+            str(ROOT / "cpp/src/cubin_plugin.cpp"),
+            f"-L{cuda / 'lib64/stubs'}",
+            "-lcuda",
+            *options,
+            "-o",
+            str(output),
+        ],
+        check=False,
+        capture_output=True,
+        text=True,
+        timeout=45,
+    )
+    assert result.returncode == 0, result.stderr
+
+
+@pytest.fixture(scope="module")
+def native_contract(native_tools, tmp_path_factory):
+    work = tmp_path_factory.mktemp("plugin-io-contract")
+    executable = work / "check"
+    _compile(FIXTURES / "check.cpp", executable, native_tools)
+    return executable
+
+
+@pytest.mark.req("REQ-TENSORACLE-003")
+@pytest.mark.parametrize("case", ["formats", "counts", "types", "shapes", "defaults", "invalid"])
+def test_native_plugin_checks_declared_io(native_contract, case):
+    """Plugins enforce each tensor's declared format, type, count, and shape validation."""
+    subprocess.run([str(native_contract), case], check=True, timeout=10)
+
+
+def _cuda(result):
+    status, *values = result
+    assert status == cudart.cudaError_t.cudaSuccess, status
+    return values[0] if values else None
+
+
+@pytest.fixture(scope="module")
+def half_plugin(native_tools, tmp_path_factory):
+    _, cuda = native_tools
+    work = tmp_path_factory.mktemp("plugin-half-io")
+    device = _cuda(cudart.cudaGetDevice())
+    properties = _cuda(cudart.cudaGetDeviceProperties(device))
+    sm = properties.major * 10 + properties.minor
+    cubin = work / "half_io.cubin"
+    subprocess.run(
+        [
+            str(cuda / "bin/nvcc"),
+            "--cubin",
+            f"-arch=sm_{sm}",
+            str(FIXTURES / "kernels.cu"),
+            "-o",
+            str(cubin),
+        ],
+        check=True,
+        capture_output=True,
+        text=True,
+        timeout=45,
+    )
+    (work / "kernel_bytes.hpp").write_text(
+        "alignas(16) unsigned char const kKernelBytes[] = {"
+        + ",".join(str(byte) for byte in cubin.read_bytes())
+        + "};\n",
+        encoding="utf-8",
+    )
+    plugin = work / "libhalf_io.so"
+    _compile(FIXTURES / "plugin.cpp", plugin, native_tools, "-shared", "-fPIC", f"-I{work}")
+    return plugin
+
+
+def _plan(plugin, kind, shapes, *, dtype=trt.float16):
+    def graph(network):
+        creator = trt.get_plugin_registry().get_creator(
+            f"TensoracleHalf{kind}", "1", "tensoracle.tests.io"
+        )
+        assert creator is not None
+        native = creator.create_plugin("io", trt.PluginFieldCollection([]), trt.TensorRTPhase.BUILD)
+        assert native is not None
+        inputs = [network.add_input(name, dtype, shape) for name, shape in shapes.items()]
+        layer = network.add_plugin_v3(inputs, [], native)
+        if layer is None:
+            raise RuntimeError("TensorRT rejected the plugin IO declaration")
+        result = layer.get_output(0)
+        result.name = "result"
+        network.mark_output(result)
+
+    return build_serialized_engine(
+        graph,
+        EngineBuild(
+            "fp16",
+            {name: InputShape.fixed(shape) for name, shape in shapes.items()},
+            BuilderSettings(workspace_limit=16 << 20, optimization_level=0),
+        ),
+        plugins=(plugin,),
+    )
+
+
+def _execute(plan, inputs):
+    logger = trt.Logger(trt.Logger.ERROR)
+    runtime = trt.Runtime(logger)
+    try:
+        engine = runtime.deserialize_cuda_engine(plan)
+        assert engine is not None
+        context = engine.create_execution_context()
+        assert context is not None
+        output = np.empty(tuple(context.get_tensor_shape("result")), dtype=np.float16)
+        buffers = {}
+        stream = _cuda(cudart.cudaStreamCreate())
+        try:
+            for name, value in {**inputs, "result": output}.items():
+                # Network I/O is LINEAR; TensorRT reformats to the plugin's HWC8 declaration.
+                assert engine.get_tensor_format(name) == trt.TensorFormat.LINEAR
+                buffers[name] = _cuda(cudart.cudaMalloc(value.nbytes))
+                assert context.set_tensor_address(name, int(buffers[name]))
+                if name in inputs:
+                    _cuda(
+                        cudart.cudaMemcpy(
+                            buffers[name],
+                            value.ctypes.data,
+                            value.nbytes,
+                            cudart.cudaMemcpyKind.cudaMemcpyHostToDevice,
+                        )
+                    )
+            assert context.execute_async_v3(int(stream))
+            _cuda(cudart.cudaStreamSynchronize(stream))
+            _cuda(
+                cudart.cudaMemcpy(
+                    output.ctypes.data,
+                    buffers["result"],
+                    output.nbytes,
+                    cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost,
+                )
+            )
+            return output
+        finally:
+            _cuda(cudart.cudaStreamSynchronize(stream))
+            del context, engine
+            for pointer in buffers.values():
+                _cuda(cudart.cudaFree(pointer))
+            _cuda(cudart.cudaStreamDestroy(stream))
+    finally:
+        del runtime
+
+
+@pytest.mark.gpu
+@pytest.mark.req("REQ-TENSORACLE-003")
+@pytest.mark.parametrize("kind", ["Unary", "Residual"])
+def test_half_hwc8_plugin_matches_numpy(half_plugin, kind, tmp_path, monkeypatch):
+    """One-input and two-input fp16 HWC8 plugins execute and match NumPy's output."""
+    monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache"))
+    shape = (1, 48, 3, 5)
+    x = (np.arange(np.prod(shape)).reshape(shape) % 31 / 8).astype(np.float16)
+    inputs = {"x": x}
+    expected = x + x
+    if kind == "Residual":
+        residual = (np.flip(x, axis=1) / 2).astype(np.float16).copy()
+        inputs["residual"] = residual
+        expected = x + residual
+    plan = _plan(half_plugin, kind, {name: value.shape for name, value in inputs.items()})
+    np.testing.assert_array_equal(_execute(plan, inputs), expected)
+
+
+@pytest.mark.gpu
+@pytest.mark.req("REQ-TENSORACLE-003")
+def test_plugin_output_shape_follows_selected_input(half_plugin, tmp_path, monkeypatch):
+    """A plugin can copy input one's shape when input zero has a different shape."""
+    monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache"))
+    inputs = {
+        "x": np.zeros((1, 48, 1, 1), dtype=np.float16),
+        "residual": np.arange(720, dtype=np.float16).reshape(1, 48, 3, 5),
+    }
+    plan = _plan(half_plugin, "Select", {name: value.shape for name, value in inputs.items()})
+    np.testing.assert_array_equal(_execute(plan, inputs), inputs["residual"])
+
+
+@pytest.mark.gpu
+@pytest.mark.req("REQ-TENSORACLE-003")
+@pytest.mark.parametrize("fault", ["missing-input", "extra-input", "shape", "type"])
+def test_plugin_rejects_network_outside_declaration(half_plugin, fault, tmp_path):
+    """A network with wrong input count, differing residual shapes, or wrong types fails."""
+    environment = dict(os.environ, XDG_CACHE_HOME=str(tmp_path / "cache"))
+    # Keep TensorRT's error state and any failed native graph in its own process.
+    result = subprocess.run(
+        [sys.executable, str(ROOT / "tests/plugin_io_runtime.py"), str(half_plugin), fault],
+        cwd=tmp_path,
+        env=environment,
+        check=False,
+        capture_output=True,
+        text=True,
+        timeout=45,
+    )
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert result.stdout.splitlines()[-1] == "declaration rejected"