tensoracle 1.0.1__tar.gz → 1.1.0__tar.gz

{tensoracle-1.0.1 → tensoracle-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tensoracle
-Version: 1.0.1
+Version: 1.1.0
 Summary: Build, save, reuse, and execute TensorRT engines from safetensors weights
 Project-URL: documentation, https://sync-dev-org.github.io/tensoracle/
 Project-URL: repository, https://github.com/sync-dev-org/tensoracle
@@ -255,10 +255,75 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
+### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ## Build a cubin plugin
 
 Follow [Prepare a TensorRT SDK](https://github.com/sync-dev-org/tensoracle/blob/main/docs/tensorrt-sdk.md) to acquire and verify the headers and native library for a build SDK, including the fixed inputs for SDK 11.2.1.2 and the `TENSORRT_ROOT` setup.

{tensoracle-1.0.1 → tensoracle-1.1.0}/README.md

@@ -224,10 +224,75 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
+### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ## Build a cubin plugin
 
 Follow [Prepare a TensorRT SDK](https://github.com/sync-dev-org/tensoracle/blob/main/docs/tensorrt-sdk.md) to acquire and verify the headers and native library for a build SDK, including the fixed inputs for SDK 11.2.1.2 and the `TENSORRT_ROOT` setup.

{tensoracle-1.0.1 → tensoracle-1.1.0}/docs/index.md

@@ -155,12 +155,77 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
 `TensorSpec` accepts optional `location=trt.TensorLocation.DEVICE` and `format=trt.TensorFormat.LINEAR`. `open_engine` checks these only when declared, together with tensor names, modes, dtypes, build-time shapes, and profiles. An `EngineIOError` names the tensor, field, observed value, expected value, and saved file path. The execution unit does not accept a second declaration.
 
+#### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ### Build a standard graph
 
 Define `graph(network)` with TensorRT's Python network API and load model weights from safetensors. Describe each graph input with `InputShape.fixed` or `InputShape.ranged`, and pass an `EngineBuild` declaration to `build_serialized_engine`. `EngineBuild` declares input shapes and builder settings; the graph function determines precision through the tensor types it places in the network. The supported precisions are fp32 (with TF32 disabled), fp16, bf16, int8, and fp8. The [bf16, int8, and fp8 guide](precision.md) shows tensor casts, explicit Q/DQ layers, and scales supplied through safetensors; fp8 targets sm89 and sm120. `open_engine` uses the same declaration plus a name, graph version, weight files, and `EngineIO` to save and reuse a checked engine. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. The [graph example](https://github.com/sync-dev-org/tensoracle#define-and-build-a-standard-graph) shows the exact calls.

{tensoracle-1.0.1 → tensoracle-1.1.0}/src/tensoracle/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.1'
-__version_tuple__ = version_tuple = (1, 0, 1)
+__version__ = version = '1.1.0'
+__version_tuple__ = version_tuple = (1, 1, 0)
 
 __commit_id__ = commit_id = None

tensoracle-1.1.0/tests/test_cuda_graphs.py

@@ -0,0 +1,191 @@
+"""Capture one enqueue and replay it with caller-owned CUDA graphs."""
+
+import json
+from contextlib import ExitStack, contextmanager
+
+import numpy as np
+import pytest
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+import tensoracle
+from test_build import _check_cuda
+from test_execution import _open_sum
+
+pytest_plugins = ("test_example_plugin",)
+
+
+@contextmanager
+def _captured(handle, batches, *, shapes):
+    """Keep one execution, its addresses, and its captured graph alive together."""
+    with handle.create_execution() as execution, ExitStack() as resources:
+        execution.set_input_shapes(shapes)
+        output = np.empty(
+            execution.get_tensor_shape("result"),
+            dtype=trt.nptype(execution.get_tensor_dtype("result")),
+        )
+        stream = _check_cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: _check_cuda(cudart.cudaStreamDestroy(stream)))
+        addresses = {}
+        for name, value in {**batches[0], "result": output}.items():
+            pointer = _check_cuda(cudart.cudaMalloc(value.nbytes))
+            resources.callback(lambda pointer=pointer: _check_cuda(cudart.cudaFree(pointer)))
+            addresses[name] = int(pointer)
+
+        def copy_inputs(batch):
+            for name, value in batch.items():
+                _check_cuda(
+                    cudart.cudaMemcpyAsync(
+                        addresses[name],
+                        value.ctypes.data,
+                        value.nbytes,
+                        cudart.cudaMemcpyKind.cudaMemcpyHostToDevice,
+                        stream,
+                    )
+                )
+
+        def read_output():
+            _check_cuda(
+                cudart.cudaMemcpyAsync(
+                    output.ctypes.data,
+                    addresses["result"],
+                    output.nbytes,
+                    cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost,
+                    stream,
+                )
+            )
+            _check_cuda(cudart.cudaStreamSynchronize(stream))
+            return output.copy()
+
+        try:
+            # The first ordinary enqueue flushes deferred shape and plugin initialization.
+            ordinary = []
+            for batch in batches:
+                copy_inputs(batch)
+                execution.enqueue(addresses, stream=int(stream))
+                ordinary.append(read_output())
+
+            _check_cuda(
+                cudart.cudaStreamBeginCapture(
+                    stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal
+                )
+            )
+            try:
+                execution.enqueue(addresses, stream=int(stream))
+            finally:
+                graph = _check_cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: _check_cuda(cudart.cudaGraphDestroy(graph)))
+            instance = _check_cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: _check_cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            def replay(index):
+                copy_inputs(batches[index])
+                # An empty graph or a launch writing to the wrong address must fail.
+                _check_cuda(
+                    cudart.cudaMemsetAsync(addresses["result"], 0xFF, output.nbytes, stream)
+                )
+                _check_cuda(cudart.cudaGraphLaunch(instance, stream))
+                actual = read_output()
+                np.testing.assert_array_equal(actual, ordinary[index])
+                return actual
+
+            yield replay
+        finally:
+            # Finish work before graphs, buffers, streams, and the execution are released.
+            _check_cuda(cudart.cudaStreamSynchronize(stream))
+
+
+def _sum_batches(rows):
+    return [
+        {
+            "features": np.arange(rows * 2, dtype=np.float32).reshape(rows, 2) / 4 + offset,
+            "extra": np.full((rows, 2), -0.25 - offset / 2, dtype=np.float32),
+        }
+        for offset in (0, 2)
+    ]
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_standard_engine_graph_replay_matches_ordinary_enqueue(tmp_path, monkeypatch):
+    """One captured enqueue on a nondefault stream reproduces ordinary sums for new input data."""
+    handle = _open_sum(tmp_path, monkeypatch, fixed=True)
+    batches = _sum_batches(3)
+    with _captured(handle, batches, shapes={}) as replay:
+        for index in (0, 1, 0):
+            np.testing.assert_array_equal(
+                replay(index), batches[index]["features"] + batches[index]["extra"]
+            )
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_plugin_engine_graph_replay_matches_ordinary_enqueue(installed_example, tmp_path):
+    """A captured cubin plugin runs repeatedly and matches ordinary enqueue with new inputs."""
+    library = installed_example["library"]
+    manifest = json.loads((library.parent / "bundle/bundle.json").read_text())
+    digest = np.frombuffer(manifest["bundle_sha256"].encode(), dtype=np.uint8)
+    bias = np.arange(16, dtype=np.float32) / 8
+
+    def graph(network):
+        creator = trt.get_plugin_registry().get_creator(
+            "TensoracleExampleBias", "1", "tensoracle.example"
+        )
+        assert creator is not None
+        fields = trt.PluginFieldCollection(
+            [
+                trt.PluginField("bundle_sha256", digest, trt.PluginFieldType.CHAR),
+                trt.PluginField("bias", bias, trt.PluginFieldType.FLOAT32),
+            ]
+        )
+        plugin = creator.create_plugin("bias", fields, trt.TensorRTPhase.BUILD)
+        assert plugin is not None
+        features = network.add_input("features", trt.float32, (3, 16))
+        result = network.add_plugin_v3([features], [], plugin).get_output(0)
+        result.name = "result"
+        network.mark_output(result)
+
+    plan = tensoracle.build_serialized_engine(
+        graph,
+        tensoracle.EngineBuild(
+            {"features": tensoracle.InputShape.fixed((3, 16))},
+            tensoracle.BuilderSettings(optimization_level=0),
+        ),
+        plugins=(library,),
+    )
+    logger = trt.Logger(trt.Logger.ERROR)
+    runtime = trt.Runtime(logger)
+    engine = runtime.deserialize_cuda_engine(plan)
+    assert engine is not None
+    handle = tensoracle.EngineHandle(engine, runtime, tmp_path / "bias.engine")
+    batches = [
+        {"features": np.arange(48, dtype=np.float32).reshape(3, 16) / 4 + offset}
+        for offset in (0, 2)
+    ]
+    with _captured(handle, batches, shapes={}) as replay:
+        for index in (0, 1, 0):
+            np.testing.assert_array_equal(replay(index), batches[index]["features"] + bias)
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_ranged_engine_keeps_a_graph_and_execution_per_shape(tmp_path, monkeypatch):
+    """Two shapes keep dedicated executions and graphs that replay correctly in alternation."""
+    handle = _open_sum(tmp_path, monkeypatch)
+    batches = [_sum_batches(rows) for rows in (2, 5)]
+    with ExitStack() as resources:
+        replays = [
+            resources.enter_context(
+                _captured(
+                    handle,
+                    values,
+                    shapes={name: value.shape for name, value in values[0].items()},
+                )
+            )
+            for values in batches
+        ]
+        for index in (0, 1, 0):
+            for replay, values in zip(replays, batches, strict=True):
+                np.testing.assert_array_equal(
+                    replay(index), values[index]["features"] + values[index]["extra"]
+                )