tensoracle 1.0.0__tar.gz → 1.1.0__tar.gz

{tensoracle-1.0.0 → tensoracle-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tensoracle
-Version: 1.0.0
+Version: 1.1.0
 Summary: Build, save, reuse, and execute TensorRT engines from safetensors weights
 Project-URL: documentation, https://sync-dev-org.github.io/tensoracle/
 Project-URL: repository, https://github.com/sync-dev-org/tensoracle
@@ -8,7 +8,7 @@ Project-URL: issues, https://github.com/sync-dev-org/tensoracle/issues
 Author-email: minamik <mia@sync.dev>
 License-Expression: MIT
 License-File: LICENSE
-Classifier: Development Status :: 4 - Beta
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: GPU :: NVIDIA CUDA :: 13
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -142,7 +142,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under the home directory's `.cache` when that variable is unset. The same rule applies on every OS; on Windows the home directory is `%USERPROFILE%`, so the default is `%USERPROFILE%\.cache\tensoracle`. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact. tensoracle closes a saved engine file after reading it, so another process using an opened engine does not block replacement. On Windows, a process that keeps the saved file itself open prevents replacement or removal with `PermissionError`; after that process closes the file, a file left empty by the failed rebuild is rejected as `CorruptEngineError` and can be rebuilt with `force_rebuild=True`.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under the home directory's `.cache` when that variable is unset. The same rule applies on every OS; on Windows the home directory is `%USERPROFILE%`, so the default is `%USERPROFILE%\.cache\tensoracle`. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, the GPU's compatibility properties, OS, CPU architecture, and TensorRT version. The GPU is the device of the current CUDA context, or device 0 when no context is current. Its properties are the product name, compute capability, memory bus width, L2 cache size, shared memory limits, texture alignment, multiprocessor count, whether the GPU is integrated, ECC mode, and total memory; another device with the same properties reuses the saved engine. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact. tensoracle closes a saved engine file after reading it, so another process using an opened engine does not block replacement. On Windows, a process that keeps the saved file itself open prevents replacement or removal with `PermissionError`; after that process closes the file, a file left empty by the failed rebuild is rejected as `CorruptEngineError` and can be rebuilt with `force_rebuild=True`.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -255,10 +255,75 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
+### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ## Build a cubin plugin
 
 Follow [Prepare a TensorRT SDK](https://github.com/sync-dev-org/tensoracle/blob/main/docs/tensorrt-sdk.md) to acquire and verify the headers and native library for a build SDK, including the fixed inputs for SDK 11.2.1.2 and the `TENSORRT_ROOT` setup.
@@ -302,7 +367,7 @@ Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in th
 
 For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
 
-`tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
+`tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records the Triton and ptxas versions and each kernel's source SHA-256, without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments
 

{tensoracle-1.0.0 → tensoracle-1.1.0}/README.md

@@ -111,7 +111,7 @@ handle = open_engine(
 engine = handle.engine  # Keep handle.runtime alive while using the engine.
 ```
 
-Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under the home directory's `.cache` when that variable is unset. The same rule applies on every OS; on Windows the home directory is `%USERPROFILE%`, so the default is `%USERPROFILE%\.cache\tensoracle`. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, GPU product name, OS, CPU architecture, and TensorRT version. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact. tensoracle closes a saved engine file after reading it, so another process using an opened engine does not block replacement. On Windows, a process that keeps the saved file itself open prevents replacement or removal with `PermissionError`; after that process closes the file, a file left empty by the failed rebuild is rejected as `CorruptEngineError` and can be rebuilt with `force_rebuild=True`.
+Saved files live in the `tensoracle` directory under `XDG_CACHE_HOME`, or under the home directory's `.cache` when that variable is unset. The same rule applies on every OS; on Windows the home directory is `%USERPROFILE%`, so the default is `%USERPROFILE%\.cache\tensoracle`. The cache identity includes the contents of all declared safetensors weight files and optional `plugins` files, the engine name, graph version, input shapes, builder settings, the GPU's compatibility properties, OS, CPU architecture, and TensorRT version. The GPU is the device of the current CUDA context, or device 0 when no context is current. Its properties are the product name, compute capability, memory bus width, L2 cache size, shared memory limits, texture alignment, multiprocessor count, whether the GPU is integrated, ECC mode, and total memory; another device with the same properties reuses the saved engine. Raise `graph_version` when graph code changes; without a version change, the previous engine is reused. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. Include in `graph_version` any other caller value that changes the engine contents but is not part of the cache identity, such as a plugin manifest or a build recipe. Pass `force_rebuild=True` to replace a cached engine even when it is intact. tensoracle closes a saved engine file after reading it, so another process using an opened engine does not block replacement. On Windows, a process that keeps the saved file itself open prevents replacement or removal with `PermissionError`; after that process closes the file, a file left empty by the failed rebuild is rejected as `CorruptEngineError` and can be rebuilt with `force_rebuild=True`.
 
 `plugins` names native TensorRT plugin libraries. Both `open_engine` and `build_serialized_engine` accept it. Each function checks that every library exports `getCreators`, loads it through TensorRT's registry, and retains it for the process before building the graph or deserializing an engine. Repeating the same library does not register it twice. `open_engine` also hashes the loaded library bytes into the engine identity; a changed library selects a different saved engine in a new process. A missing, unreadable, or incompatible library, or a library changed after loading in the same process, raises `PluginLibraryError` before a cached plan is opened. Keep the library and its cubin bundle together in the caller's wheel.
 
@@ -224,10 +224,75 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
+### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ## Build a cubin plugin
 
 Follow [Prepare a TensorRT SDK](https://github.com/sync-dev-org/tensoracle/blob/main/docs/tensorrt-sdk.md) to acquire and verify the headers and native library for a build SDK, including the fixed inputs for SDK 11.2.1.2 and the `TENSORRT_ROOT` setup.
@@ -271,7 +336,7 @@ Use `PluginBase(name, plugin_namespace, std::move(fields), residual_io())` in th
 
 For configurations the declaration cannot express, such as multiple outputs or computed output shapes, developers can override the virtual V3 methods in a derived class or ask tensoracle's maintainers to extend the declaration.
 
-`tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records Triton, ptxas, and kernel source digests without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
+`tensoracle.bundle.inspect_bundle(bundle_directory, plugin=library_path)` checks the manifest, each cubin's SHA-256 and actual ELF SM, the supported hidden arguments and zero scratch sizes, and the cubin bytes embedded in the library. `inspect_wheel(wheel_path, plugin="path/inside/wheel.so")` performs the same checks on a wheel without installing it. Their distinct exceptions identify a missing cubin, corrupt cubin, false SM claim, or plugin and bundle mismatch. The manifest records the Triton and ptxas versions and each kernel's source SHA-256, without a producer host or OS path. The TensorRT versions tested with the included example are listed below.
 
 ## Tested environments
 

{tensoracle-1.0.0 → tensoracle-1.1.0}/docs/api/index.md

@@ -12,6 +12,8 @@ The following names are available directly from `tensoracle`.
 
 ::: tensoracle._build.build_serialized_engine
 
+::: tensoracle._build.ShapeTensorOutputWarning
+
 ## Saved engines
 
 ::: tensoracle._cache.TensorSpec

{tensoracle-1.0.0 → tensoracle-1.1.0}/docs/index.md

@@ -155,12 +155,77 @@ with handle.create_execution() as execution:
 
 The caller selects the CUDA device and context, orders inputs written on other streams before enqueue, and keeps buffers, streams, plugin preparation resources, and the execution unit alive until completion. Use one execution unit serially; do not call it concurrently from multiple threads. Confirm completion before changing shapes or streams, reusing buffers, or closing. `close()` and context exit release the TensorRT context before the handle, without waiting, and repeated close calls do nothing. Operations on a closed unit raise `RuntimeError` before touching TensorRT. The unit does not retain caller arrays or expose its TensorRT context.
 
-Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. CUDA graphs, profile switching, and caller-owned TensorRT workspace memory are outside this API.
+Tensoracle neither allocates or frees device buffers, copies data, nor calls stream or device synchronization. TensorRT itself may allocate internal resources or act synchronously for some networks and plugins. Execution units reject HOST-location tensors, externally supplied shape tensor values, and data-dependent output sizes at creation. Ordinary networks that compute shapes internally from input dimensions are supported. Profile switching and caller-owned TensorRT workspace memory are outside this API.
 
 Networks with outputs marked as shape tensors emit `tensoracle.ShapeTensorOutputWarning` during building and when creating an execution unit (`create_execution()` or `EngineExecution(handle)`). The warning names the affected tensors: `create_execution()` cannot handle these engines, and calling it crashes the process. Building continues by default. To stop either operation safely, turn this warning into an exception with `warnings.filterwarnings("error", category=tensoracle.ShapeTensorOutputWarning)` after importing `warnings` and `tensoracle`. Saved engines retain the shape output names outside the cache conditions; older files without these names are reused without an execution warning.
 
 `TensorSpec` accepts optional `location=trt.TensorLocation.DEVICE` and `format=trt.TensorFormat.LINEAR`. `open_engine` checks these only when declared, together with tensor names, modes, dtypes, build-time shapes, and profiles. An `EngineIOError` names the tensor, field, observed value, expected value, and saved file path. The execution unit does not accept a second declaration.
 
+#### Capture and replay with CUDA Graphs
+
+Tensoracle supports recording one `EngineExecution.enqueue` call with CUDA stream capture and replaying the resulting graph. The application owns graph creation, recapture, and destruction; tensoracle provides no capture or replay operations.
+
+Use a dedicated execution unit (`handle.create_execution()`) for each graph. After setting input shapes, enqueue once and wait for completion before starting capture, so TensorRT finishes deferred initialization. Capture and replay on a nondefault stream; the example uses a nonblocking stream. Keep synchronous CUDA calls and work on the legacy default stream out of capture.
+
+The graph fixes the input shapes, tensor addresses, and execution unit. Keep the execution unit, its handle, buffers, and plugin preparation resources alive until the graph is destroyed and outstanding work has finished. To use different shapes or addresses, capture a new graph for that combination or select another graph with its own execution unit. Input contents may change at the same addresses between completed replays.
+
+For the `features` / `result` engine above, use the `cuda(...)` helper from the preceding example:
+
+```python
+from contextlib import ExitStack
+
+import numpy as np
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+with handle.create_execution() as execution:
+    execution.set_input_shapes({"features": (1, 4, 2)})
+    # Omit set_input_shapes for fixed inputs and single-point profiles.
+    features = np.arange(8, dtype=np.float32).reshape(1, 4, 2) / 4
+    output = np.empty(
+        execution.get_tensor_shape("result"),
+        dtype=trt.nptype(execution.get_tensor_dtype("result")),
+    )
+    with ExitStack() as resources:
+        stream = cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: cuda(cudart.cudaStreamDestroy(stream)))
+        source = cuda(cudart.cudaMalloc(features.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(source)))
+        target = cuda(cudart.cudaMalloc(output.nbytes))
+        resources.callback(lambda: cuda(cudart.cudaFree(target)))
+        addresses = {"features": int(source), "result": int(target)}
+        try:
+            cuda(cudart.cudaMemcpyAsync(
+                source, features.ctypes.data, features.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream,
+            ))
+            execution.enqueue(addresses, stream=int(stream))
+            cuda(cudart.cudaStreamSynchronize(stream))  # Warm up before capture.
+
+            cuda(cudart.cudaStreamBeginCapture(
+                stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal,
+            ))
+            try:
+                execution.enqueue(addresses, stream=int(stream))  # Exactly one call.
+            finally:
+                graph = cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: cuda(cudart.cudaGraphDestroy(graph)))
+            instance = cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            for _ in range(3):
+                cuda(cudart.cudaGraphLaunch(instance, stream))
+            cuda(cudart.cudaMemcpyAsync(
+                output.ctypes.data, target, output.nbytes,
+                cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream,
+            ))
+        finally:
+            # Wait before ExitStack destroys the graph and releases its buffers.
+            cuda(cudart.cudaStreamSynchronize(stream))
+```
+
+Engines with loops or conditionals, or layers or plugins that synchronize with the host during inference, cannot be captured. CUDA reports capture failures through `cudaErrorStreamCapture*` errors. Always end an attempted capture and release any graph resources before falling back to ordinary `enqueue`. See [TensorRT's CUDA Graph limitations](https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html#inference-with-cuda-graphs).
+
 ### Build a standard graph
 
 Define `graph(network)` with TensorRT's Python network API and load model weights from safetensors. Describe each graph input with `InputShape.fixed` or `InputShape.ranged`, and pass an `EngineBuild` declaration to `build_serialized_engine`. `EngineBuild` declares input shapes and builder settings; the graph function determines precision through the tensor types it places in the network. The supported precisions are fp32 (with TF32 disabled), fp16, bf16, int8, and fp8. The [bf16, int8, and fp8 guide](precision.md) shows tensor casts, explicit Q/DQ layers, and scales supplied through safetensors; fp8 targets sm89 and sm120. `open_engine` uses the same declaration plus a name, graph version, weight files, and `EngineIO` to save and reuse a checked engine. Use a different `name` or `graph_version` for engines with different tensor types, such as fp32 and fp16. The [graph example](https://github.com/sync-dev-org/tensoracle#define-and-build-a-standard-graph) shows the exact calls.

{tensoracle-1.0.0 → tensoracle-1.1.0}/docs/optimization.md

@@ -20,7 +20,7 @@ First check whether a combination of standard TensorRT layers can preserve the o
 
 - **tensoracle tools:** `tensoracle.producer.compile_bundle` generates cubins and a manifest for supported GPU generations. The shared C++ plugin helpers support a caller-built native plugin with declared tensor types, formats, and an input whose shape the output copies; `tensoracle.bundle.inspect_bundle` or `inspect_wheel` checks the cubins and their match with the plugin. Pass the plugin library through `plugins=` to `build_serialized_engine` or `open_engine`. Follow [Build a cubin plugin](index.md#build-a-cubin-plugin) for the `PluginIO` declaration, including fp16 / HWC8 and two-input examples, and the [cubin plugin example](https://github.com/sync-dev-org/tensoracle/blob/main/examples/cubin_plugin/README.md) for the generation, build, and inspection operations.
 - **You provide:** Identify the bottleneck, write the kernel and model-specific plugin behavior, choose its supported shapes and types, build and package its native library, and run the resulting engines. An fp32 kernel must not use TF32.
-- **Gate:** For a custom cubin plugin, the bundle and plugin inspections must pass. Reapply stage 1 to the replacement's fp32 engine against the same fixed PyTorch reference. If using fp16, reapply stage 2 against an accepted fp32 engine with the same limits and model-specific measurements. Advance only when the replacement meets those quality gates; otherwise retain the accepted implementation.
+- **Gate:** For a custom cubin plugin, the bundle and plugin inspections must pass. Reapply stage 1 to the replacement's fp32 engine against the same fixed PyTorch reference. If using a lower precision engine, reapply stage 2 against an accepted fp32 engine with the same limits and model-specific measurements. Advance only when the replacement meets those quality gates; otherwise retain the accepted implementation.
 
 ## 4. Measure speed under isolated load
 

{tensoracle-1.0.0 → tensoracle-1.1.0}/examples/cubin_plugin/README.md

@@ -30,7 +30,7 @@ Follow [Prepare a TensorRT SDK](../../docs/tensorrt-sdk.md) to obtain and verify
 CMAKE_ARGS="-DTENSORRT_ROOT=$TENSORRT_ROOT" uv build examples/cubin_plugin --wheel
 ```
 
-The example's build backend and CUDA tooling belong to this project, not to tensoracle's runtime dependencies. Before installation, inspect the wheel with `tensoracle.bundle.inspect_wheel(wheel, plugin="example_cubin_plugin/libtensoracle_example_plugin.so")`. This compares the library's embedded digest and cubin bytes with the manifest and cubin files.
+The example's build backend and CUDA tooling belong to this project, not to tensoracle's runtime dependencies. Before installation, inspect the wheel with `tensoracle.bundle.inspect_wheel(wheel, plugin="example_cubin_plugin/libtensoracle_example_plugin.so")`. For a wheel built on Windows, pass `plugin="example_cubin_plugin/tensoracle_example_plugin.dll"`. This compares the library's embedded digest and cubin bytes with the manifest and cubin files.
 
 Install the tensoracle wheel and this example wheel into a runtime Python environment with TensorRT 11 and `cuda-python`, but without Triton, torch, or the TensorRT SDK. `open_engine` registers the library before calling `graph` or opening a saved engine. The tests use two separate processes and an `XDG_CACHE_HOME` directory to verify that the second process reuses the saved engine without building a graph.
 

{tensoracle-1.0.0 → tensoracle-1.1.0}/pyproject.toml

@@ -7,7 +7,7 @@ readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
 classifiers = [
-    "Development Status :: 4 - Beta",
+    "Development Status :: 5 - Production/Stable",
     "Environment :: GPU :: NVIDIA CUDA :: 13",
     "Intended Audience :: Developers",
     "License :: OSI Approved :: MIT License",

{tensoracle-1.0.0 → tensoracle-1.1.0}/src/tensoracle/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.0'
-__version_tuple__ = version_tuple = (1, 0, 0)
+__version__ = version = '1.1.0'
+__version_tuple__ = version_tuple = (1, 1, 0)
 
 __commit_id__ = commit_id = None

tensoracle-1.1.0/tests/test_cuda_graphs.py

@@ -0,0 +1,191 @@
+"""Capture one enqueue and replay it with caller-owned CUDA graphs."""
+
+import json
+from contextlib import ExitStack, contextmanager
+
+import numpy as np
+import pytest
+import tensorrt as trt
+from cuda.bindings import runtime as cudart
+
+import tensoracle
+from test_build import _check_cuda
+from test_execution import _open_sum
+
+pytest_plugins = ("test_example_plugin",)
+
+
+@contextmanager
+def _captured(handle, batches, *, shapes):
+    """Keep one execution, its addresses, and its captured graph alive together."""
+    with handle.create_execution() as execution, ExitStack() as resources:
+        execution.set_input_shapes(shapes)
+        output = np.empty(
+            execution.get_tensor_shape("result"),
+            dtype=trt.nptype(execution.get_tensor_dtype("result")),
+        )
+        stream = _check_cuda(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))
+        resources.callback(lambda: _check_cuda(cudart.cudaStreamDestroy(stream)))
+        addresses = {}
+        for name, value in {**batches[0], "result": output}.items():
+            pointer = _check_cuda(cudart.cudaMalloc(value.nbytes))
+            resources.callback(lambda pointer=pointer: _check_cuda(cudart.cudaFree(pointer)))
+            addresses[name] = int(pointer)
+
+        def copy_inputs(batch):
+            for name, value in batch.items():
+                _check_cuda(
+                    cudart.cudaMemcpyAsync(
+                        addresses[name],
+                        value.ctypes.data,
+                        value.nbytes,
+                        cudart.cudaMemcpyKind.cudaMemcpyHostToDevice,
+                        stream,
+                    )
+                )
+
+        def read_output():
+            _check_cuda(
+                cudart.cudaMemcpyAsync(
+                    output.ctypes.data,
+                    addresses["result"],
+                    output.nbytes,
+                    cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost,
+                    stream,
+                )
+            )
+            _check_cuda(cudart.cudaStreamSynchronize(stream))
+            return output.copy()
+
+        try:
+            # The first ordinary enqueue flushes deferred shape and plugin initialization.
+            ordinary = []
+            for batch in batches:
+                copy_inputs(batch)
+                execution.enqueue(addresses, stream=int(stream))
+                ordinary.append(read_output())
+
+            _check_cuda(
+                cudart.cudaStreamBeginCapture(
+                    stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeThreadLocal
+                )
+            )
+            try:
+                execution.enqueue(addresses, stream=int(stream))
+            finally:
+                graph = _check_cuda(cudart.cudaStreamEndCapture(stream))
+            resources.callback(lambda: _check_cuda(cudart.cudaGraphDestroy(graph)))
+            instance = _check_cuda(cudart.cudaGraphInstantiate(graph, 0))
+            resources.callback(lambda: _check_cuda(cudart.cudaGraphExecDestroy(instance)))
+
+            def replay(index):
+                copy_inputs(batches[index])
+                # An empty graph or a launch writing to the wrong address must fail.
+                _check_cuda(
+                    cudart.cudaMemsetAsync(addresses["result"], 0xFF, output.nbytes, stream)
+                )
+                _check_cuda(cudart.cudaGraphLaunch(instance, stream))
+                actual = read_output()
+                np.testing.assert_array_equal(actual, ordinary[index])
+                return actual
+
+            yield replay
+        finally:
+            # Finish work before graphs, buffers, streams, and the execution are released.
+            _check_cuda(cudart.cudaStreamSynchronize(stream))
+
+
+def _sum_batches(rows):
+    return [
+        {
+            "features": np.arange(rows * 2, dtype=np.float32).reshape(rows, 2) / 4 + offset,
+            "extra": np.full((rows, 2), -0.25 - offset / 2, dtype=np.float32),
+        }
+        for offset in (0, 2)
+    ]
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_standard_engine_graph_replay_matches_ordinary_enqueue(tmp_path, monkeypatch):
+    """One captured enqueue on a nondefault stream reproduces ordinary sums for new input data."""
+    handle = _open_sum(tmp_path, monkeypatch, fixed=True)
+    batches = _sum_batches(3)
+    with _captured(handle, batches, shapes={}) as replay:
+        for index in (0, 1, 0):
+            np.testing.assert_array_equal(
+                replay(index), batches[index]["features"] + batches[index]["extra"]
+            )
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_plugin_engine_graph_replay_matches_ordinary_enqueue(installed_example, tmp_path):
+    """A captured cubin plugin runs repeatedly and matches ordinary enqueue with new inputs."""
+    library = installed_example["library"]
+    manifest = json.loads((library.parent / "bundle/bundle.json").read_text())
+    digest = np.frombuffer(manifest["bundle_sha256"].encode(), dtype=np.uint8)
+    bias = np.arange(16, dtype=np.float32) / 8
+
+    def graph(network):
+        creator = trt.get_plugin_registry().get_creator(
+            "TensoracleExampleBias", "1", "tensoracle.example"
+        )
+        assert creator is not None
+        fields = trt.PluginFieldCollection(
+            [
+                trt.PluginField("bundle_sha256", digest, trt.PluginFieldType.CHAR),
+                trt.PluginField("bias", bias, trt.PluginFieldType.FLOAT32),
+            ]
+        )
+        plugin = creator.create_plugin("bias", fields, trt.TensorRTPhase.BUILD)
+        assert plugin is not None
+        features = network.add_input("features", trt.float32, (3, 16))
+        result = network.add_plugin_v3([features], [], plugin).get_output(0)
+        result.name = "result"
+        network.mark_output(result)
+
+    plan = tensoracle.build_serialized_engine(
+        graph,
+        tensoracle.EngineBuild(
+            {"features": tensoracle.InputShape.fixed((3, 16))},
+            tensoracle.BuilderSettings(optimization_level=0),
+        ),
+        plugins=(library,),
+    )
+    logger = trt.Logger(trt.Logger.ERROR)
+    runtime = trt.Runtime(logger)
+    engine = runtime.deserialize_cuda_engine(plan)
+    assert engine is not None
+    handle = tensoracle.EngineHandle(engine, runtime, tmp_path / "bias.engine")
+    batches = [
+        {"features": np.arange(48, dtype=np.float32).reshape(3, 16) / 4 + offset}
+        for offset in (0, 2)
+    ]
+    with _captured(handle, batches, shapes={}) as replay:
+        for index in (0, 1, 0):
+            np.testing.assert_array_equal(replay(index), batches[index]["features"] + bias)
+
+
+@pytest.mark.gpu
+@pytest.mark.req("SP-TENSORACLE-028")
+def test_ranged_engine_keeps_a_graph_and_execution_per_shape(tmp_path, monkeypatch):
+    """Two shapes keep dedicated executions and graphs that replay correctly in alternation."""
+    handle = _open_sum(tmp_path, monkeypatch)
+    batches = [_sum_batches(rows) for rows in (2, 5)]
+    with ExitStack() as resources:
+        replays = [
+            resources.enter_context(
+                _captured(
+                    handle,
+                    values,
+                    shapes={name: value.shape for name, value in values[0].items()},
+                )
+            )
+            for values in batches
+        ]
+        for index in (0, 1, 0):
+            for replay, values in zip(replays, batches, strict=True):
+                np.testing.assert_array_equal(
+                    replay(index), values[index]["features"] + values[index]["extra"]
+                )

{tensoracle-1.0.0 → tensoracle-1.1.0}/tests/test_distribution.py

@@ -410,7 +410,7 @@ def test_wheel_metadata_describes_the_public_package(distributions):
     assert metadata.get_all("License-File") == ["LICENSE"]
     assert metadata["Requires-Python"] == "<3.14,>=3.12"
     assert set(metadata.get_all("Classifier", [])) == {
-        "Development Status :: 4 - Beta",
+        "Development Status :: 5 - Production/Stable",
         "Environment :: GPU :: NVIDIA CUDA :: 13",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: MIT License",