straw-queue 0.1.6__tar.gz → 0.1.7__tar.gz

{straw_queue-0.1.6 → straw_queue-0.1.7}/CHANGELOG.md

@@ -2,6 +2,40 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.7
+
+### Batched reads and validation
+
+- Add `TensorRef.load_many(..., reader=reader, workers=4)` for caller-bounded,
+  ordered tensor batches. Native workers reuse extent handles and coalesce nearby
+  reads in bounded windows (1 MiB by default, at most 8 MiB per worker), avoiding
+  read-ahead across gaps larger than 4 KiB. Each result has independent writable
+  storage; typed layout, selected frames and touched checksum chunks remain
+  authenticated. Failed attempts discard handles and windows before retrying.
+- Read tensor member envelopes through one native call and reuse canonical path
+  information within a publication, reducing Python/native transitions and
+  repeated shared-filesystem path resolution.
+- Add `store.validation_scope(coordinator)` to reuse immutable dependency proofs
+  during one local publication/queue handoff. New roots still check descriptors
+  and task/attempt identity; catalog changes still check ownership, namespace
+  and tombstones under lock. Scopes are thread-local, do not pin data, and close
+  on exit or participating errors. Explicit validation and payload reads retain
+  their normal checks.
+- Sample the coordinator lease clock after acquiring its call lock, so queued
+  calls do not submit stale timestamps. Log slow calls with separate lock-wait
+  and operation times.
+
+### Compatibility and limits
+
+- Existing Python call signatures and stored WAL/packed-record formats remain
+  compatible; the new batch and scope APIs require 0.1.7. Keep input owners alive
+  throughout reads and validation scopes. Callers must bound the total payload
+  bytes returned by a batch; the read-ahead limit only bounds worker scratch.
+- These changes reduce small-read calls and repeated immutable-graph work.
+  Throughput depends on request locality, cache state and storage; they do not
+  establish a shared-filesystem bandwidth guarantee or change GC durability
+  requirements.
+
 ## 0.1.6
 
 - Reuse confirmed writer-registry state and index operation lookup; rebuild after uncertain outcomes. Cache only successfully synchronized writer directories with pinned inode identity. Keep existing publication WAL barriers and wire formats.

{straw_queue-0.1.6 → straw_queue-0.1.7}/CHANGELOG_zh.md

@@ -2,6 +2,20 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.7
+
+### 批量读取与校验
+
+- 新增 `TensorRef.load_many(..., reader=reader, workers=4)`，按调用方限定的批量大小并行读取 tensor，并保持输入顺序。Native worker 复用 extent 句柄，在有界窗口中合并相邻读取（默认每个 worker 1 MiB，最多 8 MiB），不跨越大于 4 KiB 的空隙预读。每个结果拥有独立可写存储；typed layout、选中的 frame 和触及的 checksum chunk 仍需认证。失败尝试在重试前丢弃句柄和窗口。
+- 通过一次 native 调用读取 tensor 成员 envelope，并在同一 publication 内复用规范路径信息，减少 Python/native 边界调用和共享文件系统路径解析。
+- 新增 `store.validation_scope(coordinator)`，在一次本地 publication／queue 交接中复用不可变依赖证明。新根仍核对 descriptor 和 task／attempt 身份；catalog 变更仍在锁内检查所有权、namespace 和 tombstone。Scope 按线程隔离、不负责 pin 数据，并在退出或参与操作报错时关闭。显式 validate 和 payload 读取保留原有检查。
+- Coordinator 在取得调用锁后才读取 lease 时钟，避免排队调用提交过时的时间戳。慢调用日志分别记录锁等待与实际操作耗时。
+
+### 兼容与限制
+
+- 现有 Python 调用签名及已存储 WAL／packed-record 格式保持兼容；新增批量与 scope API 需要 0.1.7。读取和 validation scope 期间仍须保留输入 owner。调用方需限制 batch 返回的 payload 总字节数；预读上限只约束 worker 临时缓冲区。
+- 本版减少小读取调用和重复不可变图处理。吞吐仍取决于请求局部性、缓存状态和存储环境，不构成共享文件系统带宽保证，也不改变 GC 持久化要求。
+
 ## 0.1.6
 
 - 复用已确认的 writer registry 状态并为操作查找建立索引；不确定结果后重建。只缓存完整同步成功的 writer 目录，并保持 inode 身份。保留既有 publication WAL 屏障与 wire format。

{straw_queue-0.1.6 → straw_queue-0.1.7}/Cargo.lock

@@ -370,8 +370,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.6"
+version = "0.1.7"
 dependencies = [
+ "libc",
  "pyo3",
  "serde",
  "serde_json",

{straw_queue-0.1.6 → straw_queue-0.1.7}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.6"
+version = "0.1.7"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"
@@ -31,6 +31,9 @@ sha2 = "0.10"
 uuid = { version = "1.23", features = ["v4"] }
 pyo3 = { version = "0.23.5", optional = true }
 
+[target.'cfg(target_os = "linux")'.dependencies]
+libc = "0.2"
+
 [dev-dependencies]
 tempfile = "3.27"
 

{straw_queue-0.1.6 → straw_queue-0.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.6
+Version: 0.1.7
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.6 → straw_queue-0.1.7}/docs/USAGE.md

@@ -354,6 +354,26 @@ On recovery, first restore the chosen checkpoint's model/optimizer/RNG and its m
 
 ## Scoped batch reads
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` loads a caller-bounded batch
+in its original order. Bound the total requested payload bytes before calling;
+each returned tensor has independent writable storage. Native workers sort
+requests by pack/extent/ordinal, reuse the extent handle and combine nearby reads
+in a window of at most 1 MiB per worker. Gaps larger than 4 KiB are not prefetched.
+Every selected frame, typed layout and touched checksum chunk is still checked.
+Failed attempts discard both the handle and window before the ordinary bounded
+visibility retry; windows are discarded at batch exit. The lower-level
+`reader.read_tensor_ranges(..., read_ahead_bytes=0)` disables read-ahead for
+comparisons (the maximum supported setting is 8 MiB per worker).
+
+For one publish/queue handoff, `with store.validation_scope(coordinator):` reuses
+successful immutable dependency proofs across participating operations in the
+current thread. Each new root still checks its descriptor and task/attempt;
+catalog transfers still check namespace identity, ownership authorization and
+tombstones under the catalog lock. The scope never pins data, never crosses
+storage identity, closes on exit, and is invalidated by any participating error.
+Keep input owners alive throughout it. Explicit `store.validate` and later
+operations still revalidate; the scope does not cache payload read results.
+
 Use one `with store.read_session() as reader:` block for adjacent immutable publications. The reader provides `validate`, `manifest`, `read`, `read_record`, `envelope`, and checked tensor reads. Pass it to `TensorRef.from_record_set_many(store, publication, reader=reader)` and `tensor.load(reader=reader)` to avoid repeatedly parsing a shared extent index.
 
 The Rust reader caches up to eight authenticated indexes using an LRU keyed by the complete extent descriptor, with an 8 MiB encoded-metadata budget. A single oversized active index evicts all other entries. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again on a cache miss or in a new session. Calls within one session serialize; use independent sessions for parallel reads. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.

{straw_queue-0.1.6 → straw_queue-0.1.7}/docs/USAGE_zh.md

@@ -193,6 +193,22 @@ b.submit_tasks("b-1", [TaskSpec("task-b", input_ref=dependency)])
 
 ## 批量读取会话
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` 按原顺序读取调用方限定的批次。
+调用前应限制请求载荷的总字节数；返回张量各自拥有可写存储。
+Native worker 按 pack/extent/ordinal 排序，复用 extent 句柄，以每个 worker
+最多 1 MiB 的窗口合并邻近读取，不预读超过 4 KiB 的间隙。
+每个选中 frame、typed layout 和触及的 checksum chunk 仍逐一检查。
+读取失败后，先丢弃句柄和窗口，再执行原有有限可见性重试；批次结束也丢弃窗口。
+底层 `reader.read_tensor_ranges(..., read_ahead_bytes=0)` 可关闭预读用于对照，
+该参数最大为每个 worker 8 MiB。
+
+单次发布／队列交接可用 `with store.validation_scope(coordinator):`，
+在当前线程的参与操作之间复用成功的不可变依赖校验证据。
+新根仍检查描述符和 task/attempt；catalog 转移仍在锁内检查 namespace 身份、
+所有权授权和 tombstone。Scope 不保留数据所有权，不跨存储身份，退出时关闭，
+任意参与操作失败立即使其失效。整个操作期间仍须保持输入 owner。
+显式 `store.validate` 和后续操作仍重新校验；scope 不缓存载荷读取结果。
+
 连续读取不可变 publication 时，可共用一个 `with store.read_session() as reader:`。Reader 提供 `validate`、`manifest`、`read`、`read_record`、`envelope` 和带校验的张量读取。将它传给 `TensorRef.from_record_set_many(store, publication, reader=reader)` 与 `tensor.load(reader=reader)`，可避免重复解析同一 extent 的索引。
 
 Rust reader 使用最多八条已认证索引的 LRU，以完整 extent 描述符区分身份，编码元数据预算为 8 MiB；一个超大活动 index 会淘汰其他条目。每次 manifest/record 读取仍校验 frame 和载荷；张量读取仍校验触及的 chunk。每次 `validate` 都重新检查逻辑描述符与 task/attempt 权限。缓存未命中或开启新会话后重新认证索引。同一会话的调用串行，并行读取应使用独立会话。整个读取过程必须由持久 owner 或 `store.pin` 保护：读取会话本身不保留存储所有权。退出上下文会关闭会话，fork 后继承的会话拒绝所有操作。

{straw_queue-0.1.6 → straw_queue-0.1.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.6"
+version = "0.1.7"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"

{straw_queue-0.1.6 → straw_queue-0.1.7}/rust/coordinator.rs

@@ -1,7 +1,7 @@
 //! Single-owner durable scheduling and accepted-result state machine.
 use crate::journal::Journal;
 use crate::queue_state;
-use crate::store::{Config, Reader, Record, Store, mkdir, sync, sync_dir};
+use crate::store::{Config, GraphProof, Reader, Record, Store, mkdir, sync, sync_dir};
 use crate::{Error, Fault, Result, array, bytes, checked_sum, digest, fail, field, number, uid};
 use serde_json::{Value, json};
 use std::{
@@ -96,12 +96,13 @@ fn retain_commit(
     roots: &[(String, Vec<Value>)],
     storage_failed: &mut bool,
     fault: &mut Fault<'_>,
+    graph: Option<&GraphProof>,
 ) -> Result<()> {
     let retained = (|| {
         let verified = config.verify_queue_reactivation(storage_prefix, reactivated, fault)?;
         config.catalog(|catalog| {
             catalog.retain_queue_verified(config, storage_prefix, &verified, fault)?;
-            catalog.retain_queue_many(config, storage_prefix, roots, fault)
+            catalog.retain_queue_many_with_graph(config, storage_prefix, roots, graph, fault)
         })
     })();
     if retained
@@ -500,7 +501,15 @@ impl Coordinator {
         result.recovery_seconds = started.elapsed().as_secs_f64();
         Ok(result)
     }
-    fn commit(&mut self, mut events: Value, fault: &mut Fault<'_>) -> Result<()> {
+    fn commit(&mut self, events: Value, fault: &mut Fault<'_>) -> Result<()> {
+        self.commit_with_graph(events, fault, None)
+    }
+    fn commit_with_graph(
+        &mut self,
+        mut events: Value,
+        fault: &mut Fault<'_>,
+        graph: Option<&GraphProof>,
+    ) -> Result<()> {
         // No newer commit/GC may abandon the owner that protects an unfinished
         // staging handoff. Recovery also reaches this barrier before its epoch.
         self.finish_consumption(fault)?;
@@ -549,6 +558,7 @@ impl Coordinator {
             &roots,
             &mut self.storage_failed,
             fault,
+            graph,
         )?;
         crate::queue_commit::persist_and_apply::<_, crate::wal_file::NativeLibrary, _, _>(
             &mut self.journal,
@@ -570,7 +580,13 @@ impl Coordinator {
         )
         .map_err(|error| commit_error(error, &events))?;
         fault("before_staging_consume")?;
-        self.finish_consumption(fault)
+        finish_consumption_with_graph(
+            &self.store.config,
+            &self.storage_prefix,
+            &mut self.pending_consumption,
+            fault,
+            graph,
+        )
     }
     fn finish_consumption(&mut self, fault: &mut Fault<'_>) -> Result<()> {
         finish_consumption(
@@ -641,16 +657,19 @@ impl Coordinator {
     fn count(&mut self, key: &str) {
         self.counters[key] = json!(number(&self.counters, key) + 1);
     }
-    fn validate(
+    fn validate_graph(
         &mut self,
         reference: &Value,
         task: Option<&str>,
         attempt: Option<&str>,
+        graph: &mut GraphProof,
     ) -> Result<Vec<Value>> {
-        let result = self.store.config.validate(reference, task, attempt);
+        let mut reader = Reader::new(self.store.config.clone());
+        let result =
+            reader.validate_with_graph(reference, task, attempt, &mut graph.validated(), graph);
         if result
             .as_ref()
-            .is_err_and(|e| e.kind == "StorageUnavailable")
+            .is_err_and(|error| error.kind == "StorageUnavailable")
         {
             self.storage_failed = true;
         }
@@ -677,6 +696,19 @@ impl Coordinator {
             &mut self.progress_cache,
             &mut ProgressReader {
                 config: &self.store.config,
+                validated_members: None,
+                storage_failed: &mut self.storage_failed,
+            },
+            reference,
+        )
+        .map_err(progress_error)
+    }
+    fn progress_from_members(&mut self, reference: &Value, members: &[Value]) -> Result<Value> {
+        crate::queue_progress::load(
+            &mut self.progress_cache,
+            &mut ProgressReader {
+                config: &self.store.config,
+                validated_members: Some(members),
                 storage_failed: &mut self.storage_failed,
             },
             reference,
@@ -769,11 +801,11 @@ impl Coordinator {
         }
         Ok(state)
     }
-    fn validate_consumer_state(&mut self, state: &Value) -> Result<()> {
-        self.validate(&state["state_ref"], None, None)?;
+    fn validate_consumer_state(&mut self, state: &Value, graph: &mut GraphProof) -> Result<()> {
+        self.validate_graph(&state["state_ref"], None, None, graph)?;
         if !state["progress_ref"].is_null() {
-            self.validate(&state["progress_ref"], None, None)?;
-            let progress = self.progress(&state["progress_ref"])?;
+            let members = self.validate_graph(&state["progress_ref"], None, None, graph)?;
+            let progress = self.progress_from_members(&state["progress_ref"], &members)?;
             let processed = number(state, "processed_cursor");
             let fetched = number(state, "fetch_cursor");
             if array(&progress, "processed_positions")?
@@ -798,6 +830,19 @@ impl Coordinator {
         now: f64,
         fault: &mut Fault<'_>,
     ) -> Result<Value> {
+        self.call_with_graph(method, args, now, fault, None)
+    }
+    pub(crate) fn call_with_graph(
+        &mut self,
+        method: &str,
+        args: &Value,
+        now: f64,
+        fault: &mut Fault<'_>,
+        graph: Option<&mut GraphProof>,
+    ) -> Result<Value> {
+        let mut local_graph = GraphProof::new(&self.store.config);
+        let graph = graph.unwrap_or(&mut local_graph);
+        graph.check_storage(&self.store.config)?;
         if !crate::queue_application::permitted(&self.journal, self.journal.closed(), method) {
             return fail(
                 "UnsafeRecovery",
@@ -815,15 +860,16 @@ impl Coordinator {
                         return fail("StaleAttempt", "Reader lease identity mismatch");
                     }
                 }
-                self.commit(
+                self.commit_with_graph(
                     json!([{"type":"ReaderDone","leases":args["leases"]}]),
                     fault,
+                    Some(graph),
                 )?;
                 Ok(Value::Null)
             }
             "collect_garbage" => self.collect_garbage(fault),
             "release_checkpoint" => {
-                self.commit(json!([{"type":"CheckpointReleased","checkpoint_id":field(args,"checkpoint_id")?}]), fault)?;
+                self.commit_with_graph(json!([{"type":"CheckpointReleased","checkpoint_id":field(args,"checkpoint_id")?}]), fault, Some(graph))?;
                 Ok(Value::Null)
             }
             "submit_tasks" => {
@@ -874,11 +920,11 @@ impl Coordinator {
                         };
                     }
                     if !t["input_ref"].is_null() {
-                        self.validate(&t["input_ref"], None, None)?;
+                        self.validate_graph(&t["input_ref"], None, None, graph)?;
                     }
                     ids.push(json!(id));
                 }
-                self.commit(json!([Self::event("Submitted",&request,json!(ids),json!({"tasks":tasks,"producer_id":producer,"producer_state":args["producer_state"]}))]),fault)?;
+                self.commit_with_graph(json!([Self::event("Submitted",&request,json!(ids),json!({"tasks":tasks,"producer_id":producer,"producer_state":args["producer_state"]}))]),fault, Some(graph))?;
                 Ok(json!(ids))
             }
             "acquire" => {
@@ -894,7 +940,11 @@ impl Coordinator {
                     .map(|(id, _)| json!(id))
                     .collect::<Vec<_>>();
                 if !expired.is_empty() {
-                    self.commit(json!([{"type":"Expired","task_ids":expired}]), fault)?;
+                    self.commit_with_graph(
+                        json!([{"type":"Expired","task_ids":expired}]),
+                        fault,
+                        Some(graph),
+                    )?;
                     self.counters["expired_leases"] =
                         json!(number(&self.counters, "expired_leases") + expired.len() as u64);
                 }
@@ -963,7 +1013,11 @@ impl Coordinator {
                     .iter()
                     .map(|a| a["lease"].clone())
                     .collect::<Vec<_>>();
-                self.commit(json!([{"type":"Leased","leases":leases}]), fault)?;
+                self.commit_with_graph(
+                    json!([{"type":"Leased","leases":leases}]),
+                    fault,
+                    Some(graph),
+                )?;
                 for lease in leases {
                     self.deadlines
                         .insert(field(&lease, "task_id")?.into(), now + self.lease_seconds);
@@ -1027,7 +1081,7 @@ impl Coordinator {
                 last["request_digest"] = json!(request.1);
                 let response = json!(if yielding { "pending" } else { "saved" });
                 last["response"] = response.clone();
-                self.commit(json!(events), fault)?;
+                self.commit_with_graph(json!(events), fault, Some(graph))?;
                 Ok(response)
             }
             "release_tasks" => {
@@ -1062,7 +1116,7 @@ impl Coordinator {
                     last["request_key"] = json!(request.0);
                     last["request_digest"] = json!(request.1);
                     last["response"] = json!(response);
-                    self.commit(json!(events), fault)?;
+                    self.commit_with_graph(json!(events), fault, Some(graph))?;
                 }
                 Ok(json!(response))
             }
@@ -1115,15 +1169,17 @@ impl Coordinator {
                 if number(reference, "records") == 0 && !boolean(&task["spec"], "allow_empty") {
                     return fail("InvalidReference", "Task does not allow empty output");
                 }
-                self.validate(
+                self.validate_graph(
                     reference,
                     Some(field(lease, "task_id")?),
                     Some(field(lease, "attempt_id")?),
+                    graph,
                 )?;
                 let receipt = json!({"commit_id":uid(),"task_id":lease["task_id"],"submission_id":id,"digest":content,"position":array(&self.state,"commits")?.len(),"result_ref":reference});
-                self.commit(
+                self.commit_with_graph(
                     json!([{"type":"Completed","receipt":receipt,"lease":lease}]),
                     fault,
+                    Some(graph),
                 )?;
                 fault("before_complete_reply")?;
                 self.complete_seconds.push(started.elapsed().as_secs_f64());
@@ -1149,7 +1205,7 @@ impl Coordinator {
                 } else {
                     "failed"
                 });
-                self.commit(json!([Self::event("Failed",&request,response.clone(),json!({"task_ids":[lease["task_id"]],"failure":failure,"retryable":retryable}))]),fault)?;
+                self.commit_with_graph(json!([Self::event("Failed",&request,response.clone(),json!({"task_ids":[lease["task_id"]],"failure":failure,"retryable":retryable}))]),fault, Some(graph))?;
                 if matches!(
                     failure["category"].as_str(),
                     Some("QuotaExceeded" | "StorageUnavailable")
@@ -1173,9 +1229,9 @@ impl Coordinator {
                 if yielding && !crate::queue_admin::has_capacity(self.next_pending_order, 1) {
                     return fail("ValueError", "Queue scheduling order is exhausted");
                 }
-                self.validate(&args["input_ref"], None, None)?;
+                self.validate_graph(&args["input_ref"], None, None, graph)?;
                 let response = json!(if yielding { "pending" } else { "saved" });
-                self.commit(
+                self.commit_with_graph(
                     json!([Self::event(
                         if yielding { "Yielded" } else { "TaskProgress" },
                         &request,
@@ -1183,6 +1239,7 @@ impl Coordinator {
                         json!({"task_id":args["lease"]["task_id"],"input_ref":args["input_ref"]})
                     )]),
                     fault,
+                    Some(graph),
                 )?;
                 Ok(response)
             }
@@ -1195,7 +1252,7 @@ impl Coordinator {
                 if terminal(&self.state["tasks"][id]["state"]) {
                     return fail("ValueError", "Cannot cancel terminal task");
                 };
-                self.commit(
+                self.commit_with_graph(
                     json!([Self::event(
                         "Cancelled",
                         &request,
@@ -1203,15 +1260,17 @@ impl Coordinator {
                         json!({"task_id":id})
                     )]),
                     fault,
+                    Some(graph),
                 )?;
                 Ok(json!("cancelled"))
             }
             "seal_input" => {
                 let request = self.request("seal", field(args, "request_id")?, &Value::Null)?;
                 if request.2.is_none() {
-                    self.commit(
+                    self.commit_with_graph(
                         json!([Self::event("Sealed", &request, json!("sealed"), json!({}))]),
                         fault,
+                        Some(graph),
                     )?;
                 }
                 Ok(json!("sealed"))
@@ -1275,15 +1334,20 @@ impl Coordinator {
                     match self.call("fail_task", &call, now, fault) {
                         Ok(_) => {}
                         Err(e) if e.kind == "LeaseExpired" => {
-                            self.commit(
+                            self.commit_with_graph(
                                 json!([{"type":"Expired","task_ids":[lease["task_id"]]}]),
                                 fault,
+                                Some(graph),
                             )?;
                         }
                         Err(e) => return Err(e),
                     }
                 }
-                self.commit(json!([{"type":"ReaderRetired","worker_id":worker}]), fault)?;
+                self.commit_with_graph(
+                    json!([{"type":"ReaderRetired","worker_id":worker}]),
+                    fault,
+                    Some(graph),
+                )?;
                 Ok(json!(array(&self.state,"commits")?.iter().filter(|r|self.state["tasks"][r["task_id"].as_str().unwrap_or("")]["lease"]["worker_id"]==worker).cloned().collect::<Vec<_>>()))
             }
             "open_consumer" => {
@@ -1325,8 +1389,8 @@ impl Coordinator {
                 if let Some(response) = request.2 {
                     return Ok(response);
                 };
-                self.validate_consumer_state(&state)?;
-                self.commit(
+                self.validate_consumer_state(&state, graph)?;
+                self.commit_with_graph(
                     json!([Self::event(
                         "ConsumerState",
                         &request,
@@ -1334,6 +1398,7 @@ impl Coordinator {
                         json!({"consumer_id":id,"state":state})
                     )]),
                     fault,
+                    Some(graph),
                 )?;
                 Ok(state)
             }
@@ -1357,8 +1422,8 @@ impl Coordinator {
                 if request.2.is_some() {
                     return Ok(self.state["batches"][batch_id].clone());
                 };
-                self.validate(&args["plan_ref"], None, None)?;
-                self.commit(
+                self.validate_graph(&args["plan_ref"], None, None, graph)?;
+                self.commit_with_graph(
                     json!([Self::event(
                         "BatchPlanned",
                         &request,
@@ -1366,6 +1431,7 @@ impl Coordinator {
                         json!({"batch_id":batch_id,"batch":batch})
                     )]),
                     fault,
+                    Some(graph),
                 )?;
                 Ok(batch)
             }
@@ -1382,9 +1448,9 @@ impl Coordinator {
                 if let Some(response) = request.2 {
                     return Ok(response);
                 };
-                self.validate(&args["ready_ref"], None, None)?;
-                self.validate_consumer_state(&state)?;
-                self.commit(json!([Self::event("BatchReady",&request,content.clone(),json!({"consumer_id":id,"batch_id":batch,"ready_ref":args["ready_ref"],"state":state}))]),fault)?;
+                self.validate_graph(&args["ready_ref"], None, None, graph)?;
+                self.validate_consumer_state(&state, graph)?;
+                self.commit_with_graph(json!([Self::event("BatchReady",&request,content.clone(),json!({"consumer_id":id,"batch_id":batch,"ready_ref":args["ready_ref"],"state":state}))]),fault, Some(graph))?;
                 Ok(content)
             }
             "get_batch" => Ok(self.state["batches"][field(args, "batch_id")?].clone()),
@@ -1409,7 +1475,7 @@ impl Coordinator {
                     }
                     return fail("UnsafeRecovery", "Checkpoint receipt disappeared");
                 }
-                let members = self.validate(reference, None, None)?;
+                let members = self.validate_graph(reference, None, None, graph)?;
                 if members.len() != 1 {
                     return fail("InvalidReference", "Expected one JSON checkpoint record");
                 };
@@ -1421,7 +1487,7 @@ impl Coordinator {
                 if checkpoint["version"] != 1 || checkpoint["checkpoint_id"] != id {
                     return fail("InvalidReference", "Checkpoint identity/schema mismatch");
                 };
-                self.validate(&checkpoint["consumer_state"], None, None)?;
+                self.validate_graph(&checkpoint["consumer_state"], None, None, graph)?;
                 if array(&checkpoint, "batch_ids")?
                     .iter()
                     .any(|id| !boolean(&self.state["batches"][id.as_str().unwrap_or("")], "ready"))
@@ -1445,7 +1511,7 @@ impl Coordinator {
                 content["checkpoint_ref"] = reference.clone();
                 let request = self.request("checkpoint", id, &content)?;
                 if request.2.is_none() {
-                    self.commit(
+                    self.commit_with_graph(
                         json!([Self::event(
                             "Checkpoint",
                             &request,
@@ -1453,6 +1519,7 @@ impl Coordinator {
                             json!({"checkpoint_id":id,"checkpoint":content})
                         )]),
                         fault,
+                        Some(graph),
                     )?;
                 }
                 Ok(content)
@@ -1582,12 +1649,16 @@ fn recovered_consumption(
 
 struct ProgressReader<'a> {
     config: &'a Config,
+    validated_members: Option<&'a [Value]>,
     storage_failed: &'a mut bool,
 }
 impl crate::queue_progress::Reader for ProgressReader<'_> {
     type Error = Error;
     fn members(&mut self, reference: &Value) -> Result<Vec<Value>> {
-        let result = self.config.validate(reference, None, None);
+        let result = match self.validated_members {
+            Some(members) => Ok(members.to_vec()),
+            None => self.config.validate(reference, None, None),
+        };
         if result
             .as_ref()
             .is_err_and(|error| error.kind == "StorageUnavailable")
@@ -1643,6 +1714,7 @@ impl crate::queue_progress::Reader for GcPreflight<'_, '_> {
         crate::queue_progress::Reader::members(
             &mut ProgressReader {
                 config: self.config,
+                validated_members: None,
                 storage_failed: self.storage_failed,
             },
             reference,
@@ -1652,6 +1724,7 @@ impl crate::queue_progress::Reader for GcPreflight<'_, '_> {
         crate::queue_progress::Reader::read(
             &mut ProgressReader {
                 config: self.config,
+                validated_members: None,
                 storage_failed: self.storage_failed,
             },
             reference,
@@ -1663,6 +1736,16 @@ fn finish_consumption(
     prefix: &str,
     pending: &mut Vec<(String, Vec<Value>)>,
     fault: &mut Fault<'_>,
+) -> Result<()> {
+    finish_consumption_with_graph(config, prefix, pending, fault, None)
+}
+
+fn finish_consumption_with_graph(
+    config: &Config,
+    prefix: &str,
+    pending: &mut Vec<(String, Vec<Value>)>,
+    fault: &mut Fault<'_>,
+    graph: Option<&GraphProof>,
 ) -> Result<()> {
     if pending.is_empty() {
         return Ok(());
@@ -1680,7 +1763,7 @@ fn finish_consumption(
                         .collect::<Result<HashSet<_>>>()?;
                     catalog.require_retained(&owners)?;
                 }
-                let outcome = catalog.consume_once(config, id, roots, fault);
+                let outcome = catalog.consume_once_with_graph(config, id, roots, graph, fault);
                 crate::consumption::pop_completed(pending, outcome.is_ok());
                 outcome?;
             }