straw-queue 0.1.7__tar.gz → 0.1.8__tar.gz

{straw_queue-0.1.7 → straw_queue-0.1.8}/CHANGELOG.md

@@ -2,6 +2,40 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.8
+
+### Supervised workers and reclamation
+
+- Add `Coordinator(..., lease_seconds=None)` for externally supervised workers.
+  Leases do not expire with elapsed time; attempt/epoch fencing and explicit
+  worker retirement remain enforced. The default is still 300 seconds, and
+  recovery requires the queue's original lease configuration.
+- Release both Python and native coordinator locks after GC commits root
+  retirement and tombstones, allowing heartbeats, acquisitions and completions
+  during physical deletion and directory synchronization. GC calls on the same
+  coordinator remain serialized. Return separate queue-lock wait, preparation
+  and detached-collection timings.
+- Add opt-in `retire_writer(..., discard_staging=True)` to release only that
+  writer's registered transfer staging after retirement and sealing. Preserve
+  independent queue, checkpoint, reader and other writer owners. Retrying after
+  an uncertain retirement remains supported.
+- Add `retention_summary()` for owner categories, retained and reclaimable
+  packs, pending deletions and open writers, with bounded examples and no
+  payload reads. Category pack counts can overlap.
+
+### Compatibility and limits
+
+- WAL and packed-record formats remain unchanged. The new lease and staging
+  options require 0.1.8; existing timed leases and staging retention remain the
+  defaults. Supervised mode requires external failure detection and confirmation
+  that a worker has exited before retirement. Before discarding staging, stop
+  its readers and preserve accepted results through independent durable owners;
+  returned references no longer have staging protection.
+- Models retain their existing expiration, owner-release and deletion-plan
+  abstractions. Concrete regressions cover supervised fencing, concurrent queue
+  calls during GC and retirement/staging retries; they do not establish a
+  shared-filesystem latency bound or automatic failure detection.
+
 ## 0.1.7
 
 ### Batched reads and validation

{straw_queue-0.1.7 → straw_queue-0.1.8}/CHANGELOG_zh.md

@@ -2,6 +2,20 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.8
+
+### 受监督的 worker 与回收
+
+- 新增 `Coordinator(..., lease_seconds=None)`，由外部监督器管理 worker 存活性。租约不会随时间自动过期，仍执行 attempt／epoch fencing 和显式 worker 退休；默认仍为 300 秒，恢复时必须沿用队列原有的 lease 配置。
+- GC 持久化根退休和 tombstone 后，释放 Python 与原生 coordinator 锁，使 heartbeat、acquire 和 complete 可以与物理删除、目录同步并行。同一 coordinator 的 GC 调用仍串行；返回队列锁等待、准备和锁外回收的独立耗时。
+- 新增显式选项 `retire_writer(..., discard_staging=True)`，在退休、封口完成后，仅释放该 writer 已登记的 transfer staging。保留独立的 queue、checkpoint、reader 和其他 writer owner；支持退休结果不确定后的重试。
+- 新增 `retention_summary()`，报告 owner 类别、受保留／可回收 pack、待完成删除和未封口 writer，限制示例数量且不读取 payload。不同类别的 pack 计数可能重叠。
+
+### 兼容与限制
+
+- WAL 和 packed-record 格式不变。新的 lease 与 staging 选项需要 0.1.8；默认仍使用定时租约并保留 staging。受监督模式要求外部故障探测，并在退休前确认 worker 已退出。放弃 staging 前须停止相关 reader，以独立持久 owner 保留已接受结果；返回引用不再受 staging 保护。
+- 模型沿用既有的过期、owner 释放与删除计划抽象；实际回归覆盖受监督 fencing、GC 期间的队列并发调用和退休／staging 重试，不据此承诺共享文件系统延迟上限或自动故障探测。
+
 ## 0.1.7
 
 ### 批量读取与校验

{straw_queue-0.1.7 → straw_queue-0.1.8}/Cargo.lock

@@ -370,7 +370,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.7"
+version = "0.1.8"
 dependencies = [
  "libc",
  "pyo3",

{straw_queue-0.1.7 → straw_queue-0.1.8}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.7"
+version = "0.1.8"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"

{straw_queue-0.1.7 → straw_queue-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.7
+Version: 0.1.8
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.7 → straw_queue-0.1.8}/docs/DESIGN.md

@@ -109,6 +109,9 @@ payload may already have been collected. Conflicting content is an error. A new
 request must pass current task/attempt/epoch and expiry checks. Recovery revokes
 old work authority but preserves outstanding reader roots. External fencing must
 exclude concurrent coordinators; an `exclusive_owner` string is not enforcement.
+With `lease_seconds=None`, time does not expire a lease: an external supervisor
+must detect failure and confirm process exit before `retire_worker`. Attempt and
+epoch checks remain mandatory, and recovery must preserve the lease mode.
 
 <a id="c4"></a>
 
@@ -272,8 +275,11 @@ tombstone; late retain cannot revive that path. Live extents protect whole packs
 Collectors may overlap: unlink is idempotent, and every collector completes its
 own parent barriers. Never include targets added after this attempt's snapshot
 in its receipt. A failed attempt leaves durable pending work for a later call.
-The lower `LockedCatalog::collect` compatibility API retains its enclosing lock;
-one coordinator's own calls also remain serialized.
+The lower `LockedCatalog::collect` compatibility API retains its enclosing lock.
+Python coordinator GC calls serialize with each other, but release both queue
+locks after root retirement and tombstone preparation. Other queue calls may
+proceed during physical deletion; receipt completion refreshes the catalog and
+uses only the captured plan.
 Offline cleanup requires stopped clients, no shared catalog/queue markers, a
 complete accepted-root/path-binding trace and a fresh unchanged inspection.
 
@@ -288,6 +294,15 @@ persists the retirement receipt. Recover authenticates complete results and stag
 them before release; abandon makes no readable-result claim. A retry may repeat
 any already-effective step. PID, age and timeout never establish safe retirement.
 
+Explicit `discard_staging=True` releases the exact staging owners of that
+writer's registered results in a separate catalog transaction after retirement.
+It does not recursively release dependency writers' staging. The caller must
+first stop relevant readers and independently retain results still needed;
+returned references have no staging protection. A retry must finish retirement
+before releasing staging again and must preserve other owners. `WriterRetirement`
+models through the retirement receipt; subsequent explicit abandonment uses the
+owner-release rules, with concrete regressions checking the API composition.
+
 Safety holds while owners remain and errors recur. Eventual reclamation concerns
 sealed **permanently unowned** objects and additionally requires continued GC calls,
 returning lock/I/O operations, eventual stabilization and fair scheduling. No

{straw_queue-0.1.7 → straw_queue-0.1.8}/docs/DESIGN_zh.md

@@ -72,6 +72,8 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 **C2 — 身份与重试。** 稳定请求 ID 绑定内容和结果。先查询匹配的已记录回执，再考虑旧 payload 或 lease 校验；payload 可能已经回收。内容冲突必须报错。新请求必须校验当前 task／attempt／epoch 和过期状态。恢复撤销旧工作权限，但保留尚未退出的 reader 根。外部 fencing 必须排除并发协调器；`exclusive_owner` 字符串不执行 fencing。
 
+`lease_seconds=None` 时，时间不会使 lease 过期：外部监督器必须探测故障，并在调用 `retire_worker` 前确认进程退出。Attempt／epoch 检查仍然必需，恢复时必须保留原有 lease 模式。
+
 <a id="c4"></a>
 
 **C4 — Checkpoint 回退。** 持久保存 model／optimizer／RNG checkpoint 及其匹配的 consumer 位置。该 checkpoint 之后已经处理的结果，需要在确认处理前有独立 replay owner。恢复 checkpoint 的位置，而非队列最新 cursor。回退 WAL 更新前先重新保留变为必需的 receipt／batch 根；保护缺失或数据损坏必须拒绝更新。替代版本持久化后才释放旧 checkpoint／replay owner。真实 checkpoint 提交和全部 rank 完成信号由应用提供。
@@ -172,12 +174,14 @@ unlink 本轮全部目标（包括 advisory stat 声称不存在的目标）
 
 Unlink `NotFound` 是幂等缺失，不是持久删除。已经缺失的 pending 文件仍需父目录屏障。其他错误上报并保留待办。回执停止重复删除／计数，但保留永久 tombstone；迟到 retain 不得复活路径。存活 extent 保护整个 pack。离线清理要求客户端停止、无共享 catalog／queue 标记、完整接受根及路径绑定追踪，以及新的未变化检查结果。
 
-多个 collector 可以重叠执行：unlink 幂等，各自完成自己目标的父目录屏障。回执不能包含本轮快照之后加入的目标。失败后持久 pending 留给后续调用处理。底层 `LockedCatalog::collect` 兼容 API 仍持有外层锁；同一个 coordinator 自身的调用也仍然串行。
+多个 collector 可以重叠执行：unlink 幂等，各自完成自己目标的父目录屏障。回执不能包含本轮快照之后加入的目标。失败后持久 pending 留给后续调用处理。底层 `LockedCatalog::collect` 兼容 API 仍持有外层锁。Python coordinator 的 GC 调用彼此串行，但在根退休和 tombstone 准备完成后释放两层队列锁；物理删除期间其他队列调用可以继续。回执阶段重新获取新鲜 catalog，只使用已捕获的计划。
 
 <a id="g4"></a>
 
 **G4 — Writer 退休与进展。** 每次 writer 修改持有准确 incarnation 的 gate 并检查持久 fence。产生效果前先登记操作身份、准备 owner 和依赖。退休获取相同 gate，持久化 fence，排空旧 I/O，对账已登记操作，然后释放准备 owner、记录结算、seal 预留文件、持久化退休回执。Recover 在释放前认证完整结果并 staging；abandon 不声称结果可读。重试可能重复任意已生效步骤。PID、年龄和超时都不能证明退休安全。
 
+显式 `discard_staging=True` 在上述退休完成后，以独立 catalog 事务释放该 writer 已登记结果的准确 staging owner，不递归释放依赖 writer 的 staging。调用者必须先停止相关 reader，并以独立持久 owner 保留仍需使用的结果；此时返回引用不受 staging 保护。错误后的重试仍须完成退休后再释放，其他 owner 保持不变。`WriterRetirement` 模型覆盖到退休回执；后续显式放弃沿用 owner 释放规则，由实际回归检查接口组合。
+
 Owner 持续存在或错误反复发生时，安全性仍须成立。最终回收只针对 sealed 且**永久无主**的对象，还要求持续调用 GC、锁／I/O 返回、故障最终稳定及公平调度。这不提供延迟上限，也不意味着存在自动后台回收器。
 
 <details>

{straw_queue-0.1.7 → straw_queue-0.1.8}/docs/USAGE.md

@@ -463,3 +463,22 @@ matching queue view, pending/buffered/prefetched dependencies and post-checkpoin
 replay owners. A latest marker or queue acknowledgement cannot replace it. See
 [restart APIs](#what-resumes-after-a-restart). Application tests cover real schemas,
 abort/resume, shared tensors, multiple-reader completion and whole-job recovery.
+
+### Externally supervised workers
+
+`Coordinator(..., lease_seconds=None)` delegates liveness to an external supervisor;
+attempt and epoch fences remain enforced. The supervisor must detect failures,
+confirm the old process has exited, and call `retire_worker`. The default remains
+300 seconds, and recovery cannot change this queue identity setting.
+Queue GC serializes root retirement and durable tombstones, then releases queue
+locks before deleting files and syncing directories. Its result includes
+`queue_lock_wait_seconds`, `queue_prepare_seconds`, and `detached_collection_seconds`.
+
+After readers and writers have exited and accepted results have durable owners,
+`retire_writer(writer_id, policy="abandon", discard_staging=True)` also relinquishes
+that writer's registered transfer staging after retirement and sealing. This is
+opt-in; other queue, checkpoint, reader, and writer owners are preserved. Returned
+references have no staging protection in this mode and are not recovery handles.
+`retention_summary()` reports owner categories and open packs with bounded
+examples, without reading tensor payloads. Category pack counts can overlap and
+must not be added together.

{straw_queue-0.1.7 → straw_queue-0.1.8}/docs/USAGE_zh.md

@@ -272,3 +272,19 @@ receipt = peer.publication_receipt(
 Rollout 的路由／分数采集必须完整、一致后才能发布 continuation，不能仅以 token 生成结束为准。所有必要消费者完成后才确认原始输出，全部训练 rank 读完后才确认 ready batch。应用监督进程应锁存并传播后台 GC 失败、停止接收新任务、保留不确定存储，并对远端在途工作实施 fencing。
 
 联合 checkpoint 的提交与回退遵循 [C4](DESIGN_zh.md#c4)：包含模型／优化器／RNG 和匹配队列视图，保留 pending／buffered／prefetched 依赖及 post-checkpoint replay owner；不能只用 latest 标记或队列确认代替。重启 API 见[恢复说明](#重启后分别续上什么)。应用集成测试应覆盖真实 schema、中断续跑、共享张量、多 reader 完成信号与整任务恢复。
+
+### 由外部监督器管理的 worker
+
+`Coordinator(..., lease_seconds=None)` 显式关闭墙钟租约过期，仍保留 attempt/epoch fencing。
+只有外部监督器负责故障探测、确认旧进程退出并调用 `retire_worker` 时才使用此模式；
+默认仍为 300 秒。已有队列恢复时不能改变该配置。
+队列 GC 的持久化根/墓碑准备仍串行，物理删除和目录同步在队列锁外执行。
+`collect_garbage()` 返回 `queue_lock_wait_seconds`、`queue_prepare_seconds` 和 `detached_collection_seconds`。
+
+监督器确认 writer 的读写进程已退出、已接受结果由队列或其他 durable owner 接管后，
+可调用 `retire_writer(writer_id, policy="abandon", discard_staging=True)`：
+先完整退役、封口，再放弃这个 writer 已登记的 transfer staging；默认不放弃。
+这不会释放队列、checkpoint、reader 或其他 writer 的 staging。
+该模式返回的引用没有 staging 保护，不应当作待恢复的裸引用读取。
+`retention_summary()` 提供 owner 类别和未封口 pack 的元数据摘要，限制示例数量且不读取 tensor payload。
+不同类别的 pack 计数可能重叠，不能直接相加。

{straw_queue-0.1.7 → straw_queue-0.1.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.7"
+version = "0.1.8"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/coordinator.rs

@@ -78,7 +78,7 @@ pub struct Coordinator {
     owners: HashMap<String, String>,
     queue_id: String,
     storage_prefix: String,
-    lease_seconds: f64,
+    lease_seconds: Option<f64>,
     sealed: bool,
     draining: bool,
     storage_failed: bool,
@@ -391,8 +391,18 @@ impl Coordinator {
             .as_str()
             .unwrap_or("rollout")
             .to_string();
-        let lease_seconds = options["lease_seconds"].as_f64().unwrap_or(300.);
-        if lease_seconds <= 0. || queue_id.is_empty() {
+        // Null explicitly delegates failure detection to a process supervisor.
+        // Attempt/epoch fencing and reader lifetime checks still apply.
+        let lease_seconds = match options.get("lease_seconds") {
+            None => Some(300.),
+            Some(Value::Null) => None,
+            Some(value) => Some(value.as_f64().ok_or_else(|| {
+                Error::new("ValueError", "Lease duration must be a number or null")
+            })?),
+        };
+        if lease_seconds.is_some_and(|seconds| !seconds.is_finite() || seconds <= 0.)
+            || queue_id.is_empty()
+        {
             return fail("ValueError", "Invalid lease duration/queue ID");
         }
         let defaults = json!({"pending_tasks":10000,"inflight_tasks":256,"control_tasks":1,"accepted_records":100000,"accepted_bytes":68719476736u64,"accepted_tokens":100000000,"ready_bytes":68719476736u64,"max_result_bytes":4294967296u64,"max_result_tokens":100000000,"metadata_bytes":262144});
@@ -686,7 +696,7 @@ impl Coordinator {
             self.count("stale_attempts");
             return fail("StaleAttempt", "Lease is not current authorization");
         }
-        if self.deadlines.get(id).copied().unwrap_or(0.) <= now {
+        if self.lease_seconds.is_some() && self.deadlines.get(id).copied().unwrap_or(0.) <= now {
             return fail("LeaseExpired", "Lease expired");
         };
         Ok(task.clone())
@@ -1018,9 +1028,11 @@ impl Coordinator {
                     fault,
                     Some(graph),
                 )?;
-                for lease in leases {
-                    self.deadlines
-                        .insert(field(&lease, "task_id")?.into(), now + self.lease_seconds);
+                if let Some(seconds) = self.lease_seconds {
+                    for lease in leases {
+                        self.deadlines
+                            .insert(field(&lease, "task_id")?.into(), now + seconds);
+                    }
                 }
                 Ok(json!({"status":"acquired","assignments":assignments}))
             }
@@ -1126,8 +1138,10 @@ impl Coordinator {
                 for lease in leases {
                     match self.valid_lease(lease, now) {
                         Ok(_) => {
-                            self.deadlines
-                                .insert(field(lease, "task_id")?.into(), now + self.lease_seconds);
+                            if let Some(seconds) = self.lease_seconds {
+                                self.deadlines
+                                    .insert(field(lease, "task_id")?.into(), now + seconds);
+                            }
                             results.push(json!("extended"));
                         }
                         Err(e) if ["StaleAttempt", "LeaseExpired"].contains(&e.kind.as_str()) => {
@@ -1587,6 +1601,26 @@ impl Coordinator {
             .map_err(root_error)
     }
     fn collect_garbage(&mut self, fault: &mut Fault<'_>) -> Result<Value> {
+        let plan = self.prepare_garbage_collection(fault)?;
+        plan.execute(&self.store.config, fault)
+    }
+    /// Only durable queue/root admission needs exclusive queue access. Once
+    /// tombstones are committed, physical deletion can run without this owner.
+    pub(crate) fn prepare_garbage_collection(
+        &mut self,
+        fault: &mut Fault<'_>,
+    ) -> Result<crate::gc::CollectionPlan> {
+        if !crate::queue_application::permitted(
+            &self.journal,
+            self.journal.closed(),
+            "collect_garbage",
+        ) {
+            return fail(
+                "UnsafeRecovery",
+                "Coordinator unavailable; stop owner and recover",
+            );
+        }
+        self.finish_consumption(fault)?;
         if !self.store.config.gc_enabled()? {
             return fail("ValueError", "Online GC is not enabled");
         }
@@ -1610,7 +1644,7 @@ impl Coordinator {
             .map_err(gc_prepare_error)?;
         self.store
             .config
-            .collect_queue(prepared, fault)?
+            .plan_queue_collection(prepared, fault)?
             .ok_or_else(|| Error::new("ValueError", "Online GC is not enabled"))
     }
     pub fn close(&mut self) {

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/gc.rs

@@ -23,6 +23,7 @@ use std::path::{Path, PathBuf};
 
 #[path = "gc_plan.rs"]
 mod collection;
+pub(crate) use collection::Plan as CollectionPlan;
 
 fn catalog_append_error(error: crate::catalog_append::Error<Error, Error>) -> Error {
     match error {
@@ -649,6 +650,10 @@ impl LockedCatalog<'_> {
     pub fn collect(&mut self, config: &Config, fault: &mut Fault<'_>) -> Result<Value> {
         self.catalog.collect(config, fault)
     }
+    #[cfg(feature = "python")]
+    pub(crate) fn retention_summary(&self) -> Value {
+        self.catalog.retention_summary()
+    }
     #[cfg(test)]
     pub(crate) fn collect_queue(
         &mut self,
@@ -1482,6 +1487,46 @@ impl Catalog {
             "retained_packs":result.retained,"tombstones":self.log.indexes().deleted.len()})
     }
 
+    /// Metadata diagnostics with bounded examples; never reads payload files.
+    #[cfg(feature = "python")]
+    pub(crate) fn retention_summary(&self) -> Value {
+        let indexes = self.log.indexes();
+        let live = indexes.roots();
+        let mut categories =
+            std::collections::BTreeMap::<String, (usize, HashSet<String>, Vec<Value>)>::new();
+        for (owner, paths) in &indexes.owners {
+            let kind = owner.split(':').next().unwrap_or(owner).to_owned();
+            let entry = categories.entry(kind).or_default();
+            entry.0 += 1;
+            entry.1.extend(paths.iter().cloned());
+            if entry.2.len() < 4 {
+                entry.2.push(json!({"owner":owner,"packs":paths.len(),"pack_examples":paths.iter().take(3).collect::<Vec<_>>()}));
+            }
+        }
+        let categories = categories
+            .into_iter()
+            .map(|(kind, (owners, packs, examples))| {
+                (
+                    kind,
+                    json!({"owners":owners,"retained_packs":packs.len(),"examples":examples}),
+                )
+            })
+            .collect::<serde_json::Map<_, _>>();
+        let mut open = std::collections::BTreeMap::<String, usize>::new();
+        for (path, sealed) in &indexes.packs {
+            if !sealed && !indexes.deleted.contains(path) {
+                let writer = path.split('/').nth(2).unwrap_or("unknown").to_owned();
+                *open.entry(writer).or_default() += 1;
+            }
+        }
+        json!({"owners":indexes.owners.len(),"retained_packs":live.len(),
+            "unowned_sealed_packs":indexes.candidates(&live).len(),
+            "pending_deletions":indexes.pending_deletions.len(),"tombstones":indexes.deleted.len(),
+            "open_packs":open.values().sum::<usize>(),"open_writers":open.len(),
+            "open_writer_examples":open.into_iter().take(64).collect::<std::collections::BTreeMap<_,_>>(),
+            "categories":categories})
+    }
+
     #[cfg(test)]
     fn collect_queue(
         &mut self,

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/gc_writer.rs

@@ -390,6 +390,18 @@ impl Config {
         writer: &str,
         policy: RetirementPolicy,
         fault: &mut Fault<'_>,
+    ) -> Result<Vec<Value>> {
+        self.retire_writer_with_staging(writer, policy, false, fault)
+    }
+
+    /// Explicitly relinquish this writer's transfer staging only after its
+    /// retirement has committed. Queue/checkpoint/reader owners are untouched.
+    pub(crate) fn retire_writer_with_staging(
+        &self,
+        writer: &str,
+        policy: RetirementPolicy,
+        discard_staging: bool,
+        fault: &mut Fault<'_>,
     ) -> Result<Vec<Value>> {
         let mut permit = Permit::retire(&self.root, &self.run_id, writer)?;
         let registry = permit.registry(None, fault)?;
@@ -406,6 +418,22 @@ impl Config {
             pending,
         };
         crate::writer_retirement::execute(&mut retirement)?;
+        if discard_staging {
+            // Use exact durable result descriptors, not file age or path
+            // overlap. Do not recursively consume another writer's staging.
+            let owners = retirement
+                .registry
+                .described_publications()
+                .map(Catalog::staged)
+                .collect::<Result<BTreeSet<_>>>()?;
+            retirement.config.catalog(|c| {
+                let held = owners
+                    .into_iter()
+                    .filter(|owner| c.catalog.log.indexes().owners.contains_key(owner))
+                    .collect::<Vec<_>>();
+                c.release(retirement.config, &held, retirement.fault)
+            })?;
+        }
         Ok(retirement.recovered)
     }
 }

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/python.rs

@@ -675,6 +675,7 @@ impl NativeStore {
                 return Ok(serde_json::to_string(&self.config.collect_garbage(fault)?)?);
             }
             let result = self.config.catalog(|c| match method {
+                "retention_summary" => Ok(c.retention_summary()),
                 "retain" => {
                     c.retain(
                         &self.config,
@@ -706,12 +707,14 @@ impl NativeStore {
         py.allow_threads(|| Ok(self.writer_access()?.writer_id().to_owned()))
             .map_err(|e| exception(py, e))
     }
+    #[pyo3(signature = (writer_id, policy, fault, discard_staging=false))]
     fn retire_writer(
         &self,
         py: Python<'_>,
         writer_id: &str,
         policy: &str,
         fault: Py<PyAny>,
+        discard_staging: bool,
     ) -> PyResult<String> {
         self.check_writer_process()?;
         let policy = match policy {
@@ -729,7 +732,12 @@ impl NativeStore {
         };
         run(py, &fault, |fault| {
             Ok(serde_json::to_string(
-                &self.config.retire_writer(writer_id, policy, fault)?,
+                &self.config.retire_writer_with_staging(
+                    writer_id,
+                    policy,
+                    discard_staging,
+                    fault,
+                )?,
             )?)
         })
     }
@@ -758,6 +766,23 @@ impl NativeStore {
     }
 }
 #[pyclass]
+struct NativeQueueCollection {
+    config: Config,
+    plan: Mutex<Option<crate::gc::CollectionPlan>>,
+}
+#[pymethods]
+impl NativeQueueCollection {
+    fn execute(&self, py: Python<'_>, fault: Py<PyAny>) -> PyResult<String> {
+        run(py, &fault, |f| {
+            let plan =
+                self.plan.lock().unwrap().take().ok_or_else(|| {
+                    Error::new("ValueError", "Collection plan was already executed")
+                })?;
+            Ok(serde_json::to_string(&plan.execute(&self.config, f)?)?)
+        })
+    }
+}
+#[pyclass]
 struct NativeCoordinator {
     owner: Mutex<Coordinator>,
 }
@@ -809,6 +834,20 @@ impl NativeCoordinator {
     fn diagnostics(&self, py: Python<'_>) -> String {
         py.allow_threads(||{let q=self.owner.lock().unwrap();json!({"path":q.journal.path,"sequence":q.journal.sequence,"poisoned":q.journal.poisoned,"closed":q.journal.closed(),"recovery_seconds":q.recovery_seconds}).to_string()})
     }
+    fn prepare_garbage_collection(
+        &self,
+        py: Python<'_>,
+        fault: Py<PyAny>,
+    ) -> PyResult<NativeQueueCollection> {
+        run(py, &fault, |f| {
+            let mut owner = self.owner.lock().unwrap();
+            let plan = owner.prepare_garbage_collection(f)?;
+            Ok(NativeQueueCollection {
+                config: owner.store.config.clone(),
+                plan: Mutex::new(Some(plan)),
+            })
+        })
+    }
     fn close(&self, py: Python<'_>) {
         py.allow_threads(|| self.owner.lock().unwrap().close());
     }

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/store.rs

@@ -559,14 +559,12 @@ impl Config {
     ) -> Result<Option<T>> {
         self.with_catalog(|catalog| catalog.transaction(self, call))
     }
-    pub(crate) fn collect_queue(
+    pub(crate) fn plan_queue_collection(
         &self,
         prepared: crate::queue_gc::Prepared<'_>,
         fault: &mut Fault<'_>,
-    ) -> Result<Option<Value>> {
-        self.catalog(|catalog| catalog.plan_collection(self, Some(prepared), fault))?
-            .map(|plan| plan.execute(self, fault))
-            .transpose()
+    ) -> Result<Option<crate::gc::CollectionPlan>> {
+        self.catalog(|catalog| catalog.plan_collection(self, Some(prepared), fault))
     }
     /// Collect with the physical deletion and directory barriers outside the
     /// pool-wide catalog lock. Durable tombstones fence every selected name.

{straw_queue-0.1.7 → straw_queue-0.1.8}/rust/writer_gate.rs

@@ -536,6 +536,13 @@ impl Registry {
             .cloned()
             .collect()
     }
+
+    pub(crate) fn described_publications(&self) -> impl Iterator<Item = &Value> {
+        self.operations
+            .iter()
+            .filter_map(|op| op.results.as_ref())
+            .flat_map(|results| results.roots.iter().chain(results.intermediates.iter()))
+    }
 }
 
 #[cfg(test)]

{straw_queue-0.1.7 → straw_queue-0.1.8}/src/straw/coordinator.py

@@ -30,6 +30,8 @@ class _JournalView:
 
 
 class Coordinator:
+    supports_supervised_leases = True
+
     def __init__(
         self,
         store,
@@ -46,6 +48,7 @@ class Coordinator:
         self.limits = limits or Limits()
         self.lease_seconds, self.clock = lease_seconds, clock
         self._call_lock = threading.RLock()
+        self._gc_lock = threading.Lock()
         self._validation_context = threading.local()
         self._cpu_started = time.process_time()
         options = dict(
@@ -272,7 +275,27 @@ class Coordinator:
         raw results. Its finished_batches acknowledge training reads on all ranks.
         Other queues and explicit storage owners independently retain their roots.
         """
-        return self._call("collect_garbage", {})
+        # Only GC passes serialize with each other. The queue lock protects
+        # root retirement and durable tombstones, never physical filesystem I/O.
+        with self._gc_lock:
+            started = time.monotonic()
+            try:
+                with self._call_lock:
+                    acquired = time.monotonic()
+                    plan = self._native.prepare_garbage_collection(self.store._fault)
+                prepared = time.monotonic()
+                result = json.loads(plan.execute(self.store._fault))
+            except BaseException:
+                scope = getattr(self._validation_context, "scope", None)
+                if scope is not None:
+                    scope.close()
+                raise
+            result.update(
+                queue_lock_wait_seconds=acquired - started,
+                queue_prepare_seconds=prepared - acquired,
+                detached_collection_seconds=time.monotonic() - prepared,
+            )
+            return result
 
     def snapshot(self):
         result = self._call("snapshot", locals())

{straw_queue-0.1.7 → straw_queue-0.1.8}/src/straw/store.py

@@ -114,6 +114,8 @@ class SharedFilesystemStore(_RecordReader):
     max_buffer_bytes controls temporary copy memory, never logical record size.
     """
 
+    supports_writer_staging_discard = True
+
     def __init__(
         self,
         root,
@@ -215,14 +217,18 @@ class SharedFilesystemStore(_RecordReader):
         """Persistent incarnation used to explicitly retire this writer."""
         return self._native.writer_id()
 
-    def retire_writer(self, writer_id, *, policy="recover"):
+    def retire_writer(self, writer_id, *, policy="recover", discard_staging=False):
         """Fence an exact incarnation and reconcile its unfinished publications.
 
         A busy writer raises WriterBusy. Recovery returns staged results;
         explicit abandonment preserves existing queue, reader and staging roots.
         The same writer cannot resume after retirement. Create a fresh writer.
+        discard_staging=True explicitly relinquishes that writer's registered
+        transfer publications after retirement. The supervisor must first stop
+        its readers and adopt accepted results. Other durable owners survive;
+        returned references are not protected by staging in this mode.
         """
-        values = json.loads(self._native.retire_writer(writer_id, policy, self._fault))
+        values = json.loads(self._native.retire_writer(writer_id, policy, self._fault, discard_staging))
         return [
             RecordSetRef.from_dict(value) if "manifest" in value else RecordRef.from_dict(value) for value in values
         ]
@@ -277,6 +283,15 @@ class SharedFilesystemStore(_RecordReader):
         """Reclaim unowned sealed packs; retry deletions without a durable receipt."""
         return self._storage("collect")
 
+    def retention_summary(self):
+        """Report owner/pack metadata with bounded examples and no payload reads.
+
+        Category pack counts can overlap and must not be added together.
+        Open writers and retained ownership require lifecycle confirmation;
+        elapsed file age alone does not authorize reclamation.
+        """
+        return self._storage("retention_summary")
+
     def _inputs(self, records):
         metadata, payloads = [], []
         for record in records: