straw-queue 0.1.6__tar.gz → 0.1.8__tar.gz

{straw_queue-0.1.6 → straw_queue-0.1.8}/CHANGELOG.md

@@ -2,6 +2,74 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.8
+
+### Supervised workers and reclamation
+
+- Add `Coordinator(..., lease_seconds=None)` for externally supervised workers.
+  Leases do not expire with elapsed time; attempt/epoch fencing and explicit
+  worker retirement remain enforced. The default is still 300 seconds, and
+  recovery requires the queue's original lease configuration.
+- Release both Python and native coordinator locks after GC commits root
+  retirement and tombstones, allowing heartbeats, acquisitions and completions
+  during physical deletion and directory synchronization. GC calls on the same
+  coordinator remain serialized. Return separate queue-lock wait, preparation
+  and detached-collection timings.
+- Add opt-in `retire_writer(..., discard_staging=True)` to release only that
+  writer's registered transfer staging after retirement and sealing. Preserve
+  independent queue, checkpoint, reader and other writer owners. Retrying after
+  an uncertain retirement remains supported.
+- Add `retention_summary()` for owner categories, retained and reclaimable
+  packs, pending deletions and open writers, with bounded examples and no
+  payload reads. Category pack counts can overlap.
+
+### Compatibility and limits
+
+- WAL and packed-record formats remain unchanged. The new lease and staging
+  options require 0.1.8; existing timed leases and staging retention remain the
+  defaults. Supervised mode requires external failure detection and confirmation
+  that a worker has exited before retirement. Before discarding staging, stop
+  its readers and preserve accepted results through independent durable owners;
+  returned references no longer have staging protection.
+- Models retain their existing expiration, owner-release and deletion-plan
+  abstractions. Concrete regressions cover supervised fencing, concurrent queue
+  calls during GC and retirement/staging retries; they do not establish a
+  shared-filesystem latency bound or automatic failure detection.
+
+## 0.1.7
+
+### Batched reads and validation
+
+- Add `TensorRef.load_many(..., reader=reader, workers=4)` for caller-bounded,
+  ordered tensor batches. Native workers reuse extent handles and coalesce nearby
+  reads in bounded windows (1 MiB by default, at most 8 MiB per worker), avoiding
+  read-ahead across gaps larger than 4 KiB. Each result has independent writable
+  storage; typed layout, selected frames and touched checksum chunks remain
+  authenticated. Failed attempts discard handles and windows before retrying.
+- Read tensor member envelopes through one native call and reuse canonical path
+  information within a publication, reducing Python/native transitions and
+  repeated shared-filesystem path resolution.
+- Add `store.validation_scope(coordinator)` to reuse immutable dependency proofs
+  during one local publication/queue handoff. New roots still check descriptors
+  and task/attempt identity; catalog changes still check ownership, namespace
+  and tombstones under lock. Scopes are thread-local, do not pin data, and close
+  on exit or participating errors. Explicit validation and payload reads retain
+  their normal checks.
+- Sample the coordinator lease clock after acquiring its call lock, so queued
+  calls do not submit stale timestamps. Log slow calls with separate lock-wait
+  and operation times.
+
+### Compatibility and limits
+
+- Existing Python call signatures and stored WAL/packed-record formats remain
+  compatible; the new batch and scope APIs require 0.1.7. Keep input owners alive
+  throughout reads and validation scopes. Callers must bound the total payload
+  bytes returned by a batch; the read-ahead limit only bounds worker scratch.
+- These changes reduce small-read calls and repeated immutable-graph work.
+  Throughput depends on request locality, cache state and storage; they do not
+  establish a shared-filesystem bandwidth guarantee or change GC durability
+  requirements.
+
 ## 0.1.6
 
 - Reuse confirmed writer-registry state and index operation lookup; rebuild after uncertain outcomes. Cache only successfully synchronized writer directories with pinned inode identity. Keep existing publication WAL barriers and wire formats.

{straw_queue-0.1.6 → straw_queue-0.1.8}/CHANGELOG_zh.md

@@ -2,6 +2,34 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.8
+
+### 受监督的 worker 与回收
+
+- 新增 `Coordinator(..., lease_seconds=None)`，由外部监督器管理 worker 存活性。租约不会随时间自动过期，仍执行 attempt／epoch fencing 和显式 worker 退休；默认仍为 300 秒，恢复时必须沿用队列原有的 lease 配置。
+- GC 持久化根退休和 tombstone 后，释放 Python 与原生 coordinator 锁，使 heartbeat、acquire 和 complete 可以与物理删除、目录同步并行。同一 coordinator 的 GC 调用仍串行；返回队列锁等待、准备和锁外回收的独立耗时。
+- 新增显式选项 `retire_writer(..., discard_staging=True)`，在退休、封口完成后，仅释放该 writer 已登记的 transfer staging。保留独立的 queue、checkpoint、reader 和其他 writer owner；支持退休结果不确定后的重试。
+- 新增 `retention_summary()`，报告 owner 类别、受保留／可回收 pack、待完成删除和未封口 writer，限制示例数量且不读取 payload。不同类别的 pack 计数可能重叠。
+
+### 兼容与限制
+
+- WAL 和 packed-record 格式不变。新的 lease 与 staging 选项需要 0.1.8；默认仍使用定时租约并保留 staging。受监督模式要求外部故障探测，并在退休前确认 worker 已退出。放弃 staging 前须停止相关 reader，以独立持久 owner 保留已接受结果；返回引用不再受 staging 保护。
+- 模型沿用既有的过期、owner 释放与删除计划抽象；实际回归覆盖受监督 fencing、GC 期间的队列并发调用和退休／staging 重试，不据此承诺共享文件系统延迟上限或自动故障探测。
+
+## 0.1.7
+
+### 批量读取与校验
+
+- 新增 `TensorRef.load_many(..., reader=reader, workers=4)`，按调用方限定的批量大小并行读取 tensor，并保持输入顺序。Native worker 复用 extent 句柄，在有界窗口中合并相邻读取（默认每个 worker 1 MiB，最多 8 MiB），不跨越大于 4 KiB 的空隙预读。每个结果拥有独立可写存储；typed layout、选中的 frame 和触及的 checksum chunk 仍需认证。失败尝试在重试前丢弃句柄和窗口。
+- 通过一次 native 调用读取 tensor 成员 envelope，并在同一 publication 内复用规范路径信息，减少 Python/native 边界调用和共享文件系统路径解析。
+- 新增 `store.validation_scope(coordinator)`，在一次本地 publication／queue 交接中复用不可变依赖证明。新根仍核对 descriptor 和 task／attempt 身份；catalog 变更仍在锁内检查所有权、namespace 和 tombstone。Scope 按线程隔离、不负责 pin 数据，并在退出或参与操作报错时关闭。显式 validate 和 payload 读取保留原有检查。
+- Coordinator 在取得调用锁后才读取 lease 时钟，避免排队调用提交过时的时间戳。慢调用日志分别记录锁等待与实际操作耗时。
+
+### 兼容与限制
+
+- 现有 Python 调用签名及已存储 WAL／packed-record 格式保持兼容；新增批量与 scope API 需要 0.1.7。读取和 validation scope 期间仍须保留输入 owner。调用方需限制 batch 返回的 payload 总字节数；预读上限只约束 worker 临时缓冲区。
+- 本版减少小读取调用和重复不可变图处理。吞吐仍取决于请求局部性、缓存状态和存储环境，不构成共享文件系统带宽保证，也不改变 GC 持久化要求。
+
 ## 0.1.6
 
 - 复用已确认的 writer registry 状态并为操作查找建立索引；不确定结果后重建。只缓存完整同步成功的 writer 目录，并保持 inode 身份。保留既有 publication WAL 屏障与 wire format。

{straw_queue-0.1.6 → straw_queue-0.1.8}/Cargo.lock

@@ -370,8 +370,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.6"
+version = "0.1.8"
 dependencies = [
+ "libc",
  "pyo3",
  "serde",
  "serde_json",

{straw_queue-0.1.6 → straw_queue-0.1.8}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.6"
+version = "0.1.8"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"
@@ -31,6 +31,9 @@ sha2 = "0.10"
 uuid = { version = "1.23", features = ["v4"] }
 pyo3 = { version = "0.23.5", optional = true }
 
+[target.'cfg(target_os = "linux")'.dependencies]
+libc = "0.2"
+
 [dev-dependencies]
 tempfile = "3.27"
 

{straw_queue-0.1.6 → straw_queue-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.6
+Version: 0.1.8
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.6 → straw_queue-0.1.8}/docs/DESIGN.md

@@ -109,6 +109,9 @@ payload may already have been collected. Conflicting content is an error. A new
 request must pass current task/attempt/epoch and expiry checks. Recovery revokes
 old work authority but preserves outstanding reader roots. External fencing must
 exclude concurrent coordinators; an `exclusive_owner` string is not enforcement.
+With `lease_seconds=None`, time does not expire a lease: an external supervisor
+must detect failure and confirm process exit before `retire_worker`. Attempt and
+epoch checks remain mandatory, and recovery must preserve the lease mode.
 
 <a id="c4"></a>
 
@@ -272,8 +275,11 @@ tombstone; late retain cannot revive that path. Live extents protect whole packs
 Collectors may overlap: unlink is idempotent, and every collector completes its
 own parent barriers. Never include targets added after this attempt's snapshot
 in its receipt. A failed attempt leaves durable pending work for a later call.
-The lower `LockedCatalog::collect` compatibility API retains its enclosing lock;
-one coordinator's own calls also remain serialized.
+The lower `LockedCatalog::collect` compatibility API retains its enclosing lock.
+Python coordinator GC calls serialize with each other, but release both queue
+locks after root retirement and tombstone preparation. Other queue calls may
+proceed during physical deletion; receipt completion refreshes the catalog and
+uses only the captured plan.
 Offline cleanup requires stopped clients, no shared catalog/queue markers, a
 complete accepted-root/path-binding trace and a fresh unchanged inspection.
 
@@ -288,6 +294,15 @@ persists the retirement receipt. Recover authenticates complete results and stag
 them before release; abandon makes no readable-result claim. A retry may repeat
 any already-effective step. PID, age and timeout never establish safe retirement.
 
+Explicit `discard_staging=True` releases the exact staging owners of that
+writer's registered results in a separate catalog transaction after retirement.
+It does not recursively release dependency writers' staging. The caller must
+first stop relevant readers and independently retain results still needed;
+returned references have no staging protection. A retry must finish retirement
+before releasing staging again and must preserve other owners. `WriterRetirement`
+models through the retirement receipt; subsequent explicit abandonment uses the
+owner-release rules, with concrete regressions checking the API composition.
+
 Safety holds while owners remain and errors recur. Eventual reclamation concerns
 sealed **permanently unowned** objects and additionally requires continued GC calls,
 returning lock/I/O operations, eventual stabilization and fair scheduling. No

{straw_queue-0.1.6 → straw_queue-0.1.8}/docs/DESIGN_zh.md

@@ -72,6 +72,8 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 **C2 — 身份与重试。** 稳定请求 ID 绑定内容和结果。先查询匹配的已记录回执，再考虑旧 payload 或 lease 校验；payload 可能已经回收。内容冲突必须报错。新请求必须校验当前 task／attempt／epoch 和过期状态。恢复撤销旧工作权限，但保留尚未退出的 reader 根。外部 fencing 必须排除并发协调器；`exclusive_owner` 字符串不执行 fencing。
 
+`lease_seconds=None` 时，时间不会使 lease 过期：外部监督器必须探测故障，并在调用 `retire_worker` 前确认进程退出。Attempt／epoch 检查仍然必需，恢复时必须保留原有 lease 模式。
+
 <a id="c4"></a>
 
 **C4 — Checkpoint 回退。** 持久保存 model／optimizer／RNG checkpoint 及其匹配的 consumer 位置。该 checkpoint 之后已经处理的结果，需要在确认处理前有独立 replay owner。恢复 checkpoint 的位置，而非队列最新 cursor。回退 WAL 更新前先重新保留变为必需的 receipt／batch 根；保护缺失或数据损坏必须拒绝更新。替代版本持久化后才释放旧 checkpoint／replay owner。真实 checkpoint 提交和全部 rank 完成信号由应用提供。
@@ -172,12 +174,14 @@ unlink 本轮全部目标（包括 advisory stat 声称不存在的目标）
 
 Unlink `NotFound` 是幂等缺失，不是持久删除。已经缺失的 pending 文件仍需父目录屏障。其他错误上报并保留待办。回执停止重复删除／计数，但保留永久 tombstone；迟到 retain 不得复活路径。存活 extent 保护整个 pack。离线清理要求客户端停止、无共享 catalog／queue 标记、完整接受根及路径绑定追踪，以及新的未变化检查结果。
 
-多个 collector 可以重叠执行：unlink 幂等，各自完成自己目标的父目录屏障。回执不能包含本轮快照之后加入的目标。失败后持久 pending 留给后续调用处理。底层 `LockedCatalog::collect` 兼容 API 仍持有外层锁；同一个 coordinator 自身的调用也仍然串行。
+多个 collector 可以重叠执行：unlink 幂等，各自完成自己目标的父目录屏障。回执不能包含本轮快照之后加入的目标。失败后持久 pending 留给后续调用处理。底层 `LockedCatalog::collect` 兼容 API 仍持有外层锁。Python coordinator 的 GC 调用彼此串行，但在根退休和 tombstone 准备完成后释放两层队列锁；物理删除期间其他队列调用可以继续。回执阶段重新获取新鲜 catalog，只使用已捕获的计划。
 
 <a id="g4"></a>
 
 **G4 — Writer 退休与进展。** 每次 writer 修改持有准确 incarnation 的 gate 并检查持久 fence。产生效果前先登记操作身份、准备 owner 和依赖。退休获取相同 gate，持久化 fence，排空旧 I/O，对账已登记操作，然后释放准备 owner、记录结算、seal 预留文件、持久化退休回执。Recover 在释放前认证完整结果并 staging；abandon 不声称结果可读。重试可能重复任意已生效步骤。PID、年龄和超时都不能证明退休安全。
 
+显式 `discard_staging=True` 在上述退休完成后，以独立 catalog 事务释放该 writer 已登记结果的准确 staging owner，不递归释放依赖 writer 的 staging。调用者必须先停止相关 reader，并以独立持久 owner 保留仍需使用的结果；此时返回引用不受 staging 保护。错误后的重试仍须完成退休后再释放，其他 owner 保持不变。`WriterRetirement` 模型覆盖到退休回执；后续显式放弃沿用 owner 释放规则，由实际回归检查接口组合。
+
 Owner 持续存在或错误反复发生时，安全性仍须成立。最终回收只针对 sealed 且**永久无主**的对象，还要求持续调用 GC、锁／I/O 返回、故障最终稳定及公平调度。这不提供延迟上限，也不意味着存在自动后台回收器。
 
 <details>

{straw_queue-0.1.6 → straw_queue-0.1.8}/docs/USAGE.md

@@ -354,6 +354,26 @@ On recovery, first restore the chosen checkpoint's model/optimizer/RNG and its m
 
 ## Scoped batch reads
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` loads a caller-bounded batch
+in its original order. Bound the total requested payload bytes before calling;
+each returned tensor has independent writable storage. Native workers sort
+requests by pack/extent/ordinal, reuse the extent handle and combine nearby reads
+in a window of at most 1 MiB per worker. Gaps larger than 4 KiB are not prefetched.
+Every selected frame, typed layout and touched checksum chunk is still checked.
+Failed attempts discard both the handle and window before the ordinary bounded
+visibility retry; windows are discarded at batch exit. The lower-level
+`reader.read_tensor_ranges(..., read_ahead_bytes=0)` disables read-ahead for
+comparisons (the maximum supported setting is 8 MiB per worker).
+
+For one publish/queue handoff, `with store.validation_scope(coordinator):` reuses
+successful immutable dependency proofs across participating operations in the
+current thread. Each new root still checks its descriptor and task/attempt;
+catalog transfers still check namespace identity, ownership authorization and
+tombstones under the catalog lock. The scope never pins data, never crosses
+storage identity, closes on exit, and is invalidated by any participating error.
+Keep input owners alive throughout it. Explicit `store.validate` and later
+operations still revalidate; the scope does not cache payload read results.
+
 Use one `with store.read_session() as reader:` block for adjacent immutable publications. The reader provides `validate`, `manifest`, `read`, `read_record`, `envelope`, and checked tensor reads. Pass it to `TensorRef.from_record_set_many(store, publication, reader=reader)` and `tensor.load(reader=reader)` to avoid repeatedly parsing a shared extent index.
 
 The Rust reader caches up to eight authenticated indexes using an LRU keyed by the complete extent descriptor, with an 8 MiB encoded-metadata budget. A single oversized active index evicts all other entries. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again on a cache miss or in a new session. Calls within one session serialize; use independent sessions for parallel reads. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.
@@ -443,3 +463,22 @@ matching queue view, pending/buffered/prefetched dependencies and post-checkpoin
 replay owners. A latest marker or queue acknowledgement cannot replace it. See
 [restart APIs](#what-resumes-after-a-restart). Application tests cover real schemas,
 abort/resume, shared tensors, multiple-reader completion and whole-job recovery.
+
+### Externally supervised workers
+
+`Coordinator(..., lease_seconds=None)` delegates liveness to an external supervisor;
+attempt and epoch fences remain enforced. The supervisor must detect failures,
+confirm the old process has exited, and call `retire_worker`. The default remains
+300 seconds, and recovery cannot change this queue identity setting.
+Queue GC serializes root retirement and durable tombstones, then releases queue
+locks before deleting files and syncing directories. Its result includes
+`queue_lock_wait_seconds`, `queue_prepare_seconds`, and `detached_collection_seconds`.
+
+After readers and writers have exited and accepted results have durable owners,
+`retire_writer(writer_id, policy="abandon", discard_staging=True)` also relinquishes
+that writer's registered transfer staging after retirement and sealing. This is
+opt-in; other queue, checkpoint, reader, and writer owners are preserved. Returned
+references have no staging protection in this mode and are not recovery handles.
+`retention_summary()` reports owner categories and open packs with bounded
+examples, without reading tensor payloads. Category pack counts can overlap and
+must not be added together.

{straw_queue-0.1.6 → straw_queue-0.1.8}/docs/USAGE_zh.md

@@ -193,6 +193,22 @@ b.submit_tasks("b-1", [TaskSpec("task-b", input_ref=dependency)])
 
 ## 批量读取会话
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` 按原顺序读取调用方限定的批次。
+调用前应限制请求载荷的总字节数；返回张量各自拥有可写存储。
+Native worker 按 pack/extent/ordinal 排序，复用 extent 句柄，以每个 worker
+最多 1 MiB 的窗口合并邻近读取，不预读超过 4 KiB 的间隙。
+每个选中 frame、typed layout 和触及的 checksum chunk 仍逐一检查。
+读取失败后，先丢弃句柄和窗口，再执行原有有限可见性重试；批次结束也丢弃窗口。
+底层 `reader.read_tensor_ranges(..., read_ahead_bytes=0)` 可关闭预读用于对照，
+该参数最大为每个 worker 8 MiB。
+
+单次发布／队列交接可用 `with store.validation_scope(coordinator):`，
+在当前线程的参与操作之间复用成功的不可变依赖校验证据。
+新根仍检查描述符和 task/attempt；catalog 转移仍在锁内检查 namespace 身份、
+所有权授权和 tombstone。Scope 不保留数据所有权，不跨存储身份，退出时关闭，
+任意参与操作失败立即使其失效。整个操作期间仍须保持输入 owner。
+显式 `store.validate` 和后续操作仍重新校验；scope 不缓存载荷读取结果。
+
 连续读取不可变 publication 时，可共用一个 `with store.read_session() as reader:`。Reader 提供 `validate`、`manifest`、`read`、`read_record`、`envelope` 和带校验的张量读取。将它传给 `TensorRef.from_record_set_many(store, publication, reader=reader)` 与 `tensor.load(reader=reader)`，可避免重复解析同一 extent 的索引。
 
 Rust reader 使用最多八条已认证索引的 LRU，以完整 extent 描述符区分身份，编码元数据预算为 8 MiB；一个超大活动 index 会淘汰其他条目。每次 manifest/record 读取仍校验 frame 和载荷；张量读取仍校验触及的 chunk。每次 `validate` 都重新检查逻辑描述符与 task/attempt 权限。缓存未命中或开启新会话后重新认证索引。同一会话的调用串行，并行读取应使用独立会话。整个读取过程必须由持久 owner 或 `store.pin` 保护：读取会话本身不保留存储所有权。退出上下文会关闭会话，fork 后继承的会话拒绝所有操作。
@@ -256,3 +272,19 @@ receipt = peer.publication_receipt(
 Rollout 的路由／分数采集必须完整、一致后才能发布 continuation，不能仅以 token 生成结束为准。所有必要消费者完成后才确认原始输出，全部训练 rank 读完后才确认 ready batch。应用监督进程应锁存并传播后台 GC 失败、停止接收新任务、保留不确定存储，并对远端在途工作实施 fencing。
 
 联合 checkpoint 的提交与回退遵循 [C4](DESIGN_zh.md#c4)：包含模型／优化器／RNG 和匹配队列视图，保留 pending／buffered／prefetched 依赖及 post-checkpoint replay owner；不能只用 latest 标记或队列确认代替。重启 API 见[恢复说明](#重启后分别续上什么)。应用集成测试应覆盖真实 schema、中断续跑、共享张量、多 reader 完成信号与整任务恢复。
+
+### 由外部监督器管理的 worker
+
+`Coordinator(..., lease_seconds=None)` 显式关闭墙钟租约过期，仍保留 attempt/epoch fencing。
+只有外部监督器负责故障探测、确认旧进程退出并调用 `retire_worker` 时才使用此模式；
+默认仍为 300 秒。已有队列恢复时不能改变该配置。
+队列 GC 的持久化根/墓碑准备仍串行，物理删除和目录同步在队列锁外执行。
+`collect_garbage()` 返回 `queue_lock_wait_seconds`、`queue_prepare_seconds` 和 `detached_collection_seconds`。
+
+监督器确认 writer 的读写进程已退出、已接受结果由队列或其他 durable owner 接管后，
+可调用 `retire_writer(writer_id, policy="abandon", discard_staging=True)`：
+先完整退役、封口，再放弃这个 writer 已登记的 transfer staging；默认不放弃。
+这不会释放队列、checkpoint、reader 或其他 writer 的 staging。
+该模式返回的引用没有 staging 保护，不应当作待恢复的裸引用读取。
+`retention_summary()` 提供 owner 类别和未封口 pack 的元数据摘要，限制示例数量且不读取 tensor payload。
+不同类别的 pack 计数可能重叠，不能直接相加。

{straw_queue-0.1.6 → straw_queue-0.1.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.6"
+version = "0.1.8"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"