straw-queue 0.1.5__tar.gz → 0.1.7__tar.gz

{straw_queue-0.1.5 → straw_queue-0.1.7}/.github/workflows/tests.yml

@@ -71,6 +71,14 @@ jobs:
           cargo clippy --locked --all-targets --features python -- -D warnings
       - name: Run Rust tests
         run: cargo test --locked
+      - name: Check concrete protocol mutation regressions
+        run: python verification/runners/check_mutations.py
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: implementation-mutations
+          path: target/implementation-mutations/
+          if-no-files-found: warn
 
   python:
     name: Full suite / Python ${{ matrix.python }}

{straw_queue-0.1.5 → straw_queue-0.1.7}/CHANGELOG.md

@@ -2,6 +2,46 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.7
+
+### Batched reads and validation
+
+- Add `TensorRef.load_many(..., reader=reader, workers=4)` for caller-bounded,
+  ordered tensor batches. Native workers reuse extent handles and coalesce nearby
+  reads in bounded windows (1 MiB by default, at most 8 MiB per worker), avoiding
+  read-ahead across gaps larger than 4 KiB. Each result has independent writable
+  storage; typed layout, selected frames and touched checksum chunks remain
+  authenticated. Failed attempts discard handles and windows before retrying.
+- Read tensor member envelopes through one native call and reuse canonical path
+  information within a publication, reducing Python/native transitions and
+  repeated shared-filesystem path resolution.
+- Add `store.validation_scope(coordinator)` to reuse immutable dependency proofs
+  during one local publication/queue handoff. New roots still check descriptors
+  and task/attempt identity; catalog changes still check ownership, namespace
+  and tombstones under lock. Scopes are thread-local, do not pin data, and close
+  on exit or participating errors. Explicit validation and payload reads retain
+  their normal checks.
+- Sample the coordinator lease clock after acquiring its call lock, so queued
+  calls do not submit stale timestamps. Log slow calls with separate lock-wait
+  and operation times.
+
+### Compatibility and limits
+
+- Existing Python call signatures and stored WAL/packed-record formats remain
+  compatible; the new batch and scope APIs require 0.1.7. Keep input owners alive
+  throughout reads and validation scopes. Callers must bound the total payload
+  bytes returned by a batch; the read-ahead limit only bounds worker scratch.
+- These changes reduce small-read calls and repeated immutable-graph work.
+  Throughput depends on request locality, cache state and storage; they do not
+  establish a shared-filesystem bandwidth guarantee or change GC durability
+  requirements.
+
+## 0.1.6
+
+- Reuse confirmed writer-registry state and index operation lookup; rebuild after uncertain outcomes. Cache only successfully synchronized writer directories with pinned inode identity. Keep existing publication WAL barriers and wire formats.
+- Move physical GC deletion and directory synchronization outside the pool catalog lock. Bind receipts to the completed plan and support overlapping collectors without consuming newly added pending work.
+- Cache up to eight authenticated extent indexes for alternating reads. Extend protocol models and concrete regressions; add real Rust mutation checks to CI and a reproducible metadata-scaling probe. Metadata history/receipt retirement and compaction remain out of scope.
+
 ## 0.1.5
 
 ### Ownership safety

{straw_queue-0.1.5 → straw_queue-0.1.7}/CHANGELOG_zh.md

@@ -2,6 +2,26 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.7
+
+### 批量读取与校验
+
+- 新增 `TensorRef.load_many(..., reader=reader, workers=4)`，按调用方限定的批量大小并行读取 tensor，并保持输入顺序。Native worker 复用 extent 句柄，在有界窗口中合并相邻读取（默认每个 worker 1 MiB，最多 8 MiB），不跨越大于 4 KiB 的空隙预读。每个结果拥有独立可写存储；typed layout、选中的 frame 和触及的 checksum chunk 仍需认证。失败尝试在重试前丢弃句柄和窗口。
+- 通过一次 native 调用读取 tensor 成员 envelope，并在同一 publication 内复用规范路径信息，减少 Python/native 边界调用和共享文件系统路径解析。
+- 新增 `store.validation_scope(coordinator)`，在一次本地 publication／queue 交接中复用不可变依赖证明。新根仍核对 descriptor 和 task／attempt 身份；catalog 变更仍在锁内检查所有权、namespace 和 tombstone。Scope 按线程隔离、不负责 pin 数据，并在退出或参与操作报错时关闭。显式 validate 和 payload 读取保留原有检查。
+- Coordinator 在取得调用锁后才读取 lease 时钟，避免排队调用提交过时的时间戳。慢调用日志分别记录锁等待与实际操作耗时。
+
+### 兼容与限制
+
+- 现有 Python 调用签名及已存储 WAL／packed-record 格式保持兼容；新增批量与 scope API 需要 0.1.7。读取和 validation scope 期间仍须保留输入 owner。调用方需限制 batch 返回的 payload 总字节数；预读上限只约束 worker 临时缓冲区。
+- 本版减少小读取调用和重复不可变图处理。吞吐仍取决于请求局部性、缓存状态和存储环境，不构成共享文件系统带宽保证，也不改变 GC 持久化要求。
+
+## 0.1.6
+
+- 复用已确认的 writer registry 状态并为操作查找建立索引；不确定结果后重建。只缓存完整同步成功的 writer 目录，并保持 inode 身份。保留既有 publication WAL 屏障与 wire format。
+- 将 GC 物理删除与目录同步移到 pool catalog 锁外；回执绑定本轮完成的计划，允许 collector 重叠执行且不消费新加入的 pending 工作。
+- 为交替读取缓存最多八条已认证 extent 索引。扩展协议模型与实际回归，将真实 Rust 代码变异检查加入 CI，新增可重放的元数据开销探针。暂不实现元数据历史／回执退休和压缩。
+
 ## 0.1.5
 
 ### 所有权安全

{straw_queue-0.1.5 → straw_queue-0.1.7}/CONTRIBUTING.md

@@ -40,6 +40,7 @@ cargo test --locked
 python -m pytest
 python verification/runners/environment_manifest.py --check
 python verification/runners/check_tla.py
+python verification/runners/check_mutations.py
 python verification/protocol/checks/ownership_model.py --output /tmp/straw-ownership.json
 python verification/wal/checks/crash_campaign.py --output /tmp/straw-crashes.json
 ```
@@ -53,6 +54,12 @@ Logs, configs and counterexamples default to `target/tla/`. Positives must pass;
 negatives must fail the named invariant/temporal property. Parse errors, timeouts
 and launch failures never count as semantic rejection.
 
+The implementation mutation runner first passes an unmodified regression baseline,
+then edits isolated Rust copies to reuse uncertain registry state, cache a directory
+before its barriers, and skip GC parent barriers. Each must fail its named runtime
+test; compilation failures, missing tests and timeouts do not count. Reports default
+to `target/implementation-mutations/` and are retained by CI.
+
 The inventory checks declared IDs, paths and environment imports. The independent
 ownership model explores a bounded abstraction; the crash campaign checks byte
 prefixes/corruption and process exits. None certifies storage-service or power-loss

{straw_queue-0.1.5 → straw_queue-0.1.7}/CONTRIBUTING_zh.md

@@ -25,6 +25,7 @@ cargo test --locked
 python -m pytest
 python verification/runners/environment_manifest.py --check
 python verification/runners/check_tla.py
+python verification/runners/check_mutations.py
 python verification/protocol/checks/ownership_model.py --output /tmp/straw-ownership.json
 python verification/wal/checks/crash_campaign.py --output /tmp/straw-crashes.json
 ```
@@ -33,6 +34,8 @@ Rust 修改后先重新构建／安装 native 扩展，再跑 Python 测试。
 
 环境登记检查核对声明的 ID、路径与模块导入。独立所有权模型探索有限抽象；crash campaign 检查逐字节前缀／损坏和进程退出，都不认证存储服务故障或掉电行为。多客户端验证见[部署指南](docs/FILESYSTEM_zh.md#qualification)。
 
+实现 mutation runner 先通过未修改的回归基线，再在隔离 Rust 副本中复用不确定 registry 状态、提前缓存目录、跳过 GC 父目录屏障。每项必须使指定运行时测试失败；编译错误、缺失测试和超时不算检出。报告默认保存在 `target/implementation-mutations/`，由 CI 留存。
+
 <a id="build"></a>
 
 ## 构建与检查发行包

{straw_queue-0.1.5 → straw_queue-0.1.7}/Cargo.lock

@@ -370,8 +370,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.5"
+version = "0.1.7"
 dependencies = [
+ "libc",
  "pyo3",
  "serde",
  "serde_json",

{straw_queue-0.1.5 → straw_queue-0.1.7}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.5"
+version = "0.1.7"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"
@@ -31,6 +31,9 @@ sha2 = "0.10"
 uuid = { version = "1.23", features = ["v4"] }
 pyo3 = { version = "0.23.5", optional = true }
 
+[target.'cfg(target_os = "linux")'.dependencies]
+libc = "0.2"
+
 [dev-dependencies]
 tempfile = "3.27"
 

{straw_queue-0.1.5 → straw_queue-0.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.5
+Version: 0.1.7
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/BENCHMARKS.md

@@ -124,6 +124,25 @@ dead. Failed runs, including capacity exhaustion, retain their root for inspecti
 the benchmark does not reset the queue, raise limits or restart automatically.
 Root budgets do not apply to the external report/trace directory.
 
+## Metadata and small reads
+
+Run the same probe against each installed wheel to measure the cost of writer
+history and alternating extent reads:
+
+```sh
+python verification/storage/probes/metadata_scaling.py \
+  --count 1000 --output /tmp/straw-metadata.json
+```
+
+Use `--parent` to select a filesystem and `--online-gc` to include catalog work.
+The probe uses a private temporary root, closes its stores before removal, and
+records the native binary hash. It reports recent write medians, registry size,
+Linux process logical read bytes, sync hooks and 400 checked reads alternating
+between four extents. Journal/file sync hooks can name the same barrier; do not
+add them twice. Logical reads may hit caches and are not device/network traffic.
+Keep workload, mount, interpreter and GC mode fixed across comparisons; these
+small tests do not establish production throughput or tail-latency bounds.
+
 <a id="published-results"></a>
 
 ## Selected public measurements

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/BENCHMARKS_zh.md

@@ -72,6 +72,17 @@ Trace 写入统计带 framing 的 extent 字节，同时单独报告 payload 字
 
 `--max-files`、`--max-gib` 限制有限测试的规模。Benchmark 在临时 root 外保存 JSON 报告、不含载荷的 `.logs.tar.gz` 以及 `.cleanup.json`。成功结束时先确认自有 worker 已停止，再删除 root；SSH 断开并不能证明远端进程已退出。失败的测试（包括容量用尽）会保留 root 供检查，不自动重置队列、提高上限或重新启动。无法检查或停止远端机器时同样保留 root。确认 worker 死亡前不要删除。Root 预算不覆盖外部报告/trace 目录。
 
+## 元数据与小范围读取
+
+对每个已安装 wheel 运行同一探针，测量 writer 历史和交替 extent 读取的开销：
+
+```sh
+python verification/storage/probes/metadata_scaling.py \
+  --count 1000 --output /tmp/straw-metadata.json
+```
+
+`--parent` 选择文件系统，`--online-gc` 纳入 catalog 工作。探针使用私有临时 root，关闭 store 后清理，并记录原生二进制哈希。结果包括最近写入中位数、registry 大小、Linux 进程逻辑读取量、同步 hook，以及在四个 extent 之间交替进行的 400 次认证读取。Journal／file sync hook 可能指向同一次屏障，不应重复相加。逻辑读取可能命中缓存，不是设备／网络流量。对照应保持负载、挂载、解释器和 GC 模式一致；这些小型测试不代表生产吞吐或尾延迟上界。
+
 <a id="published-results"></a>
 
 ## 精选公开基准

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/DESIGN.md

@@ -158,6 +158,11 @@ and seeks to the valid end. Rebuild business indexes, receipts and derived cache
 from that same prefix before reuse. Never lose an acknowledged prefix. A complete
 uncertain frame may survive; its retry must not produce a second acceptance.
 
+An exclusively owned writer registry may reuse its confirmed, fully applied
+prefix and operation indexes after reacquiring the exact gate and checking the
+fresh log's inode and length. An uncertain append or application invalidates that
+cache and requires replay. This does not compact history or remove receipts.
+
 <a id="r1"></a>
 
 **R1 — Preparation and durability.** Use a fresh non-reused pack name. Register it
@@ -166,6 +171,10 @@ manifest. Capture the old append boundary before writing; incomplete cleanup mus
 never truncate an acknowledged prefix. Keep unresolved setup/cleanup state across
 errors. Synchronize data and required directory names, persist staging protection,
 then release preparation holds. Seal only after pending cleanup is resolved.
+Within one writer incarnation, a successfully synchronized ancestor chain may be
+reused while its directory inode stays pinned and matches the named directory.
+Failed initialization establishes no cache entry. New file names and payloads
+still require their own barriers; GC parent repair does not use this cache.
 
 <a id="r2"></a>
 
@@ -184,6 +193,10 @@ reference. Check extent bounds and selected frame/index agreement, authenticate
 the envelope and requested payload/checksum chunks, and bind tensor dtype/shape/
 length before returning or allocating payload. A cached index never skips
 record/envelope/payload validation. Empty typed ranges still validate layout.
+The session uses an LRU of at most eight indexes and an 8 MiB encoded-metadata
+budget; one oversized active index evicts all others. A miss authenticates before
+admission. A failed miss cannot select an unrelated cached index. Each session
+serializes its calls; independent sessions can read concurrently.
 
 <a id="r4"></a>
 
@@ -201,7 +214,7 @@ implementations remain external to finite protocol model checking.
 
 | Rule | TLA+ actions / properties | Deliberate abstraction |
 |---|---|---|
-| C3 | `WalRecovery`: Write/Fail/Scan/Truncate/RecoverSync, AcknowledgedSurvives, ReadyDurable. `CatalogTransaction`: BoundCaller, CacheMatchesPrefix, NoCachedAfterError. `CommitApplication`: Begin/Hook/Recover, AcceptedDataPresent | Frame parts abstract bytes; hashes/parser and lock service are external. Actual callbacks and all public guards need implementation review. |
+| C3 | `WalRecovery`: Write/Fail/Scan/Truncate/RecoverSync, AcknowledgedSurvives, ReadyDurable. `CatalogTransaction`: BoundCaller, CacheMatchesPrefix, NoCachedAfterError; `CatalogReplay`: CoherentReady, confirmed-prefix reuse and dirty-cache rejection. `CommitApplication`: Begin/Hook/Recover, AcceptedDataPresent | Frame parts abstract bytes; hashes/parser and lock service are external. Actual callbacks and all public guards need implementation review. |
 | R1 | `PublicationPreparation`, `IncompletePublication`, `DirectoryDurability`, `ProvisionalHold`, `PublicationFinalization`: reservation, rollback, directory barriers, staging, sealing | Unique immutable names and truthful filesystem barriers; a live writer owns rollback authority. |
 | R2 | `PendingPublication`: OneCompletedFile, WriterFenced, DurableVerifiedReturn. `PublicationHandoff`: NoForgottenUnreturnedFile, CompletedHandoff. `StagingConsumption`: receipt/graph recovery | Failed responses can leave effects. Different API/request/content must not consume or overwrite unresolved output. |
 | R3 | `ReadAttempt`: Index/Frame/Envelope/Payload, NoUnverifiedReturn. `RangeRead`/`TypedRead`: authenticated ranges and layout. `NamespaceIdentity`: canonical bindings | Distinct wire/index/frame/envelope faults remain distinguishable. Warm caches authorize only an exact previously authenticated index, not record bytes. |
@@ -209,8 +222,8 @@ implementations remain external to finite protocol model checking.
 
 | Rules | Actual caller chain | Regression / counterexample entry |
 |---|---|---|
-| C3 | `journal_owner::{open,append_with,recover_with}` → `journal_io` → `journal_state`, `wal_append`, `wal_recovery`, `wal_source/sink`, `file_adapter`, `file_primitives`; `catalog_runtime::Transaction` → `catalog_transaction::execute`, `catalog_log`; `queue_application::{arm,apply,confirm,restore,permitted}` | `tests/test_wal_recovery.py`, `test_task_recovery.py`; native `gc` replacement-inode, nested-entry, caught-unwind, dirty-cache tests; `queue_application` callback/partial-body tests; byte-prefix/XOR and process-exit crash campaign |
-| R1–R2 | `Store::finish/publish_attempt/close`, `Config::finalize_publication` → `incomplete`, `pending`, `finalization`, `holding`; `gc::consume_once` → `staging_consumption` | `tests/test_publication_preparation.py`, `test_pending_publication.py`, `test_provisional_holds.py`, `test_commit_consumption.py`, `test_packed_store.py`; native streamed callback tests |
+| C3 | `journal_owner::{open,append_with,recover_with}` → `journal_io` → `journal_state`, `wal_append`, `wal_recovery`, `wal_source/sink`, `file_adapter`, `file_primitives`; `catalog_runtime::Transaction` → `catalog_transaction::execute`, `catalog_log`; `Store::enter_writer` → `writer_gate::{Permit::registry,Registry::append}`; `queue_application::{arm,apply,confirm,restore,permitted}` | `tests/test_wal_recovery.py`, `test_task_recovery.py`; native `gc` and `writer_gate` identity, caught-unwind and dirty-cache tests; `queue_application` callback/partial-body tests; byte-prefix/XOR and process-exit crash campaign |
+| R1–R2 | `Store::finish/publish_attempt/close/prepare_directory`, `Config::finalize_publication` → `incomplete`, `pending`, `finalization`, `holding`; `gc::consume_once` → `staging_consumption` | `tests/test_publication_preparation.py`, `test_pending_publication.py`, `test_provisional_holds.py`, `test_commit_consumption.py`, `test_packed_store.py`; native streamed callback and directory-cache failure/identity tests |
 | R3–R4 | `Reader` and read session in `rust/store.rs` → `record_index::Cache/Attempt`, `reference_node`, `extent_retry::run`, `bounds::frame_position/authenticated_span`, native read/hash operations | `tests/test_extent_visibility.py`, `test_read_session.py`, `test_extent_bounds.py`, `test_tensor_preflight.py`, `test_tensor_granularity.py`; TLC cold/warm cache, independent visibility, wrong index/frame/envelope/cache, payload corruption and stalled retry controls |
 
 </details>
@@ -241,20 +254,26 @@ compatibility API can perform authentication while already holding the lock.
 <a id="g3"></a>
 
 **G3 — Reclamation.** After C1's durable history retirement, prune only the current
-queue's obsolete owners. Under the catalog lock:
+queue's obsolete owners. The public store/coordinator collection path orders:
 
 ```
-discover sealed packs not reachable from any owner
-append permanent tombstones; sync catalog
-unlink every pending target (including those advisory stat calls missing)
+under the catalog lock: discover sealed unowned packs; mark and sync tombstones
+capture the exact pending target set and catalog identity; release lock and mutex
+unlink every captured target (including those advisory stat calls missing)
 sync every affected parent; repair/sync missing ancestor names if needed
-append reclamation receipts; sync catalog
+reacquire and refresh the same catalog; receipt only completed targets still pending
+sync the receipt before returning
 ```
 
 Unlink `NotFound` is idempotent absence, not durable deletion. Even an absent
 pending file needs its parent barrier. Other errors propagate and keep pending
 work. A receipt stops repeated deletion/accounting but retains the permanent
 tombstone; late retain cannot revive that path. Live extents protect whole packs.
+Collectors may overlap: unlink is idempotent, and every collector completes its
+own parent barriers. Never include targets added after this attempt's snapshot
+in its receipt. A failed attempt leaves durable pending work for a later call.
+The lower `LockedCatalog::collect` compatibility API retains its enclosing lock;
+one coordinator's own calls also remain serialized.
 Offline cleanup requires stopped clients, no shared catalog/queue markers, a
 complete accepted-root/path-binding trace and a fresh unchanged inspection.
 
@@ -281,13 +300,13 @@ latency bound or automatic background collector follows from this rule.
 |---|---|---|
 | G1 | `Ownership`: ReaderProtection, CheckpointProtection, NoDanglingReferences. `DurableHandoff`: ReadersProtected; independent reader/checkpoint/peer cases | Complete application roots must be supplied by straw callers; reference possession is not a root. Permanent-owner cases check safety without assuming release. |
 | G2 | `ReactivationRace`: Prepare/Verify/Recheck/Append; AuthenticatedRoot, CompleteAuthorizedClosure, NoDanglingRestore. `DependencyTraversal`: CompleteClosure | A complete authenticated graph is the lower read boundary. Cache context mismatches cause fresh authentication; owner/root mismatch rejects before any owner append. |
-| G3 | `GCDiscovery`, `GCBatchDeletion`, `GCParentRepair`, `GCReceipt`: discovery, mark, unlink, shared-parent/ancestor barriers, receipt; DurableCompletion, ReceiptIsSafe, ReceiptKeepsFence | Multi-parent and missing-parent obligations remain separate; stat is advisory. `OfflineRoots`/`OfflineAliases` add stopped-client and binding roots, not permission for concurrent cleanup. |
+| G3 | `GCConcurrentDeletion`: PlanFenced, ReceiptScoped, OwnedPresent; `GCDiscovery`, `GCBatchDeletion`, `GCParentRepair`, `GCReceipt`: discovery, mark, unlink, shared-parent/ancestor barriers, receipt; DurableCompletion, ReceiptIsSafe, ReceiptKeepsFence | Multi-parent and missing-parent obligations remain separate; stat is advisory. `OfflineRoots`/`OfflineAliases` add stopped-client and binding roots, not permission for concurrent cleanup. The concurrent model checks safety; retry/progress obligations remain in the lower GC models. |
 | G4 | `WriterRetirement`: register/hold/submit/finish, fence/drain/authenticate/recover/release/settle/seal/receipt; AuthenticatedRecovery, ReceiptSound, RequiredProgress. `WriterLifetime`: in-flight work. `DurableHandoff`: EventuallyReclaimed | Gate handoff must finish/cancel old remote I/O. Release and registry settlement are distinct crash steps. Liveness needs stabilization, scheduling and actual caller retries. |
 
 | Rules | Actual caller chain | Regression / counterexample entry |
 |---|---|---|
 | G1–G2 | `queue_roots::collect`, `queue_reader`, `queue_gc::prepare/require`; `gc::VerifiedReactivation::verify`, `check_reactivation_owners`, `retain_verified_inner`, `closure_inner` → `closure::collect`, `reference_node`, `record_index` | `tests/test_online_gc.py`, `test_queue_gc_roots.py`, `test_path_identity.py`; `tests/native.rs` reactivation pool/run, missing-source and owner-replacement races; TLC stale snapshot, wrong context/root and missing dependency controls |
-| G3 | `Coordinator` collection → `queue_retirement`, `queue_gc_prepare`, `gc_collection::Batch::run` → `gc_mark`, `gc_batch`, `gc_io`, `directory_io`, `gc_receipt` | `tests/test_gc_batch.py`, `test_gc_receipts.py`, `test_gc_missing_parent.py`, `test_offline_aliases.py`; native absent-file/failed-barrier and shared-parent tests; TLC early receipt, lost tombstone, missing parent controls |
+| G3 | `Coordinator` collection → `queue_retirement`, `queue_gc_prepare`, `gc_plan::{plan_collection,Plan::execute,complete_collection}` → `gc_mark`, `gc_batch`, `gc_io`, `directory_io`, `gc_receipt` | `tests/test_gc_concurrency.py`, `test_gc_batch.py`, `test_gc_receipts.py`, `test_gc_missing_parent.py`, `test_offline_aliases.py`; native absent-file/failed-barrier and shared-parent tests; TLC early receipt, lost tombstone, missing parent controls |
 | G4 | `Store` mutation → `writer_gate_flow`, `writer_gate::Registry`; `Config::retire_writer` → `writer_retirement::execute`, `gc_writer::Retirement/Recovery`, `writer_registry::reconcile_registered`, `writer_ownership::authenticate_all` | `tests/test_writer_retirement.py`; native `gc_writer` corruption, recover/abandon and registered-owner tests; TLC death-unlock, skip registry/authentication, early release/receipt and conditional progress |
 
 </details>
@@ -314,6 +333,9 @@ reader/queue-checkpoint roots, receipts before directory sync and existing owner
 protecting the wrong root. The tables locate actual callers; public `checkpoint:*`
 owners and queue checkpoints require different tests. Run the checks in
 [contributing](../CONTRIBUTING.md#checks) and report current inputs and results.
+`check_mutations.py` also checks selected concrete Rust faults against named
+runtime regressions; compilation errors, missing tests and timeouts do not count
+as detections.
 
 Finite models, concrete execution tests and source review are complementary;
 they do not establish complete Rust/Python/FFI equivalence, unbounded theorems or

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/DESIGN_zh.md

@@ -104,10 +104,14 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 恢复认证 framing 和精确哈希范围，按序重放完整事务，拒绝完整损坏 frame，只截断不完整残尾，完成同步并定位到有效末尾。业务索引、回执和派生缓存必须从同一前缀重建后才能复用。不得丢失已确认前缀。完整的不确定 frame 可以存活；重试不得产生第二次接受。
 
+独占 writer registry 可以在重新取得准确 gate、核对新打开日志的 inode 和长度后，复用已确认且完整应用的前缀及操作索引。不确定追加或应用会使缓存失效，必须重放。该优化不压缩历史，也不删除回执。
+
 <a id="r1"></a>
 
 **R1 — 准备与持久化。** 使用新且不复用的 pack 名称。数据 I/O 前登记为 open，写 manifest 前 retain 依赖准备 owner。写入前捕获旧追加边界；不完整清理不得截断已确认前缀。错误后保留未完成的准备／清理状态。同步数据及必要目录名称，持久化 staging 保护，之后才释放准备持有者。解决待清理状态后才能 seal。
 
+同一 writer incarnation 内，完整同步成功的祖先链可以复用，但必须保持目录 inode 打开并核对当前名称仍指向它。初始化失败不能建立缓存。新文件名称和 payload 仍需各自屏障；GC 修复父目录不使用这个缓存。
+
 <a id="r2"></a>
 
 **R2 — 发布重试。** 未决输出绑定准确 API、请求和内容。可能结果不明的同步之前先保存描述。拒绝冲突操作；正确重试重复未知屏障、认证原字节之后才返回。关闭或轮换不能丢掉已完成但尚未返回的描述。Staging 只交给准确的完成 handoff；旧交接完成后可以开始新的 staging 生命周期。
@@ -116,6 +120,8 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 **R3 — 认证读取。** 调用者必须有 G1 保护。引用绑定 pool／run 和 namespace；index 缓存命中要求准确的不可变引用。检查 extent 边界及所选 frame 与 index 的一致性，认证 envelope 和请求 payload／checksum chunk，并在返回或分配 payload 前绑定 tensor dtype／shape／length。缓存 index 不得跳过 record／envelope／payload 校验。空 typed 区间仍需校验 layout。
 
+会话使用最多八条 index、8 MiB 编码元数据预算的 LRU；一个超大活动 index 会淘汰其他条目。未命中时先认证再缓存；失败不能选中无关的旧 index。同一会话的调用串行，独立会话可以并行读取。
+
 <a id="r4"></a>
 
 **R4 — 可见性与失败。** 文件长度、header、footer／index、envelope 和 payload 可以独立变得可见。只在共享有限预算内重试可识别的可见性失败；每次关闭失败句柄并重新打开。名称 `NotFound` 有独立有限查找预算；无关 I/O 错误停止。持久损坏必须耗尽／拒绝，不能返回未认证字节。最终可见性是部署前提，不是测得的 p99，也不保证任何输入都会成功。哈希、解析器、分配器和文件系统实现不在有限协议模型检查范围内。
@@ -125,7 +131,7 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 | 规则 | TLA+ 动作／性质 | 有意保留的抽象差异 |
 |---|---|---|
-| C3 | `WalRecovery`：Write/Fail/Scan/Truncate/RecoverSync、AcknowledgedSurvives、ReadyDurable。`CatalogTransaction`：BoundCaller、CacheMatchesPrefix、NoCachedAfterError。`CommitApplication`：Begin/Hook/Recover、AcceptedDataPresent | frame 段抽象字节；哈希／解析器和锁服务外置。实际 callback 和全部公开入口 guard 仍需实现审查。 |
+| C3 | `WalRecovery`：Write/Fail/Scan/Truncate/RecoverSync、AcknowledgedSurvives、ReadyDurable。`CatalogTransaction`：BoundCaller、CacheMatchesPrefix、NoCachedAfterError；`CatalogReplay`：CoherentReady、已确认前缀复用与 dirty cache 拒绝。`CommitApplication`：Begin/Hook/Recover、AcceptedDataPresent | frame 段抽象字节；哈希／解析器和锁服务外置。实际 callback 和全部公开入口 guard 仍需实现审查。 |
 | R1 | `PublicationPreparation`、`IncompletePublication`、`DirectoryDurability`、`ProvisionalHold`、`PublicationFinalization`：预留、回滚、目录屏障、staging、sealing | 唯一不可变名称及真实文件系统屏障；存活 writer 拥有回滚资格。 |
 | R2 | `PendingPublication`：OneCompletedFile、WriterFenced、DurableVerifiedReturn。`PublicationHandoff`：NoForgottenUnreturnedFile、CompletedHandoff。`StagingConsumption`：回执／图恢复 | 失败响应可留下效果；不同 API／请求／内容不能消费或覆盖未决输出。 |
 | R3 | `ReadAttempt`：Index/Frame/Envelope/Payload、NoUnverifiedReturn。`RangeRead`／`TypedRead`：认证区间及 layout。`NamespaceIdentity`：规范绑定 | wire／index／frame／envelope 错误分别建模；热缓存只授权准确的已认证 index，不授权 record 字节。 |
@@ -133,8 +139,8 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 | 规则 | 实际调用链 | 回归／反例入口 |
 |---|---|---|
-| C3 | `journal_owner::{open,append_with,recover_with}` → `journal_io` → `journal_state`、`wal_append/recovery/source/sink`、`file_adapter/primitives`；`catalog_runtime::Transaction` → `catalog_transaction::execute`、`catalog_log`；`queue_application::{arm,apply,confirm,restore,permitted}` | `tests/test_wal_recovery.py`、`test_task_recovery.py`；原生 gc 的 inode 替换、嵌套入口、捕获 unwind、dirty cache；queue_application 的 callback／部分应用；逐字节残尾／XOR 与进程退出实验 |
-| R1–R2 | `Store::finish/publish_attempt/close`、`Config::finalize_publication` → `incomplete`、`pending`、`finalization`、`holding`；`gc::consume_once` → `staging_consumption` | `tests/test_publication_preparation.py`、`test_pending_publication.py`、`test_provisional_holds.py`、`test_commit_consumption.py`、`test_packed_store.py`；原生 stream callback 回归 |
+| C3 | `journal_owner::{open,append_with,recover_with}` → `journal_io` → `journal_state`、`wal_append/recovery/source/sink`、`file_adapter/primitives`；`catalog_runtime::Transaction` → `catalog_transaction::execute`、`catalog_log`；`Store::enter_writer` → `writer_gate::{Permit::registry,Registry::append}`；`queue_application::{arm,apply,confirm,restore,permitted}` | `tests/test_wal_recovery.py`、`test_task_recovery.py`；原生 gc／writer_gate 的身份替换、捕获 unwind、dirty cache；queue_application 的 callback／部分应用；逐字节残尾／XOR 与进程退出实验 |
+| R1–R2 | `Store::finish/publish_attempt/close/prepare_directory`、`Config::finalize_publication` → `incomplete`、`pending`、`finalization`、`holding`；`gc::consume_once` → `staging_consumption` | `tests/test_publication_preparation.py`、`test_pending_publication.py`、`test_provisional_holds.py`、`test_commit_consumption.py`、`test_packed_store.py`；原生 stream callback 与目录缓存失败／身份回归 |
 | R3–R4 | `rust/store.rs` 的 Reader／read session → `record_index::Cache/Attempt`、`reference_node`、`extent_retry::run`、`bounds::frame_position/authenticated_span`、实际 read／hash | `tests/test_extent_visibility.py`、`test_read_session.py`、`test_extent_bounds.py`、`test_tensor_preflight.py`、`test_tensor_granularity.py`；TLC 冷／热缓存、区域可见性、index／frame／envelope／cache 错误、payload 损坏、重试停滞 |
 
 </details>
@@ -153,18 +159,21 @@ Catalog 与队列日志独立。步骤之间崩溃可以多保留保护，不能
 
 <a id="g3"></a>
 
-**G3 — 回收。** C1 的历史退休持久化后，只裁剪当前队列的旧 owner。持 catalog 锁执行：
+**G3 — 回收。** C1 的历史退休持久化后，只裁剪当前队列的旧 owner。公开 store／coordinator 回收路径按序执行：
 
 ```
-发现 sealed 且不可从任何 owner 到达的 pack
-追加永久 tombstone；同步 catalog
-unlink 全部 pending 目标（包括 advisory stat 声称不存在的目标）
+持 catalog 锁发现 sealed 且无主的 pack；追加永久 tombstone 并同步
+保存准确 pending 目标集合和 catalog 身份；释放文件锁与 mutex
+unlink 本轮全部目标（包括 advisory stat 声称不存在的目标）
 同步全部受影响父目录；必要时修复并同步缺失祖先名称
-追加回收回执；同步 catalog
+重新持锁恢复同一 catalog；只为本轮已完成且仍 pending 的目标记回执
+同步回执后返回
 ```
 
 Unlink `NotFound` 是幂等缺失，不是持久删除。已经缺失的 pending 文件仍需父目录屏障。其他错误上报并保留待办。回执停止重复删除／计数，但保留永久 tombstone；迟到 retain 不得复活路径。存活 extent 保护整个 pack。离线清理要求客户端停止、无共享 catalog／queue 标记、完整接受根及路径绑定追踪，以及新的未变化检查结果。
 
+多个 collector 可以重叠执行：unlink 幂等，各自完成自己目标的父目录屏障。回执不能包含本轮快照之后加入的目标。失败后持久 pending 留给后续调用处理。底层 `LockedCatalog::collect` 兼容 API 仍持有外层锁；同一个 coordinator 自身的调用也仍然串行。
+
 <a id="g4"></a>
 
 **G4 — Writer 退休与进展。** 每次 writer 修改持有准确 incarnation 的 gate 并检查持久 fence。产生效果前先登记操作身份、准备 owner 和依赖。退休获取相同 gate，持久化 fence，排空旧 I/O，对账已登记操作，然后释放准备 owner、记录结算、seal 预留文件、持久化退休回执。Recover 在释放前认证完整结果并 staging；abandon 不声称结果可读。重试可能重复任意已生效步骤。PID、年龄和超时都不能证明退休安全。
@@ -178,13 +187,13 @@ Owner 持续存在或错误反复发生时，安全性仍须成立。最终回
 |---|---|---|
 | G1 | `Ownership`：ReaderProtection、CheckpointProtection、NoDanglingReferences。`DurableHandoff`：ReadersProtected；独立 reader／checkpoint／peer 场景 | 完整应用根由 straw 调用者建立；持有引用不等于 owner。永久持有者场景不假定释放。 |
 | G2 | `ReactivationRace`：Prepare/Verify/Recheck/Append；AuthenticatedRoot、CompleteAuthorizedClosure、NoDanglingRestore。`DependencyTraversal`：CompleteClosure | 完整认证图是下层读取边界；缓存上下文不匹配必须重新认证，owner／root 不匹配必须在任何 owner 追加前拒绝。 |
-| G3 | `GCDiscovery`、`GCBatchDeletion`、`GCParentRepair`、`GCReceipt`：发现、mark、unlink、共享父目录／祖先屏障、回执；DurableCompletion、ReceiptIsSafe、ReceiptKeepsFence | 多父目录和缺失父目录义务独立，stat 仅供参考。`OfflineRoots`／`OfflineAliases` 增加停机及路径绑定根，不授权并发清理。 |
+| G3 | `GCConcurrentDeletion`：PlanFenced、ReceiptScoped、OwnedPresent；`GCDiscovery`、`GCBatchDeletion`、`GCParentRepair`、`GCReceipt`：发现、mark、unlink、共享父目录／祖先屏障、回执；DurableCompletion、ReceiptIsSafe、ReceiptKeepsFence | 多父目录和缺失父目录义务独立，stat 仅供参考。`OfflineRoots`／`OfflineAliases` 增加停机及路径绑定根，不授权并发清理。并发模型检查安全性；重试／进展义务仍由下层 GC 模型检查。 |
 | G4 | `WriterRetirement`：register/hold/submit/finish、fence/drain/authenticate/recover/release/settle/seal/receipt；AuthenticatedRecovery、ReceiptSound、RequiredProgress。`WriterLifetime`：在途工作。`DurableHandoff`：EventuallyReclaimed | Gate 交接必须完成／取消旧远程 I/O；release 与 registry settlement 是独立崩溃点。活性需要稳定、公平调度和真实调用者重试。 |
 
 | 规则 | 实际调用链 | 回归／反例入口 |
 |---|---|---|
 | G1–G2 | `queue_roots::collect`、`queue_reader`、`queue_gc::prepare/require`；`gc::VerifiedReactivation::verify`、`check_reactivation_owners`、`retain_verified_inner`、`closure_inner` → `closure::collect`、`reference_node`、`record_index` | `tests/test_online_gc.py`、`test_queue_gc_roots.py`、`test_path_identity.py`；`tests/native.rs` 的 pool／run、source 丢失和 owner 替换竞态；TLC 旧快照、错误上下文／根、遗漏依赖 |
-| G3 | Coordinator collection → `queue_retirement`、`queue_gc_prepare`、`gc_collection::Batch::run` → `gc_mark`、`gc_batch`、`gc_io`、`directory_io`、`gc_receipt` | `tests/test_gc_batch.py`、`test_gc_receipts.py`、`test_gc_missing_parent.py`、`test_offline_aliases.py`；原生缺失文件／失败屏障／共享父目录；TLC 提前回执、丢失 tombstone、遗漏父目录 |
+| G3 | Coordinator collection → `queue_retirement`、`queue_gc_prepare`、`gc_plan::{plan_collection,Plan::execute,complete_collection}` → `gc_mark`、`gc_batch`、`gc_io`、`directory_io`、`gc_receipt` | `tests/test_gc_concurrency.py`、 `test_gc_batch.py`、`test_gc_receipts.py`、`test_gc_missing_parent.py`、`test_offline_aliases.py`；原生缺失文件／失败屏障／共享父目录；TLC 提前回执、丢失 tombstone、遗漏父目录 |
 | G4 | Store 修改 → `writer_gate_flow`、`writer_gate::Registry`；`Config::retire_writer` → `writer_retirement::execute`、`gc_writer::Retirement/Recovery`、`writer_registry::reconcile_registered`、`writer_ownership::authenticate_all` | `tests/test_writer_retirement.py`；原生 gc_writer 的损坏、recover／abandon、已登记 owner；TLC death-unlock、跳过 registry／认证、提前 release／receipt 及条件进展 |
 
 </details>
@@ -199,4 +208,6 @@ Owner 持续存在或错误反复发生时，安全性仍须成立。最终回
 
 回归重点包括 WAL 前释放 staging、错误后复用部分状态、漏掉 reader／队列 checkpoint 根、目录同步前写回执，以及既有 owner 保护错误根。它们对应上表的真实调用点；公共 `checkpoint:*` owner 与队列 checkpoint 不能相互替代测试。按[贡献指南](../CONTRIBUTING_zh.md#checks)运行检查，报告当前输入和实际结果。
 
+`check_mutations.py` 还对实际 Rust 代码注入选定故障，要求指定运行时回归失败；编译错误、缺失测试和超时均不算检出。
+
 有限模型、具体执行回归与源码审查提供互补证据，不构成完整 Rust／Python／FFI 等价证明、无界定理或文件系统掉电认证。进程退出保留内核和存储服务；不能据此推断真实存储服务故障行为。变更规则时同步模型、调用者、环境登记和中英文正文；验证日志保存在 CI artifacts 或忽略目录中，不追加成规格正文。

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/USAGE.md

@@ -354,9 +354,29 @@ On recovery, first restore the chosen checkpoint's model/optimizer/RNG and its m
 
 ## Scoped batch reads
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` loads a caller-bounded batch
+in its original order. Bound the total requested payload bytes before calling;
+each returned tensor has independent writable storage. Native workers sort
+requests by pack/extent/ordinal, reuse the extent handle and combine nearby reads
+in a window of at most 1 MiB per worker. Gaps larger than 4 KiB are not prefetched.
+Every selected frame, typed layout and touched checksum chunk is still checked.
+Failed attempts discard both the handle and window before the ordinary bounded
+visibility retry; windows are discarded at batch exit. The lower-level
+`reader.read_tensor_ranges(..., read_ahead_bytes=0)` disables read-ahead for
+comparisons (the maximum supported setting is 8 MiB per worker).
+
+For one publish/queue handoff, `with store.validation_scope(coordinator):` reuses
+successful immutable dependency proofs across participating operations in the
+current thread. Each new root still checks its descriptor and task/attempt;
+catalog transfers still check namespace identity, ownership authorization and
+tombstones under the catalog lock. The scope never pins data, never crosses
+storage identity, closes on exit, and is invalidated by any participating error.
+Keep input owners alive throughout it. Explicit `store.validate` and later
+operations still revalidate; the scope does not cache payload read results.
+
 Use one `with store.read_session() as reader:` block for adjacent immutable publications. The reader provides `validate`, `manifest`, `read`, `read_record`, `envelope`, and checked tensor reads. Pass it to `TensorRef.from_record_set_many(store, publication, reader=reader)` and `tensor.load(reader=reader)` to avoid repeatedly parsing a shared extent index.
 
-The Rust reader retains only the last authenticated index, keyed by the complete extent descriptor. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again after an extent switch or in a new session. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.
+The Rust reader caches up to eight authenticated indexes using an LRU keyed by the complete extent descriptor, with an 8 MiB encoded-metadata budget. A single oversized active index evicts all other entries. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again on a cache miss or in a new session. Calls within one session serialize; use independent sessions for parallel reads. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.
 
 Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, owner-release rules remain unchanged; logical capacity quotas are no longer enforced.
 

{straw_queue-0.1.5 → straw_queue-0.1.7}/docs/USAGE_zh.md

@@ -193,9 +193,25 @@ b.submit_tasks("b-1", [TaskSpec("task-b", input_ref=dependency)])
 
 ## 批量读取会话
 
+`TensorRef.load_many(refs, reader=reader, workers=4)` 按原顺序读取调用方限定的批次。
+调用前应限制请求载荷的总字节数；返回张量各自拥有可写存储。
+Native worker 按 pack/extent/ordinal 排序，复用 extent 句柄，以每个 worker
+最多 1 MiB 的窗口合并邻近读取，不预读超过 4 KiB 的间隙。
+每个选中 frame、typed layout 和触及的 checksum chunk 仍逐一检查。
+读取失败后，先丢弃句柄和窗口，再执行原有有限可见性重试；批次结束也丢弃窗口。
+底层 `reader.read_tensor_ranges(..., read_ahead_bytes=0)` 可关闭预读用于对照，
+该参数最大为每个 worker 8 MiB。
+
+单次发布／队列交接可用 `with store.validation_scope(coordinator):`，
+在当前线程的参与操作之间复用成功的不可变依赖校验证据。
+新根仍检查描述符和 task/attempt；catalog 转移仍在锁内检查 namespace 身份、
+所有权授权和 tombstone。Scope 不保留数据所有权，不跨存储身份，退出时关闭，
+任意参与操作失败立即使其失效。整个操作期间仍须保持输入 owner。
+显式 `store.validate` 和后续操作仍重新校验；scope 不缓存载荷读取结果。
+
 连续读取不可变 publication 时，可共用一个 `with store.read_session() as reader:`。Reader 提供 `validate`、`manifest`、`read`、`read_record`、`envelope` 和带校验的张量读取。将它传给 `TensorRef.from_record_set_many(store, publication, reader=reader)` 与 `tensor.load(reader=reader)`，可避免重复解析同一 extent 的索引。
 
-Rust reader 只保留最后一个已认证索引，以完整 extent 描述符区分身份。每次 manifest/record 读取仍校验 frame 和载荷；张量读取仍校验触及的 chunk。每次 `validate` 都重新检查逻辑描述符与 task/attempt 权限。切换 extent 或开启新会话后重新认证索引。整个读取过程必须由持久 owner 或 `store.pin` 保护：读取会话本身不保留存储所有权。退出上下文会关闭会话，fork 后继承的会话拒绝所有操作。
+Rust reader 使用最多八条已认证索引的 LRU，以完整 extent 描述符区分身份，编码元数据预算为 8 MiB；一个超大活动 index 会淘汰其他条目。每次 manifest/record 读取仍校验 frame 和载荷；张量读取仍校验触及的 chunk。每次 `validate` 都重新检查逻辑描述符与 task/attempt 权限。缓存未命中或开启新会话后重新认证索引。同一会话的调用串行，并行读取应使用独立会话。整个读取过程必须由持久 owner 或 `store.pin` 保护：读取会话本身不保留存储所有权。退出上下文会关闭会话，fork 后继承的会话拒绝所有操作。
 
 发布依赖校验、批量续跑进度、消费者快照及 GC 遍历在 Rust 内部也复用这种有界 reader。文件格式、WAL 提交顺序、容量限制和所有权释放规则均未改变。
 

{straw_queue-0.1.5 → straw_queue-0.1.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.5"
+version = "0.1.7"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"

{straw_queue-0.1.5 → straw_queue-0.1.7}/rust/catalog_runtime.rs

@@ -13,6 +13,7 @@ pub type CollectionResult<E> = Result<
     crate::gc_collection::Collected,
     crate::gc_batch::Failure<crate::gc_collection::Error<E, E, E>>,
 >;
+#[cfg(test)]
 pub type QueueCollectionResult<E> =
     Result<crate::gc_collection::Collected, crate::gc_collection::QueueFailure<E, E, E>>;
 
@@ -309,6 +310,7 @@ impl<S: Source> Catalog<S> {
 }
 
 impl<S: Source> Catalog<S> {
+    #[cfg(test)]
     pub(crate) fn collect_queue_native<L, H, P>(
         &mut self,
         prepared: crate::queue_gc::Prepared<'_>,