straw-queue 0.1.3__tar.gz → 0.1.5__tar.gz

straw_queue-0.1.5/.agents/skills/straw-protocol-review/SKILL.md

@@ -0,0 +1,77 @@
+---
+name: straw-protocol-review
+description: Review or change straw's persistence, ownership, recovery, GC, publication and read protocols. Use for protocol review, TODO acceptance, TLA+ changes, or Rust changes affecting durable ordering, crash/retry, request/attempt/epoch admission, reader roots or authentication. Compare the human protocol document to TLA+, then TLA+ to actual implementation; identify discrepancies and validate fixes with relevant regressions.
+---
+
+# straw 协议审查
+
+以仓库根目录为工作目录。先读 `docs/DESIGN_zh.md`
+（英文版去掉 `_zh`），再按三部分中的“模型与实现对应”定位模型、调用链和测试。
+文档是人审查的规则入口；映射表只是导航，不能替代阅读当前源码。
+
+## 1. 确定规则与范围
+
+从用户要求、diff 和 `TODO.md` 确定受影响的规则：
+
+- **分布式协议 C1/C2/C4**：跨日志 handoff、请求/attempt/epoch、checkpoint 回退。
+- **WAL 与数据 I/O C3/R1–R4**：日志与派生状态恢复、准备与发布重试、认证读取、独立可见性与有限重试。
+- **所有权与 GC G1–G4**：完整根、认证与缓存、删除屏障与回执、writer 退休和条件活性。
+
+追踪一条完整流程，包括入口、成功返回、报错但已生效、崩溃恢复和重复调用。
+区分 accepted、applied、protected、reclaimed。禁止通过扩大环境假设或把失败步骤
+合成原子动作来消除反例。没有格式变化就保留现有 wire format 和 API。
+
+## 2. 文档 → TLA+
+
+逐条检查对应 `.tla`、`.cfg` 和 `verification/runners/check_tla.py` 的实际配置：
+
+1. 文档的状态、前置条件、持久化点、错误路径分别对应哪些变量和动作？
+2. 锁、write、sync、apply、release、unlink、目录屏障、receipt 的可观察顺序是否保留？
+3. 关键性质是否在运行配置中启用？正例能否成功，避免“一律拒绝”假安全？
+4. 负例是否因为指定不变量/时序性质失败？解析错误、超时和工具错误不是有效反例。
+5. 有限实例抽象了什么？跨日志模型的完整事务不能替代 WAL 的碎片写入模型。
+6. 活性是否单列公平性、锁/I/O 返回、环境稳定和实际调用者继续重试？永久 owner
+   与永久无效输入应有独立安全/拒绝场景，不能假设它们最终消失或成功。
+
+审查 `verification/environment_contracts.json` 和 `docs/DESIGN_zh.md#environment`：
+环境保证与 straw 调用者义务分别列出。尤其不能把“已认证闭包”“正确根集合”或
+“正确 callback”当成外部保证，跳过实际构造与调用。
+
+## 3. TLA+ → 实现
+
+从公开入口向下读真实 Rust/Python/FFI 调用路径，不能只核对同名 helper。
+对每个关键动作记录：实际函数、guard、数据身份、持久化效果、错误后可用状态和恢复入口。
+同时检查返回值和后续调用者是否真的遵守 guard。
+
+重点反查这些越界：
+
+- queue WAL 确认前释放 staging/根；旧 handoff 消耗新一代 staging。
+- 部分 apply、I/O 错误或 caught unwind 后仍允许写入、缓存回复或 GC；恢复用了另一条日志。
+- stale attempt/epoch 被新请求接受；旧成功请求因数据已回收而不能重放。
+- reader/checkpoint/其他队列/传递依赖根遗漏；锁外认证后没有锁内重查。
+- 缓存跨 pool/run/namespace/reference 复用；既有可变 owner 名不保护所请求的根。
+- unlink/NotFound 后目录屏障未完成就记回执；回执删除永久 tombstone。
+- writer fence 未排空旧 I/O；未认证结果就恢复 staging；release 与登记 settlement 混成一步。
+- warm index cache 跳过 frame/envelope/payload 校验；失败读取不关闭重开；永久损坏被当成功。
+
+对发现的实现错误，先保留可失败的实际回归，再修复并跑正常路径。
+修改模型和文档时同步中英文、规则 ID、映射和环境登记；不添加重复的协议正文。
+普通确定性 helper 优先编译器、Clippy 和有意义的边界测试，不重建局部证明体系。
+
+## 4. 验证与交付
+
+按风险运行 `CONTRIBUTING.md#checks` 中的命令。模型改动至少运行对应 area、
+关联组合模型和环境登记检查；完整版本验收运行全部保留模型、Rust/FFI 回归及 crash campaign。
+运行 Python 测试前确认加载的 native 扩展由当前源码构建，避免测试旧 wheel。
+只有新的修改、失败或未解决疑点才需要重复已通过的检查。
+
+输出须能让人复查：
+
+- 结论及受影响规则；文档→TLA、TLA→实现分别有哪些一致点和差异。
+- 每个问题的具体触发/崩溃窗口、影响、源码位置、复现及修复证据。
+- 实际运行命令、输入/工具版本或哈希、结果，以及未运行的部分。
+- 有意保留的抽象差异、环境前提和剩余风险；没有发现问题时也报告这些边界。
+- 删除验证时说明原来防止的错误、保留证据、减少的保证与维护成本；性能只报可比实测。
+
+有限模型检查、源码审查和具体回归是三种不同证据。不能把 CI 全绿写成完整实现等价证明，
+也不能仅删除 TODO 或改标记来宣布完成。只有对应交付物与验收结果齐全才勾选。

straw_queue-0.1.5/.agents/skills/straw-protocol-review/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "straw 协议审查"
+  short_description: "Review straw docs, TLA+ models and implementation consistency"
+  default_prompt: "Use $straw-protocol-review to review the affected protocol flows, compare docs to TLA+ and TLA+ to implementation, and validate any fixes with relevant evidence."

{straw_queue-0.1.3 → straw_queue-0.1.5}/.github/workflows/tests.yml

@@ -2,13 +2,48 @@ name: tests
 
 on:
   push:
+    branches: ['**']
   pull_request:
   workflow_dispatch:
+  workflow_call:
 
 permissions:
   contents: read
 
 jobs:
+  formal:
+    name: TLA+ / shard ${{ matrix.shard }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '17'
+      - name: Check specifications and expected counterexamples
+        run: >-
+          python verification/runners/check_tla.py
+          --shard-index ${{ matrix.shard }} --shard-count 16
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: tla-verification-${{ matrix.shard }}
+          path: |
+            target/tla/report.json
+            target/tla/environment_contracts.json
+            target/tla/**/*.log
+            target/tla/**/*.tla
+            target/tla/**/*.cfg
+          if-no-files-found: error
+
   checks:
     name: Rust tests and code checks
     runs-on: ubuntu-latest
@@ -23,6 +58,8 @@ jobs:
           python-version: '3.12'
       - name: Install formatters
         run: python -m pip install 'black>=24.8' 'isort>=5.13'
+      - name: Check environment contract inventory
+        run: python verification/runners/environment_manifest.py --check
       - name: Check formatting
         run: |
           cargo fmt --check
@@ -54,7 +91,7 @@ jobs:
           python -m pip install 'build>=1.2' 'twine>=5' 'numpy>=1.24' 'pytest>=8'
           python -m pip install 'torch>=2.3' --index-url https://download.pytorch.org/whl/cpu
       - name: Build the wheel
-        run: python -m build --wheel
+        run: python tools/release_build.py python -m build --wheel
       - name: Check package metadata
         run: python -m twine check dist/*
       - name: Run all Python and protocol tests against the installed wheel
@@ -67,3 +104,25 @@ jobs:
           name: test-report-py${{ matrix.python }}
           path: wheel-check.json
           if-no-files-found: warn
+
+  verification:
+    name: Verification passed
+    needs: [formal, checks, python]
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Require every verification job to succeed
+        env:
+          RESULTS: ${{ toJSON(needs) }}
+        run: |
+          python3 - <<'PY'
+          import json
+          import os
+
+          results = json.loads(os.environ["RESULTS"])
+          assert set(results) == {"formal", "checks", "python"}
+          failed = {name: job["result"] for name, job in results.items() if job["result"] != "success"}
+          if failed:
+              raise SystemExit(f"Verification did not pass: {failed}")
+          PY

{straw_queue-0.1.3 → straw_queue-0.1.5}/.github/workflows/wheels.yml

@@ -7,7 +7,11 @@ on:
 permissions:
   contents: read
 jobs:
+  verification:
+    name: Verify the release source
+    uses: ./.github/workflows/tests.yml
   build:
+    needs: verification
     runs-on: ubuntu-latest
     timeout-minutes: 30
     strategy:
@@ -21,7 +25,11 @@ jobs:
           target: x86_64
           # The action adds --manylinux; do not also pass --compatibility in args.
           manylinux: '2_28'
+          before-script-linux: |
+            export CARGO_ENCODED_RUSTFLAGS="$(python3 tools/release_build.py --print-flags)"
           args: --release --locked --strip --out dist --interpreter /opt/python/cp${{ matrix.python-tag }}-cp${{ matrix.python-tag }}/bin/python
+      - name: Check release artifact privacy
+        run: python3 tools/check_artifact_privacy.py dist/*.whl
       - uses: actions/upload-artifact@v4
         with:
           name: wheel-cp${{ matrix.python-tag }}-linux-x86_64
@@ -63,6 +71,7 @@ jobs:
           name: wheel-check-cp${{ matrix.tag }}
           path: wheel-check.json
   sdist:
+    needs: verification
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
@@ -73,9 +82,10 @@ jobs:
           python-version: '3.12'
       - run: python -m pip install 'build>=1.2' 'twine>=5' 'numpy>=1.24' 'pytest>=8'
       - run: python -m pip install 'torch>=2.3' --index-url https://download.pytorch.org/whl/cpu
-      - run: python -m build --sdist
+      - run: python tools/release_build.py python -m build --sdist
       - run: python -m twine check dist/*
-      - run: python -m pip wheel --no-deps dist/*.tar.gz --wheel-dir rebuilt
+      - run: python tools/release_build.py python -m pip wheel --no-deps dist/*.tar.gz --wheel-dir rebuilt
+      - run: python tools/check_artifact_privacy.py dist/*.tar.gz rebuilt/*.whl
       - run: python -m twine check rebuilt/*
       - run: python tools/check_wheel.py rebuilt/*.whl --tests --report wheel-check.json
       - uses: actions/upload-artifact@v4
@@ -90,7 +100,7 @@ jobs:
           path: wheel-check.json
   publish:
     name: Publish to PyPI
-    needs: [smoke, sdist]
+    needs: [verification, smoke, sdist]
     if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
     runs-on: ubuntu-latest
     timeout-minutes: 10

straw_queue-0.1.5/CHANGELOG.md

@@ -0,0 +1,136 @@
+# Changelog
+
+[中文版](CHANGELOG_zh.md)
+
+## 0.1.5
+
+### Ownership safety
+
+- Fix reactivation reusing an existing public owner without checking that it protects the requested root, including owner changes after authentication outside the lock. Check the retained pack closure both initially and under the fresh lock; internal queue owners bind the reference digest. Correct reuse still succeeds; public owner reuse adds closure/metadata checks.
+- Preserve public APIs, WAL/pack formats and durability ordering. Add regressions for existing-owner and post-authentication changes, and for staging remaining protected until queue WAL confirmation.
+
+### Specification, verification and maintenance
+
+- Remove all Verus source, inline specifications, ghost state, library contracts, runner, negative controls and CI; simplify Rust wrappers. Remove six deterministic local TLA+ models while retaining complete protocol models and product fault regressions.
+- Extend cross-log attempt/epoch and checkpoint rewind, WAL caller/cache, reactivation, authenticated read, writer retirement and conditional-progress scenarios. Distinguish model checking, execution tests and source review without claiming complete implementation equivalence.
+- Merge design, protocol, environmental premises and model-to-code correspondence into the [design and spec](docs/DESIGN.md). Move application integration into usage and checks/builds/releases into contributing; update both languages and add the `straw-protocol-review` skill.
+- Delete per-iteration proof, measurement and acceptance reports. Retain only three public benchmarks cited by the performance guide and explicitly list them in source-package manifests. Their results describe the original 0.1.4 candidate, not this version's performance.
+
+## 0.1.4
+
+### Recovery and continuation
+
+- Preserve producer continuation payloads and independent reader, consumer and checkpoint roots during online GC. Sampling can resume from its durable continuation after process failure. Training applications must select and load a checkpoint with matching model/optimizer state and data cursors, and retain data needed for replay; straw does not automatically load a model checkpoint or recreate reclaimed data.
+- Resolve durable receipts before reopening old references when retrying accepted consumer-state, batch-plan, batch-ready and checkpoint-registration requests. Identical retries can succeed after those references have been legitimately reclaimed, without creating new ownership.
+- Persist staging cleanup intent, dependency recipes and generation-specific completion receipts. Recovery can finish an accepted handoff after a lost reply without releasing a later staging generation of the same reference.
+- Quarantine uncertain WAL writes, partial state application and caught Rust callback panics until recovery. Reject malformed journal/catalog events before appending, preserve confirmed prefixes during rebuild and reject changed pool/run or log identity. Harden counter arithmetic and deduplicate repeated task IDs before lease selection.
+
+### Storage and reads
+
+- Retry independent visibility delays in the header, footer, index, envelope, payload and tensor checksum chunks within one bounded budget. Every returned record remains authenticated; persistent corruption fails after the budget expires.
+- Reuse the authenticated index descriptor within a read attempt and buffer catalog replay with at most 64 KiB of read-ahead. Failed read attempts reopen their files. Replace recursive dependency walks with deduplicated worklists to support deep graphs.
+- Validate tensor dtype, shape and length before reading payloads. New tensors use 4 MiB checksum chunks; older tensors retain their stored chunk sizes. Checkpoint reactivation authenticates all reachable payloads outside the catalog lock with at most 1 MiB of payload scratch, then rechecks ownership and tombstones under the lock.
+- Preserve exact finite JSON floating-point values through serialization and recovery. Reject native payload providers whose traversals have the wrong length or swallow callback failures.
+
+### Garbage collection and publication safety
+
+- Retry directory barriers after unlink, including already absent files and missing parents. Batch shared-parent barriers and persist reclamation receipts only after deletion is durable. Preserve permanent tombstones; a cold-cache retain of a reclaimed root or dependency now returns `InvalidReference` before trying to read its missing payload.
+- Protect canonical paths and aliases, peer queues, active readers, checkpoints, producer continuations and pending publications. Public owner APIs reject internal `queue:` and `publication:` names. Offline cleanup preserves control files, rejects symlink cycles on Python 3.13 as well as earlier versions, and fails closed on uncertain shared-pool discovery; dangling orphan links remain eligible for cleanup.
+- Preserve dependency holds and retry identities through failed publication setup, synchronization, staging and sealing. Repair missing ancestor barriers, roll back unpublished tails before writer reuse and seal completed standalone reservations so they can be collected. A failed GC hook with partial WAL effects forces fresh recovery before further writes.
+
+### Writer retirement
+
+- Add explicit `retire_writer(..., policy="recover" | "abandon")` with a durable fence, operation registry and reusable recovery receipts. Recovery stages complete results before releasing preparation ownership; abandonment preserves existing queue, reader and staging roots. A writer holding its gate reports `WriterBusy`, and a fenced writer cannot resume writing. Retirement depends on the documented lock, fence-visibility and in-flight-I/O contracts; timeouts alone do not establish death.
+- Reject busy writer metrics, seal and close operations without waiting on an in-flight callback. Unknown legacy preparation owners remain protected and are reported for separate recovery.
+
+### Compatibility and migration
+
+- Packed-record and WAL byte framing remain unchanged, and legacy catalog entries still replay. New queue staging-handoff events require updated coordinators: do not downgrade a queue writer after those events are stored. Upgrade all GC participants before relying on canonical-alias protection; an old collector can still delete an aliased owned file. Mixed versions do not gain a uniform cleanup/retry guarantee; historical leaks and already reclaimed legacy manifests are not automatically repaired. See [format compatibility](docs/FORMAT.md#queue-staging-handoff-receipts).
+- Rust callers must use read-only coordinator accessors (`state()`, `journal()`, `store()`, `limits()`, `epoch()`), publish through a separate `Store`, and replace explicitly typed `&mut Catalog` callbacks with `&mut LockedCatalog<'_>`. `Config::gc_enabled` returns `Result<bool>`. Existing Python call signatures are retained, with the retirement APIs added.
+- Previously rounded floats cannot be reconstructed. Old rounded request hashes and retries generated under a different tensor chunk policy can conflict after upgrade; preserve retry identities and review the [stored-format rules](docs/FORMAT.md). Catalog recipes and receipts currently grow without compaction.
+
+### Known performance regressions
+
+- In the same JuiceFS workload with online GC, median write throughput fell 20.7% and acceptance time for 64 tasks rose 34.8% versus 0.1.3; reads and synchronous WAL appends showed no comparable decline. This short comparison used two rounds per build with warm caches. It was not profiled, so the changes cannot be attributed to a particular safety protocol or proof-related refactoring. See the [complete samples and build hashes](verification/measurements/20261026-release-comparison.json).
+
+### CI and verification
+
+- Scan source distributions and wheel contents, including native libraries, for local paths and recognizable credentials. Remap Rust build paths before compilation so released binaries omit local source, home and toolchain locations; CI checks artifacts before upload and rejects findings without printing matched secrets.
+- Require TLA+/Verus checks, semantic negative controls, environment-contract audits, Rust/Python tests, formatting and Clippy before release builds and publishing. TLC records its jar hash without enforcing a checksum; Verus logs distinguish successful proofs from expected semantic rejections. Verification files are organized by protocol, WAL, GC, storage and offline recovery, with independently reviewable environment assumptions.
+- Freeze new formal-verification work for this release. Integrated proofs cover specific production bodies and models; complete retain/reactivation, cross-log recovery, authentication and liveness composition remain deferred and receive regression/fault-injection coverage. That scope describes the historical release. Passing tests does not complete these proofs, and prior performance measurements do not establish this build's throughput or latency.
+
+## 0.1.3
+
+- Retry temporarily zero-filled bytes anywhere in a published extent's header,
+  including partial and noncontiguous visibility gaps. Previously, only an
+  entirely zero-filled header triggered this retry. The retry limit and checksum
+  validation remain unchanged; other incorrect header bytes fail immediately.
+
+Existing callers and stored data remain compatible. Packed-record and WAL
+formats are unchanged.
+
+## 0.1.2
+
+- Stream large record payloads through bounded native write buffers. Records and
+  publications no longer need to fit the writer's scratch buffer; existing
+  packed-record framing and tensor reads are unchanged.
+- Remove fixed logical quotas on record and dependency counts, result sizes,
+  queue work, control messages, and metadata. Legacy limit arguments remain
+  accepted for API and queue-identity compatibility; `max_buffer_bytes` still
+  bounds temporary payload copies.
+- Retry transient shared-filesystem visibility gaps when a published extent's
+  length is visible before its header bytes. The reader reopens the file after
+  short delays; persistent failures, nonzero bad headers, and checksum errors
+  remain errors.
+
+Existing stored data and callers remain compatible. These write and read
+behaviors require 0.1.2; packed-record and WAL formats are unchanged.
+
+## 0.1.1
+
+- Persistent task scheduling with `TaskSpec.priority` (higher first),
+  `scheduling_key` (lower first), then FIFO. Both keys default to zero and accept
+  signed 64-bit integers. The Rust pending index selects tasks without reading
+  their payloads and reconstructs scheduling order during WAL recovery.
+- `Coordinator.yield_tasks()` atomically saves a batch of continuation inputs
+  and optional scheduling fields, ends their leases, and returns the tasks to
+  the pending queue without spending failure retries. Returned tasks join the
+  tail of their scheduling key's FIFO order. Identical requests are idempotent;
+  a stale lease rejects the entire batch.
+- `Coordinator.pending_tasks()` exposes pending task specifications in
+  acquisition order, with optional task-prefix and minimum-priority filters.
+  Applications coordinate paused snapshots and retain the referenced inputs.
+- Regression coverage for priority/FIFO recovery, stale leases, invalid
+  scheduling keys, and all-or-nothing batch recovery across WAL crash windows.
+
+Existing callers may omit the new scheduling fields. Packed-record and WAL
+framing are unchanged; the new APIs and scheduling behavior require 0.1.1.
+Pending-task snapshots alone do not provide a joint application checkpoint or
+arbitrary checkpoint rollback.
+
+## 0.1.0
+
+Initial release of straw, a filesystem-based durable queue and shared tensor
+store for AI applications, distributed as `straw-queue` under the MIT license.
+
+- Rust storage, WAL, queue and ownership protocols with Python APIs. Multiple
+  machines can read and write through a shared filesystem; the initial shared
+  storage target is JuiceFS.
+- Packed immutable records and embedded manifests, batched publication and
+  size-based file rotation, avoiding a file per sample or tensor.
+- Bytes and custom codecs, NumPy/PyTorch tensors, checked lazy row reads and
+  native read sessions that reuse authenticated indices.
+- Shared tensor references across queues, tensor-level copy-on-write, explicit
+  reader/checkpoint ownership and optional online GC of unreferenced sealed packs.
+- Durable task leases, continuation progress, completion receipts and consumer
+  state, with WAL recovery and retained-checkpoint restore.
+- Local and multi-host I/O benchmarks, workload profiling and replay, an
+  independent protocol model, and process-crash/recovery checks.
+- Linux x86_64 wheels for CPython 3.10–3.13, automated artifact tests and PyPI
+  Trusted Publishing, standalone examples and paired English/Chinese guides.
+
+Each queue requires one externally fenced coordinator. Automatic failover,
+live-pack compaction, journal compaction and a general distributed application
+checkpoint transaction are not provided. Additional network filesystems require
+deployment validation. See the [filesystem requirements](docs/FILESYSTEM.md) and
+[protocol guarantees](docs/DESIGN.md).

straw_queue-0.1.5/CHANGELOG_zh.md

@@ -0,0 +1,97 @@
+# 更新日志
+
+[English](CHANGELOG.md)
+
+## 0.1.5
+
+### 所有权安全
+
+- 修复 reactivation 复用既有公共 owner 名时没有确认其保护请求根的问题，包括锁外认证后 owner 被变更的竞态。在初次检查与重新加锁后核对实际 pack 闭包；内部 queue owner 绑定引用摘要。正确复用仍可成功，公共 owner 复用增加闭包／元数据检查。
+- 保持公开 API、WAL 和 pack 格式及持久化顺序。新增既有 owner／认证后变更回归，以及 queue WAL 确认前 staging 不得被消费的回归。
+
+### 规格、验证与维护
+
+- 完全移除 Verus 源码、内嵌规格、ghost state、库契约、运行器、负例与 CI，简化 Rust 包装层。删除六个确定性局部 TLA+ 模型，保留完整协议模型和产品故障回归。
+- 补齐跨日志 attempt／epoch 与 checkpoint 回退、WAL 调用者／缓存、reactivation、认证读取、writer 退休及条件进展场景。模型检查、实际回归和源码审查分别说明，不宣称完整实现等价证明。
+- 将设计、协议、环境假设及模型／实现对应合并到[设计与 spec](docs/DESIGN_zh.md)。应用集成并入使用指南，检查／构建／发布并入贡献指南；中英文同步，新增 `straw-protocol-review` skill。
+- 删除逐轮证明、测量和验收过程报告，仅保留性能文档引用的三份公开基准；源码包明确列出允许包含的数据。保留结果对应原 0.1.4 候选构建，不代表本版性能。
+
+## 0.1.4
+
+### 恢复与续读
+
+- 在线 GC 保留 producer continuation，以及独立的 reader、consumer 和 checkpoint 根。采样进程失败后可以从已持久化的 continuation 恢复。训练应用仍须选择并加载 checkpoint，恢复相匹配的模型／optimizer 状态与数据游标，并保留重放需要的数据；straw 不会自动加载模型 checkpoint，也不能恢复已回收的数据。
+- 重试已接受的 consumer state、batch plan、batch ready 和 checkpoint registration 时，先查询持久回执，再决定是否读取旧引用。即使旧数据已合法回收，相同请求仍可返回原结果，不会重新建立所有权。
+- 持久化 staging 清理意图、依赖关系和对应代际的完成回执。丢失回复后，恢复可以完成已接受的交接，不会释放同一引用后来重新建立的 staging。
+- WAL 写入结果不确定、状态部分应用或捕获 Rust callback panic 后，保持隔离直至恢复。追加前拒绝畸形 journal／catalog 事件，重建时保护已确认前缀，并拒绝池／run／日志身份变化。补齐计数溢出检查，发放 lease 前对重复 task ID 去重。
+
+### 存储与读取
+
+- header、footer、index、envelope、payload 和 tensor checksum chunk 可在同一有限预算内重试各自的可见性延迟。返回的数据始终经过认证；持续损坏会在预算耗尽后报错。
+- 同次读取尝试复用认证索引的文件描述符，catalog replay 使用至多 64 KiB 预读；失败后重新打开文件。依赖遍历改用去重工作队列，支持深层依赖图。
+- 读取 payload 前检查 tensor dtype、shape 和长度。新 tensor 使用 4 MiB checksum chunk，旧数据沿用存储的 chunk 大小。checkpoint 重新激活在 catalog 锁外认证全部可达 payload，payload scratch 至多 1 MiB，随后在锁内重新检查所有权和 tombstone。
+- JSON 有限浮点数在序列化／恢复中精确往返。拒绝遍历长度不符或吞掉 callback 错误的原生 payload provider。
+
+### GC 与发布安全
+
+- unlink 后重试目录持久化屏障，包括文件已不存在或父目录缺失的情况。共享父目录的屏障合并执行，持久删除完成后才记录回收回执。永久保留 tombstone；冷缓存 retain 遇到已回收的根或依赖时，在读取缺失 payload 前返回 `InvalidReference`。
+- 保护规范路径及别名、其他队列、活跃 reader、checkpoint、producer continuation 和未完成 publication。公共 owner API 拒绝内部 `queue:`、`publication:` 名字。离线清理保留控制文件，在 Python 3.13 及更早版本中均拒绝符号链接环，共享池发现结果不确定时拒绝继续删除；无引用的悬空链接仍可清理。
+- 发布准备、同步、staging、sealing 失败时保留依赖保护和重试身份。补齐缺失的祖先目录屏障，writer 复用前回滚未发布尾部，并封存已完成的 standalone reservation 以便 GC。GC hook 报错且已影响 WAL 时，后续写入必须先重新恢复。
+
+### Writer 退休
+
+- 新增显式 `retire_writer(..., policy="recover" | "abandon")`，通过持久 fence、操作登记和可复用恢复回执处理旧 writer。recover 先保护完整结果再释放准备所有权；abandon 保留已有 queue、reader 和 staging 根。仍持有 gate 的 writer 返回 `WriterBusy`，已被 fence 的 writer 无法恢复写入。退休依赖已列明的锁、fence 可见性和在途 I/O 契约，不能仅凭超时判断进程死亡。
+- writer 正执行 callback 时，metrics、seal、close 立即返回 busy，不再等待。无法确认来源的历史 preparation owner 继续受保护，并报告为独立恢复事项。
+
+### 兼容与迁移
+
+- packed record 和 WAL 字节 framing 不变，旧 catalog 记录仍可重放。新增 queue staging handoff 事件需要新版 coordinator：一旦写入这些事件，不要降级 queue writer。依赖规范路径／别名保护前，应升级全部 GC 参与者；旧 collector 仍可能删除通过别名持有的文件。混用版本不具备统一的清理／重试保证；历史泄漏和已回收的旧 manifest 不会自动修复。见[格式兼容说明](docs/FORMAT_zh.md#队列-staging-交接凭据)。
+- Rust 调用者需改用 coordinator 只读 accessor（`state()`、`journal()`、`store()`、`limits()`、`epoch()`），通过独立 `Store` 发布，并将显式 `&mut Catalog` callback 改为 `&mut LockedCatalog<'_>`。`Config::gc_enabled` 返回 `Result<bool>`。现有 Python 调用签名保留，另增加退休 API。
+- 旧版本已舍入的浮点数无法还原；旧请求 hash 和使用不同 tensor chunk 策略生成的重试可能在升级后冲突。保留重试身份，并检查[存储格式规则](docs/FORMAT_zh.md)。catalog 依赖关系和回执暂未实现压缩，元数据会持续增长。
+
+### 已知性能回退
+
+- 相同 JuiceFS 工作负载、启用在线 GC 时，相比 0.1.3，写吞吐中位数下降 20.7%，接受 64 个任务的耗时增加 34.8%；读取和同步 WAL 追加未见同量级下降。此次简短对照每个构建只有两轮，缓存未清空；尚未做 profile，不能归因于某项安全协议或验证相关重构。见[完整样本与构建哈希](verification/measurements/20261026-release-comparison.json)。
+
+### CI 与验证边界
+
+- 扫描源码分发包与 wheel 内容（包括原生库），检查本机路径和可识别的凭据。编译时 remap Rust 路径，避免发布二进制携带本机源码、用户目录和工具链位置；CI 上传产物前检查，发现问题即拒绝，报告不输出命中的秘密内容。
+- release 构建和发布以前置检查为条件，包括 TLA+／Verus、语义负例、environment contract 审计、Rust／Python 测试、格式和 Clippy。TLC 只记录 jar 哈希、不做校验；Verus 日志明确区分证明通过与预期语义拒绝。验证目录按协议、WAL、GC、存储与离线恢复组织，环境假设可独立审查。
+- 本次发布冻结新增形式化验证。已集成证明覆盖具体生产函数和模型；完整 retain／reactivation、跨日志恢复、认证与活性组合延期，以回归和故障注入测试覆盖。该范围描述当时的版本。测试通过不等于这些证明已完成，历史性能测量也不能代表当前构建的吞吐或延迟。
+
+## 0.1.3
+
+- 已发布 extent 的文件头在任意位置出现暂时零值时都会触发重试，支持部分字节以及不连续位置尚不可见的情况。此前只有整个文件头为零时才会触发这类重试。重试上限和 checksum 校验保持不变；其他错误的文件头字节仍会立即报错。
+
+已有调用方和存储数据保持兼容，packed record 和 WAL 格式不变。
+
+## 0.1.2
+
+- 大记录的 payload 通过有界 native 写入缓冲区分块写入。记录和批量发布不再需要适配 writer 的临时缓冲区；原有 packed record 格式和张量读取方式不变。
+- 移除记录数、依赖数、结果大小、队列任务、控制消息及元数据的固定逻辑配额。旧的限制参数继续保留，以兼容 API 和队列身份；`max_buffer_bytes` 仍约束临时 payload 复制。
+- 共享文件系统上，已发布 extent 的文件长度可能先于文件头内容可见。reader 遇到这种短暂情况会延迟并重新打开文件；持续失败、非零坏文件头和 checksum 错误仍会报错。
+
+已有存储数据和调用方保持兼容。上述写入和读取行为需要 0.1.2；packed record 和 WAL 格式不变。
+
+## 0.1.1
+
+- 持久化任务调度：按 `TaskSpec.priority` 从高到低、`scheduling_key` 从低到高、再按 FIFO 顺序领取任务。两个字段默认均为 0，接受有符号 64 位整数。Rust pending 索引无需读取载荷即可选择任务，WAL 恢复时重建调度顺序。
+- 新增 `Coordinator.yield_tasks()`：原子保存一批 continuation 输入及可选调度字段，结束这些任务的 lease 并归还到 pending 队列，不消耗失败重试次数。归还的任务进入相同调度键的 FIFO 队尾；相同请求可幂等重试，任一 lease 失效则拒绝整批操作。
+- 新增 `Coordinator.pending_tasks()`：按领取顺序返回 pending 任务的 spec，支持按任务 ID 前缀及最低优先级过滤。应用负责协调暂停后的快照，并持有所引用的输入。
+- 补充优先级/FIFO 恢复、失效 lease、非法调度键，以及 WAL 崩溃窗口下批量操作全有或全无恢复的回归测试。
+
+已有调用方可以不指定新增调度字段。Packed record 与 WAL 的 framing 不变；新增 API 和调度行为需要 0.1.1。仅有 pending 任务快照尚不构成应用联合 checkpoint，也不提供任意 checkpoint 回退。
+
+## 0.1.0
+
+straw 首次发布：面向 AI 应用、基于文件系统的持久化队列与共享张量存储，以 `straw-queue` 包分发，采用 MIT 许可证。
+
+- Rust 实现存储、WAL、队列及所有权协议，提供 Python API。多台机器通过共享文件系统并行读写，首个共享存储目标为 JuiceFS。
+- 不可变记录与 manifest 打包存储，支持批量发布和按大小轮换文件，避免每个 sample 或张量单独创建文件。
+- 支持 bytes、自定义 codec、NumPy/PyTorch 张量、带校验的惰性按行读取，以及复用已认证索引的 native 读取会话。
+- 跨队列共享张量引用、张量级写时复制、显式 reader/checkpoint 所有权，以及可选在线 GC，回收无引用的封存 pack。
+- 持久化任务 lease、continuation 进度、完成 receipt 和消费者状态，支持 WAL 恢复与已保留 checkpoint 的恢复。
+- 本地与多机 I/O benchmark、应用负载采集与重放、独立协议模型，以及进程崩溃与恢复检查。
+- CPython 3.10–3.13 的 Linux x86_64 wheel、自动化产物测试与 PyPI Trusted Publishing、独立使用示例和配套中英文指南。
+
+每个队列需要一个由外部机制隔离保护的协调器。尚不提供自动故障切换、存活 pack 整理、日志压缩或通用分布式应用 checkpoint 事务。其他网络文件系统需要单独验证部署语义。详见[文件系统要求](docs/FILESYSTEM_zh.md)与[协议保证](docs/DESIGN_zh.md)。

straw_queue-0.1.5/CONTRIBUTING.md

@@ -0,0 +1,138 @@
+# Contributing
+
+[中文版](CONTRIBUTING_zh.md)
+
+Read the [design and spec](docs/DESIGN.md) before changing durable ordering,
+recovery, ownership or authentication. Use the
+[straw-protocol-review skill](.agents/skills/straw-protocol-review/SKILL.md)
+to compare rules to TLA+, then trace the models to actual callers.
+
+Protocol changes should explain success, uncertain effects, crash/retry and
+compatibility. Add a reproducer for a real failure boundary and preserve a valid
+success path. Update the rule, model, caller and regression together. Ordinary
+local functions use compiler checks, Clippy and meaningful tests; do not rebuild
+per-function proof scaffolding. Unknown wire versions fail explicitly; framing
+changes require a format version.
+
+When changing an environmental premise, update its stable ID in
+[environment_contracts.json](verification/environment_contracts.json), the
+[spec](docs/DESIGN.md#environment), consuming models and sensitivity controls.
+Do not turn straw caller obligations into assumed filesystem guarantees.
+User/contributor guides have paired English `.md` and Chinese `_zh.md` versions;
+keep links, APIs and commands aligned. Application codecs and scheduling belong
+in the consuming application. The project uses the [MIT license](LICENSE).
+
+<a id="checks"></a>
+
+## Development and checks
+
+Use Rust 1.89+, a C linker, CPython 3.10+ and Java 17+ for TLC. Install the
+appropriate CPU/CUDA PyTorch build, then the editable development package:
+
+```sh
+python -m pip install -e '.[test,build,dev]'
+python -m black --check src tests examples tools verification
+python -m isort --check-only src tests examples tools verification
+cargo fmt --check
+cargo clippy --locked --all-targets -- -D warnings
+cargo clippy --locked --all-targets --features python -- -D warnings
+cargo test --locked
+python -m pytest
+python verification/runners/environment_manifest.py --check
+python verification/runners/check_tla.py
+python verification/protocol/checks/ownership_model.py --output /tmp/straw-ownership.json
+python verification/wal/checks/crash_campaign.py --output /tmp/straw-crashes.json
+```
+
+Rebuild/reinstall the native extension after Rust changes before running Python
+tests. For focused model work, use `check_tla.py --area wal --area gc` or
+`--case NAME`; run the full retained suite for release acceptance. Supply
+`--jar /path/to/tla2tools.jar` to select TLC; otherwise the runner downloads its
+configured release. It records the actual jar hash without enforcing a checksum.
+Logs, configs and counterexamples default to `target/tla/`. Positives must pass;
+negatives must fail the named invariant/temporal property. Parse errors, timeouts
+and launch failures never count as semantic rejection.
+
+The inventory checks declared IDs, paths and environment imports. The independent
+ownership model explores a bounded abstraction; the crash campaign checks byte
+prefixes/corruption and process exits. None certifies storage-service or power-loss
+behavior. Multi-client qualification is in [deployment](docs/FILESYSTEM.md#qualification).
+
+<a id="build"></a>
+
+## Build and check distributions
+
+Only the build machine needs Rust. Dependencies are declared in `pyproject.toml`
+and pinned by `Cargo.lock`:
+
+```sh
+python -m pip install build maturin twine
+python tools/release_build.py python -m build --wheel --sdist
+python -m twine check dist/*
+python tools/check_artifact_privacy.py dist/*
+python tools/check_wheel.py dist/<compatible-wheel>.whl --tests --report /tmp/straw-wheel-check.json
+python tools/release_build.py python -m pip wheel --no-deps dist/straw_queue-*.tar.gz --wheel-dir rebuilt
+```
+
+Use the actual compatible wheel filename and check the rebuilt wheel too.
+`check_wheel.py` creates a temporary environment, reuses installed NumPy/PyTorch,
+installs only the wheel without an index, rejects source imports and removes Rust
+from the test PATH. It runs examples, multiprocess recovery, benchmark/GC and
+trace checks; `--tests` adds the full Python suite, ownership model and crash campaign.
+
+The release wrapper removes local source/toolchain paths from Rust output.
+Artifact scans check archive contents for recognizable paths/secrets without
+printing their values. Rust SBOM generation is disabled because it embeds local
+crate paths; locked dependency versions remain in `Cargo.lock`.
+
+The release workflow targets Linux x86_64, CPython 3.10–3.13 and manylinux/glibc
+2.28+. Wheels use interpreter-specific buffer APIs, not ABI3. Local builds receive
+the platform tag they actually satisfy; never rename a tag to imply compatibility.
+Use the [manylinux workflow](.github/workflows/wheels.yml), or on Linux:
+
+```sh
+python -m pip install 'maturin[zig,patchelf]>=1.12,<2'
+python tools/release_build.py maturin build --release --locked --strip --zig \
+  --compatibility manylinux_2_28 --out dist
+```
+
+<a id="release"></a>
+
+## CI and releases
+
+[Tests](.github/workflows/tests.yml) run all TLA cases in 16 disjoint shards,
+formatting, both Clippy configurations, Rust tests and installed-wheel checks on
+CPython 3.10–3.13. [Wheels](.github/workflows/wheels.yml) requires that same-commit
+workflow, checks each portable wheel and a wheel rebuilt from the sdist, then
+uploads artifacts. Failed, cancelled, skipped or timed-out checks block release.
+Hosted CI uses a local Linux filesystem and CPU PyTorch; inspect hosted results
+separately from local checks and distributed deployment tests.
+
+For a release, align the versions in `Cargo.toml`, `pyproject.toml` and the
+`straw` entry in `Cargo.lock`; record compatibility/limits in the changelog.
+A matching new `v*` tag triggers publication of the already tested artifacts.
+PRs and manual dispatch build/test without publishing. Keep the source revision
+and wheel hash with release artifacts; rebuilding changes the artifact identity.
+Do not overwrite an existing published version/tag.
+
+The publish job uses GitHub OIDC with environment `pypi`. Configure PyPI's
+publisher for project `straw-queue`, repository `zhuzilin/straw`, workflow
+`wheels.yml`, environment `pypi`; the workflow filename has no directory prefix.
+No PyPI token is required by this workflow. See the
+[PyPI publisher setup](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)
+for service setup. Both workflows also support manual dispatch from Actions.
+
+<a id="evidence"></a>
+
+## Reports and public benchmarks
+
+Keep per-run verification reports, attempted fixes, wheels, credentials, machine
+addresses, payloads and raw logs in CI artifacts or ignored local directories.
+Use fresh bounded test roots and clean them only after owned processes stop.
+
+`verification/measurements/` contains only selected public benchmark data cited
+by [benchmarks](docs/BENCHMARKS.md#published-results). A retained result needs
+workload/method, build/probe identity, units, samples and limitations. Compare
+payload sizes, publication granularity, fanout, durability and cache conditions;
+do not imply an old build's numbers describe current code. Record one-off task
+acceptance in the PR or CI result rather than adding another dated JSON file.