straw-queue 0.1.0__tar.gz → 0.1.2__tar.gz

straw_queue-0.1.2/CHANGELOG.md

@@ -0,0 +1,69 @@
+# Changelog
+
+[中文版](CHANGELOG_zh.md)
+
+## 0.1.2
+
+- Stream large record payloads through bounded native write buffers. Records and
+  publications no longer need to fit the writer's scratch buffer; existing
+  packed-record framing and tensor reads are unchanged.
+- Remove fixed logical quotas on record and dependency counts, result sizes,
+  queue work, control messages, and metadata. Legacy limit arguments remain
+  accepted for API and queue-identity compatibility; `max_buffer_bytes` still
+  bounds temporary payload copies.
+- Retry transient shared-filesystem visibility gaps when a published extent's
+  length is visible before its header bytes. The reader reopens the file after
+  short delays; persistent failures, nonzero bad headers, and checksum errors
+  remain errors.
+
+Existing stored data and callers remain compatible. These write and read
+behaviors require 0.1.2; packed-record and WAL formats are unchanged.
+
+## 0.1.1
+
+- Persistent task scheduling with `TaskSpec.priority` (higher first),
+  `scheduling_key` (lower first), then FIFO. Both keys default to zero and accept
+  signed 64-bit integers. The Rust pending index selects tasks without reading
+  their payloads and reconstructs scheduling order during WAL recovery.
+- `Coordinator.yield_tasks()` atomically saves a batch of continuation inputs
+  and optional scheduling fields, ends their leases, and returns the tasks to
+  the pending queue without spending failure retries. Returned tasks join the
+  tail of their scheduling key's FIFO order. Identical requests are idempotent;
+  a stale lease rejects the entire batch.
+- `Coordinator.pending_tasks()` exposes pending task specifications in
+  acquisition order, with optional task-prefix and minimum-priority filters.
+  Applications coordinate paused snapshots and retain the referenced inputs.
+- Regression coverage for priority/FIFO recovery, stale leases, invalid
+  scheduling keys, and all-or-nothing batch recovery across WAL crash windows.
+
+Existing callers may omit the new scheduling fields. Packed-record and WAL
+framing are unchanged; the new APIs and scheduling behavior require 0.1.1.
+Pending-task snapshots alone do not provide a joint application checkpoint or
+arbitrary checkpoint rollback.
+
+## 0.1.0
+
+Initial release of straw, a filesystem-based durable queue and shared tensor
+store for AI applications, distributed as `straw-queue` under the MIT license.
+
+- Rust storage, WAL, queue and ownership protocols with Python APIs. Multiple
+  machines can read and write through a shared filesystem; the initial shared
+  storage target is JuiceFS.
+- Packed immutable records and embedded manifests, batched publication and
+  size-based file rotation, avoiding a file per sample or tensor.
+- Bytes and custom codecs, NumPy/PyTorch tensors, checked lazy row reads and
+  native read sessions that reuse authenticated indices.
+- Shared tensor references across queues, tensor-level copy-on-write, explicit
+  reader/checkpoint ownership and optional online GC of unreferenced sealed packs.
+- Durable task leases, continuation progress, completion receipts and consumer
+  state, with WAL recovery and retained-checkpoint restore.
+- Local and multi-host I/O benchmarks, workload profiling and replay, an
+  independent protocol model, and process-crash/recovery checks.
+- Linux x86_64 wheels for CPython 3.10–3.13, automated artifact tests and PyPI
+  Trusted Publishing, standalone examples and paired English/Chinese guides.
+
+Each queue requires one externally fenced coordinator. Automatic failover,
+live-pack compaction, journal compaction and a general distributed application
+checkpoint transaction are not provided. Additional network filesystems require
+deployment validation. See the [filesystem requirements](docs/FILESYSTEM.md) and
+[protocol guarantees](docs/PROTOCOL_AND_VERIFICATION.md).

straw_queue-0.1.2/CHANGELOG_zh.md

@@ -0,0 +1,34 @@
+# 更新日志
+
+[English](CHANGELOG.md)
+
+## 0.1.2
+
+- 大记录的 payload 通过有界 native 写入缓冲区分块写入。记录和批量发布不再需要适配 writer 的临时缓冲区；原有 packed record 格式和张量读取方式不变。
+- 移除记录数、依赖数、结果大小、队列任务、控制消息及元数据的固定逻辑配额。旧的限制参数继续保留，以兼容 API 和队列身份；`max_buffer_bytes` 仍约束临时 payload 复制。
+- 共享文件系统上，已发布 extent 的文件长度可能先于文件头内容可见。reader 遇到这种短暂情况会延迟并重新打开文件；持续失败、非零坏文件头和 checksum 错误仍会报错。
+
+已有存储数据和调用方保持兼容。上述写入和读取行为需要 0.1.2；packed record 和 WAL 格式不变。
+
+## 0.1.1
+
+- 持久化任务调度：按 `TaskSpec.priority` 从高到低、`scheduling_key` 从低到高、再按 FIFO 顺序领取任务。两个字段默认均为 0，接受有符号 64 位整数。Rust pending 索引无需读取载荷即可选择任务，WAL 恢复时重建调度顺序。
+- 新增 `Coordinator.yield_tasks()`：原子保存一批 continuation 输入及可选调度字段，结束这些任务的 lease 并归还到 pending 队列，不消耗失败重试次数。归还的任务进入相同调度键的 FIFO 队尾；相同请求可幂等重试，任一 lease 失效则拒绝整批操作。
+- 新增 `Coordinator.pending_tasks()`：按领取顺序返回 pending 任务的 spec，支持按任务 ID 前缀及最低优先级过滤。应用负责协调暂停后的快照，并持有所引用的输入。
+- 补充优先级/FIFO 恢复、失效 lease、非法调度键，以及 WAL 崩溃窗口下批量操作全有或全无恢复的回归测试。
+
+已有调用方可以不指定新增调度字段。Packed record 与 WAL 的 framing 不变；新增 API 和调度行为需要 0.1.1。仅有 pending 任务快照尚不构成应用联合 checkpoint，也不提供任意 checkpoint 回退。
+
+## 0.1.0
+
+straw 首次发布：面向 AI 应用、基于文件系统的持久化队列与共享张量存储，以 `straw-queue` 包分发，采用 MIT 许可证。
+
+- Rust 实现存储、WAL、队列及所有权协议，提供 Python API。多台机器通过共享文件系统并行读写，首个共享存储目标为 JuiceFS。
+- 不可变记录与 manifest 打包存储，支持批量发布和按大小轮换文件，避免每个 sample 或张量单独创建文件。
+- 支持 bytes、自定义 codec、NumPy/PyTorch 张量、带校验的惰性按行读取，以及复用已认证索引的 native 读取会话。
+- 跨队列共享张量引用、张量级写时复制、显式 reader/checkpoint 所有权，以及可选在线 GC，回收无引用的封存 pack。
+- 持久化任务 lease、continuation 进度、完成 receipt 和消费者状态，支持 WAL 恢复与已保留 checkpoint 的恢复。
+- 本地与多机 I/O benchmark、应用负载采集与重放、独立协议模型，以及进程崩溃与恢复检查。
+- CPython 3.10–3.13 的 Linux x86_64 wheel、自动化产物测试与 PyPI Trusted Publishing、独立使用示例和配套中英文指南。
+
+每个队列需要一个由外部机制隔离保护的协调器。尚不提供自动故障切换、存活 pack 整理、日志压缩或通用分布式应用 checkpoint 事务。其他网络文件系统需要单独验证部署语义。详见[文件系统要求](docs/FILESYSTEM_zh.md)与[协议保证](docs/PROTOCOL_AND_VERIFICATION_zh.md)。

{straw_queue-0.1.0 → straw_queue-0.1.2}/Cargo.lock

@@ -370,7 +370,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.0"
+version = "0.1.2"
 dependencies = [
  "pyo3",
  "serde",

{straw_queue-0.1.0 → straw_queue-0.1.2}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.0"
+version = "0.1.2"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"

{straw_queue-0.1.0 → straw_queue-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.0
+Version: 0.1.2
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.0 → straw_queue-0.1.2}/docs/BUILDING.md

@@ -137,17 +137,20 @@ job uses GitHub OIDC with `id-token: write`; no PyPI API token or GitHub secret
 is needed. See [PyPI's setup guide](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)
 and [publishing guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
 
-After the release commit is ready and the package versions are `0.1.0`, publish
-the first release by pushing its tag:
+After committing the release changes, ensure that `pyproject.toml`, `Cargo.toml`
+and the `straw` entry in `Cargo.lock` share the release version. For example, for
+version `0.1.2`, create and push a new tag on that commit:
 
 ```sh
-git tag v0.1.0
-git push origin v0.1.0
+git tag v0.1.2
+git push origin v0.1.2
 ```
 
 The first successful upload creates the PyPI project; registering a pending
 publisher alone does not reserve the name. Later releases use a new matching
-version and tag. If a build or test fails, fix it before publishing; failed jobs
+version and tag. An existing tag does not include later working-tree changes;
+check its target before publishing and do not overwrite a published release.
+If a build or test fails, fix it before publishing; failed jobs
 prevent the publishing job from running.
 
 Hosted CI uses local Linux filesystems and CPU PyTorch. Multi-client JuiceFS
@@ -155,8 +158,9 @@ qualification, storage-service faults and GPU training remain separate
 [deployment/application checks](VERIFICATION.md). Inspect hosted results
 separately; a local build is not proof that the matrix ran.
 
-Before distributing a release, align the versions in `Cargo.toml` and
-`pyproject.toml`, run the [verification checks](VERIFICATION.md), inspect wheel
+Before distributing a release, align the versions in `Cargo.toml`,
+`pyproject.toml` and the `straw` entry in `Cargo.lock`, run the
+[verification checks](VERIFICATION.md), inspect wheel
 and sdist contents, run `check_wheel.py`, and preserve the wheel SHA-256 plus
 the exact source revision. The changelog should state protocol/API compatibility
 and unresolved limits. Upload the already tested artifact; rebuilding afterward

{straw_queue-0.1.0 → straw_queue-0.1.2}/docs/BUILDING_zh.md

@@ -90,15 +90,15 @@ python -m pip wheel --no-deps dist/straw_queue-*.tar.gz --wheel-dir rebuilt
 
 Workflow filename 只填 `wheels.yml`，不带 `.github/workflows/`。发布前，将该文件提交并推送到配置的仓库。发布 job 通过 GitHub OIDC 和 `id-token: write` 认证，无需配置 PyPI API token 或 GitHub secret。参见 [PyPI 配置指南](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)和[发布指南](https://docs.pypi.org/trusted-publishers/using-a-publisher/)。
 
-发布 commit 准备好、包版本均为 `0.1.0` 后，推送版本 tag 触发首次发布：
+提交发布改动后，确认 `pyproject.toml`、`Cargo.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致。例如发布 `0.1.2` 时，在该 commit 上创建并推送新的 tag：
 
 ```sh
-git tag v0.1.0
-git push origin v0.1.0
+git tag v0.1.2
+git push origin v0.1.2
 ```
 
-首次上传成功时才创建 PyPI 项目；仅登记 pending publisher 不会保留包名。后续版本使用新的匹配版本号和 tag。构建或测试失败时，应先修复；失败的 job 会阻止发布 job 运行。
+首次上传成功时才创建 PyPI 项目；仅登记 pending publisher 不会保留包名。后续版本使用新的匹配版本号和 tag。已有 tag 不会包含之后的工作区改动；发布前应检查其指向，不要覆盖已发布版本。构建或测试失败时，应先修复；失败的 job 会阻止发布 job 运行。
 
 托管 CI 使用本地 Linux 文件系统和 CPU PyTorch。多客户端 JuiceFS 验证、存储服务故障及 GPU 训练属于独立的[部署与应用检查](VERIFICATION_zh.md)。托管结果需要单独检查；本地构建通过不代表矩阵已经执行。
 
-分发版本前，确认 `Cargo.toml` 与 `pyproject.toml` 版本一致，运行[验证检查](VERIFICATION_zh.md)，检查 wheel/sdist 内容，运行 `check_wheel.py`，保存 wheel SHA-256 与精确源码版本。更新日志应说明协议/API 兼容性及已知限制。上传已经测试过的那个产物；之后重新构建会产生新产物，需要重新检查。
+分发版本前，确认 `Cargo.toml`、`pyproject.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致，运行[验证检查](VERIFICATION_zh.md)，检查 wheel/sdist 内容，运行 `check_wheel.py`，保存 wheel SHA-256 与精确源码版本。更新日志应说明协议/API 兼容性及已知限制。上传已经测试过的那个产物；之后重新构建会产生新产物，需要重新检查。

{straw_queue-0.1.0 → straw_queue-0.1.2}/docs/USAGE.md

@@ -15,8 +15,8 @@ store = SharedFilesystemStore(
     "/shared/new-job", "new-job",
     online_gc=True,
     segment_target_bytes=1024**3,      # Rotation target, not a hard file-size limit.
-    max_record_bytes=256 * 1024**2,
-    max_buffer_bytes=512 * 1024**2,
+    max_record_bytes=16 * 1024**3,
+    max_buffer_bytes=16 * 1024**2,
 )
 ```
 
@@ -31,8 +31,37 @@ into a GC-enabled pool and assume its old references have owners. GC is opt-in;
 when a catalog exists, all participants must honor it. None of these calls
 create a scheduler or start an automatic GC thread.
 
+## Large records and bounded write buffers
+
+`max_record_bytes` is a compatibility argument and no longer caps records.
+`max_buffer_bytes` bounds temporary payload copies inside the native writer,
+independently of record/publication size. The writer pins Python input buffers
+and copies at most 4 MiB per chunk; large records pipeline copying and I/O using
+at most two chunks. Applications do not need to split payloads to fit scratch
+memory. Metadata/index allocations, caller-owned inputs, Python serialization
+and full-record reads are outside this payload-copy budget. There are no fixed
+record-count, dependency-count, metadata, manifest or index size quotas.
+
+Do not mutate input buffers until publication returns. The writer rechecks each
+payload while writing and rejects detected changes before committing. Chunked
+I/O retains the existing record framing and tensor slice API: it does not expose
+partial records or alter durable publication, retry, ownership or GC ordering.
+A pack rotation target is soft; a large record/publication may exceed it. Reads
+of old files retain their stored checksum chunk sizes. A reader configured with
+an explicit smaller record limit must raise it to read larger records.
+
 ## Concurrency: processes and threads
 
+Published extents are immutable, but a shared-filesystem reader can briefly see
+the new file length before the appended header bytes. The native reader closes
+and reopens the file on a short extent or an all-zero 12-byte header, with delays
+of 100, 250, 500, 1000 and 2000 ms. It still validates the header, index and record
+data normally. Persistent failures remain errors; nonzero bad headers and
+checksum failures are not retried by this visibility handling. This applies to
+store and read-session entry points, including dependency validation. Successful
+reads incur no extra I/O, and applications should not stack another retry loop
+around the resulting corruption error.
+
 The API is synchronous; straw does not launch an I/O process pool. Applications
 can read/write from multiple processes and machines using the same pool. Each
 writer needs its own store instance and pack stream. Keep these instances alive
@@ -125,7 +154,7 @@ requires a leading dimension. `.load()` returns a CPU PyTorch tensor; call
 `ref[start:stop]` reads contiguous rows and verifies intersecting 4 MiB chunks.
 Only slices with step 1 are supported. These are file reads, not mmap views.
 `load(pin_memory=True)` requests pinned CPU memory where PyTorch supports it.
-Tune store record/publication limits for large tensors; defaults still apply.
+Large tensors stream through bounded write scratch without record/publication quotas.
 
 To restore several tensor descriptors from one publication efficiently:
 
@@ -193,11 +222,22 @@ for durable intermediate inputs while retaining the lease; `yield_task` or
 `fail_task` and timeouts use the task's attempt budget. Only publish a
 continuation when all its fields describe one complete, consistent prefix.
 
-`Limits` bounds queue pending/in-flight work, accepted data, ready batches and
-control metadata. These are conservative logical budgets, not physical disk
-quotas. Pass the same limits, codecs, lease duration, run ID and queue ID when
-recovering a queue. In protocol v1 the default queue ID is `rollout`;
-applications can choose an explicit name such as `work`.
+`Limits` retains its fields solely for API and stored queue-identity compatibility.
+They no longer reject pending/in-flight work, result bytes/tokens/records, ready
+batches or control messages. Applications control concurrency through requested
+acquisition sizes and their own schedulers. Accounting metrics remain available;
+large immutable results are accepted by reference without rewriting payloads.
+Pass the same legacy limits, codecs, lease duration, run ID and queue ID when
+recovering an existing queue. In protocol v1 the default queue ID is `rollout`.
+
+The removed fixed quotas include 4 GiB per result, 64 GiB of outstanding results,
+100 million tokens, 10,000 records/dependency nodes, 64 KiB record envelopes,
+256 KiB control metadata, 8 MiB indexes/manifests/journal transactions and 64 MiB
+catalog transactions. The existing frame uses a u32 envelope length; that format
+boundary, u64 counters, authenticated file extents, checksums, task/attempt
+ownership, retry counts, leases and writer exclusion remain enforced. Removing
+quotas does not provide unlimited RAM/disk; metadata is still materialized in
+memory and payload writes retain bounded scratch buffers.
 
 ## Lifetime and online GC
 
@@ -276,6 +316,26 @@ Use one `with store.read_session() as reader:` block for adjacent immutable publ
 
 The Rust reader retains only the last authenticated index, keyed by the complete extent descriptor. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again after an extent switch or in a new session. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.
 
-Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, capacity limits and owner-release rules are unchanged.
+Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, owner-release rules remain unchanged; logical capacity quotas are no longer enforced.
 
 `store.read(publication)` also owns a scoped session for the lifetime of its iterator. Exhaust or close the iterator before releasing its data owner. Separate iterators reauthenticate the index.
+
+
+### Persistent task scheduling
+
+`TaskSpec.priority` (default 0) is ordered descending, then `scheduling_key`
+(default 0) ascending, then FIFO. Both keys are signed 64-bit integers. Keys and
+submission/return order survive WAL recovery; the Rust coordinator maintains an
+ordered pending index, without loading payloads to select the next task.
+
+Publish continuation inputs before calling `yield_tasks(updates, request_id=...)`.
+Each update contains `lease` and `input_ref`, and may replace `priority`,
+`scheduling_key` and `metadata`. The whole batch validates before one WAL commit;
+all leases end together and the tasks become pending at the tail of their key's
+FIFO order. A stale lease rejects the whole batch. Retrying the identical request
+is idempotent. Publish again only if you intend to create a different input.
+This is distinct from `save_task_progress_many`, which keeps leases active.
+`pending_tasks()` returns ordered task specifications for a paused snapshot;
+applications must coordinate the pause and retain the referenced inputs.
+
+These APIs require `straw-queue>=0.1.1`.

{straw_queue-0.1.0 → straw_queue-0.1.2}/docs/USAGE_zh.md

@@ -13,8 +13,8 @@ store = SharedFilesystemStore(
     "/shared/new-job", "new-job",
     online_gc=True,
     segment_target_bytes=1024**3,      # Rotation target, not a hard file-size limit.
-    max_record_bytes=256 * 1024**2,
-    max_buffer_bytes=512 * 1024**2,
+    max_record_bytes=16 * 1024**3,
+    max_buffer_bytes=16 * 1024**2,
 )
 ```
 
@@ -22,8 +22,16 @@ store = SharedFilesystemStore(
 
 应从创建池时就启用在线 GC。不能把已有但未跟踪所有权的队列直接切换为 GC 模式，并假定旧引用已经获得保护。GC 是可选能力；catalog 一旦存在，所有参与者都必须遵循它的规则。这些 API 不会创建调度器，也不会启动自动 GC 线程。
 
+## 大记录与有界写入缓冲区
+
+`max_record_bytes` 只保留为兼容参数，不再限制单条记录。`max_buffer_bytes` 仍限制 native writer 临时复制的 payload 内存，不限制整个 record 或 publication。每块最多复制 4 MiB，大记录采用双缓冲复制/I/O 流水线。调用方不需要自行拆分 sample。元数据/索引、调用方已有输入、Python 序列化和整条读回不包含在 scratch 预算内；记录数、依赖数、元数据和索引不再有固定容量配额。
+
+发布返回前不要修改输入 buffer。写入时会再次校验 payload，检测到变化会在提交前拒绝。分块 I/O 保持原有 record 格式及张量切片 API，不会暴露部分记录，也不改变持久化、重试、所有权和 GC 顺序。pack 大小是软轮转目标，大 record/publication 可以超过目标。旧文件仍使用其中保存的 checksum 分块大小；reader 同样不再受旧 record 大小参数限制。
+
 ## 进程与线程并发
 
+已发布的 extent 不可变，但共享文件系统的 reader 可能短暂先看到增长后的文件大小、后看到追加的文件头。对于文件长度尚不足或 12 字节文件头全零的情况，native reader 会关闭文件，依次等待 100、250、500、1000、2000 ms 后重新打开，再进行原有校验。持续失败仍报错；非零的坏文件头和 checksum 错误不走此可见性重试。该处理覆盖 store、read session 和依赖校验等入口，正常读取不增加 I/O，应用无需再叠加另一层 CorruptData 重试。
+
 API 是同步的，straw 不会启动 I/O 进程池。应用可从多个进程、多台机器访问同一个池。每个 writer 需要独立的 store 实例和 pack 写入流，并在多次发布之间复用实例。不要将可变 store/coordinator 对象跨 fork 继承后继续使用。
 
 Rust 在 native I/O、checksum 和日志操作期间释放 GIL，Python 线程可并行执行这些操作。Python 序列化、张量转换和输入缓冲区复制仍有开销。每个 store 同时只允许一个写操作；同实例的重叠写入会报 `ResourceLimitExceeded`，因此需由调用方串行化，或为每个 writer 创建独立实例。引用受到保护期间，reader 可以并发读取。
@@ -122,7 +130,9 @@ b.submit_tasks("b-1", [TaskSpec("task-b", input_ref=dependency)])
 
 通过 `heartbeat` 续租。用 `save_task_progress`/`save_task_progress_many` 在保留 lease 的同时持久化中间输入；`yield_task` 或 `release_tasks` 把未完成任务退回队列，不消耗失败重试预算。`fail_task` 与超时会消耗尝试预算。只有全部字段都对应同一个完整、一致的前缀，才能发布 continuation。
 
-`Limits` 限制 pending/in-flight 任务、已接收数据、ready batch 和控制元数据。这些是保守的逻辑预算，不是物理磁盘配额。恢复队列时保持 limits、codec、lease 时长、run ID 和 queue ID 一致。协议 v1 的默认 queue ID 是 `rollout`，应用可显式使用 `work` 等名称。
+`Limits` 的字段仅用于兼容旧调用方及队列恢复身份，不再限制 pending/in-flight 任务、结果字节/token/记录数、ready batch 或控制消息。应用通过 acquire 请求数量和自己的调度器控制并发，原有使用量统计保留。移除了单结果 4 GiB、累计 64 GiB、1 亿 token、1 万记录/依赖节点、64 KiB envelope、256 KiB 控制元数据、8 MiB 索引/manifest/事务日志以及 64 MiB catalog 事务等配额。恢复旧队列仍应传入与原先相同的 Limits 字段、codec、lease 时长、run ID 和 queue ID。
+
+保留真实格式和正确性检查：u32 envelope 长度、u64 计数、文件边界与校验和、task/attempt 归属、lease、失败重试数及同一 writer 不可重入。移除逻辑配额不代表无限内存/磁盘；元数据仍需占用内存，payload 写入继续使用有界 scratch。
 
 ## 生命周期与在线 GC
 
@@ -165,3 +175,12 @@ Rust reader 只保留最后一个已认证索引，以完整 extent 描述符区
 发布依赖校验、批量续跑进度、消费者快照及 GC 遍历在 Rust 内部也复用这种有界 reader。文件格式、WAL 提交顺序、容量限制和所有权释放规则均未改变。
 
 `store.read(publication)` 同样在迭代器生命周期内持有读取会话。释放数据 owner 前应读完或关闭迭代器。不同迭代器会重新认证索引。
+
+
+### 持久化任务排序
+
+`TaskSpec.priority`（默认 0）降序，其次 `scheduling_key`（默认 0）升序，再按 FIFO；两个字段均为有符号 64 位整数。排序字段及提交/归还顺序会从 WAL 恢复。Rust coordinator 维护有序 pending 索引，选取任务无需读取 payload。
+
+先发布续跑输入，再调用 `yield_tasks(updates, request_id=...)`。每项包含 `lease`、`input_ref`，可同时替换 `priority`、`scheduling_key` 和 `metadata`。整批校验后只提交一次 WAL；所有 lease 同时结束，任务进入其排序键的 FIFO 队尾。任何 stale lease 都会拒绝整批。相同请求重试幂等；只有确实需要另一个输入版本时才重新发布。与此不同，`save_task_progress_many` 保留 lease。
+
+`pending_tasks()` 返回按领取顺序排列的任务 spec，可用于暂停后的快照；暂停边界和输入引用持有由应用协调。这些 API 需要 `straw-queue>=0.1.1`。

{straw_queue-0.1.0 → straw_queue-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.0"
+version = "0.1.2"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"