straw-queue 0.1.0__tar.gz → 0.1.1__tar.gz

{straw_queue-0.1.0 → straw_queue-0.1.1}/CHANGELOG.md

@@ -2,6 +2,28 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.1
+
+- Persistent task scheduling with `TaskSpec.priority` (higher first),
+  `scheduling_key` (lower first), then FIFO. Both keys default to zero and accept
+  signed 64-bit integers. The Rust pending index selects tasks without reading
+  their payloads and reconstructs scheduling order during WAL recovery.
+- `Coordinator.yield_tasks()` atomically saves a batch of continuation inputs
+  and optional scheduling fields, ends their leases, and returns the tasks to
+  the pending queue without spending failure retries. Returned tasks join the
+  tail of their scheduling key's FIFO order. Identical requests are idempotent;
+  a stale lease rejects the entire batch.
+- `Coordinator.pending_tasks()` exposes pending task specifications in
+  acquisition order, with optional task-prefix and minimum-priority filters.
+  Applications coordinate paused snapshots and retain the referenced inputs.
+- Regression coverage for priority/FIFO recovery, stale leases, invalid
+  scheduling keys, and all-or-nothing batch recovery across WAL crash windows.
+
+Existing callers may omit the new scheduling fields. Packed-record and WAL
+framing are unchanged; the new APIs and scheduling behavior require 0.1.1.
+Pending-task snapshots alone do not provide a joint application checkpoint or
+arbitrary checkpoint rollback.
+
 ## 0.1.0
 
 Initial release of straw, a filesystem-based durable queue and shared tensor

{straw_queue-0.1.0 → straw_queue-0.1.1}/CHANGELOG_zh.md

@@ -2,6 +2,15 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.1
+
+- 持久化任务调度：按 `TaskSpec.priority` 从高到低、`scheduling_key` 从低到高、再按 FIFO 顺序领取任务。两个字段默认均为 0，接受有符号 64 位整数。Rust pending 索引无需读取载荷即可选择任务，WAL 恢复时重建调度顺序。
+- 新增 `Coordinator.yield_tasks()`：原子保存一批 continuation 输入及可选调度字段，结束这些任务的 lease 并归还到 pending 队列，不消耗失败重试次数。归还的任务进入相同调度键的 FIFO 队尾；相同请求可幂等重试，任一 lease 失效则拒绝整批操作。
+- 新增 `Coordinator.pending_tasks()`：按领取顺序返回 pending 任务的 spec，支持按任务 ID 前缀及最低优先级过滤。应用负责协调暂停后的快照，并持有所引用的输入。
+- 补充优先级/FIFO 恢复、失效 lease、非法调度键，以及 WAL 崩溃窗口下批量操作全有或全无恢复的回归测试。
+
+已有调用方可以不指定新增调度字段。Packed record 与 WAL 的 framing 不变；新增 API 和调度行为需要 0.1.1。仅有 pending 任务快照尚不构成应用联合 checkpoint，也不提供任意 checkpoint 回退。
+
 ## 0.1.0
 
 straw 首次发布：面向 AI 应用、基于文件系统的持久化队列与共享张量存储，以 `straw-queue` 包分发，采用 MIT 许可证。

{straw_queue-0.1.0 → straw_queue-0.1.1}/Cargo.lock

@@ -370,7 +370,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.0"
+version = "0.1.1"
 dependencies = [
  "pyo3",
  "serde",

{straw_queue-0.1.0 → straw_queue-0.1.1}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.0"
+version = "0.1.1"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"

{straw_queue-0.1.0 → straw_queue-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.0
+Version: 0.1.1
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.0 → straw_queue-0.1.1}/docs/BUILDING.md

@@ -137,17 +137,20 @@ job uses GitHub OIDC with `id-token: write`; no PyPI API token or GitHub secret
 is needed. See [PyPI's setup guide](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)
 and [publishing guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
 
-After the release commit is ready and the package versions are `0.1.0`, publish
-the first release by pushing its tag:
+After committing the release changes, ensure that `pyproject.toml`, `Cargo.toml`
+and the `straw` entry in `Cargo.lock` share the release version. For example, for
+version `0.1.1`, create and push a new tag on that commit:
 
 ```sh
-git tag v0.1.0
-git push origin v0.1.0
+git tag v0.1.1
+git push origin v0.1.1
 ```
 
 The first successful upload creates the PyPI project; registering a pending
 publisher alone does not reserve the name. Later releases use a new matching
-version and tag. If a build or test fails, fix it before publishing; failed jobs
+version and tag. An existing tag does not include later working-tree changes;
+check its target before publishing and do not overwrite a published release.
+If a build or test fails, fix it before publishing; failed jobs
 prevent the publishing job from running.
 
 Hosted CI uses local Linux filesystems and CPU PyTorch. Multi-client JuiceFS
@@ -155,8 +158,9 @@ qualification, storage-service faults and GPU training remain separate
 [deployment/application checks](VERIFICATION.md). Inspect hosted results
 separately; a local build is not proof that the matrix ran.
 
-Before distributing a release, align the versions in `Cargo.toml` and
-`pyproject.toml`, run the [verification checks](VERIFICATION.md), inspect wheel
+Before distributing a release, align the versions in `Cargo.toml`,
+`pyproject.toml` and the `straw` entry in `Cargo.lock`, run the
+[verification checks](VERIFICATION.md), inspect wheel
 and sdist contents, run `check_wheel.py`, and preserve the wheel SHA-256 plus
 the exact source revision. The changelog should state protocol/API compatibility
 and unresolved limits. Upload the already tested artifact; rebuilding afterward

{straw_queue-0.1.0 → straw_queue-0.1.1}/docs/BUILDING_zh.md

@@ -90,15 +90,15 @@ python -m pip wheel --no-deps dist/straw_queue-*.tar.gz --wheel-dir rebuilt
 
 Workflow filename 只填 `wheels.yml`，不带 `.github/workflows/`。发布前，将该文件提交并推送到配置的仓库。发布 job 通过 GitHub OIDC 和 `id-token: write` 认证，无需配置 PyPI API token 或 GitHub secret。参见 [PyPI 配置指南](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)和[发布指南](https://docs.pypi.org/trusted-publishers/using-a-publisher/)。
 
-发布 commit 准备好、包版本均为 `0.1.0` 后，推送版本 tag 触发首次发布：
+提交发布改动后，确认 `pyproject.toml`、`Cargo.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致。例如发布 `0.1.1` 时，在该 commit 上创建并推送新的 tag：
 
 ```sh
-git tag v0.1.0
-git push origin v0.1.0
+git tag v0.1.1
+git push origin v0.1.1
 ```
 
-首次上传成功时才创建 PyPI 项目；仅登记 pending publisher 不会保留包名。后续版本使用新的匹配版本号和 tag。构建或测试失败时，应先修复；失败的 job 会阻止发布 job 运行。
+首次上传成功时才创建 PyPI 项目；仅登记 pending publisher 不会保留包名。后续版本使用新的匹配版本号和 tag。已有 tag 不会包含之后的工作区改动；发布前应检查其指向，不要覆盖已发布版本。构建或测试失败时，应先修复；失败的 job 会阻止发布 job 运行。
 
 托管 CI 使用本地 Linux 文件系统和 CPU PyTorch。多客户端 JuiceFS 验证、存储服务故障及 GPU 训练属于独立的[部署与应用检查](VERIFICATION_zh.md)。托管结果需要单独检查；本地构建通过不代表矩阵已经执行。
 
-分发版本前，确认 `Cargo.toml` 与 `pyproject.toml` 版本一致，运行[验证检查](VERIFICATION_zh.md)，检查 wheel/sdist 内容，运行 `check_wheel.py`，保存 wheel SHA-256 与精确源码版本。更新日志应说明协议/API 兼容性及已知限制。上传已经测试过的那个产物；之后重新构建会产生新产物，需要重新检查。
+分发版本前，确认 `Cargo.toml`、`pyproject.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致，运行[验证检查](VERIFICATION_zh.md)，检查 wheel/sdist 内容，运行 `check_wheel.py`，保存 wheel SHA-256 与精确源码版本。更新日志应说明协议/API 兼容性及已知限制。上传已经测试过的那个产物；之后重新构建会产生新产物，需要重新检查。

{straw_queue-0.1.0 → straw_queue-0.1.1}/docs/USAGE.md

@@ -279,3 +279,23 @@ The Rust reader retains only the last authenticated index, keyed by the complete
 Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, capacity limits and owner-release rules are unchanged.
 
 `store.read(publication)` also owns a scoped session for the lifetime of its iterator. Exhaust or close the iterator before releasing its data owner. Separate iterators reauthenticate the index.
+
+
+### Persistent task scheduling
+
+`TaskSpec.priority` (default 0) is ordered descending, then `scheduling_key`
+(default 0) ascending, then FIFO. Both keys are signed 64-bit integers. Keys and
+submission/return order survive WAL recovery; the Rust coordinator maintains an
+ordered pending index, without loading payloads to select the next task.
+
+Publish continuation inputs before calling `yield_tasks(updates, request_id=...)`.
+Each update contains `lease` and `input_ref`, and may replace `priority`,
+`scheduling_key` and `metadata`. The whole batch validates before one WAL commit;
+all leases end together and the tasks become pending at the tail of their key's
+FIFO order. A stale lease rejects the whole batch. Retrying the identical request
+is idempotent. Publish again only if you intend to create a different input.
+This is distinct from `save_task_progress_many`, which keeps leases active.
+`pending_tasks()` returns ordered task specifications for a paused snapshot;
+applications must coordinate the pause and retain the referenced inputs.
+
+These APIs require `straw-queue>=0.1.1`.

{straw_queue-0.1.0 → straw_queue-0.1.1}/docs/USAGE_zh.md

@@ -165,3 +165,12 @@ Rust reader 只保留最后一个已认证索引，以完整 extent 描述符区
 发布依赖校验、批量续跑进度、消费者快照及 GC 遍历在 Rust 内部也复用这种有界 reader。文件格式、WAL 提交顺序、容量限制和所有权释放规则均未改变。
 
 `store.read(publication)` 同样在迭代器生命周期内持有读取会话。释放数据 owner 前应读完或关闭迭代器。不同迭代器会重新认证索引。
+
+
+### 持久化任务排序
+
+`TaskSpec.priority`（默认 0）降序，其次 `scheduling_key`（默认 0）升序，再按 FIFO；两个字段均为有符号 64 位整数。排序字段及提交/归还顺序会从 WAL 恢复。Rust coordinator 维护有序 pending 索引，选取任务无需读取 payload。
+
+先发布续跑输入，再调用 `yield_tasks(updates, request_id=...)`。每项包含 `lease`、`input_ref`，可同时替换 `priority`、`scheduling_key` 和 `metadata`。整批校验后只提交一次 WAL；所有 lease 同时结束，任务进入其排序键的 FIFO 队尾。任何 stale lease 都会拒绝整批。相同请求重试幂等；只有确实需要另一个输入版本时才重新发布。与此不同，`save_task_progress_many` 保留 lease。
+
+`pending_tasks()` 返回按领取顺序排列的任务 spec，可用于暂停后的快照；暂停边界和输入引用持有由应用协调。这些 API 需要 `straw-queue>=0.1.1`。

{straw_queue-0.1.0 → straw_queue-0.1.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.0"
+version = "0.1.1"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"

{straw_queue-0.1.0 → straw_queue-0.1.1}/rust/coordinator.rs

@@ -4,6 +4,7 @@ use crate::store::{Config, Reader, Record, Store, mkdir, sync, sync_dir};
 use crate::{Error, Fault, Result, array, bytes, digest, fail, field, number, uid};
 use serde_json::{Value, json};
 use std::{
+    cmp::Reverse,
     collections::{BTreeMap, HashMap, HashSet},
     fs::OpenOptions,
     io::Write,
@@ -17,9 +18,9 @@ pub struct Coordinator {
     pub limits: Value,
     pub epoch: String,
     pub recovery_seconds: f64,
-    order: Vec<String>,
-    positions: HashMap<String, usize>,
-    pending: BTreeMap<usize, String>,
+    pending: BTreeMap<(Reverse<i64>, i64, usize), String>,
+    pending_keys: HashMap<String, (Reverse<i64>, i64, usize)>,
+    next_pending_order: usize,
     leased: HashSet<String>,
     usage_cache: Option<Value>,
     deadlines: HashMap<String, f64>,
@@ -62,6 +63,17 @@ fn bounded(value: &Value, limit: u64) -> Result<()> {
     }
     Ok(())
 }
+fn validate_scheduling(value: &Value) -> Result<()> {
+    for key in ["priority", "scheduling_key"] {
+        if value.get(key).is_some_and(|v| v.as_i64().is_none()) {
+            return fail(
+                "ValueError",
+                format!("{key} must be a signed 64-bit integer"),
+            );
+        }
+    }
+    Ok(())
+}
 impl Coordinator {
     pub fn open(config: Config, options: &Value, fault: &mut Fault<'_>) -> Result<Self> {
         let started = Instant::now();
@@ -119,9 +131,9 @@ impl Coordinator {
             limits,
             epoch: String::new(),
             recovery_seconds: 0.,
-            order: vec![],
-            positions: HashMap::new(),
             pending: BTreeMap::new(),
+            pending_keys: HashMap::new(),
+            next_pending_order: 0,
             leased: HashSet::new(),
             usage_cache: None,
             deadlines: HashMap::new(),
@@ -243,8 +255,8 @@ impl Coordinator {
             _ => vec![],
         };
         for id in &affected {
-            if let Some(position) = self.positions.get(id) {
-                self.pending.remove(position);
+            if let Some(key) = self.pending_keys.remove(id) {
+                self.pending.remove(&key);
             }
             self.leased.remove(id);
             if let Some(usage) = self.usage_cache.as_mut() {
@@ -261,9 +273,9 @@ impl Coordinator {
             "Submitted" => {
                 for spec in array(e, "tasks")? {
                     let id = field(spec, "task_id")?;
-                    self.positions.insert(id.into(), self.order.len());
-                    self.order.push(id.into());
                     self.state["tasks"][id] = json!({"spec":spec,"state":"pending","generation":0,"failures":0,"lease":null});
+                    self.state["tasks"][id]["pending_order"] = json!(self.next_pending_order);
+                    self.next_pending_order += 1;
                 }
                 if let Some(id) = e["producer_id"].as_str() {
                     self.state["producers"][id] = e["producer_state"].clone();
@@ -336,9 +348,16 @@ impl Coordinator {
                 }
                 let t = &mut self.state["tasks"][id];
                 t["spec"]["input_ref"] = e["input_ref"].clone();
+                for key in ["priority", "scheduling_key", "metadata"] {
+                    if let Some(value) = e.get(key) {
+                        t["spec"][key] = value.clone();
+                    }
+                }
                 if e["type"] == "Yielded" {
                     t["state"] = json!("pending");
                     t["lease"] = Value::Null;
+                    t["pending_order"] = json!(self.next_pending_order);
+                    self.next_pending_order += 1;
                     self.deadlines.remove(id);
                 }
             }
@@ -411,7 +430,13 @@ impl Coordinator {
         for id in &affected {
             let task = &self.state["tasks"][id];
             if task["state"] == "pending" {
-                self.pending.insert(self.positions[id], id.clone());
+                let key = (
+                    Reverse(task["spec"]["priority"].as_i64().unwrap_or(0)),
+                    task["spec"]["scheduling_key"].as_i64().unwrap_or(0),
+                    number(task, "pending_order") as usize,
+                );
+                self.pending.insert(key, id.clone());
+                self.pending_keys.insert(id.clone(), key);
             }
             if task["state"] == "leased" {
                 self.leased.insert(id.clone());
@@ -745,6 +770,7 @@ impl Coordinator {
                 let mut ids = vec![];
                 let mut unique = HashSet::new();
                 for t in tasks {
+                    validate_scheduling(t)?;
                     let id = field(t, "task_id")?;
                     if !unique.insert(id) || !self.state["tasks"][id].is_null() {
                         return fail("IdempotencyConflict", "Task IDs must be new within queue");
@@ -821,7 +847,7 @@ impl Coordinator {
                     let mut selected = ids
                         .iter()
                         .filter_map(|v| v.as_str())
-                        .filter_map(|id| self.positions.get(id))
+                        .filter_map(|id| self.pending_keys.get(id))
                         .filter_map(|p| self.pending.get_key_value(p))
                         .collect::<Vec<_>>();
                     selected.sort_by_key(|(position, _)| *position);
@@ -903,11 +929,19 @@ impl Coordinator {
                 }
                 Ok(json!({"status":"acquired","assignments":assignments}))
             }
-            "save_task_progress_many" => {
+            "save_task_progress_many" | "yield_tasks" => {
+                let yielding = method == "yield_tasks";
                 let updates = array(args, "updates")?;
                 bounded(&json!(updates), metadata)?;
-                let request =
-                    self.request("progress_many", field(args, "request_id")?, &json!(updates))?;
+                let request = self.request(
+                    if yielding {
+                        "yield_many"
+                    } else {
+                        "progress_many"
+                    },
+                    field(args, "request_id")?,
+                    &json!(updates),
+                )?;
                 if !request.2.is_null() {
                     return Ok(request.2);
                 }
@@ -918,6 +952,7 @@ impl Coordinator {
                 let mut unique = HashSet::new();
                 let mut reader = Reader::new(self.store.config.clone());
                 for update in updates {
+                    validate_scheduling(update)?;
                     let lease = &update["lease"];
                     let id = field(lease, "task_id")?;
                     if !unique.insert(id) {
@@ -932,16 +967,23 @@ impl Coordinator {
                         self.storage_failed = true;
                     }
                     validated?;
-                    events.push(
-                        json!({"type":"TaskProgress","task_id":id,"input_ref":update["input_ref"]}),
-                    );
+                    let mut event = json!({"type":if yielding {"Yielded"} else {"TaskProgress"},"task_id":id,"input_ref":update["input_ref"]});
+                    if yielding {
+                        for key in ["priority", "scheduling_key", "metadata"] {
+                            if let Some(value) = update.get(key) {
+                                event[key] = value.clone();
+                            }
+                        }
+                    }
+                    events.push(event);
                 }
                 let last = events.last_mut().unwrap();
                 last["request_key"] = json!(request.0);
                 last["request_digest"] = json!(request.1);
-                last["response"] = json!("saved");
+                let response = json!(if yielding { "pending" } else { "saved" });
+                last["response"] = response.clone();
                 self.commit(json!(events), fault)?;
-                Ok(json!("saved"))
+                Ok(response)
             }
             "release_tasks" => {
                 let leases = array(args, "leases")?;
@@ -1139,6 +1181,26 @@ impl Coordinator {
             "lookup_submission" => {
                 Ok(self.state["submissions"][field(args, "submission_id")?]["receipt"].clone())
             }
+            "pending_tasks" => {
+                let prefix = args["task_prefix"].as_str();
+                let minimum = args["min_priority"].as_i64();
+                Ok(json!(
+                    self.pending
+                        .values()
+                        .filter_map(|id| {
+                            let spec = &self.state["tasks"][id]["spec"];
+                            if prefix.is_some_and(|p| !id.starts_with(p))
+                                || minimum
+                                    .is_some_and(|p| spec["priority"].as_i64().unwrap_or(0) < p)
+                            {
+                                None
+                            } else {
+                                Some(spec.clone())
+                            }
+                        })
+                        .collect::<Vec<_>>()
+                ))
+            }
             "task_status" => Ok(self.state["tasks"][field(args, "task_id")?].clone()),
             "producer_state" => Ok(self.state["producers"][field(args, "producer_id")?].clone()),
             "read_commits" => {

{straw_queue-0.1.0 → straw_queue-0.1.1}/src/straw/coordinator.py

@@ -112,6 +112,20 @@ class Coordinator:
         """Persist a bounded batch of leased continuation references atomically."""
         return self._call("save_task_progress_many", locals())
 
+    def yield_tasks(self, updates, *, request_id):
+        """Atomically persist inputs/scheduling fields and return leased tasks to pending.
+
+        Each update contains lease and input_ref, and may replace priority,
+        scheduling_key and metadata. Higher priority is acquired first; within
+        a priority, lower scheduling_key precedes FIFO return/submission order.
+        Callers must finish using the inputs before yielding their read lifetime.
+        """
+        return self._call("yield_tasks", locals())
+
+    def pending_tasks(self, *, task_prefix=None, min_priority=None):
+        """Snapshot pending task specifications in acquisition order."""
+        return [TaskSpec.from_dict(v) for v in self._call("pending_tasks", locals())]
+
     def release_tasks(self, leases, *, request_id):
         """Release existing inputs atomically without spending failure retries."""
         return self._call("release_tasks", locals())

{straw_queue-0.1.0 → straw_queue-0.1.1}/src/straw/protocol.py

@@ -98,6 +98,8 @@ class TaskSpec:
     estimated_bytes: int = 0
     estimated_tokens: int = 0
     control: bool = False
+    priority: int = 0
+    scheduling_key: int = 0
 
     @classmethod
     def from_dict(cls, value):