straw-queue 0.1.1__tar.gz → 0.1.3__tar.gz

{straw_queue-0.1.1 → straw_queue-0.1.3}/CHANGELOG.md

@@ -2,6 +2,33 @@
 
 [中文版](CHANGELOG_zh.md)
 
+## 0.1.3
+
+- Retry temporarily zero-filled bytes anywhere in a published extent's header,
+  including partial and noncontiguous visibility gaps. Previously, only an
+  entirely zero-filled header triggered this retry. The retry limit and checksum
+  validation remain unchanged; other incorrect header bytes fail immediately.
+
+Existing callers and stored data remain compatible. Packed-record and WAL
+formats are unchanged.
+
+## 0.1.2
+
+- Stream large record payloads through bounded native write buffers. Records and
+  publications no longer need to fit the writer's scratch buffer; existing
+  packed-record framing and tensor reads are unchanged.
+- Remove fixed logical quotas on record and dependency counts, result sizes,
+  queue work, control messages, and metadata. Legacy limit arguments remain
+  accepted for API and queue-identity compatibility; `max_buffer_bytes` still
+  bounds temporary payload copies.
+- Retry transient shared-filesystem visibility gaps when a published extent's
+  length is visible before its header bytes. The reader reopens the file after
+  short delays; persistent failures, nonzero bad headers, and checksum errors
+  remain errors.
+
+Existing stored data and callers remain compatible. These write and read
+behaviors require 0.1.2; packed-record and WAL formats are unchanged.
+
 ## 0.1.1
 
 - Persistent task scheduling with `TaskSpec.priority` (higher first),

{straw_queue-0.1.1 → straw_queue-0.1.3}/CHANGELOG_zh.md

@@ -2,6 +2,20 @@
 
 [English](CHANGELOG.md)
 
+## 0.1.3
+
+- 已发布 extent 的文件头在任意位置出现暂时零值时都会触发重试，支持部分字节以及不连续位置尚不可见的情况。此前只有整个文件头为零时才会触发这类重试。重试上限和 checksum 校验保持不变；其他错误的文件头字节仍会立即报错。
+
+已有调用方和存储数据保持兼容，packed record 和 WAL 格式不变。
+
+## 0.1.2
+
+- 大记录的 payload 通过有界 native 写入缓冲区分块写入。记录和批量发布不再需要适配 writer 的临时缓冲区；原有 packed record 格式和张量读取方式不变。
+- 移除记录数、依赖数、结果大小、队列任务、控制消息及元数据的固定逻辑配额。旧的限制参数继续保留，以兼容 API 和队列身份；`max_buffer_bytes` 仍约束临时 payload 复制。
+- 共享文件系统上，已发布 extent 的文件长度可能先于文件头内容可见。reader 遇到这种短暂情况会延迟并重新打开文件；持续失败、非零坏文件头和 checksum 错误仍会报错。
+
+已有存储数据和调用方保持兼容。上述写入和读取行为需要 0.1.2；packed record 和 WAL 格式不变。
+
 ## 0.1.1
 
 - 持久化任务调度：按 `TaskSpec.priority` 从高到低、`scheduling_key` 从低到高、再按 FIFO 顺序领取任务。两个字段默认均为 0，接受有符号 64 位整数。Rust pending 索引无需读取载荷即可选择任务，WAL 恢复时重建调度顺序。

{straw_queue-0.1.1 → straw_queue-0.1.3}/Cargo.lock

@@ -370,7 +370,7 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
 
 [[package]]
 name = "straw"
-version = "0.1.1"
+version = "0.1.3"
 dependencies = [
  "pyo3",
  "serde",

{straw_queue-0.1.1 → straw_queue-0.1.3}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "straw"
-version = "0.1.1"
+version = "0.1.3"
 edition = "2024"
 rust-version = "1.89"
 license = "MIT"

{straw_queue-0.1.1 → straw_queue-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: straw-queue
-Version: 0.1.1
+Version: 0.1.3
 Classifier: Development Status :: 3 - Alpha
 Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3

{straw_queue-0.1.1 → straw_queue-0.1.3}/docs/BUILDING.md

@@ -139,11 +139,11 @@ and [publishing guide](https://docs.pypi.org/trusted-publishers/using-a-publishe
 
 After committing the release changes, ensure that `pyproject.toml`, `Cargo.toml`
 and the `straw` entry in `Cargo.lock` share the release version. For example, for
-version `0.1.1`, create and push a new tag on that commit:
+version `0.1.2`, create and push a new tag on that commit:
 
 ```sh
-git tag v0.1.1
-git push origin v0.1.1
+git tag v0.1.2
+git push origin v0.1.2
 ```
 
 The first successful upload creates the PyPI project; registering a pending

{straw_queue-0.1.1 → straw_queue-0.1.3}/docs/BUILDING_zh.md

@@ -90,11 +90,11 @@ python -m pip wheel --no-deps dist/straw_queue-*.tar.gz --wheel-dir rebuilt
 
 Workflow filename 只填 `wheels.yml`，不带 `.github/workflows/`。发布前，将该文件提交并推送到配置的仓库。发布 job 通过 GitHub OIDC 和 `id-token: write` 认证，无需配置 PyPI API token 或 GitHub secret。参见 [PyPI 配置指南](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)和[发布指南](https://docs.pypi.org/trusted-publishers/using-a-publisher/)。
 
-提交发布改动后，确认 `pyproject.toml`、`Cargo.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致。例如发布 `0.1.1` 时，在该 commit 上创建并推送新的 tag：
+提交发布改动后，确认 `pyproject.toml`、`Cargo.toml` 和 `Cargo.lock` 中 `straw` 条目的版本一致。例如发布 `0.1.2` 时，在该 commit 上创建并推送新的 tag：
 
 ```sh
-git tag v0.1.1
-git push origin v0.1.1
+git tag v0.1.2
+git push origin v0.1.2
 ```
 
 首次上传成功时才创建 PyPI 项目；仅登记 pending publisher 不会保留包名。后续版本使用新的匹配版本号和 tag。已有 tag 不会包含之后的工作区改动；发布前应检查其指向，不要覆盖已发布版本。构建或测试失败时，应先修复；失败的 job 会阻止发布 job 运行。

{straw_queue-0.1.1 → straw_queue-0.1.3}/docs/USAGE.md

@@ -15,8 +15,8 @@ store = SharedFilesystemStore(
     "/shared/new-job", "new-job",
     online_gc=True,
     segment_target_bytes=1024**3,      # Rotation target, not a hard file-size limit.
-    max_record_bytes=256 * 1024**2,
-    max_buffer_bytes=512 * 1024**2,
+    max_record_bytes=16 * 1024**3,
+    max_buffer_bytes=16 * 1024**2,
 )
 ```
 
@@ -31,8 +31,37 @@ into a GC-enabled pool and assume its old references have owners. GC is opt-in;
 when a catalog exists, all participants must honor it. None of these calls
 create a scheduler or start an automatic GC thread.
 
+## Large records and bounded write buffers
+
+`max_record_bytes` is a compatibility argument and no longer caps records.
+`max_buffer_bytes` bounds temporary payload copies inside the native writer,
+independently of record/publication size. The writer pins Python input buffers
+and copies at most 4 MiB per chunk; large records pipeline copying and I/O using
+at most two chunks. Applications do not need to split payloads to fit scratch
+memory. Metadata/index allocations, caller-owned inputs, Python serialization
+and full-record reads are outside this payload-copy budget. There are no fixed
+record-count, dependency-count, metadata, manifest or index size quotas.
+
+Do not mutate input buffers until publication returns. The writer rechecks each
+payload while writing and rejects detected changes before committing. Chunked
+I/O retains the existing record framing and tensor slice API: it does not expose
+partial records or alter durable publication, retry, ownership or GC ordering.
+A pack rotation target is soft; a large record/publication may exceed it. Reads
+of old files retain their stored checksum chunk sizes. A reader configured with
+an explicit smaller record limit must raise it to read larger records.
+
 ## Concurrency: processes and threads
 
+Published extents are immutable, but a shared-filesystem reader can briefly see
+the new file length before the appended header bytes. The native reader closes
+and reopens the file on a short extent or an all-zero 12-byte header, with delays
+of 100, 250, 500, 1000 and 2000 ms. It still validates the header, index and record
+data normally. Persistent failures remain errors; nonzero bad headers and
+checksum failures are not retried by this visibility handling. This applies to
+store and read-session entry points, including dependency validation. Successful
+reads incur no extra I/O, and applications should not stack another retry loop
+around the resulting corruption error.
+
 The API is synchronous; straw does not launch an I/O process pool. Applications
 can read/write from multiple processes and machines using the same pool. Each
 writer needs its own store instance and pack stream. Keep these instances alive
@@ -125,7 +154,7 @@ requires a leading dimension. `.load()` returns a CPU PyTorch tensor; call
 `ref[start:stop]` reads contiguous rows and verifies intersecting 4 MiB chunks.
 Only slices with step 1 are supported. These are file reads, not mmap views.
 `load(pin_memory=True)` requests pinned CPU memory where PyTorch supports it.
-Tune store record/publication limits for large tensors; defaults still apply.
+Large tensors stream through bounded write scratch without record/publication quotas.
 
 To restore several tensor descriptors from one publication efficiently:
 
@@ -193,11 +222,22 @@ for durable intermediate inputs while retaining the lease; `yield_task` or
 `fail_task` and timeouts use the task's attempt budget. Only publish a
 continuation when all its fields describe one complete, consistent prefix.
 
-`Limits` bounds queue pending/in-flight work, accepted data, ready batches and
-control metadata. These are conservative logical budgets, not physical disk
-quotas. Pass the same limits, codecs, lease duration, run ID and queue ID when
-recovering a queue. In protocol v1 the default queue ID is `rollout`;
-applications can choose an explicit name such as `work`.
+`Limits` retains its fields solely for API and stored queue-identity compatibility.
+They no longer reject pending/in-flight work, result bytes/tokens/records, ready
+batches or control messages. Applications control concurrency through requested
+acquisition sizes and their own schedulers. Accounting metrics remain available;
+large immutable results are accepted by reference without rewriting payloads.
+Pass the same legacy limits, codecs, lease duration, run ID and queue ID when
+recovering an existing queue. In protocol v1 the default queue ID is `rollout`.
+
+The removed fixed quotas include 4 GiB per result, 64 GiB of outstanding results,
+100 million tokens, 10,000 records/dependency nodes, 64 KiB record envelopes,
+256 KiB control metadata, 8 MiB indexes/manifests/journal transactions and 64 MiB
+catalog transactions. The existing frame uses a u32 envelope length; that format
+boundary, u64 counters, authenticated file extents, checksums, task/attempt
+ownership, retry counts, leases and writer exclusion remain enforced. Removing
+quotas does not provide unlimited RAM/disk; metadata is still materialized in
+memory and payload writes retain bounded scratch buffers.
 
 ## Lifetime and online GC
 
@@ -276,7 +316,7 @@ Use one `with store.read_session() as reader:` block for adjacent immutable publ
 
 The Rust reader retains only the last authenticated index, keyed by the complete extent descriptor. Each manifest and record read still authenticates its frame and payload; tensor reads authenticate touched chunks. `validate` rechecks logical descriptors and task/attempt authorization on every call. Index authentication runs again after an extent switch or in a new session. Keep a durable owner or `store.pin` for the entire operation: a read session is not an ownership pin. Sessions close on context exit and reject all inherited operations after fork.
 
-Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, capacity limits and owner-release rules are unchanged.
+Publication dependency validation, continuation batches, consumer snapshots and GC traversal use the same bounded native reader internally. Formats, WAL commit ordering, owner-release rules remain unchanged; logical capacity quotas are no longer enforced.
 
 `store.read(publication)` also owns a scoped session for the lifetime of its iterator. Exhaust or close the iterator before releasing its data owner. Separate iterators reauthenticate the index.
 

{straw_queue-0.1.1 → straw_queue-0.1.3}/docs/USAGE_zh.md

@@ -13,8 +13,8 @@ store = SharedFilesystemStore(
     "/shared/new-job", "new-job",
     online_gc=True,
     segment_target_bytes=1024**3,      # Rotation target, not a hard file-size limit.
-    max_record_bytes=256 * 1024**2,
-    max_buffer_bytes=512 * 1024**2,
+    max_record_bytes=16 * 1024**3,
+    max_buffer_bytes=16 * 1024**2,
 )
 ```
 
@@ -22,8 +22,16 @@ store = SharedFilesystemStore(
 
 应从创建池时就启用在线 GC。不能把已有但未跟踪所有权的队列直接切换为 GC 模式，并假定旧引用已经获得保护。GC 是可选能力；catalog 一旦存在，所有参与者都必须遵循它的规则。这些 API 不会创建调度器，也不会启动自动 GC 线程。
 
+## 大记录与有界写入缓冲区
+
+`max_record_bytes` 只保留为兼容参数，不再限制单条记录。`max_buffer_bytes` 仍限制 native writer 临时复制的 payload 内存，不限制整个 record 或 publication。每块最多复制 4 MiB，大记录采用双缓冲复制/I/O 流水线。调用方不需要自行拆分 sample。元数据/索引、调用方已有输入、Python 序列化和整条读回不包含在 scratch 预算内；记录数、依赖数、元数据和索引不再有固定容量配额。
+
+发布返回前不要修改输入 buffer。写入时会再次校验 payload，检测到变化会在提交前拒绝。分块 I/O 保持原有 record 格式及张量切片 API，不会暴露部分记录，也不改变持久化、重试、所有权和 GC 顺序。pack 大小是软轮转目标，大 record/publication 可以超过目标。旧文件仍使用其中保存的 checksum 分块大小；reader 同样不再受旧 record 大小参数限制。
+
 ## 进程与线程并发
 
+已发布的 extent 不可变，但共享文件系统的 reader 可能短暂先看到增长后的文件大小、后看到追加的文件头。对于文件长度尚不足或 12 字节文件头全零的情况，native reader 会关闭文件，依次等待 100、250、500、1000、2000 ms 后重新打开，再进行原有校验。持续失败仍报错；非零的坏文件头和 checksum 错误不走此可见性重试。该处理覆盖 store、read session 和依赖校验等入口，正常读取不增加 I/O，应用无需再叠加另一层 CorruptData 重试。
+
 API 是同步的，straw 不会启动 I/O 进程池。应用可从多个进程、多台机器访问同一个池。每个 writer 需要独立的 store 实例和 pack 写入流，并在多次发布之间复用实例。不要将可变 store/coordinator 对象跨 fork 继承后继续使用。
 
 Rust 在 native I/O、checksum 和日志操作期间释放 GIL，Python 线程可并行执行这些操作。Python 序列化、张量转换和输入缓冲区复制仍有开销。每个 store 同时只允许一个写操作；同实例的重叠写入会报 `ResourceLimitExceeded`，因此需由调用方串行化，或为每个 writer 创建独立实例。引用受到保护期间，reader 可以并发读取。
@@ -122,7 +130,9 @@ b.submit_tasks("b-1", [TaskSpec("task-b", input_ref=dependency)])
 
 通过 `heartbeat` 续租。用 `save_task_progress`/`save_task_progress_many` 在保留 lease 的同时持久化中间输入；`yield_task` 或 `release_tasks` 把未完成任务退回队列，不消耗失败重试预算。`fail_task` 与超时会消耗尝试预算。只有全部字段都对应同一个完整、一致的前缀，才能发布 continuation。
 
-`Limits` 限制 pending/in-flight 任务、已接收数据、ready batch 和控制元数据。这些是保守的逻辑预算，不是物理磁盘配额。恢复队列时保持 limits、codec、lease 时长、run ID 和 queue ID 一致。协议 v1 的默认 queue ID 是 `rollout`，应用可显式使用 `work` 等名称。
+`Limits` 的字段仅用于兼容旧调用方及队列恢复身份，不再限制 pending/in-flight 任务、结果字节/token/记录数、ready batch 或控制消息。应用通过 acquire 请求数量和自己的调度器控制并发，原有使用量统计保留。移除了单结果 4 GiB、累计 64 GiB、1 亿 token、1 万记录/依赖节点、64 KiB envelope、256 KiB 控制元数据、8 MiB 索引/manifest/事务日志以及 64 MiB catalog 事务等配额。恢复旧队列仍应传入与原先相同的 Limits 字段、codec、lease 时长、run ID 和 queue ID。
+
+保留真实格式和正确性检查：u32 envelope 长度、u64 计数、文件边界与校验和、task/attempt 归属、lease、失败重试数及同一 writer 不可重入。移除逻辑配额不代表无限内存/磁盘；元数据仍需占用内存，payload 写入继续使用有界 scratch。
 
 ## 生命周期与在线 GC
 

{straw_queue-0.1.1 → straw_queue-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "straw-queue"
-version = "0.1.1"
+version = "0.1.3"
 description = "Rust durable queues for AI tensors and large immutable records"
 readme = "README.md"
 requires-python = ">=3.10"

{straw_queue-0.1.1 → straw_queue-0.1.3}/rust/coordinator.rs

@@ -54,15 +54,6 @@ fn retry(task: &mut Value) {
     });
     task["lease"] = Value::Null;
 }
-fn bounded(value: &Value, limit: u64) -> Result<()> {
-    if bytes(value)?.len() as u64 > limit {
-        return fail(
-            "ResourceLimitExceeded",
-            "Control metadata limit exceeded; use a manifest",
-        );
-    }
-    Ok(())
-}
 fn validate_scheduling(value: &Value) -> Result<()> {
     for key in ["priority", "scheduling_key"] {
         if value.get(key).is_some_and(|v| v.as_i64().is_none()) {
@@ -697,10 +688,8 @@ impl Coordinator {
                 "Coordinator unavailable; stop owner and recover",
             );
         }
-        let metadata = number(&self.limits, "metadata_bytes");
         match method {
             "release_task_reads" => {
-                bounded(args, metadata)?;
                 for lease in array(args, "leases")? {
                     let prior = &self.state["readers"][field(lease, "attempt_id")?];
                     if !prior.is_null() && prior["lease"] != *lease {
@@ -721,7 +710,7 @@ impl Coordinator {
             "submit_tasks" => {
                 let tasks = array(args, "tasks")?;
                 let producer = args["producer_id"].as_str();
-                if producer.is_some() != !args["producer_state"].is_null() {
+                if producer.is_some() == args["producer_state"].is_null() {
                     return fail("ValueError", "Producer ID/state must be provided together");
                 }
                 let content = if producer.is_some() {
@@ -729,7 +718,6 @@ impl Coordinator {
                 } else {
                     json!(tasks)
                 };
-                bounded(&content, metadata)?;
                 let request = self.request("submit", field(args, "request_id")?, &content)?;
                 if !request.2.is_null() {
                     self.count("duplicate_submissions");
@@ -744,29 +732,6 @@ impl Coordinator {
                         "Scheduling paused after storage failure",
                     );
                 }
-                for (control, limit) in [
-                    (false, number(&self.limits, "pending_tasks")),
-                    (true, number(&self.limits, "control_tasks")),
-                ] {
-                    let existing = vals(&self.state["tasks"])
-                        .filter(|t| {
-                            (t["state"] == "pending" || (control && t["state"] == "leased"))
-                                && boolean(&t["spec"], "control") == control
-                        })
-                        .count();
-                    if existing as u64
-                        + tasks
-                            .iter()
-                            .filter(|t| boolean(t, "control") == control)
-                            .count() as u64
-                        > limit
-                    {
-                        return fail(
-                            "ResourceLimitExceeded",
-                            "Pending/control task budget exceeded",
-                        );
-                    }
-                }
                 let mut ids = vec![];
                 let mut unique = HashSet::new();
                 for t in tasks {
@@ -785,12 +750,6 @@ impl Coordinator {
                         if boolean(t, "control") && estimate != 0 {
                             return fail("ValueError", "Control tasks cannot reserve production");
                         };
-                        if estimate > number(&self.limits, &format!("accepted_{key}")) {
-                            return fail(
-                                "ResourceLimitExceeded",
-                                "Task estimate exceeds output budget",
-                            );
-                        }
                     }
                     if !t["input_ref"].is_null() {
                         self.validate(&t["input_ref"], None, None)?;
@@ -803,11 +762,7 @@ impl Coordinator {
             "acquire" => {
                 let worker = field(args, "worker_id")?;
                 let max = args["max_tasks"].as_u64().unwrap_or(1);
-                if worker.is_empty()
-                    || worker.len() > 1024
-                    || max == 0
-                    || max > number(&self.limits, "inflight_tasks")
-                {
+                if worker.is_empty() || worker.len() > 1024 || max == 0 {
                     return fail("ValueError", "Invalid worker identity/max tasks");
                 }
                 let expired = self
@@ -835,14 +790,8 @@ impl Coordinator {
                 }
                 let control = boolean(args, "control");
                 let ids = args["task_ids"].as_array();
-                if let Some(ids) = ids {
-                    bounded(&json!(ids), metadata)?;
-                }
                 let prefix = args["task_prefix"].as_str();
-                let mut usage = self.usage()?;
                 let mut assignments = vec![];
-                let mut assignment_bytes = 64u64;
-                let mut pending = false;
                 let candidates: Vec<&String> = if let Some(ids) = ids {
                     let mut selected = ids
                         .iter()
@@ -867,61 +816,20 @@ impl Coordinator {
                     {
                         continue;
                     }
-                    pending = true;
-                    if control {
-                        if number(&usage, "control_inflight")
-                            >= number(&self.limits, "control_tasks")
-                        {
-                            continue;
-                        }
-                    } else if number(&usage, "inflight") >= number(&self.limits, "inflight_tasks")
-                        || number(&usage, "ready_bytes") >= number(&self.limits, "ready_bytes")
-                        || ["records", "bytes", "tokens"].iter().any(|k| {
-                            number(&usage, k) >= number(&self.limits, &format!("accepted_{k}"))
-                                || number(&usage, k) + number(spec, &format!("estimated_{k}"))
-                                    > number(&self.limits, &format!("accepted_{k}"))
-                        })
-                    {
-                        continue;
-                    }
                     let lease = json!({"run_id":self.store.config.run_id,"queue_id":self.queue_id,"task_id":id,"attempt_id":uid(),"generation":number(task,"generation")+1,"coordinator_epoch":self.epoch,"token":format!("{}{}",uid(),uid()),"worker_id":worker});
                     let assignment = json!({"task":spec,"lease":lease});
-                    let size = bytes(&assignment)?.len() as u64 + 1;
-                    if assignment_bytes + size > metadata {
-                        if assignments.is_empty() {
-                            return fail(
-                                "ResourceLimitExceeded",
-                                "Assignment exceeds control metadata limit",
-                            );
-                        };
-                        break;
-                    }
-                    assignment_bytes += size;
                     assignments.push(assignment);
-                    let key = if control {
-                        "control_inflight"
-                    } else {
-                        "inflight"
-                    };
-                    usage[key] = json!(number(&usage, key) + 1);
-                    for k in ["records", "bytes", "tokens"] {
-                        usage[k] =
-                            json!(number(&usage, k) + number(spec, &format!("estimated_{k}")));
-                    }
                     if assignments.len() as u64 == max {
                         break;
                     }
                 }
                 if assignments.is_empty() {
-                    return Ok(
-                        json!({"status":if pending{"backpressured"}else{"empty"},"assignments":[]}),
-                    );
+                    return Ok(json!({"status":"empty","assignments":[]}));
                 }
                 let leases = assignments
                     .iter()
                     .map(|a| a["lease"].clone())
                     .collect::<Vec<_>>();
-                bounded(&json!(leases), metadata)?;
                 self.commit(json!([{"type":"Leased","leases":leases}]), fault)?;
                 for lease in leases {
                     self.deadlines
@@ -932,7 +840,6 @@ impl Coordinator {
             "save_task_progress_many" | "yield_tasks" => {
                 let yielding = method == "yield_tasks";
                 let updates = array(args, "updates")?;
-                bounded(&json!(updates), metadata)?;
                 let request = self.request(
                     if yielding {
                         "yield_many"
@@ -987,7 +894,6 @@ impl Coordinator {
             }
             "release_tasks" => {
                 let leases = array(args, "leases")?;
-                bounded(&json!(leases), metadata)?;
                 let request =
                     self.request("release_tasks", field(args, "request_id")?, &json!(leases))?;
                 if !request.2.is_null() {
@@ -1021,7 +927,6 @@ impl Coordinator {
             }
             "heartbeat" => {
                 let leases = array(args, "leases")?;
-                bounded(&json!(leases), metadata)?;
                 let mut results = vec![];
                 for lease in leases {
                     match self.valid_lease(lease, now) {
@@ -1040,7 +945,6 @@ impl Coordinator {
             }
             "complete_task" => {
                 let started = Instant::now();
-                bounded(args, metadata)?;
                 let lease = &args["lease"];
                 let id = field(args, "submission_id")?;
                 if id.is_empty() || id.len() > 1024 {
@@ -1070,14 +974,6 @@ impl Coordinator {
                 if number(reference, "records") == 0 && !boolean(&task["spec"], "allow_empty") {
                     return fail("InvalidReference", "Task does not allow empty output");
                 }
-                if number(reference, "payload_bytes") > number(&self.limits, "max_result_bytes")
-                    || number(reference, "tokens") > number(&self.limits, "max_result_tokens")
-                {
-                    return fail(
-                        "ResourceLimitExceeded",
-                        "Per-task result upper bound exceeded",
-                    );
-                }
                 self.validate(
                     reference,
                     Some(field(lease, "task_id")?),
@@ -1100,7 +996,6 @@ impl Coordinator {
                 let retryable = args["retryable"].as_bool().unwrap_or(true);
                 let failure = &args["failure"];
                 let content = json!({"lease":lease,"failure":failure,"retryable":retryable});
-                bounded(&content, metadata)?;
                 let request = self.request("fail", field(args, "request_id")?, &content)?;
                 if !request.2.is_null() {
                     return Ok(request.2);
@@ -1124,7 +1019,6 @@ impl Coordinator {
             }
             "yield_task" | "save_task_progress" => {
                 let content = json!({"lease":args["lease"],"input_ref":args["input_ref"]});
-                bounded(&content, metadata)?;
                 let yielding = method == "yield_task";
                 let request = self.request(
                     if yielding { "yield" } else { "task_progress" },
@@ -1207,10 +1101,10 @@ impl Coordinator {
                 let commits = array(&self.state, "commits")?;
                 let cursor = number(args, "cursor") as usize;
                 let limit = args["limit"].as_u64().unwrap_or(100) as usize;
-                if cursor > commits.len() || limit == 0 || limit > 1000 {
+                if cursor > commits.len() || limit == 0 {
                     return fail("ValueError", "Invalid cursor/page limit");
                 }
-                let mut end = (cursor + limit).min(commits.len());
+                let end = cursor.saturating_add(limit).min(commits.len());
                 if cursor < number(&self.state, "retired_prefix") as usize
                     || array(&self.state, "retired_positions")?
                         .iter()
@@ -1222,15 +1116,6 @@ impl Coordinator {
                         "Requested history was released; resume from a retained checkpoint or a live cursor",
                     );
                 }
-                while bytes(&json!(&commits[cursor..end]))?.len() as u64 > metadata {
-                    end = cursor + (end - cursor) / 2;
-                }
-                if end == cursor && cursor < commits.len() {
-                    return fail(
-                        "ResourceLimitExceeded",
-                        "Commit exceeds control message limit",
-                    );
-                }
                 Ok(
                     json!({"commits":&commits[cursor..end],"cursor":end,"end_of_input":self.sealed&&end==commits.len()&&vals(&self.state["tasks"]).all(|t|terminal(&t["state"]))}),
                 )
@@ -1324,7 +1209,6 @@ impl Coordinator {
                 }
                 self.validate(&args["plan_ref"], None, None)?;
                 let batch = json!({"batch_id":batch_id,"consumer_id":id,"input_positions":positions,"plan_ref":args["plan_ref"],"ready":false});
-                bounded(&batch, metadata)?;
                 let request = self.request("plan_batch", batch_id, &batch)?;
                 if !request.2.is_null() {
                     return Ok(self.state["batches"][batch_id].clone());
@@ -1350,7 +1234,6 @@ impl Coordinator {
                 self.validate(&args["ready_ref"], None, None)?;
                 let state = self.consumer_state(args)?;
                 let content = json!({"batch_id":batch,"ready_ref":args["ready_ref"],"state":state});
-                bounded(&content, metadata)?;
                 let request = self.request("batch_ready", batch, &content)?;
                 if !request.2.is_null() {
                     return Ok(request.2);
@@ -1396,7 +1279,6 @@ impl Coordinator {
                 }
                 let mut content = checkpoint;
                 content["checkpoint_ref"] = reference.clone();
-                bounded(&content, metadata)?;
                 let request = self.request("checkpoint", id, &content)?;
                 if request.2.is_null() {
                     self.commit(

{straw_queue-0.1.1 → straw_queue-0.1.3}/rust/gc.rs

@@ -98,13 +98,16 @@ impl Catalog {
             }
             let sequence = u64::from_le_bytes(header[8..16].try_into().unwrap());
             let n = u64::from_le_bytes(header[16..24].try_into().unwrap());
-            if n > 64 * 1024 * 1024 || sequence != self.sequence {
+            if sequence != self.sequence {
                 return fail(
                     "CorruptData",
                     "Storage catalog length/sequence outside bounds",
                 );
             }
-            if remaining < 56 + n + 40 {
+            let frame_size = n.checked_add(96).ok_or_else(|| {
+                Error::new("CorruptData", "Storage catalog frame length overflow")
+            })?;
+            if remaining < frame_size {
                 break;
             }
             let mut raw = vec![0; n as usize];
@@ -173,12 +176,6 @@ impl Catalog {
     fn append(&mut self, config: &Config, mut event: Value, fault: &mut Fault<'_>) -> Result<()> {
         event["run_id"] = json!(config.run_id);
         let raw = bytes(&event)?;
-        if raw.len() > 64 * 1024 * 1024 {
-            return fail(
-                "ResourceLimitExceeded",
-                "Storage catalog transaction too large",
-            );
-        }
         let mut header = b"STRGC001".to_vec();
         header.extend_from_slice(&self.sequence.to_le_bytes());
         header.extend_from_slice(&(raw.len() as u64).to_le_bytes());

{straw_queue-0.1.1 → straw_queue-0.1.3}/rust/journal.rs

@@ -9,7 +9,6 @@ use std::{
 };
 const MAGIC: &[u8] = b"SLMTXN01";
 const END: &[u8] = b"SLMTEND1";
-const MAX_BLOCK: usize = 8 * 1024 * 1024;
 pub struct Journal {
     pub path: PathBuf,
     file: Option<File>,
@@ -89,10 +88,13 @@ impl Journal {
             }
             let sequence = u64::from_le_bytes(header[8..16].try_into().unwrap());
             let length = u64::from_le_bytes(header[16..24].try_into().unwrap());
-            if &header[..8] != MAGIC || sequence != self.sequence || length > MAX_BLOCK as u64 {
+            if &header[..8] != MAGIC || sequence != self.sequence {
                 return fail("CorruptData", "Journal sequence, magic or length mismatch");
             }
-            if remaining < 56 + length + 40 {
+            let frame_size = length
+                .checked_add(96)
+                .ok_or_else(|| Error::new("CorruptData", "Journal frame length overflow"))?;
+            if remaining < frame_size {
                 break;
             }
             let mut body = vec![0; (length + 40) as usize];
@@ -132,9 +134,6 @@ impl Journal {
             );
         }
         let payload = bytes(events)?;
-        if payload.len() > MAX_BLOCK {
-            return fail("ResourceLimitExceeded", "Journal transaction too large");
-        }
         let mut prefix = MAGIC.to_vec();
         prefix.extend_from_slice(&self.sequence.to_le_bytes());
         prefix.extend_from_slice(&(payload.len() as u64).to_le_bytes());

{straw_queue-0.1.1 → straw_queue-0.1.3}/rust/lib.rs

@@ -2,6 +2,7 @@
 pub mod coordinator;
 pub mod gc;
 pub mod journal;
+pub mod payload;
 #[cfg(feature = "python")]
 mod python;
 pub mod store;