@ewhauser/world-celld 0.4.0 → 0.4.1

package/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to this project will be documented in this file.
 
+## 0.4.1 (2026-08-24)
+
+## What's Changed
+* fix: harden remote transport failure handling by @ewhauser in https://github.com/ewhauser/world-celld/pull/34
+* test: enforce real celld restart recovery by @ewhauser in https://github.com/ewhauser/world-celld/pull/36
+* test: cover index compaction alarm recovery by @ewhauser in https://github.com/ewhauser/world-celld/pull/37
+* test: harden index compaction rollback coverage by @ewhauser in https://github.com/ewhauser/world-celld/pull/38
+* test: harden StreamDO corruption handling by @ewhauser in https://github.com/ewhauser/world-celld/pull/39
+* test: harden configuration and malformed input handling by @ewhauser in https://github.com/ewhauser/world-celld/pull/40
+* fix: enforce negative apply-event contracts by @ewhauser in https://github.com/ewhauser/world-celld/pull/41
+
+
+**Full Changelog**: https://github.com/ewhauser/world-celld/compare/v0.4.0...v0.4.1
+
 ## 0.4.0 (2026-08-22)
 
 ## What's Changed

package/README.md

@@ -180,19 +180,30 @@ variable is shown below.
 
 The deployed worker also accepts these celld variables:
 
-| Variable                          | Default  | Purpose                                                  |
-| --------------------------------- | -------- | -------------------------------------------------------- |
-| `WORLD_SECRET`                    | none     | Required bearer secret for RPC routes                    |
-| `WORKFLOW_CALLBACK_SECRET`        | none     | Sent with deliveries as `x-workflow-callback-secret`     |
-| `QUEUE_MAX_ATTEMPTS`              | `5`      | Attempts before a message is dead-lettered               |
-| `QUEUE_MAX_INFLIGHT`              | `5`      | Concurrent deliveries per queue cell (maximum `128`)     |
-| `QUEUE_DELIVERY_TIMEOUT_MS`       | `300000` | Callback timeout (maximum `300000`)                      |
-| `WORKFLOW_RETENTION_MS`           | `0`      | Maximum run age from creation; includes active runs      |
-| `WORKFLOW_RETENTION_BATCH_SIZE`   | `128`    | Runs admitted by each cron sweep (maximum `1000`)        |
-| `WORKFLOW_RETENTION_QUEUE_SHARDS` | `1`      | Queue-shard fallback for runs created before this policy |
+| Variable                          | Default  | Purpose                                              |
+| --------------------------------- | -------- | ---------------------------------------------------- |
+| `WORLD_SECRET`                    | none     | Required bearer secret for RPC routes                |
+| `WORKFLOW_CALLBACK_SECRET`        | none     | Sent with deliveries as `x-workflow-callback-secret` |
+| `QUEUE_MAX_ATTEMPTS`              | `5`      | Attempts before a message is dead-lettered           |
+| `QUEUE_MAX_INFLIGHT`              | `5`      | Concurrent deliveries per queue cell (maximum `128`) |
+| `QUEUE_DELIVERY_TIMEOUT_MS`       | `300000` | Callback timeout (maximum `300000`)                  |
+| `WORKFLOW_RETENTION_MS`           | `0`      | Maximum run age from creation; includes active runs  |
+| `WORKFLOW_RETENTION_BATCH_SIZE`   | `128`    | Runs admitted by each cron sweep (maximum `1000`)    |
+| `WORKFLOW_RETENTION_QUEUE_SHARDS` | `1`      | Queue-shard fallback for older runs                  |
 
 `queueShards` is part of queue placement and is pinned when a queue cell is
-first used. Drain pending work before changing it.
+first used. It may be any positive safe integer; the `128` storage-operation
+batch limit does not cap the fleet's total shard count. Drain pending work
+before changing it.
+
+Queue deadlines use fixed-width 13-digit epoch-millisecond keys. A requested
+`delaySeconds` or handler redelivery timeout is accepted only when its deadline
+is at most `9999999669998`, leaving 330,001 ms for a fresh alarm edge and the
+maximum delivery lease. Every derived due, retry, inflight, and GC deadline is
+checked against the absolute `9999999999999` limit. A production retry which
+cannot preserve that headroom is dead-lettered instead of clamped or rescheduled;
+the test pump terminates the equivalent in-memory message. Long test-mode waits
+are chunked at the host timer limit without changing this deadline contract.
 
 ## Workflow retention
 
@@ -219,6 +230,7 @@ does not provide the desired expiration resolution or catch-up rate.
 New runs persist the application's `queueShards` value for complete queue
 cleanup. `WORKFLOW_RETENTION_QUEUE_SHARDS` is the fallback for runs created
 before that metadata existed and must match the placement used by those runs.
+Both values may be any positive safe integer.
 
 ### Terminal payload retention
 
@@ -293,6 +305,33 @@ CELLD_WORLD_SECRET=replace-with-a-secret \
 pnpm test:integration
 ```
 
+### Real celld restart smoke
+
+The required CI smoke owns native celld v0.3.0 and MinIO processes on loopback,
+uses fresh temporary bucket and runtime state, and kills celld with `SIGKILL`
+before deleting its local working state and starting a new process against the
+bucket-backed state. It checks that:
+
+- acknowledged run and stream state survives the process restart;
+- an accepted delayed queue message that becomes due while celld is down is
+  delivered once after the alarm is restored;
+- multi-page retention cleanup continues from a persisted nonterminal phase;
+- cancelling an in-flight HTTP long poll leaves the stream writable and
+  readable.
+
+Run the same bounded smoke locally on Linux x86-64 or macOS arm64:
+
+```sh
+pnpm test:integration:celld-smoke
+```
+
+The runner downloads celld and MinIO artifacts at pinned versions and verifies
+their SHA-256 digests before use. MinIO does not implement the conditional-write
+contract celld requires for production ownership fencing, so the smoke disables
+the storage probe and runs exactly one celld process at a time. It proves the
+single-process restart boundaries above, not multi-node ownership, handoff, or
+failover correctness.
+
 ### Local MinIO performance and loss test
 
 The opt-in performance harness starts a fresh MinIO bucket and a single celld

package/celld-worker/README.md

@@ -42,9 +42,11 @@ well as terminal ones. Each cron occurrence admits at most
 state machine finishes bounded index, stream, queue, and payload cleanup.
 
 New runs persist the application's `queueShards` placement. Set
-`WORKFLOW_RETENTION_QUEUE_SHARDS` to the historical queue-shard count when
-cleaning runs created before this worker version. Edit `triggers.crons` in the
-copied config if hourly discovery is not the desired resolution.
+`WORKFLOW_RETENTION_QUEUE_SHARDS` to the historical positive-safe-integer
+queue-shard count when cleaning runs created before this worker version. The
+`128` multikey storage-operation limit is not a total-shard ceiling. Edit
+`triggers.crons` in the copied config if hourly discovery is not the desired
+resolution.
 
 Point the app at any node's public listener:
 

package/dist/index.d.ts

@@ -61,7 +61,6 @@ interface ReleaseHookIndexesResult {
 }
 interface ExpireRunIndexesRequest {
   runId: string;
-  keys: string[];
   hooks: HookIndexReference[];
   expiredAt: number;
 }
@@ -370,9 +369,11 @@ interface IndexListOptions {
   reverse?: boolean;
 }
 interface IndexListPage {
+  /** sourceShard is populated by the merged fleet index for exact follow-up mutation. */
   keys: Array<{
     name: string;
     value: string;
+    sourceShard?: string;
   }>;
   list_complete: boolean;
   cursor?: string;

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/stream-protocol.ts","../src/streamer.ts","../src/queue.ts","../src/indexes.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;;EAEX,SAAS;;EAET,cAAc;;EAEd,iBAAiB;EACjB;EACA,OAAO;EACP;EACA,OAAO;;EAEP;EACA;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;;EAEA,SAAS;EACT,cAAc;EACd,iBAAiB;EACjB,WAAW;EACX,cAAc;;EAEd,SAAS;;KAGC;KAEA,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAae;EACf;EACA,OAAO;;UAGQ;EACf;;UAGe;EACf;EACA;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;;EAEf;EACA;;UAGe;;EAEf;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;;KAGU;;EAEV;;EAEI;;EAEA;;EAEA,SAAS;;UAIE;EACf;;UAGe;EACf;EACA;;;;UCkBe;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B;;EAEA,UAAU;;KAGA;UAWK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UCnKnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,sBAAsB;IACpB;IACA;IACA;MACE;IAAU;;EACd,sBAAsB,QADlB;EAEJ,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;cCrFW;;cAEA;;cAEA;cACA;cACA;cACA;;cAEA;KAED;UAEK;EACf;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,QAAQ;EACR,OAAO;EACP;EACA,QAAQ;;;;UCXO,sBAAsB,KAAK;EAC1C,SAAS;IACP,KAAK,eAAe,cAAc,OAAO,2BAA2B;;EAEtE,cACE,cACA,gBAAgB,iBAChB,gBAAgB,aACf;EACH,oBACE,cACA,gBAAgB,iBAChB,2BAA2B,gBAC1B;EACH,YAAY,cAAc,gBAAgB,kBAAkB;EAC5D,YACE,cACA,gBAAgB,iBAChB,OAAO,2BACN;EACH,eACE,cACA,eACA,sBACC,QAAQ,eAAe;EAC1B,mBAAmB,gBAAgB;EACnC,gBACE,cACA,eACA,UAAU,mBACT,QAAQ;EACX,cAAc,cAAc,gBAAgB,QAAQ;;;UAIrC;EACf,YAAY,eAAe,QAAQ,eAAe,QAAQ;EAC1D,WAAW,SAAS,mBAAmB,SAAS,cAAc,QAAQ;EACtE,YAAY,gBAAgB;EAC5B,WAAW,eAAe,OAAO,2BAA2B;EAC5D,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eACE,eACA,mBACA;IAAY;MACX,QAAQ;EACX,iBAAiB,eAAe,oBAAoB,QAAQ;EAC5D,aACE,eACA,mBACA;IAAY;IAAgB;MAC3B,QAAQ;;UAGI;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KC1CG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,mBACA;IAAY;MACX,QAL+B;EAMlC,qBACE,eACA,SAHC,qBAIA,QADiC;;UAIrB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;UCnGlB;EACf;EACA;;EAEA;EACA;EACA;;UAGe;EACf,MAAM;IAAQ;IAAc;;EAC5B;EACA;;UAuBe;EACf;EACA;EACA;;KAGU;EACN;EAAgB,aAAa;;EAC7B;EAAiB,QAAQ;;;UAgDd;EACf,UACE,KAAK,aACL,4BACA,+BACC;IAAU;;EACb,SAAS,UAAU,mBAAmB,QAAQ;EAC9C,eAAe,gBAAgB;EAC/B,YAAY,iBAAiB;EAC7B,YAAY,eAAe,OAAO,iBAAiB,QAAQ;EAC3D,oBACE,eACA,gBACA,wBACA,OAAO,gBACP,cAAc,kBACb;EACH,uBACE,eACA,OAAO,gBACP,aAAa,kBACZ;EACH,mBAAmB,SAAS,4BAA4B,QAAQ;EAChE,UAAU,SAAS,0BAA0B,QAAQ;;;;;KC1H3C,iBAAiB;UAEZ;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA;;;;UCpBe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa,QAAQ;EAAkB,WAAW;;iBAE9C,iBAAiB,SAAS,mBAAmB"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/stream-protocol.ts","../src/streamer.ts","../src/queue.ts","../src/indexes.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;;EAEX,SAAS;;EAET,cAAc;;EAEd,iBAAiB;EACjB;EACA,OAAO;EACP;EACA,OAAO;;EAEP;EACA;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;;EAEA,SAAS;EACT,cAAc;EACd,iBAAiB;EACjB,WAAW;EACX,cAAc;;EAEd,SAAS;;KAGC;KAEA,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAae;EACf;EACA,OAAO;;UAGQ;EACf;;UAGe;EACf;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;;EAEf;EACA;;UAGe;;EAEf;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;;KAGU;;EAEV;;EAEI;;EAEA;;EAEA,SAAS;;UAIE;EACf;;UAGe;EACf;EACA;;;;UCoBe;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B;;EAEA,UAAU;;KAiFA;UAWK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UCzOnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,sBAAsB;IACpB;IACA;IACA;MACE;IAAU;;EACd,sBAAsB,QADlB;EAEJ,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;cC9FW;;cAEA;;cAEA;cACA;cACA;cACA;;cAEA;KAED;UAEK;EACf;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,QAAQ;EACR,OAAO;EACP;EACA,QAAQ;;;;UCXO,sBAAsB,KAAK;EAC1C,SAAS;IACP,KAAK,eAAe,cAAc,OAAO,2BAA2B;;EAEtE,cACE,cACA,gBAAgB,iBAChB,gBAAgB,aACf;EACH,oBACE,cACA,gBAAgB,iBAChB,2BAA2B,gBAC1B;EACH,YAAY,cAAc,gBAAgB,kBAAkB;EAC5D,YACE,cACA,gBAAgB,iBAChB,OAAO,2BACN;EACH,eACE,cACA,eACA,sBACC,QAAQ,eAAe;EAC1B,mBAAmB,gBAAgB;EACnC,gBACE,cACA,eACA,UAAU,mBACT,QAAQ;EACX,cAAc,cAAc,gBAAgB,QAAQ;;;UAIrC;EACf,YAAY,eAAe,QAAQ,eAAe,QAAQ;EAC1D,WAAW,SAAS,mBAAmB,SAAS,cAAc,QAAQ;EACtE,YAAY,gBAAgB;EAC5B,WAAW,eAAe,OAAO,2BAA2B;EAC5D,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eACE,eACA,mBACA;IAAY;MACX,QAAQ;EACX,iBAAiB,eAAe,oBAAoB,QAAQ;EAC5D,aACE,eACA,mBACA;IAAY;IAAgB;MAC3B,QAAQ;;UAGI;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KChBG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,mBACA;IAAY;MACX,QAL+B;EAMlC,qBACE,eACA,SAHC,qBAIA,QADiC;;UAIrB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;UC7HlB;EACf;EACA;;EAEA;EACA;EACA;;UAGe;;EAEf,MAAM;IAAQ;IAAc;IAAe;;EAC3C;EACA;;UAuBe;EACf;EACA;EACA;;KAGU;EACN;EAAgB,aAAa;;EAC7B;EAAiB,QAAQ;;;UAgDd;EACf,UACE,KAAK,aACL,4BACA,+BACC;IAAU;;EACb,SAAS,UAAU,mBAAmB,QAAQ;EAC9C,eAAe,gBAAgB;EAC/B,YAAY,iBAAiB;EAC7B,YAAY,eAAe,OAAO,iBAAiB,QAAQ;EAC3D,oBACE,eACA,gBACA,wBACA,OAAO,gBACP,cAAc,kBACb;EACH,uBACE,eACA,OAAO,gBACP,aAAa,kBACZ;EACH,mBAAmB,SAAS,4BAA4B,QAAQ;EAChE,UAAU,SAAS,0BAA0B,QAAQ;;;;;KC1H3C,iBAAiB;UAEZ;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA;;;;UCrBe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa,QAAQ;EAAkB,WAAW;;iBAS9C,iBAAiB,SAAS,mBAAmB"}