@ewhauser/world-celld 0.3.1 → 0.4.0

package/CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to this project will be documented in this file.
 
+## 0.4.0 (2026-08-22)
+
+## What's Changed
+* chore: test against celld v0.3.0 by @ewhauser in https://github.com/ewhauser/world-celld/pull/31
+* feat: add configurable workflow retention by @ewhauser in https://github.com/ewhauser/world-celld/pull/33
+
+
+**Full Changelog**: https://github.com/ewhauser/world-celld/compare/v0.3.2...v0.4.0
+
+## 0.3.2 (2026-08-19)
+
+## What's Changed
+* fix: compact expired-run lifecycle fences by @ewhauser in https://github.com/ewhauser/world-celld/pull/29
+
+
+**Full Changelog**: https://github.com/ewhauser/world-celld/compare/v0.3.1...v0.3.2
+
 ## 0.3.1 (2026-08-19)
 
 ## What's Changed

package/README.md

@@ -75,8 +75,9 @@ requests. It must be reachable from every celld node.
 
 ## Deploy the worker
 
-Before deploying, you need a celld fleet, `esbuild` on `PATH`, and an object
-store that meets celld's conditional-write requirements. Refer to the
+Before deploying, you need a celld v0.3.0 fleet, `esbuild` on `PATH`, and an
+object store that meets celld's conditional-write requirements. v0.3.0 is the
+currently tested runtime baseline. Refer to the
 [celld documentation](https://github.com/denoland/celld) for fleet and storage
 setup.
 
@@ -98,6 +99,7 @@ the same secret to the fleet and the application:
 
 ```sh
 CELLD_VAR_WORLD_SECRET="$CELLD_WORLD_SECRET" \
+CELLD_VAR_WORKFLOW_RETENTION_MS=7776000000 \
 celld --bucket s3://my-cells-bucket \
   --listen 0.0.0.0:8080 \
   --internal-listen 10.0.0.12:8081 \
@@ -108,6 +110,9 @@ Use a secret manager rather than putting the value in `wrangler.jsonc`. Keep
 celld's internal listener on a trusted network; the World bearer token protects
 the worker RPC routes, not celld's administrative endpoints.
 
+The example sets a fleet-wide maximum workflow age of 90 days. Leave
+`WORKFLOW_RETENTION_MS` at `0` to disable that policy.
+
 More deployment detail is in [`celld-worker/README.md`](./celld-worker/README.md).
 
 ## How it works
@@ -118,9 +123,9 @@ Workflow application
   | authenticated HTTP RPC + binary stream batches
   v
 celld worker router
+  |-- scheduled       hourly maximum-age retention discovery
   |-- WorkflowRunDO  one cell per workflow run
   |-- RunCatalogDO   16 stable run-id shards; merged ordered listing
-  |-- RunFenceDO     one lifecycle fence cell per workflow run
   |-- HookTokenDO    32 stable token-ownership shards
   |-- HookIdDO       32 stable hook-id lookup shards
   |-- StreamDO       stream chunks and run/stream indexes
@@ -141,17 +146,20 @@ Index routing is a versioned hard-cutover protocol. Run catalog writes hash the
 `runId` across 16 shards, while hook ownership and hook-ID lookup hash their
 natural keys across 32 shards each. A single stateless worker request queries
 all 16 catalog shards in parallel and merges their lexicographic pages before
-authoritative RunDO reads. A terminal or expired run is hidden through its own
-`RunFenceDO`, so no global fence cell remains. Token and hook-ID admission are
-each transactional inside their natural-key shards. The RunDO remains the
-authoritative commit domain, and exact claim IDs plus a serialized cancellation
-fence resolve ambiguous delivery without pretending the two ownership shards
-form a cross-Durable-Object transaction. Stateless cohesive index routes keep
+authoritative RunDO reads. The RunDO is also the only permanent lifecycle
+authority: hook shards consult it directly, while queue and catalog expiry
+fences compact after bounded protocol leases. Token and hook-ID admission are
+each transactional inside their natural-key shards. Exact claim IDs plus a
+serialized cancellation fence resolve ambiguous delivery without pretending
+the two ownership shards form a cross-Durable-Object transaction. Stateless cohesive index routes keep
 catalog listing, terminal commits, hook finalization, and hook release to one
 public request each while exposing the internal shard work in measurements.
 
 The routing rationale and before/after RPC, storage, latency, and contention
 evidence are recorded in [`docs/index-sharding.md`](./docs/index-sharding.md).
+The expiry authority, producer/consumer map, bounded horizons, and
+post-compaction costs are recorded in
+[`docs/lifecycle-compaction.md`](./docs/lifecycle-compaction.md).
 
 ## Configuration
 
@@ -168,22 +176,51 @@ variable is shown below.
 | `runRetentionMs`        | `CELLD_RUN_RETENTION_MS` | `0` (disabled)           |
 | `streamLongPollMs`      | —                        | `20000`                  |
 | `streamFlushIntervalMs` | —                        | `0`                      |
-| `rpcTimeoutMs`          | —                        | `30000`                  |
+| `rpcTimeoutMs`          | —                        | `30000` (max `300000`)   |
 
 The deployed worker also accepts these celld variables:
 
-| Variable                    | Default  | Purpose                                              |
-| --------------------------- | -------- | ---------------------------------------------------- |
-| `WORLD_SECRET`              | none     | Required bearer secret for RPC routes                |
-| `WORKFLOW_CALLBACK_SECRET`  | none     | Sent with deliveries as `x-workflow-callback-secret` |
-| `QUEUE_MAX_ATTEMPTS`        | `5`      | Attempts before a message is dead-lettered           |
-| `QUEUE_MAX_INFLIGHT`        | `5`      | Concurrent deliveries per queue cell (maximum `128`) |
-| `QUEUE_DELIVERY_TIMEOUT_MS` | `300000` | Timeout for an application callback                  |
+| Variable                          | Default  | Purpose                                                  |
+| --------------------------------- | -------- | -------------------------------------------------------- |
+| `WORLD_SECRET`                    | none     | Required bearer secret for RPC routes                    |
+| `WORKFLOW_CALLBACK_SECRET`        | none     | Sent with deliveries as `x-workflow-callback-secret`     |
+| `QUEUE_MAX_ATTEMPTS`              | `5`      | Attempts before a message is dead-lettered               |
+| `QUEUE_MAX_INFLIGHT`              | `5`      | Concurrent deliveries per queue cell (maximum `128`)     |
+| `QUEUE_DELIVERY_TIMEOUT_MS`       | `300000` | Callback timeout (maximum `300000`)                      |
+| `WORKFLOW_RETENTION_MS`           | `0`      | Maximum run age from creation; includes active runs      |
+| `WORKFLOW_RETENTION_BATCH_SIZE`   | `128`    | Runs admitted by each cron sweep (maximum `1000`)        |
+| `WORKFLOW_RETENTION_QUEUE_SHARDS` | `1`      | Queue-shard fallback for runs created before this policy |
 
 `queueShards` is part of queue placement and is pinned when a queue cell is
 first used. Drain pending work before changing it.
 
-## Terminal run retention
+## Workflow retention
+
+### Fleet-wide maximum age
+
+Set the deployed worker's `WORKFLOW_RETENTION_MS` variable to expire every
+workflow after a maximum age measured from `createdAt`. For example,
+`7776000000` is 90 days. This policy applies to pending, running, completed,
+failed, and cancelled runs.
+
+The packaged worker declares an hourly UTC celld cron trigger. Each occurrence
+scans one bounded creation-time catalog page and asks the authoritative run
+cells to enforce the cutoff. A run cell immediately fences reads, writes, stream
+activity, and queue work, removes its catalog entry, then finishes the existing
+bounded stream, queue, and payload cleanup phases through its durable alarm.
+Repeated cron invocations and alarm retries are idempotent.
+
+One occurrence admits at most `WORKFLOW_RETENTION_BATCH_SIZE` runs (default
+`128`, maximum `1000`). If the fleet was down or a shorter policy creates a
+backlog, later occurrences continue with the next oldest entries. Edit
+`triggers.crons` in the copied `celld-worker/wrangler.jsonc` if hourly discovery
+does not provide the desired expiration resolution or catch-up rate.
+
+New runs persist the application's `queueShards` value for complete queue
+cleanup. `WORKFLOW_RETENTION_QUEUE_SHARDS` is the fallback for runs created
+before that metadata existed and must match the placement used by those runs.
+
+### Terminal payload retention
 
 When `runRetentionMs` is greater than zero, a terminal transition atomically
 records the run's `expiredAt` and arms its cell alarm. The complete run, event,
@@ -217,7 +254,8 @@ await world.retention.rearm(runId); // recover a missed or abandoned alarm
 
 The retention deadline is pinned when the run becomes terminal. Changing the
 configuration affects newly terminal runs; call `schedule()` explicitly for
-an existing terminal run that has no cleanup record.
+an existing terminal run that has no cleanup record. If terminal retention and
+fleet-wide maximum age are both enabled, the earlier deadline wins.
 
 ## Operational notes
 
@@ -263,7 +301,7 @@ message reaches a successful callback, including forced `503` redeliveries. A
 second workload creates terminal runs with streams and delayed queue messages,
 then verifies complete payload cleanup without resurrection. Results include
 queue and cleanup throughput plus p50, p95, p99, and maximum latency and are
-saved under `.perf-results/`.
+saved under `.perf-results/`. The harness currently pins celld v0.3.0.
 
 > MinIO Community is **not a supported celld production store**. It does not
 > implement the conditional writes celld needs for ownership fencing. This
@@ -282,7 +320,7 @@ and force every twentieth message through one `503` redelivery.
 
 #### Reference result
 
-The following directional baseline was recorded on August 15, 2026. MinIO and
+The following directional baseline was recorded on August 21, 2026. MinIO and
 celld ran as native arm64 processes because Docker was unavailable on the test
 machine; container results will differ.
 
@@ -290,21 +328,25 @@ machine; container results will differ.
 | ----------- | -------------------------------------------------------------------------------------------------------- |
 | Machine     | MacBook Pro (Mac15,8), Apple M3 Max, 16 cores (12 performance, 4 efficiency), 64 GB RAM                  |
 | OS          | macOS 26.5.2 (25F84), arm64                                                                              |
-| Services    | celld v0.2.1; MinIO RELEASE.2025-09-07T16-13-09Z                                                         |
+| Services    | celld v0.3.0; MinIO RELEASE.2025-09-07T16-13-09Z                                                         |
 | Workload    | 1,000 messages; concurrency 32; 2 queue shards; 256-byte target payload; every 20th message retried once |
 
 | Metric                  |                                                                            Result |
 | ----------------------- | --------------------------------------------------------------------------------: |
 | Delivery integrity      | 1,000 accepted, 1,000 delivered, 0 missing, 0 duplicate successes, 0 dead letters |
 | Callback attempts       |                                           1,050, including 50 forced redeliveries |
-| Enqueue throughput      |                                                                 882.55 messages/s |
-| Delivery throughput     |                                                                 808.81 messages/s |
-| Enqueue latency         |                          p50 24.37 ms; p95 75.60 ms; p99 176.67 ms; max 200.36 ms |
-| All delivery latency    |                         p50 53.95 ms; p95 167.86 ms; p99 210.84 ms; max 312.03 ms |
-| First-attempt latency   |                         p50 52.55 ms; p95 166.35 ms; p99 209.80 ms; max 212.01 ms |
-| Retried-message latency |                         p50 82.52 ms; p95 262.63 ms; p99 312.03 ms; max 312.03 ms |
+| Enqueue throughput      |                                                                 865.66 messages/s |
+| Delivery throughput     |                                                                 794.60 messages/s |
+| Enqueue latency         |                          p50 26.06 ms; p95 70.72 ms; p99 146.59 ms; max 157.70 ms |
+| All delivery latency    |                         p50 65.05 ms; p95 184.47 ms; p99 225.10 ms; max 328.75 ms |
+| First-attempt latency   |                         p50 63.65 ms; p95 177.93 ms; p99 209.38 ms; max 226.86 ms |
+| Retried-message latency |                        p50 114.81 ms; p95 286.35 ms; p99 328.75 ms; max 328.75 ms |
 | Final queue state       |                         0 pending, 0 in flight, 0 dead letters across both shards |
 
+The retention workload in the same run tombstoned 100 of 100 terminal runs,
+expired all 100 reads, deleted 400 payload keys, 100 streams, and 100 delayed
+queue messages, and left both queue shards empty.
+
 Use these numbers as a smoke-test reference, not a portable performance
 guarantee. For regression tracking, compare repeated runs on the same machine
 and runtime configuration. Two shards gave the best latency balance for this

package/celld-worker/README.md

@@ -1,8 +1,8 @@
 # world-celld worker
 
-The celld-deployable half of `@ewhauser/world-celld`: seven cell classes
-(WorkflowRunDO, RunCatalogDO, RunFenceDO, HookTokenDO, HookIdDO, StreamDO, and
-QueueDO) behind an authenticated HTTP router.
+The celld-deployable half of `@ewhauser/world-celld`: six cell classes
+(WorkflowRunDO, RunCatalogDO, HookTokenDO, HookIdDO, StreamDO, and QueueDO)
+behind an authenticated HTTP router.
 Storage, control, and queue methods use fixed JSON RPC routes; stream chunks use
 bounded binary batch writes and binary long-poll reads.
 
@@ -19,11 +19,33 @@ celld deploy ./workflow-world --bucket s3://my-cells-bucket
 
 Requirements:
 
+- celld v0.3.0 (the currently tested runtime baseline).
 - `esbuild` on PATH (celld shells out to it).
 - A bucket with conditional-write support (celld's fencing requirement).
 - `WORLD_SECRET` injected at the node level (`CELLD_VAR_WORLD_SECRET=...`) —
   the router fails closed with 503 while it is empty.
 
+## Fleet-wide retention
+
+The bundled `wrangler.jsonc` declares an hourly UTC cron trigger. It does no
+catalog work by default. Set `CELLD_VAR_WORKFLOW_RETENTION_MS` on every node to
+enable a maximum workflow age measured from run creation:
+
+```sh
+CELLD_VAR_WORKFLOW_RETENTION_MS=7776000000 \
+celld --bucket s3://my-cells-bucket
+```
+
+`7776000000` is 90 days. The policy includes pending and running workflows as
+well as terminal ones. Each cron occurrence admits at most
+`WORKFLOW_RETENTION_BATCH_SIZE` runs (default `128`); the existing per-run alarm
+state machine finishes bounded index, stream, queue, and payload cleanup.
+
+New runs persist the application's `queueShards` placement. Set
+`WORKFLOW_RETENTION_QUEUE_SHARDS` to the historical queue-shard count when
+cleaning runs created before this worker version. Edit `triggers.crons` in the
+copied config if hourly discovery is not the desired resolution.
+
 Point the app at any node's public listener:
 
 ```ts

package/celld-worker/worker.ts

@@ -7,7 +7,6 @@ export {
   HookTokenDO,
   QueueDO,
   RunCatalogDO,
-  RunFenceDO,
   StreamDO,
   WorkflowRunDO,
   default,

package/celld-worker/wrangler.jsonc

@@ -10,7 +10,6 @@
       { "name": "WORKFLOW_DB", "class_name": "WorkflowRunDO" },
       { "name": "WORKFLOW_STREAMS", "class_name": "StreamDO" },
       { "name": "WORKFLOW_RUN_CATALOG", "class_name": "RunCatalogDO" },
-      { "name": "WORKFLOW_RUN_FENCES", "class_name": "RunFenceDO" },
       { "name": "WORKFLOW_HOOK_TOKENS", "class_name": "HookTokenDO" },
       { "name": "WORKFLOW_HOOK_IDS", "class_name": "HookIdDO" },
       { "name": "WORKFLOW_QUEUE", "class_name": "QueueDO" },
@@ -23,13 +22,16 @@
         "WorkflowRunDO",
         "StreamDO",
         "RunCatalogDO",
-        "RunFenceDO",
         "HookTokenDO",
         "HookIdDO",
         "QueueDO",
       ],
     },
   ],
+  // Fleet-wide maximum-age retention is discovered hourly. The handler is a
+  // no-op while WORKFLOW_RETENTION_MS is "0". Change this expression if a
+  // different sweep cadence is required; celld cron schedules are UTC.
+  "triggers": { "crons": ["0 * * * *"] },
   "vars": {
     // REQUIRED: bearer secret for the world's RPC and binary stream surface.
     // Authenticated routes fail closed with 503 while this is empty. Inject the real value
@@ -37,6 +39,14 @@
     "WORLD_SECRET": "",
     // Optional: forwarded as x-workflow-callback-secret on queue deliveries.
     "WORKFLOW_CALLBACK_SECRET": "",
+    // Optional fleet-wide maximum workflow age, measured from run creation.
+    // Applies to pending, running, and terminal runs. "0" disables it.
+    "WORKFLOW_RETENTION_MS": "0",
+    // Maximum runs admitted to cleanup by one hourly sweep (1..1000).
+    "WORKFLOW_RETENTION_BATCH_SIZE": "128",
+    // Must match createCelldWorld({ queueShards }) so every queue shard is
+    // included in cleanup for active runs discovered by the sweep.
+    "WORKFLOW_RETENTION_QUEUE_SHARDS": "1",
     // Optional queue tuning (strings): QUEUE_MAX_ATTEMPTS, QUEUE_MAX_INFLIGHT,
     // QUEUE_DELIVERY_TIMEOUT_MS.
   },

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { CreateEventParams, CreateEventRequest, Event, GetChunksOptions, Hook, RunCreatedEventRequest, Step, StreamChunksResponse, StreamInfoResponse, Streamer, TerminalWorkflowRunStatus, ValidQueueName, Wait, WorkflowRun, World } from "@workflow/world";
+import { CreateEventParams, CreateEventRequest, Event, GetChunksOptions, Hook, RunCreatedEventRequest, Step, StreamChunksResponse, StreamInfoResponse, Streamer, TerminalWorkflowRunStatus, ValidQueueName, Wait, WorkflowRun, WorkflowRunStatus, World } from "@workflow/world";
 //#region src/retention.d.ts
 type CleanupPhase = 'retained' | 'index' | 'streams' | 'queues' | 'payload' | 'tombstoned';
 interface CleanupRecord {
@@ -6,8 +6,13 @@ interface CleanupRecord {
   runId: string;
   workflowName: string;
   createdAt: Date;
-  completedAt: Date;
-  terminalStatus: TerminalWorkflowRunStatus;
+  /** Last status observed when cleanup was scheduled. */
+  status?: WorkflowRunStatus;
+  /** Present when cleanup was scheduled for a terminal run. */
+  completedAt?: Date;
+  /** Retained for persisted terminal-retention records created before max-age cleanup. */
+  terminalStatus?: TerminalWorkflowRunStatus;
+  reason?: 'terminal-retention' | 'maximum-age' | 'manual';
   dueAt: Date;
   queueShards: number;
   phase: CleanupPhase;
@@ -24,11 +29,16 @@ interface CleanupRecord {
 interface RunTombstone {
   version: 1;
   runId: string;
-  terminalStatus: TerminalWorkflowRunStatus;
-  completedAt: Date;
+  /** Last status observed before the run expired. */
+  status?: WorkflowRunStatus;
+  completedAt?: Date;
+  terminalStatus?: TerminalWorkflowRunStatus;
   expiredAt: Date;
   tombstonedAt: Date;
+  /** Final cleanup accounting retained inside the single authoritative record. */
+  cleanup: CleanupRecord;
 }
+type RunLifecycleStatus = 'active' | 'terminal' | 'expired' | 'missing';
 type RunReadOutcome<T> = {
   ok: true;
   value: T;
@@ -45,8 +55,6 @@ interface HookIndexReference {
 interface ReleaseHookIndexesRequest {
   runId: string;
   hooks: HookIndexReference[];
-  /** Install a fence that prevents delayed hook finalization after terminal state. */
-  terminal: boolean;
 }
 interface ReleaseHookIndexesResult {
   deleted: number;
@@ -77,10 +85,22 @@ interface ExpireStreamResult {
   bytes: number;
   done: boolean;
 }
-interface ExpireQueueRunResult {
+interface QueueExpiryReceipt {
+  expiredAt: number;
+  deleted: number;
+}
+type ExpireQueueRunResult = {
   /** Cumulative messages removed for this run in this queue shard. */
   deleted: number;
-  done: boolean;
+} & ({
+  done: false;
+} | {
+  done: true;
+  /** Durable receipt which the cleanup coordinator must acknowledge. */
+  receipt: QueueExpiryReceipt;
+});
+interface AcknowledgeQueueExpiryResult {
+  acknowledged: boolean;
 }
 interface ScheduleCleanupRequest {
   retentionMs: number;
@@ -146,6 +166,8 @@ interface ApplyEventSuccess {
   hasMore?: boolean;
   /** Server-owned event ceiling advertised on run lifecycle writes. */
   maxEvents?: number;
+  /** Deadline for publishing this authoritative mutation to derivative indexes. */
+  indexPublicationExpiresAt?: number;
 }
 type ApplyEventOutcome = ApplyEventSuccess | ApplyEventFailure;
 //#endregion
@@ -163,6 +185,7 @@ interface WorkflowRunDOStub {
   }): Promise<{
     committed: boolean;
   }>;
+  getLifecycleStatus(): Promise<RunLifecycleStatus>;
   getRun(): Promise<RunReadOutcome<WorkflowRun | null>>;
   getStep(stepId: string): Promise<RunReadOutcome<Step | null>>;
   getEvent(eventId: string): Promise<RunReadOutcome<Event | null>>;
@@ -326,6 +349,7 @@ interface QueueCellStub {
   expireRun(runId: string, expiredAt: number, options?: {
     limit?: number;
   }): Promise<ExpireQueueRunResult>;
+  acknowledgeExpireRun(runId: string, receipt: QueueExpiryReceipt): Promise<AcknowledgeQueueExpiryResult>;
 }
 interface QueueCellNamespace {
   idFromName(name: string): {
@@ -340,6 +364,8 @@ interface QueueCellNamespace {
 interface IndexListOptions {
   prefix?: string;
   cursor?: string;
+  /** Exclusive upper bound for creation-time retention scans. */
+  end?: string;
   limit?: number;
   reverse?: boolean;
 }
@@ -365,7 +391,7 @@ type HookReservationResult = {
 };
 /** High-level client used by the World storage layer and retention coordinator. */
 interface WorkflowIndex {
-  commitRun(run: WorkflowRun, serializedMetadata: string): Promise<{
+  commitRun(run: WorkflowRun, serializedMetadata: string, publicationExpiresAt: number): Promise<{
     stored: boolean;
   }>;
   listRuns(options?: IndexListOptions): Promise<IndexListPage>;

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/stream-protocol.ts","../src/streamer.ts","../src/queue.ts","../src/indexes.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;EACX,aAAa;EACb,gBAAgB;EAChB,OAAO;EACP;EACA,OAAO;;EAEP;EACA;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA,gBAAgB;EAChB,aAAa;EACb,WAAW;EACX,cAAc;;KAGJ,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAae;EACf;EACA,OAAO;;EAEP;;UAGe;EACf;;UAGe;EACf;EACA;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;;EAEf;EACA;;UAGe;;EAEf;;EAEA;;EAEA;EACA;;UAGe;;EAEf;EACA;;UAGe;EACf;EACA;;;;UC0Ce;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B;;EAEA,UAAU;;KAGA;UAWK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UCjKnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,sBAAsB;IACpB;IACA;IACA;MACE;IAAU;;EACd,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;cCpFW;;cAEA;;cAEA;cACA;cACA;cACA;;cAEA;KAED;UAEK;EACf;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,QAAQ;EACR,OAAO;EACP;EACA,QAAQ;;;;UCXO,sBAAsB,KAAK;EAC1C,SAAS;IACP,KAAK,eAAe,cAAc,OAAO,2BAA2B;;EAEtE,cACE,cACA,gBAAgB,iBAChB,gBAAgB,aACf;EACH,oBACE,cACA,gBAAgB,iBAChB,2BAA2B,gBAC1B;EACH,YAAY,cAAc,gBAAgB,kBAAkB;EAC5D,YACE,cACA,gBAAgB,iBAChB,OAAO,2BACN;EACH,eACE,cACA,eACA,sBACC,QAAQ,eAAe;EAC1B,mBAAmB,gBAAgB;EACnC,gBACE,cACA,eACA,UAAU,mBACT,QAAQ;EACX,cAAc,cAAc,gBAAgB,QAAQ;;;UAIrC;EACf,YAAY,eAAe,QAAQ,eAAe,QAAQ;EAC1D,WAAW,SAAS,mBAAmB,SAAS,cAAc,QAAQ;EACtE,YAAY,gBAAgB;EAC5B,WAAW,eAAe,OAAO,2BAA2B;EAC5D,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eACE,eACA,mBACA;IAAY;MACX,QAAQ;EACX,iBAAiB,eAAe,oBAAoB,QAAQ;EAC5D,aACE,eACA,mBACA;IAAY;IAAgB;MAC3B,QAAQ;;UAGI;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KC1CG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,mBACA;IAAY;MACX,QAL+B;;UAQnB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;UC/FlB;EACf;EACA;EACA;EACA;;UAGe;EACf,MAAM;IAAQ;IAAc;;EAC5B;EACA;;UAyBe;EACf;EACA;EACA;;KAGU;EACN;EAAgB,aAAa;;EAC7B;EAAiB,QAAQ;;;UAqDd;EACf,UAAU,KAAK,aAAa,6BAA6B;IAAU;;EACnE,SAAS,UAAU,mBAAmB,QAAQ;EAC9C,eAAe,gBAAgB;EAC/B,YAAY,iBAAiB;EAC7B,YAAY,eAAe,OAAO,iBAAiB,QAAQ;EAC3D,oBACE,eACA,gBACA,wBACA,OAAO,gBACP,cAAc,kBACb;EACH,uBACE,eACA,OAAO,gBACP,aAAa,kBACZ;EACH,mBAAmB,SAAS,4BAA4B,QAAQ;EAChE,UAAU,SAAS,0BAA0B,QAAQ;;;;;KC5H3C,iBAAiB;UAEZ;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA;;;;UCnBe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa,QAAQ;EAAkB,WAAW;;iBAE9C,iBAAiB,SAAS,mBAAmB"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/stream-protocol.ts","../src/streamer.ts","../src/queue.ts","../src/indexes.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;;EAEX,SAAS;;EAET,cAAc;;EAEd,iBAAiB;EACjB;EACA,OAAO;EACP;EACA,OAAO;;EAEP;EACA;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;;EAEA,SAAS;EACT,cAAc;EACd,iBAAiB;EACjB,WAAW;EACX,cAAc;;EAEd,SAAS;;KAGC;KAEA,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAae;EACf;EACA,OAAO;;UAGQ;EACf;;UAGe;EACf;EACA;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;;EAEf;EACA;;UAGe;;EAEf;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;;KAGU;;EAEV;;EAEI;;EAEA;;EAEA,SAAS;;UAIE;EACf;;UAGe;EACf;EACA;;;;UCkBe;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B;;EAEA,UAAU;;KAGA;UAWK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UCnKnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,sBAAsB;IACpB;IACA;IACA;MACE;IAAU;;EACd,sBAAsB,QADlB;EAEJ,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;cCrFW;;cAEA;;cAEA;cACA;cACA;cACA;;cAEA;KAED;UAEK;EACf;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,QAAQ;EACR,OAAO;EACP;EACA,QAAQ;;;;UCXO,sBAAsB,KAAK;EAC1C,SAAS;IACP,KAAK,eAAe,cAAc,OAAO,2BAA2B;;EAEtE,cACE,cACA,gBAAgB,iBAChB,gBAAgB,aACf;EACH,oBACE,cACA,gBAAgB,iBAChB,2BAA2B,gBAC1B;EACH,YAAY,cAAc,gBAAgB,kBAAkB;EAC5D,YACE,cACA,gBAAgB,iBAChB,OAAO,2BACN;EACH,eACE,cACA,eACA,sBACC,QAAQ,eAAe;EAC1B,mBAAmB,gBAAgB;EACnC,gBACE,cACA,eACA,UAAU,mBACT,QAAQ;EACX,cAAc,cAAc,gBAAgB,QAAQ;;;UAIrC;EACf,YAAY,eAAe,QAAQ,eAAe,QAAQ;EAC1D,WAAW,SAAS,mBAAmB,SAAS,cAAc,QAAQ;EACtE,YAAY,gBAAgB;EAC5B,WAAW,eAAe,OAAO,2BAA2B;EAC5D,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eACE,eACA,mBACA;IAAY;MACX,QAAQ;EACX,iBAAiB,eAAe,oBAAoB,QAAQ;EAC5D,aACE,eACA,mBACA;IAAY;IAAgB;MAC3B,QAAQ;;UAGI;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KC1CG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,mBACA;IAAY;MACX,QAL+B;EAMlC,qBACE,eACA,SAHC,qBAIA,QADiC;;UAIrB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;UCnGlB;EACf;EACA;;EAEA;EACA;EACA;;UAGe;EACf,MAAM;IAAQ;IAAc;;EAC5B;EACA;;UAuBe;EACf;EACA;EACA;;KAGU;EACN;EAAgB,aAAa;;EAC7B;EAAiB,QAAQ;;;UAgDd;EACf,UACE,KAAK,aACL,4BACA,+BACC;IAAU;;EACb,SAAS,UAAU,mBAAmB,QAAQ;EAC9C,eAAe,gBAAgB;EAC/B,YAAY,iBAAiB;EAC7B,YAAY,eAAe,OAAO,iBAAiB,QAAQ;EAC3D,oBACE,eACA,gBACA,wBACA,OAAO,gBACP,cAAc,kBACb;EACH,uBACE,eACA,OAAO,gBACP,aAAa,kBACZ;EACH,mBAAmB,SAAS,4BAA4B,QAAQ;EAChE,UAAU,SAAS,0BAA0B,QAAQ;;;;;KC1H3C,iBAAiB;UAEZ;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA;;;;UCpBe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa,QAAQ;EAAkB,WAAW;;iBAE9C,iBAAiB,SAAS,mBAAmB"}

package/dist/index.js

@@ -1,15 +1,17 @@
-import { S as normalizeStreamError, a as stringify, b as encodeStreamWriteBatch, c as MAX_STREAM_BATCH_BYTES, d as MAX_STREAM_LONG_POLL_MS, f as MAX_STREAM_READ_BYTES, g as decodeStreamReadResult, h as STREAM_BATCH_CONTENT_TYPE, i as parse, l as MAX_STREAM_CHUNK_BYTES, m as MAX_STREAM_WRITE_CHUNKS, n as rpcParse, o as createWorkflowIndex, p as MAX_STREAM_READ_CHUNKS, r as rpcStringify, t as compact, u as MAX_STREAM_ERROR_BYTES, v as decodeStreamWriteResult } from "./util-DmO0XkFw.js";
+import { C as decodeStreamReadResult, D as encodeStreamWriteBatch, S as STREAM_BATCH_CONTENT_TYPE, T as decodeStreamWriteResult, _ as MAX_STREAM_ERROR_BYTES, a as stringify, b as MAX_STREAM_READ_CHUNKS, g as MAX_STREAM_CHUNK_BYTES, h as MAX_STREAM_BATCH_BYTES, i as parse, k as normalizeStreamError, n as rpcParse, o as createWorkflowIndex, r as rpcStringify, t as compact, u as MAX_FLEET_RPC_TIMEOUT_MS, v as MAX_STREAM_LONG_POLL_MS, x as MAX_STREAM_WRITE_CHUNKS, y as MAX_STREAM_READ_BYTES } from "./util-ya3V_FTA.js";
 import { EventSchema, HookSchema, MessageId, SPEC_VERSION_CURRENT, StepSchema, WorkflowRunSchema, isTerminalWorkflowRunStatus, parseQueueName } from "@workflow/world";
 import { EntityConflictError, HookNotFoundError, RunExpiredError, RunNotSupportedError, TooEarlyError, WorkflowRunNotFoundError, WorkflowWorldError } from "@workflow/errors";
 import { setTimeout as setTimeout$1 } from "node:timers/promises";
 import { monotonicFactory } from "ulid";
 //#region src/config.ts
 function resolveConfig(config) {
+	const queueShards = config?.queueShards ?? 1;
+	if (!Number.isSafeInteger(queueShards) || queueShards < 1) throw new Error("world-celld: queueShards must be a positive safe integer");
 	const retentionRaw = config?.runRetentionMs ?? process.env.CELLD_RUN_RETENTION_MS ?? 0;
 	const runRetentionMs = typeof retentionRaw === "number" ? retentionRaw : Number.parseInt(retentionRaw, 10);
 	if (!Number.isSafeInteger(runRetentionMs) || runRetentionMs < 0) throw new Error("world-celld: runRetentionMs must be a non-negative safe integer");
 	const rpcTimeoutMs = config?.rpcTimeoutMs ?? 3e4;
-	if (!Number.isSafeInteger(rpcTimeoutMs) || rpcTimeoutMs < 1) throw new Error("world-celld: rpcTimeoutMs must be a positive safe integer");
+	if (!Number.isSafeInteger(rpcTimeoutMs) || rpcTimeoutMs < 1 || rpcTimeoutMs > 3e5) throw new Error(`world-celld: rpcTimeoutMs must be between 1 and ${MAX_FLEET_RPC_TIMEOUT_MS}`);
 	const streamLongPollMs = config?.streamLongPollMs ?? Math.min(2e4, Math.max(1, rpcTimeoutMs - 1e3));
 	if (!Number.isSafeInteger(streamLongPollMs) || streamLongPollMs < 1 || streamLongPollMs > 2e4) throw new Error(`world-celld: streamLongPollMs must be between 1 and ${MAX_STREAM_LONG_POLL_MS}`);
 	if (streamLongPollMs >= rpcTimeoutMs) throw new Error("world-celld: streamLongPollMs must be less than rpcTimeoutMs");
@@ -21,7 +23,7 @@ function resolveConfig(config) {
 		env: config?.env,
 		deploymentId: config?.deploymentId ?? process.env.CELLD_DEPLOYMENT_ID ?? "celld-default",
 		baseUrl: config?.baseUrl,
-		queueShards: config?.queueShards ?? 1,
+		queueShards,
 		runRetentionMs,
 		streamLongPollMs,
 		streamFlushIntervalMs,
@@ -79,6 +81,8 @@ async function callFleetRoute(transport, path, args, opts) {
 	const doFetch = transport.fetchImpl ?? fetch;
 	const url = `${transport.fleetUrl.replace(/\/$/, "")}${path}`;
 	const attempts = opts?.idempotent ? 3 : 1;
+	const timeoutMs = transport.timeoutMs ?? DEFAULT_TIMEOUT_MS$1;
+	if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 1 || timeoutMs > 3e5) throw new Error(`world-celld: fleet RPC timeout must be between 1 and ${MAX_FLEET_RPC_TIMEOUT_MS}`);
 	let lastError;
 	for (let attempt = 1; attempt <= attempts; attempt++) {
 		if (attempt > 1) await delayMs$1(100 * attempt + Math.random() * 200);
@@ -91,7 +95,7 @@ async function callFleetRoute(transport, path, args, opts) {
 					authorization: `Bearer ${transport.secret}`
 				},
 				body: rpcStringify(args),
-				signal: AbortSignal.timeout(Math.max(1, transport.timeoutMs ?? DEFAULT_TIMEOUT_MS$1))
+				signal: AbortSignal.timeout(timeoutMs)
 			});
 		} catch (error) {
 			lastError = new FleetTransportError(`world-celld: fleet unreachable at ${url}`, error);
@@ -258,6 +262,7 @@ async function readStreamChunks(transport, name, request, signal) {
 const RUNS = {
 	methods: [
 		"applyEvent",
+		"getLifecycleStatus",
 		"getRun",
 		"getStep",
 		"getEvent",
@@ -318,14 +323,16 @@ const QUEUE = {
 		"redriveDeadLetter",
 		"purgeDeadLetters",
 		"rearmAlarm",
-		"expireRun"
+		"expireRun",
+		"acknowledgeExpireRun"
 	],
 	mutating: /* @__PURE__ */ new Set([
 		"enqueue",
 		"redriveDeadLetter",
 		"purgeDeadLetters",
 		"rearmAlarm",
-		"expireRun"
+		"expireRun",
+		"acknowledgeExpireRun"
 	])
 };
 function makeStub(transport, binding, name, spec) {
@@ -351,14 +358,6 @@ const RUN_CATALOG = {
 	],
 	mutating: /* @__PURE__ */ new Set()
 };
-const RUN_FENCES = {
-	methods: [
-		"getStatus",
-		"fenceTerminal",
-		"fenceExpired"
-	],
-	mutating: /* @__PURE__ */ new Set()
-};
 const HOOK_TOKENS = {
 	methods: [
 		"get",
@@ -381,12 +380,10 @@ const HOOK_IDS = {
 };
 function createRemoteEnv(transport) {
 	const runCatalog = makeNamespace(transport, "run-catalog", RUN_CATALOG);
-	const runFences = makeNamespace(transport, "run-fences", RUN_FENCES);
 	const hookTokens = makeNamespace(transport, "hook-tokens", HOOK_TOKENS);
 	const hookIds = makeNamespace(transport, "hook-ids", HOOK_IDS);
 	const workflowIndex = createWorkflowIndex({
 		runCatalog,
-		runFences,
 		hookTokens,
 		hookIds
 	});
@@ -395,7 +392,11 @@ function createRemoteEnv(transport) {
 		WORKFLOW_STREAMS: makeStreamNamespace(transport),
 		WORKFLOW_INDEX: {
 			...workflowIndex,
-			commitRun: (run, serializedMetadata) => callFleetRoute(transport, "/v1/index/runs/commit", [run, serializedMetadata], { idempotent: true }),
+			commitRun: (run, serializedMetadata, publicationExpiresAt) => callFleetRoute(transport, "/v1/index/runs/commit", [
+				run,
+				serializedMetadata,
+				publicationExpiresAt
+			], { idempotent: true }),
 			listRuns: (options) => callFleetRoute(transport, "/v1/index/runs/list", [options], { idempotent: true }),
 			expireRun: (request) => callFleetRoute(transport, "/v1/index/runs/expire", [request], { idempotent: true }),
 			reserveHook: (token, owner) => callFleetRoute(transport, "/v1/index/hooks/reserve", [token, owner], { idempotent: true }),
@@ -940,17 +941,16 @@ function createStorage(config) {
 						createdAt: outcome.run.createdAt.toISOString(),
 						status: outcome.run.status
 					});
-					await env.WORKFLOW_INDEX.commitRun(outcome.run, meta);
+					if (outcome.indexPublicationExpiresAt === void 0) throw new Error("world-celld: authoritative run mutation omitted its index lease");
+					await env.WORKFLOW_INDEX.commitRun(outcome.run, meta, outcome.indexPublicationExpiresAt);
 				}
 				if (outcome.hookToIndex) {
 					const serialized = stringify(outcome.hookToIndex);
 					await env.WORKFLOW_INDEX.finalizeHookIndexes(outcome.hookToIndex.token, outcome.hookToIndex.hookId, serialized, hookOwner(outcome.hookToIndex), hookAdmission?.reservation);
 				} else if (hookAdmission) await env.WORKFLOW_INDEX.releaseHookReservation(hookAdmission.token, hookAdmission.owner, hookAdmission.reservation);
-				const terminal = Boolean(outcome.run && isTerminalWorkflowRunStatus(outcome.run.status));
 				if (outcome.releasedHooks.length > 0) await env.WORKFLOW_INDEX.releaseHookIndexes({
 					runId: effectiveRunId,
-					hooks: outcome.releasedHooks,
-					terminal
+					hooks: outcome.releasedHooks
 				});
 				const eventPage = outcome.events === void 0 ? {} : {
 					events: outcome.events,