@ewhauser/world-celld 0.3.0 → 0.3.2

package/CHANGELOG.md

@@ -2,6 +2,29 @@
 
 All notable changes to this project will be documented in this file.
 
+## 0.3.2 (2026-08-19)
+
+## What's Changed
+* fix: compact expired-run lifecycle fences by @ewhauser in https://github.com/ewhauser/world-celld/pull/29
+
+
+**Full Changelog**: https://github.com/ewhauser/world-celld/compare/v0.3.1...v0.3.2
+
+## 0.3.1 (2026-08-19)
+
+## What's Changed
+* perf: reduce runs.list RPC fanout by @ewhauser in https://github.com/ewhauser/world-celld/pull/21
+* perf: remove unused correlation index by @ewhauser in https://github.com/ewhauser/world-celld/pull/23
+* perf: bound QueueDO inflight deadline work by @ewhauser in https://github.com/ewhauser/world-celld/pull/24
+* chore(deps-dev): bump @types/node from 22.20.1 to 26.1.2 by @dependabot[bot] in https://github.com/ewhauser/world-celld/pull/4
+* chore(deps-dev): bump the development-dependencies group across 1 directory with 2 updates by @dependabot[bot] in https://github.com/ewhauser/world-celld/pull/25
+* perf: batch and long-poll stream I/O by @ewhauser in https://github.com/ewhauser/world-celld/pull/26
+* perf: bound cleanup and batch Durable Object storage by @ewhauser in https://github.com/ewhauser/world-celld/pull/27
+* perf: shard workflow indexes by @ewhauser in https://github.com/ewhauser/world-celld/pull/28
+
+
+**Full Changelog**: https://github.com/ewhauser/world-celld/compare/v0.3.0...v0.3.1
+
 ## 0.3.0 (2026-08-16)
 
 ## What's Changed

package/README.md

@@ -115,12 +115,14 @@ More deployment detail is in [`celld-worker/README.md`](./celld-worker/README.md
 ```text
 Workflow application
   |
-  | authenticated HTTP RPC
+  | authenticated HTTP RPC + binary stream batches
   v
 celld worker router
   |-- WorkflowRunDO  one cell per workflow run
+  |-- RunCatalogDO   16 stable run-id shards; merged ordered listing
+  |-- HookTokenDO    32 stable token-ownership shards
+  |-- HookIdDO       32 stable hook-id lookup shards
   |-- StreamDO       stream chunks and run/stream indexes
-  |-- IndexDO        run and hook lookup indexes
   `-- QueueDO        delayed delivery, retries, and dead letters
           |
           | HTTP callbacks
@@ -129,25 +131,46 @@ Workflow application /.well-known/workflow/v1/flow
 ```
 
 The application-side package implements the Workflow `World` interface and
-translates its storage, stream, and queue operations into RPC calls. The celld
-worker accepts only a fixed set of methods and routes each request to the named
-cell that owns the data.
+translates its storage and queue operations into JSON RPC calls. Stream chunk
+writes and bounded long-poll reads use a binary batch route; stream control and
+retention operations remain fixed, authenticated RPC methods. The worker routes
+each request to the named cell that owns the data.
+
+Index routing is a versioned hard-cutover protocol. Run catalog writes hash the
+`runId` across 16 shards, while hook ownership and hook-ID lookup hash their
+natural keys across 32 shards each. A single stateless worker request queries
+all 16 catalog shards in parallel and merges their lexicographic pages before
+authoritative RunDO reads. The RunDO is also the only permanent lifecycle
+authority: hook shards consult it directly, while queue and catalog expiry
+fences compact after bounded protocol leases. Token and hook-ID admission are
+each transactional inside their natural-key shards. Exact claim IDs plus a
+serialized cancellation fence resolve ambiguous delivery without pretending
+the two ownership shards form a cross-Durable-Object transaction. Stateless cohesive index routes keep
+catalog listing, terminal commits, hook finalization, and hook release to one
+public request each while exposing the internal shard work in measurements.
+
+The routing rationale and before/after RPC, storage, latency, and contention
+evidence are recorded in [`docs/index-sharding.md`](./docs/index-sharding.md).
+The expiry authority, producer/consumer map, bounded horizons, and
+post-compaction costs are recorded in
+[`docs/lifecycle-compaction.md`](./docs/lifecycle-compaction.md).
 
 ## Configuration
 
 Application options can be passed to `createCelldWorld()` unless an environment
 variable is shown below.
 
-| Option           | Environment variable     | Default                  |
-| ---------------- | ------------------------ | ------------------------ |
-| `fleetUrl`       | `CELLD_FLEET_URL`        | required                 |
-| `secret`         | `CELLD_WORLD_SECRET`     | required with `fleetUrl` |
-| `baseUrl`        | `WORKFLOW_BASE_URL`      | `http://localhost:$PORT` |
-| `deploymentId`   | `CELLD_DEPLOYMENT_ID`    | `celld-default`          |
-| `queueShards`    | —                        | `1`                      |
-| `runRetentionMs` | `CELLD_RUN_RETENTION_MS` | `0` (disabled)           |
-| `readPollMs`     | —                        | `250`                    |
-| `rpcTimeoutMs`   | —                        | `30000`                  |
+| Option                  | Environment variable     | Default                  |
+| ----------------------- | ------------------------ | ------------------------ |
+| `fleetUrl`              | `CELLD_FLEET_URL`        | required                 |
+| `secret`                | `CELLD_WORLD_SECRET`     | required with `fleetUrl` |
+| `baseUrl`               | `WORKFLOW_BASE_URL`      | `http://localhost:$PORT` |
+| `deploymentId`          | `CELLD_DEPLOYMENT_ID`    | `celld-default`          |
+| `queueShards`           | —                        | `1`                      |
+| `runRetentionMs`        | `CELLD_RUN_RETENTION_MS` | `0` (disabled)           |
+| `streamLongPollMs`      | —                        | `20000`                  |
+| `streamFlushIntervalMs` | —                        | `0`                      |
+| `rpcTimeoutMs`          | —                        | `30000` (max `300000`)   |
 
 The deployed worker also accepts these celld variables:
 
@@ -156,8 +179,8 @@ The deployed worker also accepts these celld variables:
 | `WORLD_SECRET`              | none     | Required bearer secret for RPC routes                |
 | `WORKFLOW_CALLBACK_SECRET`  | none     | Sent with deliveries as `x-workflow-callback-secret` |
 | `QUEUE_MAX_ATTEMPTS`        | `5`      | Attempts before a message is dead-lettered           |
-| `QUEUE_MAX_INFLIGHT`        | `5`      | Concurrent deliveries per queue cell                 |
-| `QUEUE_DELIVERY_TIMEOUT_MS` | `300000` | Timeout for an application callback                  |
+| `QUEUE_MAX_INFLIGHT`        | `5`      | Concurrent deliveries per queue cell (maximum `128`) |
+| `QUEUE_DELIVERY_TIMEOUT_MS` | `300000` | Callback timeout (maximum `300000`)                  |
 
 `queueShards` is part of queue placement and is pinned when a queue cell is
 first used. Drain pending work before changing it.
@@ -171,8 +194,13 @@ pending runs are never eligible for automatic cleanup.
 
 At expiration, the run cell fences new writes and removes the run's derived
 indexes, stream chunks, pending and dead-letter queue messages, and durable run
-payloads. Cleanup is a persisted, idempotent state machine: an interrupted
-phase records its error and re-arms itself with capped backoff.
+payloads. Cleanup is a persisted, idempotent state machine. Each alarm processes
+one bounded page, persists its progress, and re-arms the next alarm; an
+interrupted phase records its error and retries with capped backoff.
+
+Terminal hook and wait disposal uses the same alarm mechanism even when run
+retention is disabled, so a terminal event never performs an unbounded storage
+transaction.
 
 The final state is a metadata-only tombstone, not an empty cell. It prevents a
 delayed queue delivery or stale RPC from recreating an expired run. Reads and
@@ -199,6 +227,8 @@ an existing terminal run that has no cleanup record.
   must be idempotent.
 - Queue cells expose `stats`, `listDeadLetters`, `redriveDeadLetter`,
   `purgeDeadLetters`, and `rearmAlarm` through the authenticated RPC endpoint.
+  `purgeDeadLetters` deletes at most 128 entries per call; repeat it while its
+  result reports `hasMore`.
 - Run cells expose retention status, scheduling, immediate cleanup, and alarm
   recovery through `world.retention`.
 - A new application URL applies to newly enqueued messages. Existing messages

package/celld-worker/README.md

@@ -1,8 +1,10 @@
 # world-celld worker
 
-The celld-deployable half of `@ewhauser/world-celld`: four cell classes
-(WorkflowRunDO, StreamDO, IndexDO, QueueDO) behind an authenticated HTTP RPC
-router.
+The celld-deployable half of `@ewhauser/world-celld`: six cell classes
+(WorkflowRunDO, RunCatalogDO, HookTokenDO, HookIdDO, StreamDO, and QueueDO)
+behind an authenticated HTTP router.
+Storage, control, and queue methods use fixed JSON RPC routes; stream chunks use
+bounded binary batch writes and binary long-poll reads.
 
 ## Deploy
 

package/celld-worker/worker.ts

@@ -2,4 +2,12 @@
 // project directory; this one-liner resolves the real worker through the
 // application's node_modules so the deployed bundle stays in step with the
 // installed @ewhauser/world-celld version.
-export { IndexDO, QueueDO, StreamDO, WorkflowRunDO, default } from '@ewhauser/world-celld/worker';
+export {
+  HookIdDO,
+  HookTokenDO,
+  QueueDO,
+  RunCatalogDO,
+  StreamDO,
+  WorkflowRunDO,
+  default,
+} from '@ewhauser/world-celld/worker';

package/celld-worker/wrangler.jsonc

@@ -9,19 +9,28 @@
     "bindings": [
       { "name": "WORKFLOW_DB", "class_name": "WorkflowRunDO" },
       { "name": "WORKFLOW_STREAMS", "class_name": "StreamDO" },
-      { "name": "WORKFLOW_INDEX", "class_name": "IndexDO" },
+      { "name": "WORKFLOW_RUN_CATALOG", "class_name": "RunCatalogDO" },
+      { "name": "WORKFLOW_HOOK_TOKENS", "class_name": "HookTokenDO" },
+      { "name": "WORKFLOW_HOOK_IDS", "class_name": "HookIdDO" },
       { "name": "WORKFLOW_QUEUE", "class_name": "QueueDO" },
     ],
   },
   "migrations": [
     {
       "tag": "v1",
-      "new_sqlite_classes": ["WorkflowRunDO", "StreamDO", "IndexDO", "QueueDO"],
+      "new_sqlite_classes": [
+        "WorkflowRunDO",
+        "StreamDO",
+        "RunCatalogDO",
+        "HookTokenDO",
+        "HookIdDO",
+        "QueueDO",
+      ],
     },
   ],
   "vars": {
-    // REQUIRED: bearer secret for the world's RPC surface. All /v1/rpc
-    // routes fail closed with 503 while this is empty. Inject the real value
+    // REQUIRED: bearer secret for the world's RPC and binary stream surface.
+    // Authenticated routes fail closed with 503 while this is empty. Inject the real value
     // at the node level with CELLD_VAR_WORLD_SECRET instead of committing it.
     "WORLD_SECRET": "",
     // Optional: forwarded as x-workflow-callback-secret on queue deliveries.

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { CreateEventParams, CreateEventRequest, Event, Hook, RunCreatedEventRequest, Step, TerminalWorkflowRunStatus, ValidQueueName, Wait, WorkflowRun, World } from "@workflow/world";
+import { CreateEventParams, CreateEventRequest, Event, GetChunksOptions, Hook, RunCreatedEventRequest, Step, StreamChunksResponse, StreamInfoResponse, Streamer, TerminalWorkflowRunStatus, ValidQueueName, Wait, WorkflowRun, World } from "@workflow/world";
 //#region src/retention.d.ts
 type CleanupPhase = 'retained' | 'index' | 'streams' | 'queues' | 'payload' | 'tombstoned';
 interface CleanupRecord {
@@ -11,6 +11,8 @@ interface CleanupRecord {
   dueAt: Date;
   queueShards: number;
   phase: CleanupPhase;
+  /** Optimistic-concurrency token for alarm/RPC work that awaits another cell. */
+  generation: number;
   attempts: number;
   lastError?: string;
   lastAttemptAt?: Date;
@@ -26,7 +28,10 @@ interface RunTombstone {
   completedAt: Date;
   expiredAt: Date;
   tombstonedAt: Date;
+  /** Final cleanup accounting retained inside the single authoritative record. */
+  cleanup: CleanupRecord;
 }
+type RunLifecycleStatus = 'active' | 'terminal' | 'expired' | 'missing';
 type RunReadOutcome<T> = {
   ok: true;
   value: T;
@@ -40,6 +45,13 @@ interface HookIndexReference {
   hookId: string;
   token: string;
 }
+interface ReleaseHookIndexesRequest {
+  runId: string;
+  hooks: HookIndexReference[];
+}
+interface ReleaseHookIndexesResult {
+  deleted: number;
+}
 interface ExpireRunIndexesRequest {
   runId: string;
   keys: string[];
@@ -52,12 +64,36 @@ interface ExpireRunIndexesResult {
 interface ExpireRunStreamsResult {
   streams: string[];
 }
+interface FinalizeRunStreamsResult {
+  /** Cumulative registry entries removed for this run. */
+  deleted: number;
+  done: boolean;
+}
 interface ExpireStreamResult {
+  /** True only when this call installed the stream's expiration fence. */
   deleted: boolean;
+  /** Chunk rows removed by this bounded call. */
   chunks: number;
+  /** Payload bytes represented by the removed chunk rows. */
+  bytes: number;
+  done: boolean;
+}
+interface QueueExpiryReceipt {
+  expiredAt: number;
+  deleted: number;
 }
-interface ExpireQueueRunResult {
+type ExpireQueueRunResult = {
+  /** Cumulative messages removed for this run in this queue shard. */
   deleted: number;
+} & ({
+  done: false;
+} | {
+  done: true;
+  /** Durable receipt which the cleanup coordinator must acknowledge. */
+  receipt: QueueExpiryReceipt;
+});
+interface AcknowledgeQueueExpiryResult {
+  acknowledged: boolean;
 }
 interface ScheduleCleanupRequest {
   retentionMs: number;
@@ -72,17 +108,19 @@ interface ApplyEventRequest {
   /** Workflow 5 event-log position and preload options. */
   params?: CreateEventParams;
   /**
-   * Current holder of the hook token per the global KV token index
+   * Current holder of the hook token per the sharded token-ownership index
    * (hook_created only). `null` means the token is unclaimed.
    */
   tokenHolder?: {
     runId: string;
     hookId: string;
   } | null;
+  /** Exact token/ID reservation fenced on ambiguous-delivery resolution. */
+  hookClaimId?: string;
   /** Internal world-celld retention policy captured with the event. */
   cleanup?: ScheduleCleanupRequest;
 }
-type ApplyEventErrorCode = 'RUN_NOT_FOUND' | 'STEP_NOT_FOUND' | 'HOOK_NOT_FOUND' | 'WAIT_NOT_FOUND' | 'ENTITY_CONFLICT' | 'RUN_EXPIRED' | 'TOO_EARLY' | 'RUN_NOT_SUPPORTED';
+type ApplyEventErrorCode = 'RUN_NOT_FOUND' | 'STEP_NOT_FOUND' | 'HOOK_NOT_FOUND' | 'WAIT_NOT_FOUND' | 'ENTITY_CONFLICT' | 'HOOK_CLAIM_CANCELLED' | 'RUN_EXPIRED' | 'TOO_EARLY' | 'RUN_NOT_SUPPORTED';
 interface ApplyEventFailure {
   ok: false;
   code: ApplyEventErrorCode;
@@ -102,7 +140,7 @@ interface ApplyEventSuccess {
   wait?: Wait;
   /** True when a lazy step_started atomically created its step. */
   stepCreated?: true;
-  /** Hooks whose tokens must be released from the global KV index. */
+  /** Hooks whose token and ID indexes must be released. */
   releasedHooks: Array<{
     hookId: string;
     token: string;
@@ -112,7 +150,7 @@ interface ApplyEventSuccess {
     workflowName: string;
     createdAt: Date;
   };
-  /** Set when the global KV hook index must be (re)written. */
+  /** Set when the sharded hook indexes must be (re)written. */
   hookToIndex?: Hook;
   /** All events (ascending), preloaded for run_started responses. */
   events?: Event[];
@@ -121,6 +159,8 @@ interface ApplyEventSuccess {
   hasMore?: boolean;
   /** Server-owned event ceiling advertised on run lifecycle writes. */
   maxEvents?: number;
+  /** Deadline for publishing this authoritative mutation to derivative indexes. */
+  indexPublicationExpiresAt?: number;
 }
 type ApplyEventOutcome = ApplyEventSuccess | ApplyEventFailure;
 //#endregion
@@ -131,6 +171,14 @@ type ApplyEventOutcome = ApplyEventSuccess | ApplyEventFailure;
  */
 interface WorkflowRunDOStub {
   applyEvent(request: ApplyEventRequest): Promise<ApplyEventOutcome>;
+  resolveHookTokenClaim(request: {
+    hookId: string;
+    token: string;
+    claimId: string;
+  }): Promise<{
+    committed: boolean;
+  }>;
+  getLifecycleStatus(): Promise<RunLifecycleStatus>;
   getRun(): Promise<RunReadOutcome<WorkflowRun | null>>;
   getStep(stepId: string): Promise<RunReadOutcome<Step | null>>;
   getEvent(eventId: string): Promise<RunReadOutcome<Event | null>>;
@@ -174,32 +222,74 @@ interface DurableObjectIdLike {
   toString(): string;
 }
 //#endregion
+//#region src/stream-protocol.d.ts
+/** Metadata plus payload and size-index entries remain below the 128-key platform limit. */
+declare const MAX_STREAM_WRITE_CHUNKS = 32;
+/** A chunk remains below the 2 MiB SQLite-backed DO key/value limit. */
+declare const MAX_STREAM_CHUNK_BYTES: number;
+/** Keeps serialized RPC bodies well below the 32 MiB Workers RPC limit. */
+declare const MAX_STREAM_BATCH_BYTES: number;
+declare const MAX_STREAM_READ_CHUNKS = 32;
+declare const MAX_STREAM_READ_BYTES: number;
+declare const MAX_STREAM_ERROR_BYTES: number;
+/** Must remain below the default 30 second fleet request deadline. */
+declare const MAX_STREAM_LONG_POLL_MS = 20000;
+type StreamTerminalState = 'open' | 'closed' | 'errored' | 'expired';
+interface StreamErrorData {
+  name: string;
+  message: string;
+}
+interface StreamWriteResult {
+  startIndex: number;
+  count: number;
+  tailIndex: number;
+}
+interface StreamReadRequest {
+  runId: string;
+  startIndex: number;
+  maxChunks: number;
+  maxBytes: number;
+  waitMs: number;
+}
+interface StreamReadResult {
+  startIndex: number;
+  tailIndex: number;
+  chunks: Uint8Array[];
+  state: StreamTerminalState;
+  timedOut: boolean;
+  error?: StreamErrorData;
+}
+//#endregion
 //#region src/streamer.d.ts
-/**
- * RPC surface of StreamDO (see durable-objects/StreamDO.ts). The streamer
- * talks to the DO exclusively via these methods — there is no fetch()
- * protocol.
- */
+interface CelldStreamer extends Omit<Streamer, 'streams'> {
+  streams: Streamer['streams'] & {
+    fail(runId: string, name: string, error: StreamErrorData | string): Promise<void>;
+  };
+  writeToStream(name: string, runId: string | Promise<string>, chunk: string | Uint8Array): Promise<void>;
+  writeChunksToStream(name: string, runId: string | Promise<string>, chunks: readonly (string | Uint8Array)[]): Promise<void>;
+  closeStream(name: string, runId: string | Promise<string>): Promise<void>;
+  errorStream(name: string, runId: string | Promise<string>, error: StreamErrorData | string): Promise<void>;
+  readFromStream(name: string, runId: string, startIndex?: number): Promise<ReadableStream<Uint8Array>>;
+  listStreamsByRunId(runId: string): Promise<string[]>;
+  getStreamChunks(name: string, runId: string, options?: GetChunksOptions): Promise<StreamChunksResponse>;
+  getStreamInfo(name: string, runId: string): Promise<StreamInfoResponse>;
+}
+/** Direct StreamDO RPC surface. Remote stubs specialize read/write as binary HTTP. */
 interface StreamDOStub {
-  writeChunk(runId: string, data: Uint8Array): Promise<number>;
+  writeChunks(runId: string, chunks: Uint8Array[]): Promise<StreamWriteResult>;
+  readChunks(request: StreamReadRequest, signal?: AbortSignal): Promise<StreamReadResult>;
   closeStream(runId: string): Promise<void>;
-  getChunks(params: {
-    startIndex: number;
-    limit: number;
-  }): Promise<{
-    chunks: Uint8Array[];
-    done: boolean;
-    tailIndex: number;
-  }>;
-  getInfo(): Promise<{
-    tailIndex: number;
-    done: boolean;
-  }>;
+  failStream(runId: string, error: StreamErrorData | string): Promise<void>;
   registerStream(runId: string, name: string): Promise<void>;
   listStreams(): Promise<string[]>;
-  expireRegistry(runId: string, expiredAt: number): Promise<ExpireRunStreamsResult>;
-  finalizeRegistry(runId: string): Promise<void>;
-  expireStream(runId: string, expiredAt: number): Promise<ExpireStreamResult>;
+  expireRegistry(runId: string, expiredAt: number, options?: {
+    limit?: number;
+  }): Promise<ExpireRunStreamsResult>;
+  finalizeRegistry(runId: string, streams: string[]): Promise<FinalizeRunStreamsResult>;
+  expireStream(runId: string, expiredAt: number, options?: {
+    limit?: number;
+    byteLimit?: number;
+  }): Promise<ExpireStreamResult>;
 }
 interface StreamDONamespace {
   idFromName(name: string): StreamDOId;
@@ -249,7 +339,10 @@ type EnqueueOutcome = {
 /** RPC surface of QueueDO used by the queue producer. */
 interface QueueCellStub {
   enqueue(request: EnqueueRequest): Promise<EnqueueOutcome>;
-  expireRun(runId: string, expiredAt: number): Promise<ExpireQueueRunResult>;
+  expireRun(runId: string, expiredAt: number, options?: {
+    limit?: number;
+  }): Promise<ExpireQueueRunResult>;
+  acknowledgeExpireRun(runId: string, receipt: QueueExpiryReceipt): Promise<AcknowledgeQueueExpiryResult>;
 }
 interface QueueCellNamespace {
   idFromName(name: string): {
@@ -260,36 +353,51 @@ interface QueueCellNamespace {
   }): QueueCellStub;
 }
 //#endregion
-//#region src/config.d.ts
-/** Storage-layer KV interface (satisfied by IndexDO over HTTP, or a mock). */
-interface IndexNamespace {
-  get(key: string): Promise<string | null>;
-  put(key: string, value: string): Promise<void>;
-  delete(key: string): Promise<void>;
-  putOwned(runId: string, key: string, value: string): Promise<{
+//#region src/indexes.d.ts
+interface IndexListOptions {
+  prefix?: string;
+  cursor?: string;
+  limit?: number;
+  reverse?: boolean;
+}
+interface IndexListPage {
+  keys: Array<{
+    name: string;
+    value: string;
+  }>;
+  list_complete: boolean;
+  cursor?: string;
+}
+interface HookReservation {
+  claimId: string;
+  tokenClaimId?: string;
+  hookIdClaimId?: string;
+}
+type HookReservationResult = {
+  admitted: true;
+  reservation: HookReservation;
+} | {
+  admitted: false;
+  holder: HookTokenOwner;
+};
+/** High-level client used by the World storage layer and retention coordinator. */
+interface WorkflowIndex {
+  commitRun(run: WorkflowRun, serializedMetadata: string, publicationExpiresAt: number): Promise<{
     stored: boolean;
   }>;
+  listRuns(options?: IndexListOptions): Promise<IndexListPage>;
+  getHookByToken(token: string): Promise<string | null>;
+  getHookById(hookId: string): Promise<string | null>;
+  reserveHook(token: string, owner: HookTokenOwner): Promise<HookReservationResult>;
+  finalizeHookIndexes(token: string, hookId: string, serializedHook: string, owner: HookTokenOwner, reservation?: HookReservation): Promise<void>;
+  releaseHookReservation(token: string, owner: HookTokenOwner, reservation: HookReservation): Promise<void>;
+  releaseHookIndexes(request: ReleaseHookIndexesRequest): Promise<ReleaseHookIndexesResult>;
   expireRun(request: ExpireRunIndexesRequest): Promise<ExpireRunIndexesResult>;
-  list(options?: {
-    prefix?: string;
-    cursor?: string;
-    limit?: number;
-    reverse?: boolean;
-  }): Promise<{
-    keys: Array<{
-      name: string;
-    }>;
-    list_complete: boolean;
-    cursor?: string;
-  }>;
-  reserveHookToken(token: string, owner: HookTokenOwner): Promise<{
-    claimed: boolean;
-    holder?: HookTokenOwner;
-  }>;
-  finalizeHookIndexes(token: string, hookId: string, serializedHook: string, owner: HookTokenOwner): Promise<void>;
-  releaseHookToken(token: string, owner: HookTokenOwner): Promise<void>;
-  deleteHookIndexes(token: string, hookId: string, owner: HookTokenOwner): Promise<void>;
 }
+//#endregion
+//#region src/config.d.ts
+/** Public alias retained for custom in-process environments. */
+type IndexNamespace = WorkflowIndex;
 interface HookTokenOwner {
   runId: string;
   hookId: string;
@@ -334,8 +442,10 @@ interface CelldWorldConfig {
    * Default: process.env.CELLD_RUN_RETENTION_MS || 0
    */
   runRetentionMs?: number;
-  /** Poll interval (ms) while waiting for new chunks on a live stream. Default: 250 */
-  readPollMs?: number;
+  /** Duration of one bounded idle stream read. Default: 20000 */
+  streamLongPollMs?: number;
+  /** Runtime stream batching delay. Default: 0 */
+  streamFlushIntervalMs?: number;
   /** Per-attempt fleet RPC deadline in milliseconds. Default: 30000 */
   rpcTimeoutMs?: number;
 }
@@ -347,10 +457,10 @@ interface CelldRetentionAdmin {
   cleanupNow(runId: string): Promise<CleanupRecord | null>;
   rearm(runId: string): Promise<CleanupRecord | null>;
 }
-type CelldWorld = World & {
+type CelldWorld = World & CelldStreamer & {
   retention: CelldRetentionAdmin;
 };
 declare function createCelldWorld(config?: CelldWorldConfig): CelldWorld;
 //#endregion
-export { CelldRetentionAdmin, CelldWorld, type CelldWorldConfig, type CelldWorldEnv, type CleanupPhase, type CleanupRecord, type EnqueueOutcome, type EnqueueRequest, type IndexNamespace, type QueueCellConfig, type QueueCellNamespace, type QueueCellStub, type RunTombstone, type StreamDONamespace, type StreamDOStub, type WorkflowRunDONamespace, type WorkflowRunDOStub, createCelldWorld, createCelldWorld as createWorld };
+export { CelldRetentionAdmin, CelldWorld, type CelldWorldConfig, type CelldWorldEnv, type CleanupPhase, type CleanupRecord, type EnqueueOutcome, type EnqueueRequest, type IndexNamespace, MAX_STREAM_BATCH_BYTES, MAX_STREAM_CHUNK_BYTES, MAX_STREAM_ERROR_BYTES, MAX_STREAM_LONG_POLL_MS, MAX_STREAM_READ_BYTES, MAX_STREAM_READ_CHUNKS, MAX_STREAM_WRITE_CHUNKS, type QueueCellConfig, type QueueCellNamespace, type QueueCellStub, type RunTombstone, type StreamDONamespace, type StreamDOStub, type StreamErrorData, type StreamReadRequest, type StreamReadResult, type StreamTerminalState, type StreamWriteResult, type WorkflowRunDONamespace, type WorkflowRunDOStub, createCelldWorld, createCelldWorld as createWorld };
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/streamer.ts","../src/queue.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;EACX,aAAa;EACb,gBAAgB;EAChB,OAAO;EACP;EACA,OAAO;EACP;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA,gBAAgB;EAChB,aAAa;EACb,WAAW;EACX,cAAc;;KAGJ,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAGe;EACf;EACA;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;EACf;EACA;;UAGe;EACf;;UAGe;EACf;EACA;;;;UCiCe;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B,UAAU;;KAGA;UAUK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UC9GnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;;;;;UC/Ce;EACf,WAAW,eAAe,MAAM,aAAa;EAC7C,YAAY,gBAAgB;EAC5B,UAAU;IACR;IACA;MACE;IAAU,QAAQ;IAAc;IAAe;;EACnD,WAAW;IAAU;IAAmB;;EACxC,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eAAe,eAAe,oBAAoB,QAAQ;EAC1D,iBAAiB,gBAAgB;EACjC,aAAa,eAAe,oBAAoB,QAAQ;;UAGzC;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KCVG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,oBACC,QAJ+B;;UAOnB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;;UCzGlB;EACf,IAAI,cAAc;EAClB,IAAI,aAAa,gBAAgB;EACjC,OAAO,cAAc;EACrB,SAAS,eAAe,aAAa,gBAAgB;IAAU;;EAC/D,UACE,SAFmD,0BAGlD,QADiC;EAEpC,KAAK;IAAY;IAAiB;IAAiB;IAAgB;MAAsB;IACvF,MAAM;MAAQ;;IACd;IACA;;EAEF,iBACE,eACA,OAAO,iBACN;IAAU;IAAkB,SAAS;;EACxC,oBACE,eACA,gBACA,wBACA,OAAO,iBACN;EACH,iBAAiB,eAAe,OAAO,iBAAiB;EACxD,kBAAkB,eAAe,gBAAgB,OAAO,iBAAiB;;UAG1D;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;;;UCxDe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa;EAAU,WAAW;;iBAE9B,iBAAiB,SAAS,mBAAmB"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/retention.ts","../src/apply-event.ts","../src/storage.ts","../src/stream-protocol.ts","../src/streamer.ts","../src/queue.ts","../src/indexes.ts","../src/config.ts","../src/index.ts"],"mappings":";;KAOY;UAEK;EACf;EACA;EACA;EACA,WAAW;EACX,aAAa;EACb,gBAAgB;EAChB,OAAO;EACP;EACA,OAAO;;EAEP;EACA;EACA;EACA,gBAAgB;EAChB,eAAe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA,gBAAgB;EAChB,aAAa;EACb,WAAW;EACX,cAAc;;EAEd,SAAS;;KAGC;KAEA,eAAe;EACrB;EAAU,OAAO;;EACjB;EAAW;EAAqB;EAAiB,WAAW;;UAEjD;EACf;EACA;;UAae;EACf;EACA,OAAO;;UAGQ;EACf;;UAGe;EACf;EACA;EACA,OAAO;EACP;;UAGe;EACf;;UAGe;EACf;;UAGe;;EAEf;EACA;;UAGe;;EAEf;;EAEA;;EAEA;EACA;;UAGe;EACf;EACA;;KAGU;;EAEV;;EAEI;;EAEA;;EAEA,SAAS;;UAIE;EACf;;UAGe;EACf;EACA;;;;UCyBe;;EAEf;EACA,MAAM,qBAAqB;;EAE3B,SAAS;;;;;EAKT;IAAgB;IAAe;;;EAE/B;;EAEA,UAAU;;KAGA;UAWK;EACf;EACA,MAAM;EACN;;EAEA;;EAEA;;UAGe;EACf;;EAEA,QAAQ;EACR,MAAM;EACN,OAAO;EACP,OAAO;EACP,OAAO;;EAEP;;EAEA,eAAe;IAAQ;IAAgB;;;EAEvC;IAAe;IAAsB,WAAW;;;EAEhD,cAAc;;EAEd,SAAS;;EAET;EACA;;EAEA;;EAEA;;KAGU,oBAAoB,oBAAoB;;;;;;;UCnKnC;EACf,WAAW,SAAS,oBAAoB,QAAQ;EAChD,sBAAsB;IACpB;IACA;IACA;MACE;IAAU;;EACd,sBAAsB,QADlB;EAEJ,UAAU,QAAQ,eAAe;EACjC,QAAQ,iBAAiB,QAAQ,eAAe;EAChD,SAAS,kBAAkB,QAAQ,eAAe;EAClD,WAAW;IACT;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAS;IAAuB;;EACnE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,UAAU;IACR;IACA;IACA;MACE,QAAQ;IAAiB,MAAM;IAAQ;IAAuB;;EAClE,oBAAoB,QADhB;EAEJ,gBACE,SAFkB,yBAGjB,QADiC;EAEpC,WACE,SAFC,yBAGA,QADiC;EAEpC,gBAAgB,QADb;;UAIY;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,sBAAsB;;UAGf;EACf;;;;;cCrFW;;cAEA;;cAEA;cACA;cACA;cACA;;cAEA;KAED;UAEK;EACf;EACA;;UAGe;EACf;EACA;EACA;;UAGe;EACf;EACA;EACA;EACA;EACA;;UAGe;EACf;EACA;EACA,QAAQ;EACR,OAAO;EACP;EACA,QAAQ;;;;UCXO,sBAAsB,KAAK;EAC1C,SAAS;IACP,KAAK,eAAe,cAAc,OAAO,2BAA2B;;EAEtE,cACE,cACA,gBAAgB,iBAChB,gBAAgB,aACf;EACH,oBACE,cACA,gBAAgB,iBAChB,2BAA2B,gBAC1B;EACH,YAAY,cAAc,gBAAgB,kBAAkB;EAC5D,YACE,cACA,gBAAgB,iBAChB,OAAO,2BACN;EACH,eACE,cACA,eACA,sBACC,QAAQ,eAAe;EAC1B,mBAAmB,gBAAgB;EACnC,gBACE,cACA,eACA,UAAU,mBACT,QAAQ;EACX,cAAc,cAAc,gBAAgB,QAAQ;;;UAIrC;EACf,YAAY,eAAe,QAAQ,eAAe,QAAQ;EAC1D,WAAW,SAAS,mBAAmB,SAAS,cAAc,QAAQ;EACtE,YAAY,gBAAgB;EAC5B,WAAW,eAAe,OAAO,2BAA2B;EAC5D,eAAe,eAAe,eAAe;EAC7C,eAAe;EACf,eACE,eACA,mBACA;IAAY;MACX,QAAQ;EACX,iBAAiB,eAAe,oBAAoB,QAAQ;EAC5D,aACE,eACA,mBACA;IAAY;IAAgB;MAC3B,QAAQ;;UAGI;EACf,WAAW,eAAe;EAC1B,IAAI,IAAI,aAAa;;UAGN;EACf;;;;KC1CG;;;;;;UAiBY;EACf;EACA,WAAW;EACX,UAAU;;EAEV;;EAEA;EACA;EACA;;;;;EAKA,QAAQ;;UAGO;;EAEf;EACA;;KAGU;EACN;EAAU;EAAmB;;EAC7B;EAAW;EAAyC;;;UAGzC;EACf,QAAQ,SAAS,iBAAiB,QAAQ;EAC1C,UACE,eACA,mBACA;IAAY;MACX,QAL+B;EAMlC,qBACE,eACA,SAHC,qBAIA,QADiC;;UAIrB;EACf,WAAW;IAAiB;;EAC5B,IAAI;IAAM;MAAuB;;;;UCnGlB;EACf;EACA;EACA;EACA;;UAGe;EACf,MAAM;IAAQ;IAAc;;EAC5B;EACA;;UAkBe;EACf;EACA;EACA;;KAGU;EACN;EAAgB,aAAa;;EAC7B;EAAiB,QAAQ;;;UAgDd;EACf,UACE,KAAK,aACL,4BACA,+BACC;IAAU;;EACb,SAAS,UAAU,mBAAmB,QAAQ;EAC9C,eAAe,gBAAgB;EAC/B,YAAY,iBAAiB;EAC7B,YAAY,eAAe,OAAO,iBAAiB,QAAQ;EAC3D,oBACE,eACA,gBACA,wBACA,OAAO,gBACP,cAAc,kBACb;EACH,uBACE,eACA,OAAO,gBACP,aAAa,kBACZ;EACH,mBAAmB,SAAS,4BAA4B,QAAQ;EAChE,UAAU,SAAS,0BAA0B,QAAQ;;;;;KCnH3C,iBAAiB;UAEZ;EACf;EACA;;UAGe;EACf,aAAa;EACb,gBAAgB;EAChB,gBAAgB;EAChB,kBAAkB;;UAGH;;;;;;;EAOf;;;;;EAKA;;;;;EAKA,MAAM;;EAEN;;;;;;EAMA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA;;;;UCpBe;EACf,UAAU,gBAAgB,QAAQ;EAClC,SAAS,gBAAgB,QAAQ;EACjC,WAAW,gBAAgB,QAAQ;EACnC,MAAM,gBAAgB,QAAQ;;KAGpB,aAAa,QAAQ;EAAkB,WAAW;;iBAE9C,iBAAiB,SAAS,mBAAmB"}