@pikku/core 0.12.129 → 0.12.133

package/CHANGELOG.md

@@ -1,3 +1,114 @@
+## 0.12.133
+
+### Patch Changes
+
+- a16b0ba: The workflow run lease takes the app's `leaseService` (`holdLease(leaseService, 'workflow-run:<id>', …)`) in place of `pg_advisory_lock` / `GET_LOCK`, for DSL and graph runs alike. A message for a run already being orchestrated elsewhere wakes the run again a second later instead of failing, so a long pass never spends the retry budget the queue keeps for real failures. An inline run whose lease was lost after its body finished keeps the outcome it wrote. `PgWorkflowQueueOptions` and `RunLockHoldTimeoutError` are removed. `WorkflowService.withRunLock` is renamed `withRunLease`, since what it holds runs out unless renewed; a custom workflow service overriding it renames the method.
+
+  Pass the lease service to the workflow service — `new PgKyselyWorkflowService(db, { leaseService })` — register the same instance as the app's `leaseService`, and migrate `pikku_lease`.
+
+  A step lease is renewed by the same loop as the run lease. A graph node whose worker died is dispatched again once its lease lapses, instead of leaving the run waiting on it. A replayed step on Kysely now sees its lease, as `getStepState` does. The in-memory store refuses a superseded worker's outcome, as the database stores do. The MySQL workflow service can now record a workflow version; it used `ON CONFLICT`, which MySQL does not have.
+
+  `pikku db generate` now writes the `pikku_lease` table for any project that runs workflows, not only one that reaches `leaseService`, since every persistent workflow service holds its runs and steps there.
+
+- 21e9c3a: An SSE response stream ignores a send or close after the stream has already closed or the client has gone, instead of throwing `Controller is already closed` and taking the process down.
+- a16b0ba: Give a running workflow step a lease, so a step whose worker dies can be claimed again
+
+  A step moving from `pending` to `running` only proves it was dispatched; it says
+  nothing about whether the worker holding it is still alive. A worker that died
+  mid-step therefore left the step wedged in `running` forever: unclaimable by the
+  next dispatch, and invisible to the stalled-run sweep.
+
+  The dispatch that claims a step now stamps a lease on it, refreshes that lease on
+  an interval for as long as it is working, and releases it when the step parks on
+  a child run. A step whose lease has lapsed is claimable again, and counts as
+  in-flight no longer, so the stalled sweep can pick its run back up. A step that
+  keeps losing its worker fails with `WorkflowStepLeaseExpiredError` once its
+  attempts run out, rather than looping.
+
+  The lease duration is taken from the queue the step is dispatched through, so the
+  lease and the queue lock never disagree about who owns the job.
+
+  The refresh runs at half the lease and never later, even when that is under the
+  one-second floor the interval otherwise respects: applying that floor as a
+  maximum would renew a shorter lease for the first time after it had already
+  lapsed, which is the double-claim the lease exists to prevent. A lease that
+  short is logged once, pointing at the queue's `lockDuration`.
+
+  The lease is written and judged on the database's clock, never the worker's,
+  so a worker whose clock runs fast cannot claim a step another worker still
+  holds. A dispatch whose step has since been claimed by another stops renewing,
+  and says so: `refreshStepLease(stepId, leaseMs, attempt)` resolves `false` once
+  the step is no longer that attempt's.
+
+  The lease lives in a new `workflow_step.lease_expires_at` column, in epoch
+  milliseconds (`bigint`). Run `pikku db generate` and `pikku db migrate` after
+  upgrading: the runtime does not add columns, and every step read selects this
+  one.
+
+- a16b0ba: Persistent workflow services take a required `leaseService` (`WorkflowServiceOptions`) and lock runs and steps on it. The Postgres advisory and MySQL `GET_LOCK` step locks, and Redis's own `SET NX` run and step locks, are removed; `RedisWorkflowService` now takes `(connection, { leaseService, keyPrefix? })`.
+
+## 0.12.132
+
+### Patch Changes
+
+- d327fa5: Two gaps from the August security sweep that never reached main:
+
+  - The memoized middleware chains are capped per wire type. The key is the requested wire id, so a caller varying RPC names could otherwise grow the cache without limit.
+  - The console's generated variable brokers require `pikku:console`. They are emitted into the application rather than the console addon, so the addon's scope never reached them and any signed-in user could read and overwrite variables through `/rpc`.
+
+- 942ebdd: Add `LeaseService`: named leases shared by every process on the same store. A lease is not a mutex: a holder that stalls past its expiry loses the key without knowing, so whatever the body writes to should check the lease's `token` and refuse an older one. `acquire` never blocks and answers `null` while someone else holds the key; a lease lapses on its own if its holder dies; and each lease carries a `token` that rises every time the key changes hands, so a stale holder cannot refresh or release its successor's lease. `holdLease` refreshes the lease while its body runs, hands the body an `AbortSignal` that fires the moment the lease is lost, and throws `LeaseLostError` once the body finishes if the lease lapsed or changed hands meanwhile, so a caller never takes that result as produced under the lease.
+
+  Ships as `InMemoryLeaseService`, `PgKyselyLeaseService` (`@pikku/kysely-postgres`), `MySQLKyselyLeaseService` (`@pikku/kysely-mysql`) and `KyselyLeaseService` (SQLite), all on `pikku_lease`. The Postgres and MySQL services judge every lease by the database's clock, so a worker whose clock runs fast cannot take a live lease or stretch its own; `KyselyLeaseService` reads the process clock and is for a single-host SQLite app. `pikku db generate` writes the table for any project that reaches `leaseService`.
+
+  `RedisLeaseService` (`@pikku/redis`) keeps each lease in a hash Redis expires itself, with every check-and-set in one Lua script that reads Redis's `TIME`, so it too judges leases by the server's clock; its token counter outlives release, so the next holder always gets a higher token.
+
+- 8ac25a8: An OAuth2 credential can set `scopeSeparator` for a provider that wants scopes joined by something other than a space (Twist wants a comma). A credential with no scopes already sent no `scope` parameter; that is now tested.
+- 6606777: A webhook source can declare how its requests are signed, and the runner checks every request before `receive` runs:
+
+  ```ts
+  wireTriggerWebhookSource({
+    name: 'github',
+    verify: {
+      hmac: {
+        header: 'x-hub-signature-256',
+        prefix: 'sha256=',
+        algorithm: 'sha256',
+        encoding: 'hex',
+      },
+    },
+    receive: githubWebhookReceive,
+  })
+  ```
+
+  `verify` is an HMAC over the raw body, a shared token or a public-key signature in one header, or a function `(request, secret, services) => boolean` for anything else. A request is refused while the secret is unset or when the signature does not match. A request without a body reaches `receive` unchecked so handshakes still work, but it may only be answered: events from it are refused.
+
+  Declaring `verify` declares the secret's credential too, so it needs no `defineCredential`. It is a singleton string named `<source>WebhookSecret` in camelCase (`microsoft-outlook` → `microsoftOutlookWebhookSecret`, see `webhookSecretCredentialName`), or whatever `credential` names, described by `credentialDescription`.
+
+  `@pikku/core/hmac` gains `hmacDigest`, `verifyHmacSignature` and `verifyPublicKeySignature`. `WebhookSigningSecret` is deprecated.
+
+## 0.12.131
+
+### Patch Changes
+
+- 58cccc1: `wire.getCredential('name')` is typed by the project's own credentials, and is always on the wire. Function types were written from the setup-only inspection, which never sees `defineCredential`, so every project fell back to an untyped map; `pikku all` now rewrites them once the credentials leaf exists. `getCredential` and `getCredentials` are no longer optional on `PikkuWire`, since the function runner always sets them. Without a credentials map, `getCredential<string>('name')` returns `string` rather than `unknown`.
+- a26c60e: The workflow status stream sends a `suspended` frame with the run's reason when a run suspends, and stays open for the resume. It used to send nothing to say that no more progress was coming without action.
+
+## 0.12.130
+
+### Patch Changes
+
+- dfcd351: The deploy planner now binds a unit to the units of the functions its functions call with `rpc.invoke('name')` / `rpc.remote('name')`. Before, a caller and callee split into different units (e.g. a no-service function in `svc-base` calling a DB function in `svc-kysely`) got no service binding and failed on the deployed stage with "No service binding for function". The inspector records literal RPC names per function as `invokes` (single, double or substitution-free template quotes; `rpc!`, `wire.rpc`) and warns on a computed name, which the planner cannot see (#1883).
+- cf40182: Deployed units that call `rpc.startWorkflow('x')` now get x's meta, so the call no longer fails with `WorkflowNotFoundError`. The inspector records `startsWorkflows` for literal `rpc.startWorkflow(...)` calls. It warns when a handler computes the workflow name or passes `rpc` to a helper, because the planner cannot see those calls.
+
+  With workflow queues, the deploy planner gives a starter unit the workflow meta and orchestrator queue meta only (new `--workflowMeta` filter), plus `workflow-state` and `queue` services. Without queues the start runs inline, so the whole workflow is bundled. Core's `startWorkflow` now requires the workflow registration only for inline runs; queued runs need only the meta.
+
+- 5bce779: `pikku dev` registers webhook sources with their providers when `PIKKU_DEV_WEBHOOK_URL` is set. What it registered, signing secrets included, is kept in a git-ignored `.webhook-registrations.gen.json` next to `pikku.config.json`, so a database reset or `.pikku` wipe does not register again, and nothing reaches a provider while the url and events are unchanged. Registrations no source declares any more are warned about on every run, with the endpoint and label to delete by hand. The prefix defaults to `dev-<username>`; override it with `PIKKU_DEV_WEBHOOK_LABEL_PREFIX`. New in core: `reconcileWebhookRegistrations`.
+- 5bce779: Webhook trigger sources are off until someone turns them on, and addons declare their own.
+
+  An addon calls `wireTriggerWebhookSource` in its own package, and an app that wires the addon gets the source (its route included) without declaring it: named and routed after the addon's namespace, so two instances get one each. A source the app declares under the same name wins.
+
+  Every source now has an `enabled` switch in the `triggerSourceStore`, off by default. `reconcileTriggerSources` registers only enabled sources and records each one's `baseUrl` and `labelPrefix`; a disabled source's route answers 404 without running `receive`. New in core: `enableTriggerSource` registers a source with its provider and `disableTriggerSource` stops it receiving, then tears it down, both at the recorded address unless one is given. The admin addon exposes them as `triggerSourceEnable` and `triggerSourceDisable`. The `pikkuTriggerSource` table gains `enabled`, `baseUrl` and `labelPrefix`: run `pikku db generate` for the migration.
+
 ## 0.12.129
 
 ### Patch Changes

package/dist/function/function-meta.types.d.ts

@@ -138,12 +138,26 @@ export type FunctionMeta = FunctionRuntimeMeta & Partial<{
     isDirectFunction: boolean;
     sourceFile: string;
     exportedName: string;
+    /**
+     * RPC names this function's own body calls by literal name through
+     * `rpc.invoke(...)` / `rpc.remote(...)`. The deploy planner binds the
+     * function's unit to each callee's unit from this; a call with a computed
+     * name is not listed, and is warned about instead.
+     */
+    invokes: string[];
     /** File containing the handler body when it differs from sourceFile (imported handlers) */
     bodySourceFile?: string;
     /** 1-indexed first line of the handler body (verbose meta; coverage mapping) */
     bodyStart: number;
     /** 1-indexed last line of the handler body (verbose meta; coverage mapping) */
     bodyEnd: number;
+    /**
+     * Workflow names this function's own body starts by literal name through
+     * `rpc.startWorkflow(...)`. The deploy planner bundles each one's meta
+     * into the function's unit, which is all a queued start needs. A call
+     * made from a helper the handler passes `rpc` to is not listed.
+     */
+    startsWorkflows: string[];
 } & CommonWireMeta>;
 export type FunctionsRuntimeMeta = Record<string, FunctionRuntimeMeta>;
 export type FunctionsMeta = Record<string, FunctionMeta>;

package/dist/middleware-runner.d.ts

@@ -12,6 +12,10 @@ export declare const addTagMiddleware: <PikkuMiddleware extends CorePikkuMiddlew
  */
 export declare const addGlobalMiddleware: <PikkuMiddleware extends CorePikkuMiddleware>(middleware: CorePikkuMiddlewareGroup, packageName?: string | null) => CorePikkuMiddlewareGroup;
 export declare const clearMiddlewareCache: () => void;
+/** The per-wire-type entry cap. Exposed so the boundedness invariant is testable. */
+export declare const MIDDLEWARE_CACHE_MAX_ENTRIES = 4096;
+/** Current number of cached chains for a wire type. Test/observability only. */
+export declare const middlewareCacheSize: (wireType: PikkuWiringTypes) => number;
 export declare const combineMiddleware: (wireType: PikkuWiringTypes, uid: string, { wireInheritedMiddleware, wireMiddleware, funcInheritedMiddleware, funcMiddleware, packageName, }?: {
     wireInheritedMiddleware?: MiddlewareMetadata[];
     wireMiddleware?: CorePikkuMiddleware[];

package/dist/middleware-runner.js

@@ -65,27 +65,64 @@ const getMiddlewareByName = (name) => {
     const middleware = middlewareStore[name];
     return middleware?.[0];
 };
+/**
+ * Per-wire-type cap on the memoized middleware chains. The cache key is the
+ * wire id, which for an RPC is the requested function name — and a name can
+ * carry attacker-influenced version/alias suffixes that still resolve. Left
+ * unbounded, a caller could grow the map without limit and exhaust memory. It
+ * is a pure memoization, so evicting the oldest entry and recomputing on the
+ * next miss is free of correctness cost. The cap sits well above any realistic
+ * function count.
+ */
+const MIDDLEWARE_CACHE_MAX = 4096;
+const newCache = () => new Map();
 const middlewareCache = {
-    http: {},
-    rpc: {},
-    channel: {},
-    queue: {},
-    scheduler: {},
-    trigger: {},
-    mcp: {},
-    agent: {},
-    cli: {},
-    workflow: {},
-    gateway: {},
+    http: newCache(),
+    rpc: newCache(),
+    channel: newCache(),
+    queue: newCache(),
+    scheduler: newCache(),
+    trigger: newCache(),
+    mcp: newCache(),
+    agent: newCache(),
+    cli: newCache(),
+    workflow: newCache(),
+    gateway: newCache(),
+};
+const cacheGet = (wireType, uid) => {
+    const cache = middlewareCache[wireType];
+    const hit = cache.get(uid);
+    if (hit !== undefined) {
+        // Refresh recency: delete + re-set moves the key to the end (LRU).
+        cache.delete(uid);
+        cache.set(uid, hit);
+    }
+    return hit;
+};
+const cacheSet = (wireType, uid, value) => {
+    const cache = middlewareCache[wireType];
+    cache.set(uid, value);
+    if (cache.size > MIDDLEWARE_CACHE_MAX) {
+        // Evict the least-recently-used entry (Map iteration is insertion order).
+        const oldest = cache.keys().next().value;
+        if (oldest !== undefined)
+            cache.delete(oldest);
+    }
+    return value;
 };
 export const clearMiddlewareCache = () => {
     for (const key of Object.keys(middlewareCache)) {
-        middlewareCache[key] = {};
+        middlewareCache[key] = newCache();
     }
 };
+/** The per-wire-type entry cap. Exposed so the boundedness invariant is testable. */
+export const MIDDLEWARE_CACHE_MAX_ENTRIES = MIDDLEWARE_CACHE_MAX;
+/** Current number of cached chains for a wire type. Test/observability only. */
+export const middlewareCacheSize = (wireType) => middlewareCache[wireType].size;
 export const combineMiddleware = (wireType, uid, { wireInheritedMiddleware, wireMiddleware, funcInheritedMiddleware, funcMiddleware, packageName = null, } = {}) => {
-    if (middlewareCache[wireType][uid]) {
-        return middlewareCache[wireType][uid];
+    const cached = cacheGet(wireType, uid);
+    if (cached) {
+        return cached;
     }
     const resolved = [];
     // Global middleware is application-wide, so the root namespace always
@@ -138,6 +175,5 @@ export const combineMiddleware = (wireType, uid, { wireInheritedMiddleware, wire
         resolved.push(...funcMiddleware);
     }
     sortByPriority(resolved);
-    middlewareCache[wireType][uid] = freezeDedupe(resolved);
-    return middlewareCache[wireType][uid];
+    return cacheSet(wireType, uid, freezeDedupe(resolved));
 };

package/dist/services/in-memory-lease-service.d.ts

@@ -0,0 +1,13 @@
+import type { Lease, LeaseService } from './lease-service.js';
+export declare class InMemoryLeaseService implements LeaseService {
+    private now;
+    private leases;
+    constructor(now?: () => number);
+    acquire(key: string, holder: string, ttlMs: number): Promise<Lease | null>;
+    refresh(lease: Lease, ttlMs: number): Promise<Lease | null>;
+    release(lease: Lease): Promise<void>;
+    get(key: string): Promise<Lease | null>;
+    private lapsed;
+    private holds;
+    private set;
+}

package/dist/services/in-memory-lease-service.js

@@ -0,0 +1,49 @@
+export class InMemoryLeaseService {
+    now;
+    leases = new Map();
+    constructor(now = Date.now) {
+        this.now = now;
+    }
+    async acquire(key, holder, ttlMs) {
+        const current = this.leases.get(key);
+        const expiresAt = new Date(this.now() + ttlMs);
+        if (!current)
+            return this.set({ key, holder, token: 1, expiresAt });
+        if (this.lapsed(current)) {
+            return this.set({ key, holder, token: current.token + 1, expiresAt });
+        }
+        if (current.holder === holder)
+            return this.set({ ...current, expiresAt });
+        return null;
+    }
+    async refresh(lease, ttlMs) {
+        const current = this.leases.get(lease.key);
+        if (!this.holds(current, lease) || this.lapsed(current))
+            return null;
+        return this.set({ ...current, expiresAt: new Date(this.now() + ttlMs) });
+    }
+    async release(lease) {
+        const current = this.leases.get(lease.key);
+        if (this.holds(current, lease)) {
+            this.set({ ...current, expiresAt: new Date(0) });
+        }
+    }
+    async get(key) {
+        const current = this.leases.get(key);
+        if (!current || this.lapsed(current))
+            return null;
+        return { ...current };
+    }
+    lapsed(lease) {
+        return lease.expiresAt.getTime() <= this.now();
+    }
+    holds(current, lease) {
+        return (!!current &&
+            current.holder === lease.holder &&
+            current.token === lease.token);
+    }
+    set(lease) {
+        this.leases.set(lease.key, lease);
+        return { ...lease };
+    }
+}

package/dist/services/in-memory-workflow-service.d.ts

@@ -24,9 +24,16 @@ export declare class InMemoryWorkflowService extends PikkuWorkflowService implem
     protected insertStepStateImpl(runId: string, stepName: string, rpcName: string | null, data: any, stepOptions?: WorkflowStepOptions, fromStepName?: string): Promise<StepState>;
     getStepState(runId: string, stepName: string): Promise<StepState>;
     protected setStepRunningImpl(stepId: string): Promise<void>;
+    refreshStepLease(stepId: string, leaseMs: number | null, attempt?: number): Promise<boolean>;
     protected setStepScheduledImpl(stepId: string): Promise<void>;
-    protected setStepResultImpl(stepId: string, result: any): Promise<void>;
-    protected setStepErrorImpl(stepId: string, error: Error): Promise<void>;
+    /**
+     * The live step `stepId` names, when `attempt` still owns it. A retry is a
+     * new step with its own id, so a superseded attempt's id no longer names a
+     * live step at all.
+     */
+    private ownedStep;
+    protected setStepResultImpl(stepId: string, result: any, attempt?: number): Promise<void>;
+    protected setStepErrorImpl(stepId: string, error: Error, attempt?: number): Promise<void>;
     protected setStepChildRunIdImpl(stepId: string, childRunId: string): Promise<void>;
     protected createRetryAttemptImpl(failedStepId: string, status: 'pending' | 'running'): Promise<StepState>;
     /**
@@ -53,7 +60,7 @@ export declare class InMemoryWorkflowService extends PikkuWorkflowService implem
     }>>;
     getDistinctWorkflowNames(): Promise<string[]>;
     deleteRun(id: string): Promise<boolean>;
-    withRunLock<T>(_id: string, fn: () => Promise<T>): Promise<T>;
+    withRunLease<T>(_id: string, fn: () => Promise<T>): Promise<T>;
     withStepLock<T>(_runId: string, _stepName: string, fn: () => Promise<T>): Promise<T>;
     close(): Promise<void>;
     getCompletedGraphState(runId: string): Promise<{
@@ -68,6 +75,7 @@ export declare class InMemoryWorkflowService extends PikkuWorkflowService implem
         stepName: string;
         status: StepStatus;
         fromStepName?: string;
+        leaseExpiresAt?: Date;
     }>>;
     getNodeResults(runId: string, nodeIds: string[]): Promise<Record<string, any>>;
     setBranchKey(runId: string, nodeId: string, branchKey: string): Promise<void>;

package/dist/services/in-memory-workflow-service.js

@@ -1,6 +1,8 @@
 import { randomUUID } from 'crypto';
 import { PikkuWorkflowService } from '../wirings/workflow/pikku-workflow-service.js';
 import { isExpectedError } from '../errors/error-handler.js';
+import { isStepLeaseLive } from '../wirings/workflow/workflow-constants.js';
+import { WorkflowStepSupersededError } from '../wirings/workflow/workflow-errors.js';
 // knowledge: decisions/internals/the-in-memory-workflow-service-is-inline-only-and-single-process.md
 export class InMemoryWorkflowService extends PikkuWorkflowService {
     constructor(options = {}) {
@@ -110,42 +112,69 @@ export class InMemoryWorkflowService extends PikkuWorkflowService {
             }
         }
     }
-    async setStepScheduledImpl(stepId) {
+    async refreshStepLease(stepId, leaseMs, attempt) {
         for (const step of this.steps.values()) {
             if (step.stepId === stepId) {
-                step.status = 'scheduled';
-                step.scheduledAt = new Date();
-                step.updatedAt = new Date();
-                break;
+                if (attempt !== undefined && step.attemptCount !== attempt) {
+                    return false;
+                }
+                step.leaseExpiresAt =
+                    leaseMs === null ? undefined : new Date(Date.now() + leaseMs);
+                return true;
             }
         }
+        return false;
     }
-    async setStepResultImpl(stepId, result) {
+    async setStepScheduledImpl(stepId) {
         for (const step of this.steps.values()) {
             if (step.stepId === stepId) {
-                step.status = 'succeeded';
-                step.result = result;
-                step.succeededAt = new Date();
+                step.status = 'scheduled';
+                step.scheduledAt = new Date();
                 step.updatedAt = new Date();
                 break;
             }
         }
     }
-    async setStepErrorImpl(stepId, error) {
+    /**
+     * The live step `stepId` names, when `attempt` still owns it. A retry is a
+     * new step with its own id, so a superseded attempt's id no longer names a
+     * live step at all.
+     */
+    ownedStep(stepId, attempt) {
+        let live;
         for (const step of this.steps.values()) {
             if (step.stepId === stepId) {
-                step.status = 'failed';
-                step.error = {
-                    name: error.name,
-                    message: error.message,
-                    stack: error.stack,
-                    expected: isExpectedError(error),
-                };
-                step.failedAt = new Date();
-                step.updatedAt = new Date();
+                live = step;
                 break;
             }
         }
+        if (attempt !== undefined && live?.attemptCount !== attempt) {
+            throw new WorkflowStepSupersededError(stepId, attempt);
+        }
+        return live;
+    }
+    async setStepResultImpl(stepId, result, attempt) {
+        const step = this.ownedStep(stepId, attempt);
+        if (!step)
+            return;
+        step.status = 'succeeded';
+        step.result = result;
+        step.succeededAt = new Date();
+        step.updatedAt = new Date();
+    }
+    async setStepErrorImpl(stepId, error, attempt) {
+        const step = this.ownedStep(stepId, attempt);
+        if (!step)
+            return;
+        step.status = 'failed';
+        step.error = {
+            name: error.name,
+            message: error.message,
+            stack: error.stack,
+            expected: isExpectedError(error),
+        };
+        step.failedAt = new Date();
+        step.updatedAt = new Date();
     }
     async setStepChildRunIdImpl(stepId, childRunId) {
         for (const step of this.steps.values()) {
@@ -230,9 +259,9 @@ export class InMemoryWorkflowService extends PikkuWorkflowService {
             if (run.status !== 'running')
                 continue;
             const steps = this.stepHistory.get(runId) ?? [];
-            if (steps.some((step) => step.status === 'running' ||
-                step.status === 'scheduled' ||
-                step.status === 'suspended')) {
+            if (steps.some((step) => step.status === 'scheduled' ||
+                step.status === 'suspended' ||
+                (step.status === 'running' && isStepLeaseLive(step.leaseExpiresAt)))) {
                 continue;
             }
             const lastActivity = steps.reduce((latest, step) => (step.updatedAt > latest ? step.updatedAt : latest), run.updatedAt);
@@ -293,7 +322,7 @@ export class InMemoryWorkflowService extends PikkuWorkflowService {
         }
         return existed;
     }
-    async withRunLock(_id, fn) {
+    async withRunLease(_id, fn) {
         // knowledge: decisions/internals/the-in-memory-workflow-service-is-inline-only-and-single-process.md
         return fn();
     }
@@ -360,6 +389,7 @@ export class InMemoryWorkflowService extends PikkuWorkflowService {
                 stepName: key.substring(prefix.length),
                 status: step.status,
                 fromStepName: step.fromStepName,
+                leaseExpiresAt: step.leaseExpiresAt,
             });
         }
         return instances;

package/dist/services/index.d.ts

@@ -17,6 +17,8 @@ export { REMOTE_JOBS_SECRET_HEADER, REMOTE_JOBS_SECRET_VARIABLE, REMOTE_QUEUE_JO
 export { InMemoryQueueService } from './in-memory-queue-service.js';
 export { InMemoryTriggerService } from './in-memory-trigger-service.js';
 export { InMemoryTriggerSourceStore, type DeclaredTriggerSource, type TriggerSourceKind, type TriggerSourceResult, type TriggerSourceRow, type TriggerSourceStore, } from './trigger-source-store.js';
+export { LeaseLostError, LeaseTakenError, holdLease, type Lease, type LeaseService, } from './lease-service.js';
+export { InMemoryLeaseService } from './in-memory-lease-service.js';
 export { InMemoryAgentRunStateService } from './in-memory-agent-run-state-service.js';
 export { LocalGatewayService } from './local-gateway-service.js';
 export { FileScenarioRunStore, scenarioArtifactContentType, scenarioRunSummary, type FileScenarioRunStoreOptions, } from './file-scenario-run-store.js';

package/dist/services/index.js

@@ -17,6 +17,8 @@ export { REMOTE_JOBS_SECRET_HEADER, REMOTE_JOBS_SECRET_VARIABLE, REMOTE_QUEUE_JO
 export { InMemoryQueueService } from './in-memory-queue-service.js';
 export { InMemoryTriggerService } from './in-memory-trigger-service.js';
 export { InMemoryTriggerSourceStore, } from './trigger-source-store.js';
+export { LeaseLostError, LeaseTakenError, holdLease, } from './lease-service.js';
+export { InMemoryLeaseService } from './in-memory-lease-service.js';
 export { InMemoryAgentRunStateService } from './in-memory-agent-run-state-service.js';
 export { LocalGatewayService } from './local-gateway-service.js';
 export { FileScenarioRunStore, scenarioArtifactContentType, scenarioRunSummary, } from './file-scenario-run-store.js';

package/dist/services/lease-service.d.ts

@@ -0,0 +1,61 @@
+export type Lease = {
+    key: string;
+    holder: string;
+    token: number;
+    expiresAt: Date;
+};
+/**
+ * Named leases shared by every process on the same store. A lease is held until
+ * it is released or runs out, so it is not a mutex: a holder that stalls past
+ * its expiry loses the key without knowing. `token` rises every time the key
+ * changes hands, and a write that must not come from a stale holder checks it.
+ */
+export interface LeaseService {
+    acquire(key: string, holder: string, ttlMs: number): Promise<Lease | null>;
+    refresh(lease: Lease, ttlMs: number): Promise<Lease | null>;
+    release(lease: Lease): Promise<void>;
+    get(key: string): Promise<Lease | null>;
+}
+export declare class LeaseTakenError extends Error {
+    readonly key: string;
+    constructor(key: string);
+}
+/**
+ * Thrown by `holdLease` when the lease lapsed or passed to another holder while
+ * the body ran, so its result was not produced under the lease. The body's
+ * `signal` is aborted with it as soon as the loss is seen.
+ */
+export declare class LeaseLostError extends Error {
+    readonly key: string;
+    constructor(key: string);
+}
+/** How many renewals fit in one lease, so a failed one still leaves others before it runs out. */
+export declare const RENEWALS_PER_LEASE = 3;
+/** How long a holder waits between renewals of a `ttlMs` lease. */
+export declare const leaseRenewalIntervalMs: (ttlMs: number) => number;
+export type LeaseRenewal = {
+    /** Aborts with `LeaseLostError` once the lease is no longer this holder's. */
+    signal: AbortSignal;
+    /** Stops renewing, after waiting for a renewal already in flight. */
+    stop: () => Promise<void>;
+};
+/**
+ * Renews a held lease every third of `ttlMs` until stopped, so a renewal that
+ * fails still leaves two more before the lease runs out.
+ *
+ * `renew` resolves `true` while the lease is still this holder's and `false`
+ * once another holder has it. A throw proves neither, so the lease is only
+ * counted lost once renewals have kept failing past its expiry on this
+ * process's clock.
+ *
+ * `stop` waits for a renewal in flight, so a release made after it cannot be
+ * undone by a late one.
+ */
+export declare const keepLeaseAlive: (key: string, ttlMs: number, renew: () => Promise<boolean>) => LeaseRenewal;
+/**
+ * Runs `fn` holding `key` on `leases`, renewing the lease every third of
+ * `ttlMs` until it settles. `ttlMs` is how long a crashed holder keeps the key
+ * from everyone else. Never waits for a taken key: that throws
+ * `LeaseTakenError`.
+ */
+export declare const holdLease: <T>(leases: LeaseService, key: string, fn: (lease: Lease, signal: AbortSignal) => Promise<T>, ttlMs?: number) => Promise<T>;