@pikku/kysely 0.13.30 → 0.13.33

package/CHANGELOG.md

@@ -1,3 +1,89 @@
+## 0.13.33
+
+### Patch Changes
+
+- a16b0ba: The workflow run lease takes the app's `leaseService` (`holdLease(leaseService, 'workflow-run:<id>', …)`) in place of `pg_advisory_lock` / `GET_LOCK`, for DSL and graph runs alike. A message for a run already being orchestrated elsewhere wakes the run again a second later instead of failing, so a long pass never spends the retry budget the queue keeps for real failures. An inline run whose lease was lost after its body finished keeps the outcome it wrote. `PgWorkflowQueueOptions` and `RunLockHoldTimeoutError` are removed. `WorkflowService.withRunLock` is renamed `withRunLease`, since what it holds runs out unless renewed; a custom workflow service overriding it renames the method.
+
+  Pass the lease service to the workflow service — `new PgKyselyWorkflowService(db, { leaseService })` — register the same instance as the app's `leaseService`, and migrate `pikku_lease`.
+
+  A step lease is renewed by the same loop as the run lease. A graph node whose worker died is dispatched again once its lease lapses, instead of leaving the run waiting on it. A replayed step on Kysely now sees its lease, as `getStepState` does. The in-memory store refuses a superseded worker's outcome, as the database stores do. The MySQL workflow service can now record a workflow version; it used `ON CONFLICT`, which MySQL does not have.
+
+  `pikku db generate` now writes the `pikku_lease` table for any project that runs workflows, not only one that reaches `leaseService`, since every persistent workflow service holds its runs and steps there.
+
+- a16b0ba: Give a running workflow step a lease, so a step whose worker dies can be claimed again
+
+  A step moving from `pending` to `running` only proves it was dispatched; it says
+  nothing about whether the worker holding it is still alive. A worker that died
+  mid-step therefore left the step wedged in `running` forever: unclaimable by the
+  next dispatch, and invisible to the stalled-run sweep.
+
+  The dispatch that claims a step now stamps a lease on it, refreshes that lease on
+  an interval for as long as it is working, and releases it when the step parks on
+  a child run. A step whose lease has lapsed is claimable again, and counts as
+  in-flight no longer, so the stalled sweep can pick its run back up. A step that
+  keeps losing its worker fails with `WorkflowStepLeaseExpiredError` once its
+  attempts run out, rather than looping.
+
+  The lease duration is taken from the queue the step is dispatched through, so the
+  lease and the queue lock never disagree about who owns the job.
+
+  The refresh runs at half the lease and never later, even when that is under the
+  one-second floor the interval otherwise respects: applying that floor as a
+  maximum would renew a shorter lease for the first time after it had already
+  lapsed, which is the double-claim the lease exists to prevent. A lease that
+  short is logged once, pointing at the queue's `lockDuration`.
+
+  The lease is written and judged on the database's clock, never the worker's,
+  so a worker whose clock runs fast cannot claim a step another worker still
+  holds. A dispatch whose step has since been claimed by another stops renewing,
+  and says so: `refreshStepLease(stepId, leaseMs, attempt)` resolves `false` once
+  the step is no longer that attempt's.
+
+  The lease lives in a new `workflow_step.lease_expires_at` column, in epoch
+  milliseconds (`bigint`). Run `pikku db generate` and `pikku db migrate` after
+  upgrading: the runtime does not add columns, and every step read selects this
+  one.
+
+- a16b0ba: Persistent workflow services take a required `leaseService` (`WorkflowServiceOptions`) and lock runs and steps on it. The Postgres advisory and MySQL `GET_LOCK` step locks, and Redis's own `SET NX` run and step locks, are removed; `RedisWorkflowService` now takes `(connection, { leaseService, keyPrefix? })`.
+- Updated dependencies [a16b0ba]
+- Updated dependencies [21e9c3a]
+- Updated dependencies [a16b0ba]
+- Updated dependencies [a16b0ba]
+  - @pikku/core@0.12.133
+
+## 0.13.32
+
+### Patch Changes
+
+- 942ebdd: Add `LeaseService`: named leases shared by every process on the same store. A lease is not a mutex: a holder that stalls past its expiry loses the key without knowing, so whatever the body writes to should check the lease's `token` and refuse an older one. `acquire` never blocks and answers `null` while someone else holds the key; a lease lapses on its own if its holder dies; and each lease carries a `token` that rises every time the key changes hands, so a stale holder cannot refresh or release its successor's lease. `holdLease` refreshes the lease while its body runs, hands the body an `AbortSignal` that fires the moment the lease is lost, and throws `LeaseLostError` once the body finishes if the lease lapsed or changed hands meanwhile, so a caller never takes that result as produced under the lease.
+
+  Ships as `InMemoryLeaseService`, `PgKyselyLeaseService` (`@pikku/kysely-postgres`), `MySQLKyselyLeaseService` (`@pikku/kysely-mysql`) and `KyselyLeaseService` (SQLite), all on `pikku_lease`. The Postgres and MySQL services judge every lease by the database's clock, so a worker whose clock runs fast cannot take a live lease or stretch its own; `KyselyLeaseService` reads the process clock and is for a single-host SQLite app. `pikku db generate` writes the table for any project that reaches `leaseService`.
+
+  `RedisLeaseService` (`@pikku/redis`) keeps each lease in a hash Redis expires itself, with every check-and-set in one Lua script that reads Redis's `TIME`, so it too judges leases by the server's clock; its token counter outlives release, so the next holder always gets a higher token.
+
+- Updated dependencies [d327fa5]
+- Updated dependencies [942ebdd]
+- Updated dependencies [8ac25a8]
+- Updated dependencies [6606777]
+  - @pikku/core@0.12.132
+
+## 0.13.31
+
+### Patch Changes
+
+- 01c0209: `@pikku/kysely` re-exports `Kysely`, `PostgresDialect` and `CamelCasePlugin` from its own copy of kysely. The local-services codegen writes these imports into a project, so it now opens and types the database with the same kysely copy the generated services were built against, instead of a second copy the project may resolve on its own.
+- 5bce779: Webhook trigger sources are off until someone turns them on, and addons declare their own.
+
+  An addon calls `wireTriggerWebhookSource` in its own package, and an app that wires the addon gets the source (its route included) without declaring it: named and routed after the addon's namespace, so two instances get one each. A source the app declares under the same name wins.
+
+  Every source now has an `enabled` switch in the `triggerSourceStore`, off by default. `reconcileTriggerSources` registers only enabled sources and records each one's `baseUrl` and `labelPrefix`; a disabled source's route answers 404 without running `receive`. New in core: `enableTriggerSource` registers a source with its provider and `disableTriggerSource` stops it receiving, then tears it down, both at the recorded address unless one is given. The admin addon exposes them as `triggerSourceEnable` and `triggerSourceDisable`. The `pikkuTriggerSource` table gains `enabled`, `baseUrl` and `labelPrefix`: run `pikku db generate` for the migration.
+
+- Updated dependencies [dfcd351]
+- Updated dependencies [cf40182]
+- Updated dependencies [5bce779]
+- Updated dependencies [5bce779]
+  - @pikku/core@0.12.130
+
 ## 0.13.30
 
 ### Patch Changes

package/README.md

@@ -15,16 +15,26 @@ npm install @pikku/kysely kysely
 ## Usage
 
 ```typescript
-import { KyselyWorkflowService, KyselySecretService } from '@pikku/kysely'
+import {
+  KyselyLeaseService,
+  KyselySecretService,
+  KyselyWorkflowService,
+} from '@pikku/kysely'
 import type { KyselyPikkuDB } from '@pikku/kysely'
 import type { Kysely } from 'kysely'
 
 declare const db: Kysely<KyselyPikkuDB>
 
-const workflowService = new KyselyWorkflowService(db)
+const leaseService = new KyselyLeaseService(db)
+await leaseService.init()
+
+const workflowService = new KyselyWorkflowService(db, { leaseService })
 const secretService = new KyselySecretService(db, { key: encryptionKey })
 ```
 
+The workflow service locks runs and steps on the `leaseService` you give it;
+register the same instance as the app's `leaseService`.
+
 ## Docs
 
 https://pikku.dev/docs

package/dist/src/index.d.ts

@@ -15,17 +15,19 @@ export { KyselyFeatureFlagStore } from './kysely-feature-flag-store.js';
 export type { KyselyFeatureFlagStoreOptions } from './kysely-feature-flag-store.js';
 export { KyselyWebhookService } from './kysely-webhook-service.js';
 export { KyselyTriggerSourceStore } from './kysely-trigger-source-store.js';
+export { KyselyLeaseService } from './kysely-lease-service.js';
 export { KyselyIncomingWebhookService } from './kysely-incoming-webhook-service.js';
 export { createAuditedKysely, type CreateAuditedKyselyOptions, } from './create-audited-kysely.js';
 export { KyselyAuditService } from './kysely-audit-service.js';
 export { KyselyAnalyticsService } from './kysely-analytics-service.js';
 export { KyselyVirtualUserRunStore } from './kysely-virtual-user-run-store.js';
 export { KyselyVirtualUserScheduleStore } from './kysely-virtual-user-schedule-store.js';
+export { CamelCasePlugin, Kysely, PostgresDialect } from 'kysely';
 export { SerializePlugin, BaseSerializePlugin, type Serializer, type Deserializer, } from './serialize-plugin.js';
 export { createCoercionPlugin, type ColumnKind, type CoercionMap, type CreateCoercionPluginOptions, } from './coercion-plugin.js';
 export { createClassificationPlugin, type CreateClassificationPluginOptions, } from './classification-plugin.js';
 export { ClassificationCrypto, DEFAULT_KEY_ID, isColumnEnvelope, parseColumnEnvelope, type ClassificationCryptoOptions, type ColumnEnvelope, type KEKResolver, type ResolvedKEK, } from './classification-crypto.js';
-export { agentSchema, analyticsSchema, auditSchema, channelSchema, credentialSchema, deploymentSchema, flagSchema, incomingWebhookSchema, scopeSchema, secretSchema, sessionSchema, virtualUserSchema, webhookSchema, workflowSchema, pikkuSchemas, requiredPikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, requirePikkuSchema, resolveRequirements, type PikkuSchema, type RequiredTypes, type UnmetRequirement, type SchemaStatementFactory, } from './schema/index.js';
+export { agentSchema, analyticsSchema, auditSchema, channelSchema, credentialSchema, deploymentSchema, flagSchema, incomingWebhookSchema, leaseSchema, scopeSchema, secretSchema, sessionSchema, virtualUserSchema, webhookSchema, workflowSchema, pikkuSchemas, requiredPikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, requirePikkuSchema, resolveRequirements, type PikkuSchema, type RequiredTypes, type UnmetRequirement, type SchemaStatementFactory, } from './schema/index.js';
 export type { KyselyPikkuDB } from './kysely-tables.js';
 export type { WorkflowRunService } from '@pikku/core/workflow';
 export type { AgentRunService, AgentRunRow } from '@pikku/core/agent';

package/dist/src/index.js

@@ -14,14 +14,19 @@ export { KyselyScopeService } from './kysely-scope-service.js';
 export { KyselyFeatureFlagStore } from './kysely-feature-flag-store.js';
 export { KyselyWebhookService } from './kysely-webhook-service.js';
 export { KyselyTriggerSourceStore } from './kysely-trigger-source-store.js';
+export { KyselyLeaseService } from './kysely-lease-service.js';
 export { KyselyIncomingWebhookService } from './kysely-incoming-webhook-service.js';
 export { createAuditedKysely, } from './create-audited-kysely.js';
 export { KyselyAuditService } from './kysely-audit-service.js';
 export { KyselyAnalyticsService } from './kysely-analytics-service.js';
 export { KyselyVirtualUserRunStore } from './kysely-virtual-user-run-store.js';
 export { KyselyVirtualUserScheduleStore } from './kysely-virtual-user-schedule-store.js';
+// Re-exported so a generated file that constructs the services above opens and
+// types its database with the same copy of kysely those services were built
+// against, rather than a second copy the project may resolve on its own.
+export { CamelCasePlugin, Kysely, PostgresDialect } from 'kysely';
 export { SerializePlugin, BaseSerializePlugin, } from './serialize-plugin.js';
 export { createCoercionPlugin, } from './coercion-plugin.js';
 export { createClassificationPlugin, } from './classification-plugin.js';
 export { ClassificationCrypto, DEFAULT_KEY_ID, isColumnEnvelope, parseColumnEnvelope, } from './classification-crypto.js';
-export { agentSchema, analyticsSchema, auditSchema, channelSchema, credentialSchema, deploymentSchema, flagSchema, incomingWebhookSchema, scopeSchema, secretSchema, sessionSchema, virtualUserSchema, webhookSchema, workflowSchema, pikkuSchemas, requiredPikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, requirePikkuSchema, resolveRequirements, } from './schema/index.js';
+export { agentSchema, analyticsSchema, auditSchema, channelSchema, credentialSchema, deploymentSchema, flagSchema, incomingWebhookSchema, leaseSchema, scopeSchema, secretSchema, sessionSchema, virtualUserSchema, webhookSchema, workflowSchema, pikkuSchemas, requiredPikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, requirePikkuSchema, resolveRequirements, } from './schema/index.js';

package/dist/src/kysely-lease-clock.d.ts

@@ -0,0 +1,14 @@
+import { type RawBuilder } from 'kysely';
+/**
+ * Now, in epoch milliseconds, as a SQL expression: the clock every lease in
+ * this package is granted and judged on.
+ *
+ * A lease is compared by whichever worker reads it next, so judging it on each
+ * worker's own clock lets a worker running ahead take a lease that is still
+ * held. The dialect services read the database's clock instead, which every
+ * worker shares. This default binds the process clock, for SQLite, where the
+ * database lives in the process anyway.
+ */
+export declare const appNowMs: () => RawBuilder<number>;
+/** `now + ms`, for a lease that runs `ms` from the clock's now. */
+export declare const leaseUntil: (now: RawBuilder<number>, ms: number) => RawBuilder<number>;

package/dist/src/kysely-lease-clock.js

@@ -0,0 +1,14 @@
+import { sql } from 'kysely';
+/**
+ * Now, in epoch milliseconds, as a SQL expression: the clock every lease in
+ * this package is granted and judged on.
+ *
+ * A lease is compared by whichever worker reads it next, so judging it on each
+ * worker's own clock lets a worker running ahead take a lease that is still
+ * held. The dialect services read the database's clock instead, which every
+ * worker shares. This default binds the process clock, for SQLite, where the
+ * database lives in the process anyway.
+ */
+export const appNowMs = () => sql `cast(${Date.now()} as bigint)`;
+/** `now + ms`, for a lease that runs `ms` from the clock's now. */
+export const leaseUntil = (now, ms) => sql `${now} + ${ms}`;

package/dist/src/kysely-lease-service.d.ts

@@ -0,0 +1,37 @@
+import type { Lease, LeaseService } from '@pikku/core/services';
+import type { Kysely } from 'kysely';
+import type { KyselyPikkuDB } from './kysely-tables.js';
+/**
+ * Leases on the `pikku_lease` table.
+ *
+ * Use it directly on SQLite. On PostgreSQL and MySQL use
+ * `PgKyselyLeaseService` and `MySQLKyselyLeaseService`, which judge every lease
+ * on the database's clock rather than on each worker's own.
+ *
+ * Every statement is one MySQL can run too — no `RETURNING`, no conditional
+ * upsert — so the dialects differ only in `nowMs` and `insertIfAbsent`.
+ */
+export declare class KyselyLeaseService implements LeaseService {
+    protected db: Kysely<KyselyPikkuDB>;
+    private initialized;
+    constructor(db: Kysely<KyselyPikkuDB>);
+    init(): Promise<void>;
+    /** Now, in epoch milliseconds, on the clock leases are judged by. */
+    protected nowMs(): import("kysely").RawBuilder<number>;
+    /** Create the key's row, lapsed, unless it already has one. */
+    protected insertIfAbsent(key: string, holder: string): Promise<void>;
+    /**
+     * The row is created lapsed and then taken by the same guarded `UPDATE` as
+     * any other lapsed lease, so there is one path to holding a key. `token` is
+     * assigned first: MySQL evaluates `SET` left to right against the values
+     * already assigned, and the holder test must see the previous holder.
+     *
+     * Whether the update won is read back rather than taken from its row count,
+     * because MySQL counts rows changed, not rows matched.
+     */
+    acquire(key: string, holder: string, ttlMs: number): Promise<Lease | null>;
+    refresh(lease: Lease, ttlMs: number): Promise<Lease | null>;
+    release(lease: Lease): Promise<void>;
+    get(key: string): Promise<Lease | null>;
+    private readHeld;
+}

package/dist/src/kysely-lease-service.js

@@ -0,0 +1,116 @@
+import { appNowMs, leaseUntil } from './kysely-lease-clock.js';
+import { requirePikkuSchema } from './schema/index.js';
+import { leaseSchema } from './schema/lease.schema.js';
+const toLease = (row) => ({
+    key: row.key,
+    holder: row.holder,
+    token: Number(row.token),
+    expiresAt: new Date(Number(row.expiresAt)),
+});
+/**
+ * Leases on the `pikku_lease` table.
+ *
+ * Use it directly on SQLite. On PostgreSQL and MySQL use
+ * `PgKyselyLeaseService` and `MySQLKyselyLeaseService`, which judge every lease
+ * on the database's clock rather than on each worker's own.
+ *
+ * Every statement is one MySQL can run too — no `RETURNING`, no conditional
+ * upsert — so the dialects differ only in `nowMs` and `insertIfAbsent`.
+ */
+export class KyselyLeaseService {
+    db;
+    initialized = false;
+    constructor(db) {
+        this.db = db;
+    }
+    async init() {
+        if (this.initialized)
+            return;
+        await requirePikkuSchema(this.db, leaseSchema);
+        this.initialized = true;
+    }
+    /** Now, in epoch milliseconds, on the clock leases are judged by. */
+    nowMs() {
+        return appNowMs();
+    }
+    /** Create the key's row, lapsed, unless it already has one. */
+    async insertIfAbsent(key, holder) {
+        await this.db
+            .insertInto('pikkuLease')
+            .values({ key, holder, token: 0, expiresAt: 0 })
+            .onConflict((oc) => oc.column('key').doNothing())
+            .execute();
+    }
+    /**
+     * The row is created lapsed and then taken by the same guarded `UPDATE` as
+     * any other lapsed lease, so there is one path to holding a key. `token` is
+     * assigned first: MySQL evaluates `SET` left to right against the values
+     * already assigned, and the holder test must see the previous holder.
+     *
+     * Whether the update won is read back rather than taken from its row count,
+     * because MySQL counts rows changed, not rows matched.
+     */
+    async acquire(key, holder, ttlMs) {
+        await this.insertIfAbsent(key, holder);
+        const now = this.nowMs();
+        await this.db
+            .updateTable('pikkuLease')
+            .set((eb) => ({
+            token: eb
+                .case()
+                .when(eb.and([eb('holder', '=', holder), eb('expiresAt', '>', now)]))
+                .then(eb.ref('token'))
+                .else(eb('token', '+', 1))
+                .end(),
+            holder,
+            expiresAt: leaseUntil(now, ttlMs),
+        }))
+            .where('key', '=', key)
+            .where((eb) => eb.or([eb('holder', '=', holder), eb('expiresAt', '<=', now)]))
+            .execute();
+        return this.readHeld(key, holder);
+    }
+    async refresh(lease, ttlMs) {
+        const now = this.nowMs();
+        await this.db
+            .updateTable('pikkuLease')
+            .set({ expiresAt: leaseUntil(now, ttlMs) })
+            .where('key', '=', lease.key)
+            .where('holder', '=', lease.holder)
+            .where('token', '=', lease.token)
+            .where('expiresAt', '>', now)
+            .execute();
+        return this.readHeld(lease.key, lease.holder, lease.token);
+    }
+    async release(lease) {
+        await this.db
+            .updateTable('pikkuLease')
+            .set({ expiresAt: 0 })
+            .where('key', '=', lease.key)
+            .where('holder', '=', lease.holder)
+            .where('token', '=', lease.token)
+            .execute();
+    }
+    async get(key) {
+        const row = await this.db
+            .selectFrom('pikkuLease')
+            .selectAll()
+            .where('key', '=', key)
+            .where('expiresAt', '>', this.nowMs())
+            .executeTakeFirst();
+        return row ? toLease(row) : null;
+    }
+    async readHeld(key, holder, token) {
+        let query = this.db
+            .selectFrom('pikkuLease')
+            .selectAll()
+            .where('key', '=', key)
+            .where('holder', '=', holder)
+            .where('expiresAt', '>', this.nowMs());
+        if (token !== undefined) {
+            query = query.where('token', '=', token);
+        }
+        const row = await query.executeTakeFirst();
+        return row ? toLease(row) : null;
+    }
+}

package/dist/src/kysely-tables.d.ts

@@ -46,6 +46,13 @@ export interface WorkflowStepTable {
     fromStepName: string | null;
     /** The history attempt currently in flight; see WorkflowStepHistoryTable.attempt. */
     currentAttempt: number | null;
+    /**
+     * When the dispatch holding a `running` step stops owning it. Null is a step
+     * nothing is expected to hand back — one parked on a child run, or written
+     * before it was ever claimed. Epoch milliseconds, on the clock leases are
+     * judged by; see `appNowMs`.
+     */
+    leaseExpiresAt: number | null;
     createdAt: Generated<Date>;
     updatedAt: Generated<Date>;
 }
@@ -240,13 +247,24 @@ export interface PikkuFeatureFlagsTable {
 export interface PikkuTriggerSourceTable {
     name: string;
     kind: string;
+    baseUrl: string | null;
+    labelPrefix: string | null;
     declared: Generated<boolean>;
+    enabled: Generated<boolean>;
     status: string | null;
     /** The last setup's provider identifiers as JSON; never a secret. */
     state: string | null;
     detail: string | null;
     updatedAt: Generated<Date>;
 }
+/** A named lease, current or last. */
+export interface PikkuLeaseTable {
+    key: string;
+    holder: string;
+    token: number;
+    /** Epoch milliseconds on the clock leases are judged by; see `appNowMs`. */
+    expiresAt: number;
+}
 /** One subject's short-circuit, in either direction, over a flag's rollout. */
 export interface PikkuFeatureFlagOverridesTable {
     flag: string;
@@ -433,6 +451,7 @@ export interface KyselyPikkuDB {
     pikkuFeatureFlags: PikkuFeatureFlagsTable;
     pikkuFeatureFlagOverrides: PikkuFeatureFlagOverridesTable;
     pikkuTriggerSource: PikkuTriggerSourceTable;
+    pikkuLease: PikkuLeaseTable;
     channels: ChannelsTable;
     channelSubscriptions: ChannelSubscriptionsTable;
     workflowRuns: WorkflowRunsTable;

package/dist/src/kysely-trigger-source-store.d.ts

@@ -10,5 +10,6 @@ export declare class KyselyTriggerSourceStore implements TriggerSourceStore {
     listTriggerSources(): Promise<TriggerSourceRow[]>;
     getTriggerSource(name: string): Promise<TriggerSourceRow | null>;
     recordTriggerSource(name: string, result: TriggerSourceResult): Promise<void>;
+    setTriggerSourceEnabled(name: string, enabled: boolean): Promise<void>;
     deleteTriggerSource(name: string): Promise<void>;
 }

package/dist/src/kysely-trigger-source-store.js

@@ -15,7 +15,10 @@ const parseState = (value) => {
 const toRow = (row) => ({
     name: row.name,
     kind: row.kind,
+    baseUrl: row.baseUrl,
+    labelPrefix: row.labelPrefix,
     declared: !!row.declared,
+    enabled: !!row.enabled,
     status: row.status,
     state: parseState(row.state),
     detail: row.detail,
@@ -35,11 +38,15 @@ export class KyselyTriggerSourceStore {
     }
     async syncTriggerSources(sources) {
         await this.db.transaction().execute(async (trx) => {
-            for (const { name, kind } of sources) {
+            for (const { name, kind, baseUrl, labelPrefix } of sources) {
+                const address = {
+                    ...(baseUrl !== undefined ? { baseUrl } : {}),
+                    ...(labelPrefix !== undefined ? { labelPrefix } : {}),
+                };
                 await trx
                     .insertInto('pikkuTriggerSource')
-                    .values({ name, kind, declared: true })
-                    .onConflict((oc) => oc.column('name').doUpdateSet({ kind, declared: true }))
+                    .values({ name, kind, declared: true, ...address })
+                    .onConflict((oc) => oc.column('name').doUpdateSet({ kind, declared: true, ...address }))
                     .execute();
             }
             const markStale = trx
@@ -83,6 +90,16 @@ export class KyselyTriggerSourceStore {
             throw new Error(`Unknown trigger source: ${name}`);
         }
     }
+    async setTriggerSourceEnabled(name, enabled) {
+        const { numUpdatedRows } = await this.db
+            .updateTable('pikkuTriggerSource')
+            .set({ enabled })
+            .where('name', '=', name)
+            .executeTakeFirst();
+        if (!numUpdatedRows) {
+            throw new Error(`Unknown trigger source: ${name}`);
+        }
+    }
     async deleteTriggerSource(name) {
         await this.db
             .deleteFrom('pikkuTriggerSource')

package/dist/src/kysely-workflow-service.d.ts

@@ -1,13 +1,13 @@
 import type { SerializedError } from '@pikku/core/errors';
 import { PikkuWorkflowService } from '@pikku/core/workflow';
-import type { WorkflowPlannedStep, WorkflowQueueOptions, WorkflowRun, WorkflowRunWire, StepState, StepStatus, WorkflowStatus, WorkflowVersionStatus } from '@pikku/core/workflow';
+import type { WorkflowPlannedStep, WorkflowServiceOptions, WorkflowRun, WorkflowRunWire, StepState, StepStatus, WorkflowStatus, WorkflowVersionStatus } from '@pikku/core/workflow';
 import { type Kysely } from 'kysely';
 import type { KyselyPikkuDB } from './kysely-tables.js';
 export declare class KyselyWorkflowService extends PikkuWorkflowService {
     protected db: Kysely<KyselyPikkuDB>;
     private initialized;
     private runService;
-    constructor(db: Kysely<KyselyPikkuDB>, options?: WorkflowQueueOptions);
+    constructor(db: Kysely<KyselyPikkuDB>, options: WorkflowServiceOptions);
     init(): Promise<void>;
     protected createRunImpl(workflowName: string, input: any, inline: boolean, graphHash: string, wire: WorkflowRunWire, options?: {
         deterministic?: boolean;
@@ -27,6 +27,9 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
         stepName: string;
     }>>;
     protected setStepRunningImpl(stepId: string): Promise<void>;
+    refreshStepLease(stepId: string, leaseMs: number | null, attempt?: number): Promise<boolean>;
+    /** Now, in epoch milliseconds, on the clock step leases are judged by. */
+    protected nowMs(): import("kysely").RawBuilder<number>;
     protected setStepScheduledImpl(stepId: string): Promise<void>;
     /**
      * Move a step and its current history attempt to the same status in one
@@ -43,21 +46,10 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
     private insertHistoryRecord;
     private getTimestampFieldForStatus;
     protected setStepChildRunIdImpl(stepId: string, childRunId: string): Promise<void>;
-    protected setStepResultImpl(stepId: string, result: any): Promise<void>;
-    protected setStepErrorImpl(stepId: string, error: Error): Promise<void>;
+    protected setStepResultImpl(stepId: string, result: any, attempt?: number): Promise<void>;
+    protected setStepErrorImpl(stepId: string, error: Error, attempt?: number): Promise<void>;
     protected createRetryAttemptImpl(stepId: string, status: 'pending' | 'running'): Promise<StepState>;
-    withRunLock<T>(_id: string, fn: () => Promise<T>): Promise<T>;
-    /**
-     * A pass-through, and deliberately so: the one decision that must exclude —
-     * claiming a step to execute it — is made by `claimStepForExecution` below as
-     * a single conditional write, which needs no lock to be exclusive.
-     *
-     * What is left are the suspend and approval sections in the engine, where the
-     * lock is genuine mutual exclusion rather than a claim. A dialect with a real
-     * primitive overrides this to cover them — `kysely-postgres` with
-     * `pg_advisory_xact_lock`, `kysely-mysql` with `GET_LOCK`.
-     */
-    withStepLock<T>(_runId: string, _stepName: string, fn: () => Promise<T>): Promise<T>;
+    private readStepAttempt;
     /**
      * Claim the step with a status-guarded `UPDATE` and read the affected-row
      * count: the database decides the winner in one statement, so two dispatches
@@ -70,11 +62,29 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
      *
      * The winner then goes through the ordinary transition methods, so history
      * rows and the mirror see exactly what they saw before.
+     *
+     * A `running` step is taken back only once its lease has lapsed, and the
+     * lease is part of the same guarded `UPDATE` — checking it here and writing
+     * it afterwards would let two dispatches read one lapsed lease and both
+     * proceed. Whether it has lapsed is the database's call, on its own clock:
+     * a live lease simply matches no row.
      */
-    protected claimStepForExecution(runId: string, stepName: string, rpcName: string): Promise<StepState | null>;
+    protected claimStepForExecution(runId: string, stepName: string, rpcName: string, leaseMs: number): Promise<StepState | null>;
     /**
-     * Move a step to `running` only if it is still in one of `from`, reporting
-     * whether this caller is the one that moved it.
+     * Take back a failed or lapsed step as a new attempt. The attempt number is
+     * advanced by the same guarded `UPDATE` that grants the lease, so the worker
+     * it is taken from is fenced out of its step at once — deferring that to
+     * `createRetryAttempt` would leave a window in which its result still lands,
+     * only to be cleared and the step run again.
+     */
+    private reclaimStepAttempt;
+    /**
+     * Move a step to `running` under a fresh lease, only if it is still in one of
+     * `from`, reporting whether this caller is the one that moved it.
+     *
+     * Claiming back a `running` step additionally requires its lease to have
+     * lapsed. A null lease never matches, which is what keeps a step parked on a
+     * child run — running, with no worker on it by design — out of reach.
      */
     private claimStepStatus;
     getCompletedGraphState(runId: string): Promise<{
@@ -86,6 +96,7 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
         stepName: string;
         status: StepStatus;
         fromStepName?: string;
+        leaseExpiresAt?: Date;
     }>>;
     getNodeResults(runId: string, nodeIds: string[]): Promise<Record<string, any>>;
     protected setBranchTakenImpl(stepId: string, branchKey: string): Promise<void>;
@@ -113,7 +124,7 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
      * The undispatched-step query, shared by every dialect.
      *
      * It used to be deliberately NOT an override, because this class's
-     * `withStepLock` is a pass-through: a subclass inheriting it (kysely-sqlite)
+     * `withStepLock` was a pass-through: a subclass inheriting it (kysely-sqlite)
      * had no atomic claim, so the relay's redundant dispatches would have become
      * double executions, and only `kysely-postgres` and `kysely-mysql` opted in
      * on the strength of their real locks. `claimStepForExecution` no longer

package/dist/src/kysely-workflow-service.js

@@ -1,14 +1,52 @@
-import { PikkuWorkflowService, WorkflowStepFunctionMismatchError, } from '@pikku/core/workflow';
+import { PikkuWorkflowService, WorkflowStepFunctionMismatchError, WorkflowStepLeaseExpiredError, WorkflowStepSupersededError, leaseAttemptsExhausted, } from '@pikku/core/workflow';
 import { sql } from 'kysely';
+import { appNowMs, leaseUntil } from './kysely-lease-clock.js';
 import { KyselyWorkflowRunService } from './kysely-workflow-run-service.js';
 import { parseJson } from './kysely-json.js';
 import { requirePikkuSchema } from './schema/index.js';
 import { workflowSchema } from './schema/workflow.schema.js';
+/**
+ * What a step read returns, one list for the single read and the whole-run
+ * read alike, so the replay snapshot cannot drift from `getStepState`.
+ */
+const STEP_STATE_COLUMNS = [
+    'workflowStepId',
+    'status',
+    'rpcName',
+    'result',
+    'error',
+    'retries',
+    'retryDelay',
+    'fromStepName',
+    // `current_attempt` is bumped alongside every history insert, so it is
+    // already the count this used to aggregate — and reading it keeps the
+    // engine's hottest query off a table that grows for the life of the run.
+    'currentAttempt',
+    'leaseExpiresAt',
+    'createdAt',
+    'updatedAt',
+];
+const toStepState = (row) => ({
+    stepId: row.workflowStepId,
+    status: row.status,
+    rpcName: row.rpcName ?? null,
+    result: parseJson(row.result),
+    error: parseJson(row.error),
+    attemptCount: Number(row.currentAttempt ?? 1),
+    retries: row.retries != null ? Number(row.retries) : undefined,
+    retryDelay: row.retryDelay ?? undefined,
+    fromStepName: row.fromStepName ?? undefined,
+    leaseExpiresAt: row.leaseExpiresAt != null
+        ? new Date(Number(row.leaseExpiresAt))
+        : undefined,
+    createdAt: new Date(row.createdAt),
+    updatedAt: new Date(row.updatedAt),
+});
 export class KyselyWorkflowService extends PikkuWorkflowService {
     db;
     initialized = false;
     runService;
-    constructor(db, options = {}) {
+    constructor(db, options) {
         super(options);
         this.db = db;
         this.runService = new KyselyWorkflowRunService(db);
@@ -92,73 +130,22 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
     async getStepState(runId, stepName) {
         const row = await this.db
             .selectFrom('workflowStep')
-            .select([
-            'workflowStepId',
-            'status',
-            'rpcName',
-            'result',
-            'error',
-            'retries',
-            'retryDelay',
-            'fromStepName',
-            // `current_attempt` is bumped alongside every history insert, so it is
-            // already the count this used to aggregate — and reading it keeps the
-            // engine's hottest query off a table that grows for the life of the run.
-            'currentAttempt',
-            'createdAt',
-            'updatedAt',
-        ])
+            .select(STEP_STATE_COLUMNS)
             .where('workflowRunId', '=', runId)
             .where('stepName', '=', stepName)
             .executeTakeFirst();
         if (!row) {
             throw new Error(`Step not found: runId=${runId}, stepName=${stepName}. Use insertStepState to create it.`);
         }
-        return {
-            stepId: row.workflowStepId,
-            status: row.status,
-            rpcName: row.rpcName ?? null,
-            result: parseJson(row.result),
-            error: parseJson(row.error),
-            attemptCount: Number(row.currentAttempt ?? 1),
-            retries: row.retries != null ? Number(row.retries) : undefined,
-            retryDelay: row.retryDelay ?? undefined,
-            fromStepName: row.fromStepName ?? undefined,
-            createdAt: new Date(row.createdAt),
-            updatedAt: new Date(row.updatedAt),
-        };
+        return toStepState(row);
     }
     async listStepStates(runId) {
         const rows = await this.db
             .selectFrom('workflowStep')
-            .select([
-            'workflowStepId',
-            'stepName',
-            'status',
-            'result',
-            'error',
-            'retries',
-            'retryDelay',
-            'fromStepName',
-            'currentAttempt',
-            'createdAt',
-            'updatedAt',
-        ])
+            .select([...STEP_STATE_COLUMNS, 'stepName'])
             .where('workflowRunId', '=', runId)
             .execute();
-        return rows.map((row) => ({
-            stepId: row.workflowStepId,
-            stepName: row.stepName,
-            status: row.status,
-            result: parseJson(row.result),
-            error: parseJson(row.error),
-            attemptCount: Number(row.currentAttempt ?? 1),
-            retries: row.retries != null ? Number(row.retries) : undefined,
-            retryDelay: row.retryDelay ?? undefined,
-            fromStepName: row.fromStepName ?? undefined,
-            createdAt: new Date(row.createdAt),
-            updatedAt: new Date(row.updatedAt),
-        }));
+        return rows.map((row) => ({ ...toStepState(row), stepName: row.stepName }));
     }
     async getRunHistory(runId) {
         return this.runService.getRunHistory(runId);
@@ -168,6 +155,24 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             runningAt: new Date(),
         });
     }
+    async refreshStepLease(stepId, leaseMs, attempt) {
+        let query = this.db
+            .updateTable('workflowStep')
+            .set({
+            leaseExpiresAt: leaseMs === null ? null : leaseUntil(this.nowMs(), leaseMs),
+            updatedAt: new Date(),
+        })
+            .where('workflowStepId', '=', stepId);
+        if (attempt !== undefined) {
+            query = query.where(sql `coalesce(current_attempt, 1)`, '=', attempt);
+        }
+        const renewed = await query.executeTakeFirst();
+        return Number(renewed?.numUpdatedRows ?? 0n) > 0;
+    }
+    /** Now, in epoch milliseconds, on the clock step leases are judged by. */
+    nowMs() {
+        return appNowMs();
+    }
     async setStepScheduledImpl(stepId) {
         await this.db
             .updateTable('workflowStep')
@@ -186,7 +191,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
      * over `created_at`, so a retry that lands in the same millisecond as the
      * attempt it replaces still resolves the newer row.
      */
-    async writeStepTransition(stepId, status, historyValues) {
+    async writeStepTransition(stepId, status, historyValues, attempt) {
         const now = new Date();
         const stepValues = { status, updatedAt: now };
         if ('result' in historyValues)
@@ -199,7 +204,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
         if (status === 'failed')
             stepValues.result = null;
         await this.db.transaction().execute(async (trx) => {
-            await trx
+            let step = trx
                 .updateTable('workflowStep')
                 .set({
                 ...stepValues,
@@ -209,8 +214,17 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
                 // MySQL would reject against the table being updated.
                 currentAttempt: sql `coalesce(current_attempt, 1)`,
             })
-                .where('workflowStepId', '=', stepId)
-                .execute();
+                .where('workflowStepId', '=', stepId);
+            // Fenced to the attempt that made the write, so a dispatch whose step
+            // was claimed again after its lease lapsed cannot record over the newer
+            // attempt. Throwing rolls the transaction back.
+            if (attempt !== undefined) {
+                step = step.where(sql `coalesce(current_attempt, 1)`, '=', attempt);
+            }
+            const moved = await step.executeTakeFirst();
+            if (attempt !== undefined && Number(moved?.numUpdatedRows ?? 0n) === 0) {
+                throw new WorkflowStepSupersededError(stepId, attempt);
+            }
             const written = await trx
                 .updateTable('workflowStepHistory')
                 .set({ status, ...historyValues })
@@ -247,7 +261,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             }
         });
     }
-    async insertHistoryRecord(stepId, status, attempt, result, error) {
+    async insertHistoryRecord(stepId, status, attempt, result, error, db = this.db) {
         const now = new Date();
         const values = {
             historyId: crypto.randomUUID(),
@@ -262,7 +276,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
         if (timestampField !== 'createdAt') {
             values[timestampField] = now;
         }
-        await this.db
+        await db
             .insertInto('workflowStepHistory')
             .values(values)
             .execute();
@@ -291,13 +305,13 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             .where('workflowStepId', '=', stepId)
             .execute();
     }
-    async setStepResultImpl(stepId, result) {
+    async setStepResultImpl(stepId, result, attempt) {
         await this.writeStepTransition(stepId, 'succeeded', {
             result: JSON.stringify(result),
             succeededAt: new Date(),
-        });
+        }, attempt);
     }
-    async setStepErrorImpl(stepId, error) {
+    async setStepErrorImpl(stepId, error, attempt) {
         const serializedError = {
             message: error.message,
             stack: error.stack,
@@ -306,7 +320,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
         await this.writeStepTransition(stepId, 'failed', {
             error: JSON.stringify(serializedError),
             failedAt: new Date(),
-        });
+        }, attempt);
     }
     async createRetryAttemptImpl(stepId, status) {
         // Read before writing: the new attempt number comes from the history this
@@ -329,6 +343,9 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             .where('workflowStepId', '=', stepId)
             .execute();
         await this.insertHistoryRecord(stepId, status, attempt);
+        return this.readStepAttempt(stepId);
+    }
+    async readStepAttempt(stepId) {
         const row = await this.db
             .selectFrom('workflowStep')
             .select([
@@ -360,22 +377,6 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             updatedAt: new Date(row.updatedAt),
         };
     }
-    async withRunLock(_id, fn) {
-        return fn();
-    }
-    /**
-     * A pass-through, and deliberately so: the one decision that must exclude —
-     * claiming a step to execute it — is made by `claimStepForExecution` below as
-     * a single conditional write, which needs no lock to be exclusive.
-     *
-     * What is left are the suspend and approval sections in the engine, where the
-     * lock is genuine mutual exclusion rather than a claim. A dialect with a real
-     * primitive overrides this to cover them — `kysely-postgres` with
-     * `pg_advisory_xact_lock`, `kysely-mysql` with `GET_LOCK`.
-     */
-    async withStepLock(_runId, _stepName, fn) {
-        return fn();
-    }
     /**
      * Claim the step with a status-guarded `UPDATE` and read the affected-row
      * count: the database decides the winner in one statement, so two dispatches
@@ -388,28 +389,56 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
      *
      * The winner then goes through the ordinary transition methods, so history
      * rows and the mirror see exactly what they saw before.
+     *
+     * A `running` step is taken back only once its lease has lapsed, and the
+     * lease is part of the same guarded `UPDATE` — checking it here and writing
+     * it afterwards would let two dispatches read one lapsed lease and both
+     * proceed. Whether it has lapsed is the database's call, on its own clock:
+     * a live lease simply matches no row.
      */
-    async claimStepForExecution(runId, stepName, rpcName) {
+    async claimStepForExecution(runId, stepName, rpcName, leaseMs) {
         const stepState = await this.getStepState(runId, stepName);
         // knowledge: decisions/security/a-step-runs-the-function-the-workflow-dispatched-it-with.md
         if (stepState.rpcName !== undefined &&
             stepState.rpcName !== (rpcName ?? null)) {
             throw new WorkflowStepFunctionMismatchError(runId, stepName);
         }
-        if (stepState.status === 'succeeded' || stepState.status === 'running') {
+        if (stepState.status === 'succeeded') {
             return null;
         }
-        if (stepState.status === 'failed') {
-            if (!(await this.claimStepStatus(stepState.stepId, ['failed']))) {
+        if (stepState.status === 'running') {
+            if (leaseAttemptsExhausted(stepState)) {
+                // Fail it only if its lease is still lapsed at the moment of writing:
+                // a worker that renewed since the read above still owns the step.
+                if (!(await this.claimStepStatus(stepState.stepId, ['running'], leaseMs))) {
+                    return null;
+                }
+                await this.setStepError(stepState.stepId, new WorkflowStepLeaseExpiredError(runId, stepName, stepState.attemptCount), stepState.attemptCount);
+                // `executeWorkflowStepInner` reads a null claim as "another dispatch
+                // owns it" and returns without resuming, so the run would stay
+                // `running` until a stalled sweep. Requeue the orchestrator here.
+                await this.resumeWorkflow(runId);
                 return null;
             }
-            return this.createRetryAttempt(stepState.stepId, 'running');
+        }
+        // A failed step that has spent every attempt is settled, however it failed;
+        // a redelivered message must not buy it another one.
+        if (stepState.status === 'failed' && leaseAttemptsExhausted(stepState)) {
+            return null;
+        }
+        if (stepState.status === 'failed' || stepState.status === 'running') {
+            const attempt = stepState.attemptCount + 1;
+            return this.mirrored(() => this.reclaimStepAttempt(stepState, attempt, leaseMs), async (mirror, newStep) => {
+                if (newStep) {
+                    await mirror.createRetryAttempt(stepState.stepId, {
+                        ...newStep,
+                        stepName,
+                    });
+                }
+            });
         }
         if (stepState.status === 'pending' || stepState.status === 'scheduled') {
-            if (!(await this.claimStepStatus(stepState.stepId, [
-                'pending',
-                'scheduled',
-            ]))) {
+            if (!(await this.claimStepStatus(stepState.stepId, ['pending', 'scheduled'], leaseMs))) {
                 return null;
             }
             await this.setStepRunning(stepState.stepId);
@@ -418,16 +447,62 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
         return stepState;
     }
     /**
-     * Move a step to `running` only if it is still in one of `from`, reporting
-     * whether this caller is the one that moved it.
+     * Take back a failed or lapsed step as a new attempt. The attempt number is
+     * advanced by the same guarded `UPDATE` that grants the lease, so the worker
+     * it is taken from is fenced out of its step at once — deferring that to
+     * `createRetryAttempt` would leave a window in which its result still lands,
+     * only to be cleared and the step run again.
+     */
+    async reclaimStepAttempt(stepState, attempt, leaseMs) {
+        const now = this.nowMs();
+        const claimed = await this.db.transaction().execute(async (trx) => {
+            let query = trx
+                .updateTable('workflowStep')
+                .set({
+                status: 'running',
+                result: null,
+                error: null,
+                currentAttempt: attempt,
+                leaseExpiresAt: leaseUntil(now, leaseMs),
+                updatedAt: new Date(),
+            })
+                .where('workflowStepId', '=', stepState.stepId)
+                .where('status', '=', stepState.status)
+                .where(sql `coalesce(current_attempt, 1)`, '=', stepState.attemptCount);
+            if (stepState.status === 'running') {
+                query = query.where('leaseExpiresAt', '<', now);
+            }
+            const moved = await query.executeTakeFirst();
+            if (Number(moved?.numUpdatedRows ?? 0n) === 0)
+                return false;
+            await this.insertHistoryRecord(stepState.stepId, 'running', attempt, undefined, undefined, trx);
+            return true;
+        });
+        return claimed ? this.readStepAttempt(stepState.stepId) : null;
+    }
+    /**
+     * Move a step to `running` under a fresh lease, only if it is still in one of
+     * `from`, reporting whether this caller is the one that moved it.
+     *
+     * Claiming back a `running` step additionally requires its lease to have
+     * lapsed. A null lease never matches, which is what keeps a step parked on a
+     * child run — running, with no worker on it by design — out of reach.
      */
-    async claimStepStatus(stepId, from) {
-        const claimed = await this.db
+    async claimStepStatus(stepId, from, leaseMs) {
+        const now = this.nowMs();
+        let query = this.db
             .updateTable('workflowStep')
-            .set({ status: 'running', updatedAt: new Date() })
+            .set({
+            status: 'running',
+            leaseExpiresAt: leaseUntil(now, leaseMs),
+            updatedAt: new Date(),
+        })
             .where('workflowStepId', '=', stepId)
-            .where('status', 'in', from)
-            .executeTakeFirst();
+            .where('status', 'in', from);
+        if (from.includes('running')) {
+            query = query.where('leaseExpiresAt', '<', now);
+        }
+        const claimed = await query.executeTakeFirst();
         return Number(claimed?.numUpdatedRows ?? 0n) > 0;
     }
     async getCompletedGraphState(runId) {
@@ -465,13 +540,16 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
     async getStepInstances(runId) {
         const rows = await this.db
             .selectFrom('workflowStep')
-            .select(['stepName', 'status', 'fromStepName'])
+            .select(['stepName', 'status', 'fromStepName', 'leaseExpiresAt'])
             .where('workflowRunId', '=', runId)
             .execute();
         return rows.map((r) => ({
             stepName: r.stepName,
             status: r.status,
             fromStepName: r.fromStepName ?? undefined,
+            leaseExpiresAt: r.leaseExpiresAt != null
+                ? new Date(Number(r.leaseExpiresAt))
+                : undefined,
         }));
     }
     async getNodeResults(runId, nodeIds) {
@@ -537,7 +615,7 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
      * The undispatched-step query, shared by every dialect.
      *
      * It used to be deliberately NOT an override, because this class's
-     * `withStepLock` is a pass-through: a subclass inheriting it (kysely-sqlite)
+     * `withStepLock` was a pass-through: a subclass inheriting it (kysely-sqlite)
      * had no atomic claim, so the relay's redundant dispatches would have become
      * double executions, and only `kysely-postgres` and `kysely-mysql` opted in
      * on the strength of their real locks. `claimStepForExecution` no longer
@@ -571,7 +649,19 @@ export class KyselyWorkflowService extends PikkuWorkflowService {
             .where(({ not, exists, selectFrom }) => not(exists(selectFrom('workflowStep as s')
             .select('s.workflowStepId')
             .whereRef('s.workflowRunId', '=', 'r.workflowRunId')
-            .where('s.status', 'in', ['running', 'scheduled', 'suspended']))))
+            .where((eb) => eb.or([
+            eb('s.status', 'in', ['scheduled', 'suspended']),
+            // A `running` step counts as in flight only while someone
+            // still holds it. A lapsed lease is a dispatch that died, and
+            // a null one a step nothing was expected to hand back.
+            eb.and([
+                eb('s.status', '=', 'running'),
+                eb.or([
+                    eb('s.leaseExpiresAt', 'is', null),
+                    eb('s.leaseExpiresAt', '>', this.nowMs()),
+                ]),
+            ]),
+        ])))))
             // The run row only moves on a status change, so idleness has to hold for
             // the steps too — expressed as "no step touched since `before`" rather
             // than max(updated_at) so it stays one index-friendly NOT EXISTS.

package/dist/src/schema/index.d.ts

@@ -24,6 +24,7 @@ export { channelSchema } from './channel.schema.js';
 export { credentialSchema } from './credential.schema.js';
 export { deploymentSchema } from './deployment.schema.js';
 export { flagSchema } from './flag.schema.js';
+export { leaseSchema } from './lease.schema.js';
 export { triggerSourceSchema } from './trigger-source.schema.js';
 export { incomingWebhookSchema } from './incoming-webhook.schema.js';
 export { scopeSchema } from './scope.schema.js';

package/dist/src/schema/index.js

@@ -10,6 +10,7 @@ import { scopeSchema } from './scope.schema.js';
 import { flagSchema } from './flag.schema.js';
 import { triggerSourceSchema } from './trigger-source.schema.js';
 import { incomingWebhookSchema } from './incoming-webhook.schema.js';
+import { leaseSchema } from './lease.schema.js';
 import { secretSchema } from './secret.schema.js';
 import { sessionSchema } from './session.schema.js';
 import { webhookSchema } from './webhook.schema.js';
@@ -44,6 +45,7 @@ export const pikkuSchemas = [
     scopeSchema,
     flagSchema,
     triggerSourceSchema,
+    leaseSchema,
     auditSchema,
     analyticsSchema,
     virtualUserSchema,
@@ -56,6 +58,7 @@ export { channelSchema } from './channel.schema.js';
 export { credentialSchema } from './credential.schema.js';
 export { deploymentSchema } from './deployment.schema.js';
 export { flagSchema } from './flag.schema.js';
+export { leaseSchema } from './lease.schema.js';
 export { triggerSourceSchema } from './trigger-source.schema.js';
 export { incomingWebhookSchema } from './incoming-webhook.schema.js';
 export { scopeSchema } from './scope.schema.js';

package/dist/src/schema/lease.schema.d.ts

@@ -0,0 +1,2 @@
+import type { PikkuSchema } from './pikku-schema.types.js';
+export declare const leaseSchema: PikkuSchema;

package/dist/src/schema/lease.schema.js

@@ -0,0 +1,12 @@
+export const leaseSchema = {
+    name: 'lease',
+    ownedBy: ['leaseService', 'workflowService'],
+    statements: [
+        (db) => db.schema
+            .createTable('pikkuLease')
+            .addColumn('key', 'text', (col) => col.primaryKey())
+            .addColumn('holder', 'text', (col) => col.notNull())
+            .addColumn('token', 'integer', (col) => col.notNull())
+            .addColumn('expiresAt', 'bigint', (col) => col.notNull()),
+    ],
+};

package/dist/src/schema/trigger-source.schema.js

@@ -11,7 +11,10 @@ export const triggerSourceSchema = {
             .createTable('pikkuTriggerSource')
             .addColumn('name', 'text', (col) => col.primaryKey())
             .addColumn('kind', 'text', (col) => col.notNull())
+            .addColumn('baseUrl', 'text')
+            .addColumn('labelPrefix', 'text')
             .addColumn('declared', 'boolean', (col) => col.defaultTo(true).notNull())
+            .addColumn('enabled', 'boolean', (col) => col.defaultTo(false).notNull())
             .addColumn('status', 'text')
             .addColumn('state', 'text')
             .addColumn('detail', 'text')

package/dist/src/schema/workflow.schema.js

@@ -53,6 +53,7 @@ export const workflowSchema = {
             .addColumn('retryDelay', 'text')
             .addColumn('fromStepName', 'text')
             .addColumn('currentAttempt', 'integer')
+            .addColumn('leaseExpiresAt', 'bigint')
             .addColumn('createdAt', 'timestamp', (col) => col.defaultTo(sql `CURRENT_TIMESTAMP`).notNull())
             .addColumn('updatedAt', 'timestamp', (col) => col.defaultTo(sql `CURRENT_TIMESTAMP`).notNull())
             .addUniqueConstraint('workflow_step_run_name_unique', [

package/knowledge/decisions/a-lease-is-taken-by-one-guarded-update.md

@@ -0,0 +1,30 @@
+---
+type: decision
+title: A lease is taken by one guarded UPDATE
+description: Acquire inserts the key's row already lapsed, takes it with a guarded UPDATE, and reads the row back, because that is the one shape that behaves the same on SQLite, Postgres and MySQL
+tags: [leases, kysely, mysql]
+---
+
+# A lease is taken by one guarded UPDATE
+
+`acquire` is three statements, in an order that looks roundabout:
+
+1. insert the key's row with `token = 0, expires_at = 0` — already lapsed —
+   ignoring a conflict;
+2. `UPDATE` it to this holder where it is this holder's or has lapsed, raising
+   `token` unless this holder is re-acquiring a live lease;
+3. read the row back and answer whether this holder now has it.
+
+The natural shape, `INSERT … ON CONFLICT DO UPDATE … WHERE`, does not exist on
+MySQL, whose `ON DUPLICATE KEY UPDATE` has no guard. Inserting the row lapsed
+means there is a single path to holding a key — the guarded update — whether the
+key is new or not.
+
+Two MySQL behaviours shape the rest. `SET` is evaluated left to right against
+values already assigned in the same statement, so `token` is assigned before
+`holder`: the re-acquire test has to see the previous holder. And MySQL reports
+rows _changed_, not rows _matched_, so a holder re-acquiring with an unchanged
+row would look like a loss; the row is read back instead of trusting the count.
+
+**What this rules out:** a dialect-specific upsert per database, and deciding
+the outcome from `numUpdatedRows`.

package/knowledge/decisions/index.md

@@ -8,8 +8,11 @@ description: How encrypted columns are wrapped, and which key wraps them
 
 <!-- pikku:knowledge-index -->
 
+- [A lease is taken by one guarded UPDATE](a-lease-is-taken-by-one-guarded-update.md) — Acquire inserts the key's row already lapsed, takes it with a guarded UPDATE, and reads the row back, because that is the one shape that behaves the same on SQLite, Postgres and MySQL
 - [Classification-driven encryption happens in a Kysely plugin](classification-driven-encryption-happens-in-a-plugin.md) — The existing column classification manifest drives a transparent query plugin, rather than encrypting the whole database file or asking call sites to encrypt
 - [KEKs are scoped by purpose as well as version](keks-are-scoped-by-purpose-as-well-as-version.md) — Rows carry a keyId, and the plugin resolves KEKs through a seam rather than holding one; v1 ships a single key, but scoping never becomes a migration
+- [Leases are judged by the database's clock](leases-are-judged-by-the-database-clock.md) — A lease and a step lease are written and compared in SQL against the server's own clock, stored as epoch milliseconds, so no worker's clock decides who holds anything
 - [The plugin is given a resolver, never a key](the-plugin-is-given-a-resolver-never-a-key.md) — Assembly order for classified columns — a plain Kysely handle, then a plugin that asks a `KEKResolver` per operation rather than being handed a key it holds for the life of the process
+- [Workflows own the lease table too](workflows-own-the-lease-table-too.md) — `leaseSchema` is owned by `workflowService` as well as `leaseService`, because the engine takes the run lease without any function ever naming leaseService
 
 <!-- /pikku:knowledge-index -->

package/knowledge/decisions/leases-are-judged-by-the-database-clock.md

@@ -0,0 +1,42 @@
+---
+type: decision
+title: Leases are judged by the database's clock
+description: A lease and a step lease are written and compared in SQL against the server's own clock, stored as epoch milliseconds, so no worker's clock decides who holds anything
+tags: [leases, workflows, kysely, clocks]
+---
+
+# Leases are judged by the database's clock
+
+A lease is only as good as the clock that decides it has lapsed. When each
+worker computed `new Date(Date.now() + ttl)` and compared against its own
+`Date.now()`, a worker whose clock ran an hour fast saw every live lease as
+expired and took it — a lease and a step claim both — and wrote leases that
+outlived their ttl by the same hour for everyone else.
+
+So both leases are computed in SQL: `expires_at` is set to _now + ttl_ and
+compared against _now_, where _now_ is the database's —
+`clock_timestamp()` on Postgres, `utc_timestamp(6)` on MySQL. There is one clock,
+and every worker reads it. The columns are `bigint` epoch milliseconds rather
+than `timestamp`, because the arithmetic then means the same thing in every
+dialect and no driver gets to reinterpret a time zone.
+
+On MySQL the epoch is counted from `utc_timestamp`, never
+`unix_timestamp(now())`. `now()` is local to the session's time zone, and when
+that zone falls back an hour, two real instants share one local time: the
+epoch stood still for an hour, then jumped, so a dead holder's lease lived an
+hour longer and every lease written in the repeated hour lapsed at once.
+
+`PgKyselyLeaseService`, `MySQLKyselyLeaseService` and the dialect workflow
+services override `nowMs()`; `KyselyLeaseService` and `KyselyWorkflowService`
+keep the process clock, which is only correct when every process shares a host,
+as they do on SQLite.
+
+**What this rules out:** comparing a lease in JavaScript after reading it — the
+claim used to pre-check `isStepLeaseLive` in the app, and that check is exactly
+where a skewed worker decided wrongly.
+
+**Testing it:** PGlite runs in-process and takes its clock from the JS
+`Date.now`, so skewing `Date.now` skews the database too and a skew test on
+PGlite proves nothing. The skew assertions live in
+`verifiers/workflows/src/tests/lease-db.assert.ts`, against real Postgres and
+MySQL servers.

package/knowledge/decisions/workflows-own-the-lease-table-too.md

@@ -0,0 +1,22 @@
+---
+type: decision
+title: Workflows own the lease table too
+description: `leaseSchema` is owned by `workflowService` as well as `leaseService`, because the engine takes the run lease without any function ever naming leaseService
+tags: [leases, workflows, schema]
+---
+
+# Workflows own the lease table too
+
+`pikku db generate` writes a schema for a project only when a service in
+`ownedBy` is one the inspector found the project's code reaching. The workflow
+engine takes `workflow-run:<id>` from `leaseService` on every orchestration pass,
+but no function destructures `leaseService` to make that happen, so a workflow
+project was never given `pikku_lease` — and registering a lease service then
+failed at boot against a table nothing had migrated.
+
+`leaseSchema` is therefore owned by `workflowService` as well. A workflow project
+gets the table whether or not it registers a lease service yet.
+
+**What this rules out:** making `leaseService` a required service of every
+workflow project. That would force the in-memory and single-process apps, which
+have nothing to serialise, to wire a lease they cannot use.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pikku/kysely",
-  "version": "0.13.30",
+  "version": "0.13.33",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/pikkujs/pikku.git",
@@ -26,7 +26,7 @@
     "prepublishOnly": "bun run build"
   },
   "peerDependencies": {
-    "@pikku/core": "^0.12.128"
+    "@pikku/core": "^0.12.133"
   },
   "dependencies": {
     "kysely": "^0.29.5"