@pikku/kysely 0.13.4 → 0.13.6

package/CHANGELOG.md

@@ -1,3 +1,109 @@
+## 0.13.6
+
+### Patch Changes
+
+- 8a2c993: Make the workflow service cheaper to run, and fix two ways it lost state.
+
+  The SQL workflow tables had no indexes at all, so every step read, every
+  history walk and every orchestrator tick was a sequential scan; five indexes
+  now cover the columns the engine actually queries by. A replay used to ask for
+  each step's row individually — O(N) reads per replay, O(N^2) over a run — and
+  now takes one read of the run's steps and serves the walk from it. A step
+  transition wrote the step row and its history row as two separate statements,
+  so a crash between them left a step saying `succeeded` whose history still said
+  `running`; both halves are now one transaction, and the history row is found by
+  attempt number rather than by sorting on `created_at`, which two attempts can
+  share. Resolving a dynamic workflow read and parsed every AI-generated workflow
+  in the deployment to `.find()` one by name; it is a point lookup now.
+
+  Waiting on a run no longer polls at a fixed interval. `pollIntervalMs` became a
+  ceiling rather than a cadence: polling starts at 10ms and widens towards it, so
+  a workflow that finishes in milliseconds is no longer held for a full second,
+  and a long-running one is not read at full rate for its whole life.
+
+  Two backend-specific defects: Redis kept a run's state as one JSON blob and
+  read-modified-wrote it, so parallel branches setting different variables
+  overwrote each other — state is a field per variable now, with the old blob
+  still read underneath so runs in flight keep what they had. Mongo's
+  `setStepScheduled` never wrote history, leaving a queued step reading as never
+  dispatched.
+
+  Also: dispatch no longer JSON round-trips every step payload before handing it
+  to a queue that serialises it anyway — the in-process dev queue, which is the
+  only one that was relying on it, does it itself now.
+
+  Two more defects. A transition whose step had no live attempt wrote its status
+  to the step row and silently nothing to history — the exact divergence the
+  transaction exists to prevent — and now repairs the step and writes the
+  missing row. And resolving a dynamic workflow was non-deterministic on all
+  three backends: a name can hold several active versions, and none of them
+  ordered the candidates, so which one ran could change between two calls
+  reading identical data. The newest version wins, with the graph hash breaking
+  a tie.
+
+  The two attempt columns and the five indexes are declared in the workflow
+  schema, so a fresh database gets them at boot. An existing one gets them from
+  a migration — `pikku db generate` writes the declaration down — rather than
+  from DDL issued at boot.
+
+- a261006: **Breaking:** removed dynamic workflows — runtime-defined workflow graphs stored in the database and resolved by name instead of by codegen.
+
+  The feature was already half-gone. Its authoring surface (`createAgentWorkflow`, `saveAgentWorkflow`, `listAgentWorkflows`, `executeAgentWorkflow`, and the AI-agent instruction builder) was deleted in April 2026 along with its entire e2e suite, and nothing has written a dynamic workflow since. What remained could not execute one either: `executeAgentWorkflow` gated on `pikkuState('workflows', 'meta')`, which only codegen ever populates, so a graph that existed solely in the database was never findable. The two backend families had also drifted onto different `source` sentinels (`'ai-agent'` vs `'dynamic-workflow'`), and the two Redis implementations disagreed on key escaping — so at least one of them matched nothing. Rather than keep shipping plumbing for a path no caller could complete, it is removed until it can be reintroduced deliberately.
+
+  Removed:
+  - `getAIGeneratedWorkflows` from `WorkflowService` and `WorkflowRunService`, and from every backend (in-memory, Redis, MongoDB, Kysely, and the Cloudflare Durable Object service and client — the last two were already a `return []` stub and a rejection).
+  - The database-lookup fallbacks in `startWorkflow` and `runWorkflowJob` that resolved a workflow name against stored graphs when static meta had no match.
+  - `'dynamic-workflow'` from the `WorkflowRuntimeMeta['source']` union.
+  - `validateWorkflowWiring` and `computeEntryNodeIds` from `@pikku/core/workflow`. These validated AI-authored graphs and had no callers in core; the inspector keeps its own private entry-node computation for static graph wiring, which is unaffected.
+  - The `workflow-created` AI stream event and its AG-UI `pikku:workflow-created` custom event. Its only emitter went with the April deletion, so it could never fire.
+  - The console's `console:getAIWorkflows` RPC, the `useAIWorkflows` hook, the "Dynamic" workflow filter and badge, and the trigger-schema scraper that derived an input form from a stored graph's `$ref` bindings.
+
+  Kept, because static graph workflows depend on them and this is not a change to versioning:
+  - `upsertWorkflowVersion`, `getWorkflowVersion`, `updateWorkflowVersionStatus`, and the `workflowVersions` storage in every backend. These back version-mismatch replay: when a deployed graph's hash changes, in-flight runs continue against the exact graph they started on. No schema migration is needed — the table, its columns, and its `(workflowName, graphHash)` upsert key are unchanged.
+  - `generateMermaidDiagram`, which renders any workflow graph and is not specific to dynamic ones.
+
+  Static `pikkuWorkflowGraph` and DSL workflows are entirely unaffected: they resolve from codegen'd meta, which was always the only path that worked.
+
+  To revive this post-MVP, the deleted authoring code is recoverable in full — its prompt engineering (a compact tool table upfront, full schemas with flattened dotted output paths returned only after a validation failure) is worth reading before rewriting:
+
+  ```
+  git show f52f3308b^:packages/core/src/wirings/ai-agent/agent-dynamic-workflow.ts
+  git show f52f3308b^:packages/core/src/wirings/workflow/graph/graph-validation.ts
+  git show f52f3308b --stat   # the April removal, incl. the three e2e feature files
+  ```
+
+  Note that reviving it needs more than restoring those files: the queued-step path (`executeWorkflowStep`), `onError` compensation, and sub-workflow resolution all read static meta only and would need a fallback for a graph that exists solely in the database.
+
+- Updated dependencies [8a2c993]
+- Updated dependencies [a261006]
+- Updated dependencies [09973b9]
+  - @pikku/core@0.12.71
+
+## 0.13.5
+
+### Patch Changes
+
+- 220a8c4: fix: `ensurePikkuSchema` now sees tables on a `withSchema(...)`-bound connection
+
+  It read the table name out of its own compiled DDL with a regex that matched the
+  first quoted identifier. On a connection bound to a schema that DDL is
+  `create table "app"."workflow_runs"`, so it captured `app` — never a real table
+  name — concluded every table was missing, and issued a bare `create table` that
+  the database rejected with `relation "workflow_runs" already exists`, on every
+  boot.
+
+  The schema half of the name is now parsed, and the lookup matches on the pair
+  when the DDL is qualified (falling back to the bare name when it is not, since
+  an unqualified statement resolves against a `search_path` that is not knowable
+  from here). Error messages name the schema too.
+
+  Also: applying a schema over a `withSchema(...)`-bound **sqlite** connection now
+  explains itself. `withSchema` qualifies foreign key targets along with everything
+  else — which postgres requires and sqlite refuses, since a `REFERENCES` clause
+  there takes a bare table name — and all the engine says about it is
+  `near ".": syntax error`. The failure is now reported with the schema that
+  produced it and what to do instead, with the engine error kept as the `cause`.
+
 ## 0.13.4
 
 ### Patch Changes

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 - present Yasser Fadl and Pikku contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,30 @@
-# @pikku/pino
+# @pikku/kysely
 
-This package is part of @pikku/mono and is responsible for providing the Pino Logger
+Kysely-backed implementations of the Pikku service interfaces — workflow state,
+secrets, credentials, sessions, scopes, channel/event-hub stores and audit.
+
+Dialect-neutral. Pair it with a dialect package (`@pikku/kysely-postgres`,
+`@pikku/kysely-mysql`, `@pikku/kysely-sqlite`) where the SQL differs.
+
+## Install
+
+```bash
+npm install @pikku/kysely kysely
+```
+
+## Usage
+
+```typescript
+import { KyselyWorkflowService, KyselySecretService } from '@pikku/kysely'
+import type { KyselyPikkuDB } from '@pikku/kysely'
+import type { Kysely } from 'kysely'
+
+declare const db: Kysely<KyselyPikkuDB>
+
+const workflowService = new KyselyWorkflowService(db)
+const secretService = new KyselySecretService(db, { key: encryptionKey })
+```
+
+## Docs
+
+https://pikku.dev/docs

package/dist/src/kysely-tables.d.ts

@@ -44,6 +44,8 @@ export interface WorkflowStepTable {
     retries: number | null;
     retryDelay: string | null;
     fromStepName: string | null;
+    /** The history attempt currently in flight; see WorkflowStepHistoryTable.attempt. */
+    currentAttempt: number | null;
     createdAt: Generated<Date>;
     updatedAt: Generated<Date>;
 }
@@ -53,6 +55,12 @@ export interface WorkflowStepHistoryTable {
     status: StepStatus;
     result: string | null;
     error: string | null;
+    /**
+     * Monotonic attempt number within the step, starting at 1. Orders attempts
+     * and identifies the live one without relying on `createdAt`, which cannot
+     * separate a retry from the attempt it replaces in the same millisecond.
+     */
+    attempt: number | null;
     createdAt: Generated<Date>;
     runningAt: Date | null;
     scheduledAt: Date | null;

package/dist/src/kysely-workflow-mirror.js

@@ -77,11 +77,12 @@ export class KyselyWorkflowMirror {
             retries: step.retries ?? null,
             retryDelay: step.retryDelay?.toString() ?? null,
             fromStepName: step.fromStepName ?? null,
+            currentAttempt: 1,
             createdAt: now,
             updatedAt: now,
         })
             .execute();
-        await this.insertHistoryRecord(step.stepId, step.status);
+        await this.insertHistoryRecord(step.stepId, step.status, 1);
     }
     async setStepRunning(stepId) {
         await this.updateStepStatus(stepId, 'running');
@@ -126,17 +127,24 @@ export class KyselyWorkflowMirror {
         await this.appendOrUpdateLatestHistory(stepId, 'failed', undefined, errorJson);
     }
     async createRetryAttempt(failedStepId, newStep) {
+        const previous = await this.db
+            .selectFrom('workflowStep')
+            .select('currentAttempt')
+            .where('workflowStepId', '=', failedStepId)
+            .executeTakeFirst();
+        const attempt = (previous?.currentAttempt ?? 0) + 1;
         await this.db
             .updateTable('workflowStep')
             .set({
             status: newStep.status,
             result: null,
             error: null,
+            currentAttempt: attempt,
             updatedAt: new Date(),
         })
             .where('workflowStepId', '=', failedStepId)
             .execute();
-        await this.insertHistoryRecord(failedStepId, newStep.status);
+        await this.insertHistoryRecord(failedStepId, newStep.status, attempt);
     }
     async setBranchTaken(stepId, branchKey) {
         await this.db
@@ -192,12 +200,13 @@ export class KyselyWorkflowMirror {
             .where('workflowStepId', '=', stepId)
             .execute();
     }
-    async insertHistoryRecord(stepId, status, resultJson, errorJson) {
+    async insertHistoryRecord(stepId, status, attempt, resultJson, errorJson) {
         const now = new Date();
         const values = {
             historyId: crypto.randomUUID(),
             workflowStepId: stepId,
             status,
+            attempt,
             result: resultJson ?? null,
             error: errorJson ?? null,
             createdAt: now,
@@ -219,30 +228,33 @@ export class KyselyWorkflowMirror {
      * returns the same sequence the executor would have produced.
      */
     async appendOrUpdateLatestHistory(stepId, status, resultJson, errorJson) {
-        const latest = await this.db
-            .selectFrom('workflowStepHistory')
-            .select('historyId')
+        const update = { status };
+        if (resultJson !== undefined)
+            update.result = resultJson;
+        if (errorJson !== undefined)
+            update.error = errorJson;
+        const tsField = timestampFieldFor(status);
+        if (tsField)
+            update[tsField] = new Date();
+        const result = await this.db
+            .updateTable('workflowStepHistory')
+            .set(update)
             .where('workflowStepId', '=', stepId)
-            .orderBy('createdAt', 'desc')
-            .limit(1)
+            // Correlate through the step row rather than sorting this table by
+            // createdAt: two attempts can land in the same millisecond, and the
+            // step row already knows which attempt is current.
+            .where('attempt', '=', (eb) => eb
+            .selectFrom('workflowStep')
+            .select('currentAttempt')
+            .where('workflowStepId', '=', stepId))
             .executeTakeFirst();
-        if (latest) {
-            const update = { status };
-            if (resultJson !== undefined)
-                update.result = resultJson;
-            if (errorJson !== undefined)
-                update.error = errorJson;
-            const tsField = timestampFieldFor(status);
-            if (tsField)
-                update[tsField] = new Date();
-            await this.db
-                .updateTable('workflowStepHistory')
-                .set(update)
-                .where('historyId', '=', latest.historyId)
-                .execute();
-        }
-        else {
-            await this.insertHistoryRecord(stepId, status, resultJson, errorJson);
+        if (Number(result?.numUpdatedRows ?? 0n) === 0) {
+            const step = await this.db
+                .selectFrom('workflowStep')
+                .select('currentAttempt')
+                .where('workflowStepId', '=', stepId)
+                .executeTakeFirst();
+            await this.insertHistoryRecord(stepId, status, step?.currentAttempt ?? 1, resultJson, errorJson);
         }
     }
 }

package/dist/src/kysely-workflow-run-service.d.ts

@@ -24,11 +24,6 @@ export declare class KyselyWorkflowRunService implements WorkflowRunService {
         graph: any;
         source: string;
     } | null>;
-    getAIGeneratedWorkflows(agentName?: string): Promise<Array<{
-        workflowName: string;
-        graphHash: string;
-        graph: any;
-    }>>;
     deleteRun(id: string): Promise<boolean>;
     private mapRunRow;
 }

package/dist/src/kysely-workflow-run-service.js

@@ -143,6 +143,7 @@ export class KyselyWorkflowRunService {
             'h.status',
             'h.result',
             'h.error',
+            'h.attempt',
             'h.createdAt',
             'h.runningAt',
             'h.scheduledAt',
@@ -150,7 +151,10 @@ export class KyselyWorkflowRunService {
             'h.failedAt',
         ])
             .where('s.workflowRunId', '=', runId)
+            // `attempt` breaks the tie when a retry lands in the same millisecond as
+            // the attempt it replaces, which `createdAt` alone cannot order.
             .orderBy('h.createdAt', 'asc')
+            .orderBy('h.attempt', 'asc')
             .execute();
         let attemptCounters = {};
         return result.map((row) => {
@@ -206,22 +210,6 @@ export class KyselyWorkflowRunService {
             source: row.source,
         };
     }
-    async getAIGeneratedWorkflows(agentName) {
-        let query = this.db
-            .selectFrom('workflowVersions')
-            .select(['workflowName', 'graphHash', 'graph'])
-            .where('source', '=', 'dynamic-workflow')
-            .where('status', '=', 'active');
-        if (agentName) {
-            query = query.where('workflowName', 'like', `ai:${agentName}:%`);
-        }
-        const rows = await query.execute();
-        return rows.map((row) => ({
-            workflowName: row.workflowName,
-            graphHash: row.graphHash,
-            graph: parseJson(row.graph),
-        }));
-    }
     async deleteRun(id) {
         const result = await this.db
             .deleteFrom('workflowRuns')

package/dist/src/kysely-workflow-service.d.ts

@@ -1,6 +1,6 @@
 import type { SerializedError } from '@pikku/core';
 import { PikkuWorkflowService, type WorkflowPlannedStep, type WorkflowQueueOptions, type WorkflowRun, type WorkflowRunWire, type StepState, type StepStatus, type WorkflowStatus, type WorkflowVersionStatus } from '@pikku/core/workflow';
-import type { Kysely } from 'kysely';
+import { type Kysely } from 'kysely';
 import type { KyselyPikkuDB } from './kysely-tables.js';
 export declare class KyselyWorkflowService extends PikkuWorkflowService {
     protected db: Kysely<KyselyPikkuDB>;
@@ -19,11 +19,26 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
         retryDelay?: string | number;
     }, fromStepName?: string): Promise<StepState>;
     getStepState(runId: string, stepName: string): Promise<StepState>;
+    protected listStepStates(runId: string): Promise<Array<StepState & {
+        stepName: string;
+    }>>;
     getRunHistory(runId: string): Promise<Array<StepState & {
         stepName: string;
     }>>;
     protected setStepRunningImpl(stepId: string): Promise<void>;
     protected setStepScheduledImpl(stepId: string): Promise<void>;
+    /**
+     * Move a step and its current history attempt to the same status in one
+     * transaction.
+     *
+     * Both halves matter: a crash between them used to leave the step row saying
+     * `succeeded` while its history row still said `running`, which silently
+     * corrupts `getRunHistory` and every timeline reconstruction built on it.
+     * The history row is targeted by its `attempt` rather than by a `LIMIT 1`
+     * over `created_at`, so a retry that lands in the same millisecond as the
+     * attempt it replaces still resolves the newer row.
+     */
+    private writeStepTransition;
     private insertHistoryRecord;
     private getTimestampFieldForStatus;
     protected setStepChildRunIdImpl(stepId: string, childRunId: string): Promise<void>;
@@ -45,6 +60,25 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
     }>>;
     getNodeResults(runId: string, nodeIds: string[]): Promise<Record<string, any>>;
     protected setBranchTakenImpl(stepId: string, branchKey: string): Promise<void>;
+    /**
+     * Merge one key into the run's JSON state, as a single expression evaluated
+     * by the database.
+     *
+     * The SQL is dialect-specific, so subclasses override this rather than the
+     * caller. The default is the SQLite form; MySQL and Postgres differ in how
+     * they cast a JSON literal.
+     *
+     * @param path - JSON path to the key, already quoted (`$."key"`)
+     * @param json - The value as JSON text
+     */
+    protected jsonSetState(path: string, json: string): import("kysely").RawBuilder<string>;
+    /**
+     * A JSON path for an arbitrary key. State keys are user-supplied (a graph's
+     * `setState` name, an approval's hex-encoded reason), so the key is quoted
+     * rather than interpolated bare — an unquoted `$.a.b` would silently address
+     * a nested path instead of the key literally called `a.b`.
+     */
+    private jsonPathFor;
     protected updateRunStateImpl(runId: string, name: string, value: unknown): Promise<void>;
     getRunState(runId: string): Promise<Record<string, unknown>>;
     protected upsertWorkflowVersionImpl(name: string, graphHash: string, graph: any, source: string, status?: WorkflowVersionStatus): Promise<void>;
@@ -53,10 +87,5 @@ export declare class KyselyWorkflowService extends PikkuWorkflowService {
         graph: any;
         source: string;
     } | null>;
-    getAIGeneratedWorkflows(agentName?: string): Promise<Array<{
-        workflowName: string;
-        graphHash: string;
-        graph: any;
-    }>>;
     close(): Promise<void>;
 }