@pikku/mongodb 0.12.11 → 0.12.13

package/CHANGELOG.md

@@ -1,5 +1,411 @@
 # @pikku/mongodb
 
+## 0.12.13
+
+### Patch Changes
+
+- 6a307f0: Fix four latent correctness bugs in the function, RPC and error runtimes, and
+  remove dead code from the workflow service surface.
+
+  `WorkflowService.getNodesWithoutSteps` is gone. It was declared on the abstract
+  service and implemented by all five storage backends, and nothing ever called
+  it — hence the non-core packages in this changeset, which only lose that method.
+
+  **An RPC could execute twice.** `RPCService.invoke`, its addon path, and
+  `rpcWithWire` each wrapped the _execution_ of a resolved function in a `try`
+  whose `catch` treated `RPCNotFoundError` as "not found locally" and re-dispatched
+  the call through `deploymentService`. A nested `rpc.invoke` to a missing name,
+  raised from inside an already-running function, therefore re-ran that function on
+  a remote instance after its local side effects had already committed. Resolution
+  is now separated from execution on all three paths, so only a genuinely
+  unresolvable name reaches the fallback. What escapes to callers is unchanged.
+
+  **`addonNamespace` leaked between sibling calls.** The function runner's
+  middleware path restored `rpc`, `functionId`, `audit` and `addonNamespace` after
+  an invocation; the non-middleware path restored the first three but not the
+  fourth. A call into an addon function with no middleware left the addon's
+  namespace on the wire, so subsequent sibling calls resolved the wrong
+  per-instance singletons and `credentialOverrides`.
+
+  **Errors registered on a subclass never resolved.** `misc.errors` was typed
+  `Map<PikkuError, ErrorDetails>` — instances — while `addError` stores
+  constructors, a mismatch hidden by its `error: any` parameter. The instance
+  lookup in `getErrorResponse` was consequently dead, and lookup fell through to a
+  scan comparing `constructor.name`, so a subclass of a registered error got no
+  status mapping at all. The map is now typed `Map<PikkuErrorConstructor, …>` and
+  lookup walks the prototype chain first. Name matching is retained, deliberately,
+  as the fallback that keeps error mapping working when two copies of
+  `@pikku/core` are installed.
+
+  **`createWeakUID` collided across instances.** The prefix was
+  `Date.now().toString(36)` evaluated at module load, so any two instances loading
+  the module in the same millisecond emitted identical `channelId` and `requestId`
+  sequences — reproducibly, not just in principle. It is now seeded lazily from
+  `crypto.randomUUID()`.
+
+  Also: `pikkuState` keys its global map with `Symbol.for` rather than `Symbol`, so
+  two copies of the package share registrations instead of silently getting
+  disjoint state; and the local channel upgrade path no longer keys its middleware
+  cache on the raw request path, which grew the cache without bound while the
+  cached value never varied by path.
+
+- afef587: Close eleven security weaknesses found in a review of `@pikku/core`. Most are
+  breaking, and two invalidate data or credentials already in the wild — read the
+  migration notes before upgrading.
+
+  **Breaking: AI agent thread ownership now fails closed.** Reading, listing,
+  resuming or approving an existing thread or run requires a resolved session
+  principal (`userId`, or `orgId` for `sessionScope: 'org'`), regardless of the
+  agent's `auth` setting. Previously a request without a session had no ownership
+  model at all: the caller-supplied `resourceId` was accepted as the ownership
+  key, so any caller could read or resume another party's thread by naming its
+  `resourceId`. Worse, `threadOwnerConstraint` returned `undefined` for a
+  sessionless caller, and `undefined` means _no filter_ rather than _no rows_ —
+  so `getAgentThreads` returned every thread in the deployment. It now returns
+  `string[]`, empty for a sessionless caller, which every storage backend already
+  treats as matching nothing. Sessionless agents still run one-shot conversations,
+  each with a fresh unguessable owner; what they lose is cross-request continuity.
+  Wire a session to restore it.
+
+  **Breaking: stored secrets and credentials must be re-entered.** `deriveKey` ran
+  a single unsalted round of SHA-256 over the passphrase and used the digest
+  directly as the AES-GCM key — roughly one hash per brute-force guess, with one
+  rainbow table working against every deployment. It is now PBKDF2-HMAC-SHA256 at
+  600,000 iterations over a random salt. There is no compatibility path: every
+  value held by the kysely, mongodb and redis secret services and the kysely
+  credential service becomes undecryptable. They fail loud, naming the key and
+  `key_version`, so the app hard-fails on first secret read until each is re-set.
+
+  The KEK salt is scoped to the key version and stored alongside it, rather than
+  per secret, so a bulk read costs one derivation instead of N — `getSecrets` over
+  50 secrets went from ~2.3s to ~48ms, and rotation from ~4.6s to ~94ms. This adds
+  a salt table (kysely), hash field (redis) or collection (mongodb), created
+  automatically on first use.
+
+  **Breaking: `PIKKU_REMOTE_SECRET` must be at least 32 characters.** The
+  remote-RPC session envelope moved from PBKDF2 to HKDF, which expands
+  high-entropy key material rather than stretching a low-entropy passphrase. That
+  took a remote hop from ~269ms to ~0.4ms — PBKDF2 was running twice per request —
+  but HKDF supplies no brute-force resistance, so the secret must carry the
+  entropy itself. A shorter secret now throws `WeakKeyMaterialError` at both ends.
+  Generate one with `openssl rand -base64 32` and roll it out to every service in
+  the mesh together: existing bearer tokens are format-incompatible, so a partial
+  rollout produces 401s until every instance is updated. The Cloudflare, Lambda
+  and Azure deployment services each hand-rolled a copy of `buildRemoteHeaders`
+  and now call the shared one, which is what keeps the two sides in step.
+
+  **Breaking: previously signed content URLs stop verifying.** `LocalContent`
+  signed only `{signedAt, expiresAt, notBefore}`, so a signature proved when a URL
+  was issued but never what it was issued for — any valid token was a skeleton
+  key, and swapping the pathname from a public thumbnail to a private document
+  still verified. The signature now binds the request path. Separately, the
+  verifier returned "valid" when no JWT service was wired, which is how
+  `pikku serve` ran: a forged `?signedAt=0&expiresAt=99999999999999` was accepted.
+  It now rejects with 403, `LocalContent` requires a `JWTService`, and
+  `pikku serve`/`pikku dev` mint an ephemeral per-process signing key so local
+  development works without shipping a fail-open path. In-flight signed URLs must
+  be re-issued.
+
+  **Request body size limits now apply to every adapter.** The `maxBodySize` cap
+  existed only in `PikkuFetchHTTPRequest`. The real hole was uWebSockets, which
+  drove `res.onData` itself and concatenated every chunk with no bound and nothing
+  downstream able to intervene; it now drops chunks past the limit and replies 413
+  before routing. Fastify delegates to its native `bodyLimit` (set only when
+  `maxBodySize` is configured, so fastify's stricter 1 MB default is never
+  loosened), and `PikkuExpressServer` feeds the limit into its body parsers. Two
+  paths can only reject rather than prevent, and are documented as such:
+  `express-middleware` mounted on your own app receives an already-parsed body, so
+  that deployment must bound its own parser; Next server actions bottom out at
+  `experimental.serverActions.bodySizeLimit`.
+
+  **Breaking: the console addon's privileged functions are gated by default.**
+  `wireAddon` gains a `scopes` option that applies to every function in the
+  addon's namespace, and the console scaffold now generates
+  `wireAddon({ name: 'console', package: '@pikku/addon-console', scopes: ['admin'] })`.
+  Previously the console's entire privileged surface — around 54 functions
+  including `credentialGet`, which returns a resolved OAuth token for an arbitrary
+  `userId`, `updateFunctionBody`, and `installAddon`, which shells out to a
+  package install — was protected only by an optional host-registered
+  `addGlobalPermission`. `resolveGlobalPermissions` returns `[]` when none is
+  registered and permission checking then no-ops, so an app that never registered
+  one served those functions to anyone, and with the template's default
+  `scaffold.rpc: "no-auth"` that meant unauthenticated. All of them now return 403
+  `MissingScopeError` without an `admin`, `admin:*` or `*` scope. **Regenerating
+  is required** — an app holding an old `console.gen.ts` stays open.
+  `installAddon` and `installOpenapiAddon` additionally declare their own
+  `auth: true, scopes: ['admin']`, and `getAgentThreads` now scopes its listing to
+  the session's own threads unless the caller holds admin.
+
+  Addon scopes are enforced in `runPikkuFunc` rather than at the RPC boundary,
+  because a wiring can reference an addon function directly — the inspector
+  records the addon's `packageName` on HTTP, channel, schedule, queue, CLI,
+  trigger, gateway and MCP wirings — and those paths never call `resolveNamespace`.
+  Enforcing at the RPC seam would have covered only the `namespace:function` form
+  while reading as complete.
+
+  **Breaking: `wireAddon`'s `auth` and `tags` now apply on direct wirings too.**
+  Both were read only by `resolveAddonFunction`, so they had exactly the hole
+  scopes had: `wireAddon({ name: 'console', package: '@pikku/addon-console', auth:
+true, tags: ['admin'] })` gated `rpc('console:credentialGet')` and gated nothing
+  at all on an HTTP route wired straight to `credentialGet`. A consumer who
+  reached for the documented way to lock an addon down got a control that was
+  silently inert on every wiring except one. Both now resolve in `runPikkuFunc`.
+  `auth` merges as an OR — `auth: false` from an addon is ignored on a direct
+  wiring, because an addon may require a session the wiring did not but must never
+  waive one it did. Addon tags resolve to concrete middleware against the **root**
+  tag groups before the call rather than being folded into the function's
+  inherited middleware: `addTagMiddleware('admin', …)` is written by the consuming
+  app and registers under the root package, whereas `combineMiddleware` would look
+  the tag up under the addon's own `packageName`, where it does not exist.
+
+  One consequence worth naming: an addon that wires `auth: true` and also runs its
+  own sessionless internal work — a scheduled task or queue worker inside the
+  addon calling a sibling function — is now gated where it previously was not,
+  because a bare `rpc('fn')` from inside the addon reaches `runPikkuFunc` with the
+  addon's `packageName` like any other call. This reverses a decision that
+  deliberately scoped the gate to the namespaced boundary; that reasoning held
+  only while the boundary was real, and a direct wiring can enter an addon without
+  crossing it, so "already inside" is not something the runtime can infer. Such an
+  addon should carry authorization on the function via
+  `pikkuFunc({ permissions })`, which has always been enforced on every path. A
+  follow-up will add execution provenance so an intra-addon call can be
+  distinguished from an external one and skip the addon-level check; that needs a
+  marker no caller outside the process can set, which is its own design problem
+  and does not belong in a security fix.
+
+  **Codegen now warns when an exposed function has no gate (PKU574).** The
+  generated `POST /rpc/:rpcName` dispatcher forwards to `rpc.exposed`, which
+  refuses anything without `expose: true` — but nothing checked whether the
+  target was gated, because a dispatcher cannot know what it dispatches to. The
+  console shipped ~54 privileged functions through that gap and the toolchain was
+  silent. The inspector now reports every function that is exposed, sessionless,
+  and carries no `auth`, `scopes` or `permissions` of its own and none from a
+  governing `wireAddon`. It is a `warn`, not a critical: `expose: true` on an
+  ungated sessionless function is correct for a genuinely public endpoint, so it
+  blocks a build only under `--fail-on-warn`.
+
+  Two pieces of metadata were missing for this to be answerable statically, and
+  both are now recorded. A `pikkuSessionlessFunc`'s own `auth: true` was read at
+  runtime but never written to function meta, so a self-gated function was
+  indistinguishable from an ungated one — `sessionless` carries the baseline, and
+  `auth` now carries the tightening. And `wireAddon`'s `scopes`, `auth` and `tags`
+  were not parsed at all: the inspector recorded the addon's `rpcEndpoint` and its
+  secret, variable and credential overrides, and dropped every one of its gates.
+  An addon whose gates are not statically knowable is treated as gated, because a
+  false positive on a correctly-secured addon costs more than the one case it
+  would catch.
+
+  **Breaking: an application's global permissions now apply inside addons.**
+  `resolveGlobalPermissions` read only the bucket matching the function's own
+  package, but the generated `addGlobalPermission` wrapper takes no package
+  argument and always registers under the root. An app-wide rule like "every
+  request needs a signed-in user" therefore stopped at the addon boundary, and the
+  bucket an addon's functions did read was one no host could write to — which is
+  why the console addon's recommended `addGlobalPermission([isAdmin],
+'@pikku/addon-console')` was never a gate anybody could actually install. A
+  function now resolves the root bucket and its own package's, root first.
+  Unioning is safe in a way nothing else here would be: globals AND, so adding the
+  root ones can only tighten. Package buckets stay one-way — a package's globals
+  never reach root functions, or an installed addon could gate the whole
+  application. Apps with both a root global and addon-provided functions will see
+  those functions gated where they previously were not.
+
+  **Codegen now records whether each HTTP route requires a session, and warns
+  about inert addon tags (PKU575).** Four separate things can demand a session —
+  the function's `sessionless`, its own `auth`, the route's (or its group's)
+  `auth`, and the addon it belongs to — and answering "which routes are open?"
+  meant joining all four by hand and knowing which wins. Each route's meta now
+  carries the resolved `requiresSession` alongside the route's own `auth`. Scopes
+  count as requiring one, since they are matched against the session's and fail
+  closed. Anything not statically knowable resolves to `true`, matching PKU574:
+  a route that looks stricter than it is costs less than one that looks open and
+  isn't. Separately, `wireAddon({ tags: ['admin'] })` reads like a gate and is
+  applied like one right up until no `addTagMiddleware('admin', …)` exists, at
+  which point it resolves to an empty list and gates nothing; that now warns.
+  Only addon tags are reported — a tag on a function is as likely to be
+  organizational, and warning about those would bury the case that matters.
+
+  **Object-shorthand permissions were missing from meta.** The inspector visited
+  `ts.PropertyAssignment` but not `ts.ShorthandPropertyAssignment`, so
+  `permissions: { canAdminOrg }` — enforced identically to the longhand form at
+  runtime, since `verifyPermissions` has a non-array branch — was recorded as _no
+  permissions at all_. That is the most dangerous direction for meta to be wrong
+  in: an audit reading it sees an open door where one is shut. It cost this review
+  a false IDOR report across ~35 billing and org functions before the source
+  contradicted the metadata.
+
+  **Functions that authorize in their own body can say so.** A webhook receiver
+  verifying a signature, or a handler redeeming a signed invite, is genuinely
+  closed while carrying no session, scope or permission — indistinguishable in
+  meta from one nobody remembered to gate, and so warned about forever by PKU574.
+  `selfAuthenticated: true` on the function config records the claim and silences
+  the warning for that function. It is declarative only: nothing at runtime reads
+  it and it grants nothing. Detection was rejected deliberately — inferring it
+  from the body means a function that _looks_ like it checks something silences
+  the warning while checking nothing, and a warning that is usually wrong stops
+  being read.
+
+  **Breaking: `scaffold.<feature>` is now `boolean | { auth?, path? }`, and `true`
+  means authenticated.** The old `'auth' | 'no-auth' | false` read like a
+  starter-file preference while being a live authorization decision, set three
+  directories from the functions it governed — the shape the console incident
+  took. A surface now becomes public only by writing `{ auth: false }`, so
+  omitting a field can never open anything: the failure mode of a forgotten flag
+  is a locked door. `{ path }` additionally overrides where the file is
+  generated, which previously could only be set for all features at once via
+  `pikkuDir`.
+
+  The legacy strings are **refused, not coerced**. `resolveScaffoldFeature` throws
+  naming the key and its replacement (`"rpc": "no-auth"` → `"rpc": { "auth":
+false }`), and it does so at config load, not downstream. An earlier design used
+  a bare `string` for the output path, under which `"no-auth"` would have parsed
+  as a file named `no-auth` and every unmigrated config would have silently
+  produced nonsense; the object form makes any string invalid, so the failure is
+  loud.
+
+  The collapse is deliberately not uniform in effect. For `rpc` the flag was a
+  blanket "no anonymous RPC in this app" set on a dispatcher that cannot know what
+  it dispatches to; for `userAdmin` it was redundant, since the generated
+  functions are already `pikkuFunc` with `scopes: ['admin:users:list']`. But for
+  `agent`, `workflow`, `events` and `scenarios` it is the only gate — those
+  generate real endpoints the app never authors — so `true` keeps them
+  authenticated rather than opening them as a side effect of a config cleanup.
+  `webhook` and `remoteRpc` have no auth dimension at all (`serialize-remote-rpc.ts`
+  hardcodes `auth: false`), so an `{ auth }` on them is ignored. The three
+  configs in this repo are migrated preserving their current behaviour exactly.
+
+  **Queue job identities are signed.** A job carried the producer's `pikkuUserId`
+  as a plain string and the worker resolved a session from it with no
+  verification, so write access to the queue backend was act-as-any-user. The
+  identity is now `pq1.<claim>.<hmac>`, HMAC-SHA256 over the claim and the
+  canonicalized job payload, keyed by HKDF expansion of a new
+  `PIKKU_QUEUE_IDENTITY_SECRET`. Producers opt in by wrapping their queue service
+  with `SignedQueueService`. This fails safe rather than closed: with no secret
+  configured the identity is dropped and jobs still process, warning once per
+  process, so no existing deployment breaks on upgrade — it simply loses queue
+  identity until the secret is set. The payload rather than the job id is bound
+  because SQS, Cloudflare Queues, Azure and the in-memory service all mint ids
+  after `add` returns.
+
+  **Workflow inline state is read from the run record.** `isInline` consulted a
+  process-local `Map`, while `WorkflowRun.inline` is durable. Any instance that
+  did not start a run disagreed with the record, so one instance could dispatch a
+  queued job for a workflow another was already executing in-process. It is now
+  async and resolves through the durable identity, cached only when a context
+  already exists so a passive reader allocates nothing. The same `Map` also leaked:
+  `nextStepKey` fabricated replay state on every step, and `releaseContext`
+  refused to free anything carrying it, so runs whose steps executed outside a
+  `beginReplay` bracket — the step-worker queue path — stranded their context and
+  step state for the process lifetime. Contexts are now released by an explicit
+  execution counter. Step ordinals reset per execution rather than accumulating
+  across step-worker invocations in one process, which makes step naming
+  independent of how work was distributed.
+
+  **Secret reads fail loud in every store.** `MongoDBSecretService.getSecrets`
+  skipped rows that failed to decrypt, and the redis equivalent dropped every
+  rejection via `Promise.allSettled`, including the "No KEK available for
+  key_version N" configuration error. Both now throw, naming the key and its key
+  version, matching the kysely behaviour. This matters most alongside the KEK
+  change above: without it, an upgrade surfaces as a partial secrets map and an
+  opaque downstream failure instead of an error naming the secret to re-enter.
+
+  **A second middleware registration for a pattern no longer erases the first.**
+  `addHTTPMiddleware`, `addTagMiddleware` and `addChannelMiddleware` groups are
+  keyed by pattern or tag and held one source file each, so a second file's call
+  overwrote the first's. Codegen emits its imports from what is stored, so the
+  losing file was never imported and its middleware never registered — the
+  runtime composes repeated registrations for a pattern happily, and only codegen
+  dropped one. Adding an unrelated `addHTTPMiddleware('*', …)` to an app was
+  enough to silently unregister the generated better-auth session bridge, which
+  fails open and gives no sign until a request arrives without a session. A group
+  now carries every registration made for it, and all of them are imported.
+
+- Updated dependencies [6a307f0]
+- Updated dependencies [afef587]
+- Updated dependencies [8075f6a]
+  - @pikku/core@0.12.74
+
+## 0.12.12
+
+### Patch Changes
+
+- 8a2c993: Make the workflow service cheaper to run, and fix two ways it lost state.
+
+  The SQL workflow tables had no indexes at all, so every step read, every
+  history walk and every orchestrator tick was a sequential scan; five indexes
+  now cover the columns the engine actually queries by. A replay used to ask for
+  each step's row individually — O(N) reads per replay, O(N^2) over a run — and
+  now takes one read of the run's steps and serves the walk from it. A step
+  transition wrote the step row and its history row as two separate statements,
+  so a crash between them left a step saying `succeeded` whose history still said
+  `running`; both halves are now one transaction, and the history row is found by
+  attempt number rather than by sorting on `created_at`, which two attempts can
+  share. Resolving a dynamic workflow read and parsed every AI-generated workflow
+  in the deployment to `.find()` one by name; it is a point lookup now.
+
+  Waiting on a run no longer polls at a fixed interval. `pollIntervalMs` became a
+  ceiling rather than a cadence: polling starts at 10ms and widens towards it, so
+  a workflow that finishes in milliseconds is no longer held for a full second,
+  and a long-running one is not read at full rate for its whole life.
+
+  Two backend-specific defects: Redis kept a run's state as one JSON blob and
+  read-modified-wrote it, so parallel branches setting different variables
+  overwrote each other — state is a field per variable now, with the old blob
+  still read underneath so runs in flight keep what they had. Mongo's
+  `setStepScheduled` never wrote history, leaving a queued step reading as never
+  dispatched.
+
+  Also: dispatch no longer JSON round-trips every step payload before handing it
+  to a queue that serialises it anyway — the in-process dev queue, which is the
+  only one that was relying on it, does it itself now.
+
+  Two more defects. A transition whose step had no live attempt wrote its status
+  to the step row and silently nothing to history — the exact divergence the
+  transaction exists to prevent — and now repairs the step and writes the
+  missing row. And resolving a dynamic workflow was non-deterministic on all
+  three backends: a name can hold several active versions, and none of them
+  ordered the candidates, so which one ran could change between two calls
+  reading identical data. The newest version wins, with the graph hash breaking
+  a tie.
+
+  The two attempt columns and the five indexes are declared in the workflow
+  schema, so a fresh database gets them at boot. An existing one gets them from
+  a migration — `pikku db generate` writes the declaration down — rather than
+  from DDL issued at boot.
+
+- a261006: **Breaking:** removed dynamic workflows — runtime-defined workflow graphs stored in the database and resolved by name instead of by codegen.
+
+  The feature was already half-gone. Its authoring surface (`createAgentWorkflow`, `saveAgentWorkflow`, `listAgentWorkflows`, `executeAgentWorkflow`, and the AI-agent instruction builder) was deleted in April 2026 along with its entire e2e suite, and nothing has written a dynamic workflow since. What remained could not execute one either: `executeAgentWorkflow` gated on `pikkuState('workflows', 'meta')`, which only codegen ever populates, so a graph that existed solely in the database was never findable. The two backend families had also drifted onto different `source` sentinels (`'ai-agent'` vs `'dynamic-workflow'`), and the two Redis implementations disagreed on key escaping — so at least one of them matched nothing. Rather than keep shipping plumbing for a path no caller could complete, it is removed until it can be reintroduced deliberately.
+
+  Removed:
+  - `getAIGeneratedWorkflows` from `WorkflowService` and `WorkflowRunService`, and from every backend (in-memory, Redis, MongoDB, Kysely, and the Cloudflare Durable Object service and client — the last two were already a `return []` stub and a rejection).
+  - The database-lookup fallbacks in `startWorkflow` and `runWorkflowJob` that resolved a workflow name against stored graphs when static meta had no match.
+  - `'dynamic-workflow'` from the `WorkflowRuntimeMeta['source']` union.
+  - `validateWorkflowWiring` and `computeEntryNodeIds` from `@pikku/core/workflow`. These validated AI-authored graphs and had no callers in core; the inspector keeps its own private entry-node computation for static graph wiring, which is unaffected.
+  - The `workflow-created` AI stream event and its AG-UI `pikku:workflow-created` custom event. Its only emitter went with the April deletion, so it could never fire.
+  - The console's `console:getAIWorkflows` RPC, the `useAIWorkflows` hook, the "Dynamic" workflow filter and badge, and the trigger-schema scraper that derived an input form from a stored graph's `$ref` bindings.
+
+  Kept, because static graph workflows depend on them and this is not a change to versioning:
+  - `upsertWorkflowVersion`, `getWorkflowVersion`, `updateWorkflowVersionStatus`, and the `workflowVersions` storage in every backend. These back version-mismatch replay: when a deployed graph's hash changes, in-flight runs continue against the exact graph they started on. No schema migration is needed — the table, its columns, and its `(workflowName, graphHash)` upsert key are unchanged.
+  - `generateMermaidDiagram`, which renders any workflow graph and is not specific to dynamic ones.
+
+  Static `pikkuWorkflowGraph` and DSL workflows are entirely unaffected: they resolve from codegen'd meta, which was always the only path that worked.
+
+  To revive this post-MVP, the deleted authoring code is recoverable in full — its prompt engineering (a compact tool table upfront, full schemas with flattened dotted output paths returned only after a validation failure) is worth reading before rewriting:
+
+  ```
+  git show f52f3308b^:packages/core/src/wirings/ai-agent/agent-dynamic-workflow.ts
+  git show f52f3308b^:packages/core/src/wirings/workflow/graph/graph-validation.ts
+  git show f52f3308b --stat   # the April removal, incl. the three e2e feature files
+  ```
+
+  Note that reviving it needs more than restoring those files: the queued-step path (`executeWorkflowStep`), `onError` compensation, and sub-workflow resolution all read static meta only and would need a fallback for a graph that exists solely in the database.
+
+- Updated dependencies [8a2c993]
+- Updated dependencies [a261006]
+- Updated dependencies [09973b9]
+  - @pikku/core@0.12.71
+
 ## 0.12.11
 
 ### Patch Changes

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 - present Yasser Fadl and Pikku contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,27 @@
+# @pikku/mongodb
+
+MongoDB implementations of the Pikku service interfaces — workflow state, agent
+runs, secrets, and the channel and event-hub stores.
+
+## Install
+
+```bash
+npm install @pikku/mongodb
+```
+
+## Usage
+
+```typescript
+import { PikkuMongoDB, MongoDBWorkflowService } from '@pikku/mongodb'
+
+const { db } = new PikkuMongoDB(logger, 'mongodb://localhost:27017')
+
+const workflowService = new MongoDBWorkflowService(db)
+```
+
+`PikkuMongoDB` also accepts an existing `MongoClient`, in which case it will not
+close the connection on shutdown.
+
+## Docs
+
+https://pikku.dev/docs

package/dist/src/mongodb-secret-service.d.ts

@@ -17,9 +17,13 @@ export declare class MongoDBSecretService implements SecretService {
     private auditReads;
     private secrets;
     private secretsAudit;
+    private kekSaltDocs;
+    private kekSalts;
+    private keks;
     constructor(db: Db, config: MongoDBSecretServiceConfig);
     init(): Promise<void>;
     private logAudit;
+    private getKEKSalt;
     private getKEK;
     getSecret<T = string>(key: string): Promise<T>;
     hasSecret(key: string): Promise<boolean>;

package/dist/src/mongodb-secret-service.js

@@ -1,4 +1,4 @@
-import { envelopeEncrypt, envelopeDecrypt, envelopeRewrap, } from '@pikku/core/crypto-utils';
+import { deriveKEK, generateKEKSalt, envelopeEncrypt, envelopeDecrypt, envelopeRewrap, } from '@pikku/core/crypto-utils';
 export class MongoDBSecretService {
     db;
     initialized = false;
@@ -9,6 +9,9 @@ export class MongoDBSecretService {
     auditReads;
     secrets;
     secretsAudit;
+    kekSaltDocs;
+    kekSalts = new Map();
+    keks = new Map();
     constructor(db, config) {
         this.db = db;
         this.key = config.key;
@@ -22,6 +25,7 @@ export class MongoDBSecretService {
             return;
         this.secrets = this.db.collection('secrets');
         this.secretsAudit = this.db.collection('secrets_audit');
+        this.kekSaltDocs = this.db.collection('secret_kek_salts');
         await this.secrets.createIndex({ _id: 1 });
         if (this.audit) {
             await this.secretsAudit.createIndex({ secretKey: 1 });
@@ -40,18 +44,34 @@ export class MongoDBSecretService {
             performedAt: new Date(),
         });
     }
-    getKEK(version) {
-        if (version === this.keyVersion)
-            return this.key;
-        if (this.previousKey)
-            return this.previousKey;
-        throw new Error(`No KEK available for key_version ${version}`);
+    async getKEKSalt(version) {
+        const cached = this.kekSalts.get(version);
+        if (cached)
+            return cached;
+        const doc = await this.kekSaltDocs.findOneAndUpdate({ _id: version }, { $setOnInsert: { salt: generateKEKSalt(), createdAt: new Date() } }, { upsert: true, returnDocument: 'after' });
+        if (!doc?.salt) {
+            throw new Error(`Failed to persist a KEK salt for key_version ${version}`);
+        }
+        this.kekSalts.set(version, doc.salt);
+        return doc.salt;
+    }
+    async getKEK(version) {
+        const cached = this.keks.get(version);
+        if (cached)
+            return cached;
+        const passphrase = version === this.keyVersion ? this.key : this.previousKey;
+        if (!passphrase) {
+            throw new Error(`No KEK available for key_version ${version}`);
+        }
+        const kek = await deriveKEK(passphrase, await this.getKEKSalt(version));
+        this.keks.set(version, kek);
+        return kek;
     }
     async getSecret(key) {
         const row = await this.secrets.findOne({ _id: key });
         if (!row)
             throw new Error('Requested secret not found');
-        const kek = this.getKEK(row.keyVersion);
+        const kek = await this.getKEK(row.keyVersion);
         const result = await envelopeDecrypt(kek, row.ciphertext, row.wrappedDek);
         await this.logAudit(key, 'read');
         return result;
@@ -61,7 +81,7 @@ export class MongoDBSecretService {
         return count > 0;
     }
     async setSecret(key, value) {
-        const { ciphertext, wrappedDEK } = await envelopeEncrypt(this.key, value);
+        const { ciphertext, wrappedDEK } = await envelopeEncrypt(await this.getKEK(this.keyVersion), value);
         const now = new Date();
         await this.secrets.updateOne({ _id: key }, {
             $set: {
@@ -88,11 +108,12 @@ export class MongoDBSecretService {
         const out = {};
         for (const row of rows) {
             try {
-                const kek = this.getKEK(row.keyVersion);
+                const kek = await this.getKEK(row.keyVersion);
                 out[row._id] = await envelopeDecrypt(kek, row.ciphertext, row.wrappedDek);
             }
-            catch {
-                // skip secrets that fail to decrypt
+            catch (cause) {
+                throw new Error(`Failed to decrypt secret "${row._id}" (key_version ${row.keyVersion}): ` +
+                    `the configured KEK does not match the key it was wrapped under`, { cause });
             }
         }
         return out;
@@ -105,8 +126,10 @@ export class MongoDBSecretService {
             .find({ keyVersion: { $lt: this.keyVersion } })
             .project({ _id: 1, wrappedDek: 1 })
             .toArray();
+        const oldKEK = await this.getKEK(this.keyVersion - 1);
+        const newKEK = await this.getKEK(this.keyVersion);
         for (const row of rows) {
-            const newWrappedDEK = await envelopeRewrap(this.previousKey, this.key, row.wrappedDek);
+            const newWrappedDEK = await envelopeRewrap(oldKEK, newKEK, row.wrappedDek);
             await this.secrets.updateOne({ _id: row._id }, {
                 $set: {
                     wrappedDek: newWrappedDEK,

package/dist/src/mongodb-workflow-run-service.d.ts

@@ -26,11 +26,6 @@ export declare class MongoDBWorkflowRunService implements WorkflowRunService {
         graph: any;
         source: string;
     } | null>;
-    getAIGeneratedWorkflows(agentName?: string): Promise<Array<{
-        workflowName: string;
-        graphHash: string;
-        graph: any;
-    }>>;
     deleteRun(id: string): Promise<boolean>;
     private mapRunRow;
 }

package/dist/src/mongodb-workflow-run-service.js

@@ -105,19 +105,6 @@ export class MongoDBWorkflowRunService {
             source: row.source,
         };
     }
-    async getAIGeneratedWorkflows(agentName) {
-        const filter = { source: 'ai-agent', status: 'active' };
-        if (agentName) {
-            const escaped = agentName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
-            filter.workflowName = { $regex: `^ai:${escaped}:` };
-        }
-        const rows = await this.versions.find(filter).toArray();
-        return rows.map((row) => ({
-            workflowName: row.workflowName,
-            graphHash: row.graphHash,
-            graph: row.graph,
-        }));
-    }
     async deleteRun(id) {
         const steps = await this.steps.find({ workflowRunId: id }).toArray();
         const stepIds = steps.map((s) => s._id);

package/dist/src/mongodb-workflow-service.d.ts

@@ -26,6 +26,16 @@ export declare class MongoDBWorkflowService extends PikkuWorkflowService {
     }>>;
     protected setStepRunningImpl(stepId: string): Promise<void>;
     protected setStepScheduledImpl(stepId: string): Promise<void>;
+    /**
+     * Move the current attempt of a step's history to a status.
+     *
+     * Finding the newest attempt and then updating it by `_id` was two round
+     * trips with a window in between; `findOneAndUpdate` applies the same sort
+     * server-side and writes the row it selected, so nothing can move underneath
+     * it. The attempt is still identified by `createdAt` because Mongo keeps no
+     * attempt number — see `getStepState`, which counts history documents.
+     */
+    private writeLatestHistory;
     private insertHistoryRecord;
     private getTimestampFieldForStatus;
     protected setStepChildRunIdImpl(stepId: string, childRunId: string): Promise<void>;
@@ -39,7 +49,6 @@ export declare class MongoDBWorkflowService extends PikkuWorkflowService {
         failedNodeIds: string[];
         branchKeys: Record<string, string>;
     }>;
-    getNodesWithoutSteps(runId: string, nodeIds: string[]): Promise<string[]>;
     getStepInstances(runId: string): Promise<Array<{
         stepName: string;
         status: StepStatus;
@@ -55,10 +64,5 @@ export declare class MongoDBWorkflowService extends PikkuWorkflowService {
         graph: any;
         source: string;
     } | null>;
-    getAIGeneratedWorkflows(agentName?: string): Promise<Array<{
-        workflowName: string;
-        graphHash: string;
-        graph: any;
-    }>>;
     close(): Promise<void>;
 }

package/dist/src/mongodb-workflow-service.js

@@ -124,17 +124,29 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
     }
     async setStepRunningImpl(stepId) {
         await this.steps.updateOne({ _id: stepId }, { $set: { status: 'running', updatedAt: new Date() } });
-        const latestHistory = await this.stepHistory
-            .find({ workflowStepId: stepId })
-            .sort({ createdAt: -1 })
-            .limit(1)
-            .toArray();
-        if (latestHistory.length > 0) {
-            await this.stepHistory.updateOne({ _id: latestHistory[0]._id }, { $set: { status: 'running', runningAt: new Date() } });
-        }
+        await this.writeLatestHistory(stepId, 'running');
     }
     async setStepScheduledImpl(stepId) {
         await this.steps.updateOne({ _id: stepId }, { $set: { status: 'scheduled', updatedAt: new Date() } });
+        await this.writeLatestHistory(stepId, 'scheduled');
+    }
+    /**
+     * Move the current attempt of a step's history to a status.
+     *
+     * Finding the newest attempt and then updating it by `_id` was two round
+     * trips with a window in between; `findOneAndUpdate` applies the same sort
+     * server-side and writes the row it selected, so nothing can move underneath
+     * it. The attempt is still identified by `createdAt` because Mongo keeps no
+     * attempt number — see `getStepState`, which counts history documents.
+     */
+    async writeLatestHistory(stepId, status, values = {}) {
+        const now = new Date();
+        const update = { status, ...values };
+        const timestampField = this.getTimestampFieldForStatus(status);
+        if (timestampField !== 'createdAt') {
+            update[timestampField] = now;
+        }
+        await this.stepHistory.findOneAndUpdate({ workflowStepId: stepId }, { $set: update }, { sort: { createdAt: -1 } });
     }
     async insertHistoryRecord(stepId, status, result, error) {
         const now = new Date();
@@ -187,14 +199,7 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
                 updatedAt: new Date(),
             },
         });
-        const latestHistory = await this.stepHistory
-            .find({ workflowStepId: stepId })
-            .sort({ createdAt: -1 })
-            .limit(1)
-            .toArray();
-        if (latestHistory.length > 0) {
-            await this.stepHistory.updateOne({ _id: latestHistory[0]._id }, { $set: { status: 'succeeded', result, succeededAt: new Date() } });
-        }
+        await this.writeLatestHistory(stepId, 'succeeded', { result });
     }
     async setStepErrorImpl(stepId, error) {
         const serializedError = {
@@ -210,14 +215,9 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
                 updatedAt: new Date(),
             },
         });
-        const latestHistory = await this.stepHistory
-            .find({ workflowStepId: stepId })
-            .sort({ createdAt: -1 })
-            .limit(1)
-            .toArray();
-        if (latestHistory.length > 0) {
-            await this.stepHistory.updateOne({ _id: latestHistory[0]._id }, { $set: { status: 'failed', error: serializedError, failedAt: new Date() } });
-        }
+        await this.writeLatestHistory(stepId, 'failed', {
+            error: serializedError,
+        });
     }
     async createRetryAttemptImpl(stepId, status) {
         await this.steps.updateOne({ _id: stepId }, {
@@ -284,19 +284,6 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
         }
         return { completedNodeIds, failedNodeIds, branchKeys };
     }
-    async getNodesWithoutSteps(runId, nodeIds) {
-        if (nodeIds.length === 0)
-            return [];
-        const result = await this.steps
-            .find({
-            workflowRunId: runId,
-            stepName: { $in: nodeIds },
-        })
-            .project({ stepName: 1 })
-            .toArray();
-        const existingStepNames = new Set(result.map((r) => r.stepName));
-        return nodeIds.filter((id) => !existingStepNames.has(id));
-    }
     async getStepInstances(runId) {
         const rows = await this.steps
             .find({ workflowRunId: runId })
@@ -362,18 +349,5 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
     async getWorkflowVersion(name, graphHash) {
         return this.runService.getWorkflowVersion(name, graphHash);
     }
-    async getAIGeneratedWorkflows(agentName) {
-        const filter = { source: 'ai-agent', status: 'active' };
-        if (agentName) {
-            const escaped = agentName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
-            filter.workflowName = { $regex: `^ai:${escaped}:` };
-        }
-        const docs = await this.versions.find(filter).toArray();
-        return docs.map((doc) => ({
-            workflowName: doc.workflowName,
-            graphHash: doc.graphHash,
-            graph: doc.graph,
-        }));
-    }
     async close() { }
 }

package/package.json

@@ -1,6 +1,7 @@
 {
   "name": "@pikku/mongodb",
-  "version": "0.12.11",
+  "version": "0.12.13",
+  "description": "MongoDB services for Pikku — workflow state, agent runs, secrets and channel stores",
   "author": "yasser.fadl@gmail.com",
   "license": "MIT",
   "module": "dist/src/index.js",
@@ -17,7 +18,7 @@
     "prepublishOnly": "yarn build"
   },
   "peerDependencies": {
-    "@pikku/core": "^0.12.69"
+    "@pikku/core": "^0.12.74"
   },
   "dependencies": {
     "mongodb": "^6.12.0"

package/src/mongodb-secret-service.ts

@@ -1,6 +1,8 @@
 import type { SecretService } from '@pikku/core/services'
 import type { Db, Collection } from 'mongodb'
 import {
+  deriveKEK,
+  generateKEKSalt,
   envelopeEncrypt,
   envelopeDecrypt,
   envelopeRewrap,
@@ -23,6 +25,12 @@ interface SecretDoc {
   updatedAt: Date
 }
 
+interface KekSaltDoc {
+  _id: number
+  salt: string
+  createdAt: Date
+}
+
 interface SecretAuditDoc {
   _id: string
   secretKey: string
@@ -39,6 +47,9 @@ export class MongoDBSecretService implements SecretService {
   private auditReads: boolean
   private secrets!: Collection<SecretDoc>
   private secretsAudit!: Collection<SecretAuditDoc>
+  private kekSaltDocs!: Collection<KekSaltDoc>
+  private kekSalts = new Map<number, string>()
+  private keks = new Map<number, CryptoKey>()
 
   constructor(
     private db: Db,
@@ -56,6 +67,7 @@ export class MongoDBSecretService implements SecretService {
 
     this.secrets = this.db.collection<SecretDoc>('secrets')
     this.secretsAudit = this.db.collection<SecretAuditDoc>('secrets_audit')
+    this.kekSaltDocs = this.db.collection<KekSaltDoc>('secret_kek_salts')
 
     await this.secrets.createIndex({ _id: 1 })
 
@@ -81,17 +93,42 @@ export class MongoDBSecretService implements SecretService {
     })
   }
 
-  private getKEK(version: number): string {
-    if (version === this.keyVersion) return this.key
-    if (this.previousKey) return this.previousKey
-    throw new Error(`No KEK available for key_version ${version}`)
+  private async getKEKSalt(version: number): Promise<string> {
+    const cached = this.kekSalts.get(version)
+    if (cached) return cached
+
+    const doc = await this.kekSaltDocs.findOneAndUpdate(
+      { _id: version },
+      { $setOnInsert: { salt: generateKEKSalt(), createdAt: new Date() } },
+      { upsert: true, returnDocument: 'after' }
+    )
+    if (!doc?.salt) {
+      throw new Error(`Failed to persist a KEK salt for key_version ${version}`)
+    }
+
+    this.kekSalts.set(version, doc.salt)
+    return doc.salt
+  }
+
+  private async getKEK(version: number): Promise<CryptoKey> {
+    const cached = this.keks.get(version)
+    if (cached) return cached
+
+    const passphrase = version === this.keyVersion ? this.key : this.previousKey
+    if (!passphrase) {
+      throw new Error(`No KEK available for key_version ${version}`)
+    }
+
+    const kek = await deriveKEK(passphrase, await this.getKEKSalt(version))
+    this.keks.set(version, kek)
+    return kek
   }
 
   async getSecret<T = string>(key: string): Promise<T> {
     const row = await this.secrets.findOne({ _id: key })
     if (!row) throw new Error('Requested secret not found')
 
-    const kek = this.getKEK(row.keyVersion)
+    const kek = await this.getKEK(row.keyVersion)
     const result = await envelopeDecrypt<T>(kek, row.ciphertext, row.wrappedDek)
     await this.logAudit(key, 'read')
     return result
@@ -103,7 +140,10 @@ export class MongoDBSecretService implements SecretService {
   }
 
   async setSecret(key: string, value: unknown): Promise<void> {
-    const { ciphertext, wrappedDEK } = await envelopeEncrypt(this.key, value)
+    const { ciphertext, wrappedDEK } = await envelopeEncrypt(
+      await this.getKEK(this.keyVersion),
+      value
+    )
     const now = new Date()
 
     await this.secrets.updateOne(
@@ -131,23 +171,27 @@ export class MongoDBSecretService implements SecretService {
     await this.logAudit(key, 'delete')
   }
 
-  async getSecrets<
-    T extends Record<string, unknown> = Record<string, unknown>,
-  >(keys: (keyof T & string)[]): Promise<T> {
+  async getSecrets<T extends Record<string, unknown> = Record<string, unknown>>(
+    keys: (keyof T & string)[]
+  ): Promise<T> {
     const rows = await this.secrets
       .find({ _id: { $in: keys } } as any)
       .toArray()
     const out: Record<string, unknown> = {}
     for (const row of rows) {
       try {
-        const kek = this.getKEK(row.keyVersion)
+        const kek = await this.getKEK(row.keyVersion)
         out[row._id] = await envelopeDecrypt(
           kek,
           row.ciphertext,
           row.wrappedDek
         )
-      } catch {
-        // skip secrets that fail to decrypt
+      } catch (cause) {
+        throw new Error(
+          `Failed to decrypt secret "${row._id}" (key_version ${row.keyVersion}): ` +
+            `the configured KEK does not match the key it was wrapped under`,
+          { cause }
+        )
       }
     }
     return out as T
@@ -163,12 +207,11 @@ export class MongoDBSecretService implements SecretService {
       .project({ _id: 1, wrappedDek: 1 })
       .toArray()
 
+    const oldKEK = await this.getKEK(this.keyVersion - 1)
+    const newKEK = await this.getKEK(this.keyVersion)
+
     for (const row of rows) {
-      const newWrappedDEK = await envelopeRewrap(
-        this.previousKey,
-        this.key,
-        row.wrappedDek
-      )
+      const newWrappedDEK = await envelopeRewrap(oldKEK, newKEK, row.wrappedDek)
       await this.secrets.updateOne(
         { _id: row._id },
         {

package/src/mongodb-services.test.ts

@@ -106,6 +106,84 @@ function registerTests(name: string, getDb: () => Db) {
       assert.equal(logs[0]!.action, 'write')
     })
   })
+
+  describe(`MongoDBSecretService decryption failures [${name}]`, () => {
+    const kek = 'test-key-encryption-key-32chars!'
+    const otherKek = 'a-totally-different-kek-32chars!'
+
+    test('getSecrets throws naming a secret it cannot decrypt', async () => {
+      const writer = new MongoDBSecretService(getDb(), { key: kek })
+      await writer.init()
+      await writer.setSecret('undecryptable', { token: 'abc' })
+
+      const reader = new MongoDBSecretService(getDb(), { key: otherKek })
+      await reader.init()
+
+      await assert.rejects(
+        () => reader.getSecrets(['undecryptable']),
+        (error: Error) => {
+          assert.equal(
+            error.message,
+            'Failed to decrypt secret "undecryptable" (key_version 1): ' +
+              'the configured KEK does not match the key it was wrapped under'
+          )
+          assert.ok(error.cause, 'the underlying crypto failure is preserved')
+          return true
+        }
+      )
+    })
+
+    test('getSecrets throws when no KEK is available for a stored key_version', async () => {
+      const writer = new MongoDBSecretService(getDb(), {
+        key: kek,
+        keyVersion: 7,
+      })
+      await writer.init()
+      await writer.setSecret('old-version', 'value')
+
+      const reader = new MongoDBSecretService(getDb(), {
+        key: kek,
+        keyVersion: 8,
+      })
+      await reader.init()
+
+      await assert.rejects(
+        () => reader.getSecrets(['old-version']),
+        (error: Error) => {
+          assert.equal(
+            error.message,
+            'Failed to decrypt secret "old-version" (key_version 7): ' +
+              'the configured KEK does not match the key it was wrapped under'
+          )
+          return true
+        }
+      )
+    })
+
+    test('getSecrets returns every secret when they all decrypt', async () => {
+      const service = new MongoDBSecretService(getDb(), { key: kek })
+      await service.init()
+      await service.setSecret('batch-a', { v: 1 })
+      await service.setSecret('batch-b', 'two')
+
+      const out = await service.getSecrets<{
+        'batch-a': { v: number }
+        'batch-b': string
+      }>(['batch-a', 'batch-b'])
+
+      assert.deepEqual(out, { 'batch-a': { v: 1 }, 'batch-b': 'two' })
+    })
+
+    test('getSecrets omits keys that were never stored', async () => {
+      const service = new MongoDBSecretService(getDb(), { key: kek })
+      await service.init()
+      await service.setSecret('present', 'here')
+
+      const out = await service.getSecrets(['present', 'never-stored'])
+
+      assert.deepEqual(out, { present: 'here' })
+    })
+  })
 }
 
 describe('MongoDB Services - In-Memory', () => {

package/src/mongodb-workflow-run-service.ts

@@ -202,22 +202,6 @@ export class MongoDBWorkflowRunService implements WorkflowRunService {
     }
   }
 
-  async getAIGeneratedWorkflows(
-    agentName?: string
-  ): Promise<Array<{ workflowName: string; graphHash: string; graph: any }>> {
-    const filter: Record<string, any> = { source: 'ai-agent', status: 'active' }
-    if (agentName) {
-      const escaped = agentName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
-      filter.workflowName = { $regex: `^ai:${escaped}:` }
-    }
-    const rows = await this.versions.find(filter).toArray()
-    return rows.map((row) => ({
-      workflowName: row.workflowName,
-      graphHash: row.graphHash,
-      graph: row.graph,
-    }))
-  }
-
   async deleteRun(id: string): Promise<boolean> {
     const steps = await this.steps.find({ workflowRunId: id }).toArray()
     const stepIds = steps.map((s) => s._id)

package/src/mongodb-workflow-service.test.ts

@@ -0,0 +1,121 @@
+import { describe, test, before, after, beforeEach } from 'node:test'
+import assert from 'node:assert/strict'
+import { MongoClient, type Db } from 'mongodb'
+import { MongoMemoryServer } from 'mongodb-memory-server'
+
+import { MongoDBWorkflowService } from './mongodb-workflow-service.js'
+
+let mongod: MongoMemoryServer
+let client: MongoClient
+let db: Db
+let ws: MongoDBWorkflowService
+let dbCount = 0
+
+before(async () => {
+  mongod = await MongoMemoryServer.create()
+  client = new MongoClient(mongod.getUri())
+  await client.connect()
+})
+
+after(async () => {
+  await client?.close()
+  await mongod?.stop()
+})
+
+beforeEach(async () => {
+  db = client.db(`wf-${++dbCount}`)
+  ws = new MongoDBWorkflowService(db)
+  await ws.init()
+})
+
+const seedStep = async () => {
+  const runId = await ws.createRun('flow', {}, false, 'hash', {
+    type: 'test',
+  } as any)
+  const step = await ws.insertStepState(runId, 'step-1', 'rpc.fn', { x: 1 })
+  return { runId, step }
+}
+
+/** The history rows of a run, oldest first. */
+const historyOf = async (runId: string) => ws.getRunHistory(runId)
+
+describe('a step transition reaches the history', () => {
+  test('running stamps the attempt it is running', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepRunning(step.stepId)
+
+    const [attempt] = await historyOf(runId)
+    assert.equal(attempt!.status, 'running')
+    assert.ok(attempt!.runningAt, 'no runningAt, so the step has no duration')
+  })
+
+  test('scheduled stamps the attempt it is queueing', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepScheduled(step.stepId)
+
+    const [attempt] = await historyOf(runId)
+    assert.equal(
+      attempt!.status,
+      'scheduled',
+      'the history still says pending, so a queued step reads as never dispatched'
+    )
+    assert.ok(attempt!.scheduledAt, 'no scheduledAt on the queued attempt')
+  })
+
+  test('succeeded carries the result', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepRunning(step.stepId)
+    await ws.setStepResult(step.stepId, { ok: true })
+
+    const [attempt] = await historyOf(runId)
+    assert.equal(attempt!.status, 'succeeded')
+    assert.deepEqual(attempt!.result, { ok: true })
+    assert.ok(attempt!.succeededAt)
+  })
+
+  test('failed carries the error', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepRunning(step.stepId)
+    await ws.setStepError(step.stepId, new Error('exploded'))
+
+    const [attempt] = await historyOf(runId)
+    assert.equal(attempt!.status, 'failed')
+    assert.equal(attempt!.error?.message, 'exploded')
+    assert.ok(attempt!.failedAt)
+  })
+
+  test('a retry writes to the new attempt, leaving the failed one intact', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepRunning(step.stepId)
+    await ws.setStepError(step.stepId, new Error('first go'))
+    await ws.createRetryAttempt(step.stepId, 'pending')
+    await ws.setStepRunning(step.stepId)
+    await ws.setStepResult(step.stepId, 'second go')
+
+    const history = await historyOf(runId)
+    assert.equal(history.length, 2)
+    assert.equal(history[0]!.status, 'failed')
+    assert.equal(history[0]!.error?.message, 'first go')
+    assert.equal(history[1]!.status, 'succeeded')
+    assert.equal(history[1]!.result, 'second go')
+  })
+
+  test('the step row keeps only its latest outcome', async () => {
+    const { runId, step } = await seedStep()
+
+    await ws.setStepError(step.stepId, new Error('first go'))
+    await ws.createRetryAttempt(step.stepId, 'pending')
+    await ws.setStepResult(step.stepId, 'second go')
+
+    const state = await ws.getStepState(runId, 'step-1')
+    assert.equal(state.status, 'succeeded')
+    assert.equal(state.result, 'second go')
+    assert.equal(state.error, undefined)
+    assert.equal(state.attemptCount, 2)
+  })
+})

package/src/mongodb-workflow-service.ts

@@ -255,19 +255,7 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
       { _id: stepId },
       { $set: { status: 'running', updatedAt: new Date() } }
     )
-
-    const latestHistory = await this.stepHistory
-      .find({ workflowStepId: stepId })
-      .sort({ createdAt: -1 })
-      .limit(1)
-      .toArray()
-
-    if (latestHistory.length > 0) {
-      await this.stepHistory.updateOne(
-        { _id: latestHistory[0]!._id },
-        { $set: { status: 'running', runningAt: new Date() } }
-      )
-    }
+    await this.writeLatestHistory(stepId, 'running')
   }
 
   protected async setStepScheduledImpl(stepId: string): Promise<void> {
@@ -275,6 +263,35 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
       { _id: stepId },
       { $set: { status: 'scheduled', updatedAt: new Date() } }
     )
+    await this.writeLatestHistory(stepId, 'scheduled')
+  }
+
+  /**
+   * Move the current attempt of a step's history to a status.
+   *
+   * Finding the newest attempt and then updating it by `_id` was two round
+   * trips with a window in between; `findOneAndUpdate` applies the same sort
+   * server-side and writes the row it selected, so nothing can move underneath
+   * it. The attempt is still identified by `createdAt` because Mongo keeps no
+   * attempt number — see `getStepState`, which counts history documents.
+   */
+  private async writeLatestHistory(
+    stepId: string,
+    status: string,
+    values: Record<string, unknown> = {}
+  ): Promise<void> {
+    const now = new Date()
+    const update: Record<string, unknown> = { status, ...values }
+    const timestampField = this.getTimestampFieldForStatus(status)
+    if (timestampField !== 'createdAt') {
+      update[timestampField] = now
+    }
+
+    await this.stepHistory.findOneAndUpdate(
+      { workflowStepId: stepId },
+      { $set: update as any },
+      { sort: { createdAt: -1 } }
+    )
   }
 
   private async insertHistoryRecord(
@@ -351,18 +368,7 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
       }
     )
 
-    const latestHistory = await this.stepHistory
-      .find({ workflowStepId: stepId })
-      .sort({ createdAt: -1 })
-      .limit(1)
-      .toArray()
-
-    if (latestHistory.length > 0) {
-      await this.stepHistory.updateOne(
-        { _id: latestHistory[0]!._id },
-        { $set: { status: 'succeeded', result, succeededAt: new Date() } }
-      )
-    }
+    await this.writeLatestHistory(stepId, 'succeeded', { result })
   }
 
   protected async setStepErrorImpl(
@@ -387,18 +393,9 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
       }
     )
 
-    const latestHistory = await this.stepHistory
-      .find({ workflowStepId: stepId })
-      .sort({ createdAt: -1 })
-      .limit(1)
-      .toArray()
-
-    if (latestHistory.length > 0) {
-      await this.stepHistory.updateOne(
-        { _id: latestHistory[0]!._id },
-        { $set: { status: 'failed', error: serializedError, failedAt: new Date() } }
-      )
-    }
+    await this.writeLatestHistory(stepId, 'failed', {
+      error: serializedError,
+    })
   }
 
   protected async createRetryAttemptImpl(
@@ -490,25 +487,9 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
     return { completedNodeIds, failedNodeIds, branchKeys }
   }
 
-  async getNodesWithoutSteps(
-    runId: string,
-    nodeIds: string[]
-  ): Promise<string[]> {
-    if (nodeIds.length === 0) return []
-
-    const result = await this.steps
-      .find({
-        workflowRunId: runId,
-        stepName: { $in: nodeIds },
-      })
-      .project({ stepName: 1 })
-      .toArray()
-
-    const existingStepNames = new Set(result.map((r) => r.stepName))
-    return nodeIds.filter((id) => !existingStepNames.has(id))
-  }
-
-  async getStepInstances(runId: string): Promise<
+  async getStepInstances(
+    runId: string
+  ): Promise<
     Array<{ stepName: string; status: StepStatus; fromStepName?: string }>
   > {
     const rows = await this.steps
@@ -622,21 +603,5 @@ export class MongoDBWorkflowService extends PikkuWorkflowService {
     return this.runService.getWorkflowVersion(name, graphHash)
   }
 
-  async getAIGeneratedWorkflows(
-    agentName?: string
-  ): Promise<Array<{ workflowName: string; graphHash: string; graph: any }>> {
-    const filter: Record<string, any> = { source: 'ai-agent', status: 'active' }
-    if (agentName) {
-      const escaped = agentName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
-      filter.workflowName = { $regex: `^ai:${escaped}:` }
-    }
-    const docs = await this.versions.find(filter).toArray()
-    return docs.map((doc) => ({
-      workflowName: doc.workflowName,
-      graphHash: doc.graphHash,
-      graph: doc.graph,
-    }))
-  }
-
   async close(): Promise<void> {}
 }