@plumbus/core 0.6.4 → 0.6.6

package/CHANGELOG.md

@@ -0,0 +1,400 @@
+# @plumbus/core changelog
+
+## 0.6.6
+
+### Breaking behavior changes
+
+- **`plumbus translation status` exit code** — exits non-zero when any locale is incomplete (including `--json`). Previously incomplete locales printed a warning but exited 0. Aligns with docs that already recommend wiring the command into CI; update scripts that assumed exit 0.
+
+### Added
+
+- **`computeStatus` / `formatTranslationStatus`** — shared translation coverage util (exported from `@plumbus/core`); used by `plumbus translation status` and `plumbus ui generate`.
+- **`plumbus ui generate --skip-locale-parity`** — by default, generate refuses incomplete locale coverage (`exit 1`); the flag warns and continues.
+
+### Changed
+
+- **`plumbus ui generate` / `ui nextjs`** — no longer mirror modules into `.plumbus/generated/ui` when writing to a real frontend out-dir. That path remains only the last-resort default when no frontend directory is detected (or when `--out-dir` points there explicitly).
+
+## 0.6.5
+
+### Breaking behavior changes
+
+Contract-first defaults — no framework-wide legacy mode. Escape hatches are per-app opt-outs; see [Migration stance](../../docs/upgrading-contract-alignment.md#migration-stance-locked).
+
+- **Transactional outbox default ON (A1)** — `action` and `eventHandler` capabilities now run handler + output validation inside a single database transaction so entity writes and `ctx.events.emit()` outbox rows commit or roll back together. Auto-excluded: `kind: 'job'`, `effects.ai: true`, non-empty `effects.external`. Nested failures inside an active transaction re-throw to roll back the parent scope; `ctx.flows.start()` and `ctx.jobs.enqueue()` inside a transaction defer until commit (pre-allocated ids). Nested success capability audits defer until commit. Opt out globally with `execution.transactionalOutbox: false` (or `PLUMBUS_TRANSACTIONAL_OUTBOX=false`), or per capability with `transactional: false`. Migration: [Upgrading for contract alignment](../../docs/upgrading-contract-alignment.md#1-transactional-outbox-default-on-a1).
+
+- **`plumbus api validate` governance signals are advisory** — the command exits 1 only on hard contract findings (manifest, policy, path params, fixtures). Governance rule signals are printed but no longer fail the command unless you pass `--fail-on-governance`. Aligns with the advisory-only governance model. Migration: [Upgrading for contract alignment](../../docs/upgrading-contract-alignment.md#2-plumbus-api-validate-and-governance-signals-a4).
+
+- **AI security `mode: 'block'`** — when `aiProviders.security` or `AI_SECURITY_*` env is configured and `mode` is `block`, classified fields at the warn threshold abort the AI call instead of being redacted. Without explicit security config, classified fields are not scanned. Migration: [Upgrading for contract alignment](../../docs/upgrading-contract-alignment.md#12-ai-security-mode-block-vs-redact-a2).
+
+### Added
+
+- **`aiProviders.security` (opt-in)** — `AISecurityConfig` on multi-provider AI config (`mode`, `warnThreshold`, `redactThreshold`, auto-populated `entities` from the entity registry when security is enabled). Wired through server, worker, and MCP bootstraps only when configured.
+
+- **Structured log masking (A18)** — `maskKeys` on `StructuredLoggerConfig`, exported `getMaskedFields` / `collectMaskedFieldsFromEntities`, and `withLogMasking()` wrapper. Bootstraps derive mask keys from entity field metadata (`maskedInLogs` and sensitive classifications). Redacted values use `***MASKED***` (deep/nested key walk).
+
+- **Consumer encryption key wiring** — API server, worker pool, flow-runner, and event/job consumer contexts pass `PLUMBUS_ENCRYPTION_KEY` into `wireContextDependencies` so `encrypted: true` fields decrypt in handlers.
+
+- **`ctx.jobs.enqueue`** — when `jobQueue` is configured on `createServer`, handlers can enqueue job capabilities via `ctx.jobs.enqueue(name, input)`; calls inside a transactional outbox defer until commit (same as `ctx.flows.start`). Returns a pre-allocated job id immediately so handlers can `await` without deadlocking the transaction; the queue write runs after commit.
+
+- **Structured data/AI errors** — repository tenant/RETURNING failures, encryption config/payload errors, job dispatch validation, and AI security/budget blocks now throw `PlumbusError` subclasses with stable `code` values.
+
+- **ServerConfig.logger** — custom loggers passed to `createServer()` are now used for per-request `ctx.logger` (with masking), not only bootstrap messages.
+
+- **Field encryption at rest (A3)** — AES-256-GCM via `PLUMBUS_ENCRYPTION_KEY`; repository encrypts/decrypts `encrypted: true` string fields with plaintext fallback for legacy rows.
+
+- **Per-request locale resolution (A17)** — `resolveRequestLocale()` in the route generator reads `plumbus-ui-locale` cookie then `Accept-Language`; resolved locale is passed on `DependencyOptions` into `ctx.translations`.
+
+### Behavior fixes
+
+- **Encrypted-field queries rejected** — when `PLUMBUS_ENCRYPTION_KEY` is set, `findMany`/`count` filters, `orderBy`, and `dateFilters` on `encrypted: true` fields throw `DataValidationError` instead of silently matching or sorting ciphertext.
+
+- **Deferred post-commit isolation** — post-commit flow/job/audit callbacks catch and log failures individually so a throwing deferred start does not fail the already-committed capability or skip remaining deferred work.
+
+- **Audit-record masking** — repository audit payloads use deep masking with token `***` (structured logs remain `***MASKED***`).
+
+- **Nested capability audits** — failure audits record immediately inside nested transactions; success audits defer until parent commit.
+
+- **Deferred flow ids** — `ctx.flows.start()` inside a transaction returns the same execution id created after commit.
+
+- **LLM inside open transaction** — first nested invoke of an `effects.ai: true` capability inside an active transaction emits a one-time `ctx.logger.warn` that the parent transaction is held open for the LLM call.
+
+- **HTTP event causation** — capability-triggered HTTP emits restore dynamic `causationId` from the executing capability.
+
+- **`plumbus mcp serve` logs** — capability request logs are structured JSON at `info` minimum level (debug suppressed).
+
+- **`maskedInLogs` in structured log metadata (A18)** — capability request loggers redact masked entity field names in metadata (token `***MASKED***`, nested objects included). Escape hatch: custom logger without `maskKeys`, or omit sensitive keys from metadata.
+
+- **`encryptFieldValue('')` round-trip** — empty plaintext no longer produces an undecryptable ciphertext blob.
+
+- **Invalid `AI_SECURITY_*_THRESHOLD` env values** — bogus thresholds are ignored with a startup warning instead of matching every classified field.
+
+## 0.6.4
+
+### Added
+
+- **`defineTranslation()` typecheck key consistency** — for literal / `as const` message catalogs, TypeScript now rejects locales that do not share the same key set (native diagnostics on the locale attribute). Runtime import-time validation is unchanged. Apps with existing key drift may start failing `tsc` even though they still throw at import time today.
+
+## 0.6.3
+
+### Added
+
+- **`QueryOptions.search`** — OR-of-ILIKE across named entity columns for free-text search (usable via `findMany`/`count`). LIKE metacharacters (`%`, `_`, `\`) in the term are escaped, so the term matches literally rather than as a wildcard pattern.
+- **`QueryOptions.in`** — SQL `IN` filter (empty arrays ignored).
+- **`QueryOptions.notEq`** — SQL `<>` filter. Under SQL three-valued logic, rows whose column is `NULL` are excluded (matching `<>` semantics); the in-memory test repository mirrors this.
+- **Multi-column `QueryOptions.orderBy`** — now accepts a `string` (unchanged) or an array of `{ column, dir? }` sort specs. A spec that omits `dir` falls back to the top-level `orderDir` (then defaults to `desc`).
+- **`count`** options widened to accept `search`/`in`/`notEq` so totals match filtered `findMany` queries.
+- **`createInMemoryRepository` parity** — the in-memory test double applies `search` (literal, case-insensitive), `in`, `notEq` (NULL-excluding), and multi-column `orderBy` with the same semantics as the SQL repository.
+- **`plumbus ui generate --server-locale-cookie`** — pass-through to `@plumbus/ui` for opt-in server-side locale resolution from the `plumbus-ui-locale` cookie.
+
+Additive and backward-compatible: no methods were added to or removed from the public `Repository` interface, and existing `findMany`/`count` behavior is identical for callers that do not pass the new option keys.
+
+## 0.6.1
+
+### Added
+
+- **`plumbus translation new --locale-folders`** — scaffold per-locale message files under `app/translations/en/` and `app/translations/he/` with a thin `defineTranslation()` assembler.
+- **`plumbus ui generate --split-locale-bundles`** — pass-through to `@plumbus/ui` to emit per-locale i18n bundles under `i18n/locales/`.
+
+## 0.6.0
+
+### Breaking
+
+- **`AICostRecord.operation` union widened** — the type now includes `transcribe`, `synthesize`, and `transport` in addition to `generate`, `extract`, `classify`, and `embed`. Update exhaustive `switch` statements, persisted filters, and dashboards that assume the previous four-value union.
+
+### Added
+
+- **Voice/media cost operations** — `AICostRecord.operation` now also supports `transcribe`, `synthesize`, and `transport`, and cost rows can carry optional `mediaUsage` (`audioInputSeconds`, `audioOutputSeconds`, `characters`, `connectionMinutes`, `participantMinutes`).
+- **`ctx.ai.recordProviderCost(entry, costContext?)`** — public AI service method for recording provider-side spend when a workload does not flow through `generate*`, `extract`, or `classify` (for example STT/TTS/realtime transport). It uses the same in-memory tracker and `onAICostRecorded` hook path as text AI calls.
+- **`ctx.ai.checkProviderCostBudget({ estimatedTokens?, estimatedCostUsd? })`** — pre-check shared daily/per-request budgets before voice/media provider work.
+- **Media-aware budget pre-checks** — `CostTracker.checkBudget()` now accepts `estimatedCostUsd` so voice/media layers can pre-check shared daily USD caps before opening a session or issuing a provider call.
+
+### Upgrading
+
+- Voice/media integrations should call `ctx.ai.recordProviderCost(...)` instead of importing AI-service internals or maintaining a separate ledger path.
+- See [`docs/ai/ai-integration.md`](../../docs/ai/ai-integration.md#upgrading-to-plumbuscore-060) for ledger and budget migration notes.
+
+## 0.5.0
+
+### Added
+
+- **Workers and queues runtime** — unified bootstrap with `PLUMBUS_RUNTIME_ROLE` (`all`, `api`, `worker`), three logical queues (`events`, `flows`, `jobs`), and `plumbus worker start` for split deployments.
+- **`job_executions` table** — durable job status, auth snapshot at dequeue, and `GET /api/jobs/:jobId` for polling completion.
+- **`eventHandler` auto-registration** — optional `trigger: { event, versionConstraint? }` on `defineCapability`; worker pool wires consumers when set. Manual `ConsumerRegistry` registrations still take precedence for the same id.
+- **Optional peer dependencies** — `redis` (durable queues) and `cron-parser` (scheduled flow triggers); in-memory fallbacks with startup warnings when missing.
+- **CLI** — `plumbus worker` (health/ready/metrics), `plumbus events` ops (replay, dead-letter), `plumbus flow dead-letter`.
+- **Runtime exports** — `resolveRuntimeQueues`, `registerCapabilityConsumers`, `dispatchQueuedJob`, `createJobService`, `registerJobStatusRoute`, `RuntimeRole`, and related helpers.
+- **Governance** — advisory rules for split deploy without worker, missing `trigger.event`, and job payload compatibility.
+- **`effects.capabilities` + `ctx.capabilities.invoke`** — sanctioned synchronous capability-to-capability composition with declared dependencies, cycle detection, and runtime `dependencyViolation` enforcement.
+- **Flow auth snapshot** — `flow_executions.auth_snapshot_json` stores the caller's `AuthContext` at start; step execution restores roles/scopes from the snapshot (not worker `system` auth).
+- **`dependencyViolation` error code** — undeclared invoke targets, cycles, missing capabilities, and synchronous job invoke attempts return structured `400` errors with actionable metadata.
+
+### Breaking
+
+- **Canonical capability names only** — registry keys, flow `step.capability`, `effects.capabilities`, `ctx.capabilities.invoke`, generated `RegisteredCapabilityName`, `capability-graph.md`, and MCP manifest tool names use `<domain>.<capabilityName>` (e.g. `billing.approveRefund`). Short local names (`approveRefund`) are no longer valid references outside `defineCapability({ name })`.
+- **Flow step auth** — user-triggered flows no longer auto-inject the `system` role on every step. Steps run under the stored auth snapshot; capabilities must list the caller's actual roles (or `public`) — not rely on implicit `system` elevation.
+- **Job blocking in flows and invoke** — `kind: 'job'` capabilities cannot run synchronously inside flow steps or via `ctx.capabilities.invoke`; use job dispatch, events, or async consumers instead.
+- **Handler `__runtime` invoker stripping** — `invokeCapability`, `resolveCapability`, and `invocationEmitScope` are no longer exposed on handler-visible `ctx.__runtime`; use `ctx.capabilities.invoke` only.
+
+### Changed
+
+- **Worker pool gating** — `plumbus dev` / `plumbus start` start background workers when the app defines events, `eventHandler` or `job` capabilities, or scheduled flows (not only event-triggered flows).
+- **HTTP `kind: 'job'`** — when the API wires `jobQueue` (any job capability on `plumbus dev` / `plumbus start`, including `PLUMBUS_RUNTIME_ROLE=api` without a worker), routes return **202** with `{ data: { jobId, status: "accepted" } }` and enqueue via the jobs queue (previously ran synchronously with **200** because `jobQueue` was not wired on the server).
+- **Job dispatch** — jobs publish to the dedicated **jobs** queue with a `JobQueuePayload` envelope, not the events queue.
+- **Event handler security** — dequeue uses tenant binding from `event_outbox` (fail-closed); trusted replay via `plumbus events` ops.
+
+### Upgrading
+
+See `docs/upgrading-workers.md` in the Plumbus monorepo (not shipped in the npm `instructions/` bundle). Required steps for most apps:
+
+```bash
+plumbus migrate generate && plumbus migrate apply
+```
+
+Apps with `kind: 'job'` HTTP clients must poll `GET /api/jobs/:jobId` instead of expecting a synchronous **200** body. Split production deploys need Redis (`pnpm add redis`) and a `plumbus worker` process.
+
+For canonical capability names, `effects.capabilities`, and invoke policy, see `docs/upgrading-capability-names.md` in the Plumbus monorepo (not shipped in the npm `instructions/` bundle). Consumer apps also ship `instructions/upgrading-0.5-capabilities.md` (agent-facing checklist). Summary:
+
+1. Update flow `step.capability`, `effects.capabilities`, and `ctx.capabilities.invoke` strings to `<domain>.<name>`.
+2. Run `plumbus generate` to refresh `RegisteredCapabilityName`, `mcp-manifest.json`, and `capability-graph.md`.
+3. Run `plumbus verify` — `architecture.non-canonical-capability-reference` and related dependency rules flag stale references.
+4. Run `plumbus migrate generate && plumbus migrate apply` if upgrading to the `auth_snapshot_json` column for flow executions.
+
+### Developer experience
+
+- **Consumer AI guidance — 0.5 capability upgrade** — new `instructions/upgrading-0.5-capabilities.md` (canonical names, invoke policy, flow auth snapshot, migration checklist). `plumbus init` wires it into Copilot, Cursor, and AGENTS.md. `AGENT_WIRING_VERSION` bumped to **7**. Run `plumbus init --patch` on existing projects.
+
+### Non-breaking (default topology)
+
+- `plumbus dev` and `plumbus start` default to **`PLUMBUS_RUNTIME_ROLE=all`** — API and workers remain colocated.
+- In-memory queues remain the default when Redis is not configured.
+- `eventHandler` capabilities without `trigger.event` behave as before (manual registration only).
+- MCP job tasks without a durable Redis queue still execute in-process.
+
+## 0.4.2
+
+### Added
+
+- **Partner API exposure** — optional `exposeAs: ['api']` and inline `api` metadata on `defineCapability` (operationId, method, path, stability, auth scopes, idempotency, test intent).
+- **`@plumbus/api`** — optional peer dependency (version-locked `0.1.x`). Install with `pnpm add @plumbus/api` for manifest validation, OpenAPI/docs export, compatibility diff, test fixtures, and `registerApiRoutes()`.
+- **CLI `plumbus api`** — `validate`, `generate openapi`, `generate docs`, `diff`, `test-fixtures validate`. Commands lazy-load `@plumbus/api` and print an install hint when the package is missing.
+- **API governance** — advisory rules in `plumbus verify` for partner API surface (missing descriptions, public+test conflicts, etc.).
+- **Consumer AI guidance includes partner API** — `plumbus init` wires `@plumbus/api/instructions/*` and `instructions/api.md`. `AGENT_WIRING_VERSION` bumped to **6**.
+- **Flow docs** — convention for passing large payloads by reference through flow state (entity id, not inline blobs).
+- **Ecosystem READMEs** — sibling packages (`@plumbus/mcp`, `@plumbus/ui`, `@plumbus/chat`, `@plumbus/chat-ui`, `@plumbus/knowledge-base`, `@plumbus/browser-extension`) patch-bumped so npm READMEs list `@plumbus/api`.
+
+### Upgrading existing projects
+
+After `pnpm add @plumbus/api`, refresh agent wiring if your project predates the current template:
+
+```bash
+plumbus init --patch --agent agents-md   # or: plumbus init --patch
+plumbus doctor
+```
+
+### Non-breaking
+
+- Capabilities without `exposeAs: ['api']` behave identically on convention routes and in tests.
+- Upgrading `@plumbus/core` does **not** install `@plumbus/api`; install it explicitly when you want a partner-facing API.
+
+## 0.4.0
+
+### Added
+
+- **MCP exposure** — optional `exposeAs: ['mcp']` and `mcp?: { description?, dangerous?, agentTags? }` on capabilities; validation when MCP-exposed; `isMcpExposed()` helper.
+- **`@plumbus/core/mcp`** — `buildMcpManifest`, `renderSkillFile`; `plumbus generate` writes `mcp-manifest.json` and `skills/<domain>/<kebab>.md`.
+- **`zodInputToJsonSchema()`** — exported from `@plumbus/core` for MCP/OpenAPI-style JSON Schema from Zod inputs.
+- **`@plumbus/mcp`** — separate, optional runtime package. Declared as an optional peer dependency of `@plumbus/core` (version-locked `0.4.x`). Install with `pnpm add @plumbus/mcp` when you want to serve capabilities to AI agents; `@plumbus/core` works without it.
+- **CLI `plumbus mcp`** — `serve` (stdio / HTTP), `generate`, `list-tools`. `serve` lazy-loads `@plumbus/mcp` and prints a friendly install hint if the package is missing.
+- **`PlumbusConfig.mcp.agents`** — static agent token map for MCP auth (see `docs/mcp/agent-authentication.md`).
+- **Governance** — advisory rule for MCP-exposed capabilities missing agent-facing descriptions.
+- **Consumer AI guidance includes MCP** — `plumbus init` now wires a new `instructions/mcp.md` reference into Copilot, Cursor, and AGENTS.md so coding agents in consumer projects discover `exposeAs: ['mcp']`, `plumbus mcp serve`, and agent auth. Scaffolded capabilities include a commented-out MCP block as a discoverable next step. `AGENT_WIRING_VERSION` bumped to 3.
+
+### Upgrading existing projects
+
+If you ran `plumbus init` on an earlier version, your `.cursor/rules/`, `.github/copilot-instructions.md`, and `AGENTS.md` files were generated against `AGENT_WIRING_VERSION` 2 and do not reference the new MCP instruction. To pick up the MCP guidance (and any future framework-managed additions), run:
+
+```bash
+plumbus init --patch       # replaces only the Plumbus-managed block; preserves your edits outside the markers
+# or
+plumbus agent sync          # same as --patch plus refreshes .plumbus/briefs/project.md
+```
+
+Use `--dry-run` to preview, or `--force` to replace the entire file.
+
+### Non-breaking
+
+- Capabilities without `exposeAs: ['mcp']` behave identically on HTTP and in tests.
+- Upgrading `@plumbus/core` does **not** install `@plumbus/mcp`; install it explicitly when you want MCP support.
+- `plumbus init --patch` only replaces content between the `plumbus:agent-wiring` markers — user-written content outside those markers is preserved.
+
+## Unreleased — security hardening
+
+### Security
+
+- **Flow conditions** use a sandboxed expression evaluator (`evaluateFlowCondition`) instead of `new Function` (no arbitrary JS).
+- **`auth.internal` removed as a god-mode bypass** — flow engine uses the `system` role; capabilities must list `system` in `access.roles` (included in scaffold).
+- **`define*()` outputs are deep-frozen** (nested plain objects/arrays); Zod schemas and handlers are not frozen.
+- **HTTP/SSE errors** expose only whitelisted `metadata` keys (`field`, `hint`, `reason`, `docUrl`, `httpStatus`, `retryAfter`, …); internal errors return a generic message.
+- **CLI `.env` load** only runs inside a Plumbus project (`config/app.config.ts` or `plumbus.config.*`).
+- **AI env providers** limited to `openai` and `anthropic` only; local Ollama uses `AI_OPENAI_BASE_URL` (no URL validation).
+- **`translation import`** rejects paths outside the project root.
+- **Workers** no longer set `bypassTenantScope` when `tenantId` is absent.
+- **JWT adapter** requires secrets ≥ 32 characters.
+- **Hook failures** (`onCapabilityError`, `onAICostRecorded`, process error hooks) are logged instead of swallowed.
+
+### Fixed
+
+- AI explainability `validation.passed` reflects whether validation succeeded on the first attempt.
+- `define*()` validation throws `PlumbusError` with `code: validation`.
+
+### Developer experience
+
+- **`AGENT_WIRING_VERSION` 4** — `plumbus init` references `@plumbus/knowledge-base/instructions/*` (conventions, defining-sources, providers, chat-integration, testing, README) alongside core, UI, and chat. Run `plumbus init --patch` to refresh existing agent wiring files.
+- Centralized `ErrorHints` / `ErrorDocUrls` for flow conditions, tenant denials, and CLI messages.
+- `DatabaseConnection` + `resolveDatabaseConnection()` used by `plumbus start` / `plumbus dev`.
+- Migration snapshot parsing covered by unit tests (`migrate-snapshot-schema`).
+
+### Migration notes
+
+| Change | Action |
+|--------|--------|
+| Flow `if` expressions | Use `state.field` comparisons (`state.amount > 100`). `ctx.state.*` is normalized automatically. No method calls or arbitrary JS. |
+| `AI_OLLAMA_*` / other `AI_*_API_KEY` | Use `AI_OPENAI_API_KEY` + `AI_OPENAI_BASE_URL=http://…/v1` |
+| Capabilities called from flows | Ensure `access.roles` includes `"system"` |
+| `auth.secret` shorter than 32 chars | Lengthen to ≥ 32 characters (HS256 key strength); `createJwtAdapter` now throws on shorter secrets |
+| Custom `AI_*` providers in env | Only `openai` and `anthropic` are registered |
+| Relying on missing `tenantId` + cross-tenant data in flows | Provide `tenantId` on flow execution or use `access.tenantScoped: false` for admin capabilities |
+
+## 0.3.0
+
+### Upgrade checklist
+
+Read this before bumping to 0.3.0 — there is one mandatory step.
+
+1. **Run migrations.** 0.3.0 adds `lease_owner` and `lease_expires_at` columns
+   (plus a `flow_exec_lease_idx` index) to `flow_executions` to support
+   lease-based claiming. Workers refuse to start until these exist; the
+   startup preflight emits an actionable error pointing at this checklist.
+
+       plumbus migrate generate
+       plumbus migrate apply
+
+2. **`AIIncompleteOutputError` is now thrown when a structured-output call
+   hits `finish_reason === 'length'` / `'max_tokens'`.** Free-text prompts
+   (no `responseSchema`, no `responseFormat: 'json'`) return the partial
+   content as before — no behavior change. If you catch this error and
+   re-issue with a higher `maxTokens`, no migration is needed.
+
+3. **`AICostRecord` now carries `status` (default `'success'`) and an
+   optional `fallbackUsed` boolean.** Reading code is unaffected. If you
+   construct `AICostRecord` literals (custom tracker, test fixture), set
+   `status` explicitly. The `operation` field keeps the original union —
+   `'generate' | 'extract' | 'classify' | 'embed'` — so exhaustive
+   `switch` statements continue to compile.
+
+4. **New typed errors exported**: `AIValidationError`,
+   `AIIncompleteOutputError`, `AIRefusalError`, `LeaseLostError`,
+   `FlowCancelledError`. All extend `Error`; existing `catch` blocks are
+   unaffected. The previous "AI output validation failed after N attempts"
+   `Error` is now this typed subclass.
+
+5. **Failed AI calls now count toward `dailyCostLimit`.** Sunk provider
+   spend on validation-retry loops is real spend; tight ceilings may deny
+   retries sooner on failure-heavy workloads. Set `dailyCostLimit` with
+   headroom, or install `onAICostRecorded` to persist a ledger and tune.
+
+6. **Default AI request timeout remains 120_000 ms.** Set
+   `AIProviderConfig.requestTimeout` if your prompts need longer.
+
+### Added
+
+- `AICostContext` type (in `@plumbus/core` ai module barrel) — optional
+  per-call billing metadata (`projectId`, `serviceArea`, `operationName`,
+  `relatedEntityType`, `relatedEntityId`) you can pass to every
+  `ctx.ai.generate*` / `ctx.ai.extract` / `ctx.ai.classify` /
+  `ctx.ai.streamGenerate` call.
+- `AIServiceConfig.onAICostRecorded` and
+  `ServerConfig.onAICostRecorded(record, costContext, db)` hooks — fired
+  after every AI call completes (success *or* failure) and the in-memory
+  cost tracker has been updated. Use this to persist a ledger row per
+  provider round-trip so sunk spend on retries is visible.
+- `AICostRecord.status: 'success' | 'failed' | 'refused' | 'incomplete'`
+  and `AICostRecord.errorMessage?: string` — new fields on cost records
+  so failures are distinguishable from successes.
+- `AICostRecord.fallbackUsed?: boolean` — set when the streaming call
+  fell back to a non-streaming retry after the streamed output failed
+  validation; both attempts are billed, so this flag is the signal for
+  duplicate-billing detection.
+- `AICostRecordInput` — explicit input type for `CostTracker.record`,
+  keeps `status` optional so pre-0.3.0 call sites keep compiling.
+- `AIValidationError` now carries `usage: TokenUsage`, `model: string`,
+  and `provider: string` so failure-path cost recording knows what the
+  provider actually billed across the retry loop.
+- **`ChatMessage` type** and optional **`messages`** on `ctx.ai.generate`,
+  `ctx.ai.generateWithUsage`, and **`ctx.ai.streamGenerate`**. When `messages`
+  is a non-empty array of `{ role: 'user' | 'assistant', content }` turns,
+  OpenAI and Anthropic adapters send the thread natively instead of folding
+  everything into a single user message built from `prompt`. The rendered
+  prompt description (`buildBasePrompt`) is merged into the provider **`system`**
+  instruction so per-turn template context is preserved; `prompt` is not used
+  as an extra user turn in that mode. Sub-agents and single-turn callers can
+  omit `messages` unchanged.
+- Lease-based flow claiming: workers atomically claim flow executions
+  with `FOR UPDATE SKIP LOCKED`, hold a lease for the duration of the
+  step, and recover crashed-worker rows via expired leases. New
+  `flow_executions.lease_owner` / `lease_expires_at` columns.
+- `FlowService.heartbeat()` — extends the current flow execution lease
+  from inside a long-running step handler.
+- `ExecutionContext.signal: AbortSignal` and `workerId: string` — set
+  inside flow step execution; cooperative cancellation now flows from
+  `flows.cancel()` (and lease loss) through to in-flight AI / HTTP calls.
+- Lease-column preflight: `createWorkerPool().start()` probes
+  `flow_executions` for the new columns and refuses to start with a
+  clear migration prompt when they're missing. Also exported as
+  `assertFlowLeaseColumns(db)` for direct use.
+
+### Changed
+
+- `costTracker.record(...)` now ALSO fires on the failure path when usage
+  is known (e.g. via `AIValidationError.usage`). Previously the in-memory
+  tracker only recorded successful calls, which under-counted sunk spend.
+  Consumers who configured `budget.dailyCostLimit` will see failed-call
+  tokens start contributing to the daily running total. Tight ceilings
+  may deny retries sooner on failure-heavy workloads.
+- `AIIncompleteOutputError` is thrown only when the request is in
+  structured-output mode (`responseFormat: 'json'` or a `responseSchema`
+  is set). Text-mode generations that hit `finish_reason === 'length'`
+  return the partial content as in 0.2.x.
+- `applyMigrations(config)` now returns `Promise<MigrationApplyResult>`
+  (`{ applied, tags }`) instead of `Promise<void>`. Existing callers
+  that ignore the return value are unaffected.
+
+### Migration
+
+For consumers upgrading from `0.2.x`:
+
+1. Run `plumbus migrate generate` then `plumbus migrate apply` (mandatory —
+   see Upgrade checklist item 1).
+2. Existing `costTracker.record({ model, provider, usage, cost, ... })`
+   calls without `status` keep compiling and default to `'success'`.
+3. Existing `catch (err) { ... }` blocks reading `err.attempts`,
+   `err.rawOutput`, or `err.validationMessage` from `AIValidationError`
+   are unaffected — the new fields are additive.
+4. Existing `ServerConfig` usages without `onAICostRecorded` are
+   unaffected — the hook is optional, no hook installed = no behavior
+   change.
+
+To opt into the new failure-ledger persistence:
+
+1. Pass `costContext: { projectId, ... }` on your `ctx.ai.*` calls.
+2. Install `onAICostRecorded` in `createServer({ ... })` and write the
+   `record` (with its `status`) to your persistent ledger.
+
+For chat-style prompts (optional):
+
+- Build `messages` from your transcript (roles `user` / `assistant` only)
+  and pass it alongside `input` for template substitution. Ensure the
+  **last** message is a `user` turn when the model should reply to the latest
+  subject input. No code changes required for consumers that never pass
+  `messages`.

package/README.md

@@ -107,7 +107,7 @@ export const classifyTicket = definePrompt({
     category: z.enum(["billing", "technical", "general"]),
     priority: z.enum(["low", "medium", "high"]),
   }),
-  systemPrompt: "Classify the support ticket and return structured JSON.",
+  system: "Classify the support ticket and return structured JSON.",
 });
 ```
 
@@ -140,16 +140,26 @@ This package ships instruction files that teach AI coding agents (Copilot, Curso
 
 ```
 node_modules/@plumbus/core/instructions/
-├── framework.md      # Core abstractions and project structure
-├── capabilities.md   # Capability definitions and handlers
-├── entities.md       # Entity fields and classifications
-├── events.md         # Event emission and outbox pattern
-├── flows.md          # Workflow steps and retry logic
-├── ai.md             # AI prompts, RAG, cost tracking
-├── security.md       # Access policies and tenant isolation
-├── governance.md     # Advisory rules and compliance
-├── testing.md        # Test utilities and patterns
-└── patterns.md       # Naming conventions and best practices
+├── README.md                      # Index — start here
+├── guardrails.md                  # Mandatory framework boundaries and git safety
+├── framework.md                   # Core abstractions and project structure
+├── capabilities.md                # Capability definitions and handlers
+├── entities.md                    # Entity fields and classifications
+├── events.md                      # Event emission and outbox pattern
+├── flows.md                       # Workflow steps and retry logic
+├── prompts.md                     # Prompt content, system/description, model config
+├── ai.md                          # ctx.ai operations, RAG, cost tracking
+├── translations.md                # i18n catalogs and ctx.translations
+├── security.md                    # Access policies and tenant isolation
+├── governance.md                  # Advisory rules and compliance
+├── testing.md                     # Test utilities and patterns
+├── patterns.md                    # Naming conventions and best practices
+├── cli.md                         # CLI command reference
+├── mcp.md                         # Core MCP surface (exposeAs, plumbus mcp *)
+├── api.md                         # Core partner API surface (exposeAs: ['api'])
+├── deployment.md                  # Production deployment and workers
+├── peer-dependencies.md           # Add-on peer range literals (framework devs)
+└── upgrading-0.5-capabilities.md  # 0.5.x capability invocation migration
 ```
 
 Wire them up with `plumbus init --agent all`.
@@ -165,11 +175,11 @@ Full documentation: [github.com/plumbus-framework/plumbus/docs](https://github.c
 | **`@plumbus/core`** | **You are here.** Foundation — capabilities, entities, events, flows, prompts, translations, runtime, CLI, audit, governance. | Always (required). |
 | [`@plumbus/ui`](../ui/) | Next.js/React UI — typed API clients, auth helpers, form metadata, scaffolds. | When building a Plumbus web UI. |
 | [`@plumbus/api`](../api/) | Partner external API — manifest, OpenAPI, docs, compatibility diff, test intent. | Optional peer `0.1.x` — when publishing a documented partner-facing HTTP API. |
-| [`@plumbus/mcp`](../mcp/) | MCP runtime — serve capabilities to AI agents (`tools/*`, `tasks/*`, transports). | Optional peer `0.5.x` — when exposing capabilities to MCP clients. |
+| [`@plumbus/mcp`](../mcp/) | MCP runtime — serve capabilities to AI agents (`tools/*`, `tasks/*`, transports). | Optional peer `0.5.x \|\| 0.6.x` — when exposing capabilities to MCP clients. |
 | [`@plumbus/chat`](../chat/) | Conversational runtime — `defineChat`, policy guards, context sources, streamed events. | Optional peer `0.1.x` — when adding a chat surface. |
 | [`@plumbus/chat-ui`](../chat-ui/) | React chat UI — hooks and `<ChatPanel />` for the `@plumbus/chat` turn protocol. | Peer of `@plumbus/chat` — when adding a browser chat client. |
 | [`@plumbus/knowledge-base`](../knowledge-base/) | Knowledge providers — scoped sources, registry, chat `knowledgeContext` integration. | Optional peer of `@plumbus/chat` `0.1.x` — when sharing named knowledge across features. |
-| [`@plumbus/voice`](../voice/) | Real-time voice runtime — `defineVoice`, STT/TTS/transport providers, session worker, cost ledger. | Optional peer `0.1.x` — when adding speech I/O (not speech-to-speech); complements `@plumbus/chat` text surfaces. |
+| [`@plumbus/voice`](../voice/) | Real-time voice runtime — `defineVoice`, STT/TTS/transport providers, session worker, cost ledger. | Optional peer `0.3.x` on `@plumbus/core` `0.6.x` — when adding speech I/O (not speech-to-speech); complements `@plumbus/chat` text surfaces. |
 | [`@plumbus/browser-extension`](../browser-extension/) | Extension scaffolder — WXT Chrome/Firefox project wired to your capabilities. | With `@plumbus/ui` (`0.1.x`) — when shipping a browser extension UI. |
 
 ## License