@happyvertical/smrt-ingestion 0.57.21 → 0.57.23

package/FEEDBACK.md

@@ -30,6 +30,110 @@ an authorized existing event; it cannot supply a correctness label or fabricate
 an execution result. An initial denial does not mutate retention. Once authorized,
 a narrowed privacy ceiling survives business rollback and later revocation.
 
+Observed `action_decision` events carry the decision's `decisionMode`
+(`review` or `review_own`) and `selfDecision` flag from the durable decision
+record; `selfDecision` is true whenever the decider submitted the item, in
+either mode. A submitter confirming their own proposal (#4085) is observed as a
+decision by that actor, never as reviewer correctness evidence; consumers that
+derive reviewer agreement must exclude `selfDecision: true` events, and nothing
+in this package turns them into automation eligibility.
+
+## Reviewer routing choices
+
+A `reviewer_routing` record is appended inside the owning transaction when
+feedback is configured and `capture` is authorized, at the moment the choice is
+final: `resolveItem` (REVIEW.md) records `choice: 'none'` keyed
+`routing:<requestId>`, and `submitDecision` records `choice: 'handler'` keyed
+`routing:<decision requestId>` when a reviewer **approves** a reviewer-selected
+action. Handler-choice capture requires `mode: 'review'`: a `review_own`
+submitter approval authorizes normally but appends no reviewer-routing example or
+rule support. A reviewer using the reviewer path may still record an explicit
+routing choice for their own receipt; it does not judge any model suggestion.
+Starting an action records nothing; rejecting or deferring it never produces an
+example. The record needs no model suggestion: it binds the current
+completed/partial attempt (`attemptId`, output digest, analysis revision, frozen
+evidence hashes and the extracted text as `query`), the acting reviewer
+(`reviewer`, also `approvedBy` for a handler choice), the actor who started the
+action (`startedBy`), the start reason, and for a handler choice the
+`handlerId@version`, exact approved arguments and the approved action binding
+(revision and binding hash). Denied or oversized capture, an unavailable source
+pin or malformed evidence skip the projection and leave the authoritative
+decision or disposition intact; database write failures still roll everything
+back. There is no
+supersession: a correction after approval (`previewProposal` on the current
+revision) changes the revision, which retires the old record from retrieval and
+support until the new revision is approved and captured again; a correction
+before approval records nothing until that revision is approved.
+
+Settledness is a selection-time condition. Once a generation used an example,
+or a rule suggestion stored its support, the lineage re-checks (reading the
+generation, reviewing or deciding its actions, reading or adopting the rule,
+and replaying `suggestRoutingRule`, which returns the stored suggestion) verify:
+
+- every record: its digest, that it is not superseded, retention (no
+  redaction or deletion), the unchanged source attempt digest and frozen
+  evidence hashes, and live `retrieve` authority in the same tenant and
+  confidential scope with every generation recipient allowed by the source's
+  provider policy;
+- an `interpretation` record (unchanged from #3676): its action's current
+  revision, binding hash, attempt, handler@version and arguments; the action's
+  target preconditions at their recorded revisions unless the action
+  `succeeded`; the prompt, configuration and model versions; and that the action
+  is not reviewer-selected;
+- a `reviewer_routing` record with `choice: 'handler'`: its action's current
+  revision, binding hash, attempt, handler@version and arguments, that the
+  action is still reviewer-selected, and that its targets are still accessible
+  (not their revisions: the record means the handler choice, not target state);
+- a `reviewer_routing` record with `choice: 'none'`: it has no action, so the
+  source item is re-checked under the first currently offered receiving handler
+  that still authorizes it.
+
+Rule support (`suggestRoutingRule`, and on read, adoption and replay of a stored
+suggestion) additionally requires: at least two records from distinct source
+items; `judgment: 'correct'` with matching prompt/configuration/model versions
+for an interpretation record; `choice: 'handler'` for a routing record; each
+record's handler@version equal to the receiving binding's; and each source's
+live routing projection (or full arguments when no projection is configured)
+equal to the receiving action's. Settledness of a routing source is checked only
+when the support is first selected.
+
+A later disposition, approval lapse, outcome change or target write of a routing
+source never revokes it.
+
+Retrieval scans `interpretation` and `reviewer_routing` records together under
+the same bounds and lexical matcher. A routing example is returned only when its
+digest verifies, its attempt and evidence hashes are unchanged, the source item
+is still readable in the same tenant and confidential scope, every generation
+recipient is allowed by the source's live provider policy, and (for a chosen
+handler) the action is still reviewer-selected and settled — applied
+(`executing`, `succeeded`) or `authorized` with an unexpired approval, on an item
+without a disposition — its binding is still the current revision, and the handler with its candidate
+references is currently offered to the receiver. A `none` choice is
+revalidated under the first currently offered receiving handler the source item
+authorizes. Routing examples carry `kind: 'reviewer_routing'`,
+`judgment: 'correct'`, `routing: { choice, reason }`, `handlerId: 'none'` with an
+empty action for closed items, and no model/prompt/configuration versions, since
+no suggestion was judged. They are included in provider input and selection
+provenance exactly like interpretation examples and are rechecked by the same
+lineage gates.
+
+`suggestRoutingRule` accepts `reviewer_routing` support only for `choice: 'handler'`
+records whose action is settled in the same sense when the support is chosen
+(a lapsed, unapplied approval or a disposition excludes it) and whose handler, version and
+full arguments match the receiving binding and each other, from distinct source
+items, under the live routing projection; `none`, rejected, deferred and
+re-corrected records never support a rule. Suggestions and adoptions keep every existing #3676
+rule: no catalog, candidate, permission or automation expansion, and
+`automaticActionEligible: false`. Reviewer-selected actions themselves are refused
+by `authorizeAutomatic`. They are routing choices, never labels for a model
+suggestion: `recordFeedback` refuses an interpretation judgment on one (even
+when the item also carries a generation, and whatever its decision), retrieval
+and rule support ignore interpretation records bound to one, and `ActionReview`
+hides the correctness buttons for them. A disposition says nothing about
+whether a suggestion was right, so interpretation judgments on a resolved item
+still count. Only
+`startReviewerAction` mints that origin; `createAction` refuses it.
+
 ## Retrieval and provider boundaries
 
 `retrieveFeedback({ itemId, query })` derives the current offered catalog,

package/PROPOSALS.md

@@ -85,6 +85,25 @@ Expired, cancelled, revoked, retried or superseded workers cannot read or publis
 current lease and each offered handler/candidate are rechecked immediately before
 each outbound generative/decision boundary and before publication.
 
+### Inline interpretation
+
+Receipts created with `dispatch: 'inline'` (text/structured parts only) run the
+same stages in-process through `interpretInline(itemId, { requestId, extraction
+})`: an immutable extraction revision claimed under a fenced lease and extracted
+with the built-in UTF-8 extractor, then `prepareGeneration` and a fenced
+interpretation revision generated by `generateProposals`. The analysis lease,
+catalog digest and live catalog/candidate checks, source pin, provenance and
+`StageFailure` categories are identical to the job path; no dispatch intent is
+ever deliverable and nothing is enqueued. A retry with the same `requestId`
+replays the recorded revisions rather than re-preparing against a superseded
+source, also under concurrency. The interpretation append rechecks, under the
+item lock, that its pinned extraction is still the latest revision and refuses
+a superseded pin: first through the general latest-extraction check that
+every interpretation append runs (`GenerationSourceStaleError`), then by the
+inline check that the pinned attempt is still current (`Generation source changed`). Generated suggestions still require explicit preview and a human
+decision, either through the reviewer surface or the submitter-scoped surface
+described in [REVIEW.md](REVIEW.md).
+
 ## Providers and validation
 
 `createSDKProposalGenerator(client, identity, {maxTokens,timeoutMs})` uses an
@@ -161,6 +180,64 @@ The SDK adapter constrains replies with a `json_schema` response format built
 from the validator's schema and the fixed instructions describe the shape; a
 custom `ProposalGenerator` receives no schema, so its raw reply meets the
 validator directly and the same reason is recorded.
+The provider-facing schema is made portable (`portableResponseSchema`, #4260):
+keywords that common OpenAI-compatible structured-output backends reject are
+removed at schema positions, including inside handler argument schemas
+(`uniqueItems`, `contains`, `multipleOf`, `format`, `patternProperties`,
+`propertyNames`, `min`/`maxProperties`, `not`, `if`/`then`/`else`,
+`dependent*`, `unevaluated*`, and the `default`/`examples` annotations; the
+full list and reasons are in `src/proposal-sdk.ts`). With `uniqueItems`, for
+example, vLLM's xgrammar feature check flags the schema, its auto mode routes
+it to llguidance, and llguidance answers
+`400 Grammar error: Unimplemented keys: ["uniqueItems"]`. This only loosens the
+grammar: the validator still enforces the constraints, including unique
+`alternatives` and `missingFields`, and the fixed instructions ask for distinct
+entries. The exception is `format`, which is advisory: the validator runs Ajv
+with `validateFormats: false`, so `format` is enforced nowhere. When
+`patternProperties` is removed, the same object's `additionalProperties` is
+removed too, so the grammar does not shrink to `{}`. The `contentSchema` annotation is omitted as well, so nested unsupported
+constraints cannot leak through an unvisited schema position. A backend may
+recursively inspect typed schema-like objects even inside `enum` or `const`
+data. When such data contains an unsupported keyword, the whole containing
+`enum`/`const` constraint is omitted conservatively; literal data is never
+rewritten. Harmless property names and ordinary enum/const data stay intact.
+The original validator remains authoritative for omitted literal constraints.
+There is no automatic `json_object` fallback.
+
+Invariant: the provider copy only ever accepts a superset of what the original
+accepts. Each removed keyword is a conjunct, so removing it only widens. The two
+places where removal would narrow are rewritten:
+- every `oneOf` becomes `anyOf` (or joins an existing `anyOf` through `allOf`),
+  because loosened `oneOf` branches could both match and reject a value the
+  original accepted;
+- an object that loses `patternProperties` also loses `additionalProperties`.
+
+Provider-only `$ref`, `$dynamicRef`, and `$recursiveRef` constraints are also
+removed. Rewriting or deleting a subschema can invalidate pointer or anchor
+targets, including targets inside nested `$id` resources. Dropping references
+conservatively widens the grammar without resolving untrusted schema resources;
+property names and enum/const data named `$ref` remain unchanged. Referenced
+constraints, recursion, and dynamic scope remain enforced by the original
+validator, so provider guidance for reference-based arguments is less precise.
+
+The validator still checks the original, including `oneOf` exclusivity. A
+property-style regression checks original-valid ⇒ portable-valid across the
+removed keywords. Handler argument schemas may still carry `allOf` through
+unchanged; its support across backends is unverified, so check such a handler
+against the deployed backend.
+
+A generator call that throws ends the attempt as `provider_error` with the
+`unavailable` warning and `provenance.failure` `{category: 'unavailable',
+reason}`. The reason is value-free by construction: a provider's own text can
+quote the prompt or document, so it is never copied. The reason is built only
+from a fixed phrase and, for an SDK `AIError` only, the HTTP status (from a
+structured field, or from the exact `<status> ...` / `TypeSafe request failed
+(<status>): ...` shapes the SDK produces), the SDK error code (from the closed
+list of codes the SDK raises) and, for a schema rejection, quoted tokens that
+are JSON Schema keywords. Any other error contributes at most a structured
+numeric status. For example: `provider rejected the response schema (HTTP 400,
+code API_ERROR); unsupported response-schema keywords: uniqueItems`, or just
+`provider call failed`. It is cut at 500 characters.
 
 `createSDKProposalDecisionClient(client)` adapts an existing SDK client’s optional
 capability without changing its errors or protocol. Optional `decision` uses public core `executeDecision` and the existing SDK

package/README.md

@@ -197,11 +197,22 @@ Mount the `/svelte` inbox and review components through an authenticated host us
 the browser-safe `IntakeReviewHost` DTO contract. See [REVIEW.md](REVIEW.md) for
 exact revision-bound callbacks, live evidence access, logical split corrections,
 assignment ownership, and the maintained upload-to-domain-result browser fixture.
+`IntakeWorkspace` puts the inbox, the selected item with its original inline,
+the message box and approve-and-advance on one page with keyboard next/previous;
+it uses the same callbacks and adds no authority (REVIEW.md, "Single-page triage
+workspace").
 
 Optional explicit feedback and scoped example retrieval are documented in
 [FEEDBACK.md](FEEDBACK.md). Approval and execution outcomes remain separate from
 interpretation correctness; routing policy adoption is explicit and versioned.
 
+Reviewers can start any offered handler for an item (`startReviewerAction`) or
+close it as handled without action (`resolveItem`); the published host adapter
+wires both as `startAction`/`resolve` and exposes the offered `catalog`. Hosts
+with their own transport add the two POST operations behind the same session,
+CSRF and reviewer checks as `decide`. The `intake_dispositions` table requires
+a manifest migration before upgrading.
+
 ## Recipe preview
 
 `ingestion.inbox` and `ingestion.review` declare the user-facing inbox and
@@ -248,6 +259,121 @@ and `maxSuggestions` are each capped at 20; `timeoutMs` is capped at 2147483647.
 Invalid limits are rejected when constructing `IngestionService`, before receipt
 or interpretation. See [REVIEW.md](REVIEW.md) for the published server review host.
 
+## Trusted internal sources
+
+A host with a trusted internal source, such as chat messages recorded in
+smrt-messages, can receive **by reference**, interpret **inline**, and let the
+**submitter** confirm their own proposal (#4085). None of these modes changes
+copied receipts: their identity, validation, queueing and retention are
+untouched, and a replay that switches mode conflicts.
+
+**Receive by reference.** A part may be `{ partId, reference: { owner, id,
+version? }, snapshot: { text } | { structured } }` instead of bytes. The
+service derives UTF-8 bytes once (`text/plain` or canonical `application/json`;
+text, reference fields and owner identity containing NUL or unpaired
+surrogates are rejected `invalid` on every engine, and `expireOwned` refuses
+them, so distinct owners can never collapse onto one stored key),
+hashes them into the receipt identity, and stores only that snapshot in the
+durable evidence row; no AssetStore object or asset record is created, and
+`readEvidence` serves the snapshot after the same hash/length check. The
+reference is identity and provenance, never authority. Such parts must be
+text or structured; files still arrive as copied bytes.
+
+**Owner-bound retention.** `retention: { kind: 'owner', owner: { kind, id },
+version, expiresAt, replayUntil, acceptAfter }` binds the receipt to its
+owner's lifecycle. The time deadlines remain required and validated exactly as
+before: they are the hard privacy ceiling, and `sweepRetention` and execution's
+policy ceiling still apply. When the owner is deleted the host calls
+`expireOwned({ kind, id })` from its deletion path; every active receipt bound
+to that owner is revoked and removed through the existing `expire` /
+`repairDeletions` protocol, which redacts the snapshot together with every
+other payload, and the deleted delivery is never re-admitted. The owner key is
+recorded in the indexed `owner_key` column (`<kind>:<id>`, so `kind` cannot
+contain `:`). Deletion redaction blanks `owner_key` and `submitter_id` together
+with every payload: only the receipt identity (`delivery_key`) and the captured
+ceiling survive, and the ceiling loses `execution.principalId` unless an
+external action is still `executing` or `outcome_unknown`. So once nothing is
+left to reconcile, a redacted row no longer links an owner to an actor, and a
+repeated `expireOwned` finds nothing. Redaction also overwrites every
+identity- or content-derived digest (`payload_digest`, `scope_digest`, analysis
+`input_digest`, attempt `output_digest`, plan `digest` and proposal, decision
+and execution `binding_hash`) with the fixed marker `redacted`; successful
+result digests remain as the replay tombstone. The principal is retained only
+for an external action that was still in flight when the item was redacted,
+because reconciling it authorizes under that principal; it is dropped when that
+reconciliation reaches a final state, or by the next `repairDeletions` /
+`sweepRetention` pass. For a self-approvable receipt the principal is the
+submitter, so until then it identifies the sender. Later deletion sweeps pass
+this redacted ceiling to the host's `authorize({ operation: 'delete' })`, so
+delete authorization must not depend on `execution.principalId`: a host that
+requires it would refuse, and the sweep would stop with later deletion intents
+still pending. A successful result digest
+is a plain SHA-256 of the canonical result, so it can confirm a guessed result:
+keep handler results to record ids or opaque references, never content.
+
+A replay of a deleted delivery is rejected and never becomes fresh work. While
+deletion is still pending it returns `rejected/expired`; once redaction has
+completed the identity digests are gone, so it returns `rejected/conflict`.
+Sources treat both as terminal and must not retry. `delivery_key` stays in plaintext because it is
+the unique receipt identity and the package holds no secret to key it: hosts
+must use an opaque value (for example a random UUID recorded with the message)
+and must not encode owner ids, sender ids or content in `deliveryKey`.
+
+Host ordering for owner deletion: first stop issuing receipts for the owner,
+then call `expireOwned`, and call it again once in-flight receipts settle (or
+from a periodic sweep). A receipt still in flight when its owner is expired
+returns `{ kind: 'rejected', category: 'expired' }` rather than `accepted`, but
+one that reserved its row after the first `expireOwned` is only found by the
+later call; until then the time ceiling bounds it. Services are confidential-scope bound, so
+a host whose owner has receipts in several confidential scopes calls
+`expireOwned` from each scope's service; a foreign tenant or scope sees zero.
+
+**Inline mode.** `receive({ ..., dispatch: 'inline' })` requires text/structured
+parts only, records the route provenance but **no deliverable dispatch intent**
+for any revision (every intent is retained as `completed`, so no worker or
+`repairDispatches` ever picks it up), and does not enqueue. The host then calls
+`interpretInline(itemId, { requestId, extraction: { configurationRevision,
+limits } })`, which runs in-process exactly what the job path runs: an
+immutable `stage: 'extract'` revision claimed under a fenced lease and
+extracted with the built-in UTF-8 extractor, then `prepareGeneration` and the
+`stage: 'interpret'` revision with the same analysis lease, catalog-live
+checks, provenance and failure categories as `generateProposals`. A retry with
+the same `requestId` replays the immutable revisions, including when two
+identical calls race. The interpretation is appended under the item lock only
+while its pinned extraction is still the latest revision; if another request
+reprocessed the item meanwhile it fails with `GenerationSourceStaleError` (the
+general latest-extraction check every interpretation append runs, #4228) or,
+for a pin whose extraction attempt was replaced, `Generation source changed`. The result reports each
+stage's attempt and safe error and, when generation completed, the authorized
+`getCompletedAnalysis` snapshot. Inline is for send-triggered suggestions that
+must not depend on the scheduler; it is not a bypass of review or execution.
+
+**Submitter-scoped approval.** The service records `submitter_id` as the
+authenticated `scope.actorId` of the receiving service, server-side; payload
+and source metadata never supply it. `listOwnProposals({ cursor?, limit? })`
+lists only this actor's own active submissions with their saved review pages,
+reloaded under current item/handler/target authority, without the reviewer
+inbox. A submission the actor can no longer read is skipped before projection
+and never named by the cursor; a readable one whose reviews are denied by
+ingestion authority is returned as `{ item, reviews: null, unavailable: true }`
+with no reason, so one revoked grant never hides the others. `submitOwnDecision(input)` accepts `approve`, `reject` or `defer` (never
+`correct`) under the `review_own` authority: the host's execution `authorize`
+must return `submitter: true` for that actor, the application policy must opt
+in with `submitterReview: true` (narrower layers may only disable it), and the
+item's recorded submitter must equal the current actor. Every other check is
+the ordinary one, and `applyAction` remains the single once-only effect path
+under the approver's live grants. Because the effect runs as the receipt's
+captured `capturedCeiling.execution.principalId`, `review_own` additionally
+requires that principal to be the submitter itself (`submitter_id ===
+scope.actorId === principalId`), checked under the item lock at decision time
+and again at apply; a receipt captured for a bot or service principal can only
+be approved by a reviewer. The decision is recorded with `mode` (`review` or
+`review_own`) and `selfDecision`, which is true whenever the deciding actor is
+the item's submitter, in either mode (mirrored as `decisionMode` /
+`selfDecision` on feedback events), so a self-confirmation is never mistaken
+for independent review and never contributes to automation eligibility. See
+[REVIEW.md](REVIEW.md) and [execution integration](agents/execution.md).
+
 ## Dispatch route ownership and legacy recovery
 
 A receipt retains a versioned, server-owned `jobTarget` snapshot, and every