@happyvertical/smrt-ingestion 0.55.6 → 0.55.8

package/AGENTS.md

@@ -27,9 +27,19 @@ submission never run inside ingestion database transactions.
 
 Validation: `pnpm test`, `pnpm test:postgres`, `pnpm test:providers`, `pnpm typecheck`, `pnpm build`.
 The PostgreSQL command provisions a real database and fails if unavailable.
+Fixture teardown uses `src/test-support/postgres-cleanup.ts`: only owned
+exec_/ing_/ext_/sources_ names, normal DROP, bounded 55006-only retry. Never
+force-terminate backends, grant signal privileges, or ignore leaked sessions.
 Build includes browser import validation. The provider lane exercises local SDK
 adapters and process isolation, without claiming live remote recognition quality.
-The component/e2e/evaluation
-commands are reserved by ADR 0004 and fail explicitly until their owning children
-supply real suites.
+`src/svelte` owns transport-neutral review components and browser-safe host callbacks.
+See [FEEDBACK.md](FEEDBACK.md) for revision-bound explicit feedback, scoped example retrieval,
+provider-boundary revalidation and authorized routing policy adoption.
+See [REVIEW.md](REVIEW.md) for authenticated hosts, bounded saved-review reloads,
+assignment ownership, logical split re-extraction and the maintained browser proof.
+UI validation additionally requires `pnpm test:components` and `pnpm test:e2e`;
+`typecheck` includes Svelte and `build` includes the types-first `/svelte` entry.
+`pnpm test:evaluation` runs repository evaluation tooling/format/fault contracts.
+Measured held-out quality is a separate frozen-protocol, budget-gated run; tooling
+success does not establish reference quality or permit automatic actions.
 See ../../docs/test-matrix/3670-ingestion-foundation.md for behavior coverage.

package/FEEDBACK.md

@@ -0,0 +1,156 @@
+# Explicit ingestion feedback
+
+Configure `IngestionOptions.feedback` to opt in. The existing `intake_feedback`
+ledger owns these records; no new schema, embedding service or provider is added.
+`FeedbackConfiguration` sets a version, maximum examples, scan count, serialized
+bytes and lexical similarity threshold, plus a live scoped authorization callback.
+Hard ceilings are 20 examples, 1,000 scanned records and 1 MB. Choose substantially
+smaller application limits. The proposal's complete serialized request, including
+examples, must also fit the current minimum application/source/provider budget.
+
+`recordFeedback` accepts the exact displayed action revision, review version and
+binding hash, an explicit correct/incorrect judgment, optional reason text and a
+request ID. The server resolves reviewer, tenant/confidential scope, retained
+source evidence, extraction/generation versions and current handler arguments.
+A replay returns the same immutable receipt; changed payloads conflict.
+`supersedesId` appends a replacement for the specified prior interpretation
+judgment. A second successor conflicts. Revised actions make old bindings
+ineligible. Approval and successful execution never imply that extracted fields
+are correct.
+
+Decision and downstream execution facts are captured inside their owning
+transaction, keyed by immutable event ID and signal, when the configured capture
+authority allows the optional memory projection and it fits the configured byte
+cap. Denial or an oversized automatic projection leaves the underlying
+authoritative decision/outcome intact, including failure and reconciliation.
+Explicit `recordFeedback` and `observeFeedback` requests still reject invalid or
+oversized writes; actual database failures still propagate and roll back. Failed/rolled-back domain
+work and unknown external outcomes remain distinct. `observeFeedback` can replay
+an authorized existing event; it cannot supply a correctness label or fabricate
+an execution result. An initial denial does not mutate retention. Once authorized,
+a narrowed privacy ceiling survives business rollback and later revocation.
+
+## Retrieval and provider boundaries
+
+`retrieveFeedback({ itemId, query })` derives the current offered catalog,
+candidates and model/prompt/configuration versions on the server. Callers cannot
+provide candidate authority. The deterministic matcher uses lowercase Unicode
+word-set Jaccard similarity (minimum 0.2), then immutable record ID for ties.
+Scanning, recursive provenance checks and returned examples are bounded; a limited scan does not promise an
+exhaustive history search. Both positive and negative explicit examples are kept.
+
+Every selected source must still be readable in the same tenant and confidential
+scope, with its current action/analysis binding, target grants, handler version,
+model/prompt/configuration compatibility and retention. Each source's current
+provider policy must allow the generation recipient and any configured decision
+recipient; the source owner rechecks that policy on both initial and final live
+contexts. Receiving checks cover every offered handler, before and after source
+checks, using each live grant, actual provider recipients and the narrowest byte
+ceiling. Both retrieval and reference revalidation budget the full reconstructed
+examples. A receiving item's provider grant cannot substitute for a source grant.
+Superseded, revoked, deleted or incompatible sources are excluded. Current explicit human corrections
+take precedence: a generation carrying one does not retrieve learned examples.
+
+Full examples exist only in the authorized provider request. The SDK proposal
+adapter includes that typed selection in its serialized user message when
+present; disabled feedback leaves the original request shape unchanged. Saved generation
+provenance contains the receiving query, selection configuration digest and
+immutable feedback IDs/digests, not copied source example content. Selected
+references are rechecked before generation, capability probing, decision calls,
+publication and completed-result reads. A newly added example does not invalidate
+an older selection; a revoked or superseded selected example does. Feedback
+lineage is bounded and fails closed. Saved-action reads and pending decisions,
+automatic authorization, preview publication/replay and execution also revalidate
+selected lineage under the
+owning transaction. Revoking, deleting or superseding a selected example blocks
+pending actions; regenerate their proposals and obtain fresh approval to recover.
+A preview replay checks the exact persisted proposal it returns, including its
+analysis attempt; a newer action revision cannot supply replacement lineage.
+Plan replay and final sealed plan-step publication use the same gate.
+For already successful actions, current-authorized durable outcomes remain
+readable/replayable, but invalid learned lineage suppresses saved proposal
+display, arguments and plan payload. An empty selection carries no learned source
+and does not turn optional feedback-capture denial into action denial. Examples
+never expand the offered catalog, candidate IDs, operation permissions or
+automation eligibility.
+
+All durable feedback belongs to the source item's existing retention/redaction
+lifecycle. There is no separate memory store or delayed indexing writer to clean
+up. Hosts using other derived stores still implement the existing `purgeDerived`
+contract. Core `LearningMemory` is intentionally not used here: its success/failure
+reinforcement and default ancestor-scope fallback do not represent separate
+correctness/decision/outcome signals and strict confidential-scope eligibility.
+
+## Routing suggestions and adoption
+
+`suggestRoutingRule` requires repeated current explicit positive judgments
+from distinct source items with matching handler/version and arguments, and returns supporting immutable IDs,
+shared query terms and the policy owner's preview/version. Suggestions cannot
+execute or modify policy. `getRoutingRule` rechecks the displayed binding and all
+supporting examples. Arguments are routing preferences, never executable grants.
+
+By default matching compares **all** action arguments, preserving existing host
+behavior. Create handlers commonly have per-document content and idempotency
+keys, which must remain distinct. Such hosts can explicitly declare a routing-only
+subset in trusted server configuration before capturing judgments:
+
+```ts
+feedback: {
+  // ...existing limits, authorize and policy owner
+  routing: [{
+    handlerId: 'host:create-document',
+    handlerVersion: '1',
+    version: 'routing-fields-1',
+    fields: ['destinationId', 'category'],
+  }],
+}
+```
+
+Each declaration must name a configured handler/version and a nonempty, unique
+allowlist of its top-level `argsSchema.properties`; duplicate declarations,
+unknown properties and prototype keys reject. There are no dotted paths,
+caller-supplied selectors or implicit exclusions (including request IDs).
+Optional absent fields stay absent and differ from explicit `null`; at least one
+selected value must be present. The host must select preferences only, never
+grants, provider permissions or automation controls.
+
+Capture retains the full immutable action arguments and binding plus the
+versioned subset. For opted-in handlers `RoutingRule.args` contains only selected
+values and `RoutingRule.projection` carries the canonical sorted fields and
+version. Independent support must match that entire contract and those values;
+full original source arguments, provenance and current access are still checked.
+Existing unprojected feedback is not retroactively reinterpreted: record fresh
+explicit judgments after enabling or changing a declaration. Changing a version,
+field list or selected value invalidates old suggestions on reads, adoption and
+replay, including changes inside policy-owner callbacks. Full argument feedback
+retrieval remains unchanged; hosts may present matching examples from
+`retrieveFeedback`, but suggestion creation revalidates every supplied ID.
+
+`adoptRoutingRule` requires the exact suggestion digest and expected policy
+version plus current reviewer and adoption authority. The configured policy owner
+must perform its version compare-and-swap and audit on the supplied database
+executor. The ingestion transaction records the exact returned version/audit ID.
+Replays do not adopt twice. Revocation or permission/policy widening rolls back
+both changes. The host callback may store routing preferences only: it must not
+change grants, provider permissions or automation controls. Adoption always
+returns `automaticActionEligible: false`; subsequent proposals retain normal
+current policy and explicit review requirements.
+
+## Review UI and evidence
+
+The optional `IntakeReviewHost.feedback` callback carries the exact displayed
+revision and binding. Correct/incorrect controls are explicit and independent of
+Approve/Apply. Unsaved argument edits disable correctness controls until the
+displayed binding is current again. Hosts must authenticate that callback and connect it to
+`recordFeedback`; client-supplied reviewer/scope fields are not accepted. The
+maintained browser fixture demonstrates this transport with CSRF protection and
+rejects substituted revisions.
+
+See [the behavior matrix](../../docs/test-matrix/3676-ingestion-feedback.md) for
+validation and the frozen learning protocol, and the
+[routing projection matrix](../../docs/test-matrix/3732-ingestion-routing-projection.md)
+for the optional subset contract. The small authored deterministic
+cohort demonstrates contract-level influence, not provider quality or measured
+production improvement. Real-provider evaluation belongs to #3677. There are no
+sampled automatic outcomes in this feature; reporting that stratum as unavailable
+avoids inventing correctness labels from successful jobs.

package/README.md

@@ -65,9 +65,10 @@ Validation commands are `pnpm test`, `pnpm test:postgres`, `pnpm test:providers`
 `pnpm build` (including built browser import inspection). The PostgreSQL lane
 requires the repository's configured database service and never silently skips.
 The provider lane exercises local SDK adapters and process isolation; it does not
-claim live remote recognition quality. Component, e2e and evaluation command names are reserved by the ADR and
-explicitly unavailable until their owning children deliver the corresponding
-capability. See the [behavior matrix](../../docs/test-matrix/3670-ingestion-foundation.md).
+claim live remote recognition quality. `pnpm test:components` and `pnpm test:e2e`
+cover the maintained review surface. `pnpm test:evaluation` checks repository
+evaluation tooling contracts; measured held-out quality requires a separately
+frozen protocol and an approved aggregate provider budget. See the [behavior matrix](../../docs/test-matrix/3670-ingestion-foundation.md).
 
 ## Authoritative review and execution
 
@@ -149,3 +150,14 @@ only receipt preservation, never domain writes, AI tools or external actions.
 
 See [PROPOSALS.md](PROPOSALS.md) for bounded application catalog discovery, durable
 interpretation revisions, optional typed decisions and explicit generated preview.
+
+## Human review
+
+Mount the `/svelte` inbox and review components through an authenticated host using
+the browser-safe `IntakeReviewHost` DTO contract. See [REVIEW.md](REVIEW.md) for
+exact revision-bound callbacks, live evidence access, logical split corrections,
+assignment ownership, and the maintained upload-to-domain-result browser fixture.
+
+Optional explicit feedback and scoped example retrieval are documented in
+[FEEDBACK.md](FEEDBACK.md). Approval and execution outcomes remain separate from
+interpretation correctness; routing policy adoption is explicit and versioned.

package/REVIEW.md

@@ -0,0 +1,137 @@
+# Authenticated review surfaces
+
+`@happyvertical/smrt-ingestion/svelte` exports `IntakeInbox`, `IntakeReview` and
+`EvidenceViewer`. The browser-safe DTO entry exports their `IntakeReviewHost`
+callback contract. Import styling/theme configuration from smrt-ui in the host.
+Components use smrt-ui primitives and the `ingestion.*` i18n catalog.
+
+The host authenticates every request, resolves tenant/actor/confidential scope
+from its session, enforces CSRF on mutations, and constructs `IngestionService`.
+Never accept scope, reviewer, permissions or a principal from callback JSON.
+Only supply authorized evidence URLs and escaped text; original URLs are
+same-origin authenticated handlers that recheck `readEvidence` on every read.
+Candidate discovery uses `findCandidates` with current grants. Presentation
+state and disabled buttons provide no server authority.
+
+`listReviews(itemId, {cursor?,limit?})` reloads saved action identities and their
+current plan membership without browser storage. It returns at most 50 actions,
+with an opaque action-ID cursor. Every payload requires current item, handler,
+parent plan and target access under the owning transaction. Denied reads fail
+closed. Target reads check current authorization separately from historical
+revision freshness: unfinished drift returns state-only stale reviews without
+removing unrelated entries; succeeded results remain readable after authorized
+target changes. Apply still enforces exact bound target revisions. Expired entries
+contain only review identity/state. Stale entries never disclose old args or target
+labels; an authorized parent plan additionally exposes `stalePlan` identity, key,
+current plan revision and handler identity. Hosts must collect fresh explicit args
+from the reviewer/current authorized evidence and call `previewPlan` with that
+exact plan CAS and the current completed analysis attempt. Action revision is not
+plan revision. Re-expansion preserves action identity and requires fresh approval.
+Success results are rechecked through the durable result reader.
+Plan membership contains the exact parent args/version so edits call
+`previewPlan`, not per-step correction. An item may have more review pages; the
+host must aggregate state across pages when projecting a production inbox state.
+
+The host's `load` callback supplies a current `ItemReviewView`, using
+`getCompletedAnalysis` with no historical fallback. Generated suggestions bind
+the **current generation attempt**, not `generation.source.attemptId` (the prior
+extraction). `previewGeneratedProposals` publishes exact reviews and continues to reject stale
+generation candidate pins. When `listReviews` reports `generationStale`, the host
+omits obsolete generation output and renders authorized saved reviews/results.
+Completed saved actions do not hide healthy unprepared suggestions. The saved-binding hint is not
+an authoritative generation-availability check: a fresh human preview can bind a
+current target while the immutable generation still pins its former revision.
+`getCompletedAnalysis` remains strict and throws server-only
+`GenerationSnapshotStaleError` only after every current catalog/provider/candidate
+read is authorized and an immutable candidate revision differs. The maintained
+host catches only that class, reacquires `listReviews` under current owning
+authority, and returns that new page without generation output. It never reuses
+the earlier page after an error. Denials and unknown failures propagate; healthy
+generation still exposes unprepared sibling suggestions. Stale entries expose handler
+and attempt identity only when that attempt is still the current completed/partial
+analysis. An empty-args editor submits fresh explicit human arguments through
+`previewProposal` or `previewPlan` with the saved CAS; it never reuses old generated
+arguments, rewrites the generation snapshot or transfers approval. Analysis-stale
+entries without an attempt require normal reprocessing. Approve,
+reject, defer and correct call `submitDecision`; corrected args create a new
+revision which requires a separate human approval. After an approve/defer/reject
+decision, optional `editAction` calls `previewProposal` with the current action
+revision to publish a new review; completed/executing/unknown actions cannot be
+revised. Local dirty args disable Apply but confer no authority. Plan edits re-expand through
+`previewPlan` with the current **plan** revision. `applyAction` accepts only an
+action ID. Unknown external outcomes require server reconciliation; the UI
+never offers a blind resend. Stable result IDs survive reload and retry.
+
+Replace `contextKey` whenever authenticated tenant/session context changes.
+Changing it or itemId destroys the old view and fences late callback results.
+Denied refresh clears prior originals, candidates, arguments and results.
+Missing confidence is unavailable; page/time links exist only for observed
+locations. Provider errors and unresolved/partial results remain review states.
+
+## Assignment and logical splits
+
+Assignment is explicitly host-owned, durable and versioned. `assign` receives
+an expected assignment version and request ID; the host must authorize the
+manager, constrain eligible assignees, CAS the durable record and reject stale
+writes. Assignment does not grant read, review or execution rights. The maintained
+fixture provisions its own SQLite assignment table before serving requests; no
+production ingestion table or migration is introduced for assignment.
+
+`reviseLogicalSplit` authorizes the current generation through the same-executor
+catalog/candidate gate and the existing reviewer grant and policy reviewer list.
+Process access alone cannot mint or replay a human correction; reviewer authority
+is checked before mutation and again before publication. A generation without a
+reviewable catalogue cannot be corrected through this reviewer API.
+It requires all observed pages exactly once, an exact
+current attempt/revision and a request ID. It records the authenticated actor
+and predecessor as a **human correction** in a new immutable extraction
+configuration, queues re-extraction and immediately makes previous suggestions
+and approvals stale. It never changes original bytes/evidence or labels a human
+edit as model output. The host processes the new extraction, then calls
+`prepareGeneration` and the normal fenced interpretation stage. The supported
+immediate-predecessor extraction pin is unchanged. The generator receives a
+human correction only after its digest matches the authoritative source. Ordinary
+`analyze` calls cannot mint an extraction correction; the owning split API is the
+authenticated origin, and forged interpretation corrections fail before provider
+disclosure. If its proposed grouping differs from the correction, suggestions are withheld
+with `human_split_not_respected`. New proposals require new explicit review.
+The reference host runs these stages synchronously for browser determinism;
+production hosts dispatch the existing durable analysis jobs.
+
+`feedback` is an optional explicit correctness callback, separate from approval
+choice and execution success. No learning, feedback storage or rule promotion is
+implemented by these components; #3676 owns those services. Do not pass feedback
+or a feedback callback to readers without the relevant host grant.
+
+## Maintained browser proof
+
+`pnpm test:components` runs real Svelte component tests; `pnpm test:e2e` runs the
+package-local authenticated Vite application under Chromium. The fixture uses
+HttpOnly same-site sessions, CSRF, real SQLite/assets/ingestion/content APIs,
+production unpdf child extraction and a deterministic proposal generator. Its
+hardcoded test login is fixture-only and is never a production auth provider.
+Image/audio originals remain inspectable when no OCR/speech provider is
+configured; the suite makes no recognition-quality claim and spends no provider
+budget. `reference/review-host.ts` is a maintained development fixture, not a
+published server entry. Browser tests show actual approved args and persisted
+ContentDocument results, stale tabs, reload, retries and negative access.
+
+Run full package `test`, `test:postgres`, `test:providers`, `build`, `typecheck`,
+`test:components`, `test:e2e`; use the root Node/pnpm versions. Set a short
+`CI_TEST_TMPDIR`/`TMPDIR`, and optionally
+`PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` to the supported installed Chromium.
+`pnpm test:evaluation` checks repository evaluation tooling contracts; measured
+held-out quality remains a separate frozen-protocol, budget-gated #3677 run.
+
+The existing CI browser job includes ingestion changes and shared UI primitive
+changes in affected mode, and runs the ingestion browser suite in full mode. Its
+normal dependency build precedes the suite and failed runs upload browser evidence.
+
+The maintained host derives inbox state by traversing bounded public review pages,
+authorizing every action/parent/target before classification. It retains only
+scalar totals and the requested page; an off-page denial denies the whole view.
+The component's Load more and subsequent refresh/mutation reload every displayed
+page using fresh cursors, then publish the combined authorized set together. Old
+page payloads are cleared while loading; any unavailable page clears all payloads.
+This keeps plan controls and earlier steps visible across page boundaries without
+reusing a prior authorization decision.