@happyvertical/smrt-ingestion 0.57.29 → 0.57.31

package/PROPOSALS.md

@@ -186,7 +186,8 @@ Model output can contain only offered handler versions, schema-valid arguments,
 retained evidence locations and offered alternative handler/candidate keys.
 Candidate IDs must occur in the offered set, and evidence IDs in retained source
 results. Unknown enums, invented IDs, extra authority/tool/approval fields and
-malformed structures fail closed. Missing required fields are retained as
+malformed structures fail closed (an invented candidate target alone is dropped
+for review instead, see below). Missing required fields are retained as
 `missingFields`, with `needs_review`; values are never invented to satisfy a
 schema. Observed page/time/source granularity is preserved. Logical scan splits
 use `proposeDocumentSplits`; originals and hashes remain intact.
@@ -237,6 +238,34 @@ rejected, even when the earlier split on that evidence was dropped. Like
 reply. Stored output is never re-validated, so earlier attempts and their
 digests replay unchanged.
 
+An invented target — a value in an offered handler's candidate reference field
+that is not among the offered candidates for that field's model — no longer
+rejects the attempt (#4289). The validator drops the key from the arguments
+(no placeholder) and revalidates what remains through the required-only path
+above: a dropped required field becomes a computed `missingFields` entry, a
+dropped optional field is simply absent, and either way the suggestion is
+`needs_review` for the reviewer to choose an offered target.
+`provenance.unofferedTargetsDropped` records `{suggestions, fields}` counts
+only, never IDs, labels or field names; it is absent when nothing was dropped
+and removed whenever the suggestions are discarded. Every candidate field that
+stays in the arguments has been checked against the offered set, so no
+unoffered value survives as a target. The attempt still rejects as before for
+an unoffered handler or alternative, unknown evidence (argument or reference),
+a non-string target, a target field the handler schema does not declare in
+`properties`, any other argument error, or a drop that exposes a refusal other
+than `required`.
+
+The exact dropped value is also searched for in the kept suggestions and splits
+(JSON-escaped, keys included); an exact echo, such as the ID repeated in an
+explanation or another argument, rejects the attempt. This guard fails closed
+on short or common values: a dropped `"1"` or `"a"` almost always occurs
+somewhere else in the output, so such a reply rejects rather than being kept.
+It is an exact-match check, not a general leak filter: a case variant, a
+reformatted copy or a fragment of the ID in free text (an explanation or a
+text argument) is not detected and may reach the reviewer as ordinary model
+prose. Free text is never a target and never authority; only the checked
+candidate fields are.
+
 A rejected reply ends the attempt as `provider_error` with the `malformed_output`
 warning and a bounded, value-free `provenance.failure` `{category, reason}` (#4207).
 The reason lists schema-owned paths, keywords and schema constants, for example
@@ -314,14 +343,121 @@ A generator call that throws ends the attempt as `provider_error` with the
 `unavailable` warning and `provenance.failure` `{category: 'unavailable',
 reason}`. The reason is value-free by construction: a provider's own text can
 quote the prompt or document, so it is never copied. The reason is built only
-from a fixed phrase and, for an SDK `AIError` only, the HTTP status (from a
-structured field, or from the exact `<status> ...` / `TypeSafe request failed
-(<status>): ...` shapes the SDK produces), the SDK error code (from the closed
-list of codes the SDK raises) and, for a schema rejection, quoted tokens that
-are JSON Schema keywords. Any other error contributes at most a structured
-numeric status. For example: `provider rejected the response schema (HTTP 400,
-code API_ERROR); unsupported response-schema keywords: uniqueItems`, or just
-`provider call failed`. It is cut at 500 characters.
+from a fixed phrase and:
+
+- for an SDK `AIError` only, the HTTP status (from a structured field, or from
+  the exact `<status> ...` / `TypeSafe request failed (<status>): ...` shapes
+  the SDK produces), the SDK error code (from the closed list of codes the SDK
+  raises), for a schema rejection the quoted tokens that are JSON Schema
+  keywords, and the OpenAI SDK's exact fixed transport messages
+  (`Connection error.`, `Request timed out.`, `Request was aborted.`,
+  `(no status code or body)`), which its `mapError` keeps without a cause;
+- for the SDK adapter's own refusal of a reply it did receive, a fixed problem:
+  finish reason other than `stop` (named from the closed list `stop`, `length`,
+  `tool_calls`, `content_filter`, otherwise `other`), truncation, tool calls,
+  an unexpected response model, no text content, an empty reply, a reply over
+  the output limit, or a reply that is not valid JSON (#4285);
+- for any other error (#4285), the code-defined constructor name (an
+  identifier ending in `Error` or `Exception`; never the settable `name`),
+  an `AbortError`/`TimeoutError` name, a network `code` from a closed list
+  (`ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `EAI_AGAIN`,
+  `UND_ERR_*`, TLS certificate codes, ...) read from the error, up to three
+  `cause` levels and an `AggregateError`'s entries, and a structured numeric
+  status. A timeout or abort is named as one, except that a structured HTTP
+  status always wins over an abort name or network timeout code on a non-SDK
+  error, whether on the error itself or in its cause chain. An SDK error's
+  abort is read only from its code or exact transport message.
+
+For example: `provider rejected the response schema (HTTP 400, code API_ERROR);
+unsupported response-schema keywords: uniqueItems`,
+`provider response incomplete (finish reason length)`,
+`provider call failed (class TypeError, network ECONNREFUSED)`,
+`provider call aborted or timed out (class DOMException, name TimeoutError)`, or
+`provider call failed (class Error)`. It is cut at 500 characters.
+
+**Why some failures were a bare `provider call failed` (#4285).** The SDK
+adapter's own response checks threw plain `Error`s (finish reason not `stop`,
+e.g. `length` when the reply reached `maxTokens`; truncation; tool calls; an
+unexpected response model; non-text or oversized content) and `JSON.parse`
+threw a `SyntaxError` for an empty or non-JSON reply. The reason deliberately
+reads nothing from a non-SDK error, so all of these collapsed into the fixed
+phrase. Transport failures through the OpenAI SDK already carried
+`code API_ERROR`. The adapter now raises `ProposalResponseError` with a fixed
+problem, and other non-SDK errors contribute the transport facts above. The
+category is unchanged (`unavailable`).
+
+### Failed-attempt capture (opt-in, #4285)
+
+The value-free reason cannot show what the model actually sent. A host that
+needs that for diagnosis can opt in to keeping it:
+
+```ts
+proposals: {
+  // ...
+  failureCapture: {
+    enabled: true,      // required; absent or false keeps capture off
+    maxBytes: 16384,    // default 16384, at most 65536 (UTF-8 bytes, after redaction)
+    redact: ({ text, source }) => hostRedaction(text), // required to retain text
+  },
+},
+```
+
+`ProposalConfiguration.failureCapture` is **off by default**. When enabled with
+a host `redact` hook, a failed interpretation attempt (`provider_error`) keeps
+one of:
+
+- `raw_response`: the reply text the SDK adapter refused (for example invalid
+  JSON or a `length` finish), or
+- `generated_output`: the generator's parsed output, as JSON, that failed
+  validation, the declared-completion check or the output limit.
+
+Errors without model output (transport, timeouts, authority, decision
+failures) capture nothing. Successful attempts never store a capture.
+
+Without a `redact` hook, only `status: 'redaction_failed'` metadata is stored,
+with `redacted: false` and no text; the candidate text is not materialized.
+There is no built-in sanitizer. Hosts own the hook's semantic redaction policy;
+the service cannot verify that the returned text removes every document secret.
+
+The text, cut to the larger of the generation `maxOutputBytes` and
+`maxBytes`, goes through the host `redact` hook, and the result is cut to
+`maxBytes` without splitting a character (`truncated: true` when either cut
+applied). NUL and unpaired surrogates are stored as U+FFFD so PostgreSQL JSON accepts the record. When
+the hook throws, returns a non-string or exceeds its asynchronous wait budget,
+only `status: 'redaction_failed'` is stored. The budget is the generation
+`timeoutMs`, cut to half of the analysis lease still left when the hook starts
+(no time left: the hook is not called). The hook runs after the authority
+recheck, reserving the rest of the lease for the second recheck and publication.
+This timer cannot preempt synchronous work or cancel the hook. Hosts must use
+bounded, nonblocking redactors. Missing or failed redaction does not replace the
+attempt's real failure or value-free reason; hook errors are discarded.
+
+**Where it lives.** In the attempt row's `data` JSON, as `failureCapture`
+beside the result fields, never inside `result`: it needs no migration, is not
+part of `output_digest` or any configuration digest, never appears in
+`getCompletedAnalysis` or any other snapshot, and is not public read
+configuration (`proposalReadConfiguration` omits it, so read-only hosts still
+match). Proposal digests, replay and frozen generation configuration are
+unchanged by enabling it.
+
+**Reading it back.** Only
+`IngestionService.getFailedAttemptCapture(itemId, attemptId?)` returns it
+(`FailedAttemptCapture` from `/dto`, or `null` when nothing was stored). The
+attempt defaults to the item's current analysis attempt. It requires the
+execution catalog and the current **reviewer** authority for that item (the
+same gate as `getItemDisposition`) inside the caller's tenant and confidential
+scope; every denial, expiry, cancellation or foreign item fails closed before
+the row is read. It is never part of a list DTO, inbox row, review page,
+log line, checkpoint name, telemetry or error message.
+Publication and reads also reject a persisted `captured` record with
+`redacted: false`; capture text must be marked as processed by a configured
+host hook. Persisted caps must remain within 1–65536 bytes, and captured text
+must fit its declared UTF-8 byte cap; malformed records fail closed.
+
+**Retention and deletion.** It follows the item. There is no separate delete
+API: `expire`, `sweepRetention` and the `repairDeletions` retry of an
+interrupted propagation all empty attempt `data`, which removes the capture
+with the rest of the payload; a revoked item is unreadable before that.
 
 `createSDKProposalDecisionClient(client)` adapts an existing SDK client’s optional
 capability without changing its errors or protocol. Optional `decision` uses public core `executeDecision` and the existing SDK

package/README.md

@@ -262,6 +262,18 @@ and `maxSuggestions` are each capped at 20; `timeoutMs` is capped at 2147483647.
 Invalid limits are rejected when constructing `IngestionService`, before receipt
 or interpretation. See [REVIEW.md](REVIEW.md) for the published server review host.
 
+`proposals.failureCapture` (`{ enabled, maxBytes?, redact? }`) is off by
+default. With `enabled: true` and a host `redact` hook, a failed interpretation
+attempt keeps the model output (or the parsed output that failed validation)
+for diagnosis: cut to
+`maxBytes` (default 16384, at most 65536 UTF-8 bytes) after the host
+`redact` hook, stored beside the attempt result (not digested), readable only
+through the reviewer-authorized `getFailedAttemptCapture(itemId, attemptId?)`
+and removed with the item. A missing or failed hook stores only
+`redaction_failed` metadata, never text. Hosts own the semantic redaction policy;
+no built-in sanitizer is supplied. Invalid settings are rejected at construction.
+See [PROPOSALS.md](PROPOSALS.md#failed-attempt-capture-opt-in-4285).
+
 ## Trusted internal sources
 
 A host with a trusted internal source, such as chat messages recorded in

package/REVIEW.md

@@ -135,6 +135,15 @@ Changing it or itemId destroys the old view and fences late callback results.
 Denied refresh clears prior originals, candidates, arguments and results.
 Missing confidence is unavailable; page/time links exist only for observed
 locations. Provider errors and unresolved/partial results remain review states.
+A host that enabled `proposals.failureCapture` may fetch a failed attempt's
+redacted model output with `getFailedAttemptCapture(itemId, attemptId?)`.
+Retaining text requires a configured host redaction hook; a missing or failed
+hook yields only `redaction_failed` metadata. The host owns that redaction policy.
+The getter requires current reviewer authority for that item and fails closed
+otherwise;
+`load`, inbox rows and review pages never include it, so a host shows it only
+behind its own explicit diagnostic action (see
+[PROPOSALS.md](PROPOSALS.md#failed-attempt-capture-opt-in-4285)).
 
 ## Submitter-scoped decisions
 

package/dist/chunks/{execution-Dv3G0d62.js → execution-BxkUOtLf.js}

@@ -47,8 +47,92 @@ var __toCommonJS = (mod) => __hasOwnProp.call(mod, "module.exports") ? mod["modu
 //#region src/execution-internal.ts
 var discoveryTransaction = /* @__PURE__ */ Symbol("ingestion.discoveryTransaction");
 var generationSnapshotRead = /* @__PURE__ */ Symbol("ingestion.generationSnapshotRead");
+var completeWithFailureCapture = /* @__PURE__ */ Symbol("ingestion.completeWithFailureCapture");
+var analysisLeaseRemaining = /* @__PURE__ */ Symbol("ingestion.analysisLeaseRemaining");
 var listReadGate = /* @__PURE__ */ Symbol("ingestion.listReadGate");
 //#endregion
+//#region src/failure-capture.ts
+var FAILURE_CAPTURE_DEFAULT_MAX_BYTES = 16384;
+var FAILURE_CAPTURE_MAX_BYTES = 65536;
+function assertFailureCapture(config) {
+	if (config === void 0) return;
+	if (!config || typeof config !== "object" || typeof config.enabled !== "boolean" || config.maxBytes !== void 0 && (!Number.isSafeInteger(config.maxBytes) || config.maxBytes < 1 || config.maxBytes > 65536) || config.redact !== void 0 && typeof config.redact !== "function") throw new Error("Invalid failure capture configuration");
+}
+function truncateUtf8(text, maxBytes) {
+	const bytes = Buffer.from(text, "utf8");
+	if (bytes.length <= maxBytes) return {
+		text,
+		truncated: false
+	};
+	let end = maxBytes;
+	while (end > 0 && (bytes[end] & 192) === 128) end--;
+	return {
+		text: bytes.subarray(0, end).toString("utf8"),
+		truncated: true
+	};
+}
+function storable(text) {
+	return text.replaceAll(String.fromCharCode(0), "�").replace(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g, "�");
+}
+async function prepareFailureCapture(config, candidate, bounds) {
+	if (config?.enabled !== true) return void 0;
+	const maxBytes = config.maxBytes ?? 16384;
+	const hook = config.redact;
+	const base = {
+		version: 1,
+		source: candidate.source,
+		redacted: !!hook,
+		maxBytes,
+		capturedAt: bounds.now.toISOString()
+	};
+	if (!hook || bounds.timeoutMs < 1) return {
+		...base,
+		status: "redaction_failed",
+		truncated: false
+	};
+	let raw;
+	try {
+		raw = candidate.text();
+	} catch {
+		return;
+	}
+	if (typeof raw !== "string" || !raw.length) return void 0;
+	const input = truncateUtf8(raw, Math.max(bounds.inputBytes, maxBytes));
+	let text = input.text;
+	let timer;
+	try {
+		const redacted = await Promise.race([Promise.resolve().then(() => hook.call(config, {
+			text: input.text,
+			source: candidate.source
+		})), new Promise((_resolve, reject) => {
+			timer = setTimeout(() => reject(/* @__PURE__ */ new Error("timeout")), bounds.timeoutMs);
+		})]);
+		if (typeof redacted !== "string") throw new Error("invalid");
+		text = redacted;
+	} catch {
+		return {
+			...base,
+			status: "redaction_failed",
+			truncated: false
+		};
+	} finally {
+		clearTimeout(timer);
+	}
+	const stored = truncateUtf8(storable(text), maxBytes);
+	return {
+		...base,
+		status: "captured",
+		text: stored.text,
+		truncated: input.truncated || stored.truncated
+	};
+}
+function storedFailureCapture(value) {
+	if (!value || typeof value !== "object" || Array.isArray(value)) return void 0;
+	const record = value;
+	if (record.version !== 1 || !["raw_response", "generated_output"].includes(String(record.source)) || !["captured", "redaction_failed"].includes(String(record.status)) || typeof record.truncated !== "boolean" || typeof record.redacted !== "boolean" || typeof record.maxBytes !== "number" || !Number.isSafeInteger(record.maxBytes) || record.maxBytes < 1 || record.maxBytes > 65536 || typeof record.capturedAt !== "string" || (record.status === "captured" ? typeof record.text !== "string" || record.redacted !== true || Buffer.byteLength(record.text, "utf8") > record.maxBytes : record.text !== void 0)) return void 0;
+	return record;
+}
+//#endregion
 //#region src/proposal-errors.ts
 var HandlerArgumentsInvalidError = class extends Error {
 	issues;
@@ -7057,7 +7141,7 @@ var IntakeExecutionService = class IntakeExecutionService {
 		if (!Array.isArray(references)) throw new Error("Feedback lineage unavailable");
 		if (references.length === 0) return;
 		if (!this.options.feedback) throw new Error("Feedback configuration unavailable");
-		const { IntakeFeedbackService } = await import("./feedback-PLOZth0a.js").then((n) => n.r);
+		const { IntakeFeedbackService } = await import("./feedback-D69e1hX5.js").then((n) => n.r);
 		await new IntakeFeedbackService(this.options, this).assertActionLineage(String(bound.action.id), db, String(bound.proposal.id));
 	}
 	async getAction(actionId) {
@@ -8222,6 +8306,38 @@ var IntakeExecutionService = class IntakeExecutionService {
 			return row ? this.dispositionDTO(row) : void 0;
 		});
 	}
+	/**
+	* A failed attempt's opt-in diagnostic capture (#4285) under the current
+	* reviewer authority for this item, inside the caller's tenant and
+	* confidential scope; every denial fails closed before the row is read.
+	*/
+	async getFailedAttemptCapture(itemId, attemptId) {
+		return this.tx(async (db) => {
+			const item = await this.item(db, itemId);
+			await this.reviewerAccess(db, item);
+			let id2 = attemptId;
+			if (id2 === void 0) {
+				const [analysis] = await this.rows(db, "intake_analyses", "item_id=? AND revision=?", [item.id, item.analysis_revision]);
+				if (!analysis?.current_attempt_id) throw new Error("Analysis unavailable");
+				id2 = String(analysis.current_attempt_id);
+			}
+			const [attempt] = await this.rows(db, "intake_analysis_attempts", "id=? AND item_id=?", [id2, item.id]);
+			if (!attempt) throw new Error("Analysis unavailable");
+			const capture = storedFailureCapture(object(attempt.data).failureCapture);
+			if (!capture || !attempt.safe_error) return null;
+			return {
+				itemId: String(item.id),
+				attemptId: String(attempt.id),
+				source: capture.source,
+				status: capture.status,
+				...capture.status === "captured" ? { text: capture.text } : {},
+				truncated: capture.truncated,
+				redacted: capture.redacted,
+				maxBytes: capture.maxBytes,
+				capturedAt: capture.capturedAt
+			};
+		});
+	}
 	/** Close an item as handled without an action. Terminal; replay returns the same disposition. */
 	async resolveItem(input) {
 		const frozen = JSON.parse(canonical(JSON.parse(JSON.stringify(input))));
@@ -8600,6 +8716,6 @@ async function continueIntakeReview(service, actionId, task) {
 	return service.applyAction(actionId);
 }
 //#endregion
-export { __toESM as C, __exportAll as S, HandlerArgumentsInvalidError as _, resolveIntakePolicy as a, generationSnapshotRead as b, HANDLER_ARGUMENT_ISSUE_LIMIT as c, schemaOwnedPath as d, require_ajv as f, GenerationSourceStaleError as g, GenerationSnapshotStaleError as h, intakeBindingDigest as i, HANDLER_ARGUMENT_PATH_LIMIT as l, redactedCeiling as m, IntakeExecutionService as n, feedbackActionTransaction as o, UNFINISHED_ACTION_STATES as p, continueIntakeReview as r, feedbackDiscoveryContext as s, IntakeAccessDeniedError as t, describeArgumentRefusal as u, ProposalsNotConfiguredError as v, listReadGate as x, discoveryTransaction as y };
+export { __toESM as A, storedFailureCapture as C, generationSnapshotRead as D, discoveryTransaction as E, listReadGate as O, prepareFailureCapture as S, completeWithFailureCapture as T, HandlerArgumentsInvalidError as _, resolveIntakePolicy as a, FAILURE_CAPTURE_MAX_BYTES as b, HANDLER_ARGUMENT_ISSUE_LIMIT as c, schemaOwnedPath as d, require_ajv as f, GenerationSourceStaleError as g, GenerationSnapshotStaleError as h, intakeBindingDigest as i, __exportAll as k, HANDLER_ARGUMENT_PATH_LIMIT as l, redactedCeiling as m, IntakeExecutionService as n, feedbackActionTransaction as o, UNFINISHED_ACTION_STATES as p, continueIntakeReview as r, feedbackDiscoveryContext as s, IntakeAccessDeniedError as t, describeArgumentRefusal as u, ProposalsNotConfiguredError as v, analysisLeaseRemaining as w, assertFailureCapture as x, FAILURE_CAPTURE_DEFAULT_MAX_BYTES as y };
 
-//# sourceMappingURL=execution-Dv3G0d62.js.map
+//# sourceMappingURL=execution-BxkUOtLf.js.map