@tangle-network/agent-eval 0.220.2 → 0.221.1

package/docs/systemone.md

@@ -96,6 +96,26 @@ probabilities are not calibrated domain confidence, and a result never authorize
 sending, deletion, or another effect. Re-assessing the same prepared input and retained answer is
 an offline replay, not new inference.
 
+## Rationale and evidence limits
+
+The native request and answer types in [`systemone-protocol.ts`](../src/systemone-protocol.ts)
+declare no free-form rationale, generated evidence URI, or extracted text-span field.
+Instructions asking for an explanation do not establish a supported output contract.
+Do not issue a second explanation call to fill this gap.
+
+The same request can include another named `choice` question whose criteria keys are
+caller-supplied evidence IDs, including an explicit unresolved option. DomainMatch already
+uses this pattern for source selection. The answer selects from that finite set; it does
+not establish that the selected source caused or justifies another answer. Persist the
+immutable ID-to-source mapping with the decision (as behavioral trace definitions do), or in
+retained evidence referenced by the invocation's `evidenceRefs`.
+The journal stores state/input digests, so a mapping held only in `state` cannot support
+later inspection. Grounding and calibration require independent checks.
+
+Native response extensions may survive parsing into the evaluator's returned raw value.
+That does not promise the provider generates them or that downstream records retain them.
+The declared protocol and each consumer's persistence schema determine what survives.
+
 ## Providers, rounding and confidence
 
 The wire format is the same for every provider; how each reports its numbers is not. A
@@ -236,3 +256,90 @@ rubric lives in `examples/`, not the generic implementation, and is not a defaul
 Protocol: https://docs.typesafe.ai/api
 Clef: https://developers.cloudflare.com/workers-ai/models/clef/
 SDK (optional builders): https://github.com/typesafe-ai/typesafe-sdk-js
+
+## Decision points
+
+`decisionPoint` from `/systemone` adds durable routing to the evaluator. A point
+serves many `layers`; each invocation names its current layer and consumer.
+The same layer can use many points. Questions use typed choices; policies map
+every option to `act` (with a route), `escalate`, or `record`, with probability
+and margin thresholds per question. These are recommendations; the consumer
+performs and records the actual action. An escalation in any question takes
+precedence over `act`, which takes precedence over `record`.
+
+Use this for judgment over unstructured state only when a wrong recommendation
+is cheap and reversible and uncertainty can reach an agent or a human. It never
+grants authority to delete, deploy, merge, retire, lift a stand-down, or override
+an acceptance check. Code continues to read objective facts directly. Sample or
+trigger these calls instead of evaluating every event. Existing DomainMatch
+continues to supply its domain-specific request and assessment; this primitive
+uses the same native evaluator and answer validation, not another model adapter.
+
+A policy may drive a route only with independent labeled evidence: positive
+`gold_n`, finite Cohen κ meeting `DECISION_POINT_MIN_KAPPA` (owned in
+[`decision-point.ts`](../src/decision-point.ts)) and `evidenceRefs`.
+`decisionQuestionDigest({ id, version, model, servedModel, name, question })` binds that
+evidence to the authored question, option order and model. Changed served model
+identity, absent/stale calibration, ties and below-threshold answers escalate.
+κ measures label agreement; the joined outcome curve measures probability
+reliability. Do not describe one as proof of the other.
+
+Supply `ledger: openFindingsLedger({ path })` from `/analyst` and the existing
+`SystemOneEvaluatorOptions` as `evaluator`. `decide` accepts `runId`,
+`invocationId`, `layer`, `consumer`, `state`, `scope`, `at`, and `evidenceRefs`,
+plus the existing evaluation context for cancellation, spending and receipts.
+Inspect the typed `succeeded` result before reading its `value`.
+
+The findings ledger is the sole persistence owner. Every invocation first
+appends a 1.1.0 intent with a unique owner token. Competing callers cannot both
+infer. Completed calls record state/input/definition digests, the policy and
+calibration identity, requested/served model, probabilities, recommendation and
+receipt. Failed or cancelled evaluations record escalation with accounting when
+available. A persistence failure returns a typed error and withholds action.
+Exact `(point id, runId, invocationId)` replay reuses the original result, across
+restart; changed inputs or definitions conflict. The input includes its original
+`at`, so callers retain it. An intent without a result remains explicit and
+escalates; deliberate recovery uses a new invocation id instead of silently
+repeating a possibly paid call. The journal's normal lock/dead-owner rules apply.
+
+Decision records persist each named choice and its probabilities, so an explicit
+evidence-ID question uses the same ledger and replay path. They do not automatically
+retain arbitrary native response extensions or convert a selected ID into an `EvidenceRef`.
+The record's `reason` (for example, `uncalibrated`) is a computed policy reason, not a
+model-authored rationale; the receipt records accounting. Each additional routing question
+needs its own policy and calibration, and an uncalibrated routing question escalates the invocation.
+An explicitly `recordOnly: true` policy captures an ancillary choice without routing authority:
+all its options must use `action: 'record'`, and it cannot carry a calibration certificate.
+Its answer has `reason: 'record-only'`; provider failures still escalate. This policy does
+not bypass the calibration or thresholds of another question. Reliability lists these questions
+in `recordOnlyQuestions` and excludes them from outcome completeness and calibration curves;
+a fully labeled decision needs at least one answered routing question.
+
+`reliability()` groups the exact `definitionDigest`, including option sets and any retained
+source map. Distinct window maps therefore do not form a cross-window probability curve.
+For behavioral trace questions, use `calibrateBehavioralTraceQuestions` for semantic nominal
+agreement across maps; it binds the stable semantic question digest and reports each disagreement's
+selected event and map. A one-map result does not justify probability threshold tuning.
+
+Call `observe({ runId, invocationId, labels, at, evidenceRefs })` later to join
+independently observed option labels to a decision. This is the answer's truth,
+not whether a downstream workflow happened to succeed. Retain the original
+observation timestamp for exact replay. Conflicting labels refuse append.
+`reliability()` reads the same ledger and returns total decisions, labeled and
+missing outcomes, and per-question sample counts, empirical curves and threshold
+accuracy. A curve needs two labels; otherwise it is `null`. Threshold tables are
+development evidence for an explicit policy revision and fresh confirmation,
+not automatic tuning or activation. Failed calls remain in the denominator.
+
+Decision findings retain their measured layer and scope. The existing `/analyst`
+`promote` is the only owner for generalizing claims; it does not authorize
+routing and does not lift event/turn findings directly into run-level claims.
+
+The data model permits many-to-many associations. The only intentional
+one-to-one rules are one primary measure per hypothesis (to prevent choosing
+a winner after seeing outcomes), with any number of secondary measures, and
+one implementation per code concept (to prevent divergent owners).
+
+Decision-point construction validates the complete calibration object, including nonempty structured evidence references, before any inference. Malformed configuration throws; well-formed evidence below the calibration bar still escalates. `observe()` returns a typed `invalid` result for uncloneable, noncanonical or malformed observation data without appending an outcome. A provider identity with surrounding whitespace is rejected as a failed evaluation; its durable escalation replays and remains readable by reliability reporting.
+
+The existing behavioral analyst optionally applies bounded [trace questions](behavioral-trace-questions.md) through decision points; exact repeated tool calls stay computed.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.220.2",
+  "version": "0.221.1",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -229,15 +229,15 @@
   "dependencies": {
     "@asteasolutions/zod-to-openapi": "^9.1.0",
     "@hono/node-server": "^2.1.1",
-    "@tangle-network/agent-core": "^0.13.1",
-    "@tangle-network/agent-interface": "^5.0.0",
+    "@tangle-network/agent-core": "^0.13.2",
+    "@tangle-network/agent-interface": "^5.7.1",
     "@tangle-network/agent-trace-contract": "^2.0.0",
     "@tangle-network/harness-sessions": ">=0.1.0 <0.2.0",
     "hono": "^4.13.9",
     "linear-sum-assignment": "1.0.9",
     "re2js": "2.8.6",
     "smol-toml": "1.9.0",
-    "zod": "4.6.5"
+    "zod": "4.6.6"
   },
   "devDependencies": {
     "@arethetypeswrong/cli": "^0.18.5",