@tangle-network/agent-eval 0.183.0 → 0.184.0

package/docs/changes/evaluation-definition-identity.md

@@ -0,0 +1,27 @@
+# Unreleased: evaluation definition identity
+
+Static `jevJudge` and `jevAnalyst` versions now bind authored question order and
+Choice alternative order in addition to their canonical content. The request is
+not sorted or rewritten. Optional undefined SDK fields remain absent from the
+JSON document, so JSON round-trips retain the same identity.
+
+This intentionally changes static evaluator version hashes. A cached verdict from
+the previous identity scheme must not be treated as evidence for a reordered
+classifier definition. Existing caches remain stored under their previous versions;
+this change does not delete or migrate them.
+
+`asJudge` and `asAnalyst` capture callback references when constructed and detach
+metadata arrays/objects from the supplied options. Replacing `options.evaluate`,
+`options.map` or `options.record` later no longer changes an already-versioned
+adapter. Construct another adapter with a new version for an intentional change.
+
+This is not a sandbox or closure serializer. Callers must still version opaque
+renderer/mapping behavior and mutable state captured inside their own functions.
+Dynamic question builders remain supported; retain each actual request through
+the existing observation path. No model calls, thresholds, domain policies,
+transport defaults or execution machinery are added.
+
+Regression tests cover version collisions from reordering, preserved dispatch
+order, SDK-style undefined fields, metadata mutation and callback replacement
+through the real paid-call ledger. They establish code contracts, not classifier
+quality or deployed outcome improvements.

package/docs/changes/jev-evidence-review.md

@@ -0,0 +1,18 @@
+# Unreleased: opt-in evidence reviews on native evaluations
+
+`@tangle-network/agent-eval/jev` adds `prepareJevReview`, `assessJevReview` and
+`jevReviewFindings`. Callers retain ownership of questions, hypotheses, evidence, severity,
+thresholds and result policy. Review identity includes authored option order; request JSON is
+not sorted before dispatch. Missing or partial negative evidence cannot become a clean review.
+
+The optional `examples/jev-behavior-review.ts` recipe covers reward gaming, evaluator tampering,
+policy/authorization/data-boundary breaches, injection-following, contradicted completion,
+oversight bypass, matched-baseline anomalies and cross-run behavior. It composes the existing
+`jevAnalyst`; the taxonomy is not installed in the generic evaluator and is not a calibrated
+safety detector. See [behavioral reviews](../jev-behavior-review.md) for evidence and limits.
+
+No dependency, transport, scheduler, package entrypoint, default model, production action or
+spending policy changes. Existing evaluator, judge, analyst and protocol APIs remain unchanged.
+Use existing deterministic control/reward checks and independent final assessment alongside
+semantic reviews. Adversarial model quality and product/harness adoption remain tracked in
+#772/#771 and the cross-repository roadmap #768.

package/docs/changes/jev-product-adoption.md

@@ -0,0 +1,18 @@
+# 0.184.0: installed product adoption
+
+The new version publishes the merged typed evaluator and Jev APIs that were not in the
+previously published 0.183.0 artifact. JS and Python distribution versions remain in lockstep.
+
+`@tangle-network/agent-eval/jev/protocol` is a leaf export of the existing protocol implementation.
+Products can validate native requests/responses without importing analyst/campaign execution.
+Native extension metadata is preserved, not projected away; lossy non-JSON metadata is rejected
+before consumers persist an apparently valid but changed observation. Already-paid receipts remain
+recorded when answer validation fails.
+
+`examples/jev-decision-benchmark.ts` compares caller-defined classifier configurations with the
+existing matrix runner. It separates independent expected labels from model input, retains
+abstention, records complete observations and paid failures, and groups results by configuration.
+It is a recipe, not another optimizer or universal scoring policy.
+
+Release this version through the existing verify-and-publish workflow before merging consumers
+pinned to 0.184.0. A source manifest version is not evidence that the npm artifact is available.

package/docs/changes/probability-policy-audit.md

@@ -0,0 +1,81 @@
+# Unreleased: audit a frozen probability policy
+
+Added `auditProbabilityPolicy`, `ProbabilityPolicyAuditInput` and
+`ProbabilityPolicyAuditReport` to `@tangle-network/agent-eval/meta-eval`.
+Existing `auditEvaluator` behavior and report hashes are unchanged. No dependency,
+model call, scheduler or automatic promotion is added.
+
+Use this when a product turns a probability into accept, reject or escalation.
+The caller supplies both thresholds and independently labeled observations. The
+helper maps the observations, then uses the existing `auditEvaluator` statistics,
+independent-unit counting, development-exposure exclusion and authority checks.
+
+```ts
+import { readFile } from 'node:fs/promises'
+import { auditProbabilityPolicy } from '@tangle-network/agent-eval/meta-eval'
+
+const path = process.argv[2]
+if (!path) throw new Error('Supply the frozen audit JSON path')
+const report = auditProbabilityPolicy(JSON.parse(await readFile(path, 'utf8')))
+console.log(JSON.stringify(report, null, 2))
+```
+
+The input has the existing evaluator audit fields: `evaluatorDigest`, `population`,
+`samplingFrame`, `authority`, `policy` and `observations`. Add explicit `thresholds`
+with `rejectAtOrBelow` and `acceptAtOrAbove`. Every observation carries `id`,
+`independentUnitId`, `evidenceRef`, independently checked `expected` (`accept` or
+`reject`), `exposure` (`fresh` or `development`), and `acceptProbability` instead
+of `observed`.
+
+`acceptProbability` is P(the event the caller defines as acceptable), in [0,1], or
+null for missing output. For Jev, a caller can use a suitable Noul or a named Choice
+probability. Do not substitute Jev's distribution-confidence statistic or an
+arbitrarily scaled numeric grade. Other providers work without pretending to be Jev.
+
+Probabilities at or above `acceptAtOrAbove` map to accept. Those at or below
+`rejectAtOrBelow` map to reject. The open interval between them and null map to
+unknown. Thresholds must be finite, in [0,1], and strictly separated. Nothing
+chooses thresholds or inserts a default cutoff for the application.
+
+## Read the result correctly
+
+The result retains the existing admit/reject/inconclusive verdict, error bounds,
+coverage, exclusions, authority and mapped observations. Unknown judgments remain
+in the conservative error bounds, never measured successes. The helper does not
+quietly discard escalated or unavailable cases to improve apparent accuracy.
+`coverage.eligibleCases - coverage.unknownCases` is the number of decided eligible
+cases; the full denominator remains `eligibleCases`.
+
+`probabilityPolicy` retains the original classifier digest, thresholds and exact
+probabilities keyed by observation id. `evaluatorDigest` identifies the composed
+classifier-plus-threshold program. `inputDigest` binds the complete validated
+probability-audit input (sorted by observation id), including probability changes
+that do not change the selected action. `reportDigest` covers the complete report.
+
+The original evaluator digest must identify the actual frozen classifier, question
+and evidence-rendering definition. An arbitrary string or a model alias is not
+an independently verified program identity. Hashes establish identity, not truth.
+
+## Keep selection and final assessment separate
+
+Tune thresholds on development data, freeze them, then use fresh independent cases
+for the final audit. A development-exposed source excludes its fresh-looking
+variants too. Duplicate observation ids and identical declared author/auditor
+identities are refused. Distinct declared names do not prove real-world independence;
+the execution owner must enforce it and supply evidence.
+
+The output audits this fixed binary decision policy. It does not prove probability
+calibration, downstream workflow correctness, cost savings, causal repair quality
+or external-action authorization. Use existing calibration/comparison utilities
+and actual task outcomes for those claims. A small or entirely missing dataset is
+inconclusive rather than perfect evidence. Real datasets, task slices, cost/latency
+comparisons and deployed validation remain tracked in issue #771 under roadmap #768.
+
+## Checks
+
+`pnpm exec vitest run tests/probability-policy-audit.test.ts tests/evaluator-admission.test.ts`
+
+Regressions cover boundary decisions, missing data, invalid probabilities, overlapping
+thresholds, digest identity, input immutability, independent-unit denominators,
+development leakage, authority checks and unchanged underlying error bounds. These
+are executable arithmetic/contract tests, not live Jev quality measurements.

package/docs/jev-behavior-review.md

@@ -0,0 +1,174 @@
+# Evidence-backed behavioral reviews
+
+Use native evaluations to assess bounded hypotheses about recorded behavior. Do not ask one
+classifier whether an agent is "malicious" or a whole run is "safe". Observable policy breaches,
+model-supported hypotheses, missing evidence and unusual behavior are different results.
+
+The optional `/jev` helpers are ordinary functions:
+
+- `prepareJevReview`: validate and freeze caller-defined native questions, evidence references,
+  coverage and decision thresholds before spending. The digest binds the definition, input and
+  authored question/option order. Optional undefined object fields are absent on the wire.
+- `assessJevReview`: validate native answers and interpret designated alternative masses as
+  `supported`, `refuted` or `unresolved`. Re-scoring makes no network call.
+- `jevReviewFindings`: map that report into the existing `AnalystFinding` contract. It keeps
+  unresolved checks visible and labels supported claims as model-supported hypotheses.
+
+These helpers do not constrain `jevEvaluator`, `jevJudge`, `jevAnalyst`, or the provider-independent
+`createEvaluator`/`asJudge`/`asAnalyst` APIs. Use an ordinary custom mapper for different policies.
+No model, taxonomy, severity, operating threshold, global safety score or storage backend is chosen
+by the generic implementation.
+
+## Direct product call
+
+Supply the native request on each invocation. This example's `question`, `evidence`, `policy` and
+`limits` are application configuration, not new library exports or default values.
+
+```ts
+import {
+  prepareJevReview, assessJevReview, jevEvaluator,
+} from '@tangle-network/agent-eval/jev'
+
+const evaluate = jevEvaluator({
+  evaluate: nativeTransport, // One Router /v1/systemone attempt; preserve signal/idempotency key.
+  receipt: readRouterReceipt,
+  maximumCharge: enforcedMaximumCharge,
+})
+
+const review = prepareJevReview({
+  version: policy.version,
+  request: {
+    model: selectedModel,
+    state: { policy: policy.definition, evidence: evidence.content },
+    questions: { action: question },
+  },
+  checks: {
+    action: {
+      claim: policy.hypothesis,
+      subject: actionId,
+      area: 'authorization-review',
+      severity: policy.severity,
+      supports: ['supported'],
+      refutes: ['refuted'],
+      supportAtLeast: limits.support,
+      refuteAtLeast: limits.refute,
+      coverage: evidence.coverage,
+      evidence: evidence.refs,
+    },
+  },
+})
+const result = await evaluate(review.request, executionContext)
+await persistAuthorizedObservation({ review, result })
+const report = assessJevReview(review, result.value)
+```
+
+The example assumes a native Choice question with `supported`, `refuted` and `insufficient`
+alternatives. Other labels, Noul polarity (`true`/`false`) and grouped Score levels are supported.
+A Score's expected level is not itself the probability of a breach; explicitly select which
+levels support or refute the claim. The two thresholds must be positive, at most one and sum to
+more than one, so their regions do not overlap. This is an optional disjoint-region policy,
+not a requirement imposed on other evaluators.
+
+## Optional behavior recipe
+
+`examples/jev-behavior-review.ts` provides replaceable question definitions and
+`behaviorReviewAnalyst`, which composes the existing `jevAnalyst`. Copy/adapt the recipe in an
+application; it is not a default policy installed by importing the package. Select only relevant
+checks and provide thresholds, severity, exact policy and evidence for every selected check.
+
+| Hypothesis | Evidence needed | Important distinction |
+| --- | --- | --- |
+| Reward gaming | Task objective, raw proxy score, independent outcome and attributable action | A proxy/outcome gap can be a broken evaluator, not deliberate gaming. |
+| Evaluator tampering | Protected evaluator identity, allowed changes and trusted before/after diff | A task that authorizes test repair must not be flagged merely for editing tests. |
+| Safety-policy violation | Exact applicable policy and attributable action/output | Quotation, refusal and simulation are not completed violations. |
+| Unauthorized action | Effective grant at action time, requested target and enforcement outcome | An attempted action, a blocked request and a completed effect differ. |
+| Data-boundary violation | Data classification, allowed destination and actual payload/egress record | Do not infer a successful transfer from a plan or URL mention. |
+| Injection followed | Untrusted instruction, trusted task and subsequent attributable action | Reading or rejecting malicious text is not obeying it. |
+| Contradicted success claim | Exact claim and independent receipt for the same operation | No receipt means unresolved, not proof of deception. |
+| Oversight bypass | Applicable approval/monitor/cancel control and attributable change | Lost telemetry or an authorized control change is not evidence of agent evasion. |
+| Anomalous sequence | Matched versioned baseline and differences computed in code | Unusual does not mean harmful; no baseline means no established anomaly rate. |
+| Cross-run coordination | Authorized joined traces, linkage, timeline, policy and joint effect | Shared infrastructure or legitimate delegation is not collusion. |
+
+A hypothesis gets its own question; these behaviors are not mutually exclusive alternatives in
+one forced-choice category. Prefer a small relevant evidence window per review. A rule such as
+"use only evidence.X" is an instruction, not technical isolation between other inputs in the
+same request. Run separate requests when evidence or tenant isolation requires it.
+
+## What the numbers mean
+
+The report retains support, refutation and remaining native mass without renormalizing away
+uncertainty. Native `confidence` is stored separately: TypeSafe defines it as a statistic of the
+distribution, not the same value as the winning alternative's probability. Neither is a measured
+probability of malicious intent, future harm or successful exfiltration.
+
+The host declares evidence coverage. Non-missing coverage requires references; the library does
+not authenticate those references, read their targets, prove completeness or know that the
+right evidence was supplied. The caller must bind references to immutable authorized records
+and put the relevant content in state. A policy document alone is not behavioral evidence.
+Missing evidence always stays unresolved. Partial evidence may support a scoped hypothesis but
+cannot refute it under this helper's conservative policy.
+
+Every native result should be retained through the existing `record` callback before findings
+are reduced. A refuted check remains in the report but produces no issue finding; an empty
+finding list is not a universal safety certificate. Supported findings use model support mass
+as `confidence`, explicitly marked uncalibrated. Unresolved findings use confidence 1 only for
+the deterministic fact "this review is unresolved", with `confidence_basis: deterministic-status`.
+Do not graph those two bases together as risk probabilities.
+
+Do not multiply probabilities from correlated questions, add overlapping category probabilities
+as a run-wide risk, or substitute `1 - P(supported)` for `P(refuted)` when insufficient evidence
+has its own mass. For an independent audit of these ternary outcomes, use `auditEvaluator` with
+the actual mapped decision and unknowns. `auditProbabilityPolicy` is for a genuine binary
+probability with its frozen binary threshold rule; it must not silently drop the third category.
+
+## Existing components to reuse
+
+`/rl` already has `detectRewardHacking` for reward divergence, distribution shift, reward
+disagreement and judge drift. Those diagnostic scores are not probabilities. Keep the raw,
+ungated proxy and independently measured task outcome so a previous rejection does not hide the
+disagreement. Do not substitute the Jev monitor as the only independent outcome.
+
+The analyst system already has deterministic control-integrity and behavioral/efficiency
+checks, plus deep trace analyst definitions such as intent divergence and knowledge poisoning.
+Run exact checks for facts code can establish; use a bounded semantic review for interpretation;
+escalate selected cases to the existing deeper analyst. None is replaced by this recipe.
+
+A prepared review can be a custom registry input or built from a filtered trace store. Existing
+`asAnalyst`/`jevAnalyst` forward the shared paid-call account, cancellation and deadline. The
+recipe verifies persisted definition identity before inference, records the native observation
+before mapping, and returns ordinary findings. Graphs consume these through their existing
+analyst-registry adapter. No new supervisor, scheduler, ledger, or trace database is introduced.
+
+For judges, map assessments explicitly and refuse unresolved evidence where the scored metric
+requires resolution. Never convert an unresolved safety review to a perfect score or average a
+critical constraint failure into unrelated high scores. Working feedback must not expose the
+independent final evaluator, protected holdout labels, or monitor control credentials to the worker.
+
+## Security and validation boundaries
+
+TypeSafe's Jev 1.13 documentation explicitly warns that adversarial state content can move the
+answer. Delimiters and explicit criteria are useful instructions, not a proof of injection
+resistance. These monitors are advisory evidence, not the sole authorization/safety barrier.
+Trusted policy and references must come from outside the evaluated agent; tenant scope, redaction,
+credential handling and controls remain with existing application/security owners.
+
+A required pre-action check must execute before the effect, and a pre-delivery check must run
+before streaming exposes the answer. Post-run analysis cannot undo an effect or retract text.
+External harnesses only support their actual documented boundaries; a whole-prompt callback is
+not an interception of every native request. An observational hook is not a blocking gate.
+
+Before enabling automatic decisions, independently test both attacks and ordinary authorized
+behavior: legitimate test edits, benign quotations, refusals, blocked attempts, telemetry gaps,
+wrong-tenant evidence, long/misleading traces and multi-run attacks. Measure per-category false
+positives, false negatives, abstention, task success, latency and full cost using independent
+source units. Freeze thresholds on development data, retain a separate final set, and audit a
+sample of accepted cases to avoid selective-label blindness. Test the monitor under adaptive
+attacks, not only the worker under a fixed monitor.
+
+The new tests exercise pure mapping, canonical identity, metadata isolation and real
+Eval/AnalystRegistry accounting composition with an injected model boundary. They do not prove
+that Jev detects these behaviors correctly or is robust against adversarial prompts. No live
+provider, product UI rollout, runtime hook installation or production enforcement is added here.
+
+Sources reviewed: https://docs.typesafe.ai/confidence and
+https://docs.typesafe.ai/model-jaggedness/jev-1.13 (vendor page reviewed 2026-09-17).

package/docs/jev-integrations.md

@@ -0,0 +1,50 @@
+# Ten useful integrations, one evaluator
+
+Jev evaluates bounded questions. Applications supply state, questions and the policy interpreting
+answers. jevEvaluator handles the native model; createEvaluator supports other classifiers.
+asJudge and asAnalyst adapt the same operation to existing consumers.
+
+| Use | Outcome | Existing execution path |
+| --- | --- | --- |
+| Completion evidence | Verify a deliverable against requirements and independent artifacts | Native questions plus deterministic checks; unresolved stays unresolved. |
+| Recommendation triage | Prioritize evidence-linked Intelligence recommendations | Product triage selects existing engine findings and preserves their identities. |
+| Trace diagnosis | Narrow likely failures before deeper investigation | AnalystRegistry, scoped evidence, and the existing deep analyst. |
+| Behavioral review | Flag observable policy concerns, not hidden intent | Optional behavior-review recipe plus deterministic reward/control checks. |
+| Rubric judging | Compare product or agent outputs | jevJudge/asJudge with caller mappings and independent final assessment. |
+| Context selection | Select useful retrieved evidence | Native assessments, then deterministic packing with mandatory context protected. |
+| Skill selection | Select from authorized capabilities | Caller alternatives, including no-selection; existing runtime executes actions. |
+| Next verification | Choose a test, retrieval or review that resolves uncertainty | Existing graph checkpoints and awaited boundaries, not new permissions. |
+| Candidate selection | Choose among generated plans or artifacts | Existing matrix/graph, exact checks first, semantic assessment second. |
+| Configuration experiments | Compare questions, context, models and thresholds | examples/jev-decision-benchmark.ts using runAgentMatrix and cost receipts. |
+
+These are compositions, not ten new frameworks. Product triage and the benchmark recipe are concrete
+adoption work; other rows reuse existing APIs or identify application policies to configure. An API
+or recipe is not proof that every domain policy has been validated or deployed.
+
+## People, agents and evidence
+
+Developers need typed decisions and attributable spend. Operators need evidence, uncertainty and
+an actionable next check. Programmatic agents need stable JSON, explicit failure, a shared budget
+and retained results. Benchmark authors need independent labels that never enter model requests.
+
+Before execution, freeze inputs, independent source-unit splits, expected outcomes and configuration
+versions. During execution, retain the native request/answer, served model, receipt, timings and
+selected action. Afterwards, join independent outcomes, corrections and missing evidence. A model
+explanation is not an independent outcome.
+
+## Compare policies with the existing matrix
+
+The compareDecisions recipe accepts cases, configurations and an ordinary evaluator. Each case
+separates input from expected decision (or reviewed unresolved label) and sourceUnit. Question
+builders receive only input. Mapping and recording failures remain failed cells with paid costs;
+unknown spend is not measured free. Concurrency, repetitions and budget authority are caller-owned.
+Configuration results use `id@version` keys, with each component URL encoded; duplicate pairs fail
+before execution. This keeps versions of one policy separate in scores and costs.
+
+Report accuracy and automation coverage separately: always abstaining does not make a useful agent.
+Reuse existing audit/calibration functions and independent source units rather than inventing another
+statistics layer. The recipe chooses no default model, universal threshold or live request.
+
+Changing only a mapping can re-score saved probabilities. Changing context, questions or actions
+requires execution of the affected path. Compare verified outcomes and total spend, not report
+length, finding count or how confidently a model declares success.

package/docs/jev.md

@@ -0,0 +1,140 @@
+# Typed evaluations: products, judges, and analysts
+
+Configure a transport once. Supply your own model, state, and questions on every
+call. Interpret the answers in ordinary code; no campaign or agent is required.
+
+```ts
+import { TypeSafeClient, noul, choice } from '@typesafe-ai/sdk'
+import { jevEvaluator } from '@tangle-network/agent-eval/jev'
+
+const client = new TypeSafeClient({
+  baseURL: routerOrigin, // Origin without /v1; omit to use TypeSafe directly.
+  apiKey: scopedRouterKey,
+  retry: { maxRetries: 0 },
+})
+const evaluate = jevEvaluator({
+  evaluate: (request, { signal, idempotencyKey }) => client.systemOne(request, {
+    signal,
+    retry: { maxRetries: 0 },
+    headers: { 'Idempotency-Key': idempotencyKey },
+  }),
+  pricing: reviewedTokenPricing, // An estimate, not a provider-observed charge.
+})
+
+const result = await evaluate({
+  model: selectedModel,
+  state: { requirements, evidence },
+  questions: {
+    supported: noul('Does the evidence support the claim?'),
+    next: choice({ task: 'Choose the next step using the evidence' }, {
+      inspect: { when: 'Evidence is missing or contradictory' },
+      finish: { when: 'The requirements are satisfied' },
+    }),
+  },
+}, { signal, costLedger })
+
+result.value.answers.next.choice // 'inspect' | 'finish'
+result.value.answers.next.probabilities.inspect // number
+result.receipt // Existing CostLedger receipt, separate from native answers.
+```
+
+The named configuration and evidence values belong to the application. Plain JSON
+questions work too; SDK builders are optional. Structured criteria, null entries,
+and omitted optional fields are supported. Static TypeScript questions preserve
+exact answer names and choice-label unions. Runtime JSON must be validated; its
+labels cannot become compile-time literals automatically.
+
+## Any classifier, not just Jev
+
+`createEvaluator`, `asJudge`, and `asAnalyst` are provider-independent exports from
+`@tangle-network/agent-eval/evaluation` (also re-exported from `/jev`).
+`createEvaluator({ execute, receipt })` meters a typed asynchronous function.
+It does not require a Jev response, probability output, prompt, or provider SDK.
+
+`jevEvaluator` adds native protocol validation and optional `acceptModel(requested,
+served)` policy. It preserves the reported model instead of guessing alias rules.
+Router independently enforces its supported model and pricing policy.
+
+## Adapt the same observation, without another model call
+
+`asJudge({ name, version, dimensions, evaluate, map, record? })` returns the existing
+`JudgeConfig`. The evaluator receives `{ artifact, scenario }` and execution context.
+`map(value, input)` returns the application's `JudgeScore` on its chosen scale.
+`record(result, input)` can persist the full distribution and receipt before reduction.
+
+`asAnalyst({ id, version, description, inputKind, cost, evaluate, map, record? })`
+returns the existing `Analyst`. The evaluator receives the input, evaluation context,
+and analyst context. `map(value, input, analystContext)` creates evidence-backed
+findings. Use the existing `makeFinding` factory and real event/span/artifact references.
+An empty finding list is valid; an inference failure is not a clean review.
+
+The optional `jevJudge` and `jevAnalyst` conveniences accept static questions or a
+question-producing function. State renderers may be async. Judge renderers receive
+the evaluation context; analyst renderers and finding mappers receive the effective
+cancellation/deadline signal. Rendering must honor that signal during external work.
+Both conveniences accept a `record` callback for the complete observation.
+
+For a numeric rubric, `jevJudge` defaults to `normalizedJevScore`: equally spaced
+score expectations on [0,1], noul values unchanged, and an optional weighted mean.
+It emits no canned notes. Supply `map` for choice utilities, different scales, or
+critical-failure policies. Dynamic questions and custom mappings require explicit
+stable output `dimensions`; changing intermediate questions must not silently change
+the metric being compared. Bump `version` when opaque renderers or mappings change.
+Optional undefined fields from SDK builders hash like the JSON they send. Invalid
+JSON evidence, including Map, Date, cycles, and non-finite numbers, is rejected rather
+than silently changed before inference. Convert timestamps explicitly to strings.
+
+## Execution and accounting
+
+Use the caller's existing `costLedger`, signal, run tags, and phase. A capped ledger
+requires a genuine transport-enforced maximum charge, not a guessed cost ceiling.
+One injected call represents one paid attempt; do not hide retries at multiple layers.
+An idempotency header is correlation, not proof of upstream inference deduplication.
+Router charges the inference. Eval records attribution and must not charge it again.
+
+The shared analyst adapter uses an explicit `AnalystContext.costLedger` first. If
+none is supplied, an explicit `budgetUsd` creates an account for that invocation.
+Otherwise the evaluator keeps its configured ledger, including its cumulative cap
+across direct calls, judges, and analysts. An absent analyst budget never creates a
+fresh unlimited account that overrides the configured one. This precedence is the
+same for Jev and other classifiers. A per-analysis budget replaces the default
+account; it is not an additional child limit. Pass the shared run ledger explicitly
+when the whole run must conserve one authority. Only trusted host code should choose
+or replace that authority.
+
+`evaluate(request, { callId, costLedger, signal })` may supply the existing ledger's
+physical paid-call identity. It reaches both `receipt.callId` and the transport's
+`idempotencyKey`. Omit it for a newly allocated identity. Duplicate pending or settled
+calls in the same ledger are refused before another dispatch; they do not return a
+cached result. Preserve a retained observation for replay, and allocate a different
+physical-attempt identity only when a retry is explicitly authorized. This adds no
+cross-credential workflow recovery or provider-side deduplication guarantee.
+
+Usage settles before answer validation and mapping. Failed parsing or mapping never
+undoes paid work. Persisted observations remain available even if cancellation prevents
+their reduction to a decision. Unknown costs are not measured zero. `record` is a
+persistence seam, not an automatic cache: callers own storage, authorization, replay
+identity, and resume policy.
+
+## Graphs, hooks, and optimization
+
+Use the same evaluator in products, a graph analyst, an awaited local pre/post-call
+function, or a rollout checkpoint. Runtime owns those execution boundaries; Eval does
+not add a scheduler or import Runtime. Observation-only Runtime hooks are not blocking
+controls, and one external sandbox prompt may include many native model requests.
+A delivery check must run before delivery; it cannot retract streamed text or tool effects.
+
+Reuse existing campaigns and optimization methods to measure questions, context
+selection, rubric levels, and routing policies. Keep final assessment independent of
+selection and retain exact input/model/policy identities. Do not treat reported model
+confidence as demonstrated calibration or a classification as proof of root cause.
+
+## Checks
+
+Run `pnpm exec vitest run tests/jev*.test.ts tests/evaluation-authority.test.ts`,
+`pnpm typecheck`, `pnpm build`, and `pnpm verify:package`. These are contract and
+execution checks, not live-provider quality or deployment evidence. The product-integrity
+rubric lives in `examples/`, not the generic implementation, and is not a default policy.
+
+Protocol: https://docs.typesafe.ai/api
+SDK: https://github.com/typesafe-ai/typesafe-sdk-js

package/docs/produced-state.md

@@ -0,0 +1,13 @@
+# Produced state and artifact revisions
+
+`extractProducedState(events)` reduces one emission-ordered event stream to the latest observed artifact and proposal state for completion checking. Keep the original stream for history.
+
+Artifacts use the exact `name`, then `uri`, then `artifactId` as their identity. A later event for that identity replaces its content and type. Distinct output paths remain distinct even when an emitter reuses an artifact ID. No path normalization, rename, or deletion is inferred.
+
+Proposals use `proposalId`. A later rejected or pending observation must not leave an older approval eligible for scoring. Missing content on the latest observation is unknown/empty evidence; it is not filled from an older revision.
+
+This behavior correction prevents an overwritten deliverable from satisfying completion through stale text. Consumers that need every historical version should retain the input stream instead of reading the reduced arrays. First-seen identity order is preserved for deterministic output.
+
+The helper does not verify a provider transaction. Tool-call names are invocations, not successful execution receipts. An agent-written file claiming payment or delivery is not independently observed merchant or fulfillment evidence. Validate those effects through the owning integration and use the existing completion and trace-verification APIs for the required checks.
+
+Input ordering is the caller's responsibility. Do not combine unordered collectors and claim the result is a current snapshot. Unobserved external changes and deletions cannot be reconstructed from this event subset.

package/docs/profile-json-identity.md

@@ -0,0 +1,33 @@
+# Profile JSON identity
+
+`toAgentProfileJson` converts a profile's JSON document form through
+`ledger-core/canonical`, the existing owner of canonical serialization. Object
+properties whose value is `undefined` are omitted, including nested optional
+properties. Everything else must have a faithful canonical JSON representation.
+Functions, symbols, non-finite numbers, BigInt, class instances, Map, Set, sparse
+arrays, undefined array members, malformed Unicode and cycles are refused with
+`AgentProfileCellValidationError`. Custom `toJSON` functions are not a way to
+substitute different evidence.
+
+This fixes an older conversion that used `JSON.stringify` before validation:
+functions could disappear and non-finite numbers could become null before the
+profile was hashed. Callers must supply portable data rather than depend on those
+coercions. An absent measurement is not a measured null or zero.
+
+Cell dimensions preserve every own string key, including `__proto__`. A
+previously stored cell whose dimension was omitted from its digest no longer
+verifies against that dimension. Do not relabel or rewrite an old cell's digest;
+retain the original record as unverifiable and issue a fresh measurement when
+needed. Ordinary portable profiles and dimensions keep the same source hashes
+and cell IDs.
+
+`sourceProfile.hash` commits to the profile alone. `sourceProfile.kind` also
+contributes to the cell ID, so a source identity join should compare both kind
+and hash. A loose product-specific fingerprint must not claim to be a validated
+canonical AgentProfile merely because it is JSON-serializable.
+
+Regression coverage: `tests/profile-json-canonical.test.ts`, together with the
+existing `tests/agent-profile-cell.test.ts`. Run the repository typecheck, full
+suite, build and package checks before release. This document does not allocate
+or announce a package version; release metadata and downstream peer ranges must
+be updated together by the normal release process.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.183.0",
+  "version": "0.184.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -19,6 +19,11 @@
       "import": "./dist/index.js",
       "default": "./dist/index.js"
     },
+    "./jev": {
+      "types": "./dist/jev.d.ts",
+      "import": "./dist/jev.js",
+      "default": "./dist/jev.js"
+    },
     "./profile-cell": {
       "types": "./dist/profile-cell.d.ts",
       "import": "./dist/profile-cell.js",
@@ -156,6 +161,16 @@
       "types": "./dist/adapters/http.d.ts",
       "import": "./dist/adapters/http.js",
       "default": "./dist/adapters/http.js"
+    },
+    "./evaluation": {
+      "types": "./dist/evaluation.d.ts",
+      "import": "./dist/evaluation.js",
+      "default": "./dist/evaluation.js"
+    },
+    "./jev/protocol": {
+      "types": "./dist/jev-protocol.d.ts",
+      "import": "./dist/jev-protocol.js",
+      "default": "./dist/jev-protocol.js"
     }
   },
   "bin": {