@tangle-network/agent-eval 0.77.0 → 0.80.0

package/README.md

@@ -1,6 +1,8 @@
 # `@tangle-network/agent-eval`
 
-**Ship better agent prompts with statistical confidence.** One function call returns a decision packet: lift CI, judge calibration, contamination check, failure clusters, cost-quality Pareto, and a ranked action list. Same shape whether you've got a closed improvement loop or just production logs.
+**Decision-grade evals for agents.** One function call returns a decision packet — lift CI, judge calibration, contamination check, failure clusters, cost-quality Pareto, and a ranked action list — with the same shape whether you have a closed improvement loop or just production logs.
+
+It is the **substrate at the bottom of the stack**: [`@tangle-network/agent-runtime`](https://www.npmjs.com/package/@tangle-network/agent-runtime) runs agents and captures every run as a trace, then delegates scoring and the ship gate here. The dependency arrow only points up — agent-eval never imports the runtime.
 
 [![npm](https://img.shields.io/npm/v/@tangle-network/agent-eval.svg)](https://www.npmjs.com/package/@tangle-network/agent-eval)
 [![pypi](https://img.shields.io/pypi/v/agent-eval-rpc.svg)](https://pypi.org/project/agent-eval-rpc/)
@@ -207,28 +209,55 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
 
 | Subpath | What it gives you |
 |---|---|
-| `@tangle-network/agent-eval/contract` | **The headline surface.** `selfImprove`, `analyzeRuns`, `runImprovementLoop`, `runCampaign`, `runEval`, `diffRuns`, intake adapters (`fromFeedbackTable`, `fromOtelSpans`), drivers (`gepaDriver`, `evolutionaryDriver`), gates (`defaultProductionGate`, `heldOutGate`, `composeGate`), storage. **New code starts here.** |
-| `@tangle-network/agent-eval/hosted` | Hosted-tier wire-format types + `createHostedClient` to ship eval-run events + trace spans to any orchestrator speaking the spec |
-| `@tangle-network/agent-eval/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest |
-| `@tangle-network/agent-eval/adapters/langchain` | LangChain runnable → `Dispatch` adapter |
-| `@tangle-network/agent-eval/adapters/http` | `httpDispatch` + `runDispatchServer` for distributed campaigns across machines |
-| `@tangle-network/agent-eval/campaign` | Lower-level campaign primitives (storage, drivers, types) |
-| `@tangle-network/agent-eval/multishot` | N-shot persona × shot matrix runner |
-| `@tangle-network/agent-eval/control` | Agent control loop primitives (`runAgentControlLoop`, action policy, propose/review) |
-| `@tangle-network/agent-eval/traces` | Trace stores, emitters, OTLP-JSONL replay |
-| `@tangle-network/agent-eval/reporting` | Release confidence, paired stats, sequential e-values, launch reports |
-| `@tangle-network/agent-eval/rl` | RL bridge — verifiable rewards, preferences, OPE, PRM, tournaments, contamination, compute curves, auto-research |
-| `@tangle-network/agent-eval/matrix` | N-axis cartesian over substrate types |
-| `@tangle-network/agent-eval/wire` | HTTP/RPC server + Zod schemas (same protocol the Python client speaks) |
-| `@tangle-network/agent-eval/benchmarks` | Benchmark adapter contracts and reference wrappers |
-
-The root export remains available for backward compatibility; new code should prefer focused subpaths. Anything under `/rl`, `/pipelines`, `/meta-eval`, `/prm`, or `/builder-eval` is **only** reachable via its subpath.
+| `…/contract` | **The headline, frozen surface — new code starts here.** `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); drivers (`gepaDriver`, `evolutionaryDriver`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `Mutator` / `Gate`. |
+| `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) |
+| `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency |
+| `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep |
+| `…/adapters/http` | `httpDispatch` + `runDispatchServer` — run a campaign's worker on another machine (multi-region, driver-as-a-service) |
+| `…/campaign` | **The measurement + improvement engine** (`@experimental`): `runProfileMatrix`, `compareDrivers`, every driver (`gepaDriver`, `haloDriver`, `skillOptDriver`, `aceDriver`, `memoryCurationDriver`, …), the gates, storage backends, and loop provenance. `/contract` re-exports the stable subset. |
+| `…/rl` | RL bridge from eval artifacts to training signal: verifiable rewards, preferences, OPE, PRM, tournaments, contamination, compute curves, plus the durable corpus + `buildRlDataset` / datasheet bundle |
+| `…/reporting` | Release-decision statistics: `pairedBootstrap`, `benjaminiHochberg`, anytime-valid sequential e-values, `evaluateReleaseConfidence`, and the report renderers |
+| `…/analyst` | The trace-analyst surface: `AnalystRegistry` + `buildDefaultAnalystRegistry` (run the failure-clustering panel), `FindingsStore`, and the LLM chat transports |
+| `…/traces` | Trace stores + emitters, OTLP-JSONL deterministic replay, `analyzeTraces`, and the `traceAnalystOnRunComplete` hook |
+| `…/control` | Agent control loop: `runAgentControlLoop` (observe → validate → decide → act), action policy, propose/review |
+| `…/matrix` | `runAgentMatrix` — an N-axis cartesian over caller-supplied substrate values, per-axis pass/score/cost/duration |
+| `…/multishot` | N-shot persona × shot matrix runner (`runMultishot` / `runMultishotMatrix`) |
+| `…/wire` | The cross-language HTTP/RPC server + Zod schemas (the source-of-truth protocol the Python client speaks) + the built-in rubric registry |
+| `…/benchmarks` | `BenchmarkAdapter` contract + `deterministicSplit` + the bundled `routing` reference benchmark |
+
+**Specialized surfaces** (subpath-only): `…/prm` (process-reward grading + best-of-N), `…/meta-eval` (judge calibration + the deployment-outcome store), `…/pipelines` (trace-diagnostic views: budget breach, failure cluster, stuck loop, …), `…/governance` (EU AI Act / NIST AI RMF / SOC2 reports), `…/knowledge` (knowledge-readiness gating before a run), `…/builder-eval` (code-generator three-layer eval), `…/storyboard` (trace → watchable replay), `…/authenticity` (anti-Goodhart "real or convincing BS" scorer over produced files), `…/workflow` (workflow-trace eval + partner export), `…/telemetry` (Workers-safe telemetry client).
+
+The root export remains available for backward compatibility; new code should prefer the focused subpaths above — `/contract` first.
+
+---
+
+## Composition with the stack
+
+agent-eval is the bottom of the layering: consumers depend on it, it depends on none of them.
+
+```
+agent-runtime    Runs agents (chat turns, one-shot tasks, multi-attempt loops), captures every
+                 run as a trace, and calls optimizePrompt / runImprovementLoop. Produces the
+                 RunRecords + traces agent-eval scores. Depends on agent-eval.
+
+agent-eval       selfImprove, analyzeRuns, runCampaign + drivers (gepaDriver, …), the gates
+   (this repo)   (heldOutGate, defaultProductionGate, paretoSignificanceGate), the InsightReport
+                 decision packet, the RL bridge, the wire protocol. Depends on neither consumer.
+
+agent-knowledge  proposeKnowledgeWrites / applyKnowledgeWriteBlocks. agent-eval's analyst findings
+                 feed it; the knowledge gate consumes them. Depends on agent-eval.
+
+sandbox          AgentProfile, Sandbox.create, streamPrompt. The execution surface the runtime's
+                 loops run on; agent-eval scores what comes back.
+```
+
+The rule: **agent-eval has zero upward dependencies on a consumer.** A concept that makes sense *without* a running agent loop — a verdict, a run record, a scenario, a judge score — is substrate and lives here; a runtime-shaped one (a sandbox profile, a validation context with an abort signal) lives in agent-runtime. When in doubt, lean substrate.
 
 ---
 
 ## Concepts + design
 
-- [`docs/concepts.md`](./docs/concepts.md) — five types, three top-level functions, the layering rule, the wire protocol contract
+- [`docs/concepts.md`](./docs/concepts.md) — the three top-level functions, the layering rule, and the wire-protocol contract (the five core contract types are documented in the `/contract` barrel itself)
 - [`docs/insight-report.md`](./docs/insight-report.md) — annotated walkthrough of every section of the decision packet
 - [`docs/customer-journeys.md`](./docs/customer-journeys.md) — three end-to-end journeys with code + expected output
 - [`docs/adapters-observability.md`](./docs/adapters-observability.md) — composing agent-eval with LangSmith, Langfuse, Phoenix, OpenLLMetry, TraceAI
@@ -287,7 +316,9 @@ pnpm test
 
 ## Stability + versioning
 
-Public exports carry JSDoc stability markers visible in IDE hover + `.d.ts`:
+The `/contract` surface is the **stability contract**: its barrel freezes the API — a `0.x` minor only *adds*; nothing there changes shape or disappears. Depend on `/contract` (and the documented subpaths) rather than the root barrel.
+
+In the deeper subpaths, `@stable` / `@experimental` JSDoc markers (visible in IDE hover + `.d.ts`) call out what may still move — most granularly in `/rl` (tagged per export) and `/campaign` (whole barrel `@experimental`, since `/contract` re-exports only its settled subset).
 
 | Tag | Meaning |
 |---|---|

package/dist/adapters/http.d.ts

@@ -1,5 +1,5 @@
-import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-Bba0vl1V.js';
-import '../run-record-BgTFzO2r.js';
+import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-4mm2msnR.js';
+import '../run-record-sItO5ftF.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';
 

package/dist/adapters/langchain.d.ts

@@ -1,5 +1,5 @@
-import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-Bba0vl1V.js';
-import '../run-record-BgTFzO2r.js';
+import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-4mm2msnR.js';
+import '../run-record-sItO5ftF.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';
 

package/dist/adapters/otel.d.ts

@@ -1,10 +1,10 @@
 import { TraceSpanEvent, HostedClient } from '../hosted/index.js';
-import '../types-Bba0vl1V.js';
-import '../run-record-BgTFzO2r.js';
+import '../types-4mm2msnR.js';
+import '../run-record-sItO5ftF.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';
-import '../insight-report-Df3lxYXM.js';
-import '../summary-report-ByiOUrHj.js';
+import '../insight-report-dlpEzQDi.js';
+import '../summary-report-BTaXq1TS.js';
 import '../failure-cluster-CL7IVgkJ.js';
 import '../store-CKUAgsJz.js';
 import '../judge-calibration-DilmB3Ml.js';

package/dist/{agent-profile-DYRboYWu.d.ts → agent-profile-aSEaJ9Pl.d.ts}

@@ -1,5 +1,5 @@
 import { A as AgentEvalError } from './errors-Dwqw-T_m.js';
-import { R as RunRecord } from './run-record-BgTFzO2r.js';
+import { R as RunRecord } from './run-record-sItO5ftF.js';
 import { TCloud } from '@tangle-network/tcloud';
 
 /**

package/dist/analyst/index.d.ts

@@ -1,21 +1,21 @@
 import { AxAIService, AxFunction } from '@ax-llm/ax';
 import { M as MultiLayerVerifier, V as VerifyOptions, S as Severity } from '../multi-layer-verifier-DlWCXuxL.js';
 import { c as RunCritic, a as RunTrace } from '../run-critic-BAIjX99r.js';
-import { S as SemanticConceptJudgeOptions, a as SemanticConceptJudgeInput, B as BehavioralMetrics } from '../semantic-concept-judge-CV9Wlx4t.js';
-export { C as CreateAnalystAiConfig, D as DEFAULT_TRACE_ANALYST_KINDS, b as DefaultAnalystRegistryOptions, c as DiffPolicy, F as FAILURE_MODE_KIND_SPEC, d as FINDING_SUBJECT_GRAMMAR_PROMPT, e as FINDING_SUBJECT_KINDS, f as FindingSubject, g as FindingSubjectKind, h as FindingSubjectStringSchema, i as FindingsDiff, j as FindingsStore, I as IMPROVEMENT_KIND_SPEC, K as KIND_EXPECTED_SUBJECTS, k as KNOWLEDGE_GAP_KIND_SPEC, l as KNOWLEDGE_POISONING_KIND_SPEC, P as PersistedFinding, m as SKILL_USAGE_ANALYST, n as SkillUsageAnalyst, o as SkillUsageRecord, p as SkillUsageReport, q as SkillUsageScanConfig, r as buildDefaultAnalystRegistry, s as buildSkillUsageReport, t as createAnalystAi, u as defaultIsMaterial, v as diffFindings, w as emitSkillUsageFindings, x as parseFindingSubject, y as renderFindingSubject } from '../semantic-concept-judge-CV9Wlx4t.js';
+import { S as SemanticConceptJudgeOptions, a as SemanticConceptJudgeInput, B as BehavioralMetrics } from '../semantic-concept-judge-qXEUV2w7.js';
+export { C as CreateAnalystAiConfig, D as DEFAULT_TRACE_ANALYST_KINDS, b as DefaultAnalystRegistryOptions, c as DiffPolicy, F as FAILURE_MODE_KIND_SPEC, d as FINDING_SUBJECT_GRAMMAR_PROMPT, e as FINDING_SUBJECT_KINDS, f as FindingSubject, g as FindingSubjectKind, h as FindingSubjectStringSchema, i as FindingsDiff, j as FindingsStore, I as IMPROVEMENT_KIND_SPEC, K as KIND_EXPECTED_SUBJECTS, k as KNOWLEDGE_GAP_KIND_SPEC, l as KNOWLEDGE_POISONING_KIND_SPEC, P as PersistedFinding, m as SKILL_USAGE_ANALYST, n as SkillUsageAnalyst, o as SkillUsageRecord, p as SkillUsageReport, q as SkillUsageScanConfig, r as buildDefaultAnalystRegistry, s as buildSkillUsageReport, t as createAnalystAi, u as defaultIsMaterial, v as diffFindings, w as emitSkillUsageFindings, x as parseFindingSubject, y as renderFindingSubject } from '../semantic-concept-judge-qXEUV2w7.js';
 import { A as AnalyzeTracesOptions } from '../analyst-t7zZS3TV.js';
 import { T as TraceAnalysisStore } from '../store-GmBE2pZZ.js';
 import { b as JudgeFn, a as JudgeInput } from '../types-Croy5h7V.js';
-import { A as Analyst, h as AnalystSeverity, c as AnalystFinding } from '../types-CRD68aH7.js';
-export { a as AnalystContext, g as AnalystCost, i as AnalystInputKind, j as AnalystRequirements, f as AnalystRunEvent, e as AnalystRunInputs, d as AnalystRunResult, b as AnalystRunSummary, k as ChatCallOpts, C as ChatClient, l as ChatRequest, m as ChatResponse, n as ChatTransport, o as CliBridgeTransportOpts, p as CreateChatClientOpts, D as DirectProviderTransportOpts, E as EvidenceRef, M as MockTransportOpts, R as RouterTransportOpts, S as SandboxSdkTransportOpts, q as computeFindingId, r as createChatClient, s as makeFinding } from '../types-CRD68aH7.js';
+import { A as Analyst, h as AnalystSeverity, c as AnalystFinding } from '../types-DRvV0zRo.js';
+export { a as AnalystContext, g as AnalystCost, i as AnalystInputKind, j as AnalystRequirements, f as AnalystRunEvent, e as AnalystRunInputs, d as AnalystRunResult, b as AnalystRunSummary, k as ChatCallOpts, C as ChatClient, l as ChatRequest, m as ChatResponse, n as ChatTransport, o as CliBridgeTransportOpts, p as CreateChatClientOpts, D as DirectProviderTransportOpts, E as EvidenceRef, M as MockTransportOpts, R as RouterTransportOpts, S as SandboxSdkTransportOpts, q as computeFindingId, r as createChatClient, s as makeFinding } from '../types-DRvV0zRo.js';
 import { TCloud } from '@tangle-network/tcloud';
-export { A as ANALYST_SEVERITIES, C as CreateTraceAnalystKindOpts, R as RAW_FINDING_SCHEMA_PROMPT, a as RawAnalystFinding, b as RawAnalystFindingSchema, T as TraceAnalystGolden, c as TraceAnalystKindSpec, d as createTraceAnalystKind, p as parseRawFinding, r as renderPriorFindings } from '../kind-factory-DW9XWPvM.js';
-export { A as AnalystHooks, a as AnalystRegistry, b as AnalystRegistryOptions, B as BudgetPolicy, R as RegistryRunOpts } from '../registry-DuVYiTvw.js';
+export { A as ANALYST_SEVERITIES, C as CreateTraceAnalystKindOpts, R as RAW_FINDING_SCHEMA_PROMPT, a as RawAnalystFinding, b as RawAnalystFindingSchema, T as TraceAnalystGolden, c as TraceAnalystKindSpec, d as createTraceAnalystKind, p as parseRawFinding, r as renderPriorFindings } from '../kind-factory-DqV2t1Xk.js';
+export { A as AnalystHooks, a as AnalystRegistry, b as AnalystRegistryOptions, B as BudgetPolicy, R as RegistryRunOpts } from '../registry-BK0Zee01.js';
 import { L as LlmClientOptions } from '../llm-client-DbjLfz-K.js';
 import '../schema-m0gsnbt3.js';
 import '../store-CKUAgsJz.js';
 import 'zod';
-import '../run-record-BgTFzO2r.js';
+import '../run-record-sItO5ftF.js';
 import '../errors-Dwqw-T_m.js';
 import '../raw-provider-sink-C46HDghv.js';
 
@@ -149,6 +149,40 @@ declare function coerceJson(text: string): unknown;
  */
 declare function coerceToFindingRows(raw: unknown): unknown[];
 
+/** DESCRIPTIVE predicate: does the finding cite at least one observable
+ *  (span/event/artifact) evidence ref. Useful for ranking evidence quality or
+ *  rendering — it is NOT the steer gate. Evidence presence is the WRONG
+ *  discriminator for steering: a legitimate trace-analyst observation may cite
+ *  nothing (it would be wrongly rejected), and a judge verdict may cite an
+ *  artifact (it would be wrongly admitted). Use `assertNoJudgeVerdict` to gate
+ *  steering; use this only where "is this grounded in observable evidence" is the
+ *  literal question. */
+declare function isTraceObservable(finding: AnalystFinding): boolean;
+/** True iff the finding is a JUDGE VERDICT (an acceptance score lifted into a
+ *  finding), identified by provenance set at the lift site — independent of
+ *  whatever evidence it cites. */
+declare function isJudgeVerdict(finding: AnalystFinding): boolean;
+/**
+ * THE steer firewall. Fail-loud guard for any path that admits analyst findings
+ * as STEERING input (the `f(trace)` role): rejects — naming the offenders — any
+ * finding whose provenance is a judge verdict, rather than let `J` leak into the
+ * loop. Returns the findings unchanged for chaining.
+ *
+ * Call this at the chokepoint where a detector that ALSO scores/gates has its
+ * findings turned into a steer (the judge-and-steer dual-role case). It keys on
+ * provenance, so it correctly admits evidence-less trace-analyst observations and
+ * correctly rejects an artifact-citing judge verdict — the cases an evidence
+ * check gets backwards.
+ *
+ * It is necessary, not sufficient: it stops PROVENANCE-tagged verdicts. A judge
+ * whose output is laundered through a hand-built finding with no provenance flag
+ * is out of its reach — provenance must be honestly set at every judge→finding
+ * lift (today: createJudgeAdapter). That is why the integrity rule lives at the
+ * lift site, and why ProposeContext.judgeScores?: never is the complementary
+ * compile-time tripwire on the obvious direct channel.
+ */
+declare function assertNoJudgeVerdict(findings: ReadonlyArray<AnalystFinding>, context?: string): ReadonlyArray<AnalystFinding>;
+
 /**
  * `structureFindings` — the deferred structuring pass (DSPy TwoStepAdapter /
  * HALO `synthesize_traces` analog). The agentic actor reasons FREE-FORM and
@@ -218,4 +252,4 @@ type TraceToolGroupName =
  */
 declare function buildTraceToolsForGroup(group: TraceToolGroupName, store: TraceAnalysisStore): AxFunction[];
 
-export { Analyst, AnalystFinding, AnalystSeverity, type JudgeAdapterOpts, type RunCriticAdapterOpts, type SemanticConceptJudgeAdapterOpts, type StructureFindingsOptions, type StructureFindingsResult, type TraceAnalystAdapterOpts, type TraceToolGroupName, type VerifierAdapterOpts, behavioralAnalyst, buildTraceToolsForGroup, coerceJson, coerceToFindingRows, createJudgeAdapter, createRunCriticAdapter, createSemanticConceptJudgeAdapter, createTraceAnalystAdapter, createVerifierAdapter, deriveEfficiencyFindings, liftSeverity, stripCodeFences, structureFindings };
+export { Analyst, AnalystFinding, AnalystSeverity, type JudgeAdapterOpts, type RunCriticAdapterOpts, type SemanticConceptJudgeAdapterOpts, type StructureFindingsOptions, type StructureFindingsResult, type TraceAnalystAdapterOpts, type TraceToolGroupName, type VerifierAdapterOpts, assertNoJudgeVerdict, behavioralAnalyst, buildTraceToolsForGroup, coerceJson, coerceToFindingRows, createJudgeAdapter, createRunCriticAdapter, createSemanticConceptJudgeAdapter, createTraceAnalystAdapter, createVerifierAdapter, deriveEfficiencyFindings, isJudgeVerdict, isTraceObservable, liftSeverity, stripCodeFences, structureFindings };

package/dist/analyst/index.js

@@ -14,7 +14,7 @@ import {
   diffFindings,
   emitSkillUsageFindings,
   runSemanticConceptJudge
-} from "../chunk-7W4SM7FD.js";
+} from "../chunk-5LVWPNS5.js";
 import {
   ANALYST_SEVERITIES,
   AnalystRegistry,
@@ -41,7 +41,7 @@ import {
   renderPriorFindings,
   stripCodeFences,
   structureFindings
-} from "../chunk-WYIHD6EB.js";
+} from "../chunk-CF67I6QY.js";
 import "../chunk-IHDHUN2X.js";
 import {
   analyzeTraces
@@ -269,6 +269,11 @@ function liftJudgeScore(analyst_id, area, s) {
     severity,
     confidence: 0.8,
     evidence_refs: s.evidence ? [{ kind: "artifact", uri: "inline:evidence", excerpt: s.evidence }] : [],
+    // Provenance: this finding IS a judge verdict (an acceptance score), not an
+    // observation of behavior. The steer firewall (assertNoJudgeVerdict) rejects
+    // it from steering — even when it cites an artifact above — because letting a
+    // verdict steer the next attempt is the held-out judge leaking into the loop.
+    derived_from_judge: true,
     metadata: { judge_name: s.judgeName, dimension: s.dimension, score_10: score10 }
   });
 }
@@ -323,6 +328,28 @@ function createSemanticConceptJudgeAdapter(opts = {}) {
     }
   };
 }
+
+// src/analyst/steer-firewall.ts
+var OBSERVABLE_KINDS = /* @__PURE__ */ new Set([
+  "span",
+  "event",
+  "artifact"
+]);
+function isTraceObservable(finding) {
+  return finding.evidence_refs.some((ref) => OBSERVABLE_KINDS.has(ref.kind));
+}
+function isJudgeVerdict(finding) {
+  return finding.derived_from_judge === true;
+}
+function assertNoJudgeVerdict(findings, context = "steer") {
+  const leaks = findings.filter(isJudgeVerdict);
+  if (leaks.length > 0) {
+    throw new Error(
+      `${context}: a judge verdict cannot be admitted as steering input \u2014 that is the held-out judge leaking into the loop. Offending judge-derived findings: [${leaks.map((f) => f.finding_id).join(", ")}]. Steering consumes observations of behavior, never acceptance verdicts.`
+    );
+  }
+  return findings;
+}
 export {
   ANALYST_SEVERITIES,
   AnalystRegistry,
@@ -340,6 +367,7 @@ export {
   RawAnalystFindingSchema,
   SKILL_USAGE_ANALYST,
   SkillUsageAnalyst,
+  assertNoJudgeVerdict,
   behavioralAnalyst,
   buildDefaultAnalystRegistry,
   buildSkillUsageReport,
@@ -359,6 +387,8 @@ export {
   deriveEfficiencyFindings,
   diffFindings,
   emitSkillUsageFindings,
+  isJudgeVerdict,
+  isTraceObservable,
   liftSeverity,
   makeFinding,
   parseFindingSubject,

package/dist/analyst/index.js.map

@@ -1 +1 @@
-{"version":3,"sources":["../../src/analyst/adapters.ts"],"sourcesContent":["/**\n * Adapter factories — lift each existing agent-eval primitive into the\n * Analyst contract without re-implementing it.\n *\n * Five primitives, five factories. Each one:\n *   - Builds an Analyst with a stable id (caller chooses; defaults\n *     given), a sensible default `inputKind`, a version derived from\n *     the wrapped primitive's version + an adapter revision, and an\n *     `analyze()` that calls the primitive and lifts its output to\n *     AnalystFinding[] using `makeFinding()`.\n *   - Maps severities: the existing `Severity` ('critical' | 'major' |\n *     'minor' | 'info') projects onto AnalystSeverity ('critical' |\n *     'high' | 'medium' | 'low' | 'info'); 'major' → 'high', 'minor' →\n *     'medium'. Domain analysts that want finer-grained mapping override.\n *\n * Adapters never own state. Calling the same factory twice with the\n * same primitive instance is safe.\n */\n\nimport type { AxAIService } from '@ax-llm/ax'\nimport type {\n  Finding as LayerFinding,\n  Severity as LayerSeverity,\n  MultiLayerVerifier,\n  VerifyOptions,\n} from '../multi-layer-verifier'\nimport { RunCritic, type RunTrace } from '../run-critic'\nimport {\n  runSemanticConceptJudge,\n  SEMANTIC_CONCEPT_JUDGE_VERSION,\n  type SemanticConceptJudgeInput,\n  type SemanticConceptJudgeOptions,\n} from '../semantic-concept-judge'\nimport { type AnalyzeTracesOptions, analyzeTraces } from '../trace-analyst/analyst'\nimport type { TraceAnalysisStore } from '../trace-analyst/store'\nimport type { JudgeFn, JudgeInput, JudgeScore, TCloud } from '../types'\nimport type { Analyst, AnalystFinding, AnalystSeverity } from './types'\nimport { makeFinding } from './types'\n\nconst ADAPTER_REV = '1'\n\n// ── Severity bridges ───────────────────────────────────────────────\n\nexport function liftSeverity(s: LayerSeverity): AnalystSeverity {\n  switch (s) {\n    case 'critical':\n      return 'critical'\n    case 'major':\n      return 'high'\n    case 'minor':\n      return 'medium'\n    case 'info':\n      return 'info'\n  }\n}\n\n// ── 1. analyzeTraces → Analyst ─────────────────────────────────────\n\nexport interface TraceAnalystAdapterOpts {\n  id?: string\n  area?: string\n  /** The natural-language question(s) put to the analyst. One finding per question. */\n  questions: string[]\n  /** Caller-provided AxAI service — same one trace-analyst.ts expects. */\n  ai: AxAIService\n  model?: string\n  /** Forwarded to analyzeTraces. */\n  extra?: Omit<AnalyzeTracesOptions, 'source' | 'ai' | 'model'>\n}\n\n/**\n * @deprecated Prefer `createTraceAnalystKind` + one of the failure /\n * improvement kinds from `./kinds`. This adapter wraps the legacy\n * `analyzeTraces` flow whose output is `findings:string[]` — every\n * bullet gets flat-defaulted severity `medium` / confidence `0.6`,\n * which loses the per-finding grading kinds provide via Ax structured\n * output + Zod validation. Kept for one minor while consumers migrate.\n */\nexport function createTraceAnalystAdapter(\n  opts: TraceAnalystAdapterOpts,\n): Analyst<TraceAnalysisStore> {\n  const id = opts.id ?? 'trace-analyst'\n  const area = opts.area ?? 'agent-reasoning'\n  return {\n    id,\n    description:\n      'Runs the agent-eval trace analyst over an OTLP trace store and lifts its bulleted findings.',\n    inputKind: 'trace-store',\n    cost: { kind: 'llm', models: opts.model ? [opts.model] : undefined },\n    version: `trace-analyst-${ADAPTER_REV}`,\n    async analyze(store, ctx) {\n      const out: AnalystFinding[] = []\n      for (const question of opts.questions) {\n        if (ctx.signal?.aborted) break\n        const result = await analyzeTraces(\n          { question },\n          { source: store, ai: opts.ai, model: opts.model, ...opts.extra },\n        )\n        const subject = ctx.tags?.subject ?? question.slice(0, 60)\n        // The responder produces a list of bullet strings. Each becomes\n        // one finding; the prose answer is attached as rationale on the\n        // first (so renderers that show only top-N still get context).\n        if (result.findings.length === 0) {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject,\n              claim: result.answer.slice(0, 200),\n              rationale: result.answer,\n              severity: 'info',\n              confidence: 0.5,\n              evidence_refs: [],\n              metadata: {\n                actor_prompt_version: result.actorPromptVersion,\n                turns: result.turnCount,\n              },\n            }),\n          )\n          continue\n        }\n        result.findings.forEach((claim, i) => {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject,\n              claim,\n              rationale: i === 0 ? result.answer : undefined,\n              severity: 'medium',\n              confidence: 0.6,\n              evidence_refs: [],\n              metadata: { question, turns: result.turnCount, finding_index: i },\n            }),\n          )\n        })\n      }\n      return out\n    },\n  }\n}\n\n// ── 2. MultiLayerVerifier → Analyst ─────────────────────────────────\n\nexport interface VerifierAdapterOpts<Env> {\n  id?: string\n  area?: string\n  verifier: MultiLayerVerifier<Env>\n  /**\n   * The verifier expects an `env` per run. Adapters take it from\n   * `AnalystRunInputs.custom[<id>]` via the registry's 'custom' routing.\n   */\n  options?: Omit<VerifyOptions<Env>, 'env'>\n}\n\nexport function createVerifierAdapter<Env>(opts: VerifierAdapterOpts<Env>): Analyst<Env> {\n  const id = opts.id ?? 'multi-layer-verifier'\n  const area = opts.area ?? 'verification'\n  return {\n    id,\n    description:\n      \"Runs a MultiLayerVerifier and lifts each layer's findings into the analyst envelope.\",\n    inputKind: 'custom',\n    cost: { kind: 'deterministic' },\n    version: `verifier-${ADAPTER_REV}`,\n    async analyze(env, ctx) {\n      const report = await opts.verifier.run({ env, ...opts.options })\n      const out: AnalystFinding[] = []\n      for (const layer of report.layers) {\n        for (const finding of layer.findings) {\n          out.push(liftLayerFinding(id, area, layer.layer, finding))\n        }\n        // Layer-level signal: a failed/error layer is itself a finding\n        // even if it didn't emit per-finding rows.\n        if (layer.status === 'fail' || layer.status === 'error' || layer.status === 'timeout') {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject: layer.layer,\n              claim: `layer \"${layer.layer}\" ${layer.status}: ${layer.reason ?? 'no reason given'}`,\n              severity:\n                layer.status === 'error' ? 'high' : layer.status === 'timeout' ? 'medium' : 'high',\n              confidence: 1,\n              evidence_refs: [],\n              metadata: {\n                layer_status: layer.status,\n                duration_ms: layer.durationMs,\n                score: layer.score,\n                diagnostics: layer.diagnostics,\n              },\n            }),\n          )\n        }\n      }\n      ctx.log?.('verifier complete', {\n        layers: report.layers.length,\n        blended: report.blendedScore,\n        all_pass: report.allPass,\n      })\n      return out\n    },\n  }\n}\n\nfunction liftLayerFinding(\n  analyst_id: string,\n  area: string,\n  layer: string,\n  f: LayerFinding,\n): AnalystFinding {\n  return makeFinding({\n    analyst_id,\n    area,\n    subject: f.layer ?? layer,\n    claim: f.message,\n    severity: liftSeverity(f.severity),\n    confidence: 0.85,\n    evidence_refs: f.evidence\n      ? [{ kind: 'artifact', uri: 'inline:evidence', excerpt: f.evidence }]\n      : [],\n    metadata: f.detail,\n  })\n}\n\n// ── 3. RunCritic → Analyst ──────────────────────────────────────────\n\nexport interface RunCriticAdapterOpts {\n  id?: string\n  area?: string\n  critic?: RunCritic\n  /** Optional threshold below which a dimension is reported as a finding. Default 0.5. */\n  threshold?: number\n}\n\nexport function createRunCriticAdapter(opts: RunCriticAdapterOpts = {}): Analyst<RunTrace> {\n  const id = opts.id ?? 'run-critic'\n  const area = opts.area ?? 'run-quality'\n  const critic = opts.critic ?? new RunCritic()\n  const threshold = opts.threshold ?? 0.5\n  return {\n    id,\n    description:\n      'Scores a single run across success / grounding / drift / tool-quality and surfaces below-threshold dimensions.',\n    inputKind: 'custom',\n    cost: { kind: 'deterministic' },\n    version: `run-critic-${ADAPTER_REV}`,\n    async analyze(trace) {\n      const score = critic.scoreTrace(trace)\n      const out: AnalystFinding[] = []\n      const dims: Array<[keyof typeof score, AnalystSeverity, string]> = [\n        ['success', 'critical', 'run did not complete successfully'],\n        ['goalProgress', 'high', 'goal progress is low'],\n        ['repoGroundedness', 'high', 'output is poorly grounded in the repository'],\n        ['toolUseQuality', 'medium', 'tool use quality is low'],\n        ['patchQuality', 'medium', 'no real patch/edit evidence'],\n        ['testReality', 'high', 'no real test/build evidence'],\n        ['finalGate', 'critical', 'final gate is blocking'],\n      ]\n      for (const [dim, sev, msg] of dims) {\n        const value = score[dim] as number\n        if (typeof value === 'number' && value < threshold) {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject: dim,\n              claim: msg,\n              rationale: `${dim}=${value.toFixed(2)} below threshold ${threshold}`,\n              severity: sev,\n              confidence: 1,\n              evidence_refs: [],\n              metadata: { dimension: dim, value, threshold, run_id: trace.run.runId },\n            }),\n          )\n        }\n      }\n      // Drift penalty is high → surface as a finding (inverse threshold).\n      if (score.driftPenalty > 1 - threshold) {\n        out.push(\n          makeFinding({\n            analyst_id: id,\n            area,\n            subject: 'drift',\n            claim: 'agent output drifted from repository signal',\n            rationale: `driftPenalty=${score.driftPenalty.toFixed(2)}`,\n            severity: 'medium',\n            confidence: 0.9,\n            evidence_refs: [],\n            metadata: { drift_penalty: score.driftPenalty, notes: score.notes },\n          }),\n        )\n      }\n      return out\n    },\n  }\n}\n\n// ── 4. JudgeFn → Analyst ────────────────────────────────────────────\n\nexport interface JudgeAdapterOpts {\n  id?: string\n  area?: string\n  judge: JudgeFn\n  /** TCloud handle the JudgeFn calls. */\n  tcloud: TCloud\n  /** Optional cost classification — most judges call an LLM. */\n  cost?: Analyst['cost']\n  /** Optional threshold below which a JudgeScore becomes a finding. Default 6 (on 0-10 scale). */\n  threshold?: number\n}\n\nexport function createJudgeAdapter(opts: JudgeAdapterOpts): Analyst<JudgeInput> {\n  const id = opts.id ?? 'judge'\n  const area = opts.area ?? 'judge'\n  const threshold = opts.threshold ?? 6\n  return {\n    id,\n    description:\n      'Wraps an agent-eval JudgeFn into an analyst; below-threshold dimensions surface as findings.',\n    inputKind: 'judge-input',\n    cost: opts.cost ?? { kind: 'llm' },\n    version: `judge-${ADAPTER_REV}`,\n    async analyze(input) {\n      const scores = await opts.judge(opts.tcloud, input)\n      return scores\n        .filter((s) => normalize10(s.score) < threshold)\n        .map((s) => liftJudgeScore(id, area, s))\n    },\n  }\n}\n\nfunction normalize10(s: number): number {\n  // JudgeScore convention is 0-10 but some judges emit 0-1. Coerce to 0-10.\n  return s <= 1 ? s * 10 : s\n}\n\nfunction liftJudgeScore(analyst_id: string, area: string, s: JudgeScore): AnalystFinding {\n  const score10 = normalize10(s.score)\n  const severity: AnalystSeverity =\n    score10 < 3 ? 'critical' : score10 < 5 ? 'high' : score10 < 7 ? 'medium' : 'low'\n  return makeFinding({\n    analyst_id,\n    area,\n    subject: s.dimension,\n    claim: `${s.judgeName}/${s.dimension} scored ${score10.toFixed(1)}/10`,\n    rationale: s.reasoning,\n    severity,\n    confidence: 0.8,\n    evidence_refs: s.evidence\n      ? [{ kind: 'artifact', uri: 'inline:evidence', excerpt: s.evidence }]\n      : [],\n    metadata: { judge_name: s.judgeName, dimension: s.dimension, score_10: score10 },\n  })\n}\n\n// ── 5. SemanticConceptJudge → Analyst ──────────────────────────────\n\nexport interface SemanticConceptJudgeAdapterOpts {\n  id?: string\n  area?: string\n  options?: SemanticConceptJudgeOptions\n}\n\nexport function createSemanticConceptJudgeAdapter(\n  opts: SemanticConceptJudgeAdapterOpts = {},\n): Analyst<SemanticConceptJudgeInput> {\n  const id = opts.id ?? 'semantic-concept-judge'\n  const area = opts.area ?? 'concept-coverage'\n  return {\n    id,\n    description:\n      'Runs the semantic-concept judge and surfaces missing / weak concepts as findings.',\n    inputKind: 'custom',\n    cost: { kind: 'llm', models: opts.options?.model ? [opts.options.model] : undefined },\n    version: `${SEMANTIC_CONCEPT_JUDGE_VERSION}-adapter-${ADAPTER_REV}`,\n    async analyze(input) {\n      const result = await runSemanticConceptJudge(input, opts.options)\n      if (!result.available) {\n        return [\n          makeFinding({\n            analyst_id: id,\n            area,\n            claim: 'semantic-concept judge unavailable',\n            rationale: result.error,\n            severity: 'info',\n            confidence: 1,\n            evidence_refs: [],\n            metadata: { reason: result.error },\n          }),\n        ]\n      }\n      const out: AnalystFinding[] = []\n      for (const f of result.findings) {\n        // Only surface gaps: missing concepts or low scores. Concepts at\n        // 7+/10 with present=true are not findings — they're successes.\n        if (f.present && f.score >= 7) continue\n        out.push(\n          makeFinding({\n            analyst_id: id,\n            area,\n            subject: f.concept,\n            claim: f.present\n              ? `concept \"${f.concept}\" is weak (${f.score}/10)`\n              : `concept \"${f.concept}\" is missing`,\n            rationale: f.evidence,\n            severity: liftSeverity(f.severity),\n            confidence: 0.85,\n            evidence_refs: [{ kind: 'artifact', uri: 'inline:evidence', excerpt: f.evidence }],\n            metadata: {\n              concept: f.concept,\n              present: f.present,\n              score_10: f.score,\n              cost_usd: result.costUsd ?? undefined,\n            },\n          }),\n        )\n      }\n      return out\n    },\n  }\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAuCA,IAAM,cAAc;AAIb,SAAS,aAAa,GAAmC;AAC9D,UAAQ,GAAG;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,EACX;AACF;AAwBO,SAAS,0BACd,MAC6B;AAC7B,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,OAAO,QAAQ,KAAK,QAAQ,CAAC,KAAK,KAAK,IAAI,OAAU;AAAA,IACnE,SAAS,iBAAiB,WAAW;AAAA,IACrC,MAAM,QAAQ,OAAO,KAAK;AACxB,YAAM,MAAwB,CAAC;AAC/B,iBAAW,YAAY,KAAK,WAAW;AACrC,YAAI,IAAI,QAAQ,QAAS;AACzB,cAAM,SAAS,MAAM;AAAA,UACnB,EAAE,SAAS;AAAA,UACX,EAAE,QAAQ,OAAO,IAAI,KAAK,IAAI,OAAO,KAAK,OAAO,GAAG,KAAK,MAAM;AAAA,QACjE;AACA,cAAM,UAAU,IAAI,MAAM,WAAW,SAAS,MAAM,GAAG,EAAE;AAIzD,YAAI,OAAO,SAAS,WAAW,GAAG;AAChC,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA;AAAA,cACA,OAAO,OAAO,OAAO,MAAM,GAAG,GAAG;AAAA,cACjC,WAAW,OAAO;AAAA,cAClB,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU;AAAA,gBACR,sBAAsB,OAAO;AAAA,gBAC7B,OAAO,OAAO;AAAA,cAChB;AAAA,YACF,CAAC;AAAA,UACH;AACA;AAAA,QACF;AACA,eAAO,SAAS,QAAQ,CAAC,OAAO,MAAM;AACpC,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA;AAAA,cACA;AAAA,cACA,WAAW,MAAM,IAAI,OAAO,SAAS;AAAA,cACrC,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU,EAAE,UAAU,OAAO,OAAO,WAAW,eAAe,EAAE;AAAA,YAClE,CAAC;AAAA,UACH;AAAA,QACF,CAAC;AAAA,MACH;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAeO,SAAS,sBAA2B,MAA8C;AACvF,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,gBAAgB;AAAA,IAC9B,SAAS,YAAY,WAAW;AAAA,IAChC,MAAM,QAAQ,KAAK,KAAK;AACtB,YAAM,SAAS,MAAM,KAAK,SAAS,IAAI,EAAE,KAAK,GAAG,KAAK,QAAQ,CAAC;AAC/D,YAAM,MAAwB,CAAC;AAC/B,iBAAW,SAAS,OAAO,QAAQ;AACjC,mBAAW,WAAW,MAAM,UAAU;AACpC,cAAI,KAAK,iBAAiB,IAAI,MAAM,MAAM,OAAO,OAAO,CAAC;AAAA,QAC3D;AAGA,YAAI,MAAM,WAAW,UAAU,MAAM,WAAW,WAAW,MAAM,WAAW,WAAW;AACrF,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA,SAAS,MAAM;AAAA,cACf,OAAO,UAAU,MAAM,KAAK,KAAK,MAAM,MAAM,KAAK,MAAM,UAAU,iBAAiB;AAAA,cACnF,UACE,MAAM,WAAW,UAAU,SAAS,MAAM,WAAW,YAAY,WAAW;AAAA,cAC9E,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU;AAAA,gBACR,cAAc,MAAM;AAAA,gBACpB,aAAa,MAAM;AAAA,gBACnB,OAAO,MAAM;AAAA,gBACb,aAAa,MAAM;AAAA,cACrB;AAAA,YACF,CAAC;AAAA,UACH;AAAA,QACF;AAAA,MACF;AACA,UAAI,MAAM,qBAAqB;AAAA,QAC7B,QAAQ,OAAO,OAAO;AAAA,QACtB,SAAS,OAAO;AAAA,QAChB,UAAU,OAAO;AAAA,MACnB,CAAC;AACD,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAEA,SAAS,iBACP,YACA,MACA,OACA,GACgB;AAChB,SAAO,YAAY;AAAA,IACjB;AAAA,IACA;AAAA,IACA,SAAS,EAAE,SAAS;AAAA,IACpB,OAAO,EAAE;AAAA,IACT,UAAU,aAAa,EAAE,QAAQ;AAAA,IACjC,YAAY;AAAA,IACZ,eAAe,EAAE,WACb,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC,IAClE,CAAC;AAAA,IACL,UAAU,EAAE;AAAA,EACd,CAAC;AACH;AAYO,SAAS,uBAAuB,OAA6B,CAAC,GAAsB;AACzF,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,SAAS,KAAK,UAAU,IAAI,UAAU;AAC5C,QAAM,YAAY,KAAK,aAAa;AACpC,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,gBAAgB;AAAA,IAC9B,SAAS,cAAc,WAAW;AAAA,IAClC,MAAM,QAAQ,OAAO;AACnB,YAAM,QAAQ,OAAO,WAAW,KAAK;AACrC,YAAM,MAAwB,CAAC;AAC/B,YAAM,OAA6D;AAAA,QACjE,CAAC,WAAW,YAAY,mCAAmC;AAAA,QAC3D,CAAC,gBAAgB,QAAQ,sBAAsB;AAAA,QAC/C,CAAC,oBAAoB,QAAQ,6CAA6C;AAAA,QAC1E,CAAC,kBAAkB,UAAU,yBAAyB;AAAA,QACtD,CAAC,gBAAgB,UAAU,6BAA6B;AAAA,QACxD,CAAC,eAAe,QAAQ,6BAA6B;AAAA,QACrD,CAAC,aAAa,YAAY,wBAAwB;AAAA,MACpD;AACA,iBAAW,CAAC,KAAK,KAAK,GAAG,KAAK,MAAM;AAClC,cAAM,QAAQ,MAAM,GAAG;AACvB,YAAI,OAAO,UAAU,YAAY,QAAQ,WAAW;AAClD,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA,SAAS;AAAA,cACT,OAAO;AAAA,cACP,WAAW,GAAG,GAAG,IAAI,MAAM,QAAQ,CAAC,CAAC,oBAAoB,SAAS;AAAA,cAClE,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU,EAAE,WAAW,KAAK,OAAO,WAAW,QAAQ,MAAM,IAAI,MAAM;AAAA,YACxE,CAAC;AAAA,UACH;AAAA,QACF;AAAA,MACF;AAEA,UAAI,MAAM,eAAe,IAAI,WAAW;AACtC,YAAI;AAAA,UACF,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,SAAS;AAAA,YACT,OAAO;AAAA,YACP,WAAW,gBAAgB,MAAM,aAAa,QAAQ,CAAC,CAAC;AAAA,YACxD,UAAU;AAAA,YACV,YAAY;AAAA,YACZ,eAAe,CAAC;AAAA,YAChB,UAAU,EAAE,eAAe,MAAM,cAAc,OAAO,MAAM,MAAM;AAAA,UACpE,CAAC;AAAA,QACH;AAAA,MACF;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAgBO,SAAS,mBAAmB,MAA6C;AAC9E,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,YAAY,KAAK,aAAa;AACpC,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,KAAK,QAAQ,EAAE,MAAM,MAAM;AAAA,IACjC,SAAS,SAAS,WAAW;AAAA,IAC7B,MAAM,QAAQ,OAAO;AACnB,YAAM,SAAS,MAAM,KAAK,MAAM,KAAK,QAAQ,KAAK;AAClD,aAAO,OACJ,OAAO,CAAC,MAAM,YAAY,EAAE,KAAK,IAAI,SAAS,EAC9C,IAAI,CAAC,MAAM,eAAe,IAAI,MAAM,CAAC,CAAC;AAAA,IAC3C;AAAA,EACF;AACF;AAEA,SAAS,YAAY,GAAmB;AAEtC,SAAO,KAAK,IAAI,IAAI,KAAK;AAC3B;AAEA,SAAS,eAAe,YAAoB,MAAc,GAA+B;AACvF,QAAM,UAAU,YAAY,EAAE,KAAK;AACnC,QAAM,WACJ,UAAU,IAAI,aAAa,UAAU,IAAI,SAAS,UAAU,IAAI,WAAW;AAC7E,SAAO,YAAY;AAAA,IACjB;AAAA,IACA;AAAA,IACA,SAAS,EAAE;AAAA,IACX,OAAO,GAAG,EAAE,SAAS,IAAI,EAAE,SAAS,WAAW,QAAQ,QAAQ,CAAC,CAAC;AAAA,IACjE,WAAW,EAAE;AAAA,IACb;AAAA,IACA,YAAY;AAAA,IACZ,eAAe,EAAE,WACb,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC,IAClE,CAAC;AAAA,IACL,UAAU,EAAE,YAAY,EAAE,WAAW,WAAW,EAAE,WAAW,UAAU,QAAQ;AAAA,EACjF,CAAC;AACH;AAUO,SAAS,kCACd,OAAwC,CAAC,GACL;AACpC,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,OAAO,QAAQ,KAAK,SAAS,QAAQ,CAAC,KAAK,QAAQ,KAAK,IAAI,OAAU;AAAA,IACpF,SAAS,GAAG,8BAA8B,YAAY,WAAW;AAAA,IACjE,MAAM,QAAQ,OAAO;AACnB,YAAM,SAAS,MAAM,wBAAwB,OAAO,KAAK,OAAO;AAChE,UAAI,CAAC,OAAO,WAAW;AACrB,eAAO;AAAA,UACL,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,OAAO;AAAA,YACP,WAAW,OAAO;AAAA,YAClB,UAAU;AAAA,YACV,YAAY;AAAA,YACZ,eAAe,CAAC;AAAA,YAChB,UAAU,EAAE,QAAQ,OAAO,MAAM;AAAA,UACnC,CAAC;AAAA,QACH;AAAA,MACF;AACA,YAAM,MAAwB,CAAC;AAC/B,iBAAW,KAAK,OAAO,UAAU;AAG/B,YAAI,EAAE,WAAW,EAAE,SAAS,EAAG;AAC/B,YAAI;AAAA,UACF,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,SAAS,EAAE;AAAA,YACX,OAAO,EAAE,UACL,YAAY,EAAE,OAAO,cAAc,EAAE,KAAK,SAC1C,YAAY,EAAE,OAAO;AAAA,YACzB,WAAW,EAAE;AAAA,YACb,UAAU,aAAa,EAAE,QAAQ;AAAA,YACjC,YAAY;AAAA,YACZ,eAAe,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC;AAAA,YACjF,UAAU;AAAA,cACR,SAAS,EAAE;AAAA,cACX,SAAS,EAAE;AAAA,cACX,UAAU,EAAE;AAAA,cACZ,UAAU,OAAO,WAAW;AAAA,YAC9B;AAAA,UACF,CAAC;AAAA,QACH;AAAA,MACF;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;","names":[]}
+{"version":3,"sources":["../../src/analyst/adapters.ts","../../src/analyst/steer-firewall.ts"],"sourcesContent":["/**\n * Adapter factories — lift each existing agent-eval primitive into the\n * Analyst contract without re-implementing it.\n *\n * Five primitives, five factories. Each one:\n *   - Builds an Analyst with a stable id (caller chooses; defaults\n *     given), a sensible default `inputKind`, a version derived from\n *     the wrapped primitive's version + an adapter revision, and an\n *     `analyze()` that calls the primitive and lifts its output to\n *     AnalystFinding[] using `makeFinding()`.\n *   - Maps severities: the existing `Severity` ('critical' | 'major' |\n *     'minor' | 'info') projects onto AnalystSeverity ('critical' |\n *     'high' | 'medium' | 'low' | 'info'); 'major' → 'high', 'minor' →\n *     'medium'. Domain analysts that want finer-grained mapping override.\n *\n * Adapters never own state. Calling the same factory twice with the\n * same primitive instance is safe.\n */\n\nimport type { AxAIService } from '@ax-llm/ax'\nimport type {\n  Finding as LayerFinding,\n  Severity as LayerSeverity,\n  MultiLayerVerifier,\n  VerifyOptions,\n} from '../multi-layer-verifier'\nimport { RunCritic, type RunTrace } from '../run-critic'\nimport {\n  runSemanticConceptJudge,\n  SEMANTIC_CONCEPT_JUDGE_VERSION,\n  type SemanticConceptJudgeInput,\n  type SemanticConceptJudgeOptions,\n} from '../semantic-concept-judge'\nimport { type AnalyzeTracesOptions, analyzeTraces } from '../trace-analyst/analyst'\nimport type { TraceAnalysisStore } from '../trace-analyst/store'\nimport type { JudgeFn, JudgeInput, JudgeScore, TCloud } from '../types'\nimport type { Analyst, AnalystFinding, AnalystSeverity } from './types'\nimport { makeFinding } from './types'\n\nconst ADAPTER_REV = '1'\n\n// ── Severity bridges ───────────────────────────────────────────────\n\nexport function liftSeverity(s: LayerSeverity): AnalystSeverity {\n  switch (s) {\n    case 'critical':\n      return 'critical'\n    case 'major':\n      return 'high'\n    case 'minor':\n      return 'medium'\n    case 'info':\n      return 'info'\n  }\n}\n\n// ── 1. analyzeTraces → Analyst ─────────────────────────────────────\n\nexport interface TraceAnalystAdapterOpts {\n  id?: string\n  area?: string\n  /** The natural-language question(s) put to the analyst. One finding per question. */\n  questions: string[]\n  /** Caller-provided AxAI service — same one trace-analyst.ts expects. */\n  ai: AxAIService\n  model?: string\n  /** Forwarded to analyzeTraces. */\n  extra?: Omit<AnalyzeTracesOptions, 'source' | 'ai' | 'model'>\n}\n\n/**\n * @deprecated Prefer `createTraceAnalystKind` + one of the failure /\n * improvement kinds from `./kinds`. This adapter wraps the legacy\n * `analyzeTraces` flow whose output is `findings:string[]` — every\n * bullet gets flat-defaulted severity `medium` / confidence `0.6`,\n * which loses the per-finding grading kinds provide via Ax structured\n * output + Zod validation. Kept for one minor while consumers migrate.\n */\nexport function createTraceAnalystAdapter(\n  opts: TraceAnalystAdapterOpts,\n): Analyst<TraceAnalysisStore> {\n  const id = opts.id ?? 'trace-analyst'\n  const area = opts.area ?? 'agent-reasoning'\n  return {\n    id,\n    description:\n      'Runs the agent-eval trace analyst over an OTLP trace store and lifts its bulleted findings.',\n    inputKind: 'trace-store',\n    cost: { kind: 'llm', models: opts.model ? [opts.model] : undefined },\n    version: `trace-analyst-${ADAPTER_REV}`,\n    async analyze(store, ctx) {\n      const out: AnalystFinding[] = []\n      for (const question of opts.questions) {\n        if (ctx.signal?.aborted) break\n        const result = await analyzeTraces(\n          { question },\n          { source: store, ai: opts.ai, model: opts.model, ...opts.extra },\n        )\n        const subject = ctx.tags?.subject ?? question.slice(0, 60)\n        // The responder produces a list of bullet strings. Each becomes\n        // one finding; the prose answer is attached as rationale on the\n        // first (so renderers that show only top-N still get context).\n        if (result.findings.length === 0) {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject,\n              claim: result.answer.slice(0, 200),\n              rationale: result.answer,\n              severity: 'info',\n              confidence: 0.5,\n              evidence_refs: [],\n              metadata: {\n                actor_prompt_version: result.actorPromptVersion,\n                turns: result.turnCount,\n              },\n            }),\n          )\n          continue\n        }\n        result.findings.forEach((claim, i) => {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject,\n              claim,\n              rationale: i === 0 ? result.answer : undefined,\n              severity: 'medium',\n              confidence: 0.6,\n              evidence_refs: [],\n              metadata: { question, turns: result.turnCount, finding_index: i },\n            }),\n          )\n        })\n      }\n      return out\n    },\n  }\n}\n\n// ── 2. MultiLayerVerifier → Analyst ─────────────────────────────────\n\nexport interface VerifierAdapterOpts<Env> {\n  id?: string\n  area?: string\n  verifier: MultiLayerVerifier<Env>\n  /**\n   * The verifier expects an `env` per run. Adapters take it from\n   * `AnalystRunInputs.custom[<id>]` via the registry's 'custom' routing.\n   */\n  options?: Omit<VerifyOptions<Env>, 'env'>\n}\n\nexport function createVerifierAdapter<Env>(opts: VerifierAdapterOpts<Env>): Analyst<Env> {\n  const id = opts.id ?? 'multi-layer-verifier'\n  const area = opts.area ?? 'verification'\n  return {\n    id,\n    description:\n      \"Runs a MultiLayerVerifier and lifts each layer's findings into the analyst envelope.\",\n    inputKind: 'custom',\n    cost: { kind: 'deterministic' },\n    version: `verifier-${ADAPTER_REV}`,\n    async analyze(env, ctx) {\n      const report = await opts.verifier.run({ env, ...opts.options })\n      const out: AnalystFinding[] = []\n      for (const layer of report.layers) {\n        for (const finding of layer.findings) {\n          out.push(liftLayerFinding(id, area, layer.layer, finding))\n        }\n        // Layer-level signal: a failed/error layer is itself a finding\n        // even if it didn't emit per-finding rows.\n        if (layer.status === 'fail' || layer.status === 'error' || layer.status === 'timeout') {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject: layer.layer,\n              claim: `layer \"${layer.layer}\" ${layer.status}: ${layer.reason ?? 'no reason given'}`,\n              severity:\n                layer.status === 'error' ? 'high' : layer.status === 'timeout' ? 'medium' : 'high',\n              confidence: 1,\n              evidence_refs: [],\n              metadata: {\n                layer_status: layer.status,\n                duration_ms: layer.durationMs,\n                score: layer.score,\n                diagnostics: layer.diagnostics,\n              },\n            }),\n          )\n        }\n      }\n      ctx.log?.('verifier complete', {\n        layers: report.layers.length,\n        blended: report.blendedScore,\n        all_pass: report.allPass,\n      })\n      return out\n    },\n  }\n}\n\nfunction liftLayerFinding(\n  analyst_id: string,\n  area: string,\n  layer: string,\n  f: LayerFinding,\n): AnalystFinding {\n  return makeFinding({\n    analyst_id,\n    area,\n    subject: f.layer ?? layer,\n    claim: f.message,\n    severity: liftSeverity(f.severity),\n    confidence: 0.85,\n    evidence_refs: f.evidence\n      ? [{ kind: 'artifact', uri: 'inline:evidence', excerpt: f.evidence }]\n      : [],\n    metadata: f.detail,\n  })\n}\n\n// ── 3. RunCritic → Analyst ──────────────────────────────────────────\n\nexport interface RunCriticAdapterOpts {\n  id?: string\n  area?: string\n  critic?: RunCritic\n  /** Optional threshold below which a dimension is reported as a finding. Default 0.5. */\n  threshold?: number\n}\n\nexport function createRunCriticAdapter(opts: RunCriticAdapterOpts = {}): Analyst<RunTrace> {\n  const id = opts.id ?? 'run-critic'\n  const area = opts.area ?? 'run-quality'\n  const critic = opts.critic ?? new RunCritic()\n  const threshold = opts.threshold ?? 0.5\n  return {\n    id,\n    description:\n      'Scores a single run across success / grounding / drift / tool-quality and surfaces below-threshold dimensions.',\n    inputKind: 'custom',\n    cost: { kind: 'deterministic' },\n    version: `run-critic-${ADAPTER_REV}`,\n    async analyze(trace) {\n      const score = critic.scoreTrace(trace)\n      const out: AnalystFinding[] = []\n      const dims: Array<[keyof typeof score, AnalystSeverity, string]> = [\n        ['success', 'critical', 'run did not complete successfully'],\n        ['goalProgress', 'high', 'goal progress is low'],\n        ['repoGroundedness', 'high', 'output is poorly grounded in the repository'],\n        ['toolUseQuality', 'medium', 'tool use quality is low'],\n        ['patchQuality', 'medium', 'no real patch/edit evidence'],\n        ['testReality', 'high', 'no real test/build evidence'],\n        ['finalGate', 'critical', 'final gate is blocking'],\n      ]\n      for (const [dim, sev, msg] of dims) {\n        const value = score[dim] as number\n        if (typeof value === 'number' && value < threshold) {\n          out.push(\n            makeFinding({\n              analyst_id: id,\n              area,\n              subject: dim,\n              claim: msg,\n              rationale: `${dim}=${value.toFixed(2)} below threshold ${threshold}`,\n              severity: sev,\n              confidence: 1,\n              evidence_refs: [],\n              metadata: { dimension: dim, value, threshold, run_id: trace.run.runId },\n            }),\n          )\n        }\n      }\n      // Drift penalty is high → surface as a finding (inverse threshold).\n      if (score.driftPenalty > 1 - threshold) {\n        out.push(\n          makeFinding({\n            analyst_id: id,\n            area,\n            subject: 'drift',\n            claim: 'agent output drifted from repository signal',\n            rationale: `driftPenalty=${score.driftPenalty.toFixed(2)}`,\n            severity: 'medium',\n            confidence: 0.9,\n            evidence_refs: [],\n            metadata: { drift_penalty: score.driftPenalty, notes: score.notes },\n          }),\n        )\n      }\n      return out\n    },\n  }\n}\n\n// ── 4. JudgeFn → Analyst ────────────────────────────────────────────\n\nexport interface JudgeAdapterOpts {\n  id?: string\n  area?: string\n  judge: JudgeFn\n  /** TCloud handle the JudgeFn calls. */\n  tcloud: TCloud\n  /** Optional cost classification — most judges call an LLM. */\n  cost?: Analyst['cost']\n  /** Optional threshold below which a JudgeScore becomes a finding. Default 6 (on 0-10 scale). */\n  threshold?: number\n}\n\nexport function createJudgeAdapter(opts: JudgeAdapterOpts): Analyst<JudgeInput> {\n  const id = opts.id ?? 'judge'\n  const area = opts.area ?? 'judge'\n  const threshold = opts.threshold ?? 6\n  return {\n    id,\n    description:\n      'Wraps an agent-eval JudgeFn into an analyst; below-threshold dimensions surface as findings.',\n    inputKind: 'judge-input',\n    cost: opts.cost ?? { kind: 'llm' },\n    version: `judge-${ADAPTER_REV}`,\n    async analyze(input) {\n      const scores = await opts.judge(opts.tcloud, input)\n      return scores\n        .filter((s) => normalize10(s.score) < threshold)\n        .map((s) => liftJudgeScore(id, area, s))\n    },\n  }\n}\n\nfunction normalize10(s: number): number {\n  // JudgeScore convention is 0-10 but some judges emit 0-1. Coerce to 0-10.\n  return s <= 1 ? s * 10 : s\n}\n\nfunction liftJudgeScore(analyst_id: string, area: string, s: JudgeScore): AnalystFinding {\n  const score10 = normalize10(s.score)\n  const severity: AnalystSeverity =\n    score10 < 3 ? 'critical' : score10 < 5 ? 'high' : score10 < 7 ? 'medium' : 'low'\n  return makeFinding({\n    analyst_id,\n    area,\n    subject: s.dimension,\n    claim: `${s.judgeName}/${s.dimension} scored ${score10.toFixed(1)}/10`,\n    rationale: s.reasoning,\n    severity,\n    confidence: 0.8,\n    evidence_refs: s.evidence\n      ? [{ kind: 'artifact', uri: 'inline:evidence', excerpt: s.evidence }]\n      : [],\n    // Provenance: this finding IS a judge verdict (an acceptance score), not an\n    // observation of behavior. The steer firewall (assertNoJudgeVerdict) rejects\n    // it from steering — even when it cites an artifact above — because letting a\n    // verdict steer the next attempt is the held-out judge leaking into the loop.\n    derived_from_judge: true,\n    metadata: { judge_name: s.judgeName, dimension: s.dimension, score_10: score10 },\n  })\n}\n\n// ── 5. SemanticConceptJudge → Analyst ──────────────────────────────\n\nexport interface SemanticConceptJudgeAdapterOpts {\n  id?: string\n  area?: string\n  options?: SemanticConceptJudgeOptions\n}\n\nexport function createSemanticConceptJudgeAdapter(\n  opts: SemanticConceptJudgeAdapterOpts = {},\n): Analyst<SemanticConceptJudgeInput> {\n  const id = opts.id ?? 'semantic-concept-judge'\n  const area = opts.area ?? 'concept-coverage'\n  return {\n    id,\n    description:\n      'Runs the semantic-concept judge and surfaces missing / weak concepts as findings.',\n    inputKind: 'custom',\n    cost: { kind: 'llm', models: opts.options?.model ? [opts.options.model] : undefined },\n    version: `${SEMANTIC_CONCEPT_JUDGE_VERSION}-adapter-${ADAPTER_REV}`,\n    async analyze(input) {\n      const result = await runSemanticConceptJudge(input, opts.options)\n      if (!result.available) {\n        return [\n          makeFinding({\n            analyst_id: id,\n            area,\n            claim: 'semantic-concept judge unavailable',\n            rationale: result.error,\n            severity: 'info',\n            confidence: 1,\n            evidence_refs: [],\n            metadata: { reason: result.error },\n          }),\n        ]\n      }\n      const out: AnalystFinding[] = []\n      for (const f of result.findings) {\n        // Only surface gaps: missing concepts or low scores. Concepts at\n        // 7+/10 with present=true are not findings — they're successes.\n        if (f.present && f.score >= 7) continue\n        out.push(\n          makeFinding({\n            analyst_id: id,\n            area,\n            subject: f.concept,\n            claim: f.present\n              ? `concept \"${f.concept}\" is weak (${f.score}/10)`\n              : `concept \"${f.concept}\" is missing`,\n            rationale: f.evidence,\n            severity: liftSeverity(f.severity),\n            confidence: 0.85,\n            evidence_refs: [{ kind: 'artifact', uri: 'inline:evidence', excerpt: f.evidence }],\n            metadata: {\n              concept: f.concept,\n              present: f.present,\n              score_10: f.score,\n              cost_usd: result.costUsd ?? undefined,\n            },\n          }),\n        )\n      }\n      return out\n    },\n  }\n}\n","// The realness-oracle firewall (docs/learning-flywheel.md, \"The steer is f(trace)\").\n//\n// A realness/authenticity signal has TWO legitimate roles that must stay\n// separated by a firewall:\n//   (a) anchor judge J — write-only: scores the chosen output, gates promotion,\n//       NEVER seen by the worker/optimizer mid-run (else the loop games it).\n//   (b) steer f(trace) — an analyst observes the agent's OWN behavior in the\n//       trace (\"imported a stub\", \"used a non-crypto PRNG where encryption was\n//       required\") and steers the next attempt. Legitimate, because it is derived\n//       from OBSERVABLE BEHAVIOR, not from J's held-out verdict.\n//\n// The correct discriminator is PROVENANCE, not evidence presence. A judge verdict\n// lifted into a finding (createJudgeAdapter → liftJudgeScore) is a verdict even\n// when it cites an artifact; an evidence-less trace-analyst bullet is an\n// observation even though it cites nothing. So the firewall keys on\n// `AnalystFinding.derived_from_judge` (set at the judge lift site), NOT on whether\n// evidence_refs is populated. The instant a verdict steers the next attempt it is\n// a back-channel for J and the loop Goodharts realness exactly as it would\n// Goodhart pass-rate.\n\nimport type { AnalystFinding, EvidenceRef } from './types'\n\n/** Evidence grounded in the agent's OWN execution: OTLP trace elements\n *  (`span`/`event`) or the artifact it produced (`artifact`). */\nconst OBSERVABLE_KINDS: ReadonlySet<EvidenceRef['kind']> = new Set<EvidenceRef['kind']>([\n  'span',\n  'event',\n  'artifact',\n])\n\n/** DESCRIPTIVE predicate: does the finding cite at least one observable\n *  (span/event/artifact) evidence ref. Useful for ranking evidence quality or\n *  rendering — it is NOT the steer gate. Evidence presence is the WRONG\n *  discriminator for steering: a legitimate trace-analyst observation may cite\n *  nothing (it would be wrongly rejected), and a judge verdict may cite an\n *  artifact (it would be wrongly admitted). Use `assertNoJudgeVerdict` to gate\n *  steering; use this only where \"is this grounded in observable evidence\" is the\n *  literal question. */\nexport function isTraceObservable(finding: AnalystFinding): boolean {\n  return finding.evidence_refs.some((ref) => OBSERVABLE_KINDS.has(ref.kind))\n}\n\n/** True iff the finding is a JUDGE VERDICT (an acceptance score lifted into a\n *  finding), identified by provenance set at the lift site — independent of\n *  whatever evidence it cites. */\nexport function isJudgeVerdict(finding: AnalystFinding): boolean {\n  return finding.derived_from_judge === true\n}\n\n/**\n * THE steer firewall. Fail-loud guard for any path that admits analyst findings\n * as STEERING input (the `f(trace)` role): rejects — naming the offenders — any\n * finding whose provenance is a judge verdict, rather than let `J` leak into the\n * loop. Returns the findings unchanged for chaining.\n *\n * Call this at the chokepoint where a detector that ALSO scores/gates has its\n * findings turned into a steer (the judge-and-steer dual-role case). It keys on\n * provenance, so it correctly admits evidence-less trace-analyst observations and\n * correctly rejects an artifact-citing judge verdict — the cases an evidence\n * check gets backwards.\n *\n * It is necessary, not sufficient: it stops PROVENANCE-tagged verdicts. A judge\n * whose output is laundered through a hand-built finding with no provenance flag\n * is out of its reach — provenance must be honestly set at every judge→finding\n * lift (today: createJudgeAdapter). That is why the integrity rule lives at the\n * lift site, and why ProposeContext.judgeScores?: never is the complementary\n * compile-time tripwire on the obvious direct channel.\n */\nexport function assertNoJudgeVerdict(\n  findings: ReadonlyArray<AnalystFinding>,\n  context = 'steer',\n): ReadonlyArray<AnalystFinding> {\n  const leaks = findings.filter(isJudgeVerdict)\n  if (leaks.length > 0) {\n    throw new Error(\n      `${context}: a judge verdict cannot be admitted as steering input — that is the ` +\n        `held-out judge leaking into the loop. Offending judge-derived findings: [${leaks\n          .map((f) => f.finding_id)\n          .join(', ')}]. Steering consumes observations of behavior, never acceptance verdicts.`,\n    )\n  }\n  return findings\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAuCA,IAAM,cAAc;AAIb,SAAS,aAAa,GAAmC;AAC9D,UAAQ,GAAG;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,IACT,KAAK;AACH,aAAO;AAAA,EACX;AACF;AAwBO,SAAS,0BACd,MAC6B;AAC7B,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,OAAO,QAAQ,KAAK,QAAQ,CAAC,KAAK,KAAK,IAAI,OAAU;AAAA,IACnE,SAAS,iBAAiB,WAAW;AAAA,IACrC,MAAM,QAAQ,OAAO,KAAK;AACxB,YAAM,MAAwB,CAAC;AAC/B,iBAAW,YAAY,KAAK,WAAW;AACrC,YAAI,IAAI,QAAQ,QAAS;AACzB,cAAM,SAAS,MAAM;AAAA,UACnB,EAAE,SAAS;AAAA,UACX,EAAE,QAAQ,OAAO,IAAI,KAAK,IAAI,OAAO,KAAK,OAAO,GAAG,KAAK,MAAM;AAAA,QACjE;AACA,cAAM,UAAU,IAAI,MAAM,WAAW,SAAS,MAAM,GAAG,EAAE;AAIzD,YAAI,OAAO,SAAS,WAAW,GAAG;AAChC,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA;AAAA,cACA,OAAO,OAAO,OAAO,MAAM,GAAG,GAAG;AAAA,cACjC,WAAW,OAAO;AAAA,cAClB,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU;AAAA,gBACR,sBAAsB,OAAO;AAAA,gBAC7B,OAAO,OAAO;AAAA,cAChB;AAAA,YACF,CAAC;AAAA,UACH;AACA;AAAA,QACF;AACA,eAAO,SAAS,QAAQ,CAAC,OAAO,MAAM;AACpC,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA;AAAA,cACA;AAAA,cACA,WAAW,MAAM,IAAI,OAAO,SAAS;AAAA,cACrC,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU,EAAE,UAAU,OAAO,OAAO,WAAW,eAAe,EAAE;AAAA,YAClE,CAAC;AAAA,UACH;AAAA,QACF,CAAC;AAAA,MACH;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAeO,SAAS,sBAA2B,MAA8C;AACvF,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,gBAAgB;AAAA,IAC9B,SAAS,YAAY,WAAW;AAAA,IAChC,MAAM,QAAQ,KAAK,KAAK;AACtB,YAAM,SAAS,MAAM,KAAK,SAAS,IAAI,EAAE,KAAK,GAAG,KAAK,QAAQ,CAAC;AAC/D,YAAM,MAAwB,CAAC;AAC/B,iBAAW,SAAS,OAAO,QAAQ;AACjC,mBAAW,WAAW,MAAM,UAAU;AACpC,cAAI,KAAK,iBAAiB,IAAI,MAAM,MAAM,OAAO,OAAO,CAAC;AAAA,QAC3D;AAGA,YAAI,MAAM,WAAW,UAAU,MAAM,WAAW,WAAW,MAAM,WAAW,WAAW;AACrF,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA,SAAS,MAAM;AAAA,cACf,OAAO,UAAU,MAAM,KAAK,KAAK,MAAM,MAAM,KAAK,MAAM,UAAU,iBAAiB;AAAA,cACnF,UACE,MAAM,WAAW,UAAU,SAAS,MAAM,WAAW,YAAY,WAAW;AAAA,cAC9E,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU;AAAA,gBACR,cAAc,MAAM;AAAA,gBACpB,aAAa,MAAM;AAAA,gBACnB,OAAO,MAAM;AAAA,gBACb,aAAa,MAAM;AAAA,cACrB;AAAA,YACF,CAAC;AAAA,UACH;AAAA,QACF;AAAA,MACF;AACA,UAAI,MAAM,qBAAqB;AAAA,QAC7B,QAAQ,OAAO,OAAO;AAAA,QACtB,SAAS,OAAO;AAAA,QAChB,UAAU,OAAO;AAAA,MACnB,CAAC;AACD,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAEA,SAAS,iBACP,YACA,MACA,OACA,GACgB;AAChB,SAAO,YAAY;AAAA,IACjB;AAAA,IACA;AAAA,IACA,SAAS,EAAE,SAAS;AAAA,IACpB,OAAO,EAAE;AAAA,IACT,UAAU,aAAa,EAAE,QAAQ;AAAA,IACjC,YAAY;AAAA,IACZ,eAAe,EAAE,WACb,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC,IAClE,CAAC;AAAA,IACL,UAAU,EAAE;AAAA,EACd,CAAC;AACH;AAYO,SAAS,uBAAuB,OAA6B,CAAC,GAAsB;AACzF,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,SAAS,KAAK,UAAU,IAAI,UAAU;AAC5C,QAAM,YAAY,KAAK,aAAa;AACpC,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,gBAAgB;AAAA,IAC9B,SAAS,cAAc,WAAW;AAAA,IAClC,MAAM,QAAQ,OAAO;AACnB,YAAM,QAAQ,OAAO,WAAW,KAAK;AACrC,YAAM,MAAwB,CAAC;AAC/B,YAAM,OAA6D;AAAA,QACjE,CAAC,WAAW,YAAY,mCAAmC;AAAA,QAC3D,CAAC,gBAAgB,QAAQ,sBAAsB;AAAA,QAC/C,CAAC,oBAAoB,QAAQ,6CAA6C;AAAA,QAC1E,CAAC,kBAAkB,UAAU,yBAAyB;AAAA,QACtD,CAAC,gBAAgB,UAAU,6BAA6B;AAAA,QACxD,CAAC,eAAe,QAAQ,6BAA6B;AAAA,QACrD,CAAC,aAAa,YAAY,wBAAwB;AAAA,MACpD;AACA,iBAAW,CAAC,KAAK,KAAK,GAAG,KAAK,MAAM;AAClC,cAAM,QAAQ,MAAM,GAAG;AACvB,YAAI,OAAO,UAAU,YAAY,QAAQ,WAAW;AAClD,cAAI;AAAA,YACF,YAAY;AAAA,cACV,YAAY;AAAA,cACZ;AAAA,cACA,SAAS;AAAA,cACT,OAAO;AAAA,cACP,WAAW,GAAG,GAAG,IAAI,MAAM,QAAQ,CAAC,CAAC,oBAAoB,SAAS;AAAA,cAClE,UAAU;AAAA,cACV,YAAY;AAAA,cACZ,eAAe,CAAC;AAAA,cAChB,UAAU,EAAE,WAAW,KAAK,OAAO,WAAW,QAAQ,MAAM,IAAI,MAAM;AAAA,YACxE,CAAC;AAAA,UACH;AAAA,QACF;AAAA,MACF;AAEA,UAAI,MAAM,eAAe,IAAI,WAAW;AACtC,YAAI;AAAA,UACF,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,SAAS;AAAA,YACT,OAAO;AAAA,YACP,WAAW,gBAAgB,MAAM,aAAa,QAAQ,CAAC,CAAC;AAAA,YACxD,UAAU;AAAA,YACV,YAAY;AAAA,YACZ,eAAe,CAAC;AAAA,YAChB,UAAU,EAAE,eAAe,MAAM,cAAc,OAAO,MAAM,MAAM;AAAA,UACpE,CAAC;AAAA,QACH;AAAA,MACF;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;AAgBO,SAAS,mBAAmB,MAA6C;AAC9E,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,YAAY,KAAK,aAAa;AACpC,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,KAAK,QAAQ,EAAE,MAAM,MAAM;AAAA,IACjC,SAAS,SAAS,WAAW;AAAA,IAC7B,MAAM,QAAQ,OAAO;AACnB,YAAM,SAAS,MAAM,KAAK,MAAM,KAAK,QAAQ,KAAK;AAClD,aAAO,OACJ,OAAO,CAAC,MAAM,YAAY,EAAE,KAAK,IAAI,SAAS,EAC9C,IAAI,CAAC,MAAM,eAAe,IAAI,MAAM,CAAC,CAAC;AAAA,IAC3C;AAAA,EACF;AACF;AAEA,SAAS,YAAY,GAAmB;AAEtC,SAAO,KAAK,IAAI,IAAI,KAAK;AAC3B;AAEA,SAAS,eAAe,YAAoB,MAAc,GAA+B;AACvF,QAAM,UAAU,YAAY,EAAE,KAAK;AACnC,QAAM,WACJ,UAAU,IAAI,aAAa,UAAU,IAAI,SAAS,UAAU,IAAI,WAAW;AAC7E,SAAO,YAAY;AAAA,IACjB;AAAA,IACA;AAAA,IACA,SAAS,EAAE;AAAA,IACX,OAAO,GAAG,EAAE,SAAS,IAAI,EAAE,SAAS,WAAW,QAAQ,QAAQ,CAAC,CAAC;AAAA,IACjE,WAAW,EAAE;AAAA,IACb;AAAA,IACA,YAAY;AAAA,IACZ,eAAe,EAAE,WACb,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC,IAClE,CAAC;AAAA;AAAA;AAAA;AAAA;AAAA,IAKL,oBAAoB;AAAA,IACpB,UAAU,EAAE,YAAY,EAAE,WAAW,WAAW,EAAE,WAAW,UAAU,QAAQ;AAAA,EACjF,CAAC;AACH;AAUO,SAAS,kCACd,OAAwC,CAAC,GACL;AACpC,QAAM,KAAK,KAAK,MAAM;AACtB,QAAM,OAAO,KAAK,QAAQ;AAC1B,SAAO;AAAA,IACL;AAAA,IACA,aACE;AAAA,IACF,WAAW;AAAA,IACX,MAAM,EAAE,MAAM,OAAO,QAAQ,KAAK,SAAS,QAAQ,CAAC,KAAK,QAAQ,KAAK,IAAI,OAAU;AAAA,IACpF,SAAS,GAAG,8BAA8B,YAAY,WAAW;AAAA,IACjE,MAAM,QAAQ,OAAO;AACnB,YAAM,SAAS,MAAM,wBAAwB,OAAO,KAAK,OAAO;AAChE,UAAI,CAAC,OAAO,WAAW;AACrB,eAAO;AAAA,UACL,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,OAAO;AAAA,YACP,WAAW,OAAO;AAAA,YAClB,UAAU;AAAA,YACV,YAAY;AAAA,YACZ,eAAe,CAAC;AAAA,YAChB,UAAU,EAAE,QAAQ,OAAO,MAAM;AAAA,UACnC,CAAC;AAAA,QACH;AAAA,MACF;AACA,YAAM,MAAwB,CAAC;AAC/B,iBAAW,KAAK,OAAO,UAAU;AAG/B,YAAI,EAAE,WAAW,EAAE,SAAS,EAAG;AAC/B,YAAI;AAAA,UACF,YAAY;AAAA,YACV,YAAY;AAAA,YACZ;AAAA,YACA,SAAS,EAAE;AAAA,YACX,OAAO,EAAE,UACL,YAAY,EAAE,OAAO,cAAc,EAAE,KAAK,SAC1C,YAAY,EAAE,OAAO;AAAA,YACzB,WAAW,EAAE;AAAA,YACb,UAAU,aAAa,EAAE,QAAQ;AAAA,YACjC,YAAY;AAAA,YACZ,eAAe,CAAC,EAAE,MAAM,YAAY,KAAK,mBAAmB,SAAS,EAAE,SAAS,CAAC;AAAA,YACjF,UAAU;AAAA,cACR,SAAS,EAAE;AAAA,cACX,SAAS,EAAE;AAAA,cACX,UAAU,EAAE;AAAA,cACZ,UAAU,OAAO,WAAW;AAAA,YAC9B;AAAA,UACF,CAAC;AAAA,QACH;AAAA,MACF;AACA,aAAO;AAAA,IACT;AAAA,EACF;AACF;;;AClZA,IAAM,mBAAqD,oBAAI,IAAyB;AAAA,EACtF;AAAA,EACA;AAAA,EACA;AACF,CAAC;AAUM,SAAS,kBAAkB,SAAkC;AAClE,SAAO,QAAQ,cAAc,KAAK,CAAC,QAAQ,iBAAiB,IAAI,IAAI,IAAI,CAAC;AAC3E;AAKO,SAAS,eAAe,SAAkC;AAC/D,SAAO,QAAQ,uBAAuB;AACxC;AAqBO,SAAS,qBACd,UACA,UAAU,SACqB;AAC/B,QAAM,QAAQ,SAAS,OAAO,cAAc;AAC5C,MAAI,MAAM,SAAS,GAAG;AACpB,UAAM,IAAI;AAAA,MACR,GAAG,OAAO,sJACoE,MACzE,IAAI,CAAC,MAAM,EAAE,UAAU,EACvB,KAAK,IAAI,CAAC;AAAA,IACjB;AAAA,EACF;AACA,SAAO;AACT;","names":[]}

package/dist/authenticity/index.d.ts

@@ -63,6 +63,13 @@ interface AuthenticityResult {
     usesRealImpl: boolean;
     realInfra: boolean;
     wired: boolean;
+    /** The required artifact is actually referenced/imported by other (non-artifact)
+     *  files — i.e. wired into the rest of the system, not dead code. Domain-agnostic:
+     *  a deliverable nothing else uses is suspect in any vertical. */
+    artifactReferenced: boolean;
+    /** Convenience: the artifact is connected to the running system, via either the
+     *  domain wiring signal OR a structural reference. */
+    artifactWired: boolean;
     fakeShim: boolean;
     /** mock/stub markers per 1000 LOC, capped at 100. */
     mockDensity: number;
@@ -81,6 +88,7 @@ interface RealnessGate {
 declare function gateRealness(r: AuthenticityResult, opts?: {
     floor?: number;
     requireArtifact?: boolean;
+    requireArtifactWired?: boolean;
 }): RealnessGate;
 interface AuthenticityNuance {
     /** 0 (nothing mocked) … 100 (entirely mocked). */
@@ -104,5 +112,50 @@ declare function scoreAuthenticityNuance(files: readonly ProducedFile[], complet
     intent?: string;
     prioritize?: RegExp;
 }): Promise<AuthenticityNuance>;
+interface RealnessJudgment {
+    /** 0 (facade/simulator) … 100 (real implementation on the intended infra). */
+    isReal: number;
+    rationale: string;
+}
+/**
+ * Ask an LLM to rate realness DIRECTLY on a 0-100 scale — the axis that matched
+ * human blind-labels in validation (F1 0.80→0.88 on the gray band; a fakePct/
+ * hollowness proxy over-penalized "real core + stubbed periphery" partials, and a
+ * weak judge model over-flagged — use a strong one). Domain-agnostic skeleton; the
+ * consumer supplies `intent` (what the deliverable should be) and `rubric` (domain
+ * specifics of real-vs-fake). Fail-closed: a bad response reads as fully fake.
+ */
+declare function judgeRealnessLlm(files: readonly ProducedFile[], complete: CompleteFn, opts?: {
+    intent?: string;
+    rubric?: string;
+    prioritize?: RegExp;
+}): Promise<RealnessJudgment>;
+type RealnessBand = 'clean-real' | 'clean-fake' | 'gray';
+interface BlendedRealness extends AuthenticityResult {
+    /** Final realness after (only-when-needed) LLM adjudication, 0…100. */
+    blendedRealness: number;
+    band: RealnessBand;
+    /** True iff the LLM judge was actually consulted (gray band only). */
+    consultedLlm: boolean;
+    /** Present iff the LLM was consulted. */
+    judgment?: RealnessJudgment;
+}
+/**
+ * Score realness using the cheapest sufficient signal: trust the deterministic
+ * scorer on the CLEAN extremes (obvious fakes / obviously-real-and-wired), and only
+ * spend an LLM call on the GRAY band — cells that look real structurally but carry
+ * fakeness markers (a fake shim, an unwired/dead artifact, high mock density) or land
+ * mid-range. This caps LLM cost at the fraction of cells static analysis can't
+ * resolve, which matters at multi-vertical / multi-partner scale.
+ *
+ * Domain-agnostic: the gray-band TRIGGER is structural; the LLM judges via the
+ * consumer-supplied `intent`. Fail-closed (a bad LLM response reads as fully fake).
+ */
+declare function scoreRealnessBlended(files: readonly ProducedFile[], signals: AuthenticitySignals, complete: CompleteFn, opts?: {
+    intent?: string;
+    rubric?: string;
+    grayBand?: [number, number];
+    mockGrayThreshold?: number;
+}): Promise<BlendedRealness>;
 
-export { type AuthenticityNuance, type AuthenticityResult, type AuthenticitySignals, type CompleteFn, type ProducedFile, type RealnessGate, gateRealness, scoreAuthenticity, scoreAuthenticityNuance };
+export { type AuthenticityNuance, type AuthenticityResult, type AuthenticitySignals, type BlendedRealness, type CompleteFn, type ProducedFile, type RealnessBand, type RealnessGate, type RealnessJudgment, gateRealness, judgeRealnessLlm, scoreAuthenticity, scoreAuthenticityNuance, scoreRealnessBlended };