@tangle-network/agent-eval 0.106.3 → 0.108.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +2 -2
- package/dist/adapters/http.d.ts +1 -1
- package/dist/adapters/langchain.d.ts +1 -1
- package/dist/adapters/otel.d.ts +1 -1
- package/dist/benchmarks/index.d.ts +3 -1
- package/dist/benchmarks/index.js +49 -3
- package/dist/campaign/index.d.ts +75 -8
- package/dist/campaign/index.js +74 -3321
- package/dist/campaign/index.js.map +1 -1
- package/dist/chunk-4VLZEPJ3.js +4306 -0
- package/dist/chunk-4VLZEPJ3.js.map +1 -0
- package/dist/{chunk-3E5KUXYZ.js → chunk-OVPVM4JC.js} +1064 -643
- package/dist/chunk-OVPVM4JC.js.map +1 -0
- package/dist/chunk-T6W5ADLG.js +766 -0
- package/dist/chunk-T6W5ADLG.js.map +1 -0
- package/dist/contract/index.d.ts +16 -6
- package/dist/contract/index.js +10 -11
- package/dist/contract/index.js.map +1 -1
- package/dist/{gepa-DQ3ruj18.d.ts → gepa-B3x5Ulcv.d.ts} +11 -40
- package/dist/hosted/index.d.ts +1 -1
- package/dist/index-pPtfoIJO.d.ts +423 -0
- package/dist/index.d.ts +6 -5
- package/dist/index.js +8 -7
- package/dist/index.js.map +1 -1
- package/dist/multishot/index.d.ts +1 -1
- package/dist/openapi.json +1 -1
- package/dist/{pre-registration-BkkTmtQG.d.ts → pre-registration-BUhVPzE7.d.ts} +1 -1
- package/dist/{provenance-Bsyjc67Z.d.ts → provenance-DdDhf6cg.d.ts} +3 -2
- package/dist/rl.d.ts +1 -1
- package/dist/rl.js +6 -6
- package/dist/storage-Dw_f7WMt.d.ts +39 -0
- package/dist/{types-Ctf7XIAL.d.ts → types-BdIv5dvA.d.ts} +11 -1
- package/docs/improvement-glossary.md +5 -1
- package/package.json +1 -1
- package/dist/chunk-2HLM4SJJ.js +0 -861
- package/dist/chunk-2HLM4SJJ.js.map +0 -1
- package/dist/chunk-3E5KUXYZ.js.map +0 -1
- package/dist/chunk-6QDKWHLS.js +0 -223
- package/dist/chunk-6QDKWHLS.js.map +0 -1
- package/dist/chunk-J22CKVXN.js +0 -421
- package/dist/chunk-J22CKVXN.js.map +0 -1
- package/dist/index-C2CC_dry.d.ts +0 -159
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,27 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
|
|
|
4
4
|
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
+
## [0.108.0] — 2026-07-08 — placebo control reaches the facades
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **`neutralize` passthrough on `selfImprove`.** 0.107.0 wired the footprint-matched placebo arm at `runImprovementLoop`; the public `selfImprove` facade did not forward it, so the placebo gate was unreachable from the one-call entry point. `selfImprove({ ..., neutralize })` now scores the third placebo arm and exposes `ctx.neutralizedJudgeScores` to the gate — compose `neutralizationGate` into `gate` to act on it.
|
|
12
|
+
|
|
13
|
+
Additive (one optional field); no consumer bump required.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## [0.107.0] — 2026-07-07 — footprint-matched placebo promotion gate
|
|
18
|
+
|
|
19
|
+
### Added
|
|
20
|
+
|
|
21
|
+
- **`neutralizationGate` + `neutralizeText` (`/campaign`).** A composable promotion gate that proves a held-out lift comes from the candidate's *content* rather than from the prompt/mount *footprint* the content added. A held-out gate proves "candidate beat baseline"; it cannot tell an informative surface from one that merely added bytes the model spends attention on. `neutralizationGate` compares the candidate's lift against a footprint-matched neutralized variant (same layout + length, zero content, via `neutralizeText`) and holds any win whose lift survives blanking (decorative) — however large or significant the raw lift. Compose after significance: `composeGate(heldOutGate({ … }), neutralizationGate({ … }))`.
|
|
22
|
+
- **`runImprovementLoop({ neutralize })` + `GateContext.neutralizedJudgeScores` / `neutralizedArtifacts`.** When a `neutralize` function is supplied and the winner changed, the loop scores a third holdout arm (the blanked winner) and exposes it to the gate. Opt-in — one extra holdout campaign only when wired; existing callers are unaffected.
|
|
23
|
+
|
|
24
|
+
Additive (new exports + optional fields only); no consumer bump required.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
7
28
|
## [0.100.3] — 2026-07-01 — product benchmark contract + eval fixture UX
|
|
8
29
|
|
|
9
30
|
### Added
|
package/README.md
CHANGED
|
@@ -122,7 +122,7 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
|
|
|
122
122
|
|
|
123
123
|
| Subpath | What it gives you |
|
|
124
124
|
|---|---|
|
|
125
|
-
| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
|
|
125
|
+
| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `neutralizationGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
|
|
126
126
|
| `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) |
|
|
127
127
|
| `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency |
|
|
128
128
|
| `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep |
|
|
@@ -136,7 +136,7 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
|
|
|
136
136
|
| `…/matrix` | `runAgentMatrix` — an N-axis cartesian over caller-supplied substrate values, per-axis pass/score/cost/duration |
|
|
137
137
|
| `…/multishot` | N-shot persona × shot matrix runner (`runMultishot` / `runMultishotMatrix`) |
|
|
138
138
|
| `…/wire` | The cross-language HTTP/RPC server + Zod schemas (the source-of-truth protocol the Python client speaks) + the built-in rubric registry |
|
|
139
|
-
| `…/benchmarks` | `BenchmarkAdapter` contract + `deterministicSplit
|
|
139
|
+
| `…/benchmarks` | `BenchmarkAdapter` contract, `runBenchmarkAdapter`, `calibrateBenchmarkMetric`, standard retrieval parsers + ranked retrieval metrics, `deterministicSplit`, and the bundled `routing` reference benchmark |
|
|
140
140
|
|
|
141
141
|
**Specialized surfaces** (subpath-only): `…/prm` (process-reward grading + best-of-N), `…/meta-eval` (judge calibration + the deployment-outcome store), `…/belief-state` (decision-point extraction + selective-policy reports), `…/pipelines` (trace-diagnostic views: budget breach, failure cluster, stuck loop, …), `…/governance` (EU AI Act / NIST AI RMF / SOC2 reports), `…/knowledge` (knowledge-readiness gating before a run), `…/builder-eval` (code-generator three-layer eval), `…/storyboard` (trace → watchable replay), `…/authenticity` (anti-Goodhart "real or convincing BS" scorer over produced files), `…/workflow` (workflow-trace eval + partner export), `…/telemetry` (Workers-safe telemetry client), `…/testing` (test-only reset helpers).
|
|
142
142
|
|
package/dist/adapters/http.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-
|
|
1
|
+
import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-BdIv5dvA.js';
|
|
2
2
|
import '../run-record-I-Z3JNvO.js';
|
|
3
3
|
import '@tangle-network/agent-interface';
|
|
4
4
|
import '../errors-oeQrLqXC.js';
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-
|
|
1
|
+
import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-BdIv5dvA.js';
|
|
2
2
|
import '../run-record-I-Z3JNvO.js';
|
|
3
3
|
import '@tangle-network/agent-interface';
|
|
4
4
|
import '../errors-oeQrLqXC.js';
|
package/dist/adapters/otel.d.ts
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
|
-
export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, c as BenchmarkEvaluation, d as
|
|
1
|
+
export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, k as BenchmarkDistribution, c as BenchmarkEvaluation, d as BenchmarkFamily, l as BenchmarkMetricCalibrationOptions, m as BenchmarkMetricCalibrationResult, n as BenchmarkReport, e as BenchmarkResponder, o as BenchmarkRunOptions, p as BenchmarkRunResult, f as BenchmarkScenario, q as BenchmarkSliceSummary, g as BenchmarkSource, h as BenchmarkTaskKind, r as BuildStandardRetrievalItemsOptions, R as RetrievalIdAdapterOptions, S as StandardRetrievalArtifact, s as StandardRetrievalDocument, t as StandardRetrievalEvaluationOptions, u as StandardRetrievalPayload, v as StandardRetrievalQrel, w as StandardRetrievalQuery, x as StandardRetrievalResult, y as buildStandardRetrievalItems, z as calibrateBenchmarkMetric, A as createRetrievalIdBenchmarkAdapter, i as deterministicSplit, C as evaluateStandardRetrieval, D as normalizeRetrievedDocumentIds, E as parseBeirCorpusJsonl, F as parseBeirQueriesJsonl, G as parseJsonlRows, H as parseQrels, I as parseTsvRows, J as renderBenchmarkReportMarkdown, K as retrievalMetricsAtCutoff, L as routing, M as runBenchmarkAdapter, N as summarizeBenchmarkCampaign } from '../index-pPtfoIJO.js';
|
|
2
|
+
import '../types-BdIv5dvA.js';
|
|
2
3
|
import '../run-record-I-Z3JNvO.js';
|
|
3
4
|
import '@tangle-network/agent-interface';
|
|
4
5
|
import '../errors-oeQrLqXC.js';
|
|
5
6
|
import '../schema-m0gsnbt3.js';
|
|
7
|
+
import '../storage-Dw_f7WMt.js';
|
package/dist/benchmarks/index.js
CHANGED
|
@@ -1,12 +1,58 @@
|
|
|
1
1
|
import {
|
|
2
2
|
BENCHMARK_SPLIT_SEED,
|
|
3
|
+
buildStandardRetrievalItems,
|
|
4
|
+
calibrateBenchmarkMetric,
|
|
5
|
+
createRetrievalIdBenchmarkAdapter,
|
|
3
6
|
deterministicSplit,
|
|
4
|
-
|
|
5
|
-
|
|
7
|
+
evaluateStandardRetrieval,
|
|
8
|
+
normalizeRetrievedDocumentIds,
|
|
9
|
+
parseBeirCorpusJsonl,
|
|
10
|
+
parseBeirQueriesJsonl,
|
|
11
|
+
parseJsonlRows,
|
|
12
|
+
parseQrels,
|
|
13
|
+
parseTsvRows,
|
|
14
|
+
renderBenchmarkReportMarkdown,
|
|
15
|
+
retrievalMetricsAtCutoff,
|
|
16
|
+
routing_exports,
|
|
17
|
+
runBenchmarkAdapter,
|
|
18
|
+
summarizeBenchmarkCampaign
|
|
19
|
+
} from "../chunk-T6W5ADLG.js";
|
|
20
|
+
import "../chunk-4VLZEPJ3.js";
|
|
21
|
+
import "../chunk-OVPVM4JC.js";
|
|
22
|
+
import "../chunk-3PFZBGMR.js";
|
|
23
|
+
import "../chunk-VI2UW6B6.js";
|
|
24
|
+
import "../chunk-LOJ2QVCE.js";
|
|
25
|
+
import "../chunk-2OGPXHOB.js";
|
|
26
|
+
import "../chunk-N22ZO7FV.js";
|
|
27
|
+
import "../chunk-FUCQVFMU.js";
|
|
28
|
+
import "../chunk-45EEMHTC.js";
|
|
29
|
+
import "../chunk-XMSYF4A7.js";
|
|
30
|
+
import "../chunk-RPDDVKI7.js";
|
|
31
|
+
import "../chunk-GGE4NNQT.js";
|
|
32
|
+
import "../chunk-LNQEP766.js";
|
|
33
|
+
import "../chunk-PC4UYEBM.js";
|
|
34
|
+
import "../chunk-VK6HBGAE.js";
|
|
35
|
+
import "../chunk-XJYR7XFV.js";
|
|
36
|
+
import "../chunk-VSMTAMNK.js";
|
|
37
|
+
import "../chunk-ONWEPEDO.js";
|
|
6
38
|
import "../chunk-PZ5AY32C.js";
|
|
7
39
|
export {
|
|
8
40
|
BENCHMARK_SPLIT_SEED,
|
|
41
|
+
buildStandardRetrievalItems,
|
|
42
|
+
calibrateBenchmarkMetric,
|
|
43
|
+
createRetrievalIdBenchmarkAdapter,
|
|
9
44
|
deterministicSplit,
|
|
10
|
-
|
|
45
|
+
evaluateStandardRetrieval,
|
|
46
|
+
normalizeRetrievedDocumentIds,
|
|
47
|
+
parseBeirCorpusJsonl,
|
|
48
|
+
parseBeirQueriesJsonl,
|
|
49
|
+
parseJsonlRows,
|
|
50
|
+
parseQrels,
|
|
51
|
+
parseTsvRows,
|
|
52
|
+
renderBenchmarkReportMarkdown,
|
|
53
|
+
retrievalMetricsAtCutoff,
|
|
54
|
+
routing_exports as routing,
|
|
55
|
+
runBenchmarkAdapter,
|
|
56
|
+
summarizeBenchmarkCampaign
|
|
11
57
|
};
|
|
12
58
|
//# sourceMappingURL=index.js.map
|
package/dist/campaign/index.d.ts
CHANGED
|
@@ -1,11 +1,13 @@
|
|
|
1
|
-
import { S as SignedManifest, B as BackendIntegrityReport, C as CompletionRequirement, R as RuntimeEventLike, a as CompletionVerdict, P as ProducedState, b as CorrectnessChecker } from '../pre-registration-
|
|
2
|
-
export { L as LlmJudgeDimension, c as LlmJudgeOptions, l as llmJudge } from '../pre-registration-
|
|
1
|
+
import { S as SignedManifest, B as BackendIntegrityReport, C as CompletionRequirement, R as RuntimeEventLike, a as CompletionVerdict, P as ProducedState, b as CorrectnessChecker } from '../pre-registration-BUhVPzE7.js';
|
|
2
|
+
export { L as LlmJudgeDimension, c as LlmJudgeOptions, l as llmJudge } from '../pre-registration-BUhVPzE7.js';
|
|
3
3
|
import { A as AnalyzeTracesOptions, a as AnalyzeTracesInput, b as AnalyzeTracesResult } from '../analyst-C8HHvfJp.js';
|
|
4
|
-
import { S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig,
|
|
5
|
-
export { i as CampaignAggregates, j as CampaignArtifactWriter, k as CampaignCellResult, l as CampaignCostMeter, z as CampaignTokenUsage,
|
|
6
|
-
import {
|
|
7
|
-
export {
|
|
8
|
-
|
|
4
|
+
import { S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, f as Gate, d as GenerationRecord, J as JudgeScore, L as LabeledScenarioStore, s as LabeledScenarioWrite, t as LabeledScenarioSampleArgs, u as LabeledScenarioRecord, v as LabelTrust, e as SurfaceProposer, w as ProposedCandidate, x as ProposeContext, y as LabeledScenarioSource, C as CampaignResult, m as CodeSurface } from '../types-BdIv5dvA.js';
|
|
5
|
+
export { i as CampaignAggregates, j as CampaignArtifactWriter, k as CampaignCellResult, l as CampaignCostMeter, z as CampaignTokenUsage, c as CampaignTraceWriter, D as DispatchFn, n as GateContext, h as GateDecision, G as GateResult, o as GenerationCandidate, A as JudgeAggregate, g as JudgeDimension, p as Mutator, O as OptimizationProposer, q as OptimizerConfig, P as ParetoParent, R as RedactionStatus, B as ScenarioAggregate, r as SessionScript, T as TraceSpan, E as isProposedCandidate, F as labelTrustRank } from '../types-BdIv5dvA.js';
|
|
6
|
+
import { C as CampaignRunPlan, P as PlanCampaignRunOptions, b as RunCampaignOptions, c as RunImprovementLoopOptions } from '../gepa-B3x5Ulcv.js';
|
|
7
|
+
export { e as CampaignRunPlanCell, f as GepaProposerConstraints, G as GepaProposerOptions, O as OpenAutoPrOptions, h as OpenAutoPrResult, a as RunImprovementLoopResult, R as RunOptimizationOptions, i as RunOptimizationResult, j as countSentenceEdits, k as defaultRenderDiff, l as extractH2Sections, g as gepaProposer, o as openAutoPr, p as planCampaignRun, r as runCampaign, d as runImprovementLoop, m as runOptimization, s as surfaceHash } from '../gepa-B3x5Ulcv.js';
|
|
8
|
+
import { C as CampaignStorage } from '../storage-Dw_f7WMt.js';
|
|
9
|
+
export { f as fsCampaignStorage, i as inMemoryCampaignStorage } from '../storage-Dw_f7WMt.js';
|
|
10
|
+
export { A as AxisEvidence, a as AxisVerdict, B as BuildEvidenceVectorOptions, l as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, m as EmitLoopProvenanceArgs, n as EmitLoopProvenanceResult, E as EvidenceVector, b as EvolutionaryProposerOptions, H as HeldOutGateOptions, o as LoopProvenanceBackend, q as LoopProvenanceCandidate, L as LoopProvenanceRecord, O as ObjectiveSource, c as ParetoSignificanceGateOptions, P as PowerPreflight, s as PowerPreflightOptions, d as PromotionObjective, e as PromotionPolicy, R as RunEvalOptions, f as buildEvidenceVector, t as buildLoopProvenanceRecord, g as composeGate, h as defaultProductionGate, u as emitLoopProvenance, i as evolutionaryProposer, j as heldOutGate, v as loopProvenanceSpans, p as paretoPolicy, k as paretoSignificanceGate, w as powerPreflight, x as provenanceRecordPath, y as provenanceSpansPath, r as runEval, z as surfaceContentHash } from '../provenance-DdDhf6cg.js';
|
|
9
11
|
import { E as EProcessState, a as PairedBootstrapResult } from '../statistics-D88peojY.js';
|
|
10
12
|
import { L as LlmClientOptions } from '../llm-client-DyqEH4jH.js';
|
|
11
13
|
import { AgentProfile } from '@tangle-network/agent-interface';
|
|
@@ -435,6 +437,43 @@ declare function loadEvalFixtureScenarios(evalsDir: string, options?: LoadEvalFi
|
|
|
435
437
|
*/
|
|
436
438
|
declare function planEvalFixtureRun<TArtifact = unknown>(options: PlanEvalFixtureRunOptions<TArtifact>): EvalFixtureRunPlan;
|
|
437
439
|
|
|
440
|
+
/**
|
|
441
|
+
* @module
|
|
442
|
+
* Composable placebo / neutralization promotion gate.
|
|
443
|
+
*
|
|
444
|
+
* A held-out gate proves a candidate beat baseline. It CANNOT prove the lift came
|
|
445
|
+
* from the candidate's CONTENT rather than from the prompt/mount FOOTPRINT the
|
|
446
|
+
* content happened to add (more bytes, a longer prompt). This gate closes that
|
|
447
|
+
* hole: it compares the candidate's held-out lift against the lift of a
|
|
448
|
+
* FOOTPRINT-MATCHED neutralized variant (same layout + length, zero content, via
|
|
449
|
+
* `neutralizeText`). If the neutralized variant reproduces more than
|
|
450
|
+
* `maxDecorativeFraction` of the candidate's lift, the lift is decorative — it
|
|
451
|
+
* survives blanking the content — and the candidate is HELD regardless of how
|
|
452
|
+
* large or significant its raw lift is.
|
|
453
|
+
*
|
|
454
|
+
* Compose it AFTER the significance gate — significance says the lift is real,
|
|
455
|
+
* this says the lift is CAUSED BY THE CONTENT:
|
|
456
|
+
* composeGate(heldOutGate({ ... }), neutralizationGate({ ... }))
|
|
457
|
+
*
|
|
458
|
+
* Requires `ctx.neutralizedJudgeScores`, populated by `runImprovementLoop` when it
|
|
459
|
+
* is given a `neutralize` function. A gate composed without that wiring fails
|
|
460
|
+
* loud rather than silently passing an unproven candidate.
|
|
461
|
+
*/
|
|
462
|
+
|
|
463
|
+
interface NeutralizationGateOptions<TScenario extends Scenario = Scenario> {
|
|
464
|
+
scenarios: TScenario[];
|
|
465
|
+
/** Reject when the neutralized (content-blanked, footprint-matched) variant
|
|
466
|
+
* reproduces at least this fraction of the candidate's held-out lift. Default
|
|
467
|
+
* 0.5 — if blanking the content keeps half the lift, the content is decorative.
|
|
468
|
+
* Equality rejects: a neutralized lift == threshold·candidateLift is decorative. */
|
|
469
|
+
maxDecorativeFraction?: number;
|
|
470
|
+
}
|
|
471
|
+
/**
|
|
472
|
+
* Composable placebo gate: ships only when the candidate's held-out lift is NOT
|
|
473
|
+
* mostly reproduced by a footprint-matched neutralized variant.
|
|
474
|
+
*/
|
|
475
|
+
declare function neutralizationGate<TArtifact, TScenario extends Scenario>(options: NeutralizationGateOptions<TScenario>): Gate<TArtifact, TScenario>;
|
|
476
|
+
|
|
438
477
|
/**
|
|
439
478
|
* Anytime-valid sequential promotion gate — an e-process (betting
|
|
440
479
|
* test-martingale, see `eProcess` in `statistics.ts`) over paired
|
|
@@ -751,6 +790,34 @@ declare class FsLabeledScenarioStore implements LabeledScenarioStore {
|
|
|
751
790
|
private pathForSource;
|
|
752
791
|
}
|
|
753
792
|
|
|
793
|
+
/**
|
|
794
|
+
* @module
|
|
795
|
+
* Footprint-matched neutralization — the placebo control for content-vs-footprint
|
|
796
|
+
* attribution in a promotion gate.
|
|
797
|
+
*
|
|
798
|
+
* A promoted surface can raise a held-out score two different ways:
|
|
799
|
+
* 1. its CONTENT is informative (the thing we want to promote), or
|
|
800
|
+
* 2. it merely added prompt/mount FOOTPRINT — more bytes, more lines, a longer
|
|
801
|
+
* more authoritative-looking prompt — that the model spends attention on
|
|
802
|
+
* regardless of what the bytes say.
|
|
803
|
+
*
|
|
804
|
+
* A held-out gate proves the candidate beat baseline; it cannot separate (1) from
|
|
805
|
+
* (2). `neutralizeText` produces a variant that keeps the input's layout and
|
|
806
|
+
* length while carrying ZERO information, so scoring it isolates the footprint
|
|
807
|
+
* contribution (2). Feed the neutralized variant's scores to `neutralizationGate`:
|
|
808
|
+
* any lift it still holds over baseline is decorative, and a candidate whose lift
|
|
809
|
+
* survives neutralization is rejected however large its raw lift.
|
|
810
|
+
*/
|
|
811
|
+
/**
|
|
812
|
+
* Blank every non-whitespace character to a 1-byte filler while preserving all
|
|
813
|
+
* whitespace. Line count, indentation, and word/line lengths are unchanged — so
|
|
814
|
+
* the neutralized variant has the same layout and (for ASCII) the same byte
|
|
815
|
+
* footprint as the input, but no readable content. Whitespace is preserved
|
|
816
|
+
* deliberately: collapsing it would change the token structure and stop the
|
|
817
|
+
* variant from being a true footprint match.
|
|
818
|
+
*/
|
|
819
|
+
declare function neutralizeText(content: string): string;
|
|
820
|
+
|
|
754
821
|
/**
|
|
755
822
|
* FAPO (Fully Autonomous Prompt Optimization) is an orchestration policy, not
|
|
756
823
|
* a new prompt mutation primitive. The paper's loop evaluates an inspectable
|
|
@@ -2007,4 +2074,4 @@ declare function gitWorktreeAdapter(opts: GitWorktreeAdapterOptions): WorktreeAd
|
|
|
2007
2074
|
* as a ref under the adapter's worktree dir. */
|
|
2008
2075
|
declare function resolveWorktreePath(surface: CodeSurface, worktreeDir?: string): string;
|
|
2009
2076
|
|
|
2010
|
-
export { type AcceptedEdit, type AceProposerOptions, type AnalystArtifact, type AnalystScenario, type ApplySkillPatchResult, type BuildAnalystSurfaceDispatchOptions, type CampaignBreakdown, CampaignResult, CampaignRunPlan, CampaignStorage, CodeSurface, type CompareProposersOptions, type DimensionRegression, type DiscriminationScore, DispatchContext, type EvalFixture, type EvalFixtureFile, type EvalFixtureLoadOptions, type EvalFixtureRunPlan, type EvalFixtureScenario, type EvalFixtureValidationMode, type FailureModeRecallJudgeOptions, type FapoAttributionSignals, type FapoEntryConfig, type FapoFailureCluster, type FapoOptimizationLevel, type FapoProposerOptions, type FapoReviewInput, type FapoReviewIssue, type FapoReviewResult, type FapoScopeContract, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, Gate, GenerationRecord, type GitWorktreeAdapterOptions, type Governor, type GovernorContext, type GovernorOp, type HaloProposerOptions, type HeldoutSignificance, type HeldoutSignificanceOptions, type HeuristicGovernorOptions, type JsonPrimitive, type JsonValue, JudgeConfig, JudgeScore, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, Lineage, type LineageEdge, type LineageGraph, type LineageNode, type LineageNodeInput, type LineageStore, type LoadEvalFixtureScenariosOptions, type MemoryCurationProposerOptions, MutableSurface, type OptimizerEntryConfig, type PairedHoldout, type ParameterCandidate, type ParameterChange, type ParameterSweepProposerOptions, PlanCampaignRunOptions, type PlanEvalFixtureRunOptions, type PlaybackContext, type PlaybackDriver, type PlaybackStep, type PolicyEditProposerOptions, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, ProposeContext, type ProposePatchesArgs, ProposedCandidate, type ProposerComparison, type ProposerEntry, type ProposerPairwise, type ProposerScore, type RejectedEdit, RunCampaignOptions, RunImprovementLoopOptions, type RunLineageLoopOptions, type RunLineageLoopResult, type RunLineageLoopSeed, type RunLineageOptions, type RunLineageResult, type RunLineageSeed, type RunLineageStepResult, type RunProfileMatrixOptions, type RunProfileMatrixResult, type RunSkillOptOptions, type RunSkillOptResult, Scenario, type ScenarioRollup, type ScenarioSignal, type ScoreboardRenderOptions, type ScoreboardRow, type ScoreboardSummary, type SequentialDecideFn, type SequentialDecideOptions, type SequentialDecision, type SequentialObservation, type SequentialPairedGate, type SequentialPairedGateOptions, type SkillOptEpochRecord, type SkillOptEvidence, type SkillOptProposer, type SkillOptProposerOptions, type SkillPatch, type SkillPatchOp, SkillPatchParseError, type SkillPatchRejection, SurfaceProposer, type SurfaceScore, type TraceAnalystProposerOptions, type UserStory, type UserStoryVerdict, type Worktree, type WorktreeAdapter, WorktreeAdapterError, aceProposer, applySkillPatch, buildAnalystSurfaceDispatch, callbackGovernor, campaignBreakdown, campaignMeanComposite, compareProposers, detectScale, dimensionRegressions, discoverEvalFixtures, extractFapoAttributionSignals, failureModeRecallJudge, fapoEscalationEntry, fapoProposer, fsLineageStore, gepaParetoEntry, gepaReflectionEntry, gitWorktreeAdapter, haloProposer, heldoutSignificance, heuristicGovernor, lineageNodeId, loadEvalFixture, loadEvalFixtureScenarios, makePlaybackDispatch, memLineageStore, memoryCurationProposer, pairHoldout, parameterSweepProposer, parseSkillPatchResponse, patchEditCount, planEvalFixtureRun, policyEditProposer, renderScoreboardMarkdown, resolveRunDir, resolveWorktreePath, runLineage, runLineageLoop, runProfileMatrix, runSkillOpt, scoreDiscrimination, scoreUserStory, scoreboardSummary, selectDiscriminative, sequentialDecide, sequentialPairedGate, skillOptEntry, skillOptProposer, tangleTracesRoot, traceAnalystProposer, userStoryScoreboard };
|
|
2077
|
+
export { type AcceptedEdit, type AceProposerOptions, type AnalystArtifact, type AnalystScenario, type ApplySkillPatchResult, type BuildAnalystSurfaceDispatchOptions, type CampaignBreakdown, CampaignResult, CampaignRunPlan, CampaignStorage, CodeSurface, type CompareProposersOptions, type DimensionRegression, type DiscriminationScore, DispatchContext, type EvalFixture, type EvalFixtureFile, type EvalFixtureLoadOptions, type EvalFixtureRunPlan, type EvalFixtureScenario, type EvalFixtureValidationMode, type FailureModeRecallJudgeOptions, type FapoAttributionSignals, type FapoEntryConfig, type FapoFailureCluster, type FapoOptimizationLevel, type FapoProposerOptions, type FapoReviewInput, type FapoReviewIssue, type FapoReviewResult, type FapoScopeContract, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, Gate, GenerationRecord, type GitWorktreeAdapterOptions, type Governor, type GovernorContext, type GovernorOp, type HaloProposerOptions, type HeldoutSignificance, type HeldoutSignificanceOptions, type HeuristicGovernorOptions, type JsonPrimitive, type JsonValue, JudgeConfig, JudgeScore, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, Lineage, type LineageEdge, type LineageGraph, type LineageNode, type LineageNodeInput, type LineageStore, type LoadEvalFixtureScenariosOptions, type MemoryCurationProposerOptions, MutableSurface, type NeutralizationGateOptions, type OptimizerEntryConfig, type PairedHoldout, type ParameterCandidate, type ParameterChange, type ParameterSweepProposerOptions, PlanCampaignRunOptions, type PlanEvalFixtureRunOptions, type PlaybackContext, type PlaybackDriver, type PlaybackStep, type PolicyEditProposerOptions, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, ProposeContext, type ProposePatchesArgs, ProposedCandidate, type ProposerComparison, type ProposerEntry, type ProposerPairwise, type ProposerScore, type RejectedEdit, RunCampaignOptions, RunImprovementLoopOptions, type RunLineageLoopOptions, type RunLineageLoopResult, type RunLineageLoopSeed, type RunLineageOptions, type RunLineageResult, type RunLineageSeed, type RunLineageStepResult, type RunProfileMatrixOptions, type RunProfileMatrixResult, type RunSkillOptOptions, type RunSkillOptResult, Scenario, type ScenarioRollup, type ScenarioSignal, type ScoreboardRenderOptions, type ScoreboardRow, type ScoreboardSummary, type SequentialDecideFn, type SequentialDecideOptions, type SequentialDecision, type SequentialObservation, type SequentialPairedGate, type SequentialPairedGateOptions, type SkillOptEpochRecord, type SkillOptEvidence, type SkillOptProposer, type SkillOptProposerOptions, type SkillPatch, type SkillPatchOp, SkillPatchParseError, type SkillPatchRejection, SurfaceProposer, type SurfaceScore, type TraceAnalystProposerOptions, type UserStory, type UserStoryVerdict, type Worktree, type WorktreeAdapter, WorktreeAdapterError, aceProposer, applySkillPatch, buildAnalystSurfaceDispatch, callbackGovernor, campaignBreakdown, campaignMeanComposite, compareProposers, detectScale, dimensionRegressions, discoverEvalFixtures, extractFapoAttributionSignals, failureModeRecallJudge, fapoEscalationEntry, fapoProposer, fsLineageStore, gepaParetoEntry, gepaReflectionEntry, gitWorktreeAdapter, haloProposer, heldoutSignificance, heuristicGovernor, lineageNodeId, loadEvalFixture, loadEvalFixtureScenarios, makePlaybackDispatch, memLineageStore, memoryCurationProposer, neutralizationGate, neutralizeText, pairHoldout, parameterSweepProposer, parseSkillPatchResponse, patchEditCount, planEvalFixtureRun, policyEditProposer, renderScoreboardMarkdown, resolveRunDir, resolveWorktreePath, runLineage, runLineageLoop, runProfileMatrix, runSkillOpt, scoreDiscrimination, scoreUserStory, scoreboardSummary, selectDiscriminative, sequentialDecide, sequentialPairedGate, skillOptEntry, skillOptProposer, tangleTracesRoot, traceAnalystProposer, userStoryScoreboard };
|