instar 1.3.740 → 1.3.741
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/core/promptContract.d.ts +114 -0
- package/dist/core/promptContract.d.ts.map +1 -0
- package/dist/core/promptContract.js +120 -0
- package/dist/core/promptContract.js.map +1 -0
- package/dist/data/llmBenchCoverage.d.ts +8 -0
- package/dist/data/llmBenchCoverage.d.ts.map +1 -1
- package/dist/data/llmBenchCoverage.js +117 -0
- package/dist/data/llmBenchCoverage.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +2 -2
- package/src/data/llmBenchCoverage.ts +200 -0
- package/upgrades/1.3.741.md +85 -0
- package/upgrades/side-effects/prompt-parser-contract-standard.md +67 -0
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared prompt↔parser contract library — the mechanical arm of the
|
|
3
|
+
* "The Prompt and the Parser Are One Contract" standard (defect class 1 closure;
|
|
4
|
+
* docs/specs/prompt-parser-contract-standard.md).
|
|
5
|
+
*
|
|
6
|
+
* WHY THIS EXISTS (earned): on 2026-07-02 the INSTAR-Bench v2 defect-class
|
|
7
|
+
* review found the tone gate's PROMPT taught models the short rule id ("B15")
|
|
8
|
+
* while its production PARSER accepts only the full identifier
|
|
9
|
+
* ("B15_CONTEXT_DEATH_STOP") and fails closed on the short form. Every model
|
|
10
|
+
* through every door obeyed the prompt and "failed" — a 100%-our-fault defect.
|
|
11
|
+
* An LLM callsite whose output is machine-parsed is ONE contract with TWO halves
|
|
12
|
+
* maintained separately: the prompt promises an output vocabulary/shape; the
|
|
13
|
+
* parser accepts one. Nothing held them coherent. This module gives that
|
|
14
|
+
* contract a code artifact — the type of a co-located promise, and a pure
|
|
15
|
+
* generator for the counter-examples a contract test must prove the parser
|
|
16
|
+
* REJECTS — so a taught-but-unparsed vocabulary becomes a CI failure instead of
|
|
17
|
+
* a latent door.
|
|
18
|
+
*
|
|
19
|
+
* SCOPE HONESTY (design §2): the PREFERRED form for an enumerated-verdict
|
|
20
|
+
* callsite is single-sourcing — prompt, parser, and test all consume ONE
|
|
21
|
+
* exported verdict constant, which makes a taught-but-undeclared token
|
|
22
|
+
* structurally impossible. The `PromptContract` manifest below is the FALLBACK
|
|
23
|
+
* form for prose-shaped prompts where single-sourcing is impractical (a reviewed
|
|
24
|
+
* claim per spec §2, not an author's private call). This module ships DARK: it
|
|
25
|
+
* introduces no runtime caller and changes NO live prompt or parser behavior.
|
|
26
|
+
* It is consumed only by the per-callsite CONTRACT TESTS that graduate on the
|
|
27
|
+
* shrink-only schedule (rollout §1/§2) — each of which renders the REAL
|
|
28
|
+
* production prompt through an exported pure render function; that render
|
|
29
|
+
* refactor of the live builders is deferred to its own A/B-gated increments and
|
|
30
|
+
* is NOT part of this dark increment.
|
|
31
|
+
*/
|
|
32
|
+
/**
|
|
33
|
+
* The election of a contract's form, recorded per coverage entry (spec §2:
|
|
34
|
+
* "Form election is not self-certified"). A `manifest`-form entry on an
|
|
35
|
+
* enumerated-verdict callsite requires an X1 argued reason in the registry.
|
|
36
|
+
*/
|
|
37
|
+
export type ContractForm = 'single-source' | 'manifest';
|
|
38
|
+
/**
|
|
39
|
+
* The co-located machine-readable promise for a prose-shaped (fallback-form)
|
|
40
|
+
* callsite. Co-location is the point (spec §2): a prompt edit lands in the same
|
|
41
|
+
* file/diff as the promise, so review and CI see contract drift as ONE change.
|
|
42
|
+
*
|
|
43
|
+
* The `parser` is a FUNCTION REFERENCE, never a string — refactor-safe (spec §2).
|
|
44
|
+
* A parser returning a non-null/defined value for an input is treated as
|
|
45
|
+
* "accepted"; the contract test decides acceptance semantics per callsite.
|
|
46
|
+
*/
|
|
47
|
+
export interface PromptContract<TAccepted = unknown> {
|
|
48
|
+
/**
|
|
49
|
+
* Every terminal token/shape the prompt text tells the model to produce.
|
|
50
|
+
* Prefer the shared verdict constant even here (spec §2).
|
|
51
|
+
*/
|
|
52
|
+
readonly promisedOutputs: readonly string[];
|
|
53
|
+
/**
|
|
54
|
+
* Canonical counter-examples the parser must REJECT (fail-closed proof).
|
|
55
|
+
* MECHANICALLY DERIVED from `promisedOutputs` via `deriveRejectedForms`
|
|
56
|
+
* (case-mutation, prefix-truncation, separator-stripping) plus hand-picked
|
|
57
|
+
* extras — a hand-only list invites trivial rejects (spec §2).
|
|
58
|
+
*/
|
|
59
|
+
readonly rejectedForms: readonly string[];
|
|
60
|
+
/**
|
|
61
|
+
* Known-hazard shapes that must NOT appear anywhere in the full rendered
|
|
62
|
+
* prompt outside explicitly-declared negative sections (spec §3.1) — the
|
|
63
|
+
* inverse-direction backstop (e.g. bare rule ids the parser rejects).
|
|
64
|
+
*/
|
|
65
|
+
readonly hazardPatterns: readonly RegExp[];
|
|
66
|
+
/**
|
|
67
|
+
* The response envelope the prompt promises (JSON shape, field names).
|
|
68
|
+
*/
|
|
69
|
+
readonly envelope: {
|
|
70
|
+
readonly shape: 'json' | 'text';
|
|
71
|
+
readonly verdictField?: string;
|
|
72
|
+
};
|
|
73
|
+
/**
|
|
74
|
+
* Compatibility forms the parser DELIBERATELY accepts (documented, tested).
|
|
75
|
+
* "non-promised" for the fail-closed test means outside promised ∪ aliases.
|
|
76
|
+
*/
|
|
77
|
+
readonly acceptedAliases: readonly string[];
|
|
78
|
+
/** The REAL production parser, by reference. */
|
|
79
|
+
readonly parser: (raw: string) => TAccepted;
|
|
80
|
+
}
|
|
81
|
+
/**
|
|
82
|
+
* Options for {@link deriveRejectedForms}.
|
|
83
|
+
*/
|
|
84
|
+
export interface DeriveRejectedFormsOptions {
|
|
85
|
+
/**
|
|
86
|
+
* The separator characters whose stripping/truncation produces a rejected
|
|
87
|
+
* form. Defaults to `_`, `-`, and a space — the separators that produced the
|
|
88
|
+
* historical B15 defect ("B15_CONTEXT_DEATH_STOP" → "B15").
|
|
89
|
+
*/
|
|
90
|
+
readonly separators?: readonly string[];
|
|
91
|
+
}
|
|
92
|
+
/**
|
|
93
|
+
* Mechanically derive the canonical counter-examples a contract test feeds the
|
|
94
|
+
* REAL parser to prove its documented fail-closed behavior (spec §2/§3.3).
|
|
95
|
+
*
|
|
96
|
+
* For each promised token it produces: case-mutations, prefix-truncations at
|
|
97
|
+
* each separator, and separator-stripped forms. `extras` (hand-picked shapes,
|
|
98
|
+
* e.g. the literal short id `B15` and the empty string) are appended. The
|
|
99
|
+
* result EXCLUDES any form that collides with a genuinely-promised token — a
|
|
100
|
+
* mutation that equals a real promised output is NOT a rejected form (that would
|
|
101
|
+
* make the fail-closed assertion contradict the acceptance assertion). The
|
|
102
|
+
* output is de-duplicated and stably ordered.
|
|
103
|
+
*
|
|
104
|
+
* Pure and side-effect-free. It touches no live parser and reads no I/O.
|
|
105
|
+
*
|
|
106
|
+
* @param vocabulary the promised output vocabulary (the taught verdict tokens).
|
|
107
|
+
* @param extras hand-picked additional rejected forms (spec §2: a hand-only
|
|
108
|
+
* list invites trivial rejects, so this AUGMENTS the derived set).
|
|
109
|
+
* @param options separator configuration (defaults to `_ - <space>`).
|
|
110
|
+
* @returns the derived + hand-picked rejected forms, minus any that collide
|
|
111
|
+
* with a promised token, de-duplicated and stably ordered.
|
|
112
|
+
*/
|
|
113
|
+
export declare function deriveRejectedForms(vocabulary: readonly string[], extras?: readonly string[], options?: DeriveRejectedFormsOptions): string[];
|
|
114
|
+
//# sourceMappingURL=promptContract.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"promptContract.d.ts","sourceRoot":"","sources":["../../src/core/promptContract.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AAEH;;;;GAIG;AACH,MAAM,MAAM,YAAY,GAAG,eAAe,GAAG,UAAU,CAAC;AAExD;;;;;;;;GAQG;AACH,MAAM,WAAW,cAAc,CAAC,SAAS,GAAG,OAAO;IACjD;;;OAGG;IACH,QAAQ,CAAC,eAAe,EAAE,SAAS,MAAM,EAAE,CAAC;IAC5C;;;;;OAKG;IACH,QAAQ,CAAC,aAAa,EAAE,SAAS,MAAM,EAAE,CAAC;IAC1C;;;;OAIG;IACH,QAAQ,CAAC,cAAc,EAAE,SAAS,MAAM,EAAE,CAAC;IAC3C;;OAEG;IACH,QAAQ,CAAC,QAAQ,EAAE;QAAE,QAAQ,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAAC;QAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACvF;;;OAGG;IACH,QAAQ,CAAC,eAAe,EAAE,SAAS,MAAM,EAAE,CAAC;IAC5C,gDAAgD;IAChD,QAAQ,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,MAAM,KAAK,SAAS,CAAC;CAC7C;AAED;;GAEG;AACH,MAAM,WAAW,0BAA0B;IACzC;;;;OAIG;IACH,QAAQ,CAAC,UAAU,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CACzC;AAgDD;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAgB,mBAAmB,CACjC,UAAU,EAAE,SAAS,MAAM,EAAE,EAC7B,MAAM,GAAE,SAAS,MAAM,EAAO,EAC9B,OAAO,GAAE,0BAA+B,GACvC,MAAM,EAAE,CAqBV"}
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Shared prompt↔parser contract library — the mechanical arm of the
|
|
3
|
+
* "The Prompt and the Parser Are One Contract" standard (defect class 1 closure;
|
|
4
|
+
* docs/specs/prompt-parser-contract-standard.md).
|
|
5
|
+
*
|
|
6
|
+
* WHY THIS EXISTS (earned): on 2026-07-02 the INSTAR-Bench v2 defect-class
|
|
7
|
+
* review found the tone gate's PROMPT taught models the short rule id ("B15")
|
|
8
|
+
* while its production PARSER accepts only the full identifier
|
|
9
|
+
* ("B15_CONTEXT_DEATH_STOP") and fails closed on the short form. Every model
|
|
10
|
+
* through every door obeyed the prompt and "failed" — a 100%-our-fault defect.
|
|
11
|
+
* An LLM callsite whose output is machine-parsed is ONE contract with TWO halves
|
|
12
|
+
* maintained separately: the prompt promises an output vocabulary/shape; the
|
|
13
|
+
* parser accepts one. Nothing held them coherent. This module gives that
|
|
14
|
+
* contract a code artifact — the type of a co-located promise, and a pure
|
|
15
|
+
* generator for the counter-examples a contract test must prove the parser
|
|
16
|
+
* REJECTS — so a taught-but-unparsed vocabulary becomes a CI failure instead of
|
|
17
|
+
* a latent door.
|
|
18
|
+
*
|
|
19
|
+
* SCOPE HONESTY (design §2): the PREFERRED form for an enumerated-verdict
|
|
20
|
+
* callsite is single-sourcing — prompt, parser, and test all consume ONE
|
|
21
|
+
* exported verdict constant, which makes a taught-but-undeclared token
|
|
22
|
+
* structurally impossible. The `PromptContract` manifest below is the FALLBACK
|
|
23
|
+
* form for prose-shaped prompts where single-sourcing is impractical (a reviewed
|
|
24
|
+
* claim per spec §2, not an author's private call). This module ships DARK: it
|
|
25
|
+
* introduces no runtime caller and changes NO live prompt or parser behavior.
|
|
26
|
+
* It is consumed only by the per-callsite CONTRACT TESTS that graduate on the
|
|
27
|
+
* shrink-only schedule (rollout §1/§2) — each of which renders the REAL
|
|
28
|
+
* production prompt through an exported pure render function; that render
|
|
29
|
+
* refactor of the live builders is deferred to its own A/B-gated increments and
|
|
30
|
+
* is NOT part of this dark increment.
|
|
31
|
+
*/
|
|
32
|
+
/**
|
|
33
|
+
* Toggle a token's letter case in the three ways a lenient model most often
|
|
34
|
+
* mutates a taught identifier: all-upper, all-lower, and first-letter-flipped.
|
|
35
|
+
* Purely mechanical — no locale assumptions beyond JS default casing.
|
|
36
|
+
*/
|
|
37
|
+
function caseMutations(token) {
|
|
38
|
+
const out = new Set();
|
|
39
|
+
out.add(token.toUpperCase());
|
|
40
|
+
out.add(token.toLowerCase());
|
|
41
|
+
if (token.length > 0) {
|
|
42
|
+
const first = token[0];
|
|
43
|
+
const flippedFirst = first === first.toUpperCase() ? first.toLowerCase() : first.toUpperCase();
|
|
44
|
+
out.add(flippedFirst + token.slice(1));
|
|
45
|
+
}
|
|
46
|
+
return [...out];
|
|
47
|
+
}
|
|
48
|
+
/**
|
|
49
|
+
* Prefix-truncations at each separator boundary — the exact B15 shape: the
|
|
50
|
+
* model emits the leading segment ("B15") of a compound identifier
|
|
51
|
+
* ("B15_CONTEXT_DEATH_STOP") that the parser rejects.
|
|
52
|
+
*/
|
|
53
|
+
function prefixTruncations(token, separators) {
|
|
54
|
+
const out = new Set();
|
|
55
|
+
for (const sep of separators) {
|
|
56
|
+
if (!sep)
|
|
57
|
+
continue;
|
|
58
|
+
const idx = token.indexOf(sep);
|
|
59
|
+
if (idx > 0)
|
|
60
|
+
out.add(token.slice(0, idx));
|
|
61
|
+
}
|
|
62
|
+
return [...out];
|
|
63
|
+
}
|
|
64
|
+
/**
|
|
65
|
+
* Separator-stripped forms — the same token with each separator removed
|
|
66
|
+
* ("B15_CONTEXT_DEATH_STOP" → "B15CONTEXTDEATHSTOP").
|
|
67
|
+
*/
|
|
68
|
+
function separatorStripped(token, separators) {
|
|
69
|
+
const out = new Set();
|
|
70
|
+
for (const sep of separators) {
|
|
71
|
+
if (!sep)
|
|
72
|
+
continue;
|
|
73
|
+
if (token.includes(sep))
|
|
74
|
+
out.add(token.split(sep).join(''));
|
|
75
|
+
}
|
|
76
|
+
return [...out];
|
|
77
|
+
}
|
|
78
|
+
/**
|
|
79
|
+
* Mechanically derive the canonical counter-examples a contract test feeds the
|
|
80
|
+
* REAL parser to prove its documented fail-closed behavior (spec §2/§3.3).
|
|
81
|
+
*
|
|
82
|
+
* For each promised token it produces: case-mutations, prefix-truncations at
|
|
83
|
+
* each separator, and separator-stripped forms. `extras` (hand-picked shapes,
|
|
84
|
+
* e.g. the literal short id `B15` and the empty string) are appended. The
|
|
85
|
+
* result EXCLUDES any form that collides with a genuinely-promised token — a
|
|
86
|
+
* mutation that equals a real promised output is NOT a rejected form (that would
|
|
87
|
+
* make the fail-closed assertion contradict the acceptance assertion). The
|
|
88
|
+
* output is de-duplicated and stably ordered.
|
|
89
|
+
*
|
|
90
|
+
* Pure and side-effect-free. It touches no live parser and reads no I/O.
|
|
91
|
+
*
|
|
92
|
+
* @param vocabulary the promised output vocabulary (the taught verdict tokens).
|
|
93
|
+
* @param extras hand-picked additional rejected forms (spec §2: a hand-only
|
|
94
|
+
* list invites trivial rejects, so this AUGMENTS the derived set).
|
|
95
|
+
* @param options separator configuration (defaults to `_ - <space>`).
|
|
96
|
+
* @returns the derived + hand-picked rejected forms, minus any that collide
|
|
97
|
+
* with a promised token, de-duplicated and stably ordered.
|
|
98
|
+
*/
|
|
99
|
+
export function deriveRejectedForms(vocabulary, extras = [], options = {}) {
|
|
100
|
+
const separators = options.separators ?? ['_', '-', ' '];
|
|
101
|
+
const promised = new Set(vocabulary);
|
|
102
|
+
const derived = [];
|
|
103
|
+
for (const token of vocabulary) {
|
|
104
|
+
if (typeof token !== 'string' || token.length === 0)
|
|
105
|
+
continue;
|
|
106
|
+
derived.push(...caseMutations(token), ...prefixTruncations(token, separators), ...separatorStripped(token, separators));
|
|
107
|
+
}
|
|
108
|
+
const seen = new Set();
|
|
109
|
+
const out = [];
|
|
110
|
+
for (const form of [...derived, ...extras]) {
|
|
111
|
+
if (promised.has(form))
|
|
112
|
+
continue; // a real promised token is never a "rejected form"
|
|
113
|
+
if (seen.has(form))
|
|
114
|
+
continue;
|
|
115
|
+
seen.add(form);
|
|
116
|
+
out.push(form);
|
|
117
|
+
}
|
|
118
|
+
return out;
|
|
119
|
+
}
|
|
120
|
+
//# sourceMappingURL=promptContract.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"promptContract.js","sourceRoot":"","sources":["../../src/core/promptContract.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AA8DH;;;;GAIG;AACH,SAAS,aAAa,CAAC,KAAa;IAClC,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;IAC9B,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC;IAC7B,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC;IAC7B,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;QACvB,MAAM,YAAY,GAChB,KAAK,KAAK,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC;QAC5E,GAAG,CAAC,GAAG,CAAC,YAAY,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACzC,CAAC;IACD,OAAO,CAAC,GAAG,GAAG,CAAC,CAAC;AAClB,CAAC;AAED;;;;GAIG;AACH,SAAS,iBAAiB,CAAC,KAAa,EAAE,UAA6B;IACrE,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;IAC9B,KAAK,MAAM,GAAG,IAAI,UAAU,EAAE,CAAC;QAC7B,IAAI,CAAC,GAAG;YAAE,SAAS;QACnB,MAAM,GAAG,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC/B,IAAI,GAAG,GAAG,CAAC;YAAE,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;IAC5C,CAAC;IACD,OAAO,CAAC,GAAG,GAAG,CAAC,CAAC;AAClB,CAAC;AAED;;;GAGG;AACH,SAAS,iBAAiB,CAAC,KAAa,EAAE,UAA6B;IACrE,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;IAC9B,KAAK,MAAM,GAAG,IAAI,UAAU,EAAE,CAAC;QAC7B,IAAI,CAAC,GAAG;YAAE,SAAS;QACnB,IAAI,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC;YAAE,GAAG,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;IAC9D,CAAC;IACD,OAAO,CAAC,GAAG,GAAG,CAAC,CAAC;AAClB,CAAC;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,MAAM,UAAU,mBAAmB,CACjC,UAA6B,EAC7B,SAA4B,EAAE,EAC9B,UAAsC,EAAE;IAExC,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;IACzD,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,CAAC;IACrC,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE,CAAC;QAC/B,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,SAAS;QAC9D,OAAO,CAAC,IAAI,CACV,GAAG,aAAa,CAAC,KAAK,CAAC,EACvB,GAAG,iBAAiB,CAAC,KAAK,EAAE,UAAU,CAAC,EACvC,GAAG,iBAAiB,CAAC,KAAK,EAAE,UAAU,CAAC,CACxC,CAAC;IACJ,CAAC;IACD,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;IAC/B,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,MAAM,IAAI,IAAI,CAAC,GAAG,OAAO,EAAE,GAAG,MAAM,CAAC,EAAE,CAAC;QAC3C,IAAI,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,SAAS,CAAC,mDAAmD;QACrF,IAAI,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,SAAS;QAC7B,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACf,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACjB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}
|
|
@@ -53,4 +53,12 @@ export type JudgesClaimsFlag = {
|
|
|
53
53
|
false: string;
|
|
54
54
|
};
|
|
55
55
|
export declare const LLM_JUDGES_CLAIMS: Readonly<Record<string, JudgesClaimsFlag>>;
|
|
56
|
+
export type ParserContractFlag = {
|
|
57
|
+
contractTest: string;
|
|
58
|
+
} | {
|
|
59
|
+
pending: 'contract-wave-1' | 'contract-wave-2';
|
|
60
|
+
} | {
|
|
61
|
+
false: string;
|
|
62
|
+
};
|
|
63
|
+
export declare const LLM_PARSER_CONTRACT: Readonly<Record<string, ParserContractFlag>>;
|
|
56
64
|
//# sourceMappingURL=llmBenchCoverage.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"llmBenchCoverage.d.ts","sourceRoot":"","sources":["../../src/data/llmBenchCoverage.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AAEH,MAAM,MAAM,aAAa,GACrB;IAAE,IAAI,EAAE,MAAM,CAAA;CAAE,GAChB;IAAE,OAAO,EAAE,QAAQ,GAAG,QAAQ,CAAA;CAAE,GAChC;IAAE,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC;AAEvB,eAAO,MAAM,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,CAqFtE,CAAC;AA4BF,MAAM,MAAM,kBAAkB,GAAG,IAAI,GAAG;IAAE,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAE1D,eAAO,MAAM,mBAAmB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAiF5E,CAAC;AA+CF;;;;6EAI6E;AAC7E,MAAM,MAAM,SAAS,GAAG,iBAAiB,GAAG,aAAa,GAAG,cAAc,CAAC;AAE3E,eAAO,MAAM,WAAW,EAAE,aAAa,CAAC,SAAS,CAIhD,CAAC;AAEF;;;;oFAIoF;AACpF,MAAM,MAAM,gBAAgB,GAAG;IAAE,SAAS,EAAE,SAAS,CAAA;CAAE,GAAG,KAAK,GAAG;IAAE,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAEpF,eAAO,MAAM,iBAAiB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAmExE,CAAC"}
|
|
1
|
+
{"version":3,"file":"llmBenchCoverage.d.ts","sourceRoot":"","sources":["../../src/data/llmBenchCoverage.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AAEH,MAAM,MAAM,aAAa,GACrB;IAAE,IAAI,EAAE,MAAM,CAAA;CAAE,GAChB;IAAE,OAAO,EAAE,QAAQ,GAAG,QAAQ,CAAA;CAAE,GAChC;IAAE,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC;AAEvB,eAAO,MAAM,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,CAqFtE,CAAC;AA4BF,MAAM,MAAM,kBAAkB,GAAG,IAAI,GAAG;IAAE,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAE1D,eAAO,MAAM,mBAAmB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAiF5E,CAAC;AA+CF;;;;6EAI6E;AAC7E,MAAM,MAAM,SAAS,GAAG,iBAAiB,GAAG,aAAa,GAAG,cAAc,CAAC;AAE3E,eAAO,MAAM,WAAW,EAAE,aAAa,CAAC,SAAS,CAIhD,CAAC;AAEF;;;;oFAIoF;AACpF,MAAM,MAAM,gBAAgB,GAAG;IAAE,SAAS,EAAE,SAAS,CAAA;CAAE,GAAG,KAAK,GAAG;IAAE,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAEpF,eAAO,MAAM,iBAAiB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAmExE,CAAC;AAgDF,MAAM,MAAM,kBAAkB,GAC1B;IAAE,YAAY,EAAE,MAAM,CAAA;CAAE,GACxB;IAAE,OAAO,EAAE,iBAAiB,GAAG,iBAAiB,CAAA;CAAE,GAClD;IAAE,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAEtB,eAAO,MAAM,mBAAmB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAmJ5E,CAAC"}
|
|
@@ -239,4 +239,121 @@ export const LLM_JUDGES_CLAIMS = {
|
|
|
239
239
|
CartographerSweep: false, // authors doc-tree summaries over code
|
|
240
240
|
StandardsCoverageEnrichment: false, // enriches standards-coverage rows
|
|
241
241
|
};
|
|
242
|
+
export const LLM_PARSER_CONTRACT = {
|
|
243
|
+
// ── WAVE-1: the four spec-named highest-stakes parsed callsites (rollout §0) ──
|
|
244
|
+
MessagingToneGate: { pending: 'contract-wave-1' }, // THE motivating defect: prompt taught "B15", parser accepts only "B15_CONTEXT_DEATH_STOP"
|
|
245
|
+
ExternalOperationGate: { pending: 'contract-wave-1' }, // parses a closed mutability/reversibility classification
|
|
246
|
+
CompletionEvaluator: { pending: 'contract-wave-1' }, // the stop-judge surface parses a closed done/blocked/continue verdict
|
|
247
|
+
InputClassifier: { pending: 'contract-wave-1' }, // parses a closed auto-approve vs relay decision
|
|
248
|
+
// ── WAVE-2: every other enumerated-verdict / decision callsite (rollout §2) ──
|
|
249
|
+
MessageSentinel: { pending: 'contract-wave-2' }, // closed intent set (pause / emergency / normal)
|
|
250
|
+
LLMSanitizer: { pending: 'contract-wave-2' }, // parses a closed sanitize verdict/decision
|
|
251
|
+
WarrantsReplyGate: { pending: 'contract-wave-2' }, // closed should-reply yes/no verdict
|
|
252
|
+
InputGuard: { pending: 'contract-wave-2' }, // closed input-coherence verdict
|
|
253
|
+
StallTriageNurse: { pending: 'contract-wave-2' }, // closed stall-triage diagnosis label
|
|
254
|
+
CommitmentSentinel: { pending: 'contract-wave-2' }, // closed commitment-detected verdict + structured envelope
|
|
255
|
+
PresenceProxy: { pending: 'contract-wave-2' }, // closed tier-3 stall verdict
|
|
256
|
+
ProjectDriftChecker: { pending: 'contract-wave-2' }, // closed on-project verdict
|
|
257
|
+
TemporalCoherenceChecker: { pending: 'contract-wave-2' }, // closed temporal-coherence verdict
|
|
258
|
+
SessionWatchdog: { pending: 'contract-wave-2' }, // closed stuck verdict
|
|
259
|
+
ResumeQueueDrainer: { pending: 'contract-wave-2' }, // closed resume-sanity verdict
|
|
260
|
+
TopicIntentArcCheck: { pending: 'contract-wave-2' }, // closed arc-check classification label
|
|
261
|
+
TelegramAdapter: { pending: 'contract-wave-2' }, // stall-confirm — closed genuinely-stalled verdict
|
|
262
|
+
SlackAdapter: { pending: 'contract-wave-2' }, // stall-confirm (byte-identical prompt to Telegram's; parity)
|
|
263
|
+
PromptGate: { pending: 'contract-wave-2' }, // closed injection-detected verdict
|
|
264
|
+
UnjustifiedStopGate: { pending: 'contract-wave-2' }, // closed stop-justified verdict
|
|
265
|
+
OverrideDetector: { pending: 'contract-wave-2' }, // closed override-intent verdict
|
|
266
|
+
TaskClassifier: { pending: 'contract-wave-2' }, // closed task-type label set
|
|
267
|
+
ResumeValidator: { pending: 'contract-wave-2' }, // closed resume-UUID match yes/no verdict
|
|
268
|
+
CoherenceReviewer: { pending: 'contract-wave-2' }, // gate-triage — closed coherence verdict
|
|
269
|
+
// ── Argued false (pinned shrink-only) — no closed-vocabulary verdict parse ──
|
|
270
|
+
// No live LLM callsite, a fixed canary, or free-text / open-set content.
|
|
271
|
+
InputDetector: {
|
|
272
|
+
false: 'attribution-manifest alias only (a legacy prompt-pattern matcher); the live matcher calls with attribution PromptGate, contracted there',
|
|
273
|
+
},
|
|
274
|
+
SessionActivitySentinel: {
|
|
275
|
+
false: 'authors a free-text activity DIGEST — the product is prose, not a closed verdict vocabulary a prompt teaches and a parser gates on',
|
|
276
|
+
},
|
|
277
|
+
PromiseBeacon: {
|
|
278
|
+
false: 'no live LLM prompt — generateStatusLine/classifyProgress hooks are unwired at the construction site; nothing parses a taught vocabulary (matches its bench-coverage exemption)',
|
|
279
|
+
},
|
|
280
|
+
InteractivePoolCanaryJudge: {
|
|
281
|
+
false: 'judges a FIXED known-answer canary probe — the expected output is a constant, so the canary is its own contract; a prompt↔parser contract test would re-test the same constant',
|
|
282
|
+
},
|
|
283
|
+
SessionSummarySentinel: {
|
|
284
|
+
false: 'extracts task/phase/files as open-set free-text FIELDS — there is no closed taught verdict vocabulary, so the B15 prompt↔parser drift cannot arise here',
|
|
285
|
+
},
|
|
286
|
+
AutoApprover: {
|
|
287
|
+
false: 'mechanical key injection + audit logging, no LLM prompt of its own; the upstream parsed decision is InputClassifier.classify(), contracted there',
|
|
288
|
+
},
|
|
289
|
+
IntegrationGate: {
|
|
290
|
+
false: 'no LLM prompt of its own — delegates to JobReflector.reflect(); zero LLM-provider callsites of its own that parse a taught vocabulary',
|
|
291
|
+
},
|
|
292
|
+
CoherenceGate: {
|
|
293
|
+
false: 'no callsite carries attribution CoherenceGate — all LLM calls flow through CoherenceReviewer.callApi(), contracted there',
|
|
294
|
+
},
|
|
295
|
+
JobReflector: {
|
|
296
|
+
false: 'reflection produces free-text content over a job — no closed taught verdict vocabulary a parser gates on (its own bench coverage is wave-3)',
|
|
297
|
+
},
|
|
298
|
+
crossModelReviewer: {
|
|
299
|
+
false: 'produces a free-text review of a SPEC document — no closed output vocabulary the prompt teaches and a parser must accept',
|
|
300
|
+
},
|
|
301
|
+
SelfKnowledgeTree: {
|
|
302
|
+
false: 'extracts self-knowledge tree fragments as content that is stored/merged — no closed verdict token gates a branch on a taught vocabulary',
|
|
303
|
+
},
|
|
304
|
+
TreeTriage: {
|
|
305
|
+
false: 'triages knowledge-tree fragments into content — no closed taught output vocabulary a parser must accept or reject',
|
|
306
|
+
},
|
|
307
|
+
TopicSummarizer: {
|
|
308
|
+
false: 'produces a free-text topic summary — the prose is the product; there is no closed vocabulary the prompt teaches and a parser gates on',
|
|
309
|
+
},
|
|
310
|
+
ContextualEvaluator: {
|
|
311
|
+
false: 'evaluates context relevance into content — no closed taught verdict vocabulary that a parser accepts a promised form of',
|
|
312
|
+
},
|
|
313
|
+
RelationshipManager: {
|
|
314
|
+
false: 'extracts relationship facts as open-set structured content — no closed taught vocabulary a prompt promises and a parser gates on',
|
|
315
|
+
},
|
|
316
|
+
StandardsConformanceReviewer: {
|
|
317
|
+
false: 'reviews artifact-vs-standard conformance as content — no closed output vocabulary the prompt teaches and a parser must accept',
|
|
318
|
+
},
|
|
319
|
+
DiscoveryEvaluator: {
|
|
320
|
+
false: 'evaluates serendipity discoveries into content — no closed taught verdict vocabulary a parser accepts a promised form of',
|
|
321
|
+
},
|
|
322
|
+
Usher: {
|
|
323
|
+
false: 'routes a turn to candidate TOPIC IDS — an OPEN, machine-supplied set, not a closed taught vocabulary the prompt fixes and a parser gates on',
|
|
324
|
+
},
|
|
325
|
+
TopicIntentExtractor: {
|
|
326
|
+
false: 'extracts a topic-intent description from a turn — free-text content, not a closed taught verdict vocabulary a parser accepts',
|
|
327
|
+
},
|
|
328
|
+
PreCompactionFlush: {
|
|
329
|
+
false: 'extracts durable facts before compaction as free-text content — no closed taught output vocabulary a parser gates on',
|
|
330
|
+
},
|
|
331
|
+
TreeSynthesis: {
|
|
332
|
+
false: 'synthesizes knowledge fragments into a free-text answer — the prose is the product, no closed taught verdict vocabulary a parser accepts',
|
|
333
|
+
},
|
|
334
|
+
LLMConflictResolver: {
|
|
335
|
+
false: 'resolves divergent multi-machine state into a merged value/content — no closed taught verdict vocabulary a prompt promises and a parser gates on',
|
|
336
|
+
},
|
|
337
|
+
openConversationBrief: {
|
|
338
|
+
false: 'generates a free-text A2A conversation brief — the prose is the product, no closed output vocabulary the prompt teaches and a parser must accept',
|
|
339
|
+
},
|
|
340
|
+
'a2a-checkin': {
|
|
341
|
+
false: 'summarizes A2A check-in threads into free-text content — no closed taught verdict vocabulary a parser accepts a promised form of',
|
|
342
|
+
},
|
|
343
|
+
'correction-learning': {
|
|
344
|
+
false: 'distills recurring corrections into a preference (content) — no closed taught output vocabulary a parser gates on',
|
|
345
|
+
},
|
|
346
|
+
'mentor-stage-b': {
|
|
347
|
+
false: 'classifies mentor signals over mentee output into differential content — no closed taught verdict vocabulary a parser gates on (its own bench coverage is wave-3)',
|
|
348
|
+
},
|
|
349
|
+
PipeSessionSpawner: {
|
|
350
|
+
false: 'spawns sessions from task descriptions — no LLM output parsed into a closed taught verdict vocabulary',
|
|
351
|
+
},
|
|
352
|
+
CartographerSweep: {
|
|
353
|
+
false: 'authors doc-tree summaries over code as free text — the prose is the product, no closed taught output vocabulary a parser gates on',
|
|
354
|
+
},
|
|
355
|
+
StandardsCoverageEnrichment: {
|
|
356
|
+
false: 'enriches standards-coverage rows with content — no closed taught verdict vocabulary a prompt promises and a parser accepts',
|
|
357
|
+
},
|
|
358
|
+
};
|
|
242
359
|
//# sourceMappingURL=llmBenchCoverage.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"llmBenchCoverage.js","sourceRoot":"","sources":["../../src/data/llmBenchCoverage.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AAOH,MAAM,CAAC,MAAM,kBAAkB,GAA4C;IACzE,mEAAmE;IACnE,eAAe,EAAE,EAAE,IAAI,EAAE,mBAAmB,EAAE;IAC9C,iBAAiB,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE;IACxC,6DAA6D;IAC7D,mBAAmB,EAAE,EAAE,IAAI,EAAE,iCAAiC,EAAE;IAChE,qBAAqB,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IACnD,YAAY,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IAC7C,iBAAiB,EAAE,EAAE,IAAI,EAAE,gBAAgB,EAAE;IAC7C,eAAe,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IAC7C,KAAK,EAAE,EAAE,IAAI,EAAE,OAAO,EAAE;IACxB,qBAAqB,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IACvD,iBAAiB,EAAE,EAAE,IAAI,EAAE,aAAa,EAAE;IAE1C,kEAAkE;IAClE,0BAA0B,EAAE;QAC1B,MAAM,EACJ,0HAA0H;KAC7H;IAED,mFAAmF;IACnF,UAAU,EAAE,EAAE,IAAI,EAAE,uBAAuB,EAAE;IAC7C,uBAAuB,EAAE,EAAE,IAAI,EAAE,iBAAiB,EAAE;IACpD,gBAAgB,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IACpD,kBAAkB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACnD,aAAa,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IAC/C,mBAAmB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACpD,wBAAwB,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IACxD,eAAe,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IACjD,kBAAkB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACnD,mBAAmB,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IACnD,eAAe,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IACnD,mEAAmE;IACnE,sEAAsE;IACtE,iEAAiE;IACjE,YAAY,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IAChD,UAAU,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IAC1C,mBAAmB,EAAE,EAAE,IAAI,EAAE,uBAAuB,EAAE;IACtD,gBAAgB,EAAE,EAAE,IAAI,EAAE,mBAAmB,EAAE;IAC/C,cAAc,EAAE,EAAE,IAAI,EAAE,iBAAiB,EAAE;IAC3C,eAAe,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IAC7C,sBAAsB,EAAE,EAAE,IAAI,EAAE,0BAA0B,EAAE;IAC5D,oBAAoB,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IAExD,yFAAyF;IACzF,eAAe,EAAE;QACf,MAAM,EACJ,yKAAyK;KAC5K;IACD,aAAa,EAAE;QACb,MAAM,EACJ,qKAAqK;KACxK;IACD,YAAY,EAAE;QACZ,MAAM,EACJ,6IAA6I;KAChJ;IACD,aAAa,EAAE;QACb,MAAM,EACJ,mLAAmL;KACtL;IACD,aAAa,EAAE;QACb,MAAM,EACJ,+LAA+L;KAClM;IAED,mDAAmD;IACnD,YAAY,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACnC,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,iBAAiB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACxC,UAAU,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACjC,eAAe,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACtC,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,4BAA4B,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACnD,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,iBAAiB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACxC,2BAA2B,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAClD,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,aAAa,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACpC,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,qBAAqB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC5C,aAAa,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACpC,gBAAgB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;CACxC,CAAC;AA8BF,MAAM,CAAC,MAAM,mBAAmB,GAAiD;IAC/E,yEAAyE;IACzE,UAAU,EAAE,IAAI;IAChB,uBAAuB,EAAE,IAAI;IAC7B,gBAAgB,EAAE,IAAI;IACtB,kBAAkB,EAAE,IAAI;IACxB,aAAa,EAAE,IAAI;IACnB,eAAe,EAAE,IAAI;IACrB,mBAAmB,EAAE,IAAI;IACzB,wBAAwB,EAAE,IAAI;IAC9B,mBAAmB,EAAE,IAAI;IACzB,eAAe,EAAE,IAAI;IACrB,kBAAkB,EAAE,IAAI;IACxB,mBAAmB,EAAE,IAAI;IACzB,YAAY,EAAE,IAAI;IAClB,eAAe,EAAE,IAAI;IACrB,sBAAsB,EAAE,IAAI;IAC5B,eAAe,EAAE,IAAI;IAErB,4DAA4D;IAC5D,UAAU,EAAE,IAAI;IAChB,qBAAqB,EAAE,IAAI,EAAE,uEAAuE;IACpG,iBAAiB,EAAE,IAAI;IACvB,mBAAmB,EAAE,IAAI;IACzB,iBAAiB,EAAE,IAAI,EAAE,oEAAoE;IAC7F,iBAAiB,EAAE,IAAI;IACvB,YAAY,EAAE,IAAI,EAAE,kDAAkD;IACtE,gBAAgB,EAAE,IAAI;IACtB,cAAc,EAAE,IAAI;IACpB,eAAe,EAAE,IAAI,EAAE,wEAAwE;IAE/F,oFAAoF;IACpF,YAAY,EAAE,IAAI;IAClB,kBAAkB,EAAE,IAAI;IACxB,iBAAiB,EAAE,IAAI;IACvB,UAAU,EAAE,IAAI;IAChB,eAAe,EAAE,IAAI;IACrB,mBAAmB,EAAE,IAAI;IACzB,mBAAmB,EAAE,IAAI;IACzB,4BAA4B,EAAE,IAAI;IAClC,kBAAkB,EAAE,IAAI;IACxB,KAAK,EAAE,IAAI;IACX,oBAAoB,EAAE,IAAI;IAC1B,kBAAkB,EAAE,IAAI;IACxB,aAAa,EAAE,IAAI;IACnB,mBAAmB,EAAE,IAAI,EAAE,sDAAsD;IACjF,qBAAqB,EAAE,IAAI;IAC3B,aAAa,EAAE,IAAI,EAAE,4BAA4B;IACjD,qBAAqB,EAAE,IAAI;IAC3B,gBAAgB,EAAE,IAAI;IAEtB,+DAA+D;IAC/D,kBAAkB,EAAE,IAAI,EAAE,0DAA0D;IACpF,iBAAiB,EAAE,IAAI,EAAE,gGAAgG;IACzH,2BAA2B,EAAE,IAAI;IAEjC,0FAA0F;IAC1F,aAAa,EAAE;QACb,KAAK,EACH,8KAA8K;KACjL;IACD,0BAA0B,EAAE;QAC1B,KAAK,EACH,iIAAiI;KACpI;IACD,YAAY,EAAE;QACZ,KAAK,EACH,2IAA2I;KAC9I;IACD,eAAe,EAAE;QACf,KAAK,EACH,mIAAmI;KACtI;IACD,aAAa,EAAE;QACb,KAAK,EACH,+HAA+H;KAClI;IACD,aAAa,EAAE;QACb,KAAK,EACH,8IAA8I;KACjJ;CACF,CAAC;AAsDF,MAAM,CAAC,MAAM,WAAW,GAA6B;IACnD,iBAAiB;IACjB,aAAa;IACb,cAAc;CACf,CAAC;AASF,MAAM,CAAC,MAAM,iBAAiB,GAA+C;IAC3E,8EAA8E;IAC9E,kEAAkE;IAClE,mBAAmB,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,wDAAwD;IAC/G,mBAAmB,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,iFAAiF;IACxI,YAAY,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,yHAAyH;IAEzK,iFAAiF;IACjF,eAAe,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,cAAc;IAC7D,aAAa,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,qBAAqB;IAClE,gBAAgB,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,yBAAyB;IACzE,eAAe,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,kIAAkI;IACjL,YAAY,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,kEAAkE;IAE9G,qDAAqD;IACrD,gBAAgB,EAAE,EAAE,SAAS,EAAE,cAAc,EAAE,EAAE,4EAA4E;IAE7H,+EAA+E;IAC/E,iFAAiF;IACjF,aAAa,EAAE,KAAK,EAAE,sCAAsC;IAC5D,UAAU,EAAE,KAAK,EAAE,iDAAiD;IACpE,uBAAuB,EAAE,KAAK,EAAE,8FAA8F;IAC9H,kBAAkB,EAAE,KAAK,EAAE,8BAA8B;IACzD,aAAa,EAAE,KAAK,EAAE,+EAA+E;IACrG,eAAe,EAAE,KAAK,EAAE,sEAAsE;IAC9F,mBAAmB,EAAE,KAAK,EAAE,8EAA8E;IAC1G,eAAe,EAAE,KAAK,EAAE,6CAA6C;IACrE,sBAAsB,EAAE,KAAK,EAAE,yEAAyE;IACxG,mBAAmB,EAAE,KAAK,EAAE,8DAA8D;IAC1F,wBAAwB,EAAE,KAAK,EAAE,oCAAoC;IACrE,kBAAkB,EAAE,KAAK,EAAE,uFAAuF;IAClH,0BAA0B,EAAE,KAAK,EAAE,kEAAkE;IAErG,qFAAqF;IACrF,UAAU,EAAE,KAAK,EAAE,sCAAsC;IACzD,YAAY,EAAE,KAAK,EAAE,qDAAqD;IAC1E,eAAe,EAAE,KAAK,EAAE,uDAAuD;IAC/E,qBAAqB,EAAE,KAAK,EAAE,wEAAwE;IACtG,iBAAiB,EAAE,KAAK,EAAE,oDAAoD;IAC9E,aAAa,EAAE,KAAK,EAAE,oDAAoD;IAC1E,iBAAiB,EAAE,KAAK,EAAE,8BAA8B;IACxD,iBAAiB,EAAE,KAAK,EAAE,6BAA6B;IACvD,YAAY,EAAE,KAAK,EAAE,sCAAsC;IAC3D,gBAAgB,EAAE,KAAK,EAAE,oCAAoC;IAC7D,cAAc,EAAE,KAAK,EAAE,uBAAuB;IAC9C,eAAe,EAAE,KAAK,EAAE,qEAAqE;IAE7F,4FAA4F;IAC5F,kBAAkB,EAAE,KAAK,EAAE,4DAA4D;IACvF,iBAAiB,EAAE,KAAK,EAAE,0BAA0B;IACpD,UAAU,EAAE,KAAK,EAAE,yBAAyB;IAC5C,eAAe,EAAE,KAAK,EAAE,qBAAqB;IAC7C,mBAAmB,EAAE,KAAK,EAAE,8BAA8B;IAC1D,mBAAmB,EAAE,KAAK,EAAE,8BAA8B;IAC1D,4BAA4B,EAAE,KAAK,EAAE,2EAA2E;IAChH,kBAAkB,EAAE,KAAK,EAAE,oCAAoC;IAC/D,KAAK,EAAE,KAAK,EAAE,oCAAoC;IAClD,oBAAoB,EAAE,KAAK,EAAE,oCAAoC;IACjE,kBAAkB,EAAE,KAAK,EAAE,2CAA2C;IACtE,aAAa,EAAE,KAAK,EAAE,2CAA2C;IACjE,mBAAmB,EAAE,KAAK,EAAE,yCAAyC;IACrE,qBAAqB,EAAE,KAAK,EAAE,sCAAsC;IACpE,aAAa,EAAE,KAAK,EAAE,kCAAkC;IACxD,qBAAqB,EAAE,KAAK,EAAE,8CAA8C;IAC5E,kBAAkB,EAAE,KAAK,EAAE,gCAAgC;IAC3D,iBAAiB,EAAE,KAAK,EAAE,uCAAuC;IACjE,2BAA2B,EAAE,KAAK,EAAE,mCAAmC;CACxE,CAAC"}
|
|
1
|
+
{"version":3,"file":"llmBenchCoverage.js","sourceRoot":"","sources":["../../src/data/llmBenchCoverage.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AAOH,MAAM,CAAC,MAAM,kBAAkB,GAA4C;IACzE,mEAAmE;IACnE,eAAe,EAAE,EAAE,IAAI,EAAE,mBAAmB,EAAE;IAC9C,iBAAiB,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE;IACxC,6DAA6D;IAC7D,mBAAmB,EAAE,EAAE,IAAI,EAAE,iCAAiC,EAAE;IAChE,qBAAqB,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IACnD,YAAY,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IAC7C,iBAAiB,EAAE,EAAE,IAAI,EAAE,gBAAgB,EAAE;IAC7C,eAAe,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IAC7C,KAAK,EAAE,EAAE,IAAI,EAAE,OAAO,EAAE;IACxB,qBAAqB,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IACvD,iBAAiB,EAAE,EAAE,IAAI,EAAE,aAAa,EAAE;IAE1C,kEAAkE;IAClE,0BAA0B,EAAE;QAC1B,MAAM,EACJ,0HAA0H;KAC7H;IAED,mFAAmF;IACnF,UAAU,EAAE,EAAE,IAAI,EAAE,uBAAuB,EAAE;IAC7C,uBAAuB,EAAE,EAAE,IAAI,EAAE,iBAAiB,EAAE;IACpD,gBAAgB,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IACpD,kBAAkB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACnD,aAAa,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IAC/C,mBAAmB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACpD,wBAAwB,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IACxD,eAAe,EAAE,EAAE,IAAI,EAAE,sBAAsB,EAAE;IACjD,kBAAkB,EAAE,EAAE,IAAI,EAAE,qBAAqB,EAAE;IACnD,mBAAmB,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IACnD,eAAe,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IACnD,mEAAmE;IACnE,sEAAsE;IACtE,iEAAiE;IACjE,YAAY,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IAChD,UAAU,EAAE,EAAE,IAAI,EAAE,oBAAoB,EAAE;IAC1C,mBAAmB,EAAE,EAAE,IAAI,EAAE,uBAAuB,EAAE;IACtD,gBAAgB,EAAE,EAAE,IAAI,EAAE,mBAAmB,EAAE;IAC/C,cAAc,EAAE,EAAE,IAAI,EAAE,iBAAiB,EAAE;IAC3C,eAAe,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE;IAC7C,sBAAsB,EAAE,EAAE,IAAI,EAAE,0BAA0B,EAAE;IAC5D,oBAAoB,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE;IAExD,yFAAyF;IACzF,eAAe,EAAE;QACf,MAAM,EACJ,yKAAyK;KAC5K;IACD,aAAa,EAAE;QACb,MAAM,EACJ,qKAAqK;KACxK;IACD,YAAY,EAAE;QACZ,MAAM,EACJ,6IAA6I;KAChJ;IACD,aAAa,EAAE;QACb,MAAM,EACJ,mLAAmL;KACtL;IACD,aAAa,EAAE;QACb,MAAM,EACJ,+LAA+L;KAClM;IAED,mDAAmD;IACnD,YAAY,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACnC,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,iBAAiB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACxC,UAAU,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACjC,eAAe,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACtC,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,4BAA4B,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACnD,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,iBAAiB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACxC,2BAA2B,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAClD,kBAAkB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACzC,aAAa,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACpC,mBAAmB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC1C,qBAAqB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IAC5C,aAAa,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;IACpC,gBAAgB,EAAE,EAAE,OAAO,EAAE,QAAQ,EAAE;CACxC,CAAC;AA8BF,MAAM,CAAC,MAAM,mBAAmB,GAAiD;IAC/E,yEAAyE;IACzE,UAAU,EAAE,IAAI;IAChB,uBAAuB,EAAE,IAAI;IAC7B,gBAAgB,EAAE,IAAI;IACtB,kBAAkB,EAAE,IAAI;IACxB,aAAa,EAAE,IAAI;IACnB,eAAe,EAAE,IAAI;IACrB,mBAAmB,EAAE,IAAI;IACzB,wBAAwB,EAAE,IAAI;IAC9B,mBAAmB,EAAE,IAAI;IACzB,eAAe,EAAE,IAAI;IACrB,kBAAkB,EAAE,IAAI;IACxB,mBAAmB,EAAE,IAAI;IACzB,YAAY,EAAE,IAAI;IAClB,eAAe,EAAE,IAAI;IACrB,sBAAsB,EAAE,IAAI;IAC5B,eAAe,EAAE,IAAI;IAErB,4DAA4D;IAC5D,UAAU,EAAE,IAAI;IAChB,qBAAqB,EAAE,IAAI,EAAE,uEAAuE;IACpG,iBAAiB,EAAE,IAAI;IACvB,mBAAmB,EAAE,IAAI;IACzB,iBAAiB,EAAE,IAAI,EAAE,oEAAoE;IAC7F,iBAAiB,EAAE,IAAI;IACvB,YAAY,EAAE,IAAI,EAAE,kDAAkD;IACtE,gBAAgB,EAAE,IAAI;IACtB,cAAc,EAAE,IAAI;IACpB,eAAe,EAAE,IAAI,EAAE,wEAAwE;IAE/F,oFAAoF;IACpF,YAAY,EAAE,IAAI;IAClB,kBAAkB,EAAE,IAAI;IACxB,iBAAiB,EAAE,IAAI;IACvB,UAAU,EAAE,IAAI;IAChB,eAAe,EAAE,IAAI;IACrB,mBAAmB,EAAE,IAAI;IACzB,mBAAmB,EAAE,IAAI;IACzB,4BAA4B,EAAE,IAAI;IAClC,kBAAkB,EAAE,IAAI;IACxB,KAAK,EAAE,IAAI;IACX,oBAAoB,EAAE,IAAI;IAC1B,kBAAkB,EAAE,IAAI;IACxB,aAAa,EAAE,IAAI;IACnB,mBAAmB,EAAE,IAAI,EAAE,sDAAsD;IACjF,qBAAqB,EAAE,IAAI;IAC3B,aAAa,EAAE,IAAI,EAAE,4BAA4B;IACjD,qBAAqB,EAAE,IAAI;IAC3B,gBAAgB,EAAE,IAAI;IAEtB,+DAA+D;IAC/D,kBAAkB,EAAE,IAAI,EAAE,0DAA0D;IACpF,iBAAiB,EAAE,IAAI,EAAE,gGAAgG;IACzH,2BAA2B,EAAE,IAAI;IAEjC,0FAA0F;IAC1F,aAAa,EAAE;QACb,KAAK,EACH,8KAA8K;KACjL;IACD,0BAA0B,EAAE;QAC1B,KAAK,EACH,iIAAiI;KACpI;IACD,YAAY,EAAE;QACZ,KAAK,EACH,2IAA2I;KAC9I;IACD,eAAe,EAAE;QACf,KAAK,EACH,mIAAmI;KACtI;IACD,aAAa,EAAE;QACb,KAAK,EACH,+HAA+H;KAClI;IACD,aAAa,EAAE;QACb,KAAK,EACH,8IAA8I;KACjJ;CACF,CAAC;AAsDF,MAAM,CAAC,MAAM,WAAW,GAA6B;IACnD,iBAAiB;IACjB,aAAa;IACb,cAAc;CACf,CAAC;AASF,MAAM,CAAC,MAAM,iBAAiB,GAA+C;IAC3E,8EAA8E;IAC9E,kEAAkE;IAClE,mBAAmB,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,wDAAwD;IAC/G,mBAAmB,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,iFAAiF;IACxI,YAAY,EAAE,EAAE,SAAS,EAAE,iBAAiB,EAAE,EAAE,yHAAyH;IAEzK,iFAAiF;IACjF,eAAe,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,cAAc;IAC7D,aAAa,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,qBAAqB;IAClE,gBAAgB,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,yBAAyB;IACzE,eAAe,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,kIAAkI;IACjL,YAAY,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,EAAE,kEAAkE;IAE9G,qDAAqD;IACrD,gBAAgB,EAAE,EAAE,SAAS,EAAE,cAAc,EAAE,EAAE,4EAA4E;IAE7H,+EAA+E;IAC/E,iFAAiF;IACjF,aAAa,EAAE,KAAK,EAAE,sCAAsC;IAC5D,UAAU,EAAE,KAAK,EAAE,iDAAiD;IACpE,uBAAuB,EAAE,KAAK,EAAE,8FAA8F;IAC9H,kBAAkB,EAAE,KAAK,EAAE,8BAA8B;IACzD,aAAa,EAAE,KAAK,EAAE,+EAA+E;IACrG,eAAe,EAAE,KAAK,EAAE,sEAAsE;IAC9F,mBAAmB,EAAE,KAAK,EAAE,8EAA8E;IAC1G,eAAe,EAAE,KAAK,EAAE,6CAA6C;IACrE,sBAAsB,EAAE,KAAK,EAAE,yEAAyE;IACxG,mBAAmB,EAAE,KAAK,EAAE,8DAA8D;IAC1F,wBAAwB,EAAE,KAAK,EAAE,oCAAoC;IACrE,kBAAkB,EAAE,KAAK,EAAE,uFAAuF;IAClH,0BAA0B,EAAE,KAAK,EAAE,kEAAkE;IAErG,qFAAqF;IACrF,UAAU,EAAE,KAAK,EAAE,sCAAsC;IACzD,YAAY,EAAE,KAAK,EAAE,qDAAqD;IAC1E,eAAe,EAAE,KAAK,EAAE,uDAAuD;IAC/E,qBAAqB,EAAE,KAAK,EAAE,wEAAwE;IACtG,iBAAiB,EAAE,KAAK,EAAE,oDAAoD;IAC9E,aAAa,EAAE,KAAK,EAAE,oDAAoD;IAC1E,iBAAiB,EAAE,KAAK,EAAE,8BAA8B;IACxD,iBAAiB,EAAE,KAAK,EAAE,6BAA6B;IACvD,YAAY,EAAE,KAAK,EAAE,sCAAsC;IAC3D,gBAAgB,EAAE,KAAK,EAAE,oCAAoC;IAC7D,cAAc,EAAE,KAAK,EAAE,uBAAuB;IAC9C,eAAe,EAAE,KAAK,EAAE,qEAAqE;IAE7F,4FAA4F;IAC5F,kBAAkB,EAAE,KAAK,EAAE,4DAA4D;IACvF,iBAAiB,EAAE,KAAK,EAAE,0BAA0B;IACpD,UAAU,EAAE,KAAK,EAAE,yBAAyB;IAC5C,eAAe,EAAE,KAAK,EAAE,qBAAqB;IAC7C,mBAAmB,EAAE,KAAK,EAAE,8BAA8B;IAC1D,mBAAmB,EAAE,KAAK,EAAE,8BAA8B;IAC1D,4BAA4B,EAAE,KAAK,EAAE,2EAA2E;IAChH,kBAAkB,EAAE,KAAK,EAAE,oCAAoC;IAC/D,KAAK,EAAE,KAAK,EAAE,oCAAoC;IAClD,oBAAoB,EAAE,KAAK,EAAE,oCAAoC;IACjE,kBAAkB,EAAE,KAAK,EAAE,2CAA2C;IACtE,aAAa,EAAE,KAAK,EAAE,2CAA2C;IACjE,mBAAmB,EAAE,KAAK,EAAE,yCAAyC;IACrE,qBAAqB,EAAE,KAAK,EAAE,sCAAsC;IACpE,aAAa,EAAE,KAAK,EAAE,kCAAkC;IACxD,qBAAqB,EAAE,KAAK,EAAE,8CAA8C;IAC5E,kBAAkB,EAAE,KAAK,EAAE,gCAAgC;IAC3D,iBAAiB,EAAE,KAAK,EAAE,uCAAuC;IACjE,2BAA2B,EAAE,KAAK,EAAE,mCAAmC;CACxE,CAAC;AAqDF,MAAM,CAAC,MAAM,mBAAmB,GAAiD;IAC/E,iFAAiF;IACjF,iBAAiB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,2FAA2F;IAC9I,qBAAqB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,0DAA0D;IACjH,mBAAmB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,uEAAuE;IAC5H,eAAe,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,iDAAiD;IAElG,gFAAgF;IAChF,eAAe,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,iDAAiD;IAClG,YAAY,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,4CAA4C;IAC1F,iBAAiB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,qCAAqC;IACxF,UAAU,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,iCAAiC;IAC7E,gBAAgB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,sCAAsC;IACxF,kBAAkB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,2DAA2D;IAC/G,aAAa,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,8BAA8B;IAC7E,mBAAmB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,4BAA4B;IACjF,wBAAwB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,oCAAoC;IAC9F,eAAe,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,uBAAuB;IACxE,kBAAkB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,+BAA+B;IACnF,mBAAmB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,wCAAwC;IAC7F,eAAe,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,mDAAmD;IACpG,YAAY,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,8DAA8D;IAC5G,UAAU,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,oCAAoC;IAChF,mBAAmB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,gCAAgC;IACrF,gBAAgB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,iCAAiC;IACnF,cAAc,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,6BAA6B;IAC7E,eAAe,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,0CAA0C;IAC3F,iBAAiB,EAAE,EAAE,OAAO,EAAE,iBAAiB,EAAE,EAAE,yCAAyC;IAE5F,+EAA+E;IAC/E,yEAAyE;IACzE,aAAa,EAAE;QACb,KAAK,EACH,yIAAyI;KAC5I;IACD,uBAAuB,EAAE;QACvB,KAAK,EACH,oIAAoI;KACvI;IACD,aAAa,EAAE;QACb,KAAK,EACH,gLAAgL;KACnL;IACD,0BAA0B,EAAE;QAC1B,KAAK,EACH,gLAAgL;KACnL;IACD,sBAAsB,EAAE;QACtB,KAAK,EACH,yJAAyJ;KAC5J;IACD,YAAY,EAAE;QACZ,KAAK,EACH,kJAAkJ;KACrJ;IACD,eAAe,EAAE;QACf,KAAK,EACH,uIAAuI;KAC1I;IACD,aAAa,EAAE;QACb,KAAK,EACH,0HAA0H;KAC7H;IACD,YAAY,EAAE;QACZ,KAAK,EACH,6IAA6I;KAChJ;IACD,kBAAkB,EAAE;QAClB,KAAK,EACH,0HAA0H;KAC7H;IACD,iBAAiB,EAAE;QACjB,KAAK,EACH,yIAAyI;KAC5I;IACD,UAAU,EAAE;QACV,KAAK,EACH,mHAAmH;KACtH;IACD,eAAe,EAAE;QACf,KAAK,EACH,uIAAuI;KAC1I;IACD,mBAAmB,EAAE;QACnB,KAAK,EACH,yHAAyH;KAC5H;IACD,mBAAmB,EAAE;QACnB,KAAK,EACH,kIAAkI;KACrI;IACD,4BAA4B,EAAE;QAC5B,KAAK,EACH,+HAA+H;KAClI;IACD,kBAAkB,EAAE;QAClB,KAAK,EACH,0HAA0H;KAC7H;IACD,KAAK,EAAE;QACL,KAAK,EACH,6IAA6I;KAChJ;IACD,oBAAoB,EAAE;QACpB,KAAK,EACH,8HAA8H;KACjI;IACD,kBAAkB,EAAE;QAClB,KAAK,EACH,sHAAsH;KACzH;IACD,aAAa,EAAE;QACb,KAAK,EACH,0IAA0I;KAC7I;IACD,mBAAmB,EAAE;QACnB,KAAK,EACH,kJAAkJ;KACrJ;IACD,qBAAqB,EAAE;QACrB,KAAK,EACH,kJAAkJ;KACrJ;IACD,aAAa,EAAE;QACb,KAAK,EACH,kIAAkI;KACrI;IACD,qBAAqB,EAAE;QACrB,KAAK,EACH,mHAAmH;KACtH;IACD,gBAAgB,EAAE;QAChB,KAAK,EACH,mKAAmK;KACtK;IACD,kBAAkB,EAAE;QAClB,KAAK,EACH,uGAAuG;KAC1G;IACD,iBAAiB,EAAE;QACjB,KAAK,EACH,oIAAoI;KACvI;IACD,2BAA2B,EAAE;QAC3B,KAAK,EACH,4HAA4H;KAC/H;CACF,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "./builtin-manifest.schema.json",
|
|
3
3
|
"schemaVersion": 1,
|
|
4
|
-
"generatedAt": "2026-07-03T23:
|
|
5
|
-
"instarVersion": "1.3.
|
|
4
|
+
"generatedAt": "2026-07-03T23:25:24.796Z",
|
|
5
|
+
"instarVersion": "1.3.741",
|
|
6
6
|
"entryCount": 202,
|
|
7
7
|
"entries": {
|
|
8
8
|
"hook:session-start": {
|
|
@@ -360,3 +360,203 @@ export const LLM_JUDGES_CLAIMS: Readonly<Record<string, JudgesClaimsFlag>> = {
|
|
|
360
360
|
CartographerSweep: false, // authors doc-tree summaries over code
|
|
361
361
|
StandardsCoverageEnrichment: false, // enriches standards-coverage rows
|
|
362
362
|
};
|
|
363
|
+
|
|
364
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
365
|
+
// Prompt↔parser contract standard (defect class 1 — docs/specs/prompt-parser-
|
|
366
|
+
// contract-standard.md §4)
|
|
367
|
+
//
|
|
368
|
+
// The `contract` axis of the program's shared per-callsite metadata record
|
|
369
|
+
// (class-closure-gate.md §"Program-shared machinery"). It co-locates in THIS
|
|
370
|
+
// file with the bench-coverage record it extends, exactly like the sibling
|
|
371
|
+
// `untrustedInput` (authority-clause) and `judgesClaims` (evidence-bar) axes.
|
|
372
|
+
//
|
|
373
|
+
// THE FIELD IS REQUIRED AND EXPLICIT FOR EVERY COMPONENT_CATEGORY KEY — there is
|
|
374
|
+
// NO DEFAULT. A callsite whose output is machine-parsed into a CLOSED, taught
|
|
375
|
+
// verdict/decision vocabulary (the B15 failure surface: prompt teaches "B15",
|
|
376
|
+
// parser accepts only "B15_CONTEXT_DEATH_STOP") either NAMES its contract-test
|
|
377
|
+
// file (`{ contractTest: '<path>' }`, covered) or is queued in the shrink-only
|
|
378
|
+
// pending set (`{ pending: 'contract-wave-1' | 'contract-wave-2' }`). A callsite
|
|
379
|
+
// with NO such closed-vocabulary parse — no live LLM prompt, a fixed canary, or
|
|
380
|
+
// free-text / open-set CONTENT consumed as data (summaries, extractions,
|
|
381
|
+
// syntheses, briefs, reviews, open-set id routing) — is `{ false: '<reason>' }`
|
|
382
|
+
// and pinned shrink-only in
|
|
383
|
+
// tests/unit/parser-contract-classification-ratchet.test.ts. A silent omission
|
|
384
|
+
// is red CI, so the flag can NEVER default toward the un-contracted state.
|
|
385
|
+
//
|
|
386
|
+
// POLARITY (spec §3: "Undeclared content is hazard-scanned by default"): the
|
|
387
|
+
// default pulls TOWARD "needs a contract" — a callsite is `false` only with an
|
|
388
|
+
// argued reason, exactly like the sibling `untrustedInput` axis defaults toward
|
|
389
|
+
// `true`.
|
|
390
|
+
//
|
|
391
|
+
// WAVE-1 is the spec-named SHIPPED-FIX set (rollout §0/§1): the four
|
|
392
|
+
// highest-stakes parsed callsites — MessagingToneGate (the motivating B15
|
|
393
|
+
// defect), ExternalOperationGate, CompletionEvaluator (stop-judge), and
|
|
394
|
+
// InputClassifier. It is pinned as a seed floor in the ratchet so a
|
|
395
|
+
// highest-stakes callsite can never silently slip out of scope. WAVE-2 is every
|
|
396
|
+
// other enumerated-verdict/decision callsite, graduating on the shrink-only
|
|
397
|
+
// schedule (rollout §2).
|
|
398
|
+
//
|
|
399
|
+
// DARK / REPORT-ONLY: this record is build-time metadata read ONLY by the new
|
|
400
|
+
// pinned ratchet. Nothing here is a contract test yet (those render the REAL
|
|
401
|
+
// production prompt and need the live-builder render refactor, deferred to its
|
|
402
|
+
// own A/B-gated increments — spec rollout §0/§1). The pending set IS the report
|
|
403
|
+
// (spec §4: "report-only inventory happens by construction").
|
|
404
|
+
//
|
|
405
|
+
// A cross-check lint flags any GATE/SENTINEL-category callsite marked `false`
|
|
406
|
+
// for review (these categories most often parse a verdict) — see the ratchet's
|
|
407
|
+
// REVIEWED_FALSE_PARSER_GATE pin.
|
|
408
|
+
// ───────────────────────────────────────────────────────────────────────────
|
|
409
|
+
|
|
410
|
+
export type ParserContractFlag =
|
|
411
|
+
| { contractTest: string }
|
|
412
|
+
| { pending: 'contract-wave-1' | 'contract-wave-2' }
|
|
413
|
+
| { false: string };
|
|
414
|
+
|
|
415
|
+
export const LLM_PARSER_CONTRACT: Readonly<Record<string, ParserContractFlag>> = {
|
|
416
|
+
// ── WAVE-1: the four spec-named highest-stakes parsed callsites (rollout §0) ──
|
|
417
|
+
MessagingToneGate: { pending: 'contract-wave-1' }, // THE motivating defect: prompt taught "B15", parser accepts only "B15_CONTEXT_DEATH_STOP"
|
|
418
|
+
ExternalOperationGate: { pending: 'contract-wave-1' }, // parses a closed mutability/reversibility classification
|
|
419
|
+
CompletionEvaluator: { pending: 'contract-wave-1' }, // the stop-judge surface parses a closed done/blocked/continue verdict
|
|
420
|
+
InputClassifier: { pending: 'contract-wave-1' }, // parses a closed auto-approve vs relay decision
|
|
421
|
+
|
|
422
|
+
// ── WAVE-2: every other enumerated-verdict / decision callsite (rollout §2) ──
|
|
423
|
+
MessageSentinel: { pending: 'contract-wave-2' }, // closed intent set (pause / emergency / normal)
|
|
424
|
+
LLMSanitizer: { pending: 'contract-wave-2' }, // parses a closed sanitize verdict/decision
|
|
425
|
+
WarrantsReplyGate: { pending: 'contract-wave-2' }, // closed should-reply yes/no verdict
|
|
426
|
+
InputGuard: { pending: 'contract-wave-2' }, // closed input-coherence verdict
|
|
427
|
+
StallTriageNurse: { pending: 'contract-wave-2' }, // closed stall-triage diagnosis label
|
|
428
|
+
CommitmentSentinel: { pending: 'contract-wave-2' }, // closed commitment-detected verdict + structured envelope
|
|
429
|
+
PresenceProxy: { pending: 'contract-wave-2' }, // closed tier-3 stall verdict
|
|
430
|
+
ProjectDriftChecker: { pending: 'contract-wave-2' }, // closed on-project verdict
|
|
431
|
+
TemporalCoherenceChecker: { pending: 'contract-wave-2' }, // closed temporal-coherence verdict
|
|
432
|
+
SessionWatchdog: { pending: 'contract-wave-2' }, // closed stuck verdict
|
|
433
|
+
ResumeQueueDrainer: { pending: 'contract-wave-2' }, // closed resume-sanity verdict
|
|
434
|
+
TopicIntentArcCheck: { pending: 'contract-wave-2' }, // closed arc-check classification label
|
|
435
|
+
TelegramAdapter: { pending: 'contract-wave-2' }, // stall-confirm — closed genuinely-stalled verdict
|
|
436
|
+
SlackAdapter: { pending: 'contract-wave-2' }, // stall-confirm (byte-identical prompt to Telegram's; parity)
|
|
437
|
+
PromptGate: { pending: 'contract-wave-2' }, // closed injection-detected verdict
|
|
438
|
+
UnjustifiedStopGate: { pending: 'contract-wave-2' }, // closed stop-justified verdict
|
|
439
|
+
OverrideDetector: { pending: 'contract-wave-2' }, // closed override-intent verdict
|
|
440
|
+
TaskClassifier: { pending: 'contract-wave-2' }, // closed task-type label set
|
|
441
|
+
ResumeValidator: { pending: 'contract-wave-2' }, // closed resume-UUID match yes/no verdict
|
|
442
|
+
CoherenceReviewer: { pending: 'contract-wave-2' }, // gate-triage — closed coherence verdict
|
|
443
|
+
|
|
444
|
+
// ── Argued false (pinned shrink-only) — no closed-vocabulary verdict parse ──
|
|
445
|
+
// No live LLM callsite, a fixed canary, or free-text / open-set content.
|
|
446
|
+
InputDetector: {
|
|
447
|
+
false:
|
|
448
|
+
'attribution-manifest alias only (a legacy prompt-pattern matcher); the live matcher calls with attribution PromptGate, contracted there',
|
|
449
|
+
},
|
|
450
|
+
SessionActivitySentinel: {
|
|
451
|
+
false:
|
|
452
|
+
'authors a free-text activity DIGEST — the product is prose, not a closed verdict vocabulary a prompt teaches and a parser gates on',
|
|
453
|
+
},
|
|
454
|
+
PromiseBeacon: {
|
|
455
|
+
false:
|
|
456
|
+
'no live LLM prompt — generateStatusLine/classifyProgress hooks are unwired at the construction site; nothing parses a taught vocabulary (matches its bench-coverage exemption)',
|
|
457
|
+
},
|
|
458
|
+
InteractivePoolCanaryJudge: {
|
|
459
|
+
false:
|
|
460
|
+
'judges a FIXED known-answer canary probe — the expected output is a constant, so the canary is its own contract; a prompt↔parser contract test would re-test the same constant',
|
|
461
|
+
},
|
|
462
|
+
SessionSummarySentinel: {
|
|
463
|
+
false:
|
|
464
|
+
'extracts task/phase/files as open-set free-text FIELDS — there is no closed taught verdict vocabulary, so the B15 prompt↔parser drift cannot arise here',
|
|
465
|
+
},
|
|
466
|
+
AutoApprover: {
|
|
467
|
+
false:
|
|
468
|
+
'mechanical key injection + audit logging, no LLM prompt of its own; the upstream parsed decision is InputClassifier.classify(), contracted there',
|
|
469
|
+
},
|
|
470
|
+
IntegrationGate: {
|
|
471
|
+
false:
|
|
472
|
+
'no LLM prompt of its own — delegates to JobReflector.reflect(); zero LLM-provider callsites of its own that parse a taught vocabulary',
|
|
473
|
+
},
|
|
474
|
+
CoherenceGate: {
|
|
475
|
+
false:
|
|
476
|
+
'no callsite carries attribution CoherenceGate — all LLM calls flow through CoherenceReviewer.callApi(), contracted there',
|
|
477
|
+
},
|
|
478
|
+
JobReflector: {
|
|
479
|
+
false:
|
|
480
|
+
'reflection produces free-text content over a job — no closed taught verdict vocabulary a parser gates on (its own bench coverage is wave-3)',
|
|
481
|
+
},
|
|
482
|
+
crossModelReviewer: {
|
|
483
|
+
false:
|
|
484
|
+
'produces a free-text review of a SPEC document — no closed output vocabulary the prompt teaches and a parser must accept',
|
|
485
|
+
},
|
|
486
|
+
SelfKnowledgeTree: {
|
|
487
|
+
false:
|
|
488
|
+
'extracts self-knowledge tree fragments as content that is stored/merged — no closed verdict token gates a branch on a taught vocabulary',
|
|
489
|
+
},
|
|
490
|
+
TreeTriage: {
|
|
491
|
+
false:
|
|
492
|
+
'triages knowledge-tree fragments into content — no closed taught output vocabulary a parser must accept or reject',
|
|
493
|
+
},
|
|
494
|
+
TopicSummarizer: {
|
|
495
|
+
false:
|
|
496
|
+
'produces a free-text topic summary — the prose is the product; there is no closed vocabulary the prompt teaches and a parser gates on',
|
|
497
|
+
},
|
|
498
|
+
ContextualEvaluator: {
|
|
499
|
+
false:
|
|
500
|
+
'evaluates context relevance into content — no closed taught verdict vocabulary that a parser accepts a promised form of',
|
|
501
|
+
},
|
|
502
|
+
RelationshipManager: {
|
|
503
|
+
false:
|
|
504
|
+
'extracts relationship facts as open-set structured content — no closed taught vocabulary a prompt promises and a parser gates on',
|
|
505
|
+
},
|
|
506
|
+
StandardsConformanceReviewer: {
|
|
507
|
+
false:
|
|
508
|
+
'reviews artifact-vs-standard conformance as content — no closed output vocabulary the prompt teaches and a parser must accept',
|
|
509
|
+
},
|
|
510
|
+
DiscoveryEvaluator: {
|
|
511
|
+
false:
|
|
512
|
+
'evaluates serendipity discoveries into content — no closed taught verdict vocabulary a parser accepts a promised form of',
|
|
513
|
+
},
|
|
514
|
+
Usher: {
|
|
515
|
+
false:
|
|
516
|
+
'routes a turn to candidate TOPIC IDS — an OPEN, machine-supplied set, not a closed taught vocabulary the prompt fixes and a parser gates on',
|
|
517
|
+
},
|
|
518
|
+
TopicIntentExtractor: {
|
|
519
|
+
false:
|
|
520
|
+
'extracts a topic-intent description from a turn — free-text content, not a closed taught verdict vocabulary a parser accepts',
|
|
521
|
+
},
|
|
522
|
+
PreCompactionFlush: {
|
|
523
|
+
false:
|
|
524
|
+
'extracts durable facts before compaction as free-text content — no closed taught output vocabulary a parser gates on',
|
|
525
|
+
},
|
|
526
|
+
TreeSynthesis: {
|
|
527
|
+
false:
|
|
528
|
+
'synthesizes knowledge fragments into a free-text answer — the prose is the product, no closed taught verdict vocabulary a parser accepts',
|
|
529
|
+
},
|
|
530
|
+
LLMConflictResolver: {
|
|
531
|
+
false:
|
|
532
|
+
'resolves divergent multi-machine state into a merged value/content — no closed taught verdict vocabulary a prompt promises and a parser gates on',
|
|
533
|
+
},
|
|
534
|
+
openConversationBrief: {
|
|
535
|
+
false:
|
|
536
|
+
'generates a free-text A2A conversation brief — the prose is the product, no closed output vocabulary the prompt teaches and a parser must accept',
|
|
537
|
+
},
|
|
538
|
+
'a2a-checkin': {
|
|
539
|
+
false:
|
|
540
|
+
'summarizes A2A check-in threads into free-text content — no closed taught verdict vocabulary a parser accepts a promised form of',
|
|
541
|
+
},
|
|
542
|
+
'correction-learning': {
|
|
543
|
+
false:
|
|
544
|
+
'distills recurring corrections into a preference (content) — no closed taught output vocabulary a parser gates on',
|
|
545
|
+
},
|
|
546
|
+
'mentor-stage-b': {
|
|
547
|
+
false:
|
|
548
|
+
'classifies mentor signals over mentee output into differential content — no closed taught verdict vocabulary a parser gates on (its own bench coverage is wave-3)',
|
|
549
|
+
},
|
|
550
|
+
PipeSessionSpawner: {
|
|
551
|
+
false:
|
|
552
|
+
'spawns sessions from task descriptions — no LLM output parsed into a closed taught verdict vocabulary',
|
|
553
|
+
},
|
|
554
|
+
CartographerSweep: {
|
|
555
|
+
false:
|
|
556
|
+
'authors doc-tree summaries over code as free text — the prose is the product, no closed taught output vocabulary a parser gates on',
|
|
557
|
+
},
|
|
558
|
+
StandardsCoverageEnrichment: {
|
|
559
|
+
false:
|
|
560
|
+
'enriches standards-coverage rows with content — no closed taught verdict vocabulary a prompt promises and a parser accepts',
|
|
561
|
+
},
|
|
562
|
+
};
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: minor -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
The mechanical arm of the **"The Prompt and the Parser Are One Contract"** standard
|
|
9
|
+
(`docs/specs/prompt-parser-contract-standard.md`, defect class 1 / `prompt-parser-contract`
|
|
10
|
+
closure), shipped as a self-contained DARK increment — no runtime wiring, no operator-gated
|
|
11
|
+
registry text, no config key, and **no change to any live prompt or parser**. It is the
|
|
12
|
+
structural answer to the 2026-07-02 INSTAR-Bench v2 finding: the tone gate's PROMPT taught
|
|
13
|
+
models the short rule id (`B15`) while its production PARSER accepts only the full identifier
|
|
14
|
+
(`B15_CONTEXT_DEATH_STOP`) and fails closed on the short form — so every model through every
|
|
15
|
+
door obeyed the prompt and "failed." A single CI test comparing what the prompt promises
|
|
16
|
+
against what the parser accepts would have made that defect unrepresentable.
|
|
17
|
+
|
|
18
|
+
This increment ships **the shared contract library + the per-callsite classification + its CI
|
|
19
|
+
ratchet only**:
|
|
20
|
+
|
|
21
|
+
- A **shared contract library** (`src/core/promptContract.ts`): the `PromptContract` manifest
|
|
22
|
+
type (the co-located machine-readable promise a prose-shaped callsite carries next to its
|
|
23
|
+
prompt) and `deriveRejectedForms` — a pure generator for the counter-examples a contract
|
|
24
|
+
test feeds the REAL parser to prove its fail-closed behavior (case-mutation,
|
|
25
|
+
prefix-truncation at each separator — the exact `B15` shape — and separator-stripping, plus
|
|
26
|
+
hand-picked extras, minus any form that collides with a promised token). Machine-derived by
|
|
27
|
+
design: a hand-only reject list invites trivial rejects that prove nothing.
|
|
28
|
+
- The **`contract` classification** (`LLM_PARSER_CONTRACT` in `src/data/llmBenchCoverage.ts`):
|
|
29
|
+
the `contract` axis of the program's ONE shared per-callsite metadata record (sibling of the
|
|
30
|
+
authority-clause `untrustedInput` and evidence-bar `judgesClaims` axes), required-explicit
|
|
31
|
+
for every LLM component — no default, so a silent omission is red CI and the flag can never
|
|
32
|
+
default toward the un-contracted state. The four spec-named highest-stakes parsed callsites
|
|
33
|
+
(tone gate, external-op gate, stop judge, input classifier) seed `contract-wave-1`; every
|
|
34
|
+
other enumerated-verdict callsite is `contract-wave-2`; a callsite with no closed-vocabulary
|
|
35
|
+
parse (no live prompt, a fixed canary, or free-text/open-set content) is an argued `false`.
|
|
36
|
+
- A **classification ratchet** (`tests/unit/parser-contract-classification-ratchet.test.ts`):
|
|
37
|
+
required-explicit + no-dangling + valid-wave + the wave-1 seed pinned as a floor (a
|
|
38
|
+
highest-stakes callsite can never silently slip scope) + the pending set pinned shrink-only
|
|
39
|
+
+ the argued-false set pinned shrink-only with a real-reason floor + a gate/sentinel
|
|
40
|
+
cross-check. Same pinned-baseline family as `llm-bench-coverage-ratchet` and the sibling
|
|
41
|
+
`untrusted-input-classification-ratchet`.
|
|
42
|
+
|
|
43
|
+
Deliberately OUT of scope (not orphan deferrals — see `upgrades/side-effects/`):
|
|
44
|
+
|
|
45
|
+
- The **per-callsite contract tests** and the **live-builder render refactor** (spec rollout
|
|
46
|
+
§0/§1). A contract test renders the REAL production prompt through an exported pure render
|
|
47
|
+
function; several production builders are private instance methods with live deps and need
|
|
48
|
+
that refactor, which TOUCHES live parsing code. It is deferred to its own A/B-gated
|
|
49
|
+
increments, one callsite at a time — never inside this dark inventory increment.
|
|
50
|
+
- The **runtime contract-drift warning** (spec Frontloaded Decision #3) — a prompt-build-time
|
|
51
|
+
signal that also touches the live builders; it lands with the single-sourcing migrations.
|
|
52
|
+
- The **registry / constitution text** (spec §1) — operator-gated; ships ONLY with Justin's
|
|
53
|
+
explicit sign-off. It is DRAFTED in the spec; this run does not edit the standards registry.
|
|
54
|
+
|
|
55
|
+
## What to Tell Your User
|
|
56
|
+
|
|
57
|
+
Nothing changes for you right now — this ships dark, and it is maintainer-only machinery (a
|
|
58
|
+
no-op on your install unless you develop instar itself). What it does is give my own AI checks
|
|
59
|
+
a safety net: the words a check's prompt tells a model to answer with, and the words the code
|
|
60
|
+
behind that check will actually accept, are now recorded together and held in sync by an
|
|
61
|
+
automatic test. That closes a class of bug where a check would quietly reject every correct
|
|
62
|
+
answer because the two halves had drifted apart — a mistake that was entirely mine, never the
|
|
63
|
+
model's. The real per-check tests and any wording changes come later, one at a time, and only
|
|
64
|
+
after careful review.
|
|
65
|
+
|
|
66
|
+
## Summary of New Capabilities
|
|
67
|
+
|
|
68
|
+
None active for end users in this increment — everything ships dark and additive. (For instar
|
|
69
|
+
maintainers: a shared prompt↔parser contract library plus a required per-callsite `contract`
|
|
70
|
+
classification with a shrink-only ratchet and a pinned highest-stakes seed floor, so a
|
|
71
|
+
machine-parsed callsite can never silently ship a taught output vocabulary its own parser
|
|
72
|
+
rejects.)
|
|
73
|
+
|
|
74
|
+
## Evidence
|
|
75
|
+
|
|
76
|
+
- `npx vitest run tests/unit/parser-contract-classification-ratchet.test.ts tests/unit/promptContract.test.ts tests/unit/llm-bench-coverage-ratchet.test.ts`
|
|
77
|
+
→ **3 files, 26 tests, 0 failures** (required-explicit + no-dangling + wave-1 seed floor +
|
|
78
|
+
shrink-only pending + shrink-only argued-false + real-reason floor + gate/sentinel
|
|
79
|
+
cross-check + the derive-rejected-forms mutation logic incl. the B15 prefix shape; the
|
|
80
|
+
pre-existing coverage ratchet still green against the additive record).
|
|
81
|
+
- `npx tsc --noEmit` → exit 0. `npm run lint` → exit 0.
|
|
82
|
+
- Dark-by-construction: `LLM_PARSER_CONTRACT` is build-time metadata read only by the new
|
|
83
|
+
pinned ratchet; `promptContract.ts` has no runtime caller. It changes NO prompt text and
|
|
84
|
+
wires NO runtime parser — verified by the absence of any src import of the new module
|
|
85
|
+
outside tests.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Side-Effects Review — Prompt↔Parser Contract Standard (defect class 1, dark increment)
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `prompt-parser-contract-standard`
|
|
4
|
+
**Date:** `2026-07-03`
|
|
5
|
+
**Author:** `echo`
|
|
6
|
+
**Tier:** `1 (dark increment — additive library + classification + pinned ratchet; no runtime wiring, no live prompt/parser change, no operator-gated text)`
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
The mechanical arm of the "The Prompt and the Parser Are One Contract" standard (defect class 1 closure; `docs/specs/prompt-parser-contract-standard.md`), shipped as a self-contained DARK increment. It ships:
|
|
11
|
+
|
|
12
|
+
1. **`src/core/promptContract.ts`** — the shared prompt↔parser contract library. The `PromptContract` manifest type (a co-located, machine-readable promise carried next to a prose-shaped prompt), the `ContractForm` election type, and `deriveRejectedForms(vocabulary, extras, options)` — a PURE generator for the fail-closed counter-examples a per-callsite contract test feeds the REAL parser (case-mutation, prefix-truncation at each separator — the exact B15 shape — separator-stripping, plus hand-picked extras, minus any form that collides with a promised token). No runtime caller in this increment.
|
|
13
|
+
2. **`src/data/llmBenchCoverage.ts`** (additive) — `LLM_PARSER_CONTRACT`, the `contract` axis of the program's shared per-callsite metadata record. Required-explicit for every `COMPONENT_CATEGORY` key; the four spec-named highest-stakes callsites seed `contract-wave-1`, other enumerated-verdict callsites are `contract-wave-2`, and a no-closed-vocabulary callsite is `{ false: '<reason>' }`.
|
|
14
|
+
3. **`tests/unit/parser-contract-classification-ratchet.test.ts`** — required-explicit + no-dangling + valid-wave + wave-1 seed floor + shrink-only pending + shrink-only argued-false + real-reason floor + the gate/sentinel cross-check lint.
|
|
15
|
+
4. **`tests/unit/promptContract.test.ts`** — the derive-rejected-forms mutation logic (incl. the B15 prefix shape, collision-exclusion, purity) and the manifest type surface.
|
|
16
|
+
5. **`docs/specs/prompt-parser-contract-standard.md`** + **`.eli16.md`** — the converged spec and its plain-English overview.
|
|
17
|
+
|
|
18
|
+
## What is DELIBERATELY out of scope (not orphan deferrals)
|
|
19
|
+
|
|
20
|
+
- **The per-callsite contract tests + the live-builder render refactor (spec rollout §0/§1).** A contract test renders the REAL production prompt through an exported pure render function; several production builders (e.g. `MessagingToneGate`'s prompt builder) are private instance methods with live deps and need that render refactor, which TOUCHES live parsing code. Per the CAREFUL constraint on this defect class (live, load-bearing parsers), that refactor is deferred to its own A/B-gated increments, one callsite at a time, so live parse behavior is UNCHANGED on this merge. The four siblings deferred their behavioral/render arms for exactly this reason; this increment mirrors that.
|
|
21
|
+
- **The runtime contract-drift warning (spec Frontloaded Decision #3 / "Decision points touched" (a)).** A signal-only prompt-build-time assertion that also touches the live builders; it lands with the by-construction single-sourcing migrations, not this inventory increment.
|
|
22
|
+
- **The registry / constitution text (spec §1).** Operator-gated — ships ONLY with Justin's explicit sign-off (spec front-matter `operator-gate`). The registry entry is DRAFTED in the spec; this run does not write `docs/STANDARDS-REGISTRY.md`.
|
|
23
|
+
|
|
24
|
+
## Decision-point inventory
|
|
25
|
+
|
|
26
|
+
- `src/core/promptContract.ts` — **add** — a pure library (types + string/array helpers). NO runtime caller in this increment (dark by construction); it changes NO prompt text and NO parser until a per-callsite contract test + render refactor lands through its own gate. No allow/deny surface.
|
|
27
|
+
- `LLM_PARSER_CONTRACT` classification (`src/data/llmBenchCoverage.ts`) — **add** — build-time metadata only. Read by the new pinned ratchet; no runtime consumer. A SIGNAL producer (which callsites parse a taught output vocabulary), never a runtime authority.
|
|
28
|
+
- The two new vitest tests — **add** — CI-only pinned baselines (same family as `llm-bench-coverage-ratchet`). The ratchet gates the build (adding an unclassified callsite / growing the pending or false set silently is red CI), never a runtime path.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## 1. Over-block
|
|
33
|
+
|
|
34
|
+
**What legitimate inputs does this change reject that it shouldn't?**
|
|
35
|
+
|
|
36
|
+
None at runtime — nothing is wired to a runtime allow/deny path, and no live prompt or parser changes, so no model answer that was accepted before is rejected now. At CI/build time the only new red-CI surfaces are: (a) adding a new LLM component to `COMPONENT_CATEGORY` without an explicit `contract` classification, (b) growing the pending or argued-false set without editing the pinned baseline, and (c) marking a gate/sentinel `false` without the reviewed allowlist. All three are intentional, self-describing failures with fix-instructions in the assertion message — the "visible, reviewed act" the standard exists to force, not an over-block of legitimate work.
|
|
37
|
+
|
|
38
|
+
## 2. Under-block
|
|
39
|
+
|
|
40
|
+
**What failure modes does this still miss?**
|
|
41
|
+
|
|
42
|
+
- This increment adds NO test that a callsite's prompt and parser are ACTUALLY coherent — that is the per-callsite contract test, deferred with its render refactor. So a classified-`pending` callsite can still ship a drifted prompt↔parser until its contract test graduates. This is the known staged-rollout gap, tracked in the spec's rollout §0→§3, not a silent miss — the pending set IS the honest inventory of that remaining work.
|
|
43
|
+
- Classification accuracy is author-judged against a stated criterion (a CLOSED taught verdict vocabulary → needs a contract; free-text / open-set / no-live-prompt → `false`). A mislabel of a gate/sentinel is partly caught by the cross-check (the highest-risk category); a reflector/job mislabeled `false` would pass. The argued-false shrink-only pin makes every such call visible and reviewed at PR time; this seeding PR IS that review.
|
|
44
|
+
|
|
45
|
+
## 3. Level-of-abstraction fit
|
|
46
|
+
|
|
47
|
+
**Is this at the right layer?**
|
|
48
|
+
|
|
49
|
+
Yes. The contract library is a pure `src/core/` module (the layer where shared prompt/parser scaffolding belongs — sibling to `promptClauses.ts` from the authority-clause standard); the classification extends the ONE shared metadata record the program mandates (`src/data/llmBenchCoverage.ts`), not a new parallel registry; the ratchet is in the established pinned-baseline vitest family. No higher gate already owns "does the prompt's taught vocabulary match the parser"; no lower primitive is duplicated.
|
|
50
|
+
|
|
51
|
+
## 4. Signal vs authority compliance
|
|
52
|
+
|
|
53
|
+
The classification and the ratchet are SIGNALS (which callsites parse a taught vocabulary; is the pending/false inventory shrink-only) — they inform the build and the reviewer, never grant or deny a runtime action. `deriveRejectedForms` returns an array; it is inert until a contract test consumes it. No model-produced field is wired to satisfy any authorization check. The library changes no live parser's accept/reject decision.
|
|
54
|
+
|
|
55
|
+
## 5. Interactions with existing systems
|
|
56
|
+
|
|
57
|
+
- **`llm-bench-coverage-ratchet` / `untrusted-input-classification-ratchet`** — unaffected: `LLM_PARSER_CONTRACT` is a NEW additive export; the existing `LLM_BENCH_COVERAGE` and `LLM_UNTRUSTED_INPUT` records and their ratchets are untouched (all green post-change). The argued-false membership is consistent with the sibling axes' "no live judging/parsing callsite" reasoning where they overlap (PromiseBeacon, InteractivePoolCanaryJudge, AutoApprover, IntegrationGate, CoherenceGate, InputDetector), and adds the free-text-content reflectors/summarizers on top (which parse no closed verdict vocabulary).
|
|
58
|
+
- **`lint-scrape-fixture-realness`** — the new helper is named `deriveRejectedForms` (not `parse*`/`scrape*`), so it is deliberately outside that lint's `parse*`/`scrape*` surface; it consumes no untrusted real-world text (it operates on a callsite's own declared vocabulary constant).
|
|
59
|
+
- **green-PR auto-merge protected paths** — `src/data/llmBenchCoverage.ts` is already in the class-closure agent-authored-artifact predicate; the pinned ratchet baselines route every future classification/pending/false edit to operator review while a fully-conforming graduation (pending→contractTest) keeps auto-merge (program-shared machinery).
|
|
60
|
+
|
|
61
|
+
## 6. Failure modes / rollback
|
|
62
|
+
|
|
63
|
+
Pure additive TypeScript + tests + docs. Rollback = revert the commit; nothing persists state, nothing runs at runtime, no migration, no config key (no agent-side config key exists or is wanted — repo posture only, so no Migration Parity work). `tsc --noEmit` clean, `npm run lint` exit 0, all new + adjacent ratchets green under bounded single-file vitest runs (the machine has intermittent external CPU pressure; verification used bounded runs per the CI-as-gate pattern, full suite gated by CI).
|
|
64
|
+
|
|
65
|
+
## 7. Second-pass reviewer
|
|
66
|
+
|
|
67
|
+
Self-review (bounded machine-pressure build). The change is additive, dark, and test-pinned; no runtime surface and no live prompt/parser touched. Key self-checks: (1) confirmed `deriveRejectedForms` never emits a promised token as a "rejected form" (the collision-exclusion test), so a future contract test's accept/reject assertions cannot contradict; (2) confirmed the classification covers all 53 `COMPONENT_CATEGORY` keys exactly once (the required-explicit + no-dangling ratchet asserts it); (3) confirmed the wave-1 seed is exactly the four spec-named highest-stakes callsites; (4) confirmed no `src/` file outside tests imports `promptContract.ts`, so the increment is dark by construction.
|