@typedstandards/civic-typed-harness 0.4.0 → 0.4.1

package/CHANGELOG.md

@@ -3,6 +3,97 @@
 Factual record of what changed per published version. Section references are
 to the Typed Standards specification unless noted otherwise.
 
+## 0.4.1 — 2026-09-12
+
+Patch: one existing export changes its output for one input shape, and no
+export is added, removed or retyped. That is measured across the release
+boundary rather than asserted — the published `0.4.0` and this build each
+expose 49 runtime exports from the package root, none added, none removed,
+none changing runtime type, and no declared top-level name in `dist/*.d.ts`
+differing between them. The tarball carries the same 28 files, and 22 of the
+24 under `dist/` are byte-identical to `0.4.0`; the two that are not are
+`dist/rubric/adversarial-eval-core.js` and its `.d.ts`, the latter differing
+by a doc comment alone.
+
+**What this release claims about golden bytes, and what it does not.** Every
+golden byte is unchanged. This time that sentence is not the evidence, and
+saying so is the point: **no golden fixture reaches the export that changed.**
+`buildEvaluationPrompt` is named in two test files, both under `src/rubric/`,
+and in neither golden suite. `__fixtures__/website-golden.json` carries no
+`queries` array at all; `__fixtures__/reference-golden.json` carries eight and
+**zero** occurrences of the `failed` key that is this release's only new
+branch. So the golden suites are green for this change the way they are green
+for a change to a file they never import, and their green is not evidence
+about it. This is the same narrowing
+[civic-ai-tools#199](https://github.com/npstorey/civic-ai-tools/issues/199)
+records for the graph builder, met from the other side.
+
+What **is** evidence is a direct measurement of the changed export either side
+of the release boundary: `buildEvaluationPrompt` imported from the published
+`0.4.0` and from this build, run over the same package.
+
+- A package in which **no entry records `failed: true`** renders
+  **byte-identical** on both — 492 bytes, `sha256 ed77c84e…` — over five entry
+  shapes: `resultRows: 12`, `resultRows: 0`, no `resultRows` at all,
+  `failed: false`, and a bare `failureKind` carrying no `failed`.
+- The same package plus one entry recorded `failed: true` renders
+  **differently** (539 bytes against 607). That case is the reason the first
+  one is worth reporting: an instrument that answers "identical" whatever it is
+  fed has not measured anything, and this one is shown answering "different"
+  on the shape it must flag.
+
+**The evaluator reads a rejected call as rejected**
+([civic-ai-tools#203](https://github.com/npstorey/civic-ai-tools/issues/203),
+Wave N11 P-H1). Behaviour change, confined to one input shape.
+
+- **The line that changes.** A `queries[]` entry the producer recorded as
+  rejected rendered `→ ? rows` — the same line an entry whose row count was
+  simply never recorded produces. A rejection and an unrecorded count were one
+  line to the evaluator, and the rubric asks that evaluator to cross-check the
+  answer's figures against the data the tool calls returned, so the conflation
+  invited it to read data that was never returned as merely uncounted. Its
+  score becomes a signed attestation. The entry now reads `→ REJECTED by the
+  source — no data was returned, and no row count is claimed`.
+- **`failed: true` is the whole assertion.** `failureKind` is declared on the
+  shape and **deliberately not rendered**: it is producer-controlled text, and
+  the turn built here is read by a model whose score is signed, so the
+  rejection is stated in this module's own fixed words, interpolating nothing
+  the producer wrote and claiming no row count.
+- **Absence stays absence.** An entry that records no outcome renders exactly
+  as it did before the fields existed. `failed: false` is not a rejection, and
+  a `failureKind` with no `failed` is not a rejection — the posture
+  `ToolCallSummary.failed` already takes in `capture/data-sources.ts`.
+- **No dependency moves.** The two keys are read off a locally-declared shape
+  rather than imported, following that same file: produce-core `0.3.0`, which
+  the lockfile resolves, names neither key. No manifest change, no lockfile
+  bump, no new dependency.
+
+**`RUBRIC_VERSION_SHA256` does not move, and the claim around it is corrected
+rather than left standing**
+([civic-ai-tools#207](https://github.com/npstorey/civic-ai-tools/issues/207)).
+The hash is `sha256Hex(EVALUATION_RUBRIC)` — the rubric text alone. It does not
+cover `buildEvaluationPrompt`, the template that assembles that rubric with the
+package's tool calls, data sources and model into the turn the evaluator
+actually reads. So this release changes what an evaluator is shown without
+changing the value an attestation pins as `methodology.promptSetVersion`: two
+evaluations either side of it carry the same value, measured here as
+`b62b193c992fd430cc82dff9a80a057bbe0b2b656fbdace450ca1a5d99fffa02` on both the
+published `0.4.0` and this build.
+
+- **The overclaim is retired in the same change.** ADR-0015 §3, Q26's
+  resolution note and the two module comments said the value pinned the whole
+  prompt set that produced the scores. They now say it pins the rubric
+  *wording*, and that the prompt template is outside the hash. The phrase they
+  used for the wider claim appears nowhere in the tree at this release.
+- **The gap is filed, not resolved.** #207 records what the hash covers, what
+  it does not, and the two spec-level options — widen the hash, or add a second
+  field — with no recommendation. Which one is right is a specification
+  decision, not a package's.
+
+A consumer sees none of this until its own lockfile moves: `^0.4.0` admits
+`0.4.1`, so the manifest range need not change and the refresh is the whole of
+the upgrade.
+
 ## 0.4.0 — 2026-09-04
 
 Minor: two optional input fields are added and no existing export is removed

package/dist/rubric/adversarial-eval-core.d.ts

@@ -6,7 +6,10 @@ export declare const EVALUATION_CRITERIA: readonly ["dataSourceIdentification",
 export declare const EVALUATION_RUBRIC = "You are an independent evaluator assessing an AI-generated civic data analysis.\n\nYou will receive:\n1. The original prompt (question asked)\n2. The tool calls made (MCP queries to Socrata open data portals)\n3. The AI's final output/analysis\n\nEvaluate the analysis against these 6 criteria, scoring each 1-10:\n\na) **Data Source Identification** (1-10): Does the output correctly identify the data source(s) and time period? Are dataset IDs, portal domains, and date ranges accurate?\n\nb) **Quantitative Claim Support** (1-10): Are the quantitative claims (numbers, percentages, rankings) supported by the data returned in the tool calls? Cross-check key figures against the raw data.\n\nc) **Confounders and Bias** (1-10): Does the analysis acknowledge obvious confounders, selection biases, or framing issues? Are there lurking variables or cherry-picked timeframes?\n\nd) **Geographic Scope** (1-10): Is the geographic scope appropriate for the question? Does the analysis avoid over-generalizing from one jurisdiction?\n\ne) **Limitations Noted** (1-10): Are limitations and caveats noted? Does the analysis flag data quality issues, missing fields, or incomplete coverage?\n\nf) **Contradictory Conclusion** (1-10): Could the same data reasonably support a contradictory conclusion? Does the analysis consider alternative interpretations?\n\nRespond in this exact JSON format (no markdown fences, just raw JSON):\n{\n  \"dataSourceIdentification\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"quantitativeClaimSupport\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"confoundersAndBias\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"geographicScope\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"limitationsNoted\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"contradictoryConclusion\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n  \"overallScore\": <average of all 6 scores, one decimal>,\n  \"assessment\": \"<2-4 sentence overall assessment>\"\n}";
 /** The methodology's `promptSetVersion`: SHA-256 of the rubric text, computed
  *  once at module load. Any wording change to the rubric produces a new
- *  version, so an attestation pins the exact prompt set that scored it. */
+ *  version, so an attestation pins the rubric WORDING that scored it — and
+ *  that wording only. The prompt template around it (`buildEvaluationPrompt`)
+ *  is outside this hash, so one value does not identify one set of evaluator
+ *  inputs (civic-ai-tools#207). */
 export declare const RUBRIC_VERSION_SHA256: string;
 /** Build the user-turn evaluation content from a package's signed fields.
  *  Takes `RecordPackage` — produce-core 0.3.0's settlement-era name for the

package/dist/rubric/adversarial-eval-core.js

@@ -5,8 +5,14 @@
 // brief §1. The model runner and DB/blob emission (`adversarial-eval.ts`)
 // stay app-side — this module never performs I/O.
 //
-// ⚠ RUBRIC_VERSION_SHA256 is a Q26-pinned version hash: attestations pin the
-// exact prompt set that scored them by this value. The rubric text below is
+// ⚠ RUBRIC_VERSION_SHA256 is a Q26-pinned version hash over THE RUBRIC TEXT
+// ALONE (`sha256Hex(EVALUATION_RUBRIC)`). It does NOT cover
+// `buildEvaluationPrompt` below — the template that assembles the rubric with
+// the package's tool calls, data sources and model into the turn the evaluator
+// actually reads. So two evaluations either side of a change to that template
+// carry the SAME `methodology.promptSetVersion` while the evaluator saw
+// different inputs; civic-ai-tools#207 files that gap for the specification to
+// settle, and this module does not resolve it. The rubric text below is
 // byte-exact with the reference implementation (the relocation changes no
 // hashes), and the test suite asserts the literal digest. Any wording change
 // to EVALUATION_RUBRIC is a NEW rubric version — never edit the text without
@@ -57,8 +63,24 @@ Respond in this exact JSON format (no markdown fences, just raw JSON):
 }`;
 /** The methodology's `promptSetVersion`: SHA-256 of the rubric text, computed
  *  once at module load. Any wording change to the rubric produces a new
- *  version, so an attestation pins the exact prompt set that scored it. */
+ *  version, so an attestation pins the rubric WORDING that scored it — and
+ *  that wording only. The prompt template around it (`buildEvaluationPrompt`)
+ *  is outside this hash, so one value does not identify one set of evaluator
+ *  inputs (civic-ai-tools#207). */
 export const RUBRIC_VERSION_SHA256 = sha256Hex(EVALUATION_RUBRIC);
+/** Did the producer record this entry as REJECTED by the source? Only a
+ *  literal `failed: true` is a rejection (see `RecordedCallOutcome`). */
+function wasRecordedRejected(query) {
+    return query.failed === true;
+}
+/** What the evaluator is told about a call the source rejected. A FIXED
+ *  string: it interpolates nothing the producer wrote, and it claims no row
+ *  count — there were no rows to count, which is neither "zero rows" nor "an
+ *  unknown number of rows". The rubric asks the evaluator to cross-check
+ *  figures against the data returned in the tool calls, so a rejected call
+ *  reading `→ ? rows` invited it to treat absent data as merely unrecorded
+ *  (civic-ai-tools#203). */
+const REJECTED_CALL_RENDERING = 'REJECTED by the source — no data was returned, and no row count is claimed';
 /** Build the user-turn evaluation content from a package's signed fields.
  *  Takes `RecordPackage` — produce-core 0.3.0's settlement-era name for the
  *  same object (spec Appendix J, migration class alias-and-deprecate). The
@@ -70,7 +92,10 @@ export function buildEvaluationPrompt(pkg) {
         const argStr = Object.entries(q.arguments)
             .map(([k, v]) => `${k}=${JSON.stringify(v)}`)
             .join(', ');
-        return `  ${i + 1}. ${q.tool}(${argStr}) → ${q.resultRows ?? '?'} rows`;
+        const outcome = wasRecordedRejected(q)
+            ? REJECTED_CALL_RENDERING
+            : `${q.resultRows ?? '?'} rows`;
+        return `  ${i + 1}. ${q.tool}(${argStr}) → ${outcome}`;
     })
         .join('\n');
     const dataSources = pkg.dataSources

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@typedstandards/civic-typed-harness",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "Civic-domain harness for Typed Standards evidence packages — the DOMAIN layer of the ADR-0021 format/domain line: civic PROV-O vocabulary and provenance builder, datHere envelope-policy derivations, and the capture machinery (trace builder, skill-metadata extraction, data-source population, adversarial rubric core). Derives; @typedstandards/produce-core assembles.",
   "license": "MIT",
   "author": "Nathan Storey",