@typedstandards/civic-typed-harness 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +219 -0
- package/dist/capture/provenance.d.ts +4 -1
- package/dist/capture/provenance.js +42 -25
- package/dist/format/dathere.d.ts +31 -5
- package/dist/format/dathere.js +37 -10
- package/dist/format/sources.d.ts +20 -3
- package/dist/format/sources.js +8 -1
- package/dist/format/vocabulary.d.ts +77 -0
- package/dist/format/vocabulary.js +95 -9
- package/dist/rubric/adversarial-eval-core.d.ts +4 -1
- package/dist/rubric/adversarial-eval-core.js +29 -4
- package/package.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,225 @@
|
|
|
3
3
|
Factual record of what changed per published version. Section references are
|
|
4
4
|
to the Typed Standards specification unless noted otherwise.
|
|
5
5
|
|
|
6
|
+
## 0.5.0 — 2026-09-18
|
|
7
|
+
|
|
8
|
+
The list is derived from
|
|
9
|
+
`git log 33a3aff..<this release> -- packages/civic-typed-harness/` — `33a3aff`
|
|
10
|
+
is the 0.4.1 release bump — so it is the commit range rather than a
|
|
11
|
+
recollection.
|
|
12
|
+
Three commits landed after that bump and had no entry; this phase's two
|
|
13
|
+
changes join them.
|
|
14
|
+
|
|
15
|
+
**Together, the two changes below make a record name the servers the run
|
|
16
|
+
actually had, and say nothing where it does not know.** They are the harness
|
|
17
|
+
half of
|
|
18
|
+
[civic-ai-tools#205](https://github.com/npstorey/civic-ai-tools/issues/205)
|
|
19
|
+
and
|
|
20
|
+
[civic-ai-tools-website#449](https://github.com/npstorey/civic-ai-tools-website/issues/449);
|
|
21
|
+
the caller half (a registry built from the instance's own configuration, and
|
|
22
|
+
the packager passing the list) lands in the reference application.
|
|
23
|
+
|
|
24
|
+
### The environment extension can name every server, not only the skill's
|
|
25
|
+
|
|
26
|
+
- **`buildDatHereEnvironment` accepts a list of MCP servers**, additively. Its
|
|
27
|
+
second parameter is now `string | readonly DatHereMcpServer[] | undefined`:
|
|
28
|
+
the bare string is the pre-0.5.0 shape — the trace's skill-fetch URL, the one
|
|
29
|
+
server the builder could name — and a list is emitted in the order the caller
|
|
30
|
+
gives, each entry carrying `name` only when the caller supplied one. Spec
|
|
31
|
+
§8.7.1 requirement 3 always specified `mcpServers` as "an array of objects
|
|
32
|
+
with `url` and optional `name`"; until now the harness could only ever put
|
|
33
|
+
one object in it.
|
|
34
|
+
- **The single-URL shape is byte-identical to before**, and a test pins those
|
|
35
|
+
exact bytes rather than deep-equality, because an already-signed datHere
|
|
36
|
+
package reproduces through this function. A one-entry list with no `name`
|
|
37
|
+
produces the same bytes as the string, which is the same statement from the
|
|
38
|
+
other side.
|
|
39
|
+
- **`DatHerePolicyInput` gains `mcpServers`**, consulted in preference to
|
|
40
|
+
`skillMcpServerUrl` and falling back to it, so `deriveDatHereEnvelopeFields`
|
|
41
|
+
carries the list through without any existing caller changing.
|
|
42
|
+
- **New export:** the `DatHereMcpServer` interface. An entry whose `url` is
|
|
43
|
+
empty contributes nothing — the same rule the single-URL shape always
|
|
44
|
+
applied: a server with no address is stated by absence, never by an empty
|
|
45
|
+
string.
|
|
46
|
+
|
|
47
|
+
### A source agent names the configured address, or names no address at all
|
|
48
|
+
|
|
49
|
+
- **`CivicSourceInfo.serverUrl` is optional.** The address a run reached is the
|
|
50
|
+
operator's configuration, not a fact the harness knows. A caller passes a
|
|
51
|
+
registry whose entries carry the addresses that instance is pointed at.
|
|
52
|
+
- **`buildProvenanceGraph` omits `civic:serverUrl` when it has no address.**
|
|
53
|
+
Two shapes reach that: a registry entry that carries no `serverUrl`, and a
|
|
54
|
+
source id the registry does not carry at all. Both now emit an agent with no
|
|
55
|
+
`civic:serverUrl` key — the key's absence, not an empty string — so the
|
|
56
|
+
signed bytes say the address is unknown instead of naming a server the
|
|
57
|
+
analysis may never have contacted. When an address IS known the key keeps its
|
|
58
|
+
position in the emitted key order, which is the byte contract on the legacy
|
|
59
|
+
chain.
|
|
60
|
+
- **The source-id fallback is dropped.** An agent for a source outside the
|
|
61
|
+
registry used to carry `civic:serverUrl` equal to its own source id — a
|
|
62
|
+
string that was never an address. That is the asserted-default class
|
|
63
|
+
([civic-ai-tools#129](https://github.com/npstorey/civic-ai-tools/issues/129)'s
|
|
64
|
+
family): a stand-in standing in a signed record for a fact the run had and
|
|
65
|
+
did not write.
|
|
66
|
+
- **`CIVIC_SOURCE_REGISTRY` keeps its three `serverUrl` values**, and the
|
|
67
|
+
comment now says what they are for: the reference deployment's own default
|
|
68
|
+
addresses for display, and — on the two aggregate sources — the companions of
|
|
69
|
+
`aggregatePortalUrl`. They are not what a record asserts about a run.
|
|
70
|
+
- **Type-level consequence, stated rather than glossed.** Widening
|
|
71
|
+
`serverUrl` to optional is additive for a caller that BUILDS a registry and
|
|
72
|
+
narrowing for one that READS `info.serverUrl` and expects a `string`. A
|
|
73
|
+
TypeScript consumer doing the latter has to handle `undefined`.
|
|
74
|
+
|
|
75
|
+
**Golden bytes.** Two of the three fixtures are untouched;
|
|
76
|
+
`__fixtures__/website-golden.json` and `__fixtures__/reference-golden.json` are
|
|
77
|
+
frozen reference captures and no byte of either is edited. The 2026-08-01
|
|
78
|
+
website capture carries one MCP agent for a source its registry does not know,
|
|
79
|
+
and that agent's `civic:serverUrl` is the dropped fallback, so the suites lift
|
|
80
|
+
the capture over exactly that one key — derived from the registry, with the
|
|
81
|
+
number of lifted nodes asserted, the same device already used for the two
|
|
82
|
+
Appendix J literals and for the synthetic prompt hash. The lifted graph is
|
|
83
|
+
inside the hashed bytes of the ninth golden-reproduction case, so
|
|
84
|
+
`__fixtures__/span-carrying-golden.json`'s two recorded hashes are regenerated,
|
|
85
|
+
intentionally and by that one dropped key: `envelopeHash` `eaa41755…b9b87d` →
|
|
86
|
+
`2a57c911…c0af89`, `contentHashSha256` `cd036569…0e0202` → `e14adecf…a6af5f`.
|
|
87
|
+
The fixture records the reason beside them. **A package signed before this
|
|
88
|
+
change whose graph carried the source-id fallback no longer reproduces at the
|
|
89
|
+
defaults** — that is the intended consequence of dropping an asserted default,
|
|
90
|
+
and it is named here so nobody reads a moved hash as an accident.
|
|
91
|
+
|
|
92
|
+
### Already on `main` since 0.4.1, and unreleased
|
|
93
|
+
|
|
94
|
+
- **The test files type-check**
|
|
95
|
+
([civic-ai-tools#197](https://github.com/npstorey/civic-ai-tools/issues/197),
|
|
96
|
+
`0a861b1`). `tsconfig.test.json` extends the build config, emits nothing, and
|
|
97
|
+
adds the Node types the suite needs; the package's `typecheck` script runs
|
|
98
|
+
both configs, and `src/the-suite-type-checks.test.ts` guards that every test
|
|
99
|
+
file is inside the checking config and none is inside the emitting one. The
|
|
100
|
+
build config now sets `types: []`, so shipped source still cannot reach a
|
|
101
|
+
Node global and type-check. No runtime export changes.
|
|
102
|
+
- **The `civic:` vocabulary is named, and a rejected call states its elapsed**
|
|
103
|
+
([civic-ai-tools#199](https://github.com/npstorey/civic-ai-tools/issues/199)
|
|
104
|
+
§2, `e4e32e7`). **Fourteen new `CIVIC_TERM_*` exports** —
|
|
105
|
+
`CIVIC_TERM_CONTENT_HASH`, `CIVIC_TERM_SERVER_URL`, `CIVIC_TERM_SOURCE_ID`,
|
|
106
|
+
`CIVIC_TERM_URL`, `CIVIC_TERM_PROMPT_TOKENS`, `CIVIC_TERM_COMPLETION_TOKENS`,
|
|
107
|
+
`CIVIC_TERM_TOOL_NAME`, `CIVIC_TERM_OPERATION_TYPE`, `CIVIC_TERM_DATASET_ID`,
|
|
108
|
+
`CIVIC_TERM_PORTAL_DOMAIN`, `CIVIC_TERM_DATASET_URL`,
|
|
109
|
+
`CIVIC_TERM_CROISSANT_METADATA_URL`, `CIVIC_TERM_RESPONSE_ROWS`,
|
|
110
|
+
`CIVIC_TERM_DURATION_MS` — replacing inline literals at twenty sites in
|
|
111
|
+
`capture/provenance.ts`, with no byte moving (a computed key inserts in the
|
|
112
|
+
same position with the same value). `purity.test.ts`'s guard derives its
|
|
113
|
+
universe rather than listing literals. The same commit corrected two comments
|
|
114
|
+
that said a rejected span carries no `tool.duration_ms`: the reference
|
|
115
|
+
producer records it, so a rejected call's activity emits `civic:durationMs`
|
|
116
|
+
beside `civic:failed`, the duration in its 0.3.1 position and the two markers
|
|
117
|
+
appended after it.
|
|
118
|
+
- **A golden case that carries `mcp_tool_call` spans**
|
|
119
|
+
([civic-ai-tools#199](https://github.com/npstorey/civic-ai-tools/issues/199)
|
|
120
|
+
§1, `92efbae`). `__fixtures__/span-carrying-golden.json` — a ninth
|
|
121
|
+
golden-reproduction case, in its own file so the verbatim upstream copy stays
|
|
122
|
+
a copy. The other eight carry zero spans of any kind, so until it existed the
|
|
123
|
+
byte-compat suite was green for everything the tool-span loop does. Test-only;
|
|
124
|
+
no export changes.
|
|
125
|
+
|
|
126
|
+
**Correction to the 0.4.0 entry.** That entry's type-level-gate bullet says
|
|
127
|
+
"`tsconfig.json` excludes `src/**/*.test.ts`, so the suite type-checks nothing
|
|
128
|
+
— a test can drive a field the type does not have and `npm run typecheck` stays
|
|
129
|
+
green." **That has been false since `0a861b1`.** The suite is type-checked by
|
|
130
|
+
`tsconfig.test.json`, `npm run typecheck` runs it, and a type error in a test
|
|
131
|
+
file fails that gate. The sentence is true of 0.4.0 as published and is left in
|
|
132
|
+
place as the record of it; this is the forward correction.
|
|
133
|
+
|
|
134
|
+
## 0.4.1 — 2026-09-12
|
|
135
|
+
|
|
136
|
+
Patch: one existing export changes its output for one input shape, and no
|
|
137
|
+
export is added, removed or retyped. That is measured across the release
|
|
138
|
+
boundary rather than asserted — the published `0.4.0` and this build each
|
|
139
|
+
expose 49 runtime exports from the package root, none added, none removed,
|
|
140
|
+
none changing runtime type, and no declared top-level name in `dist/*.d.ts`
|
|
141
|
+
differing between them. The tarball carries the same 28 files, and 22 of the
|
|
142
|
+
24 under `dist/` are byte-identical to `0.4.0`; the two that are not are
|
|
143
|
+
`dist/rubric/adversarial-eval-core.js` and its `.d.ts`, the latter differing
|
|
144
|
+
by a doc comment alone.
|
|
145
|
+
|
|
146
|
+
**What this release claims about golden bytes, and what it does not.** Every
|
|
147
|
+
golden byte is unchanged. This time that sentence is not the evidence, and
|
|
148
|
+
saying so is the point: **no golden fixture reaches the export that changed.**
|
|
149
|
+
`buildEvaluationPrompt` is named in two test files, both under `src/rubric/`,
|
|
150
|
+
and in neither golden suite. `__fixtures__/website-golden.json` carries no
|
|
151
|
+
`queries` array at all; `__fixtures__/reference-golden.json` carries eight and
|
|
152
|
+
**zero** occurrences of the `failed` key that is this release's only new
|
|
153
|
+
branch. So the golden suites are green for this change the way they are green
|
|
154
|
+
for a change to a file they never import, and their green is not evidence
|
|
155
|
+
about it. This is the same narrowing
|
|
156
|
+
[civic-ai-tools#199](https://github.com/npstorey/civic-ai-tools/issues/199)
|
|
157
|
+
records for the graph builder, met from the other side.
|
|
158
|
+
|
|
159
|
+
What **is** evidence is a direct measurement of the changed export either side
|
|
160
|
+
of the release boundary: `buildEvaluationPrompt` imported from the published
|
|
161
|
+
`0.4.0` and from this build, run over the same package.
|
|
162
|
+
|
|
163
|
+
- A package in which **no entry records `failed: true`** renders
|
|
164
|
+
**byte-identical** on both — 492 bytes, `sha256 ed77c84e…` — over five entry
|
|
165
|
+
shapes: `resultRows: 12`, `resultRows: 0`, no `resultRows` at all,
|
|
166
|
+
`failed: false`, and a bare `failureKind` carrying no `failed`.
|
|
167
|
+
- The same package plus one entry recorded `failed: true` renders
|
|
168
|
+
**differently** (539 bytes against 607). That case is the reason the first
|
|
169
|
+
one is worth reporting: an instrument that answers "identical" whatever it is
|
|
170
|
+
fed has not measured anything, and this one is shown answering "different"
|
|
171
|
+
on the shape it must flag.
|
|
172
|
+
|
|
173
|
+
**The evaluator reads a rejected call as rejected**
|
|
174
|
+
([civic-ai-tools#203](https://github.com/npstorey/civic-ai-tools/issues/203),
|
|
175
|
+
Wave N11 P-H1). Behaviour change, confined to one input shape.
|
|
176
|
+
|
|
177
|
+
- **The line that changes.** A `queries[]` entry the producer recorded as
|
|
178
|
+
rejected rendered `→ ? rows` — the same line an entry whose row count was
|
|
179
|
+
simply never recorded produces. A rejection and an unrecorded count were one
|
|
180
|
+
line to the evaluator, and the rubric asks that evaluator to cross-check the
|
|
181
|
+
answer's figures against the data the tool calls returned, so the conflation
|
|
182
|
+
invited it to read data that was never returned as merely uncounted. Its
|
|
183
|
+
score becomes a signed attestation. The entry now reads `→ REJECTED by the
|
|
184
|
+
source — no data was returned, and no row count is claimed`.
|
|
185
|
+
- **`failed: true` is the whole assertion.** `failureKind` is declared on the
|
|
186
|
+
shape and **deliberately not rendered**: it is producer-controlled text, and
|
|
187
|
+
the turn built here is read by a model whose score is signed, so the
|
|
188
|
+
rejection is stated in this module's own fixed words, interpolating nothing
|
|
189
|
+
the producer wrote and claiming no row count.
|
|
190
|
+
- **Absence stays absence.** An entry that records no outcome renders exactly
|
|
191
|
+
as it did before the fields existed. `failed: false` is not a rejection, and
|
|
192
|
+
a `failureKind` with no `failed` is not a rejection — the posture
|
|
193
|
+
`ToolCallSummary.failed` already takes in `capture/data-sources.ts`.
|
|
194
|
+
- **No dependency moves.** The two keys are read off a locally-declared shape
|
|
195
|
+
rather than imported, following that same file: produce-core `0.3.0`, which
|
|
196
|
+
the lockfile resolves, names neither key. No manifest change, no lockfile
|
|
197
|
+
bump, no new dependency.
|
|
198
|
+
|
|
199
|
+
**`RUBRIC_VERSION_SHA256` does not move, and the claim around it is corrected
|
|
200
|
+
rather than left standing**
|
|
201
|
+
([civic-ai-tools#207](https://github.com/npstorey/civic-ai-tools/issues/207)).
|
|
202
|
+
The hash is `sha256Hex(EVALUATION_RUBRIC)` — the rubric text alone. It does not
|
|
203
|
+
cover `buildEvaluationPrompt`, the template that assembles that rubric with the
|
|
204
|
+
package's tool calls, data sources and model into the turn the evaluator
|
|
205
|
+
actually reads. So this release changes what an evaluator is shown without
|
|
206
|
+
changing the value an attestation pins as `methodology.promptSetVersion`: two
|
|
207
|
+
evaluations either side of it carry the same value, measured here as
|
|
208
|
+
`b62b193c992fd430cc82dff9a80a057bbe0b2b656fbdace450ca1a5d99fffa02` on both the
|
|
209
|
+
published `0.4.0` and this build.
|
|
210
|
+
|
|
211
|
+
- **The overclaim is retired in the same change.** ADR-0015 §3, Q26's
|
|
212
|
+
resolution note and the two module comments said the value pinned the whole
|
|
213
|
+
prompt set that produced the scores. They now say it pins the rubric
|
|
214
|
+
*wording*, and that the prompt template is outside the hash. The phrase they
|
|
215
|
+
used for the wider claim appears nowhere in the tree at this release.
|
|
216
|
+
- **The gap is filed, not resolved.** #207 records what the hash covers, what
|
|
217
|
+
it does not, and the two spec-level options — widen the hash, or add a second
|
|
218
|
+
field — with no recommendation. Which one is right is a specification
|
|
219
|
+
decision, not a package's.
|
|
220
|
+
|
|
221
|
+
A consumer sees none of this until its own lockfile moves: `^0.4.0` admits
|
|
222
|
+
`0.4.1`, so the manifest range need not change and the refresh is the whole of
|
|
223
|
+
the upgrade.
|
|
224
|
+
|
|
6
225
|
## 0.4.0 — 2026-09-04
|
|
7
226
|
|
|
8
227
|
Minor: two optional input fields are added and no existing export is removed
|
|
@@ -31,7 +31,10 @@ export interface ProvenanceInput {
|
|
|
31
31
|
export interface ProvenanceConfig {
|
|
32
32
|
/** Platform-agent identity — the deployment publishing the record. */
|
|
33
33
|
platformAgent: PlatformAgentConfig;
|
|
34
|
-
/** Source registry supplying agent titles and MCP server URLs.
|
|
34
|
+
/** Source registry supplying agent titles and MCP server URLs. Caller-
|
|
35
|
+
* supplied configuration: an instance passes the addresses it is pointed
|
|
36
|
+
* at, and an entry with no `serverUrl` makes its agent omit
|
|
37
|
+
* `civic:serverUrl` (civic-ai-tools#205). */
|
|
35
38
|
sourceRegistry: CivicSourceRegistry;
|
|
36
39
|
/** Source id untagged tool spans fall back to. Default `socrata`
|
|
37
40
|
* (pre-source-tagging captures were Socrata-only). */
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
// order is preserved throughout — the legacy hash chain's byte contract).
|
|
18
18
|
import { makeProvGraph, makeEntityNode, makeActivityNode, makeAgentNode, provRef, provUsed, provWasGeneratedBy, provWasDerivedFrom, provWasAssociatedWith, xsdDateTime, } from '@typedstandards/produce-core';
|
|
19
19
|
import { hash } from "./trace.js";
|
|
20
|
-
import { CIVIC_VOCABULARY, CIVICAITOOLS_PLATFORM_AGENT, CIVIC_TERM_FAILED, CIVIC_TERM_FAILURE_KIND, } from "../format/vocabulary.js";
|
|
20
|
+
import { CIVIC_VOCABULARY, CIVICAITOOLS_PLATFORM_AGENT, CIVIC_TERM_COMPLETION_TOKENS, CIVIC_TERM_CONTENT_HASH, CIVIC_TERM_CROISSANT_METADATA_URL, CIVIC_TERM_DATASET_ID, CIVIC_TERM_DATASET_URL, CIVIC_TERM_DURATION_MS, CIVIC_TERM_FAILED, CIVIC_TERM_FAILURE_KIND, CIVIC_TERM_OPERATION_TYPE, CIVIC_TERM_PORTAL_DOMAIN, CIVIC_TERM_PROMPT_TOKENS, CIVIC_TERM_RESPONSE_ROWS, CIVIC_TERM_SERVER_URL, CIVIC_TERM_SOURCE_ID, CIVIC_TERM_TOOL_NAME, CIVIC_TERM_URL, } from "../format/vocabulary.js";
|
|
21
21
|
import { CIVIC_SOURCE_REGISTRY, FALLBACK_SOURCE_ID, isDatasetKeyedSource, } from "../format/sources.js";
|
|
22
22
|
/** The civicaitools.org reference deployment's values. Passed explicitly by
|
|
23
23
|
* the reference app — never applied as a default, and never spread into
|
|
@@ -88,7 +88,7 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
88
88
|
// --- Entities ---
|
|
89
89
|
// User prompt
|
|
90
90
|
graph.push(makeEntityNode(vocab.urn(packageId, 'prompt', promptHash), {
|
|
91
|
-
|
|
91
|
+
[CIVIC_TERM_CONTENT_HASH]: `sha256:${promptHash}`,
|
|
92
92
|
'dcterms:description': 'User query prompt',
|
|
93
93
|
...(promptText ? { 'prov:value': promptText } : {}),
|
|
94
94
|
}));
|
|
@@ -101,13 +101,13 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
101
101
|
const skillServerUrl = skillSpan ? getAttr(skillSpan.attributes, 'skill.mcp_server_url') : undefined;
|
|
102
102
|
if (skillHash) {
|
|
103
103
|
graph.push(makeEntityNode(vocab.urn(packageId, 'skill', skillHash), {
|
|
104
|
-
|
|
104
|
+
[CIVIC_TERM_CONTENT_HASH]: `sha256:${skillHash}`,
|
|
105
105
|
'dcterms:description': 'Composed MCP skill guidance (system prompt)',
|
|
106
106
|
}, ['prov:Plan']));
|
|
107
107
|
}
|
|
108
108
|
// Final output
|
|
109
109
|
graph.push(makeEntityNode(vocab.urn(packageId, 'output', outputHash), {
|
|
110
|
-
|
|
110
|
+
[CIVIC_TERM_CONTENT_HASH]: `sha256:${outputHash}`,
|
|
111
111
|
'dcterms:description': 'AI-generated analysis output',
|
|
112
112
|
}));
|
|
113
113
|
// --- Agents ---
|
|
@@ -127,6 +127,13 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
127
127
|
// configured fallback source (socrata — the only source at the time) for
|
|
128
128
|
// backwards compatibility. Agent coordinates come from the source registry;
|
|
129
129
|
// the skill-fetch span URL overrides the skill source's server URL.
|
|
130
|
+
//
|
|
131
|
+
// THE ADDRESS IS THE OPERATOR'S, OR IT IS ABSENT (civic-ai-tools#205, 0.5.0).
|
|
132
|
+
// `serverUrl` is optional on a registry entry and there is no stand-in for a
|
|
133
|
+
// source whose address this run does not know: a source the registry does
|
|
134
|
+
// not carry, and a registry entry with no address, both emit an agent with
|
|
135
|
+
// no `civic:serverUrl` key at all. The signed bytes then say the address is
|
|
136
|
+
// unknown instead of naming a server the analysis may never have contacted.
|
|
130
137
|
const toolSpans = spans.filter(s => s.name === 'mcp_tool_call');
|
|
131
138
|
const sourceAgentMap = {};
|
|
132
139
|
for (const [id, info] of Object.entries(registry)) {
|
|
@@ -145,12 +152,13 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
145
152
|
const meta = sourceAgentMap[sourceId] ?? {
|
|
146
153
|
urn: vocab.sourceAgentUrn(encodeURIComponent(sourceId)),
|
|
147
154
|
title: `${sourceId} MCP Server`,
|
|
148
|
-
serverUrl: sourceId,
|
|
149
155
|
};
|
|
150
156
|
graph.push(makeAgentNode(meta.urn, {
|
|
151
157
|
'dcterms:title': meta.title,
|
|
152
|
-
|
|
153
|
-
|
|
158
|
+
// Conditional spread in place, so a known address keeps its
|
|
159
|
+
// position in the key order the legacy chain hashes.
|
|
160
|
+
...(meta.serverUrl ? { [CIVIC_TERM_SERVER_URL]: meta.serverUrl } : {}),
|
|
161
|
+
[CIVIC_TERM_SOURCE_ID]: sourceId,
|
|
154
162
|
}, ['prov:SoftwareAgent']));
|
|
155
163
|
}
|
|
156
164
|
// Resolve a source id to its agent URN, falling back to the configured
|
|
@@ -165,7 +173,7 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
165
173
|
// Platform
|
|
166
174
|
graph.push(makeAgentNode(vocab.platformUrn(config.platformAgent.id), {
|
|
167
175
|
'dcterms:title': config.platformAgent.title,
|
|
168
|
-
|
|
176
|
+
[CIVIC_TERM_URL]: config.platformAgent.url,
|
|
169
177
|
}, ['prov:SoftwareAgent']));
|
|
170
178
|
// --- Activities ---
|
|
171
179
|
// LLM inference spans
|
|
@@ -190,8 +198,8 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
190
198
|
...(span.endTimeUnixNano
|
|
191
199
|
? { 'prov:endedAtTime': xsdDateTime(nanoToIso(span.endTimeUnixNano)) }
|
|
192
200
|
: {}),
|
|
193
|
-
...(promptTokens ? {
|
|
194
|
-
...(completionTokens ? {
|
|
201
|
+
...(promptTokens ? { [CIVIC_TERM_PROMPT_TOKENS]: Number(promptTokens) } : {}),
|
|
202
|
+
...(completionTokens ? { [CIVIC_TERM_COMPLETION_TOKENS]: Number(completionTokens) } : {}),
|
|
195
203
|
}));
|
|
196
204
|
}
|
|
197
205
|
// MCP tool call spans.
|
|
@@ -233,10 +241,10 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
233
241
|
.sort((a, b) => Number(b.endTimeUnixNano || '0') - Number(a.endTimeUnixNano || '0'))[0];
|
|
234
242
|
const queryUrn = vocab.urn(packageId, 'query', queryHash);
|
|
235
243
|
graph.push(makeEntityNode(queryUrn, {
|
|
236
|
-
|
|
244
|
+
[CIVIC_TERM_CONTENT_HASH]: `sha256:${queryHash}`,
|
|
237
245
|
// Omitted — not placeholdered — when the span named no tool.
|
|
238
|
-
...(toolName ? {
|
|
239
|
-
|
|
246
|
+
...(toolName ? { [CIVIC_TERM_TOOL_NAME]: toolName } : {}),
|
|
247
|
+
[CIVIC_TERM_OPERATION_TYPE]: opType,
|
|
240
248
|
'dcterms:description': `MCP tool arguments (${opType})`,
|
|
241
249
|
...(precedingInference
|
|
242
250
|
? provWasGeneratedBy(vocab.urn(packageId, 'inference', precedingInference.spanId))
|
|
@@ -264,9 +272,9 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
264
272
|
: `Data response from ${sourceAgentMap[toolSource]?.title || toolSource}`;
|
|
265
273
|
const responseRows = getAttr(span.attributes, 'tool.response_rows');
|
|
266
274
|
graph.push(makeEntityNode(dataUrn, {
|
|
267
|
-
|
|
275
|
+
[CIVIC_TERM_CONTENT_HASH]: `sha256:${responseHash}`,
|
|
268
276
|
'dcterms:description': description,
|
|
269
|
-
|
|
277
|
+
[CIVIC_TERM_SOURCE_ID]: toolSource,
|
|
270
278
|
...provWasGeneratedBy(toolCallUrn),
|
|
271
279
|
// Croissant 1.1 placeholder — only meaningful for dataset-keyed
|
|
272
280
|
// sources today. The dataset id is stated whenever the span carried
|
|
@@ -274,17 +282,17 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
274
282
|
// the portal as well. Key order is the byte contract.
|
|
275
283
|
...(toolSourceDatasetKeyed && datasetId
|
|
276
284
|
? {
|
|
277
|
-
|
|
285
|
+
[CIVIC_TERM_DATASET_ID]: datasetId,
|
|
278
286
|
...(portalDomain
|
|
279
287
|
? {
|
|
280
|
-
|
|
281
|
-
|
|
288
|
+
[CIVIC_TERM_PORTAL_DOMAIN]: portalDomain,
|
|
289
|
+
[CIVIC_TERM_DATASET_URL]: `https://${portalDomain}/d/${datasetId}`,
|
|
282
290
|
}
|
|
283
291
|
: {}),
|
|
284
|
-
|
|
292
|
+
[CIVIC_TERM_CROISSANT_METADATA_URL]: null, // hook for future Croissant integration
|
|
285
293
|
}
|
|
286
294
|
: {}),
|
|
287
|
-
...(responseRows ? {
|
|
295
|
+
...(responseRows ? { [CIVIC_TERM_RESPONSE_ROWS]: Number(responseRows) } : {}),
|
|
288
296
|
}));
|
|
289
297
|
}
|
|
290
298
|
// Tool call activity — associated with the MCP source agent that handled
|
|
@@ -317,7 +325,7 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
317
325
|
'dcterms:description': toolName
|
|
318
326
|
? `MCP tool call: ${toolName} (${opType})`
|
|
319
327
|
: `MCP tool call (${opType})`,
|
|
320
|
-
|
|
328
|
+
[CIVIC_TERM_SOURCE_ID]: toolSource,
|
|
321
329
|
...provUsed([queryUrn]),
|
|
322
330
|
...provWasAssociatedWith(toolAgentUrn),
|
|
323
331
|
...(span.startTimeUnixNano
|
|
@@ -326,16 +334,25 @@ export function buildProvenanceGraph(trace, input, config) {
|
|
|
326
334
|
...(span.endTimeUnixNano
|
|
327
335
|
? { 'prov:endedAtTime': xsdDateTime(nanoToIso(span.endTimeUnixNano)) }
|
|
328
336
|
: {}),
|
|
329
|
-
...(durationMs ? {
|
|
337
|
+
...(durationMs ? { [CIVIC_TERM_DURATION_MS]: Number(durationMs) } : {}),
|
|
330
338
|
// Appended LAST and spread conditionally, exactly as `civic:durationMs`
|
|
331
339
|
// above is: a span that recorded no rejection yields the key list it
|
|
332
340
|
// yielded at 0.3.1, in the same order, and property insertion order is
|
|
333
341
|
// the legacy chain's byte contract. `false` is never emitted — a
|
|
334
342
|
// producer that stated "not failed" and one that stated nothing must
|
|
335
343
|
// read the same, which is what makes a marker that IS present mean
|
|
336
|
-
// something.
|
|
337
|
-
//
|
|
338
|
-
//
|
|
344
|
+
// something.
|
|
345
|
+
//
|
|
346
|
+
// A REJECTED SPAN NOW CARRIES A DURATION. Until
|
|
347
|
+
// civic-ai-tools-website#413 shipped (Wave N11 P4, merged `ec7f173`)
|
|
348
|
+
// the reference producer measured a rejected call's elapsed and
|
|
349
|
+
// discarded it, so this comment said `civic:durationMs` "simply does
|
|
350
|
+
// not fire beside these two". It fires: the producer writes
|
|
351
|
+
// `tool.duration_ms` on the rejection path, and the conditional above
|
|
352
|
+
// states it — BEFORE the two markers, in the position it has held
|
|
353
|
+
// since 0.3.1, because that conditional is unchanged and this one is
|
|
354
|
+
// still appended after it. `capture/provenance.test.ts` drives that
|
|
355
|
+
// combination and pins the whole key list for it.
|
|
339
356
|
...(failed === true ? { [CIVIC_TERM_FAILED]: true } : {}),
|
|
340
357
|
...(failureKind ? { [CIVIC_TERM_FAILURE_KIND]: failureKind } : {}),
|
|
341
358
|
}));
|
package/dist/format/dathere.d.ts
CHANGED
|
@@ -39,18 +39,37 @@ export declare function selectContentCanonicalization(contentProfile?: string):
|
|
|
39
39
|
* value to pass as the envelope's `summary` field, or `undefined` to omit.
|
|
40
40
|
*/
|
|
41
41
|
export declare function deriveSummaryEmission(contentProfile?: string, summary?: string): string | undefined;
|
|
42
|
+
/** One MCP server the run had available, as the environment extension carries
|
|
43
|
+
* it — spec §8.7.1 requirement 3: an object with `url` and optional `name`.
|
|
44
|
+
* The CALLER supplies the order and the names; the harness neither sorts nor
|
|
45
|
+
* invents them. */
|
|
46
|
+
export interface DatHereMcpServer {
|
|
47
|
+
/** The server's configured address. */
|
|
48
|
+
url: string;
|
|
49
|
+
/** Optional label for the server (the reference caller passes the source
|
|
50
|
+
* id). Omitted from the emitted object when absent, so a nameless entry
|
|
51
|
+
* stays byte-identical to the pre-0.5.0 single-server shape. */
|
|
52
|
+
name?: string;
|
|
53
|
+
}
|
|
42
54
|
/**
|
|
43
55
|
* Build the `org.civicaitools.environment` extension content for a datHere
|
|
44
56
|
* package. Per OES §9.1.1 requirement 3 the extension MUST carry
|
|
45
57
|
* `modelVersion`, `temperature`, `mcpServers`, `toolDefinitions`, `host`.
|
|
46
58
|
*
|
|
59
|
+
* `mcpServers` accepts EITHER shape (additive since 0.5.0,
|
|
60
|
+
* civic-ai-tools-website#449): a single URL string — the trace's skill-fetch
|
|
61
|
+
* span URL, the only server the pre-0.5.0 builder could name — or the list of
|
|
62
|
+
* servers the run actually had, in the caller's order, each with an optional
|
|
63
|
+
* `name`. The single-URL shape produces byte-identical output to before, so
|
|
64
|
+
* an already-signed package still reproduces.
|
|
65
|
+
*
|
|
47
66
|
* Prototype limitations carried over from the reference implementation
|
|
48
67
|
* (known gaps, tightened in follow-up work): `temperature` placeholder `0`,
|
|
49
|
-
* `toolDefinitions` placeholder `[]
|
|
50
|
-
*
|
|
51
|
-
*
|
|
68
|
+
* `toolDefinitions` placeholder `[]`. Fields captured honestly:
|
|
69
|
+
* `modelVersion`, `host`, and — when the caller passes the list — the servers
|
|
70
|
+
* the run had.
|
|
52
71
|
*/
|
|
53
|
-
export declare function buildDatHereEnvironment(model: string,
|
|
72
|
+
export declare function buildDatHereEnvironment(model: string, mcpServers: string | readonly DatHereMcpServer[] | undefined, config: DatHereEnvironmentConfig): Record<string, unknown>;
|
|
54
73
|
/** Input to the composite datHere-policy derivation. */
|
|
55
74
|
export interface DatHerePolicyInput {
|
|
56
75
|
/** Model identifier (becomes `environment.modelVersion`). */
|
|
@@ -62,8 +81,15 @@ export interface DatHerePolicyInput {
|
|
|
62
81
|
/** Candidate summary text (emitted only under the datHere profile). */
|
|
63
82
|
summary?: string;
|
|
64
83
|
/** MCP server URL from the trace's skill-fetch span (capture-side
|
|
65
|
-
* extraction — see src/capture/skill-metadata.ts).
|
|
84
|
+
* extraction — see src/capture/skill-metadata.ts). Consulted only when
|
|
85
|
+
* `mcpServers` is absent, so the pre-0.5.0 call shape keeps producing
|
|
86
|
+
* byte-identical envelopes. */
|
|
66
87
|
skillMcpServerUrl?: string;
|
|
88
|
+
/** Every MCP server the run had available, in the order the caller wants
|
|
89
|
+
* them emitted, each with an optional `name`
|
|
90
|
+
* (civic-ai-tools-website#449). When supplied this REPLACES
|
|
91
|
+
* `skillMcpServerUrl` as the source of `environment.mcpServers`. */
|
|
92
|
+
mcpServers?: readonly DatHereMcpServer[];
|
|
67
93
|
/** Caller-supplied extensions to layer the environment extension onto. */
|
|
68
94
|
extensions?: Record<string, unknown>;
|
|
69
95
|
}
|
package/dist/format/dathere.js
CHANGED
|
@@ -69,26 +69,50 @@ export function deriveSummaryEmission(contentProfile, summary) {
|
|
|
69
69
|
? summary
|
|
70
70
|
: undefined;
|
|
71
71
|
}
|
|
72
|
+
/** Normalize the two accepted `mcpServers` inputs to the emitted array.
|
|
73
|
+
*
|
|
74
|
+
* A bare string is the pre-0.5.0 shape — the trace's skill-fetch URL — and
|
|
75
|
+
* yields at most the one `{ url }` entry, exactly as before. A list is
|
|
76
|
+
* emitted in the order given, each entry carrying `name` only when the
|
|
77
|
+
* caller supplied one. An entry with an empty `url` contributes nothing,
|
|
78
|
+
* which is the same rule the single-URL shape always applied: a server with
|
|
79
|
+
* no address is stated by absence, never by an empty string. */
|
|
80
|
+
function normalizeMcpServers(servers) {
|
|
81
|
+
if (servers === undefined)
|
|
82
|
+
return [];
|
|
83
|
+
if (typeof servers === 'string') {
|
|
84
|
+
return servers ? [{ url: servers }] : [];
|
|
85
|
+
}
|
|
86
|
+
return servers
|
|
87
|
+
.filter((server) => Boolean(server.url))
|
|
88
|
+
.map((server) => ({
|
|
89
|
+
url: server.url,
|
|
90
|
+
...(server.name ? { name: server.name } : {}),
|
|
91
|
+
}));
|
|
92
|
+
}
|
|
72
93
|
/**
|
|
73
94
|
* Build the `org.civicaitools.environment` extension content for a datHere
|
|
74
95
|
* package. Per OES §9.1.1 requirement 3 the extension MUST carry
|
|
75
96
|
* `modelVersion`, `temperature`, `mcpServers`, `toolDefinitions`, `host`.
|
|
76
97
|
*
|
|
98
|
+
* `mcpServers` accepts EITHER shape (additive since 0.5.0,
|
|
99
|
+
* civic-ai-tools-website#449): a single URL string — the trace's skill-fetch
|
|
100
|
+
* span URL, the only server the pre-0.5.0 builder could name — or the list of
|
|
101
|
+
* servers the run actually had, in the caller's order, each with an optional
|
|
102
|
+
* `name`. The single-URL shape produces byte-identical output to before, so
|
|
103
|
+
* an already-signed package still reproduces.
|
|
104
|
+
*
|
|
77
105
|
* Prototype limitations carried over from the reference implementation
|
|
78
106
|
* (known gaps, tightened in follow-up work): `temperature` placeholder `0`,
|
|
79
|
-
* `toolDefinitions` placeholder `[]
|
|
80
|
-
*
|
|
81
|
-
*
|
|
107
|
+
* `toolDefinitions` placeholder `[]`. Fields captured honestly:
|
|
108
|
+
* `modelVersion`, `host`, and — when the caller passes the list — the servers
|
|
109
|
+
* the run had.
|
|
82
110
|
*/
|
|
83
|
-
export function buildDatHereEnvironment(model,
|
|
84
|
-
const mcpServers = [];
|
|
85
|
-
if (skillMcpServerUrl) {
|
|
86
|
-
mcpServers.push({ url: skillMcpServerUrl });
|
|
87
|
-
}
|
|
111
|
+
export function buildDatHereEnvironment(model, mcpServers, config) {
|
|
88
112
|
return {
|
|
89
113
|
modelVersion: model,
|
|
90
114
|
temperature: 0,
|
|
91
|
-
mcpServers,
|
|
115
|
+
mcpServers: normalizeMcpServers(mcpServers),
|
|
92
116
|
toolDefinitions: [],
|
|
93
117
|
host: config.host,
|
|
94
118
|
};
|
|
@@ -102,7 +126,10 @@ export function buildDatHereEnvironment(model, skillMcpServerUrl, config) {
|
|
|
102
126
|
export function deriveDatHereEnvelopeFields(input, config) {
|
|
103
127
|
const extensions = { ...(input.extensions ?? {}) };
|
|
104
128
|
if (input.contentProfile === DATHERE_CONTENT_PROFILE) {
|
|
105
|
-
extensions[ENVIRONMENT_EXTENSION_KEY] = buildDatHereEnvironment(input.model,
|
|
129
|
+
extensions[ENVIRONMENT_EXTENSION_KEY] = buildDatHereEnvironment(input.model,
|
|
130
|
+
// The list when the caller has one, the single skill-fetch URL
|
|
131
|
+
// otherwise — an input carrying neither still emits `mcpServers: []`.
|
|
132
|
+
input.mcpServers ?? input.skillMcpServerUrl, config);
|
|
106
133
|
}
|
|
107
134
|
return {
|
|
108
135
|
producerProfile: deriveProducerProfile(input.producerProfile, input.contentProfile),
|
package/dist/format/sources.d.ts
CHANGED
|
@@ -5,8 +5,18 @@ export interface CivicSourceInfo {
|
|
|
5
5
|
displayName: string;
|
|
6
6
|
/** PROV agent `dcterms:title`, e.g. "Socrata MCP Server". */
|
|
7
7
|
agentTitle: string;
|
|
8
|
-
/**
|
|
9
|
-
|
|
8
|
+
/**
|
|
9
|
+
* MCP endpoint URL — the PROV agent's `civic:serverUrl`, and this source's
|
|
10
|
+
* display address.
|
|
11
|
+
*
|
|
12
|
+
* OPTIONAL since 0.5.0 (civic-ai-tools#205). The address a run reached is
|
|
13
|
+
* the OPERATOR's configuration, not a fact the harness knows, so a caller
|
|
14
|
+
* passes a registry whose entries carry the addresses that instance is
|
|
15
|
+
* pointed at. An entry that carries no address makes the provenance graph
|
|
16
|
+
* OMIT `civic:serverUrl` from that source's agent entirely, rather than
|
|
17
|
+
* assert a stand-in: honest absence over an asserted default.
|
|
18
|
+
*/
|
|
19
|
+
serverUrl?: string;
|
|
10
20
|
/** `dataSources[].catalogType` value for entries from this source. */
|
|
11
21
|
catalogType: string;
|
|
12
22
|
/**
|
|
@@ -22,7 +32,14 @@ export interface CivicSourceInfo {
|
|
|
22
32
|
* order for aggregate `dataSources` entries. */
|
|
23
33
|
export type CivicSourceRegistry = Record<string, CivicSourceInfo>;
|
|
24
34
|
/** The civic reference registry — the three demo sources. Passed explicitly
|
|
25
|
-
* by the reference app — never applied as a default.
|
|
35
|
+
* by the reference app — never applied as a default.
|
|
36
|
+
*
|
|
37
|
+
* THE `serverUrl` VALUES BELOW ARE THE REFERENCE DEPLOYMENT'S OWN, and since
|
|
38
|
+
* 0.5.0 (civic-ai-tools#205) that is the whole of their role: a default
|
|
39
|
+
* display address, and — on the two aggregate sources — the companion of
|
|
40
|
+
* `aggregatePortalUrl`. They are not what a record asserts about a run. An
|
|
41
|
+
* instance pointed at other addresses passes a registry carrying those; an
|
|
42
|
+
* entry with no address emits no `civic:serverUrl` at all. */
|
|
26
43
|
export declare const CIVIC_SOURCE_REGISTRY: CivicSourceRegistry;
|
|
27
44
|
/** The source id untagged/pre-M9.1 captures fall back to. Socrata was the
|
|
28
45
|
* only source when those packages were written. */
|
package/dist/format/sources.js
CHANGED
|
@@ -13,7 +13,14 @@
|
|
|
13
13
|
// demo map is exported as `CIVIC_SOURCE_REGISTRY` for explicit use, never
|
|
14
14
|
// applied as a default.
|
|
15
15
|
/** The civic reference registry — the three demo sources. Passed explicitly
|
|
16
|
-
* by the reference app — never applied as a default.
|
|
16
|
+
* by the reference app — never applied as a default.
|
|
17
|
+
*
|
|
18
|
+
* THE `serverUrl` VALUES BELOW ARE THE REFERENCE DEPLOYMENT'S OWN, and since
|
|
19
|
+
* 0.5.0 (civic-ai-tools#205) that is the whole of their role: a default
|
|
20
|
+
* display address, and — on the two aggregate sources — the companion of
|
|
21
|
+
* `aggregatePortalUrl`. They are not what a record asserts about a run. An
|
|
22
|
+
* instance pointed at other addresses passes a registry carrying those; an
|
|
23
|
+
* entry with no address emits no `civic:serverUrl` at all. */
|
|
17
24
|
export const CIVIC_SOURCE_REGISTRY = {
|
|
18
25
|
socrata: {
|
|
19
26
|
displayName: 'Socrata',
|
|
@@ -35,6 +35,83 @@ export declare const CIVIC_TERM_FAILED = "civic:failed";
|
|
|
35
35
|
* wider vocabulary widens this field rather than putting prose in it.
|
|
36
36
|
*/
|
|
37
37
|
export declare const CIVIC_TERM_FAILURE_KIND = "civic:failureKind";
|
|
38
|
+
/**
|
|
39
|
+
* `civic:contentHash` — the SHA-256 of the bytes an entity stands for, as
|
|
40
|
+
* `sha256:<hex>`. Emitted on the prompt, skill, output, tool-argument and
|
|
41
|
+
* data-response entities: a reader hashes the content it holds and compares.
|
|
42
|
+
*/
|
|
43
|
+
export declare const CIVIC_TERM_CONTENT_HASH = "civic:contentHash";
|
|
44
|
+
/**
|
|
45
|
+
* `civic:serverUrl` — the MCP server URL an source agent answered from, as
|
|
46
|
+
* the deployment's source registry states it (or, for the skill source, as
|
|
47
|
+
* the trace's skill-fetch span carried it).
|
|
48
|
+
*/
|
|
49
|
+
export declare const CIVIC_TERM_SERVER_URL = "civic:serverUrl";
|
|
50
|
+
/**
|
|
51
|
+
* `civic:sourceId` — the stable id of the source that answered a call, on
|
|
52
|
+
* both the tool-call activity and the data-response entity. It is the
|
|
53
|
+
* registry key, not a display name: a reader resolves it, and the graph never
|
|
54
|
+
* spells the reader-facing name.
|
|
55
|
+
*/
|
|
56
|
+
export declare const CIVIC_TERM_SOURCE_ID = "civic:sourceId";
|
|
57
|
+
/**
|
|
58
|
+
* `civic:url` — the public URL of the platform agent that published the
|
|
59
|
+
* record (the deployment's own address, a typed config input).
|
|
60
|
+
*/
|
|
61
|
+
export declare const CIVIC_TERM_URL = "civic:url";
|
|
62
|
+
/** `civic:promptTokens` — prompt tokens the span reported for one inference.
|
|
63
|
+
* Emitted only when the span carried the count; never zero-filled. */
|
|
64
|
+
export declare const CIVIC_TERM_PROMPT_TOKENS = "civic:promptTokens";
|
|
65
|
+
/** `civic:completionTokens` — completion tokens the span reported for one
|
|
66
|
+
* inference. Emitted only when the span carried the count. */
|
|
67
|
+
export declare const CIVIC_TERM_COMPLETION_TOKENS = "civic:completionTokens";
|
|
68
|
+
/**
|
|
69
|
+
* `civic:toolName` — the tool a call invoked, verbatim from the span.
|
|
70
|
+
* Omitted, never placeholdered, when the span named no tool: the graph states
|
|
71
|
+
* absence as absence.
|
|
72
|
+
*/
|
|
73
|
+
export declare const CIVIC_TERM_TOOL_NAME = "civic:toolName";
|
|
74
|
+
/**
|
|
75
|
+
* `civic:operationType` — the producer's classification of what a call did
|
|
76
|
+
* (`query`, `metadata`, `unknown`, …). An open string the graph states
|
|
77
|
+
* verbatim and never re-derives.
|
|
78
|
+
*/
|
|
79
|
+
export declare const CIVIC_TERM_OPERATION_TYPE = "civic:operationType";
|
|
80
|
+
/**
|
|
81
|
+
* `civic:datasetId` — the dataset a data response came from, stated whenever
|
|
82
|
+
* the span carried one and the source is dataset-keyed.
|
|
83
|
+
*/
|
|
84
|
+
export declare const CIVIC_TERM_DATASET_ID = "civic:datasetId";
|
|
85
|
+
/**
|
|
86
|
+
* `civic:portalDomain` — the portal host a tool span carried. Never the run's
|
|
87
|
+
* selected portal: a call that addressed no portal yields a response
|
|
88
|
+
* attributed to none.
|
|
89
|
+
*/
|
|
90
|
+
export declare const CIVIC_TERM_PORTAL_DOMAIN = "civic:portalDomain";
|
|
91
|
+
/**
|
|
92
|
+
* `civic:datasetUrl` — the canonical dataset URL, minted only when the span
|
|
93
|
+
* carried BOTH the portal host and the dataset id (a URL needs a host the
|
|
94
|
+
* span actually stated).
|
|
95
|
+
*/
|
|
96
|
+
export declare const CIVIC_TERM_DATASET_URL = "civic:datasetUrl";
|
|
97
|
+
/**
|
|
98
|
+
* `civic:croissantMetadataUrl` — the Croissant 1.1 metadata URL for a
|
|
99
|
+
* dataset-keyed data response. Emitted as an explicit `null` placeholder for
|
|
100
|
+
* a future integration; the key is inside signed bytes, so it is stated here
|
|
101
|
+
* rather than left as an undocumented literal.
|
|
102
|
+
*/
|
|
103
|
+
export declare const CIVIC_TERM_CROISSANT_METADATA_URL = "civic:croissantMetadataUrl";
|
|
104
|
+
/** `civic:responseRows` — the row count a tool response carried, when the
|
|
105
|
+
* span reported one. */
|
|
106
|
+
export declare const CIVIC_TERM_RESPONSE_ROWS = "civic:responseRows";
|
|
107
|
+
/**
|
|
108
|
+
* `civic:durationMs` — the elapsed the producer measured for one tool call.
|
|
109
|
+
* Emitted whenever the span carried it, on a call that answered and on one
|
|
110
|
+
* the source REFUSED alike: since civic-ai-tools-website#413 the reference
|
|
111
|
+
* producer records the elapsed on its rejection path too, so this term and
|
|
112
|
+
* {@link CIVIC_TERM_FAILED} appear together on the same activity.
|
|
113
|
+
*/
|
|
114
|
+
export declare const CIVIC_TERM_DURATION_MS = "civic:durationMs";
|
|
38
115
|
/**
|
|
39
116
|
* One era of the civic vocabulary: the two literals plus the emitters bound
|
|
40
117
|
* to them. The vocabulary is not deployment configuration (a deployment does
|
|
@@ -37,16 +37,25 @@ export const PRIOR_ERA_CIVIC_URN_PREFIX = 'urn:civic-evidence';
|
|
|
37
37
|
// A `civic:` property name is vocabulary as much as the namespace it hangs
|
|
38
38
|
// under: it is the word a reader of a signed graph interprets, so it is
|
|
39
39
|
// declared here and imported by the builder that emits it, never spelled as a
|
|
40
|
-
// literal inside capture
|
|
41
|
-
//
|
|
42
|
-
//
|
|
43
|
-
// time after it.
|
|
40
|
+
// literal inside capture/. Every term is era-independent — the 2026-08-19
|
|
41
|
+
// settlement moved the NAMESPACE, not the property names — so none carries an
|
|
42
|
+
// era qualifier.
|
|
44
43
|
//
|
|
45
|
-
//
|
|
46
|
-
//
|
|
47
|
-
//
|
|
48
|
-
//
|
|
49
|
-
//
|
|
44
|
+
// COMPLETE, AND GUARDED AS COMPLETE (civic-ai-tools#199 §2). Until this wave
|
|
45
|
+
// two terms were declared here and the other fourteen were spelled inline in
|
|
46
|
+
// capture/provenance.ts and declared nowhere, which the note that stood in
|
|
47
|
+
// this place recorded as "a real inconsistency ... a phase of its own". This
|
|
48
|
+
// is that phase. `purity.test.ts` no longer holds a list of the terms it
|
|
49
|
+
// knows about — it derives the universe from both ends: no string literal
|
|
50
|
+
// anywhere in capture/ may be a `civic:` term, and every `civic:` key a
|
|
51
|
+
// DRIVEN graph emits must be the value of a constant declared here. A term
|
|
52
|
+
// invented inline tomorrow fails both legs without anyone editing a list.
|
|
53
|
+
//
|
|
54
|
+
// NO BYTE MOVES. Naming a key changes no emitted byte: a computed key
|
|
55
|
+
// (`{ [CIVIC_TERM_SOURCE_ID]: v }`) inserts in the same position with the same
|
|
56
|
+
// value as the literal it replaces, and property insertion order is the legacy
|
|
57
|
+
// chain's byte contract. Both golden suites and the span-carrying golden case
|
|
58
|
+
// added in the same phase measure that rather than assume it.
|
|
50
59
|
/**
|
|
51
60
|
* `civic:failed` — a tool-call activity whose call the SOURCE REFUSED.
|
|
52
61
|
*
|
|
@@ -71,6 +80,83 @@ export const CIVIC_TERM_FAILED = 'civic:failed';
|
|
|
71
80
|
* wider vocabulary widens this field rather than putting prose in it.
|
|
72
81
|
*/
|
|
73
82
|
export const CIVIC_TERM_FAILURE_KIND = 'civic:failureKind';
|
|
83
|
+
/**
|
|
84
|
+
* `civic:contentHash` — the SHA-256 of the bytes an entity stands for, as
|
|
85
|
+
* `sha256:<hex>`. Emitted on the prompt, skill, output, tool-argument and
|
|
86
|
+
* data-response entities: a reader hashes the content it holds and compares.
|
|
87
|
+
*/
|
|
88
|
+
export const CIVIC_TERM_CONTENT_HASH = 'civic:contentHash';
|
|
89
|
+
/**
|
|
90
|
+
* `civic:serverUrl` — the MCP server URL an source agent answered from, as
|
|
91
|
+
* the deployment's source registry states it (or, for the skill source, as
|
|
92
|
+
* the trace's skill-fetch span carried it).
|
|
93
|
+
*/
|
|
94
|
+
export const CIVIC_TERM_SERVER_URL = 'civic:serverUrl';
|
|
95
|
+
/**
|
|
96
|
+
* `civic:sourceId` — the stable id of the source that answered a call, on
|
|
97
|
+
* both the tool-call activity and the data-response entity. It is the
|
|
98
|
+
* registry key, not a display name: a reader resolves it, and the graph never
|
|
99
|
+
* spells the reader-facing name.
|
|
100
|
+
*/
|
|
101
|
+
export const CIVIC_TERM_SOURCE_ID = 'civic:sourceId';
|
|
102
|
+
/**
|
|
103
|
+
* `civic:url` — the public URL of the platform agent that published the
|
|
104
|
+
* record (the deployment's own address, a typed config input).
|
|
105
|
+
*/
|
|
106
|
+
export const CIVIC_TERM_URL = 'civic:url';
|
|
107
|
+
/** `civic:promptTokens` — prompt tokens the span reported for one inference.
|
|
108
|
+
* Emitted only when the span carried the count; never zero-filled. */
|
|
109
|
+
export const CIVIC_TERM_PROMPT_TOKENS = 'civic:promptTokens';
|
|
110
|
+
/** `civic:completionTokens` — completion tokens the span reported for one
|
|
111
|
+
* inference. Emitted only when the span carried the count. */
|
|
112
|
+
export const CIVIC_TERM_COMPLETION_TOKENS = 'civic:completionTokens';
|
|
113
|
+
/**
|
|
114
|
+
* `civic:toolName` — the tool a call invoked, verbatim from the span.
|
|
115
|
+
* Omitted, never placeholdered, when the span named no tool: the graph states
|
|
116
|
+
* absence as absence.
|
|
117
|
+
*/
|
|
118
|
+
export const CIVIC_TERM_TOOL_NAME = 'civic:toolName';
|
|
119
|
+
/**
|
|
120
|
+
* `civic:operationType` — the producer's classification of what a call did
|
|
121
|
+
* (`query`, `metadata`, `unknown`, …). An open string the graph states
|
|
122
|
+
* verbatim and never re-derives.
|
|
123
|
+
*/
|
|
124
|
+
export const CIVIC_TERM_OPERATION_TYPE = 'civic:operationType';
|
|
125
|
+
/**
|
|
126
|
+
* `civic:datasetId` — the dataset a data response came from, stated whenever
|
|
127
|
+
* the span carried one and the source is dataset-keyed.
|
|
128
|
+
*/
|
|
129
|
+
export const CIVIC_TERM_DATASET_ID = 'civic:datasetId';
|
|
130
|
+
/**
|
|
131
|
+
* `civic:portalDomain` — the portal host a tool span carried. Never the run's
|
|
132
|
+
* selected portal: a call that addressed no portal yields a response
|
|
133
|
+
* attributed to none.
|
|
134
|
+
*/
|
|
135
|
+
export const CIVIC_TERM_PORTAL_DOMAIN = 'civic:portalDomain';
|
|
136
|
+
/**
|
|
137
|
+
* `civic:datasetUrl` — the canonical dataset URL, minted only when the span
|
|
138
|
+
* carried BOTH the portal host and the dataset id (a URL needs a host the
|
|
139
|
+
* span actually stated).
|
|
140
|
+
*/
|
|
141
|
+
export const CIVIC_TERM_DATASET_URL = 'civic:datasetUrl';
|
|
142
|
+
/**
|
|
143
|
+
* `civic:croissantMetadataUrl` — the Croissant 1.1 metadata URL for a
|
|
144
|
+
* dataset-keyed data response. Emitted as an explicit `null` placeholder for
|
|
145
|
+
* a future integration; the key is inside signed bytes, so it is stated here
|
|
146
|
+
* rather than left as an undocumented literal.
|
|
147
|
+
*/
|
|
148
|
+
export const CIVIC_TERM_CROISSANT_METADATA_URL = 'civic:croissantMetadataUrl';
|
|
149
|
+
/** `civic:responseRows` — the row count a tool response carried, when the
|
|
150
|
+
* span reported one. */
|
|
151
|
+
export const CIVIC_TERM_RESPONSE_ROWS = 'civic:responseRows';
|
|
152
|
+
/**
|
|
153
|
+
* `civic:durationMs` — the elapsed the producer measured for one tool call.
|
|
154
|
+
* Emitted whenever the span carried it, on a call that answered and on one
|
|
155
|
+
* the source REFUSED alike: since civic-ai-tools-website#413 the reference
|
|
156
|
+
* producer records the elapsed on its rejection path too, so this term and
|
|
157
|
+
* {@link CIVIC_TERM_FAILED} appear together on the same activity.
|
|
158
|
+
*/
|
|
159
|
+
export const CIVIC_TERM_DURATION_MS = 'civic:durationMs';
|
|
74
160
|
/** Bind the emitters to one era's two literals. */
|
|
75
161
|
export function makeCivicVocabulary(ns, urnPrefix) {
|
|
76
162
|
return {
|
|
@@ -6,7 +6,10 @@ export declare const EVALUATION_CRITERIA: readonly ["dataSourceIdentification",
|
|
|
6
6
|
export declare const EVALUATION_RUBRIC = "You are an independent evaluator assessing an AI-generated civic data analysis.\n\nYou will receive:\n1. The original prompt (question asked)\n2. The tool calls made (MCP queries to Socrata open data portals)\n3. The AI's final output/analysis\n\nEvaluate the analysis against these 6 criteria, scoring each 1-10:\n\na) **Data Source Identification** (1-10): Does the output correctly identify the data source(s) and time period? Are dataset IDs, portal domains, and date ranges accurate?\n\nb) **Quantitative Claim Support** (1-10): Are the quantitative claims (numbers, percentages, rankings) supported by the data returned in the tool calls? Cross-check key figures against the raw data.\n\nc) **Confounders and Bias** (1-10): Does the analysis acknowledge obvious confounders, selection biases, or framing issues? Are there lurking variables or cherry-picked timeframes?\n\nd) **Geographic Scope** (1-10): Is the geographic scope appropriate for the question? Does the analysis avoid over-generalizing from one jurisdiction?\n\ne) **Limitations Noted** (1-10): Are limitations and caveats noted? Does the analysis flag data quality issues, missing fields, or incomplete coverage?\n\nf) **Contradictory Conclusion** (1-10): Could the same data reasonably support a contradictory conclusion? Does the analysis consider alternative interpretations?\n\nRespond in this exact JSON format (no markdown fences, just raw JSON):\n{\n \"dataSourceIdentification\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"quantitativeClaimSupport\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"confoundersAndBias\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"geographicScope\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"limitationsNoted\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"contradictoryConclusion\": { \"score\": <1-10>, \"comment\": \"<1-2 sentences>\" },\n \"overallScore\": <average of all 6 scores, one decimal>,\n \"assessment\": \"<2-4 sentence overall assessment>\"\n}";
|
|
7
7
|
/** The methodology's `promptSetVersion`: SHA-256 of the rubric text, computed
|
|
8
8
|
* once at module load. Any wording change to the rubric produces a new
|
|
9
|
-
* version, so an attestation pins the
|
|
9
|
+
* version, so an attestation pins the rubric WORDING that scored it — and
|
|
10
|
+
* that wording only. The prompt template around it (`buildEvaluationPrompt`)
|
|
11
|
+
* is outside this hash, so one value does not identify one set of evaluator
|
|
12
|
+
* inputs (civic-ai-tools#207). */
|
|
10
13
|
export declare const RUBRIC_VERSION_SHA256: string;
|
|
11
14
|
/** Build the user-turn evaluation content from a package's signed fields.
|
|
12
15
|
* Takes `RecordPackage` — produce-core 0.3.0's settlement-era name for the
|
|
@@ -5,8 +5,14 @@
|
|
|
5
5
|
// brief §1. The model runner and DB/blob emission (`adversarial-eval.ts`)
|
|
6
6
|
// stay app-side — this module never performs I/O.
|
|
7
7
|
//
|
|
8
|
-
// ⚠ RUBRIC_VERSION_SHA256 is a Q26-pinned version hash
|
|
9
|
-
//
|
|
8
|
+
// ⚠ RUBRIC_VERSION_SHA256 is a Q26-pinned version hash over THE RUBRIC TEXT
|
|
9
|
+
// ALONE (`sha256Hex(EVALUATION_RUBRIC)`). It does NOT cover
|
|
10
|
+
// `buildEvaluationPrompt` below — the template that assembles the rubric with
|
|
11
|
+
// the package's tool calls, data sources and model into the turn the evaluator
|
|
12
|
+
// actually reads. So two evaluations either side of a change to that template
|
|
13
|
+
// carry the SAME `methodology.promptSetVersion` while the evaluator saw
|
|
14
|
+
// different inputs; civic-ai-tools#207 files that gap for the specification to
|
|
15
|
+
// settle, and this module does not resolve it. The rubric text below is
|
|
10
16
|
// byte-exact with the reference implementation (the relocation changes no
|
|
11
17
|
// hashes), and the test suite asserts the literal digest. Any wording change
|
|
12
18
|
// to EVALUATION_RUBRIC is a NEW rubric version — never edit the text without
|
|
@@ -57,8 +63,24 @@ Respond in this exact JSON format (no markdown fences, just raw JSON):
|
|
|
57
63
|
}`;
|
|
58
64
|
/** The methodology's `promptSetVersion`: SHA-256 of the rubric text, computed
|
|
59
65
|
* once at module load. Any wording change to the rubric produces a new
|
|
60
|
-
* version, so an attestation pins the
|
|
66
|
+
* version, so an attestation pins the rubric WORDING that scored it — and
|
|
67
|
+
* that wording only. The prompt template around it (`buildEvaluationPrompt`)
|
|
68
|
+
* is outside this hash, so one value does not identify one set of evaluator
|
|
69
|
+
* inputs (civic-ai-tools#207). */
|
|
61
70
|
export const RUBRIC_VERSION_SHA256 = sha256Hex(EVALUATION_RUBRIC);
|
|
71
|
+
/** Did the producer record this entry as REJECTED by the source? Only a
|
|
72
|
+
* literal `failed: true` is a rejection (see `RecordedCallOutcome`). */
|
|
73
|
+
function wasRecordedRejected(query) {
|
|
74
|
+
return query.failed === true;
|
|
75
|
+
}
|
|
76
|
+
/** What the evaluator is told about a call the source rejected. A FIXED
|
|
77
|
+
* string: it interpolates nothing the producer wrote, and it claims no row
|
|
78
|
+
* count — there were no rows to count, which is neither "zero rows" nor "an
|
|
79
|
+
* unknown number of rows". The rubric asks the evaluator to cross-check
|
|
80
|
+
* figures against the data returned in the tool calls, so a rejected call
|
|
81
|
+
* reading `→ ? rows` invited it to treat absent data as merely unrecorded
|
|
82
|
+
* (civic-ai-tools#203). */
|
|
83
|
+
const REJECTED_CALL_RENDERING = 'REJECTED by the source — no data was returned, and no row count is claimed';
|
|
62
84
|
/** Build the user-turn evaluation content from a package's signed fields.
|
|
63
85
|
* Takes `RecordPackage` — produce-core 0.3.0's settlement-era name for the
|
|
64
86
|
* same object (spec Appendix J, migration class alias-and-deprecate). The
|
|
@@ -70,7 +92,10 @@ export function buildEvaluationPrompt(pkg) {
|
|
|
70
92
|
const argStr = Object.entries(q.arguments)
|
|
71
93
|
.map(([k, v]) => `${k}=${JSON.stringify(v)}`)
|
|
72
94
|
.join(', ');
|
|
73
|
-
|
|
95
|
+
const outcome = wasRecordedRejected(q)
|
|
96
|
+
? REJECTED_CALL_RENDERING
|
|
97
|
+
: `${q.resultRows ?? '?'} rows`;
|
|
98
|
+
return ` ${i + 1}. ${q.tool}(${argStr}) → ${outcome}`;
|
|
74
99
|
})
|
|
75
100
|
.join('\n');
|
|
76
101
|
const dataSources = pkg.dataSources
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@typedstandards/civic-typed-harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "Civic-domain harness for Typed Standards evidence packages — the DOMAIN layer of the ADR-0021 format/domain line: civic PROV-O vocabulary and provenance builder, datHere envelope-policy derivations, and the capture machinery (trace builder, skill-metadata extraction, data-source population, adversarial rubric core). Derives; @typedstandards/produce-core assembles.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Nathan Storey",
|
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
"scripts": {
|
|
27
27
|
"build": "tsc -p tsconfig.json",
|
|
28
28
|
"clean": "rm -rf dist tsconfig.tsbuildinfo",
|
|
29
|
-
"typecheck": "tsc -p tsconfig.json --noEmit",
|
|
29
|
+
"typecheck": "tsc -p tsconfig.json --noEmit && tsc -p tsconfig.test.json",
|
|
30
30
|
"lint": "eslint .",
|
|
31
31
|
"test": "node --test --experimental-strip-types \"src/**/*.test.ts\"",
|
|
32
32
|
"prepare": "npm run build",
|
|
@@ -37,6 +37,7 @@
|
|
|
37
37
|
},
|
|
38
38
|
"devDependencies": {
|
|
39
39
|
"@typedstandards/verify-core": "^0.9.0",
|
|
40
|
+
"@types/node": "^22.20.2",
|
|
40
41
|
"@typescript-eslint/parser": "^8",
|
|
41
42
|
"eslint": "^9",
|
|
42
43
|
"typescript": "^5.9.3"
|