@dogfood-lab/study-swarm 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/PROTOCOL.md +18 -0
- package/README.es.md +27 -2
- package/README.fr.md +47 -22
- package/README.hi.md +27 -2
- package/README.it.md +58 -33
- package/README.ja.md +60 -35
- package/README.md +27 -2
- package/README.pt-BR.md +27 -2
- package/README.zh.md +27 -2
- package/bin/study-swarm.mjs +452 -1
- package/examples/study-swarm-canon-rollback.dispatch.md +97 -0
- package/examples/study-swarm-canon-rollback.lock.json +78 -0
- package/examples/study-swarm-canon-rollback.orchestration.json +761 -0
- package/examples/study-swarm-ci.yml +3 -0
- package/examples/study-swarm-lock.dispatch.md +137 -0
- package/examples/study-swarm-lock.lock.json +62 -0
- package/examples/study-swarm-lock.orchestration.json +369 -0
- package/package.json +1 -1
|
@@ -26,3 +26,6 @@ jobs:
|
|
|
26
26
|
# Lint every dispatch under dispatches/ (a file, a dir, or '-' for stdin all work).
|
|
27
27
|
# Exit 1 on any sourcing violation fails the check. Add --json for machine-readable output.
|
|
28
28
|
- run: npx @dogfood-lab/study-swarm@latest lint dispatches/
|
|
29
|
+
# Halt the build while any finding that became canon is withdrawn and not yet removed or
|
|
30
|
+
# re-grounded — the canon-rollback andon. Exit 1 on any unresolved evidence-withdrawn flag.
|
|
31
|
+
- run: npx @dogfood-lab/study-swarm@latest requalify --check dispatches/
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
<!-- study-swarm v1.2.0 · protocol-sha256:4cfba7e8f7ef0915 · created:2026-06-29 -->
|
|
2
|
+
# Study-swarm dispatch: dispatch.lock.json (the PIN_PER_STEP feature)
|
|
3
|
+
|
|
4
|
+
> **Design dispatch.** This grounds the design of `dispatch.lock.json` — a per-dispatch lockfile that
|
|
5
|
+
> makes a study-swarm research dispatch **byte-replayable** by pinning, per step, the resolved model id,
|
|
6
|
+
> the SHA-256 of the byte-exact agent prompt, the SHA-256 of the tool JSONSchemas the agent had, and the
|
|
7
|
+
> external-verifier receipt — plus a top-level `lock_sha256` rollup. It implements the **PIN_PER_STEP**
|
|
8
|
+
> workflow standard (heritage: Snakemake 2012, Pegasus 2001). Five load-bearing questions were sent to
|
|
9
|
+
> parallel retrieval-grounded research agents; every finding below was fetched this session, and the whole
|
|
10
|
+
> set is gated through Step 4 (`roleos verify-citations` → prism, a different model family) **before** it
|
|
11
|
+
> informs the architecture. The synthesizer is Claude/Opus; the groundedness lens is Mistral; the existence
|
|
12
|
+
> oracle is deterministic retrieval — none of them Claude. Run `study-swarm lint study-swarm-lock.dispatch.md`
|
|
13
|
+
> (it passes).
|
|
14
|
+
|
|
15
|
+
## Step 1 — Load-bearing questions
|
|
16
|
+
|
|
17
|
+
Each passes the load-bearing test (two real designs hinge on the answer; an adjacent field has measured it; the current spec is silent or hand-wavy):
|
|
18
|
+
|
|
19
|
+
- **Q1 — Replay-manifest structure.** How do reproducible-workflow and package/build systems structure a replay manifest, and how do they detect & surface DRIFT between the lock and a re-run? (What is pinned, how it is content-addressed, how a mismatch fails.)
|
|
20
|
+
- **Q2 — Canonicalization & rollup hashing.** How do you canonicalize structured data so a hash is STABLE across platforms (Windows/macOS/Linux) and re-serializations, and how should per-step hashes roll up to one dispatch hash without introducing a collision?
|
|
21
|
+
- **Q3 — Step-level provenance/attestation.** How do supply-chain frameworks capture STEP-LEVEL provenance, and which parts map to pinning "model + prompt + tool-schema + verifier receipt"? What is the honest verifiable-not-unforgeable claim for an offline CLI with an ephemeral key?
|
|
22
|
+
- **Q4 — LLM replay determinism reality.** Can pinning model + prompt + temperature (+ seed) yield reproducible OUTPUTS, or only reproducible INPUTS? (This decides whether the lock may claim "deterministic replay" or must claim "replayable inputs + drift-detectable outputs.")
|
|
23
|
+
- **Q5 — Tool-schema drift.** How do agent frameworks pin/version the tool/function schemas an agent had, so a replay with the same prompt but a CHANGED tool surface is detected? (The half PIN explicitly flags as missing.)
|
|
24
|
+
|
|
25
|
+
## Step 2 — Research dispatch
|
|
26
|
+
|
|
27
|
+
Five parallel research agents (one per question), each retrieval-required — a source an agent could not fetch did not enter the dispatch — followed by a per-question **coverage-recovery sweep** that did a second retrieval pass AND a retrieval-based **existence audit** of the first sweep's citations. The recovery pass earned its keep: it caught a real **misattribution** (PEP 658's author is Tzu-ping Chung, not the byline the first sweep guessed), a **mis-sourced** claim (MoE-routing nondeterminism was attributed to a blog that does not state it; the correct source is the Hugging Face MoE post), and two **over-attributed URLs** (an SLSA reproducibility caveat that is a paraphrase, and a Sigstore ephemeral-key claim documented on the Fulcio pages, not the Rekor logging page). One first-sweep finding was `retrieved:false` (a Pact claim) and was **dropped**; an `oasdiff` finding whose maintainer attribution could not be verified was demoted to a design note rather than a numbered citation. All corrections are folded into Step 3 below; the audit is recorded in Step 4.
|
|
28
|
+
|
|
29
|
+
## Step 3 — Research grounding
|
|
30
|
+
|
|
31
|
+
<!-- Every finding: author + year + a resolvable identifier (arXiv / DOI / RFC-or-spec URL), one-sentence finding, design implication. All gated by Step 4 before Step 5. Claims are phrased to what the retrieved source supports; precise figures that live only in a paper's body were softened to the abstract-grounded claim. -->
|
|
32
|
+
|
|
33
|
+
1. **(Q1) Snakemake re-runs a rule only on output-absence or an input modification-time newer than the output, and deletes incomplete outputs on resume.** Köster & Rahmann 2012 (DOI:10.1093/bioinformatics/bts480). Implication: mtime-based re-run detection is the weak baseline — `dispatch.lock.json` pins CONTENT (SHA-256 of prompt/schema), never timestamps, so drift survives file-touch noise.
|
|
34
|
+
2. **(Q1) Nextflow decides a task is cached by a single hash folding its inputs, command script, container image, environment, and parameters, re-executing on any change.** Di Tommaso et al. 2017 (DOI:10.1038/nbt.3820). Implication: each lock step's record folds resolved-model-id + `prompt_sha256` + `tool_schema_sha256` together, and `lock --verify` treats any field's drift as a replay miss.
|
|
35
|
+
3. **(Q1) Nextflow's default file hashing keys on path + last-modified + size rather than content, producing spurious cache misses it must paper over with a "lenient" mode.** Seqera 2025 (https://docs.seqera.io/nextflow/cache-and-resume). Implication: a cautionary tale — hash byte-exact CONTENT via `node:crypto`, never path+mtime+size, so the lock does not inherit metadata-hash false positives.
|
|
36
|
+
4. **(Q1) npm's package-lock.json pins each package's exact version, resolved URL, and an integrity hash, describing the tree so installs reproduce it.** npm, Inc. 2024 (https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json). Implication: models the per-step fields one-to-one — resolved id (≈version), source, content hash (≈integrity) — per research-agent step.
|
|
37
|
+
5. **(Q1) npm ci treats the lockfile as the source of truth and errors out rather than rewriting it on any package.json/lock mismatch.** npm, Inc. 2024 (https://docs.npmjs.com/cli/v10/commands/npm-ci). Implication: the exact pattern for `lock --verify` — re-derive the hashes and FAIL CLOSED (non-zero exit) on mismatch; never silently auto-heal.
|
|
38
|
+
6. **(Q1) pip's --require-hashes mode demands a SHA-256-or-stronger hash for ALL requirements and rejects md5/sha1/sha224 as too weak.** Python Packaging Authority 2024 (https://pip.pypa.io/en/stable/topics/secure-installs/). Implication: SHA-256 is the floor and completeness is all-or-nothing — every step is hashed, and one missing/mismatched hash fails the whole verify.
|
|
39
|
+
7. **(Q1) PEP 658 lets a repository advertise a distribution's metadata hash as hashname=hashvalue so clients verify fetched metadata before trusting it.** Chung 2021 (https://peps.python.org/pep-0658/). Implication: precedent for hashing the capability METADATA, not just a payload — `tool_schema_sha256` hashes the tool schemas (what the agent COULD call), independent of any output.
|
|
40
|
+
8. **(Q1) A build is reproducible only if, given the same source, environment, and instructions, any party can recreate bit-identical artifacts.** Reproducible Builds 2024 (https://reproducible-builds.org/docs/definition/). Implication: frames the honest ceiling — an LLM step has an uncontrolled input, so the lock pins the declarable inputs bit-exact and only drift-detects outputs.
|
|
41
|
+
9. **(Q2) RFC 8785 (JCS) fixes a canonical JSON form — recursive UTF-16 code-unit key sorting, no inter-token whitespace, ECMAScript number serialization, UTF-8 output — so equal objects serialize to equal bytes.** Rundgren et al. 2020 (https://www.rfc-editor.org/rfc/rfc8785). Implication: run every CLI-assembled JSON object (the tool surface, the lock body) through JCS before SHA-256 so the harness's key order or spacing never causes false drift.
|
|
42
|
+
10. **(Q2) UAX #15 guarantees canonically-equivalent Unicode strings (e.g. the NFC vs NFD encodings of the same text) share one identical normalized form.** Whistler 2025 (https://unicode.org/reports/tr15/). Implication: because JCS deliberately does NOT normalize Unicode, the CLI must NFC-normalize strings before JCS+SHA-256, or text authored on macOS (often NFD) silently drifts from the same text on Windows/Linux.
|
|
43
|
+
11. **(Q2) RFC 8259 requires interchange JSON be UTF-8 and forbids emitting a byte-order mark, while declaring member order and whitespace insignificant to JSON semantics.** Bray 2017 (https://datatracker.ietf.org/doc/html/rfc8259). Implication: the Windows-authored CLI strips any BOM, never writes one, and relies on JCS rather than trusting the harness's key order/whitespace.
|
|
44
|
+
12. **(Q2) RFC 7515 (JWS) signs over the exact base64url octets rather than re-canonicalized JSON, so whitespace, member order, and re-serialization can never alter the signature.** Jones et al. 2015 (https://www.rfc-editor.org/rfc/rfc7515). Implication: for inputs whose exact bytes are already known — the byte-exact prompt and the verifier receipt blob — hash the captured bytes directly; reserve JCS for JSON the CLI assembles itself.
|
|
45
|
+
13. **(Q2) Merkle's hash tree authenticates many leaves under a single root hash by hashing pairwise up a tree.** Merkle 1987 (DOI:10.1007/3-540-48184-2_32). Implication: `lock_sha256` is a rollup committing to every per-step record, so any single-step drift changes the dispatch's content-address.
|
|
46
|
+
14. **(Q2) RFC 6962 hashes Merkle leaves as SHA-256(0x00 then data) and interior nodes as SHA-256(0x01 then children), stating the domain separation is required for second-preimage resistance.** Laurie et al. 2013 (https://www.rfc-editor.org/rfc/rfc6962.html). Implication: distinct per-field keys inside the canonical lock object provide the same domain separation, so a single step's hash cannot be passed off as the whole-lock hash.
|
|
47
|
+
15. **(Q2) Bitcoin's CVE-2012-2459 showed that hashing leaves and interior nodes identically, plus duplicating an odd last leaf, lets two distinct trees collide to one root.** Bitcoin Optech 2012 (https://bitcoinops.org/en/topics/merkle-tree-vulnerabilities/). Implication: do NOT hand-roll an odd-leaf-duplicating Merkle tree — hash the ordered steps as one canonical JSON array whose explicit length commits to exactly N steps.
|
|
48
|
+
16. **(Q2) RFC 8032 specifies Ed25519 signatures as deterministic — the per-signature nonce is hash-derived, not random — so signing the same bytes under the same key yields byte-identical signatures.** Josefsson & Liusvaara 2017 (https://www.rfc-editor.org/rfc/rfc8032.html). Implication: the prism Ed25519 receipt is a stable, re-derivable pin — the lock stores the receipt id plus a hash of the signed blob, and `--verify` flags drift if those bytes change, without the CLI verifying the signature itself.
|
|
49
|
+
17. **(Q3) in-toto link metadata records, per step, materials (input hashes), products (output hashes), the command, byproducts, and environment, with MATCH rules chaining one step's products to the next step's materials.** Torres-Arias et al. 2019 (https://www.usenix.org/conference/usenixsecurity19/presentation/torres-arias). Implication: a lock step mirrors the link shape — materials = prompt + tool-schema hashes, product = output hash, command = resolved model + params, actor = verifier receipt.
|
|
50
|
+
18. **(Q3) The in-toto attestation Statement binds a predicate to artifacts purely by digest, matching subjects regardless of name or content type.** in-toto Attestation Framework 2023 (https://github.com/in-toto/attestation/blob/main/spec/v1/statement.md). Implication: pin every input by its digest, never by filename or alias — the digest is the anchor, the name is cosmetic.
|
|
51
|
+
19. **(Q3) SLSA provenance splits buildDefinition (external/internal parameters, resolvedDependencies with digests) from runDetails (builder.id, metadata, byproducts) and documents inputs for rebuild rather than guaranteeing reproducibility.** OpenSSF SLSA 2023 (https://slsa.dev/spec/v1.0/provenance). Implication: licenses the lock's honest ceiling and tells it to record the RESOLVED model id as the builder.id-equivalent — aliases float and break replay.
|
|
52
|
+
20. **(Q3) SLSA verification is a strict comparison against pre-declared expectations that rejects unrecognized external parameters rather than ignoring them.** OpenSSF SLSA 2023 (https://slsa.dev/spec/v1.0/verifying-artifacts). Implication: `lock --verify` is strict-match — every pinned field must equal the re-derived value AND unknown/extra fields fail, not pass silently.
|
|
53
|
+
21. **(Q3) SCITT issues a Receipt — an offline-verifiable inclusion proof — for a registered Signed Statement, so a third party can verify registration without re-contacting the service.** Birkholz et al. 2025 (https://datatracker.ietf.org/doc/html/draft-ietf-scitt-architecture-22). Implication: store the verifier run_id + receipt chain hash so `lock --verify` confirms the verification happened, offline, without re-running the verifier.
|
|
54
|
+
22. **(Q3) Sigstore's Rekor is an append-only transparency log whose entries are immutable, making tampering detectable rather than impossible, and it pairs ephemeral keys with the log so a discarded key still yields a verifiable record.** Newman et al. 2022 (DOI:10.1145/3548606.3560596). Implication: the lock makes drift/tampering DETECTABLE, not outputs unforgeable — the correct, honest claim for an offline CLI whose signing key is ephemeral.
|
|
55
|
+
23. **(Q3) DSSE signs over a length-prefixed pre-authentication encoding of (type, body) and forbids verifiers from re-parsing the envelope to extract the payload.** Secure Systems Lab 2021 (https://github.com/secure-systems-lab/dsse/blob/master/protocol.md). Implication: hash the byte-exact prompt/tool-schema with their length and type bound in, and have `--verify` hash the STORED bytes rather than re-canonicalizing parsed JSON — re-parsing reintroduces the canonicalization attack surface DSSE was built to remove.
|
|
56
|
+
24. **(Q3) Git is content-addressable: an object's key is the hash of a typed header (the type plus byte-length plus a NUL) concatenated with its exact content.** Chacon & Straub 2014 (https://git-scm.com/book/en/v2/Git-Internals-Git-Objects). Implication: compute `lock_sha256` over a typed, length-framed canonical encoding so a one-byte change flips the address — the proven content-address recipe.
|
|
57
|
+
25. **(Q3) W3C PROV models a step as an Activity that used inputs, wasGeneratedBy outputs, and wasAssociatedWith an agent, chained by wasInformedBy / wasDerivedFrom.** Lebo et al. 2013 (https://www.w3.org/TR/prov-o/). Implication: confirms the minimal per-step quadruple — input hashes, output hashes, actor (model + verifier), and a rollup link — is complete lineage; nothing more is required.
|
|
58
|
+
26. **(Q4) The dominant cause of LLM inference nondeterminism is lack of batch invariance — kernel reduction order shifts with batch size (server load) — so temperature-0 prompts still vary; standard vLLM gave 80 unique completions from 1000 identical samples.** He & Thinking Machines Lab 2025 (https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/). Implication: batch size is outside the lock's control, so it records output hashes for drift and never promises identical outputs.
|
|
59
|
+
27. **(Q4) Under greedy decoding, BF16 LLM outputs diverge across hardware from floating-point non-associativity, with up to ~9% accuracy variation and large response-length differences.** Yuan et al. 2025 (arXiv:2506.09501). Implication: fixed decoding params do not yield reproducible outputs across infrastructure, so an output-hash mismatch is expected drift to flag, not a failure of the lock.
|
|
60
|
+
28. **(Q4) Five LLMs configured at temperature 0 rarely produced identical raw output strings across reruns, with accuracy varying up to ~15%.** Atil et al. 2024 (arXiv:2408.04667). Implication: the strongest direct evidence that "deterministic settings" do not give byte-identical outputs — pin INPUTS byte-exact, only drift-detect OUTPUTS.
|
|
61
|
+
29. **(Q4) The same commercial model silently changed behavior over months — GPT-4's prime-vs-composite accuracy fell from 84% to 51% between March and June 2023.** Chen, Zaharia & Zou 2023 (arXiv:2307.09009). Implication: an alias (or even a named model) can be re-tuned server-side, so the lock pins the RESOLVED model id and drift-detects to surface provider drift at re-run.
|
|
62
|
+
30. **(Q4) OpenAI's API frames reproducibility as inputs-pinned plus a drift sentinel: identical seed and params are only "mostly" deterministic, and clients must watch the returned system_fingerprint for backend changes.** OpenAI 2025 (https://developers.openai.com/api/docs/guides/advanced-usage). Implication: a frontier provider already ships the lock's exact pattern — pin inputs, treat a fingerprint/hash mismatch on re-run as drift to flag, not a lock error.
|
|
63
|
+
31. **(Q4) MoE serving adds routing-level nondeterminism: expert capacity and token-dropping mean a sequence's processing depends on which other sequences share its batch.** Sanseviero et al. 2023 (https://huggingface.co/blog/moe). Implication: for MoE-backed frontier models the lock can never promise output replay, reinforcing "replayable inputs + drift-detectable outputs."
|
|
64
|
+
32. **(Q4) Achieving truly deterministic LLM output needs a runtime verify-and-rollback loop that replays nondeterministic tokens under consistent reduction schedules.** Gond et al. 2026 (arXiv:2601.17768). Implication: output determinism requires machinery far outside a zero-dependency offline CLI, so the lock defers any determinism claim to the harness and stays at the input-pin + output-drift layer.
|
|
65
|
+
33. **(Q5) An MCP tool is defined by name + description + inputSchema (+ optional outputSchema/annotations) returned by tools/list, but the protocol ships no per-tool version or content hash.** MCP Spec 2025 (https://modelcontextprotocol.io/specification/2025-06-18/server/tools). Implication: `tool_schema_sha256` supplies the missing content-level detector — hash the canonicalized tools/list array the agent actually saw.
|
|
66
|
+
34. **(Q5) MCP signals tool changes only via a runtime listChanged flag and versions just the protocol (a session date-string), not the individual tool schemas.** MCP Spec 2025 (https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle). Implication: record the protocolVersion but never rely on it for drift — two runs can share it with a totally different tool set, so the content hash is the real guard.
|
|
67
|
+
35. **(Q5) Anthropic injects tool definitions as JSON Schema verbatim into the constructed system prompt the model conditions on.** Anthropic 2025 (https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use). Implication: because the schema bytes condition the model, `tool_schema_sha256` must hash the exact schema — a changed description or an added enum value is real conditioning drift, not cosmetic.
|
|
68
|
+
36. **(Q5) OpenAI strict-mode function calling requires additionalProperties false and every field marked required, rejecting a tool definition that violates the constraints.** OpenAI 2025 (https://developers.openai.com/api/docs/guides/function-calling). Implication: the CLI validates each captured tool schema is well-formed before hashing, so the lock pins a schema the run could actually have used.
|
|
69
|
+
37. **(Q5) The same JSON Schema body can be evaluated differently under a different dialect, and MCP's unspecified dialect caused real cross-SDK validation failures resolved by defaulting to draft 2020-12.** JSON Schema Org 2025 (https://json-schema.org/understanding-json-schema/reference/schema). Implication: capture the effective `$schema` dialect inside the hashed bytes so a draft-07 to 2020-12 change flips the hash even when the properties are unchanged.
|
|
70
|
+
38. **(Q5) W3C Subresource Integrity pins an algorithm-prefixed digest (e.g. the algorithm name then a base64 hash) and requires the agent to refuse content whose computed hash does not exactly match, erroring rather than warning.** Braun 2026 (https://www.w3.org/TR/sri-2/). Implication: adopt SRI's self-describing, algorithm-agile digest form and its fail-closed semantics for `lock --verify`.
|
|
71
|
+
39. **(Q5) OpenAPI separates the spec-dialect version (openapi) from the document's own version (info.version) as distinct fields.** OpenAPI Initiative 2025 (https://spec.openapis.org/oas/v3.2.0.html). Implication: separate the schema-dialect version from any tool-surface version string the harness supplies, but treat the content hash — not either label — as the authoritative drift guard.
|
|
72
|
+
|
|
73
|
+
## Step 4 — External verification
|
|
74
|
+
|
|
75
|
+
**Run against this dispatch's 39 citations through the LIVE runner before Step 5 was locked.** Synthesizer = Claude/Opus; verifier = the deterministic arXiv→Crossref retrieval oracle + a groundedness lens on **`mistral-small:24b`** (ModelFamily `local`, reasoning-stripped — `--caller-family anthropic` excludes the synthesizer's family by construction). Command: `roleos verify-citations examples/study-swarm-lock.dispatch.md --provider ollama` → `prism verify --type citations` (prism v1.6.0). **No verifier was Claude — the protocol did not grade its own homework.**
|
|
76
|
+
|
|
77
|
+
- [x] existence established by **retrieval, not memory** — the structured oracle for the academic subset, plus an independent in-session retrieval existence-audit for the rest. **0 fabricated.**
|
|
78
|
+
- [x] groundedness checked by a **different family** (`mistral-small:24b`, reasoning-stripped), where the oracle returned an abstract — abstract-less and oracle-timeout citations **escalated "retrieve manually," never auto-passed**.
|
|
79
|
+
- [x] **≥ 3 decorrelated lenses** (mechanism diversity, per the ensemble rule): the deterministic arXiv/Crossref oracle + the Mistral groundedness lens + the independent coverage-recovery retrieval existence-audit.
|
|
80
|
+
|
|
81
|
+
**Verdict: `escalate` (advisory, non-blocking; prism exit 30). 0 fabricated, 0 refused.** The gate *discriminated* by source type rather than rubber-stamping:
|
|
82
|
+
|
|
83
|
+
- **8 citations parsed** by the runner's arXiv/DOI extractor (the academic subset). **4 Crossref DOIs existence-RESOLVED:** Snakemake (`10.1093/bioinformatics/bts480`), Nextflow (`10.1038/nbt.3820`), Merkle (`10.1007/3-540-48184-2_32`), Sigstore (`10.1145/3548606.3560596`). Groundedness rendered a **real verdict on Snakemake** — `not_addressed` ("the claim is not in the title+abstract"), correct: the mtime-re-run behavior lives in the paper body, exactly the partial-support escalation the protocol exists to surface. The other three resolved but Crossref served no abstract, so groundedness could not run → escalate.
|
|
84
|
+
- **4 arXiv papers** (Yuan `2506.09501`, Atil `2408.04667`, Chen `2307.09009`, Gond `2601.17768`) → **arXiv oracle ReadTimeout**: this rig's IP was rate-limited by arXiv after the dispatch's heavy same-session traffic (confirmed — a direct combined `export.arxiv.org` request also returned empty during the cooldown). The runner escalated **"RETRIEVE MANUALLY," never fabrication** — the oracle-unavailable halt rule firing live. All four were existence-confirmed earlier this session by the coverage-recovery existence-audit.
|
|
85
|
+
- **31 spec / RFC / standard / vendor-doc citations** carry direct URLs — outside the arXiv/Crossref structured oracle's resolver — so the extractor reported them **`unparsed` (not arXiv/DOI), which is NOT fabrication**. Every one was existence-verified by **direct retrieval this session**: the research + coverage-recovery agents' existence-audit resolved all of them (and caught the corrections folded into Step 3), and a live spot-check returned **HTTP 200** for RFC 8785, the in-toto USENIX page, W3C SRI, and the MCP spec.
|
|
86
|
+
- **A first run flagged the in-toto citation `fabricated`** — a **Crossref-coverage gap**, not a real fabrication: the ACM/USENIX legacy `10.5555` DOI namespace is not indexed by Crossref. The paper is real (USENIX Security 2019). Corrected to its primary USENIX URL (finding 17); the false flag cleared. This is the "the oracle can't resolve *this identifier type* ≠ fabricated" lesson, observed live.
|
|
87
|
+
|
|
88
|
+
**Receipt (captured + cryptographically verified).** prism Ed25519 receipt **`prism-01kwbajx31dj9gcf5xn3cn5ydg`**, chained into the `roleos-citation-receipt/v1` (`chain_sha256 499b63905064a5e25fd1801c5530504c94742f2183c4d3c8eb545a20cfbb112e`; each retrieval pin carries a `source_sha256` → drift-detectable). `prism replay <id>` → `prism verify-receipt --public-key <pub.pem>` → **`signature_valid: true`, exit 0** — verified with the **public key alone, no shared secret** (the cross-tool path a consumer uses to confirm a verdict it did not produce). Honest ceiling: the signing key is ephemeral and scratchpad-local, so the receipt buys third-party *verifiability*, not anti-forgery (prism's own disclosed limit). **This dispatch is the first to pin its own verifier receipt into its `dispatch.lock.json` (L10) — the feature closing the loop on the exact run that gated it.**
|
|
89
|
+
|
|
90
|
+
## Step 5 — Architecture (`dispatch.lock.json`)
|
|
91
|
+
|
|
92
|
+
Each choice traces to findings by number. The shape:
|
|
93
|
+
|
|
94
|
+
```json
|
|
95
|
+
{
|
|
96
|
+
"schema": "dispatch.lock/v1",
|
|
97
|
+
"study_swarm_version": "1.2.0",
|
|
98
|
+
"protocol_sha256": "<full sha256 of the vendored PROTOCOL.md>",
|
|
99
|
+
"dispatch_sha256": "<sha256 of the dispatch .md bytes>",
|
|
100
|
+
"steps": [
|
|
101
|
+
{
|
|
102
|
+
"question_id": "Q1-replay-manifest",
|
|
103
|
+
"resolved_model": "claude-opus-4-8",
|
|
104
|
+
"prompt_sha256": "sha256-<base64>",
|
|
105
|
+
"tool_schema_sha256": "sha256-<base64>",
|
|
106
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
107
|
+
"params": { "effort": "high" },
|
|
108
|
+
"output_sha256": "sha256-<base64>"
|
|
109
|
+
}
|
|
110
|
+
],
|
|
111
|
+
"verification": {
|
|
112
|
+
"runner": "roleos verify-citations",
|
|
113
|
+
"tool": "prism verify --type citations",
|
|
114
|
+
"verifier_model": "mistral-small:24b",
|
|
115
|
+
"verifier_family": "local",
|
|
116
|
+
"receipt_id": "prism-...",
|
|
117
|
+
"receipt_chain_sha256": "sha256-<base64>"
|
|
118
|
+
},
|
|
119
|
+
"lock_sha256": "sha256-<base64>"
|
|
120
|
+
}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
- **L1 — One lock per dispatch; the lock IS the dispatch's content-address.** `lock_sha256` binds every per-step record and the verifier block together, so a replay cannot stitch a step from one dispatch onto another. (findings 4, 13, 17, 18)
|
|
124
|
+
- **L2 — The harness emits the record; the CLI canonicalizes + hashes + validates it.** This producer/verifier split is universal across the provenance literature, and it is what keeps the CLI zero-dependency, network-free, and deterministic — it never calls a model. (findings 17, 19, 21, 22)
|
|
125
|
+
- **L3 — Hash the prompt as normalized text, not JCS-restructured JSON.** The prompt is the literal string the model conditioned on, so it is hashed directly rather than canonicalized as JSON (the JWS/DSSE hash-known-bytes rule) — under one necessary text normalization (BOM strip + CRLF→LF + NFC), without which the same prompt hashes differently across platforms. (findings 12, 23; 10, 11)
|
|
126
|
+
- **L4 — Normalize every text input before hashing (BOM strip + CRLF→LF + NFC), and JCS-canonicalize the structured JSON (tool surface, lock body).** This is the only way the same dispatch hashes identically on Windows, macOS, and Linux — the prompt, the dispatch text, `PROTOCOL.md`, and every JSON string value all pass through it. (findings 9, 10, 11) *(CI caught a real CRLF drift here when an early build hashed raw `PROTOCOL.md` bytes — the fix is exactly this normalization, and a line-ending-invariance test now guards it.)*
|
|
127
|
+
- **L5 — Capture the tool surface as the canonicalized array `{name, description, inputSchema, outputSchema}` plus the effective JSON Schema dialect.** Neither MCP nor the provider APIs ship a per-tool version or hash, so the lock's content hash is the missing drift detector, and the dialect is part of the contract. (findings 7, 33, 34, 35, 37)
|
|
128
|
+
- **L6 — Pin the RESOLVED model id, never an alias.** A named model can be re-tuned server-side; the concrete platform identity is what makes a step replayable. (findings 19, 29)
|
|
129
|
+
- **L7 — `output_sha256` records outputs for DRIFT DETECTION, not determinism.** Pinning model + prompt + temperature does not yield bit-identical outputs (batch-invariance, FP non-associativity, MoE routing, provider drift), so the honest claim is **"replayable inputs + drift-detectable outputs,"** never "deterministic replay." (findings 8, 19, 26, 27, 28, 30, 31, 32)
|
|
130
|
+
- **L8 — `lock --verify` is fail-closed strict-match.** Re-derive every deterministic hash and assert equality; any mismatch — and any unrecognized field — exits non-zero, never auto-heals. (findings 5, 6, 20, 38)
|
|
131
|
+
- **L9 — Self-describing digests + domain separation in the rollup, hashed as one flat canonical array.** Digests carry their algorithm prefix; the canonical JSON object's distinct keys and the array's explicit length supply domain separation and sidestep the odd-leaf-duplication collision class. (findings 14, 15, 24, 38)
|
|
132
|
+
- **L10 — Pin the verifier receipt offline-verifiably.** Store `receipt_id` + `receipt_chain_sha256`; `lock --verify` confirms the verification happened without re-contacting the verifier. The Ed25519 receipt is a stable pin; the ephemeral key buys verifiability, not anti-forgery — stated, not oversold. (findings 16, 21, 22)
|
|
133
|
+
- **L11 — Pin content, not mtimes.** The lock is robust to file-touch noise by construction. (findings 1, 3)
|
|
134
|
+
|
|
135
|
+
**Optional actionable drift output (design note, not a numbered citation):** beyond a flipped SHA, `lock --verify` may classify *what* changed in the tool surface (added tool = additive, removed/renamed parameter = breaking), mirroring the breaking/non-breaking split that the [oasdiff](https://github.com/oasdiff/oasdiff) OpenAPI differ surfaces in CI. This is a usability layer over the authoritative hash check, not a substitute for it.
|
|
136
|
+
|
|
137
|
+
**Net:** the lock turns a study-swarm dispatch into a content-addressed, byte-replayable manifest — resolved model + prompt + tool-schema + verifier receipt pinned per step, rolled into one `lock_sha256`, drift-checked fail-closed — while telling the truth about its ceiling: it makes inputs replayable and outputs drift-detectable, not LLM outputs deterministic.
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schema": "dispatch.lock/v1",
|
|
3
|
+
"study_swarm_version": "1.3.0",
|
|
4
|
+
"protocol_sha256": "sha256-seELxwXtj+xVxa3mas2cP0J0iMTTpq4vHqFMiUGqmuU=",
|
|
5
|
+
"dispatch_sha256": "sha256-mPdwnlPuRlCtky20zG6l5AFJ8iRAZjHISnoGDUuGSOo=",
|
|
6
|
+
"steps": [
|
|
7
|
+
{
|
|
8
|
+
"question_id": "Q1-replay-manifest",
|
|
9
|
+
"resolved_model": "claude-opus-4-8",
|
|
10
|
+
"prompt_sha256": "sha256-hcdcTxvqJNij0z9ItUPIzQhBWbrVZy3VvY4umPHaM5I=",
|
|
11
|
+
"tool_schema_sha256": "sha256-kUC7s+wAFV4lzwrFKYp36khxCRQZWlWeFDOwhPIqGCk=",
|
|
12
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
13
|
+
"output_sha256": "sha256-k5rkOOyAHGBCVacEk8LPULp+s5YHVzaUyCHJY4tD5jE="
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"question_id": "Q2-canonicalization",
|
|
17
|
+
"resolved_model": "claude-opus-4-8",
|
|
18
|
+
"prompt_sha256": "sha256-Gj+ri3gWVWxP7M9fv50c0xsgiZ2j3T6kn3A3c3EhSos=",
|
|
19
|
+
"tool_schema_sha256": "sha256-kUC7s+wAFV4lzwrFKYp36khxCRQZWlWeFDOwhPIqGCk=",
|
|
20
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
21
|
+
"output_sha256": "sha256-ymPE5lJqoygN0MpzqftgE1tYlyp7z+poCK2Z1Zj3mi8="
|
|
22
|
+
},
|
|
23
|
+
{
|
|
24
|
+
"question_id": "Q3-provenance-attestation",
|
|
25
|
+
"resolved_model": "claude-opus-4-8",
|
|
26
|
+
"prompt_sha256": "sha256-g5/dW+zJvMObKyqnAdy70pUPtJcGD0DrsjkeEx/AcIY=",
|
|
27
|
+
"tool_schema_sha256": "sha256-kUC7s+wAFV4lzwrFKYp36khxCRQZWlWeFDOwhPIqGCk=",
|
|
28
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
29
|
+
"output_sha256": "sha256-OSUAIhytytKihfFM1Y+p1BllUSTC+KEfb/NX86X+Kc0="
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
"question_id": "Q4-llm-determinism",
|
|
33
|
+
"resolved_model": "claude-opus-4-8",
|
|
34
|
+
"prompt_sha256": "sha256-OfQ00IyAulvEOnZ/UpiLp+JTwF4jYUzQRaqCp3zgIbE=",
|
|
35
|
+
"tool_schema_sha256": "sha256-kUC7s+wAFV4lzwrFKYp36khxCRQZWlWeFDOwhPIqGCk=",
|
|
36
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
37
|
+
"output_sha256": "sha256-aFDm9p4/94p97NJg/vWKbYXchv6N1b0swpIfYoep1iw="
|
|
38
|
+
},
|
|
39
|
+
{
|
|
40
|
+
"question_id": "Q5-tool-schema-drift",
|
|
41
|
+
"resolved_model": "claude-opus-4-8",
|
|
42
|
+
"prompt_sha256": "sha256-qIr9Dl8GmGeikUrgCJ62QQh8FvTNUhvdxL0Ivk/Di3M=",
|
|
43
|
+
"tool_schema_sha256": "sha256-kUC7s+wAFV4lzwrFKYp36khxCRQZWlWeFDOwhPIqGCk=",
|
|
44
|
+
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
|
|
45
|
+
"output_sha256": "sha256-Y1lsUdKoplgdos0r4faN6UgxXX74D/Se0AVHCiUA9dM="
|
|
46
|
+
}
|
|
47
|
+
],
|
|
48
|
+
"verification": {
|
|
49
|
+
"runner": "roleos verify-citations",
|
|
50
|
+
"runner_source": "role-os local clone E:/AI/role-os",
|
|
51
|
+
"tool": "prism verify --type citations",
|
|
52
|
+
"tool_version": "prism 1.6.0",
|
|
53
|
+
"verifier_model": "mistral-small:24b",
|
|
54
|
+
"verifier_family": "local",
|
|
55
|
+
"caller_family_excluded": "anthropic",
|
|
56
|
+
"verdict": "escalate",
|
|
57
|
+
"receipt_id": "prism-01kwbajx31dj9gcf5xn3cn5ydg",
|
|
58
|
+
"receipt_signature": "272c892124e3bc13a76b2674fa361b1d65aee6a588c74604cf4ae4e7c9440a8ba7888175b9ec1286fe87490121f694f64cd30adc3ffc0e1b31cd3365b7b38901",
|
|
59
|
+
"receipt_chain_sha256": "499b63905064a5e25fd1801c5530504c94742f2183c4d3c8eb545a20cfbb112e"
|
|
60
|
+
},
|
|
61
|
+
"lock_sha256": "sha256-AOzuFtiRt4iXnsVqEvcB5hms1oO1LcjRrin96aQnUHE="
|
|
62
|
+
}
|