@ara-commons/ara-skills 0.3.0 → 0.4.0

package/skills/research-visualizer/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: research-visualizer
+description: |
+  Research Visualizer. Renders an existing Agent-Native Research Artifact (ARA) into ONE
+  self-contained, interactive HTML file showing the AI scientist's step-by-step research process:
+  a clickable process map of the exploration tree (branches and dead ends included) on the left,
+  and a per-step drill-down on the right — what the step did, why (the linked claim), the real
+  result (verbatim grounded numbers + inline figures + tables), and the code/artifact pointer.
+  Read-only consumer of the artifact — it never changes how research is done.
+  When the ARA carries them, it also surfaces (each optional, only when present) the related-work
+  dependency graph, the problem framing, a concepts glossary with in-text term popovers, and the
+  solution recipes — reached from header disclosures without leaving the process map.
+  Accepts either an existing ARA or raw research input (a paper, repo, run logs, or notes); when the
+  input is not yet an ARA it is compiled into one first, then visualized.
+
+  TRIGGERS: visualize, visualizer, trajectory view, render the ARA, see the steps, step-by-step view,
+  process map, replay the trajectory, watch the agent work, drill into steps,
+  visualize a paper, visualize a repo, visualize a run
+argument-hint: "[ara-dir] [--output <path>]"
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash(python3 *|base64 *|find *|ls *|open *)
+metadata:
+  author: ara-commons
+  category: research-tooling
+  version: "1.0.0"
+  tags: [research, visualization, trajectory, exploration-tree, html]
+---
+
+# Research Visualizer
+
+You render an existing ARA into a single portable HTML view of the agent's step-by-step process.
+You are a **read-only consumer**: you read the artifact and emit a file; you never edit the ARA.
+
+You operate as a first-class agent — use your native tools directly. The heavy rendering logic is
+**already written** in `references/trajectory-template.html`; you do NOT rewrite it. Your job is to
+parse the ARA into one `ARA_DATA` JSON object, inline the figures, and inject that object into the
+template's data slot.
+
+## What you produce
+
+One self-contained file, default `<ara-dir>/trajectory.html` (override with `--output`):
+- All data, tables, and figures (base64) inlined — no server, no network, no CDN. Double-click to open.
+- Built by populating the canonical scaffold, so every generated view is structurally consistent.
+
+## v1 boundaries (do not exceed)
+
+- **Post-hoc** visualization of a finished/in-progress ARA. No live/real-time mode.
+- **Self-contained from the ARA directory alone.** Do NOT open or inline anything outside the ARA dir.
+  `src/artifacts.md` run-store pointers and node `source_refs` (external journal `file:line`) are shown
+  **as pointers/chips, not resolved**. (External resolution is a planned future extension — out of scope.)
+- **Single ARA.** No cross-ARA comparison.
+
+## Pipeline
+
+1. **Args.** Resolve `<ara-dir>` (default: the ARA in the current working context / most-recently
+   referenced). Resolve `--output` (default `<ara-dir>/trajectory.html`).
+1b. **Precondition — the input must be an ARA; if it is not, compile it first.** Decide with one
+   observable test: does the resolved input expose a parseable `trace/exploration_tree.yaml`
+   (**≥1 node**) — directly, or as a standard ARA directory layout?
+   - **It is an ARA** → continue to Validate unchanged.
+   - **It is not an ARA** — the input is raw research material (a paper/PDF, a code repository, a
+     run/log directory, notes, or any directory with no exploration tree) → **invoke the `compiler`
+     skill on that input to produce an ARA**, then set `<ara-dir>` to the compiler's output artifact
+     and continue. Do not hand-roll an ARA yourself; the compiler is the only path that builds one.
+     Default `--output` to `<compiled-ara-dir>/trajectory.html` unless the user set it.
+   Only if the compiler still yields no exploration tree does the Validate step's "no process" message apply.
+2. **Validate — the exploration tree is the ONLY hard requirement.** Confirm
+   `trace/exploration_tree.yaml` exists and parses to **≥1 node**; if not (and the precondition's
+   compile step has already run), tell the user there is no process to show (this replaces the old
+   `PAPER.md` "is-this-an-ARA?" guard). Everything else —
+   `PAPER.md`, `logic/`, `src/`, `evidence/`, and the four enrichment layers — is **optional
+   enrichment**: glob whatever is present. If `PAPER.md` is absent, synthesize a minimal `meta` (title
+   from a tree-level `title:` or the dir name; empty `abstract` hides the disclosure). This is the
+   **raw-trajectory path**: the skill produces a useful step-by-step view from *just the tree* (a raw
+   agent run), not only a fully-compiled ARA — see `references/parsing.md` §7.
+3. **Parse the trace** into normalized nodes. The field conventions vary across ARAs — follow
+   `references/parsing.md` exactly (handles `tree:` vs `root:`, generic vs type-named fields,
+   `evidence:` routing, `isolated`, `also_depends_on`). Every node must yield a `title` + `body`.
+4. **Parse the hub layers — each only when present (all optional now):** `logic/claims.md` (the
+   binding hub *when it exists*), `logic/experiments.md`, `evidence/README.md` (figure/table ↔ claim
+   reverse index), `src/artifacts.md`, `logic/solution/*`. A missing layer simply contributes nothing;
+   the node still renders from its own `title`/`body`/`thinking`.
+   **Also parse the four OPTIONAL enrichment layers when present, per `references/parsing.md` §8:**
+   `logic/problem.md`→`context`, `logic/concepts.md`→`glossary` (+ build the `lexicon`),
+   `logic/related_work.md`→`dependencies`, `logic/solution/*.md`→`recipes` (role-classify by content, not
+   filename). A missing file/dir omits its key entirely. Reproduce statements/deltas/definitions/relations/
+   headings/quotes/cells **verbatim**.
+5. **Build each node's drill-down.** When `logic/claims.md` exists, follow the claim-hub chain in
+   `references/binding.md` (node → `evidence:[C##]` → claims → {Sources quotes, figures/tables,
+   experiments, artifact pointers}). When it does **not**, the drill-down is just the node's own
+   narrative (`thinking`/`body`) — every claim/result/verified block is empty and omitted.
+5b. **Bind the enrichment layers** per the "four enrichment layers" section of `references/binding.md`:
+   build `claimIds`/`nodeByClaim`/`conceptNames`/`rwIds`; resolve every `refs[].target` (drop danglers,
+   never link off-ARA); derive each node's `built_on`/`rejected_here` (dependency→claim→node, bucketed by
+   `relation_norm`), `concepts` (whole-word name-match), and `recipe_refs` (recipe→claim→node); mark
+   cross-agent entries. All per-node enrichment fields default `[]`.
+6. **Inline figures.** For each referenced figure that has a real raster (`evidence/figures/*.png`),
+   base64-encode it and put the `data:` URI in `figures[].img`. Use Bash, e.g.
+   `python3 -c "import base64,sys;print('data:image/png;base64,'+base64.b64encode(open(sys.argv[1],'rb').read()).decode())" <path>`.
+   For data-only figure markdown (no raster), render its data table instead (as a `tables[]` entry).
+   **Also inline code diffs + the artifact index:** for each node with `code_change.diff_file`, read that
+   tracked `evidence/changes/<id>.diff.md` sidecar and inline its fenced diff text into `code_change.diff`
+   (parallel to figures); build the top-level `artifacts[]` index from `src/artifacts.md` so
+   `base_artifact`/`variant_artifact` ids resolve; carry each node's verbatim `thinking` straight through.
+   Sanitize all three verbatim fields per the Injection contract. The visualizer never computes a diff
+   itself and never opens the external store — it only inlines what the ARA already contains.
+7. **Assemble `ARA_DATA`** (exact schema in `references/binding.md`) and **inject** it: replace ONLY
+   the JSON between `/* __ARA_DATA_BEGIN__ */` and `/* __ARA_DATA_END__ */` in the
+   `<script id="ara-data">` block of a copy of the template. Write the result to the output path.
+   Include `context`/`glossary`/`dependencies`/`recipes` and the per-node `built_on`/`rejected_here`/
+   `recipe_refs`/`concepts` **only when their sources exist** — omit absent keys entirely (no empty
+   stubs). A payload omitting all of them stays byte-compatible with the v1.0 schema.
+8. **Report** the output path. Optionally open it (`open <path>` on macOS). Print a one-line summary
+   (node count, dead ends, figures inlined, which of the four enrichment overlays were emitted with their
+   term/dependency/recipe counts, danglers dropped, any pointers left unresolved).
+
+## Injection contract (critical)
+
+- The injected payload MUST be valid JSON (it is read with `JSON.parse`). The template strips only the
+  two named marker comments before parsing, so the payload is otherwise pure JSON.
+- It must not contain the literal substring `</script>`, nor the literal marker strings
+  `/* __ARA_DATA_BEGIN__ */` / `/* __ARA_DATA_END__ */`. Escape any `<` in inlined markdown/text as `&lt;`
+  (or `<`) — this also neutralizes `</script>`. (A bare `*/` inside a string value is harmless to
+  `JSON.parse`; only the exact marker strings would be stripped.)
+- **The verbatim free-text fields `thinking` and `code_change.diff` are the high-risk carriers** (source
+  code routinely contains `/* … */`). If either marker token would appear in their text, break it (e.g.
+  insert a zero-width space inside `__ARA_DATA_…`) so the global marker-strip can't delete it from inside
+  a value. Re-validate: a node whose `thinking`/`diff` contains a marker token MUST round-trip intact.
+- Do not touch anything else in the template — only the bytes between the two markers.
+- After writing, re-validate: the file still parses (the embedded JSON loads). If a figure pushed the
+  file very large, apply the size guards in `references/binding.md` (truncate logs/tables, keep figures).
+
+## Faithfulness (hard rules)
+
+- Reproduce claim `Statement`s, `Sources` quotes, and table numbers **verbatim** — never paraphrase,
+  never invent. Missing data → set the field empty/omit (the viewer shows "No …"); never fabricate.
+- Provenance, `support_level`, and `status` are shown **only if present** in the source; do not guess.
+- Dead-end nodes and `isolated` subtrees must be carried through faithfully — they are the most
+  valuable things to display, not noise to drop.
+- For the enrichment layers: relation strings, definitions, constraint headings, and footprint citations
+  are reproduced **verbatim**; relation enums are open (compound `bounds / refutes` / transition
+  `extends → quarantined` kept as written; `relation_norm` is for color only). Never normalize a heading
+  or invent a typed sub-field. A `refs[].target` is set only on real in-ARA resolution; dangling refs are
+  flagged, never silently corrected or dropped. `built_on`/`concepts`/"used by" name-matches are
+  best-effort hints (marked "inferred"), never asserted as facts.
+
+## Verify
+
+Run on any ARA and confirm these properties — no named fixtures required:
+- Opens by double-click: no server, no network, no console errors.
+- Full process map: nesting, branches, dead ends marked, any `isolated` subtree boxed, `depends_on` chips.
+- Drill-down renders whichever blocks are present (what / why / result-with-inline-figure / how-verified /
+  code-or-pointer), correctly under **both** field dialects in `references/parsing.md`.
+- Verbatim quotes/numbers; nothing fabricated; self-contained from the ARA dir (no needed external refs).
+- Re-running reproduces the same structure (data differs only as the ARA differs).
+- **Enrichment layers:** a layer's header button appears only when its source exists; an ARA with none of
+  the four layers renders identically to v1.0 (no layer bar, no node chips). Open each emitted overlay and
+  confirm verbatim relations/definitions/recipe cells, ungrounded/dangling/cross-agent markers, and that
+  the `built_on`/`rejected_here` chips + the `⊕/⊘` map marker deep-link into Dependencies. Glossary
+  popovers fire on body terms; inline `$LaTeX$` renders with no network.
+- **Degradation:** a `minimal-artifact` (only `problem.md`) shows only the Context button, others absent,
+  popovers off, per-node chips empty, zero console errors.
+- **Compile-first (non-ARA input):** pointing the skill at raw research material with no exploration
+  tree (a paper, a repo, a run/log dir) triggers the `compiler` skill first, then visualizes the
+  resulting ARA — the output is identical to running the compiler then the visualizer by hand.
+- **Raw trajectory (the decoupled path):** a **tree-only** ARA — just `trace/exploration_tree.yaml`, no
+  `PAPER.md`, no `logic/`, no `evidence/` — still renders the full process map + each step's narrative
+  (`thinking`/`body`), with no layer bar and no claim/result/verified blocks, zero console errors. This
+  is a first-class supported input, not a failure mode.
+
+Cover the variant axes with whatever ARAs you have: both root forms (`tree:`/`root:`), both field
+dialects (generic / type-named), figures present as real raster vs. data-markdown-only, `src/` as a
+pointer index vs. transcribed code, and an `isolated` subtree if any artifact has one.

package/skills/research-visualizer/references/binding.md

@@ -0,0 +1,245 @@
+# Binding — the claim-hub drill-down chain + the `ARA_DATA` schema
+
+This is the read step. For each normalized trace node you build one drill-down bundle by following
+the artifact's **claim-mediated** cross-layer links, then you emit the whole thing as one `ARA_DATA`
+object that the scaffold renders.
+
+> **The claim hub is OPTIONAL — the tree is the only hard requirement.** Every layer this file binds
+> (`logic/claims.md`, `logic/experiments.md`, `evidence/`, `src/artifacts.md`, the four enrichment
+> layers) is enrichment: when its dir/file is absent, that binding step **no-ops** and the
+> corresponding `ARA_DATA` arrays stay `[]` (omitted by the renderer). A node ALWAYS renders from its
+> own normalized `title` / `body` / `thinking` (parsing.md §3, §7a) — so a **raw, tree-only
+> trajectory** with no `logic/` and no `evidence/` produces a complete process map + per-step
+> narrative. The chain below is what runs *when the hub exists*; nothing here may hard-fail on an
+> absent layer.
+
+## Why claim-mediated (and not node → `.py` + table)
+
+Current ARAs usually have **no per-step source file**: `src/` is `artifacts.md` (a pointer index into
+an external run store) + `environment.md`. The binding hub is `logic/claims.md`. A trace node points
+at claims via `evidence: [C##]`; the claim carries the *why*, the grounded *result* (verbatim
+`Sources` quotes with `file:line`), the verification (`Proof: [E##]`), and the figures/tables that
+cite it. So the real grounded numbers are **already inside the ARA** — you do not need the external
+store (v1 leaves those as pointers).
+
+## Resolution chain (per node)
+
+```
+NODE (type, content+outcome, support_level, source_refs, evidence:[C##], also_depends_on, isolated)
+  ├─ WHAT        → node title + body (+ outcome) ............................ → node.title / node.body
+  ├─ WHY         → evidence:[C##] → logic/claims.md ......................... → node.why[]
+  │                 each claim: Statement, Status, Conditions, Falsification,
+  │                 Dependencies, provenance
+  ├─ HOW VERIFIED→ claim.Proof:[E##] → logic/experiments.md ................. → node.verified_by[]
+  │                 each experiment: Run (pointer), Setup, Metrics
+  ├─ RESULT      → claim.Sources «quote ← file:line» + Evidence basis ....... → node.result.sources[]
+  │              + evidence/README.md reverse-lookup: figures/tables citing C## → node.result.figures[]/tables[]
+  │              + evidence/data/*.json (raw, in-artifact) .................. → node.result.data[]
+  └─ CODE/ARTIFACT→ src/artifacts.md pointer(s) + logic/solution/* recipe ... → node.artifact[]
+```
+
+### evidence/README.md reverse-lookup
+`evidence/README.md` has a `Claims` column mapping each figure/table file → the `C##` it grounds.
+Build a map `claim_id → [evidence files]` once. For a node, union the evidence of all its claims.
+Fallback if no README row: scan the figure/table `.md` for an inline `Supports C##` / `grounds C##`.
+
+### Figures
+- Real raster (`evidence/figures/<name>.png` beside the `.md`) → base64-inline into `figures[].img`.
+- Data-only figure markdown (no raster) → render its data table as a `tables[]` entry instead.
+- A figure's caption/title comes from the first heading or the `What it shows` section of its `.md`.
+
+### What counts as "code" now
+There is usually no `src/execution/*.py`. Populate `artifact[]` from `src/artifacts.md` (the run-index
+row / `record_configs/` path / named submitted variant file) plus the relevant
+`logic/solution/*.md` recipe section. Only when a real transcribed `src/execution/*.py` exists
+(legacy / paper-only code) do you point at that file. **Never resolve the external store in v1** —
+the pointer text is the value.
+
+### The changed-code diff (`node.code_change`) — compiler-produced, visualizer-rendered
+A full ARA may carry, per experiment node, the **unified diff** the step represents. The addresses
+live in ONE place and are referenced by id (weak coupling):
+
+`node.code_change` → `evidence/changes/<node-id>.diff.md` (the diff **text**) → `artifacts[]` /
+`src/artifacts.md` entry (path + sha256 + original location) → the original repo.
+
+- The diff **text** is grounded by citing the two **artifact ids** (`base_artifact`, `variant_artifact`),
+  never an embedded path. Whole scripts stay pointers in `src/artifacts.md` (Rule 14) — the diff is a
+  derived, grounded view (≈ a `derived_subset` table), not a copy of the artifact.
+- **`diff_file` → `diff` inlining** (parallel to figures' `.md`→base64 `img`): on disk the node carries
+  `code_change.diff_file: "evidence/changes/<id>.diff.md"`; the visualizer reads that **tracked** sidecar
+  and inlines its fenced diff text into `code_change.diff` in `ARA_DATA`, so the rendered HTML stays
+  self-contained (the sidecar lives inside the ARA dir).
+- **`artifactById`**: the visualizer builds an `id → artifacts[] entry` map (parallel to `nodeByClaim`)
+  and resolves `base_artifact`/`variant_artifact` into the shown-not-resolved pointer chip under the diff.
+- **Degrade**: when the scripts don't resolve at compile time (store absent), the compiler emits
+  `code_change` with the artifact ids + a `note` but no diff; the viewer shows a pointer chip, not a diff.
+- **Marker safety**: `diff` and `thinking` are verbatim, so the producer MUST ensure neither the literal
+  `/* __ARA_DATA_BEGIN__ */` / `/* __ARA_DATA_END__ */` tokens nor `</script>` appears in any inlined
+  string (escape `<`→`&lt;`; break the marker tokens). See SKILL.md "Injection contract".
+
+## The `ARA_DATA` object (exact schema the scaffold reads)
+
+```jsonc
+{
+  "meta": {
+    "title":  "<PAPER.md frontmatter title>",
+    "authors": ["..."],            // [] if none
+    "year":   "", "venue": "", "ara_dir": "<dir name>",
+    "abstract": "<PAPER.md abstract>"   // "" to hide the Abstract disclosure
+  },
+
+  // Traversal order for Replay and ← →. A flat list of node ids in the order to step through
+  // (typically a pre-order DFS of the tree). If omitted, the scaffold derives a DFS from `parent`.
+  "order": ["N01", "N02", "N03", "..."],
+
+  // OPTIONAL addressable artifact index (from src/artifacts.md). Each script the compiler points at
+  // gets a stable id so a node's code_change can reference it BY ID (no embedded path). Omit if absent.
+  "artifacts": [
+    { "id":"A01", "name":"<artifact name>", "path":"<repo-relative path>", "sha256":"<...>",
+      "original_location":"<store/repo ref>", "pointer":"<src/artifacts.md pointer text>" }
+  ],
+
+  "nodes": [
+    {
+      "id": "N02",
+      "type": "experiment",        // question|experiment|decision|dead_end|pivot|insight|<other ok>
+      "parent": "N01",             // id of the nesting parent, or null for a root
+      "title": "<normalized step title>",            // see parsing.md
+      "body":  "<what the step did / its outcome>",
+      "thinking": "<verbatim agent deliberation — why it did/branched; OPTIONAL>", // primary block; falls back to body
+      "support_level": "explicit", // "explicit"|"inferred"|null
+      "isolated": false,           // true → rendered in a separated dashed box
+      "depends_on": ["N00"],       // also_depends_on cross-edges (ids); [] if none
+      "source_refs": ["<external path:line — shown as a chip, NOT resolved>"],
+
+      "why": [                     // from evidence:[C##] → claims.md
+        { "id":"C01", "title":"...", "statement":"<verbatim Statement>",
+          "status":"supported", "conditions":"...", "falsification":"...",
+          "dependencies":["C00"], "provenance":"<as stated in source, else null>" }
+      ],
+
+      "result": {
+        "sources": [ { "quote":"<verbatim Sources quote>", "ref":"<file:line>" } ],
+        "figures": [ { "id":"<figure id>", "caption":"...", "kind":"quantitative_plot",
+                       "img":"data:image/png;base64,<...>" } ],
+        "tables":  [ { "id":"<table id>", "caption":"...", "markdown":"| col | ... |\n|---|...|" } ],
+        "data":    [ { "id":"<data id>", "path":"evidence/data/<name>.json", "note":"raw" } ]
+      },
+
+      "verified_by": [             // from claim.Proof:[E##] → experiments.md
+        { "id":"E01", "title":"...", "run":"<Run pointer text>", "setup":"...", "metrics":"..." }
+      ],
+
+      "artifact": [                // src/artifacts.md pointers + solution recipe refs (pointer text only)
+        { "name":"<artifact / family name>", "pointer":"<src/artifacts.md pointer text>", "what":"pointer index entry" }
+      ],
+
+      "code_change": {             // OPTIONAL — the changed-code diff for this step (compiler-produced)
+        "base_artifact":"A01",     // → artifacts[].id (holds path+sha+original_location)
+        "variant_artifact":"A07",  // → artifacts[].id
+        "lang":"python",
+        "diff":"<unified-diff text, inlined by the visualizer from evidence/changes/<id>.diff.md>",
+        "note":""                  // set (with diff absent) when the scripts didn't resolve → pointer-only chip
+      }
+    }
+    // ... one object per trace node
+  ]
+}
+```
+
+### Field rules
+- Every node MUST have `id`, `type`, `title`, `parent` (or null). All other arrays default to `[]`,
+  scalars to `null`/`""`. The scaffold tolerates missing optional fields.
+- Put **only what the source contains**. Empty `why`/`result`/`verified_by`/`artifact` is fine and
+  common (e.g. a bare `decision` node) — the viewer simply omits those blocks. `thinking` and
+  `code_change` are likewise optional; omit when absent (a payload without them is byte-compatible).
+- `status` is lower-cased by the viewer for styling; pass it as written (`Supported`, `hypothesis`, …).
+
+### Size guards
+- Inline every figure (they are the point), but cap a single inlined table/log render at a few hundred
+  lines; if longer, truncate and append a final row/line `… +N more (truncated)`.
+- If the assembled file would be very large, prefer truncating `tables`/`data` text over dropping
+  figures, and never drop nodes.
+
+---
+
+# The four enrichment layers — schema additions + cross-linking
+
+These surface `logic/problem.md` / `logic/concepts.md` / `logic/related_work.md` / `logic/solution/*`
+(parsed per `parsing.md` §8). **Everything here is OPTIONAL and additive** — omit any key whose source
+is absent; a payload with none of these stays byte-compatible with the prior schema and the renderer is
+inert for any key it doesn't see.
+
+## Schema additions
+
+A shared **typed-ref** primitive (used by every layer's `refs`/`grounding`):
+```jsonc
+{ "raw": "<verbatim token, always shown>",
+  "kind": "claim|concept|related_work|observation|gap|assumption|experiment|node|source|figure|pr|arxiv|doi|url|unknown",
+  "target": "<in-ARA anchor id, or null>" }   // non-null ONLY when it resolves inside THIS ARA
+```
+
+Four OPTIONAL top-level keys:
+```jsonc
+"context": {                 // logic/problem.md ; omit if absent
+  "title":"Problem", "summary":"",
+  "sections":[ { "role":"setting|observations|gaps|insight|assumptions|other", "heading":"<verbatim H2>",
+    "present":true,           // false ⇒ greyed "(not specified)" stub for an expected-but-absent role
+    "entries":[ { "ent_id":"O2", "inferred_id":false, "ent_title":"", "text":"<verbatim>",
+                  "unstructured":false, "fields":[{"label":"<verbatim>","value":"<verbatim>"}], "refs":[/*typed-ref*/] } ] } ] },
+"glossary": {                // logic/concepts.md ; omit ⇒ popovers disabled
+  "title":"Concepts", "groups":[{"name":"","term_ids":["G07"]}],
+  "terms":[ { "term_id":"G07", "name":"<verbatim, the match key>", "aliases":[], "group":"",
+              "definition":"<verbatim, may hold $LaTeX$>", "fields":[{"label":"","value":""}], "related":["G08"], "refs":[] } ],
+  "lexicon": { "<lowercased surface>":"G07" } },   // precomputed surface→term_id for popovers
+"dependencies": {            // logic/related_work.md ; omit if absent
+  "title":"Related Work / Dependencies", "preamble":"", "legend":[{"type":"extends","gloss":""}], "attribution":"",
+  "entries":[ { "rw_id":"RW02", "inferred_id":false, "name":"", "relation_raw":"<verbatim, compound/transition kept>",
+                "relation_norm":"baseline|imports|extends|bounds|refutes|", "delta":"", "adopted":"",
+                "grounding":[/*typed-ref: pr|arxiv|doi|source|url*/], "claims":["C03"], "cross_agent":false, "is_footprint":false } ],
+  "footprint":[ {"ref_id":"16","citation":"<verbatim one-liner>"} ] },
+"recipes": {                 // logic/solution/* ; omit if dir absent
+  "title":"Solution",
+  "files":[ { "file":"method.md", "role":"constraints|method|heuristics|algorithm|architecture|recipe", "file_title":"",
+    "sections":[ { "sec_id":"", "heading":"<verbatim, never normalized>", "kind":"table|steps|kv|code|math|prose",
+                   "markdown":"", "steps":[], "fields":[], "code":"", "text":"<verbatim fallback>", "warn":"", "refs":[] } ] } ] }
+```
+
+Four OPTIONAL per-NODE fields (default `[]`):
+```jsonc
+"built_on":     [ {"rw_id":"RW02","name":"","relation_raw":""} ],   // baseline/imports/extends deps a node's claims cite
+"rejected_here":[ {"rw_id":"RW10","name":"","relation_raw":""} ],   // bounds/refutes deps
+"recipe_refs":  [ {"file":"","sec_id":"","heading":"","role":""} ], // recipe sections a node's claims cite
+"concepts":     [ "<verbatim term name>" ]                          // glossary terms mentioned in this node (popover anchors)
+```
+Field rules: every new key omittable; `text`/`definition`/`delta`/`value`/relations/headings/quotes/cells
+**verbatim**; `relation_raw` always preserved (compound/transition survive), `relation_norm` is color-only
+and may be `""`; `target` non-null only on in-ARA resolution (off-ARA `pr`/`arxiv`/`doi`/`§`/file:line stay
+`null` → inert chip); no fabrication.
+
+## Cross-linking (the binder runs AFTER nodes + claims are parsed)
+
+The hub stays `logic/claims.md`. All linking is best-effort, computed once into lookup maps, then attached.
+
+- **B.0 anchors:** `claimIds` (the `## C\d+` set), `nodeIds`, `nodeByClaim` (invert each node's `why[].id`),
+  `conceptNames` (lowercased `glossary` names+aliases), `rwIds`.
+- **B.1 claim-ref resolution (single pass over all layers' refs):** `kind:"claim"` ref → `target` if
+  `∈ claimIds`, else `target:null` and **flag dangling** (renderer greys it "broken link" — never a live
+  dead link). Same for `related_work`/`concept`/`node` targets against their id sets.
+- **B.2 dependencies ↔ claims/nodes:** forward edge is in-data (`entries[].claims[]`). Build the reverse
+  `claim_id → [rw_id]`; also harvest `claims.md`'s own `**Dependencies.**` lines as extra rw-edges (so the
+  edge exists even when related_work omits the back-ref).
+- **B.3 per-node `built_on` / `rejected_here` (the chips):** `rwSet = ⋃ over n.why[].id of reverse(claim→rw)`.
+  For each rw, bucket by `relation_norm`: `∈{baseline,imports,extends}` → `built_on`; `∈{bounds,refutes}` →
+  `rejected_here`; `""`/transition → first keyword, default `built_on`, **keep `relation_raw` literal**.
+  Dedupe by `rw_id`; empty buckets stay `[]` (no chip row). Cross-agent deps render a distinct "↔ other-agent"
+  ribbon.
+- **B.4 concepts ↔ everything:** concepts are leaf provenance (no `trace:N` refs). Build `node.concepts[]` by
+  **sideways name-match** — whole-word, case-insensitive scan of each node's `title`+`body`+claim statements
+  against `conceptNames`. The Concept overlay's "used by" is that scan inverted, marked **"mentions
+  (inferred)"** and visually distinct from id-resolved links. Outbound concept `refs` resolve normally.
+- **B.5 recipes ↔ claims/nodes (`recipe_refs[]`):** a node's `recipe_refs` = recipe sections whose `refs[]`
+  cite one of the node's `why` claims (∪ sections a node's claim cites back); dedupe by `(file,sec_id)`.
+  Unresolvable `C##` ⇒ inert muted chip ("referenced but not linked"), never dropped.
+- **B.6 determinism & cost:** all maps built once, O(nodes+claims+refs); stable source order; a wrong/no
+  match never blocks rendering. The browser-side `lexicon` is precomputed so no source regex runs at runtime.

package/skills/research-visualizer/references/parsing.md

@@ -0,0 +1,211 @@
+# Parsing tolerance — normalize the exploration tree across ARA variants
+
+The canonical `exploration-tree-spec.md` uses **generic** field names; many real artifacts instead
+use **type-named** fields. Parse against the *model*, tolerate the *variants*. Never
+hardcode to one example; never assume a fixed field set. The goal: **every node yields an `id`,
+`type`, `title`, `body`, and a `parent`** (plus optional metadata), no matter which dialect.
+
+## 1. Root form
+
+Accept both:
+- `tree:` → a **list** of root nodes (canonical).
+- `root:` → a **single** root node. Treat it as a one-element root list.
+
+Children nest under each node's `children:` list. A node's `parent` is the nearest enclosing node;
+top-level nodes have `parent: null`.
+
+## 2. Node identity & type
+
+- `id` — required (any stable identifier the tree uses, e.g. `N01`, `E07`). Keep verbatim.
+- `type` — one of `question | experiment | decision | dead_end | pivot | insight` (or anything else;
+  unknown types still render with a neutral glyph). Keep verbatim.
+- `support_level` — `explicit | inferred` if present, else `null`.
+- `source_refs` — list of strings if present, else `[]`. **External pointers; shown, never resolved.**
+- `isolated: true` — carry through (renders in a separated box).
+- `also_depends_on: [ids]` → emit as `depends_on` (DAG cross-edges).
+- `thinking` — verbatim agent deliberation, **passed straight through** (the primary reasoning block).
+  Absent ⇒ omit. Never paraphrase or synthesize it.
+- `code_change` — when the compiler wrote one onto the node (`base_artifact` / `variant_artifact` /
+  `lang` / `diff_file`), **pass it through**. The `diff_file`→`diff` inlining and the top-level
+  `artifacts[]` index are done in the binding/inline step (binding.md); the visualizer never computes a
+  diff itself. Absent ⇒ omit.
+
+## 3. Title + body normalization (the dialect bridge)
+
+For each node, derive a **display title** and a **body** by probing known fields **in order**, per type.
+Use the first non-empty match; fall back gracefully so nothing is blank.
+
+| type        | title — try in order                          | body — try in order                                   |
+|-------------|-----------------------------------------------|-------------------------------------------------------|
+| question    | `title` → `question`                          | `description` → `question` (if used as title, leave body empty) |
+| experiment  | `title` → first sentence of `experiment`      | `result` → `outcome` → `experiment`                   |
+| decision    | `title` → first sentence of `decision`        | `choice` (+ `alternatives`) → `decision`              |
+| dead_end    | `title` → first sentence of `dead_end`        | `failure_mode`/`why_failed` (+ `hypothesis`,`lesson`) → `dead_end` |
+| pivot       | `title` → `trigger`                           | `from` → `to` → `trigger`                              |
+| insight     | `title` → first sentence of `insight`         | `description` → `insight`                              |
+| *(other)*   | `title` → `<type>` field → `description`       | the `<type>` field → `description`                     |
+
+Rules:
+- "first sentence of X" = X truncated at the first `. ` or ~80 chars — only when there is no separate
+  `title`. If a single field is both the only content and long, use a truncated form as title and the
+  full text as body.
+- `decision`: append `alternatives` to the body as "alternatives: a; b" when present.
+- `dead_end`: prefer to show *why it failed* in the body — it is the most valuable content.
+- Never emit an empty title. If truly nothing usable, use the `id`.
+
+## 4. `evidence:` routing
+
+A node's `evidence:` (and a decision's `evidence:` string) may mix kinds. Split tokens and route:
+- `C\d+` → claim IDs → drive the `why` / `result` binding (see `binding.md`).
+- `Figure N` / `Table N` / a figure/table filename → evidence file refs → `result.figures`/`tables`.
+- `§...` / prose → keep as a `source_refs`-style chip (context only).
+
+## 5. Provenance
+
+May appear as:
+- a per-claim `provenance:` / `Provenance:` field, **or**
+- prose in the `claims.md` header ("all claims are `ai-executed`; C06, C08 are `user-revised`").
+
+Capture whichever exists and attach the right tag to each claim in `why[]`. If neither exists, omit
+provenance — do not guess.
+
+## 6. Claims, experiments, evidence index (the hub layers)
+
+- `logic/claims.md` — split on `## C\d+` headings. Pull `Statement`, `Status`, `Conditions`,
+  `Falsification criteria`, `Proof` (→ `E##`), `Sources` (verbatim «quote» + `← file:line`),
+  `Dependencies`. Field labels are bold-prefixed (`**Statement.**` or `- **Statement**:`) — match both.
+- `logic/experiments.md` — split on `## E\d+`. Pull `Verifies` (→ `C##`), `Run`, `Setup`, `Metrics`.
+- `evidence/README.md` — parse the Tables/Figures index to build `claim_id → [evidence files]`.
+
+## 7. Degrade, don't fail — the tree is the only hard requirement
+
+Any missing/oddly-shaped field → fall back per the tables above and continue. A smaller honest view
+is correct. **Hard-stop on exactly one condition: `trace/exploration_tree.yaml` is absent or parses to
+zero nodes** (nothing to show). Minimal-validity guard = *the tree parses AND yields ≥1 node* — this
+replaces the old "`PAPER.md` missing ⇒ not an ARA" guard, which no longer hard-stops (a missing
+`PAPER.md` just means the visualizer synthesizes a minimal `meta` from a tree-level `title:` / the dir
+name). Everything else — `logic/`, `evidence/`, `src/`, the four enrichment layers — is optional; absent
+⇒ contributes nothing, never an error.
+
+### 7a. Raw-trajectory input mode (first-class)
+
+The minimum a node needs to render a useful step is `id` + (`title` **or** a type-named text field);
+`body` / `thinking` are optional but make the step legible. So a **bare exploration tree with no
+`logic/`, no `evidence/`, and no `PAPER.md`** — i.e. a raw agent run — is a fully supported input, not a
+degraded one. Each node renders from its own normalized `title` / `body` (per §3) plus its `thinking`
+(the agent's deliberation, when the source carries it — a verbatim pass-through field); the
+`why` / `result` / `how-verified` blocks are simply empty and omitted.
+
+**Adapter recipe (generic agent run → minimal tree).** A typical agent log is a sequence of steps, each
+a `{thought, action, observation/result}`. Map it onto the tree:
+- one tree node per step (or per meaningful decision/experiment); `id` = the step index/label.
+- `type` from the step kind: a tried approach → `experiment`; a chosen direction → `decision`; an
+  abandoned/failed approach → `dead_end`; an opening/guiding question → `question`.
+- `title` = a one-line summary of the step (first sentence of the action, ≤80 chars).
+- `thinking` = the agent's thought/deliberation for the step (**verbatim** — why it did/branched).
+- `body` = what it actually did + what came back (action + observation).
+- `source_refs` = a pointer back to the log line(s) (shown, never resolved).
+- nesting via `children`; convergence via `also_depends_on`; a discarded branch via `isolated`.
+
+No `logic/` or `evidence/` is required; enrich the same tree later (via the compiler) to add claims,
+evidence, and per-node `code_change` diffs.
+
+# 8. The four `logic/` enrichment layers (all optional)
+
+These produce the OPTIONAL `context` / `glossary` / `dependencies` / `recipes` keys (and the per-node
+`built_on` / `rejected_here` / `recipe_refs` / `concepts` fields, derived in `binding.md`). **Each is
+independent and absent ⇒ omit its key entirely** (no empty stubs). The renderer is inert for any key
+it doesn't see, so an ARA with none of these renders exactly as before.
+
+**Governing rule — classify by markdown SHAPE + generic cross-ref regexes; NEVER key on field
+vocabulary.** Do not look for "optimizer", "Fig.", "loss", "shortcut", etc. — only headings, bold-lead
+labels, id tokens, and the ref battery below. Verbatim everywhere: `text`/`definition`/`delta`/`value`,
+all numbers, quotes, table cells, relation strings, and headings are reproduced exactly; a missing
+field is `""`/`[]`; positive-absence signals (`present:false`, `unstructured`, `is_footprint`,
+`inferred_id`) are *shown*, never invented.
+
+## 8.0 Shared sub-routines (define once, reuse for every layer)
+
+- **`splitEntries(sectionText)`** — try delimiters in order, take the first that yields ≥1 hit:
+  (a) `^### ` H3 headings; (b) `^## ` with a leading id token (`^##\s*([A-Za-z]{1,3}\d+|RW\d+|[OGA]\d+)`);
+  (c) top-level bold-lead bullets `^\s*-\s*\*\*(.+?)\.?\*\*`; (d) blank-line-separated paragraph blocks.
+  (d) always succeeds ⇒ prose-only dialects still yield entries.
+- **`probeFields(entryText)`** — collect every `^\s*-?\s*\*\*([^*]+?)\*\*\s*[:—]` labeled leader into an
+  **open** `[{label,value}]` list. **Labels verbatim, no whitelist.** Empty ⇒ `unstructured:true`,
+  `fields:[]`.
+- **`harvestRefs(text)` → `typed-ref[]`** — run the FULL union of patterns; never assume one scheme.
+  The shared **typed-ref** is `{ "raw":"<verbatim token>", "kind":"…", "target":null }`; `target` is set
+  later in the binding pass (it stays `null` for anything that doesn't resolve inside this ARA).
+
+  | pattern | kind |
+  |---|---|
+  | `\[?\b(C\d{1,3})\b\]?` (optionally `(claims.md…)`) | `claim` |
+  | `\bRW\d{1,3}\b` | `related_work` |
+  | `\b([OGA])\d{1,3}\b` | `observation` / `gap` / `assumption` |
+  | `\bK\d{1,3}\b` or a minted term-id | `concept` |
+  | `PR\s*#?\d+` | `pr` |
+  | `arXiv[:\s]\S+` | `arxiv` |
+  | `10\.\d{4,}/\S+` | `doi` |
+  | `[\w./-]+\.\w+:\d+(?:-\d+)?` · `trace:N\d+` · `logic/\S+#\w+` | `source` / `node` |
+  | `§[\w".]+` · `Table\s*\S+` · `Fig\.?\s*\S+` · `Eqn\.?\s*\S+` | `figure` |
+  | `https?://\S+` | `url` |
+
+## 8.1 `logic/problem.md` → `context`
+Absent ⇒ omit (do NOT fabricate context from claims). Split on `^## `; classify each heading by a
+**case-insensitive role-synonym map** (NOT exact names): `setting ⇐ {setting,task,constraints,
+background,context,problem statement}`; `observations ⇐ {observation*,findings,evidence}`;
+`gaps ⇐ {gap*,open question*,challenges,limitations of prior work}`; `insight ⇐ {key insight,insight,
+core idea,approach,thesis}`; `assumptions ⇐ {assumption*,scope,caveats}`. Unmatched ⇒ `role:"other"`,
+heading kept verbatim, **never dropped**. **Merge** same-role sections. Any canonical role that never
+appeared ⇒ emit a `{present:false}` stub (signals the compile, greys the chip). Per section run
+`splitEntries`; `ent_id = \b([OGA]\d+)\b` (width-agnostic) else positional + `inferred_id:true`;
+`ent_title` = text to the first `:`/`—`/`.` else first sentence; `probeFields` (empty ⇒
+`unstructured:true`); `text` = verbatim body (mandatory); `harvestRefs`. `summary` = the insight
+entry's first sentence (else `""`).
+
+## 8.2 `logic/concepts.md` → `glossary`
+Absent ⇒ omit (⇒ popovers globally disabled). **Detect grouping:** a `##` is a *group* (not a term)
+iff it has no `—`/`:` id-separator AND the next non-blank line is a `- ` bullet AND no following
+`**Definition`. Split terms via `splitEntries` (`^##\s+(.+)$` → `^- \*\*(.+?)\.?\*\*` → paragraphs).
+Per term: `term_id` = leading `^(K|C|RW|[A-Z]{1,3})\d{2,}` before the separator, else **mint**
+`G01,G02,…` positionally (a stable popover anchor); `name` = heading minus id/separator (mandatory);
+`aliases` = split a trailing `(…)` and `/`; `definition` = `**Definition**:` value else whole body
+(mandatory); `probeFields` (captures resnet-style `Notation`/`Boundary conditions`; nothing for
+codex/ptb — correct); `related` = any `/relat/i`-labeled field split on commas (resolved to `term_id`s
+in binding); `harvestRefs`. **Build `lexicon`** = lowercased `name` + each alias → `term_id`, skipping
+tokens <3 chars and pure-numeric aliases, first-wins on collision (the renderer uses it for popovers).
+
+## 8.3 `logic/related_work.md` → `dependencies`
+Absent ⇒ omit. Strip the preamble before the first `## ` → `preamble`; parse `**word** (gloss)` pairs →
+`legend[]`; capture a `> **Cross-agent attribution.**` blockquote → `attribution`. Split on `^## `,
+match `^##\s*(RW\d+)?\s*[—:-]?\s*(.*)$`. If a heading has no `RW\d+` and matches `/brief|additional
+referenced|citations/i`, parse its bullets into `footprint[]` (`{ref_id,citation}`) and skip structured
+probing (`is_footprint` entries, or just the `footprint` tail). Per entry: `relation_raw` from a
+` — **type** ` in the heading ELSE a `**Type:**` line — **kept verbatim**, incl. compound
+(`bounds / refutes`) and transition (`extends → quarantined`); `relation_norm` = the first of
+`{baseline,imports,extends,bounds,refutes}` found, else `""` (color only); `name` = heading after the
+separator; `delta` = `**Delta**`/`What changed:`+`Why:` (concatenate) else body; `adopted` =
+`**Adopted elements**`; `grounding`/`claims` = `harvestRefs` partitioned by kind; `cross_agent:true` if
+the entry is named in `attribution` or its relation mentions another agent. Unnumbered ⇒ positional +
+`inferred_id`. **Degrade:** an entry with no source and no claim keeps `grounding:[]`+`claims:[]` (the
+renderer shows a muted "ungrounded").
+
+## 8.4 `logic/solution/*` → `recipes`
+Glob `logic/solution/*.md`; dir absent ⇒ omit. **Role-classify each file by content signal (priority):**
+filename `constraints.md` → `constraints`; filename `method.md` OR title `/method|recipe|procedure|
+process|pipeline/i` → `method`; uniform `^##\s*[A-Z]\d+:` entries → `heuristics`;
+`## Mathematical formulation`/pseudocode fence/`$…$` → `algorithm`; repeated component `###` micro-schema
+→ `architecture`; else → `recipe`. (Only the `constraints.md` filename is special — a universal ARA
+convention, not a field term.) Split each file on `^## ` (recurse `###`). **`heading` kept verbatim —
+never normalized.** Per section pick the **dominant `kind`** in order: table `| … |` → `table`
+(`markdown`); typed bullets `- **K**: v` → `kv` (`fields`); numbered `^\d+\.` → `steps`;
+fenced/ASCII-DAG → `code`; `$$…$$`/`$…$` → `math` (put TeX in `code`); else → `prose`. `text` = verbatim
+body (mandatory fallback); `sec_id` = leading id; `warn` = body under a `/confound|cut-off|incomplete|
+not specified|unverified/i` heading; `refs` = `harvestRefs` (harvest **bare** `C05` too, not only
+`[C05]`). **Degrade:** only `constraints.md` present ⇒ render just it (no "method missing" stub);
+`method.md` absent but `algorithm.md`/`architecture.md` present ⇒ those carry the method role.
+
+## 8.5 Universal absence rule
+Each key is independent; any combination is valid. A `minimal-artifact` (only `problem.md`+`claims.md`)
+yields `context` only — `glossary`/`dependencies`/`recipes` omitted, all per-node enrichment fields
+`[]`, nothing errors.