@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/shared/knowledge_harvest.md

@@ -0,0 +1,220 @@
+---
+name: knowledge-harvest-protocol
+description: Shared protocol for extracting reusable knowledge at project completion — all specialists
+type: reference
+---
+
+# Knowledge Ledger — Harvest Protocol
+
+This protocol runs once, after all reviews in the final phase are complete and before marking `Status: Complete`. It extracts reusable knowledge from the project and writes it to the workspace-wide Knowledge Ledger (`.shards/knowledge/`).
+
+---
+
+## Special case: Autonomous Research (`[AR]`) fan-out
+
+When an AR session ran as a **fan-out** (multiple parallel branches converged by Syn Arbiter — see `autonomous_research.md` Section H):
+
+- **Losing branches do NOT run this protocol independently.** Running harvest in each branch would flood the ledger with duplicate or conflicting candidates across branches exploring adjacent territory.
+- **Only the parent specialist runs harvest**, after promotion (`diverge_protocol.md` Section G), as part of the consolidated Phase 3.
+- Harvest candidates come from:
+  1. The **winning branch's** artifacts (research_brief.md, results.json, iteration files).
+  2. **Cross-branch patterns** that Syn Arbiter flagged in the leaderboard (e.g., "three of four branches hit the same data leakage issue"). These are worth harvesting even when no single branch would have flagged them, because they reveal structural properties of the data or methodology.
+
+For **solo AR** sessions, run this protocol normally at Phase 3 per the agent's `research.md`. Candidate sources are the AR artifacts (brief, results.json, per-iteration files) and the final `research_summary.md` patterns section.
+
+---
+
+---
+
+## Steps
+
+### 1. Review the project
+
+Read the full `project-specs.md` for this project. Identify candidates across four categories:
+
+| Category | Directory | What to look for |
+|----------|-----------|-----------------|
+| **Entities** | `entities/` | Data table quirks, column semantics that surprised you, grain discoveries, type mismatches (e.g., "user_id is a string UUID in billing but an integer in events"), unexpected nullability |
+| **Infrastructure** | `infrastructure/` | Warehouse behaviors, API rate limits, system quirks, connection patterns, freshness guarantees (or lack thereof) |
+| **Patterns** | `patterns/` | Reusable SQL snippets, Python patterns, transformation techniques, join strategies that solved a non-obvious problem |
+| **Features** | `features/` | Verified ML features with SQL, grain, and performance data — **Data Scientist and ML Engineer only** |
+
+### 1b. Check for contradiction resolutions
+
+Search `project-specs.md` for lines matching the structured template:
+
+```
+**Knowledge contradiction:** "<title>" claims ...
+```
+
+This template is written by the checkpoint protocol (`knowledge_checkpoint.md`) whenever an agent observes data that contradicts a ledger entry during execution.
+
+For each contradiction found where "Ledger update needed: Yes":
+
+- Identify the existing knowledge file path from INDEX.md (match by title)
+- Draft an **update candidate** with:
+  - **Action:** Update (not a new entry — overwrite the existing file)
+  - **Existing entry:** `.shards/knowledge/<category>/<filename>.md`
+  - **Change:** old claim → new claim (extracted from the contradiction template)
+  - **New confidence:** re-assess based on observed data (usually downgrade or upgrade from original)
+
+When multiple contradictions exist in a single project, group them in the harvest candidate list as a subsection:
+
+```
+Ledger updates from contradictions:
+1. Update "<title>" — old claim: <X> → new claim: <Y> (confidence: <new>)
+2. Update "<title>" — old claim: <X> → new claim: <Y> (confidence: <new>)
+```
+
+Present these alongside new candidates in Step 3 so the user can confirm or reject them as a batch.
+
+**On confirm:** Overwrite the existing knowledge file — update the `date`, `confidence`, and content fields in place. Do **not** create a `_v2` file for contradiction resolutions. Update the INDEX.md row to reflect the new date and confidence.
+
+If no contradiction lines are found in `project-specs.md`, skip this step.
+
+### 1c. Check for validation findings worth harvesting
+
+Read the `## Validation` section of `project-specs.md` (per `shared/validation_protocol.md`). Validation that caught a real issue is often the highest-signal source of harvest candidates — the check surfaced something that would have quietly been wrong otherwise.
+
+Look for:
+
+- **Checks with `Pass/Fail: ✗` that were fixed.** The issue that was caught is a candidate entity/infrastructure/pattern entry ("billing.revenue column contains negative values on refunds"; "incremental predicate on `updated_at` misses rows when source clock drifts").
+- **Checks with surprising `Observed` values even when Pass/Fail = ✓.** Distribution shifts, null-rate anomalies, unexpected fan-out multipliers — the check passed but the value is worth remembering (e.g., "orders-to-items fan-out is 4.8x on this warehouse, not the industry-typical 2-3x — use for capacity planning").
+- **`n/a` with an instructive justification.** When a check is genuinely inapplicable for a non-obvious reason, that reason is a pattern (e.g., "DL-09 inference parity is n/a for this service because training and serving share the same feature module — note the pattern, not the skip").
+- **Downstream impact discoveries.** Consumers that were affected in non-obvious ways. These are often infrastructure or feature-level knowledge.
+
+These candidates blend in with the Step 1 / Step 2 draft list — tag them clearly so the user sees that they came from the validation section:
+
+```
+4. [patterns] "orders-items fan-out is 4.8x in this warehouse" (high confidence, from Validation AE-06)
+   — Verified across two quarters of data. Use for capacity planning and query tier assessment.
+```
+
+If the `## Validation` section is absent or contains nothing worth harvesting, skip this step.
+
+### 2. Draft candidates
+
+For each candidate, draft:
+
+- **Title:** short, specific, grep-friendly (e.g., "billing.user_id is string UUID not integer")
+- **Category:** entities | infrastructure | patterns | features
+- **Domain tags:** 2–4 keywords for INDEX.md matching
+- **Confidence:** high (verified in production data) | medium (observed in this project) | low (inferred, not directly tested)
+- **Content:** 3–10 lines explaining the knowledge. Be specific — include table names, column names, SQL snippets, system names. Vague entries are worthless.
+
+For **feature** candidates (Data Scientist / ML Engineer only), also draft:
+- **SQL snippet:** the feature computation
+- **Feature type:** numeric | categorical | boolean | temporal | embedding
+- **Grain:** one row per what
+- **Verified by:** which agent, in which project, with what metric impact
+
+**Minimum:** explicit assessment — 1+ candidates or "None — trivial project". For Quick tracks, default to "None" unless something genuinely surprising was discovered during the fix.
+
+### 3. Present to user (GATE)
+
+Present all candidates to the user in a numbered list:
+
+```
+Knowledge harvest — candidates for the Knowledge Ledger:
+
+1. [entities] "billing.user_id is string UUID not integer" (high confidence)
+   — billing.users.user_id is VARCHAR(36) UUID, not INT. Joins to events.user_id require CAST.
+
+2. [patterns] "incremental merge for slowly changing dims in BigQuery" (medium confidence)
+   — MERGE pattern that handles late-arriving updates without full refresh. SQL included.
+
+3. [features] "days_since_last_login" (high confidence, grain: user-day)
+   — Verified: +3.2% AUC in churn model. SQL: DATEDIFF(CURRENT_DATE, last_login_at).
+
+Add, edit, or remove? Or confirm to write.
+```
+
+::GATE:: id=specific-instructions-shared-knowledge-harvest-phase0 phase=0 kind=phase
+Do not write any files until the user confirms.
+::ENDGATE::
+
+The user may edit titles, remove candidates, adjust confidence, or add new ones.
+
+### 4. Create directory structure
+
+If `.shards/knowledge/` does not exist, create it with subdirectories:
+
+```
+.shards/knowledge/
+  INDEX.md          (copy from templates/knowledge-index.md if missing)
+  entities/
+  infrastructure/
+  patterns/
+  features/
+```
+
+### 5. Write knowledge files
+
+For each confirmed candidate, write a markdown file:
+
+**File path:** `.shards/knowledge/<category>/<slugified-title>.md`
+
+Slugify: lowercase, replace spaces with hyphens, remove special characters, max 60 characters.
+
+**Conflict handling:** if a file with that name already exists, append `_v2` (or `_v3`, etc.) to the filename and note the prior entry in the content.
+
+**File format:**
+
+```markdown
+---
+title: <title>
+domain: [<keyword1>, <keyword2>, <keyword3>]
+source_project: <project directory path>
+contributed_by: <agent name>
+date: <YYYY-MM-DD>
+type: <entities | infrastructure | patterns | features>
+confidence: <high | medium | low>
+---
+
+<content — 3-10 lines, specific, with table/column/system names>
+```
+
+**Extended frontmatter for features:**
+
+```markdown
+---
+title: <title>
+domain: [<keyword1>, <keyword2>, <keyword3>]
+source_project: <project directory path>
+contributed_by: <agent name>
+date: <YYYY-MM-DD>
+type: features
+confidence: <high | medium | low>
+sql_snippet: |
+  <the SQL that computes this feature>
+feature_type: <numeric | categorical | boolean | temporal | embedding>
+grain: <one row per what>
+verified_by: <agent name> in <source_project> — <metric>: <impact>
+---
+
+<content — what this feature captures, when to use it, caveats>
+```
+
+### 6. Update INDEX.md
+
+Append one row per entry to the table in `.shards/knowledge/INDEX.md`:
+
+```
+| <YYYY-MM-DD> | <type> | <title> | <domain keywords> | <confidence> | <category>/<filename>.md |
+```
+
+**Size warning:** if INDEX.md exceeds 200 entries after appending, warn the user:
+
+> "INDEX.md now has <N> entries. Consider pruning stale or low-confidence entries to keep retrieval fast."
+
+### 7. Report
+
+Tell the user what was written:
+
+```
+Knowledge harvested:
+- <title> → .shards/knowledge/<type>/<filename>.md
+- <title> → .shards/knowledge/<type>/<filename>.md
+
+INDEX.md updated (<N> total entries).
+```

package/src/agents/specific_instructions/shared/knowledge_retrieval.md

@@ -0,0 +1,100 @@
+---
+name: knowledge-retrieval-protocol
+description: Shared protocol for checking the Knowledge Ledger before Phase 1 — all specialists
+type: reference
+---
+
+# Knowledge Ledger — Retrieval Protocol
+
+This protocol runs once, between Phase 0 confirmation and Phase 1 start. It is non-blocking — if the ledger doesn't exist or is empty, document "N/A" and proceed.
+
+**Scope:** Build mode (Phase 0 → final), and Autonomous Research (`[AR]`) mode (Phase 0 setup). Do NOT run in Review, Advise, Service, Explain, Experiment (`[EX]`/`[EXP]`), or Prompt Lab modes.
+
+**AR entry point.** `[AR]` mode calls this protocol from `autonomous_research.md` Section A.1, before the research brief is drafted. Prior AR runs in the ledger may have already established saturation points, known-leaky features, or architectural dead ends that should shape hypotheses from iteration 1. The match criteria most useful for AR are:
+
+- **Metric** — has a prior AR or study measured the same primary metric on similar data?
+- **Domain / dataset / entity** — normal domain match.
+- **Approach family** — prior runs that explored the same model family (gradient boosting, transformer, prompt-chain, etc.) — these often document why a family is or is not a fit.
+
+Add these keywords to the extraction step alongside the normal domain keywords.
+
+**Syn handoff sessions:** Still run this protocol. The specialist appends the Knowledge Ledger subsection to the existing Phase 0 docs in `project-specs.md`.
+
+---
+
+## Steps
+
+### 1. Check for the ledger
+
+Check if `.shards/knowledge/INDEX.md` exists. If it does not exist or is empty (only the header row), document:
+
+```
+### Knowledge Ledger
+- **Entries checked:** N/A — ledger not found
+```
+
+Append this to the Phase 0 section in `project-specs.md` and proceed to Phase 1. Stop here — skip the remaining steps.
+
+### 2. Extract domain keywords
+
+From the Phase 0 answers already documented in `project-specs.md`, extract 3–5 domain keywords that describe the project's data domain, business area, and technical focus. Examples: "billing", "churn", "user_id", "Stripe", "incremental", "recommender".
+
+### 3. Scan INDEX.md
+
+Read `.shards/knowledge/INDEX.md` in full. Scan the table for rows whose **Title** or **Domains** columns contain any of your keywords (case-insensitive partial match). Collect up to 5 matching entries.
+
+### 4. Deep-read relevant entries
+
+For each matching entry (up to 5), read the file at `.shards/knowledge/<File column value>` (the File column contains paths relative to `.shards/knowledge/`). Assess relevance to the current project:
+
+- **Relevant:** the entry describes a data quirk, pattern, or infrastructure behavior that applies to the tables, systems, or domain this project will touch.
+- **Not relevant:** keyword match was coincidental (e.g., same word, different context).
+
+Discard entries that are not relevant after reading.
+
+For entries with a **Date** older than 6 months from today, flag as `(possibly stale)` in the output.
+
+### 4b. Validation-pattern retrieval
+
+In addition to the general keyword scan, match ledger entries against the checks your agent's validation checklist will apply in later phases. Entries that describe:
+
+- a **known distribution anomaly** relevant to a check like `AE-05` / `ML-02` / `DS-02`
+- a **grain or fan-out surprise** relevant to `AE-03` / `AE-06`
+- a **historical downstream break** relevant to the Downstream Impact analysis for this agent
+- a **prior validation failure** on a similar entity (check title or content references check IDs or validation findings)
+
+...should be surfaced specifically so you can run those checks knowing the history. Add a note in the Phase 0 documentation when such entries are found:
+
+```
+- <title> — relevant to validation check <ID>: <1-line relevance note>
+```
+
+If the ledger has nothing validation-specific, skip without writing an empty line.
+
+### 5. Feature Registry check (Data Scientist and ML Engineer only)
+
+If the current agent is the Data Scientist or ML Engineer, also check `.shards/knowledge/features/`:
+
+1. List files in `.shards/knowledge/features/` (skip if directory doesn't exist or is empty).
+2. For each feature file, read the YAML frontmatter and check if its `domain` tags overlap with the project's data domain keywords.
+3. Include relevant features in the output below, noting their `feature_type`, `grain`, and `verified_by` metadata.
+
+This is a preliminary scan for awareness only. A deeper Feature Registry import check happens in Phase 4.
+
+### 6. Document in project-specs.md
+
+Append the following subsection to the Phase 0 section in `project-specs.md`:
+
+```markdown
+### Knowledge Ledger
+- **Entries checked:** <N entries in INDEX.md>
+- **Relevant entries found:** <N>
+  - <title> (<type>, <confidence>) — <1-line relevance note>
+  - <title> (<type>, <confidence>, possibly stale) — <1-line relevance note>
+- **Or:** No relevant entries found
+- **Relevant features:** <N> (Data Scientist / ML Engineer only)
+  - <feature title> (<feature_type>, grain: <grain>, verified by: <agent> in <source_project>)
+  - Or: No relevant features found | N/A — not a DS/ML project
+```
+
+Read the subsection back to the user as part of Phase 0 confirmation (or as an addendum if Phase 0 was already confirmed). Then proceed to Phase 1.

package/src/agents/specific_instructions/shared/notebook_walkthrough_protocol.md

@@ -0,0 +1,367 @@
+# Notebook Walkthrough — Shared Protocol
+
+This document defines the **interactive notebook walkthrough** mode shared by
+the Data Scientist, ML Engineer, and Syn shards. The per-agent variant files
+under `specific_instructions/<agent>/notebook_walkthrough.md` set the persona
+spin and reference this document for the protocol.
+
+The walkthrough lets the agent guide the user through a Jupyter notebook
+**cell by cell**: execute the cell against a live kernel, explain it in chat,
+take questions, and (when asked) edit, insert, or delete cells and re-run
+downstream. It is **not** a Phase 0+ project workflow — there is no
+project-specs.md, no gate sequence, and no Syn final review. It is a live,
+read-and-react session.
+
+---
+
+## Architecture
+
+Three pieces:
+
+1. **The notebook on disk** (`.ipynb`) — the source of truth for cell source
+   and cell outputs. Mutated only via `NotebookEdit` (for source) or via the
+   kernel helper's `exec` subcommand (for outputs after execution).
+2. **The walkthrough state JSON** — the agent's view of the walkthrough
+   (current cell, which cells are stale, transcript of explanations and
+   user questions). Lives at:
+   `<project_root>/.shards/notebook-walkthrough.json`
+   The Shards UI watches this file via the panel's `--source` flag and
+   re-renders on every change.
+3. **The kernel helper** — `python .shards/ui/notebook-kernel.py` keeps a
+   persistent Jupyter kernel alive between agent invocations. Connection
+   files live under `.shards/notebooks/<session_id>/`.
+
+The agent owns the choreography between the three. The UI is read-mostly:
+it shows the current state and dispatches user actions back as chat messages.
+
+---
+
+## Bootstrap sequence (on activation)
+
+1. **Ask** the user for the notebook path. Validate it exists and ends in
+   `.ipynb`. (If invoked via `/notebook-walkthrough` slash command, the path
+   may already be in the user's first message.)
+
+2. **Generate a session id**:
+   ```bash
+   python .shards/ui/notebook-kernel.py new-session
+   ```
+   The helper prints `{"ok": true, "sessionId": "<uuid>"}`. Capture the
+   sessionId — every subsequent helper call needs it.
+
+3. **Start the kernel**:
+   ```bash
+   python .shards/ui/notebook-kernel.py start <session_id> <notebook_path>
+   ```
+   Helper writes the connection file, spawns ipykernel, and creates the
+   initial walkthrough state at `.shards/notebooks/<session_id>/state.json`.
+   On success the response includes `cellCount` and `stateFile`.
+
+4. **Read** the notebook from disk so you have the cell sources in
+   memory. Read both `<notebook_path>` and the helper's `state.json`.
+
+5. **Compose** the UI-facing walkthrough state JSON. Required shape:
+   ```json
+   {
+     "sessionId": "<uuid>",
+     "notebookPath": "<absolute or project-relative path>",
+     "currentCellIndex": 0,
+     "status": "ready",
+     "cells": [
+       {
+         "index": 0,
+         "type": "code|markdown",
+         "executed": false,
+         "stale": false,
+         "explained": false,
+         "lastRunAt": null,
+         "explanation": null,
+         "outputSummary": null
+       }
+     ],
+     "notebookCells": [
+       { "cell_type": "code|markdown", "source": "...", "outputs": [] }
+     ],
+     "transcript": []
+   }
+   ```
+   - `cells[].*` is per-cell walkthrough state (the helper's `state.json`
+     already gives you this — copy it forward).
+   - `notebookCells` is a flat snapshot of the on-disk `.ipynb` cells. The
+     UI uses this to render cell source and existing outputs without making
+     a separate file fetch.
+   - `transcript` accumulates `{ role, kind, cellIndex, text }` entries.
+
+   Write this JSON to `<project_root>/.shards/notebook-walkthrough.json`.
+
+6. **Push the panel** to the UI:
+   ```bash
+   node .shards/ui/ui-push.js notebook-walkthrough \
+     --title "Walkthrough: <notebook_basename>" \
+     --agent "<agent-name>" \
+     --panel-id "nw-<session_id>" \
+     --source "<project_root>/.shards/notebook-walkthrough.json"
+   ```
+   The `--source` flag tells the server to watch the file. Subsequent state
+   updates need only re-write the file — no further pushes required.
+
+7. **Explain cell 0** to the user in chat. Set `cells[0].explained = true`
+   and append an entry to `transcript`. Update the state JSON.
+
+8. **Wait** for the user to send a `[NOTEBOOK-WALKTHROUGH] ...` message or
+   to reply in chat.
+
+If the UI is not running, the `ui-push.js` call exits silently — that is
+fine, the walkthrough still works in chat-only mode (the user just won't
+see the panel; they can still read explanations and request actions in plain
+chat).
+
+---
+
+## Message protocol
+
+The UI dispatches user actions as chat messages prefixed with
+`[NOTEBOOK-WALKTHROUGH]`. Recognize and handle:
+
+| Pattern | Action |
+|---|---|
+| `Run cell <N>` | Execute cell N via the kernel helper, update state |
+| `Next cell` | Advance `currentCellIndex` by 1, explain the new cell |
+| `Re-explain cell <N>` | Generate a fresh explanation for cell N |
+| `Question on cell <N>: <text>` | Answer in chat; record in transcript; do not advance |
+| `Edit cell <N>: <new content>` | Mutate cell source via NotebookEdit, mark stale, re-run if user asked |
+| `Insert cell after <N> (<type>): <content>` | Insert a cell, shift indices, update state |
+| `Delete cell <N>` | Delete via NotebookEdit, shift indices, mark downstream stale |
+| `Re-run from cell <N>` | Execute cells N..end, clearing stale flags as you go |
+| `Restart kernel` | Call helper `restart`; mark all executed cells stale |
+| `Restart & run all` | Call helper `run-all`; fresh-kernel top-to-bottom reproducibility check; report per-cell pass/fail and the first error |
+| `End walkthrough` | Call helper `stop`; finalize transcript; offer summary |
+
+In `Insert` and `Edit` payloads, `\n` is the literal escape sequence — replace
+`\n` with newline before applying.
+
+User messages **without** the `[NOTEBOOK-WALKTHROUGH]` prefix are normal
+chat: treat them as questions or instructions, answer in chat, and record
+relevant ones in the transcript with `kind: "question"`.
+
+---
+
+## Cell execution
+
+Always go through the helper:
+
+```bash
+python .shards/ui/notebook-kernel.py exec <session_id> <cell_index>
+```
+
+The helper:
+- Reads cell source from the `.ipynb` on disk (so any prior `NotebookEdit`
+  is picked up automatically).
+- Sends the source to the live kernel.
+- Captures stream/result/error outputs.
+- Writes outputs back into the `.ipynb` file (so the file pane reflects the
+  fresh outputs).
+- Updates the helper's `state.json` with `executed`, `lastRunAt`, and
+  `outputSummary` for that cell.
+
+After every `exec` call:
+1. Re-read the helper's `state.json` (or just the relevant cell entry).
+2. Re-read the relevant `.ipynb` cell so `notebookCells` in the UI state JSON
+   reflects the new outputs.
+3. Update `<project_root>/.shards/notebook-walkthrough.json`. Set
+   `currentCellIndex` to the cell that just ran.
+4. Explain the result in chat — what the cell did, what to read in the
+   output, anything surprising.
+
+If `status` in the helper response is `"error"`, do not advance to the next
+cell. Explain the error in business/methodology terms and ask the user how
+to proceed.
+
+---
+
+## Cell mutation (edit / insert / delete)
+
+All cell mutations go through the **NotebookEdit** tool — never write the
+`.ipynb` directly.
+
+After any mutation:
+1. **Reload** the `.ipynb` from disk. Cell indices may have shifted.
+2. **Recompute the staleness pass**:
+   - Find the smallest index `K` that was mutated.
+   - Every cell at index `≥ K` that was previously executed gets
+     `stale: true` in walkthrough state.
+   - The mutated cell itself becomes `stale: true` and `executed: false`
+     (its old execution count is no longer meaningful).
+3. **Update `notebookCells`** in the walkthrough state JSON to mirror the
+   new cell list, and update `cells[]` so length matches.
+4. If the user asked for "Edit and re-run" (edit followed immediately by a
+   run), call `exec` for that cell now; otherwise leave it stale and tell
+   the user it is staged for re-run.
+5. Save the walkthrough state JSON.
+
+When inserting a cell, decide whether to insert with empty source (and let
+the user fill it later) or generate plausible code/markdown. **Default: ask
+the user.** Never invent analysis code without confirmation — this is the
+"facilitate, don't generate" rule still in force, even in walkthrough mode.
+
+---
+
+## Re-run from cell N
+
+When the user sends `Re-run from cell N`:
+
+1. Iterate `i` from `N` through the last executed-or-stale cell.
+2. For each `i`, call `exec <session_id> <i>`.
+3. After each successful run, clear `stale` and update `lastRunAt` /
+   `outputSummary`. Save the walkthrough state JSON between iterations so
+   the UI shows progress.
+4. If any cell errors, **stop** the loop, leave subsequent stale flags
+   intact, explain the error, and ask the user.
+
+---
+
+## Restart & run all
+
+When the user sends `Restart & run all` (the final reproducibility check —
+does the notebook run clean top-to-bottom on a *fresh* kernel?):
+
+1. Call the helper once:
+   ```bash
+   python .shards/ui/notebook-kernel.py run-all <session_id>
+   ```
+   It restarts the kernel, executes every code cell in order on the fresh
+   kernel, writes outputs back into the `.ipynb`, updates `state.json` per
+   cell, and stops at the first failing cell.
+2. Parse the roll-up: `cellsTotal`, `cellsRun`, `cellsPassed`, `perCell`
+   (`[{index, status}]`), and `firstError` (`{cellIndex, ename, evalue}` or
+   `null`).
+3. Re-read `state.json` and the `.ipynb`, then update
+   `<project_root>/.shards/notebook-walkthrough.json` to mirror the fresh
+   run — every passing cell `executed: true`, `stale: false`.
+4. Report the outcome in chat:
+   - **All passed** (`firstError == null`): say so plainly — the notebook is
+     reproducible from a cold start. This is the evidence DS-11 wants.
+   - **Failed**: name the failing cell, translate `ename`/`evalue` into
+     business/methodology terms, and **stop** (honoring "kernel errors halt
+     progress"). Every cell after `firstError.cellIndex` did not run.
+5. This is an *offer-and-run on request* action — never trigger it on your
+   own; the user clicks "Run All" or types the command.
+
+---
+
+## Explanations — how to write them
+
+Carried over from the existing Explain mode:
+
+- **Explain intent and logic, not syntax.** "We're imputing missing income
+  with the median because the column is right-skewed and the mean would be
+  dragged by outliers." Not: "We call `df['income'].median()` and pass it
+  to `fillna()`."
+- **Tie to the question.** Every cell either gathers data, transforms it,
+  models it, evaluates it, or visualizes it — name the role.
+- **Flag what to look at in the output.** "The KS statistic of 0.31 is what
+  matters here — anything above 0.2 is a meaningful distributional shift."
+- **Be honest about gaps.** If a cell's logic seems off, say so —
+  walkthrough mode is also implicit review.
+- Keep each explanation short — 2–4 sentences, occasionally a small list.
+  The user can ask for more.
+
+After explaining a cell, set `cells[N].explained = true` and append the
+explanation to `transcript`:
+
+```json
+{ "role": "agent", "kind": "explanation", "cellIndex": N, "text": "..." }
+```
+
+---
+
+## Question handling
+
+When the user asks a question (either with `[NOTEBOOK-WALKTHROUGH] Question
+on cell N: ...` or as a free-form chat message):
+
+1. Answer in chat. Reference the cell by index when relevant.
+2. Append to `transcript`:
+   ```json
+   { "role": "user", "kind": "question", "cellIndex": N, "text": "..." }
+   { "role": "agent", "kind": "answer", "cellIndex": N, "text": "..." }
+   ```
+3. Save the walkthrough state JSON.
+4. Do **not** advance `currentCellIndex`. The user is on this cell.
+
+If the question implies a code change ("could you also compute the median
+imputation?"), confirm the change in chat first, then apply it via the Edit
+flow above.
+
+---
+
+## End walkthrough
+
+When the user sends `End walkthrough` or otherwise signals completion:
+
+1. **Optional reproducibility check:** if cells were edited or run during the
+   walkthrough, offer to run `Restart & run all` once before ending — the
+   fresh-kernel top-to-bottom check that proves the notebook still works as a
+   whole. If the user accepts and an artifact is written (step 4), fold the
+   roll-up (`cellsPassed / cellsTotal`, first error if any) into the summary.
+   Offer only — never run it without confirmation.
+2. Call `python .shards/ui/notebook-kernel.py stop <session_id>`.
+3. Set walkthrough state `status = "ended"` and save.
+4. **Optional artifact:** if the notebook lives under `studies/` (Data
+   Scientist) or `models/` / `services/` / `research/` (ML Engineer / Syn),
+   offer to write a brief transcript summary to
+   `<project_dir>/walkthroughs/<session_id>.md` — list of cells walked
+   through, key explanations, user questions, and any edits applied.
+   Do not write this without explicit user confirmation.
+5. Close the panel:
+   ```bash
+   node .shards/ui/ui-push.js close --panel-id "nw-<session_id>"
+   ```
+
+---
+
+## Behavioral rules
+
+- **Never auto-run cells the user did not ask to run.** The user clicks Run
+  or types `Run cell N` / `Next cell`. Even when the kernel is fresh and
+  the notebook obviously needs all cells executed in order, ask before
+  running through.
+- **Never edit cells without confirmation**, even if the user is clearly
+  hinting. Confirm the change first.
+- **Never silently skip a stale cell.** If the user asks to advance past
+  a stale cell, point it out and ask whether to re-run it first.
+- **Save the walkthrough state JSON after every state-changing action.**
+  The UI is read-only — if you forget to write, the UI goes stale and the
+  user sees stale info.
+- **One action at a time.** Don't batch. The walkthrough is conversational.
+- **Kernel errors halt progress** — surface them and wait for the user.
+- **Keep outputs clean.** Every `exec` writes the cell's output back into the
+  `.ipynb` *and* into your context. Before running a cell that would print a
+  secret/token or dump a full DataFrame, suggest a `.head()` / `.shape` /
+  summary edit first — full dumps bloat the notebook and burn context on every
+  state read.
+- **Persona stays in character.** Use the per-agent variant's voice for
+  explanations and confirmations. The protocol mechanics here apply
+  unchanged across all three agents.
+
+---
+
+## Resume
+
+If `<project_root>/.shards/notebook-walkthrough.json` already exists when
+the agent activates:
+
+1. Offer to **resume** the existing session, with the same notebook and
+   `currentCellIndex`. State which cell you'd resume on and how many cells
+   are stale.
+2. If the user agrees, check the kernel:
+   ```bash
+   python .shards/ui/notebook-kernel.py status <session_id>
+   ```
+   - If `alive`: re-push the panel pointed at the existing state JSON,
+     re-explain the current cell, wait.
+   - If `dead` or `missing`: ask whether to **restart** (fresh kernel, all
+     executed cells become stale) or **start over** (new session id, drop
+     existing state).
+3. If the user prefers a fresh walkthrough, run the bootstrap sequence
+   from scratch — pick a new session id and overwrite the state JSON.