@hippo-digital/hippocampus 1.0.0-rc.1

package/docs/research-knowledge-graph.md

@@ -0,0 +1,552 @@
+# Research Knowledge Graph and Import Process
+
+This project treats design knowledge as linked source data. The frontend renders users, scenarios, journeys, blueprints, prototype screens, evidence impact and research artefacts from JSON in `hippocampus/source`.
+
+The goal is traceability:
+
+```text
+research round
+  -> participant
+  -> insight
+  -> design action
+  -> scenario / need / journey / screen / decision
+```
+
+The current implementation supports the graph model and a conservative research artefact import pipeline. Imported PowerPoint decks and text-based PDFs become reviewable drafts first. Reviewed drafts can then be promoted into the canonical knowledge source.
+
+Use `docs/repeatable-research-round-ingestion.md` as the canonical staged ingestion workflow. This document focuses on the graph model, importer behavior, draft shape, validation and promotion semantics.
+
+## Hippocampus Layout
+
+The `hippocampus/` folder contains the knowledge base and the material being absorbed into it:
+
+```text
+hippocampus/
+  source/              canonical structured knowledge
+  inbox/project/       new project overview artefacts to scan
+  inbox/research/      new research artefacts to scan
+  imports/project/     reviewable project context drafts
+  imports/research/    reviewable import drafts
+  manifests/           import and promotion ledgers
+  source-artefacts/    copied evidence files that support source knowledge
+```
+
+Generated outputs belong under `artefacts/`. The app should still validate and render if `artefacts/` is deleted.
+
+## Core Source Files
+
+The main source files live in `hippocampus/source`:
+
+- `users.json`
+- `project-context.json`
+- `needs.json`
+- `scenarios.json`
+- `journeys.json`
+- `screens.json`
+- `evidence.json`
+- `decisions.json`
+- `assumptions.json`
+- `relationships.json`
+- `research-rounds.json`
+- `participants.json`
+- `insights.json`
+- `design-actions.json`
+
+The schema lives in `node_modules/@hippo-digital/hippocampus/lib/schema.js`.
+
+The loader, index builder and cross-reference validation live in `node_modules/@hippo-digital/hippocampus/lib/load-knowledge.js`.
+
+## Relationship Graph
+
+`relationships.json` is the explicit graph layer. It records typed links between entities:
+
+```json
+{
+  "id": "rel-policy-scenario-journey",
+  "from": "scenario-policy-administrator-household-eligibility",
+  "type": "drives-journey",
+  "to": "policy-household-eligibility"
+}
+```
+
+Relationship IDs use lowercase kebab-case.
+
+Supported relationship types currently include:
+
+- `has-scenario`
+- `has-need`
+- `tests-need`
+- `supported-by`
+- `drives-journey`
+- `renders-screen`
+- `addresses-need`
+- `uses-evidence`
+- `records-decision`
+- `has-participant`
+- `produced-insight`
+- `contributed-to`
+- `supports-insight`
+- `responds-to-insight` (design action -> insight)
+- `resolves-action` (decision -> design action)
+- `leads-to-action` (legacy direction, no longer emitted)
+
+Validation checks that every `from` and `to` points to a known entity.
+
+## Frontend Views
+
+Key routes:
+
+- `/hippocampus`
+- `/hippocampus/users/:userId`
+- `/hippocampus/graph`
+- `/hippocampus/evidence/:evidenceId/impact`
+- `/hippocampus/research`
+- `/hippocampus/journeys/:journeyId`
+- `/hippocampus/journeys/:journeyId/blueprint`
+- `/hippocampus/journeys/:journeyId/blueprint/classic`
+- `/hippocampus/journeys/:journeyId/flow`
+- `/hippocampus/journeys/:journeyId/handover`
+
+The policy administrator example is useful for checking the graph:
+
+- `/hippocampus/users/policy-administrator`
+- `/hippocampus/journeys/policy-household-eligibility`
+- `/hippocampus/evidence/evidence-policy-administrators-scenario-map/impact`
+
+## Project Context Import Flow
+
+Project context is for generic overview material rather than research findings. Use it for service overviews, mapping boards, policy summaries, channel information, volumes, constraints, milestones and risks.
+
+The default project context workflow is:
+
+```bash
+npx hippocampus import project-context
+```
+
+This scans `hippocampus/inbox/project`, fingerprints supported files, imports only new files, updates `hippocampus/manifests/project.json`, and asks before promoting unpromoted drafts.
+
+Use flags when needed:
+
+```bash
+npx hippocampus import project-context -- --dry-run
+npx hippocampus import project-context -- --no-promote
+npx hippocampus import project-context -- --yes
+npx hippocampus import project-context -- --source path/to/folder
+```
+
+Single-artefact import is also available:
+
+```bash
+npm run import:project-context -- --document path/to/overview.pdf --context-id overview-id --title "Project overview"
+```
+
+Review generated drafts in:
+
+```text
+hippocampus/imports/project/
+```
+
+Promote after review:
+
+```bash
+npx hippocampus promote project-context -- --draft hippocampus/imports/project/<draft-id>.json
+```
+
+Promoted project context is written to:
+
+```text
+hippocampus/source/project-context.json
+```
+
+and rendered at:
+
+```text
+/hippocampus/project
+```
+
+## Research Artefact Import Flow
+
+The default research workflow is:
+
+```bash
+npx hippocampus research sync
+```
+
+This scans `hippocampus/inbox/research`, fingerprints supported files, imports only new files, updates `hippocampus/manifests/research.json`, and asks before promoting unpromoted drafts.
+
+Use flags when needed:
+
+```bash
+npx hippocampus research sync -- --dry-run
+npx hippocampus research sync -- --no-promote
+npx hippocampus research sync -- --yes
+npx hippocampus research sync -- --source path/to/folder
+```
+
+The lower-level single-artefact and batch commands are still available when you need explicit control.
+
+The single-artefact import flow has two stages:
+
+```text
+PowerPoint deck or text-based PDF
+  -> import draft JSON
+  -> human review
+  -> promotion into canonical source data
+  -> validation
+  -> frontend graph views
+```
+
+This is deliberate. An artefact may contain useful research, but extracted text often lacks stable structure. The importer should not silently mutate source data.
+
+## Script And AI Workflow
+
+The intended pattern remains:
+
+```text
+script extracts text
+  -> skill structures or reviews the draft
+  -> human checks the result
+  -> promotion script validates and merges
+```
+
+For the step-by-step team workflow, use `docs/repeatable-research-round-ingestion.md`.
+
+The active skill entry points are:
+
+- `.github/skills/ingest-research-round.skill.md`
+- `.github/skills/ingest-project-context.skill.md`
+- `.github/skills/structure-research-draft.skill.md`
+- `.github/skills/review-research-import-draft.skill.md`
+- `.github/skills/map-research-to-graph.skill.md`
+
+The skills should not promote drafts. Promotion remains a deliberate command so source changes stay reviewable and auditable.
+
+### 1. Import an Artefact
+
+Run:
+
+```bash
+npx hippocampus import research-deck -- --deck path/to/artefact.pptx-or.pdf --round-id round-id --title "Round title" --date YYYY-MM-DD
+```
+
+The importer:
+
+- reads slide XML from `.pptx` files
+- extracts page text from text-based `.pdf` files
+- does not OCR scanned PDFs yet
+- looks for prefixed lines:
+  - `Participant:`
+  - `User:`
+  - `Attendee:`
+  - `Insight:`
+  - `Finding:`
+  - `Observation:`
+  - `Action:`
+  - `Design action:`
+  - `Recommendation:`
+- builds a draft JSON file under `hippocampus/imports/research/`
+- writes a paired `.review.md` file for human review
+- validates the draft against `ResearchImportDraftSchema`
+- reports fit warnings when the artefact structure does not map well to the research graph
+
+Example warnings:
+
+- no participants detected
+- no insights detected
+- no design actions detected
+- low insight density for the number of slides
+
+The draft includes the extracted slide or page text so a human can inspect what the importer saw.
+
+Use the `.review.md` file for review first. Use the `.json` file when editing data for promotion.
+
+## Batch Import Flow
+
+Use batch import when you have many files to process at once.
+
+```bash
+npx hippocampus research batch -- --source path/to/files --batch-id batch-id
+```
+
+Optional arguments:
+
+```bash
+npx hippocampus research batch -- --source path/to/files --batch-id batch-id --date YYYY-MM-DD --concurrency 4
+```
+
+The batch importer:
+
+- recursively scans the source folder
+- processes supported files in parallel
+- currently supports `.pptx` and text-based `.pdf`
+- writes one draft per supported source file
+- writes a `manifest.json`
+- records unsupported files and failed files separately
+
+Batch output:
+
+```text
+hippocampus/imports/research/batches/<batch-id>/
+  drafts/
+    deck-a.json
+    deck-a.review.md
+    deck-b.json
+    deck-b.review.md
+  manifest.json
+```
+
+The manifest records:
+
+- source folder
+- import timestamp
+- concurrency level
+- files found
+- drafts created
+- unsupported files
+- failed files
+- draft and review paths
+
+Parallel import is safe because it only writes draft files. Promotion should stay deliberate and reviewed.
+
+### 2. Review the Draft
+
+Open the generated file in `hippocampus/imports/research/`.
+
+For batch imports, review files in:
+
+```text
+hippocampus/imports/research/batches/<batch-id>/drafts/
+```
+
+Before promotion, check:
+
+- Does the `researchRounds` record describe the round correctly?
+- Are participant labels and roles useful?
+- Are insights genuine research findings rather than slide headings?
+- Are design actions phrased as concrete design work?
+- Do relationships connect the right entities?
+- Are IDs stable and lowercase kebab-case?
+
+Edit the draft if needed before promotion.
+
+### 3. Dry-Run Promotion
+
+Run:
+
+```bash
+npx hippocampus promote research -- --draft hippocampus/imports/research/round-id.json --dry-run
+```
+
+Dry-run promotion:
+
+- validates the draft against `ResearchImportDraftSchema`
+- checks duplicate IDs against canonical source data
+- checks references inside the draft
+- checks relationship endpoints against known canonical and draft entities
+- does not write source files
+
+### 4. Promote the Draft
+
+Run:
+
+```bash
+npx hippocampus promote research -- --draft hippocampus/imports/research/round-id.json
+```
+
+Promotion appends records into:
+
+- `research-rounds.json`
+- `participants.json`
+- `insights.json`
+- `design-actions.json`
+- `relationships.json`
+
+After writing, it runs:
+
+```bash
+npx hippocampus validate
+```
+
+If validation fails, review the promoted data before continuing.
+
+## Validation
+
+Run:
+
+```bash
+npx hippocampus validate
+```
+
+This checks:
+
+- JSON shape through Zod schemas
+- ID formats
+- enum values
+- route formats
+- cross-file references
+- relationship endpoints
+
+Run:
+
+```bash
+npx hippocampus doctor
+```
+
+This starts the app on a smoke-test port and checks generated routes, user routes, evidence impact routes and prototype screen routes return 200.
+
+## Current Limitations
+
+- The importer uses visible PowerPoint slide text or text extracted from PDF pages only.
+- Speaker notes are not imported yet.
+- Slide layout is not interpreted.
+- Decks without clear `Participant:`, `Insight:` or `Action:` prefixes will produce weak drafts.
+- Promotion appends records; it does not update existing records.
+- Research detail pages beyond the index are not built yet.
+
+## Recommended Deck Format
+
+For best import results, use explicit prefixes in the artefact text:
+
+```text
+Participant: P1 - Policy administrator
+Insight: Users need caveats when household relationships are inferred from address history
+Action: Add caveat capture to the eligibility evidence screen
+```
+
+Keep one insight or action per line where possible.
+
+## Safe Editing Rules
+
+- Import first, review second, promote third.
+- Do not promote unrelated decks just because they parse.
+- Prefer fit warnings over false confidence.
+- Keep source data fictional unless explicitly approved.
+- Run `npx hippocampus validate` and `npx hippocampus doctor` before committing.
+
+## Testing the Import Flow
+
+Use the safest route for testing:
+
+```text
+import -> inspect draft -> dry-run promotion -> promote only if the draft is right
+```
+
+### 1. Create a Test Source Folder
+
+```bash
+mkdir -p hippocampus/imports/research/batches/test-batch/source
+```
+
+Copy one or more `.pptx` or text-based `.pdf` files into:
+
+```text
+hippocampus/imports/research/batches/test-batch/source/
+```
+
+For the first test, use a small artefact with clear text markers:
+
+```text
+Participant: P1 - Policy administrator
+Insight: Users need caveats when household relationships are inferred
+Action: Add caveat capture to the evidence screen
+```
+
+### 2. Run Batch Import
+
+```bash
+npx hippocampus research batch -- --source hippocampus/imports/research/batches/test-batch/source --batch-id test-batch --date 2026-07-07 --concurrency 2
+```
+
+Expected output:
+
+```text
+hippocampus/imports/research/batches/test-batch/
+  drafts/
+  manifest.json
+```
+
+### 3. Inspect the Manifest
+
+```bash
+cat hippocampus/imports/research/batches/test-batch/manifest.json
+```
+
+Check:
+
+- `filesFound`
+- `draftsCreated`
+- `unsupported`
+- `failed`
+- `drafts`
+
+If `failed` is not empty, inspect the error before continuing.
+
+### 4. Inspect the Draft
+
+```bash
+ls hippocampus/imports/research/batches/test-batch/drafts
+```
+
+Open the generated `.review.md` file first. It is the human-readable view of the import and includes:
+
+- fit warnings
+- counts
+- participants
+- insights
+- design actions
+- extracted slide or page text
+
+Then open the `.json` draft if you need to edit the data before promotion. Review:
+
+- `warnings`
+- `researchRounds`
+- `participants`
+- `insights`
+- `designActions`
+- `relationships`
+- `slides`
+
+Warnings are not automatic failures. They mean the extracted text did not map cleanly to the research graph. Fix the draft or improve the source markers before promotion.
+
+### 5. Dry-Run Promotion
+
+```bash
+npx hippocampus promote research -- --draft hippocampus/imports/research/batches/test-batch/drafts/YOUR-DRAFT.json --dry-run
+```
+
+Dry-run promotion should pass before real promotion. It checks:
+
+- draft schema
+- duplicate IDs
+- participant-to-round links
+- insight-to-round and insight-to-participant links
+- action-to-insight links
+- relationship endpoints
+
+### 6. Promote Only After Review
+
+```bash
+npx hippocampus promote research -- --draft hippocampus/imports/research/batches/test-batch/drafts/YOUR-DRAFT.json
+```
+
+Then run:
+
+```bash
+npx hippocampus validate
+npx hippocampus doctor
+```
+
+View the result:
+
+- `/hippocampus/research`
+- `/hippocampus/graph`
+
+### 7. Clean Up Throwaway Tests
+
+If the batch was only a smoke test and should not remain in the repo, remove it with:
+
+```bash
+trash hippocampus/imports/research/batches/test-batch
+```
+
+Use `trash` rather than `rm` so the generated test files remain recoverable.

package/docs/research-source-file-conventions.md

@@ -0,0 +1,121 @@
+# Research source file conventions
+
+How to write and store research artefacts so they ingest cleanly into Hippocampus.
+
+This is written for researchers and designers, not for the scripts. Everything here
+is a change to how a file is *authored or named*, not a change to the research
+itself. Each convention exists because the ingestion pipeline currently loses
+something real without it.
+
+The reference corpus behind this guidance is the NCRS user research drive:
+599 files, 2.3 million extractable words, rounds 4 to 16.
+
+## The one thing that matters most
+
+**Write findings under a heading that says what they are.**
+
+The importer scopes detection to section headings. A slide or Word heading called
+`Insights`, `Findings`, `What we learned`, `Themes`, `Pain points`,
+`Design actions`, `Recommendations`, `Participants` or `Assumptions` turns every
+substantive line beneath it into a candidate record.
+
+A heading called `Global Insights` works. `Slide 12` does not. A deck of genuine
+findings under generic headings imports as nothing at all.
+
+## Conventions by artefact type
+
+### All artefacts
+
+| Do | Why |
+| --- | --- |
+| Keep the `01 Planning / 02 Doing / 03 Analysing / 04 Sharing / 05 Design Actions` folder pattern | It is read as research-stage provenance. It already works well across rounds 4 to 16 — keep it. |
+| Put the round number in the folder name (`Round 12 - ...`) | The round is derived from the path, so every artefact inherits the right `roundId`. |
+| Put the fieldwork date in the file name as `YYYY-MM` or `YYYY-MM-DD` | Copying files off SharePoint resets every modification date. On this corpus, all 599 files carried the copy date, so file dates cannot be trusted as evidence dates. The name is the only surviving provenance. |
+| Keep one canonical version, not `- Copy`, `(1)`, `OLD`, `Archive` siblings | Near-identical duplicates each generate their own draft and their own IDs. |
+| Use participant codes (`P27`, `P104`) rather than names | Codes are safe to store, and they are what links a quote to a participant record. |
+
+### Slide decks (.pptx)
+
+- Section headings drive detection — see above.
+- **Speaker notes are now read.** They were previously discarded, which lost about
+  10% of the text in this corpus and over 3,500 words in each of the round 16
+  playbacks. Notes are captured as context, but a finding stated only in the notes
+  will not seed an insight on its own — put the finding on the slide.
+- Attribute quotes as `P104 - "the quote"`. An unattributed quote cannot be traced
+  to a participant and the importer will warn about it.
+- One finding per bullet. A bullet that wraps into the next one imports as two
+  fragments.
+
+### Documents (.docx)
+
+- **Use real Word heading styles**, not bold body text. This is the single biggest
+  quality gap in the corpus: the round 16 `P## Detailed Scenario Map` documents are
+  beautifully structured with numbered sections, but every paragraph is styled
+  `Normal`, so the whole document imports as one undivided block. The persona
+  documents use `Heading 2` correctly and import as clean, separately addressable
+  sections.
+- Keep the numbered section pattern used by the scenario maps. It maps directly onto
+  `scenario.locationSearch` in the schema.
+- Transcripts should keep the `P##:` speaker prefix on every turn.
+
+### Spreadsheets (.xlsx, .csv)
+
+- **Put a header row in row 1 of every sheet.** Column headers are read as field
+  names, so `Insight / Feedback Summary` becomes the insight text and
+  `Design Action` becomes the action.
+- The `Insight and Design Action Tracker` format already in use is close to ideal.
+  Its columns (`ID`, `Date Received`, `User Groups Affected`, `Feedback Source`,
+  `Need`, `Theme`, `Insight / Feedback Summary`, `Impact Level`, `Signal Strength`,
+  `Evidence Link 1-3`, `Design Action`, `Status`) map almost one-for-one onto
+  `insights`, `designActions`, `evidence` and `relationships`. Use it for every round.
+- Keep date columns as real dates. Excel serials in date-named columns are converted
+  back to `YYYY-MM-DD`; serials elsewhere are left as opaque numbers.
+- Do not merge cells across a header row.
+- Save legacy `.xls` files as real `.xlsx`. Two files in this corpus carry an `.xlsx`
+  extension over a non-zip payload and cannot be opened by any tool in the pipeline.
+
+### Links (.url)
+
+- These are kept, not skipped. A MURAL board or Confluence page shortcut becomes an
+  evidence record pointing at where the analysis lives.
+- Name the shortcut after what it points at (`Round 11 MURAL analysis.url`), because
+  the file name becomes the evidence title.
+
+### Media (.mp4, .jpg, .png)
+
+- Video and images carry no extractable text and are registered as evidence
+  artefacts only. On this corpus that is 3.4 GB of the 3.7 GB total.
+- **Name clips as `P## - what the clip shows.mp4`.** The round 15 accessibility clips
+  do this well: `P94 - Not knowing how to navigate with a keyboard.mp4` is a usable
+  finding and a participant link from the file name alone.
+- If a PDF is a scan or an exported board image, say so in the file name. It will
+  extract as near-zero words and would otherwise look like a failed import.
+
+## What good looks like
+
+A round folder that ingests cleanly:
+
+```text
+Round 17 - <topic>/
+  01 Planning research/
+    Round 17 Research Plan 2026-10.docx        <- Heading styles
+  02 Doing the research/
+    P## transcripts, discussion guide
+  03 Analysing the research/
+    P## Detailed Scenario Map.docx             <- Heading styles, numbered sections
+    Scenario_Evidence_Map.xlsx                 <- header row, one scenario per row
+  04 Sharing the research/
+    Round 17 Show and Tell 2026-11.pptx        <- "Insights - <group>" headings,
+                                                  P## attributed quotes, notes used
+  05 Design Actions/
+    R17 research insights.xlsx                 <- the tracker template
+```
+
+## Checking your own files before handing them over
+
+```bash
+npx hippocampus research triage -- --source "/path/to/Round 17" --topic round-17
+```
+
+Read the report in `artefacts/research-triage/`. If your finished playback deck lands
+in Tier 3, the content is fine but the structure is hiding it.