create-egregore 0.20.2 → 0.21.0-runtime-mvp.2

package/runtime/claude/.claude/skills/ingest/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: ingest
+description: "Route and run organizational ingestion through Egregore Runtime for files, folders, meetings, Google Workspace, Notion, interviews, or bulk corpora. Use for 'ingest', 'bring this into Egregore', 'index this folder', or 'import our docs'."
+---
+
+# Ingest
+
+Route external material into an authorized intake plane, then promote selected
+evidence into curated organizational memory. Do not collapse intake and
+promotion.
+
+## When to invoke
+
+Route and run organizational ingestion through Egregore Runtime for files, folders, meetings, Google Workspace, Notion, interviews, or bulk corpora. Use for 'ingest', 'bring this into Egregore', 'index this folder', or 'import our docs'.
+
+## Route
+
+- One URL/document/reference → use the `add` skill and `bin/ingest-item.sh`.
+- `meeting ...` → use the meeting skill.
+- `user-interview ...` → use the ingest-user-interview skill.
+- `google ...` → use the ingest-google skill.
+- `notion ...` → use the ingest-notion skill.
+- A large question-answering corpus or “build a knowledge base” → use the
+  ingest-corpus skill and keep its Connected capability gate.
+- No readable path → `bash bin/ingest.sh select`.
+- File/folder/export/bulk path → use the corpus pipeline below.
+- Ambiguous → ask which source class applies; do not guess a connector.
+
+Preserve connector authorization/revocation in connector-specific Runtime
+adapters. This skill must not write connector lifecycle files or project state.
+
+## Invariants
+
+- Resolve organization only from the active Egregore instance, never content or
+  arguments.
+- Keep selected/unverified bytes in gitignored local staging/quarantine.
+- Canonical shared state contains admitted Markdown or reference manifests with
+  stable ids, revisions, provenance, boundaries, errors, and tombstones.
+- Register canonical state successfully before any derived retrieval refresh.
+- Hard boundaries are authorization filters, never descriptive tags. Missing
+  required boundary evidence fails closed.
+- Graph projection is optional and cannot complete, block, or override intake.
+- Telemetry contains structured counts/status only, never source content.
+
+## Local selection
+
+When no path was provided:
+
+```bash
+bash bin/ingest.sh select
+```
+
+If cancelled, stop. If it returns a Google or Notion connector, continue in
+that connector skill. Otherwise always submit the reviewed selection:
+
+```bash
+bash bin/ingest.sh add-selection "$SELECTION_PATH"
+```
+
+The selection adapter verifies staging containment, relative paths, byte
+counts, and hashes before extraction. Register harness-native extraction for a
+selected PDF only when the reader actually produced normalized text:
+
+```bash
+bash bin/ingest.sh register-extraction "$SELECTION_PATH" \
+  "relative/file.pdf" --extractor "$EXTRACTOR" < normalized.txt
+```
+
+Report unreadable/scanned pages; never imply complete extraction.
+
+## Corpus intake
+
+Infer only safe metadata. Ask for a stable source id, sensitivity/visibility,
+hard boundaries, and whether the directory is an authoritative snapshot.
+
+```bash
+bash bin/ingest.sh add "$PATH" \
+  --source "$SOURCE_ID" \
+  --name "$NAME" \
+  --kind "$KIND" \
+  --boundary "$KEY=$VALUE"
+```
+
+Repeat boundaries. Use `--prune` only for an explicitly authoritative full
+snapshot. The adapter owns deterministic extraction, stable document/chunk
+identity, local quarantine, canonical manifest registration, retrieval refresh,
+and background embedding. Skills do not invoke those mechanics separately.
+
+## Verify
+
+```bash
+bash bin/ingest.sh status
+bash bin/ingest.sh search "a known phrase" --source "$SOURCE_ID" -n 5
+```
+
+Include every required hard boundary in verification searches. Open exact
+evidence before making consequential claims. Report indexed/skipped/failed
+counts and gaps honestly.
+
+## Promote
+
+Intake is not curated knowledge. Retrieve candidate evidence, open its exact
+source, then use the appropriate Runtime ritual for a decision, finding,
+research synthesis, meeting, or domain claim. Preserve source id, document id,
+revision/content hash, exact chunk ids, boundaries, reviewer, confidence, and
+conflicts. No evidence chunk means no promoted claim.
+
+## Completion
+
+Report source id, documents/chunks admitted, extraction failures, boundaries,
+canonical manifest/writeback status, semantic-build state, and next review or
+promotion step. Do not run a second save, index, telemetry, or projection step.

package/runtime/claude/.claude/skills/ingest-corpus/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: ingest-corpus
+description: "Turn a large folder of documents into a Connect-only knowledge base that answers questions with sources. Say 'build a knowledge base'. Not a few files (/ingest) or a meeting/Notion/Google source."
+---
+
+Turn a folder of documents into a knowledge base that answers questions with its sources.
+
+## When to invoke
+
+`/ingest` routes here when the user brings **a body of documents to be asked questions of** —
+a research archive, a contract set, a manual library, a regulatory corpus. Signals: hundreds
+or thousands of files, subject folders, "make this searchable", "build a knowledge base",
+"I want to ask these questions".
+
+Not this: a handful of files to keep for reference → the plain `/ingest` path · a meeting
+recording → `ingest-meeting` · a Notion or Google source → those connectors.
+
+## Step 0 — Connected mode only
+
+```bash
+MODE=$(jq -r '.mode // "connected"' egregore.json 2>/dev/null)
+```
+
+If `mode` is `local`, stop and tell the user:
+
+> Building a knowledge base needs Egregore Connect. The statements it produces
+> are shared through the graph, so your team asks one archive rather than each
+> keeping a private copy. This configuration has no graph.
+>
+> Plain `/ingest` still works — it stores your documents and makes them
+> searchable on this machine.
+
+Then stop. Do not run the survey, and do not offer a way to turn Connect on.
+
+## What makes this different from plain ingest
+
+Plain ingest stores documents and makes them searchable. This additionally works out **which
+document may answer which question**, extracts the sentences that carry advice, and checks
+each one against the sentence it came from.
+
+That needs a profile: which folder means which region or client, which source outranks which,
+and what a sentence carrying advice looks like in the language the documents are written in.
+Nobody can write that file cold. So it is generated from a survey plus two answers.
+
+## Step 1 — Look before asking
+
+```bash
+python3 bin/corpus_survey.py  # or: python3 -c "import sys;sys.path.append('bin');import corpus_survey as cs;print(cs.summarise(cs.survey('<path>')))"
+```
+
+Show the user what was found, verbatim — document count, folders with counts, file types,
+detected language, and every note. **Do not summarise away the notes.** They carry the things
+that will otherwise be discovered from an empty result: a language with no grammar, files that
+cannot be read, a spreadsheet sitting beside the documents.
+
+## Step 2 — Ask what must be kept apart
+
+Only if the survey reports more than one group. One group means there is nothing to separate,
+so ask nothing.
+
+Use `AskUserQuestion` with the question `corpus_survey.boundary_question()` returns. It already
+carries the user's own folder names and states the consequence. Keep both options in the order
+given: **keeping them together is first**, because most folder structures are subjects, and
+separating subjects removes real answers.
+
+If the user separates them, ask what the groups are — clients, regions, products, versions —
+and use that word as the boundary key.
+
+## Step 3 — Ask which source wins
+
+Use `corpus_survey.trust_question()`. The user ranks the groups, or picks "Don't know".
+
+"Don't know" is a real answer and must stay on offer. It leaves every source equal, so a
+disagreement is shown rather than resolved — which is honest, and better than a ranking nobody
+chose.
+
+## Step 4 — Write the profile
+
+```python
+import corpus_profile as cp
+profile = cp.build(survey, separate_groups=<bool>, trust_order=[...], boundary_key="<word>")
+cp.write(profile, "memory/knowledge/tools/<name>-ontology/publisher-profile.json")
+```
+
+Show `cp.summarise(profile)` to the user. Read out every line under `open` — those are the
+things this profile will **not** do. A language with no grammar means those documents yield no
+advice at all, and that must be said now rather than found later.
+
+## Step 5 — Ingest
+
+```bash
+python3 bin/ingest.py add <path> --source <id> --boundary <key>=<value>
+```
+
+The profile is found automatically once it is under `memory/knowledge/tools/*/`. If ingest
+warns that no profile was found, stop — every document will carry no region, and every later
+question will be refused.
+
+Expect roughly two hours per thousand documents. Run it in the background.
+
+## Correcting the profile after ingest
+
+A profile is usually wrong the first time in a way nothing reveals until questions
+are asked of it — a publisher classed regional that is national, a tier too low. Fix
+the profile, then apply it:
+
+```bash
+python3 bin/ingest.py reresolve --source <id> --dry-run   # what would change
+python3 bin/ingest.py reresolve --source <id>             # apply it
+```
+
+**Do not re-run `add` to apply a profile change.** It skips on the content hash and
+returns before it consults the profile, so it reports every document unchanged and
+applies nothing — which reads as the correction not working.
+
+`reresolve` re-runs the resolver against each document's stored path. No file is
+reopened and no text is re-extracted, so a corpus that took hours to ingest is
+corrected in seconds. A boundary the operator stated with `--boundary`, or one the
+catalogue recorded, is reported as `protected` and never overwritten.
+
+Report `placed` (documents that had no zone and now have one) first. Those were
+invisible: a zoneless passage is dropped before ranking, so they could not be
+retrieved at all, however well they matched.
+
+## Step 6 — Check it before handing it over
+
+```bash
+python3 bin/retrieval_eval.py --fixture <questions.json> --manifest <meta>/sources/<id>/manifest.json \
+  --data-root <root> --vocabulary <ontology>/*.json
+```
+
+Report **boundary violations** first — the target is zero and anything else is a failure
+regardless of how good the answers look.
+
+Report the citation hit rate as **unavailable** unless someone who knows the subject has
+confirmed which documents should answer each question. A hit rate computed from unconfirmed
+expectations measures agreement with whoever wrote the fixture, not with the domain.
+
+## Step 7 — Build the map, then spend only on demand
+
+Intake made the documents retrievable. Do **not** run statement extraction over
+the corpus wholesale — most passages are never retrieved, and extracting them
+buys nothing. Build the map instead:
+
+```bash
+python3 bin/corpus_map.py build --source <id> --map-dir memory/corpora/<name>
+```
+
+One card per document — publisher, tier, zone, genre, the author's own abstract
+verbatim — at **zero model cost**, in the memory repo where every agent can read
+it. `index.md` is the whole corpus in one read; `map.jsonl` is the same thing
+for grep and jq. Never re-ingest to find a document; read the map.
+
+From then on the corpus runs a demand loop, and every step writes `ledger.md`:
+
+```bash
+python3 bin/corpus_map.py retrieve --source <id> --question "…" --zone <zone>   # BM25, zone-bounded
+python3 bin/corpus_map.py touch <doc> --source <id> --question "…"              # usage = curation signal
+python3 bin/corpus_map.py promote <doc> --source <id> --reason "<the question>" # THIS document only
+python3 bin/corpus_map.py validate <doc> --source <id>                          # claims checked against source
+python3 bin/corpus_map.py answer --source <id> --question "…" --zone <zone>     # invariants enforced
+```
+
+Promotion is judgment applied to what a question needs, never to the corpus at
+large. The deterministic path records the source's own sentences verbatim and
+cannot misquote; a refined claim (slots file) is validated against the sentence
+it came from. A refused answer is recorded in `gaps.md` — that file is the
+curator's acquisition queue, not an apology. The ledger records the exact local
+scan performed and its trigger. `corpus_map.py` makes no model call; any later
+model or expert refinement must record its own actual cost rather than treating
+promoted text as model input.
+
+## What to tell the user at the end
+
+- how many documents went in, and how many were skipped and why
+- that answers carry their source and trust level, and that questions outside the boundary are refused
+- what the profile will not do, from its `open` list
+- that nobody has checked the answers yet, if nobody has
+
+## Rules
+
+- **Never write the profile without showing its `open` list.** Those are the silent failures.
+- **Never present an unconfirmed accuracy figure.** Withhold it and say why.
+- **A survey note is not noise.** Files that could not be read, an undetected language, and a
+  found spreadsheet all change what the knowledge base can do.
+- The user answers questions about their own folders, by name. Never ask them to describe their
+  corpus in the abstract.

package/runtime/claude/.claude/skills/ingest-notion/SKILL.md

@@ -0,0 +1,214 @@
+---
+name: ingest-notion
+description: "Bring selected Notion pages into shared Egregore memory via Notion's official MCP. Routed from /ingest when the source is Notion — say '/ingest notion'. Selected pages only, not a full-workspace sync."
+---
+
+Bring selected Notion pages into shared Egregore memory through Notion's official MCP.
+
+Arguments: $ARGUMENTS (Optional: a page URL/ID, a topic or title to search, or `--auto`)
+
+## When to invoke
+
+Routed from `/ingest` when the source is Notion. Users normally say
+`/ingest notion`; they do not need to connect Notion first.
+
+## What this flow does
+
+Notion MCP is the live bridge. This skill adds the memory layer:
+
+1. connect to Notion when needed;
+2. search or fetch what the user names;
+3. show exactly what will be imported;
+4. write only the approved pages into Egregore memory with provenance.
+
+This is a selected-page flow. Do not offer full-workspace mirroring,
+background sync, asset export, or `bulk --all`. Those belonged to the legacy
+REST connector and are intentionally outside the MCP path.
+
+## Step 0 — Ensure Notion MCP is available
+
+Discover tools using the current runtime's registry:
+
+- Claude Code: ToolSearch for `notion`.
+- Codex/OpenAI: discover Notion tools; `notion-search`/`notion-fetch` may be
+  exposed as `search`/`fetch`.
+- Other MCP clients: inspect tools from the server named `notion`.
+
+Require both of these capabilities:
+
+- search: `notion-search` or the Notion server's `search`
+- fetch: `notion-fetch` or the Notion server's `fetch`
+
+Do not match an unrelated web-search or generic fetch tool. Confirm the tool's
+server/provider is Notion.
+
+If either capability is missing, follow `.claude/skills/notion-connect/SKILL.md`
+inside this flow. If the runtime loads the tools immediately, continue. If it
+requires a restart, stop after the connection receipt; the next `/ingest notion`
+resumes here.
+
+There is no Connected-tier gate. Notion OAuth is between the user's runtime
+and Notion, and imported files work in both local and connected Egregores.
+
+## Step 1 — Choose the scope
+
+- A Notion URL or page ID → fetch that page directly.
+- A topic/title → search Notion for it.
+- No useful argument → ask one question:
+
+  - **Search by topic** — they name what they want to find.
+  - **Paste a page link** — they provide a Notion URL.
+
+Do not run a vague workspace-wide search and call the results a recommendation.
+The user supplies the topic or page.
+
+If they ask to import everything, say:
+
+> Notion MCP is built for finding and fetching selected pages, not mirroring a whole workspace. Name a topic or paste the first page you want in memory.
+
+Then wait.
+
+## Step 2 — Find candidate pages
+
+For a search request, call the Notion MCP search tool with the user's words.
+Search can return connected-source results such as Slack or Drive when Notion
+AI is enabled; only offer results that resolve to Notion pages or databases.
+
+Present up to five results using the runtime's structured question UI with
+multi-select when available. Each option contains:
+
+- page title;
+- short location or last-edited context when returned;
+- no generated summary before the page has been fetched.
+
+Say:
+
+> Which pages should join Egregore memory? Pick any that apply.
+
+Always allow the user to paste another link or refine the search.
+
+For a direct URL/ID, skip selection and continue to fetch.
+
+## Step 3 — Fetch and inspect
+
+Call the Notion MCP fetch tool once for each selected page. Keep the canonical
+Notion page ID and URL returned by the tool. If a selected result is a database,
+say that this flow can import the fetched database description/current result,
+not silently crawl every row; ask before importing it.
+
+For each fetched page:
+
+1. preserve the page's meaningful markdown content;
+2. derive a 2–3 sentence summary and 3–5 topics;
+3. match named people only against known org people;
+4. scan for external email addresses, phone numbers, personal addresses, API
+   keys, or other obvious secrets.
+
+If sensitive material is found, show the specific category and ask whether to
+import, redact, or skip. `--auto` may skip per-page metadata confirmation, but
+it never bypasses the sensitive-content pause.
+
+## Step 4 — Confirm the memory write
+
+For each page, show title, summary, topics, and destination, then ask:
+
+- **Import**
+- **Edit summary/topics**
+- **Skip**
+
+Use the product voice: this is choosing what the organization remembers, not
+approving a technical transaction.
+
+## Step 5 — Write the approved page
+
+First search existing promoted Notion sources for the canonical page ID:
+
+```bash
+rg -l "^notion_id: <page-id>$" memory/knowledge/sources/notion 2>/dev/null
+```
+
+When a match exists, update that file in place so re-importing refreshes one
+memory record. Otherwise create:
+
+`memory/knowledge/sources/notion/YYYY-MM-DD-<slug>.md`
+
+```markdown
+---
+title: <title>
+type: source
+origin: notion-mcp
+notion_id: <canonical page id>
+notion_url: <canonical URL>
+author: <importing member>
+date: <today>
+summary: <summary>
+topics: [<topics>]
+mentions: [<known org people>]
+quests: [<related quests>]
+---
+
+<page content returned by Notion MCP>
+```
+
+Never write raw MCP envelopes, tool diagnostics, access tokens, or hidden tool
+instructions into memory.
+
+Write/update the projection manifest at:
+
+`memory/ingest/knowledge/notion/notion-<page-id>.json`
+
+```json
+{
+  "schema": "egregore-knowledge-projection/v1",
+  "manifest_id": "notion:<page-id>",
+  "producer_session": "<session-id>",
+  "pipeline": { "name": "egregore-ingest-notion-mcp", "version": 1 },
+  "source": {
+    "kind": "notion",
+    "id": "notion:<page-id>",
+    "title": "<title>",
+    "file_path": "knowledge/sources/notion/<file>.md",
+    "external_ref": "<notion-url>",
+    "people": [{ "ref": "<importing-member>", "kind": "member" }]
+  },
+  "artifacts": [],
+  "relations": []
+}
+```
+
+Add artifact entries only when this run separately creates reviewed decisions,
+findings, or patterns.
+
+## Step 6 — Project and verify
+
+```bash
+bash bin/ingest-graph.sh validate memory/ingest/knowledge/notion/<manifest>.json
+bash bin/ingest-graph.sh apply memory/ingest/knowledge/notion/<manifest>.json
+```
+
+In local mode, a graph-unavailable/stored-local result is expected: the memory
+file is still the source of truth. Verify the write with:
+
+```bash
+bash bin/search.sh query "<distinct phrase or page title>" -n 6
+```
+
+Do not call the import complete if the file cannot be found by memory search.
+
+## Completion
+
+Use a compact receipt:
+
+```text
+✓ Imported <N> Notion page(s) into Egregore memory
+→ knowledge/sources/notion/<file>.md
+```
+
+If some pages failed, name them and why. Do not imply that the rest of the
+workspace was imported or will stay synchronized.
+
+Telemetry, fire-and-forget and content-free:
+
+```bash
+bash bin/telemetry.sh emit "command" '{"command":"ingest","kind":"notion-mcp"}' 2>/dev/null &
+```

package/runtime/claude/.claude/skills/ingest-user-interview/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: ingest-user-interview
+description: "Analyze a user interview from Granola, pasted text, or a file into an evidence-backed briefing, journey insights, product findings, patterns, and actions. Use for /ingest user-interview, onboarding interviews, research calls, or requests to process user feedback."
+---
+
+# Ingest user interview
+
+Acquire the transcript, understand the research intent, analyze it with
+appropriate lenses, preview the result, then persist one approved typed package
+through Egregore Runtime.
+
+Read [references/analysis-contract.md](references/analysis-contract.md) before
+analysis or package construction.
+
+## When to invoke
+
+Analyze a user interview from Granola, pasted text, or a file into an evidence-backed briefing, journey insights, product findings, patterns, and actions. Use for /ingest user-interview, onboarding interviews, research calls, or requests to process user feedback.
+
+## Source
+
+Accept one of:
+
+- Granola connector: search/select, then fetch transcript and stable metadata;
+- pasted transcript;
+- a user-authorized `.txt`, `.md`, or `.json` file.
+
+Infer participant, researcher, interview type, date, and source revision when
+reliable; otherwise ask compact questions. Do not put connector credentials or
+the raw transcript into canonical memory merely to analyze it.
+
+## Context and analysis
+
+Reuse sufficient `EGREGORE_ORG_CONTEXT_V1`. Open selected canonical evidence
+through Runtime when verification is necessary. If cross-interview comparison
+is requested and context is insufficient, make at most one Egregore retrieval
+request. Never query graph/QMD directly; canonical Markdown/Git wins.
+
+Start with a compact scaffold. Analyze inline for short interviews. For long or
+complex material, parallel Journey, Sentiment, and Product lenses may run once,
+then synthesize their evidence and disagreements. Do not force multi-agent work
+when it adds latency without analytical value.
+
+Show the emotional arc, evidence-backed product reading, proposed durable
+insights, confidence, priority, tensions, and researcher actions. Ask `Save`,
+`Edit`, or `Skip`. Do not write before `Save`.
+
+## Canonical writeback
+
+Build one temporary `egregore-research-ingest/v1` package containing the
+approved interview briefing, participant journey document where needed, and
+typed findings/patterns/decisions. Then make exactly one deterministic call:
+
+```bash
+bash bin/research-ingest.sh interview --input "$PACKAGE_JSON"
+```
+
+Runtime owns authorization, schema/path validation, stable identity,
+provenance, canonical Markdown, Git provenance, derived-index refresh,
+background embedding, and telemetry. Do not separately create directories,
+append indexes, write files, run `/save`, or manage graph projection.
+
+Report accepted/partial/rejected status and canonical paths. Cross-interview
+synthesis reads canonical artifacts; graph is optional enrichment only.
+Publication and notifications are separate later effects and require their own
+explicit approval.