@event4u/agent-config 3.1.1 → 3.2.0

package/docs/guides/local-knowledge.md

@@ -0,0 +1,127 @@
+---
+stability: beta
+keep-beta-until: 2026-08-24
+---
+
+# Local knowledge — 5-minute walkthrough
+
+Point the agent at a folder of local files (PDFs, Markdown, Word docs, spreadsheets). It chunks, redacts PII + secrets, and persists into the agent memory namespace — local-only, single-user, no OAuth, no remote fetch.
+
+Contract: [`local-knowledge-ingestion`](../contracts/local-knowledge-ingestion.md).
+Roadmap home: `agents/roadmaps/road-to-employee-product-and-external-proof.md` Phase 2.
+
+## Prerequisites
+
+- Python 3.10+ on the host.
+- `markitdown` on `PATH` if the corpus contains PDF / DOCX / XLSX / EPUB / images. Pure markdown / text corpora work without it.
+- An `agents/` directory in the project (created by the installer). The `agents/memory/knowledge/` subdirectory is created lazily on first ingest.
+
+## Step 1 — Pick a folder
+
+Anything local works: a customer folder, a project drop, a `.zip` archive, a single PDF. The walk skips hidden dirs (`.git`, `.venv`, `node_modules`) and does not follow symlinks.
+
+For this walkthrough we use a folder with one PDF and three markdown notes:
+
+```text
+/Users/maintainer/clients/acme/
+├── brief.pdf
+├── kickoff-notes.md
+├── meeting-2026-05-12.md
+└── pricing-v3.md
+```
+
+## Step 2 — Ingest
+
+```bash
+/knowledge ingest /Users/maintainer/clients/acme/
+```
+
+Realistic output (your ingest-id will differ — uuid7s are time-ordered):
+
+```text
+✅ ingested 01927f4a-2b1c from /Users/maintainer/clients/acme/
+   documents: 4, chunks: 18, bytes_stored: 47312
+   PII redacted: EMAIL=3, PHONE=1, IBAN=0, CC=0, SSN=0
+   secrets redacted: 0
+   skipped: 0 unsupported MIME
+```
+
+What just happened:
+
+- Each file routed through `markitdown` (PDF) or passthrough (Markdown).
+- Chunks split at ~2 KB boundaries, written to `agents/memory/knowledge/<ingest-id>/chunks/<n>.md`.
+- A `manifest.json` recorded the source path, doc count, redaction counters, and `created_at`.
+- PII regex pass replaced 3 emails + 1 phone with `[EMAIL]` / `[PHONE]` placeholders **before** the chunk hit disk.
+
+> Want the raw text in (no redaction)? `--no-redact`. The manifest captures the flag so the audit row names every bypass. Default is always redact.
+
+## Step 3 — Ask the agent
+
+Use the host model normally. The MCP tool `memory_retrieve` now returns knowledge chunks alongside curated and intake entries — same envelope, with an additional `body.source_kind: knowledge` tag so the model knows the source is user-supplied, not maintainer-curated.
+
+Example prompt:
+
+> *"What does the acme pricing-v3 note say about volume discounts?"*
+
+The agent retrieves the matching chunks (pinned chunks rank slightly higher than unpinned; knowledge entries are discounted ~15 % vs curated so hand-reviewed content still wins on equal relevance) and answers with a citation back to the source path stored in the manifest.
+
+If nothing matches, the model says so. The retrieval surface does not invent a citation.
+
+## Step 4 — List + pin
+
+See what's been ingested:
+
+```bash
+/knowledge list
+```
+
+```text
+ID        DOCS  CHUNKS  BYTES   PINNED  REDACTED  CREATED              SOURCE
+01927f4a  4     18      47312   no      yes       2026-05-25T08:14:02  /Users/maintainer/clients/acme
+```
+
+Pin so it survives LRU eviction when the 500 MB namespace cap is crossed:
+
+```bash
+/knowledge list --pin 01927f4a
+```
+
+```text
+✅ pinned 01927f4a
+```
+
+Prefix must be unambiguous — if it matches > 1 ingest, the command rejects with a structured error and asks for a longer prefix.
+
+## Step 5 — Forget
+
+When the work is done, drop the ingest atomically:
+
+```bash
+/knowledge forget 01927f4a
+```
+
+```text
+✅ forgot 01927f4a — removed 18 chunks, 47312 bytes
+```
+
+Forget is atomic — no partial state. Pinned ingests are dropped the same as unpinned; pinning protects from LRU, not from explicit forget.
+
+## What the guide does **not** cover
+
+- Multi-user share — single-user by design. Multi-user lives behind ADR-024 workspace work and Phase 4 of the parent roadmap.
+- Remote sources — every input must resolve to a local path. `http://`, `https://`, `s3://`, `gs://`, `azure://` are rejected at the input validator.
+- Connector contracts (GitHub / Jira / Confluence) — those sit behind Hard-Floor OAuth and stay cancelled in `road-to-internal-ai-os-deployment.md` Phase 5.
+
+## Troubleshooting
+
+- **"Bound exceeded: total_ingest_size"** — the corpus is > 100 MB. Split it, or ingest a sub-folder.
+- **"Bound exceeded: document_count"** — > 1000 files. Same fix.
+- **"unsupported MIME"** — file skipped, counted in the summary, no chunk written. Add the file as `.md` if you need it indexed.
+- **OCR confidence < 0.7** — the chunk is tagged `low_confidence`. The model still receives it but the citation surface flags the lower confidence.
+- **markitdown not on PATH** — install it (`pip install 'markitdown[all]'`) or pass `--markitdown=<bin>`. Markdown-only corpora work without it.
+
+## See also
+
+- [`local-knowledge-ingestion`](../contracts/local-knowledge-ingestion.md) — contract (input shapes, bounds, storage, redaction).
+- [`/knowledge ingest`](../../.agent-src/commands/knowledge/ingest.md) · [`/knowledge list`](../../.agent-src/commands/knowledge/list.md) · [`/knowledge forget`](../../.agent-src/commands/knowledge/forget.md)
+- [`markitdown` skill](../../.agent-src/skills/markitdown/SKILL.md) — peer-side adapter for binary formats.

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@event4u/agent-config",
-    "version": "3.1.1",
+    "version": "3.2.0",
     "description": "Universal AI Agent OS \u2014 audited skills, governance rules, commands, and templates for AI coding tools (Claude Code, Cursor, Windsurf, Copilot).",
     "license": "MIT",
     "private": false,
@@ -26,7 +26,9 @@
         "skills",
         "prompt-engineering",
         "typescript",
-        "python"
+        "python",
+        "agent-skills",
+        "cinematic-ai-video"
     ],
     "files": [
         ".agent-src/",

package/scripts/_lib/changelog_eras.py

@@ -0,0 +1,330 @@
+"""Shared constants + helpers for CHANGELOG.md era discipline.
+
+The drift gate (``tests/test_changelog_eras.py``) and the release
+automation (``scripts/release.py``) both reason about the same era
+shape: a single ``# Era: X.Y.x — current`` header followed by inline
+entries, then ``# Era: pre-X.Y.0 — archived`` pointers to files under
+``docs/archive/``. Keeping the regex / cap / path constants in one
+place prevents drift between the gate and the auto-split logic.
+
+Normative source: ``docs/contracts/CHANGELOG-conventions.md § Era splits``.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent.parent
+CHANGELOG = REPO_ROOT / "CHANGELOG.md"
+CONVENTIONS = REPO_ROOT / "docs" / "contracts" / "CHANGELOG-conventions.md"
+ARCHIVE_DIR = REPO_ROOT / "docs" / "archive"
+
+# Drift cap — entries between the current era header and the next era
+# header may not exceed this many lines. Raising the cap is a contract
+# change (see CHANGELOG-conventions.md § Era splits).
+CURRENT_ERA_BODY_CAP = 250
+
+ERA_HEADER_RE = re.compile(
+    r"^# Era: (?P<label>[^\n]+?)(?: — (?P<state>current|archived))?\s*$"
+)
+ARCHIVE_LINK_RE = re.compile(r"\(docs/archive/(CHANGELOG-pre-[^)\s]+\.md)\)")
+VERSION_HEADING_RE = re.compile(r"^## \[?(?P<version>\d+\.\d+\.\d+)")
+ERA_LABEL_RE = re.compile(r"^(?P<major>\d+)\.(?P<minor>\d+)\.x$")
+
+
+@dataclass(frozen=True)
+class EraSpan:
+    """One era header in CHANGELOG.md, with its line index."""
+
+    line_index: int
+    label: str
+    state: str  # "current" | "archived" | ""
+
+
+def read_changelog_lines() -> list[str]:
+    """Return CHANGELOG.md split into lines (no trailing newlines)."""
+    return CHANGELOG.read_text(encoding="utf-8").splitlines()
+
+
+def era_spans(lines: list[str]) -> list[EraSpan]:
+    """Return every era header in line-order."""
+    spans: list[EraSpan] = []
+    for i, line in enumerate(lines):
+        m = ERA_HEADER_RE.match(line)
+        if m:
+            spans.append(
+                EraSpan(
+                    line_index=i,
+                    label=m.group("label"),
+                    state=m.group("state") or "",
+                )
+            )
+    return spans
+
+
+def current_era_index(spans: list[EraSpan]) -> int | None:
+    """Return the line index of the ``— current`` era header, or None."""
+    for span in spans:
+        if span.state == "current":
+            return span.line_index
+    return None
+
+
+def current_era_body_size(lines: list[str] | None = None) -> int:
+    """Return the number of lines between the current era header and
+    the next era header (exclusive of both)."""
+    if lines is None:
+        lines = read_changelog_lines()
+    spans = era_spans(lines)
+    current_idx = current_era_index(spans)
+    if current_idx is None:
+        return 0
+    next_era_line = len(lines)
+    for span in spans:
+        if span.line_index > current_idx:
+            next_era_line = span.line_index
+            break
+    return next_era_line - current_idx - 1
+
+
+def parse_era_label(label: str) -> tuple[int, int] | None:
+    """Parse ``M.N.x`` into ``(M, N)``; return None for archived labels."""
+    m = ERA_LABEL_RE.match(label.strip())
+    if not m:
+        return None
+    return int(m.group("major")), int(m.group("minor"))
+
+
+def archive_path_for_boundary(boundary: str) -> Path:
+    """Return ``docs/archive/CHANGELOG-pre-<boundary>.md``."""
+    return ARCHIVE_DIR / f"CHANGELOG-pre-{boundary}.md"
+
+
+def collapsed_era_block(boundary: str) -> str:
+    """Render the standard ``# Era: pre-<boundary> — archived`` pointer
+    block that replaces archived entries in CHANGELOG.md.
+
+    Mirrors the wording the manual splits already used (verified against
+    every existing collapsed era as of 3.2.x).
+    """
+    archive_rel = f"docs/archive/CHANGELOG-pre-{boundary}.md"
+    return (
+        f"# Era: pre-{boundary} — archived\n"
+        "\n"
+        f"> All entries before `{boundary}` live in\n"
+        f"> [`{archive_rel}`]({archive_rel}).\n"
+        "> The archive is read-only; git tags remain the canonical\n"
+        "> source for what shipped. Splitting them out of the main file\n"
+        "> keeps the active era under the 250-line drift cap enforced by\n"
+        "> `tests/test_changelog_eras.py`.\n"
+    )
+
+
+def archive_file_header(boundary: str) -> str:
+    """Return the standard prologue for ``docs/archive/CHANGELOG-pre-<boundary>.md``."""
+    return (
+        f"# Changelog Archive — pre-{boundary}\n"
+        "\n"
+        f"> Frozen snapshot of `event4u/agent-config` changelog entries\n"
+        f"> released before `{boundary}`, split out of the main\n"
+        "> [`CHANGELOG.md`](../../CHANGELOG.md) by `scripts/release.py`\n"
+        "> once the active era's body crossed the drift cap enforced by\n"
+        "> `tests/test_changelog_eras.py`.\n"
+        ">\n"
+        "> **Read-only.** New entries land in `CHANGELOG.md`. Entries\n"
+        "> here are not amended — git tags remain the canonical source\n"
+        "> for what shipped.\n"
+        ">\n"
+        "> Entry shape follows\n"
+        "> [`../contracts/CHANGELOG-conventions.md`](../contracts/CHANGELOG-conventions.md).\n"
+        "\n"
+    )
+
+
+# ─── split planning + execution ────────────────────────────────────────────────
+
+
+_RELEASE_VERSION_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)$")
+
+
+@dataclass(frozen=True)
+class SplitPlan:
+    """Recipe for an era split during release of ``release_version``."""
+
+    release_version: str  # e.g. "3.3.0"
+    boundary: str  # e.g. "3.3.0" — used in archive filename + pointer
+    new_era_label: str  # e.g. "3.3.x"
+    old_era_label: str  # e.g. "3.2.x"
+    archive_path: Path
+
+    @property
+    def commit_subject(self) -> str:
+        return (
+            f"chore(changelog): split era {self.old_era_label} "
+            f"→ pre-{self.boundary}"
+        )
+
+
+def plan_split(release_version: str) -> SplitPlan | None:
+    """Plan an era split when releasing ``release_version``.
+
+    Returns None when no split is needed (release is a patch within the
+    current era, or no current era header exists). Returns a SplitPlan
+    when the release crosses a minor or major boundary; the caller
+    decides whether to invoke ``perform_split`` based on era body size.
+
+    Raises ValueError when ``release_version`` is not bare semver, or
+    when it would move backward relative to the current era label.
+    """
+    m = _RELEASE_VERSION_RE.match(release_version.strip())
+    if not m:
+        raise ValueError(f"not a bare semver (X.Y.Z): {release_version!r}")
+    rel_major, rel_minor, _rel_patch = (int(m.group(i)) for i in (1, 2, 3))
+
+    lines = read_changelog_lines()
+    spans = era_spans(lines)
+    current = next((s for s in spans if s.state == "current"), None)
+    if current is None:
+        return None
+
+    parsed = parse_era_label(current.label)
+    if parsed is None:
+        return None
+    era_major, era_minor = parsed
+
+    if (rel_major, rel_minor) < (era_major, era_minor):
+        raise ValueError(
+            f"release {release_version!r} is older than current era "
+            f"{current.label!r}; refusing to plan a backwards split"
+        )
+    if (rel_major, rel_minor) == (era_major, era_minor):
+        # Patch release within the current era — no era boundary crossed,
+        # so an auto-split would create a nonsensical archive name. The
+        # caller is expected to die() with the manual-intervention message.
+        return None
+
+    boundary = f"{rel_major}.{rel_minor}.0"
+    return SplitPlan(
+        release_version=release_version,
+        boundary=boundary,
+        new_era_label=f"{rel_major}.{rel_minor}.x",
+        old_era_label=current.label,
+        archive_path=archive_path_for_boundary(boundary),
+    )
+
+
+def new_era_intro_block(new_era_label: str, boundary: str) -> str:
+    """Render the header + blockquote intro for a freshly-split current era."""
+    parsed = parse_era_label(new_era_label)
+    if parsed is None:
+        next_example = "# Era: <next>.x"
+    else:
+        m, n = parsed
+        next_example = f"# Era: {m}.{n + 1}.x"
+    return (
+        f"# Era: {new_era_label} — current\n"
+        "\n"
+        f"> Started at `{boundary}`. Full entries live inline below.\n"
+        "> The drift test caps this era at 250 lines of entry body; growth past\n"
+        f"> that forces a new era split (`{next_example}`, etc.) — see\n"
+        "> [`docs/contracts/CHANGELOG-conventions.md § Era splits`](docs/contracts/CHANGELOG-conventions.md).\n"
+    )
+
+
+def _era_body_bounds(
+    lines: list[str], current_idx: int
+) -> tuple[int, int, int]:
+    """Return ``(body_start, body_end, next_era_line)`` for the era at
+    ``current_idx``.
+
+    * ``body_start`` — first line after the header + leading blockquote
+      intro + the blank line that follows.
+    * ``body_end`` — exclusive; one line before the next era marker (or
+      end of file). Trailing blank lines are NOT trimmed; the caller
+      reattaches them on splice.
+    * ``next_era_line`` — index of the next ``# Era:`` line, or
+      ``len(lines)`` when none follows.
+    """
+    next_era_line = len(lines)
+    for i in range(current_idx + 1, len(lines)):
+        if ERA_HEADER_RE.match(lines[i]):
+            next_era_line = i
+            break
+
+    cursor = current_idx + 1
+    # Skip leading blank lines between header and blockquote intro.
+    while cursor < next_era_line and lines[cursor].strip() == "":
+        cursor += 1
+    # Skip the leading blockquote intro (consecutive `>`-prefixed lines).
+    while cursor < next_era_line and lines[cursor].startswith(">"):
+        cursor += 1
+    # Skip the blank separator between intro and entries.
+    while cursor < next_era_line and lines[cursor].strip() == "":
+        cursor += 1
+
+    return cursor, next_era_line, next_era_line
+
+
+def current_era_insertion_point(lines: list[str]) -> int | None:
+    """Return the line index at which a new release entry should be
+    prepended within the current era.
+
+    Strategy:
+    * If the current era body contains one or more ``## [X.Y.Z]``
+      headings, return the line of the topmost (newest) one.
+    * Otherwise, return the first line after the era intro blockquote.
+
+    Returns None when no current era header exists.
+    """
+    spans = era_spans(lines)
+    current_idx = current_era_index(spans)
+    if current_idx is None:
+        return None
+    body_start, body_end, _ = _era_body_bounds(lines, current_idx)
+    for i in range(body_start, body_end):
+        if VERSION_HEADING_RE.match(lines[i]):
+            return i
+    return body_start
+
+
+def perform_split(plan: SplitPlan) -> None:
+    """Execute ``plan`` against the on-disk CHANGELOG.md.
+
+    * Refuses to overwrite an existing archive file.
+    * Moves every entry in the current era body into the new archive.
+    * Replaces the current era block with the collapsed pointer + the
+      freshly-labelled new current era header (empty body).
+    """
+    if plan.archive_path.exists():
+        raise FileExistsError(
+            f"archive already exists at {plan.archive_path} — "
+            "likely a previous --resume run; inspect manually"
+        )
+
+    lines = read_changelog_lines()
+    spans = era_spans(lines)
+    current_idx = current_era_index(spans)
+    if current_idx is None:
+        raise RuntimeError("no current era header found in CHANGELOG.md")
+
+    body_start, _, next_era_line = _era_body_bounds(lines, current_idx)
+    entries = lines[body_start:next_era_line]
+    # Trim trailing blank lines so the archive doesn't accumulate them.
+    while entries and entries[-1].strip() == "":
+        entries.pop()
+
+    collapsed = collapsed_era_block(plan.boundary).rstrip("\n").splitlines()
+    new_era = new_era_intro_block(plan.new_era_label, plan.boundary).rstrip("\n").splitlines()
+
+    head = lines[:current_idx]
+    tail = lines[next_era_line:]
+    new_lines = head + collapsed + [""] + new_era + [""] + tail
+    new_text = "\n".join(new_lines).rstrip() + "\n"
+
+    archive_body = "\n".join(entries).rstrip() + "\n" if entries else ""
+    archive_text = archive_file_header(plan.boundary) + archive_body
+
+    plan.archive_path.parent.mkdir(parents=True, exist_ok=True)
+    plan.archive_path.write_text(archive_text, encoding="utf-8")
+    CHANGELOG.write_text(new_text, encoding="utf-8")

package/scripts/memory_lookup.py

@@ -39,6 +39,7 @@ from typing import Any, Callable, Iterable, Optional, Union
 
 MEMORY_ROOT = Path("agents/memory")
 INTAKE_ROOT = MEMORY_ROOT / "intake"
+KNOWLEDGE_ROOT = MEMORY_ROOT / "knowledge"
 
 CURATED_TYPES = {
     "ownership",
@@ -49,6 +50,12 @@ CURATED_TYPES = {
     "product-rules",
 }
 
+# `knowledge` is its own type: user-ingested local documents that live
+# under `agents/memory/knowledge/<ingest-id>/chunks/*.md`. They are
+# repo-side (file-backed) but not "curated" and not intake — the
+# conflict rule still treats them as repo entries against operational.
+KNOWLEDGE_TYPE = "knowledge"
+
 
 @dataclass
 class Hit:
@@ -167,6 +174,58 @@ def _iter_intake_entries(mtype: str) -> Iterable[tuple[Path, dict]]:
             yield jsonl, obj
 
 
+def _iter_knowledge_entries() -> Iterable[tuple[Path, dict]]:
+    """Yield (chunk-file, entry) pairs from `agents/memory/knowledge/`.
+
+    Layout (frozen in `docs/contracts/local-knowledge-ingestion.md`):
+
+        agents/memory/knowledge/<ingest-id>/
+            manifest.json
+            chunks/<n>.md
+
+    Each chunk becomes one retrieval entry. The chunk body, the
+    manifest source path, and pinned flag are surfaced into the entry
+    so `_score()` can match on either the source path or the chunk
+    text. The entry id is ``<ingest-id>:<chunk-stem>`` so callers can
+    locate the exact file on disk.
+    """
+    if not KNOWLEDGE_ROOT.is_dir():
+        return
+    for ingest_dir in sorted(KNOWLEDGE_ROOT.iterdir()):
+        if not ingest_dir.is_dir():
+            continue
+        manifest_path = ingest_dir / "manifest.json"
+        manifest: dict = {}
+        if manifest_path.is_file():
+            try:
+                manifest = json.loads(
+                    manifest_path.read_text(encoding="utf-8")
+                )
+            except (ValueError, OSError):
+                manifest = {}
+        ingest_id = str(manifest.get("ingest_id") or ingest_dir.name)
+        source = str(manifest.get("source") or "")
+        pinned = bool(manifest.get("pinned", False))
+        chunks_dir = ingest_dir / "chunks"
+        if not chunks_dir.is_dir():
+            continue
+        for chunk in sorted(chunks_dir.glob("*.md")):
+            try:
+                body = chunk.read_text(encoding="utf-8")
+            except OSError:
+                continue
+            entry = {
+                "id": f"{ingest_id}:{chunk.stem}",
+                "ingest_id": ingest_id,
+                "source": source,
+                "path": source,
+                "body": body,
+                "pinned": pinned,
+                "source_kind": "knowledge",
+            }
+            yield chunk, entry
+
+
 def _score(entry: dict, keys: list[str]) -> float:
     """Naive relevance score: max over keys of (glob-match | substring).
 
@@ -378,6 +437,24 @@ def retrieve(
     """
     repo_hits: list[Hit] = []
     for mtype in types:
+        if mtype == KNOWLEDGE_TYPE:
+            for path, entry in _iter_knowledge_entries():
+                base = _score(entry, keys)
+                # Pinned entries get a slight ranking boost so the
+                # `/knowledge:list --pin` flag has retrieval effect.
+                if entry.get("pinned"):
+                    base = min(1.0, base + 0.05)
+                repo_hits.append(Hit(
+                    id=str(entry.get("id", "")),
+                    type=KNOWLEDGE_TYPE,
+                    source="knowledge",
+                    path=str(path),
+                    # Discount vs curated/intake so hand-reviewed repo
+                    # entries still win on equal relevance.
+                    score=base * 0.85,
+                    entry=entry,
+                ))
+            continue
         if mtype not in CURATED_TYPES:
             continue
         for path, entry in _iter_curated_entries(mtype):
@@ -426,7 +503,7 @@ CONTRACT_VERSION = 1
 
 # Memory types this file-backed backend can answer. Types outside this
 # set map to `unknown_type` per the retrieval contract.
-_KNOWN_TYPES = CURATED_TYPES
+_KNOWN_TYPES = CURATED_TYPES | {KNOWLEDGE_TYPE}
 
 
 def retrieve_v1(