@x12i/memorix-docs 1.0.2 → 1.1.0

package/files/use-cases/ops-pack-fixtures.md

@@ -0,0 +1,168 @@
+# Package ops fixtures and acceptance harness
+
+> **Use case id:** `ops-pack-fixtures`
+> **Goal:** Build or extend an agent pack with g12ops* acceptance, goldens, and harness commands.
+> **Audiences:** developers
+> **Tags:** ops-pack, fixtures, acceptance
+
+## Reading path
+
+1. **Agent Packs & Ops** (developers) → chapters: `2-pack-layout`, `3-harness-up-down-reset-accept`, `7-install-into-a-fresh-acceptance-org`, `6-golden-v2-validation`
+
+## From: Agent Packs & Ops — 2. Pack layout
+
+```text
+memorix-ops-pack/
+  metadata/
+    manifest.json
+    agents/ops.json
+    agents/project-ops.json
+    abstract-types.json
+    abstract-bindings.json
+    object-types.json
+    relationship-definitions.json
+    view-definitions.json
+    write-and-source-mappings.json
+    pipelines-and-sources.json
+  fixtures/                 # deterministic source data
+  ops-source/server.mjs     # local REST fixture (no internet)
+  scripts/harness.mjs       # up / down / reset / accept
+  examples/v2/              # memorix-record/2 goldens
+```
+
+Diagram (same idea):
+
+```text
+pack/
+  metadata/
+  fixtures/
+  ops-source/
+  scripts/
+  examples/v2/
+```
+
+### Manifest
+
+`metadata/manifest.json` lists includes for agent `ops` (versioned). Install tooling resolves those files into installable metadata items.
+
+Typical includes: parent agent `ops`, child `project-ops`, abstracts + bindings, object types, relationship definitions, views, write/source mappings, pipelines and sources.
+
+### Subjects in the narrative
+
+| Subject | Role in fixtures |
+|---|---|
+| Employee **Alex** | Stable `recordId` across snapshot / workflow / optional knowx |
+| Work item **WI-100** | Prepare quarterly operations review |
+| Procedure **PROC-2** | Knowledge-category procedure content |
+| Activity events | Event-category samples |
+| Malformed / deleted items | Negative paths for quarantine / 404 checks |
+
+---
+
+## From: Agent Packs & Ops — 3. Harness: up / down / reset / accept
+
+```bash
+cd memorix-ops-pack
+npm install && npm test
+node scripts/harness.mjs up          # OPS_SOURCE_PORT default 5102
+node scripts/harness.mjs reset       # reset fixture control state
+node scripts/harness.mjs accept      # smoke: metadata files + source routes
+node scripts/harness.mjs down
+```
+
+Prefer the monorepo stack when exercising Studio + service together:
+
+```bash
+node scripts/dev-stack.mjs
+# includes ops-source on :5102 (even ports = APIs), stack-status :5103, /_live on APIs
+```
+
+| Command | Purpose |
+|---|---|
+| `up` | Start localhost `ops-source` (PID under `.run/`) |
+| `down` | Stop ops-source via PID file |
+| `reset` | POST fixture control reset; re-establish known source state |
+| `accept` | Verify manifest files exist; hit key source routes; assert negative paths |
+
+### What `accept` is (and is not)
+
+**Is:** a deterministic smoke that the pack’s metadata files and fixture server behave.
+
+**Is not:** a license to use live customer orgs; not a substitute for service-level gates on a fresh `g12ops*` with explicit metadata install + pipeline green.
+
+Platform product acceptance always pairs pack smoke with:
+
+```http
+x-memorix-org-id: g12ops<suffix>
+x-memorix-agent-ids: ops
+```
+
+and an explicit `/api/metadata/install` (or Studio install) into that org.
+
+### Harness sketch in CI
+
+```bash
+node scripts/harness.mjs up
+node scripts/harness.mjs accept
+npx memorix-format validate-v2 examples/v2
+node scripts/harness.mjs down
+```
+
+---
+
+## From: Agent Packs & Ops — 7. Install into a fresh acceptance org
+
+```bash
+ORG=$(node -e "console.log('g12ops'+Math.random().toString(36).slice(2,10))")
+H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: ops")
+
+curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
+  -H "content-type: application/json" \
+  -d '{"packId":"ops","confirm":true}'
+
+curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | head
+
+# Memory pull from ops-source (credentialRef per Memory contracts)
+# Pipeline start → assert WI-100 composition + links
+```
+
+Assertions for a full gate:
+
+- [ ] Manifest files present; `harness accept` green
+- [ ] `validate-v2` green on `examples/v2`
+- [ ] Install with `confirm: true` into `g12ops*`
+- [ ] Memory pull lands raw for employees / work-items / …
+- [ ] Pipeline green; Explore/HTTP composition for `wi-100`
+- [ ] Relationship verify for documented definitions
+- [ ] Custom kinds (if any) visible via `/api/metadata/kinds` and **not** executed by pipelines
+- [ ] No live customer org; agents `["ops"]`
+
+---
+
+## From: Agent Packs & Ops — 6. Golden V2 validation
+
+Ops pack ships goldens under `examples/v2/` (snapshots, raw, workflow, analysis, decisions, lightweight-refs, …).
+
+```bash
+npx memorix-format validate-v2 examples/v2
+```
+
+Rules:
+
+- Every golden must be `format: "memorix-record/2"`.
+- No copied associated payloads (`V2-ASSOC-COPIED`).
+- Discriminators present when required (`V2-DISCRIMINATOR`).
+- Workflow / analysis / decisions siblings stay separate content types.
+- Relationship **navigation** is not proven by fat goldens — use four-field links + engine verify in service gates.
+
+Migration note: present → v2 belongs in book `13` / Format §§18–19. Pack authors ship **V2**, not present envelopes, as target truth.
+
+---
+
+## Also see
+
+- **Platform Overview** (`00-platform-overview`) — Surfaces, packages, hard rules, and where to start — developers and builders twins.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._

package/files/use-cases/orient-platform.md

@@ -0,0 +1,172 @@
+# Orient on Memorix after G12
+
+> **Use case id:** `orient-platform`
+> **Goal:** Understand surfaces, packages, hard rules, and which ebook to open next.
+> **Audiences:** developers, builders
+> **Tags:** platform, start-here
+
+## Reading path
+
+1. **Platform Overview** (developers) → chapters: `the-central-mental-model`, `2-architecture`, `8-hard-product-rules-integrator-checklist`, `10-decision-tree-which-ebook-next`
+
+## From: Platform Overview — The central mental model
+
+Memorix is a **descriptor-driven data platform**.
+
+> **One kind of content about one stable subject** — composed by `recordId`, navigated by four-field relationships, projected by virtual abstracts.
+
+| Idea | Meaning |
+|---|---|
+| Subject | The stable thing (`WI-100`, employee Alex) |
+| Content | One kind of information *about* that subject (snapshot, workflow, analysis, …) |
+| Metadata | Declares object types, content types, abstracts, relationships, views, services, pipelines |
+| Runtime data | Stored as `memorix-record/2` documents |
+| Scope | Every request names a data universe (`orgId`) and effective metadata (`agentIds[]`) |
+
+Integrators never invent list columns or write shapes in the host. They load metadata, then read/write through the supported boundary.
+
+**Published npm lineup:** all public `@x12i/memorix-*` packages are version **`3.0.0`**. Install with `^3.0.0` (or omit the range and take `latest`).
+
+---
+
+## From: Platform Overview — 2. Architecture
+
+```mermaid
+flowchart TB
+  YourApp[Your solution] --> Service[memorix-service]
+  Studio[memorix-studio] --> Service
+  Service --> MD[metadata]
+  Service --> Data[memorix-data]
+  Service --> Rel[relationships]
+  Service --> Mem[memory]
+  Service --> Pipe[pipeline]
+  Service --> Intel[intelligence]
+  Service --> Wf[workflow]
+  Data --> Mongo[(Mongo)]
+```
+
+See also [_shared/diagrams/platform-planes.md](../../_shared/diagrams/platform-planes.md) and [diagrams/surfaces.md](../diagrams/surfaces.md).
+
+### How packages connect
+
+```mermaid
+flowchart TB
+  App[Your solution] --> Svc[memorix-service]
+  Studio[memorix-studio] --> Svc
+  Svc --> Meta[metadata + runtime]
+  Svc --> DataPlane[memorix-data]
+  Svc --> RelEng[relationships]
+  Svc --> MemEng[memory]
+  Svc --> PipeEng[pipeline]
+  Svc --> IntelEng[intelligence]
+  Fmt[format] --> DataPlane
+  Map[mapping] --> PipeEng
+  DataPlane --> Mongo[(Mongo)]
+  RelEng --> Store[relationship-store]
+  PipeEng --> PipeRt[pipeline-runtime]
+  MemEng --> DataPlane
+  PipeEng --> DataPlane
+  PipeEng --> RelEng
+  IntelEng -.->|proposes| Meta
+```
+
+### Request lifecycle
+
+1. Client sends `x-memorix-org-id` + `x-memorix-agent-ids`.
+2. Service resolves **effective metadata** for that agent set.
+3. The requested engine (data, relationships, memory, pipelines, …) runs against scoped Mongo.
+4. Response may include composition and/or relationship expansion (virtual at read time).
+
+### Ingest path (ops journey)
+
+```text
+External source (ops-source)
+  → Memory pull lands exact raw payload (*-raw)
+  → Pipeline maps / validates / writes subject content
+  → Relationship engine materializes navigation links
+  → Explore / host apps see composition + links (never copied payloads)
+```
+
+Example subjects in the permanent narrative: employee **Alex**, work item **WI-100**, procedure **PROC-2**. See [_shared/ops-scenario.md](../../_shared/ops-scenario.md).
+
+### Trusted writers
+
+| Concern | Writer |
+|---|---|
+| Subject content | `memorix-data` (via service HTTP) |
+| Links | `memorix-relationships` only |
+| Raw landing | `memorix-memory` → data plane |
+| Metadata install | Explicit metadata-runtime / HTTP — **not** intelligence |
+
+---
+
+## From: Platform Overview — 8. Hard product rules (integrator checklist)
+
+These invariants apply to every integration, acceptance example, and ebook:
+
+1. Acceptance orgs are fresh `g12ops*` with `agentIds: ["ops"]`.
+2. Permanent narrative is non-cyber **ops** (employees, work-items, procedures).
+3. Relationships are four-field links only — no root `associated*` storage, no copied payloads.
+4. Abstracts are virtual projections — no abstract collections / abstract `recordId`.
+5. KnowX is one content type + `data.kind` (`NODE` | `EDGE`) + one abstract `KnowXEntity`.
+6. Provenance (`docHash` + `SourceSpan`) is mandatory for KnowX; `epistemic.state` ≠ extraction confidence.
+7. Deterministic fixtures only for acceptance — no live AI required.
+8. Mutations go through the data plane (`memorix-service` / `@x12i/memorix-data`), never direct Mongo domain writes.
+
+Full text: [HARD-RULES.md](../../HARD-RULES.md).
+
+---
+
+## From: Platform Overview — 10. Decision tree — which ebook next?
+
+```text
+Need the record envelope?
+  → Format for Dummies
+Need to call the API correctly?
+  → Runtime Data Client
+Need list/cursor/bulk semantics?
+  → Lists, Queries, Pagination
+Need “which call for this job?”
+  → Data Access Patterns
+Need agents / OT / CT / abstracts / mappings?
+  → Metadata, Agents, Abstracts & Mappings
+Need links or source landing?
+  → Relationships & Memory
+Need custom code / AI jobs?
+  → Building Services
+Need ordered multi-step work?
+  → Building Pipelines
+Need Studio / workflow siblings?
+  → Workflow & Studio
+Need profiling proposals?
+  → Structural Intelligence
+Need fixtures / harness?
+  → Agent Packs & Ops
+Building knowledge claims?
+  → Building KnowX
+Migrating or stuck?
+  → Migration & Troubleshooting
+```
+
+### Recommended reading order
+
+1. [Format for Dummies](../../01-format-for-dummies/developers/BOOK.md)
+2. [Runtime Data Client](../../02-runtime-data-client/developers/BOOK.md)
+3. [Data Access Patterns](../../04-data-access-patterns/developers/BOOK.md)
+4. Then topic books as your product needs them
+
+### Glossary
+
+Shared vocabulary: [_shared/glossary.md](../../_shared/glossary.md).
+
+### Twin for Studio audiences
+
+Builders who design and operate without API detail: [Builders version](../builders/BOOK.md).
+
+## Also see
+
+- **Platform Overview** (`00-platform-overview`) — Surfaces, packages, hard rules, and where to start — developers and builders twins.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._

package/files/use-cases/pipeline-retry-evidence.md

@@ -0,0 +1,121 @@
+# Retry a failed pipeline step with evidence
+
+> **Use case id:** `pipeline-retry-evidence`
+> **Goal:** Operate failed attempts, retry safely, and keep artifacts for forensics.
+> **Audiences:** developers
+> **Tags:** pipelines, ops, retry
+
+## Reading path
+
+1. **Building Pipelines** (developers) → chapters: `10-operations`, `11-errors-catalog`, `14-antipatterns-and-troubleshooting`
+
+## From: Building Pipelines — 10. Operations
+
+| Need | Where |
+|---|---|
+| Start / retry runs | `/api/pipelines` or Studio Operate |
+| Run graph / ledger | Pipeline runtime collections (org-scoped operational DB) |
+| Step / attempt inspection | Attempt documents with status, error, item outcomes |
+| Artifacts | Artifact store + DataRef |
+| Schedules | Design schedules + Operate history |
+| Memory diagnostics | Memory pull/quarantine ledgers |
+| Relationship diagnostics | `/api/relationships` verify/rebuild |
+| AI evidence | Request/response artifacts |
+
+### Operator diagnosis
+
+1. Open failed attempt — read structured `error.code`.
+2. Compare captured context artifact vs current Explore data.
+3. Fix mapping/service/metadata; reinstall if needed.
+4. Retry failed step or failed batch items only.
+5. After process restart, confirm resume without redoing succeeded steps.
+6. Confirm Explore for **WI-100** / **Alex** / **PROC-2**.
+
+### Retention
+
+Keep run evidence needed for audit; clean up expired context artifacts per Memory/policy. Do not delete ledgers that explain production incidents until retention allows.
+
+---
+
+## From: Building Pipelines — 11. Errors (catalog)
+
+| Code class | Layer | Retryable? | Safe action |
+|---|---|---|---|
+| Invalid definition / unknown step / cycle | Validate | No | Fix pack |
+| Missing dependency / invalid reference | Validate | No | Fix graph |
+| Missing service / port | Startup/run | No | Bind registry / ports |
+| Missing prod ledger/artifact store | Startup | No | Configure durable stores |
+| Invalid scope / input | Run | No | Fix caller |
+| Mapping / validation failure | Step | No | Fix mapping/data |
+| Identity conflict | Step | No | Resolve identity |
+| Stale revision | Write | No auto | Re-read; new revision |
+| Relationship discover/write failure | Step | Often partial | Retry relationship portion |
+| Memory / cursor failure | Step | Often yes | Fix source; resume cursor |
+| Artifact missing / wrong org | DataRef | No | Restore / fix scope |
+| Timeout / cancelled | Attempt | Policy | Retry or accept cancel |
+| Retry exhausted | Step | No | Manual intervene |
+| Partial batch | Run | Mixed | Retry failed items |
+| Invalid AI output | Step | Policy | Repair or fail |
+| External action uncertain/failed | Step | Keyed only | Inspect receipt |
+| Unsupported atomicity | Validate | No | Disable atomicBatch |
+| Schedule error | Scheduler | No | Fix cron/config |
+
+Sanitize all evidence — no secrets, no unrestricted payloads.
+
+---
+
+## From: Building Pipelines — 14. Antipatterns and troubleshooting
+
+### Antipatterns
+
+| Antipattern | Prefer |
+|---|---|
+| Pipeline for a pure rename | Mapping |
+| Service that hides persistence | Capture → service → validate → write |
+| Auto-promote raw to snapshot | Explicit map/validate/`write-record` |
+| Ad-hoc relationship `PUT` | Discover → relationship-write via engine |
+| Secrets in pipeline JSON | `credentialRef` on sources/services |
+| AI auto-installs metadata | Proposal artifact + human install |
+| Global transaction assumption | Explicit partial success; retry failed portion |
+| Live AI for ops acceptance | Recorded adapter / fixtures |
+| Re-schedule to “fix” resume | Resume from durable ledger |
+| Direct Mongo domain writes | Data plane + relationship engine |
+
+### Troubleshooting decision tree
+
+```text
+Fails before any step?
+├─ Invalid definition / cycle / unknown step → fix pack
+├─ Missing service/port/ledger → bind durable stores
+└─ Wrong scope → fresh g12ops* + ops
+
+Map/validate fails?
+└─ Fix mapping against raw for PROC-2 / WI-100
+
+Write stale?
+└─ Re-read or new run; do not force overwrite blindly
+
+Green run, empty Explore?
+├─ Missing write-record?
+└─ Wrong objectType/contentType?
+
+Links missing?
+├─ Discover without write?
+└─ Verify/rebuild not run?
+
+Partial batch?
+└─ Retry failed items only — do not redo succeeded writes
+
+AI flaky in acceptance?
+└─ Recorded adapter; inspect request/response artifacts
+```
+
+---
+
+## Also see
+
+- **Workflow & Studio** (`09-workflow-and-studio`) — Design · Operate · Explore — scope bar ≡ headers, workflow siblings, embedding guidance. Builders primary.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._

package/files/use-cases/relationship-rebuild-verify.md

@@ -0,0 +1,128 @@
+# Rebuild and verify relationship links
+
+> **Use case id:** `relationship-rebuild-verify`
+> **Goal:** Run discover / verify / rebuild / cleanup through the relationship engine.
+> **Audiences:** developers
+> **Tags:** relationships, ops
+
+## Reading path
+
+1. **Relationships & Memory** (developers) → chapters: `4-engine-lifecycle-discover-verify-rebuild-cleanup`, `6-ops-narrative-walkthrough`
+
+## From: Relationships & Memory — 4. Engine lifecycle — discover / verify / rebuild / cleanup
+
+The relationship engine owns a deliberate lifecycle. Product callers do **not** PUT/DELETE individual links.
+
+### Responsibilities
+
+1. **Compile** abstract relationship definitions → concrete binding plans.
+2. **Discover** candidate links (bounded, non-Cartesian).
+3. **Materialize** idempotent four-field links via the store.
+4. **Verify** existing links against current definitions and subject content.
+5. **Rebuild** a definition’s links when definitions or source evidence change.
+6. **Cleanup** stale links for a definition according to policy.
+
+Conceptual trusted call:
+
+```ts
+import { createRelationshipEngine } from "@x12i/memorix-relationships";
+
+await engine.discoverAndMaterialize(scope, "people-assigned-to-work-items");
+```
+
+### HTTP surface
+
+| Capability | Via |
+|---|---|
+| List definitions | `GET /api/relationships/definitions` (and related) |
+| Verify / rebuild / query | `/api/relationships` operator routes |
+| Ad-hoc link PUT/DELETE | **405** — forbidden for product callers |
+
+There is:
+
+- **No** public manual put/delete relationship API for product callers
+- **No** auto-run on metadata change (installing a definition does not silently rewrite all links)
+- **No** AI on the deterministic path
+
+### Discover (bounded)
+
+Discovery rules read evidence (often source fields on raw/snapshots) and resolve endpoints across concrete types that bind to the abstract sides.
+
+Guarantees you should expect:
+
+- Candidates are **bounded** — not “every person × every work item.”
+- Unresolved targets are **first-class** (quarantined / reported), not silently invented.
+- Repeated discover+materialize is **idempotent** for the same logical link (`relationshipId` is deterministic).
+
+### Verify / rebuild / cleanup
+
+| Operation | Intent |
+|---|---|
+| Verify | Compare materialized links to what discovery would produce now; report drift |
+| Rebuild | Re-run discover+materialize for one definition; reconcile store |
+| Cleanup | Remove stale links for one definition per policy |
+
+Always operate **one definition at a time** unless a higher-level pipeline explicitly sequences multiple definitions.
+
+### Pipeline pairing (ops)
+
+The ops pack includes an explicit relationship pipeline pattern:
+
+```text
+relationship-discover → relationship-write → relationship-verify
+```
+
+Pipelines **invoke** the engine; they do not become a second link writer.
+
+---
+
+## From: Relationships & Memory — 6. Ops narrative walkthrough
+
+### Scope
+
+```http
+x-memorix-org-id: g12ops7
+x-memorix-agent-ids: ops
+```
+
+### Happy path
+
+1. **Metadata installed** for `ops` (sources, object types, relationship definitions).
+2. **Memory knowledge pull** from `ops-source` using `credentialRef` → `employees-raw`, `work-items-raw`, …
+3. **Pipeline promote** → snapshots for Alex, WI-100, PROC-2, …
+4. **Relationship discover/materialize** for `people-assigned-to-work-items` (and others as needed).
+5. **Verify** the definition — no unexpected drift.
+6. **Read** WI-100 or Alex with `relationships.include: true` — navigation only, no copied payloads.
+7. If a target is missing, Explore shows **unresolved** — fix data or rebuild; do not hand-edit links.
+
+### Integrator checklist
+
+- [ ] Source uses `credentialRef`, not pasted secrets
+- [ ] `*-raw` exists before you expect snapshots
+- [ ] Snapshots exist on both sides before you expect links
+- [ ] Relationship definition installed for the agent
+- [ ] Links created via discover/materialize or Operate verify/rebuild — never ad-hoc PUT
+- [ ] Expansion shows four-field navigation; unresolved list explained or empty
+- [ ] No `associated*` blobs on subject records
+
+### Failure modes
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| 405 on link PUT/DELETE | Calling forbidden ad-hoc API | Use verify/rebuild/query only |
+| Empty links after promote | Discovery not run / wrong definition id | Run discover+materialize for that definition |
+| Unresolved targets | Missing subject content or bad source ids | Seed/fix data; rebuild |
+| Cartesian explosion fear | Misread discovery | Confirm bounded rules; report if unbounded |
+| Secrets in quarantine | Credential misuse | Switch to `credentialRef` |
+
+Tutorial companion: [documentation/tutorials/05-relationships.md](../../../documentation/tutorials/05-relationships.md).
+
+---
+
+## Also see
+
+- **Agent Packs & Ops** (`11-agent-packs-and-ops`) — Pack layout, harness, g12ops* acceptance, custom kinds, golden V2 validation.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._