@x12i/memorix-docs 1.0.1 → 1.1.0

package/files/use-cases/learn-record-envelope.md

@@ -0,0 +1,233 @@
+# Learn the memorix-record/2 envelope
+
+> **Use case id:** `learn-record-envelope`
+> **Goal:** Internalize OT/CT, composition, discriminators, and what must not live on the record.
+> **Audiences:** developers
+> **Tags:** format, record
+
+## Reading path
+
+1. **Format for Dummies** (developers) → chapters: `2-anatomy-of-memorix-record-2`, `4-object-type-content-type-and-data-category`, `9-composition-same-subject-different-content`, `10-discriminated-content-instances`, `12-relationships-are-not-record-content`
+
+## From: Format for Dummies — 2. Anatomy of `memorix-record/2`
+
+```json
+{
+  "format": "memorix-record/2",
+  "recordId": "WI-100",
+  "revision": 1,
+  "objectType": "work-items",
+  "contentType": "snapshots",
+  "dataCategory": "entity",
+  "concept": {},
+  "data": {},
+  "_system": {}
+}
+```
+
+*(The empty `{}` above are illustration only — real records must not ship empty optional buckets.)*
+
+| Root | Meaning |
+|---|---|
+| `format` | Must be exactly `"memorix-record/2"`. |
+| `recordId` | Stable subject identity (org-wide). |
+| `revision` | Integer ≥ 1 for *this* content instance; starts at 1. |
+| `objectType` | Concrete subject type (`work-items`, `employees`, …). |
+| `contentType` | Kind of content (`snapshots`, `raw`, `workflow`, …). |
+| `dataCategory` | Data universe: `entity` \| `event` \| `knowledge`. |
+| `concept` | Stable identifying / business-facing info (title + identifiers). |
+| `data` | Content-type-specific payload. |
+| `_system` | Machine state, timestamps, optional lifecycle & provenance. |
+
+**Also allowed (narrowly):**
+
+- Snapshot-like types may carry non-empty `analytics`, `enrichment`, `insights`.
+- Storage-only `_id` may exist; format law ignores it as business identity.
+- Format validation still recognizes lightweight root `associated*` refs (compatibility). Production V2 writers reject *new* root `associated*` writes — authoritative links live in central `relationships` collections.
+
+**Rules you will feel every day:**
+
+- Unknown root properties are **rejected**.
+- Empty optional buckets are **omitted** — never emit meaningless `{}`.
+- Only `_system` and `_id` underscore roots are allowed on the record.
+
+---
+
+## From: Format for Dummies — 4. Object type, content type, and data category
+
+Explain the three separately — they answer different questions.
+
+| Field | Meaning | Example |
+|---|---|---|
+| `objectType` | Concrete subject type | `work-items` |
+| `contentType` | Content about that subject | `snapshots` |
+| `dataCategory` | Database category | `entity` |
+
+### Common data categories
+
+| Category | Typical subjects |
+|---|---|
+| `entity` | employees, work-items, contractors |
+| `event` | activity-events |
+| `knowledge` | procedures, documents |
+
+Operational runs, ledgers, artifacts, and quarantines belong to **operational** storage. They are not ordinary subject content and are not written as `memorix-record/2` subject envelopes.
+
+### Common content types
+
+`raw` · `snapshots` · `analysis` · `decisions` · `discovery` · `workflow` · `context`
+
+Metadata defines which object/content types exist for an agent pack. This list is **not** permanently closed — profiles come from effective metadata (with built-in defaults for well-known types).
+
+Physical collection name: `<objectType>-<contentType>`  
+Example: `work-items-snapshots` in database `g12opsabc123_entity`.
+
+---
+
+## From: Format for Dummies — 9. Composition: same subject, different content
+
+### Core rule
+
+> **Same `recordId` means composition, not a relationship.**
+
+### Ops journey example
+
+All of these are sibling content about `wi-100`:
+
+| Content | Instance |
+|---|---|
+| Snapshot | `wi-100` + `snapshots` |
+| Workflow | `wi-100` + `workflow` |
+| Analysis | `wi-100` + `analysis` + `capacity` |
+| Analysis | `wi-100` + `analysis` + `delivery-risk` |
+| Decision | `wi-100` + `decisions` + `priority` |
+
+### Read behavior
+
+- Default reads (`getContent`) return **one** requested content instance.
+- Composition must be **explicitly** requested (`composeSubject`).
+- Composed results contain **arrays by content type** (even for singletons).
+- Siblings are **never** flattened into the snapshot.
+- Workflow and context are ordinary sibling content.
+- Composition **never** creates relationship definitions or links.
+
+### Composition response shape (virtual)
+
+```json
+{
+  "recordId": "wi-100",
+  "objectType": "work-items",
+  "dataCategory": "entity",
+  "contents": {
+    "snapshots": [ /* V2 */ ],
+    "analysis": [ /* zero or more */ ],
+    "decisions": [ /* zero or more */ ],
+    "workflow": [ /* V2 */ ]
+  },
+  "warnings": []
+}
+```
+
+---
+
+## From: Format for Dummies — 10. Discriminated content instances
+
+### Why some content types have more than one instance
+
+Analysis, decisions, and discovery can exist many times per subject (capacity vs delivery-risk; priority vs schedule; …).
+
+### Built-in discriminator paths (defaults)
+
+| Content type | Path | Default |
+|---|---|---|
+| `analysis` | `data.analysisType` | `"default"` |
+| `decisions` | `data.decisionType` | `"default"` |
+| `discovery` | `data.discoveryType` | `"default"` |
+
+Metadata defines path / requiredness / default / allowed values. Defaults are applied and **persisted** before addressing so the record stays self-describing.
+
+### Addressing
+
+Same `recordId` + `contentType` + discriminator value → one instance.
+
+- Discriminator values are **immutable** for an existing instance.
+- Changing the discriminator requires **remove + add**.
+- Each instance has its own revision.
+- Mongo’s deterministic storage key is internal; `recordId` remains the subject.
+
+### Two analyses on one work item
+
+```json
+{
+  "recordId": "wi-100",
+  "contentType": "analysis",
+  "data": { "analysisType": "delivery-risk", "risk": "low" }
+}
+```
+
+```json
+{
+  "recordId": "wi-100",
+  "contentType": "analysis",
+  "data": { "analysisType": "capacity", "load": "nominal" }
+}
+```
+
+Validation: missing required discriminator → `V2-DISCRIMINATOR`. Missing with default → warning `V2-DISCRIMINATOR-DEFAULT` (writers apply default).
+
+---
+
+## From: Format for Dummies — 12. Relationships are not record content
+
+### Core rule
+
+> **Different `recordId` values are connected using relationship links.**
+
+### Four-field stored link
+
+```json
+{
+  "relationshipId": "…",
+  "relationshipDefinitionId": "people-assigned-to-work-items",
+  "fromRecordId": "employee-10",
+  "toRecordId": "wi-100"
+}
+```
+
+These are the **only** persisted link fields.
+
+| Concept | Role |
+|---|---|
+| Relationship **definition** | Metadata (meaning, directionality, path) |
+| Relationship **link** | Data (four fields) |
+
+### Rules
+
+- Links do not contain copied names, titles, or target records.
+- Links live in centralized per-category `relationships` collections (`g12opsabc123_entity.relationships`, etc.).
+- Forward and inverse traversal query the **same** physical link.
+- Expansion loads **current** target content when requested.
+- Missing targets appear as `unresolved` (calculated) — the link is not deleted.
+- Updating the target changes expansion without changing the link.
+- Relationship history / provenance belongs in operational runs.
+
+### Compatibility nuance
+
+| Shape | Status |
+|---|---|
+| Lightweight root `associated*` | Format-valid for validation/migration; **not** authoritative storage |
+| Production V2 writers | Reject new root `associated*` writes |
+| Sibling `contentType: "relationships"` | Format-valid profile only; **not** authoritative storage — writers reject it |
+| Nested `.data.associated*` | Valid when it is untouched **provider raw** data |
+
+Authoritative storage is the central per-category `relationships` collection (not root `associated*`, not a `contentType: "relationships"` sibling).
+
+---
+
+## Also see
+
+- **Format for Dummies** (`01-format-for-dummies`) — Mental model of the record envelope — OT/CT, composition, discriminators, and what is not stored on the record.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._

package/files/use-cases/lists-cursors-pagination.md

@@ -0,0 +1,196 @@
+# List, filter, and paginate correctly
+
+> **Use case id:** `lists-cursors-pagination`
+> **Goal:** Use cursors, pageInfo, and query grammar without inventing offsets.
+> **Audiences:** developers
+> **Tags:** lists, cursor, http
+
+## Reading path
+
+1. **Lists, Queries, Pagination** (developers) → chapters: `3-common-list-response-pageinfo-contract`, `8-cursor-contract`, `9-pagination-guarantees`, `21-client-side-iteration-patterns`
+
+## From: Lists, Queries, Pagination — 3. Common list response / `pageInfo` contract
+
+### 3.1 Concrete content page (HTTP)
+
+```http
+GET /api/data/content?objectType=work-items&contentType=snapshots&limit=50&cursor=<opaque>
+x-memorix-org-id: g12opsabc123
+x-memorix-agent-ids: ops
+```
+
+```json
+{
+  "scope": { "orgId": "g12opsabc123", "agentIds": ["ops"] },
+  "addressing": {
+    "orgId": "g12opsabc123",
+    "objectType": "work-items",
+    "contentType": "snapshots",
+    "id": null
+  },
+  "records": [ /* memorix-record/2 … */ ],
+  "pageInfo": { "nextCursor": "mrxq1...." },
+  "unresolvedRelationships": [],
+  "currentRevision": null
+}
+```
+
+### 3.2 Abstract page (HTTP)
+
+```json
+{
+  "scope": { "orgId": "g12opsabc123", "agentIds": ["ops"] },
+  "abstractType": "People",
+  "records": [
+    {
+      "abstractType": "People",
+      "recordId": "employee-10",
+      "objectType": "employees",
+      "dataCategory": "entity",
+      "contentType": "snapshots",
+      "properties": { }
+    }
+  ],
+  "pageInfo": { "nextCursor": null }
+}
+```
+
+### 3.3 Package-level result
+
+Inside `@x12i/memorix-data`, list results use `items` + optional `nextCursor`. HTTP maps that to `records` + `pageInfo.nextCursor` (null when exhausted). Relationship sections expose `items` + `pageInfo: { limit, nextCursor? }`.
+
+### 3.4 Clarifications
+
+| Situation | Contract |
+|---|---|
+| Empty list | HTTP 200, empty array, `nextCursor` null/omitted |
+| Missing single get | Not-found class error — not an empty list |
+| Totals | **May be absent** even when pagination works — do not require `totalCount` |
+| Relationship pagination | Independent from source-content pagination |
+| Expanded targets | Virtual; never part of stored source |
+| Diagnostics | Related-property and some operational lists may include plan diagnostics |
+
+`PageInfo` (OpenAPI): `nextCursor` (string \| null) and optionally `limit`. Do not assume `hasMore`, query-plan dumps, or correlation ids unless the accepted schema includes them.
+
+---
+
+## From: Lists, Queries, Pagination — 8. Cursor contract
+
+Cursors are opaque tokens. Product encoding uses a versioned prefix (e.g. `mrxq1.…`) wrapping a payload that includes:
+
+- cursor version + kind (`concrete` | `abstract`)  
+- **query shape hash** bound to filters + sort + addressing  
+- continuation position (sort values + internal storage key)  
+
+### 8.1 Public rules
+
+| Rule | Why |
+|---|---|
+| Opaque | Clients never parse or mint cursors |
+| Bound to query shape | Changing filters/sort/addressing → `READ_CURSOR_MISMATCH` |
+| Bound to kind | Concrete cursor ≠ abstract cursor |
+| No credentials | Cursor must not carry secrets or unrestricted payloads |
+| Forward continuation | Resume with the **same** query + `nextCursor` |
+| Exhausted | `nextCursor` null/omitted → done |
+
+### 8.2 Error matrix
+
+| Condition | Code (package) | Client action |
+|---|---|---|
+| Malformed / wrong prefix | `READ_CURSOR_INVALID` | Restart query from the beginning |
+| Unsupported version | `READ_CURSOR_INVALID` | Restart; upgrade client expectations |
+| Shape / kind mismatch | `READ_CURSOR_MISMATCH` | Restart with one consistent query |
+| Cursor from another endpoint | mismatch / invalid | Do not reuse across routes |
+| Referring to removed data | Continue or shorten pages | Treat as concurrent mutation (ch. 20) |
+
+**When the query shape changes, restart.** Do not “fix” a cursor by editing it.
+
+---
+
+## From: Lists, Queries, Pagination — 9. Pagination guarantees
+
+### 9.1 What is guaranteed
+
+- Forward cursor pagination with stable ordering **for a fixed sort + tie-breaker** under quiet data  
+- Page size clamped to an accepted max (content lists: default **50**, max **200**)  
+- Empty final page signal via null/omitted `nextCursor`  
+- Org isolation: another org’s WI-100 never appears in your page  
+
+### 9.2 What is **not** promised
+
+| Question | Honest answer |
+|---|---|
+| Snapshot-isolated pages? | **No** — do not promise snapshot isolation unless a read token exists |
+| Concurrent inserts? | May appear on later pages |
+| Concurrent deletes? | May shorten later pages |
+| Updates to sort fields? | May move an item across boundaries (rare duplicate/gap under change) |
+| Reverse pagination? | Not a public contract |
+| Unlimited cursor TTL? | Do not retain cursors indefinitely; restart if mismatch/expiry occurs |
+| Metadata change mid-abstract-iteration? | Effective union may change with `agentIds[]` / pack updates — restart |
+
+### 9.3 Safe client guidance
+
+**Stable data:** paginate once; process; optionally re-list with a watermark filter (`_system.modifiedAt` / business field) if metadata allows.
+
+**Concurrent data:** design handlers idempotent; prefer keys (`recordId` + `contentType` + discriminator) for side effects; for exports use an operational job (ch. 16) instead of a hours-long cursor walk.
+
+---
+
+## From: Lists, Queries, Pagination — 21. Client-side iteration patterns
+
+### 21.1 Safe page loop (HTTP)
+
+```ts
+async function iterateWorkItemSnapshots(baseUrl: string, headers: HeadersInit) {
+  let cursor: string | null = null;
+  const filter = encodeURIComponent(
+    JSON.stringify([{ field: "data.status", op: "eq", value: "in-progress" }]),
+  );
+  const sort = encodeURIComponent(
+    JSON.stringify([{ field: "recordId", direction: "asc" }]),
+  );
+
+  for (;;) {
+    const qs = new URLSearchParams({
+      objectType: "work-items",
+      contentType: "snapshots",
+      limit: "50",
+    });
+    // Pass filter/sort only when your service build wires them (see OpenAPI).
+    qs.set("filter", filter);
+    qs.set("sort", sort);
+    if (cursor) qs.set("cursor", cursor);
+
+    const res = await fetch(`${baseUrl}/api/data/content?${qs}`, { headers });
+    if (!res.ok) throw new Error(await res.text());
+    const body = await res.json();
+
+    for (const record of body.records ?? []) {
+      await handle(record); // idempotent on recordId+revision
+    }
+
+    cursor = body.pageInfo?.nextCursor ?? null;
+    if (!cursor) break;
+  }
+}
+```
+
+### 21.2 Checklist
+
+- Preserve the **exact** query between pages  
+- On process restart, reuse stored cursor **only** with the same shape; on mismatch, restart  
+- Avoid duplicate side effects (upsert by business key / revision)  
+- Iterate abstract unions with abstract cursors — do not merge concrete pages yourself unless you reimplement the merge contract  
+- Paginate relationships independently from content  
+- Prefer export jobs for huge extracts  
+
+---
+
+## Also see
+
+- **Runtime Data Client** (`02-runtime-data-client`) — Connection, scope, exact reads, writes, compose, abstracts, and client anti-patterns through Memorix Service / OpenAPI.
+- **Data Access Patterns** (`04-data-access-patterns`) — Decision matrix — best / acceptable / forbidden access methods for real client tasks.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._

package/files/use-cases/local-acceptance-g12ops.md

@@ -0,0 +1,160 @@
+# Run local acceptance on g12ops*
+
+> **Use case id:** `local-acceptance-g12ops`
+> **Goal:** Bring up the stack with ops-pack and prove the ops narrative in an isolated org.
+> **Audiences:** developers
+> **Tags:** acceptance, ops, local
+
+## Reading path
+
+1. **Platform Overview** (developers) → chapters: `6-the-ops-mental-path-end-to-end`, `7-local-stack-quick-start`
+1. **Agent Packs & Ops** (developers) → chapters: `3-harness-up-down-reset-accept`, `7-install-into-a-fresh-acceptance-org`
+
+## From: Platform Overview — 6. The ops mental path end to end
+
+1. **Scope** — create/select fresh `g12ops*`, set `agentIds: ["ops"]`.
+2. **Resolve metadata** — install ops pack; load types, views, pipelines for that scope.
+3. **Write / read** — data plane (`/api/data`) for subject content.
+4. **Ingest** — Memory lands exact raw → pipelines promote → relationships link.
+5. **Operate** — verify/rebuild links, inspect runs, install proposed metadata (**human** confirm).
+
+Composition example: request all content about `WI-100` and receive siblings (snapshot, workflow, analyses) without flattening them into one document.
+
+![Composed subject](../screenshots/10-composed-subject.png)
+
+Agent inheritance (Design) selects which metadata definitions contribute to effective resolution:
+
+![Agents inheritance](../screenshots/02-agents-inheritance.png)
+
+---
+
+## From: Platform Overview — 7. Local stack (quick start)
+
+From the monorepo after workspace install:
+
+```bash
+node scripts/dev-stack.mjs
+# opens browser tabs: stack status, /_live views, Studio
+# use --no-open to skip browser tabs
+```
+
+| Surface | Default URL |
+|---|---|
+| Studio | http://127.0.0.1:5101/ |
+| Stack status | http://127.0.0.1:5103/ (hub — not Studio; links to each `/_live`) |
+| Service live view | http://127.0.0.1:5100/_live (`@x12i/api-live-view` via `@x12i/core-service`) |
+| Service OpenAPI | http://127.0.0.1:5100/api/openapi.json |
+| Service health | http://127.0.0.1:5100/health |
+| Ops-source live / health | http://127.0.0.1:5102/_live · `/health` |
+
+Ports: Memorix zone **5100–5119** (even = API, odd = UI) from [`@x12i/ports-manager`](../../ports-manager/). Avoid **5000/7000** (macOS AirPlay). Quarantine **5000–5099**. How-to for new HTTP workers: [Building Services — HTTP process compliance](../../07-services/developers/BOOK.md#http-process-compliance-local-services). Remote data clients call `/api/*` only — do not install ports kits.
+
+```bash
+curl -sS \
+  -H 'x-memorix-org-id: g12ops-demo' \
+  -H 'x-memorix-agent-ids: ops' \
+  http://127.0.0.1:5100/api/openapi.json
+```
+
+What to install in *your* app:
+
+```bash
+# Default remote host: nothing required for HTTP
+npm install @x12i/memorix-format@^3.0.1   # optional offline validate/migrate
+
+# Trusted platform process only:
+npm install @x12i/memorix-data@^3.0.1 @x12i/memorix-metadata-runtime@^3.0.1
+
+# New HTTP services in x12i/exellix (bind a port):
+npm install @x12i/core-service
+```
+
+---
+
+## From: Agent Packs & Ops — 3. Harness: up / down / reset / accept
+
+```bash
+cd memorix-ops-pack
+npm install && npm test
+node scripts/harness.mjs up          # OPS_SOURCE_PORT default 5102
+node scripts/harness.mjs reset       # reset fixture control state
+node scripts/harness.mjs accept      # smoke: metadata files + source routes
+node scripts/harness.mjs down
+```
+
+Prefer the monorepo stack when exercising Studio + service together:
+
+```bash
+node scripts/dev-stack.mjs
+# includes ops-source on :5102 (even ports = APIs), stack-status :5103, /_live on APIs
+```
+
+| Command | Purpose |
+|---|---|
+| `up` | Start localhost `ops-source` (PID under `.run/`) |
+| `down` | Stop ops-source via PID file |
+| `reset` | POST fixture control reset; re-establish known source state |
+| `accept` | Verify manifest files exist; hit key source routes; assert negative paths |
+
+### What `accept` is (and is not)
+
+**Is:** a deterministic smoke that the pack’s metadata files and fixture server behave.
+
+**Is not:** a license to use live customer orgs; not a substitute for service-level gates on a fresh `g12ops*` with explicit metadata install + pipeline green.
+
+Platform product acceptance always pairs pack smoke with:
+
+```http
+x-memorix-org-id: g12ops<suffix>
+x-memorix-agent-ids: ops
+```
+
+and an explicit `/api/metadata/install` (or Studio install) into that org.
+
+### Harness sketch in CI
+
+```bash
+node scripts/harness.mjs up
+node scripts/harness.mjs accept
+npx memorix-format validate-v2 examples/v2
+node scripts/harness.mjs down
+```
+
+---
+
+## From: Agent Packs & Ops — 7. Install into a fresh acceptance org
+
+```bash
+ORG=$(node -e "console.log('g12ops'+Math.random().toString(36).slice(2,10))")
+H=(-H "x-memorix-org-id: $ORG" -H "x-memorix-agent-ids: ops")
+
+curl -s "${H[@]}" -X POST "$SERVICE/api/metadata/install" \
+  -H "content-type: application/json" \
+  -d '{"packId":"ops","confirm":true}'
+
+curl -s "${H[@]}" "$SERVICE/api/metadata/effective" | head
+
+# Memory pull from ops-source (credentialRef per Memory contracts)
+# Pipeline start → assert WI-100 composition + links
+```
+
+Assertions for a full gate:
+
+- [ ] Manifest files present; `harness accept` green
+- [ ] `validate-v2` green on `examples/v2`
+- [ ] Install with `confirm: true` into `g12ops*`
+- [ ] Memory pull lands raw for employees / work-items / …
+- [ ] Pipeline green; Explore/HTTP composition for `wi-100`
+- [ ] Relationship verify for documented definitions
+- [ ] Custom kinds (if any) visible via `/api/metadata/kinds` and **not** executed by pipelines
+- [ ] No live customer org; agents `["ops"]`
+
+---
+
+## Also see
+
+- **Migration & Troubleshooting** (`13-migration-troubleshooting`) — associated* rejection, stale revisions, HTTP failure matrix, escalate packets.
+
+---
+
+_Generated use-case pack for agents and humans. See `agent-manifest.json` for discovery._