@x12i/memorix-metadata 3.6.2 → 3.6.3

package/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 3.6.3
+
+- Updated `docs/memorix-format-reference.md` (copied from `@x12i/memorix-format` at pack time). It now shows how a content type and its `profile` become the `ContentTypeProfileV2` that record validation uses, and how to validate a record against a pack offline. No validation change.
+
 ## 3.6.2
 
 - Ship `docs/memorix-format-reference.md` and `CHANGELOG.md`. Object types, content types, and descriptors were already this package; the reference was not. No validation change.

package/README.md

@@ -9,7 +9,7 @@ Install it on its own. The only runtime dependency is `@x12i/memorix-mapping`. R
 ## Install
 
 ```bash
-npm install @x12i/memorix-metadata@3.6.2
+npm install @x12i/memorix-metadata@3.6.3
 ```
 
 | Audience | Use this package? |
@@ -83,5 +83,6 @@ Abstracts stay **virtual** — no abstract collections and no abstract `recordId
 
 ## Docs
 
-- [`docs/memorix-format-reference.md`](./docs/memorix-format-reference.md) — records, object types, content types, descriptors, and mappings. The same file ships in `@x12i/memorix-format`.
+- [`docs/memorix-format-reference.md`](./docs/memorix-format-reference.md) — records, object types, content types, descriptors, and mappings. Section 5 covers object types, content types, and the descriptor; section 5.7 covers validating a record against a pack.
+- [`examples/`](./examples/) — valid packs, and `examples/invalid/` with one failing rule each.
 - Hub: [https://docs.memorix.x12i.com](https://docs.memorix.x12i.com)

package/docs/memorix-format-reference.md

@@ -6,15 +6,16 @@ This document puts in one place everything that defines how Memorix data is shap
 2. The **metadata pack** (`memorix-metadata/1`): where an agent declares its **object types (OTs)**, their **content types (CTs)**, the CT **descriptor** (`profile`), fields, abstracts, and relationships.
 3. **Mappings** (the mapping AST): how provider payloads land on an OT/CT, and how concrete fields project to abstract properties.
 
-Each section names the source file that is the law for it. When this document and the code disagree, the code wins.
+Each section names the package and export that is the law for it. When this document and the code disagree, the code wins.
 
 ## Using these packages on their own
 
 Memorix depends on these packages. Anyone else can install them without the service, the database layer, or the pipeline runtime.
 
 ```bash
-npm install @x12i/memorix-format
-npm install @x12i/memorix-metadata
+npm install @x12i/memorix-format     # records
+npm install @x12i/memorix-metadata   # object types, content types, descriptors (brings @x12i/memorix-mapping)
+npm install @x12i/memorix-mapping    # mappings only
 ```
 
 | Package | What you get | Runtime dependencies |
@@ -25,7 +26,7 @@ npm install @x12i/memorix-metadata
 
 `@x12i/memorix-metadata` installs `@x12i/memorix-mapping` for you. Validating a record or a pack does not require `@x12i/memorix-data`, `@x12i/memorix-service`, or Mongo. Routing and the write path live in `@x12i/memorix-data`; they are listed below so the error codes make sense, and they are not needed to use the format or the descriptors.
 
-This file ships as `docs/memorix-format-reference.md` in both `@x12i/memorix-format` and `@x12i/memorix-metadata`.
+This file ships as `docs/memorix-format-reference.md` in all three packages. The copy in `@x12i/memorix-format` is the source; the other two are copied from it when they are packed.
 
 ```ts
 import { validateMemorixRecordV2 } from "@x12i/memorix-format/v2";
@@ -36,14 +37,14 @@ import {
 } from "@x12i/memorix-metadata";
 ```
 
-| Layer | Package | Source of truth |
+| Layer | Package | Exports and files |
 |---|---|---|
-| Record envelope | `@x12i/memorix-format` | `memorix-format/docs/memorix-record-v2.md`, `memorix-format/src/v2/` |
-| OT / CT / descriptor / abstracts / relationships / mapping declarations | `@x12i/memorix-metadata` | `memorix-metadata/src/types.ts`, `memorix-metadata/src/validate.ts` |
-| Platform interfaces (Entity / Event / Knowledge) | `@x12i/memorix-metadata` | `memorix-metadata/src/platform-interfaces.ts` |
-| Mapping AST and execution | `@x12i/memorix-mapping` | `memorix-mapping/src/types.ts` |
-| Physical routing (database / collection names) | `@x12i/memorix-data` | `memorix-data/src/routing.ts` |
-| Write-time enforcement of the descriptor | `@x12i/memorix-data` | `memorix-data/src/write/record.ts` |
+| Record envelope | `@x12i/memorix-format` | `MemorixRecordV2`, `validateMemorixRecordV2` (from `@x12i/memorix-format/v2`); `docs/memorix-record-v2.md` |
+| Format-level content-type profile | `@x12i/memorix-format` | `ContentTypeProfileV2`, `BUILTIN_CONTENT_PROFILES`, `resolveContentProfile` |
+| OT / CT / descriptor / abstracts / relationships / mapping declarations | `@x12i/memorix-metadata` | `MetadataPack`, `ObjectTypeDefinition`, `ContentTypeSpec`, `ContentTypeProfileDecl`, `validateMetadataPack` |
+| Platform interfaces (Entity / Event / Knowledge) | `@x12i/memorix-metadata` | `PLATFORM_INTERFACES` |
+| Mapping AST and execution | `@x12i/memorix-mapping` | `MappingDocument`, `MappingExpr`, `executeMapping`, `validateMappingDocument` |
+| Physical routing and write-time checks | `@x12i/memorix-data` (Memorix runtime; not needed to use the format) | `databaseNameFor`, `contentCollectionName`, the content writer |
 
 ---
 
@@ -59,7 +60,7 @@ import {
 | **Abstract type** | A metadata-only type that one or more OTs project into (for example `People`). Never has a collection or its own `recordId`. Views, agent services, and relationships speak abstracts. |
 | **Platform interface** | The sealed abstracts `Entity`, `Event`, `Knowledge`. Every OT implements the one that matches its data category. |
 | **Mapping** | A declarative transform written in the mapping AST. Purpose is `source`, `abstract`, or `completion`. |
-| **Managed descriptor (legacy)** | Older Catalox "entity descriptors" in `memorix-retrieval/catalox-seeds`. Only an inventory of them is kept here (`memorix-format/meta/inventory/managed-descriptors.json`). They are not format law and not agent packs. |
+| **Managed descriptor (legacy)** | Older Catalox "entity descriptors" in `memorix-retrieval/catalox-seeds`. Only an inventory of them is kept, in `meta/inventory/managed-descriptors.json` of `@x12i/memorix-format`. They are not format law and not agent packs. |
 
 ---
 
@@ -99,8 +100,10 @@ type MemorixRecordV2 = {
     };
   };
   data: Record<string, unknown>;
+  subject?: { objectType: string; recordId: string };
   context?: Record<string, unknown>;
   _metadata?: Record<string, unknown>;
+  _markdown?: { content: string; source: "ai" | "json" | "content"; metadata?: Record<string, unknown> };
   _audit?: Record<string, unknown>;
   _history?: Array<Record<string, unknown>>;
   analytics?: Record<string, unknown>;
@@ -112,8 +115,14 @@ type MemorixRecordV2 = {
     createdAt: string;
     modifiedAt: string;
     observedAt?: string;
+    writtenAt?: string;
     lifecycle?: Array<{ process: string; processedAt: string }>;
     provenance?: Record<string, unknown>;
+    _datasource?: {
+      service: string; ingress: string; basis: string; recordedAt: string;
+      origin?: string; channel?: string; originRecordId?: string; originAt?: string;
+      sources?: Array<{ system: string; name: string }>;
+    };
     deletion?: { deletedAt: string; by: string; via: "content" | "patch"; patchId?: string; reason?: string };
   };
 };
@@ -133,8 +142,10 @@ type MemorixRecordV2 = {
 | `concept.identifiers.primary` | yes | `{ kind, value }`, optional `issuer` and `normalizedValue`. |
 | `concept.identifiers.alternates` | no | Omit when empty. |
 | `data` | yes | The subject content itself. JSON markdown is rendered from `data` only. |
+| `subject` | no | `{ objectType, recordId }` of the subject this row is about, when that subject is a different OT. The row keeps its own `recordId`. List filters use `subject.recordId`. |
 | `context` | no | Context payloads that are not the subject content. Omit when empty. Not rendered to markdown. |
 | `_metadata` | no | Producer context (for example `stepId`, `writtenBy`). A key here must not also be in `data`. Not `_system.provenance` and not `_markdown.metadata`. |
+| `_markdown` | no | Original text beside `data`. `source` is `ai`, `json`, or `content`. Writers fill `source: "json"` from `data` when it is omitted. Readers omit it unless the fetch asks for markdown. |
 | `_audit` | no | Audit of the record. When present, `provider`, `model`, and `reasoningEffort` must be non-empty strings. Other keys are open. |
 | `_history` | no | Earlier products of this record, oldest first. Each entry is an object stored as given. An empty array is valid and differs from an absent key. A write that removes fields appends `{ action: "patch", caller, modifiedAt, removed: string[] }`. |
 | `analytics` | no | Numeric rollups. Allowed only when the CT descriptor has `allowSnapshotBuckets: true`. |
@@ -144,6 +155,8 @@ type MemorixRecordV2 = {
 | `_system.state` | yes | `active`, `archived`, `superseded`, or `deleted`. |
 | `_system.createdAt` / `modifiedAt` | yes | ISO timestamps. |
 | `_system.observedAt` | no | Default source for the `Event.occurredAt` interface property. |
+| `_system.writtenAt` | no | Clock time of the write that produced this revision. |
+| `_system._datasource` | no | Memorix-owned document source (`service`, `ingress`, `basis`, `recordedAt`, optional `origin`, `channel`, `sources`). Not part of `data`. |
 | `_system.lifecycle` | no | Appended only for meaningful process events. Omit when empty. Independent of `revision`. |
 | `_system.provenance` | no | Source attribution such as `sourceId` or `endpoint`. |
 | `_system.deletion` | conditional | Present if and only if `state` is `deleted` (`V2-DELETION`). |
@@ -154,8 +167,9 @@ type MemorixRecordV2 = {
 - `analytics`, `enrichment`, and `insights` appear only on CTs whose descriptor sets `allowSnapshotBuckets: true`.
 - If the CT declares a discriminator with a `defaultValue` and the record lacks it, validation warns `V2-DISCRIMINATOR-DEFAULT` and the writer fills the default. If it is required with no default, validation fails with `V2-DISCRIMINATOR`.
 - What a delete does is declared on the CT (`profile.deletion`), see section 5.2.
+- Section 5.7 shows how the metadata descriptor becomes the profile that `validateMemorixRecordV2` checks.
 
-Golden examples live in `memorix-format/examples/v2/`.
+Golden examples live in `examples/v2/` of `@x12i/memorix-format`.
 
 ---
 
@@ -245,7 +259,7 @@ type ContentTypeProfileDecl = {
 | `deletion.mode` | `logical`: the row stays with `_system.state: "deleted"` and reads hide it. `tracked`: the row is removed and a `memorix-deletion/1` entry is written to `deletions`. `untracked`: the row is removed and nothing is recorded. Omitted `deletion` means `{ mode: "tracked", keepBody: false }`. |
 | `deletion.keepBody` | Only valid with `tracked`. Keeps the removed body on the ledger entry. Purges and legacy-instance deletions never keep the body. A CT declares the same policy on every OT that uses it. |
 
-Write-time enforcement: if the effective catalog has no descriptor for the CT, the writer throws `METADATA_DESCRIPTOR_MISSING` (HTTP 400, with a `next` hint pointing at `/api/metadata/ensure`). See `memorix-data/src/write/record.ts`.
+Write-time enforcement: if the effective catalog has no descriptor for the CT, the writer throws `METADATA_DESCRIPTOR_MISSING` (HTTP 400, with a `next` hint pointing at `/api/metadata/ensure`).
 
 ### 5.3 Fields (`ObjectTypeFieldSpec`)
 
@@ -275,7 +289,7 @@ Published by the skill API and run with `POST /api/v1/agent-functions/agents/{ag
 
 ### 5.5 Content-type vocabulary in use
 
-Any routing-safe id may be declared. These are the CTs the platform and current packs use (`memorix-format/meta/content-types.json`):
+Any routing-safe id may be declared. These are the CTs the platform and current packs use (`meta/content-types.json` in `@x12i/memorix-format`):
 
 | CT | Typical descriptor | Purpose |
 |---|---|---|
@@ -293,6 +307,80 @@ Any routing-safe id may be declared. These are the CTs the platform and current
 
 The lander declares OTs and their source CTs. A skill appends only its target CTs to an OT that already exists, and never re-declares source CTs. Because ensure replaces the catalog, a skill always merges into the full pack from `/api/metadata/raw` before it ensures.
 
+### 5.7 From metadata descriptor to format profile
+
+`@x12i/memorix-format` does not read metadata packs. `validateMemorixRecordV2` takes a `ContentTypeProfileV2`, the format-level descriptor:
+
+```ts
+type ContentTypeProfileV2 = {
+  contentType: string;
+  allowSnapshotBuckets: boolean;
+  discriminator?: { path: string; required: boolean; defaultValue?: string; allowedValues?: string[] };
+  allowedExtraRoots?: string[];
+  deletion?: { mode: "logical" | "tracked" | "untracked"; keepBody?: boolean };
+};
+```
+
+Memorix builds it from the metadata content type like this:
+
+| `ContentTypeProfileV2` | Comes from (`ContentTypeSpec`) |
+|---|---|
+| `contentType` | `id` |
+| `allowSnapshotBuckets` | `profile.allowSnapshotBuckets` |
+| `deletion` | `profile.deletion`, parsed with `parseDeletionPolicy`. An invalid declaration reads as absent. |
+| `discriminator.path` | `discriminator.path`. With no path, there is no discriminator. |
+| `discriminator.required` | `discriminator.required === true` (default `false`) |
+| `discriminator.defaultValue` | `discriminator.defaultValue` when it is a non-empty string |
+| `allowedValues`, `allowedExtraRoots` | Not set from metadata. Available to callers that validate offline. |
+
+The lookup is by content-type id across the caller's effective object types, and the first match wins. That is why one CT id must declare the same `allowSnapshotBuckets` and `deletion` on every OT (`META-CT-PROFILE`).
+
+When no profile is passed, `validateMemorixRecordV2` falls back to `resolveContentProfile(contentType)`, which uses `BUILTIN_CONTENT_PROFILES`:
+
+| CT | `allowSnapshotBuckets` | Built-in discriminator |
+|---|---|---|
+| `snapshots`, `knowledge` | `true` | none |
+| `raw`, `workflow`, `context`, `relationships`, `events` | `false` | none |
+| `discovery` | `false` | `data.discoveryType`, default `default` |
+| `analysis` | `false` | `data.analysisType`, default `default` |
+| `decisions` | `false` | `data.decisionType`, default `default` |
+| any other id | `false` | none |
+
+A pack's descriptor always replaces the built-in profile, including its discriminator.
+
+To validate a record offline against a pack, without Memorix:
+
+```ts
+import {
+  validateMemorixRecordV2,
+  parseDeletionPolicy,
+  type ContentTypeProfileV2,
+} from "@x12i/memorix-format/v2";
+import type { ContentTypeSpec } from "@x12i/memorix-metadata";
+
+function toProfile(ct: ContentTypeSpec): ContentTypeProfileV2 {
+  const deletion = ct.profile.deletion ? parseDeletionPolicy(ct.profile.deletion) : undefined;
+  return {
+    contentType: ct.id,
+    allowSnapshotBuckets: ct.profile.allowSnapshotBuckets,
+    ...(deletion?.ok ? { deletion: deletion.policy } : {}),
+    ...(ct.discriminator?.path
+      ? {
+          discriminator: {
+            path: ct.discriminator.path,
+            required: ct.discriminator.required === true,
+            ...(ct.discriminator.defaultValue ? { defaultValue: ct.discriminator.defaultValue } : {}),
+          },
+        }
+      : {}),
+  };
+}
+
+const ct = pack.objectTypes.find((o) => o.id === record.objectType)!
+  .contentTypes.find((c) => c.id === record.contentType)!;
+const result = validateMemorixRecordV2(record, { profile: toProfile(ct), enforceDiscriminator: true });
+```
+
 ---
 
 ## 6. Abstract types and platform interfaces
@@ -414,7 +502,7 @@ Run with `executeMapping`, `simulateMapping` (adds coverage), and `executeRevers
 
 ## 9. Worked example
 
-A pack fragment with one OT, two CTs, an abstract with a direct binding, a source mapping, and an abstract mapping. A complete real pack is in `memorix-metadata/examples/opx/`.
+A pack fragment with one OT, two CTs, an abstract with a direct binding, a source mapping, and an abstract mapping. A complete real pack is `examples/opx/normalized/pack.json` in `@x12i/memorix-metadata`.
 
 ```json
 {
@@ -566,12 +654,20 @@ A stored record for that OT/CT, in collection `work-items-snapshots` of database
 
 ## Related documents
 
-These ship inside `@x12i/memorix-format`:
+In `@x12i/memorix-format`:
 
 - `docs/memorix-record-v2.md`: the full envelope contract and its locked decisions.
-- `docs/migrate-present-to-v2.md`: migrating the older present envelope.
+- `docs/migrate-present-to-v2.md`: migrating the older present envelope to `memorix-record/2`.
 - `docs/memorix-workflow-format.md`: the `workflow` content type.
-- `docs/association-format.md`, `docs/memorix-format.md`, `docs/memorix-format-requirements.md`.
-- `meta/`: envelope, open-bucket, and content-type vocabulary.
+- `docs/memorix-format.md`, `docs/memorix-format-requirements.md`, `docs/association-format.md`: the older present envelope. Read them only to migrate existing data.
+- `meta/`: envelope, open-bucket, and content-type vocabulary as JSON.
+- `examples/v2/`: valid `memorix-record/2` documents.
+
+In `@x12i/memorix-metadata`:
+
+- `examples/opx/`, `examples/knowx/`, `examples/analytics/`, `examples/custom-kinds/`: valid packs.
+- `examples/invalid/`: packs that fail validation, one rule each.
+
+In `@x12i/memorix-mapping`:
 
-`@x12i/memorix-metadata` ships this reference, its `README.md`, and `examples/`. `@x12i/memorix-mapping` ships its API in `README.md`.
+- `examples/`: source, abstract, and completion mappings.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@x12i/memorix-metadata",
-  "version": "3.6.2",
+  "version": "3.6.3",
   "description": "Memorix object-type and content-type descriptors, metadata pack contracts, and validation. Depends only on @x12i/memorix-mapping.",
   "type": "module",
   "main": "./dist/index.js",
@@ -23,7 +23,8 @@
     "build": "tsc -p tsconfig.build.json",
     "test": "vitest run",
     "test:watch": "vitest",
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit",
+    "prepack": "node ../scripts/sync-format-reference.mjs && npm run build"
   },
   "engines": {
     "node": ">=18.0.0"