@ai-matrx/content-ir 0.18.6 → 0.19.1

package/CHANGELOG.md

@@ -1,12 +1,34 @@
 # Changelog
 
-## 0.18.6
+## 0.19.1
 
 Automatic changed-only republish (docs/metadata drift since the last tag — see
-`git diff npm/content-ir-core/v0.18.5..npm/content-ir-core/v0.18.6 -- apps/shared/content-ir-core`).
+`git diff npm/content-ir-core/v0.19.0..npm/content-ir-core/v0.19.1 -- apps/shared/content-ir-core`).
 No source changes intended and no consumer action required.
 
-## 0.19.0
+## 0.19.0 — kind validation behind a schema-source port (Matrx Alchemy ALC-13)
+
+`/registry` (and the root) now export `createKindValidator(source, options?)`,
+`SchemaSourcePort`, `SchemaUnavailableReason`, `KindValidationResult`,
+`KindValidator` and `Json`. This is matrx-frontend's `validateAgainstKind`,
+moved into the package with its database read inverted: the host implements
+`SchemaSourcePort.kindSchema(kind, signal)` once and answers `{ schema,
+version }` or `{ unavailable: "kind_not_registered" | "schema_unavailable" |
+"schema_invalid" | "catalog_unreachable" }`. The verdict (`checked`, `ok`,
+`errors`, `degradedReason`) and its messages are unchanged; a source that
+throws reads as `catalog_unreachable`; an uncompilable schema reads as
+`schema_invalid`. The schema cache (30 s default, `invalidate`,
+`cachedSchema`) belongs to each validator instance, never to the module.
+
+### Consumer action
+
+matrx-frontend: implement `SchemaSourcePort` once over
+`content_ir.kind_definition`, build one validator with `createKindValidator`,
+repoint `features/surfaces/runtime/surface-writeback.ts` and
+`build-tool-injection.ts` to it, and delete
+`features/content-ir/registry/validate-against-kind.ts`.
+
+### Also in 0.19.0 — the RC-B3 fix round (RC-B3 lane)
 
 **Carries: RC-B3 fix round (verify-RC-B3 § 0.18.3) and RC-B3r R3 — builder:
 the RC-B3 standard lane.** (aidream 1174edb640 — RC-B10, the print lane —
@@ -16,7 +38,7 @@ below — shipped in 0.18.4; both under generic "no source changes" notes.)
 **`/source` streaming: settled means settled** (F3/F4). Found with the
 verifier's generator (realistic model fragments, every prefix, with and
 without a final newline), now permanent in `__tests__/source-attack.test.ts`
-(`LLM-fragment documents…`; 12,000 documents in `SOURCE_ATTACK=long`) plus 27
+(`LLM-fragment documents…`; 12,000 documents in `SOURCE_ATTACK=long`) plus 30
 pinned reproducers. `tokenizeSource(prefix, { streaming: true })` now also
 holds back, as prose or as an open island:
 - lines a box-drawing tree still forming at the end may claim as its root
@@ -24,12 +46,14 @@ holds back, as prose or as an open island:
 - a paragraph cut short by a block that may still dissolve (a `$$` pair
   closing on the open line, an unbalanced JSON head, any opener still
   arriving on the open line) — and a footnote before one;
-- everything inline on the open last line, and a block island ending there;
+- a block island ending on the open last line (inlines there are covered by
+  the tree-root hold);
 - a nested-fence closer found only by the strict-CommonMark retry, a tilde
   fence whose inner block may still grow past it, braces that balance on
   the open last line;
 - a code span that a blank-so-far or `>`-only open line may still close
-  (`pendingCodeSpanStart`, `source/code-ranges.ts`).
+  (`pendingCodeSpanStart`, `source/code-ranges.ts`);
+- a block whose opener is provisional text, and a paragraph break there.
 The two lone-HTML-tag holdbacks (M-N6/N7) collapsed into the open-tail block
 rule. Each holdback turns the attack suite red on its own when removed
 (planted mutations on a scratch copy). The H2 exception is unchanged.
@@ -53,6 +77,12 @@ guard case `echo $HOME $PATH`).
   `singleDollarAfter(text, close)` instead of `text[close + 1]` (a one-character
   `after` still works; it just cannot see a variable chain).
 
+## 0.18.6
+
+Automatic republish. Despite the generated note, this version DID carry source
+changes: most of the 0.19.0 work above (through aidream cd35751bea). 0.19.0
+completes it (aidream d6241c5496, f1e9156e82).
+
 ## 0.18.5
 
 Automatic republish. Despite the generated note, this version DID carry source
@@ -952,3 +982,4 @@ unchanged. Two things are worth doing at your next working session:
 - Added ESM and declaration builds, package linting, packed-artifact analysis, and an isolated consumer canary.
 - Preserved all parser, session, normalization, schema conversion, and structural-validation behavior with the existing shared fixtures and fingerprint vectors.
 
+

package/README.md

@@ -15,7 +15,7 @@ The root export is the full surface and always will be. Since 0.10.0 each intern
 | `@ai-matrx/content-ir` | everything below |
 | `@ai-matrx/content-ir/core` | IR types + tree, JSON tokenizer, kind parser, snapshots, normalize, fingerprint, discriminator, envelope read/value/cache, region, schema structure |
 | `@ai-matrx/content-ir/session` | `ParseSession` and the one-writer-per-identity registry |
-| `@ai-matrx/content-ir/registry` | kind registry types, storage transform, the dual gate |
+| `@ai-matrx/content-ir/registry` | kind registry types, storage transform, the dual gate, `createKindValidator` over a host `SchemaSourcePort` |
 | `@ai-matrx/content-ir/convert` | kind ⇄ JSON Schema, the OpenAI structured-output converter |
 | `@ai-matrx/content-ir/wire` | partial-kind events, runtime wrapper readers, emit payload |
 | `@ai-matrx/content-ir/directives` | Kind Directives: slug grammar, the ONE decoder |

package/dist/index.cjs

@@ -2968,6 +2968,93 @@ function describeDualGateFailure(kind, result) {
   return `kind "${kind}" failed the dual gate \u2014 ${parts.join(" | ")}`;
 }
 
+// registry/kind-validator.ts
+var SCHEMA_COMPILE_FAILURE_PREFIX = "emitted_json_schema failed to compile:";
+var DEFAULT_CACHE_TTL_MS = 3e4;
+function unavailableMessage(kind, reason, detail) {
+  switch (reason) {
+    case "kind_not_registered":
+      return `kind "${kind}" is not registered in content_ir.kind_definition`;
+    case "schema_unavailable":
+      return `kind "${kind}" has no emitted_json_schema \u2014 nothing to validate against`;
+    case "schema_invalid":
+      return `kind "${kind}" has an uncompilable emitted_json_schema${detail ? `: ${detail}` : ""}`;
+    case "catalog_unreachable":
+      return `could not read the contract for kind "${kind}"${detail ? `: ${detail}` : ""}`;
+  }
+}
+function degraded(kind, reason, detail) {
+  return {
+    kind,
+    checked: false,
+    ok: false,
+    errors: [unavailableMessage(kind, reason, detail)],
+    degradedReason: reason
+  };
+}
+function createKindValidator(source, options = {}) {
+  const ttl = options.cacheTtlMs ?? DEFAULT_CACHE_TTL_MS;
+  const now = options.now ?? Date.now;
+  const cache = /* @__PURE__ */ new Map();
+  async function fetchSchema(kind, signal) {
+    const cached = cache.get(kind);
+    if (cached && now() - cached.fetchedAt <= ttl) {
+      return { schema: cached.schema, version: cached.version };
+    }
+    let answer;
+    try {
+      answer = await source.kindSchema(kind, signal);
+    } catch (err) {
+      return {
+        unavailable: "catalog_unreachable",
+        detail: err instanceof Error ? err.message : String(err)
+      };
+    }
+    if ("unavailable" in answer) return { unavailable: answer.unavailable };
+    if (answer.schema == null) {
+      return { unavailable: "schema_unavailable" };
+    }
+    cache.set(kind, {
+      schema: answer.schema,
+      version: answer.version,
+      fetchedAt: now()
+    });
+    return answer;
+  }
+  return {
+    async validate(value, kind, signal = new AbortController().signal) {
+      if (!kind || typeof kind !== "string") {
+        throw new TypeError(
+          `validateAgainstKind: "kind" must be a non-empty slug (received ${typeof kind})`
+        );
+      }
+      const fetched = await fetchSchema(kind, signal);
+      if ("unavailable" in fetched) {
+        return degraded(kind, fetched.unavailable, fetched.detail);
+      }
+      const leg = validateStructuralLeg(value, fetched.schema);
+      if (!leg.ok && leg.detail?.startsWith(SCHEMA_COMPILE_FAILURE_PREFIX)) {
+        return degraded(kind, "schema_invalid", leg.detail);
+      }
+      return {
+        kind,
+        checked: true,
+        ok: leg.ok,
+        errors: leg.ok ? [] : [leg.detail ?? "validation failed"],
+        degradedReason: null
+      };
+    },
+    async cachedSchema(kind, signal = new AbortController().signal) {
+      const fetched = await fetchSchema(kind, signal);
+      return "unavailable" in fetched ? null : fetched.schema;
+    },
+    invalidate(kind) {
+      if (kind) cache.delete(kind);
+      else cache.clear();
+    }
+  };
+}
+
 // convert/openai-schema-converter.ts
 function requiredNullableFlags(required, nullable) {
   return {
@@ -5243,12 +5330,6 @@ function fenceOpenerOf(content) {
   return null;
 }
 var FenceReader = class {
-  constructor(opener, options = {}) {
-    this.opener = opener;
-    this.isJson = opener.char === "`" && opener.lang === "json";
-    this.nesting = opener.char === "`" && (options.nesting ?? fenceNestsInnerFences(opener.lang));
-    this.tildeCloser = opener.char === "~" ? new RegExp(`^ {0,3}~{${opener.ticks},}[ \\t]*$`) : null;
-  }
   opener;
   state = { inString: false, escaped: false };
   depth = 0;
@@ -5257,6 +5338,12 @@ var FenceReader = class {
   isJson;
   nesting;
   tildeCloser;
+  constructor(opener, options = {}) {
+    this.opener = opener;
+    this.isJson = opener.char === "`" && opener.lang === "json";
+    this.nesting = opener.char === "`" && (options.nesting ?? fenceNestsInnerFences(opener.lang));
+    this.tildeCloser = opener.char === "~" ? new RegExp(`^ {0,3}~{${opener.ticks},}[ \\t]*$`) : null;
+  }
   /** How many nested fences are open right now. */
   get nestedDepth() {
     return this.depth;
@@ -5561,7 +5648,7 @@ function pendingCodeSpanStart(text) {
   const n = text.length;
   if (n === 0 || /\n[ \t]*\n[ \t]*$/.test(text)) return -1;
   const lastLine = text.slice(text.lastIndexOf("\n") + 1);
-  if (lastLine && /^(?: {0,3}>[ \t]?)*[ \t]*$/.test(lastLine)) return pendingCodeSpanStart(`${text}x`);
+  if (lastLine && /^(?: {0,3}>[ \t]?)*[ \t\r]*$/.test(lastLine)) return pendingCodeSpanStart(`${text}x`);
   const s = scan(text);
   const para = s.pendingPara;
   if (s.endsInFence || para === -1) return -1;
@@ -5725,7 +5812,7 @@ function isVariableChain(content, after) {
   const run = /^[A-Za-z0-9_]+/.exec(after)?.[0];
   if (!run) return false;
   if (after.length === 1) return false;
-  if (/^[A-Za-z_][A-Za-z0-9_]*[.@-]$/.test(content)) return true;
+  if (/^[A-Za-z_][A-Za-z0-9_]*(?:[.@-]|->|::)$/.test(content)) return true;
   return IDENTIFIER.test(content) && !MATH_SUFFIX.test(run);
 }
 var FUNCTION_NAMES = /* @__PURE__ */ new Set([
@@ -5874,9 +5961,6 @@ function isEscapedBacktick(line, offset) {
   return slashes % 2 === 1;
 }
 var XmlContainerTracker = class {
-  constructor(rootTag) {
-    this.rootTag = rootTag;
-  }
   rootTag;
   depth = 0;
   inComment = false;
@@ -5884,6 +5968,9 @@ var XmlContainerTracker = class {
   inlineTicks = 0;
   /** The open fence, read by THE one closer (source/code-ranges.ts). */
   fence = null;
+  constructor(rootTag) {
+    this.rootTag = rootTag;
+  }
   consumeLine(line, startOffset = 0) {
     if (this.fence) {
       if (this.fence.feed(line)) this.fence = null;
@@ -6095,6 +6182,23 @@ function isSplitterTreeLine(line) {
 }
 var MEDIA_REF_PREFIXES = ["[Image URL:", "[Video URL:", "[Audio URL:"];
 var Tokenizer = class {
+  text;
+  streaming;
+  meter;
+  n;
+  lineStarts = [];
+  contentEnds = [];
+  lineEnds = [];
+  blocks = [];
+  blockCache = /* @__PURE__ */ new Map();
+  /** Lazily built: for line i, first line j >= i where brace net from i drops to <= 0. */
+  braceStop = null;
+  braceNet = null;
+  /** Every occurrence offset of each needle searched so far (overlapping, like indexOf). */
+  occurrences = /* @__PURE__ */ new Map();
+  /** `$$` opener → closer for the pairs the core renders as math. */
+  mathPairs;
+  finder = (needle, from) => this.next(needle, from);
   constructor(text, streaming = false, meter = void 0) {
     this.text = text;
     this.streaming = streaming;
@@ -6113,7 +6217,7 @@ var Tokenizer = class {
           if (close >= span) this.provisionalFrom = Math.min(this.provisionalFrom, open);
         }
       }
-      for (const pad of ["`", "``", "```", "~~~"]) {
+      for (const pad of ["`", "``", "```", "~~~", "$"]) {
         const alt = pairDisplayMath(text + pad);
         for (const [open, close] of this.mathPairs) {
           if (alt.get(open) !== close) this.provisionalFrom = Math.min(this.provisionalFrom, open);
@@ -6141,40 +6245,10 @@ var Tokenizer = class {
       this.lineEnds.push(text.length);
     }
   }
-  text;
-  streaming;
-  meter;
-  n;
-  lineStarts = [];
-  contentEnds = [];
-  lineEnds = [];
-  blocks = [];
-  blockCache = /* @__PURE__ */ new Map();
-  /** Lazily built: for line i, first line j >= i where brace net from i drops to <= 0. */
-  braceStop = null;
-  braceNet = null;
-  /** Every occurrence offset of each needle searched so far (overlapping, like indexOf). */
-  occurrences = /* @__PURE__ */ new Map();
-  /** `$$` opener → closer for the pairs the core renders as math. */
-  mathPairs;
-  finder = (needle, from) => this.next(needle, from);
   /** Streaming: where provisional text starts (a pending `$$`), or +∞. */
   provisionalFrom = Number.POSITIVE_INFINITY;
   /** Streaming: the text ends in a lone CR whose LF has not arrived yet. */
   pendingCR = false;
-  /**
-   * Streaming: an open last line that begins with `{` is undecided — each new
-   * byte can balance or unbalance its braces and flip the whole line between
-   * prose and a JSON island — so it is provisional until it ends (G5).
-   * (A lone HTML tag on that line is held by {@link holdUndecidedTail}.)
-   */
-  markOpenJsonTail() {
-    const last = this.lineStarts.length - 1;
-    if (!this.isOpenTail(last)) return;
-    if (this.lineText(last).trim().startsWith("{")) {
-      this.provisionalFrom = Math.min(this.provisionalFrom, this.lineStarts[last]);
-    }
-  }
   /**
    * `text.indexOf(needle, from)` in O(log k): the first query for a needle
    * indexes all of its occurrences once, so repeated searches for a closer
@@ -6209,7 +6283,7 @@ var Tokenizer = class {
     const rest = this.n - pe;
     if (rest === 0) return true;
     const tail = this.text.slice(pe);
-    return /^(?:\r\n|\r|\n)[ \t]*$/.test(tail);
+    return /^(?:\r\n|\r|\n)[ \t\r]*$/.test(tail);
   }
   /**
    * Streaming: an open block that may still turn back into prose — a `$$`
@@ -6218,7 +6292,9 @@ var Tokenizer = class {
    * or any opener that is itself still arriving on the open last line.
    */
   mayDissolve(block) {
-    if (!this.streaming || block.complete) return false;
+    if (!this.streaming) return false;
+    if (block.start >= this.provisionalFrom) return true;
+    if (block.complete) return false;
     if (block.type === "math_block" || block.type === "json") return true;
     return this.isOpenTail(this.lineOf(block.start));
   }
@@ -6257,7 +6333,6 @@ var Tokenizer = class {
     return this.text.slice(this.lineStarts[li], this.contentEnds[li]);
   }
   run() {
-    if (this.streaming) this.markOpenJsonTail();
     let pos = 0;
     while (pos < this.n) {
       const li = this.lineOf(pos);
@@ -6304,12 +6379,6 @@ var Tokenizer = class {
     const last = this.lineStarts.length - 1;
     if (last < 0) return;
     if (this.isOpenTail(last)) {
-      const lineStart = this.lineStarts[last];
-      for (const block2 of this.blocks) {
-        if (block2.kind === "prose" && block2.end > lineStart) {
-          block2.inlines = block2.inlines.filter((inline) => inline.end <= lineStart);
-        }
-      }
       let b = this.blocks.length - 1;
       while (b >= 0 && this.blocks[b].kind === "gap") b--;
       const block = this.blocks[b];
@@ -6501,7 +6570,7 @@ var Tokenizer = class {
     return { type, start, end, complete, meta };
   }
   frontMatter(li, t) {
-    const fence = this.lineText(li);
+    const fence = this.lineText(li).replace(/^\uFEFF/, "");
     if (fence !== "---" && fence !== "+++" || t !== fence) return null;
     const toml = fence === "+++";
     let sawContent = false;
@@ -6512,9 +6581,13 @@ var Tokenizer = class {
       }
       if (!NOT_WHITESPACE.test(line)) continue;
       const shaped = toml ? /^\s*(?:[A-Za-z0-9_."-]+\s*=|\[|#)/.test(line) : /^(?:[A-Za-z0-9_"'-][^:]*:(?:\s|$)|\s+\S|-\s|#)/.test(line);
-      if (!shaped) return null;
+      if (!shaped) {
+        if (this.isOpenTail(k)) break;
+        return null;
+      }
       sawContent = true;
     }
+    if (this.streaming) return this.island("front_matter", 0, this.n, false, { format: toml ? "toml" : "yaml" });
     return null;
   }
   /**
@@ -6915,7 +6988,9 @@ var Tokenizer = class {
       const block = this.detectBlock(this.lineStarts[k], k, false);
       if (block) {
         breakLine = k;
-        if (this.mayDissolve(block)) this.undecidedBreaks.add(end);
+        if (this.mayDissolve(block)) {
+          this.undecidedBreaks.add(end);
+        }
         break;
       }
       end = this.contentEnds[k];
@@ -7260,12 +7335,12 @@ function blockAt(blocks, offset) {
 
 // source/splice.ts
 var SourceSpliceError = class extends Error {
+  code;
   constructor(code, message) {
     super(message);
     this.code = code;
     this.name = "SourceSpliceError";
   }
-  code;
 };
 function blockEdit(block, text) {
   return { start: block.start, end: block.end, text };
@@ -7597,6 +7672,7 @@ exports.compareWithExistingKindSchema = compareWithExistingKindSchema;
 exports.convertAiSchemaToBlockFields = convertAiSchemaToBlockFields;
 exports.createFingerprinter = createFingerprinter;
 exports.createKindStreamParser = createKindStreamParser;
+exports.createKindValidator = createKindValidator;
 exports.decodeDirective = decodeDirective;
 exports.describeDualGateFailure = describeDualGateFailure;
 exports.directiveDisplay = directiveDisplay;