@ai-matrx/content-ir 0.3.0 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,77 @@
 # Changelog
 
+## 0.6.0 — 2026-08-29
+
+- **`$ref` / `$defs` resolution — a kind's child kinds are no longer invisible
+  to JSON Schema → KindSchema.** `convertAiSchemaToBlockFields` resolves `#`,
+  `#/$defs/<name>` and `#/definitions/<name>` in field position, in array
+  `items`, and inside `anyOf`/`oneOf` (including `items: {anyOf: [$ref, …]}`,
+  the multi-kind array). Referenced defs are registered as their own drafts,
+  with a cycle brake so a self-referencing kind terminates. An unresolvable
+  ref is a loud error, never a guess.
+
+  WHY: `kindSchemaToJsonSchema` externalizes every child kind as a `$def`, and
+  the reverse direction rejected `$ref` outright — so every
+  `{type:"object", kind}` and `{type:"array", itemKinds}` field came back
+  EMPTY. Measured over the 62 live kinds carrying both a stored field list and
+  a schema, 36 lost fields this way (sections, slides, nodes, edges, criteria,
+  questions, segments, …).
+
+- **A `__kind`-carrying record reads back as a record.** An object whose only
+  declared property is `__kind`, with typed `additionalProperties`, is the
+  forward converter's emission for `{type:"record", values}` — it was widening
+  to an open `inline_object` and losing the value type.
+
+- Live-corpus result: schema-derived KindSchemas now match the stored field
+  list byte-for-byte for 48 of 62 kinds, with 1 strictly richer. The remaining
+  13 are DIVERGENCE BETWEEN THE TWO STORED COPIES, not conversion gaps — 4
+  stored lists carry `__kind` as a data field (it is the discriminator, never
+  a field), and 9 disagree on enums, bounds, required flags or descriptions
+  the schema states and the copy does not. That is the evidence for retiring
+  the stored copy.
+
+## 0.5.0 — 2026-08-29
+
+- **A multi-type union is no longer silently narrowed to its first member.**
+  `resolvePrimaryType` now reports every non-null member, and
+  `convertAiSchemaToBlockFields` branches on them: a union spanning
+  objects/arrays becomes `json`, a scalar-only union becomes
+  `{type:"union", scalars}`, and anything unnameable stays `json` with a
+  warning. Single-typed fields are untouched.
+
+  WHY: `type: ["string","number","boolean","object","array","null"]` —
+  pydantic's `Any`, and the commonest construct in user-authored kinds —
+  converted to plain `string`. The loss was invisible and it inverted the
+  field's meaning: an `estimated_count` of `1` then FAILED validation, so a
+  correct payload read as a broken instance. Widening is safe; silently
+  narrowing is not.
+
+- New `__tests__/json-schema-to-kind.test.ts` pins JSON Schema → KindSchema
+  over the constructs a stored field list historically could not hold —
+  pydantic-Any unions, `items: {}` arrays, arrays of `__kind` children,
+  nested `__kind` objects, and markerless inline objects. All convert with
+  zero errors, which is the evidence that `emitted_json_schema` is sufficient
+  on its own and a second stored copy of a kind's fields is unnecessary.
+
+## 0.4.0 — 2026-08-29
+
+- **`unverified` — "we never checked it" is no longer "we checked it and it
+  failed."** `IrKindState` gains `unverified`, and `raw_object` events carry a
+  `cause` (`"unverified" | "invalid"`, absent reads as `"invalid"`). The four
+  no-schema-registered degrades — the two pending-schema resolutions, plus the
+  typed and speculated finalizers — now report `unverified`; every validation
+  failure, array-item mismatch, duplicate key, bad placement, and contradicted
+  speculation still reports `invalid`. `IrTree` carries the cause onto
+  `root.kindState` and into `nodeIndex`.
+
+  WHY THIS EXISTS: `raw` meant both things, and on 2026-08-28 the render route
+  began reading `raw` as "broken instance" and diverting it away from the
+  kind's component. Every kind whose schema cannot be reconstructed — nested
+  objects, arrays of child kinds, loose fields — degrades on the no-schema
+  path, so ~221 live kinds with purpose-built components silently started
+  rendering as key/value dumps. Consumers MUST NOT treat `unverified` as a
+  failure: the value is intact and was never examined.
+
 ## 0.3.0 — 2026-08-29
 
 - **KIND PRESERVATION now covers STRUCTURAL raws.** A node that degrades to

package/dist/index.cjs

@@ -111,11 +111,20 @@ var IrTree = class {
    */
   earlyFields = /* @__PURE__ */ new Map();
   /**
-   * pathKey → identified kind preserved through a SCHEMA-AVAILABILITY raw
-   * fallback (parser stamped `kind` on the raw_object event). Structural raws
-   * (missing __kind, duplicate key, validation failure) never land here.
+   * pathKey → identified kind preserved through a raw fallback (parser stamped
+   * `kind` on the raw_object event) — schema-availability degrades and, since
+   * 2026-08-29, structural ones too. Only a node nothing ever identified (no
+   * `__kind` at all) is absent here.
    */
   rawKinds = /* @__PURE__ */ new Map();
+  /**
+   * pathKey → WHY the node degraded. `"unverified"` means no schema was
+   * available and nothing was ever checked; `"invalid"` means a check ran and
+   * failed. THE RENDER ROUTE BRANCHES ON THIS — see `IrKindState`. Absent =
+   * `"invalid"`, so a parser that predates the `cause` field (or any consumer
+   * hand-building events) keeps the strict, safe reading.
+   */
+  rawCauses = /* @__PURE__ */ new Map();
   regionStatus = "streaming";
   errorReason = null;
   rootRawValue = null;
@@ -168,6 +177,7 @@ var IrTree = class {
         this.pendingSchemaPaths.delete(pathKey);
         this.earlyFields.delete(pathKey);
         if (event.kind) this.rawKinds.set(pathKey, event.kind);
+        this.rawCauses.set(pathKey, event.cause ?? "invalid");
         this.markRaw(event.path, event.reason, event.value);
         return;
       }
@@ -366,10 +376,11 @@ var IrTree = class {
     const isRaw = rootRawReason !== null;
     const identifiedKind = this.identifiedKinds.get("") ?? "";
     const rootKind = isRaw ? this.rawKinds.get("") ?? "" : rootNode?.kind ?? (this.completedKind || identifiedKind);
+    const rootRawState = this.rawCauses.get("") === "unverified" ? "unverified" : "raw";
     const root = {
       role: "structured",
       kind: rootKind,
-      kindState: isRaw ? "raw" : rootNode ? rootNode.kindState : this.pendingSchemaPaths.has("") ? "pending_schema" : this.regionStatus === "streaming" ? identifiedKind ? "pending_schema" : "pending_kind" : "raw",
+      kindState: isRaw ? rootRawState : rootNode ? rootNode.kindState : this.pendingSchemaPaths.has("") ? "pending_schema" : this.regionStatus === "streaming" ? identifiedKind ? "pending_schema" : "pending_kind" : "raw",
       discriminator: JSON_DISCRIMINATOR,
       path: [],
       status: this.regionStatus,
@@ -392,7 +403,7 @@ var IrTree = class {
       if (pathKey === "") continue;
       nodeIndex[pathKey] = {
         kind: this.rawKinds.get(pathKey) ?? "",
-        kindState: "raw",
+        kindState: this.rawCauses.get(pathKey) === "unverified" ? "unverified" : "raw",
         status: "complete"
       };
     }
@@ -732,7 +743,8 @@ var KindStreamParser = class {
           safeCopy(value),
           `No block schema registered for "${kind}".`,
           at,
-          kind
+          kind,
+          "unverified"
         );
       }
     }
@@ -766,7 +778,8 @@ var KindStreamParser = class {
           safeCopy(value ?? {}),
           `No block schema registered for "${kind}".`,
           at,
-          kind
+          kind,
+          "unverified"
         );
         this.closedPendingPaths.delete(pathKey);
         continue;
@@ -1312,7 +1325,8 @@ var KindStreamParser = class {
         objectValue,
         `No block schema registered for "${kind}".`,
         at,
-        kind
+        kind,
+        "unverified"
       );
       return;
     }
@@ -1346,7 +1360,8 @@ var KindStreamParser = class {
         objectValue,
         `No block schema registered for "${kind}".`,
         at,
-        kind
+        kind,
+        "unverified"
       );
       return;
     }
@@ -1419,7 +1434,15 @@ var KindStreamParser = class {
     this.speculativeKinds.delete(pathKey);
     this.emitRawObject(path, safeCopy(liveValue), reason, at, identifiedKind);
   }
-  emitRawObject(path, value, reason, at, identifiedKind) {
+  /**
+   * Degrade ONE node off the resolved path.
+   *
+   * `cause` defaults to `"invalid"` deliberately: every call site that omits
+   * it is a real failure (validation, duplicate key, placement, contradicted
+   * speculation). ONLY the "no schema registered" sites pass `"unverified"`,
+   * and they are the reason the parameter exists — see `IrKindState`.
+   */
+  emitRawObject(path, value, reason, at, identifiedKind, cause = "invalid") {
     const pathKey = this.pathKey(path);
     if (this.rawObjectPaths.has(pathKey)) return;
     this.rawObjectPaths.add(pathKey);
@@ -1430,6 +1453,7 @@ var KindStreamParser = class {
       value,
       reason,
       ...identifiedKind !== void 0 && { kind: identifiedKind },
+      cause,
       at
     });
   }
@@ -2999,21 +3023,26 @@ function fieldSchemaSummary(field) {
 function resolvePrimaryType(node) {
   const raw = node.type;
   if (typeof raw === "string") {
-    return { type: raw === "integer" ? "number" : raw, nullable: false };
+    const t = raw === "integer" ? "number" : raw;
+    return { type: t, nullable: false, members: [t] };
   }
   if (Array.isArray(raw)) {
     const types = raw.filter((t) => typeof t === "string");
     const nullable = types.includes("null");
-    const primary = types.find((t) => t !== "null") ?? (nullable && types.length === 1 ? "null" : null);
-    if (primary === "integer") {
-      return { type: "number", nullable };
-    }
-    return { type: primary ?? null, nullable };
-  }
-  if (node.enum) return { type: "string", nullable: false };
-  if (node.properties) return { type: "object", nullable: false };
-  if (node.items) return { type: "array", nullable: false };
-  return { type: null, nullable: false };
+    const members = [
+      ...new Set(
+        types.filter((t) => t !== "null").map((t) => t === "integer" ? "number" : t)
+      )
+    ];
+    const primary = members[0] ?? (nullable && types.length === 1 ? "null" : null);
+    return { type: primary ?? null, nullable, members };
+  }
+  if (node.enum) return { type: "string", nullable: false, members: ["string"] };
+  if (node.properties)
+    return { type: "object", nullable: false, members: ["object"] };
+  if (node.items)
+    return { type: "array", nullable: false, members: ["array"] };
+  return { type: null, nullable: false, members: [] };
 }
 function carriedMetadataKeys(field) {
   if (field === null) return /* @__PURE__ */ new Set();
@@ -3148,6 +3177,29 @@ function buildAgentSchemaWithRenderBlockSupport(input, rootKindSlug, arrayBindin
   }
   return updatedRoot;
 }
+function resolveRef(ref, ctx) {
+  if (ref === "#" || ref === "#/") {
+    return { slug: ctx.schemaName, node: ctx.rootSchema };
+  }
+  const match = /^#\/(?:\$defs|definitions)\/(.+)$/.exec(ref);
+  if (!match) return null;
+  const name = decodeURIComponent(
+    (match[1] ?? "").replace(/~1/g, "/").replace(/~0/g, "~")
+  );
+  const node = ctx.defs[name];
+  if (!isRecord4(node)) return null;
+  const declared = isRecord4(node.properties) ? readBlockKindFromProperties(node.properties) : null;
+  return { slug: declared ?? name, node };
+}
+function refToObjectField(resolved, path, ctx) {
+  if (ctx.refsInProgress.has(resolved.slug)) return;
+  ctx.refsInProgress.add(resolved.slug);
+  try {
+    registerDeclaredKindDraft(resolved.slug, resolved.node, path, ctx);
+  } finally {
+    ctx.refsInProgress.delete(resolved.slug);
+  }
+}
 function isAnyValueSchema(node) {
   return node.type === void 0 && node.enum === void 0 && node.const === void 0 && node.properties === void 0 && node.items === void 0 && node.additionalProperties === void 0 && node.anyOf === void 0 && node.oneOf === void 0 && node.allOf === void 0 && node.$ref === void 0;
 }
@@ -3184,12 +3236,21 @@ function convertProperty(fieldName, node, required, path, ctx) {
 }
 function convertPropertyCore(fieldName, node, required, path, ctx) {
   if (typeof node.$ref === "string") {
-    ctx.problems.push({
-      severity: "error",
-      path,
-      message: `$ref is not supported ("${node.$ref}"). Inline the schema manually.`
-    });
-    return null;
+    const resolved = resolveRef(node.$ref, ctx);
+    if (!resolved) {
+      ctx.problems.push({
+        severity: "error",
+        path,
+        message: `Unresolvable $ref ("${node.$ref}") \u2014 only "#", "#/$defs/<name>" and "#/definitions/<name>" are supported.`
+      });
+      return null;
+    }
+    refToObjectField(resolved, path, ctx);
+    return {
+      ...requiredNullableFlags(required),
+      type: "object",
+      kind: resolved.slug
+    };
   }
   if (Array.isArray(node.anyOf) || Array.isArray(node.oneOf)) {
     const variants = node.anyOf ?? node.oneOf;
@@ -3205,12 +3266,18 @@ function convertPropertyCore(fieldName, node, required, path, ctx) {
         continue;
       }
       if (typeof variant.$ref === "string") {
-        ctx.problems.push({
-          severity: "error",
-          path,
-          message: `$ref inside anyOf/oneOf is not supported ("${variant.$ref}"). Inline the schema manually.`
-        });
-        unsupported = true;
+        const resolvedVariant = resolveRef(variant.$ref, ctx);
+        if (!resolvedVariant) {
+          ctx.problems.push({
+            severity: "error",
+            path,
+            message: `Unresolvable $ref inside anyOf/oneOf ("${variant.$ref}").`
+          });
+          unsupported = true;
+          continue;
+        }
+        refToObjectField(resolvedVariant, path, ctx);
+        memberKinds.push(resolvedVariant.slug);
         continue;
       }
       const { type: type2, nullable: nullable2 } = resolvePrimaryType(variant);
@@ -3331,7 +3398,34 @@ function convertPropertyCore(fieldName, node, required, path, ctx) {
     });
     return { ...requiredNullableFlags(required), type: "json" };
   }
-  const { type, nullable } = resolvePrimaryType(node);
+  const { type, nullable, members } = resolvePrimaryType(node);
+  if (members.length > 1) {
+    const objectish = members.some((m) => m === "object" || m === "array");
+    if (objectish) {
+      ctx.problems.push({
+        severity: "info",
+        path,
+        message: `Union of [${members.join(", ")}] at "${path || "(root)"}" spans objects/arrays \u2014 carried as any JSON value (json).`
+      });
+      return { ...requiredNullableFlags(required), type: "json" };
+    }
+    const scalars = members.filter(
+      (m) => m === "string" || m === "number" || m === "boolean"
+    );
+    if (scalars.length === members.length && scalars.length > 1) {
+      return {
+        ...requiredNullableFlags(required, nullable),
+        type: "union",
+        scalars
+      };
+    }
+    ctx.problems.push({
+      severity: "warning",
+      path,
+      message: `Union of [${members.join(", ")}] at "${path || "(root)"}" has no exact field type \u2014 carried as any JSON value (json) rather than narrowed to "${type}".`
+    });
+    return { ...requiredNullableFlags(required), type: "json" };
+  }
   if (type === "string") {
     if (Array.isArray(node.enum)) {
       const values = node.enum.filter(
@@ -3386,6 +3480,27 @@ function convertPropertyCore(fieldName, node, required, path, ctx) {
       });
       return null;
     }
+    if (typeof itemNode.$ref === "string") {
+      const resolvedItem = resolveRef(itemNode.$ref, ctx);
+      if (!resolvedItem) {
+        ctx.problems.push({
+          severity: "error",
+          path: `${path}[]`,
+          message: `Unresolvable $ref in array items ("${itemNode.$ref}").`
+        });
+        return null;
+      }
+      refToObjectField(resolvedItem, `${path}[]`, ctx);
+      ctx.arrayBindings.push({
+        arrayField: fieldName,
+        itemKindSlug: resolvedItem.slug
+      });
+      return {
+        ...requiredNullableFlags(required, nullable),
+        type: "array",
+        itemKinds: [resolvedItem.slug]
+      };
+    }
     if (isAnyValueSchema(itemNode)) {
       return {
         ...requiredNullableFlags(required, nullable),
@@ -3420,11 +3535,29 @@ function convertPropertyCore(fieldName, node, required, path, ctx) {
       }
       const itemKinds = [];
       for (const variant of itemVariants) {
+        if (isRecord4(variant) && typeof variant.$ref === "string") {
+          const resolvedVariant = resolveRef(variant.$ref, ctx);
+          if (!resolvedVariant) {
+            ctx.problems.push({
+              severity: "error",
+              path: `${path}[]`,
+              message: `Unresolvable $ref in array items anyOf ("${variant.$ref}").`
+            });
+            return null;
+          }
+          itemKinds.push(resolvedVariant.slug);
+          refToObjectField(resolvedVariant, `${path}[]<${resolvedVariant.slug}>`, ctx);
+          ctx.arrayBindings.push({
+            arrayField: fieldName,
+            itemKindSlug: resolvedVariant.slug
+          });
+          continue;
+        }
         if (!isRecord4(variant) || !isRecord4(variant.properties)) {
           ctx.problems.push({
             severity: "error",
             path: `${path}[]`,
-            message: "Array items anyOf/oneOf variants must be inline objects declaring __kind."
+            message: "Array items anyOf/oneOf variants must be inline objects declaring __kind, or $refs to defs."
           });
           return null;
         }
@@ -3558,6 +3691,14 @@ function convertPropertyCore(fieldName, node, required, path, ctx) {
           kind: referencedKind
         };
       }
+      if (Object.keys(nestedFields).length === 0 && !apIsOpen && isRecord4(ap) && KIND_KEY in node.properties) {
+        const apType = resolvePrimaryType(ap).type;
+        return {
+          ...requiredNullableFlags(required, nullable),
+          type: "record",
+          values: apType === "string" || apType === "number" || apType === "boolean" ? apType : "json"
+        };
+      }
       let open = apIsOpen;
       if (!apIsOpen && isRecord4(ap)) {
         ctx.problems.push({
@@ -3674,12 +3815,21 @@ function normalizeAiSchemaInput(input) {
   return { name: null, strict: null, rootSchema: null, parseErrors };
 }
 function convertAiSchemaToBlockFields(schemaName, rootSchema, strict) {
+  const rawDefs = isRecord4(rootSchema.$defs) ? rootSchema.$defs : isRecord4(rootSchema.definitions) ? rootSchema.definitions : {};
+  const defs = {};
+  for (const [name, node] of Object.entries(rawDefs)) {
+    if (isRecord4(node)) defs[name] = node;
+  }
   const ctx = {
     schemaName,
+    strict,
     problems: [],
     droppedMetadata: [],
     blockSchemas: [],
-    arrayBindings: []
+    arrayBindings: [],
+    defs,
+    rootSchema,
+    refsInProgress: /* @__PURE__ */ new Set()
   };
   ctx.droppedMetadata.push(...collectDropped(rootSchema, ""));
   const rootAp = rootSchema.additionalProperties;