@ai-markdown/core 3.0.0 → 3.0.2

package/README.md

@@ -1,13 +1,15 @@
 # @ai-markdown/core
 
+[Documentation](https://ai-markdown.github.io/docs/core/) · [Examples](https://ai-markdown.github.io/examples/) · [Website](https://ai-markdown.github.io/)
+
 [![@ai-markdown/core stable](https://img.shields.io/npm/v/@ai-markdown/core?label=npm&color=blue)](https://www.npmjs.com/package/@ai-markdown/core?activeTab=versions)
 [![@ai-markdown/core monthly downloads](https://img.shields.io/npm/dm/@ai-markdown/core?label=downloads%2Fmonth&color=blue)](https://www.npmjs.com/package/@ai-markdown/core)
 [![TypeScript declarations included](https://img.shields.io/badge/TypeScript-included-3178c6?logo=typescript&logoColor=white)](https://github.com/ai-markdown/ai-markdown/tree/main/packages/core)
 [![MIT license](https://img.shields.io/badge/license-MIT-blue)](https://github.com/ai-markdown/ai-markdown/blob/main/LICENSE)
 
-Framework-independent orchestration for ai-markdown adapters, on the **3.0.0** train alongside the React and Vue adapters. It supplies pipeline sessions, block planning, cross-chunk contributions, aggregate footnote trees and streaming coordination. Applications install `@ai-markdown/react` or `@ai-markdown/vue`; adapter authors can install `@ai-markdown/core` and `@ai-markdown/engine` directly.
+Framework-independent orchestration for ai-markdown adapters, on the same release train as the React and Vue adapters. It supplies pipeline sessions, block planning, cross-chunk contributions, aggregate footnote trees and streaming coordination. Applications install `@ai-markdown/react` or `@ai-markdown/vue`; adapter authors can install `@ai-markdown/core` and `@ai-markdown/engine` directly.
 
-This package was the private runtime in the legacy v2.14.1 release. It is now a real external dependency of both framework adapters, with an explicit public export list. The old `@ai-react-markdown/core` React package maps to `@ai-markdown/react`, not this package. See the [migration guide](../../docs/framework-transition.md). Public contracts follow semantic versioning from 3.0.0; keep the shared packages at the same exact train version.
+This package was the private runtime in the legacy v2.14.1 release. It is now a real external dependency of both framework adapters, with an explicit public export list. The old `@ai-react-markdown/core` React package maps to `@ai-markdown/react`, not this package. See the [migration guide](https://ai-markdown.github.io/docs/guides/framework-transition/). Public contracts follow semantic versioning from 3.0.0; keep the shared packages at the same exact train version.
 
 ## Responsibility and dependency direction
 
@@ -20,17 +22,17 @@ This package was the private runtime in the legacy v2.14.1 release. It is now a
 
 The adapter may also consume engine primitives directly. Shared core does not duplicate the engine's public barrel or wrap every engine function merely to rename it. It owns reusable orchestration that would otherwise have to be copied into another framework adapter.
 
-| Module                       | Responsibility                                                                                                                                             | State owner                                             |
-| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
-| `coordinationPreparation.ts` | Derive phantom targets, select coordinated/orphan handlers and harvesting policy, build contribution invalidation tuples                                   | Pure decisions; previous snapshots belong to the caller |
-| `pipelineSession.ts`         | Merge coordinated parse options, append protected phantom suffixes, select full/incremental parsing, invalidate failed state and fall back to a full parse | One session per rendered chunk                          |
-| `blockPlan.ts`               | Associate transformed HAST with source MDAST, produce stable keys, classify reference dependencies and swallowed HTML, compute registry fingerprints       | Pure functions over the supplied trees and registry     |
-| `blockPlanner.ts`            | Reuse eligible retained prefix plans while preserving whole-document reference context                                                                     | One planner per rendered chunk                          |
-| `contribution.ts`            | Compare source and policy fingerprints, harvest transformed definition bodies, publish changed committed contributions                                     | One publisher per mounted chunk                         |
-| `aggregateFootnotes.ts`      | Assemble a document's ordered footnotes and occurrence backrefs as HAST                                                                                    | Returned tree belongs to the caller                     |
-| `cloneHastForRender.ts`      | Clone node/children/properties/data and original-URL containers before render-time mutation                                                                | Returned clone belongs to the caller                    |
-| `smoothCoordinator.ts`       | Order chunk reveal, maintain sticky completion, defer cleanup and coalesce notifications                                                                   | One coordinator per logical document                    |
-| `tailSignal.ts`              | Classify a source tail that renders inside a footnote or as an invisible link definition                                                                   | Pure function over MDAST                                |
+| Module                       | Responsibility                                                                                                                                                                                                          | State owner                                             |
+| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
+| `coordinationPreparation.ts` | Derive phantom targets, select coordinated/orphan handlers and harvesting policy, build contribution invalidation tuples                                                                                                | Pure decisions; previous snapshots belong to the caller |
+| `pipelineSession.ts`         | Merge coordinated parse options, append protected phantom suffixes, select full/incremental parsing, invalidate failed state, fall back to a full parse or, on the engine's raw-depth error only, to a plain-text frame | One session per rendered chunk                          |
+| `blockPlan.ts`               | Associate transformed HAST with source MDAST, produce stable keys, classify reference dependencies and swallowed HTML, compute registry fingerprints                                                                    | Pure functions over the supplied trees and registry     |
+| `blockPlanner.ts`            | Reuse eligible retained prefix plans while preserving whole-document reference context                                                                                                                                  | One planner per rendered chunk                          |
+| `contribution.ts`            | Compare source and policy fingerprints, harvest transformed definition bodies, publish changed committed contributions                                                                                                  | One publisher per mounted chunk                         |
+| `aggregateFootnotes.ts`      | Assemble a document's ordered footnotes and occurrence backrefs as HAST                                                                                                                                                 | Returned tree belongs to the caller                     |
+| `cloneHastForRender.ts`      | Clone node/children/properties/data and original-URL containers before render-time mutation                                                                                                                             | Returned clone belongs to the caller                    |
+| `smoothCoordinator.ts`       | Order chunk reveal, maintain sticky completion, defer cleanup and coalesce notifications                                                                                                                                | One coordinator per logical document                    |
+| `tailSignal.ts`              | Classify a source tail that renders inside a footnote or as an invisible link definition                                                                                                                                | Pure function over MDAST                                |
 
 The engine remains responsible for grammar, preprocessing, plugin chains, scanners, incremental parsing algorithms, registry storage/indexing, reference resolution and URL policy primitives. Shared core uses those algorithms to implement reusable rendering decisions.
 
@@ -77,7 +79,7 @@ This is a standalone example using the public package entries. A coordinated ada
 
 The host chooses `incrementalParse: false` for a one-shot server render. Shared core does not inspect `window` to infer the host. A later incremental frame starts from fresh state after a non-incremental frame. Call `reset()` when the host invalidates retained parse state because render policy changed. Parse-input identities are also checked by the engine's dependency key; phantom suffix changes remain always-tail input rather than invalidating all retained source.
 
-An incremental failure clears the retained state before retrying the full pipeline. A failed full parse still propagates its exception to the host. The fallback is not a substitute for an error boundary, and the shared core does not silently return a stale frame. Each session and planner is mutable local computation state: do not share one across independent chunks or concurrently executing consumers.
+An incremental failure clears the retained state before retrying the full pipeline. The full pipeline has one degraded outcome: when the engine's raw-HTML step reports `EngineRawHtmlDepthError` (element nesting deeper than the engine's bound, `RAW_HTML_MAX_DEPTH`, which hundreds of nested raw `<div>` tags exceed), the session logs the error in development builds and returns the frame as one plain-text paragraph whose positions cover the whole source, so one hostile message cannot take the host surface down; the next frame parses normally. That last resort renders escaped text, not a stale frame. Every other exception from the full pipeline propagates to the host, including a `RangeError` that is not that overflow and any throw from a remark or rehype plugin or handler the host supplied: those are bugs to surface, not frames to render as text, and the fallback is not a substitute for an error boundary around adapter code. Each session and planner is mutable local computation state: do not share one across independent chunks or concurrently executing consumers.
 
 ## Shared preparation decisions
 
@@ -89,7 +91,7 @@ An incremental failure clears the retained state before retrying the full pipeli
 
 These functions do not register, subscribe, publish or mutate their supplied snapshots. Treat label sets and returned objects as immutable; pass a previous result only from the same logical consumer. React retains the previous snapshots in refs and uses memoization; Vue uses local variables behind computed values. The engine and planner still own correctness when a source is replaced or a retained render is discarded.
 
-The published [Vue adapter](../vue/README.md) uses these decisions in its component lifecycle while keeping AST/registry objects out of deep reactive proxies. It has SSR, hydration, slots, base styles and cursor support, with separate adapter and Chromium lifecycle tests. The earlier private prototype is archived under `prototypes/` and is not the application entry point.
+The published [Vue adapter](../vue/README.md) uses these decisions in its component lifecycle while keeping AST/registry objects out of deep reactive proxies. It has SSR, hydration, slots, base styles and cursor support, with functional adapter checks in Chromium, Firefox and WebKit, and forced-GC lifecycle checks in Chromium. The earlier private prototype is archived under `prototypes/` and is not the application entry point.
 
 ## Preparation and commit have different effects
 
@@ -124,16 +126,16 @@ pnpm preflight
 
 Build before running distribution tests: they load the actual ESM/CJS production and development files in separate Node processes, reject UI framework resolution transitively and parse without browser globals. Other tests compare session output with an independent full engine pipeline, exercise resets/fallback, prove explicit contribution timing and check aggregate body immutability. Existing React tests still exercise the adapter's delegation, memoization, byte equivalence, coordination and browser behavior.
 
-The dedicated `test:core-contracts` gate builds core and its workspace dependencies, typechecks it, and runs all core tests independently of React/Vue. It runs in preflight, a dedicated CI task and the release workflow. Pure planner, coordinator, tail-signal and block-plan tests live with core; React retains renderer/cache integration tests. Fixed-seed sequences compare retained pipeline/planning and contribution state against full reconstruction, and coordinator lifecycles against a separate model. See [core testing](../../docs/core-testing.md) for the module map, fixed budgets, replay commands and limitations.
+The dedicated `test:core-contracts` gate builds core and its workspace dependencies, typechecks it, and runs all core tests independently of React/Vue. It runs in preflight, a dedicated CI task and the release workflow. Pure planner, coordinator, tail-signal and block-plan tests live with core; React retains renderer/cache integration tests. Fixed-seed sequences compare retained pipeline/planning and contribution state against full reconstruction, and coordinator lifecycles against a separate model. See [core testing](https://ai-markdown.github.io/docs/guides/core-testing/) for the module map, fixed budgets, replay commands and limitations.
 
 `assert-boundary.mjs` checks public-package identity, allowed production dependencies, source import direction and folded environment gates. The React distribution guard requires external core and engine imports. Core's declarations must not expose `RegistryInternal`, `SmoothCoordinatorInternal` or private refcount/subscriber containers.
 
-The release train is engine/core/react/react-mantine/vue at the same version. Vue now consumes the same shared contracts; see the [API contracts](../../docs/api/core-engine-contracts.md) for the beta.1-to-beta.2 signature changes. Core depends on engine through `workspace:*`, which becomes the exact train version in the published manifest. Both ESM and CJS have production and development entries; every build folds environment gates separately. Core has no `use client` directive and does not inline a second engine implementation.
+The release train is engine/core/react/react-mantine/vue at the same version. Vue now consumes the same shared contracts; see the [API contracts](https://ai-markdown.github.io/docs/guides/api/core-engine-contracts/) for stable signatures and migration from beta.1. Core depends on engine through `workspace:*`, which becomes the exact train version in the published manifest. Both ESM and CJS have production and development entries; every build folds environment gates separately. Core has no `use client` directive and does not inline a second engine implementation.
 
 ## Public API and write capabilities
 
-The root entry explicitly lists supported beta exports. Session creation, parsing, planning, pure coordination preparation, contribution commit, aggregate construction, structural cloning and source-tail classification are adapter contracts. The existing block digest and fingerprint functions remain available because React consumes them; their source/ownership assumptions above still apply.
+The root entry explicitly lists supported stable exports. Session creation, parsing, planning, pure coordination preparation, contribution commit, aggregate construction, structural cloning and source-tail classification are adapter contracts. The existing block digest and fingerprint functions remain available because React consumes them; their source/ownership assumptions above still apply.
 
 `createSmoothCoordinator` returns the read-only state and documented methods of `SmoothCoordinator`, excluding internal refcounts and notification containers. `ContributionOptions.registry` accepts only a `ContributionRegistry` write capability with `contributeChunkData`; it does not require an implementation registry. Create a registry through engine's `createRegistry`, whose `RegistryController` adds registration and publication to the read-only `Registry` contract. Pair every registration with release in the owning adapter.
 
-These are type-level API boundaries, not object freezing or arbitrary deep cloning. Mutating returned sets, tree data or undocumented implementation fields is unsupported. Public contracts and checked signatures are recorded in the [API contract document](../../docs/api/core-engine-contracts.md).
+These are type-level API boundaries, not object freezing or arbitrary deep cloning. Mutating returned sets, tree data or undocumented implementation fields is unsupported. Public contracts and checked signatures are recorded in the [API contract document](https://ai-markdown.github.io/docs/guides/api/core-engine-contracts/).

package/dist/index.cjs

@@ -383,19 +383,49 @@ function createBlockPlanner() {
     let h = 0;
     let reusedContext = false;
     if (previous && source.startsWith(previous.source) && options.phantomFootnoteLabels === previous.phantoms) {
+      const prevHast = previous.hast.children;
+      const prevPlan = previous.result.plan;
+      const isRetainedGap = (i) => {
+        const gap = hast.children[i];
+        return gap === prevHast[i] && gap.type === "text" && !/\S/.test(gap.value);
+      };
       while (m < mdast.children.length) {
         const node = mdast.children[m];
         const el = hast.children[h];
-        const item = previous.result.plan[h];
-        if (node !== previous.mdast.children[m] || el !== previous.hast.children[h] || !el || el.type !== "element" || !item || item.kind !== "block" || item.info.startOffset !== node.position?.start.offset || !isEligible(node))
+        const item = prevPlan[h];
+        if (node !== previous.mdast.children[m] || el !== prevHast[h] || !el || el.type !== "element" || !item || item.kind !== "block" || item.info.startOffset !== node.position?.start.offset || !isEligible(node))
           break;
+        const startOffset = item.info.startOffset;
+        let next = h + 1;
+        let complete = true;
+        for (; ; ) {
+          let probe = next;
+          while (probe < hast.children.length && isRetainedGap(probe)) probe++;
+          const sibling = prevPlan[probe];
+          if (!sibling || sibling.kind !== "block" || sibling.info.startOffset !== startOffset) break;
+          if (hast.children[probe] !== prevHast[probe]) {
+            complete = false;
+            break;
+          }
+          next = probe + 1;
+        }
+        if (!complete) break;
         reusedContext ||= item.info.hasReference;
         m++;
-        h++;
-        while (h < hast.children.length) {
-          const gap = hast.children[h];
-          if (gap !== previous.hast.children[h] || gap.type !== "text" || /\S/.test(gap.value)) break;
-          h++;
+        h = next;
+        while (h < hast.children.length && isRetainedGap(h)) h++;
+      }
+      if (m > 0) {
+        const prefixEnd = mdast.children[m - 1].position?.end.offset ?? 0;
+        for (let i = h; i < hast.children.length; i++) {
+          const child = hast.children[i];
+          const at = child.type === "element" ? child.position?.start.offset : void 0;
+          if (at !== void 0 && at < prefixEnd) {
+            m = 0;
+            h = 0;
+            reusedContext = false;
+            break;
+          }
         }
       }
     }
@@ -442,7 +472,12 @@ function createContributionSession() {
         phantomFootnoteLabels: targetPhantoms.missingFootnotes
       })) {
         if (node.kind === "ref") {
-          refs.push({ label: node.label, kind: node.refKind, referenceType: node.referenceType });
+          refs.push({
+            label: node.label,
+            kind: node.refKind,
+            referenceType: node.referenceType,
+            ...node.nestedIn !== void 0 ? { nestedIn: node.nestedIn } : {}
+          });
         } else if (node.kind === "fnDef") {
           defMeta.set(node.label, {
             identifier: node.label,
@@ -476,14 +511,14 @@ function createContributionSession() {
       if (last?.registry === registry && last.symbol === sym && last.fp === fp && last.chain.length === chain.length && last.chain.every((dep, i) => dep === chain[i])) {
         return;
       }
-      const bodiesByLabel = (0, import_engine2.extractDefBodiesFromHast)(pipeline.hast, clobberPrefix);
+      const bodiesBySafeId = (0, import_engine2.extractDefBodiesFromHast)(pipeline.hast, clobberPrefix);
       const defs = /* @__PURE__ */ new Map();
       for (const [label, meta] of defMeta) {
         defs.set(label, {
           identifier: meta.identifier,
           sourceIdentifier: meta.sourceIdentifier,
           contentSource: meta.contentSource,
-          bodyHast: bodiesByLabel.get(label) ?? []
+          bodyHast: bodiesBySafeId.get((0, import_engine2.footnoteSafeId)(meta.sourceIdentifier)) ?? []
         });
       }
       lastContribution = { registry, symbol: sym, fp, chain };
@@ -691,6 +726,7 @@ function buildAggregateTree(registry, clobberPrefix, preserveOrphanReferences =
       });
     }
   }
+  ordered.sort((a, b) => (a.n ?? 0) - (b.n ?? 0));
   if (preserveOrphanReferences) {
     for (const sym of registry.chunkOrder) {
       const data = registry.chunkData.get(sym);
@@ -778,6 +814,30 @@ function buildAggregateTree(registry, clobberPrefix, preserveOrphanReferences =
 // src/pipelineSession.ts
 var import_engine4 = require("@ai-markdown/engine");
 var unmeasured = (_stage, fn) => fn();
+function endPoint(text) {
+  let line = 1;
+  let lineStart = 0;
+  for (let i = 0; i < text.length; i++) {
+    const code = text.charCodeAt(i);
+    if (code === 10 || code === 13 && text.charCodeAt(i + 1) !== 10) {
+      line += 1;
+      lineStart = i + 1;
+    }
+  }
+  return { line, column: text.length - lineStart + 1, offset: text.length };
+}
+function plainTextTrees(content) {
+  const position = { start: { line: 1, column: 1, offset: 0 }, end: endPoint(content) };
+  const text = { type: "text", value: content, position };
+  return {
+    mdast: { type: "root", children: [{ type: "paragraph", children: [text], position }], position },
+    hast: {
+      type: "root",
+      children: [{ type: "element", tagName: "p", properties: {}, children: [text], position }],
+      position
+    }
+  };
+}
 function createPipelineSession() {
   let state = null;
   return {
@@ -819,19 +879,33 @@ function createPipelineSession() {
       } : {
         ...remarkRehypeOptions
       };
+      const fullPipeline = () => {
+        try {
+          const parsed = measureHere(
+            "parse",
+            () => (0, import_engine4.parseStage)({
+              children: augmented,
+              remarkPlugins,
+              rehypePlugins,
+              remarkRehypeOptions: mergedRemarkRehypeOptions
+            })
+          );
+          const hastRoot = measureHere("transform", () => (0, import_engine4.transformStage)(parsed));
+          return { mdast: parsed.mdast, hast: hastRoot };
+        } catch (error) {
+          if (!(error instanceof import_engine4.EngineRawHtmlDepthError)) throw error;
+          if (false) {
+            console.error(
+              "[ai-react-markdown] raw HTML nested past the engine depth bound \u2014 rendering this frame as plain text:",
+              error
+            );
+          }
+          return plainTextTrees(content ?? "");
+        }
+      };
       if (!incrementalParse) {
         state = null;
-        const parsed = measureHere(
-          "parse",
-          () => (0, import_engine4.parseStage)({
-            children: augmented,
-            remarkPlugins,
-            rehypePlugins,
-            remarkRehypeOptions: mergedRemarkRehypeOptions
-          })
-        );
-        const hastRoot = measureHere("transform", () => (0, import_engine4.transformStage)(parsed));
-        return { mdast: parsed.mdast, hast: hastRoot };
+        return fullPipeline();
       }
       try {
         const result = (0, import_engine4.advanceIncrementalParse)(state, content ?? "", {
@@ -854,7 +928,12 @@ function createPipelineSession() {
             // Explicit even though `rehypePlugins` already carries it: a
             // credential change must never reuse trees stamped under the
             // old one.
-            provenance
+            provenance,
+            // The boundary scanner's grammar profile. The scan checkpoint
+            // already refuses to resume under a different profile, but the
+            // retained TREES were still spliced against — a prefix frozen
+            // under one profile must not survive into frames under the other.
+            defListEnabled
           ],
           defListEnabled,
           phantomSuffix,
@@ -867,17 +946,7 @@ function createPipelineSession() {
         if (false) {
           console.error("[ai-react-markdown] incremental parse failed \u2014 full parse fallback for this frame:", error);
         }
-        const parsed = measureHere(
-          "parse",
-          () => (0, import_engine4.parseStage)({
-            children: augmented,
-            remarkPlugins,
-            rehypePlugins,
-            remarkRehypeOptions: mergedRemarkRehypeOptions
-          })
-        );
-        const hastRoot = measureHere("transform", () => (0, import_engine4.transformStage)(parsed));
-        return { mdast: parsed.mdast, hast: hastRoot };
+        return fullPipeline();
       }
     }
   };