@spintax/core 0.3.1 → 0.3.3

package/CHANGELOG.md

@@ -3,6 +3,116 @@
 All notable changes to `@spintax/core` are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.3 — 2026-08-07
+
+`validate()` scales. No output changes anywhere: a 464-document differential over every
+definition-graph, diagnostic-position and fuzz shape is byte-identical before and after, and
+three deliberate mutations (drop the prune, shift a column, freeze the line counter) each
+turn it red — the harness can see what it guards.
+
+### Fixed
+
+- **`validate()` is no longer super-linear on definition graphs — and no longer hangs.** Four
+  independent costs, each measured before and after (Node 22, same machine):
+
+  | shape | 0.3.2 | 0.3.3 |
+  |---|---|---|
+  | chain of 1600 `#set`s | 13.1 s | 18 ms |
+  | converging diamond, 20 levels (~1 KB) | 4.8 s | <1 ms |
+  | converging diamond, 26+ levels | **never returns** | <1 ms (2000 levels: 22 ms) |
+  | one cycle of 1600 | 55.6 s | 0.8 s |
+  | 6400 duplicate definitions | 9.3 s | 29 ms |
+  | 6400 undefined references | 2.9 s | 11 ms |
+
+  The circular-reference walk re-parsed every value's references at every visit, tested the
+  path with an array `includes`, copied the path at every step, and restarted from every
+  definition with no memory of silent subtrees — on a converging diamond that re-explored
+  shared subgraphs exponentially, so a one-kilobyte template could pin a CPU forever (the
+  validator is exposed by hosts; that is a denial-of-service shape, not just a slow one).
+  References are now parsed once, the path is a Set plus a shared push/pop array, the walk is
+  iterative (a chain as deep as the document cannot overflow the call stack), and one colour
+  walk computes up front which names can reach a cycle — a subtree that cannot reach one
+  cannot report, so skipping it is output-neutral by construction. Emission order, count and
+  messages are exactly the old walk's, duplicated edges and all.
+
+  The taint sweep behind `plural.count-macro` was a fixpoint that re-read every macro once per
+  newly tainted name; it is now a reverse-edge worklist computing the same closure. Diagnostic
+  positions were each computed by scanning from offset 0; a line-start index built once per
+  call now answers them by binary search. And `extractDirectives` counted the line of each
+  occurrence from the start of the text — quadratic on directive-heavy documents, and on the
+  **render** path too, since the renderer extracts directives the same way; a resuming counter
+  fixes both surfaces.
+
+  Two boundaries stay as they are, deliberately. A giant single cycle still costs seconds at
+  6400 definitions because the output itself is quadratic — every diagnostic's message carries
+  the full cycle path, so ~300 MB of message text is the answer, not the overhead. And a
+  converging diamond that FEEDS a cycle still emits one diagnostic per path — exponentially
+  many — because that is the reference emission the corpus family just aligned on
+  (spintax-win, 2026-08-07); whether that emission should be bounded is a family question, not
+  a patch.
+
+Post-process robustness and scaling: `postProcess()` no longer emits its own U+0000 sentinel, and
+the placeholder restore is linear instead of quadratic. Output is unchanged on ordinary text; the
+two behaviour changes are on shapes that previously produced invalid output (a raw U+0000).
+
+### Fixed
+
+- **`postProcess()` no longer emits a raw U+0000** into the returned text — on input carrying
+  none. A URI body runs to the first delimiter, so the URL rule and the `mailto:`/`tel:` rule
+  overlap whenever one URI carries the other's scheme, and shielding them in two passes let the
+  second run into a placeholder the first had minted:
+
+  ```js
+  render('mailto:sales@example.com?body=see%20https://shop.example.com/cart');
+  // 'mailto:sales@example.com?body=see%20\x00URL_0\x00'  ← before this release
+  ```
+
+  The swallowed key never restored, so the engine returned its own placeholder delimiter: illegal
+  in an XML document, replaced with U+FFFD by an HTML parser, rejected by PostgreSQL in
+  `text`/`varchar` — and a live key again as soon as an edit detaches it from the `mailto:` prefix
+  that was shielding it, at which point the next render substitutes an unrelated URL into the
+  contact link. It also silently disabled the linear restore below, whose guard is
+  `input.includes('\x00')`.
+
+  The two rules are now **one alternation**, so the leftmost match takes the whole token whichever
+  scheme it is. Reordering the passes was the other candidate and is not equivalent: it moves the
+  damage onto a URL whose path carries a `mailto:`, where the leading half then loses its trailing
+  dot to the punctuation pass. Both directions are fixtures now, the second one negative. U+0000 is
+  also excluded from the URI body class, for a caller-supplied one.
+
+  Mirrored into the WordPress plugin engine, and gated by three golden-corpus fixtures — the
+  cross-engine contract, so `spintax-php`, `spintax-py` and `spintax-win` inherit the gate.
+  ([#53](https://github.com/investblog/spintax-js/issues/53))
+
+### Performance
+
+- **Post-process no longer goes quadratic on shield-heavy text.** The restore step replaced each
+  shielded placeholder across the whole text one key at a time — O(text × placeholders) — and
+  since URLs, `mailto:`/`tel:` URIs, emails, domains, decimals and abbreviations are all shielded,
+  the placeholder count grows with the input. The stage came to dominate the render:
+
+  | input  | `postProcess: false` | before     | after   |
+  |--------|----------------------|------------|---------|
+  | 47 KB  | 0.007 s              | 0.18 s     | 0.014 s |
+  | 189 KB | 0.006 s              | 3.06 s     | 0.057 s |
+  | 756 KB | 0.081 s              | **43.9 s** | 0.25 s  |
+
+  A single left-to-right pass replaces the loop, behind a guard on the input. The two are not the
+  same function: the loop is a repeated *substring* substitution, so it rewrites every occurrence
+  of a key rather than the one the shield placed. Most of the disagreement needs a literal `\x00`
+  from the caller, and the guard sends that input to the original loop.
+
+  **One shape survives the guard**, and on it the behaviour changed deliberately: two adjacent
+  placeholders can sandwich caller text that spells a key, so one token's closing delimiter, that
+  text, and the next token's opening delimiter form a third occurrence of a real key. Rendering
+  `https://a.io e.g. URL_0mailto:x@y.io` — no `\x00` anywhere in it — the loop substituted the
+  forgery, destroyed two real tokens and returned raw `\x00` bytes; the single pass returns the
+  text intact. Measured over 456 976 probes, 12 `\x00`-free inputs distinguish the two restores and
+  the loop emits a raw `\x00` on all 12. A corpus fixture now pins the surviving answer, since the
+  engines disagreed on it. `npm run bench:postprocess` records the scaling.
+  ([#52](https://github.com/investblog/spintax-js/issues/52),
+  [#54](https://github.com/investblog/spintax-js/issues/54))
+
 ## 0.3.1 — 2026-07-21
 
 Author markup is sanitised in one place, so a handle renders exactly like its source.

package/dist/index.cjs

@@ -61,6 +61,15 @@ function extractDirectives(text) {
   const setDefs = {};
   const defDefs = {};
   const occurrences = [];
+  let cursorOffset = 0;
+  let cursorLine = 1;
+  const lineAt = (offset) => {
+    for (let i = cursorOffset; i < offset; i += 1) {
+      if (text.charCodeAt(i) === 10) cursorLine += 1;
+    }
+    cursorOffset = offset;
+    return cursorLine;
+  };
   DIRECTIVE_RE.lastIndex = 0;
   const stripped = text.replace(
     DIRECTIVE_RE,
@@ -70,7 +79,7 @@ function extractDirectives(text) {
         kind,
         name,
         value,
-        line: countLines(text, offset)
+        line: lineAt(offset)
       });
       if (kind === "def") defDefs[name] = value;
       else setDefs[name] = value;
@@ -79,11 +88,6 @@ function extractDirectives(text) {
   );
   return { body: stripped.replace(/\n{3,}/gu, "\n\n"), setDefs, defDefs, occurrences };
 }
-function countLines(text, offset) {
-  let line = 1;
-  for (let i = 0; i < offset; i += 1) if (text[i] === "\n") line += 1;
-  return line;
-}
 function stripComments(text) {
   return text.replace(/\/#[\s\S]*?#\//g, "");
 }
@@ -420,13 +424,14 @@ var INCLUDE_RE = /^[ \t]*#include[ \t\n\r\f\x0B]+"([^"]+)"[ \t\n\r\f\x0B]*$/gmu;
 function validateTemplate(src, opts = {}) {
   const diagnostics = [];
   const text = stripComments(src);
+  const idx = buildLineIndex(text);
   checkBrackets(text, diagnostics);
   checkDirectives(text, diagnostics);
-  checkPermutationConfigs(text, diagnostics);
-  checkPlurals(text, opts.locale, diagnostics);
-  checkVariableReferences(text, opts.knownVariables, diagnostics);
+  checkPermutationConfigs(text, idx, diagnostics);
+  checkPlurals(text, idx, opts.locale, diagnostics);
+  checkVariableReferences(text, idx, opts.knownVariables, diagnostics);
   if (opts.knownIncludes && opts.knownIncludes.length > 0) {
-    checkIncludeTargets(text, opts.knownIncludes, diagnostics);
+    checkIncludeTargets(text, idx, opts.knownIncludes, diagnostics);
   }
   return diagnostics;
 }
@@ -436,21 +441,27 @@ function err(code, message, pos) {
 function warn(code, message, pos) {
   return { severity: "warning", code, message, ...pos };
 }
-function offsetToLineCol(text, offset) {
-  let line = 1;
-  let lineStart = 0;
-  const end = Math.min(Math.max(offset, 0), text.length);
-  for (let i = 0; i < end; i += 1) {
-    if (text.charAt(i) === "\n") {
-      line += 1;
-      lineStart = i + 1;
-    }
+function buildLineIndex(text) {
+  const starts = [0];
+  for (let i = 0; i < text.length; i += 1) {
+    if (text.charCodeAt(i) === 10) starts.push(i + 1);
   }
-  return { line, column: end - lineStart + 1 };
+  return { starts, length: text.length };
+}
+function offsetToLineCol(idx, offset) {
+  const end = Math.min(Math.max(offset, 0), idx.length);
+  let lo = 0;
+  let hi = idx.starts.length - 1;
+  while (lo < hi) {
+    const mid = lo + hi + 1 >> 1;
+    if ((idx.starts[mid] ?? 0) <= end) lo = mid;
+    else hi = mid - 1;
+  }
+  return { line: lo + 1, column: end - (idx.starts[lo] ?? 0) + 1 };
 }
-function span(text, offset, length) {
-  const s = offsetToLineCol(text, offset);
-  const e = offsetToLineCol(text, offset + length);
+function span(idx, offset, length) {
+  const s = offsetToLineCol(idx, offset);
+  const e = offsetToLineCol(idx, offset + length);
   return { line: s.line, column: s.column, endLine: e.line, endColumn: e.column };
 }
 function checkBrackets(text, out) {
@@ -518,7 +529,7 @@ function checkDirectives(text, out) {
     }
   }
 }
-function checkPermutationConfigs(text, out) {
+function checkPermutationConfigs(text, idx, out) {
   for (const m of text.matchAll(/\[<([^>]*?)>/gu)) {
     const configStr = m[1] ?? "";
     if (!/\w+\s*=/.test(configStr)) continue;
@@ -529,7 +540,7 @@ function checkPermutationConfigs(text, out) {
         out.push(err(
           "permutation.unknown-key",
           `Unknown permutation config key: '${km[1]}'.`,
-          { ...span(text, configBase + (km.index ?? 0), (km[1] ?? "").length), data: { key: km[1] } }
+          { ...span(idx, configBase + (km.index ?? 0), (km[1] ?? "").length), data: { key: km[1] } }
         ));
       }
     }
@@ -538,7 +549,7 @@ function checkPermutationConfigs(text, out) {
       out.push(err(
         "permutation.minsize-not-integer",
         `minsize must be a positive integer, got '${min[1]}'.`,
-        { ...span(text, configBase + min.index, min[0].length), data: { value: min[1] } }
+        { ...span(idx, configBase + min.index, min[0].length), data: { value: min[1] } }
       ));
     }
     const max = /maxsize\s*=\s*([^;>\s]+)/i.exec(configStr);
@@ -546,17 +557,17 @@ function checkPermutationConfigs(text, out) {
       out.push(err(
         "permutation.maxsize-not-integer",
         `maxsize must be a positive integer, got '${max[1]}'.`,
-        { ...span(text, configBase + max.index, max[0].length), data: { value: max[1] } }
+        { ...span(idx, configBase + max.index, max[0].length), data: { value: max[1] } }
       ));
     }
   }
 }
-function checkPlurals(text, locale, out) {
+function checkPlurals(text, idx, locale, out) {
   const base = locale && locale !== "" ? normalizeBaseLang(locale) : "";
   const arity = base !== "" ? pluralArity(base) : 0;
   const tainted = macroTaintedNames(text);
   for (const block of findPluralBlocks(text)) {
-    const at = span(text, block.start, block.end - block.start);
+    const at = span(idx, block.start, block.end - block.start);
     for (const m of block.countSlot.matchAll(/%(\w+)%/gu)) {
       const name = (m[1] ?? "").toLowerCase();
       if (!tainted.has(name)) continue;
@@ -586,25 +597,34 @@ var UNRESOLVED_AT_PLURAL_TIME = /\[|\{(?!\?)/u;
 function macroTaintedNames(text) {
   const macros = extractDirectives(text).setDefs;
   const tainted = /* @__PURE__ */ new Set();
+  const queue = [];
   for (const [name, value] of Object.entries(macros)) {
-    if (UNRESOLVED_AT_PLURAL_TIME.test(value)) tainted.add(name);
+    if (UNRESOLVED_AT_PLURAL_TIME.test(value)) {
+      tainted.add(name);
+      queue.push(name);
+    }
   }
-  let grew = true;
-  while (grew) {
-    grew = false;
-    for (const [name, value] of Object.entries(macros)) {
-      if (tainted.has(name)) continue;
-      for (const m of value.matchAll(/%(\w+)%/gu)) {
-        if (!tainted.has((m[1] ?? "").toLowerCase())) continue;
+  const dependents = /* @__PURE__ */ new Map();
+  for (const [name, value] of Object.entries(macros)) {
+    for (const m of value.matchAll(/%(\w+)%/gu)) {
+      const ref = (m[1] ?? "").toLowerCase();
+      const list = dependents.get(ref);
+      if (list === void 0) dependents.set(ref, [name]);
+      else list.push(name);
+    }
+  }
+  while (queue.length > 0) {
+    const source = queue.pop();
+    for (const name of dependents.get(source) ?? []) {
+      if (!tainted.has(name)) {
         tainted.add(name);
-        grew = true;
-        break;
+        queue.push(name);
       }
     }
   }
   return tainted;
 }
-function checkVariableReferences(text, known, out) {
+function checkVariableReferences(text, idx, known, out) {
   const knownSet = new Set((known ?? []).map((n) => n.toLowerCase()));
   const defs = /* @__PURE__ */ new Map();
   const defPos = /* @__PURE__ */ new Map();
@@ -612,7 +632,7 @@ function checkVariableReferences(text, known, out) {
     const name = (m[1] ?? "").toLowerCase();
     defs.set(name, m[2] ?? "");
     const nameOffset = (m.index ?? 0) + m[0].indexOf("%");
-    defPos.set(name, span(text, nameOffset, name.length + 2));
+    defPos.set(name, span(idx, nameOffset, name.length + 2));
   }
   const somewhere = { line: 1, column: 1 };
   for (const [name, value] of defs) {
@@ -620,8 +640,15 @@ function checkVariableReferences(text, known, out) {
       out.push(err("variable.self-reference", `Variable '${name}' references itself.`, defPos.get(name) ?? somewhere));
     }
   }
+  const refsOf = /* @__PURE__ */ new Map();
+  for (const [name, value] of defs) {
+    const refs = [];
+    for (const m of value.matchAll(/%(\w+)%/gu)) refs.push((m[1] ?? "").toLowerCase());
+    refsOf.set(name, refs);
+  }
+  const reachesCycle = namesThatReachACycle(defs, refsOf);
   for (const name of defs.keys()) {
-    detectCycle(name, defs, [name], defPos.get(name) ?? somewhere, out);
+    detectCycle(name, defs, refsOf, reachesCycle, defPos.get(name) ?? somewhere, out);
   }
   const body = text.replace(/^[ \t]*#(?:set|def)[ \t]+%\w+%[ \t]*=[ \t]*.*?$/gmu, (m) => m.replace(/[^\n]/g, " "));
   const seen = /* @__PURE__ */ new Set();
@@ -632,7 +659,7 @@ function checkVariableReferences(text, known, out) {
     out.push(warn(
       "variable.undefined",
       `Variable '${name}' is not defined \u2014 may be a runtime variable.`,
-      { ...span(text, offset, length), data: { name } }
+      { ...span(idx, offset, length), data: { name } }
     ));
   };
   for (const m of body.matchAll(/%(\w+)%/gu)) undefinedAt(m[1] ?? "", m.index ?? 0, m[0].length);
@@ -640,19 +667,70 @@ function checkVariableReferences(text, known, out) {
     undefinedAt(m[1] ?? "", (m.index ?? 0) + m[0].indexOf(m[1] ?? ""), (m[1] ?? "").length);
   }
 }
-function detectCycle(current, defs, visited, rootPos, out) {
-  const value = defs.get(current) ?? "";
-  for (const m of value.matchAll(/%(\w+)%/gu)) {
-    const ref = (m[1] ?? "").toLowerCase();
-    if (ref === current) continue;
-    if (visited.includes(ref)) {
-      out.push(err("variable.circular-reference", `Circular variable reference: ${[...visited, ref].join(" \u2192 ")}.`, rootPos));
-      return;
+function namesThatReachACycle(defs, refsOf) {
+  const GREY = 1;
+  const BLACK = 2;
+  const color = /* @__PURE__ */ new Map();
+  const reaches = /* @__PURE__ */ new Set();
+  for (const root of defs.keys()) {
+    if (color.has(root)) continue;
+    const stack = [{ name: root, refs: refsOf.get(root) ?? [], i: 0 }];
+    color.set(root, GREY);
+    while (stack.length > 0) {
+      const top = stack[stack.length - 1];
+      if (top.i < top.refs.length) {
+        const ref = top.refs[top.i];
+        top.i += 1;
+        if (ref === top.name || !defs.has(ref)) continue;
+        const c = color.get(ref);
+        if (c === GREY) reaches.add(top.name);
+        else if (c === BLACK) {
+          if (reaches.has(ref)) reaches.add(top.name);
+        } else {
+          stack.push({ name: ref, refs: refsOf.get(ref) ?? [], i: 0 });
+          color.set(ref, GREY);
+        }
+      } else {
+        stack.pop();
+        color.set(top.name, BLACK);
+        if (stack.length > 0 && reaches.has(top.name)) reaches.add(stack[stack.length - 1].name);
+      }
+    }
+  }
+  return reaches;
+}
+function detectCycle(root, defs, refsOf, reachesCycle, rootPos, out) {
+  if (!reachesCycle.has(root)) return;
+  const path = [root];
+  const onPath = /* @__PURE__ */ new Set([root]);
+  const stack = [{ name: root, refs: refsOf.get(root) ?? [], i: 0 }];
+  const leave = (frame) => {
+    stack.pop();
+    onPath.delete(frame.name);
+    path.pop();
+  };
+  while (stack.length > 0) {
+    const top = stack[stack.length - 1];
+    if (top.i >= top.refs.length) {
+      leave(top);
+      continue;
+    }
+    const ref = top.refs[top.i];
+    top.i += 1;
+    if (ref === top.name) continue;
+    if (onPath.has(ref)) {
+      out.push(err("variable.circular-reference", `Circular variable reference: ${[...path, ref].join(" \u2192 ")}.`, rootPos));
+      leave(top);
+      continue;
+    }
+    if (defs.has(ref) && reachesCycle.has(ref)) {
+      stack.push({ name: ref, refs: refsOf.get(ref) ?? [], i: 0 });
+      onPath.add(ref);
+      path.push(ref);
     }
-    if (defs.has(ref)) detectCycle(ref, defs, [...visited, ref], rootPos, out);
   }
 }
-function checkIncludeTargets(text, known, out) {
+function checkIncludeTargets(text, idx, known, out) {
   const set = new Set(known);
   INCLUDE_RE.lastIndex = 0;
   for (const m of text.matchAll(INCLUDE_RE)) {
@@ -662,7 +740,7 @@ function checkIncludeTargets(text, known, out) {
       out.push(err(
         "include.unknown-target",
         `#include target '${ref}' does not match any known template.`,
-        { ...span(text, refOffset, ref.length), data: { target: ref } }
+        { ...span(idx, refOffset, ref.length), data: { target: ref } }
       ));
     }
   }
@@ -1038,8 +1116,9 @@ var SINGLE_ABBREVS = [
 var WS = " \\t\\r\\n\\f\\x0B";
 var S = `[${WS}]`;
 var DOMAIN_PART = "(?:(?:(?:xn--)?[\\p{L}\\p{N}]+(?:-[\\p{L}\\p{N}]+)*)\\.)+(?:xn--[a-z0-9\\-]{2,59}|[\\p{L}][\\p{L}\\p{N}-]{1,62})";
-var URL_RE = new RegExp(`(?:https?|ftp):\\/\\/[^${WS}<>"')\\]]+`, "giu");
-var MAILTEL_RE = new RegExp(`(?:mailto|tel):[^${WS}<>"')\\]]+`, "giu");
+var URI_BODY = `[^\\x00${WS}<>"')\\]]`;
+var URI_RE = new RegExp(`(?:(?:https?|ftp):\\/\\/|(?:mailto|tel):)${URI_BODY}+`, "giu");
+var MAILTEL_PREFIX_RE = /^(?:mailto|tel):/iu;
 var EMAIL_RE = new RegExp(`[a-z0-9._%+\\-]+@${DOMAIN_PART}\\b`, "giu");
 var DOMAIN_RE = new RegExp(`\\b${DOMAIN_PART}\\b`, "giu");
 var DECIMAL_RE = /\b\d+\.\d+\b/gu;
@@ -1060,6 +1139,18 @@ var CAP_AFTER_BLOCK_RE = new RegExp(
 );
 var CAP_AFTER_BREAK_RE = new RegExp(`(\\n${LEAD})(\\p{Ll})`, "gu");
 var up = (ch) => ch.toUpperCase();
+var SHIELD_PREFIXES = ["URL", "URI", "EMAIL", "DOM", "NUM", "ABBR"];
+var RESTORE_RE2 = new RegExp(`\\x00(?:${SHIELD_PREFIXES.join("|")})_\\d+\\x00`, "gu");
+function restore(text, input, placeholders) {
+  if (!input.includes("\0")) {
+    return text.replace(RESTORE_RE2, (key) => placeholders.get(key) ?? key);
+  }
+  let out = text;
+  for (const [key, value] of placeholders) {
+    out = out.split(key).join(value);
+  }
+  return out;
+}
 function postProcess(input) {
   const placeholders = /* @__PURE__ */ new Map();
   let counter = 0;
@@ -1079,8 +1170,10 @@ function postProcess(input) {
     return store(value, prefix);
   };
   let text = input;
-  text = text.replace(URL_RE, (m) => storeWithTrailingPunct(m, "URL"));
-  text = text.replace(MAILTEL_RE, (m) => storeWithTrailingPunct(m, "URI"));
+  text = text.replace(
+    URI_RE,
+    (m) => storeWithTrailingPunct(m, MAILTEL_PREFIX_RE.test(m) ? "URI" : "URL")
+  );
   text = text.replace(EMAIL_RE, (m) => store(m, "EMAIL"));
   text = text.replace(DOMAIN_RE, (m) => store(m, "DOM"));
   text = text.replace(DECIMAL_RE, (m) => store(m, "NUM"));
@@ -1095,10 +1188,7 @@ function postProcess(input) {
   text = text.replace(CAP_AFTER_SENTENCE_RE, (_m, p, gap, ch) => p + gap + up(ch));
   text = text.replace(CAP_AFTER_BLOCK_RE, (_m, tag, ch) => tag + up(ch));
   text = text.replace(CAP_AFTER_BREAK_RE, (_m, br, ch) => br + up(ch));
-  for (const [key, value] of placeholders) {
-    text = text.split(key).join(value);
-  }
-  return text.trim();
+  return restore(text, input, placeholders).trim();
 }
 
 // src/internal/pipeline.ts