@spintax/core 0.3.0 → 0.3.2

package/CHANGELOG.md

@@ -3,6 +3,98 @@
 All notable changes to `@spintax/core` are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.2 — 2026-07-23
+
+Post-process robustness and scaling: `postProcess()` no longer emits its own U+0000 sentinel, and
+the placeholder restore is linear instead of quadratic. Output is unchanged on ordinary text; the
+two behaviour changes are on shapes that previously produced invalid output (a raw U+0000).
+
+### Fixed
+
+- **`postProcess()` no longer emits a raw U+0000** into the returned text — on input carrying
+  none. A URI body runs to the first delimiter, so the URL rule and the `mailto:`/`tel:` rule
+  overlap whenever one URI carries the other's scheme, and shielding them in two passes let the
+  second run into a placeholder the first had minted:
+
+  ```js
+  render('mailto:sales@example.com?body=see%20https://shop.example.com/cart');
+  // 'mailto:sales@example.com?body=see%20\x00URL_0\x00'  ← before this release
+  ```
+
+  The swallowed key never restored, so the engine returned its own placeholder delimiter: illegal
+  in an XML document, replaced with U+FFFD by an HTML parser, rejected by PostgreSQL in
+  `text`/`varchar` — and a live key again as soon as an edit detaches it from the `mailto:` prefix
+  that was shielding it, at which point the next render substitutes an unrelated URL into the
+  contact link. It also silently disabled the linear restore below, whose guard is
+  `input.includes('\x00')`.
+
+  The two rules are now **one alternation**, so the leftmost match takes the whole token whichever
+  scheme it is. Reordering the passes was the other candidate and is not equivalent: it moves the
+  damage onto a URL whose path carries a `mailto:`, where the leading half then loses its trailing
+  dot to the punctuation pass. Both directions are fixtures now, the second one negative. U+0000 is
+  also excluded from the URI body class, for a caller-supplied one.
+
+  Mirrored into the WordPress plugin engine, and gated by three golden-corpus fixtures — the
+  cross-engine contract, so `spintax-php`, `spintax-py` and `spintax-win` inherit the gate.
+  ([#53](https://github.com/investblog/spintax-js/issues/53))
+
+### Performance
+
+- **Post-process no longer goes quadratic on shield-heavy text.** The restore step replaced each
+  shielded placeholder across the whole text one key at a time — O(text × placeholders) — and
+  since URLs, `mailto:`/`tel:` URIs, emails, domains, decimals and abbreviations are all shielded,
+  the placeholder count grows with the input. The stage came to dominate the render:
+
+  | input  | `postProcess: false` | before     | after   |
+  |--------|----------------------|------------|---------|
+  | 47 KB  | 0.007 s              | 0.18 s     | 0.014 s |
+  | 189 KB | 0.006 s              | 3.06 s     | 0.057 s |
+  | 756 KB | 0.081 s              | **43.9 s** | 0.25 s  |
+
+  A single left-to-right pass replaces the loop, behind a guard on the input. The two are not the
+  same function: the loop is a repeated *substring* substitution, so it rewrites every occurrence
+  of a key rather than the one the shield placed. Most of the disagreement needs a literal `\x00`
+  from the caller, and the guard sends that input to the original loop.
+
+  **One shape survives the guard**, and on it the behaviour changed deliberately: two adjacent
+  placeholders can sandwich caller text that spells a key, so one token's closing delimiter, that
+  text, and the next token's opening delimiter form a third occurrence of a real key. Rendering
+  `https://a.io e.g. URL_0mailto:x@y.io` — no `\x00` anywhere in it — the loop substituted the
+  forgery, destroyed two real tokens and returned raw `\x00` bytes; the single pass returns the
+  text intact. Measured over 456 976 probes, 12 `\x00`-free inputs distinguish the two restores and
+  the loop emits a raw `\x00` on all 12. A corpus fixture now pins the surviving answer, since the
+  engines disagreed on it. `npm run bench:postprocess` records the scaling.
+  ([#52](https://github.com/investblog/spintax-js/issues/52),
+  [#54](https://github.com/investblog/spintax-js/issues/54))
+
+## 0.3.1 — 2026-07-21
+
+Author markup is sanitised in one place, so a handle renders exactly like its source.
+
+### Fixed
+
+- **`render(parse(src))` no longer diverges from `render(src)`** when the template contains an
+  author-typed engine sentinel (the reserved range U+E000–U+E005). The strip that keeps stray
+  sentinels out of a tree lived at the render entry points, so `parse()` and `analyze(str)` — two
+  of the three doors into the parser — skipped it, and the mandatory safety-restore then rewrote
+  the author's character into a structural glyph they never wrote:
+
+  ```js
+  const src = `a${String.fromCharCode(0xe000)}b`;
+  render(src, { postProcess: false }); // "ab"
+  render(parse(src), { postProcess: false }); // "a{b"  ← before this release
+  ```
+
+  The strip now lives in `parseTemplate`, the single door from author source into an AST, so all
+  three entry points agree. `parseSequence` is deliberately **not** sanitised — it re-parses a
+  variable's *value*, where sentinels a host `neutralize()`d are legitimate and must survive to the
+  restore — and `ParsedAst.source` still keeps the original bytes so diagnostics point at what was
+  typed. ([#51](https://github.com/investblog/spintax-js/issues/51))
+
+  Templates that contain no reserved-range characters are unaffected: the strip is a no-op on them.
+  The Python port fixed the same defect the same way; the PHP engines never had it (no `parse()`
+  handle, no PUA sentinels).
+
 ## 0.3.0 — 2026-07-19
 
 `#set` goes back to being a macro and a new `#def` carries roll-once. Breaking: it changes what

package/README.md

@@ -153,7 +153,7 @@ render('%bio%', { context: { bio: neutralize('Save {50|60}% today') }, postProce
 
 With a `seed`, `render` is reproducible within this engine. Cross-engine RNG-sequence parity
 with the PHP plugin is a **non-goal** — only the *deterministic* behavior (validation verdicts,
-plural buckets, conditional truthiness, `#set` collapse, post-process output) is parity-gated.
+plural buckets, conditional truthiness, `#set`/`#def` semantics, post-process output) is parity-gated.
 
 ## Links
 

package/dist/index.cjs

@@ -23,13 +23,38 @@ function isParsedAst(value) {
   return typeof value === "object" && value !== null && value.astVersion === AST_VERSION && typeof value.source === "string" && Array.isArray(value.nodes);
 }
 
+// src/internal/neutralize.ts
+var STRUCTURAL = ["{", "}", "[", "]", "%", "#"];
+var SENTINEL_BASE = 57344;
+var SHIELD = /* @__PURE__ */ new Map();
+var RESTORE = /* @__PURE__ */ new Map();
+STRUCTURAL.forEach((ch, i) => {
+  const sentinel = String.fromCharCode(SENTINEL_BASE + i);
+  SHIELD.set(ch, sentinel);
+  RESTORE.set(sentinel, ch);
+});
+var SHIELD_RE = /[{}[\]%#]/gu;
+var RESTORE_RE = new RegExp(
+  `[${String.fromCharCode(SENTINEL_BASE)}-${String.fromCharCode(SENTINEL_BASE + STRUCTURAL.length - 1)}]`,
+  "gu"
+);
+function neutralize(value) {
+  return value.replace(SHIELD_RE, (ch) => SHIELD.get(ch) ?? ch);
+}
+function safetyRestore(text) {
+  return text.replace(RESTORE_RE, (ch) => RESTORE.get(ch) ?? ch);
+}
+function stripSentinels(text) {
+  return text.replace(RESTORE_RE, "");
+}
+
 // src/internal/parser.ts
 var VARIABLE_RE = /^%(\w+)%/;
 var DIRECTIVE_RE = /^[ \t]*#(set|def)[ \t]+%(\w+)%[ \t]*=[ \t]*(.*?)[ \t]*\r?$/gmu;
 var CONDITIONAL_NAME_RE = /^[A-Za-z_]\w*/;
 var PLURAL_PREFIX = "plural ";
 function parseTemplate(src) {
-  const { body, setDefs, defDefs } = extractDirectives(stripComments(src));
+  const { body, setDefs, defDefs } = extractDirectives(stripComments(stripSentinels(src)));
   return { astVersion: AST_VERSION, source: src, setDefs, defDefs, nodes: parseSequence(body) };
 }
 function extractDirectives(text) {
@@ -702,31 +727,6 @@ var NotImplementedError = class extends SpintaxError {
   }
 };
 
-// src/internal/neutralize.ts
-var STRUCTURAL = ["{", "}", "[", "]", "%", "#"];
-var SENTINEL_BASE = 57344;
-var SHIELD = /* @__PURE__ */ new Map();
-var RESTORE = /* @__PURE__ */ new Map();
-STRUCTURAL.forEach((ch, i) => {
-  const sentinel = String.fromCharCode(SENTINEL_BASE + i);
-  SHIELD.set(ch, sentinel);
-  RESTORE.set(sentinel, ch);
-});
-var SHIELD_RE = /[{}[\]%#]/gu;
-var RESTORE_RE = new RegExp(
-  `[${String.fromCharCode(SENTINEL_BASE)}-${String.fromCharCode(SENTINEL_BASE + STRUCTURAL.length - 1)}]`,
-  "gu"
-);
-function neutralize(value) {
-  return value.replace(SHIELD_RE, (ch) => SHIELD.get(ch) ?? ch);
-}
-function safetyRestore(text) {
-  return text.replace(RESTORE_RE, (ch) => RESTORE.get(ch) ?? ch);
-}
-function stripSentinels(text) {
-  return text.replace(RESTORE_RE, "");
-}
-
 // src/internal/render.ts
 var MAX_VARIABLE_DEPTH = 50;
 var INCLUDE_LINE_RE = /^[ \t]*#include[ \t\n\r\f\x0B]+"([^"]+)"[ \t\n\r\f\x0B]*$/gmu;
@@ -753,7 +753,7 @@ function resolveIncludes(text, ctx) {
       throw new IncludeResolverError(`includeResolver threw for "${ref}"`, { cause });
     }
     if (included === null) return "";
-    return renderAst(parseTemplate(stripSentinels(included)), {
+    return renderAst(parseTemplate(included), {
       ...ctx,
       runtimeContext: ctx.runtimeContext,
       // child inherits runtime, not parent #set
@@ -879,7 +879,7 @@ function renderPlural(node, opts) {
   if (/[{}[\]]/u.test(formsRaw)) {
     report({
       code: "plural.nested-brackets",
-      message: "Plural form slot contains nested spintax brackets; extract via #set first.",
+      message: "Plural form slot contains nested spintax brackets; extract via #def first \u2014 a #set is substituted verbatim and would put the brackets straight back.",
       construct: rawConstruct(countRaw, formsRaw),
       locale: base
     });
@@ -1038,8 +1038,9 @@ var SINGLE_ABBREVS = [
 var WS = " \\t\\r\\n\\f\\x0B";
 var S = `[${WS}]`;
 var DOMAIN_PART = "(?:(?:(?:xn--)?[\\p{L}\\p{N}]+(?:-[\\p{L}\\p{N}]+)*)\\.)+(?:xn--[a-z0-9\\-]{2,59}|[\\p{L}][\\p{L}\\p{N}-]{1,62})";
-var URL_RE = new RegExp(`(?:https?|ftp):\\/\\/[^${WS}<>"')\\]]+`, "giu");
-var MAILTEL_RE = new RegExp(`(?:mailto|tel):[^${WS}<>"')\\]]+`, "giu");
+var URI_BODY = `[^\\x00${WS}<>"')\\]]`;
+var URI_RE = new RegExp(`(?:(?:https?|ftp):\\/\\/|(?:mailto|tel):)${URI_BODY}+`, "giu");
+var MAILTEL_PREFIX_RE = /^(?:mailto|tel):/iu;
 var EMAIL_RE = new RegExp(`[a-z0-9._%+\\-]+@${DOMAIN_PART}\\b`, "giu");
 var DOMAIN_RE = new RegExp(`\\b${DOMAIN_PART}\\b`, "giu");
 var DECIMAL_RE = /\b\d+\.\d+\b/gu;
@@ -1060,6 +1061,18 @@ var CAP_AFTER_BLOCK_RE = new RegExp(
 );
 var CAP_AFTER_BREAK_RE = new RegExp(`(\\n${LEAD})(\\p{Ll})`, "gu");
 var up = (ch) => ch.toUpperCase();
+var SHIELD_PREFIXES = ["URL", "URI", "EMAIL", "DOM", "NUM", "ABBR"];
+var RESTORE_RE2 = new RegExp(`\\x00(?:${SHIELD_PREFIXES.join("|")})_\\d+\\x00`, "gu");
+function restore(text, input, placeholders) {
+  if (!input.includes("\0")) {
+    return text.replace(RESTORE_RE2, (key) => placeholders.get(key) ?? key);
+  }
+  let out = text;
+  for (const [key, value] of placeholders) {
+    out = out.split(key).join(value);
+  }
+  return out;
+}
 function postProcess(input) {
   const placeholders = /* @__PURE__ */ new Map();
   let counter = 0;
@@ -1079,8 +1092,10 @@ function postProcess(input) {
     return store(value, prefix);
   };
   let text = input;
-  text = text.replace(URL_RE, (m) => storeWithTrailingPunct(m, "URL"));
-  text = text.replace(MAILTEL_RE, (m) => storeWithTrailingPunct(m, "URI"));
+  text = text.replace(
+    URI_RE,
+    (m) => storeWithTrailingPunct(m, MAILTEL_PREFIX_RE.test(m) ? "URI" : "URL")
+  );
   text = text.replace(EMAIL_RE, (m) => store(m, "EMAIL"));
   text = text.replace(DOMAIN_RE, (m) => store(m, "DOM"));
   text = text.replace(DECIMAL_RE, (m) => store(m, "NUM"));
@@ -1095,10 +1110,7 @@ function postProcess(input) {
   text = text.replace(CAP_AFTER_SENTENCE_RE, (_m, p, gap, ch) => p + gap + up(ch));
   text = text.replace(CAP_AFTER_BLOCK_RE, (_m, tag, ch) => tag + up(ch));
   text = text.replace(CAP_AFTER_BREAK_RE, (_m, br, ch) => br + up(ch));
-  for (const [key, value] of placeholders) {
-    text = text.split(key).join(value);
-  }
-  return text.trim();
+  return restore(text, input, placeholders).trim();
 }
 
 // src/internal/pipeline.ts
@@ -1113,7 +1125,7 @@ function renderWith(input, rng, opts = {}) {
     includeStack: [],
     onPluralError: opts.onPluralError
   };
-  const ast = resolveAst(typeof input === "string" ? stripSentinels(input) : input);
+  const ast = resolveAst(input);
   let out = renderAst(ast, ctx);
   if (opts.postProcess !== false) out = postProcess(out);
   return safetyRestore(out);