@spintax/core 0.7.0 → 0.9.0

package/CHANGELOG.md

@@ -3,6 +3,441 @@
 All notable changes to `@spintax/core` are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.9.0 — 2026-09-17
+
+### Changed, visibly
+
+- **A TLD is a label in ONE case (#79).** The bare-domain and email shields took any `word.Word` for a
+  domain, so a sentence glued to the next one kept its missing space and its lower-case start:
+  `kept compact.Game categories`, `конец.Начало`. On a production host's stored content that shape was
+  a sentence start 369 times in 56 tenants and a domain never. Now a TLD is all lower case or all upper
+  case — letters without case (CJK, Arabic, Thai) fit either — so those sentences get their space and
+  their capital, while `example.com`, `ASP.NET`, `info@Example.COM`, `ПРИМЕР.РФ`, `例子.中国` and a punycode
+  TLD in any case stay whole. The accepted cost: a Title-case second half reads as a sentence too —
+  `Yandex.Money` renders `Yandex. Money`, and `info@example.Com` is no longer shielded as an email. Both
+  PHP engines take the same rule.
+
+### Fixed
+
+- **A titlecase letter after a block tag stays as PHP leaves it.** The plugin writes that capitalizer
+  `/ui`, and PCRE2 does not fold a Unicode property, so its `\p{Ll}` stays lower case only. This engine
+  read `\p{Ll}` under JavaScript's `i`, which takes every cased letter, and `<p>Dživot</p>` became
+  `<p>DŽivot</p>`. Found by the post-process differential once its alphabet held a titlecase digraph; as
+  old as the capitalizer.
+
+### Corpus
+
+Nine new cases in `render-postprocess.json`, every expectation taken from both PHP engines with the rule
+applied: `postprocess/title-case-tld-is-a-sentence` (`compact.Game`), `postprocess/mixed-case-tld-is-not-a-domain`
+(`cOM` — one case means the whole label), `postprocess/title-case-email-tld-is-not-shielded` (the cost,
+pinned so nobody files it), the negatives `postprocess/uppercase-tld-is-a-domain`,
+`postprocess/uppercase-email-domain-is-shielded`, `postprocess/caseless-script-tld-is-a-domain`,
+`postprocess/uppercase-punycode-tld-is-a-domain` and `postprocess/punycode-tld-folds-long-s-and-kelvin`, and
+`postprocess/titlecase-letter-after-block-tag-is-kept`. `postprocess/glued-cyrillic-sentence-is-a-domain`
+becomes `postprocess/glued-cyrillic-sentence-is-spaced`: it pinned the reading before #79, and its note said
+deciding #79 would move it.
+
+### Notes
+
+**One JavaScript trap behind both.** Under `i`, a JavaScript `\p{Ll}` matches capitals and a `\p{Lu}`
+matches lower case; PCRE2 leaves a property alone under `/i`. PHP can also switch `i` off for one
+alternative with `(?-i:…)`, which JavaScript has only from ES2025 — Node 24, not the Node 18–22 this engine
+still supports. So the domain
+patterns carry no `i` and spell out the one part that is caseless, the punycode prefix and class — U+017F
+and U+212A included, as PCRE2's caseless `[a-z]` takes them.
+
+**Verified.** 24 000 generated post-process inputs — the earlier alphabet with caseless scripts, mixed-case
+TLDs and a titlecase digraph added, plus dotted chains whose TLD candidate is lower, upper, title, mixed or
+caseless, in prose and in emails — through both PHP engines with the rule applied: they agree with each other
+everywhere, this engine differs on none but the recorded final `trim`, and 0.8.0 on 4 552. Control mutations:
+`i` restored on either shield, the caseless letters dropped from one branch or both, the any-case TLD restored,
+U+017F and U+212A dropped from the punycode class — each turns the suite red. The last one first looked
+output-equivalent here, because a lower-case `xn--` prefix always has a one-case letter reading from the same
+start; a mixed-case prefix has none (`site.xN--ſA`, found by the Codex gate), and
+`postprocess/punycode-tld-folds-long-s-and-kelvin` pins it. Post-process cost is unchanged — 1 000 prose calls
+18 ms against 19, HTML blocks 27 against 26, a 756 KB shield-heavy text 147 ms against 154.
+
+**The PHP engines skip what they cannot shield.** Rejecting `Game` made `a.a.…a.Game` — one domain from its
+first start before — a chain PCRE retried from every label: 2 000 one-character labels (about 4 KB) took 48 ms where they had taken
+0.7 (found by the Codex gate). Both PHP shields now take a run or a chain they cannot shield and
+`(*SKIP)(*FAIL)` past it — the skips this engine's scanners already prove output-neutral — which also
+removes the older retry on a chain whose last label is too short to be a TLD: 2 000 labels, 39 ms before,
+0.1 ms now. Ordinary text costs 2–4 % more there. JavaScript has no backtracking verbs, which is why this
+engine's shields are scanners.
+
+**The two costs bounded by the source are gone** — 0.8.0's notes recorded them as left alone.
+`validate()` formatted a capped circular-reference message by walking the rest of the cycle again for each
+of its names: one cycle of 8 000 names took 6 s, now 68 ms, because the route lengths are measured once for
+all names. `render()` ordered `#def`s by rescanning every pending name each round and handed every roll a
+fresh copy of the map of rolled values: a 1 600-definition chain took 1.6 s and 6 400 independent
+definitions 4.1 s, now 6 ms and 24 ms. The order is counted round by round in source order, and every roll
+reads one map that grows by each rolled value, dropping the truthiness a conditional cached for it after
+each write. Output is byte-identical over 3 000 generated definition-graph documents — chains, cycles with
+tails long enough for a capped message, shadowed globals, outranked definitions, conditionals on definition
+names — through `validate`, `analyze` and five renders, with three control mutations caught first: a round
+left unsorted, the truthiness cache kept across a write, a route one name longer.
+
+## 0.8.0 — 2026-09-13
+
+Two defects found by the ports while they mirrored 0.7.0, both places where this engine read a
+template differently from the PHP engines — and both measured on those engines before anything
+changed: the character classes of the post-process, and the key that decides which constructs are
+re-read as text (#80).
+
+**The post-process reads characters the way PHP does.** PHP compiles every pattern of the cosmetic
+stage with `/u`, and `/u` turns on PCRE2_UCP: `\s`, `\d`, `\w` and `\b` are Unicode classes there.
+This engine took them for ASCII — the belief was written into its own comments ("no PCRE_UCP") — and
+the three ports that copied the classes did the same. Measured on both PHP engines:
+
+| input | 0.7.0 | both PHP engines, and now this one |
+|---|---|---|
+| `и т.д. и т.п. всё` | `И т. Д. И т. П. Всё` | `И т.д. и т.п. всё` |
+| `открой пример.рф сегодня` | `Открой пример. Рф сегодня` | `Открой пример.рф сегодня` |
+| `пишите на info@сайт.рф` | `Пишите на info@сайт. Рф` | `Пишите на info@сайт.рф` |
+| `end.<NBSP>next` | `End. <NBSP>next` | `End.<NBSP>Next` |
+| `в 5<NBSP>тыс.<NBSP>руб.` | `В 5<NBSP>тыс. <NBSP>руб.` | `В 5<NBSP>тыс.<NBSP>руб.` |
+
+**A conditional or a config reference inside `{…}`/`[…]` is text before the split, too (#80).** 0.7.0
+re-read a construct as text when a `%var%` sat directly in it, and decided that from a key narrower
+than its own rule:
+
+| template | 0.7.0 | both PHP engines, and now this one |
+|---|---|---|
+| `[<minsize=%n%;maxsize=%n%>a\|b\|c]`, `n=1` | all three elements | one element |
+| `[<sep=%S%>a\|b]`, `S=", "` | `b a` | `b, a` |
+| `[{?f?a\|b\|x}\|c]` | `c b\|x` — a raw pipe | three elements |
+| `[<sep=", ";lastsep=" and ">{?f?live casino}\|slots\|poker]` | `slots, poker and ` | `poker and slots` |
+| `[<sep=", ">slots\|{live casino\|}\|poker]`, the empty option picked | `slots, , poker` | `slots, poker` |
+
+Minor rather than patch: rendered text changes for every template of these shapes, and
+`AST_VERSION` moves to 4.
+
+### Fixed
+
+- **The post-process classes are PHP's UCP classes** (`internal/charclass.ts`). Whitespace is PCRE2's
+  `\s` — NBSP, the thin spaces, NEL — and not JavaScript's, which adds U+FEFF and misses U+0085 and
+  U+180E. `\b` counts a letter of any script as a word character. The digit in the two spacing
+  lookaheads is any `\p{Nd}`. The decimal shield stays ASCII: PHP writes that one pattern without `/u`.
+- **Conditional truthiness uses the same whitespace.** A value of U+FEFF alone is truthy (it was blank
+  here); a value of U+0085 alone is blank (it was truthy).
+- **The permutation-config patterns go the other way.** PHP writes them without `/u`, so their `\s` is
+  ASCII, while JavaScript's `\s` is Unicode on any flags. A no-break space around `=` is no longer
+  config whitespace — `[<minsize<NBSP>=<NBSP>1>a|b|c]` is one separator, as in PHP — and the validator
+  agrees: `[<foo<NBSP>=1>a|b]` is valid, `[<minsize=2<NBSP>>a|b]` is `permutation.minsize-not-integer`.
+- **A reference anywhere in a permutation's `<config>` marks it for the re-read** (#80). 0.7.0 tested
+  the PARSED `sep` and `lastsep`, where a size reference never arrives (a size that is not digits
+  parses to nothing) and neither does an unquoted separator (it parses to the default). The raw
+  header is tested now, so `minsize=%n%`, `maxsize=%n%` and `sep=%S%` take their values from the
+  context, as they always have in PHP. So does a whole `{?…}` there or in a per-element separator —
+  `[<lastsep="{?en? and | и }">a|b|c]` printed the raw conditional unless a neighbour happened to
+  trigger the re-read (found in review). A bare `{?` does not mark: the first cut of that check made
+  every level of `[<{?}>a|[<{?}>a|…]]` rescan its body for a conditional that was not there, twice
+  0.7.0's time on deep nesting (found by the Codex gate).
+- **A `{?…}` conditional directly in `{…}`/`[…]` marks the construct** (#80), whatever its branches
+  hold — 0.7.0 marked it only when a `%var%` sat in a branch. The plugin resolves conditionals at
+  Stage 6a, before any bracket is read, so a taken branch's `|` separates options, an empty branch
+  leaves an empty permutation element that is dropped, and whitespace at a branch's edge is the
+  element's edge. 0.7.0's notes kept the empty element on purpose as a pathological divergence; a
+  list item gated by a flag is an ordinary template, and in one it rendered `Есть покер, слоты и.`.
+- **A permutation element is its rendered text, trimmed, and one that renders empty is dropped** (#80).
+  The plugin resolves every nested enumeration before it splits a permutation, so an optional item
+  written `{live casino|}` that picks its empty option leaves `slots||poker` there — two elements —
+  while this engine kept a blank third and printed `slots, , poker`; `[<minsize=3;maxsize=3>a|{b|}|c]`
+  counted three and printed `a  c`. The conditional fix above did not reach it: the re-read drops a
+  part that is empty as TEXT, and `{b|}` is not empty until it is rendered. Found by the differential
+  below after the conditional half was in; the size pick and the shuffle count the elements that
+  remain, and an element that is neither empty nor padded draws as before.
+- **`AST_VERSION` 3 → 4.** The node shape is 3's, but an `Ast` cached by 0.7.0 lacks `raw` exactly
+  where the wider key puts it and would render the old output; the guard turns that into
+  `AstVersionError`. Nothing persists a handle across versions.
+- **Three patterns no longer scan a long run once per character** (found in review). Trimming each
+  assembled element with `/[…]+$/` retried from every position of a whitespace run inside the text —
+  the first cut of the element fix made `[a%v%y|z]` over 51 000 spaces take 3 s, and 13 s from a
+  362-byte macro template; the PHP trims are loops now, in the parser as well, where the same regex
+  already cost seconds on a re-read. And two post-process patterns that were quadratic IN 0.7.0 got
+  new triggers from the wider whitespace class: removing whitespace before punctuation rescanned a run
+  with no punctuation after it (100 000 form feeds: 10 s in 0.7.0; a macro of NBSP or U+3000: 10–31 s
+  on the first cut), and capitalizing after a line break rescanned a run of breaks and spaces with no
+  letter after it (20 000 of `\n` plus a space: 4 s in 0.7.0). Each now starts a match only where the
+  run starts — the same matches, checked over 500 000 generated strings — and all of these take
+  milliseconds.
+- **`parse()` and `render()` no longer throw on a long tag-shaped permutation config.** Whether
+  `[<name>…]` opens with an HTML tag was decided by compiling `</name\s*>` from the name itself, and V8
+  will not compile that past about 7.8 KB of name: `SyntaxError: Invalid regular expression … Stack
+  overflow`, straight out of `parse()`. A 7.8 KB source was enough — under the reference Worker's 8 KB
+  cap — and so were 333 bytes of `#set` doubling a letter inside a re-read config, which spell the name
+  at render time. §9.2 says the engine never throws on content; 0.7.0 threw on both.
+  The closing tag is found by a scan now, matching exactly what the pattern matched: any case, and the
+  two non-ASCII letters `iu` folds into ASCII (U+017F, U+212A). Found by the scaling probe behind the
+  note on template scans below.
+- **`constructor` and `__proto__` are ordinary variable names.** The variable and definition maps were
+  plain JavaScript objects, so both names found `Object`'s own members, and assigning `__proto__` stored
+  nothing. `render('a %constructor% b')` threw `TypeError` — §9.2 lets `render()` throw only on a resolver
+  or a foreign `Ast` — while `validate()` called the name merely undefined, so nothing could screen it out.
+  `%__proto__%` rendered as nothing, `{?constructor?…}` and `{?__proto__?…}` took the then-branch, a
+  `#set`, a `#def` or a context value under `__proto__` was lost, a runtime list under it spliced
+  `[object Object]`, and the validator withheld `plural.arity`, `plural.count-macro` and
+  `plural.nested-brackets` for a `#set` of either name — valid where both PHP validators say invalid.
+  Both PHP engines read variables from arrays and always treated them as ordinary names. The maps have
+  no prototype now (`internal/name-map.ts`). Found by the Codex gate on the nesting rewrite, where
+  `%__proto__%` had left the expansion budget `NaN`.
+
+### Changed, visibly
+
+- A no-break space behaves like a space around punctuation: removed before `,.;:!?`, no second space
+  inserted after one, and the next sentence capitalized through it. French typography with U+202F
+  before `!` or `?` loses it — as a plain `bonjour !` always has, in every engine.
+- A glued Cyrillic sentence stays glued: `конец.Начало` matches the bare-domain shield in PHP and now
+  here, exactly as `compact.Game` always has in ASCII. Whether the shield should take a capitalized
+  TLD at all is #79.
+
+### Corpus
+
+Twenty fixtures, every expectation taken from BOTH PHP engines (docker, real code): fifteen in
+`render-postprocess.json` — Cyrillic multi-dot abbreviations, an IDN domain, email and TLD, NBSP before
+and after punctuation and after a whitelisted abbreviation, a URL that ends at NBSP, an Arabic-Indic
+digit, NEL as whitespace and U+FEFF as not, a glued Cyrillic sentence, `_` before an abbreviation — plus
+`conditional/bom-only-is-truthy`, `conditional/nel-only-is-falsy`,
+`perm/config-nbsp-is-not-config-whitespace` and its two `validate/*` twins. Nineteen fail on 0.7.0; the
+twentieth is a negative guard. No earlier fixture put a non-ASCII character next to a boundary or a
+whitespace class — every domain, email and multi-dot case was Latin — which is how a premise stated in
+a comment stayed green for two months. `postprocess.test.ts` had pinned the mangled `т.д.` "in both
+engines", with an assertion that passed on the capital letter alone.
+
+Seven more `splice/*` fixtures for #80, again from both PHP engines: a size and an unquoted separator
+taken from a variable, a taken branch carrying a pipe in `[…]` and in `{…}`, a flag-gated list item,
+a branch trimmed at an element's edge — six fail on 0.7.0 — and the negative
+`splice/conditional-without-pipes-keeps-draws`, which pins that a conditional changing nothing
+structural draws exactly where 0.7.0 drew. Three `perm/*` fixtures for an element that renders empty or
+padded: `perm/nested-empty-option-drops-element`, `perm/nested-option-edge-whitespace-trimmed` and
+`perm/dropped-element-narrows-the-size-range`, all failing on 0.7.0. Four from the review, all failing on
+0.7.0: `splice/conditional-in-per-element-separator`, `splice/conditional-in-config-lastsep`, and the
+separator of a dropped element — `perm/dropped-element-takes-its-separator` (`[a<1>|{x|}|b]` is `a b`)
+and `perm/dropped-element-passes-on-its-trailing-separator` (`[a<1>|{x|}<2>|b]` is `a2b`).
+
+**The corpus can pin how many** (#74). `validate` cases take an optional `diagnosticCount` — the exact
+number of diagnostics per code — where the subset match used to be the only assertion, and that
+subset match is what passed two million circular-reference diagnostics for eleven days (#59). Two
+cases carry it: `validate/cycle-diamond-terminates` (22, one per name) and the new
+`validate/plural-count-macro-per-reference`, which records #73's decision: `plural.count-macro` once
+per tainted reference in the count slot, as this engine, `spintax-core` and both PHP validators already
+emit it. No engine output changed for either.
+
+Twelve fixtures for the prototype names, every expectation taken from both PHP engines: eight in
+`render-semantics.json` — undefined references, conditionals, a `#set` of each name, context keys, a
+`#def` rolled once, a `#def` naming them undefined, a runtime list spliced into a permutation, a form
+slot — and four in `validate.json`: the three withheld plural verdicts and the negative
+`validate/prototype-names-defined-and-undefined`. Eleven fail on 0.7.0. A JSON-decoded context carries
+`__proto__` as an ordinary key in every engine's harness; a JavaScript object literal would not.
+
+### Notes
+
+**Cost.** Against 0.7.0, the post-process of this whole batch is faster on prose and on HTML blocks and
+about a third slower on shield-heavy text — V8 runs a Unicode-class lookbehind more slowly than its ASCII
+`\b`, and every shield pattern carries one now. Per 1 000 calls, the four builds side by side:
+
+| input | 0.7.0 | the classes alone (regexes) | first linear scanners | now |
+|---|---|---|---|---|
+| prose, 857 characters | 34 ms | 32 ms | 108 ms | 19 ms |
+| HTML blocks, about 700 characters | 33 ms | 31 ms | 86 ms | 27 ms |
+| a URL, email, domain, decimal and abbreviation line, 237 characters | 36 ms | 46 ms | 70 ms | 49 ms |
+| the same line ×3 200, 756 KB, once | 113 ms | 151 ms | 223 ms | 158 ms |
+
+**The post-process is linear — and was a live denial of service in every release before this.** Eight
+of its passes were global regex replaces that a long run made retry from every start inside it: the
+email shield (one long word), the bare-domain shield (a dotted chain in any script), the
+trailing-punctuation cut of a URL, the space after a run of sentence marks, and the three capitalizers
+after a sentence end, a block tag and a line break (an unclosed `<`, a run of `<p>`). None of those
+runs needs a long template: 336 bytes of `#set` doubling one letter expand to 131 000 characters, whose
+post-process took 25 s, and the dotted, mark and tag units ran longer — so any host rendering untrusted
+templates with post-process on, the reference Worker and the hosted MCP server among them, could be held
+for as long as its CPU limit allowed. The expansion budget bounded the size of that text, never the time
+to post-process it. One more stage was quadratic on its own terms (found by the Codex gate): the restore
+of a text that carries U+0000, where #54 keeps the reference loop's reading — one `split`/`join` per
+placeholder — so a few hundred bytes doubling a NUL and a decimal took 2.1 s at 16 000 decimals, four
+times that per doubling.
+
+The shields and the capitalizers are scanners now. They run the plugin's own patterns at the same starts,
+in the same order, and skip only starts that provably fail. Every start inside one run of email-local
+characters meets the same `@` or none, so the email shield works back from each `@` — a text without one
+costs nothing. An attempt that fails at the start of a chain of labels fails at every later start in that
+chain, because the chain's own labels prefix any match further in, so the domain shield is one regex that
+matches at the starts it tries and consumes the chain when there is no domain — and is skipped when no dot
+in the text is followed by a label's first character. A lead has exactly one tokenization — a tag ends at
+the first `>` — so the capitalizers find their boundaries natively, walk a short lead, and build an index of
+where each lead ends, from the right, only when a lead holds a tag or runs past 32 characters; one index
+serves all three passes unless a capital grew the text (`ß` → `SS`). The URL cut is a loop, and the
+sentence-mark space starts a match only where its run starts. Output is byte-identical: 7.9 million strings
+against the regex implementation — exhaustive over small alphabets aimed at each pass, plus random soup — and
+5.8 million more aimed at the final form of each scanner, with every real control mutation caught, and the
+20 000-input post-process differential against both PHP engines unchanged at zero. Those macro templates now
+render half a megabyte to a megabyte in 0.1–0.2 s.
+
+The first cut of these scanners paid for linearity on every call: a regex call per word, a JavaScript step
+per character, three times — 3.4 times the regex passes on a short paragraph (the table above). The scaling
+bench, one large text, could not show that, and this note first said the bench was no slower on prose. The
+shapes above are what came after measuring a thousand short renders instead.
+
+The NUL-path restore computes the loop's result in one pass. An occurrence of a key is always a stretch
+of the original text between two `\x00`s — no stored value holds a `\x00`, and no whole stored value
+fits inside a key name — so the only way one replacement touches another is by taking a delimiter they
+share, and visiting the candidates in the loop's key order, each only while both its delimiters survive,
+is the loop. 1.9 million NUL-carrying strings come out identical, three control mutations were caught,
+and the 16 000-decimal case takes 130 ms.
+
+**The template scans are linear too — and part of that was live the same way.** They had the
+post-process's flaw. The parser matched each `{` and `[` by counting forward to its closer, which
+for an opener that never closes is the end of the text, from every such opener; the plural scan did the
+same for `{plural …}`, and the validator ran `/\[<([^>]*?)>/` from every `[<`. Lazy or overlapping
+patterns restarted from every character of a run in four more places: a `/# … #/` comment that never
+closes, a `#set`/`#def` value with a long whitespace run inside, a tag-shaped config holding a quoted
+`>`, and the validator's config-key test on a long word. The re-read of a construct hands the parser
+whatever its macros spell, so the parser's half was reachable from a few hundred bytes:
+
+| input (Node 22) | 0.7.0 | now |
+|---|---|---|
+| 333–341 bytes of `#set` spelling `[<`, `{?a?`, `[<li>`, `[<sep="` or `{plural 1:` inside `{…}` — render | 1.3–6.2 s, ×4 per doubling | 63–117 ms |
+| 387 bytes spelling form feeds into a tag-shaped config that holds a quoted `>` — render | 8.6 s, ×4 per doubling | 117 ms |
+| `[<` repeated, 32 KB — render / validate | 1.4 s / 0.27 s | 29 ms / 23 ms |
+| `{plural 1:` repeated, 32 KB — render / validate | 0.32 s / 0.20 s | 41 ms / 17 ms |
+| `/#a` repeated, 32 KB — parse / validate | 0.14 s / 0.14 s | 10 ms / 12 ms |
+| a `#set` value holding 32 KB of spaces — parse / validate | 2.8 s / 11.8 s | 4 ms / 14 ms |
+| `[<a`, 32 KB of spaces, a quoted `>` — parse | 2.0 s | 10 ms |
+| `[<` and a 32 KB word `>` — validate | 0.9 s | 10 ms |
+
+Brackets are paired once for the whole text, and the plural scan and the renderer's conditional pass use
+the same tables — the deep-nesting note below says how a construct is read now. The comment strip and the
+config scan are `indexOf` loops. The directive value ends at its last non-blank character instead of
+growing lazily, the tag pattern takes one whitespace character where two runs used to trade them, and a
+config key starts only where a word does — each matching exactly what it matched before. Output is
+byte-identical to the code before: 4.3 million generated strings through `parse`, `render` (three rng
+strategies, with and without post-process), `validate`, `analyze`, `extract` and the plural scan —
+exhaustive over alphabets aimed at each rewritten scan, plus random soup — with all twelve control
+mutations caught first.
+
+**Deep nesting no longer costs its depth again at every level — and the re-read had made that a live denial
+of service.** #68 kept nesting super-linear on the ground that depth costs source: at the hosted 8 KB cap,
+4 000 levels answered in about a second. 0.7.0's re-read took that ground away, because a construct re-read
+as text hands the parser whatever its macros spell (found by the Codex gate):
+
+| input (Node 22, the two measured side by side) | 0.7.0 | now |
+|---|---|---|
+| 705 bytes: `{%o15%x%c15%\|y}` over fifteen `#set` doublings of `{` and of `}` — render | 40 s | 0.14 s |
+| a `#def` rolling those 32 768 levels, referenced once — render | 40 s | 0.28 s |
+| `{a…x…b}` 16 384 levels deep, every level adding text — render | 19 s | 0.14 s |
+| `[a\|…x…]` 16 384 levels deep — render | 20 s | 0.24 s |
+| `{a%u%…x…}` 8 192 levels, each marked for the re-read by an undefined reference — render | 20 s | 0.10 s |
+| 20 000 nested conditionals naming a `#def` of 2^17 form feeds — render | 13 s | 0.30 s |
+| `{` 100 000 levels deep — parse | > 90 s | 0.21 s |
+
+Four costs were paid once per level, and each is gone:
+
+- **The parser read every construct's content in full** — to find its closer, to split it on top-level
+  pipes, to find a conditional's pipe, a config's end, a closing tag, a trailing separator — and handed its
+  children copies. A construct's children are spans of one text now, and every one of those questions is
+  answered for a span from tables built once over the text (`internal/text-index.ts`): bracket pairs; the
+  top-level pipes grouped by the brace and bracket totals in front of them, since `split_top_level`'s two
+  signed counters split exactly where both totals equal the ones at the span's start; a conditional's pipe
+  by jumping each opener to where one stack of both bracket kinds closes it, which is what the clamped
+  counter counts; the quote parity of every `>`; every closing tag by name; every `<` and `>`. Most of those
+  answers are a binary search in a sorted list of positions, so a template of n characters parses in
+  O(n log n) at worst rather than O(n) — the logarithm shows on many small siblings: 200 000 `{a|b}` parse in
+  150 ms against 0.7.0's 137, 100 000 `[<sep=",">a|b]` in 208 against 161 (found by the Codex gate).
+- **The walk joined each frame's output into a new string**, copying a construct's text once for every
+  level above it. Frames hand fragments up instead — joins, and windows for a trimmed permutation element,
+  every join knowing how many PHP trim characters sit at its ends — and the text is made once.
+- **A construct marked for the re-read ran both conditional passes and the fixpoint over its body to find
+  nothing to change**, once per marked level. The index answers first: the body can change only if it holds
+  a conditional the pass would resolve — a well-formed head whose brace closes inside it — or a reference
+  expansion would substitute, a name the variable map defines while the budget lasts. When neither, the
+  splice returns what it always returned. The conditional pass finds its heads once, too, instead of
+  searching the text again from the start of every span.
+- **Truthiness scanned a value's leading whitespace at every conditional**; it is computed once per
+  variable map.
+
+Output is byte-identical to the code before this change: the 4.3 million alphabet strings above again, and
+20 000 generated documents — long outputs, padded and empty elements, constructs marked for the re-read
+with defined, undefined and budget-starved references, conditionals in bodies, count slots and `#def`
+rolls, nests up to 300 deep — through eight probes each (`parse`, `validate`, `analyze`, four rng
+strategies with and without post-process, and a pre-parsed `Ast`), with the control mutations caught first:
+a join that stops counting trim characters after its first piece, one that never counts them at its end, a
+trim window a character late, a re-read check that forgets conditionals or inverted heads or compares names
+case-sensitively, one truthiness cache for every map, a conditional pass that skips a head at a span's start,
+top-level pipes grouped by one depth, a flipped quote parity, a closing tag on the span's end counted
+outside it. The PHP engines are no linear reference here: both resolve innermost-first with a
+full-text pass per level and stop at 10 000 levels with an exception — in `spintax/core`, 8 192 nested
+permutations took 5.9 s and 32 768 nested enumerations threw. Real templates did not pay for any of this —
+the render cost note below has the numbers. An `Ast` copied out of the process (it is not a serialization
+format) loses the side table the index lives in, and renders the same text by reading bodies as before.
+
+**Two costs bounded by the source, recorded rather than changed.** The same gate found `validate()`
+formatting a large definition cycle in quadratic time — each name walks the cycle to count what its
+printed route leaves out: 8 000 names 6.5 s, the 320 that fit in 8 KB 23 ms — and `render()` ordering
+`#def`s in quadratic-to-cubic time: 1 600 in a chain 2 s, 400 in 90 ms. A macro cannot spell either,
+both were so in 0.7.0, and the hosted surfaces cap a source at 8 KB.
+
+**Two PHP builds differ at the margin.** PCRE2 10.43 made non-spacing marks and connector punctuation
+word characters, so PHP 8.3 sees a boundary between `x` and U+0301 that PHP 8.4 does not. The corpus
+runs PHP 8.4 and this engine follows it; recorded in the conformance README.
+
+**What the wider re-read does not move.** A construct re-read because it holds a conditional draws
+exactly as the tree did whenever the taken branch changes nothing structural — measured on 0.7.0
+before the change, 500 seeds over five shapes (a permutation with separators, one with a size range,
+an enumeration, a nested permutation in a branch, an else branch): identical renders. What changes is
+the three kinds of text the plugin always produced here — a pipe in a branch, an empty element, a
+trimmed edge — and only in the renders where one occurs: a permutation whose elements all render
+non-empty and unpadded picks and shuffles exactly as before. `validate()` still reports `minsize=%n%`
+as `permutation.minsize-not-integer`, as both PHP validators do: a verdict about the template as
+written, unchanged.
+
+**Verified against PHP, not only against 0.7.0.** A generator of construct-heavy templates —
+conditionals with empty, piped and padded branches, references in configs and elements, macros
+carrying conditionals, per-element separators — 3 000 renders with `rng: first` and `last` through
+both PHP engines: 0.7.0 differed on 416, this engine on 0. The first cut of #80 still differed on
+147, every one an element that rendered empty through a nested construct; that is how the trim-and-drop
+above was found. And 20 000 generated post-process inputs (Latin, Cyrillic, marks, every space the two
+dialects disagree on, shield triggers): 0.7.0 differed on 6 229, this engine on none but the recorded
+final `trim`, which accounts for all 3 689 of its edge-only differences. A zero holds for what those
+generators can build, and is stated here as that.
+
+**On a production host's templates.** 2 522 template-and-flag combinations taken from a live
+deployment's migrations, 15 132 renders against 0.7.0: 30 change, and `validate`/`analyze`/`extract`
+change on none. They are the two defects, in real copy: Russian brand articles stop printing
+`Т. Е. Каждый перевод…` and `…х5, х7 и т. Д.`, and a payment FAQ whose list gates one item behind
+`{?CasinoHasCrypto?…|}` stops printing `…are offered.. Traditional methods…` for a casino without
+crypto — the blank element and its `. ` separator, which post-process had turned into a double stop.
+The same generated documents without any of the changed shapes — 600 of them, 3 600 renders plus
+`validate`/`analyze`/`extract` — are byte-identical, and the harness caught all three deliberate control
+mutations first.
+
+**Render cost.** A construct re-read because it holds a conditional is parsed again on every render, so
+the payment FAQ whose list gates one item behind a flag is the one real template that got slower: 1 000
+renders take 120 ms against 0.7.0's 86. Everything else measured got faster once the index answered most
+re-reads without reading, the walk stopped copying and the post-process lost its per-call overhead: the FAQ
+with a providers gate four times over, 513 ms against 1 322; plain prose, 145 against 198; all 1 262
+construct-bearing literals of the production host's migrations, rendered once each, 178 ms against 262.
+
+**The prototype names, verified against a renamed oracle.** 0.7.0's output under those names was the
+defect, so the fixed build is compared with the build before it on each document with `__proto__` and
+`constructor` renamed to ordinary names of the same length, renamed back in the output: 20 000 generated
+documents, 19 875 carrying one of the names in a reference, a conditional, a `#set`, a `#def`, a plural
+slot or a context key, through `parse`, `validate`, `analyze` and five renders — no difference, with three
+control mutations caught first (an unrelated truthiness change, and the alias map or the `#def` map left a
+plain object). The 449 009 alphabet strings, which name neither, are unchanged. The maps are merged by a
+loop: `Object.assign` into a map with no prototype took five times the spread it replaced, once per
+`#def` — 1 000 renders of a template with 60 context variables and 20 definitions went from 920 ms to
+1 336 — and the loop brings that to 968. The production templates, the FAQ and prose measure as before.
+
+**Recorded, not closed** — in the conformance README, under the known divergences. A value carrying an
+unbalanced bracket (`[a|{%L%}]` with `L = "x}|y"`) re-cuts the enclosing construct in PHP and only its
+own here. Four shapes of the same family, where a nested pick leaves markup the permutation's reader
+sees in PHP and a tree parsed before the pick: a leading element that renders empty exposes its
+`<…>` as the permutation's config, a pick ending in `<…>` becomes a per-element separator, a
+construct inside the config is resolved first, and a nested permutation joined by `|` re-splits the
+outer one (`[[<|>a|b]|c]`). All four were already so in 0.7.0. A generated differential aimed at exactly these meets
+the first twice in 3 000 renders. And the Unicode tables: Node 22's are 17.0, PCRE2 10.44's are 15.0,
+so a character assigned since then can sit on the other side of a boundary.
+
 ## 0.7.0 — 2026-09-12
 
 **A `%variable%` written directly inside `{…}` or `[…]` is now spliced as TEXT before the construct

package/README.md

@@ -81,10 +81,10 @@ dedupe in the host and cap the retries: a template may simply not have N combina
 | --- | --- | --- |
 | Enumeration | `{a\|b\|c}` | pick one (nestable: `{a\|{b\|c}}`) |
 | Permutation | `[a\|b\|c]` | pick N, shuffle, join — `[<minsize=1;maxsize=2;sep=", ">a\|b\|c]` |
-| Variable | `%var%` | substitute a context value |
+| Variable | `%var%` | substitute a context value — inside `{…}`/`[…]` the value is spliced as text first, so a `|` it carries separates options |
 | Local set | `#set %v% = value` | define a macro — re-picked at every use |
 | Local def | `#def %v% = value` | define a value — picked once per render, held at every use |
-| Conditional | `{?VAR?then\|else}` | `then` if `VAR` is truthy, else `else` |
+| Conditional | `{?VAR?then\|else}` | `then` if `VAR` is truthy, else `else` — inside `{…}`/`[…]` it resolves first, so a `\|` in the taken branch separates options and an empty one drops its element |
 | Plural | `{plural %n%: one\|few\|many}` | grammatical agreement by locale |
 | Include | `#include "slug-or-id"` | embed another template (host-resolved) |
 | Comment | `/# … #/` | stripped before rendering |
@@ -154,6 +154,11 @@ render('%bio%', { context: { bio: neutralize('Save {50|60}% today') }, postProce
 // → "Save {50|60}% today"   (the braces stay literal, not a random pick)
 ```
 
+The pipe is deliberately **not** shielded: it means nothing outside a construct, and inside one it
+is the separator. A neutralized value an author places in `{…}`/`[…]` is still split on its `|`, in
+this engine as in every other in the family. Keep such a value out of a construct if it must stay
+one option.
+
 ## Using it without writing code
 
 - **n8n** — install [`n8n-nodes-spintax`](https://www.npmjs.com/package/n8n-nodes-spintax)