@spintax/core 0.11.0 → 0.13.0

package/CHANGELOG.md

@@ -3,6 +3,84 @@
 All notable changes to `@spintax/core` are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.13.0 — 2026-10-19
+
+A minor: the post-process moves rendered text for a token that starts with a dot, for a bare `//`
+scheme, and for a sentence opening on `i` under Turkish or Azerbaijani. No verdict changes and no API
+change; `render()` already took the `locale` the post-process now also reads.
+
+### Fixed
+
+- **A dot that starts a token is not a sentence end (#91).** `.NET`, a dotfile, a file extension were
+  taken for sentence punctuation: the space before the dot was removed, a space was inserted after it and
+  the token was capitalized — `and .NET` rendered `and. NET`, `the .apk file` rendered `the. Apk file`.
+  A token that begins with a dot — `.NET`, `.gitignore`, `.env.local`, `.tar.gz`, `.xlsx`, `.5` — is now
+  shielded like a domain or a URL when it follows the text start, whitespace, an opening bracket, a quote,
+  a tag or a path separator (`src/.env`, `dir\.env`); trailing punctuation stays outside it (`.NET.`).
+  Unchanged: `word .` → `Word.`, `end . Next` → `End. Next`, `end.Next`, `ASP.NET`, `3.5`, and an ellipsis
+  glued to a word (`see...and` → `See... And`). The cost: `end .Next` — a sentence typed with the space on
+  the wrong side of the dot — is no longer repaired.
+- **Turkish and Azerbaijani capital İ (#94).** Under `locale: 'tr'` or `'az'` (any region, `-` or `_`) a
+  sentence that opens on dotted `i` is capitalized to `İ`, not `I` — `istanbul` → `İstanbul`; dotless `ı`
+  still becomes `I`. An explicit mapping, not `toLocaleUpperCase`, so the result does not depend on the
+  host's ICU data. Other locales are unchanged.
+- **A bare `//` scheme is a URL.** Prose that names the protocol — `(https://)` — got a space after its
+  colon: `(https: //)`. A `//` scheme with no body is now shielded like a URL, in any case (`HTTPS://`).
+  `mailto:` and `tel:` still need a body, so `Tel: 555` opening a sentence is capitalized and spaced as
+  before.
+
+Measured before the change over the family's stored content (2 581 production templates in four
+languages, 163 more in ten): every changed document is a fix — 21 renders of a published article that
+read `the. Apk file`, 11 that read `(https: //)` — and nothing else moves.
+
+### Corpus
+
+424 cases (+16), all in `render-postprocess.json`: `postprocess/turkish-*`, `postprocess/azerbaijani-*`,
+`postprocess/turkic-locale-case-and-underscore`, `postprocess/dotted-capital-only-under-turkic-locales`,
+`postprocess/dot-token-*` (openers, punctuation, text and line start, a digit, UCP whitespace, both path
+separators, and the negatives with the accepted cost), `postprocess/bare-scheme-*` and
+`postprocess/bare-mailto-and-tel-are-not-uris`.
+
+## 0.12.0 — 2026-10-06
+
+A minor: rendered text moves for permutation separators in Arabic, Hebrew, Thai and Lao. No verdict
+changes and no API change; `render()` already took the `locale` it now also reads here.
+
+### Fixed
+
+- **An Arabic list can be written (#90).** The conjunction و ("and") — and ف ("and then") — is written
+  attached to the next word: `الكازينو والبث`, never `الكازينو و البث`. A letter-only separator was always
+  padded on both sides, whatever spaces the author wrote, so no shape of `lastsep` could produce it. Now,
+  under `locale: 'ar'`, a separator that is exactly و or ف keeps the space before it and none after when
+  the next element starts with an Arabic letter: `[<sep="، ";lastsep="و">…]` renders
+  `الكازينو، المراهنات الرياضية والبث المباشر`. Before anything else — a Latin brand name, a digit — both
+  spaces stay: `و Evolution`. Hebrew ו under `locale: 'he'` works the same way. The rule is keyed by
+  LANGUAGE, not script: Persian and Urdu write the same letter و as a word of its own, so under `fa` and
+  `ur` it is padded as before, and so is every word conjunction (أو, ثم, או) and و with no locale.
+- **Thai and Lao separators join bare (#90),** as Han and kana have since 0.11.0: `[<lastsep="และ">แมว|สุนัข]`
+  renders `แมวและสุนัข`. Khmer and Myanmar take the same rule. A separator carrying a combining mark —
+  Thai `หรือ`, most Khmer and Myanmar words — is not all letters, so it was never padded and still renders
+  exactly as written, spaces included.
+
+No template in the family's stored content uses any of these separators (measured over casino-platform,
+content-gen, both sites and the plugin: the only hits were the probes that found the issue), so no
+existing text changes.
+
+### Corpus
+
+408 cases (+22), all in `render-semantics.json`. `perm/sep-arabic-*` (و and ف attached, the author's
+spaces, a regional locale, before a Latin word, a digit, an astral Arabic letter and U+0640, a word
+conjunction, the issue's whole list with the post-process on), `perm/sep-persian-waw-padded`,
+`perm/sep-urdu-waw-padded`, `perm/sep-waw-without-locale-padded`, `perm/sep-hebrew-*` (ו attached, before a Latin word, the word conjunction או), `perm/sep-thai-*`,
+`perm/sep-lao-not-padded`, `perm/sep-khmer-not-padded`, `perm/sep-myanmar-not-padded`, and
+`perm/sep-script-not-script-extensions`.
+
+That last one pins a divergence 0.11.0 shipped in both PHP engines without anyone seeing it: PCRE2 10.40
+and later read `\p{Han}` as Script_Extensions, so U+3006 〆, U+303C 〼, the kana repeat marks U+3031–3035
+and the halfwidth sound marks U+FF9E/FF9F joined bare there and were padded here and in every port. The
+PHP engines now exclude them, which is Script on any PCRE2 version. The same trap sat under the new Arabic
+test (U+0640, tatweel); the next element is read by code point, not by UTF-16 unit.
+
 ## 0.11.0 — 2026-10-03
 
 A minor: one verdict changes (a two-form `ar` plural is now an error) and rendered text moves for

package/README.md

@@ -19,7 +19,7 @@ parse, render, validate, extract, analyze, and neutralize spintax templates.
   [`spintax-core`](https://pypi.org/project/spintax-core/) (Python), an Object Pascal engine,
   [`Spintax.Core`](https://www.nuget.org/packages/Spintax.Core/) (.NET), and
   the [Spintax WordPress plugin](https://wordpress.org/plugins/spintax/) are all held to the same
-  behavior contract by a **shared golden corpus of 386 fixtures** — deterministic verdicts, plural
+  behavior contract by a **shared golden corpus of 424 fixtures** — deterministic verdicts, plural
   buckets, conditionals, `#set`/`#def` semantics and post-processing agree everywhere, so a
   template is an asset you can move between runtimes, not a lock-in.
 - **MIT** licensed.

package/dist/index.cjs

@@ -1389,6 +1389,37 @@ function trimFragment(fragment) {
   if (fragment.lead === 0 && fragment.trail === 0) return fragment;
   return new Window(fragment, fragment.lead, fragment.length - fragment.trail);
 }
+function firstCodePointOf(fragment) {
+  const hi = charCodeAtOf(fragment, 0);
+  if (hi >= 55296 && hi <= 56319) {
+    const lo = charCodeAtOf(fragment, 1);
+    if (lo >= 56320 && lo <= 57343) return (hi - 55296) * 1024 + (lo - 56320) + 65536;
+  }
+  return hi;
+}
+function charCodeAtOf(fragment, index) {
+  let piece = fragment;
+  let at = index;
+  for (; ; ) {
+    if (at < 0 || at >= lengthOf(piece)) return -1;
+    if (typeof piece === "string") return piece.charCodeAt(at);
+    if (piece instanceof Window) {
+      at += piece.from;
+      piece = piece.base;
+      continue;
+    }
+    let offset = 0;
+    for (const child of piece.pieces) {
+      const length = lengthOf(child);
+      if (at < offset + length) {
+        piece = child;
+        at -= offset;
+        break;
+      }
+      offset += length;
+    }
+  }
+}
 function flatten(fragment) {
   if (typeof fragment === "string") return fragment;
   const out = [];
@@ -1782,7 +1813,12 @@ function assemblePermutation(node, rendered, opts) {
   max = Math.max(min, Math.min(max, total));
   const pick = randomInt(opts.rng, min, max);
   shuffle(elements, opts.rng);
-  return joinWithSeparators(elements.slice(0, pick), config.sep, config.lastsep ?? config.sep);
+  return joinWithSeparators(
+    elements.slice(0, pick),
+    config.sep,
+    config.lastsep ?? config.sep,
+    normalizeBaseLang(opts.locale)
+  );
 }
 function shuffle(arr, rng) {
   for (let i = arr.length - 1; i > 0; i -= 1) {
@@ -1792,7 +1828,7 @@ function shuffle(arr, rng) {
     arr[j] = tmp;
   }
 }
-function joinWithSeparators(elements, globalSep, globalLastsep) {
+function joinWithSeparators(elements, globalSep, globalLastsep, lang) {
   const count = elements.length;
   if (count === 0) return "";
   if (count === 1) return elements[0].text;
@@ -1800,14 +1836,23 @@ function joinWithSeparators(elements, globalSep, globalLastsep) {
   for (let i = 1; i < count; i += 1) {
     const el = elements[i];
     const sep = el.sep ?? (i === count - 1 ? globalLastsep : globalSep);
-    pieces.push(padSeparator(sep), el.text);
+    pieces.push(padSeparator(sep, lang, el.text), el.text);
   }
   return joinFragments(pieces);
 }
-var UNSPACED_SCRIPT_RE = /^[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}ーｰ]+$/u;
-function padSeparator(sep) {
+var UNSPACED_SCRIPT_RE = /^[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}ーｰ\p{Script=Thai}\p{Script=Lao}\p{Script=Khmer}\p{Script=Myanmar}]+$/u;
+var PROCLITICS = /* @__PURE__ */ new Map([
+  ["ar", { words: /* @__PURE__ */ new Set(["\u0648", "\u0641"]), letter: /^(?=\p{L})\p{Script=Arabic}$/u }],
+  ["he", { words: /* @__PURE__ */ new Set(["\u05D5"]), letter: /^(?=\p{L})\p{Script=Hebrew}$/u }]
+]);
+function padSeparator(sep, lang, next) {
   const trimmed = phpTrim(sep);
   if (trimmed === "") return sep;
+  const proclitic = PROCLITICS.get(lang);
+  if (proclitic !== void 0 && proclitic.words.has(trimmed)) {
+    const first = firstCodePointOf(next);
+    return first >= 0 && proclitic.letter.test(String.fromCodePoint(first)) ? ` ${trimmed}` : ` ${trimmed} `;
+  }
   if (/^\p{L}+$/u.test(trimmed) && !UNSPACED_SCRIPT_RE.test(trimmed)) return ` ${trimmed} `;
   return sep;
 }
@@ -1874,7 +1919,7 @@ var TLD_UPPER = "\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}";
 var TLD = `(?:${XN}[a-zA-Z0-9\\-\\u017F\\u212A]{2,59}|[${TLD_LOWER}][${TLD_LOWER}\\p{N}-]{1,62}|[${TLD_UPPER}][${TLD_UPPER}\\p{N}-]{1,62})`;
 var DOMAIN_PART = `(?:${LABEL}\\.)+${TLD}`;
 var URI_BODY = `[^\\x00${WS}<>"')\\]]`;
-var URI_RE = new RegExp(`(?:(?:https?|ftp):\\/\\/|(?:mailto|tel):)${URI_BODY}+`, "giu");
+var URI_RE = new RegExp(`(?:(?:https?|ftp):\\/\\/${URI_BODY}*|(?:mailto|tel):${URI_BODY}+)`, "giu");
 var MAILTEL_PREFIX_RE = /^(?:mailto|tel):/iu;
 var DOMAIN_AT_RE = new RegExp(`${DOMAIN_PART}${WORD_BOUNDARY}`, "uy");
 var DOMAIN_SCAN_RE = new RegExp(`${AFTER_NON_WORD}(?:(${DOMAIN_PART}${WORD_BOUNDARY})|(?:${LABEL}\\.)*${LABEL})`, "gu");
@@ -1923,12 +1968,15 @@ var SENTENCE_OPENERS = "\xBF\xA1";
 var LEAD = `(?:<[^>]+>|[${SENTENCE_OPENERS}]|${S})*`;
 var QUOTES = `"'\xAB\xBB\u2039\u203A\u201C\u201D\u2018\u2019`;
 var CLOSER = `[)\\]]|[${QUOTES}]+(?=${S}|$|<|\\/?>|[.,;:!?\u2026)\\]\u2014\u2013])`;
+var DOT_TOKEN_RE = new RegExp(`(?<=^|${S}|[(\\[{>/\\\\${QUOTES}])\\.[\\p{L}\\p{N}][\\p{L}\\p{N}_.\\-]*`, "gu");
 var SPACE_BEFORE_PUNCT_RE = new RegExp(`(?<!${S})${S}+([,;:!?.])`, "gu");
 var SPACE_AFTER_COMMA_RE = new RegExp(`([,;:])(?!\\p{Nd})(?!${S}|$|<|${CLOSER})`, "gu");
 var SPACE_AFTER_SENTENCE_RE = new RegExp(`(?<![.!?])([.!?]+)(?![.!?])(?!\\p{Nd})(?!${S}|$|<|${CLOSER})`, "gu");
 var SPACE_AFTER_OPENER_RE = new RegExp(`([${SENTENCE_OPENERS}])${S}+`, "gu");
 var CAP_FIRST_RE = new RegExp(`^(${LEAD})(\\p{Ll})`, "u");
 var up = (ch) => ch.toUpperCase();
+var upTurkic = (ch) => ch === "i" ? "\u0130" : ch.toUpperCase();
+var TURKIC_LANGS = /* @__PURE__ */ new Set(["tr", "az"]);
 function indexLeads(text) {
   const n = text.length;
   const leadEnd = new Int32Array(n + 1);
@@ -1961,7 +2009,7 @@ function leadEndFrom(text, i, leads) {
   }
   return j < text.length ? leads().leadEnd[j] : j;
 }
-function capitalizeAfter(text, lower, next, context) {
+function capitalizeAfter(text, lower, next, context, upper) {
   const leads = () => {
     if (context.leads === null || context.leads.leadEnd.length !== text.length + 1) context.leads = indexLeads(text);
     return context.leads;
@@ -1976,7 +2024,7 @@ function capitalizeAfter(text, lower, next, context) {
     if (cp === void 0) continue;
     const ch = String.fromCodePoint(cp);
     if (!lower.test(ch)) continue;
-    out += text.slice(emitted, at) + up(ch);
+    out += text.slice(emitted, at) + upper(ch);
     emitted = at + ch.length;
     from = emitted;
   }
@@ -2009,7 +2057,7 @@ var afterLineBreak = (text, from) => {
   const at = text.indexOf("\n", from);
   return at === -1 ? null : { leadStart: at + 1, resume: at + 1 };
 };
-var SHIELD_PREFIXES = ["URL", "URI", "EMAIL", "DOM", "NUM", "ABBR"];
+var SHIELD_PREFIXES = ["URL", "URI", "EMAIL", "DOTTOK", "DOM", "NUM", "ABBR"];
 var RESTORE_RE2 = new RegExp(`\\x00(?:${SHIELD_PREFIXES.join("|")})_\\d+\\x00`, "gu");
 function restore(text, input, placeholders) {
   if (!input.includes("\0")) {
@@ -2044,7 +2092,8 @@ function restoreAsTheLoopDoes(text, placeholders) {
   }
   return out;
 }
-function postProcess(input) {
+function postProcess(input, locale = "") {
+  const upper = TURKIC_LANGS.has(normalizeBaseLang(locale)) ? upTurkic : up;
   const placeholders = /* @__PURE__ */ new Map();
   let counter = 0;
   const store = (value, prefix) => {
@@ -2064,6 +2113,7 @@ function postProcess(input) {
     (m) => storeWithTrailingPunct(m, MAILTEL_PREFIX_RE.test(m) ? "URI" : "URL")
   );
   text = shieldEmails(text, (m) => store(m, "EMAIL"));
+  text = text.replace(DOT_TOKEN_RE, (m) => storeWithTrailingPunct(m, "DOTTOK"));
   text = shieldDomains(text, (m) => store(m, "DOM"));
   text = text.replace(DECIMAL_RE, (m) => store(m, "NUM"));
   text = text.replace(MULTI_ABBR_RE, (m) => store(m, "ABBR"));
@@ -2073,11 +2123,11 @@ function postProcess(input) {
   text = text.replace(SPACE_AFTER_COMMA_RE, "$1 ");
   text = text.replace(SPACE_AFTER_SENTENCE_RE, "$1 ");
   text = text.replace(SPACE_AFTER_OPENER_RE, "$1");
-  text = text.replace(CAP_FIRST_RE, (_m, lead, ch) => lead + up(ch));
+  text = text.replace(CAP_FIRST_RE, (_m, lead, ch) => lead + upper(ch));
   const leadContext = { leads: null };
-  text = capitalizeAfter(text, LOWER_RE, afterSentenceEnd, leadContext);
-  text = capitalizeAfter(text, LOWER_RE, afterBlockTag(), leadContext);
-  text = capitalizeAfter(text, LOWER_RE, afterLineBreak, leadContext);
+  text = capitalizeAfter(text, LOWER_RE, afterSentenceEnd, leadContext, upper);
+  text = capitalizeAfter(text, LOWER_RE, afterBlockTag(), leadContext, upper);
+  text = capitalizeAfter(text, LOWER_RE, afterLineBreak, leadContext, upper);
   return restore(text, input, placeholders).trim();
 }
 
@@ -2098,7 +2148,7 @@ function renderWith(input, rng, opts = {}) {
   };
   const ast = resolveAst(input);
   let out = renderAst(ast, ctx);
-  if (opts.postProcess !== false) out = postProcess(out);
+  if (opts.postProcess !== false) out = postProcess(out, ctx.locale);
   return safetyRestore(out);
 }
 function resolveAst(input) {