langsys-js-typescript 0.6.2 → 0.6.4

package/CHANGELOG.md

@@ -1,3 +1,51 @@
+## 0.6.4 - 2026-08-16
+
+### Fixed
+
+- **A missing ICU argument no longer dumps the message source to the page.** `intl-messageformat` throws when an argument is absent, and the fallback ran the simple interpolator — whose pattern deliberately cannot match an ICU slot, so the entire construct passed through untouched. A phrase promoted to `{name_gender, select, …}` rendered as literal ICU source in front of users.
+
+  This is reachable with no caller mistake: langsys-ai's ICU promoter *introduces* a select argument the source phrase never had — a plain `{name}` becomes `{name_gender, select, …}` in gendered target locales. The application cannot supply `name_gender`; it doesn't exist in the phrase the developer wrote, and nothing tells them the target grew one. Every app translating into a gendered locale hits it.
+
+  Recovery, agreed with `langsys-php` so a shared catalog cannot render two different sentences for one input:
+
+  | slot | missing-argument behavior | outcome |
+  |---|---|---|
+  | `select` | take the `other` branch | a correct sentence — `other` is what an unknown gender should render |
+  | `plural` | take `other`, render `#` as `{argName}` | sentence survives with a visible gap; nothing can be inferred for a count |
+  | `{name}`, `{n, number}`, dates | render as `{argName}` | matches the simple path's rule that unknown keys stay visible |
+
+  The asymmetry is deliberate: `select` is genuinely recoverable, `plural` is only made less bad. Arguments that ARE supplied are untouched and recursed into, so a missing argument nested inside a satisfied branch still recovers. `plural`/`select` without an `other` branch is malformed ICU and is left alone, falling through to the simple path as before.
+
+  **A `null` argument counts as missing**, matching `langsys-php`. It doesn't throw — `intl-messageformat` coerces null to `0` — so `{count, plural, …}` rendered `0 items`. That is worse than it looks: a genuine `0` also renders `0 items`, so a bug and valid data became indistinguishable, in the output, in a screenshot, and in a bug report. A real `0` still renders `0 items`; only null and absent produce `{count} items`.
+
+  Found by `langsys-php` from a report against their SDK; both implementations had it, by different mechanisms. Verified against their shipped v1.3.1 behavior, including a nested case where an *unsatisfied* branch's missing argument correctly costs nothing (`{g, select, f {Ella} other {{n, plural …}}}` with `g='f'` renders `Ella`).
+
+## 0.6.3 - 2026-08-16
+
+### Fixed
+
+- **`<select>` option text was harvested twice, diverging `custom_id` from every other Langsys SDK.** A `<select>` special case pushed each `<option>`'s text, and the ordinary text-node walk then pushed it again — `<option>` cannot contain elements, so the walker already covered it. Any content block containing a `<select>` therefore produced `["S","S"]` where `langsys-php` produced `["S"]`, and had a different `custom_id` in the two SDKs since before either was audited. The fixture suite could not see it: those fixtures are synthetic token lists that never touch a DOM.
+
+  **Migration is automatic and lookup-only**, extending the mechanism 0.6.0 introduced. Three historical id shapes are now tried before registering:
+
+  | tokens | hash | registered by |
+  |---|---|---|
+  | corrected | corrected | 0.6.3+ (current) |
+  | legacy (duplicated) | corrected | 0.6.0 – 0.6.2 |
+  | legacy (duplicated) | legacy | before 0.6.0 |
+
+  Registration always uses the corrected id, so the legacy-keyed population can only shrink. Content without a `<select>` is unaffected — its token list is identical either way, so nothing needlessly rebases. Without this fallback the upgrade would have been silent in the worst direction: a dropdown quietly reverting to source text, with nothing erroring to explain it.
+
+- **`legacyTokenizeElement(element)`** — the pre-0.6.3 token list, exported for migration tooling. Lookup-only; never register under an id derived from it. Deprecated on arrival.
+
+### Added
+
+- **Four standard attributes are now harvested: `label`, `aria-description`, `aria-valuetext`, `aria-roledescription`.** `langsys-php` harvests a superset of our list; diffing the two showed nothing JS-only and 16 PHP-only, of which these four carry real user-visible text. `label` is what a user reads in a `<select>` picker (`<option>`, `<optgroup>`, `<track>`) and is invisible to sighted testing that only inspects option text; the three ARIA attributes are read aloud by screen readers, so leaving them untranslated degrades accessibility specifically for the users least able to work around it.
+
+  The remaining twelve PHP-only attributes are framework conventions (Bootstrap `data-bs-*`, Rails-style `data-confirm`, loading/notification text) and are deliberately **not** mirrored — they're written by server-rendered templates, and a JS app renders those strings through its own components.
+
+  **Note:** a content block containing one of these attributes now yields a different token list, and therefore a different `custom_id`. Such blocks re-register and their existing translations orphan. The 0.6.0 legacy-id fallback cannot help here — it re-hashes the *same* tokens, and this changes the tokens themselves. Affected population is only blocks that use these four attributes.
+
 ## 0.6.2 - 2026-08-16
 
 ### Added

package/dist/index.d.mts

@@ -738,6 +738,23 @@ declare function tokenizeElement(element: HTMLElement): {
     tokens: string[];
     content: string;
 };
+/**
+ * The token list this element would have produced before 0.6.3, when every
+ * `<option>`'s text was harvested twice — once by a `<select>` special case and
+ * again by the ordinary text-node walk.
+ *
+ * **Lookup only.** Content blocks registered by an older SDK are keyed by an id
+ * derived from this list, so `Translate` falls back to it when the corrected id
+ * misses; without that, every block containing a `<select>` would silently
+ * re-key and lose its translations on upgrade. Registration always uses the
+ * corrected list, so the legacy-keyed population can only shrink.
+ *
+ * Skips the computed-style capture that `tokenizeElement` performs — this is a
+ * cache-key computation, not a snapshot for translators.
+ *
+ * @deprecated Migration aid; will be removed once catalogs have been rebased.
+ */
+declare function legacyTokenizeElement(element: HTMLElement): string[];
 
 /**
  * Create a Signal whose value is persisted to localStorage (when available).
@@ -876,4 +893,4 @@ interface iProject {
 declare const t: TFunction;
 declare const tSignal: Signal<TFunction>;
 
-export { type EncodedRichText, type ExtractParamKeys, LangsysApp, LangsysAppAPI, type LocaleSource, Logger, PHRASE_MARKER_ATTR, PHRASE_MARKER_ATTRS, type ParamPrimitive, type ParamsFor, Phrase, type PhraseOptions, type RichSlot, type Signal, type Subscriber, type TArgs, type TFunction, Translate, type TranslateOptions, type TranslationParams, Translations, type Unsubscriber, type Updater, canonicalizeLocale, createSignal, currentlyLoadedLocale, encodeRichText, findUnusedParamKeys, generateCustomId, generateLegacyCustomId, getValue, type iCategories, type iContentBlock, type iCountry, type iCountryDialCode, type iCountryList, type iCurrency, type iCurrencyList, type iLangsysConfig, type iLangsysInitConfig, type ResponseObject as iLangsysResponse, type iLanguageName, type iLocaleData, type iLocaleDefault, type iLocaleFlat, type iProject, type iTranslations, interpolate, isContentBlockKnown, isEmpty, isICU, isPhraseMarked, isTranslationExcluded, logger, markupTokenValues, md5, md5Legacy, normalizeMarkupPlaceholders, persist, reconstitute, registerContentBlock, sTranslations, stripSentinels, t, tSignal, tokenizeElement };
+export { type EncodedRichText, type ExtractParamKeys, LangsysApp, LangsysAppAPI, type LocaleSource, Logger, PHRASE_MARKER_ATTR, PHRASE_MARKER_ATTRS, type ParamPrimitive, type ParamsFor, Phrase, type PhraseOptions, type RichSlot, type Signal, type Subscriber, type TArgs, type TFunction, Translate, type TranslateOptions, type TranslationParams, Translations, type Unsubscriber, type Updater, canonicalizeLocale, createSignal, currentlyLoadedLocale, encodeRichText, findUnusedParamKeys, generateCustomId, generateLegacyCustomId, getValue, type iCategories, type iContentBlock, type iCountry, type iCountryDialCode, type iCountryList, type iCurrency, type iCurrencyList, type iLangsysConfig, type iLangsysInitConfig, type ResponseObject as iLangsysResponse, type iLanguageName, type iLocaleData, type iLocaleDefault, type iLocaleFlat, type iProject, type iTranslations, interpolate, isContentBlockKnown, isEmpty, isICU, isPhraseMarked, isTranslationExcluded, legacyTokenizeElement, logger, markupTokenValues, md5, md5Legacy, normalizeMarkupPlaceholders, persist, reconstitute, registerContentBlock, sTranslations, stripSentinels, t, tSignal, tokenizeElement };

package/dist/index.d.ts

@@ -738,6 +738,23 @@ declare function tokenizeElement(element: HTMLElement): {
     tokens: string[];
     content: string;
 };
+/**
+ * The token list this element would have produced before 0.6.3, when every
+ * `<option>`'s text was harvested twice — once by a `<select>` special case and
+ * again by the ordinary text-node walk.
+ *
+ * **Lookup only.** Content blocks registered by an older SDK are keyed by an id
+ * derived from this list, so `Translate` falls back to it when the corrected id
+ * misses; without that, every block containing a `<select>` would silently
+ * re-key and lose its translations on upgrade. Registration always uses the
+ * corrected list, so the legacy-keyed population can only shrink.
+ *
+ * Skips the computed-style capture that `tokenizeElement` performs — this is a
+ * cache-key computation, not a snapshot for translators.
+ *
+ * @deprecated Migration aid; will be removed once catalogs have been rebased.
+ */
+declare function legacyTokenizeElement(element: HTMLElement): string[];
 
 /**
  * Create a Signal whose value is persisted to localStorage (when available).
@@ -876,4 +893,4 @@ interface iProject {
 declare const t: TFunction;
 declare const tSignal: Signal<TFunction>;
 
-export { type EncodedRichText, type ExtractParamKeys, LangsysApp, LangsysAppAPI, type LocaleSource, Logger, PHRASE_MARKER_ATTR, PHRASE_MARKER_ATTRS, type ParamPrimitive, type ParamsFor, Phrase, type PhraseOptions, type RichSlot, type Signal, type Subscriber, type TArgs, type TFunction, Translate, type TranslateOptions, type TranslationParams, Translations, type Unsubscriber, type Updater, canonicalizeLocale, createSignal, currentlyLoadedLocale, encodeRichText, findUnusedParamKeys, generateCustomId, generateLegacyCustomId, getValue, type iCategories, type iContentBlock, type iCountry, type iCountryDialCode, type iCountryList, type iCurrency, type iCurrencyList, type iLangsysConfig, type iLangsysInitConfig, type ResponseObject as iLangsysResponse, type iLanguageName, type iLocaleData, type iLocaleDefault, type iLocaleFlat, type iProject, type iTranslations, interpolate, isContentBlockKnown, isEmpty, isICU, isPhraseMarked, isTranslationExcluded, logger, markupTokenValues, md5, md5Legacy, normalizeMarkupPlaceholders, persist, reconstitute, registerContentBlock, sTranslations, stripSentinels, t, tSignal, tokenizeElement };
+export { type EncodedRichText, type ExtractParamKeys, LangsysApp, LangsysAppAPI, type LocaleSource, Logger, PHRASE_MARKER_ATTR, PHRASE_MARKER_ATTRS, type ParamPrimitive, type ParamsFor, Phrase, type PhraseOptions, type RichSlot, type Signal, type Subscriber, type TArgs, type TFunction, Translate, type TranslateOptions, type TranslationParams, Translations, type Unsubscriber, type Updater, canonicalizeLocale, createSignal, currentlyLoadedLocale, encodeRichText, findUnusedParamKeys, generateCustomId, generateLegacyCustomId, getValue, type iCategories, type iContentBlock, type iCountry, type iCountryDialCode, type iCountryList, type iCurrency, type iCurrencyList, type iLangsysConfig, type iLangsysInitConfig, type ResponseObject as iLangsysResponse, type iLanguageName, type iLocaleData, type iLocaleDefault, type iLocaleFlat, type iProject, type iTranslations, interpolate, isContentBlockKnown, isEmpty, isICU, isPhraseMarked, isTranslationExcluded, legacyTokenizeElement, logger, markupTokenValues, md5, md5Legacy, normalizeMarkupPlaceholders, persist, reconstitute, registerContentBlock, sTranslations, stripSentinels, t, tSignal, tokenizeElement };

package/dist/index.js

@@ -289,14 +289,76 @@ function warnUnmatchedParams(source, texts, params, context) {
 }
 function interpolate(template, params, locale) {
   if (isICU(template)) {
-    try {
-      return new intlMessageformat.IntlMessageFormat(template, locale || "en").format(params);
-    } catch {
-      return simpleInterpolate(template, params, locale);
+    const resolved = locale || "en";
+    const hasNullParam = !!params && Object.values(params).some((value) => value === null);
+    if (!hasNullParam) {
+      try {
+        return new intlMessageformat.IntlMessageFormat(template, resolved).format(params);
+      } catch {
+      }
+    }
+    {
+      try {
+        const ast = new intlMessageformat.IntlMessageFormat(template, resolved).ast;
+        if (!Array.isArray(ast)) return simpleInterpolate(template, params, locale);
+        const recovered = _recoverMissingArgs(JSON.parse(JSON.stringify(ast)), params);
+        return new intlMessageformat.IntlMessageFormat(recovered, resolved).format(params);
+      } catch {
+        return simpleInterpolate(template, params, locale);
+      }
     }
   }
   return simpleInterpolate(template, params, locale);
 }
+var ICU_LITERAL = 0;
+var ICU_ARGUMENT = 1;
+var ICU_NUMBER = 2;
+var ICU_DATE = 3;
+var ICU_TIME = 4;
+var ICU_SELECT = 5;
+var ICU_PLURAL = 6;
+var ICU_POUND = 7;
+var ICU_TAG = 8;
+function _recoverMissingArgs(nodes, params) {
+  const out = [];
+  for (const node of nodes) {
+    const name = node.value;
+    const value = name !== void 0 && name in params ? params[name] : void 0;
+    const supplied = value !== void 0 && value !== null;
+    if (node.type === ICU_TAG) {
+      out.push({ ...node, children: _recoverMissingArgs(node.children ?? [], params) });
+      continue;
+    }
+    if (node.type === ICU_SELECT || node.type === ICU_PLURAL) {
+      if (supplied) {
+        const options = {};
+        for (const [key, branch2] of Object.entries(node.options ?? {})) {
+          options[key] = { value: _recoverMissingArgs(branch2.value, params) };
+        }
+        out.push({ ...node, options });
+        continue;
+      }
+      const other = node.options?.other;
+      if (!other) {
+        out.push(node);
+        continue;
+      }
+      const branch = _recoverMissingArgs(other.value, params);
+      out.push(
+        ...branch.map(
+          (child) => child.type === ICU_POUND ? { type: ICU_LITERAL, value: `{${name}}` } : child
+        )
+      );
+      continue;
+    }
+    if (!supplied && (node.type === ICU_ARGUMENT || node.type === ICU_NUMBER || node.type === ICU_DATE || node.type === ICU_TIME)) {
+      out.push({ type: ICU_LITERAL, value: `{${name}}` });
+      continue;
+    }
+    out.push(node);
+  }
+  return out;
+}
 function simpleInterpolate(template, params, locale) {
   return template.replace(/\{([^{},]+)\}/g, (match, rawKey) => {
     const key = rawKey.trim();
@@ -1076,8 +1138,14 @@ var TRANSLATABLE_ATTRIBUTES = [
   "placeholder",
   "alt",
   "title",
+  "label",
+  // <option>, <optgroup>, <track> — the text a user reads in the picker
   "aria-label",
   "aria-placeholder",
+  "aria-description",
+  "aria-valuetext",
+  // the spoken value of a slider/meter
+  "aria-roledescription",
   "data-error",
   "data-error-message",
   "data-validation-message",
@@ -1194,9 +1262,15 @@ async function registerContentBlock(contentBlock) {
 function tokenizeElement(element) {
   const tokens = [];
   const clone = element.cloneNode(true);
-  _walkForTokens(element, Array.from(clone.childNodes), tokens, []);
+  _walkForTokens(element, Array.from(clone.childNodes), tokens, [], false, true);
   return { tokens, content: normalizeMarkupPlaceholders(clone.outerHTML) };
 }
+function legacyTokenizeElement(element) {
+  const tokens = [];
+  const clone = element.cloneNode(true);
+  _walkForTokens(element, Array.from(clone.childNodes), tokens, [], true, false);
+  return tokens;
+}
 function _writeKnownContentBlockToCache(category, customId) {
   const lookupCat = category || "__uncategorized__";
   sTranslations.update((current) => {
@@ -1210,18 +1284,18 @@ function _writeKnownContentBlockToCache(category, customId) {
     return { ...current };
   });
 }
-function _walkForTokens(liveRoot, cloneNodes, tokens, indices) {
+function _walkForTokens(liveRoot, cloneNodes, tokens, indices, duplicateSelectOptions, applyStyles) {
   cloneNodes.forEach((node, index) => {
     if (node.nodeType === Node.ELEMENT_NODE) {
       const el = node;
       if (isTranslationExcluded(el)) return;
       if (isPhraseMarked(el)) return;
     }
-    if (node.hasChildNodes()) {
+    if (applyStyles && node.hasChildNodes()) {
       _applyStylesToClone(liveRoot, node, [...indices, index]);
     }
     if (node.nodeType === Node.ELEMENT_NODE) {
-      _tokenizeAttributes(node, tokens);
+      _tokenizeAttributes(node, tokens, duplicateSelectOptions);
     }
     const contentToken = node.nodeValue?.replace(/\s+/g, " ").trim();
     if (node.nodeType === Node.TEXT_NODE && contentToken) {
@@ -1229,10 +1303,10 @@ function _walkForTokens(liveRoot, cloneNodes, tokens, indices) {
       return;
     }
     if (!node.hasChildNodes()) return;
-    _walkForTokens(liveRoot, Array.from(node.childNodes), tokens, [...indices, index]);
+    _walkForTokens(liveRoot, Array.from(node.childNodes), tokens, [...indices, index], duplicateSelectOptions, applyStyles);
   });
 }
-function _tokenizeAttributes(element, tokens) {
+function _tokenizeAttributes(element, tokens, duplicateSelectOptions) {
   const tagName = element.tagName.toLowerCase();
   if (tagName === "img") {
     const img = element;
@@ -1253,7 +1327,7 @@ function _tokenizeAttributes(element, tokens) {
       if (value) tokens.push(normalizeMarkupPlaceholders(value));
     }
   }
-  if (tagName === "select") {
+  if (duplicateSelectOptions && tagName === "select") {
     const options = element.querySelectorAll("option");
     options.forEach((option) => {
       const optionText = option.textContent?.trim();
@@ -1379,9 +1453,15 @@ var Translate = class {
       return;
     }
     if (derivedId) {
-      const legacyId = generateLegacyCustomId(contentBlock.category, contentBlock.tokens);
-      if (legacyId !== this.custom_id && isContentBlockKnown(contentBlock.category, legacyId)) {
-        this.custom_id = legacyId;
+      const legacyTokens = legacyTokenizeElement(this.element);
+      const candidates = [
+        generateCustomId(contentBlock.category, legacyTokens),
+        generateLegacyCustomId(contentBlock.category, legacyTokens)
+      ];
+      for (const candidate of candidates) {
+        if (candidate === this.custom_id) continue;
+        if (!isContentBlockKnown(contentBlock.category, candidate)) continue;
+        this.custom_id = candidate;
         if (this.tokens.length > 1 && this.element) {
           this.translate(Array.from(this.element.childNodes));
           this.lastTranslatedLocale = currentlyLoadedLocale.get();
@@ -1667,6 +1747,7 @@ exports.isEmpty = isEmpty;
 exports.isICU = isICU;
 exports.isPhraseMarked = isPhraseMarked;
 exports.isTranslationExcluded = isTranslationExcluded;
+exports.legacyTokenizeElement = legacyTokenizeElement;
 exports.logger = logger;
 exports.markupTokenValues = markupTokenValues;
 exports.md5 = md5;