@mailwoman/neural-weights-en-us 5.1.0 → 5.2.0

package/model-card.json

@@ -1,13 +1,13 @@
 {
 	"name": "neural-weights-en-us",
-	"version": "5.1.0",
-	"model_lineage": "v1.9.3a3-anchor-absorption encoder (UNCHANGED, step 80000 — see the v4.15.0 lineage below) + the #884 UNIGRAM vocab-splice with FVT mean-init embeddings — the training-free CZ/PL/SK/SI diacritic fix. The 48k SentencePiece vocab carried the diacritic CHARACTERS but no multi-char subwords containing them, so every diacritic isolated its own piece (Vysoká → [▁V, ys, ok, á], ~3.3× English fertility) and the fragmented span resolved to the WRONG CITY. A shard retrain (v196) was falsified (held US, regressed CZ wrong-city 44→58% at convergence). The fix: train a CZ/PL SP unigram, splice ONLY its diacritic-bearing pieces into the vocab (48,000 → 58,582; a diacritic piece can never match an ASCII span, so English tokenizes byte-identically BY CONSTRUCTION — asserted 0/2000), and FVT mean-init the 10,582 new embedding rows from their old-tokenizer constituents. The encoder is byte-for-byte v1.9.3a3; a 2k fine-tune was ablated and retired as slightly harmful. Slovak + Slovenian ride the same pieces for free (shared codepoints + morphology). PRIOR LINEAGE: v1.9.3a3-anchor-absorption / step 80000 — the #220/#723 postcode-anchor-pollution fix on top of v1.9.2-multilocale-au (shipped v4.14.0). Pre-seeded from the v1.9.2 step-040000 checkpoint, continued to 80k on A100. TWO coordinated levers: (1) anchor_paint_mode=shaped — the postcode anchor is painted at TRAIN time on the same postcode-SHAPED spans inference fires on (neural/postcode-anchor.ts collectMatches), closing the #723 WHERE gap (training only ever saw the anchor on GOLD postcode spans, so the model never learned to OVERRIDE a present anchor when a leading 5-digit token is a house number that looks like a ZIP); (2) the synth-anchor-absorption counter-shard (source-weight 6.0) carrying the locality discriminator — leading-5-digit + trailing-postcode → house_number, + locality → house_number, + NO locality + state → postcode (the VT E911 format). RESULT: SLICE-H anchor-ON 20→100 (the #723 leading-5-digit house# bug fixed) at ZERO coordinate cost, confirmed by the non-circular oa-resolver-eval (2000 real OA-US points, full ship-config, v192 vs v1.9.3a3): whole-US locality-match 97.7→97.8 (+0.1), region 99.9 flat, coord p50 3.4 flat, p90 10.7 flat, p99 275.3→259.1; VT — the E911 leading-postcode state where the entire trade lives — FLAT at the coordinate (loc 92.6=92.6), SD +1.0. Trades us.postcode LABEL-F1 −2 (95.0) + fr.postcode −0.2 (99.3), both coordinate-invisible; gate revised (scripts/eval/gates/v4.15.0-boundary.json) with a stated reason, operator-approved. mask-regression gate (#718) PASS; int8↔fp32 delta 0. Ship-config (requires) UNCHANGED from v1.9.2.",
+	"version": "5.2.0",
+	"model_lineage": "v5.1.0 bsplice-expanded (UNCHANGED encoder — see below) + the #912-lever-4 NORDIC vocab-splice with FVT mean-init embeddings (the same training-free #884 mechanism, second application): +5,331 Nordic diacritic pieces from OA fi/se/no/dk/is STREET/CITY text + the gazetteer ALIAS names (the exonym feed — 'Åbo' is Swedish for Turku and appears in no native OA text; without alias mass the piece is never minted and the decode drop survives, measured as the 134km v1 residual). Tokenizer v0.7.1-nsplice = v0.6.0-bsplice + the Nordic pieces; English tokenization byte-identical; US resolution byte-identical 2000/2000. Gate: 14/14 per-locale ni PASS (incl. the five #900-accepted acute-vowel overlap locales CZ/ES/PL/SI/SK), SE resolve 83.8→90.2% (p90res 113→12.7km), NO 92.7→98.1%, DK 91.4→95.2%, Åbo→Turku 0.12km. PRIOR LINEAGE: v1.9.3a3-anchor-absorption encoder (UNCHANGED, step 80000 — see the v4.15.0 lineage below) + the #884 UNIGRAM vocab-splice with FVT mean-init embeddings — the training-free CZ/PL/SK/SI diacritic fix. The 48k SentencePiece vocab carried the diacritic CHARACTERS but no multi-char subwords containing them, so every diacritic isolated its own piece (Vysoká → [▁V, ys, ok, á], ~3.3× English fertility) and the fragmented span resolved to the WRONG CITY. A shard retrain (v196) was falsified (held US, regressed CZ wrong-city 44→58% at convergence). The fix: train a CZ/PL SP unigram, splice ONLY its diacritic-bearing pieces into the vocab (48,000 → 58,582; a diacritic piece can never match an ASCII span, so English tokenizes byte-identically BY CONSTRUCTION — asserted 0/2000), and FVT mean-init the 10,582 new embedding rows from their old-tokenizer constituents. The encoder is byte-for-byte v1.9.3a3; a 2k fine-tune was ablated and retired as slightly harmful. Slovak + Slovenian ride the same pieces for free (shared codepoints + morphology). PRIOR LINEAGE: v1.9.3a3-anchor-absorption / step 80000 — the #220/#723 postcode-anchor-pollution fix on top of v1.9.2-multilocale-au (shipped v4.14.0). Pre-seeded from the v1.9.2 step-040000 checkpoint, continued to 80k on A100. TWO coordinated levers: (1) anchor_paint_mode=shaped — the postcode anchor is painted at TRAIN time on the same postcode-SHAPED spans inference fires on (neural/postcode-anchor.ts collectMatches), closing the #723 WHERE gap (training only ever saw the anchor on GOLD postcode spans, so the model never learned to OVERRIDE a present anchor when a leading 5-digit token is a house number that looks like a ZIP); (2) the synth-anchor-absorption counter-shard (source-weight 6.0) carrying the locality discriminator — leading-5-digit + trailing-postcode → house_number, + locality → house_number, + NO locality + state → postcode (the VT E911 format). RESULT: SLICE-H anchor-ON 20→100 (the #723 leading-5-digit house# bug fixed) at ZERO coordinate cost, confirmed by the non-circular oa-resolver-eval (2000 real OA-US points, full ship-config, v192 vs v1.9.3a3): whole-US locality-match 97.7→97.8 (+0.1), region 99.9 flat, coord p50 3.4 flat, p90 10.7 flat, p99 275.3→259.1; VT — the E911 leading-postcode state where the entire trade lives — FLAT at the coordinate (loc 92.6=92.6), SD +1.0. Trades us.postcode LABEL-F1 −2 (95.0) + fr.postcode −0.2 (99.3), both coordinate-invisible; gate revised (scripts/eval/gates/v4.15.0-boundary.json) with a stated reason, operator-approved. mask-regression gate (#718) PASS; int8↔fp32 delta 0. Ship-config (requires) UNCHANGED from v1.9.2.",
 	"phase": "v5.1.0 vocab-splice (#884) — training-free CZ/PL/SK/SI diacritic fix: spliced tokenizer (48k→58.6k) + mean-init rows on the unchanged v1.9.3a3 encoder. Wrong-city at n=1k: CZ 22.4→14.8%, PL 27.9→7.9%, SK 22.2→13.2%, SI 18.9→10.6%; US byte-identical (int8 row dumps identical to v4.15.0)",
-	"license": "AGPL-3.0-only",
+	"license": "AGPL-3.0-only OR LicenseRef-Commercial",
 	"locale": "en-us",
 	"training": {
 		"corpus_version": "corpus-v0.9.3a3 = v0.9.2-multilocale-au base + synth-anchor-absorption counter-shard @ source-weight 6.0 (the #220/#723 leading-5-digit discriminator, built by scripts/build-anchor-absorption-shard.mjs from 42,312 real US ZIPs)",
-		"tokenizer_version": "0.6.0-bsplice",
+		"tokenizer_version": "0.7.1-nsplice",
 		"steps": 80000,
 		"best_step": 80000,
 		"hardware": "NVIDIA A100-SXM4-40GB (Modal cloud)",
@@ -303,9 +303,9 @@
 		}
 	},
 	"files_md5": {
-		"$comment": "Source artifact md5s (int8 model-bsplice-meaninit-int8.onnx + tokenizer v0.6.0-bsplice). The PUBLISHED files are re-verified against these at release Step 4 (npm pack); update if the publish pipeline post-processes either.",
-		"model.onnx": "12fa2d645162e90cf028dbc333218f2c",
-		"tokenizer.model": "b0397ce3d89ab459be6dd901eae8f0a1"
+		"$comment": "Source artifact md5s (int8 model-nsplice-v2-int8.onnx + tokenizer v0.7.1-nsplice). The PUBLISHED files are re-verified against these at release Step 4 (npm pack); update if the publish pipeline post-processes either.",
+		"model.onnx": "2c18d8545048e116564069647978532b",
+		"tokenizer.model": "8e1cab7c501af834fbfcd6c4b8d78cad"
 	},
 	"capabilities": {
 		"server": {

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "@mailwoman/neural-weights-en-us",
-  "version": "5.1.0",
+  "version": "5.2.0",
   "description": "Mailwoman neural-classifier weights for locale 'en-us'. Data-only package — loaded by @mailwoman/neural at runtime.",
-  "license": "AGPL-3.0-only",
+  "license": "AGPL-3.0-only OR LicenseRef-Commercial",
   "repository": {
     "type": "git",
     "url": "https://github.com/sister-software/mailwoman.git",