whatlanguage 2.0.1 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 53fea55810c7a8eb102970d6d8c3fe039b031680ff0f23eb6a1c7fec6ad1a602
-  data.tar.gz: f960efb55e0d04b5e457ef53f6c382434a334eea9d44fab2065e7ce9d4376747
+  metadata.gz: 93967e970b5e62eb58b9cb835866d4abe885f98761f7a339d0817dad8414eac8
+  data.tar.gz: b7e983bcd0a56f7b6f77675fdedd6d7724da71b42e26e3229014a92a3c32166b
 SHA512:
-  metadata.gz: 99e1be592f692ba51e3a973a489703f955238da71985feb91b424e9203b8ae29d1ff635d5479ee6cea0226a06cd74c10c4f859a722c76914f277df3d41c3c779
-  data.tar.gz: e13bf56aa67c969d86213e9f1bd34bb2f0aa4ab0ab5db4c1b735ef84c9c3d2ad8e7891f0104a0881833fb5f54c42235132cf48f7a75af38f1cf5f71280529a31
+  metadata.gz: f7c259a244e6c0fe0aa9084037fb29381bc96c9b5a4e549382b660b95943d82bf651d5e706fc4290cb93ff2bc820b1c15495f70bec025c5b98bfa3fb0ff91d6a
+  data.tar.gz: 6b2d67686a98ec148d4a3f3314cd270beee7ad3b3437411b8a516d69d6078d3a1a3bd230a4d7e06a5904f7b286339e8be1b2cfef29ba9df73e0da76efb0044a3

data/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## 3.0.0 / 2026-10-20
+
+### Breaking change: nine ISO output codes
+
+- Normalized nine individual-language ISO outputs to CLDR macrolanguage aliases
+  (#51): `swh → sw`, `ayr → ay`, `knc → kr`, `quz → qu`, `khk → mn`,
+  `kng → kg`, `src → sc`, `uzn → uz`, and `als → sq`. This changes
+  `Result#iso`, instance/class `language_iso`, and CLI output; callers comparing
+  the previous codes must update those comparisons. Language names and detection
+  scores are unchanged. Northern Uzbek and Uzbek now both return `uz`.
+
+### Improvements
+
+- Preserved the individual codes `quy`, `qug`, `koi`, and `min`, and documented
+  the selected CLDR aliases and historical overrides used for ISO output.
+- Added shared trigram indexes to speed up repeated detection over the full
+  language set, without changing scores or rankings. First calls and restricted
+  selections retain the original scan.
+- Reduced text-processing allocations with rolling trigram windows, a combined
+  script/letter-count pass, and an ASCII fast path. Cached restricted candidates
+  and avoided sorting when only the winning language is requested.
+- Added equivalence tests and reproducible benchmarks for
+  [indexed scoring](benchmark/README.md) and
+  [text processing](benchmark/long-inputs.md), including allocation and memory costs.
+
+## 2.1.0 / 2026-09-05
+
+- Added a fixed short-text preference for English, Chinese, Hindi, Spanish, and
+  French: a 600-point trigram bonus through 10 letters, fading to zero at 50.
+  Script routing, candidate restrictions, and minimum-length rules still apply.
+- Replaced hard-coded ranges with Unicode script properties, recognizing
+  uppercase Georgian, extended Greek, and supplementary Han letters.
+- Combined Han and kana when detecting Japanese instead of misclassifying
+  Han-heavy Japanese text as Chinese.
+- Excluded digits, punctuation, and emoji from script votes and minimum-length
+  counting, and treated them as trigram separators while preserving combining marks.
+- Added regression coverage for short texts and Unicode handling. Short-text
+  preferences are a heuristic; ambiguous fragments can still be misclassified.
+
 ## 2.0.1 / 2026-07-11
 
 - Added a `whatlanguage` command-line executable that reads from files given as

data/README.md

@@ -17,14 +17,22 @@ WhatLanguage.language("Que linguagem é essa? É uma pergunta sobre a língua po
 - **Best on sentence-length text or longer.** Short texts can have ambiguous results.
 
 > [!IMPORTANT]
-> v2.0 has many breaking changes as the entire library has been reimplemented, though the core `WhatLanguage.language` API remains similar. Versions 1.0.6 and earlier (so the 2007-2025 run of the library) used a Bloom-filter technique and had 5MB of binary files to handle ~20 languages. Version 2.0 is more accurate, faster, and supports more languages from a single 220KB JSON file :-)
+> Version 3.0 changes nine ISO output codes, speeds up detection, and reduces
+> allocations. See [Upgrading to 3.0](#upgrading-to-30) for the code changes.
 
 ## How it works
 
-Detection is in two stages. First, the dominant Unicode script is detected; scripts used by a single language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.
+Detection is in two stages. First, the dominant Unicode script is detected from letters; scripts used by a single supported language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. When kana is present, Han, Hiragana, and Katakana count together as Japanese, while a dominant unrelated script can still win. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.
+
+Digits, punctuation, and emoji do not count toward the minimum text length and are treated as separators when extracting trigrams. Combining marks are preserved in trigrams but do not count as additional letters. Unicode script coverage follows the Unicode version supported by your Ruby runtime.
 
 The trigram profiles are vendored from [whatlang](https://github.com/greyblake/whatlang-rs), a port of [Franc](https://github.com/wooorm/franc), whose models are built from the public-domain UDHR corpus (see Credits). The model is a ~220 KB JSON file.
 
+Repeated calls over the full language set use a shared trigram index. A rolling
+window counts input trigrams without building a character array. Scores and
+rankings are unchanged. See the [indexing benchmarks](benchmark/README.md) and
+[long-input benchmarks](benchmark/long-inputs.md) for timings, allocations, and
+memory costs.
 
 ## Usage
 
@@ -40,7 +48,17 @@ result.score      # => 79018
 result.ranked     # => [[:german, 79018], [:dutch, 77631], ... ]
 ```
 
-Return ranked scores, or the raw score hash:
+`result.iso` and `language_iso` return ISO 639-1 symbols with an ISO 639-3
+fallback. Selected individual languages use CLDR's macrolanguage aliases, such
+as Swahili `swh → sw`, Mandarin `cmn → zh`, and Western Farsi `pes → fa`;
+historical code mappings are also preserved. These aliases are a normalization
+policy, not a BCP 47 requirement: `swh` remains a valid individual-language
+subtag. Languages without a selected alias retain their individual codes.
+The CLI uses the same mapping. Codes are not necessarily unique: both
+`:northern_uzbek` and `:uzbek` return `:uz`. Use `result.language` to distinguish
+the language names.
+
+Return ranked scores, or the score hash:
 
 ```ruby
 wl.ranked(text)       # => [[:german, 79018], [:dutch, 77631], ... ]
@@ -53,12 +71,46 @@ Restrict candidate languages:
 wl = WhatLanguage.new(only: [:english, :german, :french])
 ```
 
-Short Latin-script fragments are ignored by default because there is not enough signal to rank shared-script languages reliably. The threshold applies to the statistical trigram stage; scripts that identify a single supported language, such as Greek, Korean, or Thai, can still resolve from shorter text. The threshold can be adjusted:
+For short texts, English, Chinese (Mandarin), Hindi, Spanish, and French receive
+a built-in preference. These five were selected using the
+[2025 total-speaker ranking](https://www.visualcapitalist.com/ranked-the-worlds-most-spoken-languages-in-2025/)
+(including second-language speakers). Eligible trigram candidates receive the
+same bonus: 600 ranking points through 10 letters, decreasing linearly to zero
+at 50 letters. Longer texts keep their original scores. The preference is not
+configurable; it respects `only:` and script detection. Chinese already resolves
+from its script, so it does not need a statistical bonus. Scores exposed by the
+API include the bonus and remain ranking values, not probabilities.
+
+Short shared-script fragments are ignored by default because there is not enough signal to rank their languages reliably. The threshold applies to the statistical trigram stage; scripts that identify a single supported language, such as Greek, Korean, or Thai, can still resolve from shorter text. The threshold can be adjusted:
 
 ```ruby
 wl = WhatLanguage.new(min_chars: 0)
 ```
 
+## Upgrading to 3.0
+
+Update comparisons against `result.iso`, `language_iso`, and CLI output:
+
+| Language | Previous code | 3.0 code |
+| --- | --- | --- |
+| Swahili | `swh` | `sw` |
+| Central Aymara | `ayr` | `ay` |
+| Central Kanuri | `knc` | `kr` |
+| Cusco Quechua | `quz` | `qu` |
+| Halh Mongolian | `khk` | `mn` |
+| Koongo | `kng` | `kg` |
+| Logudorese Sardinian | `src` | `sc` |
+| Northern Uzbek | `uzn` | `uz` |
+| Tosk Albanian | `als` | `sq` |
+
+Both Northern Uzbek and Uzbek now return `uz`; use `result.language` to
+distinguish them. Language-name symbols, detection scores, rankings, options,
+and return types are unchanged from 2.1.
+
+If upgrading from 1.x, see the [2.0 changelog](CHANGELOG.md#200--2026-06-10) too.
+Version 2.0 replaced the Bloom-filter engine with trigram profiles and removed
+the `String#language` monkeypatch.
+
 ## Command line
 
 Installing the gem also installs a `whatlanguage` executable that reads from files given as arguments (or stdin) and prints the ISO 639 code of the detected language, or `und` if undetermined:
@@ -72,11 +124,11 @@ de
 
 ## Known limitations
 
-- Short fragments are unreliable. For languages resolved by statistical comparison, fewer than 10 significant characters returns `nil` by default. **This is the biggest thing I want to fix.** I think there should be a mechanism for "weighting" languages so that less likely languages don't frequently appear for small English samples, say.
-- Scores are relative ranking values, not probabilities. Use `#ranked` or `#detect.ranked` when close runners-up matter.
-- Closely related written languages can be hard to separate, especially Norwegian Bokmål/Danish, Hebrew/Yiddish, and similar language pairs.
-- Kanji-only Japanese text can classify as Chinese because Han characters alone do not identify the language.
-- Romanized text is classified by Latin-script trigram profiles; it is not treated as native-script text.
+- Short texts can be ambiguous. Built-in preferences help some phrases but can bias others toward preferred languages. Statistical detection returns `nil` below 10 letters by default.
+- Scores indicate relative rank, not probability or confidence. Use `#ranked` to inspect alternatives.
+- Similar written languages, such as Norwegian Bokmål/Danish and Hebrew/Yiddish, can be confused.
+- Japanese without kana is classified as Chinese when Han is the dominant script.
+- Romanized text is compared with Latin-script profiles, so it may not resolve to its original language.
 
 ## Credits
 

data/lib/whatlanguage/languages.rb

@@ -1,8 +1,12 @@
 # frozen_string_literal: true
 
-# AUTO-GENERATED from the ISO 639-3 registry + whatlang dataset codes.
+# Generated from the ISO 639-3 registry + whatlang dataset codes, with overrides.
 # Maps ISO 639-3 code => [language name symbol, ISO 639-1 (or 639-3 fallback) symbol].
 # Original gem languages keep their historical symbols and 2-letter codes.
+# Selected individual languages use CLDR's 2-letter macrolanguage aliases:
+# als, ayr, cmn, khk, knc, kng, pes, quz, src, swh, uzn. Preserve these overrides
+# when regenerating; do not map every macrolanguage member to its parent code.
+# Source: https://github.com/unicode-org/cldr/blob/main/common/supplemental/supplementalMetadata.xml
 class WhatLanguage
   CODE_INFO = {
     "ace"   => [:achinese,               :ace],
@@ -30,16 +34,16 @@ class WhatLanguage
     "cat"   => [:catalan,                :ca],
     "ceb"   => [:cebuano,                :ceb],
     "tzm"   => [:central_atlas_tamazight, :tzm],
-    "ayr"   => [:central_aymara,         :ayr],
+    "ayr"   => [:central_aymara,         :ay],
     "bcl"   => [:central_bikol,          :bcl],
-    "knc"   => [:central_kanuri,         :knc],
+    "knc"   => [:central_kanuri,         :kr],
     "nhn"   => [:central_nahuatl,        :nhn],
     "nya"   => [:chichewa,               :ny],
     "qug"   => [:chimborazo_highland_quichua, :qug],
     "cmn"   => [:chinese,                :zh],
     "cjk"   => [:chokwe,                 :cjk],
     "hrv"   => [:croatian,               :hr],
-    "quz"   => [:cusco_quechua,          :quz],
+    "quz"   => [:cusco_quechua,          :qu],
     "ces"   => [:czech,                  :cs],
     "dan"   => [:danish,                 :da],
     "nld"   => [:dutch,                  :nl],
@@ -61,7 +65,7 @@ class WhatLanguage
     "ell"   => [:greek,                  :el],
     "guj"   => [:gujarati,               :gu],
     "hat"   => [:haitian,                :ht],
-    "khk"   => [:halh_mongolian,         :khk],
+    "khk"   => [:halh_mongolian,         :mn],
     "hau"   => [:hausa,                  :ha],
     "heb"   => [:hebrew,                 :he],
     "hil"   => [:hiligaynon,             :hil],
@@ -82,14 +86,14 @@ class WhatLanguage
     "kin"   => [:kinyarwanda,            :rw],
     "kir"   => [:kirghiz,                :ky],
     "koi"   => [:komi_permyak,           :koi],
-    "kng"   => [:koongo,                 :kng],
+    "kng"   => [:koongo,                 :kg],
     "kor"   => [:korean,                 :ko],
     "kur"   => [:kurdish,                :ku],
     "lat"   => [:latin,                  :la],
     "lav"   => [:latvian,                :lv],
     "lin"   => [:lingala,                :ln],
     "lit"   => [:lithuanian,             :lt],
-    "src"   => [:logudorese_sardinian,   :src],
+    "src"   => [:logudorese_sardinian,   :sc],
     "nds"   => [:low_german,             :nds],
     "lua"   => [:luba_lulua,             :lua],
     "lun"   => [:lunda,                  :lun],
@@ -108,7 +112,7 @@ class WhatLanguage
     "ndo"   => [:ndonga,                 :ng],
     "nep"   => [:nepali,                 :ne],
     "dip"   => [:northeastern_dinka,     :dip],
-    "uzn"   => [:northern_uzbek,         :uzn],
+    "uzn"   => [:northern_uzbek,         :uz],
     "nob"   => [:norwegian,              :no],
     "nno"   => [:norwegian_nynorsk,      :nn],
     "nyn"   => [:nyankole,               :nyn],
@@ -137,7 +141,7 @@ class WhatLanguage
     "spa"   => [:spanish,                :es],
     "suk"   => [:sukuma,                 :suk],
     "sun"   => [:sundanese,              :su],
-    "swh"   => [:swahili,                :swh],
+    "swh"   => [:swahili,                :sw],
     "ssw"   => [:swati,                  :ss],
     "swe"   => [:swedish,                :sv],
     "tgl"   => [:tagalog,                :tl],
@@ -151,7 +155,7 @@ class WhatLanguage
     "tiv"   => [:tiv,                    :tiv],
     "tpi"   => [:tok_pisin,              :tpi],
     "toi"   => [:tonga,                  :toi],
-    "als"   => [:tosk_albanian,          :als],
+    "als"   => [:tosk_albanian,          :sq],
     "tso"   => [:tsonga,                 :ts],
     "tsn"   => [:tswana,                 :tn],
     "tur"   => [:turkish,                :tr],

data/lib/whatlanguage/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class WhatLanguage
-  VERSION = '2.0.1'
+  VERSION = '3.0.0'
 end

data/lib/whatlanguage.rb

@@ -9,12 +9,15 @@ class WhatLanguage
   MAX_TOTAL_DISTANCE   = MAX_TRIGRAM_DISTANCE * MAX_TRIGRAM_DISTANCE # 90_000
   TEXT_TRIGRAMS_SIZE   = 600
   DEFAULT_MIN_CHARS    = 10
+  SHORT_TEXT_PREFERENCES = %i[english chinese hindi spanish french].freeze
+  SHORT_TEXT_BONUS = 600
+  SHORT_TEXT_FADE_CHARS = 50
 
   Result = Struct.new(:language, :iso, :score, :ranked, keyword_init: true) do
     alias scores ranked
   end
 
-  # Scripts that resolve to a single language by their Unicode block alone.
+  # Scripts that resolve to a single supported language from their letters.
   # (Hiragana and Katakana both indicate Japanese.) Scripts NOT listed here but
   # present in the trigram dataset are disambiguated statistically instead.
   DETERMINISTIC = {
@@ -27,43 +30,13 @@ class WhatLanguage
     'Katakana'  => 'jpn'
   }.freeze
 
-  # Unicode ranges per script, in detection priority order (mirrors whatlang's
-  # scripts/detect.rs). The first script whose range contains a character claims
-  # that character; the script with the most characters wins.
-  SCRIPT_RANGES = [
-    ['Latin',      [[0x61,0x7A],[0x41,0x5A],[0x80,0xFF],[0x100,0x17F],[0x180,0x24F],
-                    [0x250,0x2AF],[0x1D00,0x1D7F],[0x1D80,0x1DBF],[0x1E00,0x1EFF],
-                    [0x2100,0x214F],[0x2C60,0x2C7F],[0xA720,0xA7FF],[0xAB30,0xAB6F]]],
-    ['Cyrillic',   [[0x400,0x484],[0x487,0x52F],[0x2DE0,0x2DFF],[0xA640,0xA69D],
-                    [0x1D2B,0x1D2B],[0x1D78,0x1D78],[0xA69F,0xA69F]]],
-    ['Arabic',     [[0x600,0x6FF],[0x750,0x7FF],[0x8A0,0x8FF],[0xFB50,0xFDFF],
-                    [0xFE70,0xFEFF],[0x10E60,0x10E7F],[0x1EE00,0x1EEFF]]],
-    ['Mandarin',   [[0x2E80,0x2E99],[0x2E9B,0x2EF3],[0x2F00,0x2FD5],[0x3005,0x3005],
-                    [0x3007,0x3007],[0x3021,0x3029],[0x3038,0x303B],[0x3400,0x4DB5],
-                    [0x4E00,0x9FCC],[0xF900,0xFA6D],[0xFA70,0xFAD9]]],
-    ['Devanagari', [[0x900,0x97F],[0xA8E0,0xA8FF],[0x1CD0,0x1CFF]]],
-    ['Hebrew',     [[0x590,0x5FF]]],
-    ['Ethiopic',   [[0x1200,0x139F],[0x2D80,0x2DDF],[0xAB00,0xAB2F]]],
-    ['Georgian',   [[0x10A0,0x10FF]]],
-    ['Bengali',    [[0x980,0x9FF]]],
-    ['Hangul',     [[0xAC00,0xD7AF],[0x1100,0x11FF],[0x3130,0x318F],[0x3200,0x32FF],
-                    [0xA960,0xA97F],[0xD7B0,0xD7FF]]],
-    ['Hiragana',   [[0x3040,0x309F]]],
-    ['Katakana',   [[0x30A0,0x30FF]]],
-    ['Greek',      [[0x370,0x3FF]]],
-    ['Kannada',    [[0xC80,0xCFF]]],
-    ['Tamil',      [[0xB80,0xBFF]]],
-    ['Thai',       [[0xE00,0xE7F]]],
-    ['Gujarati',   [[0xA80,0xAFF]]],
-    ['Gurmukhi',   [[0xA00,0xA7F]]],
-    ['Telugu',     [[0xC00,0xC7F]]],
-    ['Malayalam',  [[0xD00,0xD7F]]],
-    ['Oriya',      [[0xB00,0xB7F]]],
-    ['Myanmar',    [[0x1000,0x109F]]],
-    ['Sinhala',    [[0xD80,0xDFF]]],
-    ['Khmer',      [[0x1780,0x17FF],[0x19E0,0x19FF]]],
-    ['Armenian',   [[0x530,0x58F],[0xFB13,0xFB17]]]
-  ].freeze
+  # Unicode script properties include letters outside the original BMP blocks.
+  # Keep the model's historical "Mandarin" key for the Han script.
+  SCRIPT_PATTERNS = %w[
+    Latin Cyrillic Arabic Han Devanagari Hebrew Ethiopic Georgian Bengali
+    Hangul Hiragana Katakana Greek Kannada Tamil Thai Gujarati Gurmukhi
+    Telugu Malayalam Oriya Myanmar Sinhala Khmer Armenian
+  ].map { |script| [script == 'Han' ? 'Mandarin' : script, Regexp.new("\\p{#{script}}")] }.freeze
 
   # ISO 639-1 (with 639-3 fallback) lookup by language-name symbol, plus the
   # historical nil => nil entry. Internal; kept for backward compatibility.
@@ -76,8 +49,9 @@ class WhatLanguage
   end.freeze
 
   private_constant :MAX_TRIGRAM_DISTANCE, :MAX_TOTAL_DISTANCE, :TEXT_TRIGRAMS_SIZE,
-                   :DEFAULT_MIN_CHARS, :DETERMINISTIC, :SCRIPT_RANGES, :ISO_CODES,
-                   :NAME_TO_CODE
+                   :DEFAULT_MIN_CHARS, :DETERMINISTIC, :SCRIPT_PATTERNS, :ISO_CODES,
+                   :NAME_TO_CODE, :SHORT_TEXT_PREFERENCES, :SHORT_TEXT_BONUS,
+                   :SHORT_TEXT_FADE_CHARS
 
   class << self
     def detect(text)
@@ -118,6 +92,30 @@ class WhatLanguage
     def default_detector
       @default_detector ||= new
     end
+
+    # Shared index for a script: trigram => flat [candidate, rank, ...].
+    # Use the direct scan on the first call: one-shot CLI use should not pay
+    # to build an index. Flat arrays avoid an object for every posting.
+    def profile_index(script)
+      @profile_indexes ||= {}
+      unless @profile_indexes.key?(script)
+        @profile_indexes[script] = nil
+        return
+      end
+
+      @profile_indexes[script] ||= begin
+        candidates = profiles.fetch(script)
+        postings = {}
+        candidates.each_with_index do |(_code, trigrams), candidate|
+          trigrams.each_with_index do |trigram, rank|
+            (postings[trigram] ||= []).push(candidate, rank)
+          end
+        end
+        postings.each_value(&:freeze)
+        missing = candidates.map { |_code, trigrams| trigrams.size * MAX_TRIGRAM_DISTANCE }
+        [missing.freeze, postings.freeze].freeze
+      end
+    end
   end
 
   def initialize(*selection, only: nil, min_chars: DEFAULT_MIN_CHARS)
@@ -145,7 +143,7 @@ class WhatLanguage
   def score_hash(text)
     results = Hash.new(0)
     text = normalize_text(text)
-    script = detect_script(text)
+    script, char_count = script_and_letter_count(text)
     return results unless script
 
     if (code = DETERMINISTIC[script])
@@ -154,16 +152,31 @@ class WhatLanguage
       return results
     end
 
-    candidates = self.class.profiles[script]
+    candidates = candidates_for(script)
     return results unless candidates
-    return results if significant_char_count(text) < @min_chars
+    return results if char_count < @min_chars
 
+    bonus = short_text_bonus(char_count)
     positions = trigram_positions(text)
-    candidates.each do |code, trigrams|
-      name = CODE_INFO[code].first
-      next unless allowed?(name)
+    # Small selections retain the direct scan rather than visiting postings for
+    # languages they cannot return.
+    distances = indexed_distances(script, positions) if @selection.include?(:all)
+    if distances
+      candidates.each_with_index do |(code, _trigrams), i|
+        name = CODE_INFO[code].first
+        next unless allowed?(name)
+
+        results[name] = MAX_TOTAL_DISTANCE - distances[i]
+        results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
+      end
+    else
+      candidates.each do |code, trigrams|
+        name = CODE_INFO[code].first
+        next unless allowed?(name)
 
-      results[name] = MAX_TOTAL_DISTANCE - distance(trigrams, positions)
+        results[name] = MAX_TOTAL_DISTANCE - distance(trigrams, positions)
+        results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
+      end
     end
     results
   end
@@ -188,16 +201,26 @@ class WhatLanguage
 
   # Most likely language as a name symbol, or nil when no language is detected.
   def language(text)
-    detect(text)&.language
+    score_hash(text).max_by { |_name, score| score }&.first
   end
 
   # Most likely language as an ISO 639-1 symbol (639-3 fallback), or nil.
+  # Includes selected CLDR macrolanguage aliases and historical code overrides;
+  # see whatlanguage/languages.rb. Codes need not uniquely identify a language.
   def language_iso(text)
-    detect(text)&.iso
+    ISO_CODES[language(text)]
   end
 
   private
 
+  # A small fixed prior for widely spoken languages: full strength through ten
+  # letters, fading linearly to zero at fifty. These are ranking points, not
+  # probabilities. Script routing and the caller's candidate selection still win.
+  def short_text_bonus(char_count)
+    remaining = (SHORT_TEXT_FADE_CHARS - char_count).clamp(0, SHORT_TEXT_FADE_CHARS - DEFAULT_MIN_CHARS)
+    SHORT_TEXT_BONUS * remaining / (SHORT_TEXT_FADE_CHARS - DEFAULT_MIN_CHARS)
+  end
+
   def normalize_text(text)
     text.to_s.unicode_normalize(:nfkc)
   end
@@ -206,6 +229,20 @@ class WhatLanguage
     @selection.include?(:all) || languages.include?(name)
   end
 
+  def candidates_for(script)
+    candidates = self.class.profiles[script]
+    return candidates unless candidates && !@selection.include?(:all)
+
+    # languages returns a mutable array. Invalidate the cached selection when
+    # callers change it, preserving the existing selection behavior.
+    wanted = languages
+    if @selected_languages != wanted
+      @selected_languages = wanted.dup
+      @selected_profiles = {}
+    end
+    @selected_profiles[script] ||= candidates.select { |code, _trigrams| allowed?(CODE_INFO[code].first) }
+  end
+
   def validate_selection!
     requested = @selection.reject { |name| name == :all }
     unknown = requested.reject { |name| self.class.languages.include?(NAME_ALIASES.fetch(name, name)) }
@@ -214,53 +251,134 @@ class WhatLanguage
     raise ArgumentError, "Unknown language selection: #{unknown.map(&:inspect).join(', ')}"
   end
 
-  def significant_char_count(text)
-    text.each_char.count { |ch| !stop_char?(ch.ord) }
-  end
+  # Count all letters, including those outside the recognized scripts.
+  def script_and_letter_count(text)
+    if text.ascii_only?
+      letter_count = text.count('A-Za-z')
+      return [letter_count.zero? ? nil : 'Latin', letter_count]
+    end
 
-  # Dominant Unicode script of the text, or nil if it has no script characters.
-  def detect_script(text)
     counts = Hash.new(0)
+    letter_count = 0
     text.each_char do |ch|
-      cp = ch.ord
-      next if stop_char?(cp)
+      next unless /\p{L}/.match?(ch)
+      letter_count += 1
 
-      SCRIPT_RANGES.each do |name, ranges|
-        if ranges.any? { |lo, hi| cp >= lo && cp <= hi }
+      i = 0
+      while i < SCRIPT_PATTERNS.size
+        name, pattern = SCRIPT_PATTERNS[i]
+        if pattern.match?(ch)
           counts[name] += 1
           break
         end
+        i += 1
       end
     end
-    return nil if counts.empty?
+    return [nil, letter_count] if counts.empty?
+
+    # Japanese uses Han, Hiragana, and Katakana together. Combine their votes
+    # when kana is present, but still let a dominant unrelated script win.
+    if counts['Hiragana'] + counts['Katakana'] > 0
+      japanese = counts['Mandarin'] + counts['Hiragana'] + counts['Katakana']
+      counts.delete('Mandarin')
+      counts.delete('Katakana')
+      counts['Hiragana'] = japanese
+    end
 
-    counts.max_by { |_name, n| n }.first
+    [counts.max_by { |_name, n| n }.first, letter_count]
   end
 
   # Text trigrams ranked by descending frequency, mapped to their rank index.
-  # Mirrors whatlang's trigram extraction: punctuation/digits become spaces,
-  # the stream is bounded by spaces, and runs of spaces are collapsed.
+  # Non-letter/mark characters become spaces. Preserve combining marks used
+  # by the profiles; bound the stream by spaces and ignore repeated spaces.
   def trigram_positions(text)
-    chars = text.downcase.each_char.map { |c| stop_char?(c.ord) ? ' ' : c }
-    return {} if chars.empty?
+    text = text.downcase
+    occurrences = text.ascii_only? ? ascii_trigram_counts(text) : unicode_trigram_counts(text)
+    ranked = occurrences.to_a.sort! do |a, b|
+      order = b[1] <=> a[1]
+      order.zero? ? b[0] <=> a[0] : order
+    end.first(TEXT_TRIGRAMS_SIZE)
+    positions = {}
+    ranked.each_with_index { |(trigram, _count), i| positions[trigram] = i }
+    positions
+  end
 
+  def unicode_trigram_counts(text)
     occurrences = Hash.new(0)
-    c1 = ' '
-    c2 = chars[0]
-    (chars[1..] + [' ']).each do |c3|
-      occurrences[c1 + c2 + c3] += 1 unless c2 == ' ' && (c1 == ' ' || c3 == ' ')
+    trigram = +''
+    c1 = nil
+    c2 = ' '
+    text.each_char do |character|
+      c3 = /[\p{L}\p{M}]/.match?(character) ? character : ' '
+      if c1 && !(c2 == ' ' && (c1 == ' ' || c3 == ' '))
+        # Hash copies new String keys, so this buffer can be reused for every
+        # window without changing already-counted trigrams.
+        trigram.replace(c1) << c2 << c3
+        occurrences[trigram] += 1
+      end
       c1 = c2
       c2 = c3
     end
+    if c1 && c2 != ' '
+      trigram.replace(c1) << c2 << ' '
+      occurrences[trigram] += 1
+    end
 
-    ranked = occurrences.to_a.sort { |a, b| [b[1], b[0]] <=> [a[1], a[0]] }.first(TEXT_TRIGRAMS_SIZE)
-    positions = {}
-    ranked.each_with_index { |(trigram, _count), i| positions[trigram] = i }
-    positions
+    occurrences
+  end
+
+  # Lowercased ASCII has only a-z letters and no combining marks. Use byte
+  # values and a reusable three-byte key to avoid allocating character strings.
+  def ascii_trigram_counts(text)
+    occurrences = Hash.new(0)
+    trigram = +"\0\0\0"
+    c1 = nil
+    c2 = 32
+    text.each_byte do |byte|
+      c3 = byte.between?(97, 122) ? byte : 32
+      if c1 && !(c2 == 32 && (c1 == 32 || c3 == 32))
+        trigram.setbyte(0, c1)
+        trigram.setbyte(1, c2)
+        trigram.setbyte(2, c3)
+        occurrences[trigram] += 1
+      end
+      c1 = c2
+      c2 = c3
+    end
+    if c1 && c2 != 32
+      trigram.setbyte(0, c1)
+      trigram.setbyte(1, c2)
+      trigram.setbyte(2, 32)
+      occurrences[trigram] += 1
+    end
+    occurrences
   end
 
   # Out-of-place distance between a language's ordered trigram profile and the
   # text's ranked trigrams. Lower means a closer match.
+  def indexed_distances(script, positions)
+    index = self.class.send(:profile_index, script)
+    return unless index
+
+    missing, postings = index
+    totals = missing.dup
+    positions.each do |trigram, position|
+      matches = postings[trigram]
+      next unless matches
+
+      i = 0
+      while i < matches.size
+        # Replace the missing-trigram penalty with the exact rank distance.
+        # This correction can be positive when the text rank is above 300.
+        totals[matches[i]] += (position - matches[i + 1]).abs - MAX_TRIGRAM_DISTANCE
+        i += 2
+      end
+    end
+
+    adjustment = [MAX_TRIGRAM_DISTANCE - positions.size, 0].max * MAX_TRIGRAM_DISTANCE
+    totals.map! { |total| (total - adjustment).clamp(0, MAX_TOTAL_DISTANCE) }
+  end
+
   def distance(profile, positions)
     total = 0
     profile.each_with_index do |trigram, i|
@@ -272,9 +390,4 @@ class WhatLanguage
     total -= (MAX_TRIGRAM_DISTANCE - count) * MAX_TRIGRAM_DISTANCE if MAX_TRIGRAM_DISTANCE > count
     total.clamp(0, MAX_TOTAL_DISTANCE)
   end
-
-  # Space, ASCII punctuation, or digit: no value for script/language detection.
-  def stop_char?(codepoint)
-    codepoint <= 0x40 || (codepoint >= 0x5B && codepoint <= 0x60) || (codepoint >= 0x7B && codepoint <= 0x7E)
-  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: whatlanguage
 version: !ruby/object:Gem::Version
-  version: 2.0.1
+  version: 3.0.0
 platform: ruby
 authors:
 - Peter Cooper