whatlanguage 2.1.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4707fce379ca4e085886aef33b8bd45f3480c79bd93c1a7d040c068aff16a94c
-  data.tar.gz: e4b0156f7d3ee2dda4efb77cd2cd8c6abd6f96eac8c27d821166f927fe7421e6
+  metadata.gz: 93967e970b5e62eb58b9cb835866d4abe885f98761f7a339d0817dad8414eac8
+  data.tar.gz: b7e983bcd0a56f7b6f77675fdedd6d7724da71b42e26e3229014a92a3c32166b
 SHA512:
-  metadata.gz: fa04498461b8e07de91debf3ddf5ebece5d701a2006fe45a08a0baa7f1d8e7947a1c90fa726b175cbe08a3201ac510041f3f952f4b89abfd61ba0d42de0e5c77
-  data.tar.gz: 7f2ba53f9248b9f8559c06758205874b61a624c485a1e6bad02f5d192a5a93e09ae38018c73a5ac46bc93dc9e742a648757b6e00fc75ed12698588530f0124a9
+  metadata.gz: f7c259a244e6c0fe0aa9084037fb29381bc96c9b5a4e549382b660b95943d82bf651d5e706fc4290cb93ff2bc820b1c15495f70bec025c5b98bfa3fb0ff91d6a
+  data.tar.gz: 6b2d67686a98ec148d4a3f3314cd270beee7ad3b3437411b8a516d69d6078d3a1a3bd230a4d7e06a5904f7b286339e8be1b2cfef29ba9df73e0da76efb0044a3

data/CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## 3.0.0 / 2026-10-20
+
+### Breaking change: nine ISO output codes
+
+- Normalized nine individual-language ISO outputs to CLDR macrolanguage aliases
+  (#51): `swh → sw`, `ayr → ay`, `knc → kr`, `quz → qu`, `khk → mn`,
+  `kng → kg`, `src → sc`, `uzn → uz`, and `als → sq`. This changes
+  `Result#iso`, instance/class `language_iso`, and CLI output; callers comparing
+  the previous codes must update those comparisons. Language names and detection
+  scores are unchanged. Northern Uzbek and Uzbek now both return `uz`.
+
+### Improvements
+
+- Preserved the individual codes `quy`, `qug`, `koi`, and `min`, and documented
+  the selected CLDR aliases and historical overrides used for ISO output.
+- Added shared trigram indexes to speed up repeated detection over the full
+  language set, without changing scores or rankings. First calls and restricted
+  selections retain the original scan.
+- Reduced text-processing allocations with rolling trigram windows, a combined
+  script/letter-count pass, and an ASCII fast path. Cached restricted candidates
+  and avoided sorting when only the winning language is requested.
+- Added equivalence tests and reproducible benchmarks for
+  [indexed scoring](benchmark/README.md) and
+  [text processing](benchmark/long-inputs.md), including allocation and memory costs.
+
 ## 2.1.0 / 2026-09-05
 
 - Added a fixed short-text preference for English, Chinese, Hindi, Spanish, and

data/README.md

@@ -17,21 +17,22 @@ WhatLanguage.language("Que linguagem é essa? É uma pergunta sobre a língua po
 - **Best on sentence-length text or longer.** Short texts can have ambiguous results.
 
 > [!IMPORTANT]
-> v2.0 has many breaking changes as the entire library has been reimplemented, though the core `WhatLanguage.language` API remains similar. Versions 1.0.6 and earlier (so the 2007-2025 run of the library) used a Bloom-filter technique and had 5MB of binary files to handle ~20 languages. Version 2.0 is more accurate, faster, and supports more languages from a single 220KB JSON file :-)
+> Version 3.0 changes nine ISO output codes, speeds up detection, and reduces
+> allocations. See [Upgrading to 3.0](#upgrading-to-30) for the code changes.
 
 ## How it works
 
-> [!NOTE]
-> Version 2.1 improves Unicode and mixed-script Japanese handling and adds a fixed
-> preference for five widely spoken languages on short texts. The public API and
-> default 10-letter minimum for statistical detection are unchanged.
-
 Detection is in two stages. First, the dominant Unicode script is detected from letters; scripts used by a single supported language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. When kana is present, Han, Hiragana, and Katakana count together as Japanese, while a dominant unrelated script can still win. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.
 
 Digits, punctuation, and emoji do not count toward the minimum text length and are treated as separators when extracting trigrams. Combining marks are preserved in trigrams but do not count as additional letters. Unicode script coverage follows the Unicode version supported by your Ruby runtime.
 
 The trigram profiles are vendored from [whatlang](https://github.com/greyblake/whatlang-rs), a port of [Franc](https://github.com/wooorm/franc), whose models are built from the public-domain UDHR corpus (see Credits). The model is a ~220 KB JSON file.
 
+Repeated calls over the full language set use a shared trigram index. A rolling
+window counts input trigrams without building a character array. Scores and
+rankings are unchanged. See the [indexing benchmarks](benchmark/README.md) and
+[long-input benchmarks](benchmark/long-inputs.md) for timings, allocations, and
+memory costs.
 
 ## Usage
 
@@ -47,6 +48,16 @@ result.score      # => 79018
 result.ranked     # => [[:german, 79018], [:dutch, 77631], ... ]
 ```
 
+`result.iso` and `language_iso` return ISO 639-1 symbols with an ISO 639-3
+fallback. Selected individual languages use CLDR's macrolanguage aliases, such
+as Swahili `swh → sw`, Mandarin `cmn → zh`, and Western Farsi `pes → fa`;
+historical code mappings are also preserved. These aliases are a normalization
+policy, not a BCP 47 requirement: `swh` remains a valid individual-language
+subtag. Languages without a selected alias retain their individual codes.
+The CLI uses the same mapping. Codes are not necessarily unique: both
+`:northern_uzbek` and `:uzbek` return `:uz`. Use `result.language` to distinguish
+the language names.
+
 Return ranked scores, or the score hash:
 
 ```ruby
@@ -76,6 +87,30 @@ Short shared-script fragments are ignored by default because there is not enough
 wl = WhatLanguage.new(min_chars: 0)
 ```
 
+## Upgrading to 3.0
+
+Update comparisons against `result.iso`, `language_iso`, and CLI output:
+
+| Language | Previous code | 3.0 code |
+| --- | --- | --- |
+| Swahili | `swh` | `sw` |
+| Central Aymara | `ayr` | `ay` |
+| Central Kanuri | `knc` | `kr` |
+| Cusco Quechua | `quz` | `qu` |
+| Halh Mongolian | `khk` | `mn` |
+| Koongo | `kng` | `kg` |
+| Logudorese Sardinian | `src` | `sc` |
+| Northern Uzbek | `uzn` | `uz` |
+| Tosk Albanian | `als` | `sq` |
+
+Both Northern Uzbek and Uzbek now return `uz`; use `result.language` to
+distinguish them. Language-name symbols, detection scores, rankings, options,
+and return types are unchanged from 2.1.
+
+If upgrading from 1.x, see the [2.0 changelog](CHANGELOG.md#200--2026-06-10) too.
+Version 2.0 replaced the Bloom-filter engine with trigram profiles and removed
+the `String#language` monkeypatch.
+
 ## Command line
 
 Installing the gem also installs a `whatlanguage` executable that reads from files given as arguments (or stdin) and prints the ISO 639 code of the detected language, or `und` if undetermined:
@@ -89,11 +124,11 @@ de
 
 ## Known limitations
 
-- Short fragments remain unreliable. For languages resolved by statistical comparison, fewer than 10 letters returns `nil` by default. The built-in preference helps some common phrases, but can also favor a preferred language over a less widely spoken one when evidence is weak. It does not lower the minimum length.
-- Scores are relative ranking values, not probabilities. Use `#ranked` or `#detect.ranked` when close runners-up matter.
-- Closely related written languages can be hard to separate, especially Norwegian Bokmål/Danish, Hebrew/Yiddish, and similar language pairs.
-- Kanji-only Japanese text can classify as Chinese because Han characters alone do not identify the language.
-- Romanized text is classified by Latin-script trigram profiles; it is not treated as native-script text.
+- Short texts can be ambiguous. Built-in preferences help some phrases but can bias others toward preferred languages. Statistical detection returns `nil` below 10 letters by default.
+- Scores indicate relative rank, not probability or confidence. Use `#ranked` to inspect alternatives.
+- Similar written languages, such as Norwegian Bokmål/Danish and Hebrew/Yiddish, can be confused.
+- Japanese without kana is classified as Chinese when Han is the dominant script.
+- Romanized text is compared with Latin-script profiles, so it may not resolve to its original language.
 
 ## Credits
 

data/lib/whatlanguage/languages.rb

@@ -1,8 +1,12 @@
 # frozen_string_literal: true
 
-# AUTO-GENERATED from the ISO 639-3 registry + whatlang dataset codes.
+# Generated from the ISO 639-3 registry + whatlang dataset codes, with overrides.
 # Maps ISO 639-3 code => [language name symbol, ISO 639-1 (or 639-3 fallback) symbol].
 # Original gem languages keep their historical symbols and 2-letter codes.
+# Selected individual languages use CLDR's 2-letter macrolanguage aliases:
+# als, ayr, cmn, khk, knc, kng, pes, quz, src, swh, uzn. Preserve these overrides
+# when regenerating; do not map every macrolanguage member to its parent code.
+# Source: https://github.com/unicode-org/cldr/blob/main/common/supplemental/supplementalMetadata.xml
 class WhatLanguage
   CODE_INFO = {
     "ace"   => [:achinese,               :ace],
@@ -30,16 +34,16 @@ class WhatLanguage
     "cat"   => [:catalan,                :ca],
     "ceb"   => [:cebuano,                :ceb],
     "tzm"   => [:central_atlas_tamazight, :tzm],
-    "ayr"   => [:central_aymara,         :ayr],
+    "ayr"   => [:central_aymara,         :ay],
     "bcl"   => [:central_bikol,          :bcl],
-    "knc"   => [:central_kanuri,         :knc],
+    "knc"   => [:central_kanuri,         :kr],
     "nhn"   => [:central_nahuatl,        :nhn],
     "nya"   => [:chichewa,               :ny],
     "qug"   => [:chimborazo_highland_quichua, :qug],
     "cmn"   => [:chinese,                :zh],
     "cjk"   => [:chokwe,                 :cjk],
     "hrv"   => [:croatian,               :hr],
-    "quz"   => [:cusco_quechua,          :quz],
+    "quz"   => [:cusco_quechua,          :qu],
     "ces"   => [:czech,                  :cs],
     "dan"   => [:danish,                 :da],
     "nld"   => [:dutch,                  :nl],
@@ -61,7 +65,7 @@ class WhatLanguage
     "ell"   => [:greek,                  :el],
     "guj"   => [:gujarati,               :gu],
     "hat"   => [:haitian,                :ht],
-    "khk"   => [:halh_mongolian,         :khk],
+    "khk"   => [:halh_mongolian,         :mn],
     "hau"   => [:hausa,                  :ha],
     "heb"   => [:hebrew,                 :he],
     "hil"   => [:hiligaynon,             :hil],
@@ -82,14 +86,14 @@ class WhatLanguage
     "kin"   => [:kinyarwanda,            :rw],
     "kir"   => [:kirghiz,                :ky],
     "koi"   => [:komi_permyak,           :koi],
-    "kng"   => [:koongo,                 :kng],
+    "kng"   => [:koongo,                 :kg],
     "kor"   => [:korean,                 :ko],
     "kur"   => [:kurdish,                :ku],
     "lat"   => [:latin,                  :la],
     "lav"   => [:latvian,                :lv],
     "lin"   => [:lingala,                :ln],
     "lit"   => [:lithuanian,             :lt],
-    "src"   => [:logudorese_sardinian,   :src],
+    "src"   => [:logudorese_sardinian,   :sc],
     "nds"   => [:low_german,             :nds],
     "lua"   => [:luba_lulua,             :lua],
     "lun"   => [:lunda,                  :lun],
@@ -108,7 +112,7 @@ class WhatLanguage
     "ndo"   => [:ndonga,                 :ng],
     "nep"   => [:nepali,                 :ne],
     "dip"   => [:northeastern_dinka,     :dip],
-    "uzn"   => [:northern_uzbek,         :uzn],
+    "uzn"   => [:northern_uzbek,         :uz],
     "nob"   => [:norwegian,              :no],
     "nno"   => [:norwegian_nynorsk,      :nn],
     "nyn"   => [:nyankole,               :nyn],
@@ -137,7 +141,7 @@ class WhatLanguage
     "spa"   => [:spanish,                :es],
     "suk"   => [:sukuma,                 :suk],
     "sun"   => [:sundanese,              :su],
-    "swh"   => [:swahili,                :swh],
+    "swh"   => [:swahili,                :sw],
     "ssw"   => [:swati,                  :ss],
     "swe"   => [:swedish,                :sv],
     "tgl"   => [:tagalog,                :tl],
@@ -151,7 +155,7 @@ class WhatLanguage
     "tiv"   => [:tiv,                    :tiv],
     "tpi"   => [:tok_pisin,              :tpi],
     "toi"   => [:tonga,                  :toi],
-    "als"   => [:tosk_albanian,          :als],
+    "als"   => [:tosk_albanian,          :sq],
     "tso"   => [:tsonga,                 :ts],
     "tsn"   => [:tswana,                 :tn],
     "tur"   => [:turkish,                :tr],

data/lib/whatlanguage/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class WhatLanguage
-  VERSION = '2.1.0'
+  VERSION = '3.0.0'
 end

data/lib/whatlanguage.rb

@@ -92,6 +92,30 @@ class WhatLanguage
     def default_detector
       @default_detector ||= new
     end
+
+    # Shared index for a script: trigram => flat [candidate, rank, ...].
+    # Use the direct scan on the first call: one-shot CLI use should not pay
+    # to build an index. Flat arrays avoid an object for every posting.
+    def profile_index(script)
+      @profile_indexes ||= {}
+      unless @profile_indexes.key?(script)
+        @profile_indexes[script] = nil
+        return
+      end
+
+      @profile_indexes[script] ||= begin
+        candidates = profiles.fetch(script)
+        postings = {}
+        candidates.each_with_index do |(_code, trigrams), candidate|
+          trigrams.each_with_index do |trigram, rank|
+            (postings[trigram] ||= []).push(candidate, rank)
+          end
+        end
+        postings.each_value(&:freeze)
+        missing = candidates.map { |_code, trigrams| trigrams.size * MAX_TRIGRAM_DISTANCE }
+        [missing.freeze, postings.freeze].freeze
+      end
+    end
   end
 
   def initialize(*selection, only: nil, min_chars: DEFAULT_MIN_CHARS)
@@ -119,7 +143,7 @@ class WhatLanguage
   def score_hash(text)
     results = Hash.new(0)
     text = normalize_text(text)
-    script = detect_script(text)
+    script, char_count = script_and_letter_count(text)
     return results unless script
 
     if (code = DETERMINISTIC[script])
@@ -128,19 +152,31 @@ class WhatLanguage
       return results
     end
 
-    candidates = self.class.profiles[script]
+    candidates = candidates_for(script)
     return results unless candidates
-    char_count = significant_char_count(text)
     return results if char_count < @min_chars
 
     bonus = short_text_bonus(char_count)
     positions = trigram_positions(text)
-    candidates.each do |code, trigrams|
-      name = CODE_INFO[code].first
-      next unless allowed?(name)
+    # Small selections retain the direct scan rather than visiting postings for
+    # languages they cannot return.
+    distances = indexed_distances(script, positions) if @selection.include?(:all)
+    if distances
+      candidates.each_with_index do |(code, _trigrams), i|
+        name = CODE_INFO[code].first
+        next unless allowed?(name)
+
+        results[name] = MAX_TOTAL_DISTANCE - distances[i]
+        results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
+      end
+    else
+      candidates.each do |code, trigrams|
+        name = CODE_INFO[code].first
+        next unless allowed?(name)
 
-      results[name] = MAX_TOTAL_DISTANCE - distance(trigrams, positions)
-      results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
+        results[name] = MAX_TOTAL_DISTANCE - distance(trigrams, positions)
+        results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
+      end
     end
     results
   end
@@ -165,12 +201,14 @@ class WhatLanguage
 
   # Most likely language as a name symbol, or nil when no language is detected.
   def language(text)
-    detect(text)&.language
+    score_hash(text).max_by { |_name, score| score }&.first
   end
 
   # Most likely language as an ISO 639-1 symbol (639-3 fallback), or nil.
+  # Includes selected CLDR macrolanguage aliases and historical code overrides;
+  # see whatlanguage/languages.rb. Codes need not uniquely identify a language.
   def language_iso(text)
-    detect(text)&.iso
+    ISO_CODES[language(text)]
   end
 
   private
@@ -191,6 +229,20 @@ class WhatLanguage
     @selection.include?(:all) || languages.include?(name)
   end
 
+  def candidates_for(script)
+    candidates = self.class.profiles[script]
+    return candidates unless candidates && !@selection.include?(:all)
+
+    # languages returns a mutable array. Invalidate the cached selection when
+    # callers change it, preserving the existing selection behavior.
+    wanted = languages
+    if @selected_languages != wanted
+      @selected_languages = wanted.dup
+      @selected_profiles = {}
+    end
+    @selected_profiles[script] ||= candidates.select { |code, _trigrams| allowed?(CODE_INFO[code].first) }
+  end
+
   def validate_selection!
     requested = @selection.reject { |name| name == :all }
     unknown = requested.reject { |name| self.class.languages.include?(NAME_ALIASES.fetch(name, name)) }
@@ -199,24 +251,30 @@ class WhatLanguage
     raise ArgumentError, "Unknown language selection: #{unknown.map(&:inspect).join(', ')}"
   end
 
-  def significant_char_count(text)
-    text.each_char.count { |ch| /\p{L}/.match?(ch) }
-  end
+  # Count all letters, including those outside the recognized scripts.
+  def script_and_letter_count(text)
+    if text.ascii_only?
+      letter_count = text.count('A-Za-z')
+      return [letter_count.zero? ? nil : 'Latin', letter_count]
+    end
 
-  # Dominant Unicode script of the text, or nil if it has no script characters.
-  def detect_script(text)
     counts = Hash.new(0)
+    letter_count = 0
     text.each_char do |ch|
       next unless /\p{L}/.match?(ch)
+      letter_count += 1
 
-      SCRIPT_PATTERNS.each do |name, pattern|
+      i = 0
+      while i < SCRIPT_PATTERNS.size
+        name, pattern = SCRIPT_PATTERNS[i]
         if pattern.match?(ch)
           counts[name] += 1
           break
         end
+        i += 1
       end
     end
-    return nil if counts.empty?
+    return [nil, letter_count] if counts.empty?
 
     # Japanese uses Han, Hiragana, and Katakana together. Combine their votes
     # when kana is present, but still let a dominant unrelated script win.
@@ -227,33 +285,100 @@ class WhatLanguage
       counts['Hiragana'] = japanese
     end
 
-    counts.max_by { |_name, n| n }.first
+    [counts.max_by { |_name, n| n }.first, letter_count]
   end
 
   # Text trigrams ranked by descending frequency, mapped to their rank index.
   # Non-letter/mark characters become spaces. Preserve combining marks used
   # by the profiles; bound the stream by spaces and ignore repeated spaces.
   def trigram_positions(text)
-    chars = text.downcase.each_char.map { |c| /[\p{L}\p{M}]/.match?(c) ? c : ' ' }
-    return {} if chars.empty?
+    text = text.downcase
+    occurrences = text.ascii_only? ? ascii_trigram_counts(text) : unicode_trigram_counts(text)
+    ranked = occurrences.to_a.sort! do |a, b|
+      order = b[1] <=> a[1]
+      order.zero? ? b[0] <=> a[0] : order
+    end.first(TEXT_TRIGRAMS_SIZE)
+    positions = {}
+    ranked.each_with_index { |(trigram, _count), i| positions[trigram] = i }
+    positions
+  end
 
+  def unicode_trigram_counts(text)
     occurrences = Hash.new(0)
-    c1 = ' '
-    c2 = chars[0]
-    (chars[1..] + [' ']).each do |c3|
-      occurrences[c1 + c2 + c3] += 1 unless c2 == ' ' && (c1 == ' ' || c3 == ' ')
+    trigram = +''
+    c1 = nil
+    c2 = ' '
+    text.each_char do |character|
+      c3 = /[\p{L}\p{M}]/.match?(character) ? character : ' '
+      if c1 && !(c2 == ' ' && (c1 == ' ' || c3 == ' '))
+        # Hash copies new String keys, so this buffer can be reused for every
+        # window without changing already-counted trigrams.
+        trigram.replace(c1) << c2 << c3
+        occurrences[trigram] += 1
+      end
       c1 = c2
       c2 = c3
     end
+    if c1 && c2 != ' '
+      trigram.replace(c1) << c2 << ' '
+      occurrences[trigram] += 1
+    end
 
-    ranked = occurrences.to_a.sort { |a, b| [b[1], b[0]] <=> [a[1], a[0]] }.first(TEXT_TRIGRAMS_SIZE)
-    positions = {}
-    ranked.each_with_index { |(trigram, _count), i| positions[trigram] = i }
-    positions
+    occurrences
+  end
+
+  # Lowercased ASCII has only a-z letters and no combining marks. Use byte
+  # values and a reusable three-byte key to avoid allocating character strings.
+  def ascii_trigram_counts(text)
+    occurrences = Hash.new(0)
+    trigram = +"\0\0\0"
+    c1 = nil
+    c2 = 32
+    text.each_byte do |byte|
+      c3 = byte.between?(97, 122) ? byte : 32
+      if c1 && !(c2 == 32 && (c1 == 32 || c3 == 32))
+        trigram.setbyte(0, c1)
+        trigram.setbyte(1, c2)
+        trigram.setbyte(2, c3)
+        occurrences[trigram] += 1
+      end
+      c1 = c2
+      c2 = c3
+    end
+    if c1 && c2 != 32
+      trigram.setbyte(0, c1)
+      trigram.setbyte(1, c2)
+      trigram.setbyte(2, 32)
+      occurrences[trigram] += 1
+    end
+    occurrences
   end
 
   # Out-of-place distance between a language's ordered trigram profile and the
   # text's ranked trigrams. Lower means a closer match.
+  def indexed_distances(script, positions)
+    index = self.class.send(:profile_index, script)
+    return unless index
+
+    missing, postings = index
+    totals = missing.dup
+    positions.each do |trigram, position|
+      matches = postings[trigram]
+      next unless matches
+
+      i = 0
+      while i < matches.size
+        # Replace the missing-trigram penalty with the exact rank distance.
+        # This correction can be positive when the text rank is above 300.
+        totals[matches[i]] += (position - matches[i + 1]).abs - MAX_TRIGRAM_DISTANCE
+        i += 2
+      end
+    end
+
+    adjustment = [MAX_TRIGRAM_DISTANCE - positions.size, 0].max * MAX_TRIGRAM_DISTANCE
+    totals.map! { |total| (total - adjustment).clamp(0, MAX_TOTAL_DISTANCE) }
+  end
+
   def distance(profile, positions)
     total = 0
     profile.each_with_index do |trigram, i|

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: whatlanguage
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 3.0.0
 platform: ruby
 authors:
 - Peter Cooper