whatlanguage 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0939c1109ab44fd32e9e611c20696ee315a980877c0f04d5990fe42f51c17201'
-  data.tar.gz: e5002ebde89b8c6acfa313355d2aa588b31cbb7db27870690bbeae761ecaeb2c
+  metadata.gz: 4707fce379ca4e085886aef33b8bd45f3480c79bd93c1a7d040c068aff16a94c
+  data.tar.gz: e4b0156f7d3ee2dda4efb77cd2cd8c6abd6f96eac8c27d821166f927fe7421e6
 SHA512:
-  metadata.gz: 2a06fa18163cee6de9278291856c1b04c633880be1a2816476ce4b47b03c374242c9eb0ea012e1fb04322de753d45492a9ca96856ec70582ea1287fb19034c2e
-  data.tar.gz: e57243d9fccc9b5c43e5da591db26085d2eedd7776a4bc3e6dfd9b8ddcfcc59112de6309f63a41f0b2f38bc171bce2d80e638b2c3a8645230657aeb643f852ef
+  metadata.gz: fa04498461b8e07de91debf3ddf5ebece5d701a2006fe45a08a0baa7f1d8e7947a1c90fa726b175cbe08a3201ac510041f3f952f4b89abfd61ba0d42de0e5c77
+  data.tar.gz: 7f2ba53f9248b9f8559c06758205874b61a624c485a1e6bad02f5d192a5a93e09ae38018c73a5ac46bc93dc9e742a648757b6e00fc75ed12698588530f0124a9

data/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 2.1.0 / 2026-09-05
+
+- Added a fixed short-text preference for English, Chinese, Hindi, Spanish, and
+  French: a 600-point trigram bonus through 10 letters, fading to zero at 50.
+  Script routing, candidate restrictions, and minimum-length rules still apply.
+- Replaced hard-coded ranges with Unicode script properties, recognizing
+  uppercase Georgian, extended Greek, and supplementary Han letters.
+- Combined Han and kana when detecting Japanese instead of misclassifying
+  Han-heavy Japanese text as Chinese.
+- Excluded digits, punctuation, and emoji from script votes and minimum-length
+  counting, and treated them as trigram separators while preserving combining marks.
+- Added regression coverage for short texts and Unicode handling. Short-text
+  preferences are a heuristic; ambiguous fragments can still be misclassified.
+
+## 2.0.1 / 2026-07-11
+
+- Added a `whatlanguage` command-line executable that reads from files given as
+  arguments (or stdin) and prints the ISO 639 code of the detected language, or
+  `und` if undetermined. (#50, suggested by Keith Bennett)
+
 ## 2.0.0 / 2026-06-10
 
 - Rewrote the detection engine from scratch. Bloom-filter dictionary lookups are

data/README.md

@@ -13,17 +13,25 @@ WhatLanguage.language("Que linguagem é essa? É uma pergunta sobre a língua po
 - No runtime dependencies.
 - Supports 20+ writing systems.
 - Ships a compact ~220 KB trigram model.
-- Requires Ruby 3.0+
-- Best on sentence-length text or longer.
+- Requires Ruby 3.0+ (JRuby and TruffleRuby also good)
+- **Best on sentence-length text or longer.** Short texts can have ambiguous results.
+
+> [!IMPORTANT]
+> v2.0 has many breaking changes as the entire library has been reimplemented, though the core `WhatLanguage.language` API remains similar. Versions 1.0.6 and earlier (so the 2007-2025 run of the library) used a Bloom-filter technique and had 5MB of binary files to handle ~20 languages. Version 2.0 is more accurate, faster, and supports more languages from a single 220KB JSON file :-)
 
 ## How it works
 
-Detection is in two stages. First, the dominant Unicode script is detected; scripts used by a single language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.
+> [!NOTE]
+> Version 2.1 improves Unicode and mixed-script Japanese handling and adds a fixed
+> preference for five widely spoken languages on short texts. The public API and
+> default 10-letter minimum for statistical detection are unchanged.
+
+Detection is in two stages. First, the dominant Unicode script is detected from letters; scripts used by a single supported language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. When kana is present, Han, Hiragana, and Katakana count together as Japanese, while a dominant unrelated script can still win. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.
+
+Digits, punctuation, and emoji do not count toward the minimum text length and are treated as separators when extracting trigrams. Combining marks are preserved in trigrams but do not count as additional letters. Unicode script coverage follows the Unicode version supported by your Ruby runtime.
 
 The trigram profiles are vendored from [whatlang](https://github.com/greyblake/whatlang-rs), a port of [Franc](https://github.com/wooorm/franc), whose models are built from the public-domain UDHR corpus (see Credits). The model is a ~220 KB JSON file.
 
-> [!IMPORTANT]
-> v2.0 has many breaking changes as the entire library has been rewritten, though the core `WhatLanguage.language` API remains similar. Versions 1.0.6 and earlier (so the 2007-2025 run of the library) used a Bloom-filter technique and had 5MB of binary files to handle ~20 languages. Version 2.0 is more accurate, faster, and supports more languages from a single 220KB JSON file :-)
 
 ## Usage
 
@@ -39,7 +47,7 @@ result.score      # => 79018
 result.ranked     # => [[:german, 79018], [:dutch, 77631], ... ]
 ```
 
-Return ranked scores, or the raw score hash:
+Return ranked scores, or the score hash:
 
 ```ruby
 wl.ranked(text)       # => [[:german, 79018], [:dutch, 77631], ... ]
@@ -52,15 +60,36 @@ Restrict candidate languages:
 wl = WhatLanguage.new(only: [:english, :german, :french])
 ```
 
-Short Latin-script fragments are ignored by default because there is not enough signal to rank shared-script languages reliably. The threshold applies to the statistical trigram stage; scripts that identify a single supported language, such as Greek, Korean, or Thai, can still resolve from shorter text. The threshold can be adjusted:
+For short texts, English, Chinese (Mandarin), Hindi, Spanish, and French receive
+a built-in preference. These five were selected using the
+[2025 total-speaker ranking](https://www.visualcapitalist.com/ranked-the-worlds-most-spoken-languages-in-2025/)
+(including second-language speakers). Eligible trigram candidates receive the
+same bonus: 600 ranking points through 10 letters, decreasing linearly to zero
+at 50 letters. Longer texts keep their original scores. The preference is not
+configurable; it respects `only:` and script detection. Chinese already resolves
+from its script, so it does not need a statistical bonus. Scores exposed by the
+API include the bonus and remain ranking values, not probabilities.
+
+Short shared-script fragments are ignored by default because there is not enough signal to rank their languages reliably. The threshold applies to the statistical trigram stage; scripts that identify a single supported language, such as Greek, Korean, or Thai, can still resolve from shorter text. The threshold can be adjusted:
 
 ```ruby
 wl = WhatLanguage.new(min_chars: 0)
 ```
 
+## Command line
+
+Installing the gem also installs a `whatlanguage` executable that reads from files given as arguments (or stdin) and prints the ISO 639 code of the detected language, or `und` if undetermined:
+
+```bash
+$ whatlanguage README.md
+en
+$ echo "Wie geht es dir heute?" | whatlanguage
+de
+```
+
 ## Known limitations
 
-- Short fragments are unreliable. For languages resolved by statistical comparison, fewer than 10 significant characters returns `nil` by default.
+- Short fragments remain unreliable. For languages resolved by statistical comparison, fewer than 10 letters returns `nil` by default. The built-in preference helps some common phrases, but can also favor a preferred language over a less widely spoken one when evidence is weak. It does not lower the minimum length.
 - Scores are relative ranking values, not probabilities. Use `#ranked` or `#detect.ranked` when close runners-up matter.
 - Closely related written languages can be hard to separate, especially Norwegian Bokmål/Danish, Hebrew/Yiddish, and similar language pairs.
 - Kanji-only Japanese text can classify as Chinese because Han characters alone do not identify the language.
@@ -68,6 +97,6 @@ wl = WhatLanguage.new(min_chars: 0)
 
 ## Credits
 
-Contributions from Konrad Reiche, Salimane Adjao Moustapha, Andrew Cone, Lasse Skindstad Ebert, Henrik Nyh, Daniel Sandbecker, Michael Hartl, Pedro Lambert, Tobias Preuss, Pepijn Looije, and others appreciated.
+Contributions from Konrad Reiche, Salimane Adjao Moustapha, Andrew Cone, Lasse Skindstad Ebert, Henrik Nyh, Daniel Sandbecker, Michael Hartl, Pedro Lambert, Tobias Preuss, Pepijn Looije, Keith Bennett, and others appreciated.
 
 The trigram language profiles in `lib/whatlanguage/trigrams.json` are taken from [whatlang](https://github.com/greyblake/whatlang-rs) (MIT, © Sergey Potapov), itself a derivative of [Franc](https://github.com/wooorm/franc) (MIT, © Titus Wormer). Those profiles are derived from the public-domain Universal Declaration of Human Rights translations.

data/exe/whatlanguage

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'whatlanguage'
+
+# Reads from files given as arguments, or stdin otherwise. Prints the ISO 639
+# code of the detected language, or "und" (undetermined) if detection fails.
+puts WhatLanguage.language_iso(ARGF.read) || 'und'

data/lib/whatlanguage/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class WhatLanguage
-  VERSION = '2.0.0'
+  VERSION = '2.1.0'
 end

data/lib/whatlanguage.rb

@@ -9,12 +9,15 @@ class WhatLanguage
   MAX_TOTAL_DISTANCE   = MAX_TRIGRAM_DISTANCE * MAX_TRIGRAM_DISTANCE # 90_000
   TEXT_TRIGRAMS_SIZE   = 600
   DEFAULT_MIN_CHARS    = 10
+  SHORT_TEXT_PREFERENCES = %i[english chinese hindi spanish french].freeze
+  SHORT_TEXT_BONUS = 600
+  SHORT_TEXT_FADE_CHARS = 50
 
   Result = Struct.new(:language, :iso, :score, :ranked, keyword_init: true) do
     alias scores ranked
   end
 
-  # Scripts that resolve to a single language by their Unicode block alone.
+  # Scripts that resolve to a single supported language from their letters.
   # (Hiragana and Katakana both indicate Japanese.) Scripts NOT listed here but
   # present in the trigram dataset are disambiguated statistically instead.
   DETERMINISTIC = {
@@ -27,43 +30,13 @@ class WhatLanguage
     'Katakana'  => 'jpn'
   }.freeze
 
-  # Unicode ranges per script, in detection priority order (mirrors whatlang's
-  # scripts/detect.rs). The first script whose range contains a character claims
-  # that character; the script with the most characters wins.
-  SCRIPT_RANGES = [
-    ['Latin',      [[0x61,0x7A],[0x41,0x5A],[0x80,0xFF],[0x100,0x17F],[0x180,0x24F],
-                    [0x250,0x2AF],[0x1D00,0x1D7F],[0x1D80,0x1DBF],[0x1E00,0x1EFF],
-                    [0x2100,0x214F],[0x2C60,0x2C7F],[0xA720,0xA7FF],[0xAB30,0xAB6F]]],
-    ['Cyrillic',   [[0x400,0x484],[0x487,0x52F],[0x2DE0,0x2DFF],[0xA640,0xA69D],
-                    [0x1D2B,0x1D2B],[0x1D78,0x1D78],[0xA69F,0xA69F]]],
-    ['Arabic',     [[0x600,0x6FF],[0x750,0x7FF],[0x8A0,0x8FF],[0xFB50,0xFDFF],
-                    [0xFE70,0xFEFF],[0x10E60,0x10E7F],[0x1EE00,0x1EEFF]]],
-    ['Mandarin',   [[0x2E80,0x2E99],[0x2E9B,0x2EF3],[0x2F00,0x2FD5],[0x3005,0x3005],
-                    [0x3007,0x3007],[0x3021,0x3029],[0x3038,0x303B],[0x3400,0x4DB5],
-                    [0x4E00,0x9FCC],[0xF900,0xFA6D],[0xFA70,0xFAD9]]],
-    ['Devanagari', [[0x900,0x97F],[0xA8E0,0xA8FF],[0x1CD0,0x1CFF]]],
-    ['Hebrew',     [[0x590,0x5FF]]],
-    ['Ethiopic',   [[0x1200,0x139F],[0x2D80,0x2DDF],[0xAB00,0xAB2F]]],
-    ['Georgian',   [[0x10A0,0x10FF]]],
-    ['Bengali',    [[0x980,0x9FF]]],
-    ['Hangul',     [[0xAC00,0xD7AF],[0x1100,0x11FF],[0x3130,0x318F],[0x3200,0x32FF],
-                    [0xA960,0xA97F],[0xD7B0,0xD7FF]]],
-    ['Hiragana',   [[0x3040,0x309F]]],
-    ['Katakana',   [[0x30A0,0x30FF]]],
-    ['Greek',      [[0x370,0x3FF]]],
-    ['Kannada',    [[0xC80,0xCFF]]],
-    ['Tamil',      [[0xB80,0xBFF]]],
-    ['Thai',       [[0xE00,0xE7F]]],
-    ['Gujarati',   [[0xA80,0xAFF]]],
-    ['Gurmukhi',   [[0xA00,0xA7F]]],
-    ['Telugu',     [[0xC00,0xC7F]]],
-    ['Malayalam',  [[0xD00,0xD7F]]],
-    ['Oriya',      [[0xB00,0xB7F]]],
-    ['Myanmar',    [[0x1000,0x109F]]],
-    ['Sinhala',    [[0xD80,0xDFF]]],
-    ['Khmer',      [[0x1780,0x17FF],[0x19E0,0x19FF]]],
-    ['Armenian',   [[0x530,0x58F],[0xFB13,0xFB17]]]
-  ].freeze
+  # Unicode script properties include letters outside the original BMP blocks.
+  # Keep the model's historical "Mandarin" key for the Han script.
+  SCRIPT_PATTERNS = %w[
+    Latin Cyrillic Arabic Han Devanagari Hebrew Ethiopic Georgian Bengali
+    Hangul Hiragana Katakana Greek Kannada Tamil Thai Gujarati Gurmukhi
+    Telugu Malayalam Oriya Myanmar Sinhala Khmer Armenian
+  ].map { |script| [script == 'Han' ? 'Mandarin' : script, Regexp.new("\\p{#{script}}")] }.freeze
 
   # ISO 639-1 (with 639-3 fallback) lookup by language-name symbol, plus the
   # historical nil => nil entry. Internal; kept for backward compatibility.
@@ -76,8 +49,9 @@ class WhatLanguage
   end.freeze
 
   private_constant :MAX_TRIGRAM_DISTANCE, :MAX_TOTAL_DISTANCE, :TEXT_TRIGRAMS_SIZE,
-                   :DEFAULT_MIN_CHARS, :DETERMINISTIC, :SCRIPT_RANGES, :ISO_CODES,
-                   :NAME_TO_CODE
+                   :DEFAULT_MIN_CHARS, :DETERMINISTIC, :SCRIPT_PATTERNS, :ISO_CODES,
+                   :NAME_TO_CODE, :SHORT_TEXT_PREFERENCES, :SHORT_TEXT_BONUS,
+                   :SHORT_TEXT_FADE_CHARS
 
   class << self
     def detect(text)
@@ -156,14 +130,17 @@ class WhatLanguage
 
     candidates = self.class.profiles[script]
     return results unless candidates
-    return results if significant_char_count(text) < @min_chars
+    char_count = significant_char_count(text)
+    return results if char_count < @min_chars
 
+    bonus = short_text_bonus(char_count)
     positions = trigram_positions(text)
     candidates.each do |code, trigrams|
       name = CODE_INFO[code].first
       next unless allowed?(name)
 
       results[name] = MAX_TOTAL_DISTANCE - distance(trigrams, positions)
+      results[name] += bonus if SHORT_TEXT_PREFERENCES.include?(name)
     end
     results
   end
@@ -198,6 +175,14 @@ class WhatLanguage
 
   private
 
+  # A small fixed prior for widely spoken languages: full strength through ten
+  # letters, fading linearly to zero at fifty. These are ranking points, not
+  # probabilities. Script routing and the caller's candidate selection still win.
+  def short_text_bonus(char_count)
+    remaining = (SHORT_TEXT_FADE_CHARS - char_count).clamp(0, SHORT_TEXT_FADE_CHARS - DEFAULT_MIN_CHARS)
+    SHORT_TEXT_BONUS * remaining / (SHORT_TEXT_FADE_CHARS - DEFAULT_MIN_CHARS)
+  end
+
   def normalize_text(text)
     text.to_s.unicode_normalize(:nfkc)
   end
@@ -215,18 +200,17 @@ class WhatLanguage
   end
 
   def significant_char_count(text)
-    text.each_char.count { |ch| !stop_char?(ch.ord) }
+    text.each_char.count { |ch| /\p{L}/.match?(ch) }
   end
 
   # Dominant Unicode script of the text, or nil if it has no script characters.
   def detect_script(text)
     counts = Hash.new(0)
     text.each_char do |ch|
-      cp = ch.ord
-      next if stop_char?(cp)
+      next unless /\p{L}/.match?(ch)
 
-      SCRIPT_RANGES.each do |name, ranges|
-        if ranges.any? { |lo, hi| cp >= lo && cp <= hi }
+      SCRIPT_PATTERNS.each do |name, pattern|
+        if pattern.match?(ch)
           counts[name] += 1
           break
         end
@@ -234,14 +218,23 @@ class WhatLanguage
     end
     return nil if counts.empty?
 
+    # Japanese uses Han, Hiragana, and Katakana together. Combine their votes
+    # when kana is present, but still let a dominant unrelated script win.
+    if counts['Hiragana'] + counts['Katakana'] > 0
+      japanese = counts['Mandarin'] + counts['Hiragana'] + counts['Katakana']
+      counts.delete('Mandarin')
+      counts.delete('Katakana')
+      counts['Hiragana'] = japanese
+    end
+
     counts.max_by { |_name, n| n }.first
   end
 
   # Text trigrams ranked by descending frequency, mapped to their rank index.
-  # Mirrors whatlang's trigram extraction: punctuation/digits become spaces,
-  # the stream is bounded by spaces, and runs of spaces are collapsed.
+  # Non-letter/mark characters become spaces. Preserve combining marks used
+  # by the profiles; bound the stream by spaces and ignore repeated spaces.
   def trigram_positions(text)
-    chars = text.downcase.each_char.map { |c| stop_char?(c.ord) ? ' ' : c }
+    chars = text.downcase.each_char.map { |c| /[\p{L}\p{M}]/.match?(c) ? c : ' ' }
     return {} if chars.empty?
 
     occurrences = Hash.new(0)
@@ -272,9 +265,4 @@ class WhatLanguage
     total -= (MAX_TRIGRAM_DISTANCE - count) * MAX_TRIGRAM_DISTANCE if MAX_TRIGRAM_DISTANCE > count
     total.clamp(0, MAX_TOTAL_DISTANCE)
   end
-
-  # Space, ASCII punctuation, or digit: no value for script/language detection.
-  def stop_char?(codepoint)
-    codepoint <= 0x40 || (codepoint >= 0x5B && codepoint <= 0x60) || (codepoint >= 0x7B && codepoint <= 0x7E)
-  end
 end

data/whatlanguage.gemspec

@@ -16,6 +16,7 @@ Gem::Specification.new do |gem|
   gem.required_ruby_version = '>= 3.0'
 
   gem.files = Dir['lib/**/*'] + [
+    'exe/whatlanguage',
     'README.md',
     'CHANGELOG.md',
     'LICENSE.txt',
@@ -23,6 +24,8 @@ Gem::Specification.new do |gem|
     'Rakefile',
     'whatlanguage.gemspec'
   ]
+  gem.bindir        = 'exe'
+  gem.executables   = ['whatlanguage']
   gem.require_paths = ['lib']
 
   gem.add_development_dependency 'minitest', '~> 5.0'

metadata

@@ -1,11 +1,11 @@
 --- !ruby/object:Gem::Specification
 name: whatlanguage
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Peter Cooper
-bindir: bin
+bindir: exe
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
@@ -40,7 +40,8 @@ dependencies:
 description: WhatLanguage rapidly detects the language of a sample of text
 email:
 - git@peterc.org
-executables: []
+executables:
+- whatlanguage
 extensions: []
 extra_rdoc_files: []
 files:
@@ -49,6 +50,7 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
+- exe/whatlanguage
 - lib/whatlanguage.rb
 - lib/whatlanguage/languages.rb
 - lib/whatlanguage/trigrams.json