psychowl 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d819bc7757a98dbd6a87c9716fc74433c4c28709b5a4262a0211e407a7c8c68b
+  data.tar.gz: 04fd01219832809efce54feed609c28977ba4c2991f100c7b7983daaceaf6763
+SHA512:
+  metadata.gz: 2654f694df5a85b1ad9f5731bcbde84e505bbdfc24ac6bb9c4b253e437b76aa06a8ff9f651d0bf1e3c86206241454de34e8bd2efc7299fc49a00d27402dd5947
+  data.tar.gz: 400a6c781067e429ab9c619f184b4750b6882d38703258ec53fc10bb86b440694fc7e4058591e76b34faf6c09538cd7251ba697013678c3fc9e922014c81396d

data/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+## 0.1.0 (2026-10-08)
+
+First release, for Ruby 4.0+ (CRuby, JRuby 10.1+, TruffleRuby 40+): 18 languages
+and 25 scripts. Wave 1: Arabic, English, French, German, Italian, Japanese,
+Mandarin, Portuguese, Russian, Spanish. Wave 2: Dutch, Hindi, Indonesian,
+Korean, Polish, Turkish, Ukrainian, Vietnamese.
+
+- Pure Ruby engine plus the `psychowl` Rust crate, loaded as an optional
+  native extension; a parity suite keeps both bit-identical.
+- Precompiled gems for Linux (x86_64/aarch64, glibc and musl), macOS
+  (arm64/x86_64) and FreeBSD 15.1 (amd64); a pure Ruby gem for everything
+  else.
+- `detect`, `detect_lang`, `detect_script`, `candidates`, `segments`,
+  `detect_many`, `scripts`; allowlists and denylists that also apply to
+  single-language scripts and Han.
+- `Lang` with ISO 639-3/639-1 codes, `#locale` and `#pg_regconfig`.
+- ActiveModel `language:` validator, `psychowl` CLI.
+- Language tables as plain text, a whatlang importer and a trigram profile
+  trainer.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Abdelkader Boudih
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,212 @@
+# psychowl
+
+It knows what language you speak. It knows you skipped your lesson.
+
+Language and script detection for Ruby and Rust. psychowl follows the
+[Matryoshka](https://github.com/seuros/matryoshka) FFI Hybrid pattern:
+
+- a **pure Ruby engine** that runs everywhere (CRuby, JRuby 10.1+, TruffleRuby 40+),
+- the same algorithm in **Rust** (the `psychowl` crate), loaded as a native
+  extension when available, 20 to 500 times faster,
+- one set of **plain-text language tables** both engines read, and a parity
+  test suite holding them to bit-identical results.
+
+```ruby
+require "psychowl"
+
+info = Psychowl.detect("El presidente dijo que la economía va muy bien. Los precios todavía no se han enterado.")
+info.lang.code     # => "spa"
+info.lang.locale   # => :es
+info.script.name   # => "Latin"
+info.confidence    # => 1.0  (more than the economy gets)
+info.reliable?     # => true
+```
+
+## The owl has heard things
+
+Every line below is detected correctly, at full confidence, by both engines.
+Heads of state make the best test data: they talk a lot and say nothing.
+
+| | Text | Roughly |
+|---|---|---|
+| eng | Nobody detects languages better than me, believe me. Tremendous trigrams, the best trigrams, everybody says so. | The best trigrams. Everybody says so. |
+| spa | El presidente dijo que la economía va muy bien. Los precios todavía no se han enterado. | The president said the economy is doing great. Prices haven't heard yet. |
+| fra | Traversez la rue, je vous trouverai un travail, a dit le président avant de repartir en jet privé. | "Cross the street, I'll find you a job," said the president, boarding his private jet. |
+| deu | Die Deutsche Bahn ist heute pünktlich. Wir ermitteln. | Deutsche Bahn is on time today. We are investigating. |
+| ita | Il presidente ha promesso che il ponte sarà finito entro le prossime elezioni. Quali elezioni non lo ha detto. | The bridge will be done by the next election. He didn't say which one. |
+| por | O presidente prometeu que desta vez a obra termina no prazo. A obra começou em mil novecentos e oitenta e dois. | This time the construction finishes on schedule. It started in 1982. |
+| rus | Это не война, это специальная военная операция. | It's not a war, it's a special military operation. |
+| ara | فاز الرئيس في الانتخابات بنسبة تسعة وتسعين في المئة. مرة أخرى. | The president won the election with 99%. Again. |
+| cmn | 小熊维尼今天又被禁止了。 | Winnie the Pooh was banned again today. |
+| jpn | 首相は「前向きに検討します」と言いました。 | The prime minister said he will "consider it positively". |
+| kor | 대통령은 부동산 가격이 안정되었다고 말했다. 집값은 그 말을 듣지 못했다. | The president said housing prices have stabilized. Housing prices didn't hear him. |
+| hin | नेताजी ने वादा किया कि इस बार बिजली नहीं जाएगी। फिर बिजली चली गई। | The politician promised the power won't go out this time. Then the power went out. |
+| tur | Bakan, ekonominin uçuşa geçtiğini söyledi. Kimse paraşüt dağıtmadı. | The minister said the economy has taken off. Nobody handed out parachutes. |
+| nld | De regering is gevallen. Dat is al de derde keer deze maand, en het is pas dinsdag. | The government has fallen. Third time this month, and it's only Tuesday. |
+| pol | Prezydent obiecał, że pociąg przyjedzie punktualnie. Pociąg nie został o tym poinformowany. | The president promised the train will be on time. Nobody told the train. |
+| ukr | Сова нагадує: урок не пройдено. Вона вже їде до тебе. | The owl reminds you: lesson not done. It is already on its way to you. |
+| ind | Pejabat itu bilang jalan tol akan selesai tahun depan. Dia mengatakan hal yang sama sepuluh tahun lalu. | The official says the toll road is done next year. He said the same ten years ago. |
+| vie | Ông quan chức hứa sẽ sửa đường ngay tuần sau. Tuần sau đó đã kéo dài ba năm. | The official promised to fix the road next week. That week has lasted three years. |
+
+## Installation
+
+Requires Ruby 4.0 or newer.
+
+```ruby
+gem "psychowl"
+```
+
+Precompiled native gems ship for Linux (x86_64 and aarch64; glibc 2.34+ such
+as Debian 13, Ubuntu 26.04, RHEL 10, Fedora, Arch or Omarchy, plus musl for
+Alpine), macOS 13+ (arm64, x86_64) and FreeBSD 15.1 (amd64). No Rust, no
+compiler, nothing to build.
+Windows, JRuby and TruffleRuby get the pure Ruby gem: same results, just
+slower. Where a shipped native extension can't load (glibc older than 2.34,
+Alpine without `libgcc`), psychowl warns and uses its Ruby engine.
+
+```ruby
+Psychowl.backend # => :native or :ruby
+```
+
+Set `DISABLE_PSYCHOWL_NATIVE=1` (or `DISABLE_MATRYOSHKA_NATIVE=1`) to force
+the Ruby engine.
+
+## Usage
+
+```ruby
+Psychowl.detect_lang("Die Deutsche Bahn ist heute pünktlich. Wir ermitteln.")
+# => #<Psychowl::Lang deu (German)>
+
+Psychowl.detect_script("Это не война, это специальная военная операция.")
+# => #<Psychowl::Script Cyrillic>  (the owl calls it what it is)
+
+promise = "O presidente prometeu que desta vez a obra termina no prazo."
+
+# Restrict the candidates (ISO 639-3 or 639-1 codes, or Lang objects)
+Psychowl.detect_lang(promise, allowlist: %i[en es]) # => #<Psychowl::Lang spa (Spanish)>
+Psychowl.detect_lang(promise, denylist: [:por])     # => #<Psychowl::Lang spa (Spanish)>
+
+# Reuse a configured detector (frozen, thread-safe, unlike the coalition)
+detector = Psychowl::Detector.new(allowlist: %i[en fr de])
+detector.detect_lang("Traversez la rue, je vous trouverai un travail, a dit le président.")
+# => #<Psychowl::Lang fra (French)>
+
+# Every candidate with its score, best first
+Psychowl.candidates(promise, limit: 3)
+# => [[#<Psychowl::Lang por (Portuguese)>, 0.71], [#<Psychowl::Lang spa (Spanish)>, 0.67], [#<Psychowl::Lang ita (Italian)>, 0.66]]
+
+# Mixed-language text, per sentence, with character ranges (summit transcripts)
+summit = "Nobody detects languages better than me, believe me. Traversez la rue, je vous trouverai un travail."
+Psychowl.segments(summit).map { [_1.lang.code, _1.range] }
+# => [["eng", 0...53], ["fra", 53...100]]
+
+# Many texts at once (spread over all cores with the native engine)
+Psychowl.detect_many(press_releases.map(&:body))
+
+# Writing systems in a text
+Psychowl.scripts("Error 404: смысл жизни not found")
+# => {#<Psychowl::Script Latin> => 0.565, #<Psychowl::Script Cyrillic> => 0.435}
+
+# Pattern matching
+case Psychowl.detect(speech)
+in {lang: {code: "eng"}, reliable: true} then :tremendous
+in {reliable: false} then :fake_news
+in nil then :no_comment
+end
+```
+
+Invalid input raises: `TypeError` for non-strings, `EncodingError` for bytes
+that are not valid UTF-8 (other encodings are converted, binary strings are
+accepted when they hold valid UTF-8), `ArgumentError` for unsupported
+language codes.
+
+### Short texts
+
+Detection works on character trigrams, so it needs some text. A couple of
+words is a guess: "Hello" comes out as Italian with a confidence of 0.08,
+and the owl stands by it. Check `reliable?` or `confidence` before trusting a
+result, or restrict the candidates with an allowlist. Same rule as campaign
+promises.
+
+## Rails
+
+The `language:` validator loads automatically in Rails apps (elsewhere:
+`require "psychowl/active_model"`).
+
+```ruby
+class CampaignPromise < ApplicationRecord
+  validates :text, language: { in: %i[en es] }
+  validates :fine_print, language: { reliable: true }, allow_blank: true
+  validates :excuse, language: { minimum_confidence: 0.5 } # "we will consider it positively" is not an excuse
+end
+```
+
+Error keys are `:language_undetected`, `:language_not_allowed` and
+`:language_uncertain`, each with a `%{language}` interpolation; English
+messages ship with the gem.
+
+PostgreSQL full-text search configuration for a language:
+
+```ruby
+config = Psychowl.detect(speech.transcript)&.lang&.pg_regconfig || "simple" # => "german"
+Speech.where("to_tsvector(?::regconfig, transcript) @@ plainto_tsquery(?::regconfig, ?)", config, config, "pünktlich")
+# => [] (as expected)
+```
+
+## CLI
+
+```console
+$ echo "Nobody detects languages better than me, believe me." | psychowl
+eng	English	Latin	0.89	unreliable
+$ psychowl --candidates 3 campaign_promises.txt
+$ psychowl --lines --json press_conference.txt   # one result per line, batch
+$ psychowl --segments --allow en,fr summit_transcript.txt
+$ psychowl --scripts leaked_memo.txt
+$ psychowl --languages
+```
+
+Exit status: 0 when a language was found, 1 when not, 2 on bad input.
+
+## Supported languages
+
+Languages are added in waves; adding one is a data change, not a code change
+(see [`data/README.md`](ext/psychowl_native/core/data/README.md)).
+
+| Wave | Languages |
+|---|---|
+| 1 (0.1) | Arabic, English, French, German, Italian, Japanese, Mandarin, Portuguese, Russian, Spanish |
+| 2 (0.1) | Dutch, Hindi, Indonesian, Korean, Polish, Turkish, Ukrainian, Vietnamese |
+| 3 | the rest of whatlang's 70 |
+| 4 | new languages trained from our own corpora |
+
+Script detection covers 25 writing systems already. A text in a script with
+no supported language yet (Greek, Thai...) has a script but no language.
+Text in an unsupported language written in a supported script gets the
+closest supported language, usually with a low confidence.
+
+## Rust
+
+The engine is a standalone crate, see
+[`ext/psychowl_native/core`](ext/psychowl_native/core/README.md).
+
+## Development
+
+```console
+bundle install
+rake test:all        # cargo test, then the suite on both engines (incl. parity)
+rake test:ruby       # pure Ruby engine only
+rake lint            # cargo fmt --check + clippy
+rake "psychowl:train[eng]"   # rebuild a profile from corpus/eng.txt
+ruby bench/engines.rb        # Ruby vs Rust timings
+```
+
+## Credits
+
+The algorithm and the initial language profiles are ported from
+[whatlang](https://github.com/greyblake/whatlang-rs) by Sergey Potapov and
+contributors (MIT); see
+[`LICENSE-whatlang`](ext/psychowl_native/core/data/LICENSE-whatlang).
+
+## License
+
+MIT

data/exe/psychowl

@@ -0,0 +1,106 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'json'
+require 'optparse'
+require 'psychowl'
+
+options = { mode: :detect, json: false }
+
+parser = OptionParser.new do |opts|
+  opts.banner = <<~BANNER
+    Usage: psychowl [options] [FILE...]
+
+    Detects the language of each FILE, or of standard input.
+
+  BANNER
+  opts.on('-j', '--json', 'JSON output') { options[:json] = true }
+  opts.on('-l', '--lines', 'Detect every line separately (batch)') { options[:mode] = :lines }
+  opts.on('-c', '--candidates [N]', Integer, 'Rank candidate languages (top N)') do |limit|
+    options[:mode] = :candidates
+    options[:limit] = limit
+  end
+  opts.on('-s', '--segments', 'Split mixed-language text into segments') { options[:mode] = :segments }
+  opts.on('--scripts', 'Show the writing systems used') { options[:mode] = :scripts }
+  opts.on('-a', '--allow LANGS', Array, 'Only consider these languages, e.g. en,fr') do |langs|
+    options[:allowlist] = langs
+  end
+  opts.on('-d', '--deny LANGS', Array, 'Never consider these languages') { options[:denylist] = it }
+  opts.on('--languages', 'List supported languages') { options[:mode] = :languages }
+  opts.on('-v', '--version', 'Show version and engine') do
+    puts "psychowl #{Psychowl::VERSION} (#{Psychowl.backend} engine)"
+    exit
+  end
+  opts.on('-h', '--help', 'Show this help') do
+    puts opts
+    exit
+  end
+end
+
+def info_hash(info)
+  return nil unless info
+
+  { lang: info.lang.code, iso639_1: info.lang.iso639_1, name: info.lang.eng_name,
+    script: info.script.name, confidence: info.confidence, reliable: info.reliable? }
+end
+
+def info_line(info)
+  return 'unknown' unless info
+
+  [info.lang.code, info.lang.eng_name, info.script.name, format('%.2f', info.confidence),
+   info.reliable? ? 'reliable' : 'unreliable'].join("\t")
+end
+
+def emit(options, json, text)
+  puts options[:json] ? JSON.generate(json) : text
+end
+
+begin
+  parser.parse!
+  filter = options.slice(:allowlist, :denylist)
+  detector = Psychowl::Detector.new(**filter)
+
+  if options[:mode] == :languages
+    langs = Psychowl::Lang.all
+    emit(options, langs.map { |lang| { code: lang.code, iso639_1: lang.iso639_1, name: lang.eng_name } },
+         langs.map { |lang| [lang.code, lang.iso639_1, lang.eng_name, lang.name].join("\t") }.join("\n"))
+    exit
+  end
+
+  input = ARGF.read.force_encoding(Encoding::UTF_8)
+  detected = true
+
+  case options[:mode]
+  when :detect
+    info = detector.detect(input)
+    detected = !info.nil?
+    emit(options, info_hash(info), info_line(info))
+  when :lines
+    lines = input.lines(chomp: true)
+    infos = detector.detect_many(lines)
+    emit(options, infos.map { info_hash(it) }, infos.map { info_line(it) }.join("\n"))
+  when :candidates
+    ranked = detector.candidates(input, limit: options[:limit])
+    detected = !ranked.empty?
+    emit(options, ranked.map { |lang, score| { lang: lang.code, name: lang.eng_name, score: } },
+         ranked.map { |lang, score| [lang.code, lang.eng_name, format('%.4f', score)].join("\t") }.join("\n"))
+  when :segments
+    segments = detector.segments(input)
+    detected = !segments.empty?
+    emit(options,
+         segments.map do |segment|
+           info_hash(segment.info).merge(start: segment.range.begin, end: segment.range.end, text: segment.text)
+         end,
+         segments.map { |segment| [segment.range, info_line(segment.info), segment.text.strip].join("\t") }.join("\n"))
+  when :scripts
+    shares = detector.scripts(input)
+    detected = !shares.empty?
+    emit(options, shares.to_h { |script, share| [script.name, share] },
+         shares.map { |script, share| [script.name, format('%.1f%%', share * 100)].join("\t") }.join("\n"))
+  end
+
+  exit(detected ? 0 : 1)
+rescue OptionParser::ParseError, ArgumentError, EncodingError, SystemCallError => e
+  warn "psychowl: #{e.message}"
+  exit 2
+end

data/ext/psychowl_native/core/data/LICENSE-whatlang

@@ -0,0 +1,26 @@
+(The MIT License)
+
+Copyright (c) 2017 Sergey Potapov <blake131313@gmail.com>
+Copyright (c) 2014 Titus Wormer <tituswormer@gmail.com>
+Copyright (c) 2008 Kent S Johnson
+Copyright (c) 2006 Jacob R Rideout <kde@jacobrideout.net>
+Copyright (c) 2004 Maciej Ceglowski
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/ext/psychowl_native/core/data/README.md

@@ -0,0 +1,32 @@
+# psychowl language data
+
+Plain-text tables shared by both engines: the Rust crate compiles them in
+(`build.rs`), the pure Ruby engine reads them at load time. Both must give
+identical results, so these files are the single source of truth.
+
+Lines starting with `#` are comments. Fields are tab-separated.
+
+| File | Content |
+|---|---|
+| `languages.tsv` | `code`, `iso639_1`, `eng_name`, `name`. Sorted by code; line order is the language index. |
+| `scripts.tsv` | `name`, `languages` (comma-separated codes, `-` for none), `ranges` (hex code points, inclusive, disjoint across scripts). Line order is the script index and the tie-break priority. |
+| `alphabets.tsv` | `code`, `letters`: lowercase letters a language commonly uses. Needed for languages sharing a script. |
+| `trigrams/<code>.txt` | The language's 300 most frequent character trigrams, most frequent first, one per line. `_` stands for a word boundary (space). Needed for languages sharing a script. |
+
+## Adding a language
+
+1. Add its row to `languages.tsv` (keep it sorted).
+2. Add its code to its script(s) in `scripts.tsv`.
+3. If the script has other languages: add `alphabets.tsv` letters and a
+   `trigrams/<code>.txt` profile (`rake psychowl:train[code]` builds one from
+   `corpus/<code>.txt`).
+4. Add samples to `core/tests/samples/<code>.txt`; both test suites pick
+   them up.
+
+## Provenance
+
+The initial profiles, alphabets and script ranges were ported from
+[whatlang](https://github.com/greyblake/whatlang-rs) 0.18.0 with
+`tools/import_whatlang.rb`; see `LICENSE-whatlang` (MIT). Changes from the
+original: the Latin range no longer overlaps the Cyrillic letters U+1D2B and
+U+1D78, and Mandarin is named by its Unicode script name, Han.

data/ext/psychowl_native/core/data/alphabets.tsv

@@ -0,0 +1,16 @@
+# Letters commonly used by a language, lowercase. Only used for scripts
+# shared by several languages (Latin, Cyrillic).
+# code	letters
+deu	abcdefghijklmnopqrstuvwxyzßäöü
+eng	abcdefghijklmnopqrstuvwxyz
+fra	abcdefghijklmnopqrstuvwxyzàâçèéêëîïôùûüÿœ
+ind	abcdefghijklmnopqrstuvwxyz
+ita	abcdefghijklmnopqrstuvwxyzàèéìòù
+nld	abcdefghijklmnopqrstuvwxyzàèéëïij
+pol	abcdefghijklmnopqrstuvwxyzóąćęłńśźż
+por	abcdefghijklmnopqrstuvwxyzàáâãçéêíóôõú
+rus	абвгдежзийклмнопрстуфхцчшщъыьэюяё
+spa	abcdefghijklmnopqrstuvwxyz¡¿áéíñóúü
+tur	abcdefghijklmnopqrstuvwxyzçöüğış̇
+ukr	абвгдежзийклмнопрстуфхцчшщьюяєіїґ
+vie	abcdefghijklmnopqrstuvwxyzàáâãèéêìíòóôõùúýăđĩũơưạảấầẩẫậắằẳẵặẹẻẽếềểễệỉịọỏốồổỗộớờởỡợụủứừửữựỳỵỷỹ

data/ext/psychowl_native/core/data/languages.tsv

@@ -0,0 +1,20 @@
+# Supported languages, sorted by code. Line order defines language indices.
+# code (ISO 639-3)	ISO 639-1	English name	native name
+ara	ar	Arabic	العربية
+cmn	zh	Mandarin	普通话
+deu	de	German	Deutsch
+eng	en	English	English
+fra	fr	French	Français
+hin	hi	Hindi	हिन्दी
+ind	id	Indonesian	Bahasa Indonesia
+ita	it	Italian	Italiano
+jpn	ja	Japanese	日本語
+kor	ko	Korean	한국어
+nld	nl	Dutch	Nederlands
+pol	pl	Polish	Polski
+por	pt	Portuguese	Português
+rus	ru	Russian	Русский
+spa	es	Spanish	Español
+tur	tr	Turkish	Türkçe
+ukr	uk	Ukrainian	Українська
+vie	vi	Vietnamese	Tiếng Việt

data/ext/psychowl_native/core/data/scripts.tsv

@@ -0,0 +1,28 @@
+# Writing systems in detection priority order (ties go to the earlier one).
+# Code point ranges are disjoint. "-" means no supported language yet.
+# name	languages	ranges (hex, inclusive)
+Latin	deu,eng,fra,ind,ita,nld,pol,por,spa,tur,vie	0061-007A,0041-005A,0080-00FF,0100-017F,0180-024F,0250-02AF,1D00-1D2A,1D2C-1D77,1D79-1D7F,1D80-1DBF,1E00-1EFF,2100-214F,2C60-2C7F,A720-A7FF,AB30-AB6F
+Cyrillic	rus,ukr	0400-0484,0487-052F,2DE0-2DFF,A640-A69D,1D2B,1D78,A69F
+Arabic	ara	0600-06FF,0750-07FF,08A0-08FF,FB50-FDFF,FE70-FEFF,10E60-10E7F,1EE00-1EEFF
+Han	cmn,jpn	2E80-2E99,2E9B-2EF3,2F00-2FD5,3005,3007,3021-3029,3038-303B,3400-4DB5,4E00-9FCC,F900-FA6D,FA70-FAD9
+Devanagari	hin	0900-097F,A8E0-A8FF,1CD0-1CFF
+Hebrew	-	0590-05FF
+Ethiopic	-	1200-139F,2D80-2DDF,AB00-AB2F
+Georgian	-	10A0-10FF
+Bengali	-	0980-09FF
+Hangul	kor	AC00-D7AF,1100-11FF,3130-318F,3200-32FF,A960-A97F,D7B0-D7FF,FF00-FFEF
+Hiragana	jpn	3040-309F
+Katakana	jpn	30A0-30FF
+Greek	-	0370-03FF
+Kannada	-	0C80-0CFF
+Tamil	-	0B80-0BFF
+Thai	-	0E00-0E7F
+Gujarati	-	0A80-0AFF
+Gurmukhi	-	0A00-0A7F
+Telugu	-	0C00-0C7F
+Malayalam	-	0D00-0D7F
+Oriya	-	0B00-0B7F
+Myanmar	-	1000-109F
+Sinhala	-	0D80-0DFF
+Khmer	-	1780-17FF,19E0-19FF
+Armenian	-	0530-058F,FB13-FB17