lingua_rs 0.4.4 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 151633d9585ecca170847b63a380b257178573a6f8fcb395d8ff731acfc0df4d
-  data.tar.gz: 97ddfa0dcf4cf4943afcf65cb4240dd3f7e7d0b39936d36f38deb1a5224d3994
+  metadata.gz: 9ec0e0ba5844b9b3e47cd60babbc90ac03791bc83e3327e78b7f2d83ba4a71d8
+  data.tar.gz: 55af5915e63a1b34cbc5c73d2e2c63b6a1ce2030183112c54aad888b299bc889
 SHA512:
-  metadata.gz: 634d5762de12f35056462e61160e7b2dc730f9970ba9a4248c13daf8c93ab5d6c35d010576a2b1cf5255756e0319ec0409b3cd21fedf36aff970c7ac22e2efeb
-  data.tar.gz: f27c9d123beee999fc57d13ab41f2b3029855bb957e363621092dafb9e6ccad8246a37a097b0b5ee4a5025762df04048e70d8a292984f856558fd67b8ece8855
+  metadata.gz: b6bfc5d1605bf171e7be4dd899e7b82460f4c8b81ab541ff97a18fb59205a752e9cfe12de640c20b6a60a4814bc8e89e884b9d501a2ec3b5a4e4f0b4bc2e094d
+  data.tar.gz: ecf2866f980b9ddd7189d639ea1330a4f47f8f0a528dcb636358c7c6de30add2d8a854cee8252eac36540fc528b1836eedebbd03313eeb1de167cd18c241afbf

data/README.md

@@ -5,7 +5,7 @@
 [![Downloads](https://img.shields.io/gem/dt/lingua_rs?color=0969da)](https://rubygems.org/gems/lingua_rs)
 [![License: MIT](https://img.shields.io/badge/License-MIT-6e7781.svg)](https://opensource.org/licenses/MIT)
 
-A Ruby gem wrapping the [Lingua](https://github.com/pemistahl/lingua-rs) Rust library for language detection.
+Native Ruby bindings for the [Lingua](https://github.com/pemistahl/lingua-rs) Rust library, bringing highly accurate language detection for short text and mixed-language text, with confidence scores and ISO 639-1/639-3 language codes.
 
 ## Installation
 
@@ -25,12 +25,7 @@ Then run `bundle install`.
 
 ### Build requirements
 
-A Rust toolchain is required to compile the native extension. `rb-sys` currently depends on `bindgen 0.69` which does not support LLVM 22+. If your system uses LLVM 22+, point to LLVM 21:
-
-```bash
-export LIBCLANG_PATH=/usr/lib/llvm21/lib
-export PATH=/usr/lib/llvm21/bin:$PATH
-```
+A Rust toolchain is required to compile the native extension.
 
 ## Usage
 
@@ -71,8 +66,6 @@ Lingua.detect('Bonjour', languages: %w[en french deu])
 
 ### Options
 
-All options accept both symbol and string keys.
-
 | Option | Type | Description |
 |---|---|---|
 | `languages` | `Array<String>` | Restrict detection to these languages (names, ISO 639-1 or ISO 639-3 codes) |
@@ -137,7 +130,52 @@ detector = Lingua::Detector.new(languages: %w[en fr de])
 detector.detect_multiple(text)
 ```
 
-### `Lingua::Language` methods
+### Parallel batch detection
+
+`Lingua::Detector` provides batch methods that process multiple texts in parallel using [Rayon](https://github.com/rayon-rs/rayon). These are significantly faster than calling single-text methods in a loop.
+
+```ruby
+detector = Lingua::Detector.new(languages: %w[en fr de])
+
+# Detect language of multiple texts
+detector.detect_batch(['Bonjour le monde', 'Hello world', 'Hallo Welt'])
+# => [#<Lingua::Language French>, #<Lingua::Language English>, #<Lingua::Language German>]
+
+# Confidence for a specific language across multiple texts
+detector.confidence_batch(['Bonjour le monde', 'Hello world'], :fr)
+# => [0.8217, 0.0215]
+
+# Confidence values for multiple texts
+detector.confidence_values_batch(['Bonjour le monde', 'Hello world'])
+# => [[#<Lingua::ConfidenceResult French (0.82)>, ...], [#<Lingua::ConfidenceResult English (0.91)>, ...]]
+
+# Mixed-language detection on multiple texts
+detector.detect_multiple_batch(['Bonjour le monde. Hello world.', 'Ich bin müde. Salut tout le monde.'])
+# => [[#<Lingua::Segment French ...>, #<Lingua::Segment English ...>], [#<Lingua::Segment German ...>, #<Lingua::Segment French ...>]]
+```
+
+| Method | Return type | Description |
+|---|---|---|
+| `detect_batch(texts)` | `Array<Language\|nil>` | Detect language of each text |
+| `detect_multiple_batch(texts)` | `Array<Array<Segment>>` | Mixed-language detection for each text |
+| `confidence_values_batch(texts)` | `Array<Array<ConfidenceResult>>` | Confidence values for each text |
+| `confidence_batch(texts, language)` | `Array<Float>` | Confidence for a specific language across texts |
+
+### Error handling
+
+`Lingua::UnknownLanguageError` (subclass of `ArgumentError`) is raised when an unrecognized language name or code is passed:
+
+```ruby
+begin
+  Lingua.detect('Hello', languages: %w[en zzzz])
+rescue Lingua::UnknownLanguageError => e
+  puts e.message # => unknown language: "zzzz"
+end
+```
+
+## API Reference
+
+### `Lingua::Language`
 
 `Lingua::Language` objects support equality (`==`) and can be used as Hash keys. You can look up a language by name, ISO 639-1 code, or ISO 639-3 code using `[]`:
 
@@ -148,6 +186,12 @@ Lingua::Language['fra']     # => #<Lingua::Language French>
 Lingua::Language['xxx']     # => nil
 ```
 
+| Method | Return type | Description |
+|---|---|---|
+| `Lingua::Language.all` | `Array<Lingua::Language>` | All supported languages |
+| `Lingua::Language.names` | `Array<String>` | All language names (e.g. `'French'`) |
+| `Lingua::Language.iso_codes` | `Array<String>` | All ISO 639-1 codes (e.g. `'fr'`) |
+
 | Method | Return type | Example |
 |---|---|---|
 | `name` | `String` | `'French'` |
@@ -157,20 +201,12 @@ Lingua::Language['xxx']     # => nil
 | `to_iso` | `String` | `'fr'` (alias for `to_iso6391`) |
 | `to_iso6391` | `String` | `'fr'` |
 | `to_iso6393` | `String` | `'fra'` |
-| `french?` | `Boolean` | `true` (works with name, ISO 639-1 or ISO 639-3: `fr?`, `fra?`) |
+| `<name>?` | `Boolean` | Dynamic predicate for any language (`french?`, `fr?`, `fra?`) |
 | `inspect` | `String` | `'#<Lingua::Language French>'` |
 | `==` | `Boolean` | Compare two languages |
 | `hash` | `Integer` | Hash value (usable as Hash key) |
 
-Class methods:
-
-| Method | Return type | Description |
-|---|---|---|
-| `Lingua::Language.all` | `Array<Lingua::Language>` | All supported languages |
-| `Lingua::Language.names` | `Array<String>` | All language names (e.g. `'French'`) |
-| `Lingua::Language.iso_codes` | `Array<String>` | All ISO 639-1 codes (e.g. `'fr'`) |
-
-### `Lingua::ConfidenceResult` methods
+### `Lingua::ConfidenceResult`
 
 Returned by `confidence_values`.
 
@@ -181,7 +217,7 @@ Returned by `confidence_values`.
 | `to_s` | `String` | `'French (0.82)'` |
 | `inspect` | `String` | `'#<Lingua::ConfidenceResult French (0.8217)>'` |
 
-### `Lingua::Segment` methods
+### `Lingua::Segment`
 
 Returned by `detect_multiple`.
 
@@ -195,22 +231,6 @@ Returned by `detect_multiple`.
 | `to_s` | `String` | `'French (0-22): Parlez-vous français? '` |
 | `inspect` | `String` | `'#<Lingua::Segment French (0-22) "Parlez-vous français? ">'` |
 
-### Error handling
-
-`Lingua::UnknownLanguageError` (subclass of `ArgumentError`) is raised when an unrecognized language name or code is passed:
-
-```ruby
-Lingua.detect('Hello', languages: %w[en zzzz])
-# => Lingua::UnknownLanguageError: unknown language: "zzzz"
-
-# Can also be rescued as ArgumentError
-begin
-  Lingua.detect('Hello', languages: %w[zzzz])
-rescue ArgumentError => e
-  puts e.message
-end
-```
-
 ## Optimization: selecting languages
 
 By default, all 75 languages are compiled into the native extension (~278 MB). If you only need a subset, select languages at build time to reduce binary size and improve detection speed.
@@ -277,6 +297,8 @@ Supported languages (from [Lingua Rust's official list](https://github.com/pemis
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake compile` to build the native extension and `rake test` to run the tests.
 
+To run benchmarks (sequential vs batch performance comparison): `rake bench`
+
 ## Acknowledgements
 
 This gem is built on top of [Lingua](https://github.com/pemistahl/lingua-rs) by [Peter M. Stahl](https://github.com/pemistahl), a highly accurate natural language detection library written in Rust.

data/Rakefile

@@ -19,4 +19,10 @@ RbSys::ExtensionTask.new('lingua', GEMSPEC) do |ext|
   ext.lib_dir = 'lib/lingua'
 end
 
+desc 'Run benchmarks (sequential vs batch)'
+task bench: :compile do
+  ENV['BENCH'] = '1'
+  ruby '-Itest', 'test/test_benchmark.rb'
+end
+
 task default: %i[compile test rubocop]

data/ext/lingua/src/detector.rs

@@ -13,6 +13,10 @@ pub fn define(ruby: &Ruby, module: &RModule) -> Result<(), Error> {
     class.define_method("confidence", method!(RubyDetector::confidence, 2))?;
     class.define_method("confidence_values", method!(RubyDetector::confidence_values, 1))?;
     class.define_method("detect_multiple", method!(RubyDetector::detect_multiple, 1))?;
+    class.define_method("detect_batch", method!(RubyDetector::detect_batch, 1))?;
+    class.define_method("detect_multiple_batch", method!(RubyDetector::detect_multiple_batch, 1))?;
+    class.define_method("confidence_values_batch", method!(RubyDetector::confidence_values_batch, 1))?;
+    class.define_method("confidence_batch", method!(RubyDetector::confidence_batch, 2))?;
     Ok(())
 }
 
@@ -81,6 +85,73 @@ impl RubyDetector {
     pub fn detect_multiple(&self, subject: String) -> Result<RArray, Error> {
         compute_detect_multiple(&self.detector, &subject)
     }
+
+    pub fn detect_batch(&self, texts: Vec<String>) -> Result<RArray, Error> {
+        let ruby = Ruby::get().unwrap();
+        let results = self.detector.detect_languages_in_parallel_of(&texts);
+        let array = ruby.ary_new_capa(results.len());
+        for result in results {
+            array.push(result.map(WrappedLanguage))?;
+        }
+        Ok(array)
+    }
+
+    pub fn detect_multiple_batch(&self, texts: Vec<String>) -> Result<RArray, Error> {
+        let ruby = Ruby::get().unwrap();
+        let results = self.detector.detect_multiple_languages_in_parallel_of(&texts);
+        let outer = ruby.ary_new_capa(results.len());
+        for (detection_results, subject) in results.iter().zip(texts.iter()) {
+            let inner = ruby.ary_new_capa(detection_results.len());
+            for r in detection_results {
+                let text = subject[r.start_index()..r.end_index()].to_string();
+                let start_index = subject[..r.start_index()].chars().count();
+                let end_index = start_index + text.chars().count();
+                inner.push(Segment {
+                    language: r.language(),
+                    start_index,
+                    end_index,
+                    word_count: r.word_count(),
+                    text,
+                })?;
+            }
+            outer.push(inner)?;
+        }
+        Ok(outer)
+    }
+
+    pub fn confidence_values_batch(&self, texts: Vec<String>) -> Result<RArray, Error> {
+        let ruby = Ruby::get().unwrap();
+        let results = self.detector.compute_language_confidence_values_in_parallel(&texts);
+        let outer = ruby.ary_new_capa(results.len());
+        for values in results {
+            let inner = ruby.ary_new_capa(values.len());
+            for (language, confidence) in values {
+                inner.push(ConfidenceResult {
+                    language,
+                    confidence,
+                })?;
+            }
+            outer.push(inner)?;
+        }
+        Ok(outer)
+    }
+
+    pub fn confidence_batch(&self, texts: Vec<String>, language: magnus::Value) -> Result<RArray, Error> {
+        let ruby = Ruby::get().unwrap();
+        let language_str = value_to_string(language)?;
+        let lang = parse_language(&language_str).ok_or_else(|| {
+            Error::new(
+                unknown_language_error(&ruby),
+                format!("unknown language: \"{language_str}\""),
+            )
+        })?;
+        let results = self.detector.compute_language_confidence_in_parallel(&texts, lang);
+        let array = ruby.ary_new_capa(results.len());
+        for score in results {
+            array.push(score)?;
+        }
+        Ok(array)
+    }
 }
 
 pub fn compute_detect_multiple(

data/lib/lingua/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Lingua
-  VERSION = '0.4.4'
+  VERSION = '0.5.0'
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: lingua_rs
 version: !ruby/object:Gem::Version
-  version: 0.4.4
+  version: 0.5.0
 platform: ruby
 authors:
 - Sébastien Vrillaud
 bindir: exe
 cert_chain: []
-date: 2026-04-06 00:00:00.000000000 Z
+date: 2026-04-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rb_sys
@@ -50,7 +50,6 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
-- ext/lingua/Cargo.lock
 - ext/lingua/Cargo.toml
 - ext/lingua/extconf.rb
 - ext/lingua/src/confidence_result.rs

data/ext/lingua/Cargo.lock

@@ -1,289 +0,0 @@
-# This file is automatically @generated by Cargo.
-# It is not intended for manual editing.
-version = 4
-
-[[package]]
-name = "aho-corasick"
-version = "1.1.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
-dependencies = [
- "memchr",
-]
-
-[[package]]
-name = "bindgen"
-version = "0.69.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088"
-dependencies = [
- "bitflags",
- "cexpr",
- "clang-sys",
- "itertools",
- "lazy_static",
- "lazycell",
- "proc-macro2",
- "quote",
- "regex",
- "rustc-hash",
- "shlex",
- "syn",
-]
-
-[[package]]
-name = "bitflags"
-version = "2.11.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af"
-
-[[package]]
-name = "cexpr"
-version = "0.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
-dependencies = [
- "nom",
-]
-
-[[package]]
-name = "cfg-if"
-version = "1.0.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
-
-[[package]]
-name = "clang-sys"
-version = "1.8.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
-dependencies = [
- "glob",
- "libc",
- "libloading",
-]
-
-[[package]]
-name = "either"
-version = "1.15.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
-
-[[package]]
-name = "glob"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
-
-[[package]]
-name = "itertools"
-version = "0.12.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569"
-dependencies = [
- "either",
-]
-
-[[package]]
-name = "lazy_static"
-version = "1.5.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
-
-[[package]]
-name = "lazycell"
-version = "1.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"
-
-[[package]]
-name = "libc"
-version = "0.2.183"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d"
-
-[[package]]
-name = "libloading"
-version = "0.8.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
-dependencies = [
- "cfg-if",
- "windows-link",
-]
-
-[[package]]
-name = "lingua"
-version = "0.1.0"
-dependencies = [
- "magnus",
-]
-
-[[package]]
-name = "magnus"
-version = "0.8.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b36a5b126bbe97eb0d02d07acfeb327036c6319fd816139a49824a83b7f9012"
-dependencies = [
- "magnus-macros",
- "rb-sys",
- "rb-sys-env",
- "seq-macro",
-]
-
-[[package]]
-name = "magnus-macros"
-version = "0.8.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
-
-[[package]]
-name = "memchr"
-version = "2.8.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
-
-[[package]]
-name = "minimal-lexical"
-version = "0.2.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
-
-[[package]]
-name = "nom"
-version = "7.1.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
-dependencies = [
- "memchr",
- "minimal-lexical",
-]
-
-[[package]]
-name = "proc-macro2"
-version = "1.0.106"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
-dependencies = [
- "unicode-ident",
-]
-
-[[package]]
-name = "quote"
-version = "1.0.45"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
-dependencies = [
- "proc-macro2",
-]
-
-[[package]]
-name = "rb-sys"
-version = "0.9.124"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c85c4188462601e2aa1469def389c17228566f82ea72f137ed096f21591bc489"
-dependencies = [
- "rb-sys-build",
-]
-
-[[package]]
-name = "rb-sys-build"
-version = "0.9.124"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "568068db4102230882e6d4ae8de6632e224ca75fe5970f6e026a04e91ed635d3"
-dependencies = [
- "bindgen",
- "lazy_static",
- "proc-macro2",
- "quote",
- "regex",
- "shell-words",
- "syn",
-]
-
-[[package]]
-name = "rb-sys-env"
-version = "0.2.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cca7ad6a7e21e72151d56fe2495a259b5670e204c3adac41ee7ef676ea08117a"
-
-[[package]]
-name = "regex"
-version = "1.12.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276"
-dependencies = [
- "aho-corasick",
- "memchr",
- "regex-automata",
- "regex-syntax",
-]
-
-[[package]]
-name = "regex-automata"
-version = "0.4.14"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
-dependencies = [
- "aho-corasick",
- "memchr",
- "regex-syntax",
-]
-
-[[package]]
-name = "regex-syntax"
-version = "0.8.10"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
-
-[[package]]
-name = "rustc-hash"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
-
-[[package]]
-name = "seq-macro"
-version = "0.3.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
-
-[[package]]
-name = "shell-words"
-version = "1.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dc6fe69c597f9c37bfeeeeeb33da3530379845f10be461a66d16d03eca2ded77"
-
-[[package]]
-name = "shlex"
-version = "1.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
-
-[[package]]
-name = "syn"
-version = "2.0.117"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
-dependencies = [
- "proc-macro2",
- "quote",
- "unicode-ident",
-]
-
-[[package]]
-name = "unicode-ident"
-version = "1.0.24"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
-
-[[package]]
-name = "windows-link"
-version = "0.2.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"