whatlanguage 1.0.5 → 1.0.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ccd0489249d639e17bda9ae90c496d33e2e6048a
+  data.tar.gz: 83a969f3f186de199996e19bd6c6f391ba7cae24
+SHA512:
+  metadata.gz: a1fbd9f4745e74637c8eddf8a0c4ac74fed02493d00b011c8e3de80887000253b0d6e39260d54afed9cc1918270c3dad227c963c07591840ad4692ecd775682a
+  data.tar.gz: 59a45845d1f19f073d0f1af57526fcedea5813126f93a7795fe4218842bcf6212d7169f6a34d04d060d587f24c172558c7548ae7d063345f7e5a4800c105f078

data/History.txt

@@ -1,3 +1,8 @@
+== 1.0.6 / 2016-01-28
+
+* Minor test fixes and tweaks
+* New release taking into account a handful of pull requests
+
 == 1.0.5 / 2013-10-05
 
 * Many more languages supported

data/README.md

@@ -2,59 +2,72 @@
 
 by Peter Cooper
 
-Text language detection. Quick, fast, memory efficient, and all in pure Ruby. Uses Bloom filters for aforementioned speed and memory benefits.
+Text language detection. Quick, fast, memory efficient, and all in pure Ruby. Uses Bloom filters for aforementioned speed and memory benefits. It works well on texts of over 10 words in length (e.g. blog posts or comments) and *very poorly* on short or Twitter-esque text, so be aware.
 
 Works with Dutch, English, Farsi, French, German, Italian, Pinyin, Swedish, Portuguese, Russian, Arabic, Finnish, Greek, Hebrew, Hungarian, Korean, Norwegian, Polish and Spanish out of the box.
 
 ## Important note
-  
-This library was first built in 2007 and has received a few minor updates over the years. There are now more efficient and effective algorithms for doing language detection which I am investigating for a WhatLanguage 2.0.
 
-This library has been updated to be distributed and to work on modern Ruby implementations but other than that, has had no improvements.
+This library was first built in 2007 and has received only a few minor updates over the years. There are now more efficient and effective algorithms for doing language detection which I am investigating for a future WhatLanguage.
+
+This library has been updated to be distributed and to work on modern Ruby implementations but other than that, has had no significant improvements.
 
 ## Synopsis
 
 Full Example
 
-    require 'whatlanguage'
-    
-    texts = []
-    texts << %q{Deux autres personnes ont été arrêtées durant la nuit}
-    texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer}
-    texts << %q{En estado de máxima alertaen su nivel de crítico}
-    texts << %q{Returns the object in enum with the maximum value.}
-    texts << %q{Propose des données au sujet de la langue espagnole.}
-    texts << %q{La palabra "mezquita" se usa en español para referirse a todo tipo de edificios dedicados.}
-    texts << %q{اللغة التي هي هذه؟}
-    texts << %q{Mitä kieltä tämä on?}
-    texts << %q{Ποια γλώσσα είναι αυτή;}
-    texts << %q{באיזו שפה זה?}
-    texts << %q{Milyen nyelv ez?}
-    texts << %q{이 어떤 언어인가?}
-    texts << %q{Hvilket språk er dette?}
-    texts << %q{W jakim języku to jest?}
-
-    texts.each { |text| puts "#{text[0..18]}... is in #{text.language.to_s.capitalize}" }
+```ruby
+require 'whatlanguage/string'
+
+texts = []
+texts << %q{Deux autres personnes ont été arrêtées durant la nuit}
+texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer}
+texts << %q{En estado de máxima alertaen su nivel de crítico}
+texts << %q{Returns the object in enum with the maximum value.}
+texts << %q{Propose des données au sujet de la langue espagnole.}
+texts << %q{La palabra "mezquita" se usa en español para referirse a todo tipo de edificios dedicados.}
+texts << %q{اللغة التي هي هذه؟}
+texts << %q{Mitä kieltä tämä on?}
+texts << %q{Ποια γλώσσα είναι αυτή;}
+texts << %q{באיזו שפה זה?}
+texts << %q{Milyen nyelv ez?}
+texts << %q{이 어떤 언어인가?}
+texts << %q{Hvilket språk er dette?}
+texts << %q{W jakim języku to jest?}
+
+texts.each { |text| puts "#{text[0..18]}... is in #{text.language.to_s.capitalize}" }
+```
 
 Initialize WhatLanguage with all filters
 
-    wl = WhatLanguage.new(:all)
+```ruby
+wl = WhatLanguage.new(:all)
+```
 
 Return language with best score
 
-    wl.language(text)
+```ruby
+wl.language(text)
+```
 
 Return hash with scores for all relevant languages
 
-    wl.process_text(text)
+```ruby
+wl.process_text(text)
+```
 
-Convenience method on String
+Convenience methods on String
 
-    "This is a test".language   # => "English"
+```ruby
+"This is a test".language   # => :english
+"This is a test".language_iso   # => :en
+```
 
 Initialize WhatLanguage with certain languages
 
-    wl = WhatLanguage.new(:english, :german, :french)
+```ruby
+wl = WhatLanguage.new(:english, :german, :french)
+```
 
 ## Requirements
 
@@ -66,18 +79,20 @@ None, minor libraries (BloominSimple and BitField) included with this release.
 
 To test, go into irb, then:
 
-    require 'whatlanguage'
-    "Je suis un homme".language
+```ruby
+require 'whatlanguage'
+"Je suis un homme".language
+```
 
 ## Credits
 
-Contributions from Konrad Reiche and Salimane Adjao Moustapha appreciated.
+Contributions from Konrad Reiche, Salimane Adjao Moustapha, and others appreciated.
 
 ## License
 
 MIT License
 
-Copyright (c) 2007-2013 Peter Cooper
+Copyright (c) 2007-2016 Peter Cooper
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/lib/whatlanguage.rb

@@ -2,65 +2,100 @@ require 'whatlanguage/bloominsimple'
 require 'whatlanguage/bitfield'
 require 'digest/sha1'
 
-class WhatLanguage  
+class WhatLanguage
   HASHER = lambda { |item| Digest::SHA1.digest(item.downcase.strip).unpack("VV") }
-  
+
   BITFIELD_WIDTH = 2_000_000
-  
+
+  ISO_CODES = {
+    nil => nil,
+    :arabic => :ar,
+    :danish => :da,
+    :dutch  => :nl,
+    :english => :en,
+    :farsi => :fa,
+    :finnish => :fi,
+    :french => :fr,
+    :german => :de,
+    :greek => :el,
+    :hebrew => :he,
+    :hungarian => :hu,
+    :italian => :it,
+    :korean => :ko,
+    :norwegian => :no,
+    :pinyin => :zh,
+    :polish => :pl,
+    :portuguese => :pt,
+    :russian => :ru,
+    :spanish => :es,
+    :swedish => :sv
+  }
+
   @@data = {}
-  
+
   def initialize(*selection)
     @selection = (selection.empty?) ? [:all] : selection
-    languages_folder = File.join(File.dirname(__FILE__), "..", "lang")
-    Dir.entries(languages_folder).grep(/\.lang/).each do |lang|
-      @@data[lang[/\w+/].to_sym] ||= BloominSimple.from_dump(File.new(File.join(languages_folder, lang), 'rb').read, &HASHER)
+    if @@data.empty?
+      languages_folder = File.join(File.dirname(__FILE__), "..", "lang")
+      Dir.entries(languages_folder).grep(/\.lang/).each do |lang|
+        @@data[lang[/\w+/].to_sym] ||= BloominSimple.from_dump(File.new(File.join(languages_folder, lang), 'rb').read, &HASHER)
+      end
     end
   end
-  
+
+  def languages
+    @languages ||=
+      begin
+        if @selection.include?(:all)
+          languages = @@data.keys
+        else
+          languages = @@data.keys & @selection  # intersection
+        end
+      end
+  end
+
   # Very inefficient method for now.. but still beats the non-Bloom alternatives.
   # Change to better bit comparison technique later..
   def process_text(text)
     results = Hash.new(0)
     it = 0
-    text.downcase.split.each do |word|
+    to_lowercase(text).split.each do |word|
       it += 1
 
-      if @selection.include?(:all)
-        languages = @@data.keys
-      else
-        languages = @@data.keys & @selection  # intersection
-      end
-
       languages.each do |lang|
         results[lang] += 1 if @@data[lang].includes?(word)
       end
-      
+
       # Every now and then check to see if we have a really convincing result.. if so, exit early.
       if it % 4 == 0 && results.size > 1
         top_results = results.sort_by{|a,b| -b}[0..1]
-        
+
         # Next line may need some tweaking one day..
         break if top_results[0][1] > 4 && ((top_results[0][1] > top_results[1][1] * 2) || (top_results[0][1] - top_results[1][1] > 25))
       end
-      
+
       #break if it > 100
     end
     results
   end
-  
+
   def language(text)
     process_text(text).max { |a,b| a[1] <=> b[1] }.first rescue nil
   end
-  
+
+  def language_iso(text)
+    ISO_CODES[language(text)]
+  end
+
   def self.filter_from_dictionary(filename)
     bf = BloominSimple.new(BITFIELD_WIDTH, &HASHER)
     File.open(filename).each { |word| bf.add(word) }
     bf
   end
-end
 
-class String
-  def language
-    WhatLanguage.new(:all).language(self)
+  if !defined? UnicodeUtils
+    define_method(:to_lowercase) { |str| str.downcase }
+  else
+    define_method(:to_lowercase) { |str| UnicodeUtils.casefold(str) }
   end
 end

data/lib/whatlanguage/string.rb

@@ -0,0 +1,11 @@
+require 'whatlanguage'
+
+class String
+  def language
+    WhatLanguage.new(:all).language(self)
+  end
+
+  def language_iso
+    WhatLanguage.new(:all).language_iso(self)
+  end
+end

data/lib/whatlanguage/version.rb

@@ -1,3 +1,3 @@
 class WhatLanguage
-  VERSION = '1.0.5'
+  VERSION = '1.0.6'
 end

data/test/test_whatlanguage.rb

@@ -1,17 +1,27 @@
 # encoding: utf-8
 require "test/unit"
 
-require 'whatlanguage'
+# not a dependency
+begin
+  require 'unicode_utils'
+rescue LoadError
+end
+
+require 'whatlanguage/string'
 
 class TestWhatLanguage < Test::Unit::TestCase
   def setup
     @wl = WhatLanguage.new(:all)
   end
-  
+
   def test_string_method
     assert_equal :english, "This is a test".language
   end
 
+  def test_string_iso_method
+    assert_equal :en, "this is a test".language_iso
+  end
+
   def test_arabic
     assert_equal :arabic, @wl.language("اللغة التي هي هذه؟")
   end
@@ -29,9 +39,9 @@ class TestWhatLanguage < Test::Unit::TestCase
   end
 
   def test_french
-    assert_equal :french, @wl.language("Bonjour, je m'appelle Sandrine. Voila ma chatte.")
+    assert_equal :french, @wl.language("Bonjour, je m'appelle Sandrine. Voila mon chat.")
   end
-  
+
   def test_german
     assert_equal :german, @wl.language("Welche Sprache ist das?")
   end
@@ -79,23 +89,23 @@ class TestWhatLanguage < Test::Unit::TestCase
   def test_swedish
     assert_equal :swedish, @wl.language("Vilket språk är detta?")
   end
-  
+
+  def test_danish
+    assert_equal :danish, @wl.language("Dansk er et nord-germansk sprog af den østnordiske (kontinentale) gruppe, der tales af ca. seks millioner mennesker.")
+  end
+
   def test_nothing
     assert_nil @wl.language("")
   end
-  
+
   def test_something
     assert_not_nil @wl.language("test")
   end
-  
+
   def test_processor
     assert_kind_of Hash, @wl.process_text("this is a test")
   end
 
-  def test_italian
-    assert_equal :italian, @wl.language("Roma, capitale dell'impero romano, è stata per secoli il centro politico e culturale della civiltà occidentale.")
-  end
-
   def test_language_selection
     selective_wl = WhatLanguage.new(:german, :english)
     assert_equal :german, selective_wl.language("der die das")
@@ -110,4 +120,10 @@ class TestWhatLanguage < Test::Unit::TestCase
     selective_wl = WhatLanguage.new(:german, :all, :english)
     assert_equal :russian, selective_wl.language("Все новости в хронологическом порядке")
   end
-end
+
+  if defined? UnicodeUtils
+    def test_casing_conversion
+      assert_equal "âncora cor âmbar".language, "ÂNCORA COR ÂMBAR".language
+    end
+  end
+end

metadata

@@ -1,15 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: whatlanguage
 version: !ruby/object:Gem::Version
-  version: 1.0.5
-  prerelease: 
+  version: 1.0.6
 platform: ruby
 authors:
 - Peter Cooper
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-05 00:00:00.000000000 Z
+date: 2016-01-28 00:00:00.000000000 Z
 dependencies: []
 description: WhatLanguage rapidly detects the language of a sample of text
 email:
@@ -18,7 +17,7 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
+- ".gitignore"
 - Gemfile
 - History.txt
 - LICENSE.txt
@@ -30,6 +29,7 @@ files:
 - copyright-en
 - example.rb
 - lang/arabic.lang
+- lang/danish.lang
 - lang/dutch.lang
 - lang/english.lang
 - lang/farsi.lang
@@ -51,32 +51,32 @@ files:
 - lib/whatlanguage.rb
 - lib/whatlanguage/bitfield.rb
 - lib/whatlanguage/bloominsimple.rb
+- lib/whatlanguage/string.rb
 - lib/whatlanguage/version.rb
 - test/test_whatlanguage.rb
 - whatlanguage.gemspec
 homepage: https://github.com/peterc/whatlanguage
 licenses: []
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 2.4.5
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: Natural language detection for text samples
 test_files:
 - test/test_whatlanguage.rb