whatlanguage 1.0.0

data/History.txt

@@ -0,0 +1,4 @@
+== 1.0.0 / 2007-07-02
+
+* First version with pre-built English, French, and Spanish filters
+

data/Manifest.txt

@@ -0,0 +1,19 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+build_filter.rb
+example.rb
+lang/dutch.lang
+lang/farsi.lang
+lang/german.lang
+lang/pinyin.lang
+lang/russian.lang
+lang/english.lang
+lang/portuguese.lang
+lang/french.lang
+lang/spanish.lang
+lib/bitfield.rb
+lib/bloominsimple.rb
+lib/whatlanguage.rb
+test/test_whatlanguage.rb

data/README.txt

@@ -0,0 +1,71 @@
+whatlanguage
+    by Peter Cooper
+    http://www.petercooper.co.uk/
+    http://www.rubyinside.com/
+
+== DESCRIPTION:
+  
+Text language detection. Quick, fast, memory efficient, and all in pure Ruby. Uses Bloom filters for aforementioned speed and memory benefits.
+
+== FEATURES/PROBLEMS:
+  
+* Only does French, English and Spanish out of the box. Very easy to train new languages though.
+* It can be made far more efficient at the comparison stage, but all in good time..! It still beats literal dictionary approaches.
+* No filter selection yet, you get 'em all loaded.
+* Tests are reasonably light.
+
+== SYNOPSIS:
+
+  Full Example
+    require 'whatlanguage'
+    
+    texts = []
+    texts << %q{Deux autres personnes ont été arrêtées durant la nuit}
+    texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer}
+    texts << %q{En estado de máxima alertaen su nivel de crítico}
+    texts << %q{Returns the object in enum with the maximum value.}
+    texts << %q{Propose des données au sujet de la langue espagnole.}
+    texts << %q{La palabra "mezquita" se usa en español para referirse a todo tipo de edificios dedicados.}
+    
+    texts.each { |text| puts "#{text[0..18]}... is in #{text.language.to_s.capitalize}" }
+
+  Initialize WhatLanguage with all filters
+    wl = WhatLanguage.new(:all)
+
+  Return language with best score
+    wl.language(text)
+
+  Return hash with scores for all relevant languages
+    wl.process_text(text)
+
+  Convenience method on String
+    "This is a test".language   # => "English"
+
+== REQUIREMENTS:
+
+* None, minor libraries (BloominSimple and BitField) included with this release.
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2007-2008 Peter Cooper
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,17 @@
+# -*- ruby -*-
+
+require 'rubygems'
+require 'hoe'
+require './lib/whatlanguage.rb'
+
+Hoe.new('whatlanguage', WhatLanguage::VERSION) do |p|
+  p.rubyforge_name = 'whatlanguage'
+  p.author = 'Peter Cooper'
+  p.email = 'whatlanguage@peterc.org'
+  p.summary = 'Fast, quick, textual language detection'
+  p.description = p.paragraphs_of('README.txt', 2..5).join("\n\n")
+  p.url = "http://rubyforge.org/projects/whatlanguage/"
+  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
+end
+
+# vim: syntax=Ruby

data/build_filter.rb

@@ -0,0 +1,9 @@
+# Use this to build new filters (for other languages, ideally) from /usr/share/dict/words style dictionaries..
+#
+# Call like so..
+#   ruby build_filter.rb /usr/share/dict/words lang/english.lang
+# (replace params as necessary)
+
+require 'lib/whatlanguage'
+filter = WhatLanguage.filter_from_dictionary(ARGV[0])
+File.open(ARGV[1], 'w') { |f| f.write filter.dump }

data/example.rb

@@ -0,0 +1,51 @@
+require 'lib/whatlanguage'
+require 'benchmark'
+
+texts = []
+
+texts << %q{Deux autres personnes ont été arrêtées durant la nuit. "Induite par le quinquennat, la modernisation de nos institutions (...) est un facteur de modernité et d'efficacité", a déclaré François Fillon. "Devant cet exécutif plus resserré et plus efficace, les pouvoirs du Parlement doivent être renforcés", a-t-il ajouté, en évoquant par exemple le contrôle parlementaire des nominations à "certains postes publics."
+
+Le premier ministre, indiquant que le président Nicolas Sarkozy entendait "réunir une commission réunissant des personnalités incontestables pour leurs compétences et représentatives de notre diversité politique qui sera chargée d'éclairer ses choix" en matière de modernisation des institutions.
+
+"Faut-il faire élire quelques députés au scrutin proportionnel? (...) Aucun sujet ne doit être tabou", a-t-il lâché. "Enfin nous devrons engager, comme le demande le Conseil constitutionnel, une révision de la carte des circonscriptions législatives. Ce travail sera engagé dans la transparence et en y associant l'opposition".}
+
+texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer. Seven are believed to be doctors or medical students, while one formerly worked as a laboratory technician.
+
+Australian media have identified a man arrested at Brisbane airport as Dr Mohammed Haneef, 27.
+
+Two men have been arrested in Blackburn under terror laws but police have not confirmed a link with the car bombs.
+
+The pair were detained on an industrial estate and are being held at a police station in Lancashire on suspicion of offences under the Terrorism Act 2000.
+
+Thousands of passengers travelling from Heathrow Airport's Terminal 4 face major delays after a suspect bag sparked a security alert.
+
+BAA said the departure lounge was partially evacuated and departing passengers were being re-screened, causing some cancellations and delays.}
+
+texts << %q{En estado de máxima alertaen su nivel de crítico. Cinco detenciones hasta el momento parecen indicar que los coches bomba de Londres y Glasgow fueron obra de una misma célula de terroristas islámicos con residencia en el Reino Unido, posiblemente con alguna conexión con otros grupos previamente desarticulados. La relación con éstos, singularmente con el núcleo de Dhiren Barot (condenado a 30 años por sus planes de llenar limusinas con bombonas de gas para provocar una masiva explosión), podría haber llevado a su detención tiempo atrás y su puesta en libertad al no existir suficientes pruebas contra ellos.}
+
+texts << %q{Fern- und Regionalzüge, aber auch die S-Bahnen in den Großstädten stehen still. Gerade hat die Kanzlerin die Ergebnisse des Energiegipfels mit der Wirtschaft, den Energieerzeugern und Verbraucherschützern referiert, die "sachliche Atmosphäre" gelobt. Da wird der Umweltminister von einem Journalisten an seinen Ausspruch vom "Wirtschaftsstalinisten" erinnert, mit dem er jüngst den BASF-Vorstandschef Jürgen Hambrecht belegt hat im Streit um die ehrgeizigen Ziele der deutschen Klimapolitik. Wie haben denn seine Beiträge in der Runde für eine sachliche Atmosphäre ausgesehen? Gabriel überlegt, aber die Kanzlerin ist schneller.}
+
+#texts << %q{Deux autres personnes ont été arrêtées durant la nuit}
+#texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer}
+#texts << %q{En estado de máxima alertaen su nivel de crítico}
+#texts << %q{Returns the object in enum with the maximum value.}
+#texts << %q{Propose des données au sujet de la langue espagnole.}
+#texts << %q{La palabra "mezquita" se usa en español para referirse a todo tipo de edificios dedicados.}
+#texts << %q{Fern- und Regionalzüge, aber auch die S-Bahnen in den Großstädten stehen still.}
+
+#texts.collect! { |t| (t + " ") * 5 }
+
+@wl = WhatLanguage.new(:all)
+
+puts Benchmark.measure {
+
+100.times do  
+texts.each { |text| 
+  lang = text.language.to_s.capitalize
+#  puts "#{text[0..18]}... is in #{lang}"
+#  puts @wl.process_text(text).sort_by{|a,b| b }.reverse.inspect
+#  puts "---"
+}
+end
+
+}

data/lib/bitfield.rb

@@ -0,0 +1,64 @@
+#        NAME: BitField
+#      AUTHOR: Peter Cooper
+#     LICENSE: MIT ( http://www.opensource.org/licenses/mit-license.php )
+#   COPYRIGHT: (c) 2007 Peter Cooper (http://www.petercooper.co.uk/)
+#     VERSION: v4
+#     HISTORY: v4 (better support for loading and dumping fields)
+#              v3 (supports dynamic bitwidths for array elements.. now doing 32 bit widths default)
+#              v2 (now uses 1 << y, rather than 2 ** y .. it's 21.8 times faster!)
+#              v1 (first release)
+#
+# DESCRIPTION: Basic, pure Ruby bit field. Pretty fast (for what it is) and memory efficient.
+#              I've written a pretty intensive test suite for it and it passes great. 
+#              Works well for Bloom filters (the reason I wrote it).
+#
+#              Create a bit field 1000 bits wide
+#                bf = BitField.new(1000)
+#
+#              Setting and reading bits
+#                bf[100] = 1
+#                bf[100]    .. => 1
+#                bf[100] = 0
+#
+#              More
+#                bf.to_s = "10101000101010101"  (example)
+#                bf.total_set         .. => 10  (example - 10 bits are set to "1")
+
+class BitField
+  attr_reader :size
+  attr_accessor :field
+  include Enumerable
+  
+  ELEMENT_WIDTH = 32
+  
+  def initialize(size)
+    @size = size
+    @field = Array.new(((size - 1) / ELEMENT_WIDTH) + 1, 0)
+  end
+  
+  # Set a bit (1/0)
+  def []=(position, value)
+    value == 1 ? @field[position / ELEMENT_WIDTH] |= 1 << (position % ELEMENT_WIDTH) : @field[position / ELEMENT_WIDTH] ^= 1 << (position % ELEMENT_WIDTH)
+  end
+  
+  # Read a bit (1/0)
+  def [](position)
+    @field[position / ELEMENT_WIDTH] & 1 << (position % ELEMENT_WIDTH) > 0 ? 1 : 0
+  end
+  
+  # Iterate over each bit
+  def each(&block)
+    @size.times { |position| yield self[position] }
+  end
+  
+  # Returns the field as a string like "0101010100111100," etc.
+  def to_s
+    inject("") { |a, b| a + b.to_s }
+  end
+  
+  # Returns the total number of bits that are set
+  # (The technique used here is about 6 times faster than using each or inject direct on the bitfield)
+  def total_set
+    @field.inject(0) { |a, byte| a += byte & 1 and byte >>= 1 until byte == 0; a }
+  end
+end

data/lib/bloominsimple.rb

@@ -0,0 +1,88 @@
+#        NAME: BloominSimple
+#      AUTHOR: Peter Cooper
+#     LICENSE: MIT ( http://www.opensource.org/licenses/mit-license.php )
+#   COPYRIGHT: (c) 2007 Peter Cooper
+# DESCRIPTION: Very basic, pure Ruby Bloom filter. Uses my BitField, pure Ruby
+#              bit field library (http://snippets.dzone.com/posts/show/4234).
+#              Supports custom hashing (default is 3).
+#
+#              Create a Bloom filter that uses default hashing with 1Mbit wide bitfield
+#                bf = BloominSimple.new(1_000_000)
+#              
+#              Add items to it
+#                File.open('/usr/share/dict/words').each { |a| bf.add(a) }
+#
+#              Check for existence of items in the filter
+#                bf.includes?("people")     # => true
+#                bf.includes?("kwyjibo")    # => false
+#
+#              Add better hashing (c'est easy!)
+#                require 'digest/sha1'
+#                b = BloominSimple.new(1_000_000) do |item|
+#                  Digest::SHA1.digest(item.downcase.strip).unpack("VVVV")
+#                end
+#
+#              More
+#                %w{wonderful ball stereo jester flag shshshshsh nooooooo newyorkcity}.each do |a|
+#                  puts "#{sprintf("%15s", a)}: #{b.includes?(a)}"
+#                end
+#
+#                 #  =>   wonderful: true
+#                 #  =>        ball: true
+#                 #  =>      stereo: true
+#                 #  =>      jester: true
+#                 #  =>        flag: true
+#                 #  =>  shshshshsh: false
+#                 #  =>    nooooooo: false
+#                 #  => newyorkcity: false
+
+require File.join(File.dirname(__FILE__), 'bitfield')
+
+class BloominSimple
+  attr_accessor :bitfield, :hasher
+  
+  def initialize(bitsize, &block)
+    @bitfield = BitField.new(bitsize)
+    @size = bitsize
+    @hasher = block || lambda do |word|
+      word = word.downcase.strip
+      [h1 = word.sum, h2 = word.hash, h2 + h1 ** 3]
+    end
+  end
+  
+  # Add item to the filter
+  def add(item)
+    @hasher[item].each { |hi| @bitfield[hi % @size] = 1 }
+  end
+  
+  # Find out if the filter possibly contains the supplied item
+  def includes?(item)
+    @hasher[item].each { |hi| return false unless @bitfield[hi % @size] == 1 } and true
+  end
+  
+  # Allows comparison between two filters. Returns number of same bits.
+  def &(other)
+    raise "Wrong sizes" if self.bitfield.size != other.bitfield.size
+    same = 0
+    #self.bitfield.size.times do |pos|
+    #  same += 1 if self.bitfield[pos] & other.bitfield[pos] == 1
+    #end
+    self.bitfield.total_set.to_s + "--" + other.bitfield.total_set.to_s
+  end
+
+  # Dumps the bitfield for a bloom filter for storage
+  def dump
+    [@size, *@bitfield.field].pack("I*")
+    #Marshal.dump([@size, @bitfield])
+  end
+  
+  # Creates a new bloom filter object from a stored dump (hasher has to be resent though for additions)
+  def self.from_dump(data, &block)
+    data = data.unpack("I*")
+    #data = Marshal.load(data)
+    temp = new(data[0], &block)
+    temp.bitfield.field = data[1..-1]
+    temp
+  end
+end
+

data/lib/whatlanguage.rb

@@ -0,0 +1,59 @@
+require File.join(File.dirname(__FILE__), 'bloominsimple')
+require 'digest/sha1'
+
+class WhatLanguage
+  VERSION = '1.0.0'
+  
+  HASHER = lambda { |item| Digest::SHA1.digest(item.downcase.strip).unpack("VV") }
+  
+  BITFIELD_WIDTH = 2_000_000
+  
+  @@data = {}
+  
+  def initialize(options)
+    languages_folder = File.join(File.dirname(__FILE__), "..", "lang")
+    Dir.entries(languages_folder).grep(/\.lang/).each do |lang|
+      @@data[lang[/\w+/].to_sym] ||= BloominSimple.from_dump(File.read(File.join(languages_folder, lang)), &HASHER)
+    end
+  end
+  
+  # Very inefficient method for now.. but still beats the non-Bloom alternatives.
+  # Change to better bit comparison technique later..
+  def process_text(text)
+    results = Hash.new(0)
+    it = 0
+    text.split.collect {|a| a.downcase }.each do |word|
+      it += 1
+      @@data.keys.each do |lang|
+        results[lang] += 1 if @@data[lang].includes?(word)
+      end
+      
+      # Every now and then check to see if we have a really convincing result.. if so, exit early.
+      if it % 4 == 0 && results.size > 1
+        top_results = results.sort_by{|a,b| b}.reverse[0..1]
+        
+        # Next line may need some tweaking one day..
+        break if top_results[0][1] > 4 && ((top_results[0][1] > top_results[1][1] * 2) || (top_results[0][1] - top_results[1][1] > 25))
+      end
+      
+      #break if it > 100
+    end
+    results
+  end
+  
+  def language(text)
+    process_text(text).max { |a,b| a[1] <=> b[1] }.first rescue nil
+  end
+  
+  def self.filter_from_dictionary(filename)
+    bf = BloominSimple.new(BITFIELD_WIDTH, &HASHER)
+    File.open(filename).each { |word| bf.add(word) }
+    bf
+  end
+end
+
+class String
+  def language
+    WhatLanguage.new(:all).language(self)
+  end
+end

data/test/test_whatlanguage.rb

@@ -0,0 +1,33 @@
+require "test/unit"
+
+require File.join(File.dirname(__FILE__), "..", "lib", "whatlanguage")
+
+class TestWhatLanguage < Test::Unit::TestCase
+  def setup
+    @wl = WhatLanguage.new(:all)
+  end
+  
+  def test_string_method
+    assert_equal :english, "This is a test".language
+  end
+  
+  def test_french
+    assert_equal :french, @wl.language("Bonjour, je m'appelle Sandrine. Voila ma chatte.")
+  end
+  
+  def test_spanish
+    assert_equal :spanish, @wl.language("La palabra mezquita se usa en español para referirse a todo tipo de edificios dedicados.")
+  end
+  
+  def test_nothing
+    assert_nil @wl.language("")
+  end
+  
+  def test_something
+    assert_not_nil @wl.language("test")
+  end
+  
+  def test_processor
+    assert_kind_of Hash, @wl.process_text("this is a test")
+  end
+end

metadata

@@ -0,0 +1,83 @@
+--- !ruby/object:Gem::Specification 
+name: whatlanguage
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Peter Cooper
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-08-22 00:00:00 +01:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: hoe
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.7.0
+    version: 
+description: "== FEATURES/PROBLEMS:  * Only does French, English and Spanish out of the box. Very easy to train new languages though. * It can be made far more efficient at the comparison stage, but all in good time..! It still beats literal dictionary approaches. * No filter selection yet, you get 'em all loaded. * Tests are reasonably light.  == SYNOPSIS:  Full Example require 'whatlanguage'  texts = [] texts << %q{Deux autres personnes ont \xC3\xA9t\xC3\xA9 arr\xC3\xAAt\xC3\xA9es durant la nuit} texts << %q{The links between the attempted car bombings in Glasgow and London are becoming clearer} texts << %q{En estado de m\xC3\xA1xima alertaen su nivel de cr\xC3\xADtico} texts << %q{Returns the object in enum with the maximum value.} texts << %q{Propose des donn\xC3\xA9es au sujet de la langue espagnole.} texts << %q{La palabra \"mezquita\" se usa en espa\xC3\xB1ol para referirse a todo tipo de edificios dedicados.}  texts.each { |text| puts \"#{text[0..18]}... is in #{text.language.to_s.capitalize}\" }  Initialize WhatLanguage with all filters wl = WhatLanguage.new(:all)"
+email: whatlanguage@peterc.org
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- History.txt
+- Manifest.txt
+- README.txt
+files: 
+- History.txt
+- Manifest.txt
+- README.txt
+- Rakefile
+- build_filter.rb
+- example.rb
+- lang/dutch.lang
+- lang/farsi.lang
+- lang/german.lang
+- lang/pinyin.lang
+- lang/russian.lang
+- lang/english.lang
+- lang/portuguese.lang
+- lang/french.lang
+- lang/spanish.lang
+- lib/bitfield.rb
+- lib/bloominsimple.rb
+- lib/whatlanguage.rb
+- test/test_whatlanguage.rb
+has_rdoc: true
+homepage: http://rubyforge.org/projects/whatlanguage/
+post_install_message: 
+rdoc_options: 
+- --main
+- README.txt
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: whatlanguage
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Fast, quick, textual language detection
+test_files: 
+- test/test_whatlanguage.rb