treat 0.1.1

data/examples/texts/to_infinity_and_beyond.txt

@@ -0,0 +1,15 @@
+To Infinity and Beyond
+
+THE annual Consumer Electronics Show (CES) feels right at home amid the blazing neon excess of Las Vegas. For four days each January, thousands of companies gather to showcase their flashiest technologies at America's biggest trade show. This year over 20,000 brand new gadgets competed for attendees' attention. Each has its own power cord or battery, and almost every one is either bigger or faster—and thus more power-hungry—than last year's model.
+
+The International Energy Agency expects consumer electronics' appetite for electricity to double over the next decade, from 15% to 30% of residential consumption worldwide. Even supposedly energy-saving innovations, such as the organic-LED (OLED) screen in the massive 55-inch television proudly displayed by South Korea's LG, consume oodles of power, just by dint of being so big.
+
+But this year's show also sees a welcome counter-trend. Several companies launched products labelled as having "infinite power". Such devices are meant to generate at least as much power as they consume. Buy one of these gizmos, the theory goes, and you need never connect it to a wall socket.
+
+Eton Corporation announced Rukus, a portable boom box that can stream music from Bluetooth devices such as smartphones and tablets. For every hour that the Rukus is in full sunlight, it can play an hour of music, harvesting solar energy from 40 square inches (260 square centimetres) of photovoltaic panels. It has an internal battery which stores sunshine for cloudy picnics and can also be tapped to recharge other mobile gadgets.
+
+Similarly impressive is a high-tech cover for Amazon's Kindle ebook reader by Solar Focus (see picture). A solar panel on the outer face gives 90 minutes of reading time for every hour of sunlight. Surplus energy is stored in a small lithium-ion battery and allows an LED reading lamp to run for up to 50 hours without drawing on the Kindle's own battery.
+
+Both the Kindle and the Rukus have frugal monochrome E Ink displays that consume a fraction of the power of the colourful LCD screens found on most gadgets. Even the best solar-panel case for Apple's iPhone, for instance, can do no more than slow the rate at which the smartphone runs down. You might think, then, that something as large as a practical solar-powered electric car would be utterly impossible. Not so, says Ford. At CES, the carmaker showed a domestic solar panel kit it claims will offset all the electricity used over the lifetime of its new Focus Electric plug-in vehicle. The 150 square-foot (14 square-metre) array, to be installed on owners' houses, should feed enough power to the grid as the car draws to recharge its battery. to account for every mile a typical motorist drives. Panels on roofs in rainy Seattle might allow 12,000 miles (19,300km) of driving each year; denizens of sunny Tucson may squeeze out 15,000 miles or more.
+
+The solar kit will set a Ford owner back around $10,000 (the price would be higher were it not for American federal incentives). It comes with a 25-year guarantee—22 years more than the warranty on the car. This is still some way from the dream of a self-contained solar runabout, but it is a start. The increasingly power-hungry electronics industry might will no doubt need to steer itself in a similar direction.

data/lib/treat.rb

@@ -0,0 +1,91 @@
+# This file requires all source code files for the Treat module.
+
+#
+# Main Treat namespace.
+#
+# Textual model:
+#
+# - Tree - Contains abstract tree node and leaf structures.
+# - Entities - Contains concrtypee node and leaf structures
+#   that represent textual entities.
+#
+# Algorithm namespaces:
+#
+# - Dtypeectors - Namespace for language, encoding, and format
+#   detectors.
+# - Extractors - Namespace for algorithms that extract
+#   information from entities.
+# - Formatters - Namespace for algorithms that handle
+#   conversion to and from different formats.
+# - Inflectors - Namespace for algorithms that supply
+#   the base form, inflections and declensions of a word.
+# - Lexicalizers - Namespace for algorithms that supply
+#   lexical information about a word (part of speech,
+#   synstypes, klass.)
+# - Processors - Namespace for algorithms that process an
+#   entity into a tree of sub-entities.
+#
+# Other modules:
+#
+# - Group - Creates functions for algorithm groups.
+# - Proxies - Provide proxies for Treat functions on String,
+#   Numeric and Array classes.
+# - Utilities - Supply utility functions used across the library.
+#
+module Treat
+
+  # Make sure that we are running on Ruby 1.9 or higher.
+  if RUBY_VERSION <= '1.9'
+    raise 'Treat requires Ruby 1.9 or higher.'
+  end
+
+  # The current version of Treat.
+  VERSION = "0.1.1"
+
+  # Require all files for the Treat library.
+  require 'treat/exception'
+  require 'treat/utilities'
+  require 'treat/resources'
+  require 'treat/entities'
+  require 'treat/categories'
+  require 'treat/proxies'
+
+  # Provides syntactic sugar.
+  require 'treat/sugar'
+  extend Sugar
+  
+  # Create class variables for the Treat module.
+  class << self
+    # Default language to use when detect_language is false
+    attr_accessor :default_language
+    # Default encoding to use.
+    attr_accessor :default_encoding
+    # Boolean - detect language or use default?
+    attr_accessor :detect_language
+    # Identifier - the ideal entity level to detect language at
+    # (:entity, :sentence, :zone, :text, :document, klass.)
+    attr_accessor :language_detection_level
+    # String - main folder for executable files.
+    attr_accessor :bin
+  end
+
+  # Folder paths.
+  @@lib = File.dirname(__FILE__)
+  @@test = @@lib + '/../test/'
+  @@tmp = @@lib + '/../tmp/'
+  @@bin = @@lib + '/../bin'
+  def self.lib; @@lib; end
+  def self.test; @@test; end
+  def self.tmp; @@tmp; end
+
+  # Stype the default language to english.
+  self.default_language = :eng
+  # Stype the default encoding to utf-8.
+  self.default_encoding = :utf_8
+  # Turn language detection off by default.
+  self.detect_language = false
+  # Dtypeect the language once per text by default.
+  self.language_detection_level = :text
+  # Stype the bin path to the gem's bin folder by default.
+  self.bin = @@bin  
+end

data/lib/treat/buildable.rb

@@ -0,0 +1,115 @@
+module Treat
+  module Buildable
+
+    def from_anything(file_or_value, id)
+      if File.readable?(file_or_value.to_s)
+        from_file(file_or_value)
+      elsif file_or_value.is_a?(String)
+        from_string(file_or_value)
+      elsif file_or_value.is_a?(Numeric)
+        from_numeric(file_or_value)
+      else
+        raise "Unrecognizable input #{file_or_value}. "+
+        "Use filename, folder, text or a number."
+      end
+    end
+    
+    def from_string(string)
+      if self == Treat::Entities::Document ||
+        self == Treat::Entities::Collection
+        raise Treat::Exception,
+        "Cannot create a document or collection from " +
+        "a string (need a readable file/folder)."
+      end
+      string = string.to_s
+      dot = string.count('.') + string.count('!') + string.count('?')
+      return Treat::Entities::Text.new(string) if dot > 1 ||
+      (string.count("\n") > 0 && dot == 1)
+      return Treat::Entities::Sentence.new(string) if dot == 1 && string.size > 5
+      if string.count(' ') == 0
+        return Treat::Entities::Clitic.new(string) if string == "'s"
+        return Treat::Entities::Word.new(string) if string =~ /^[[:alpha:]\-']+$/
+        return Treat::Entities::Number.new(string) if string =~ /^[[:digit:]]+$/
+        return Treat::Entities::Punctuation.new(string) if string =~ /^[[:punct:]]+$/
+        return Treat::Entities::Symbol.new(string)
+      else
+        return Treat::Entities::Phrase.new(string)
+      end
+      return Treat::Entities::Unknown.new(string)
+    end
+
+    def from_numeric(numeric)
+      unless self == Treat::Entities::Number
+        raise Treat::Exception,
+        "Cannot create something else than a " +
+        " number from a numeric object."
+      end
+      Treat::Entities::Number.new(numeric.to_s)
+    end
+
+    def from_folder(folder)
+      unless FileTest.directory?(folder)
+        raise Treat::Exception,
+        "Path '#{folder}' does not point to a folder."
+      end
+      unless File.readable?(folder)
+        raise Treat::Exception,
+        "Folder '#{folder}' is not readable."
+      end
+      unless self == Treat::Entities::Collection
+        raise Treat::Exception,
+        "Cannot create something else than a " +
+        "collection from folder '#{folder}'."
+      end
+      c = Treat::Entities::Collection.new
+      folder += '/' unless folder[-1] == '/'
+      Dir[folder + '*'].each do |f|
+        next if FileTest.directory?(f)
+        c << Treat::Entities::Document.from_file(f)
+      end
+      c
+    end
+
+    def from_file(file)
+      unless File.readable?(file)
+        raise Treat::Exception,
+        "Path '#{file}' does not point to a readable file."
+      end
+      if FileTest.directory?(file)
+        from_folder(file)
+      else
+        ext = file.split('.')[-1]
+        ext = 'yaml' if ext == 'yml' # Humanize the yml extension.
+        if Treat::Formatters::Unserializers.list.
+          include?(ext.downcase.intern)
+          from_serialized_file(file)
+        else
+          from_raw_file(file)
+        end
+      end
+    end
+
+    def from_raw_file(file)
+      unless self == Treat::Entities::Document
+        raise Treat::Exception,
+        "Cannot create something else than a " +
+        "document from raw file '#{file}'."
+      end
+      d = Treat::Entities::Document.new(file)
+      d.read
+    end
+
+    def from_serialized_file(file)
+      unless [Treat::Entities::Document, Treat::Entities::Collection].include?(self)
+        raise Treat::Exception,
+        "Cannot create something else than a " +
+        "document from raw file '#{file}'."
+      end
+      d = Treat::Entities::Document.new(file)
+      d.unserialize
+      d.children[0].set_as_root!
+      d.children[0]
+    end
+
+  end
+end

data/lib/treat/categories.rb

@@ -0,0 +1,29 @@
+module Treat
+  module Categories
+    # Modify the module that includes Category to
+    # setup autoload, delegators and provide a list
+    # of methods.
+    class << self; attr_accessor :list; end
+    self.list = []
+    # Boolean - does any of the categories
+    # groups respond to the symbol.
+    def self.have_method?(sym); methods.include?(sym); end
+    # Provide a list of all methods implemented
+    # by all Treat categories.
+    @@methods = []
+    def self.methods
+      return @@methods unless @@methods.empty?
+      self.list.each do |ns|
+        ns.methods.each { |method| @@methods << method }
+      end
+      @@methods
+    end
+    require 'treat/category'
+    require 'treat/detectors'
+    require 'treat/formatters'
+    require 'treat/processors'
+    require 'treat/lexicalizers'
+    require 'treat/extractors'
+    require 'treat/inflectors'
+  end
+end

data/lib/treat/category.rb

@@ -0,0 +1,28 @@
+module Treat
+  # Provides functions common to all algorithm categories.
+  module Category
+    require 'treat/group'
+    def self.extended(category)
+      Treat::Categories.list << category
+      category.module_eval do
+        groups.each do |group|
+          group = const_get(group)
+          group.targets.each do |entity_type|
+            entity = Entities.const_get(cc(entity_type))
+            entity.class_eval { add_delegators group }
+          end
+        end
+      end
+    end
+    def groups; self.constants; end
+    # Provide a list of methods implemented in
+    # the groups contained within that
+    def methods
+      methods = []
+      groups.each do |group|
+        methods << const_get(group).method
+      end
+      methods
+    end
+  end
+end

data/lib/treat/delegatable.rb

@@ -0,0 +1,90 @@
+module Treat
+  # Makes a class delegatable, allowing calls on it to be forwarded
+  # to a delegate class performing the appropriate call.
+  module Delegatable
+
+    # Get the default delegate for that language
+    # inside the given group.
+    def get_language_delegate(language, group)
+      lang = Treat::Resources::Languages.describe(language)
+      lclass = cc(lang).intern
+      if Treat::Resources::Delegates.constants.include?(lclass)
+        cat = group.to_s.split('::')[-2].intern
+        lclass = Treat::Resources::Delegates.
+        const_get(lclass).const_get(cat)
+        g = ucc(cl(group)).intern
+        if !lclass[g] || !lclass[g][0]
+          d = ucc(cl(group))
+          d.gsub!('_', ' ')
+          d = d[0..-2] if d[-1] == 's'
+          d = 'delegator to find ' + d
+          raise Treat::Exception, "No #{d}" +
+          " is available for the #{lang} language."
+        end
+        return lclass[g][0]
+      else
+        raise Treat::Exception,
+        "Language '#{lang}' is not supported (yet)."
+      end
+    end
+
+    # Add decorator methods to entities.
+    def decorate(group, m)
+      decorators = group.methods -
+      Object.methods -
+      [:type, :type=, :targets, :targets=,
+        :default, :default=, :add,
+      :has_target?, :list]
+      decorators.each do |decorator_m|
+        define_method(decorator_m) do |delegate=nil, options={}|
+          options[:decorator] = decorator_m
+          send(m, delegate, options)
+        end
+      end
+    end
+
+    # Raise an exception and suggest alternatives.
+    def delegate_not_found(klass, group)
+      "Algorithm '#{ucc(klass)}' couldn't be found in group #{group}." +
+      did_you_mean?(group.list.map { |c| ucc(c) }, ucc(klass))
+    end
+
+    # Add delegator group to all entities of a class.
+    def add_delegators(group)
+      # Define each method in group.
+      self.class_eval do
+        m = group.method
+        decorate(group, m)
+        define_method(m) do |delegate=nil, options={}|
+          decorator = options.delete(:decorator)
+          puts self.id if !@features
+          if !@features[m].nil?
+            @features[m]
+          else
+            if delegate.nil?
+              delegate = group.default.nil? ? 
+              self.class.get_language_delegate(language, group) :
+              group.default
+              raise "No default delegate for #{group}." if delegate == :none
+            end
+            if not group.list.include?(delegate)
+              raise Treat::Exception,
+              self.class.delegate_not_found(delegate, group)
+            else
+              delegate_klass = group.const_get(:"#{cc(delegate.to_s)}")
+              result = accept(group, delegate_klass, m, options)
+              if decorator
+                result = group.send(decorator, self, result)
+              end
+              if group.type == :annotator
+                f = decorator.nil? ? m : decorator
+                @features[f] = result
+              end
+              result
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/treat/detectors.rb

@@ -0,0 +1,28 @@
+module Treat
+  # Detectors detect a specific meta-information about
+  # an entity, such as encoding, format and language.
+  module Detectors
+    # Group for algorithms that detect encoding.
+    module Encoding 
+      extend Group
+      self.type = :annotator
+      self.targets = [:entity]
+      self.default = :native
+    end
+    # Group for algorithms that support format detection.
+    module Format
+      extend Group
+      self.type = :annotator
+      self.targets = [:entity]
+      self.default = :file
+    end
+    # Group for algorithms that do language detection.
+    module Language
+      extend Group
+      self.type = :annotator
+      self.targets = [:entity]
+      self.default = :what_language
+    end
+    extend Treat::Category
+  end
+end

data/lib/treat/detectors/encoding/native.rb

@@ -0,0 +1,12 @@
+module Treat
+  module Detectors
+    module Encoding
+      class Native
+        def self.encoding(entity, options={})
+          entity.value.encoding.name.
+          gsub('-', '_').downcase.intern
+        end
+      end
+    end
+  end
+end

data/lib/treat/detectors/encoding/r_chardet19.rb

@@ -0,0 +1,24 @@
+module Treat
+  module Detectors
+    module Encoding
+      # Require the 'rchardet19' gem.
+      silently { require 'rchardet19' }
+      # A wrapper for the 'rchardet19' gem, which
+      # detects the encoding of a file.
+      class RChardet19
+        # Returns an Encoding object representing
+        # the encoding of the supplied entity's 
+        # text value.
+        #
+        # Options: none.
+        def self.encoding(entity, options={})
+          r = CharDet.detect(entity.to_s)
+          Treat::Feature.new({
+            r.encoding.
+            gsub('-', '_').intern => 
+            r.confidence}).best
+        end
+      end
+    end
+  end
+end

data/lib/treat/detectors/format/file.rb

@@ -0,0 +1,36 @@
+module Treat
+  module Detectors
+    module Format
+      # A wrapper for the *NIX 'file' command,
+      # witch uses etc/magic to detect the format
+      # of a file.
+      class File
+        # Returns an identifier representing
+        # the format of a file using the *NIX
+        # 'file' command.
+        # 
+        # Options: none.
+        def self.format(entity, options = {})
+          format = nil
+          create_temp_file(:txt, entity.to_s) do |tmp|
+            format = `file #{tmp}` 
+          end
+          if format.scan('text')
+            :txt
+          elsif format.scan('XML')
+            :xml
+          elsif format.scan('HTML')
+            :html
+          elsif format.scan('image')
+            :image
+          elsif format.scan('PDF')
+            :pdf
+          else
+            raise Treat::Exception,
+            "Unsupported text format #{format}."
+          end
+        end
+      end
+    end
+  end
+end