treat 0.2.5 → 1.0.0

data/LICENSE

@@ -1,4 +1,4 @@
-Treat - Text Retrieval and Annotation Toolkit
+Treat - Text Retrieval, Extraction and Annotation Toolkit
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -11,9 +11,9 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 
 You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
+along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2012.
+Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2011-12.
 
 Non-trivial amount of code has been incorporated and modified from other libraries:
 

data/README.md

@@ -0,0 +1,33 @@
+![Build Status](https://secure.travis-ci.org/louismullie/treat.png) ![Dependency Status](https://gemnasium.com/louismullie/treat.png)
+
+Treat is a toolkit for natural language processing and computational linguistics in Ruby. It provides a common API for a number of gems and external libraries for document retrieval, parsing, annotation, and information extraction.
+
+**Current features**
+
+* Text extractors for PDF, HTML, XML, Word, AbiWord, OpenOffice and image formats (Ocropus)
+* Text retrieval with indexation and full-text search (Ferret)
+* Text chunkers, sentence segmenters, tokenizers, and parsers for several languages (Stanford & Enju)
+* Word inflectors, including stemmers, conjugators, declensors, and number inflection
+* Lexical resources (WordNet interface, several POS taggers for English, Stanford taggers for several languages)
+* Language, date/time, topic words (LDA) and keyword (TF*IDF) extraction.
+* Simple text statistics (frequency, TF*IDF)
+* Serialization of annotated entities to YAML or XML format
+* Visualization in ASCII tree, directed graph (DOT) and tag-bracketed (standoff) formats
+* Linguistic resources, including full ISO-639-1 and ISO-639-2 support, and tag alignments for several treebanks
+
+<br>
+
+**Resources**
+
+* Read the [latest documentation](http://rubydoc.info/github/louismullie/treat/master/frames).
+* See how to [install Treat](https://github.com/louismullie/treat/wiki/Installing-Treat).
+* Learn how to [use Treat](https://github.com/louismullie/treat/wiki/Using-Treat).
+* Help out by [contributing to the project](https://github.com/louismullie/treat/wiki/Contributing-to-Treat).
+* View a list of [papers](https://github.com/louismullie/treat/wiki/Papers) about tools included in this toolkit.
+* Open an [issue](https://github.com/louismullie/treat/issues).
+ 
+<br>
+
+**License**
+
+This software is released under the [GPL License](https://github.com/louismullie/treat/wiki/License-Information) and includes software released under the GPL, Ruby, Apache 2.0 and MIT licenses.

data/files/INFO

@@ -0,0 +1 @@
+This is a folder containing the files downloaded by Treat.

data/lib/treat.rb

@@ -1,127 +1,62 @@
-# Main namespace for Treat modules.
-#
-# === Entities
-#
-# Entities are Tree structures that represent textual entities
-# (from a collection of texts down to an individual word), with
-# a value, features, children and dependencies linking it to other
-# textual entities.
-#
-# Here are some example of how to create entities:
-#
-#     Treat.sweeten!
-#
-#     c = Collection 'folder_with_documents'
-#     d = Document 'filename.txt'
-#     p = Paragraph 'A short story. The end.'
-#     s = Sentence 'That is not a sentence.'
-#     w = Word 'fox'
-#
-# Here is a list of entities and their description:
-#
-# - A Collection represents a folder with different textual documents.
-# - A Document represents a file with a textual content.
-# - A Zone can be a Section, Title, a Paragraph or a List and represents an intra-section division of content.
-# - A Sentence represents just that.
-# - A Phrase is a group of words; a Sentence is a Phrase with a sentence ender (.!?)
-# - A Token can be a Word, a Number, a Punctuation or a Symbol (non-punctuation, non-alphanumeric character).
-#
-# === Functions
-#
-# A worker class is defined for each implemented algorithm performing a given
-# task. These classes are clustered into workers performing the same given task
-# differently (Group), and the groups are clustered into Categories
-# of groups of workers that perform related tasks.
-#
-# Here are the different Categories and their description:
-#
-# - Processors perform the building of tree of entities representing texts (chunking, segmenting, tokenizing, parsing).
-# - Lexicalizers give lexical information about words (synsets, semantic relationships, tag, word category).
-# - Extractors extract semantic information about an entity (topic, date, time, named entity).
-# - Inflectors allow to retrieve the different inflections of a word (declensors, conjugators, stemmers, lemmatizers).
-# - Formatters handle the conversion of entities to and from different formats (readers, serializers, unserializers, visualizers).
-# - Retrievers allow to index and search collections of documents.
-#
-# === Linguistic Resources
-#
-# The Languages module contains linguistic information about
-# languages (full ISO-639-1 and 2 language list, tag alignments
-# for three treebanks, word categories, etc.)
-#
-# === Exception Class.
-#
-# Treat::Exception defines a custom exception class for the Treat module.
-#
 module Treat
+  
+  # Require custom exception cass.
+  require 'treat/exception'
 
-  # Make sure that we are running on Ruby 1.9 or higher.
+  # Treat requires Ruby 1.9 or higher.
   if RUBY_VERSION <= '1.9'
-    raise 'Treat requires Ruby 1.9 or higher.'
+    raise Treat::Exception,
+    'Treat requires Ruby 1.9 or higher.'
   end
 
   # The current version of Treat.
-  VERSION = "0.2.5"
-
-  #$LOAD_PATH << '/ruby/gems/treat/lib/' # Remove for release
-
-  # Create class variables for the Treat module.
+  VERSION = "1.0.0"
+
+  # Add methods to handle syntactic sugar,
+  # language configuration options, and paths.
+  require 'treat/configurable'
+  extend Treat::Configurable
+
+  # The folders in the library and descriptions.
+  Paths = {
+    :tmp => 'temporary files',
+    :lib => 'class and module definitions',
+    :bin => 'binary files',
+    :files => 'user-saved files',
+    :data => 'data set files',
+    :models => 'model files',
+    :spec => 'spec test files'
+  }
+
+  # Add methods to provide access to common paths.
   class << self
-    # Boolean - output debug information.
-    attr_accessor :debug
-    # Symbol - default language to use when detect_language is false.
-    attr_accessor :default_language
-    # Symbol - default encoding to use.
-    attr_accessor :default_encoding
-    # Boolean - detect language or use default?
-    attr_accessor :detect_language
-    # Symbol - the ideal entity level to detect language at
-    # (e.g., :entity, :sentence, :zone, :section, :document)
-    attr_accessor :language_detection_level
-    # String - folder of this file.
-    attr_accessor :lib
-    # String - folder for tests.
-    attr_accessor :test
+    Paths.each do |path, _|
+      define_method(path) do
+        (File.dirname(__FILE__).
+        split('/')[0..-2].join('/') + 
+        '/' + path.to_s + '/').gsub(
+        'lib/../', '')
+      end
+    end
   end
 
-  # Turn off debug by default.
-  self.debug = false
-  # Set the default language to english.
-  self.default_language = :eng
-  # Set the default encoding to utf-8.
-  self.default_encoding = :utf_8
-  # Turn language detection off by default.
-  self.detect_language = false
-  # Detect the language once per text by default.
-  self.language_detection_level = :zone
-  # Set the lib path to that of this file.
-  self.lib = File.dirname(__FILE__)
-  # Set the paths to the test folder.
-  self.test = self.lib + '/../test'
-
-  # Require inline C
-  # require 'inline'
-
-  # Require modified core classes.
   require 'treat/object'
   require 'treat/kernel'
-  # Require all files for the Treat library.
-  require 'treat/exception'
+  require 'treat/downloader'
   require 'treat/languages'
+  require 'treat/linguistics'
   require 'treat/entities'
   require 'treat/categories'
+  require 'treat/data_set'
   require 'treat/proxies'
-  require 'treat/sugar'
-
-  # Make sugar available when needed.
-  extend Treat::Sugar
 
   # Install packages for a given language.
   def self.install(language = :english)
-    require 'treat/install'
+    require 'treat/installer'
     Treat::Installer.install(language)
   end
 
-  # Turn on detect language.
-  def self.detect!; self.detect_language = true; end
-
+  # Enable syntactic sugar by default.
+  Treat.sweeten!
+  
 end

data/lib/treat/ai.rb

@@ -0,0 +1,12 @@
+module Treat::AI
+  
+  module Classifiers
+    extend Treat::Groupable
+    self.type = :annotator
+    self.targets = [:entity]
+    self.default = :id3
+  end
+  
+  extend Treat::Categorizable
+  
+end

data/lib/treat/ai/classifiers/id3.rb

@@ -0,0 +1,27 @@
+class Treat::AI::Classifiers::ID3
+  
+  require 'decisiontree'
+  
+  @@classifiers = {}
+  
+  def self.classify(entity, options = {})
+  
+    set = options[:training]
+    cl = set.classification
+    
+    if !@@classifiers[cl]
+      dec_tree = DecisionTree::ID3Tree.new(
+      set.labels, set.items, 
+      cl.default, :continuous)
+      dec_tree.train
+    else
+      dec_tree = @@classifiers[cl]
+    end
+    
+    dec_tree.predict(
+      cl.export_item(entity, false)
+    )[0]
+    
+  end
+  
+end

data/lib/treat/categories.rb

@@ -1,43 +1,90 @@
-module Treat
-  # This module keeps track of all categories that
-  # exist and the methods they implement.
-  module Categories
-    class << self
-      # A list of all categories.
-      attr_accessor :list
+# This module keeps track of all the Treat::Categorizable
+# modules that exist and the methods they define.
+#
+#
+# - Processors perform the building of tree of
+#   entities representing texts (chunking,
+#   segmenting, tokenizing, parsing).
+# - Lexicalizers give lexical information about
+#   words (synsets, semantic relationships,
+#   tag, word category).
+# - Extractors extract semantic information about
+#   an entity (language, topic, date, time, named
+#   entity, coreferences).
+# - Inflectors allow to retrieve the different
+#   inflections of a word (declensors, conjugators,
+#   stemmers, lemmatizers).
+# - Formatters handle the conversion of entities to
+#   and from different formats(readers, serializers,
+#   unserializers, visualizers).
+# - Retrievers allow to index and search collections
+#   of documents.
+module Treat::Categories
+
+  class << self
+    # A list of all categories.
+    attr_accessor :list
+  end
+
+  # Array - list of all categories.
+  self.list = []
+  # A lookup table for entity types.
+  @@lookup = {}
+
+  # Require all categories.
+  require 'treat/categorizable'
+  require 'treat/formatters'
+  require 'treat/processors'
+  require 'treat/lexicalizers'
+  require 'treat/inflectors'
+  require 'treat/extractors'
+  require 'treat/retrievers'
+  require 'treat/ai'
+  
+  # Create the lookup table.
+  self.list.each do |category|
+    category.groups.each do |group|
+      group = category.const_get(group)
+      @@lookup[group.method] = group
+      group.presets.each do |x,y|
+        @@lookup[x] = group
+      end if group.presets
     end
-    # Array - list of all categories.
-    self.list = []
-    @@lookup = nil
-    # Find the class of a group given its method.
-    def self.lookup(method)
-      return @@lookup[method] if @@lookup
-      @@lookup = {}
+  end
+
+  # Find the class of a group given its method.
+  def self.lookup(method)
+    @@lookup[method]
+  end
+
+  # Fix -- This must be moved urgently.
+  Treat::Entities::Entity.class_eval do
+
+    alias :true_language :language
+    
+    def language(extractor = nil, options = {})
       
-      self.list.each do |category|
-        category.groups.each do |group|
-          group = category.const_get(group)
-          @@lookup[group.method] = group
-          methods = group.presets.merge(
-            group.preprocessors.merge(
-              group.postprocessors
-            )
-          )
-          methods.each do |x,y|
-            @@lookup[x] = group
-          end
+      if is_a?(Treat::Entities::Symbol) ||
+        is_a?(Treat::Entities::Number)
+        return Treat.default_language
+      end
+      
+      if !Treat.detect_language
+        return Treat.default_language
+      else
+        dlvl = Treat.language_detection_level
+        if (Treat::Entities.rank(type) <
+          Treat::Entities.rank(dlvl)) &&
+          has_parent?
+          anc = ancestor_with_type(dlvl)
+          return anc.language if anc
         end
       end
       
-      @@lookup[method]
+      true_language(extractor, options)
+      
     end
-    # Require all categories.
-    require 'treat/category'
-    require 'treat/formatters'
-    require 'treat/processors'
-    require 'treat/lexicalizers'
-    require 'treat/inflectors'
-    require 'treat/extractors'
-    require 'treat/retrievers'
+
   end
+
 end

data/lib/treat/categorizable.rb

@@ -0,0 +1,44 @@
+# A categorizable module brings together groups 
+# of algorithms that perform similar functions.
+module Treat::Categorizable
+  
+  # The contents of each categorizable 
+  # module are groupable.
+  require 'treat/groupable'
+  
+  # Add workers to the Entities based on the
+  # configuration for a given category.
+  def self.extended(category)
+    Treat::Categories.list << category
+    category.module_eval do
+      groups.each do |group|
+        group = const_get(group)
+        group.targets.each do |entity_type|
+          entity = Treat::Entities.
+          const_get(cc(entity_type))
+          entity.class_eval do
+            add_workers group
+          end
+        end
+      end
+    end
+  end
+  
+  # Get the list of groups defined
+  # under this module.
+  @@groups = self.constants
+  
+  # Populate a list of methods.
+  @@methods = []
+  @@groups.each do |group|
+    @@methods << const_get(group).method
+  end
+  
+  # Provide a list of methods implemented in
+  # the groups contained within this category.
+  def methods; @@methods; end
+  
+  # Provides a list of groups within this category.
+  def groups; self.constants; end
+  
+end

data/lib/treat/classification.rb

@@ -0,0 +1,61 @@
+class Treat::Classification
+
+  attr_reader :types
+  attr_reader :features
+  attr_reader :question
+  attr_reader :labels
+  attr_reader :default
+
+  def initialize(type_or_types, feature_or_features, question, default = false)
+
+    @types, @features,
+    @question, @default = 
+    [*type_or_types],
+    [*feature_or_features], 
+    question, default
+    
+    @labels = []
+
+    @features.each do |cmd|
+      if cmd.is_a?(Array)
+        @labels << cmd[0]
+      else
+        @labels << cmd
+      end
+    end
+
+  end
+
+  def export_item(e, include_question = true)
+
+    line = []
+
+    @features.each do |cmd|
+      begin
+        if cmd.is_a?(Array)
+          line << cmd[1].call(e)
+        else
+          line << e.send(cmd)
+        end
+      rescue Treat::Exception
+        dflt = (
+        (cmd.is_a?(Array) && cmd[2]) ?
+        cmd[2] : nil
+        )
+        line << dflt
+      end
+    end
+
+    begin
+      if include_question
+        line << e.send(@question)
+      end
+    rescue Treat::Exception
+      line << @default
+    end
+    line[-1] = '' if line[-1].nil?
+    line
+
+  end
+
+end