treat 1.2.0 → 2.0.0rc1

data/LICENSE

@@ -1,4 +1,4 @@
-Treat - Text Retrieval, Extraction and Annotation Toolkit, v. 1.1.2
+Treat - Text Retrieval, Extraction and Annotation Toolkit, v. 2.0.0rc1
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -15,7 +15,7 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2011-12.
 
-Non-trivial amount of code has been incorporated and modified from other libraries:
+A non-trivial amount of code has been incorporated and modified from other libraries:
 
 - formatters/readers/odt.rb - Mark Watson (GPL license)
 - processors/tokenizers/tactful.rb - Matthew Bunday (GPL license)

data/README.md

@@ -1,33 +1,24 @@
 [![Build Status](https://secure.travis-ci.org/louismullie/treat.png)](http://travis-ci.org/#!/louismullie/treat)
-[![Dependency Status](https://gemnasium.com/louismullie/treat.png)](https://gemnasium.com/louismullie/treat)
- 
-Treat is a framework for natural language processing and computational linguistics in Ruby. It provides a common API for a number of gems and external libraries for document retrieval, parsing, annotation, and information extraction.
+[![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/louismullie/treat)
 
-**Current features**
+Treat is a toolkit for natural language processing and computational linguistics in Ruby. The Treat project aims to build a language- and algorithm- agnostic NLP framework for Ruby with support for tasks such as document retrieval, text chunking, segmentation and tokenization, natural language parsing, part-of-speech tagging, keyword extraction and named entity recognition. Learn more by taking a [quick tour](https://github.com/louismullie/treat/wiki/Quick-Tour) or by reading the [manual](https://github.com/louismullie/treat/wiki/Manual).
+
+**Features**
 
 * Text extractors for PDF, HTML, XML, Word, AbiWord, OpenOffice and image formats (Ocropus).
-* Text retrieval with indexation and full-text search (Ferret).
-* Text chunkers, sentence segmenters, tokenizers, and parsers for several languages (Stanford & Enju).
-* Word inflectors, including stemmers, conjugators, declensors, and number inflection.
-* Lexical resources (WordNet interface, several POS taggers for English, Stanford taggers for several languages).
+* Text chunkers, sentence segmenters, tokenizers, and parsers (Stanford & Enju).
+* Lexical resources (WordNet interface, several POS taggers for English).
 * Language, date/time, topic words (LDA) and keyword (TF*IDF) extraction.
-* Serialization of annotated entities to YAML, XML formats or to MongoDB.
+* Word inflectors, including stemmers, conjugators, declensors, and number inflection.
+* Serialization of annotated entities to YAML, XML or to MongoDB.
 * Visualization in ASCII tree, directed graph (DOT) and tag-bracketed (standoff) formats.
 * Linguistic resources, including language detection and tag alignments for several treebanks.
-* Decision tree and multilayer perceptron classification (liblinear coming soon!)
-
-<br>
+* Machine learning (decision tree, multilayer perceptron, LIBLINEAR, LIBSVM).
+* Text retrieval with indexation and full-text search (Ferret).
 
-**Resources**
+**Contributing**
 
-* Read the [latest documentation](http://rubydoc.info/github/louismullie/treat/frames).
-* See how to [install Treat](https://github.com/louismullie/treat/wiki/Installation).
-* Learn how to [use Treat](https://github.com/louismullie/treat/wiki/Manual).
-* Help out by [contributing to the project](https://github.com/louismullie/treat/wiki/Contributing).
-* View a list of [papers](https://github.com/louismullie/treat/wiki/Papers) about tools included in this toolkit.
-* Open an [issue](https://github.com/louismullie/treat/issues).
- 
-<br>
+I am actively seeking developers that can help maintain and expand this project. You can find a list of ideas for contributing to the project [here](https://github.com/louismullie/treat/wiki/Contributing).
 
 **License**
 

data/lib/treat/autoload.rb

@@ -0,0 +1,44 @@
+# Basic mixin for all the main modules;
+# takes care of requiring the right files 
+# in the right order for each one.
+# 
+# If a module's folder (e.g. /entities) 
+# contains a file with a corresponding
+# singular name (e.g. /entity), that 
+# base class is required first. Then, 
+# all the files that are found directly 
+# under that folder are required (but
+# not those found in sub-folders).
+module Treat::Autoload
+  
+  # Loads all the files for the base
+  # module in the appropriate order.
+  def self.included(base)
+    m = self.get_module_name(base)
+    d = self.get_module_path(m)
+    n = self.singularize(m) + '.rb'
+    f, p = File.join(d, n), "#{d}/*.rb"
+    require f if File.readable?(f)
+    Dir.glob(p).each { |f| require f }
+  end
+
+  # Returns the path to a module's dir.
+  def self.get_module_path(name)
+    file = File.expand_path(__FILE__)
+    dirs = File.dirname(file).split('/')
+    File.join(*dirs[0..-1], name)
+  end
+  
+  # Return the downcased form of the
+  # module's last name (e.g. "entities").
+  def self.get_module_name(mod)
+    mod.to_s.split('::')[-1].downcase
+  end
+    
+  # Helper method to singularize words.
+  def self.singularize(w)
+    if w[-3..-1] == 'ies'; w[0..-4] +  'y'
+    else; (w[-1] == 's' ? w[0..-2] : w); end
+  end
+  
+end

data/lib/treat/config/config.rb

@@ -0,0 +1,38 @@
+# This module uses structs to represent the 
+# configuration options that are stored in 
+# the /config folder.
+module Treat::Config
+
+  # Require configurable mix in.
+  require_relative 'importable'
+  
+  # Make all configuration importable.
+  extend Treat::Config::Importable
+  
+  # Core configuration options for entities.
+  class Treat::Config::Entities; end
+  
+  # Configuration for paths to models, binaries,
+  # temporary storage and file downloads.
+  class Treat::Config::Paths; end
+  
+  # Configuration for all Treat workers.
+  class Treat::Config::Workers; end
+
+  # Helpful linguistic options.
+  class Treat::Config::Linguistics; end
+
+  # Supported workers for each language.
+  class Treat::Config::Languages; end
+
+  # Configuration options for external libraries.
+  class Treat::Config::Libraries; end
+  
+  # Configuration options for database 
+  # connectivity (host, port, etc.)
+  class Treat::Config::Databases; end
+
+  # Configuration options for Treat core.
+  class Treat::Config::Core; end
+
+end

data/lib/treat/config/configurable.rb

@@ -0,0 +1,51 @@
+# Provide default functionality to load configuration
+# options from flat files into their respective modules.
+module Treat::Config::Configurable
+  
+  # When extended, add the .config property to
+  # the class that is being operated on.
+  def self.extended(base)
+    class << base; attr_accessor :config; end
+    base.class_eval { self.config = {} }
+  end
+  
+  # Provide base functionality to configure 
+  # all modules. The behaviour is as follows:
+  # 
+  # 1 - Check if a file named data/$CLASS$.rb
+  # exists; if so, load that file as the base 
+  # configuration, i.e. "Treat.$CLASS$"; e.g. 
+  # "Treat.core"
+  # 
+  # 2 - Check if a folder named data/$CLASS$
+  # exists; if so, load each file in that folder
+  # as a suboption of the main configuration,
+  # i.e. "Treat.$CLASS$.$FILE$"; e.g. "Treat.workers"
+  # 
+  # (where $CLASS$ is the lowercase name of 
+  # the concrete class being extended by this.)
+  def configure!
+    path = File.dirname(File.expand_path(         # FIXME
+    __FILE__)).split('/')[0..-4].join('/') + '/'
+    main_dir = path + 'lib/treat/config/data/'
+    mod_name = self.name.split('::')[-1].downcase
+    conf_dir = main_dir + mod_name
+    base_file = main_dir + mod_name + '.rb'
+    if File.readable?(base_file)
+      self.config = eval(File.read(base_file))
+    elsif FileTest.directory?(conf_dir)
+      self.config = self.from_dir(conf_dir)
+    else; raise Treat::Exception,
+      "No config file found for #{mod_name}."
+    end
+  end
+  
+  # * Helper methods for configuraton * #
+  def from_dir(conf_dir)
+    Hash[Dir[conf_dir + '/*'].map do |path|
+      name = File.basename(path, '.*').intern
+      [name, eval(File.read(path))]
+    end]
+  end
+  
+end

data/lib/treat/config/data/config.rb

@@ -0,0 +1,50 @@
+{acronyms: 
+  ['xml', 'html', 'txt', 'odt',
+  'abw', 'doc', 'yaml', 'uea',
+  'lda', 'pdf', 'ptb', 'dot',
+  'ai', 'id3', 'svo', 'mlp',
+  'svm', 'srx'],
+  
+encodings: 
+  {language_to_code: {
+    arabic: 'UTF-8',
+    chinese: 'GB18030',
+    english: 'UTF-8',
+    french: 'UTF-8',
+    german: 'UTF-8',
+    hebrew: 'UTF-8'
+}},
+
+entities: 
+    {list: 
+      [:entity, :unknown, :email, 
+       :url, :symbol, :sentence, 
+       :punctuation, :number, 
+       :enclitic, :word, :token, 
+       :fragment, :phrase, :paragraph, 
+       :title, :zone, :list, :block, 
+       :page, :section, :collection, 
+       :document],
+    order: 
+      [:token, :fragment, :phrase, 
+       :sentence, :zone, :section, 
+       :document, :collection]},
+    language: {
+      default: :english, 
+      detect: false, 
+      detect_at: :document
+    },
+    paths: {
+      description: {
+        tmp: 'temporary files',
+        lib: 'class and module definitions',
+        bin: 'binary files',
+        files: 'user-saved files',
+        models: 'model files',
+        spec: 'spec test files'
+      }
+    },
+    
+syntax: { sweetened: false },
+
+verbosity: { debug: false, silence: true}}

data/lib/treat/config/data/core.rb

@@ -0,0 +1,52 @@
+{
+  acronyms: 
+    ['xml', 'html', 'txt', 'odt',
+    'abw', 'doc', 'yaml', 'uea',
+    'lda', 'pdf', 'ptb', 'dot',
+    'ai', 'id3', 'svo', 'mlp',
+    'svm', 'srx'],
+  
+  encodings: 
+    {language_to_code: {
+      arabic: 'UTF-8',
+      chinese: 'GB18030',
+      english: 'UTF-8',
+      french: 'ISO_8859-1',
+      ferman: 'ISO_8859-1',
+      hebrew: 'UTF-8'
+  }},
+
+  entities: 
+      {list: 
+        [:entity, :unknown, :email, 
+         :url, :symbol, :sentence, 
+         :punctuation, :number, 
+         :enclitic, :word, :token, :group,
+         :fragment, :phrase, :paragraph, 
+         :title, :zone, :list, :block, 
+         :page, :section, :collection, 
+         :document],
+      order: 
+        [:token, :fragment, :group, 
+         :sentence, :zone, :section, 
+         :document, :collection]},
+      language: {
+        default: :english, 
+        detect: false, 
+        detect_at: :document
+      },
+      paths: {
+        description: {
+          tmp: 'temporary files',
+          lib: 'class and module definitions',
+          bin: 'binary files',
+          files: 'user-saved files',
+          models: 'model files',
+          spec: 'spec test files'
+        }
+      },
+    
+  syntax: { sweetened: false },
+
+  verbosity: { debug: false, silence: true}
+}

data/lib/treat/config/data/databases.rb

@@ -0,0 +1,10 @@
+{
+  default: {
+    adapter: :mongo
+  },
+  mongo: {
+    host: 'localhost', 
+    port: '27017', 
+    db: nil
+  }
+}

data/lib/treat/config/data/entities.rb

@@ -0,0 +1,15 @@
+{
+  list:
+    [:entity, :unknown, :email,
+      :url, :symbol, :sentence,
+      :punctuation, :number,
+      :enclitic, :word, :token,
+      :fragment, :phrase, :paragraph,
+      :title, :zone, :list, :block,
+      :page, :section, :collection,
+    :document],
+  order:
+    [:token, :fragment, :phrase,
+      :sentence, :zone, :section,
+    :document, :collection]
+}

data/lib/treat/config/data/languages/agnostic.rb

@@ -0,0 +1,31 @@
+{
+  dependencies: [
+    'nokogiri', 'ferret',
+    'bson_ext', 'mongo', 'lda-ruby',
+    'stanford-core-nlp', 'linguistics',
+    'ruby-readability', 'whatlanguage',
+    'chronic', 'nickel', 'decisiontree',
+    'rb-libsvm', 'ruby-fann', 'zip',
+    'tf-idf-similarity', 'narray'
+  ],
+  workers: {
+    learners: {
+      classifiers: [:id3, :linear, :mlp, :svm]
+    },
+    extractors: {
+      keywords: [:tf_idf],
+      language: [:what_language],
+      topic_words: [:lda],
+      tf_idf: [:native]
+    },
+    formatters: {
+      serializers: [:xml, :yaml, :mongo],
+      unserializers: [:xml, :yaml, :mongo],
+      visualizers: [:dot, :standoff, :tree] 
+    },
+    retrievers: {
+      searchers: [:ferret],
+      indexers: [:ferret]
+    }
+  }
+}

data/lib/treat/config/{languages → data/languages}/dutch.rb

@@ -6,7 +6,7 @@
   workers: {
     processors: {
       segmenters: [:punkt],
-      tokenizers: [:tactful]
+      tokenizers: []
     }
   }
 }

data/lib/treat/config/data/languages/english.rb

@@ -0,0 +1,95 @@
+{
+  dependencies: [
+    'rbtagger', 
+    'ruby-stemmer', 
+    'punkt-segmenter', 
+    'tactful_tokenizer',
+    'nickel', 
+    'rwordnet', 
+    'uea-stemmer', 
+    'engtagger', 
+    'activesupport',
+    'srx-english',
+    'scalpel'
+  ],
+  workers: {
+    extractors: {
+      time: [:chronic, :ruby, :nickel],
+      topics: [:reuters],
+      name_tag: [:stanford]
+    },
+    inflectors: {
+      conjugators: [:linguistics],
+      declensors: [:english, :linguistics],
+      stemmers: [:porter, :porter_c, :uea],
+      ordinalizers:  [:linguistics],
+      cardinalizers:  [:linguistics]
+    },
+    lexicalizers: {
+      taggers: [:lingua, :brill, :stanford],
+      sensers: [:wordnet],
+      categorizers: [:from_tag]
+    },
+    processors: {
+      parsers: [:stanford],
+      segmenters: [:scalpel, :srx, :tactful, :punkt, :stanford],
+      tokenizers: [:ptb, :stanford, :punkt]
+    }
+  },
+  stop_words:
+    [
+      "about",
+      "also",
+      "are",
+      "away",
+      "because",
+      "been",
+      "beside",
+      "besides",
+      "between",
+      "but",
+      "cannot",
+      "could",
+      "did",
+      "etc",
+      "even",
+      "ever",
+      "every",
+      "for",
+      "had",
+      "have",
+      "how",
+      "into",
+      "isn",
+      "maybe",
+      "non",
+      "nor",
+      "now",
+      "should",
+      "such",
+      "than",
+      "that",
+      "then",
+      "these",
+      "this",
+      "those",
+      "though",
+      "too",
+      "was",
+      "wasn",
+      "were",
+      "what",
+      "when",
+      "where",
+      "which",
+      "while",
+      "who",
+      "whom",
+      "whose",
+      "will",
+      "with",
+      "would",
+      "wouldn",
+      "yes"
+    ]
+}

data/lib/treat/config/data/languages/french.rb

@@ -0,0 +1,148 @@
+{
+  dependencies: [
+    'punkt-segmenter', 
+    'tactful_tokenizer', 
+    'stanford-core-nlp'
+  ],
+  workers: {
+    processors: {
+      segmenters: [:scalpel],
+      tokenizers: [:ptb,:stanford],
+      parsers: [:stanford]
+    },
+    lexicalizers: {
+      taggers: [:stanford],
+      categorizers: [:from_tag]
+    }
+  },
+  stop_words:
+    [
+      "ailleurs",
+      "ainsi",
+      "alors",
+      "aucun",
+      "aucune",
+      "auquel",
+      "aurai",
+      "auras",
+      "aurez",
+      "aurons",
+      "auront",
+      "aussi",
+      "autre",
+      "autres",
+      "aux",
+      "auxquelles",
+      "auxquels",
+      "avaient",
+      "avais",
+      "avait",
+      "avec",
+      "avez",
+      "aviez",
+      "avoir",
+      "avons",
+      "celui",
+      "cependant",
+      "certaine",
+      "certaines",
+      "certains",
+      "ces",
+      "cet",
+      "cette",
+      "ceux",
+      "chacun",
+      "chacune",
+      "chaque",
+      "comme",
+      "constamment",
+      "davantage",
+      "depuis",
+      "des",
+      "desquelles",
+      "desquels",
+      "dessous",
+      "dessus",
+      "donc",
+      "dont",
+      "duquel",
+      "egalement",
+      "elles",
+      "encore",
+      "enfin",
+      "ensuite",
+      "etaient",
+      "etais",
+      "etait",
+      "etes",
+      "etiez",
+      "etions",
+      "etre",
+      "eux",
+      "guere",
+      "ici",
+      "ils",
+      "jamais",
+      "jusqu",
+      "laquelle",
+      "legerement",
+      "lequel",
+      "les",
+      "lesquelles",
+      "lesquels",
+      "leur",
+      "leurs",
+      "lors",
+      "lui",
+      "maintenant",
+      "mais",
+      "malgre",
+      "moi",
+      "moins",
+      "notamment",
+      "parce",
+      "plupart",
+      "pourtant",
+      "presentement",
+      "presque",
+      "puis",
+      "puisque",
+      "quand",
+      "quant",
+      "que",
+      "quel",
+      "quelqu",
+      "quelque",
+      "quelques",
+      "qui",
+      "quoi",
+      "quoique",
+      "rien",
+      "selon",
+      "serai",
+      "seras",
+      "serez",
+      "serons",
+      "seront",
+      "soient",
+      "soit",
+      "sommes",
+      "sont",
+      "sous",
+      "suis",
+      "telle",
+      "telles",
+      "tels",
+      "toi",
+      "toujours",
+      "tout",
+      "toutes",
+      "tres",
+      "trop",
+      "une",
+      "vos",
+      "votre",
+      "vous"
+    ]
+  
+}