open-nlp 0.0.1

data/LICENSE

@@ -0,0 +1,18 @@
+Ruby bindings for the OpenNLP tools.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+This license also applies to the included Stanford CoreNLP files.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2012.

data/README.md

@@ -0,0 +1,112 @@
+**Warning**
+
+This is an alpha release. Expect things to break and/or change in the near future! Also, keep the following in mind:
+
+- Currently, this gem is only tested on JRuby, but support for MRI through Rjb is coming very soon.
+- Currently, the parser and chunker classes are not working.
+
+**About**
+
+This library provides high-level Ruby bindings to the Open NLP package, a Java machine learning toolkit for natural language processing (NLP). 
+
+This gem only provides a thin wrapper over the OpenNLP API. If you are looking for a Ruby natural language processing framework, have a look at [Treat](https://github.com/louismullie/treat).
+
+**Installing**
+
+First, install the gem: `gem install open-nlp`. Then, individually download the appropriate models from the [open-nlp website](http://opennlp.sourceforge.net/models-1.5/) or just get [all english language models](louismullie.com/treat/open-nlp-english.zip) in one package (80 MB).
+
+Place the contents of the extracted archive inside the /bin/ folder of the stanford-core-nlp gem (e.g. [...]/gems/open-nlp-0.x.x/bin/).
+
+**Configuration**
+
+After installing and requiring the gem (`require 'open-nlp'`), you may want to set some optional configuration options. Here are some examples:
+
+```ruby
+# Set an alternative path to look for the JAR files
+# Default is gem's bin folder.
+OpenNLP.jar_path = '/path_to_jars/'
+
+# Set an alternative path to look for the model files
+# Default is gem's bin folder.
+OpenNLP.model_path = '/path_to_models/'
+
+# Pass some alternative arguments to the Java VM.
+# Default is ['-Xms512M', '-Xmx1024M'].
+OpenNLP.jvm_args = ['-option1', '-option2']
+
+# Redirect VM output to log.txt
+OpenNLP.log_file = 'log.txt'
+
+# WARNING: Not implemented yet.
+
+# Use the model files for a different language than English.
+# OpenNLP.use(:french) # or :german
+# 
+# Change a specific model file.
+# StanfordCoreNLP.set_model('pos.model', 'english-left3words-distsim.tagger')
+```
+
+**Using the gem**
+
+```ruby
+text = 'Angela Merkel met Nicolas Sarkozy on January 25th in ' +
+       'Berlin to discuss a new $25 billion austerity package.' +
+       'Sarkozy looked pleased, but Merkel was dismayed.'
+
+tokenizer   = OpenNLP::TokenizerME.new
+segmenter   = OpenNLP::SentenceDetectorME.new
+tagger      = OpenNLP::POSTaggerME.new
+ner_models  = ['person', 'time', 'money']
+
+ner_finders = ner_models.map do |model|
+ OpenNLP::NameFinderME.new("en-ner-#{model}.bin")
+end
+
+sentences = segmenter.sent_detect(text)
+all_entities = []
+
+sentences.each do |sentence|
+
+ tokens = tokenizer.tokenize(sentence)
+ tags   = tagger.tag(tokens)
+ 
+ # Get a list of all tokens.
+ puts tokens.to_a.inspect
+ # Get the sentence's text.
+ puts sentence.to_s.inspect
+ # Get the sentence's tags.
+ puts tags.to_a.inspect
+
+ # Run three NER models and find entities.
+ ner_models.each_with_index do |model,i|
+   finder = ner_finders[i]
+   name_spans = finder.find(tokens)
+   name_spans.each do |name_span|
+     start = name_span.get_start
+     stop  = name_span.get_end-1
+     slice = tokens[start..stop].to_a
+     all_entities << [slice, model]
+   end
+ end
+
+end
+
+# Show all named entities.
+puts all_entities.inspect
+```
+
+**Loading specific classes**
+
+You may also want to load your own classes from the Stanford NLP to do more specific tasks. The gem provides an API to do this:
+
+```ruby
+# Default base class is opennlp.tools.
+OpenNLP.load_class('SomeClassName')  
+
+# Here, we specify another base class.
+OpenNLP.load_class('SomeOtherClass', 'opennlp.tools.namefind') 
+```
+
+**Contributing**
+
+Feel free to fork the project and send me a pull request!

data/lib/open-nlp.rb

@@ -0,0 +1,162 @@
+module OpenNLP
+  
+  # Library version.
+   VERSION = '0.0.1'
+
+   # Require configuration.
+   require 'open-nlp/config'
+  
+  # ############################ #
+  # BindIt Configuration Options #
+  # ############################ #
+  
+   require 'bind-it'
+   extend BindIt::Binding
+
+  # The path in which to look for JAR files, with
+  # a trailing slash (default is gem's bin folder).
+  self.jar_path = File.dirname(__FILE__) + '/../bin/'
+  
+  # Load the JVM with a minimum heap size of 512MB,
+  # and a maximum heap size of 1024MB.
+  self.jvm_args = ['-Xms512M', '-Xmx1024M']
+  
+  # Turn logging off by default.
+  self.log_file = nil
+  
+  # Default JARs to load.
+  self.default_jars = [
+    'jwnl-1.3.3.jar',
+    'opennlp-tools-1.5.2-incubating.jar',
+    'opennlp-maxent-3.0.2-incubating.jar',
+    'opennlp-uima-1.5.2-incubating.jar'
+  ]
+  
+  # Default namespace.
+  self.default_namespace = 'opennlp.tools'
+  
+  # Default classes.
+  self.default_classes = [
+    ['DocumentCategorizerME', 'opennlp.tools.doccat'],
+    ['ChunkerME', 'opennlp.tools.chunker'],
+    ['DictionaryDetokenizer', 'opennlp.tools.tokenize'],
+    ['NameFinderME', 'opennlp.tools.namefind'],
+    ['Parse', 'opennlp.tools.parser'],
+    ['ParserFactory', 'opennlp.tools.parser'],
+    ['POSTaggerME', 'opennlp.tools.postag'],
+    ['SentenceDetectorME', 'opennlp.tools.sentdetect'],
+    ['SimpleTokenizer', 'opennlp.tools.tokenize'],
+    ['TokenizerME', 'opennlp.tools.tokenize']
+  ]
+  
+  # Redefine the Bind-It class loader to redefine 
+  # a new constructor for classes that require a model.
+  def self.load_klass(klass, base, name = nil)
+    super(klass,base,name)
+    requires_model = OpenNLP::Config::RequiresModel
+    return unless requires_model.include?(klass)
+    new_class = Class.new(const_get(klass)) do
+      def initialize(file = nil, *args)
+        klass = OpenNLP.last_name(self.class)
+        if !file && !OpenNLP.has_default_model?(klass)
+          raise 'This class intentionally has no default ' +
+          'model. Please supply a file name as an argument ' +
+          'to the class constructor.'
+        else
+          model = OpenNLP.get_model(klass, file)
+          super(*([model] + args))
+        end
+      end
+    end
+    remove_const(klass)
+    const_set(klass, new_class)
+  end
+  
+  # Make the bindings.
+  self.bind
+  
+  # Load utility classes.
+  self.load_class('FileInputStream', 'java.io')
+  
+  # ############################ #
+  #   OpenNLP bindings proper    #
+  # ############################ #
+
+  class <<self
+    # A hash containing loaded models.
+    attr_accessor :models
+    # A hash containing the names of loaded models.
+    attr_accessor :model_files
+    # The folder in which to look for models.
+    attr_accessor :model_path
+    # Store the language currently being used.
+    attr_accessor :language
+  end
+  
+  # The loaded models.
+  self.models = {}
+  
+  # The names of loaded models.
+  self.model_files = {}
+  
+  # The path to the main folder containing the folders
+  # with the individual models inside. By default, this
+  # is the same as the JAR path.
+  self.model_path = self.jar_path
+  
+  # Default the language to English.
+  self.language = :english
+  
+  # Use a given language for default models.
+  def self.use(language)
+    self.language = language
+  end
+  
+  def self.set_model
+    # Implement
+  end
+
+  def self.has_default_model?(klass)
+    name = OpenNLP::Config::ClassToName[klass]
+    if !OpenNLP::Config::DefaultModels[name]
+      raise 'No default model files are available ' +
+      "for the class #{klass}. Please supply a model " +
+      'as an argument to the constructor.'
+    end
+    !OpenNLP::Config::DefaultModels[name].empty?
+  end
+  
+  def self.get_model(klass, file=nil)
+    name = OpenNLP::Config::ClassToName[klass]
+    if !self.language and !file
+      raise 'No model file was supplied to the ' +
+      'constructor. Please supply a model file ' +
+      'or call OpenNLP.use(:some_language), to ' +
+      'load the default models for a language.'
+    end
+    OpenNLP.load_model(name, file)
+    model = OpenNLP.models[name]
+  end
+
+  def self.load_model(name, file = nil)
+    if self.models[name] && file ==
+       self.model_files[name]
+       return self.models[name]
+    end
+    models = Config::DefaultModels[name]
+    file ||= models[self.language]
+    path = self.model_path + file
+    stream = FileInputStream.new(path)
+    klass = Config::NameToClass[name]
+    load_class(*klass)
+    klass = const_get(klass[0])
+    model = klass.new(stream)
+    self.model_files[name] = file
+    self.models[name] = model
+  end
+
+  def self.last_name(klass)
+    klass.to_s.split('::')[-1]
+  end
+  
+end

data/lib/open-nlp/config.rb

@@ -0,0 +1,55 @@
+module OpenNLP::Config
+  
+  NameToClass = {
+    categorizer: ['DoccatModel', 'opennlp.tools.doccat'],
+    chunker: ['ChunkerModel', 'opennlp.tools.chunker'],
+    detokenizer: ['DetokenizationDictionary', 'opennlp.tools.tokenize'],
+    name_finder: ['TokenNameFinderModel', 'opennlp.tools.namefind'],
+    parser: ['ParserModel', 'opennlp.tools.parser'],
+    pos_tagger: ['POSModel', 'opennlp.tools.postag'],
+    sentence_detector: ['SentenceModel', 'opennlp.tools.sentdetect'],
+    tokenizer: ['TokenizerModel', 'opennlp.tools.tokenize']
+  }
+  
+  ClassToName = {
+    'ChunkerME' => :chunker,
+    'DictionaryDetokenizer' => :detokenizer,
+    'DocumentCategorizerME' => :categorizer,
+    'NameFinderME' => :name_finder,
+    'POSTaggerME' => :pos_tagger,
+    'ParserME' => :parser,
+    'SentenceDetectorME' => :sentence_detector,
+    'TokenizerME' => :tokenizer,
+  }
+
+  DefaultModels = {
+    chunker: {
+      english: 'en-chunker.bin'
+    },
+    detokenizer: {
+      english: 'en-detokenizer.xml'
+    },
+    # Intentionally left empty.
+    name_finder: {},
+    parser: {
+      english: 'en-parser-chunking.bin'
+    },
+    pos_tagger: { 
+      english: 'en-pos-maxent.bin'
+    },
+    sentence_detector: {
+      english: 'en-sent.bin'
+    },
+    tokenizer: {
+      english: 'en-token.bin'
+    }
+  }
+
+  # Classes that require a model as first argument to constructor.
+  RequiresModel = [
+    'SentenceDetectorME', 'NameFinderME', 'DictionaryDetokenizer',
+    'TokenizerME', 'ChunkerME', 'POSTaggerME'
+  ]
+
+  
+end

data/spec/english_spec.rb

@@ -0,0 +1,103 @@
+# encoding: utf-8
+require_relative 'spec_helper'
+
+describe OpenNLP do
+
+  # Failing spec #1
+  context "the maximum entropy chunker is run after tokenization and POS tagging" do
+    it "should find the accurate chunks" do
+      
+      chunker   = OpenNLP::ChunkerME.new
+      tokenizer = OpenNLP::TokenizerME.new
+      tagger    = OpenNLP::POSTaggerME.new
+
+      sent   = "The death of the poet was kept from his poems."
+      tokens = tokenizer.tokenize(sent)
+      tags   = tagger.tag(tokens)
+      
+      chunks = chunker.chunk(tokens.to_java(:String), pos_tags.to_java(:String))
+      # cannot convert instance of class org.jruby.java.proxies.ArrayJavaProxy to class java.lang.String
+      
+      tokens.to_a.should eql %w[The death of the poet was kept from his poems .]
+      tags.should eql ['put tags here']
+      
+    end
+  end
+
+  # Failing spec #2
+  context "the maximum entropy parser is run after tokenization" do
+    it "parses the text accurately" do
+      sent      = "The death of the poet was kept from his poems."
+      tokenizer = OpenNLP::TokenizerME.new
+      p_model   = OpenNLP.load_model(:parser)
+      parser    = OpenNLP::ParserFactory.create(p_model)
+      tokens = tokenizer.tokenize(sent)
+      result = parser.parse(tokens.to_java(:String))
+      # cannot convert instance of class org.jruby.java.proxies.ArrayJavaProxy to class java.lang.String
+      # org/jruby/java/addons/KernelJavaAddons.java:70:in `to_java'
+      # /ruby/gems/open-nlp/spec/english_spec.rb in `(root)'
+      puts result.to_a.inspect
+    end
+  end
+
+  context "the SimpleTokenizer is run" do
+    it "tokenizes the text accurately" do
+      sent = "The death of the poet was kept from his poems."
+      tokenizer = OpenNLP::SimpleTokenizer.new
+      tokens = tokenizer.tokenize(sent).to_a
+      tokens.should eql %w[The death of the poet was kept from his poems .]
+    end
+  end
+
+  context "the maximum entropy sentence detector, tokenizer, POS tagger " +
+  "and NER finders are run with the default models for English" do
+
+    it "should accurately detect tokens, sentences and named entities" do
+
+      text = File.read('./spec/sample.txt').gsub!("\n", "")
+      
+      tokenizer   = OpenNLP::TokenizerME.new
+      segmenter   = OpenNLP::SentenceDetectorME.new
+      tagger      = OpenNLP::POSTaggerME.new
+      ner_models  = ['person', 'time', 'money']
+
+      ner_finders = ner_models.map do |model|
+        OpenNLP::NameFinderME.new("en-ner-#{model}.bin")
+      end
+
+      sentences = segmenter.sent_detect(text)
+      all_entities, all_tags, all_sentences, all_tokens = [], [], [], []
+
+      sentences.each do |sentence|
+
+        tokens = tokenizer.tokenize(sentence)
+        tags   = tagger.tag(tokens)
+        
+        ner_models.each_with_index do |model,i|
+          finder = ner_finders[i]
+          name_spans = finder.find(tokens)
+          name_spans.each do |name_span|
+            start = name_span.get_start
+            stop  = name_span.get_end-1
+            slice = tokens[start..stop].to_a
+            all_entities << [slice, model]
+          end
+        end
+
+        all_tokens << tokens.to_a
+        all_sentences << sentence
+        all_tags << tags.to_a
+      end
+
+      all_tokens.should eql [["To", "describe", "2009", "as", "a", "stellar", "year", "for", "Petrofac", "(", "LON:PFC)", "would", "be", "a", "huge", "understatement", "."], ["The", "group", "finished", "the", "year", "with", "an", "order", "backlog", "twice", "the", "size", "than", "it", "had", "at", "the", "outset", "."], ["The", "group", "has", "since", "been", "awarded", "a", "US", "600", "million", "contract", "and", "spun", "off", "its", "North", "Sea", "assets", "."], ["The", "group", "’s", "recently", "released", "full", "year", "results", "show", "a", "jump", "in", "revenues", ",", "pre-tax", "profits", "and", "order", "backlog", "."], ["Whilst", "group", "revenue", "rose", "by", "10", "%", "from", "$", "3.3", "billion", "to", "$", "3.7", "billion", ",", "pre-tax", "profits", "rose", "by", "25", "%", "from", "$", "358", "million", "to", "$", "448", "million", ".All", "the", "more", "impressive", ",", "the", "group", "’s", "order", "backlog", "doubled", "to", "over", "$", "8", "billion", "paying", "no", "attention", "to", "the", "15", "%", "cut", "in", "capital", "expenditure", "witnessed", "across", "the", "oil", "and", "gas", "industry", "as", "whole", "in", "2009", ".Focussing", "in", "on", "which", "the", "underlying", "performances", "of", "the", "individual", "segments", ",", "the", "group", "cash", "cow", ",", "its", "Engineering", "and", "Construction", "division", ",", "saw", "operating", "profit", "rise", "33", "%", "over", "the", "year", "to", "$", "322", "million", ",", "thanks", "to", "US$", "6.3", "billion", "worth", "of", "new", "contract", "wins", "during", "the", "year", "which", "included", "a", "$", "100", "million", "contract", "with", "Turkmengaz", ",", "the", "Turkmenistan", "national", "energy", "company", "."], ["The", "division", "has", "picked", "up", "in", "2010", "where", "it", "left", "off", "in", "2009", "and", "has", "been", "awarded", "a", "contract", "worth", "more", "than", "US600", "million", "for", "a", "gas", "sweetening", "facilities", "project", "by", "Qatar", "Petroleum.Elsewhere", "the", "group", "’s", "Offshore", "Engineering", "&", "Operations", "division", "may", "have", "seen", "a", "pullback", "in", "revenue", "and", "earnings", "vis-a-vis", "2008", ",", "but", "it", "did", "secure", "a", "£75", "million", "contract", "with", "Apache", "to", "provideengineering", "and", "construction", "services", "for", "the", "Forties", "field", "in", "the", "UK", "North", "Sea", "."], ["And", "to", "underscore", "the", "fact", "that", "there", "is", "life", "beyond", "NOC’s", "for", "Petrofac", "(", "LON:PFC)", "the", "division", "was", "awarded", "a", "£100", "million", "5-year", "contract", "by", "BP", "(", "LON:BP.", ")", "to", "deliver", "integrated", "maintenance", "management", "support", "services", "for", "all", "of", "BP", "'s", "UK", "offshore", "assets", "and", "onshore", "Dimlington", "plant", "."], ["The", "laggard", "of", "the", "group", "was", "the", "Engineering", ",", "Training", "Services", "and", "Production", "Solutions", "division", "."], ["The", "business", "suffered", "as", "the", "oil", "price", "tailed", "off", "and", "the", "economic", "outlook", "deteriorated", "forcing", "a", "number", "ofmajor", "customers", "to", "postpone", "early", "stage", "engineering", "studies", "or", "re-phased", "work", "upon", "which", "the", "division", "depends", "."], ["Although", "the", "fall", "in", "activity", "was", "notable", ",", "the", "division’s", "operational", "performance", "in", "service", "operator", "role", "for", "production", "of", "Dubai", "'s", "offshore", "oil", "&", "gas", "proved", "a", "highlight.Energy", "Developments", "meanwhile", "saw", "the", "start", "of", "oil", "production", "from", "the", "West", "Don", "field", "during", "the", "first", "half", "of", "the", "year", "less", "than", "a", "year", "from", "Field", "Development", "Programme", "approval", "."], ["In", "addition", "output", "from", "Don", "Southwest", "field", "began", "in", "June", "."], ["Despite", "considerably", "lower", "oil", "prices", "in", "2009", "compared", "to", "the", "prior", "year", ",", "Energy", "Developments", "'", "revenue", "reached", "almost", "US$", "250", "million", "(", "significantly", "higher", "than", "the", "US$", "153", "million", "of", "2008", ")", "due", "not", "only", "to", "the", "‘Don", "fields", "effect", "’", "but", "also", "a", "full", "year", "'s", "contribution", "from", "the", "Chergui", "gas", "plant", ",", "which", "began", "exports", "in", "August", "2008.In", "order", "to", "maximize", "the", "earnings", "potential", "of", "the", "division’s", "North", "Sea", "assets", ",", "including", "the", "Don", "assets", ",", "the", "group", "has", "demerged", "them", "providing", "its", "shareholders", "with", "shares", "in", "a", "newly", "listed", "independent", "exploration", "and", "production", "company", "called", "EnQuest", "(", "LON:ENQ", ")", "."], ["EnQuest", "is", "a", "product", "of", "the", "Petrofac’s", "North", "Sea", "Assets", "with", "those", "off", "of", "Swedish", "explorer", "Lundin", "with", "both", "companies", "divesting", "for", "different", "reasons", "."], ["Upon", "listing", "(", "April", "6th", ")", ",", "Petrofac", "(", "LON:PFC)", "shareholders", "owned", "around", "45", "%", "of", "the", "new", "EnQuest", "entity", "with", "Lundin", "shareholders", "owning", "approximately", "55", "%", "."], ["It", "is", "important", "to", "note", "that", "post", "demerger", "the", "Energy", "Developments", "business", "unit", "is", "still", "a", "key", "constituent", "of", "Petrofac", "'s", "business", "portfolio", ",", "and", "will", "continue", "to", "hold", "significant", "assets", "Tunisia", ",", "Malaysia", ",", "Algeria", "and", "Kyrgyz", "Republic", "-", "sandwiched", "between", "Kazakhstan", "and", "China", "."]]
+
+      all_sentences.should eql ["To describe 2009 as a stellar year for Petrofac (LON:PFC) would be a huge understatement.", "The group finished the year with an order backlog twice the size than it had at the outset.", "The group has since been awarded a US 600 million contract and spun off its North Sea assets.", "The group’s recently released full year results show a jump in revenues, pre-tax profits and order backlog.", "Whilst group revenue rose by 10% from $3.3 billion to $3.7 billion, pre-tax profits rose by 25% from $358 million to $448 million.All the more impressive, the group’s order backlog doubled to over $8 billion paying no attention to the 15% cut in capital expenditure witnessed across the oil and gas industry as whole in 2009.Focussing in on which the underlying performances of the individual segments, the group cash cow, its Engineering and Construction division, saw operating profit rise 33% over the year to $322 million, thanks to US$6.3 billion worth of new contract wins during the year which included a $100 million contract with Turkmengaz, the Turkmenistan national energy company.", "The division has picked up in 2010 where it left off in 2009 and has been awarded a contract worth more than US600 million for a gas sweetening facilities project by Qatar Petroleum.Elsewhere the group’s Offshore Engineering & Operations division may have seen a pullback in revenue and earnings vis-a-vis 2008, but it did secure a £75 million contract with Apache to provideengineering and construction services for the Forties field in the UK North Sea.", "And to underscore the fact that there is life beyond NOC’s for Petrofac (LON:PFC) the division was awarded a £100 million 5-year contract by BP (LON:BP.) to deliver integrated maintenance management support services for all of BP's UK offshore assets and onshore Dimlington plant.", "The laggard of the group was the Engineering, Training Services and Production Solutions division.", "The business suffered as the oil price tailed off and the economic outlook deteriorated forcing a number ofmajor customers to postpone early stage engineering studies or re-phased work upon which the division depends.", "Although the fall in activity was notable, the division’s operational performance in service operator role for production of Dubai's offshore oil & gas proved a highlight.Energy Developments meanwhile saw the start of oil production from the West Don field during the first half of the year less than a year from Field Development Programme approval.", "In addition output from Don Southwest field began in June.", "Despite considerably lower oil prices in 2009 compared to the prior year, Energy Developments' revenue reached almost US$250 million (significantly higher than the US$153 million of 2008) due not only to the ‘Don fields effect’ but also a full year's contribution from the Chergui gas plant, which began exports in August 2008.In order to maximize the earnings potential of the division’s North Sea assets, including the Don assets, the group has demerged them providing its shareholders with shares in a newly listed independent exploration and production company called EnQuest (LON:ENQ).", "EnQuest is a product of the Petrofac’s North Sea Assets with those off of Swedish explorer Lundin with both companies divesting for different reasons.", "Upon listing (April 6th), Petrofac (LON:PFC) shareholders owned around 45% of the new EnQuest entity with Lundin shareholders owning approximately 55%.", "It is important to note that post demerger the Energy Developments business unit is still a key constituent of Petrofac's business portfolio, and will continue to hold significant assets Tunisia, Malaysia, Algeria and Kyrgyz Republic - sandwiched between Kazakhstan and China."]
+
+      all_entities.should eql [[["$", "3.3", "billion"], "money"], [["$", "3.7", "billion"], "money"], [["$", "358", "million", "to", "$", "448", "million"], "money"], [["$", "8", "billion"], "money"], [["$", "322", "million"], "money"], [["$", "100", "million"], "money"], [["Lundin"], "person"], [["Lundin"], "person"]]
+      
+      all_tags.should eql [["TO", "VB", "CD", "IN", "DT", "NN", "NN", "IN", "NNP", "-LRB-", "NNP", "MD", "VB", "DT", "JJ", "NN", "."], ["DT", "NN", "VBD", "DT", "NN", "IN", "DT", "NN", "NN", "RB", "DT", "NN", "IN", "PRP", "VBD", "IN", "DT", "NN", "."], ["DT", "NN", "VBZ", "RB", "VBN", "VBN", "DT", "PRP", "CD", "CD", "NN", "CC", "VBD", "RP", "PRP$", "NNP", "NNP", "NNS", "."], ["DT", "NN", "VBD", "RB", "VBN", "JJ", "NN", "NNS", "VBP", "DT", "NN", "IN", "NNS", ",", "JJ", "NNS", "CC", "NN", "NN", "."], ["NNP", "NN", "NN", "VBD", "IN", "CD", "NN", "IN", "$", "CD", "CD", "TO", "$", "CD", "CD", ",", "JJ", "NNS", "VBD", "IN", "CD", "NN", "IN", "$", "CD", "CD", "TO", "$", "CD", "CD", "PDT", "DT", "RBR", "JJ", ",", "DT", "NN", "VBZ", "NN", "NN", "VBD", "TO", "RP", "$", "CD", "CD", "VBG", "DT", "NN", "TO", "DT", "CD", "NN", "NN", "IN", "NN", "NN", "VBN", "IN", "DT", "NN", "CC", "NN", "NN", "IN", "JJ", "IN", "CD", "NN", "IN", "IN", "WDT", "DT", "JJ", "NNS", "IN", "DT", "JJ", "NNS", ",", "DT", "NN", "NN", "NN", ",", "PRP$", "NNP", "CC", "NNP", "NN", ",", "VBD", "NN", "NN", "VB", "CD", "NN", "IN", "DT", "NN", "TO", "$", "CD", "CD", ",", "NNS", "TO", "$", "CD", "CD", "NN", "IN", "JJ", "NN", "VBZ", "IN", "DT", "NN", "WDT", "VBD", "DT", "$", "CD", "CD", "NN", "IN", "NNP", ",", "DT", "NNP", "JJ", "NN", "NN", "."], ["DT", "NN", "VBZ", "VBN", "RP", "IN", "CD", "WRB", "PRP", "VBD", "RP", "IN", "CD", "CC", "VBZ", "VBN", "VBN", "DT", "NN", "NN", "JJR", "IN", "CD", "CD", "IN", "DT", "NN", "VBG", "NNS", "NN", "IN", "NNP", "NNP", "DT", "NN", "JJ", "NNP", "NNP", "CC", "NNP", "NN", "MD", "VB", "VBN", "DT", "NN", "IN", "NN", "CC", "NNS", "NN", "CD", ",", "CC", "PRP", "VBD", "VB", "DT", "CD", "CD", "NN", "IN", "NNP", "TO", "VB", "CC", "NN", "NNS", "IN", "DT", "NNP", "NN", "IN", "DT", "NNP", "NNP", "NNP", "."], ["CC", "TO", "VB", "DT", "NN", "IN", "EX", "VBZ", "NN", "IN", "NNP", "IN", "NNP", "-LRB-", "NNP", "DT", "NN", "VBD", "VBN", "DT", "CD", "CD", "JJ", "NN", "IN", "NNP", "-LRB-", "NNP", "-RRB-", "TO", "VB", "JJ", "NN", "NN", "NN", "NNS", "IN", "DT", "IN", "NNP", "POS", "NN", "JJ", "NNS", "CC", "RB", "NNP", "NN", "."], ["DT", "NN", "IN", "DT", "NN", "VBD", "DT", "NNP", ",", "NNP", "NNP", "CC", "NNP", "NNP", "NN", "."], ["DT", "NN", "VBD", "IN", "DT", "NN", "NN", "VBN", "RB", "CC", "DT", "JJ", "NN", "VBD", "VBG", "DT", "NN", "IN", "NNS", "TO", "VB", "JJ", "NN", "NN", "NNS", "CC", "JJ", "NN", "IN", "WDT", "DT", "NN", "VBZ", "."], ["IN", "DT", "NN", "IN", "NN", "VBD", "JJ", ",", "DT", "JJ", "JJ", "NN", "IN", "NN", "NN", "NN", "IN", "NN", "IN", "NNP", "POS", "JJ", "NN", "CC", "NN", "VBD", "DT", "RB", "NNPS", "RB", "VBD", "DT", "NN", "IN", "NN", "NN", "IN", "DT", "NNP", "NNP", "NN", "IN", "DT", "JJ", "NN", "IN", "DT", "NN", "RBR", "IN", "DT", "NN", "IN", "NNP", "NNP", "NNP", "NN", "."], ["IN", "NN", "NN", "IN", "NNP", "NNP", "NN", "VBD", "IN", "NNP", "."], ["IN", "RB", "JJR", "NN", "NNS", "IN", "CD", "VBN", "TO", "DT", "JJ", "NN", ",", "NNP", "NNPS", "POS", "NN", "VBD", "RB", "$", "CD", "CD", "-LRB-", "RB", "JJR", "IN", "DT", "$", "CD", "CD", "IN", "CD", "-RRB-", "RB", "RB", "RB", "TO", "DT", "JJ", "NNS", "NN", ",", "CC", "RB", "DT", "JJ", "NN", "POS", "NN", "IN", "DT", "NNP", "NN", "NN", ",", "WDT", "VBD", "NNS", "IN", "NNP", "IN", "NN", "TO", "VB", "DT", "NNS", "NN", "IN", "DT", "JJ", "NNP", "NNP", "NNS", ",", "VBG", "DT", "NNP", "NNS", ",", "DT", "NN", "VBZ", "VBN", "PRP", "VBG", "PRP$", "NNS", "IN", "NNS", "IN", "DT", "RB", "VBN", "JJ", "NN", "CC", "NN", "NN", "VBD", "NNP", "-LRB-", "NN", "-RRB-", "."], ["NNP", "VBZ", "DT", "NN", "IN", "DT", "NNP", "NNP", "NNP", "NNS", "IN", "DT", "IN", "IN", "JJ", "NN", "NN", "IN", "DT", "NNS", "VBG", "IN", "JJ", "NNS", "."], ["IN", "VBG", "-LRB-", "NNP", "NN", "-RRB-", ",", "NNP", "-LRB-", "NNP", "NNS", "VBD", "IN", "CD", "NN", "IN", "DT", "JJ", "NNP", "NN", "IN", "NNP", "NNS", "VBG", "RB", "CD", "NN", "."], ["PRP", "VBZ", "JJ", "TO", "VB", "IN", "NN", "NN", "DT", "NNP", "NNPS", "NN", "NN", "VBZ", "RB", "DT", "JJ", "NN", "IN", "NNP", "POS", "NN", "NN", ",", "CC", "MD", "VB", "TO", "VB", "JJ", "NNS", "NNP", ",", "NNP", ",", "NNP", "CC", "NNP", "NNP", ":", "VBD", "IN", "NNP", "CC", "NNP", "."]]
+      
+    end
+  end
+
+end

data/spec/sample.txt

@@ -0,0 +1,20 @@
+To describe 2009 as a stellar year for Petrofac (LON:PFC) would be a huge understatement. The group finished the year with an order backlog twice the size than it had at the outset. The group has since been awarded a US 600 million contract and spun off its North Sea assets. 
+The group’s recently released full year results show a jump in revenues, pre-tax profits and order backlog. Whilst group revenue rose by 10% from $3.3 billion to $3.7 billion, pre-tax profits rose by 25% from $358 million to $448 million.
+
+All the more impressive, the group’s order backlog doubled to over $8 billion paying no attention to the 15% cut in capital expenditure witnessed across the oil and gas industry as whole in 2009.
+
+Focussing in on which the underlying performances of the individual segments, the group cash cow, its Engineering and Construction division, saw operating profit rise 33% over the year to $322 million, thanks to US$6.3 billion worth of new contract wins during the year which included a $100 million contract with Turkmengaz, the Turkmenistan national energy company. The division has picked up in 2010 where it left off in 2009 and has been awarded a contract worth more than US600 million for a gas sweetening facilities project by Qatar Petroleum.
+
+Elsewhere the group’s Offshore Engineering & Operations division may have seen a pullback in revenue and earnings vis-a-vis 2008, but it did secure a £75 million contract with Apache to provide
+engineering and construction services for the Forties field in the UK North Sea. And to underscore the fact that there is life beyond NOC’s for Petrofac (LON:PFC) the division was awarded a £100 million 5-year contract by BP (LON:BP.) to deliver integrated maintenance management support services for all of BP's UK offshore assets and onshore Dimlington plant.   
+
+The laggard of the group was the Engineering, Training Services and Production Solutions division. The business suffered as the oil price tailed off and the economic outlook deteriorated forcing a number of
+major customers to postpone early stage engineering studies or re-phased work upon which the division depends. Although the fall in activity was notable, the division’s operational performance in service operator role for production of Dubai's offshore oil & gas proved a highlight.
+
+Energy Developments meanwhile saw the start of oil production from the West Don field during the first half of the year less than a year from Field Development Programme approval. In addition output from Don Southwest field began in June. Despite considerably lower oil prices in 2009 compared to the prior year, Energy Developments' revenue reached almost US$250 million (significantly higher than the US$153 million of 2008) due not only to the ‘Don fields effect’ but also a full year's contribution from the Chergui gas plant, which began exports in August 2008.
+
+In order to maximize the earnings potential of the division’s North Sea assets, including the Don assets, the group has demerged them providing its shareholders with shares in a newly listed independent exploration and production company called EnQuest (LON:ENQ). 
+
+EnQuest is a product of the Petrofac’s North Sea Assets with those off of Swedish explorer Lundin with both companies divesting for different reasons. Upon listing (April 6th), Petrofac (LON:PFC) shareholders owned around 45% of the new EnQuest entity with Lundin shareholders owning approximately 55%. 
+
+It is important to note that post demerger the Energy Developments business unit is still a key constituent of Petrofac's business portfolio, and will continue to hold significant assets Tunisia, Malaysia, Algeria and Kyrgyz Republic - sandwiched between Kazakhstan and China.

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+require 'rspec'
+require_relative '../lib/open-nlp'

metadata

@@ -0,0 +1,74 @@
+--- !ruby/object:Gem::Specification
+name: open-nlp
+version: !ruby/object:Gem::Version
+  prerelease:
+  version: 0.0.1
+platform: ruby
+authors:
+- Louis Mullie
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2012-12-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bind-it
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+    none: false
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+    none: false
+  prerelease: false
+  type: :runtime
+description: ! ' Ruby bindings to the OpenNLP tools, a Java machine learning toolkit
+  for natural language processing (NLP). '
+email:
+- louis.mullie@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- bin/jwnl-1.3.3.jar
+- bin/opennlp-maxent-3.0.2-incubating.jar
+- bin/opennlp-tools-1.5.2-incubating.jar
+- bin/opennlp-uima-1.5.2-incubating.jar
+- lib/open-nlp.rb
+- lib/open-nlp/config.rb
+- spec/english_spec.rb
+- spec/sample.txt
+- spec/spec_helper.rb
+- README.md
+- LICENSE
+homepage: https://github.com/louismullie/open-nlp
+licenses: []
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+  none: false
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+  none: false
+requirements: []
+rubyforge_project:
+rubygems_version: 1.8.24
+signing_key:
+specification_version: 3
+summary: Ruby bindings to the OpenNLP Java toolkit.
+test_files: []
+...