corenlp 0.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 072b1b153bb4591c16e6242713e9b4431ba003da
+  data.tar.gz: 0e71dd5289c128e0245f082ace874d29b51cd92f
+SHA512:
+  metadata.gz: 7969ddc18c42ca6c832c06bf677df56212f4ec54bc35bebbd4d9e4925425804015ef2e333cbc0463a88af76204ed46f814c355c10c703586761c9a7db501442d
+  data.tar.gz: 646eb3e03f42182e5a957fe6d52db3a7767cd52c9cf48bb4bd4bc36d0e07c03f1f4890c652addad58ccaf845f9a430c5d877143902b3824efc60c42e44757f31

data/README.md

@@ -0,0 +1,95 @@
+# Corenlp
+
+Corenlp is a Ruby gem that uses the [Stanford CoreNLP Java tools](http://nlp.stanford.edu/software/corenlp.shtml) to parse text. The gem takes the output from Stanford CoreNLP and builds objects in Ruby, for use in Ruby applications.
+
+Stanford CoreNLP requires Java version 1.6 or higher. Installations vary so that will need to be installed by the developer on their own before continuing.
+
+Development has been done with 64-bit Java on a OS X machine. 3G of RAM is allocated for the Java process, which can be set any time the parser is called.
+
+The following Java version has been used in development.
+
+    $ java -version
+    java version "1.7.0_45"
+    Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
+    Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
+
+## Installing Stanford CoreNLP
+
+Run `rake corenlp:download_deps` to download the Stanford CoreNLP dependencies. The files will be extracted to the `lib/ext` directory, which is ignored from git. The dependencies directory can be customized. The download URL can also be customized.
+
+The rake task will use the values from the following environment variables if they are set.
+
+ * `CORENLP_DOWNLOAD_URL` - This is set to "http://nlp.stanford.edu/software/stanford-corenlp-full-2014-06-16.zip" which was the latest version when this was written.
+ * `CORENLP_DEPS_DIR` - This is set to "./lib/ext/", which is a directory that exists in our project where we want to place the Stanford CoreNLP files.
+
+To customize these values, supply environment variable arguments when calling the rake task like this:
+
+    rake corenlp:download_deps CORENLP_DEPS_DIR='./my_directory'
+
+## Testing the output in a IRB console
+
+Corenlp gem builds up a treebank of structured parts that define tokens, sentences, and dependencies between the tokens. This treebank structure is represented as a nested Ruby hash. Token objects that are part of a sentence are nested within the sentence. Token dependencies are nested within the sentence, and so on.
+
+The following code will build up a treebank structure for the raw text "Put the book down.". On my machine this takes around 10 seconds to run.
+
+    bundle exec irb
+    Bundler.require
+    Corenlp::Treebank.new(raw_text: "Put the book down.").parse
+
+## Options
+
+The Treebank object can be initialize with various options.
+
+ * `java_max_memory` - set to 3GB by default. This can be customized via the Treebank initializer to be `-Xmx2g`, which would use a max of 2GB of memory, for example.
+ * `threads_to_use` - number of threads Stanford CoreNLP uses to parse text. This is set to 4 by default. This option is passed to the Java executable.
+ * `output_directory` - by default this is `./tmp/language_processing`, which already exists. This is where Stanford CoreNLP XML files are placed. These XML files represented the structured parser output.
+
+## Tests
+
+Minitest is used as a test suite for the Ruby objects. New code should include test coverage. Manually testing is also useful. Internally we have some more test methods to verify parser output on the same content over time, but they are not included at this time.
+
+    rake
+
+To run a single test:
+
+    ruby path/to/file.rb --name test_method_name
+
+## Terminology
+
+Stanford CoreNLP uses a lot of terminology from the natural language processing field, and defines its own terminology. Refer to the [Stanford CoreNLP documentation](http://nlp.stanford.edu/software/corenlp.shtml) to learn more.
+
+## Contributors
+
+This gem was developed at Lengio by Andy Atkinson as an extraction of some of our natural language processing tools.
+
+  * Andy Atkinson, gem author and maintainer
+  * Kamran Khan
+  * Rodolfo Carvalho
+
+## Rubygems badge
+
+[![Gem Version](https://badge.fury.io/rb/corenlp.svg)](http://badge.fury.io/rb/corenlp)
+
+## License
+
+The MIT License (MIT)
+
+Copyright (c) 2014 Lengio Corporation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/lib/corenlp/downloader.rb

@@ -0,0 +1,56 @@
+require 'net/http'
+require 'zip/zip'
+require 'fileutils'
+require 'uri'
+
+module Corenlp
+  class Downloader
+    attr_accessor :url, :destination, :local_file
+    def initialize(url, destination)
+      self.url = url
+      self.destination = destination
+      self.local_file = nil
+    end
+
+    def extract
+      puts "extracting file..."
+      Zip::ZipFile.open(local_file) do |zip_file|
+        zip_file.each do |file|
+          file_path = File.join(destination, file.name)
+          zip_file.extract(file, file_path) unless File.exist?(file_path)
+        end
+
+        puts "moving files into directory..."
+        dirname = local_file[0...-4]
+        dir = File.join(destination, dirname)
+        if File.exists?(dir)
+          Dir.glob(File.join(dir, "*")).each do |file|
+            FileUtils.mv(file, File.join(destination, File.basename(file)))
+          end
+          FileUtils.rm_rf(dir)
+        end
+
+        puts "deleting original zip file..."
+        FileUtils.rm(local_file)
+        puts "done."
+      end
+    end
+
+    def download
+      return unless url
+      puts "downloading zip file from url #{url} to #{destination}..."
+      self.local_file = File.basename(url)
+      uri = URI.parse(url)
+      if local_file && uri
+        Net::HTTP.start(uri.host) do |http|
+          resp = http.get(uri.request_uri)
+          open(local_file, "wb") do |file|
+            file.write(resp.body)
+          end
+        end
+        puts "done. Downloaded file #{local_file}."
+        extract
+      end
+    end
+  end
+end

data/lib/corenlp/enclitic.rb

@@ -0,0 +1,22 @@
+module Corenlp
+  class Enclitic < Token
+    def enclitic_map
+      # Note: This isn't really one-to-one (e.g. "'d" could be "had" or "would"):
+      {
+        "'ll" => "will",
+        "'m"  => "am",
+        "'re" => "are",
+        "'s"  => "is",
+        "'d"  => "would",
+        "'t"  => "not",
+        "'ve" => "have",
+        "'nt" => "not",
+        "n't" => "not"
+      }
+    end
+
+    def expanded
+      enclitic_map[text]
+    end
+  end
+end

data/lib/corenlp/number.rb

@@ -0,0 +1,4 @@
+module Corenlp
+  class Number < Token
+  end
+end

data/lib/corenlp/punctuation.rb

@@ -0,0 +1,20 @@
+module Corenlp
+  class Punctuation < Token
+    # From http://www.unicode.org/charts/PDF/U20A0.pdf
+    CURRENCY_SYMBOLS = %W(\u0024 \u00A2 \u00A3 \u00A4 \u00A5 \u20AC)
+    DASH_SYMBOLS = %W(\u2010 \u2011 \u2012 \u2013 \u2014)
+    OPENING_SYMBOLS = %W(\u201C \u2018 \u00A1 \u00BF \( [ {)
+
+    def currency?
+      CURRENCY_SYMBOLS.include?(text)
+    end
+
+    def dash?
+      DASH_SYMBOLS.include?(text)
+    end
+
+    def opening?
+      OPENING_SYMBOLS.include?(text)
+    end
+  end
+end

data/lib/corenlp/sentence.rb

@@ -0,0 +1,28 @@
+module Corenlp
+  class Sentence
+    attr_accessor :index, :tokens, :token_dependencies, :parse_tree_raw
+
+    def initialize(attrs = {})
+      @index = attrs[:index]
+      @tokens = []
+      @token_dependencies = []
+      @parse_tree_raw = ''
+    end
+
+    def governor_dependencies(token)
+      token_dependencies.select{|td| td.governor == token}
+    end
+
+    def next_token(token)
+      tokens.sort_by(&:index).detect{|t| t.index > token.index}
+    end
+
+    def previous_token(token)
+      tokens.sort_by(&:index).reverse.detect{|t| t.index < token.index}
+    end
+
+    def get_dependency_token_by_index(index)
+      tokens.detect{|t| t.index == index}
+    end
+  end
+end

data/lib/corenlp/token.rb

@@ -0,0 +1,73 @@
+module Corenlp
+  class Token
+    attr_accessor :index, :text, :penn_treebank_tag, :stanford_lemma, :type, :ner
+
+    def initialize(attrs = {})
+      @index = attrs[:index]
+      @text = attrs[:text]
+      @penn_treebank_tag = attrs[:penn_treebank_tag]
+      @stanford_lemma = attrs[:stanford_lemma]
+      @type = attrs[:type]
+      @ner = attrs[:ner]
+    end
+
+    IGNORED_ENTITIES = ["PERSON"]
+
+    def content?
+      is_a?(Word) || is_a?(Enclitic)
+    end
+
+    def top_level_penn_treebank_category
+      penn_treebank_tag[0]
+    end
+
+    def ==(other)
+      index == other.index && \
+        penn_treebank_tag == other.penn_treebank_tag && type == other.type
+    end
+
+    def website_text?
+      text =~ /http:\/\//
+    end
+
+    def self.clean_stanford_text(text)
+      Token::STANFORD_TEXT_REPLACEMENTS.each_pair do |original, replacement|
+        text.gsub!(replacement, original)
+      end
+      text
+    end
+
+    Enclitics = %w{'ll 'm 're 's 't 've 'nt n't 'd ’ll ’m ’re ’s ’t ’ve ’nt n’t ’d}
+    WordRegexp = /^[[:alpha:]\-'\/]+$/
+    NumberRegexp = /^#?(\d+)(,\d+)*(\.\d+)?$/
+    PunctRegexp = /^[[:punct:]'"\$]+$/
+    WebsiteRegexp = /https?:\/\/[\S]+/
+
+    # The character replacements that Stanford performs which we reverse:
+    STANFORD_TEXT_REPLACEMENTS = {
+      '”' => "''",    '“' => '``',    '(' => '-LRB-',
+      ')' => '-RRB-', '[' => '-LSB-', ']' => '-RSB-',
+      '{' => '-LCB-', '}' => '-RCB-',
+      '‘' => '`', '’' => '\'', '—' => '--', '/' => '\\/'
+    }
+
+    def ignored_entity?
+      IGNORED_ENTITIES.include?(self.ner)
+    end
+
+    def self.token_subclass_from_text(text)
+      case
+      when Enclitics.include?(text)
+        Enclitic
+      when (text =~ WordRegexp && text != '-') || (text =~ WebsiteRegexp)
+        Word
+      when text =~ PunctRegexp
+        Punctuation
+      when text =~ NumberRegexp
+        Number
+      else
+        Token
+      end
+    end
+  end
+end

data/lib/corenlp/token_dependency.rb

@@ -0,0 +1,11 @@
+module Corenlp
+  class TokenDependency
+    attr_accessor :dependent, :governor, :relation
+
+    def initialize(attrs = {})
+      @dependent = attrs[:dependent]
+      @governor = attrs[:governor]
+      @relation = attrs[:relation]
+    end
+  end
+end

data/lib/corenlp/version.rb

@@ -0,0 +1,3 @@
+module Corenlp
+  VERSION = "0.0.3"
+end

data/lib/corenlp/word.rb

@@ -0,0 +1,4 @@
+module Corenlp
+  class Word < Token
+  end
+end

data/test/downloader_test.rb

@@ -0,0 +1,9 @@
+require "test_helper"
+
+class DownloaderTest < Minitest::Test
+  def test_initialized_ok
+    zip_file_url = "http://nlp.stanford.edu/software/stanford-corenlp-full-2014-06-16.zip"
+    destination = File.join('./lib/ext/')
+    assert Downloader.new(zip_file_url, destination)
+  end
+end

data/test/enclitic_test.rb

@@ -0,0 +1,8 @@
+require "test_helper"
+
+class EncliticTest < Minitest::Test
+  def test_expaneded_returns_the_expanded_version
+    enclitic = Enclitic.new(text: "'ll")
+    assert_equal "will", enclitic.expanded
+  end
+end

data/test/number_test.rb

@@ -0,0 +1,8 @@
+require "test_helper"
+
+class NumberTest < Minitest::Test
+  def test_number_is_initialized_ok
+    number = Number.new(text: "1")
+    assert number.is_a?(Number)
+  end
+end

data/test/punctuation_test.rb

@@ -0,0 +1,38 @@
+require "test_helper"
+
+class PunctuationTest < Minitest::Test
+  def test_punctuation_is_initialized_ok
+    punctuation = Punctuation.new(text: "$")
+    assert punctuation.is_a?(Punctuation)
+  end
+
+  def test_punctuation_is_currency
+    Punctuation::CURRENCY_SYMBOLS.each do |s|
+      token = Punctuation.new text: s
+      assert token.currency?, "Token #{token.text} should be a currency"
+    end
+
+    token = Punctuation.new text: "a"
+    assert !token.currency?
+  end
+
+  def test_punctuation_is_dash
+    Punctuation::DASH_SYMBOLS.each do |s|
+      token = Punctuation.new text: s
+      assert token.dash?, "Token #{token.text} should be a dash"
+    end
+
+    token = Punctuation.new text: "{"
+    assert !token.dash?
+  end
+
+  def test_punctuation_is_opening
+    Punctuation::OPENING_SYMBOLS.each do |s|
+      token = Punctuation.new text: s
+      assert token.opening?, "Token #{token.text} should be an opening"
+    end
+
+    token = Punctuation.new text: "$"
+    assert !token.opening?
+  end
+end

data/test/sentence_test.rb

@@ -0,0 +1,33 @@
+require "test_helper"
+
+class SentenceTest < Minitest::Test
+  def test_initialized_ok
+    sentence = Sentence.new(index: 0, text: "some text in a sentence.")
+    t1 = Token.new(index: 0, text: "some", type: "Word", sentence: sentence)
+    t2 = Token.new(index: 1, text: "text", type: "Word", sentence: sentence)
+    t3 = Token.new(index: 2, text: "in", type: "Word", sentence: sentence)
+    t4 = Token.new(index: 3, text: "a", type: "Word", sentence: sentence)
+    td1 = TokenDependency.new(dependent: t1, governor: t2, relation: "det")
+    sentence.tokens << t1 << t2 << t3 << t4
+    sentence.token_dependencies << td1
+    assert_equal 4, sentence.tokens.size
+    assert_equal 1, sentence.token_dependencies.size
+    assert_equal [td1], sentence.governor_dependencies(t2)
+    assert_equal [td1], sentence.token_dependencies
+    assert_equal 0, sentence.index
+  end
+
+  def test_calculate_previous_and_next_token_from_token_in_a_sentence
+    sentence = Sentence.new(index: 0, text: "some text in a sentence.")
+    t0 = Word.new(index: 0, text: "some", sentence: sentence)
+    t1 = Word.new(index: 1, text: "text", sentence: sentence)
+    t2 = Word.new(index: 2, text: "in", sentence: sentence)
+    t3 = Word.new(index: 3, text: "a", sentence: sentence)
+    sentence.tokens << t0 << t1 << t2 << t3
+    assert_equal nil, sentence.previous_token(t0)
+    assert_equal nil, sentence.next_token(t3)
+    assert_equal t1, sentence.next_token(t0)
+    assert_equal t2, sentence.previous_token(t3)
+    assert_equal t1, sentence.previous_token(t2)
+  end
+end

data/test/test_helper.rb

@@ -0,0 +1,5 @@
+require 'minitest/autorun'
+lib = File.expand_path('../../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "corenlp"
+include Corenlp

data/test/token_test.rb

@@ -0,0 +1,41 @@
+require "test_helper"
+
+class TokenTest < Minitest::Test
+  def test_initialized_ok
+    assert token = Word.new(index: 0, text: "some", penn_treebank_tag: "NNP",
+                      stanford_lemma: "some")
+    assert_equal "N", token.top_level_penn_treebank_category
+    assert token.content?
+  end
+
+  def test_token_equality
+    t0 = Word.new(index: 0, text: "some", penn_treebank_tag: "NNP")
+    t1 = Word.new(index: 0, text: "more", penn_treebank_tag: "NNP")
+    assert t0 == t1
+  end
+
+  def test_token_person_ner_value_is_ignored
+    assert Token.new(text: "Walter", ner: "PERSON").ignored_entity?
+  end
+
+  def test_number_recognition
+    text_samples = ["33,333", "20", "30.00", "30,000,000.00"]
+
+    text_samples.each do |text|
+      assert_equal Number, Token.token_subclass_from_text(text), text
+    end
+
+    text_samples = ["33A333", "20", "30F00", "30X000;000"]
+    text_samples.each do |text|
+      assert !Token.token_subclass_from_text(text).is_a?(Number)
+    end
+  end
+
+  def test_enclitic_recognition
+    text_samples = %w{n't 'nt 'll n’t ’nt ’ll}
+
+    text_samples.each do |text|
+      assert_equal Enclitic, Token.token_subclass_from_text(text)
+    end
+  end
+end

data/test/treebank_test.rb

@@ -0,0 +1,12 @@
+require 'test_helper'
+
+class TreebankTest < Minitest::Test
+  @@treebank = Treebank.new(raw_text: 'I put the book down on the coffee table.').parse
+
+  def test_treebank_has_all_the_parsed_parts
+    # earlier verions of stanford had some different dependencies
+    # ["I put nsubj", "the book det", "book put dobj", "down put prt", "on put prep", "the table det", "coffee table nn", "table on pobj"]
+    expected = ["I put nsubj", "the book det", "book put dobj", "down put prt", "the table det", "coffee table nn", "table put prep_on"]
+    assert_equal expected, @@treebank.sentences.map(&:token_dependencies).flatten.map{|x| "#{x.dependent.text} #{x.governor.text} #{x.relation}"}
+  end
+end

data/test/word_test.rb

@@ -0,0 +1,8 @@
+require "test_helper"
+
+class WordTest < Minitest::Test
+  def test_word_is_initialized_ok
+    word = Word.new(text: "here")
+    assert word.is_a?(Word)
+  end
+end

metadata

@@ -0,0 +1,150 @@
+--- !ruby/object:Gem::Specification
+name: corenlp
+version: !ruby/object:Gem::Version
+  version: 0.0.3
+platform: ruby
+authors:
+- Lengio Corporation
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-06-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.6.1
+- !ruby/object:Gem::Dependency
+  name: rubyzip
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.5'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Corenlp is a Ruby gem that uses the Stanford CoreNLP Java tools to parse
+  text.
+email:
+- engineering@leng.io
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/corenlp/downloader.rb
+- lib/corenlp/enclitic.rb
+- lib/corenlp/number.rb
+- lib/corenlp/punctuation.rb
+- lib/corenlp/sentence.rb
+- lib/corenlp/token.rb
+- lib/corenlp/token_dependency.rb
+- lib/corenlp/version.rb
+- lib/corenlp/word.rb
+- test/downloader_test.rb
+- test/enclitic_test.rb
+- test/number_test.rb
+- test/punctuation_test.rb
+- test/sentence_test.rb
+- test/test_helper.rb
+- test/token_test.rb
+- test/treebank_test.rb
+- test/word_test.rb
+- README.md
+homepage: https://github.com/lengio/corenlp
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: Corenlp is a Ruby gem that uses the Stanford CoreNLP Java tools to parse
+  text.
+test_files: []
+has_rdoc: