stanford-core-nlp 0.5.1 → 0.5.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f58c7f886c5f1f7fae0ff5915adb48c9fd9dc8df
+  data.tar.gz: 8d0b101b58a6584f3f19ed714a9e9c004d73acc8
+SHA512:
+  metadata.gz: c47c18b4d11651d6e64f48a87d9a5318b5506a92f2dc0ec3453e31bbc1a75f96b7cef72b09dfc77eba98f35b8779bc0dc0ab1befef96107e3611ab2dd34fec65
+  data.tar.gz: 4ff45b83b273e2b7bc29a6614124f857c016fed38c2382a064db8a355c54d998a5c850d60ed204a76aef3ed277c0c4513eabb7cae98a815d5f2d43c0fa1b4c2c

data/README.md

@@ -1,21 +1,25 @@
-[![Build Status](https://secure.travis-ci.org/louismullie/stanford-core-nlp.png)](http://travis-ci.org/louismullie/stanford-core-nlp)
+# Stanford CoreNLP [![Build Status](https://secure.travis-ci.org/louismullie/stanford-core-nlp.png)](http://travis-ci.org/louismullie/stanford-core-nlp) [![Awesome RubyNLP](https://img.shields.io/badge/Awesome-RubyNLP-brightgreen.svg)](https://github.com/arbox/nlp-with-ruby)
 
-**About**
-  
-This gem provides high-level Ruby bindings to the [Stanford Core NLP package](http://nlp.stanford.edu/software/corenlp.shtml), a set natural language processing tools for tokenization, sentence segmentation, part-of-speech tagging, lemmatization, and parsing of English, French and German. The package also provides named entity recognition and coreference resolution for English.
+> Ruby bindings for the Stanford [CoreNLP Toolchain](http://stanfordnlp.github.io/CoreNLP/).
 
-This gem is compatible with Ruby 1.9.2 and 1.9.3 as well as JRuby 1.7.1. It is tested on both Java 6 and Java 7.
+This gem provides high-level Ruby bindings to the
+[Stanford Core NLP package](http://nlp.stanford.edu/software/corenlp.shtml),
+a set natural language processing tools for tokenization, sentence segmentation,
+part-of-speech tagging, lemmatization, and parsing of English, French and German.
+The package also provides named entity recognition and coreference resolution for English.
 
-**Installing**
+This gem is compatible with Ruby 1.9.2 and 1.9.3 as well as JRuby 1.7.1.
+It is tested on both Java 6 and Java 7.
+Newer Ruby version should work as well.
 
-First, install the gem: `gem install stanford-core-nlp`. Then, download the Stanford Core NLP JAR and model files. Two packages are available:
+## Installation
 
-* A [minimal package](http://louismullie.com/treat/stanford-core-nlp-minimal.zip) with the default tagger and parser models for English, French and German.
-* A [full package](http://louismullie.com/treat/stanford-core-nlp-full.zip), with all of the tagger and parser models for English, French and German, as well as named entity and coreference resolution models for English.
+First, install the gem: `gem install stanford-core-nlp`.
+Then, download the Stanford Core NLP JAR and model files: [Stanford CoreNLP](http://nlp.stanford.edu/software/stanford-postagger-full-2014-10-26.zip)
 
 Place the contents of the extracted archive inside the /bin/ folder of the stanford-core-nlp gem (e.g. [...]/gems/stanford-core-nlp-0.x/bin/).
 
-**Configuration**
+## Configuration
 
 You may want to set some optional configuration options. Here are some examples:
 
@@ -40,7 +44,7 @@ StanfordCoreNLP.log_file = 'log.txt'
 StanfordCoreNLP.set_model('pos.model', 'english-left3words-distsim.tagger')
 ```
 
-**Using the gem**
+## Using the gem
 
 ```ruby
 # Use the model files for a different language than English.
@@ -71,7 +75,7 @@ text.get(:sentences).each do |sentence|
     puts token.get(:named_entity_tag).to_s
     # Coreference
     puts token.get(:coref_cluster_id).to_s
-    # Also of interest: coref, coref_chain, 
+    # Also of interest: coref, coref_chain,
     # coref_cluster, coref_dest, coref_graph.
   end
 end
@@ -79,9 +83,13 @@ end
 
 > Important: You need to load the StanfordCoreNLP pipeline before using the StanfordCoreNLP::Annotation class.
 
-The Ruby symbol (e.g. `:named_entity_tag`) corresponding to a Java annotation class is the `snake_case` of the class name, with 'Annotation' at the end removed. For example, `NamedEntityTagAnnotation` translates to `:named_entity_tag`, `PartOfSpeechAnnotation` to `:part_of_speech`, etc.
+The Ruby symbol (e.g. `:named_entity_tag`) corresponding to a Java annotation
+class is the `snake_case` of the class name, with 'Annotation' at the end removed.
+For example, `NamedEntityTagAnnotation` translates to `:named_entity_tag`,
+`PartOfSpeechAnnotation` to `:part_of_speech`, etc.
 
-A good reference for names of annotations are the Stanford Javadocs for [CoreAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/ling/CoreAnnotations.html), [CoreCorefAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/dcoref/CorefCoreAnnotations.html), and [TreeCoreAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/trees/TreeCoreAnnotations.html). For a full list of all possible annotations, see the `config.rb` file inside the gem. 
+A good reference for names of annotations are the Stanford Javadocs
+for [CoreAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/ling/CoreAnnotations.html), [CoreCorefAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/dcoref/CorefCoreAnnotations.html), and [TreeCoreAnnotations](http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/trees/TreeCoreAnnotations.html). For a full list of all possible annotations, see the `config.rb` file inside the gem.
 
 
 **Loading specific classes**
@@ -90,17 +98,17 @@ You may want to load additional Java classes (including any class from the Stanf
 
 ```ruby
 # Default base class is edu.stanford.nlp.pipeline.
-StanfordCoreNLP.load_class('PTBTokenizerAnnotator')  
+StanfordCoreNLP.load_class('PTBTokenizerAnnotator')
 puts StanfordCoreNLP::PTBTokenizerAnnotator.inspect
   # => #<Rjb::Edu_stanford_nlp_pipeline_PTBTokenizerAnnotator>
 
 # Here, we specify another base class.
-StanfordCoreNLP.load_class('MaxentTagger', 'edu.stanford.nlp.tagger') 
+StanfordCoreNLP.load_class('MaxentTagger', 'edu.stanford.nlp.tagger')
 puts StanfordCoreNLP::MaxentTagger.inspect
   # => <Rjb::Edu_stanford_nlp_tagger_maxent_MaxentTagger:0x007f88491e2020>
 ```
 
-**List of annotator classes**
+## List of annotator classes
 
 Here is a full list of annotator classes provided by the Stanford Core NLP package. You can load these classes individually using `StanfordCoreNLP.load_class` (see above). Once this is done, you can use them like you would from a Java program. Refer to the Java documentation for a list of functions provided by each of these classes.
 
@@ -119,7 +127,7 @@ Here is a full list of annotator classes provided by the Stanford Core NLP packa
 * DeterministicCorefAnnotator - implements anaphora resolution using a deterministic model.
 * NFLAnnotator - implements entity and relation mention extraction for the NFL domain.
 
-**List of model files**
+## List of model files
 
 Here is a full list of the default models for the Stanford Core NLP pipeline. You can change these models individually using `StanfordCoreNLP.set_model` (see above).
 
@@ -137,9 +145,75 @@ Here is a full list of the default models for the Stanford Core NLP pipeline. Yo
 * 'dcoref.states' - 'state-abbreviations.txt'
 * 'dcoref.extra.gender' - 'namegender.combine.txt'
 
-**Contributing**
+## Testing
+
+To run the specs for each language (after copying the JARs into the `bin` folder):
+
+``` shell
+rake spec[english]
+rake spec[german]
+rake spec[french]
+```
+
+## Using the latest version of the Stanford CoreNLP
+
+Using the latest version of the Stanford CoreNLP (version 3.5.0 as of 31/10/2014) requires some additional manual steps:
+
+* Download [Stanford CoreNLP version 3.5.0](http://nlp.stanford.edu/software/stanford-corenlp-full-2014-10-31.zip) from http://nlp.stanford.edu/.
+* Place the contents of the extracted archive inside the /bin/ folder of the stanford-core-nlp gem (e.g. [...]/gems/stanford-core-nlp-0.x/bin/) or inside the directory location configured by setting StanfordCoreNLP.jar_path.
+* Download [the full Stanford Tagger version 3.5.0](http://nlp.stanford.edu/software/stanford-postagger-full-2014-10-26.zip) from http://nlp.stanford.edu/.
+* Make a directory named 'taggers' inside the /bin/ folder of the stanford-core-nlp gem (e.g. [...]/gems/stanford-core-nlp-0.x/bin/) or inside the directory configured by setting StanfordCoreNLP.jar_path.
+* Place the contents of the extracted archive inside taggers directory.
+* Download [the bridge.jar file](https://github.com/louismullie/stanford-core-nlp/blob/master/bin/bridge.jar?raw=true) from https://github.com/louismullie/stanford-core-nlp.
+* Place the downloaded bridger.jar file inside the /bin/ folder of the stanford-core-nlp gem (e.g. [...]/gems/stanford-core-nlp-0.x/bin/taggers/) or inside the directory configured by setting StanfordCoreNLP.jar_path.
+* Configure your setup (for English) as follows:
+```ruby
+StanfordCoreNLP.use :english
+StanfordCoreNLP.model_files = {}
+StanfordCoreNLP.default_jars = [
+  'joda-time.jar',
+  'xom.jar',
+  'stanford-corenlp-3.5.0.jar',
+  'stanford-corenlp-3.5.0-models.jar',
+  'jollyday.jar',
+  'bridge.jar'
+]
+end
+```
+Or configure your setup (for French) as follows:
+```ruby
+StanfordCoreNLP.use :french
+StanfordCoreNLP.model_files = {}
+StanfordCoreNLP.set_model('pos.model', 'french.tagger')
+StanfordCoreNLP.default_jars = [
+  'joda-time.jar',
+  'xom.jar',
+  'stanford-corenlp-3.5.0.jar',
+  'stanford-corenlp-3.5.0-models.jar',
+  'jollyday.jar',
+  'bridge.jar'
+]
+end
+```
+Or configure your setup (for German) as follows:
+```ruby
+StanfordCoreNLP.use :german
+StanfordCoreNLP.model_files = {}
+StanfordCoreNLP.set_model('pos.model', 'german-fast.tagger')
+StanfordCoreNLP.default_jars = [
+  'joda-time.jar',
+  'xom.jar',
+  'stanford-corenlp-3.5.0.jar',
+  'stanford-corenlp-3.5.0-models.jar',
+  'jollyday.jar',
+  'bridge.jar'
+]
+end
+```
+
+## Contributing
 
 Simple.
 
 1. Fork the project.
-2. Send me a pull request!
+2. Send me a pull request!

data/bin/taggers/stanford-postagger-full-2014-10-26/TaggerDemo.java

@@ -0,0 +1,28 @@
+
+import java.io.BufferedReader;
+import java.io.FileReader;
+import java.util.List;
+
+import edu.stanford.nlp.ling.Sentence;
+import edu.stanford.nlp.ling.TaggedWord;
+import edu.stanford.nlp.ling.HasWord;
+import edu.stanford.nlp.tagger.maxent.MaxentTagger;
+
+class TaggerDemo {
+
+  private TaggerDemo() {}
+
+  public static void main(String[] args) throws Exception {
+    if (args.length != 2) {
+      System.err.println("usage: java TaggerDemo modelFile fileToTag");
+      return;
+    }
+    MaxentTagger tagger = new MaxentTagger(args[0]);
+    List<List<HasWord>> sentences = MaxentTagger.tokenizeText(new BufferedReader(new FileReader(args[1])));
+    for (List<HasWord> sentence : sentences) {
+      List<TaggedWord> tSentence = tagger.tagSentence(sentence);
+      System.out.println(Sentence.listToString(tSentence, false));
+    }
+  }
+
+}

data/bin/taggers/stanford-postagger-full-2014-10-26/TaggerDemo2.java

@@ -0,0 +1,58 @@
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.util.List;
+
+import edu.stanford.nlp.ling.Sentence;
+import edu.stanford.nlp.ling.TaggedWord;
+import edu.stanford.nlp.ling.HasWord;
+import edu.stanford.nlp.ling.CoreLabel;
+import edu.stanford.nlp.process.CoreLabelTokenFactory;
+import edu.stanford.nlp.process.DocumentPreprocessor;
+import edu.stanford.nlp.process.PTBTokenizer;
+import edu.stanford.nlp.process.TokenizerFactory;
+import edu.stanford.nlp.tagger.maxent.MaxentTagger;
+
+/** This demo shows user-provided sentences (i.e., {@code List<HasWord>})
+ *  being tagged by the tagger. The sentences are generated by direct use
+ *  of the DocumentPreprocessor class.
+ *
+ *  @author Christopher Manning
+ */
+class TaggerDemo2 {
+
+  private TaggerDemo2() {}
+
+  public static void main(String[] args) throws Exception {
+    if (args.length != 2) {
+      System.err.println("usage: java TaggerDemo2 modelFile fileToTag");
+      return;
+    }
+    MaxentTagger tagger = new MaxentTagger(args[0]);
+    TokenizerFactory<CoreLabel> ptbTokenizerFactory = PTBTokenizer.factory(new CoreLabelTokenFactory(),
+									   "untokenizable=noneKeep");
+    BufferedReader r = new BufferedReader(new InputStreamReader(new FileInputStream(args[1]), "utf-8"));
+    PrintWriter pw = new PrintWriter(new OutputStreamWriter(System.out, "utf-8"));
+    DocumentPreprocessor documentPreprocessor = new DocumentPreprocessor(r);
+    documentPreprocessor.setTokenizerFactory(ptbTokenizerFactory);
+    for (List<HasWord> sentence : documentPreprocessor) {
+      List<TaggedWord> tSentence = tagger.tagSentence(sentence);
+      pw.println(Sentence.listToString(tSentence, false));
+    }
+
+    // print the adjectives in one more sentence. This shows how to get at words and tags in a tagged sentence.
+    List<HasWord> sent = Sentence.toWordList("The", "slimy", "slug", "crawled", "over", "the", "long", ",", "green", "grass", ".");
+    List<TaggedWord> taggedSent = tagger.tagSentence(sent);
+    for (TaggedWord tw : taggedSent) {
+      if (tw.tag().startsWith("JJ")) {
+        pw.println(tw.word());
+      }
+    }
+
+    pw.close();
+  }
+
+}

data/lib/stanford-core-nlp.rb

@@ -1,10 +1,7 @@
+require 'bind-it'
 require 'stanford-core-nlp/config'
 
 module StanfordCoreNLP
-
-  VERSION = '0.5.1'
-
-  require 'bind-it'
   extend BindIt::Binding
 
   # ############################ #
@@ -29,9 +26,7 @@ module StanfordCoreNLP
   StanfordCoreNLP.default_jars = [
     'joda-time.jar',
     'xom.jar',
-    'stanford-parser.jar',
     'stanford-corenlp.jar',
-    'stanford-segmenter.jar',
     'jollyday.jar',
     'bridge.jar'
   ]
@@ -57,7 +52,7 @@ module StanfordCoreNLP
 
   require 'stanford-core-nlp/bridge'
   extend StanfordCoreNLP::Bridge
-  
+
   class << self
     # The model file names for a given language.
     attr_accessor :model_files
@@ -65,13 +60,17 @@ module StanfordCoreNLP
     attr_accessor :model_path
     # Store the language currently being used.
     attr_accessor :language
+    #Custom properties
+    attr_accessor :custom_properties
   end
 
+  self.custom_properties = {}
+
   # The path to the main folder containing the folders
   # with the individual models inside. By default, this
   # is the same as the JAR path.
   self.model_path = self.jar_path
-  
+
   # ########################### #
   # Public configuration params #
   # ########################### #
@@ -102,7 +101,7 @@ module StanfordCoreNLP
 
   # Use english by default.
   self.use :english
-  
+
   # Set a model file.
   def self.set_model(name, file)
     n = name.split('.')[0].intern
@@ -114,7 +113,7 @@ module StanfordCoreNLP
   # ########################### #
 
   def self.bind
-    
+
     # Take care of Windows users.
     if self.running_on_windows?
       self.jar_path.gsub!('/', '\\')
@@ -129,16 +128,16 @@ module StanfordCoreNLP
       klass = const_get(info.first)
       self.inject_get_method(klass)
     end
-  
+
   end
-  
+
   # Load a StanfordCoreNLP pipeline with the
   # specified JVM flags and StanfordCoreNLP
   # properties.
   def self.load(*annotators)
-    
+
     self.bind unless self.bound
-    
+
     # Prepend the JAR path to the model files.
     properties = {}
     self.model_files.each do |k,v|
@@ -156,7 +155,7 @@ module StanfordCoreNLP
       end
       properties[k] = f
     end
-    
+
     properties['annotators'] = annotators.map { |x| x.to_s }.join(', ')
 
     unless self.language == :english
@@ -168,45 +167,46 @@ module StanfordCoreNLP
       # Otherswise throws java.lang.NullPointerException: null.
       properties['parse.buildgraphs'] = 'false'
     end
-    
+
     # Bug fix for NER system. Otherwise throws:
     # Error initializing binder 1 at edu.stanford.
     # nlp.time.Options.<init>(Options.java:88)
     properties['sutime.binders'] = '0'
-    
+
     # Manually include SUTime models.
     if annotators.include?(:ner)
-      properties['sutime.rules'] = 
+      properties['sutime.rules'] =
       self.model_path + 'sutime/defs.sutime.txt, ' +
       self.model_path + 'sutime/english.sutime.txt'
     end
-    
+
     props = get_properties(properties)
-    
+
     # Hack for Java7 compatibility.
     bridge = const_get(:AnnotationBridge)
     bridge.getPipelineWithProperties(props)
 
   end
-  
+
   # Hack in order not to break backwards compatibility.
   def self.const_missing(const)
     if const == :Text
       puts "WARNING: StanfordCoreNLP::Text has been deprecated." +
       "Please use StanfordCoreNLP::Annotation instead."
       Annotation
-    else 
+    else
       super(const)
     end
   end
 
   private
-  
+
   # Create a java.util.Properties object from a hash.
   def self.get_properties(properties)
+    properties = properties.merge(self.custom_properties)
     props = Properties.new
     properties.each do |property, value|
-      props.set_property(property, value)
+      props.set_property(property.to_s, value.to_s)
     end
     props
   end

data/lib/stanford-core-nlp/config.rb

@@ -41,7 +41,7 @@ module StanfordCoreNLP
       },
       
       :ner => {
-        :english => 'all.3class.distsim.crf.ser.gz'
+        :english => 'english.all.3class.distsim.crf.ser.gz'
         # :german => {} # Add this at some point.
       },
       
@@ -58,7 +58,8 @@ module StanfordCoreNLP
           'states' => 'state-abbreviations.txt',
           'countries' => 'countries', 
           'states.provinces' => 'statesandprovinces',
-          'extra.gender' => 'namegender.combine.txt'
+          'extra.gender' => 'namegender.combine.txt',
+          'singleton.predictor' => 'singleton.predictor.ser'
         },
         :german => {},
         :french  => {}
@@ -351,7 +352,7 @@ module StanfordCoreNLP
         'ConstraintAnnotation'
       ],
 
-      'nlp.trees.semgraph.SemanticGraphCoreAnnotations' => [
+      'nlp.semgraph.SemanticGraphCoreAnnotations' => [
         'BasicDependenciesAnnotation',
         'CollapsedCCProcessedDependenciesAnnotation',
         'CollapsedDependenciesAnnotation'
@@ -364,7 +365,8 @@ module StanfordCoreNLP
 
       'nlp.time.TimeExpression' => [
         'Annotation',
-        'ChildrenAnnotation'
+        'ChildrenAnnotation',
+        'TimeIndexAnnotation'
       ],
 
       'nlp.trees.TreeCoreAnnotations' => [

data/lib/stanford-core-nlp/version.rb

@@ -0,0 +1,3 @@
+module StanfordCoreNLP
+  VERSION = '0.5.3'
+end

metadata

@@ -1,91 +1,70 @@
 --- !ruby/object:Gem::Specification
 name: stanford-core-nlp
 version: !ruby/object:Gem::Version
-  version: 0.5.1
-  prerelease:
+  version: 0.5.3
 platform: ruby
 authors:
 - Louis Mullie
-autorequire:
+autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-07 00:00:00.000000000 Z
+date: 2016-12-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bind-it
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 0.2.5
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.2.5
-    none: false
-  prerelease: false
+        version: 0.2.7
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: rspec
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: !binary |-
-          MA==
-    none: false
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ! '>='
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: !binary |-
-          MA==
-    none: false
-  prerelease: false
-  type: :development
-description: " High-level Ruby bindings to the Stanford CoreNLP package, a set natural\
-  \ language processing \ntools that provides tokenization, part-of-speech tagging\
-  \ and parsing for several languages, as well as named entity \nrecognition and coreference\
-  \ resolution for English. "
+        version: 0.2.7
+description: High-level Ruby bindings to the Stanford CoreNLP package, a set natural
+  language processing tools that provides tokenization, part-of-speech tagging and
+  parsing for several languages, as well as named entity recognition and coreference
+  resolution for English, German, French and other languages.
 email:
 - louis.mullie@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- LICENSE
+- README.md
+- bin/AnnotationBridge.java
+- bin/taggers/stanford-postagger-full-2014-10-26/TaggerDemo.java
+- bin/taggers/stanford-postagger-full-2014-10-26/TaggerDemo2.java
 - lib/stanford-core-nlp.rb
 - lib/stanford-core-nlp/bridge.rb
 - lib/stanford-core-nlp/config.rb
-- bin/AnnotationBridge.java
-- bin/bridge.jar
-- README.md
-- LICENSE
+- lib/stanford-core-nlp/version.rb
 homepage: https://github.com/louismullie/stanford-core-nlp
-licenses: []
-post_install_message:
+licenses:
+- GPL-3.0
+metadata: {}
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
-      version: !binary |-
-        MA==
-  none: false
+      version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
-      version: !binary |-
-        MA==
-  none: false
+      version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 1.8.24
-signing_key:
-specification_version: 3
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
 summary: Ruby bindings to the Stanford Core NLP tools.
 test_files: []