logstash-filter-stanford-nlp 0.0.1-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ff7cc127d889fca197a10767b6090798e15c26f3
+  data.tar.gz: 43502d6e32406d82ed13487239bc99054102c733
+SHA512:
+  metadata.gz: 297bb707f2876d2c3f6a38a31725ec814e9dd3d45833cfb1584d73e39d9ea85e7aa8dec59f8f0bf3d5c7fe8931b1137e7dc00ba5275cabae3043914244e25e10
+  data.tar.gz: ce2d3fa01e89d595c89b1f760c66d4e435fe16afd7f78b32cc4973a116102c76208b6a5e13853786def230ad2356aa66471355c04b7c7984d4db958ea7467c7c

data/.github/CONTRIBUTING.md

@@ -0,0 +1,65 @@
+# Contributing to Logstash
+
+All contributions are welcome: ideas, patches, documentation, bug reports,
+complaints, etc!
+
+Programming is not a required skill, and there are many ways to help out!
+It is more important to us that you are able to contribute.
+
+That said, some basic guidelines, which you are free to ignore :)
+
+## Want to learn?
+
+Want to lurk about and see what others are doing with Logstash? 
+
+* The irc channel (#logstash on irc.freenode.org) is a good place for this
+* The [forum](https://discuss.elastic.co/c/logstash) is also
+  great for learning from others.
+
+## Got Questions?
+
+Have a problem you want Logstash to solve for you? 
+
+* You can ask a question in the [forum](https://discuss.elastic.co/c/logstash)
+* Alternately, you are welcome to join the IRC channel #logstash on
+irc.freenode.org and ask for help there!
+
+## Have an Idea or Feature Request?
+
+* File a ticket on [GitHub](https://github.com/elastic/logstash/issues). Please remember that GitHub is used only for issues and feature requests. If you have a general question, the [forum](https://discuss.elastic.co/c/logstash) or IRC would be the best place to ask.
+
+## Something Not Working? Found a Bug?
+
+If you think you found a bug, it probably is a bug.
+
+* If it is a general Logstash or a pipeline issue, file it in [Logstash GitHub](https://github.com/elasticsearch/logstash/issues)
+* If it is specific to a plugin, please file it in the respective repository under [logstash-plugins](https://github.com/logstash-plugins)
+* or ask the [forum](https://discuss.elastic.co/c/logstash).
+
+# Contributing Documentation and Code Changes
+
+If you have a bugfix or new feature that you would like to contribute to
+logstash, and you think it will take more than a few minutes to produce the fix
+(ie; write code), it is worth discussing the change with the Logstash users and developers first! You can reach us via [GitHub](https://github.com/elastic/logstash/issues), the [forum](https://discuss.elastic.co/c/logstash), or via IRC (#logstash on freenode irc)
+Please note that Pull Requests without tests will not be merged. If you would like to contribute but do not have experience with writing tests, please ping us on IRC/forum or create a PR and ask our help.
+
+## Contributing to plugins
+
+Check our [documentation](https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html) on how to contribute to plugins or write your own! It is super easy!
+
+## Contribution Steps
+
+1. Test your changes! [Run](https://github.com/elastic/logstash#testing) the test suite
+2. Please make sure you have signed our [Contributor License
+   Agreement](https://www.elastic.co/contributor-agreement/). We are not
+   asking you to assign copyright to us, but to give us the right to distribute
+   your code without restriction. We ask this of all contributors in order to
+   assure our users of the origin and continuing existence of the code. You
+   only need to sign the CLA once.
+3. Send a pull request! Push your changes to your fork of the repository and
+   [submit a pull
+   request](https://help.github.com/articles/using-pull-requests). In the pull
+   request, describe what your changes do and mention any bugs/issues related
+   to the pull request.
+
+

data/.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,9 @@
+Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.
+
+For all general issues, please provide the following details for fast resolution:
+
+- Version:
+- Operating System:
+- Config File (if you have sensitive info, please remove it):
+- Sample Data:
+- Steps to Reproduce:

data/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1 @@
+Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

data/.gitignore

@@ -0,0 +1,7 @@
+*.gem
+Gemfile.lock
+Gemfile.bak
+.bundle
+vendor
+
+lib/**/*.jar

data/.ruby-env

@@ -0,0 +1 @@
+JRUBY_OPTS=-J-Xmx2048m

data/.ruby-gemset

@@ -0,0 +1 @@
+logstash_filter_stanford_nlp

data/.ruby-version

@@ -0,0 +1 @@
+jruby-1.7.19

data/.travis.yml

@@ -0,0 +1,7 @@
+sudo: false
+language: ruby
+cache: bundler
+rvm:
+  - jruby-1.7.23
+script: 
+  - bundle exec rspec spec

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Pier-Hugues Pellerin (ph)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-filter-example
+Example filter plugin. This should help bootstrap your effort to write your own filter plugin!

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,99 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash). It integrates
+the Logstash with the [Stanford NLP library](http://nlp.stanford.edu/software)
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+The Stanford NLP library relies on ***large*** (~400mb) model files stored in
+JAR files. You will need to increase the size of your Java heap to run the tests
+without crashing.
+
+```sh
+export JRUBY_OPTS="-J-Xmx2048m"
+```
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-stanford-nlp", :path => "/your/local/logstash-filter-nlp"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# need to install a dependency
+ mkdir -p lib/edu/stanford/nlp/stanford-corenlp/3.6.0/ && curl http://nlp.stanford.edu/software/stanford-english-corenlp-2016-01-10-models.jar -o lib/edu/stanford/nlp/stanford-corenlp/3.6.0/stanford-corenlp-3.6.0-models.jar
+
+# Prior to Logstash 2.3 - not supported
+```
+- Run Logstash with your plugin
+
+You may need to increase the heap size of logstash, just prepend LS_HEAP_SIZE=2048m
+to the logstash invocation.
+
+```sh
+bin/logstash -p lib -e 'input { stdin {} } filter { ner {} } output { stdout { codec => rubydebug } }'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# need to install a dependency
+mkdir -p lib/edu/stanford/nlp/stanford-corenlp/3.6.0/ && curl http://nlp.stanford.edu/software/stanford-english-corenlp-2016-01-10-models.jar -o lib/edu/stanford/nlp/stanford-corenlp/3.6.0/stanford-corenlp-3.6.0-models.jar
+
+# Prior to Logstash 2.3 - not supported
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/Rakefile

@@ -0,0 +1 @@
+require "logstash/devutils/rake"

data/lib/logstash-filter-stanford-nlp_jars.rb

@@ -0,0 +1,16 @@
+# this is a generated file, to avoid over-writing it just delete this comment
+require 'jar_dependencies'
+
+require_jar( 'com.io7m.xom', 'xom', '1.2.10' )
+require_jar( 'javax.xml.bind', 'jaxb-api', '2.2.7' )
+require_jar( 'javax.json', 'javax.json-api', '1.0' )
+require_jar( 'org.slf4j', 'slf4j-simple', '1.7.21' )
+require_jar( 'xerces', 'xercesImpl', '2.8.0' )
+require_jar( 'xalan', 'xalan', '2.7.0' )
+require_jar( 'joda-time', 'joda-time', '2.9' )
+require_jar( 'com.googlecode.efficient-java-matrix-library', 'ejml', '0.23' )
+require_jar( 'org.slf4j', 'slf4j-api', '1.7.12' )
+require_jar( 'xml-apis', 'xml-apis', '1.3.03' )
+require_jar( 'edu.stanford.nlp', 'stanford-corenlp', '3.6.0' )
+require_jar( 'de.jollyday', 'jollyday', '0.4.7' )
+require_jar( 'com.google.protobuf', 'protobuf-java', '2.6.1' )

data/lib/logstash/filters/ner.rb

@@ -0,0 +1,45 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash-filter-stanford-nlp_jars.rb"
+require_jar( 'edu.stanford.nlp', 'stanford-corenlp', 'models', '3.6.0' )
+
+
+class LogStash::Filters::Ner < LogStash::Filters::Base
+
+  config_name "ner"
+  config :message, :validate => :string, :default => ""
+
+  module NlpSimple
+    include_package "edu.stanford.nlp.simple"
+  end
+
+  public
+  def register
+  end
+
+  public
+  def filter(event)
+    if @message
+      names = []
+      locations = []
+      organizations = []
+      dates = []
+
+      document = NlpSimple::Document.new(event["message"])
+      document.sentences().each do |sentence|
+        names.concat(sentence.mentions("PERSON"))
+        locations.concat(sentence.mentions("LOCATION"))
+        organizations.concat(sentence.mentions("ORGANIZATION"))
+        dates.concat(sentence.mentions("DATE"))
+      end
+      event["ner.names"] = names
+      event["ner.locations"] = locations
+      event["ner.organizations"] = organizations
+      event["ner.dates"] = dates
+    end
+
+    # filter_matched should go in the last line of our successful code
+    filter_matched(event)
+  end
+end

data/logstash-filter-stanford-nlp.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-filter-stanford-nlp'
+  s.version         = '0.0.1'
+  s.licenses = ['Apache License (2.0)']
+  s.summary = "This filter extracts named entities from the message and adds them as attributes to the message."
+  s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
+  s.authors = ["Jonathan Randall"]
+  s.email = 'jonathar@users.noreply.github.com'
+  s.homepage = "http://www.github.com/rahtanoj/logstash-filter-stanford-nlp"
+  s.require_paths = ["lib"]
+  s.platform = 'java'
+
+  # Files
+  s.files =  `git ls-files -z`.split("\x0")
+
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.requirements << "jar 'edu.stanford.nlp:stanford-corenlp', '3.6.0'"
+  s.requirements << "jar 'org.slf4j:slf4j-simple', '1.7.21'"
+  s.requirements << "jar 'com.google.protobuf:protobuf-java', '2.6.1'"
+  s.add_runtime_dependency "jar-dependencies", "~> 0"
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
+  s.add_development_dependency "logstash-devutils", "= 0.0.19" 
+end

data/spec/filters/ner_spec.rb

@@ -0,0 +1,43 @@
+# encoding: utf-8
+require 'spec_helper'
+require "logstash/filters/ner"
+
+describe LogStash::Filters::Ner do
+  describe "should extract named entities" do
+    let(:config) { {} }
+    subject { described_class.new(config) }
+
+    describe "from the message attribute and expose" do
+      let(:data) { "Jeffrey Alan Mott and Michelle Mott, individuals Dda Integrity Landscape 3756 Independence Avenue Sanger, CA 93637 CSLB#774222 Decision 04/04/2016. Aldan, Inc. P.O. Box 9428, Brea, CA 92822 CSLB #949229 Decision"}
+      let(:event) { LogStash::Event.new("message" => data) }
+
+      it "a list of PERSONs" do
+        subject.register
+        subject.filter(event)
+        expect(event['message']).to eq(data)
+        expect(event['ner.names']).to include('Jeffrey Alan Mott')
+      end
+
+      it "a list of LOCATIONS" do
+        subject.register
+        subject.filter(event)
+        expect(event['message']).to eq(data)
+        expect(event['ner.locations']).to include('Brea')
+      end
+
+      it "a list of ORGANIZATIONs" do
+        subject.register
+        subject.filter(event)
+        expect(event['message']).to eq(data)
+        expect(event['ner.organizations']).to include('Aldan , Inc.')
+      end
+
+      it "a list of DATEs" do
+        subject.register
+        subject.filter(event)
+        expect(event['message']).to eq(data)
+        expect(event['ner.dates']).to include('04/04/2016')
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,119 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-stanford-nlp
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: java
+authors:
+- Jonathan Randall
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2016-05-10 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: jar-dependencies
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.0.19
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.0.19
+description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
+email: jonathar@users.noreply.github.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .github/CONTRIBUTING.md
+- .github/ISSUE_TEMPLATE.md
+- .github/PULL_REQUEST_TEMPLATE.md
+- .gitignore
+- .ruby-env
+- .ruby-gemset
+- .ruby-version
+- .travis.yml
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- Rakefile
+- lib/logstash-filter-stanford-nlp_jars.rb
+- lib/logstash/filters/ner.rb
+- logstash-filter-stanford-nlp.gemspec
+- spec/filters/ner_spec.rb
+- spec/spec_helper.rb
+homepage: http://www.github.com/rahtanoj/logstash-filter-stanford-nlp
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements:
+- jar 'edu.stanford.nlp:stanford-corenlp', '3.6.0'
+- jar 'org.slf4j:slf4j-simple', '1.7.21'
+- jar 'com.google.protobuf:protobuf-java', '2.6.1'
+rubyforge_project:
+rubygems_version: 2.4.5
+signing_key:
+specification_version: 4
+summary: This filter extracts named entities from the message and adds them as attributes to the message.
+test_files:
+- spec/filters/ner_spec.rb
+- spec/spec_helper.rb