logstash-output-solr_post 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 79da3dd21bdb0425816f168fce414d11a8763a16
+  data.tar.gz: 8d3aeb443b8f9521ff4f3288e25c59b406285c21
+SHA512:
+  metadata.gz: 0e31426fb6b39ff2da93ba760203fb46fb5988cbcf61527d5847c26a4cf7c87320f9278b4fb235b4025c57cfb6f721f94b07178ca21c70fc69855f7ed25628ac
+  data.tar.gz: cfd045529ea9d0e4a6b0b8cd1db448988eed3c5f7aae3059a8d09ceff8e12ec5cdc544ffc4b92abd185a7eb186f7d3db7528e4810ce77931481df54fdc485252

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/NOTICE.TXT

@@ -0,0 +1 @@
+

data/README.md

@@ -0,0 +1,109 @@
+# Logstash Plugin
+
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Why this plugin
+
+When setting up logging stacks, we weren't able to use the [solr_http](https://rubygems.org/gems/logstash-output-solr_http) output plugin in logstash conf to talk to solr
+On the other hand, we found SiLK 1.5 stack is shipping its own customized version of logstash, which is a java jar file.
+Though the example provided with customized version of [SiLK logstash](https://docs.lucidworks.com/display/SiLK/Solr+Writer+for+Logstash+Initial+Setup) was able to pipe events to solr server, our needs were
+
+1. We already had a logstash server running for a while and it had a great ecosystem of input, filter and output plugins
+2. We would like to use this existing logstash server and NOT have to re-invent the wheel
+3. We want multiple conf files to be loaded by logstash server, which the SiLK 1.5 solrWriter for logstash did not allow as per their documentation
+
+Specifically, we want the same [logstash server](https://www.elastic.co/products/logstash) (that elastic search provides) to work with solr. It should be as simple as install a plugin, write the output conf and see the pipeline work, by viewing the events on the other side of solr  
+
+## Installing and using  
+
+1. git clone 
+2. JRUBY >=1.7 (We tried with MRI Ruby, however that did not go well. Guess because logstash is developed in JRUBY)
+3. If you are on windows, use pik and switch to use jruby (pik use <identifier>). If on *nix, use rvm to switch to jruby
+4. jgem build logstash-output-solr_post.gemspec (output should be a gem like logstash-output-solr_post-0.1.gem
+5. Copy this gem to ~/logstash root folder
+6. Install as output plugin to logstash using "./bin/plugin install logstash-output-solr_post-0.1.gem"
+7. ./bin/plugin list to verify that the plugin installed successfully and registered as output plugin
+8. See the next section on usage instructions
+
+## How does this plugin do it  
+
+It is possible to do a simple http POST to solr end point and post documents. Hence, we wrote simple logic extending logstash output base class and POST the event to solr endpoint. Of course, since solr expects the http post payload to be in a certain format, we format the event and then directly do a POST. That is it! 
+
+## Working Example  
+
+### Solr  
+Version: 5.3.1  
+**Note:** We tried starting solr in schemaless mode (gettingstarted collection automatically created), however we did not have success posting events directly to http://localhost:8983/solr/update?commit=true&wt=json endpoint
+
+### Logstash (2.0.0) conf
+```
+input {
+  tcp {
+    type => "eventlog"
+    port => 5544
+    codec=> "json"
+    }
+  }
+output {
+  solr_post{
+    solr_url => "http://localhost:8983/solr/mycollection/update?commit=true&wt=json"
+    }
+  }
+```  
+where 'mycollection' is the name of your collection. We require collection name in the url   
+
+### Windows nxlog (nxlog-ce-2.9.1347) conf
+```
+<Extension json>
+  Module xm_json
+</Extension>
+
+<Extension xml>
+  Module xm_xml
+</Extension>
+
+<Input in>
+    Module im_msvistalog
+  Exec $raw_event=to_json(); 
+</Input>
+
+<Output out>
+  Module om_tcp
+  Host 127.0.0.1
+  Port 5544
+</Output>
+
+<Route r>
+  Path in => out
+</Route>
+```  
+
+We have used the above conf and were able to use this plugin to pipe events through the pipeline. We were also able to view the events in Banana. Below is how it looked for us  
+
+![Banana Dashboard](https://github.com/machzqcq/logstash-output-solr_post/blob/master/images/banana_dashboard.JPG "Banana Dashboard")  
+
+## TODO
+
+1. Use idle flush time, flush size & documents parameters (currently it is declared inside the code but not used)
+2. Write rspec unit tests
+3. Support for XML (depends on need for the community)
+4. Register this plugin for logstash plugin binary
+5. Miscellaneous
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/lib/logstash/outputs/solr_post.rb

@@ -0,0 +1,71 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "stud/buffer"
+require "rubygems"
+require "uuidtools"
+require "net/http"
+require "uri"
+require "json"
+
+# You can learn more at https://lucene.apache.org/solr/[the Solr home page]
+
+class LogStash::Outputs::SolrPost < LogStash::Outputs::Base
+  include Stud::Buffer
+
+  config_name "solr_post"
+
+
+  # URL used to connect to Solr
+  config :solr_url, :validate => :string
+
+  # Number of events to queue up before writing to Solr
+  config :flush_size, :validate => :number, :default => 100
+
+  # Amount of time since the last flush before a flush is done even if
+  # the number of buffered events is smaller than flush_size
+  config :idle_flush_time, :validate => :number, :default => 1
+
+  # Solr document ID for events. You'd typically have a variable here, like
+  # '%{foo}' so you can assign your own IDs
+  config :document_id, :validate => :string, :default => nil
+
+  public
+  def register
+    buffer_initialize(
+      :max_items => @flush_size,
+      :max_interval => @idle_flush_time,
+      :logger => @logger
+    )
+  end #def register
+
+  public
+  def receive(event)
+    
+    buffer_receive(event)
+  end #def receive
+
+  public
+  def flush(events, close=false)
+    # The documents array below is NOT being used currently, however the next version will utilize this.
+    documents = []  #this is the array of hashes that we push to Solr as documents
+
+    events.each do |event|
+        document = event.to_hash()
+        document["@timestamp"] = document["@timestamp"].iso8601 #make the timestamp ISO
+        if @document_id.nil?
+          document["id"] = UUIDTools::UUID.random_create    #add a unique ID
+        else
+          document["id"] = event.sprintf(@document_id)      #or use the one provided
+        end
+        url = URI.parse(@solr_url)
+        req = Net::HTTP::Post.new(@solr_url, initheader = {'Content-Type' =>'application/json'})
+        req.body = '{add:{ doc:' + JSON.generate(document) + ',boost:1.0,overwrite:true,commitWithin:1000}}'
+        res = Net::HTTP.start(url.host,url.port) do |http|
+          http.request(req)
+        end
+    end
+    rescue Exception => e
+      @logger.warn("An error occurred while indexing: #{e.message}")
+  end #def flush
+end #class LogStash::Outputs::SolrPost

data/logstash-output-solr_post.gemspec

@@ -0,0 +1,31 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-output-solr_post'
+  s.version         = '0.1'
+  s.licenses        = ['MIT']
+  s.email           = ['pradeep@seleniumframework.com']
+  s.licenses        = ['MIT']
+  s.summary         = "This output lets you index&store your logs in Solr."
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Pradeep K. Macharla","Venkatesh R. Peruvemba"]
+  s.homepage        = "https://github.com/machzqcq/logstash-output-solr_post.git"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
+
+  s.add_runtime_dependency 'stud'
+  s.add_runtime_dependency 'uuidtools'
+
+  s.add_development_dependency 'logstash-devutils'
+end
+

data/spec/outputs/solr_post_spec.rb

@@ -0,0 +1 @@
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,116 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-solr_post
+version: !ruby/object:Gem::Version
+  version: '0.1'
+platform: ruby
+authors:
+- Pradeep K. Macharla
+- Venkatesh R. Peruvemba
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2015-12-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: stud
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: uuidtools
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email:
+- pradeep@seleniumframework.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/logstash/outputs/solr_post.rb
+- spec/outputs/solr_post_spec.rb
+- logstash-output-solr_post.gemspec
+- README.md
+- Gemfile
+- NOTICE.TXT
+homepage: https://github.com/machzqcq/logstash-output-solr_post.git
+licenses:
+- MIT
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.1.9
+signing_key:
+specification_version: 4
+summary: This output lets you index&store your logs in Solr.
+test_files:
+- spec/outputs/solr_post_spec.rb