logstash-codec-loginsight 0.1.48

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ff605c480d5ebf65f598c5012b05cce22be807ab
+  data.tar.gz: 5d49e4b6ab13c92be8e91cc94300ecca8773a55f
+SHA512:
+  metadata.gz: 94eb279833bd8387916f0e10f7639948870db278a28e4d0108de9f4ee90231be03dc1a8c64974e3fc122e0f89c4e712b4c8f849e70a06c0c6268593b4640bfcb
+  data.tar.gz: c961ed872c861ceab49d18dfdde26b0809856a413f7d37358eb6fe69f4e8ee2266a73204fb2cd7b41d99951922f9707770d23676bc98c781d76f533c9826cde0

data/CONTRIBUTORS

@@ -0,0 +1,14 @@
+Contributors:
+* Alan J Castonguay (alanjcastonguay)
+
+Inspired by logstash-codec-json, from these fine people:
+
+Contributors:
+* Colin Surprenant (colinsurprenant)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Kurt Hurtado (kurtado)
+* Nick Ethier (nickethier)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Tal Levy (talevy)

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,123 @@
+# logstash-codec-loginsight
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash), converting events received from [VMware vRealize Log Insight](https://www.vmware.com/support/pubs/log-insight-pubs.html) via [logstash-input-http](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Installation from rubygems
+
+[logstash-codec-loginsight](http://rubygems.org/gems/logstash-codec-loginsight) is hosted on rubygems.org. [Download and install the latest gem](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html) in your Logstash deployment:
+
+```sh
+bin/logstash-plugin install logstash-codec-loginsight
+```
+
+Verify installed version:
+```sh
+bin/logstash-plugin list --verbose logstash-codec-loginsight
+logstash-codec-loginsight (x.y.z)
+```
+
+## Usage
+
+The codec is designed to be chained with the `logstash-input-http` plugin. Log events are forwarded from Log Insight with CFAPI on port 9000 (non-ssl) or 9543 (ssl), targetting and instance of `logstash-input-http` configured with additional codec.
+
+```
+input {
+  http {
+    port=>9000
+    additional_codecs=>{ "application/json" => "loginsight"}
+  }
+}
+```
+
+| option | default | notes |
+| --- | --- | --- |
+| `charset`  | `UTF-8` | The character encoding used in this codec.
+
+## AsciiDocs
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running the local, unpublished plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-codec-loginsight", :path => "/your/local/logstash-codec-loginsight"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash --debug --log.level=debug -e 'input { http {port=>9000 additional_codecs=>{ "application/json" => "loginsight"} } } output { stdout {codec=>rubydebug} }'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/codecs/loginsight.rb

@@ -0,0 +1,106 @@
+# encoding: utf-8
+# Copyright © 2017 VMware, Inc. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+require "logstash/codecs/base"
+require "logstash/util/charset"
+require "logstash/json"
+require "logstash/event"
+require "logstash/timestamp"
+
+# This codec may be used to decode (via inputs) JSON-encoded messages,
+# with timestamp and key=value fields, from VMware vRealize Log Insight via CFAPI.
+#
+# If this codec recieves a payload from an input that is not valid JSON,
+# then it it will add a tag `_jsonparsefailure` with the payload stored
+# in the `message` field.
+#
+# If this codec receives a payload from an input that is valid JSON,
+# but which is not structured as Log Insight is expected to produce,
+# then it will add a tag `_eventparsefailure` with the payload stored
+# in the `message` field.
+#
+# input { http { port=>9000 additional_codecs=>{ "application/json" => "loginsight"} } }
+
+class LogStash::Codecs::LogInsight < LogStash::Codecs::Base
+  config_name "loginsight"
+
+  # The character encoding used in this codec. Examples include "UTF-8" and
+  # "CP1252".
+  config :charset, :validate => ::Encoding.name_list, :default => "UTF-8"
+
+  def register
+    @logger.info("Log Insight Codec Registered")
+    @converter = LogStash::Util::Charset.new(@charset)
+    @converter.logger = @logger
+  end
+
+  def decode(data, &block)
+    parse(@converter.convert(data), &block)
+  end
+
+  private
+  
+  def map_loginsight_messages(item, &block)
+    # map the fields and values to a hash, preserving an existing `message`
+    begin
+      event_hash = Hash[item["fields"].map { |f| [f["name"], f["content"]] }]
+      if event_hash.has_key?("message")
+        event_hash["_message"] = event_hash["message"]
+      end
+    rescue
+      event_hash = {}
+    end
+
+    # Add the message field, so it overwrites the above
+    event = LogStash::Event.new(event_hash.merge("message" => item["text"]))
+
+    # Make a Timestamp object from the source message (UTC Epoch in Milliseconds).
+    begin
+      newtime = LogStash::Timestamp.at( item["timestamp"].to_i / 1000.0 )
+      event.set( LogStash::Event::TIMESTAMP, newtime )
+    rescue
+      event.set( LogStash::Event::TIMESTAMP_FAILURE_FIELD, item["timestamp"])
+    end
+
+    yield event
+  end
+
+  def parse(json, &block)
+    decoded = LogStash::Json.load(json)
+
+    case decoded
+    when Hash
+      @logger.debug("parse handling hash", :decoded => decoded)
+
+      if decoded.has_key?("messages") and decoded["messages"].kind_of?(Array)
+        decoded["messages"].each { |item|
+          begin
+            map_loginsight_messages(item, &block)
+          rescue StandardError => e
+            @logger.error("Event parse failure.", :error => e, :item => item)
+            yield LogStash::Event.new("message" => item, "tags" => ["_eventparsefailure"])
+          end
+        }  # end decoded["message"].each
+      else
+        @logger.error("Log Insight codec expects 'messages' to contain an array", :data => json)
+        yield LogStash::Event.new("message" => decoded, "tags" => ["_eventparsefailure"])
+      end
+    else
+      @logger.error("Log Insight expeects a hash containing 'messages'", :data => json)
+      yield LogStash::Event.new("message" => decoded, "tags" => ["_eventparsefailure"])
+    end
+  rescue LogStash::Json::ParserError => e
+    @logger.error("JSON parse failure.", :error => e, :data => json)
+    yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
+  rescue StandardError => e
+    # This should NEVER happen.
+    @logger.warn(
+      "An unexpected error occurred parsing JSON data",
+      :data => json,
+      :message => e.message,
+      :class => e.class.name,
+      :backtrace => e.backtrace
+    )
+  end
+end

data/logstash-codec-loginsight.gemspec

@@ -0,0 +1,26 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-codec-loginsight'
+  s.version         = '0.1.48'
+  s.licenses        = ['Apache-2.0']
+  s.summary         = "This codec may be used to decode (via inputs) and encode (via outputs) full JSON messages"
+  s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Alan J Castonguay"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths   = ["lib"]
+
+  # Files
+  s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/codecs/loginsight_spec.rb

@@ -0,0 +1,166 @@
+# encoding: utf-8
+# Copyright © 2017 VMware, Inc. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/codecs/loginsight"
+require "logstash/event"
+require "logstash/json"
+require "logstash/timestamp"
+
+describe LogStash::Codecs::LogInsight do
+  subject do
+    LogStash::Codecs::LogInsight.new
+  end
+
+  context "#map_loginsight_messages" do
+    it "works" do
+      message = {"text" => "message body"}
+
+      expect { |b|
+        subject.send(:map_loginsight_messages, message, &b)
+      }.to yield_control.exactly(1).times
+
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+      end
+    end
+
+    it "honors existing milliseconds-since-epoch timestamp" do
+      message = {"timestamp" => "5000"}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("@timestamp")).to be_a(LogStash::Timestamp)
+        expect(event.get("@timestamp").to_i).to eql(5)  # 5000 ms == 5 seconds
+      end
+    end
+
+    it "adds missing timestamp" do
+      message = {"text" => "message body"}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("@timestamp")).to be_a(LogStash::Timestamp)
+        expect(event.get("message")).to eql("message body")
+      end
+    end
+
+    it "maps arbitrary fields" do
+      message = {"text"=>"message body", "fields"=>[{"name"=>"hostname", "content"=>"172.16.44.1", "startPosition"=>-1, "length"=>-2147483648}, {"name"=>"extratag", "content"=>"5", "startPosition"=>-1, "length"=>-2147483648}, {"name"=>"__li_source_path", "content"=>"172.16.44.1", "startPosition"=>-1, "length"=>-2147483648}]}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("@timestamp")).to be_a(LogStash::Timestamp)
+        expect(event.get("message")).to eql("message body")
+        expect(event.get("hostname")).to eql("172.16.44.1")
+        expect(event.get("__li_source_path")).to eql("172.16.44.1")
+        expect(event.get("extratag")).to eql("5")
+      end
+    end
+
+    it "preserves an existing 'message' field" do
+      message = {"text"=>"message body", "fields"=>[{"name"=>"message", "content"=>"message tag"}]}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("message")).to eql(message["text"])
+        expect(event.get("_message")).to eql(message["fields"][0]["content"]["message tag"])
+      end
+    end
+
+    it "handles missing 'message' field" do
+      message = {"fields"=>[{"name"=>"foo", "content"=>"bar"}]}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("message")).to eql(nil)
+      end
+    end
+
+    it "ignores other fields" do
+      message = {"foo"=>"bar"}
+      subject.send(:map_loginsight_messages, message) do |event|
+        expect(event).to be_a(LogStash::Event)
+        expect(event.get("message")).to eql(nil)
+        expect(event).not_to include("foo")
+      end
+    end
+
+  end
+
+  context "#decode" do
+
+    it "silently yields no events for an empty input messages list" do
+      data = {"messages" => []}
+      expect { |b|
+        subject.decode(LogStash::Json.dump(data), &b)
+      }.to yield_control.exactly(0).times
+    end
+
+    describe "passes through unknown, but valid json" do
+      shared_examples "given" do |value_arg|
+        context "where input is '#{value_arg}'" do
+          let(:value) { value_arg }
+          let(:event) do
+            e = nil
+            subject.decode(LogStash::Json.dump(value)) do |decoded|
+              e = decoded
+            end
+            e
+          end
+
+          it "stores the value in 'message'" do
+            expect(event.get("message")).to eql(value)
+          end
+
+          it "adds the _eventparsefailure tag" do
+            expect(event.get("tags")).to include("_eventparsefailure")
+          end
+        end
+      end
+
+      include_examples "given", 123
+      include_examples "given", "scalar"
+      include_examples "given", "-1"
+      include_examples "given", " "
+      include_examples "given", {"foo" => "bar"}
+      include_examples "given", [{"foo" => "bar"}]
+      include_examples "given", {"foo" => "bar", "baz" => {"quux" => ["a","b","c"]}}
+
+    end
+
+    describe "cannot parse json" do
+      shared_examples "json" do |value_arg|
+        context "where input is '#{value_arg}'" do
+          let(:value) { value_arg }
+          let(:event) do
+            e = nil
+            subject.decode(value) do |decoded|
+              e = decoded
+            end
+            e
+          end
+
+          it "stores the value in 'message'" do
+            expect(event.get("message")).to eql(value)
+          end
+
+          it "adds the failure tag" do
+            expect(event).to include "tags"
+          end
+
+          it "uses an array to store the tags" do
+            expect(event.get("tags")).to be_a Array
+          end
+
+          it "adds the _jsonparsefailure tag" do
+            expect(event.get("tags")).to include("_jsonparsefailure")
+          end
+        end
+      end
+
+      include_examples "json", "scalar"
+      include_examples "json", 'random_{message'
+      include_examples "json", "{"
+      include_examples "json", "[{]"
+    end
+
+  end
+
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: logstash-codec-loginsight
+version: !ruby/object:Gem::Version
+  version: 0.1.48
+platform: ruby
+authors:
+- Alan J Castonguay
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-05-16 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
+email: info@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/codecs/loginsight.rb
+- logstash-codec-loginsight.gemspec
+- spec/codecs/loginsight_spec.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: codec
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.8
+signing_key:
+specification_version: 4
+summary: This codec may be used to decode (via inputs) and encode (via outputs) full JSON messages
+test_files:
+- spec/codecs/loginsight_spec.rb