logstash-filter-edn 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 3dd1567b8932d848fe00c0b0f6202363666578c3
+  data.tar.gz: b3a2895db4c74c7c52295015e3783b70ce460fbe
+SHA512:
+  metadata.gz: 43b571d82db0e6cef985630d1c02def78b76ec623a9351f3e6a26b32a91b4990df55afd1edf48432a50353eb578c71dd4d7f14e7540823c2d398647af2e1c9a4
+  data.tar.gz: 93a9114102302f496b71361d475daca5312329346f6d28022fa4d1ad7b69c4c3729539cc0002f6187b53b434ffeab9e841bcad88a1d8be7f3c130f500cca60e5

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Evan Niessen-Derry - eniessenderry@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,3 @@
+# logstash-filter-edn
+
+Convert EDN data into JSON data. Similar in structure to `logstash-filter-json`.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/edn.rb

@@ -0,0 +1,112 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash/json"
+require "edn"
+require "json"
+
+# This is an EDN parsing filter which takes an existing field containing EDN
+# and expands it into a data structure within the Logstash event. It takes a
+# lot of inspiration from the JSON filter.
+#
+# By default, it will place the parsed EDN in the root of the Logstash event,
+# but you can specify any arbitrary event field using the `target`
+# configuration option.
+#
+# 
+class LogStash::Filters::EDN < LogStash::Filters::Base
+
+  config_name "edn"
+
+  # The configuration options for the EDN filter:
+  # [source, ruby]
+  #     source => source_field, required
+  #     target => target_field, optional
+  #
+  # Which field contains the EDN data?
+  # [source, ruby]
+  # -------------------------------
+  # filter {
+  #   edn {
+  #     source => "message"
+  #   }
+  # }
+  # -------------------------------
+  config :source, :validate => :string, :required => true
+
+  # Where should we store the parsed data? If this setting is omitted, the EDN
+  # data will be store at the top level of the event.
+  #
+  # NB: This is required if the EDN data isn't a map
+  #
+  # [source, ruby]
+  # -------------------------------
+  # filter {
+  #   edn {
+  #     source => "message"
+  #     target => "edn"
+  #   }
+  # }
+  # -------------------------------
+  #
+  # NB: The `target` field will be overwitten if it exists
+  config :target, :validate => :string
+
+  # Array of values to append to the event tags upon parsing failure
+  config :tag_on_failure, :validate => :array, :default => [ "_ednparsefailure" ]
+
+  # Should we pass the event through without error/tagging upon parsing failure?
+  config :skip_on_invalid_edn, :validate => :boolean, :default => false
+
+  public
+  def register
+    # This space intentionally empty
+  end
+
+  public
+  def filter(event)
+
+    @logger.debug? && @logger.debug("Running EDN filter", :event => event)
+
+    # Let's grab the field we're parsing; bail if it's not there
+    source = event.get(@source)
+    return unless source
+
+    begin
+      # Since ES needs this in JSON, we're going to first parse the EDN to Ruby
+      # values, then into JSON values
+      parsed = LogStash::Json.load(EDN.read(source).to_json)
+    rescue Exception => e
+      unless @skip_on_invalid_edn
+        @tag_on_failure.each{|tag| event.tag(tag)}
+        @logger.warn("Error parsing edn", :source => @source, :raw => source, :exception => e)
+      end
+      return
+    end
+
+    # The user told us where they'd like the data, so let's put it there
+    if @target
+      event.set(@target, parsed)
+    else
+      # If we don't have a hash, then we don't have keys to tie the values to.
+      # We could guess about the key to tie them to, but instead we'll just
+      # force people to specify one.
+      unless parsed.is_a?(Hash)
+        @tag_on_failure.each{|tag| event.tag(tag)}
+        @logger.warn("Non-map EDN values require a :target to be specified; exiting EDN filter.", :source => @source, :raw => source)
+        return
+      end
+
+      # Persist every parsed field into the event
+      # JSON doesn't have Symbol/Keyword types, so we convert them to JSON.
+      parsed.each{|k, v| event.set(k, v)}
+    end
+
+    @logger.debug? && @logger.debug("Event post EDN filtering", :event => event)
+
+    # filter_matched should go in the last line of our successful code
+    filter_matched(event)
+  end # def filter
+end # class LogStash::Filters::Edn
+
+# vim: set et ts=2 sw=2 :

data/logstash-filter-edn.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-filter-edn'
+  s.version       = '0.1.0'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'A Logstash filter for EDN data.'
+  s.homepage      = "https://github.com/kidblog/logstash-filter-edn"
+  s.authors       = ['Evan Niessen-Derry']
+  s.email         = 'eniessenderry@gmail.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.1"
+  s.add_development_dependency 'logstash-devutils', "~> 1.3"
+  #s.add_development_dependency 'pry-byebug', '~>3.6'
+  #s.add_development_dependency 'pry-debugger', '~>0.2'
+  s.add_development_dependency 'ruby-debug', '~> 0.10'
+
+  # Our particular dependencies
+  s.add_runtime_dependency 'edn', "~> 1.1"
+end

data/spec/filters/edn_spec.rb

@@ -0,0 +1,80 @@
+# encoding: utf-8
+require_relative '../spec_helper'
+require "logstash/filters/edn"
+
+describe LogStash::Filters::EDN do
+  # Parses an event's source and puts the parsed data on the top level if no
+  # :target
+  describe "parse EDN into event" do
+    config <<-CONFIG
+      filter {
+        edn {
+          # Parse message as edn
+          source => "message"
+        }
+      }
+    CONFIG
+
+    sample '{:hello "world", "list" [1 2 3], :map { "foo" "bar" }, 1 "one cookie"}' do
+      #debugger
+      insist { subject.get("hello") } == "world"
+      insist { subject.get("list").to_a } == [1,2,3]
+      insist { subject.get("map") } == { "foo"=>"bar" }
+      insist { subject.get("1") } == "one cookie"
+    end
+  end
+
+  # Puts the parsed data under :target if specified
+  # Throws an error if passed non-map data and no :target spec'd
+  describe "Put parsed EDN into target field" do
+    config <<-CONFIG
+      filter {
+        edn {
+          # Parse message as edn
+          source => "message"
+          target => "edn"
+        }
+      }
+    CONFIG
+
+    sample '{:hello "world"}' do
+      insist { subject.get("edn") } == LogStash::Json.load('{"hello":"world"}')
+    end
+  end
+
+  # Throws an error if passed non-map data and no :target spec'd
+  describe "Put parsed EDN into target field" do
+    config <<-CONFIG
+      filter {
+        edn {
+          # Parse message as edn
+          source => "message"
+        }
+      }
+    CONFIG
+
+    sample '{:hello}' do
+      expect(subject.get('tags')).to include("_ednparsefailure")
+    end
+  end
+
+  # Throws an error if passed non-map data and no :target spec'd
+  # except not if :skip_on_invalid_edn
+  describe "Put parsed EDN into target field" do
+    config <<-CONFIG
+      filter {
+        edn {
+          # Parse message as edn
+          source => "message"
+          skip_on_invalid_edn => true
+        }
+      }
+    CONFIG
+
+    sample '{:hello}' do
+      expect(subject.get('tags')).to be_nil
+    end
+  end
+end
+
+# vim: set et ts=2 sw=2 :

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,113 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-edn
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Evan Niessen-Derry
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2018-03-21 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+  name: ruby-debug
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  name: edn
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+description:
+email: eniessenderry@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/filters/edn.rb
+- logstash-filter-edn.gemspec
+- spec/filters/edn_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/kidblog/logstash-filter-edn
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.14.1
+signing_key:
+specification_version: 4
+summary: A Logstash filter for EDN data.
+test_files:
+- spec/filters/edn_spec.rb
+- spec/spec_helper.rb