logstash-codec-cef 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 87b1c006ad2c6b9f408a1916584c8395aaa1055e
-  data.tar.gz: 3be4bfb1d8b3b2ef05cb22a35adb2f734bfe57a4
+  metadata.gz: aff6387623a45295597fec46cc743482346abc96
+  data.tar.gz: 2f33a701cc5dc39ca149a94e781e68956fa0242a
 SHA512:
-  metadata.gz: cdde709159e93287bdfa22bb4b233d2e5d044132ea29c1a4610e260f3cc5b9a3c6b3b6095bffb8a82cba4dead39eba9553106b2da1d22dbe0ed97da482238697
-  data.tar.gz: cc9f48c5c29e2f922fbe291b50e0ec907e96fcbe47ec32bd8f5e82c3b597e63c4419a3f2dfb9ffef0a81baa003e6aa737c2583cb16b0486749e47f894d8b1e7b
+  metadata.gz: 63891ca7444ab5129cb2d0ac7c374b2be40f50d475fc555791e825d24631e13ee3a9f6add3bf35f2a59a1fdc598bae9239028a820ce74090059120a5db3e34b1
+  data.tar.gz: f8a9479c006edd2385ed9a6b2bf82043c04587b8dce485d7e496bbd01353501415d4539783634206c8d6c89a949c9be249ba84360896f7e615952ce2ce10c7b9

data/.gitignore

@@ -0,0 +1,4 @@
+build
+vendor
+tools
+.VERSION.mk

data/CONTRIBUTORS

@@ -0,0 +1,17 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Colin Surprenant (colinsurprenant)
+* Jason Kendall (coolacid)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Nick Ethier (nickethier)
+* Pete Fritchman (fetep)
+* Pier-Hugues Pellerin (ph)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -1,3 +1,2 @@
 source 'https://rubygems.org'
-gemspec
-gem "logstash", :github => "elasticsearch/logstash", :branch => "1.5"
+gemspec

data/README.md

@@ -22,7 +22,7 @@ Need help? Try #logstash on freenode IRC or the logstash-users@googlegroups.com
 #### Code
 - To get started, you'll need JRuby with the Bundler gem installed.
 
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization.
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
 
 - Install dependencies
 ```sh
@@ -31,26 +31,15 @@ bundle install
 
 #### Test
 
-```sh
-bundle exec rspec
-```
+- Update your dependencies
 
-The Logstash code required to run the tests/specs is specified in the `Gemfile` by the line similar to:
-```ruby
-gem "logstash", :github => "elasticsearch/logstash", :branch => "1.5"
-```
-To test against another version or a local Logstash, edit the `Gemfile` to specify an alternative location, for example:
-```ruby
-gem "logstash", :github => "elasticsearch/logstash", :ref => "master"
-```
-```ruby
-gem "logstash", :path => "/your/local/logstash"
+```sh
+bundle install
 ```
 
-Then update your dependencies and run your tests:
+- Run tests
 
 ```sh
-bundle install
 bundle exec rspec
 ```
 
@@ -58,13 +47,13 @@ bundle exec rspec
 
 #### 2.1 Run in a local Logstash clone
 
-- Edit Logstash `tools/Gemfile` and add the local plugin path, for example:
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
 ```ruby
 gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
 ```
-- Update Logstash dependencies
+- Install plugin
 ```sh
-rake vendor:gems
+bin/plugin install --no-verify
 ```
 - Run Logstash with your plugin
 ```sh
@@ -74,6 +63,8 @@ At this point any modifications to the plugin code will be applied to this local
 
 #### 2.2 Run in an installed Logstash
 
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
 - Build your plugin gem
 ```sh
 gem build logstash-filter-awesome.gemspec
@@ -90,6 +81,6 @@ All contributions are welcome: ideas, patches, documentation, bug reports, compl
 
 Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
 
-It is more important to me that you are able to contribute.
+It is more important to the community that you are able to contribute.
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/codecs/cef.rb

@@ -24,40 +24,65 @@ class LogStash::Codecs::CEF < LogStash::Codecs::Base
     # %{SYSLOGDATE} %{HOST} CEF:Version|Device Vendor|Device Product|Device Version|SignatureID|Name|Severity|Extension
     event = LogStash::Event.new()
     if @syslog
-        @logger.debug("Expecting SYSLOG headers")
-        event['syslog'], data = data.split('CEF:', 1)
-        # Since we have the syslog headers, lets pull them out first and put them into their own field to be handled
+      @logger.debug("Expecting SYSLOG headers")
+      event['syslog'], data = data.split('CEF:', 2)
+      # Since we have the syslog headers, lets pull them out first and put them into their own field to be handled
     else 
-        # We don't have syslog headers, so we just need to remove CEF:
-        data.sub! /^CEF:/, ''
+      # We don't have syslog headers, so we just need to remove CEF:
+      data.sub! /^CEF:/, ''
     end #if @syslog
     # Now, break out the rest of the headers
     event['cef_version'], event['cef_vendor'], event['cef_product'], event['cef_device_version'], event['cef_sigid'], event['cef_name'], event['cef_severity'], event['message'] =  data.scan /(?:[^\|\\]|\\.)+/
+    # Now, try to break out the Extension Dictionary
+    message=event['message']
+    if message.to_s.strip.length != 0
+      message = message.split(/ ([\w\.]+)=/)
+
+      key, value = message.shift.split('=',2)
+      @logger.debug(message)
+      kv = Hash[*message]
+      @logger.debug(kv)
+      addKey(kv,key,value)
+      event.to_hash.merge!(Hash[kv.map{ |k,v| ["cef_ext_"+k,v] }])
+    end #
     yield event
   end
 
   public
   def encode(data)
-	# "CEF:0|Elasticsearch|Logstash|1.0|Signature|Name|Sev|"
+    # "CEF:0|Elasticsearch|Logstash|1.0|Signature|Name|Sev|"
 
-	# TODO: Need to check that fields are set!
+    # TODO: Need to check that fields are set!
 
-	# Signature, Name, and Sev should be set in the config, with ref to fields
-	# Should also probably set the fields sent
-	header = ["CEF:0", "Elasticsearch", "Logstash", "1.0", @signature, @name, @sev].join("|")
-	values = @fields.map {|name| get_value(name, data)}.join(" ")
-	# values = values.map {|k,v| "#{k}=#{v}"}.join(" ")
-	@on_event.call(header + " " + values + "\n")
+    # Signature, Name, and Sev should be set in the config, with ref to fields
+    # Should also probably set the fields sent
+    header = ["CEF:0", "Elasticsearch", "Logstash", "1.0", @signature, @name, @sev].join("|")
+    values = @fields.map {|name| get_value(name, data)}.join(" ")
+    # values = values.map {|k,v| "#{k}=#{v}"}.join(" ")
+    @on_event.call(header + " " + values + "\n")
   end
 
+  private
+  def addKey(kv_keys, key, value)
+    if kv_keys.has_key?(key)
+      if kv_keys[key].is_a? Array
+        kv_keys[key].push(value)
+      else
+        kv_keys[key] = [kv_keys[key], value]
+      end
+    else
+      kv_keys[key] = value
+    end
+  end # addKey
+
   private
   def get_value(name, event)
     val = event[name]
     case val
-      when Hash
-        return name + "=" + val.to_json
-      else
-        return name + "=" + val
+    when Hash
+      return name + "=" + val.to_json
+    else
+      return name + "=" + val
     end
   end
 

data/logstash-codec-cef.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-codec-cef'
-  s.version         = '0.1.1'
+  s.version         = '0.1.3'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "CEF codec to parse CEF formated logs"
   s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
@@ -20,7 +20,7 @@ Gem::Specification.new do |s|
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash', '>= 1.4.0', '< 2.0.0'
+  s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0'
 
   s.add_development_dependency 'logstash-devutils'
 end

data/spec/codecs/cef_spec.rb

@@ -30,7 +30,14 @@ describe LogStash::Codecs::CEF do
       end
     end
 
-    it "should parse the cef body"
+    it "should parse the cef body" do
+      subject.decode(message) do |e|
+        insist { e["cef_ext_src"] } == "10.0.0.192"
+        insist { e["cef_ext_dst"] } == "12.121.122.82"
+        insist { e["cef_ext_spt"] } == "1232"
+      end
+    end
+
     it "should handle values in the body that contain spaces"
   end
 

metadata

@@ -1,18 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: logstash-codec-cef
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - Elasticsearch
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-01-22 00:00:00.000000000 Z
+date: 2015-02-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: logstash
-  version_requirements: !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
@@ -20,7 +19,10 @@ dependencies:
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
-  requirement: !ruby/object:Gem::Requirement
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
@@ -28,29 +30,28 @@ dependencies:
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
-  prerelease: false
-  type: :runtime
 - !ruby/object:Gem::Dependency
-  name: logstash-devutils
-  version_requirements: !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  requirement: !ruby/object:Gem::Requirement
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  prerelease: false
-  type: :development
 description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
 email: info@elasticsearch.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- DEVELOPER.md
+- .gitignore
+- CONTRIBUTORS
 - Gemfile
 - LICENSE
 - README.md
@@ -81,7 +82,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project:
-rubygems_version: 2.1.9
+rubygems_version: 2.4.5
 signing_key:
 specification_version: 4
 summary: CEF codec to parse CEF formated logs

data/DEVELOPER.md

@@ -1,49 +0,0 @@
-logstash-CEF
-============
-
-Logstash Codec to handle CEF encoded data
-
-Build
-=====
-
-Run 'make tarball' to build the project. A tarball will end up in ./build. Extract the file over top of your logstash directory. 
-(Hint: or, just copy the ./lib and ./vendor directories to your logstash folder)
-
-
-Config
-======
-
-This is an example input config. 
-
-```
-input {
-    generator {
-	message => "TODO"
-	count => 1
-	codec => cef
-    }
-}
-
-output {
-    stdout { 
-	codec => "rubydebug"
-    }
-}
-```
-
-This is an example output config. 
-
-```
-input {
-    generator {
-	message => "TODO"
-	count => 1
-    }
-}
-
-output {
-    stdout { 
-	codec => cef
-    }
-}
-```