logstash-codec-cef 3.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a841a39cd9b28c0794c9cb08e8295c5eea3bb521
+  data.tar.gz: 9d6a5196496f277d25bf0064da05499c8240f2e1
+SHA512:
+  metadata.gz: 1517164e07a5e5d89d1a2a768b9d0dd676c3175cbf6fbe40742e41d9fd8f163f24d15605dae02d1c2e241be9815e8de6fc6e617812fbb076038512f85af82a11
+  data.tar.gz: 32de38bce405918a55b762161d1d1ebd603660caeeeb5b0f86fbce8f5ee9b7320f49b961ec13b6714163a75201582d872238b6198fa574f1abe547b682e3dd43

data/CHANGELOG.md

@@ -0,0 +1,21 @@
+## 3.0.0
+ - breaking: Updated plugin to use new Java Event APIs
+
+## 2.1.3
+ - Switch in-place sub! to sub when extracting `cef_version`. new Logstash Java Event does not support in-place String changes.
+
+## 2.1.2
+ - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
+
+## 2.1.1
+ - New dependency requirements for logstash-core for the 5.0 release
+
+## 2.1.0
+ - Implements `encode` with escaping according to the [CEF specification](https://protect724.hp.com/docs/DOC-1072).
+ - Config option `sev` is deprecated, use `severity` instead.
+
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,22 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Maintainers:
+* Lucas Bremgartner (breml)
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Colin Surprenant (colinsurprenant)
+* Jason Kendall (coolacid)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Nick Ethier (nickethier)
+* Pete Fritchman (fetep)
+* Pier-Hugues Pellerin (ph)
+* Karl Stoney (Stono)
+* Lucas Bremgartner (breml)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,107 @@
+# Logstash Plugin
+
+[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-codec-cef.svg)](https://travis-ci.org/logstash-plugins/logstash-codec-cef)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing with Docker
+You can use a docker container with all of the requirements pre installed to save you installing the development environment on your host.
+
+### 1. Starting the container
+Simply type `docker-compose run devenv` and you'll be entered into the container.  Then you'll need to do `jruby -S bundle install` to get all the dependencies down.
+
+### 2. Running tests
+Once you've done #1 above, you can run your tests with `jruby -S bundle exec rspec`
+
+## Developing without Docker
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+# Logstash 2.3 and higher
+bin/logstash-plugin install --no-verify
+
+# Prior to Logstash 2.3
+bin/plugin install --no-verify
+
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/codecs/cef.rb

@@ -0,0 +1,234 @@
+# encoding: utf-8
+require "logstash/codecs/base"
+require "json"
+
+# Implementation of a Logstash codec for the ArcSight Common Event Format (CEF)
+# Based on Revision 20 of Implementing ArcSight CEF, dated from June 05, 2013
+# https://protect724.hp.com/servlet/JiveServlet/downloadBody/1072-102-6-4697/CommonEventFormat.pdf
+class LogStash::Codecs::CEF < LogStash::Codecs::Base
+  config_name "cef"
+
+  # Device vendor field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  config :vendor, :validate => :string, :default => "Elasticsearch"
+
+  # Device product field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  config :product, :validate => :string, :default => "Logstash"
+
+  # Device version field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  config :version, :validate => :string, :default => "1.0"
+
+  # Signature ID field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  config :signature, :validate => :string, :default => "Logstash"
+
+  # Name field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  config :name, :validate => :string, :default => "Logstash"
+
+  # Deprecated severity field for CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  #
+  # This field is used only if :severity is unchanged set to the default value.
+  #
+  # Defined as field of type string to allow sprintf. The value will be validated
+  # to be an integer in the range from 0 to 10 (including).
+  # All invalid values will be mapped to the default of 6.
+  config :sev, :validate => :string, :default => "6", :deprecated => "This setting is being deprecated, use :severity instead."
+
+  # Severity field in CEF header. The new value can include `%{foo}` strings
+  # to help you build a new value from other parts of the event.
+  #
+  # Defined as field of type string to allow sprintf. The value will be validated
+  # to be an integer in the range from 0 to 10 (including).
+  # All invalid values will be mapped to the default of 6.
+  config :severity, :validate => :string, :default => "6"
+
+  # Fields to be included in CEV extension part as key/value pairs
+  config :fields, :validate => :array, :default => []
+
+  HEADER_FIELDS = ['cef_version','cef_vendor','cef_product','cef_device_version','cef_sigid','cef_name','cef_severity']
+  
+  public
+  def initialize(params={})
+    super(params)
+  end
+
+  private
+  def store_header_field(event,field_name,field_data)
+    #Unescape pipes and backslash in header fields
+    event.set(field_name,field_data.gsub(/\\\|/, '|').gsub(/\\\\/, '\\')) unless field_data.nil?
+  end
+
+  public
+  def decode(data)
+    # Strip any quotations at the start and end, flex connectors seem to send this
+    if data[0] == "\""
+      data = data[1..-2]
+    end
+    event = LogStash::Event.new
+
+    # Split by the pipes, pipes in the extension part are perfectly valid and do not need escaping
+    # The better solution for the splitting regex would be /(?<!\\(\\\\)*)[\|]/, but this
+    # gives an "SyntaxError: (RegexpError) invalid pattern in look-behind" for the variable length look behind.
+    # Therefore one edge case is not handled properly: \\| (this should split, but it does not, because the escaped \ is not recognized)
+    # TODO: To solve all unescaping cases, regex is not suitable. A little parse should be written.
+    split_data = data.split /(?<=[^\\]\\\\)[\|]|(?<!\\)[\|]/
+
+    # Store header fields
+    HEADER_FIELDS.each_with_index do |field_name, index|
+      store_header_field(event,field_name,split_data[index])
+    end
+    #Remainder is message
+    message = split_data[HEADER_FIELDS.size..-1].join('|')
+
+    # Try and parse out the syslog header if there is one
+    if event.get('cef_version').include? ' '
+      split_cef_version= event.get('cef_version').rpartition(' ')
+      event.set('syslog', split_cef_version[0])
+      event.set('cef_version',split_cef_version[2])
+    end
+
+    # Get rid of the CEF bit in the version
+    event.set('cef_version', event.get('cef_version').sub(/^CEF:/, ''))
+
+    # Strip any whitespace from the message
+    if not message.nil? and message.include? '='
+      message = message.strip
+
+      # If the last KVP has no value, add an empty string, this prevents hash errors below
+      if message.end_with?('=')
+        message=message + ' ' unless message.end_with?('\=')
+      end
+
+      # Now parse the key value pairs into it
+      extensions = {}
+      message = message.split(/ ([\w\.]+)=/)
+      key, value = message.shift.split('=', 2)
+      extensions[key] = value.gsub(/\\=/, '=').gsub(/\\\\/, '\\')
+      Hash[*message].each{ |k, v| extensions[k] = v }
+      # And save the new has as the extensions
+      event.set('cef_ext', extensions)
+    end
+
+    yield event
+  end
+
+  public
+  def encode(event)
+    # "CEF:0|Elasticsearch|Logstash|1.0|Signature|Name|Sev|"
+
+    vendor = sanitize_header_field(event.sprintf(@vendor))
+    vendor = self.class.get_config["vendor"][:default] if vendor == ""
+
+    product = sanitize_header_field(event.sprintf(@product))
+    product = self.class.get_config["product"][:default] if product == ""
+
+    version = sanitize_header_field(event.sprintf(@version))
+    version = self.class.get_config["version"][:default] if version == ""
+
+    signature = sanitize_header_field(event.sprintf(@signature))
+    signature = self.class.get_config["signature"][:default] if signature == ""
+
+    name = sanitize_header_field(event.sprintf(@name))
+    name = self.class.get_config["name"][:default] if name == ""
+
+    # :sev is deprecated and therefore only considered if :severity equals the default setting or is invalid
+    severity = sanitize_severity(event, @severity)
+    if severity == self.class.get_config["severity"][:default]
+      # Use deprecated setting sev
+      severity = sanitize_severity(event, @sev)
+    end
+
+    # Should also probably set the fields sent
+    header = ["CEF:0", vendor, product, version, signature, name, severity].join("|")
+    values = @fields.map {|fieldname| get_value(fieldname, event)}.compact.join(" ")
+
+    @on_event.call(event, "#{header}|#{values}\n")
+  end
+
+  private
+
+  # Escape pipes and backslashes in the header. Equal signs are ok.
+  # Newlines are forbidden.
+  def sanitize_header_field(value)
+    output = ""
+
+    value = value.to_s.gsub(/\r\n/, "\n")
+
+    value.each_char{|c|
+      case c
+      when "\\", "|"
+        output += "\\" + c
+      when "\n", "\r"
+        output += " "
+      else
+        output += c
+      end
+    }
+
+    return output
+  end
+
+  # Keys must be made up of a single word, with no spaces
+  # must be alphanumeric
+  def sanitize_extension_key(value)
+    value = value.to_s.gsub(/[^a-zA-Z0-9]/, "")
+    return value
+  end
+
+  # Escape equal signs in the extensions. Canonicalize newlines.
+  # CEF spec leaves it up to us to choose \r or \n for newline.
+  # We choose \n as the default.
+  def sanitize_extension_val(value)
+    output = ""
+
+    value = value.to_s.gsub(/\r\n/, "\n")
+
+    value.each_char{|c|
+      case c
+      when "\\", "="
+        output += "\\" + c
+      when "\n", "\r"
+        output += "\\n"
+      else
+        output += c
+      end
+    }
+
+    return output
+  end
+
+  def get_value(fieldname, event)
+    val = event.get(fieldname)
+
+    return nil if val.nil?
+
+    case val
+    when Array, Hash
+      return "#{sanitize_extension_key(fieldname)}=#{sanitize_extension_val(val.to_json)}"
+    when LogStash::Timestamp
+      return "#{sanitize_extension_key(fieldname)}=#{val.to_s}"
+    else
+      return "#{sanitize_extension_key(fieldname)}=#{sanitize_extension_val(val)}"
+    end
+  end
+
+  def sanitize_severity(event, severity)
+    severity = sanitize_header_field(event.sprintf(severity)).strip
+    severity = self.class.get_config["severity"][:default] unless valid_severity?(severity)
+    severity = severity.to_i.to_s
+  end
+
+  def valid_severity?(sev)
+    f = Float(sev)
+    # check if it's an integer or a float with no remainder
+    # and if the value is between 0 and 10 (inclusive)
+    (f % 1 == 0) && f.between?(0,10)
+  rescue TypeError, ArgumentError
+    false
+  end
+
+end

data/logstash-codec-cef.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-codec-cef'
+  s.version         = '3.0.0'
+  s.platform        = 'java'
+  s.licenses        = ['Apache License (2.0)']
+  s.summary         = "CEF codec to parse and encode CEF formated logs"
+  s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Elastic"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+
+  s.add_development_dependency 'logstash-devutils'
+end
+

data/spec/codecs/cef_spec.rb

@@ -0,0 +1,487 @@
+# encoding: utf-8
+
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/codecs/cef"
+require "logstash/event"
+require "json"
+
+describe LogStash::Codecs::CEF do
+  subject do
+    next LogStash::Codecs::CEF.new
+  end
+
+  context "#encode" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    let(:results)   { [] }
+
+    it "should not fail if fields is nil" do
+      codec.on_event{|data, newdata| results << newdata}
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should assert all header fields are present" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should use default values for empty header fields" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.vendor = ""
+      codec.product = ""
+      codec.version = ""
+      codec.signature = ""
+      codec.name = ""
+      codec.severity = ""
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should use configured values for header fields" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.vendor = "vendor"
+      codec.product = "product"
+      codec.version = "2.0"
+      codec.signature = "signature"
+      codec.name = "name"
+      codec.severity = "1"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|vendor\|product\|2.0\|signature\|name\|1\|$/m)
+    end
+
+    it "should use sprintf for header fields" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.vendor = "%{vendor}"
+      codec.product = "%{product}"
+      codec.version = "%{version}"
+      codec.signature = "%{signature}"
+      codec.name = "%{name}"
+      codec.severity = "%{severity}"
+      codec.fields = []
+      event = LogStash::Event.new("vendor" => "vendor", "product" => "product", "version" => "2.0", "signature" => "signature", "name" => "name", "severity" => "1")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|vendor\|product\|2.0\|signature\|name\|1\|$/m)
+    end
+
+    it "should use default, if severity is not numeric" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.severity = "foo"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should use default, if severity is > 10" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.severity = "11"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should use default, if severity is < 0" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.severity = "-1"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should use default, if severity is float with decimal part" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.severity = "5.4"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+
+    it "should append fields as key/value pairs in cef extension part" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo", "bar" ]
+      event = LogStash::Event.new("foo" => "foo value", "bar" => "bar value")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=foo value bar=bar value$/m)
+    end
+
+    it "should ignore fields in fields if not present in event" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo", "bar", "baz" ]
+      event = LogStash::Event.new("foo" => "foo value", "baz" => "baz value")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=foo value baz=baz value$/m)
+    end
+
+    it "should sanitize header fields" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.vendor = "ven\ndor"
+      codec.product = "pro|duct"
+      codec.version = "ver\\sion"
+      codec.signature = "sig\r\nnature"
+      codec.name = "na\rme"
+      codec.severity = "4\n"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|ven dor\|pro\\\|duct\|ver\\\\sion\|sig nature\|na me\|4\|$/m)
+    end
+
+    it "should sanitize extension keys" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "f o\no", "@b-a_r" ]
+      event = LogStash::Event.new("f o\no" => "foo value", "@b-a_r" => "bar value")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=foo value bar=bar value$/m)
+    end
+
+    it "should sanitize extension values" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo", "bar", "baz" ]
+      event = LogStash::Event.new("foo" => "foo\\value\n", "bar" => "bar=value\r")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=foo\\\\value\\n bar=bar\\=value\\n$/m)
+    end
+
+    it "should encode a hash value" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo" ]
+      event = LogStash::Event.new("foo" => { "bar" => "bar value", "baz" => "baz value" })
+      codec.encode(event)
+      foo = results.first[/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=(.*)$/, 1]
+      expect(foo).not_to be_nil
+      foo_hash = JSON.parse(foo)
+      expect(foo_hash).to eq({"bar" => "bar value", "baz" => "baz value"})
+    end
+
+    it "should encode an array value" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo" ]
+      event = LogStash::Event.new("foo" => [ "bar", "baz" ])
+      codec.encode(event)
+      foo = results.first[/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=(.*)$/, 1]
+      expect(foo).not_to be_nil
+      foo_array = JSON.parse(foo)
+      expect(foo_array).to eq(["bar", "baz"])
+    end
+
+    it "should encode a hash in an array value" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo" ]
+      event = LogStash::Event.new("foo" => [ { "bar" => "bar value" }, "baz" ])
+      codec.encode(event)
+      foo = results.first[/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=(.*)$/, 1]
+      expect(foo).not_to be_nil
+      foo_array = JSON.parse(foo)
+      expect(foo_array).to eq([{"bar" => "bar value"}, "baz"])
+    end
+
+    it "should encode a LogStash::Timestamp" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.fields = [ "foo" ]
+      event = LogStash::Event.new("foo" => LogStash::Timestamp.new)
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|foo=[0-9TZ.:-]+$/m)
+    end
+
+    it "should use severity (instead of depricated sev), if severity is set)" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.sev = "4"
+      codec.severity = "5"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|5\|$/m)
+    end
+
+    it "should use deprecated sev, if severity is not set (equals default value)" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.sev = "4"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
+    end
+
+    it "should use deprecated sev, if severity is explicitly set to default value)" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.sev = "4"
+      codec.severity = "6"
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
+    end
+
+    it "should use deprecated sev, if severity is invalid" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.sev = "4"
+      codec.severity = ""
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|4\|$/m)
+    end
+
+    it "should use default value, if severity is not set and sev is invalid" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.sev = ""
+      codec.fields = []
+      event = LogStash::Event.new("foo" => "bar")
+      codec.encode(event)
+      expect(results.first).to match(/^CEF:0\|Elasticsearch\|Logstash\|1.0\|Logstash\|Logstash\|6\|$/m)
+    end
+  end
+
+  context "sanitize header field" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    it "should sanitize" do
+      expect(codec.send(:sanitize_header_field, "foo")).to be == "foo"
+      expect(codec.send(:sanitize_header_field, "foo\nbar")).to be == "foo bar"
+      expect(codec.send(:sanitize_header_field, "foo\rbar")).to be == "foo bar"
+      expect(codec.send(:sanitize_header_field, "foo\r\nbar")).to be == "foo bar"
+      expect(codec.send(:sanitize_header_field, "foo\r\nbar\r\nbaz")).to be == "foo bar baz"
+      expect(codec.send(:sanitize_header_field, "foo\\bar")).to be == "foo\\\\bar"
+      expect(codec.send(:sanitize_header_field, "foo|bar")).to be == "foo\\|bar"
+      expect(codec.send(:sanitize_header_field, "foo=bar")).to be == "foo=bar"
+      expect(codec.send(:sanitize_header_field, 123)).to be == "123" # Input value is a Fixnum
+      expect(codec.send(:sanitize_header_field, 123.123)).to be == "123.123" # Input value is a Float
+      expect(codec.send(:sanitize_header_field, [])).to be == "[]" # Input value is an Array
+      expect(codec.send(:sanitize_header_field, {})).to be == "{}" # Input value is a Hash
+    end
+  end
+
+  context "sanitize extension key" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    it "should sanitize" do
+      expect(codec.send(:sanitize_extension_key, " foo ")).to be == "foo"
+      expect(codec.send(:sanitize_extension_key, " FOO 123 ")).to be == "FOO123"
+      expect(codec.send(:sanitize_extension_key, "foo\nbar\rbaz")).to be == "foobarbaz"
+      expect(codec.send(:sanitize_extension_key, "Foo_Bar\r\nBaz")).to be == "FooBarBaz"
+      expect(codec.send(:sanitize_extension_key, "foo-@bar=baz")).to be == "foobarbaz"
+      expect(codec.send(:sanitize_extension_key, "[foo]|bar.baz")).to be == "foobarbaz"
+      expect(codec.send(:sanitize_extension_key, 123)).to be == "123" # Input value is a Fixnum
+      expect(codec.send(:sanitize_extension_key, 123.123)).to be == "123123" # Input value is a Float, "." is not allowed and therefore removed
+      expect(codec.send(:sanitize_extension_key, [])).to be == "" # Input value is an Array, "[" and "]" are not allowed and therefore removed
+      expect(codec.send(:sanitize_extension_key, {})).to be == "" # Input value is a Hash, "{" and "}" are not allowed and therefore removed
+    end
+  end
+
+  context "sanitize extension value" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    it "should sanitize" do
+      expect(codec.send(:sanitize_extension_val, "foo")).to be == "foo"
+      expect(codec.send(:sanitize_extension_val, "foo\nbar")).to be == "foo\\nbar"
+      expect(codec.send(:sanitize_extension_val, "foo\rbar")).to be == "foo\\nbar"
+      expect(codec.send(:sanitize_extension_val, "foo\r\nbar")).to be == "foo\\nbar"
+      expect(codec.send(:sanitize_extension_val, "foo\r\nbar\r\nbaz")).to be == "foo\\nbar\\nbaz"
+      expect(codec.send(:sanitize_extension_val, "foo\\bar")).to be == "foo\\\\bar"
+      expect(codec.send(:sanitize_extension_val, "foo|bar")).to be == "foo|bar"
+      expect(codec.send(:sanitize_extension_val, "foo=bar")).to be == "foo\\=bar"
+      expect(codec.send(:sanitize_extension_val, 123)).to be == "123" # Input value is a Fixnum
+      expect(codec.send(:sanitize_extension_val, 123.123)).to be == "123.123" # Input value is a Float
+      expect(codec.send(:sanitize_extension_val, [])).to be == "[]" # Input value is an Array
+      expect(codec.send(:sanitize_extension_val, {})).to be == "{}" # Input value is a Hash
+    end
+  end
+
+  context "valid_severity?" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    it "should validate severity" do
+      expect(codec.send(:valid_severity?, nil)).to be == false
+      expect(codec.send(:valid_severity?, "")).to be == false
+      expect(codec.send(:valid_severity?, "foo")).to be == false
+      expect(codec.send(:valid_severity?, "1.5")).to be == false
+      expect(codec.send(:valid_severity?, "-1")).to be == false
+      expect(codec.send(:valid_severity?, "11")).to be == false
+      expect(codec.send(:valid_severity?, "0")).to be == true
+      expect(codec.send(:valid_severity?, "10")).to be == true
+      expect(codec.send(:valid_severity?, "1.0")).to be == true
+      expect(codec.send(:valid_severity?, 1)).to be == true
+      expect(codec.send(:valid_severity?, 1.0)).to be == true
+    end
+  end
+
+  context "#decode" do
+    let (:message) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
+
+    def validate(e) 
+      insist { e.is_a?(LogStash::Event) }
+      insist { e.get('cef_version') } == "0"
+      insist { e.get('cef_device_version') } == "1.0"
+      insist { e.get('cef_sigid') } == "100"
+      insist { e.get('cef_name') } == "trojan successfully stopped"
+      insist { e.get('cef_severity') } == "10"
+    end
+
+    it "should parse the cef headers" do
+      subject.decode(message) do |e|
+        validate(e)
+        ext = e.get('cef_ext')
+        insist { e.get("cef_vendor") } == "security"
+        insist { e.get("cef_product") } == "threatmanager"
+      end
+    end
+
+    it "should parse the cef body" do
+      subject.decode(message) do |e|
+        ext = e.get('cef_ext')
+        insist { ext['src'] } == "10.0.0.192"
+        insist { ext['dst'] } == "12.121.122.82"
+        insist { ext['spt'] } == "1232"
+      end
+    end
+
+    let (:no_ext) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|" }
+    it "should be OK with no extension dictionary" do
+      subject.decode(no_ext) do |e|
+        validate(e)
+        insist { e.get("cef_ext") } == nil
+      end 
+    end
+
+    let (:missing_headers) { "CEF:0|||1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
+    it "should be OK with missing CEF headers (multiple pipes in sequence)" do
+      subject.decode(missing_headers) do |e|
+        validate(e)
+        insist { e.get("cef_vendor") } == ""
+        insist { e.get("cef_product") } == ""
+      end 
+    end
+
+    let (:leading_whitespace) { "CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10| src=10.0.0.192 dst=12.121.122.82 spt=1232" }
+    it "should strip leading whitespace from the message" do
+      subject.decode(leading_whitespace) do |e|
+        validate(e)
+      end 
+    end
+
+    let (:escaped_pipes) { 'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this\|has an escaped pipe' }
+    it "should be OK with escaped pipes in the message" do
+      subject.decode(escaped_pipes) do |e|
+        ext = e.get('cef_ext')
+        insist { ext['moo'] } == 'this\|has an escaped pipe'
+      end 
+    end
+
+    let (:pipes_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this|has an pipe'}
+    it "should be OK with not escaped pipes in the message" do
+      subject.decode(pipes_in_message) do |e|
+        ext = e.get('cef_ext')
+        insist { ext['moo'] } == 'this|has an pipe'
+      end
+    end
+
+    let (:escaped_equal_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this \=has escaped \= equals\='}
+    it "should be OK with escaped equal in the message" do
+      subject.decode(escaped_equal_in_message) do |e|
+        ext = e.get('cef_ext')
+        insist { ext['moo'] } == 'this =has escaped = equals='
+      end
+    end
+
+    let (:escaped_backslash_in_header) {'CEF:0|secu\\\\rity|threat\\\\manager|1.\\\\0|10\\\\0|tro\\\\jan successfully stopped|\\\\10|'}
+    it "should be OK with escaped backslash in the headers" do
+      subject.decode(escaped_backslash_in_header) do |e|
+        insist { e.get("cef_version") } == '0'
+        insist { e.get("cef_vendor") } == 'secu\\rity'
+        insist { e.get("cef_product") } == 'threat\\manager'
+        insist { e.get("cef_device_version") } == '1.\\0'
+        insist { e.get("cef_sigid") } == '10\\0'
+        insist { e.get("cef_name") } == 'tro\\jan successfully stopped'
+        insist { e.get("cef_severity") } == '\\10'
+      end
+    end
+
+    let (:escaped_backslash_in_header_edge_case) {'CEF:0|security\\\\\\||threatmanager\\\\|1.0|100|trojan successfully stopped|10|'}
+    it "should be OK with escaped backslash in the headers (edge case: escaped slash in front of pipe)" do
+      subject.decode(escaped_backslash_in_header_edge_case) do |e|
+        validate(e)
+        insist { e.get("cef_vendor") } == 'security\\|'
+        insist { e.get("cef_product") } == 'threatmanager\\'
+      end
+    end
+
+    let (:escaped_pipes_in_header) {'CEF:0|secu\\|rity|threatmanager\\||1.\\|0|10\\|0|tro\\|jan successfully stopped|\\|10|'}
+    it "should be OK with escaped pipes in the headers" do
+      subject.decode(escaped_pipes_in_header) do |e|
+        insist { e.get("cef_version") } == '0'
+        insist { e.get("cef_vendor") } == 'secu|rity'
+        insist { e.get("cef_product") } == 'threatmanager|'
+        insist { e.get("cef_device_version") } == '1.|0'
+        insist { e.get("cef_sigid") } == '10|0'
+        insist { e.get("cef_name") } == 'tro|jan successfully stopped'
+        insist { e.get("cef_severity") } == '|10'
+      end
+    end
+
+    let (:escaped_backslash_in_message) {'CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|moo=this \\\\has escaped \\\\ backslashs\\\\'}
+    it "should be OK with escaped backslashs in the message" do
+      subject.decode(escaped_backslash_in_message) do |e|
+        ext = e.get('cef_ext')
+        insist { ext['moo'] } == 'this \\has escaped \\ backslashs\\'
+      end
+    end
+
+    let (:equal_in_header) {'CEF:0|security|threatmanager=equal|1.0|100|trojan successfully stopped|10|'}
+    it "should be OK with equal in the headers" do
+      subject.decode(equal_in_header) do |e|
+        validate(e)
+        insist { e.get("cef_product") } == "threatmanager=equal"
+      end
+    end
+
+    let (:syslog) { "Syslogdate Sysloghost CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232" }
+    it "Should detect headers before CEF starts" do
+      subject.decode(syslog) do |e|
+        validate(e)
+        insist { e.get('syslog') } == 'Syslogdate Sysloghost'
+      end 
+    end
+  end
+
+  context "encode and decode" do
+    subject(:codec) { LogStash::Codecs::CEF.new }
+
+    let(:results)   { [] }
+
+    it "should return an equal event if encoded and decoded again" do
+      codec.on_event{|data, newdata| results << newdata}
+      codec.vendor = "%{cef_vendor}"
+      codec.product = "%{cef_product}"
+      codec.version = "%{cef_device_version}"
+      codec.signature = "%{cef_sigid}"
+      codec.name = "%{cef_name}"
+      codec.severity = "%{cef_severity}"
+      codec.fields = [ "foo" ]
+      event = LogStash::Event.new("cef_vendor" => "vendor", "cef_product" => "product", "cef_device_version" => "2.0", "cef_sigid" => "signature", "cef_name" => "name", "cef_severity" => "1", "foo" => "bar")
+      codec.encode(event)
+      codec.decode(results.first) do |e|
+        expect(e.get('cef_vendor')).to be == event.get('cef_vendor')
+        expect(e.get('cef_product')).to be == event.get('cef_product')
+        expect(e.get('cef_device_version')).to be == event.get('cef_device_version')
+        expect(e.get('cef_sigid')).to be == event.get('cef_sigid')
+        expect(e.get('cef_name')).to be == event.get('cef_name')
+        expect(e.get('cef_severity')).to be == event.get('cef_severity')
+        # decode saves extensions as hash to 'cef_ext'
+        expect(e.get('[cef_ext][foo]')).to be == event.get('foo')
+      end
+    end
+  end
+
+end

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification
+name: logstash-codec-cef
+version: !ruby/object:Gem::Version
+  version: 3.0.0
+platform: java
+authors:
+- Elastic
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2016-09-22 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
+email: info@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/codecs/cef.rb
+- logstash-codec-cef.gemspec
+- spec/codecs/cef_spec.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: codec
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: CEF codec to parse and encode CEF formated logs
+test_files:
+- spec/codecs/cef_spec.rb