logstash-filter-dissect 1.0.12 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: edba4758d6e759bae681c335eec23281c0bc8505
-  data.tar.gz: 7cb793e36f94954a21540f6b5e2f584826ee4223
+SHA256:
+  metadata.gz: c56d7ab014406bf2a21c22cb3f3d40bd09dba45fd3b5d26db58fed30461b56c4
+  data.tar.gz: 1985e43ac6547d4493fb1493ba3f7065a314b769a3de5a8d7a6b99d60453d059
 SHA512:
-  metadata.gz: e4fa09ae61de294335c0eda1c88cc07bb41e672ff50a81dbd69d8175354f9b225c1bbe5a79ad3bd7a35757c158daae51c639d0a3041a2470561fdac245d8c17e
-  data.tar.gz: 9c9dfe6585d8b6a20114d9ba73c49a0d7df252f0fff297dc39f1808fb00ef51e60b3d67738b24fbbe211bc57d584f5143b4ee1b929c075323d293e1a9b8642d8
+  metadata.gz: ae8a49d90ef97ff73010debef4334404fa2b5c994f81388eb8f4de5f9d4849d7bd6afee722dba8afddc3440c3f034176498a15163d092aeb799a1f9ecf294960
+  data.tar.gz: 2d8d5e3db0c3131866b54123a64f64b441e5c5433f398a19bae4a757fda45bec60b2dd487a6a9e7eca91ce08851234771ac14afab1134c0f09e550c0cc900ce7

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 1.1.1
+  - Fix for "Missing field values cause dissected fields to be out of position" issue. See updated documentation.
+  - Fix for "Check empty fields" issue, empty fields handled better.
+  - Fix for "Integer conversion does not handle big integers".
+  
 ## 1.0.12
   - Fix some documentation issues
 

data/Gemfile

@@ -2,7 +2,7 @@ source 'https://rubygems.org'
 
 gemspec
 
-logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash" # <- is where travis has the source
 use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
 
 if Dir.exist?(logstash_path) && use_logstash_source

data/VERSION

@@ -1 +1 @@
-1.0.12
+1.1.1

data/docs/index.asciidoc

@@ -20,10 +20,12 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
-The Dissect filter is a kind of split operation. Unlike a regular split operation where one delimiter is applied to the whole string, this operation applies a set of delimiters # to a string value. +
+The Dissect filter is a kind of split operation. Unlike a regular split operation where one delimiter is applied to
+the whole string, this operation applies a set of delimiters to a string value. +
 Dissect does not use regular expressions and is very fast. +
 However, if the structure of your text varies from line to line then Grok is more suitable. +
-There is a hybrid case where Dissect can be used to de-structure the section of the line that is reliably repeated and then Grok can be used on the remaining field values with # more regex predictability and less overall work to do. +
+There is a hybrid case where Dissect can be used to de-structure the section of the line that is reliably repeated and
+then Grok can be used on the remaining field values with more regex predictability and less overall work to do. +
 
 A set of fields and delimiters is called a *dissection*.
 
@@ -34,10 +36,10 @@ The dissection is described using a set of `%{}` sections:
 
 A *field* is the text from `%` to `}` inclusive.
 
-A *delimiter* is the text between `}` and `%` characters.
+A *delimiter* is the text between a `}` and next `%{` characters.
 
 [NOTE]
-delimiters can't contain these `}{%` characters.
+Any set of characters that do not fit `%{`, `'not }'`, `}` pattern is a delimiter.
 
 The config might look like this:
 ....
@@ -49,7 +51,8 @@ The config might look like this:
     }
   }
 ....
-When dissecting a string from left to right, text is captured upto the first delimiter - this captured text is stored in the first field. This is repeated for each field/# delimiter pair thereafter until the last delimiter is reached, then *the remaining text is stored in the last field*. +
+When dissecting a string from left to right, text is captured upto the first delimiter - this captured text is stored in the first field.
+This is repeated for each field/# delimiter pair thereafter until the last delimiter is reached, then *the remaining text is stored in the last field*. +
 
 *The Key:* +
 The key is the text between the `%{` and `}`, exclusive of the ?, +, & prefixes and the ordinal suffix. +
@@ -57,7 +60,7 @@ The key is the text between the `%{` and `}`, exclusive of the ?, +, & prefixes
 `%{+bbb/3}` - key is `bbb` +
 `%{&ccc}` - key is `ccc` +
 
-*Normal field notation:* +
+===== Normal field notation
 The found value is added to the Event using the key. +
 `%{some_field}` - a normal field has no prefix or suffix
 
@@ -69,7 +72,7 @@ The key, if supplied, is prefixed with a `?`.
 
 `%{?foo}` is a named skip field.
 
-*Append field notation:* +
+===== Append field notation
 The value is appended to another value or stored if its the first field seen. +
 The key is prefixed with a `+`. +
 The final value is stored in the Event using the key. +
@@ -88,7 +91,7 @@ e.g. for a text of `1 2 3 go`, this `%{+a/2} %{+a/1} %{+a/4} %{+a/3}` will build
 Append fields without an order modifier will append in declared order. +
 e.g. for a text of `1 2 3 go`, this `%{a} %{b} %{+a}` will build two key/values of `a => 1 3 go, b => 2` +
 
-*Indirect field notation:* +
+===== Indirect field notation
 The found value is added to the Event using the found value of another field as the key. +
 The key is prefixed with a `&`. +
 `%{&some_field}` - an indirect field where the key is indirectly sourced from the value of `some_field`. +
@@ -109,20 +112,87 @@ append and indirect cannot be combined and will fail validation. +
 `%{&+something}` will add a value to the `+something` key, again probably unintended. +
 ===============================
 
-*Delimiter repetition:* +
-In the source text if a field has variable width padded with delimiters, the padding will be ignored. +
-e.g. for texts of:
+==== Multiple Consecutive Delimiter Handling
+
+[IMPORTANT]
+===============================
+Starting from version 1.1.1 of this plugin, multiple found delimiter handling has changed.
+Now multiple consecutive delimiters will be seen as missing fields by default and not padding.
+If you are already using Dissect and your source text has fields padded with extra delimiters,
+you will need to change your config. Please read the section below.
+===============================
+
+===== Empty data between delimiters
+Given this text as the sample used to create a dissection:
+....
+John Smith,Big Oaks,Wood Lane,Hambledown,Canterbury,CB34RY
+....
+The created dissection, with 6 fields, is:
+....
+%{name},%{addr1},%{addr2},%{addr3},%{city},%{zip}
+....
+When a line like this is processed:
 ....
-00000043 ViewReceiver  I
-000000b3 Peer          I
+Jane Doe,4321 Fifth Avenue,,,New York,87432
+....
+Dissect will create an event with empty fields for `addr2 and addr3` like so:
+....
+{
+  "name": "Jane Doe",
+  "addr1": "4321 Fifth Avenue",
+  "addr2": "",
+  "addr3": "",
+  "city": "New York"
+  "zip": "87432"
+}
 ....
-with a dissection of `%{a} %{b} %{c}`; the padding is ignored, `event.get([c]) -> "I"`
 
-[NOTE]
-====
+===== Delimiters used as padding to visually align fields
+*Padding to the right hand side*
+
+Given these texts as the samples used to create a dissection:
+....
+00000043 ViewReceive     machine-321
+f3000a3b Calc            machine-123
+....
+The dissection, with 3 fields, is:
+....
+%{id} %{function->} %{server}
+....
+Note, above, the second field has a `->` suffix which tells Dissect to ignore padding to its right. +
+Dissect will create these events:
+....
+{
+  "id": "00000043",
+  "function": "ViewReceive",
+  "server": "machine-123"
+}
+{
+  "id": "f3000a3b",
+  "function": "Calc",
+  "server": "machine-321"
+}
+....
+[IMPORTANT]
+Always add the `->` suffix to the field on the left of the padding.
+
+*Padding to the left hand side (to the human eye)*
+
+Given these texts as the samples used to create a dissection:
+....
+00000043     ViewReceive machine-321
+f3000a3b            Calc machine-123
+....
+The dissection, with 3 fields, is now:
+....
+%{id->} %{function} %{server}
+....
+Here the `->` suffix moves to the `id` field because Dissect sees the padding as being to the right of the `id` field. +
+
+==== Conditional processing
+
 You probably want to use this filter inside an `if` block. +
 This ensures that the event contains a field value with a suitable structure for the dissection.
-====
 
 For example...
 ....
@@ -156,7 +226,7 @@ filter plugins.
  
 
 [id="plugins-{type}s-{plugin}-convert_datatype"]
-===== `convert_datatype` 
+===== `convert_datatype`
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
@@ -177,7 +247,7 @@ filter {
 }
 
 [id="plugins-{type}s-{plugin}-mapping"]
-===== `mapping` 
+===== `mapping`
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
@@ -200,7 +270,7 @@ This is useful if you want to keep the field `description` but also
 dissect it some more.
 
 [id="plugins-{type}s-{plugin}-tag_on_failure"]
-===== `tag_on_failure` 
+===== `tag_on_failure`
 
   * Value type is <<array,array>>
   * Default value is `["_dissectfailure"]`
@@ -210,4 +280,4 @@ Append values to the `tags` field when dissection fails
 
 
 [id="plugins-{type}s-{plugin}-common-options"]
-include::{include_path}/{type}.asciidoc[]
+include::{include_path}/{type}.asciidoc[]

data/lib/jruby-dissect-library_jars.rb

@@ -1,4 +1,4 @@
 # AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.
 
 require 'jar_dependencies'
-require_jar('org.logstash.dissect', 'jruby-dissect-library', '1.0.12')
+require_jar('org.logstash.dissect', 'jruby-dissect-library', '1.1.1')

data/lib/logstash/filters/dissect.rb

@@ -168,22 +168,35 @@ module LogStash module Filters class Dissect < LogStash::Filters::Base
   public
 
   def register
-    @dissector = LogStash::Dissector.new(@mapping)
+    needs_decoration = @add_field.size + @add_tag.size + @remove_field.size + @remove_tag.size > 0
+    @dissector = LogStash::Dissector.new(@mapping, self, @convert_datatype, needs_decoration)
   end
 
   def filter(event)
     # all plugin functions happen in the JRuby extension:
     # debug, warn and error logging, filter_matched, tagging etc.
-    @dissector.dissect(event, self)
+    @dissector.dissect(event)
   end
 
   def multi_filter(events)
     LogStash::Util.set_thread_plugin(self)
-    @dissector.dissect_multi(events, self)
+    @dissector.dissect_multi(events)
     events
   end
 
+  # this method is stubbed during testing
+  # a reference to it in the JRuby Extension `initialize` may not be valid
   def metric_increment(metric_name)
     metric.increment(metric_name)
   end
+
+  # the JRuby Extension `initialize` method stores a DynamicMethod reference to this method
+  def increment_matches_metric
+    metric_increment(:matches)
+  end
+
+  # the JRuby Extension `initialize` method stores a DynamicMethod reference to this method
+  def increment_failures_metric
+    metric_increment(:failures)
+  end
 end end end

data/spec/filters/dissect_spec.rb

@@ -3,59 +3,50 @@ require 'spec_helper'
 require "logstash/filters/dissect"
 
 describe LogStash::Filters::Dissect do
-  class LoggerMock
-    attr_reader :msgs, :hashes
-    def initialize()
-      @msgs = []
-      @hashes = []
-    end
-
-    def error(*msg)
-      @msgs.push(msg[0])
-      @hashes.push(msg[1])
-    end
-
-    def warn(*msg)
-      @msgs.push(msg[0])
-      @hashes.push(msg[1])
-    end
-
-    def debug?() true; end
 
-    def debug(*msg)
-      @msgs.push(msg[0])
-      @hashes.push(msg[1])
-    end
-
-    def fatal(*msg)
-      @msgs.push(msg[0])
-      @hashes.push(msg[1])
+  describe "Basic dissection" do
+    let(:config) do <<-CONFIG
+      filter {
+        dissect {
+          mapping => {
+            message => "[%{occurred_at}] %{code} %{service->} %{ic} %{svc_message}"
+          }
+        }
+      }
+    CONFIG
     end
 
-    def trace(*msg)
-      @msgs.push(msg[0])
-      @hashes.push(msg[1])
+    sample("message" => "[25/05/16 09:10:38:425 BST] 00000001 SystemOut     O java.lang:type=MemoryPool,name=class storage") do
+      expect(subject.get("occurred_at")).to eq("25/05/16 09:10:38:425 BST")
+      expect(subject.get("code")).to eq("00000001")
+      expect(subject.get("service")).to eq("SystemOut")
+      expect(subject.get("ic")).to eq("O")
+      expect(subject.get("svc_message")).to eq("java.lang:type=MemoryPool,name=class storage")
+      expect(subject.get("tags")).to be_nil
     end
   end
 
-  describe "Basic dissection" do
+  describe "Basic dissection, like CSV with missing fields" do
     let(:config) do <<-CONFIG
       filter {
         dissect {
           mapping => {
-            message => "[%{occurred_at}] %{code} %{service} %{ic} %{svc_message}"
+            message => '[%{occurred_at}] %{code} %{service} values: "%{v1}","%{v2}","%{v3}"%{rest}'
           }
         }
       }
     CONFIG
     end
 
-    sample("message" => "[25/05/16 09:10:38:425 BST] 00000001 SystemOut     O java.lang:type=MemoryPool,name=class storage") do
+    sample("message" => '[25/05/16 09:10:38:425 BST] 00000001 SystemOut values: "f1","","f3"') do
       expect(subject.get("occurred_at")).to eq("25/05/16 09:10:38:425 BST")
       expect(subject.get("code")).to eq("00000001")
       expect(subject.get("service")).to eq("SystemOut")
-      expect(subject.get("ic")).to eq("O")
-      expect(subject.get("svc_message")).to eq("java.lang:type=MemoryPool,name=class storage")
+      expect(subject.get("v1")).to eq("f1")
+      expect(subject.get("v2")).to eq("")
+      expect(subject.get("v3")).to eq("f3")
+      expect(subject.get("rest")).to eq("")
+      expect(subject.get("tags")).to be_nil
     end
   end
 
@@ -118,32 +109,82 @@ describe LogStash::Filters::Dissect do
           "mapping" => {"message" => "[%{occurred_at}] %{code} %{service} %{?ic}=%{&ic}% %{svc_message}"},
           "convert_datatype" => {
             "ccu" => "float", # ccu field -> nil
-            "code" => "integer", # only int is supported
             "other" => "int" # other field -> hash - not coercible
           }
       }
     end
     let(:event)      { LogStash::Event.new("message" => message, "other" => {}) }
-    let(:loggr)      { LoggerMock.new }
-
-    before(:each) do
-      filter.class.instance_variable_set("@logger", loggr)
-    end
 
     it "tags and log messages are created" do
       filter.register
       filter.filter(event)
       expect(event.get("code")).to eq("00000001")
-      expect(event.get("tags")).to eq(["_dataconversionnullvalue_ccu_float", "_dataconversionmissing_code_integer", "_dataconversionuncoercible_other_int"])
-      expect(loggr.msgs).to eq(
-          [
-              "Event before dissection",
-              "Dissector datatype conversion, value cannot be coerced, key: ccu, value: null",
-              "Dissector datatype conversion, datatype not supported: integer",
-              "Dissector datatype conversion, value cannot be coerced, key: other, value: {}",
-              "Event after dissection"
-          ]
-      )
+      tags = event.get("tags")
+      expect(tags).to include("_dataconversionnullvalue_ccu_float")
+      expect(tags).to include("_dataconversionuncoercible_other_int")
+      # Logging moved to java can't mock ruby logger anymore
+    end
+  end
+
+  describe "Basic dissection when the source field does not exist" do
+    let(:event) { LogStash::Event.new("message" => "foo", "other" => {}) }
+    subject(:filter) {  LogStash::Filters::Dissect.new(config)  }
+    let(:config)     do
+      {
+        "mapping" => {"msg" => "[%{occurred_at}] %{code} %{service} %{?ic}=%{&ic}% %{svc_message}"},
+      }
+    end
+    it "does not raise an error" do
+      filter.register
+      expect(subject).to receive(:metric_increment).once.with(:failures)
+      # it should log a warning, but we can't test that
+      expect { filter.filter(event) }.not_to raise_exception
+    end
+  end
+
+  describe "Invalid datatype conversion specified integer instead of int" do
+    subject(:filter) {  LogStash::Filters::Dissect.new(config)  }
+    let(:config)     do
+      {
+        "convert_datatype" => {
+          "code" => "integer", # only int is supported
+        }
+      }
+    end
+    it "raises an error" do
+      expect { filter.register }.to raise_exception(LogStash::ConvertDatatypeFormatError)
+    end
+  end
+
+  describe "Integer datatype conversion, handle large integers" do
+    let(:config) do <<-CONFIG
+      filter {
+        dissect {
+          convert_datatype => {
+            "big_number" => "int"
+          }
+        }
+      }
+    CONFIG
+    end
+    sample("big_number" => "43947404257507186289") do
+      expect(subject.get("big_number")).to eq(43947404257507186289)
+    end
+  end
+
+  describe "Float datatype conversion, handle large floats" do
+    let(:config) do <<-CONFIG
+      filter {
+        dissect {
+          convert_datatype => {
+            "big_number" => "float"
+          }
+        }
+      }
+    CONFIG
+    end
+    sample("big_number" => "43947404257507186289.345324") do
+      expect(subject.get("big_number")).to eq(BigDecimal.new("43947404257507186289.345324"))
     end
   end
 
@@ -177,11 +218,6 @@ describe LogStash::Filters::Dissect do
     let(:message)    { "very random message :-)" }
     let(:config)     { {"mapping" => {"blah-di-blah" => "%{timestamp} %{+timestamp}"}} }
     let(:event)      { LogStash::Event.new("message" => message) }
-    let(:loggr)      { LoggerMock.new }
-
-    before(:each) do
-      filter.class.instance_variable_set("@logger", loggr)
-    end
 
     it "does not raise any exceptions" do
       expect{filter.register}.not_to raise_exception
@@ -189,8 +225,8 @@ describe LogStash::Filters::Dissect do
 
     it "dissect failure key missing is logged" do
       filter.register
-      filter.filter(event)
-      expect(loggr.msgs).to eq(["Event before dissection", "Dissector mapping, key not found in event", "Event after dissection"])
+      expect{filter.filter(event)}.not_to raise_exception
+      # Logging moved to java, can't Mock Logger
     end
   end
 
@@ -209,7 +245,7 @@ describe LogStash::Filters::Dissect do
     context "when field is defined as Append and Indirect (+&)" do
       let(:config)     { {"mapping" => {"message" => "%{+&timestamp}"}}}
       it "raises an error in register" do
-        msg = "org.logstash.dissect.fields.InvalidFieldException: Field cannot prefix with both Append and Indirect Prefix (+&): +&timestamp"
+        msg = /\Aorg\.logstash\.dissect\.fields\.InvalidFieldException: Field cannot prefix with both Append and Indirect Prefix .+/
         expect{filter.register}.to raise_exception(LogStash::FieldFormatError, msg)
       end
     end
@@ -217,7 +253,7 @@ describe LogStash::Filters::Dissect do
     context "when field is defined as Indirect and Append (&+)" do
       let(:config)     { {"mapping" => {"message" => "%{&+timestamp}"}}}
       it "raises an error in register" do
-        msg = "org.logstash.dissect.fields.InvalidFieldException: Field cannot prefix with both Append and Indirect Prefix (&+): &+timestamp"
+        msg = /\Aorg\.logstash\.dissect\.fields\.InvalidFieldException: Field cannot prefix with both Append and Indirect Prefix .+/
         expect{filter.register}.to raise_exception(LogStash::FieldFormatError, msg)
       end
     end
@@ -249,6 +285,22 @@ describe LogStash::Filters::Dissect do
     end
   end
 
+  describe "When the delimiters contain '{' and '}'" do
+    let(:options) { { "mapping" => { "message" => "{%{a}}{%{b}}%{rest}" } } }
+    subject { described_class.new(options) }
+    let(:event) { LogStash::Event.new({ "message" => "{foo}{bar}" }) }
+    before(:each) do
+      subject.register
+      subject.filter(event)
+    end
+    it "should dissect properly and not add tags to the event" do
+      expect(event.get("a")).to eq("foo")
+      expect(event.get("b")).to eq("bar")
+      expect(event.get("rest")).to eq("")
+      expect(event.get("tags")).to be_nil
+    end
+  end
+
   describe "Basic dissection" do
 
     let(:options) { { "mapping" => { "message" => "%{a} %{b}" } } }
@@ -270,7 +322,9 @@ describe LogStash::Filters::Dissect do
     context "when field is empty" do
       let(:event_data) { { "message" => "" } }
       it "should add tags to the event" do
-        expect(event.get("tags")).to include("_dissectfailure")
+        tags = event.get("tags")
+        expect(tags).not_to be_nil
+        expect(tags).to include("_dissectfailure")
       end
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-dissect
 version: !ruby/object:Gem::Version
-  version: 1.0.12
+  version: 1.1.1
 platform: ruby
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-08-15 00:00:00.000000000 Z
+date: 2017-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -72,7 +72,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
+description: This gem is a Logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
+  gem is not a stand-alone program
 email: info@elastic.co
 executables: []
 extensions: []
@@ -93,7 +95,7 @@ files:
 - logstash-filter-dissect.gemspec
 - spec/filters/dissect_spec.rb
 - spec/spec_helper.rb
-- vendor/jars/org/logstash/dissect/jruby-dissect-library/1.0.12/jruby-dissect-library-1.0.12.jar
+- vendor/jars/org/logstash/dissect/jruby-dissect-library/1.1.1/jruby-dissect-library-1.1.1.jar
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
@@ -117,7 +119,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project:
-rubygems_version: 2.4.8
+rubygems_version: 2.6.13
 signing_key:
 specification_version: 4
 summary: This dissect filter will de-structure text into multiple fields.