logstash-filter-java_stack_digest 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a0dd81d1918a7eb930634c65283b9716c753c3721f87659aff1ff471ed40f6db
-  data.tar.gz: 174b1436664b23d69003d2ad96423e9162b60e62e16ede20173ae2f996c98732
+  metadata.gz: 600d4f38d95662574e1cfa1c0e3b743d1537de52b292ae3258411822383ba947
+  data.tar.gz: ee436c70b22307b5a3ceeee14057a28a7fac98d71327afe1097537cb105d3c03
 SHA512:
-  metadata.gz: 0d95b2391c05acea7d7da3fad911930d9ef2d69cb95f61d6d643685b15c7a1b946c8c89ef4a509efc291d954d3f9e16dcc018a6840e151dd8c6967b2df7bd884
-  data.tar.gz: 412c45a60b44075a197c714bcbe3ec2b18ae91569ec98393d1e44a4afc467fee7ea81d3e2068bb208290fc4eb714f07e453705738adffaecbfd3e4e2ac122747
+  metadata.gz: a34404d3ed44c6dc7deaf970fa7392600d01a09dcfb6543cb17da275896b4b5a41ad01fb87ef3a74330eed802ab1d1147d34f5f0c86403cda9c6b936654a4df4
+  data.tar.gz: 9ccbdfa5e831fc0f67c3edd2e0bc78bf18884e2e49988ee0de5ea08785bd0932a179db9fec15046849770f31b9c5d2c870cc2801cac4ad19f7fe5618c4088b3b

data/CHANGELOG.md

@@ -1,2 +1,5 @@
 ## 0.1.0
   - Plugin created with the logstash plugin generator
+
+## 0.1.1
+  - added `includes` patterns

data/README.md

@@ -1,79 +1,53 @@
 # Logstash Plugin
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
+This is a filter plugin for [Logstash](https://github.com/elastic/logstash) that enables computing a *stable* digest 
+(MD5) from a Java stack trace.
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
-## Documentation
-
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
-
-### 1. Plugin Developement and Testing
 
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
+## Documentation
 
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+Goal and principles are described in the project [GitHub pages](https://pismy.github.io/logstash-filter-java_stack_digest/).
 
-- Install dependencies
-```sh
-bundle install
-```
+You'll also find an [online debugger](https://pismy.github.io/logstash-filter-java_stack_digest/debugger.html) to help 
+you configure and debug your plugin options.
 
-#### Test
 
-- Update your dependencies
+## Install in Logstash
 
+- Download the plugin from [RubyGems](https://rubygems.org/gems/logstash-filter-java_stack_digest).
 ```sh
-bundle install
+gem install logstash-filter-java_stack_digest
 ```
-
-- Run tests
-
+- Install the plugin from the Logstash home
 ```sh
-bundle exec rspec
+bin/logstash-plugin install logstash-filter-java_stack_digest.gem
 ```
+- Configure the plugin in Logstash configuration
+- Start Logstash and proceed to test the plugin
 
-### 2. Running your unpublished Plugin in Logstash
+## Configuration
 
-#### 2.1 Run in a local Logstash clone
+This plugin supports the following configuration options:
+* `source` (type `string`): the name of the field containing the Java stack trace option (default value: `"stack_trace"`)
+* `target` (type `string`): the name of the field to assign the computed stack trace digest (default value: `"stack_digest"`)
+* `exclude_no_source` (type `boolean`): whether stack trace elements without source info (no filename or line number) should be excluded from the digest  (default value: `true`)
+* `includes` (type `array` of `string`): RegExp patterns determining whether stack trace elements should be **included** from digest (defaults to none - _exclusion patterns only_)
+* `excludes` (type `array` of `string`): RegExp patterns determining whether stack trace elements should be **excluded** from digest (defaults to standard dynamic Java reflection and generated classes patterns)
 
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
+Filter configuration example:
 ```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+  filter {
+    java_stack_digest {
+      source => 'java_stack'
+      target => 'error_digest'
+      exclude_no_source => true
+      includes => ['^com\\.xyz\\.', '^java\\.']
+      excludes => ['\\$\\$FastClassByCGLIB\\$\\$', '\\$\\$EnhancerBySpringCGLIB\\$\\$']
+    }
+  }
 ```
-- Install plugin
-```sh
-bin/logstash-plugin install --no-verify
-```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'filter {awesome {}}'
-```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
-
-#### 2.2 Run in an installed Logstash
-
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
-
-- Build your plugin gem
-```sh
-gem build logstash-filter-awesome.gemspec
-```
-- Install the plugin from the Logstash home
-```sh
-bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
-```
-- Start Logstash and proceed to test the plugin
 
 ## Contributing
 

data/lib/logstash/filters/java_stack_digest.rb

@@ -17,9 +17,6 @@ class LogStash::Filters::JavaStackDigest < LogStash::Filters::Base
   #
   config_name "java_stack_digest"
 
-  # activates debug traces
-  config :debug, :validate => :boolean, :default => false
-
   # The name of the input field supposed to contain the Java stack trace (default 'stack_trace').
   config :source, :validate => :string, :default => "stack_trace"
 
@@ -29,9 +26,12 @@ class LogStash::Filters::JavaStackDigest < LogStash::Filters::Base
   # Determines whether stack trace elements without source info (no filename or line number) should be excluded from the digest (default 'true')
   config :exclude_no_source, :validate => :boolean, :default => true
 
+  # Array of regular expressions to include stack trace elements
+  # defaults: empty; if non-empty, matching will start with includes, then filter out excludes
+  config :includes, :validate => :array, :default => []
+
   # Array of regular expressions to exclude stack trace elements
   # defaults to standard dynamic classes and method invocations
-  # config :excludes, :validate => :array, :default => %w["\\$\\$FastClassByCGLIB\\$\\$", "\\$\\$EnhancerBySpringCGLIB\\$\\$", "^sun\\.reflect\\..*\\.invoke", "^com\\.sun\\.", "^sun\\.net\\.", "^java\\.lang\\.reflect\\.Method\\.invoke", "^net\\.sf\\.cglib\\.proxy\\.MethodProxy\\.invoke", "^java\\.util\\.concurrent\\.ThreadPoolExecutor\\.runWorker","^java\\.lang\\.Thread\\.run$" ]
   config :excludes, :validate => :array, :default => [/\$\$FastClassByCGLIB\$\$/, /\$\$EnhancerBySpringCGLIB\$\$/, /^sun\.reflect\..*\.invoke/, /^com\.sun\./, /^sun\.net\./, /^java\.lang\.reflect\.Method\.invoke/, /^net\.sf\.cglib\.proxy\.MethodProxy\.invoke/, /^java\.util\.concurrent\.ThreadPoolExecutor\.runWorker/, /^java\.lang\.Thread\.run$/ ]
 
   public
@@ -48,6 +48,9 @@ class LogStash::Filters::JavaStackDigest < LogStash::Filters::Base
     # group 3: line number (optional)
     @stack_element_pattern = /^\s+at\s+((?:[\w$]+\.){2,}[\w$]+)\((?:([^:]+)(?::(\d+))?)?\)/
 
+    # coerce includes to an array of Regexp
+    @includes = @includes.collect {|pattern| Regexp::new(pattern)}
+
     # coerce excludes to an array of Regexp
     @excludes = @excludes.collect {|pattern| Regexp::new(pattern)}
   end # def register
@@ -59,35 +62,35 @@ class LogStash::Filters::JavaStackDigest < LogStash::Filters::Base
 
     return if stack_trace.nil? || stack_trace.empty?
 
-    event.set(@target, compute_digest(stack_trace.split("\n"), 0))
+    # compute digest add to the event
+    event.set(@target, compute_digest(stack_trace.split("\n")))
 
     # filter_matched should go in the last line of our successful code
     filter_matched(event)
   end # def filter
 
   # computes a Java stack trace digest
-  def compute_digest(stack_trace, level)
+  def compute_digest(stack_trace)
+    md5 = Digest::MD5.new
 
     # 1: extract error class from first line
-    error_class = @error_pattern.match(stack_trace.shift)
-    puts "Error (#{level}) #{error_class}:" if @debug
-    md5 = Digest::MD5.new
+    cur_stack_trace_line = stack_trace.shift
+    error_class = @error_pattern.match(cur_stack_trace_line)
+
     # digest: error classname
     md5.update error_class[1]
 
     # 2: read all stack trace elements until stack trace is empty or we hit the next error
     ste_count = 0
     while not stack_trace.empty?
-      if stack_trace.first.start_with?(' ') or stack_trace.first.start_with?("\t")
+      cur_stack_trace_line = stack_trace.first
+      if cur_stack_trace_line.start_with?(' ') or cur_stack_trace_line.start_with?("\t")
         # current line starts with a whitespace: is it a stack trace element ?
-        stack_element = @stack_element_pattern.match(stack_trace.first)
+        stack_element = @stack_element_pattern.match(cur_stack_trace_line)
         if stack_element
           # current line is a stack trace element
           ste_count+=1
-          if is_excluded?(stack_element)
-            puts "  (-) at #{stack_element[1]}(#{stack_element[2]}:#{stack_element[3]})" if @debug
-          else
-            puts "  (+) at #{stack_element[1]}(#{stack_element[2]}:#{stack_element[3]})" if @debug
+          if not is_excluded?(stack_element)
             # digest: STE classname and method
             md5.update stack_element[1]
             # digest: line number (if present)
@@ -97,29 +100,43 @@ class LogStash::Filters::JavaStackDigest < LogStash::Filters::Base
           end
         end
       elsif(ste_count > 0)
-        # current line doesn't start with a whitespace and we've already read stack trace elements: is it the next error in the stack
+        # current line doesn't start with a whitespace and we've already read stack trace elements: it looks like the next error in the stack
         break
       end
       # move to next line
       stack_trace.shift
     end
 
+
     # 3: if stack trace not empty, compute digest for next error
     if not stack_trace.empty?
-      md5.update compute_digest(stack_trace, level+1)
+      md5.update compute_digest(stack_trace)
     end
 
     return md5.hexdigest
   end
 
+  # Determines whether the given stack trace element (Regexp match) should be excluded from digest computation
   def is_excluded?(stack_element)
     # 1: exclude elements without source info ?
-    if @exclude_no_source and (stack_element[2].nil? or stack_element[2].empty?) and (stack_element[3].nil? or stack_element[3].empty?)
+    if @exclude_no_source and (stack_element[3].nil? or stack_element[3].empty?)
       return true
     end
-    # 2: Regex based exclusion
+
+    # 2: Regex based inclusions
+    classname_and_method = stack_element[1]
+    if not @includes.empty?
+      match_idx = @includes.index do |pattern|
+        pattern.match(classname_and_method)
+      end
+      if match_idx.nil?
+        return true
+      end
+    end
+
+    # 3: Regex based exclusions
     @excludes.each do |pattern|
-      if pattern.match(stack_element[1])
+      if pattern.match(classname_and_method)
         return true
       end
     end

data/logstash-filter-java_stack_digest.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-filter-java_stack_digest'
-  s.version       = '0.1.0'
+  s.version       = '0.1.1'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'Logstash filter that computes a digest of Java stack traces.'
   s.description   = 'Logstash filter that computes a digest of Java stack traces.'

data/spec/filters/java_stack_digest_spec.rb

@@ -143,6 +143,25 @@ Caused by: java.net.SocketTimeoutException: Read timed out
     end
   end
 
+  # ---------------------------------------------------------------------
+  describe "Digest with inclusion and exclusion config" do
+    let(:config) do <<-CONFIG
+      filter {
+        java_stack_digest {
+          exclude_no_source => true
+          includes => ['^com\\.xyz\\.']
+          excludes => ['\\$\\$FastClassByCGLIB\\$\\$', '\\$\\$EnhancerBySpringCGLIB\\$\\$']
+        }
+      }
+    CONFIG
+    end
+
+    sample("stack_trace" => stack_trace) do
+      expect(subject).to include("stack_digest")
+      expect(subject.get('stack_digest')).to eq('fa54771601828e9a2964ed0651e8c09c')
+    end
+  end
+
   # ---------------------------------------------------------------------
   describe "Digest should be computed if stack trace present with non default config" do
     let(:config) do <<-CONFIG

data/stack-digest.md

@@ -0,0 +1,210 @@
+# Details about stack digest
+
+This page gives details about the **stack digest** feature (goal and implementation).
+
+
+## Why generating stack digests?
+
+A full Java stack trace is very helpful to analyse and debug a single error, but is not very handy to compare two
+errors.
+
+The idea is to turn a full Java stack trace into a **short** and **stable** digest, that could help matching several 
+distinct occurrences of the same type of error:
+
+* **short** for easing elasticsearch indexing, and take advantage of it (we use a MD5 hash),
+* **stable** is the tricky part, as the same type of error occurring twice may not generate exactly the same stack trace 
+(see below).
+
+This done, it becomes easy with elasticsearch or any other logs centralization and indexation system to:
+
+* **count** distinct type of errors that occur in your code over time,
+* **count** occurrences and frequency of a given type of error,
+* **detect** when a (new) type of error occurred for the first time (maybe linking this to a new version being deployed?).
+
+The stack digest may also become a simple error id that you can link your bug tracker with...
+
+
+## Stack digest stability challenge by examples
+
+### Let's consider error stack 1
+
+*(the stack trace presented here has been cut by half from useless lines)*
+
+<pre>
+<b>com.xyz.MyApp$MyClient$MyClientException</b>: <strike>An error occurred while getting Alice's things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)</b>
+  at <b>com.xyz.MyApp$MyService.displayThings(MyApp.java:16)</b>
+  at <strike>com.xyz.MyApp$MyService$$FastClassByCGLIB$$e7645040.invoke()</strike><sup>(aop)</sup>
+  at <i>net.sf.cglib.proxy.MethodProxy.invoke()</i><sup>(aop)</sup>
+  at <i>org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()</i><sup>(fwk)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke0()</i><sup>(aop)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>sun.reflect.DelegatingMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>java.lang.reflect.Method.invoke()</i><sup>(aop)</sup>
+  at <i>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.AspectJAroundAdvice.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.transaction.interceptor.TransactionInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept()</i><sup>(fwk)</sup>
+  at <strike>com.xyz.MyApp$MyService$$EnhancerBySpringCGLIB$$c673c675.displayThings(&lt;generated&gt;)</strike><sup>(aop)</sup>
+  at <strike>sun.reflect.GeneratedMethodAccessor647.invoke(Unknown Source)</strike><sup>(aop)</sup>
+  at <i>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</i><sup>(aop)</sup>
+  at <i>java.lang.reflect.Method.invoke(Method.java:498)</i><sup>(aop)</sup>
+  at <i>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)</i><sup>(fwk)</sup>
+  at <i>javax.servlet.http.HttpServlet.service(HttpServlet.java:624)</i><sup>(jee)</sup>
+  at <i>org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)</i><sup>(fwk)</sup>
+  at <i>javax.servlet.http.HttpServlet.service(HttpServlet.java:731)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)</i><sup>(jee)</sup>
+  at <i>org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)</i><sup>(jee)</sup>
+  ...
+  at <i>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)</i><sup>(fwk)</sup>
+  at <i>org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)</i><sup>(fwk)</sup>
+  at <i>org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)</i><sup>(fwk)</sup>
+  at <i>org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)</i><sup>(fwk)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)</i><sup>(jee)</sup>
+  ...
+  at <i>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)</i><sup>(jee)</sup>
+  at <i>org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)</i><sup>(jee)</sup>
+  at <i>org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)</i><sup>(jee)</sup>
+  at <i>org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)</i><sup>(jee)</sup>
+  at <i>org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)</i><sup>(jee)</sup>
+  at <i>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)</i><sup>(jee)</sup>
+  at <i>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)</i><sup>(jee)</sup>
+  at <i>org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)</i><sup>(jee)</sup>
+  at <i>java.lang.Thread.run(Thread.java:748)</i><sup>(jee)</sup>
+  ...
+Caused by: <b>com.xyz.MyApp$HttpStack$HttpError</b>: <strike>I/O error on GET http://dummy/user/alice/things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$HttpStack.get(MyApp.java:40)</b>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)</b>
+  ... 23 common frames omitted
+Caused by: <b>java.net.SocketTimeoutException</b>: <strike>Read timed out</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$HttpStack.get(MyApp.java:38)</b>
+  ... 24 common frames omitted
+</pre>
+
+---
+
+<strike>Strike out elements</strike> may vary from one occurrence to the other:
+
+* <strike>error messages</strike><sup>(msg)</sup> often contain stuff related to the very error occurrence context,
+* <strike>AOP generated classes</strike><sup>(aop)</sup> may vary from one execution to another.
+
+*Italic* elements are somewhat not stable, or at least useless (purely technical). Ex:
+
+* <i>JEE container stuff</i><sup>(jee)</sup>: may change when you upgrade your JEE container version or add/remove/reorganize your servlet filters chain for instance,
+* <i>Spring Framework</i><sup>(fwk)</sup> underlying stacks (MVC, security) for pretty much the same reason,
+* <i>AOP and dynamic invocation</i><sup>(aop)</sup>: purely technical, and quite implementation-dependent.
+
+Only **bolded elements** are supposed to be stable.
+
+
+### Now let's consider error stack 2
+
+*(shortened)*
+
+<pre>
+<b>com.xyz.MyApp$MyClient$MyClientException</b>: <strike>An error occurred while getting <b>Bob</b>'s things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)</b>
+  at <b>com.xyz.MyApp$MyService.displayThings(MyApp.java:16)</b>
+  at <strike>com.xyz.MyApp$MyService$$FastClassByCGLIB$$<b>07e70d1e</b>.invoke()</strike><sup>(aop)</sup>
+  at <i>net.sf.cglib.proxy.MethodProxy.invoke()</i><sup>(aop)</sup>
+  at <i>org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()</i><sup>(fwk)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke0()</i><sup>(aop)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>sun.reflect.DelegatingMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>java.lang.reflect.Method.invoke()</i><sup>(aop)</sup>
+  at <i>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.AspectJAroundAdvice.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.transaction.interceptor.TransactionInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept()</i><sup>(fwk)</sup>
+  at <strike>com.xyz.MyApp$MyService$$EnhancerBySpringCGLIB$$<b>e3f570b1</b>.displayThings(&lt;generated&gt;)</strike><sup>(aop)</sup>
+  at <strike>sun.reflect.<b>GeneratedMethodAccessor737</b>.invoke(Unknown Source)</strike><sup>(aop)</sup>
+  ...
+Caused by: <b>com.xyz.MyApp$HttpStack$HttpError</b>: <strike>I/O error on GET http://dummy/user/<b>bob</b>/things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$HttpStack.get(MyApp.java:40)</b>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)</b>
+  ... 23 common frames omitted
+Caused by: <b>java.net.SocketTimeoutException</b>: <strike>Read timed out</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$HttpStack.get(MyApp.java:38)</b>
+  ... 24 common frames omitted
+</pre>
+
+---
+
+You may see in this example that most of the <strike>strike elements have slight <b>differences</b></strike> from error stack
+1 (messages and generated classes names).
+
+Nevertheless it is the same exact error (despite the context is different as it applies to another user), and the goal
+here is to be able to count them as *two occurrences of the same error*.
+
+### Now let's consider error stack 3
+
+*(shortened)*
+
+<pre>
+<b>com.xyz.MyApp$MyClient$MyClientException</b>: <strike>An error occurred while getting Alice's things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:26)</b>
+  at <b>com.xyz.MyApp$MyService.displayThings(MyApp.java:16)</b>
+  at <strike>com.xyz.MyApp$MyService$$FastClassByCGLIB$$e7645040.invoke()</strike><sup>(aop)</sup>
+  at <i>net.sf.cglib.proxy.MethodProxy.invoke()</i><sup>(aop)</sup>
+  at <i>org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()</i><sup>(fwk)</sup>
+  at <i>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed()</i><sup>(fwk)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke0()</i><sup>(aop)</sup>
+  at <i>sun.reflect.NativeMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>sun.reflect.DelegatingMethodAccessorImpl.invoke()</i><sup>(aop)</sup>
+  at <i>java.lang.reflect.Method.invoke()</i><sup>(aop)</sup>
+  ...
+Caused by: <b>com.xyz.MyApp$HttpStack$HttpError</b>: <strike>I/O error on GET http://dummy/user/alice/things</strike><sup>(msg)</sup>
+  at <b>com.xyz.MyApp$HttpStack.get(MyApp.java:40)</b>
+  at <b>com.xyz.MyApp$MyClient.getTheThings(MyApp.java:24)</b>
+  ... 23 common frames omitted
+Caused by: <b>javax.net.ssl.SSLException</b>: <strike>Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown</strike><sup>(msg)</sup>
+  at <b>com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1172)</b>
+  ... 24 common frames omitted
+</pre>
+
+---
+
+Here, you can see that the first and second errors are the same as in error stack 1, but the root cause is different (`SSLException` instead of `SocketTimeoutException`).
+
+So in that case we don't want the top error digest computed for error stack 3 to be the same as for error stack 1.
+
+## Stack digest computation rules
+
+As a conclusion, stack digest computation applies the following rules:
+
+1. a stack digest shall **not compute with the error message**
+2. a stack digest shall **compute with it's parent cause** (recurses)
+3. in order to stabilize the stack digest (over time and space), it's recommended to **exclude non-stable elements**

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-java_stack_digest
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Pierre Smeyers
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2018-03-30 00:00:00.000000000 Z
+date: 2018-04-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -54,6 +54,7 @@ files:
 - logstash-filter-java_stack_digest.gemspec
 - spec/filters/java_stack_digest_spec.rb
 - spec/spec_helper.rb
+- stack-digest.md
 homepage: https://github.com/pismy/logstash-filter-java_stack_digest
 licenses:
 - Apache-2.0