logstash-filter-aggregate 2.3.1 → 2.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: eaa83c85ffda851fdae41182c4ec2d010483dbbe
-  data.tar.gz: bfbb53a989e67654304b4e4c9cf89f2b91411c6a
+  metadata.gz: 86bf41e2f4183eb1bca1053dcdd54bba16ce715d
+  data.tar.gz: edd15d9f860d6becda2d0faabf246de87cb404bd
 SHA512:
-  metadata.gz: 0adfe2d3916570f84230a5823ad6b038f661a5837bb78600e66c3a58d4b066630cf31cf974f939a8071f2ab2549698756a24cb768e6b15df983a2546f0dc6490
-  data.tar.gz: 7720f2d38e7757145294e05903376874dcd88a600bcd9174c5da1ac2ef0ca3ba4bd2545c01a107317cf037f99d8be8005cd3830aba62af32d85c6e8e5f57f1c7
+  metadata.gz: adf8addee7fcfd1efeddf980a85933b9ccb35e89f1d3e1e784fb302902a0308ec31209decb5c3d3da83151860d212e2d7c79f41349bc5e2cc4a16acdc2352d5c
+  data.tar.gz: 320a2b40266aa4cc506ef01ec6e61e6b0f92985f2ccd4914a621b725fc9ace12f50e8d3314c36afedff97c5e0bb886a2460c2bc87549ebf41ff92389754b6842

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 2.4.0
+ - new feature: You can now define timeout options per task_id pattern (#42)  
+ timeout options are : `timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_task_id_field, timeout_tags`
+ - validation: a configuration error is thrown at startup if you define any timeout option on several aggregate filters for the same task_id pattern
+ - breaking: if you use `aggregate_maps_path` option, storage format has changed. So you have to delete `aggregate_maps_path` file before starting Logstash
+ 
 ## 2.3.1
  - new feature: Add new option "timeout_tags" so that you can add tags to generated timeout events
  

data/README.md

@@ -204,7 +204,9 @@ In that case, you don't want to wait task timeout to flush aggregation map.
 - after the final event, the map attached to task is deleted
 - in one filter configuration, it is recommanded to define a timeout option to protect the filter against unterminated tasks. It tells the filter to delete expired maps
 - if no timeout is defined, by default, all maps older than 1800 seconds are automatically deleted
-- finally, if `code` execution raises an exception, the error is logged and event is tagged '_aggregateexception'
+- all timeout options have to be defined in only one aggregate filter per task_id pattern.  
+Timeout options are : `timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_task_id_field, timeout_tags`
+- if `code` execution raises an exception, the error is logged and event is tagged '_aggregateexception'
 
 ## Use Cases
 - extract some cool metrics from task logs and push them into task final log event (like in example #1 and #2)
@@ -236,29 +238,20 @@ Tell the filter what to do with aggregate map (default :  "create_or_update").
 Default value: `create_or_update`  
 
 - **end_of_task:**  
-Tell the filter that task is ended, and therefore, to delete map after code execution.  
+Tell the filter that task is ended, and therefore, to delete aggregate map after code execution.  
 Default value: `false`  
 
-- **timeout:**  
-The amount of seconds after a task "end event" can be considered lost.  
-When timeout occurs for a task, The task "map" is evicted.  
-If no timeout is defined, default timeout will be applied : 1800 seconds.  
-
 - **aggregate_maps_path:**  
 The path to file where aggregate maps are stored when logstash stops and are loaded from when logstash starts.  
 If not defined, aggregate maps will not be stored at logstash stop and will be lost.   
 Must be defined in only one aggregate filter (as aggregate maps are global).  
 Example value : `"/path/to/.aggregate_maps"`
 
-- **push_previous_map_as_event:**  
-When this option is enabled, each time aggregate plugin detects a new task id, it pushes previous aggregate map as a new logstash event, 
-and then creates a new empty map for the next task.  
-_WARNING:_ this option works fine only if tasks come one after the other. It means : all task1 events, then all task2 events, etc...  
-Default value: `false`  
-
-- **push_map_as_event_on_timeout**  
-When this option is enabled, each time a task timeout is detected, it pushes task aggregation map as a new logstash event.  
-This enables to detect and process task timeouts in logstash, but also to manage tasks that have no explicit end event.
+- **timeout:**  
+The amount of seconds after a task "end event" can be considered lost.  
+When timeout occurs for a task, The task "map" is evicted.  
+Timeout can be defined for each "task_id" pattern.  
+If no timeout is defined, default timeout will be applied : 1800 seconds.  
 
 - **timeout_code**  
 The code to execute to complete timeout generated event, when 'push_map_as_event_on_timeout' or 'push_previous_map_as_event' is set to true.  
@@ -266,6 +259,17 @@ The code block will have access to the newly generated timeout event that is pre
 If 'timeout_task_id_field' is set, the event is also populated with the task_id value  
 Example value: `"event['state'] = 'timeout'"`
 
+- **push_map_as_event_on_timeout**  
+When this option is enabled, each time a task timeout is detected, it pushes task aggregation map as a new logstash event.  
+This enables to detect and process task timeouts in logstash, but also to manage tasks that have no explicit end event.  
+Default value: `false`  
+
+- **push_previous_map_as_event:**  
+When this option is enabled, each time aggregate plugin detects a new task id, it pushes previous aggregate map as a new logstash event, 
+and then creates a new empty map for the next task.  
+_WARNING:_ this option works fine only if tasks come one after the other. It means : all task1 events, then all task2 events, etc...  
+Default value: `false`  
+
 - **timeout_task_id_field**  
 This option indicates the timeout generated event's field for the "task_id" value.  
 The task id will then be set into the timeout event. This can help correlate which tasks have been timed out.  

data/lib/logstash/filters/aggregate.rb

@@ -220,7 +220,8 @@ require "logstash/util/decorators"
 # * after the final event, the map attached to task is deleted
 # * in one filter configuration, it is recommanded to define a timeout option to protect the feature against unterminated tasks. It tells the filter to delete expired maps
 # * if no timeout is defined, by default, all maps older than 1800 seconds are automatically deleted
-# * finally, if `code` execution raises an exception, the error is logged and event is tagged '_aggregateexception'
+# * all timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_task_id_field, timeout_tags 
+# * if `code` execution raises an exception, the error is logged and event is tagged '_aggregateexception'
 #
 #
 # ==== Use Cases
@@ -252,30 +253,6 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   # Example value : `"map['sql_duration'] += event['duration']"`
   config :code, :validate => :string, :required => true
 
-
-
-  # The code to execute to complete timeout generated event, when 'push_map_as_event_on_timeout' or 'push_previous_map_as_event' is set to true. 
-  # The code block will have access to the newly generated timeout event that is pre-populated with the aggregation map. 
-  #
-  # If 'timeout_task_id_field' is set, the event is also populated with the task_id value 
-  #
-  # Example value: `"event['state'] = 'timeout'"`
-  config :timeout_code, :validate => :string, :required => false
-
-
-  # This option indicates the timeout generated event's field for the "task_id" value. 
-  # The task id will then be set into the timeout event. This can help correlate which tasks have been timed out.  
-  #
-  # This field has no default value and will not be set on the event if not configured.
-  #
-  # Example:
-  #
-  # If the task_id is "12345" and this field is set to "my_id", the generated event will have:
-  # event[ "my_id" ] = "12345"
-  #
-  config :timeout_task_id_field, :validate => :string, :required => false
-
-
   # Tell the filter what to do with aggregate map.
   #
   # `create`: create the map, and execute the code only if map wasn't created before
@@ -285,16 +262,9 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   # `create_or_update`: create the map if it wasn't created before, execute the code in all cases
   config :map_action, :validate => :string, :default => "create_or_update"
 
-  # Tell the filter that task is ended, and therefore, to delete map after code execution.  
+  # Tell the filter that task is ended, and therefore, to delete aggregate map after code execution.  
   config :end_of_task, :validate => :boolean, :default => false
 
-  # The amount of seconds after a task "end event" can be considered lost.
-  #
-  # When timeout occurs for a task, The task "map" is evicted.
-  #
-  # If no timeout is defined, default timeout will be applied : 1800 seconds.
-  config :timeout, :validate => :number, :required => false
-
   # The path to file where aggregate maps are stored when logstash stops
   # and are loaded from when logstash starts.
   #
@@ -304,15 +274,44 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   # Example value : `"/path/to/.aggregate_maps"`
   config :aggregate_maps_path, :validate => :string, :required => false
   
+  # The amount of seconds after a task "end event" can be considered lost.
+  #
+  # When timeout occurs for a task, The task "map" is evicted.
+  #
+  # Timeout can be defined for each "task_id" pattern.
+  #
+  # If no timeout is defined, default timeout will be applied : 1800 seconds.
+  config :timeout, :validate => :number, :required => false
+
+  # The code to execute to complete timeout generated event, when 'push_map_as_event_on_timeout' or 'push_previous_map_as_event' is set to true. 
+  # The code block will have access to the newly generated timeout event that is pre-populated with the aggregation map. 
+  #
+  # If 'timeout_task_id_field' is set, the event is also populated with the task_id value 
+  #
+  # Example value: `"event['state'] = 'timeout'"`
+  config :timeout_code, :validate => :string, :required => false
+
+  # When this option is enabled, each time a task timeout is detected, it pushes task aggregation map as a new logstash event.  
+  # This enables to detect and process task timeouts in logstash, but also to manage tasks that have no explicit end event.
+  config :push_map_as_event_on_timeout, :validate => :boolean, :required => false, :default => false
+
   # When this option is enabled, each time aggregate plugin detects a new task id, it pushes previous aggregate map as a new logstash event, 
   # and then creates a new empty map for the next task.
   #
   # WARNING: this option works fine only if tasks come one after the other. It means : all task1 events, then all task2 events, etc...
   config :push_previous_map_as_event, :validate => :boolean, :required => false, :default => false
   
-  # When this option is enabled, each time a task timeout is detected, it pushes task aggregation map as a new logstash event.  
-  # This enables to detect and process task timeouts in logstash, but also to manage tasks that have no explicit end event.
-  config :push_map_as_event_on_timeout, :validate => :boolean, :required => false, :default => false
+  # This option indicates the timeout generated event's field for the "task_id" value. 
+  # The task id will then be set into the timeout event. This can help correlate which tasks have been timed out.  
+  #
+  # This field has no default value and will not be set on the event if not configured.
+  #
+  # Example:
+  #
+  # If the task_id is "12345" and this field is set to "my_id", the generated event will have:
+  # event[ "my_id" ] = "12345"
+  #
+  config :timeout_task_id_field, :validate => :string, :required => false
 
   # Defines tags to add when a timeout event is generated and yield
   config :timeout_tags, :validate => :array, :required => false, :default => []
@@ -329,11 +328,15 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   # Mutex used to synchronize access to 'aggregate_maps'
   @@mutex = Mutex.new
 
-  # Aggregate instance which will evict all zombie Aggregate elements (older than timeout)
-  @@eviction_instance = nil
+  # Default timeout for task_id patterns where timeout is not defined in logstash filter configuration
+  @@default_timeout = nil
+
+  # For each "task_id" pattern, defines which Aggregate instance will evict all expired Aggregate elements (older than timeout)
+  # For each entry, key is "task_id pattern" and value is "aggregate instance"
+  @@eviction_instance_map = {}
 
-  # last time where eviction was launched
-  @@last_eviction_timestamp = nil
+  # last time where eviction was launched, per "task_id" pattern
+  @@last_eviction_timestamp_map = {}
 
   # flag indicating if aggregate_maps_path option has been already set on one aggregate instance
   @@aggregate_maps_path_set = false
@@ -349,30 +352,44 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     if @timeout_code
       eval("@timeout_codeblock = lambda { |event| #{@timeout_code} }", binding, "(aggregate filter timeout code)")
     end
-
+    
     @@mutex.synchronize do
-      # define eviction_instance
-      if (!@timeout.nil? && (@@eviction_instance.nil? || @timeout < @@eviction_instance.timeout))
-        @@eviction_instance = self
-        @logger.info("Aggregate, timeout: #{@timeout} seconds")
+      
+      # timeout management : define eviction_instance for current task_id pattern
+      if has_timeout_options?
+        if @@eviction_instance_map.has_key?(@task_id)
+          # all timeout options have to be defined in only one aggregate filter per task_id pattern
+          raise LogStash::ConfigurationError, "Aggregate plugin: For task_id pattern #{@task_id}, there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : #{display_timeout_options}"
+        end
+        @@eviction_instance_map[@task_id] = self
+        @logger.info("Aggregate plugin: timeout for '#{@task_id}' pattern: #{@timeout} seconds")
+      end
+
+      # timeout management : define default_timeout 
+      if !@timeout.nil? && (@@default_timeout.nil? || @timeout < @@default_timeout)
+        @@default_timeout = @timeout
+        @logger.info("Aggregate plugin: default timeout: #{@timeout} seconds")
       end
 
       # check if aggregate_maps_path option has already been set on another instance
-      if (!@aggregate_maps_path.nil?)
-        if (@@aggregate_maps_path_set)
+      if !@aggregate_maps_path.nil?
+        if @@aggregate_maps_path_set
           @@aggregate_maps_path_set = false
-          raise LogStash::ConfigurationError, "Option 'aggregate_maps_path' must be set on only one aggregate filter"
+          raise LogStash::ConfigurationError, "Aggregate plugin: Option 'aggregate_maps_path' must be set on only one aggregate filter"
         else
           @@aggregate_maps_path_set = true
         end
       end
       
       # load aggregate maps from file (if option defined)
-      if (!@aggregate_maps_path.nil? && File.exist?(@aggregate_maps_path))
+      if !@aggregate_maps_path.nil? && File.exist?(@aggregate_maps_path)
         File.open(@aggregate_maps_path, "r") { |from_file| @@aggregate_maps = Marshal.load(from_file) }
         File.delete(@aggregate_maps_path)
-        @logger.info("Aggregate, load aggregate maps from : #{@aggregate_maps_path}")
+        @logger.info("Aggregate plugin: load aggregate maps from : #{@aggregate_maps_path}")
       end
+      
+      # init aggregate_maps
+      @@aggregate_maps[@task_id] ||= {}
     end
   end
 
@@ -380,18 +397,21 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   public
   def close
 
-    # Protection against logstash reload
-    @@aggregate_maps_path_set = false if @@aggregate_maps_path_set
-    @@eviction_instance = nil unless @@eviction_instance.nil?
-
+    # store aggregate maps to file (if option defined)
     @@mutex.synchronize do
-      # store aggregate maps to file (if option defined)
-      if (!@aggregate_maps_path.nil? && !@@aggregate_maps.empty?)
+      @@aggregate_maps.delete_if { |key, value| value.empty? }
+      if !@aggregate_maps_path.nil? && !@@aggregate_maps.empty?
         File.open(@aggregate_maps_path, "w"){ |to_file| Marshal.dump(@@aggregate_maps, to_file) }
-        @@aggregate_maps.clear()
-        @logger.info("Aggregate, store aggregate maps to : #{@aggregate_maps_path}")
+        @logger.info("Aggregate plugin: store aggregate maps to : #{@aggregate_maps_path}")
       end
+      @@aggregate_maps.clear()
     end
+
+    # Protection against logstash reload
+    @@aggregate_maps_path_set = false if @@aggregate_maps_path_set
+    @@default_timeout = nil unless @@default_timeout.nil?
+    @@eviction_instance_map = {} unless @@eviction_instance_map.empty?
+    
   end
   
   # This method is invoked each time an event matches the filter
@@ -409,19 +429,19 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     @@mutex.synchronize do
     
       # retrieve the current aggregate map
-      aggregate_maps_element = @@aggregate_maps[task_id]
+      aggregate_maps_element = @@aggregate_maps[@task_id][task_id]
       
 
       # create aggregate map, if it doesn't exist
-      if (aggregate_maps_element.nil?)
+      if aggregate_maps_element.nil?
         return if @map_action == "update"
         # create new event from previous map, if @push_previous_map_as_event is enabled
-        if (@push_previous_map_as_event and !@@aggregate_maps.empty?)
-          previous_map = @@aggregate_maps.shift[1].map
+        if @push_previous_map_as_event && !@@aggregate_maps[@task_id].empty?
+          previous_map = @@aggregate_maps[@task_id].shift[1].map
           event_to_yield = create_timeout_event(previous_map, task_id)
         end
         aggregate_maps_element = LogStash::Filters::Aggregate::Element.new(Time.now);
-        @@aggregate_maps[task_id] = aggregate_maps_element
+        @@aggregate_maps[@task_id][task_id] = aggregate_maps_element
       else
         return if @map_action == "create"
       end
@@ -437,7 +457,7 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
       end
       
       # delete the map if task is ended
-      @@aggregate_maps.delete(task_id) if @end_of_task
+      @@aggregate_maps[@task_id].delete(task_id) if @end_of_task
       
     end
 
@@ -484,15 +504,20 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   # This method is invoked by LogStash every 5 seconds.
   def flush(options = {})
     # Protection against no timeout defined by logstash conf : define a default eviction instance with timeout = DEFAULT_TIMEOUT seconds
-    if (@@eviction_instance.nil?)
-      @@eviction_instance = self
-      @timeout = DEFAULT_TIMEOUT
+    if @@default_timeout.nil?
+      @@default_timeout = DEFAULT_TIMEOUT
+    end
+    if !@@eviction_instance_map.has_key?(@task_id)
+      @@eviction_instance_map[@task_id] = self
+      @timeout = @@default_timeout
+    elsif @@eviction_instance_map[@task_id].timeout.nil?
+      @@eviction_instance_map[@task_id].timeout = @@default_timeout
     end
     
     # Launch eviction only every interval of (@timeout / 2) seconds
-    if (@@eviction_instance == self && (@@last_eviction_timestamp.nil? || Time.now > @@last_eviction_timestamp + @timeout / 2))
+    if @@eviction_instance_map[@task_id] == self && (!@@last_eviction_timestamp_map.has_key?(@task_id) || Time.now > @@last_eviction_timestamp_map[@task_id] + @timeout / 2)
       events_to_flush = remove_expired_maps()
-      @@last_eviction_timestamp = Time.now
+      @@last_eviction_timestamp_map[@task_id] = Time.now
       return events_to_flush
     end
 
@@ -507,9 +532,9 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     
     @@mutex.synchronize do
 
-      @@aggregate_maps.delete_if do |key, element| 
-        if (element.creation_timestamp < min_timestamp)
-          if (@push_previous_map_as_event) || (@push_map_as_event_on_timeout)
+      @@aggregate_maps[@task_id].delete_if do |key, element| 
+        if element.creation_timestamp < min_timestamp
+          if @push_previous_map_as_event || @push_map_as_event_on_timeout
             events_to_flush << create_timeout_event(element.map, key)
           end
           next true
@@ -520,6 +545,30 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     
     return events_to_flush
   end
+  
+  # return if this filter instance has any timeout option enabled in logstash configuration
+  def has_timeout_options?()
+    return (
+      timeout ||
+      timeout_code ||
+      push_map_as_event_on_timeout ||
+      push_previous_map_as_event ||
+      timeout_task_id_field ||
+      !timeout_tags.empty?
+    )
+  end
+
+  # display all possible timeout options
+  def display_timeout_options()
+    return [
+      "timeout",
+      "timeout_code",
+      "push_map_as_event_on_timeout",
+      "push_previous_map_as_event",
+      "timeout_task_id_field",
+      "timeout_tags"
+    ].join(", ")
+  end
 
 end # class LogStash::Filters::Aggregate
 

data/logstash-filter-aggregate.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-filter-aggregate'
-  s.version = '2.3.1'
+  s.version = '2.4.0'
   s.licenses = ['Apache License (2.0)']
   s.summary = "The aim of this filter is to aggregate information available among several events (typically log lines) belonging to a same task, and finally push aggregated information into final task event."
   s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

data/spec/filters/aggregate_spec.rb

@@ -6,7 +6,7 @@ require_relative "aggregate_spec_helper"
 describe LogStash::Filters::Aggregate do
 
   before(:each) do
-    set_eviction_instance(nil)
+    reset_timeout_management()
     aggregate_maps.clear()
     @start_filter = setup_filter({ "map_action" => "create", "code" => "map['sql_duration'] = 0" })
     @update_filter = setup_filter({ "map_action" => "update", "code" => "map['sql_duration'] += event['duration']" })
@@ -17,7 +17,7 @@ describe LogStash::Filters::Aggregate do
     describe "and receiving an event without task_id" do
       it "does not record it" do
         @start_filter.filter(event())
-        expect(aggregate_maps).to be_empty
+        expect(aggregate_maps["%{taskid}"]).to be_empty
       end
     end
     describe "and receiving an event with task_id" do
@@ -25,10 +25,10 @@ describe LogStash::Filters::Aggregate do
         event = start_event("taskid" => "id123")
         @start_filter.filter(event)
 
-        expect(aggregate_maps.size).to eq(1)
-        expect(aggregate_maps["id123"]).not_to be_nil
-        expect(aggregate_maps["id123"].creation_timestamp).to be >= event["@timestamp"]
-        expect(aggregate_maps["id123"].map["sql_duration"]).to eq(0)
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
+        expect(aggregate_maps["%{taskid}"]["id123"]).not_to be_nil
+        expect(aggregate_maps["%{taskid}"]["id123"].creation_timestamp).to be >= event["@timestamp"]
+        expect(aggregate_maps["%{taskid}"]["id123"].map["sql_duration"]).to eq(0)
       end
     end
 
@@ -45,9 +45,9 @@ describe LogStash::Filters::Aggregate do
         second_start_event = start_event("taskid" => "id124")
         @start_filter.filter(second_start_event)
 
-        expect(aggregate_maps.size).to eq(1)
-        expect(aggregate_maps["id124"].creation_timestamp).to be < second_start_event["@timestamp"]
-        expect(aggregate_maps["id124"].map["sql_duration"]).to eq(first_update_event["duration"])
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
+        expect(aggregate_maps["%{taskid}"]["id124"].creation_timestamp).to be < second_start_event["@timestamp"]
+        expect(aggregate_maps["%{taskid}"]["id124"].map["sql_duration"]).to eq(first_update_event["duration"])
       end
     end
   end
@@ -59,7 +59,7 @@ describe LogStash::Filters::Aggregate do
           end_event = end_event("taskid" => "id124")
           @end_filter.filter(end_event)
 
-          expect(aggregate_maps).to be_empty
+          expect(aggregate_maps["%{taskid}"]).to be_empty
           expect(end_event["sql_duration"]).to be_nil
         end
       end
@@ -72,7 +72,7 @@ describe LogStash::Filters::Aggregate do
         @task_id_value = "id_123"
         @start_event = start_event({"taskid" => @task_id_value})
         @start_filter.filter(@start_event)
-        expect(aggregate_maps.size).to eq(1)
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
       end
 
       describe "and receiving an end event" do
@@ -80,7 +80,7 @@ describe LogStash::Filters::Aggregate do
           it "does nothing" do
             end_event = end_event()
             @end_filter.filter(end_event)
-            expect(aggregate_maps.size).to eq(1)
+            expect(aggregate_maps["%{taskid}"].size).to eq(1)
             expect(end_event["sql_duration"]).to be_nil
           end
         end
@@ -90,21 +90,21 @@ describe LogStash::Filters::Aggregate do
             different_id_value = @task_id_value + "_different"
             @end_filter.filter(end_event("taskid" => different_id_value))
 
-            expect(aggregate_maps.size).to eq(1)
-            expect(aggregate_maps[@task_id_value]).not_to be_nil
+            expect(aggregate_maps["%{taskid}"].size).to eq(1)
+            expect(aggregate_maps["%{taskid}"][@task_id_value]).not_to be_nil
           end
         end
 
         describe "and the same id of the 'start event'" do
           it "add 'sql_duration' field to the end event and deletes the aggregate map associated to taskid" do
-            expect(aggregate_maps.size).to eq(1)
+            expect(aggregate_maps["%{taskid}"].size).to eq(1)
 
             @update_filter.filter(update_event("taskid" => @task_id_value, "duration" => 2))
 
             end_event = end_event("taskid" => @task_id_value)
             @end_filter.filter(end_event)
 
-            expect(aggregate_maps).to be_empty
+            expect(aggregate_maps["%{taskid}"]).to be_empty
             expect(end_event["sql_duration"]).to eq(2)
           end
 
@@ -117,7 +117,7 @@ describe LogStash::Filters::Aggregate do
     it "works as well as with a string task id" do
       start_event = start_event("taskid" => 124)
       @start_filter.filter(start_event)
-      expect(aggregate_maps.size).to eq(1)
+      expect(aggregate_maps["%{taskid}"].size).to eq(1)
     end
   end
 
@@ -138,25 +138,25 @@ describe LogStash::Filters::Aggregate do
       @task_id_value = "id_123"
       @start_event = start_event({"taskid" => @task_id_value})
       @start_filter.filter(@start_event)
-      expect(aggregate_maps.size).to eq(1)
+      expect(aggregate_maps["%{taskid}"].size).to eq(1)
     end
 
     describe "no timeout defined in none filter" do
       it "defines a default timeout on a default filter" do
-        set_eviction_instance(nil)
-        expect(eviction_instance).to be_nil
+        reset_timeout_management()
+        expect(taskid_eviction_instance).to be_nil
         @end_filter.flush()
-        expect(eviction_instance).to eq(@end_filter)
+        expect(taskid_eviction_instance).to eq(@end_filter)
         expect(@end_filter.timeout).to eq(LogStash::Filters::Aggregate::DEFAULT_TIMEOUT)
       end
     end
 
     describe "timeout is defined on another filter" do
-      it "eviction_instance is not updated" do
-        expect(eviction_instance).not_to be_nil
+      it "taskid eviction_instance is not updated" do
+        expect(taskid_eviction_instance).not_to be_nil
         @start_filter.flush()
-        expect(eviction_instance).not_to eq(@start_filter)
-        expect(eviction_instance).to eq(@end_filter)
+        expect(taskid_eviction_instance).not_to eq(@start_filter)
+        expect(taskid_eviction_instance).to eq(@end_filter)
       end
     end
 
@@ -164,20 +164,20 @@ describe LogStash::Filters::Aggregate do
       it "event is not removed" do
         sleep(2)
         @start_filter.flush()
-        expect(aggregate_maps.size).to eq(1)
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
       end
     end
 
     describe "timeout defined on the filter" do
       it "event is not removed if not expired" do
         entries = @end_filter.flush()
-        expect(aggregate_maps.size).to eq(1)
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
         expect(entries).to be_empty
       end
       it "removes event if expired and creates a new timeout event" do
         sleep(2)
         entries = @end_filter.flush()
-        expect(aggregate_maps).to be_empty
+        expect(aggregate_maps["%{taskid}"]).to be_empty
         expect(entries.size).to eq(1)
         expect(entries[0]['my_id']).to eq("id_123") # task id 
         expect(entries[0]["sql_duration"]).to eq(0) # Aggregation map
@@ -186,21 +186,29 @@ describe LogStash::Filters::Aggregate do
       end
     end
 
+    describe "timeout defined on another filter with another task_id pattern" do
+      it "does not remove event" do
+        another_filter = setup_filter({ "task_id" => "%{another_taskid}", "code" => "", "timeout" => 1 })
+        sleep(2)
+        entries = another_filter.flush()
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
+        expect(entries).to be_empty
+      end
+    end
   end
 
   context "aggregate_maps_path option is defined, " do
     describe "close event append then register event append, " do
       it "stores aggregate maps to configured file and then loads aggregate maps from file" do
-        
         store_file = "aggregate_maps"
         expect(File.exist?(store_file)).to be false
           
         store_filter = setup_filter({ "code" => "map['sql_duration'] = 0", "aggregate_maps_path" => store_file })
-        expect(aggregate_maps).to be_empty
+        expect(aggregate_maps["%{taskid}"]).to be_empty
 
         start_event = start_event("taskid" => 124)
         filter = store_filter.filter(start_event)
-        expect(aggregate_maps.size).to eq(1)
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
         
         store_filter.close()
         expect(File.exist?(store_file)).to be true
@@ -208,44 +216,49 @@ describe LogStash::Filters::Aggregate do
 
         store_filter = setup_filter({ "code" => "map['sql_duration'] = 0", "aggregate_maps_path" => store_file })
         expect(File.exist?(store_file)).to be false
-        expect(aggregate_maps.size).to eq(1)
-        
+        expect(aggregate_maps["%{taskid}"].size).to eq(1)
       end
     end
 
     describe "when aggregate_maps_path option is defined in 2 instances, " do
       it "raises Logstash::ConfigurationError" do
-
         expect {
           setup_filter({ "code" => "", "aggregate_maps_path" => "aggregate_maps1" })
           setup_filter({ "code" => "", "aggregate_maps_path" => "aggregate_maps2" })
         }.to raise_error(LogStash::ConfigurationError)
-        
       end
     end
   end
   
   context "push_previous_map_as_event option is defined, " do 
+    describe "when push_previous_map_as_event option is activated on another filter with same task_id pattern" do
+      it "should throw a LogStash::ConfigurationError" do
+        expect {
+          setup_filter({"code" => "map['taskid'] = event['taskid']", "push_previous_map_as_event" => true})
+        }.to raise_error(LogStash::ConfigurationError)
+      end
+    end
+    
     describe "when a new task id is detected, " do
       it "should push previous map as new event" do
-        push_filter = setup_filter({ "code" => "map['taskid'] = event['taskid']", "push_previous_map_as_event" => true, "timeout" => 5 })
-        push_filter.filter(event({"taskid" => "1"})) { |yield_event| fail "task 1 shouldn't have yield event" }
-        push_filter.filter(event({"taskid" => "2"})) { |yield_event| expect(yield_event["taskid"]).to eq("1") }
-        expect(aggregate_maps.size).to eq(1)
+        push_filter = setup_filter({ "task_id" => "%{ppm_id}", "code" => "map['ppm_id'] = event['ppm_id']", "push_previous_map_as_event" => true, "timeout" => 5 })
+        push_filter.filter(event({"ppm_id" => "1"})) { |yield_event| fail "task 1 shouldn't have yield event" }
+        push_filter.filter(event({"ppm_id" => "2"})) { |yield_event| expect(yield_event["ppm_id"]).to eq("1") }
+        expect(aggregate_maps["%{ppm_id}"].size).to eq(1)
       end
     end
 
     describe "when timeout happens, " do
       it "flush method should return last map as new event" do
-        push_filter = setup_filter({ "code" => "map['taskid'] = event['taskid']", "push_previous_map_as_event" => true, "timeout" => 1, "timeout_code" => "event['test'] = 'testValue'" })
-        push_filter.filter(event({"taskid" => "1"}))
+        push_filter = setup_filter({ "task_id" => "%{ppm_id}", "code" => "map['ppm_id'] = event['ppm_id']", "push_previous_map_as_event" => true, "timeout" => 1, "timeout_code" => "event['test'] = 'testValue'" })
+        push_filter.filter(event({"ppm_id" => "1"}))
         sleep(2)
         events_to_flush = push_filter.flush()
         expect(events_to_flush).not_to be_nil
         expect(events_to_flush.size).to eq(1)
-        expect(events_to_flush[0]["taskid"]).to eq("1")
+        expect(events_to_flush[0]["ppm_id"]).to eq("1")
         expect(events_to_flush[0]['test']).to eq("testValue")
-        expect(aggregate_maps.size).to eq(0)
+        expect(aggregate_maps["%{ppm_id}"].size).to eq(0)
       end
     end
   end

data/spec/filters/aggregate_spec_helper.rb

@@ -39,11 +39,11 @@ def aggregate_maps()
 	LogStash::Filters::Aggregate.class_variable_get(:@@aggregate_maps)
 end
 
-def eviction_instance()
-	LogStash::Filters::Aggregate.class_variable_get(:@@eviction_instance)
+def taskid_eviction_instance()
+	LogStash::Filters::Aggregate.class_variable_get(:@@eviction_instance_map)["%{taskid}"]
 end
 
-def set_eviction_instance(new_value)
-	LogStash::Filters::Aggregate.class_variable_set(:@@eviction_instance, new_value)
+def reset_timeout_management()
+	LogStash::Filters::Aggregate.class_variable_set(:@@default_timeout, nil)
+  LogStash::Filters::Aggregate.class_variable_get(:@@eviction_instance_map).clear()
 end
-

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-aggregate
 version: !ruby/object:Gem::Version
-  version: 2.3.1
+  version: 2.4.0
 platform: ruby
 authors:
 - Elastic
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2016-10-01 00:00:00.000000000 Z
+date: 2016-10-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement