logstash-filter-aggregate 2.6.4 → 2.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dae791a411f617516bd80f0481456ce0b06e2197
-  data.tar.gz: b7a082089b57954c853d4966f6f5c7d7f82509a8
+  metadata.gz: 155d6cb60a93bfdc3fb4d14e67c775e59e0b25cc
+  data.tar.gz: 3d61aa6dbd824619fe613d295f958d5e4abe70a7
 SHA512:
-  metadata.gz: f5c3902f05ae72c511c48f7fb6df713f1a904c0528bcdca11b2bed0f8af00afd30ad054688f34d5ff8f436b20aebaeb2ae2ce27b0572e63797a604296bc7b047
-  data.tar.gz: a82642ab7984e484c7b4744dc4c6f12f46abe86c6b39ae3eb914d2c822d15d5f6ba0563d2f9965707080fbca9baf822a3010eac2c69b23d65c681e130c6f90a5
+  metadata.gz: ab53e0c19de46018254cfc5f545d96a6796556321a8841775aff1c69d1b1ed02c0e3f8df1b89a9934f3e6475f1d720b6215b81f530000275c4ff7b7cd8f6ecc4
+  data.tar.gz: 1b7a7d1fc22c049b0593c84cde188c3bcc1654ae267840789786110d9bb78e1fe0917f3dcb7393173a9b0bf37588d14358252b959b438716761034466731cf46

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 2.7.0
+ - new feature: add support for multiple pipelines (for Logstash 6.0+)  
+   aggregate maps, timeout options, and aggregate_maps_path are now stored per pipeline.  
+   each pipeline is independant.
+ - docs: fix break lines in documentation examples  
+
 ## 2.6.4
 - bugfix: fix a NPE issue at Logstash 6.0 shutdown
 - docs: remove all redundant documentation in aggregate.rb (now only present in docs/index.asciidoc)

data/docs/index.asciidoc

@@ -195,9 +195,11 @@ filter {
 [id="plugins-{type}s-{plugin}-example4"]
 ==== Example #4 : no end event and tasks come one after the other
 
-Fourth use case : like example #3, you have no specific end event, but also, tasks come one after the other. +
-That is to say : tasks are not interlaced. All task1 events come, then all task2 events come, ... +
-In that case, you don't want to wait task timeout to flush aggregation map. +
+Fourth use case : like example #3, you have no specific end event, but also, tasks come one after the other.
+
+That is to say : tasks are not interlaced. All task1 events come, then all task2 events come, ...
+
+In that case, you don't want to wait task timeout to flush aggregation map.
 
 * A typical case is aggregating results from jdbc input plugin.
 * Given that you have this SQL query : `SELECT country_name, town_name FROM town`
@@ -245,20 +247,31 @@ In that case, you don't want to wait task timeout to flush aggregation map. +
 [id="plugins-{type}s-{plugin}-example5"]
 ==== Example #5 : no end event and push events as soon as possible
 
-Fifth use case: like example #3, there is no end event. +
-Events keep comming for an indefinite time and you want to push the aggregation map as soon as possible after the last user interaction without waiting for the `timeout`. +
-This allows to have the aggregated events pushed closer to real time. +
+Fifth use case: like example #3, there is no end event.
+
+Events keep comming for an indefinite time and you want to push the aggregation map as soon as possible after the last user interaction without waiting for the `timeout`.
+
+This allows to have the aggregated events pushed closer to real time.
+
+
+A typical case is aggregating or tracking user behaviour.
+
+We can track a user by its ID through the events, however once the user stops interacting, the events stop coming in.
+
+There is no specific event indicating the end of the user's interaction.
+
+The user ineraction will be considered as ended when no events for the specified user (task_id) arrive after the specified inactivity_timeout`.
+
+If the user continues interacting for longer than `timeout` seconds (since first event), the aggregation map will still be deleted and pushed as a new event when timeout occurs.
 
-A typical case is aggregating or tracking user behaviour. +
-We can track a user by its ID through the events, however once the user stops interacting, the events stop coming in. +
-There is no specific event indicating the end of the user's interaction. +
-The user ineraction will be considered as ended when no events for the specified user (task_id) arrive after the specified inactivity_timeout`. +
-If the user continues interacting for longer than `timeout` seconds (since first event), the aggregation map will still be deleted and pushed as a new event when timeout occurs. +
 The difference with example #3 is that the events will be pushed as soon as the user stops interacting for `inactivity_timeout` seconds instead of waiting for the end of `timeout` seconds since first event.
 
-In this case, we can enable the option 'push_map_as_event_on_timeout' to enable pushing the aggregation map as a new event when inactivity timeout occurs. +
-In addition, we can enable 'timeout_code' to execute code on the populated timeout event. +
-We can also add 'timeout_task_id_field' so we can correlate the task_id, which in this case would be the user's ID. +
+In this case, we can enable the option 'push_map_as_event_on_timeout' to enable pushing the aggregation map as a new event when inactivity timeout occurs.
+
+In addition, we can enable 'timeout_code' to execute code on the populated timeout event.
+
+We can also add 'timeout_task_id_field' so we can correlate the task_id, which in this case would be the user's ID.
+
 
 * Given these logs:
 
@@ -315,7 +328,7 @@ filter {
 * an aggregate map is tied to one task_id value which is tied to one task_id pattern. So if you have 2 filters with different task_id patterns, even if you have same task_id value, they won't share the same aggregate map.
 * in one filter configuration, it is recommanded to define a timeout option to protect the feature against unterminated tasks. It tells the filter to delete expired maps
 * if no timeout is defined, by default, all maps older than 1800 seconds are automatically deleted
-* all timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_task_id_field, timeout_tags
+* all timeout options have to be defined in only one aggregate filter per task_id pattern (per pipeline). Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_task_id_field, timeout_tags
 * if `code` execution raises an exception, the error is logged and event is tagged '_aggregateexception'
 
 
@@ -366,7 +379,7 @@ The path to file where aggregate maps are stored when Logstash stops
 and are loaded from when Logstash starts.
 
 If not defined, aggregate maps will not be stored at Logstash stop and will be lost.
-Must be defined in only one aggregate filter (as aggregate maps are global).
+Must be defined in only one aggregate filter per pipeline (as aggregate maps are shared at pipeline level).
 
 Example:
 [source,ruby]

data/lib/logstash/filters/aggregate.rb

@@ -41,6 +41,15 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   config :timeout_tags, :validate => :array, :required => false, :default => []
 
 
+  # ################## #
+  # INSTANCE VARIABLES #
+  # ################## #
+  
+
+  # pointer to current pipeline context
+  attr_accessor :current_pipeline
+
+
   # ################ #
   # STATIC VARIABLES #
   # ################ #
@@ -48,29 +57,9 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
 
   # Default timeout (in seconds) when not defined in plugin configuration
   DEFAULT_TIMEOUT = 1800
-
-  # This is the state of the filter.
-  # For each entry, key is "task_id" and value is a map freely updatable by 'code' config
-  @@aggregate_maps = {}
-
-  # Mutex used to synchronize access to 'aggregate_maps'
-  @@mutex = Mutex.new
-
-  # Default timeout for task_id patterns where timeout is not defined in Logstash filter configuration
-  @@default_timeout = nil
-
-  # For each "task_id" pattern, defines which Aggregate instance will process flush() call, processing expired Aggregate elements (older than timeout)
-  # For each entry, key is "task_id pattern" and value is "aggregate instance"
-  @@flush_instance_map = {}
-
-  # last time where timeout management in flush() method was launched, per "task_id" pattern
-  @@last_flush_timestamp_map = {}
-
-  # flag indicating if aggregate_maps_path option has been already set on one aggregate instance
-  @@aggregate_maps_path_set = false
-
-  # defines which Aggregate instance will close Aggregate static variables
-  @@static_close_instance = nil
+  
+  # Store all shared aggregate attributes per pipeline id
+  @@pipelines = {}
 
 
   # ####### #
@@ -88,7 +77,7 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     if !@task_id.match(/%\{.+\}/)
       raise LogStash::ConfigurationError, "Aggregate plugin: task_id pattern '#{@task_id}' must contain a dynamic expression like '%{field}'"
     end
-
+    
     # process lambda expression to call in each filter call
     eval("@codeblock = lambda { |event, map| #{@code} }", binding, "(aggregate filter code)")
 
@@ -97,54 +86,60 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
       eval("@timeout_codeblock = lambda { |event| #{@timeout_code} }", binding, "(aggregate filter timeout code)")
     end
 
-    @@mutex.synchronize do
+    # init pipeline context
+    @@pipelines[pipeline_id] ||= LogStash::Filters::Aggregate::Pipeline.new();
+    @current_pipeline = @@pipelines[pipeline_id]
+
+    @current_pipeline.mutex.synchronize do
 
       # timeout management : define eviction_instance for current task_id pattern
       if has_timeout_options?
-        if @@flush_instance_map.has_key?(@task_id)
+        if @current_pipeline.flush_instance_map.has_key?(@task_id)
           # all timeout options have to be defined in only one aggregate filter per task_id pattern
           raise LogStash::ConfigurationError, "Aggregate plugin: For task_id pattern '#{@task_id}', there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : #{display_timeout_options}"
         end
-        @@flush_instance_map[@task_id] = self
+        @current_pipeline.flush_instance_map[@task_id] = self
         @logger.debug("Aggregate timeout for '#{@task_id}' pattern: #{@timeout} seconds")
       end
 
       # timeout management : define default_timeout
-      if !@timeout.nil? && (@@default_timeout.nil? || @timeout < @@default_timeout)
-        @@default_timeout = @timeout
+      if !@timeout.nil? && (@current_pipeline.default_timeout.nil? || @timeout < @current_pipeline.default_timeout)
+        @current_pipeline.default_timeout = @timeout
         @logger.debug("Aggregate default timeout: #{@timeout} seconds")
       end
 
       # inactivity timeout management: make sure it is lower than timeout
-      if !@inactivity_timeout.nil? && ((!@timeout.nil? && @inactivity_timeout > @timeout) || (!@@default_timeout.nil? && @inactivity_timeout > @@default_timeout))
+      if !@inactivity_timeout.nil? && ((!@timeout.nil? && @inactivity_timeout > @timeout) || (!@current_pipeline.default_timeout.nil? && @inactivity_timeout > @current_pipeline.default_timeout))
         raise LogStash::ConfigurationError, "Aggregate plugin: For task_id pattern #{@task_id}, inactivity_timeout must be lower than timeout"
       end
 
-      # reinit static_close_instance (if necessary)
-      if !@@aggregate_maps_path_set && !@@static_close_instance.nil?
-        @@static_close_instance = nil
+      # reinit pipeline_close_instance (if necessary)
+      if !@current_pipeline.aggregate_maps_path_set && !@current_pipeline.pipeline_close_instance.nil?
+        @current_pipeline.pipeline_close_instance = nil
       end
 
-      # check if aggregate_maps_path option has already been set on another instance else set @@aggregate_maps_path_set
+      # check if aggregate_maps_path option has already been set on another instance else set @current_pipeline.aggregate_maps_path_set
       if !@aggregate_maps_path.nil?
-        if @@aggregate_maps_path_set
-          @@aggregate_maps_path_set = false
+        if @current_pipeline.aggregate_maps_path_set
+          @current_pipeline.aggregate_maps_path_set = false
           raise LogStash::ConfigurationError, "Aggregate plugin: Option 'aggregate_maps_path' must be set on only one aggregate filter"
         else
-          @@aggregate_maps_path_set = true
-          @@static_close_instance = self
+          @current_pipeline.aggregate_maps_path_set = true
+          @current_pipeline.pipeline_close_instance = self
         end
       end
 
       # load aggregate maps from file (if option defined)
       if !@aggregate_maps_path.nil? && File.exist?(@aggregate_maps_path)
-        File.open(@aggregate_maps_path, "r") { |from_file| @@aggregate_maps.merge!(Marshal.load(from_file)) }
+        File.open(@aggregate_maps_path, "r") { |from_file| @current_pipeline.aggregate_maps.merge!(Marshal.load(from_file)) }
         File.delete(@aggregate_maps_path)
         @logger.info("Aggregate maps loaded from : #{@aggregate_maps_path}")
       end
 
       # init aggregate_maps
-      @@aggregate_maps[@task_id] ||= {}
+      @current_pipeline.aggregate_maps[@task_id] ||= {}
+      
+      
     end
   end
 
@@ -154,25 +149,21 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
 
     @logger.debug("Aggregate close call", :code => @code)
 
-    # define static close instance if none is already defined
-    @@static_close_instance = self if @@static_close_instance.nil?
+    # define pipeline close instance if none is already defined
+    @current_pipeline.pipeline_close_instance = self if @current_pipeline.pipeline_close_instance.nil?
 
-    if @@static_close_instance == self
+    if @current_pipeline.pipeline_close_instance == self
       # store aggregate maps to file (if option defined)
-      @@mutex.synchronize do
-        @@aggregate_maps.delete_if { |key, value| value.empty? }
-        if !@aggregate_maps_path.nil? && !@@aggregate_maps.empty?
-          File.open(@aggregate_maps_path, "w"){ |to_file| Marshal.dump(@@aggregate_maps, to_file) }
+      @current_pipeline.mutex.synchronize do
+        @current_pipeline.aggregate_maps.delete_if { |key, value| value.empty? }
+        if !@aggregate_maps_path.nil? && !@current_pipeline.aggregate_maps.empty?
+          File.open(@aggregate_maps_path, "w"){ |to_file| Marshal.dump(@current_pipeline.aggregate_maps, to_file) }
           @logger.info("Aggregate maps stored to : #{@aggregate_maps_path}")
         end
-        @@aggregate_maps.clear()
       end
 
-      # reinit static variables for Logstash reload
-      @@default_timeout = nil
-      @@flush_instance_map = {}
-      @@last_flush_timestamp_map = {}
-      @@aggregate_maps_path_set = false
+      # remove pipeline context for Logstash reload
+      @@pipelines.delete(pipeline_id)
     end
 
   end
@@ -189,21 +180,21 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     event_to_yield = nil
 
     # protect aggregate_maps against concurrent access, using a mutex
-    @@mutex.synchronize do
+    @current_pipeline.mutex.synchronize do
 
       # retrieve the current aggregate map
-      aggregate_maps_element = @@aggregate_maps[@task_id][task_id]
+      aggregate_maps_element = @current_pipeline.aggregate_maps[@task_id][task_id]
 
 
       # create aggregate map, if it doesn't exist
       if aggregate_maps_element.nil?
         return if @map_action == "update"
         # create new event from previous map, if @push_previous_map_as_event is enabled
-        if @push_previous_map_as_event && !@@aggregate_maps[@task_id].empty?
+        if @push_previous_map_as_event && !@current_pipeline.aggregate_maps[@task_id].empty?
           event_to_yield = extract_previous_map_as_event()
         end
         aggregate_maps_element = LogStash::Filters::Aggregate::Element.new(Time.now);
-        @@aggregate_maps[@task_id][task_id] = aggregate_maps_element
+        @current_pipeline.aggregate_maps[@task_id][task_id] = aggregate_maps_element
       else
         return if @map_action == "create"
       end
@@ -225,7 +216,7 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
       end
 
       # delete the map if task is ended
-      @@aggregate_maps[@task_id].delete(task_id) if @end_of_task
+      @current_pipeline.aggregate_maps[@task_id].delete(task_id) if @end_of_task
 
     end
 
@@ -272,7 +263,7 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
 
   # Extract the previous map in aggregate maps, and return it as a new Logstash event
   def extract_previous_map_as_event
-    previous_entry = @@aggregate_maps[@task_id].shift()
+    previous_entry = @current_pipeline.aggregate_maps[@task_id].shift()
     previous_task_id = previous_entry[0]
     previous_map = previous_entry[1].map
     return create_timeout_event(previous_map, previous_task_id)
@@ -289,26 +280,26 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
     @logger.debug("Aggregate flush call with #{options}")
 
     # Protection against no timeout defined by Logstash conf : define a default eviction instance with timeout = DEFAULT_TIMEOUT seconds
-    if @@default_timeout.nil?
-      @@default_timeout = DEFAULT_TIMEOUT
+    if @current_pipeline.default_timeout.nil?
+      @current_pipeline.default_timeout = DEFAULT_TIMEOUT
     end
-    if !@@flush_instance_map.has_key?(@task_id)
-      @@flush_instance_map[@task_id] = self
-      @timeout = @@default_timeout
-    elsif @@flush_instance_map[@task_id].timeout.nil?
-      @@flush_instance_map[@task_id].timeout = @@default_timeout
+    if !@current_pipeline.flush_instance_map.has_key?(@task_id)
+      @current_pipeline.flush_instance_map[@task_id] = self
+      @timeout = @current_pipeline.default_timeout
+    elsif @current_pipeline.flush_instance_map[@task_id].timeout.nil?
+      @current_pipeline.flush_instance_map[@task_id].timeout = @current_pipeline.default_timeout
     end
 
-    if @@flush_instance_map[@task_id].inactivity_timeout.nil?
-      @@flush_instance_map[@task_id].inactivity_timeout = @@flush_instance_map[@task_id].timeout
+    if @current_pipeline.flush_instance_map[@task_id].inactivity_timeout.nil?
+      @current_pipeline.flush_instance_map[@task_id].inactivity_timeout = @current_pipeline.flush_instance_map[@task_id].timeout
     end
 
     # Launch timeout management only every interval of (@inactivity_timeout / 2) seconds or at Logstash shutdown
-    if @@flush_instance_map[@task_id] == self && !@@aggregate_maps[@task_id].nil? && (!@@last_flush_timestamp_map.has_key?(@task_id) || Time.now > @@last_flush_timestamp_map[@task_id] + @inactivity_timeout / 2 || options[:final])
+    if @current_pipeline.flush_instance_map[@task_id] == self && !@current_pipeline.aggregate_maps[@task_id].nil? && (!@current_pipeline.last_flush_timestamp_map.has_key?(@task_id) || Time.now > @current_pipeline.last_flush_timestamp_map[@task_id] + @inactivity_timeout / 2 || options[:final])
       events_to_flush = remove_expired_maps()
 
       # at Logstash shutdown, if push_previous_map_as_event is enabled, it's important to force flush (particularly for jdbc input plugin)
-      if options[:final] && @push_previous_map_as_event && !@@aggregate_maps[@task_id].empty?
+      if options[:final] && @push_previous_map_as_event && !@current_pipeline.aggregate_maps[@task_id].empty?
         events_to_flush << extract_previous_map_as_event()
       end
 
@@ -318,7 +309,7 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
       end
 
       # update last flush timestamp
-      @@last_flush_timestamp_map[@task_id] = Time.now
+      @current_pipeline.last_flush_timestamp_map[@task_id] = Time.now
 
       # return events to flush into Logstash pipeline
       return events_to_flush
@@ -329,18 +320,18 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
   end
 
 
-  # Remove the expired Aggregate maps from @@aggregate_maps if they are older than timeout or if no new event has been received since inactivity_timeout.
+  # Remove the expired Aggregate maps from @current_pipeline.aggregate_maps if they are older than timeout or if no new event has been received since inactivity_timeout.
   # If @push_previous_map_as_event option is set, or @push_map_as_event_on_timeout is set, expired maps are returned as new events to be flushed to Logstash pipeline.
   def remove_expired_maps()
     events_to_flush = []
     min_timestamp = Time.now - @timeout
     min_inactivity_timestamp = Time.now - @inactivity_timeout
 
-    @@mutex.synchronize do
+    @current_pipeline.mutex.synchronize do
 
-      @logger.debug("Aggregate remove_expired_maps call with '#{@task_id}' pattern and #{@@aggregate_maps[@task_id].length} maps")
+      @logger.debug("Aggregate remove_expired_maps call with '#{@task_id}' pattern and #{@current_pipeline.aggregate_maps[@task_id].length} maps")
 
-      @@aggregate_maps[@task_id].delete_if do |key, element|
+      @current_pipeline.aggregate_maps[@task_id].delete_if do |key, element|
         if element.creation_timestamp < min_timestamp || element.lastevent_timestamp < min_inactivity_timestamp
           if @push_previous_map_as_event || @push_map_as_event_on_timeout
             events_to_flush << create_timeout_event(element.map, key)
@@ -379,6 +370,15 @@ class LogStash::Filters::Aggregate < LogStash::Filters::Base
       "timeout_tags"
     ].join(", ")
   end
+  
+  # return current pipeline id
+  def pipeline_id()
+    if @execution_context
+      return @execution_context.pipeline_id
+    else
+      return pipeline_id = "main"
+    end
+  end
 
 end # class LogStash::Filters::Aggregate
 
@@ -393,3 +393,33 @@ class LogStash::Filters::Aggregate::Element
     @map = {}
   end
 end
+
+# shared aggregate attributes for each pipeline
+class LogStash::Filters::Aggregate::Pipeline
+  
+  attr_accessor :aggregate_maps, :mutex, :default_timeout, :flush_instance_map, :last_flush_timestamp_map, :aggregate_maps_path_set, :pipeline_close_instance
+
+  def initialize()
+    # Stores all aggregate maps, per task_id pattern, then per task_id value
+    @aggregate_maps = {}
+  
+    # Mutex used to synchronize access to 'aggregate_maps'
+    @mutex = Mutex.new
+  
+    # Default timeout for task_id patterns where timeout is not defined in Logstash filter configuration
+    @default_timeout = nil
+  
+    # For each "task_id" pattern, defines which Aggregate instance will process flush() call, processing expired Aggregate elements (older than timeout)
+    # For each entry, key is "task_id pattern" and value is "aggregate instance"
+    @flush_instance_map = {}
+  
+    # last time where timeout management in flush() method was launched, per "task_id" pattern
+    @last_flush_timestamp_map = {}
+  
+    # flag indicating if aggregate_maps_path option has been already set on one aggregate instance
+    @aggregate_maps_path_set = false
+  
+    # defines which Aggregate instance will close Aggregate variables associated to current pipeline
+    @pipeline_close_instance = nil
+  end
+end

data/logstash-filter-aggregate.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-filter-aggregate'
-  s.version         = '2.6.4'
+  s.version         = '2.7.0'
   s.licenses = ['Apache License (2.0)']
   s.summary = 'The aim of this filter is to aggregate information available among several events (typically log lines) belonging to a same task, and finally push aggregated information into final task event.'
   s.description = 'This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program'

data/spec/filters/aggregate_spec.rb

@@ -6,7 +6,7 @@ require_relative "aggregate_spec_helper"
 describe LogStash::Filters::Aggregate do
 
   before(:each) do
-    reset_static_variables()
+    reset_pipeline_variables()
     @start_filter = setup_filter({ "map_action" => "create", "code" => "map['sql_duration'] = 0" })
     @update_filter = setup_filter({ "map_action" => "update", "code" => "map['sql_duration'] += event.get('duration')" })
     @end_filter = setup_filter({"timeout_task_id_field" => "my_id", "push_map_as_event_on_timeout" => true, "map_action" => "update", "code" => "event.set('sql_duration', map['sql_duration'])", "end_of_task" => true, "timeout" => 5, "inactivity_timeout" => 2, "timeout_code" => "event.set('test', 'testValue')", "timeout_tags" => ["tag1", "tag2"] })
@@ -268,6 +268,7 @@ describe LogStash::Filters::Aggregate do
     describe "close event append then register event append, " do
       it "stores aggregate maps to configured file and then loads aggregate maps from file" do
         store_file = "aggregate_maps"
+        File.delete(store_file) if File.exist?(store_file)
         expect(File.exist?(store_file)).to be false
 
         one_filter = setup_filter({ "task_id" => "%{one_special_field}", "code" => ""})
@@ -284,7 +285,7 @@ describe LogStash::Filters::Aggregate do
 
         store_filter.close()
         expect(File.exist?(store_file)).to be true
-        expect(aggregate_maps).to be_empty
+        expect(current_pipeline).to be_nil
 
         one_filter = setup_filter({ "task_id" => "%{one_special_field}", "code" => ""})
         store_filter = setup_filter({ "code" => "map['sql_duration'] = 0", "aggregate_maps_path" => store_file })
@@ -306,15 +307,14 @@ describe LogStash::Filters::Aggregate do
 
   context "Logstash reload occurs, " do
     describe "close method is called, " do
-      it "reinitializes static variables" do
+      it "reinitializes pipelines" do
         @end_filter.close()
-        expect(aggregate_maps).to be_empty
-        expect(taskid_eviction_instance).to be_nil
-        expect(static_close_instance).not_to be_nil
-        expect(aggregate_maps_path_set).to be false
+        expect(current_pipeline).to be_nil
 
         @end_filter.register()
-        expect(static_close_instance).to be_nil
+        expect(current_pipeline).not_to be_nil
+        expect(aggregate_maps).not_to be_nil
+        expect(pipeline_close_instance).to be_nil
       end
     end
   end

data/spec/filters/aggregate_spec_helper.rb

@@ -33,31 +33,40 @@ def filter(event)
 	@end_filter.filter(event)
 end
 
+def pipelines()
+  LogStash::Filters::Aggregate.class_variable_get(:@@pipelines)
+end
+
+def current_pipeline()
+  pipelines()['main']
+end
+
 def aggregate_maps()
-	LogStash::Filters::Aggregate.class_variable_get(:@@aggregate_maps)
+  current_pipeline().aggregate_maps
 end
 
 def taskid_eviction_instance()
-	LogStash::Filters::Aggregate.class_variable_get(:@@flush_instance_map)["%{taskid}"]
+  current_pipeline().flush_instance_map["%{taskid}"]
 end
 
-def static_close_instance()
-  LogStash::Filters::Aggregate.class_variable_get(:@@static_close_instance)
+def pipeline_close_instance()
+  current_pipeline().pipeline_close_instance
 end
 
 def aggregate_maps_path_set()
-  LogStash::Filters::Aggregate.class_variable_get(:@@aggregate_maps_path_set)
+  current_pipeline().aggregate_maps_path_set
 end
 
 def reset_timeout_management()
-	LogStash::Filters::Aggregate.class_variable_set(:@@default_timeout, nil)
-  LogStash::Filters::Aggregate.class_variable_get(:@@flush_instance_map).clear()
-  LogStash::Filters::Aggregate.class_variable_get(:@@last_flush_timestamp_map).clear()
+  current_pipeline().default_timeout = nil
+  current_pipeline().flush_instance_map.clear()
+  current_pipeline().last_flush_timestamp_map.clear()
 end
 
-def reset_static_variables()
-  reset_timeout_management()
-  aggregate_maps().clear()
-  LogStash::Filters::Aggregate.class_variable_set(:@@static_close_instance, nil)
-  LogStash::Filters::Aggregate.class_variable_set(:@@aggregate_maps_path_set, false)
+def reset_pipeline_variables()
+  pipelines().clear()
+#  reset_timeout_management()
+#  aggregate_maps().clear()
+#  current_pipeline().pipeline_close_instance = nil
+#  current_pipeline().aggregate_maps_path_set = false
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-aggregate
 version: !ruby/object:Gem::Version
-  version: 2.6.4
+  version: 2.7.0
 platform: ruby
 authors:
 - Elastic
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-10-10 00:00:00.000000000 Z
+date: 2017-11-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement