logstash-input-crowdstrike_fdr 2.1.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: c8e0b47a277210807baaf165fa9a0b1acd18fa52
+  data.tar.gz: 553647a85d8c20fd05c74bf530fee9df2b7d053f
+SHA512:
+  metadata.gz: 726387033c60271642d500faa8ec4cbf0d039a71413036615bc09bcb64582966b4bdddee342df2c6c8703822eacba5751ed10028c48fada1eacfb2337de142ab
+  data.tar.gz: e47d188d9294cfb83c2c28306cd63cddbf03351a8c918826701ada77377271dfb1729f6468ae1db4978e7ef72a185a8571d680b4871126e02c2bc74482074cec

data/CHANGELOG.md

@@ -0,0 +1,141 @@
+##2.1.2
+- FEATURE: Now it´s possible to use queue urls and names.
+- FEATURE: Add sqs long polling config parameter: sqs_wait_time_seconds
+- FIX: Valid UTF-8 byte sequences in logs are munged
+- CLEANUP: Remove tests. (as a begin for clean testing)
+##2.1.1
+- FEATURE: Enable Multiregion Support for included S3 client.
+- Add region by bucket feature
+##2.1.0
+- FEATURE: Add S3 metadata -> config :include_object_properties
+- FEATURE: Watch for threads in exception state and restart...
+##2.0.9
+-gzip dectection should return false for files smaller than gzip_signiture_bytes
+##2.0.8
+-fix nil class error
+##2.0.7
+-fix gem error
+##2.0.6
+-fix: fix crash of extender
+##2.0.5
+-fix: crash on 0 byte file
+-fix: type by folder function
+##2.0.4
+-fix: type-by-folder repair
+-fix: crash on 0 byte file
+##2.0.3
+- Increase max parsing time -> drop event if reached
+- watcher thread should raise an error in poller loop if timeout reached.
+- remove some debug logs
+
+##2.0.2
+FIX:
+- Terminate every input line by \n (BufferedReader does not)
+- Wrong input for type folder, leads to empty types
+##2.0.1
+FIX:
+- Deadlock while message decoding
+- make method stop? public
+
+##2.0.0
+Breaking Changes:
+- s3_key_prefix was never functional and will be removed. Actually only used for metadata.folder backward compatibility.
+  config for s3 paths are regex (if not exact match)
+- s3_options_by_bucket substitutes all s3_* options
+  We will merge deprecated options into the new structure for one release
+Changes:
+- Refactor plugin structure to be more modular 
+- Rework threadding design
+- introduce s3_options_by_bucket to configure settings (e.g aws_options_hash or type)
+##1.6.1
+- Fix typo in gzip error logging
+##1.6.0
+- add a test to tmp file deletion
+- revert type folder regex
+##1.5.9
+- Fix regex for type folder
+##1.5.8
+- Add some debug and a toggle for delete
+##1.5.7
+- Remove Debug poutput
+##1.5.6
+-BugFix
+##1.5.5
+- Memo to me: better testing :-) Fix msg -> message
+##1.5.4
+- BugFix
+##1.5.3
+- Try to fix requeue Problem
+## 1.5.2
+- BugFix Set Metadata bucket,key,folder
+- Feature: Possibility to fall back to old threadding model by unset consumer_threads
+## 1.5.1
+- BugFix: rescue all AWS::S3 errors
+## 1.5.0
+- Feature: Use own magicbyte detector (small&fast)
+## 1.4.9
+- Feature: Detect filetype with MagicByte 
+## 1.4.8
+- Bufix: CF Metadata events Bug #7
+- Feature: use aws-role for s3 client connection.
+## 1.4.7
+Remove from rubygems.org
+## 1.4.6
+- BugFix: jRuby > 2 : No return from block
+- BugFix: No exit on gzip error
+## 1.4.5
+- BugFix: undeclared var in rescue 
+## 1.4.4
+- Feature: make set_codec_by_folder match as regex
+  e.g.: set_codec_by_folder => { ".*-ELB-logs" => "plain"} 
+## 1.4.3
+- Fix: skip_delete on S3::Errors::AccessDenied
+- Feature: codec per s3 folder
+- Feature: Alpha phase: different credentials for s3 / default credentials for sqs
+- Feature: Find files folder. 
+## 1.4.2
+- Fix: Thread shutdown method should kill in case of wakeup fails
+## 1.4.1
+- Fix: Last event in file not decorated
+- Adjust metadata namings
+- Event decoration in private method now.
+## 1.4.0
+- Filehandling rewritten THX to logstash-input-s3 for inspiration
+- Improve performance of gzip decoding by 10x by using Java's Zlib
+- Added multithreading via config Use: consumer_threads in config
+## 1.2.0
+- Add codec suggestion by content-type
+- enrich metadata 
+- Fix some bugs
+## 1.1.9
+- Add config for s3 folder prefix, auto codec and auto type
+## 1.1.8
+- Add config switch for delivery with or without SNS
+## 1.1.6
+- Fix a nil exception in message parsing
+## 1.1.5
+- Fix loglevel for some debug messages
+## 1.1.4
+- Add Account-ID to config
+## 1.1.2
+- Fix a Bug in the S3 Key generation
+- Enable shipping throug SNS Topic (needs another toJSON)
+## 1.1.1
+- Added the ability to remove objects from S3 after processing.
+- Workaround an issue with the Ruby autoload that causes "uninitialized constant `Aws::Client::Errors`" errors.
+
+## 1.1.0
+- Logstash 5 compatibility
+
+## 1.0.3
+- added some metadata to the event (bucket and object name as commited by joshuaspence)
+- also try to unzip files ending with ".gz" (ALB logs are zipped but not marked with proper Content-Encoding)
+
+## 1.0.2
+- fix for broken UTF-8 (so we won't lose a whole s3 log file because of a single invalid line, ruby's split will die on those)
+
+## 1.0.1
+- same (because of screwed up rubygems.org release)
+
+## 1.0.0
+- Initial Release

data/CONTRIBUTORS

@@ -0,0 +1,14 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* cherweg(this fork + some bugfixes)
+* holgerjenczewski1007 (thank you for the refactoring)
+* joshuaspence (event metadata)
+* Heiko-san (initial contributor)
+* logstash-input-sqs plugin as code base
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,147 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0.
+
+## Documentation
+
+ Get logs from AWS s3 buckets as issued by an object-created event via sqs.
+
+ This plugin is based on the logstash-input-sqs plugin but doesn't log the sqs event itself.
+ Instead it assumes, that the event is an s3 object-created event and will then download
+ and process the given file.
+
+ Some issues of logstash-input-sqs, like logstash not shutting down properly, have been
+ fixed for this plugin.
+
+ In contrast to logstash-input-sqs this plugin uses the "Receive Message Wait Time"
+ configured for the sqs queue in question, a good value will be something like 10 seconds
+ to ensure a reasonable shutdown time of logstash.
+ Also use a "Default Visibility Timeout" that is high enough for log files to be downloaded
+ and processed (I think a good value should be 5-10 minutes for most use cases), the plugin will
+ avoid removing the event from the queue if the associated log file couldn't be correctly
+ passed to the processing level of logstash (e.g. downloaded content size doesn't match sqs event).
+
+ This plugin is meant for high availability setups, in contrast to logstash-input-s3 you can safely
+ use multiple logstash nodes, since the usage of sqs will ensure that each logfile is processed
+ only once and no file will get lost on node failure or downscaling for auto-scaling groups.
+ (You should use a "Message Retention Period" >= 4 days for your sqs to ensure you can survive
+ a weekend of faulty log file processing)
+ The plugin will not delete objects from s3 buckets, so make sure to have a reasonable "Lifecycle"
+ configured for your buckets, which should keep the files at least "Message Retention Period" days.
+
+ A typical setup will contain some s3 buckets containing elb, cloudtrail or other log files.
+ These will be configured to send object-created events to a sqs queue, which will be configured
+ as the source queue for this plugin.
+ (The plugin supports gzipped content if it is marked with "contend-encoding: gzip" as it is the
+ case for cloudtrail logs)
+
+ The logstash node therefore must have sqs permissions + the permissions to download objects
+ from the s3 buckets that send events to the queue.
+ (If logstash nodes are running on EC2 you should use a ServerRole to provide permissions)
+ [source,json]
+   {
+       "Version": "2012-10-17",
+       "Statement": [
+           {
+               "Effect": "Allow",
+               "Action": [
+                   "sqs:Get*",
+                   "sqs:List*",
+                   "sqs:ReceiveMessage",
+                   "sqs:ChangeMessageVisibility*",
+                   "sqs:DeleteMessage*"
+               ],
+               "Resource": [
+                   "arn:aws:sqs:us-east-1:123456789012:my-elb-log-queue"
+               ]
+           },
+           {
+               "Effect": "Allow",
+               "Action": [
+                   "s3:Get*",
+                   "s3:List*",
+                   "s3:DeleteObject"
+               ],
+               "Resource": [
+                   "arn:aws:s3:::my-elb-logs",
+                   "arn:aws:s3:::my-elb-logs/*"
+               ]
+           }
+       ]
+   }
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/codec_factory.rb

@@ -0,0 +1,37 @@
+# CodecFactory:
+# lazy-fetch codec plugins
+
+class CodecFactory
+  def initialize(logger, options)
+    @logger = logger
+    @default_codec = options[:default_codec]
+    @codec_by_folder = options[:codec_by_folder]
+    @codecs = {
+      'default' => @default_codec
+    }
+  end
+
+  def get_codec(record)
+    codec = find_codec(record)
+    if @codecs[codec].nil?
+      @codecs[codec] = get_codec_plugin(codec)
+    end
+    @logger.debug("Switching to codec #{codec}") if codec != 'default'
+    return @codecs[codec].clone
+  end
+
+  private
+
+  def find_codec(record)
+    bucket, key, folder = record[:bucket], record[:key], record[:folder]
+    unless @codec_by_folder[bucket].nil?
+      @logger.debug("Looking up codec for folder #{folder}", :codec =>  @codec_by_folder[bucket][folder])
+      return @codec_by_folder[bucket][folder] unless @codec_by_folder[bucket][folder].nil?
+    end
+    return 'default'
+  end
+
+  def get_codec_plugin(name, options = {})
+    LogStash::Plugin.lookup('codec', name).new(options)
+  end
+end

data/lib/logstash/inputs/crowdstrike_fdr.rb

@@ -0,0 +1,343 @@
+# encoding: utf-8
+require "logstash/inputs/threadable"
+require "logstash/namespace"
+require "logstash/timestamp"
+require "logstash/plugin_mixins/aws_config"
+require "logstash/shutdown_watcher"
+require "logstash/errors"
+require 'logstash/inputs/s3sqs/patch'
+require "aws-sdk"
+
+# "object-oriented interfaces on top of API clients"...
+# => Overhead. FIXME: needed?
+#require "aws-sdk-resources"
+require "fileutils"
+require "concurrent"
+require 'tmpdir'
+# unused in code:
+#require "stud/interval"
+#require "digest/md5"
+
+require 'java'
+java_import java.io.InputStream
+java_import java.io.InputStreamReader
+java_import java.io.FileInputStream
+java_import java.io.BufferedReader
+java_import java.util.zip.GZIPInputStream
+java_import java.util.zip.ZipException
+import java.lang.StringBuilder
+
+# our helper classes
+# these may go into this file for brevity...
+require_relative 'sqs/poller'
+require_relative 's3/client_factory'
+require_relative 's3/downloader'
+require_relative 'codec_factory'
+require_relative 's3snssqs/log_processor'
+
+Aws.eager_autoload!
+
+# Get logs from AWS s3 buckets as issued by an object-created event via sqs.
+#
+# This plugin is based on the logstash-input-sqs plugin but doesn't log the sqs event itself.
+# Instead it assumes, that the event is an s3 object-created event and will then download
+# and process the given file.
+#
+# Some issues of logstash-input-sqs, like logstash not shutting down properly, have been
+# fixed for this plugin.
+#
+# In contrast to logstash-input-sqs this plugin uses the "Receive Message Wait Time"
+# configured for the sqs queue in question, a good value will be something like 10 seconds
+# to ensure a reasonable shutdown time of logstash.
+# Also use a "Default Visibility Timeout" that is high enough for log files to be downloaded
+# and processed (I think a good value should be 5-10 minutes for most use cases), the plugin will
+# avoid removing the event from the queue if the associated log file couldn't be correctly
+# passed to the processing level of logstash (e.g. downloaded content size doesn't match sqs event).
+#
+# This plugin is meant for high availability setups, in contrast to logstash-input-s3 you can safely
+# use multiple logstash nodes, since the usage of sqs will ensure that each logfile is processed
+# only once and no file will get lost on node failure or downscaling for auto-scaling groups.
+# (You should use a "Message Retention Period" >= 4 days for your sqs to ensure you can survive
+# a weekend of faulty log file processing)
+# The plugin will not delete objects from s3 buckets, so make sure to have a reasonable "Lifecycle"
+# configured for your buckets, which should keep the files at least "Message Retention Period" days.
+#
+# A typical setup will contain some s3 buckets containing elb, cloudtrail or other log files.
+# These will be configured to send object-created events to a sqs queue, which will be configured
+# as the source queue for this plugin.
+# (The plugin supports gzipped content if it is marked with "contend-encoding: gzip" as it is the
+# case for cloudtrail logs)
+#
+# The logstash node therefore must have sqs permissions + the permissions to download objects
+# from the s3 buckets that send events to the queue.
+# (If logstash nodes are running on EC2 you should use a ServerRole to provide permissions)
+# [source,json]
+#   {
+#       "Version": "2012-10-17",
+#       "Statement": [
+#           {
+#               "Effect": "Allow",
+#               "Action": [
+#                   "sqs:Get*",
+#                   "sqs:List*",
+#                   "sqs:ReceiveMessage",
+#                   "sqs:ChangeMessageVisibility*",
+#                   "sqs:DeleteMessage*"
+#               ],
+#               "Resource": [
+#                   "arn:aws:sqs:us-east-1:123456789012:my-elb-log-queue"
+#               ]
+#           },
+#           {
+#               "Effect": "Allow",
+#               "Action": [
+#                   "s3:Get*",
+#                   "s3:List*",
+#                   "s3:DeleteObject"
+#               ],
+#               "Resource": [
+#                   "arn:aws:s3:::my-elb-logs",
+#                   "arn:aws:s3:::my-elb-logs/*"
+#               ]
+#           }
+#       ]
+#   }
+#
+class LogStash::Inputs::CrowdStrikeFDR < LogStash::Inputs::Threadable
+  include LogStash::PluginMixins::AwsConfig::V2
+  include LogProcessor
+
+
+  config_name "crowdstrike_fdr"    # needs to match the name of this file
+
+  default :codec, "json"  #  FDR uses json,  base project defaulted to "plain"
+
+  config :s3_key_prefix, :validate => :string, :default => '', :deprecated => true #, :obsolete => " Will be moved to s3_options_by_bucket/types"
+
+  config :s3_access_key_id, :validate => :string, :deprecated => true #, :obsolete => "Please migrate to :s3_options_by_bucket. We will remove this option in the next Version"
+  config :s3_secret_access_key, :validate => :string, :deprecated => true #, :obsolete => "Please migrate to :s3_options_by_bucket. We will remove this option in the next Version"
+  config :s3_role_arn, :validate => :string, :deprecated => true #, :obsolete => "Please migrate to :s3_options_by_bucket. We will remove this option in the next Version"
+
+  config :set_codec_by_folder, :validate => :hash, :default => {}, :deprecated => true #, :obsolete => "Please migrate to :s3_options_by_bucket. We will remove this option in the next Version"
+
+  # Default Options for the S3 clients
+  config :s3_default_options, :validate => :hash, :required => false, :default => {}
+  # We need a list of buckets, together with role arns and possible folder/codecs:
+  config :s3_options_by_bucket, :validate => :array, :required => false # TODO: true
+  # Session name to use when assuming an IAM role
+  config :s3_role_session_name, :validate => :string, :default => "logstash"
+  config :delete_on_success, :validate => :boolean, :default => false
+  # Whether or not to include the S3 object's properties (last_modified, content_type, metadata)
+  # into each Event at [@metadata][s3]. Regardless of this setting, [@metdata][s3][key] will always
+  # be present.
+  config :include_object_properties, :validate => :array, :default => [:last_modified, :content_type, :metadata]
+
+  ### sqs
+  # Name of the SQS Queue to pull messages from. Note that this is just the name of the queue, not the URL or ARN.
+  config :queue, :validate => :string, :required => true
+  config :queue_owner_aws_account_id, :validate => :string, :required => false
+  
+  # CrowdStrike FDR does not use SNS so set the default to match
+  config :from_sns, :validate => :boolean, :default => false
+  config :sqs_skip_delete, :validate => :boolean, :default => false
+  config :sqs_wait_time_seconds, :validate => :number, :required => false
+  config :sqs_delete_on_failure, :validate => :boolean, :default => true
+
+  config :visibility_timeout, :validate => :number, :default => 120
+  config :max_processing_time, :validate => :number, :default => 8000
+  ### system
+  config :temporary_directory, :validate => :string, :default => File.join(Dir.tmpdir, "logstash")
+  # To run in multiple threads use this
+  config :consumer_threads, :validate => :number, :default => 1
+
+
+  public
+
+  # --- BEGIN plugin interface ----------------------------------------#
+
+  # initialisation
+  def register
+    # prepare system
+    FileUtils.mkdir_p(@temporary_directory) unless Dir.exist?(@temporary_directory)
+    @id ||= "Unknown" #Use INPUT{ id => name} for thread identifier
+    @credentials_by_bucket = hash_key_is_regex({})
+    @region_by_bucket = hash_key_is_regex({})
+    # create the bucket=>folder=>codec lookup from config options
+    @codec_by_folder = hash_key_is_regex({})
+    @type_by_folder = hash_key_is_regex({})
+
+    # use deprecated settings only if new config is missing:
+    if @s3_options_by_bucket.nil?
+      # We don't know any bucket name, so we must rely on a "catch-all" regex
+      s3_options = {
+        'bucket_name' => '.*',
+        'folders' => @set_codec_by_folder.map { |key, codec|
+          { 'key' => key, 'codec' => codec }
+        }
+      }
+      if @s3_role_arn.nil?
+        # access key/secret key pair needed
+        unless @s3_access_key_id.nil? or @s3_secret_access_key.nil?
+          s3_options['credentials'] = {
+            'access_key_id' => @s3_access_key_id,
+            'secret_access_key' => @s3_secret_access_key
+          }
+        end
+      else
+        s3_options['credentials'] = {
+          'role' => @s3_role_arn
+        }
+      end
+      @s3_options_by_bucket = [s3_options]
+    end
+
+    @s3_options_by_bucket.each do |options|
+      bucket = options['bucket_name']
+      if options.key?('credentials')
+        @credentials_by_bucket[bucket] = options['credentials']
+      end
+      if options.key?('region')
+        @region_by_bucket[bucket] = options['region']
+      end
+      if options.key?('folders')
+        # make these hashes do key lookups using regex matching
+        folders = hash_key_is_regex({})
+        types = hash_key_is_regex({})
+        options['folders'].each do |entry|
+          @logger.debug("options for folder ", :folder => entry)
+          folders[entry['key']] = entry['codec'] if entry.key?('codec')
+          types[entry['key']] = entry['type'] if entry.key?('type')
+        end
+        @codec_by_folder[bucket] = folders unless folders.empty?
+        @type_by_folder[bucket] = types unless types.empty?
+      end
+    end
+
+    @received_stop = Concurrent::AtomicBoolean.new(false)
+
+    # instantiate helpers
+    @sqs_poller = SqsPoller.new(@logger, @received_stop,
+      {
+        visibility_timeout: @visibility_timeout,
+        skip_delete: @sqs_skip_delete,
+        wait_time_seconds: @sqs_wait_time_seconds
+      },
+      {
+        sqs_queue: @queue,
+        queue_owner_aws_account_id: @queue_owner_aws_account_id,
+        from_sns: @from_sns,
+        max_processing_time: @max_processing_time,
+        sqs_delete_on_failure: @sqs_delete_on_failure
+      },
+      aws_options_hash)
+    @s3_client_factory = S3ClientFactory.new(@logger, {
+      aws_region: @region,
+      s3_default_options: @s3_default_options,
+      s3_credentials_by_bucket: @credentials_by_bucket,
+      s3_region_by_bucket: @region_by_bucket,
+      s3_role_session_name: @s3_role_session_name
+    }, aws_options_hash)
+    @s3_downloader = S3Downloader.new(@logger, @received_stop, {
+      s3_client_factory: @s3_client_factory,
+      delete_on_success: @delete_on_success,
+      include_object_properties: @include_object_properties
+    })
+    @codec_factory = CodecFactory.new(@logger, {
+      default_codec: @codec,
+      codec_by_folder: @codec_by_folder
+    })
+    #@log_processor = LogProcessor.new(self)
+
+    # administrative stuff
+    @worker_threads = []
+  end
+
+  # startup
+  def run(logstash_event_queue)
+    @control_threads = @consumer_threads.times.map do |thread_id|
+      Thread.new do
+        restart_count = 0
+        while not stop?
+          #make thead start async to prevent polling the same message from sqs
+          sleep 0.5
+          worker_thread = run_worker_thread(logstash_event_queue, thread_id)
+          worker_thread.join
+          restart_count += 1
+          thread_id = "#{thread_id}_#{restart_count}"
+          @logger.info("[control_thread] restarting a thread #{thread_id}... ", :thread => worker_thread.inspect)
+        end
+      end
+    end
+    @control_threads.each { |t| t.join }
+  end
+
+  # shutdown
+  def stop
+    @received_stop.make_true
+
+    unless @worker_threads.nil?
+      @worker_threads.each do |worker|
+        begin
+          @logger.info("Stopping thread ... ", :thread => worker.inspect)
+          worker.wakeup
+        rescue
+          @logger.error("Cannot stop thread ... try to kill him", :thread => worker.inspect)
+          worker.kill
+        end
+      end
+    end
+  end
+
+  def stop?
+    @received_stop.value
+  end
+
+  # --- END plugin interface ------------------------------------------#
+
+  private
+  def run_worker_thread(queue, thread_id)
+    Thread.new do
+      LogStash::Util.set_thread_name("Worker #{@id}/#{thread_id}")
+      @logger.info("[#{Thread.current[:name]}] started (#{Time.now})") #PROFILING
+      temporary_directory = Dir.mktmpdir("#{@temporary_directory}/")
+      @sqs_poller.run do |record|
+        throw :skip_delete if stop?
+        # record is a valid object with the keys ":bucket", ":key", ":size"
+        record[:local_file] = File.join(temporary_directory, File.basename(record[:key]))
+        if @s3_downloader.copy_s3object_to_disk(record)
+          completed = catch(:skip_delete) do
+            process(record, queue)
+          end
+          @s3_downloader.cleanup_local_object(record)
+          # re-throw if necessary:
+          throw :skip_delete unless completed
+          @s3_downloader.cleanup_s3object(record)
+        end
+      end
+    end
+  end
+
+  # Will be removed in further releases:
+  def get_object_folder(key)
+    if match = /#{s3_key_prefix}\/?(?<type_folder>.*?)\/.*/.match(key)
+      return match['type_folder']
+    else
+      return ""
+    end
+  end
+
+  def hash_key_is_regex(myhash)
+    myhash.default_proc = lambda do |hash, lookup|
+      result = nil
+      hash.each_pair do |key, value|
+        if %r[#{key}] =~ lookup
+          result = value
+          break
+        end
+      end
+      result
+    end
+    # return input hash (convenience)
+    return myhash
+  end
+end # class