logstash-input-sqs_to_s3 1.5.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1709d15716d6889de3283f18eee924966eddd2c3cc345fbeeea9d5fa40853fc8
+  data.tar.gz: 14a7d289d2f2d992a365024717b1d5996a3cc48c1b814bcc198542758fc2f686
+SHA512:
+  metadata.gz: 1939b901ff07dda96895f1f45e14563298a9eb96339008d3dacd207c3c2ec501642eb32969a53999d507c7ad1daf10ecc584efd78262f172017731047ce052b2
+  data.tar.gz: 525aadcbc4ef730228f3ba3d3d7ad43c7c5d1c71be2feec832e5688f9b5a863dda628496f645582901e4f02c59a237ddc6d01251cd6478733a13f2ab10fffec0

data/CHANGELOG.md

@@ -0,0 +1,15 @@
+## 1.1.0
+- Logstash 5 compatibility
+
+## 1.0.3
+- added some metadata to the event (bucket and object name as commited by joshuaspence)
+- also try to unzip files ending with ".gz" (ALB logs are zipped but not marked with proper Content-Encoding)
+
+## 1.0.2
+- fix for broken UTF-8 (so we won't lose a whole s3 log file because of a single invalid line, ruby's split will die on those)
+
+## 1.0.1
+- same (because of screwed up rubygems.org release)
+
+## 1.0.0
+- Initial Release

data/CONTRIBUTORS

@@ -0,0 +1,12 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* joshuaspence (event metadata)
+* Heiko-san (initial contributor)
+* logstash-input-sqs plugin as code base
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/s3sqs.rb

@@ -0,0 +1,220 @@
+# encoding: utf-8
+#
+require "logstash/inputs/threadable"
+require "logstash/namespace"
+require "logstash/timestamp"
+require "logstash/plugin_mixins/aws_config"
+require "logstash/errors"
+
+# Get logs from AWS s3 buckets as issued by an object-created event via sqs.
+#
+# This plugin is based on the logstash-input-sqs plugin but doesn't log the sqs event itself.
+# Instead it assumes, that the event is an s3 object-created event and will then download
+# and process the given file.
+#
+# Some issues of logstash-input-sqs, like logstash not shutting down properly, have been
+# fixed for this plugin.
+#
+# In contrast to logstash-input-sqs this plugin uses the "Receive Message Wait Time"
+# configured for the sqs queue in question, a good value will be something like 10 seconds
+# to ensure a reasonable shutdown time of logstash.
+# Also use a "Default Visibility Timeout" that is high enough for log files to be downloaded
+# and processed (I think a good value should be 5-10 minutes for most use cases), the plugin will
+# avoid removing the event from the queue if the associated log file couldn't be correctly
+# passed to the processing level of logstash (e.g. downloaded content size doesn't match sqs event).
+#
+# This plugin is meant for high availability setups, in contrast to logstash-input-s3 you can safely
+# use multiple logstash nodes, since the usage of sqs will ensure that each logfile is processed
+# only once and no file will get lost on node failure or downscaling for auto-scaling groups.
+# (You should use a "Message Retention Period" >= 4 days for your sqs to ensure you can survive
+# a weekend of faulty log file processing)
+# The plugin will not delete objects from s3 buckets, so make sure to have a reasonable "Lifecycle"
+# configured for your buckets, which should keep the files at least "Message Retention Period" days.
+#
+# A typical setup will contain some s3 buckets containing elb, cloudtrail or other log files.
+# These will be configured to send object-created events to a sqs queue, which will be configured
+# as the source queue for this plugin.
+# (The plugin supports gzipped content if it is marked with "contend-encoding: gzip" as it is the
+# case for cloudtrail logs)
+#
+# The logstash node therefore must have sqs permissions + the permissions to download objects
+# from the s3 buckets that send events to the queue.
+# (If logstash nodes are running on EC2 you should use a ServerRole to provide permissions)
+# [source,json]
+#   {
+#       "Version": "2012-10-17",
+#       "Statement": [
+#           {
+#               "Effect": "Allow",
+#               "Action": [
+#                   "sqs:Get*",
+#                   "sqs:List*",
+#                   "sqs:ReceiveMessage",
+#                   "sqs:ChangeMessageVisibility*",
+#                   "sqs:DeleteMessage*"
+#               ],
+#               "Resource": [
+#                   "arn:aws:sqs:us-east-1:123456789012:my-elb-log-queue"
+#               ]
+#           },
+#           {
+#               "Effect": "Allow",
+#               "Action": [
+#                   "s3:Get*",
+#                   "s3:List*"
+#               ],
+#               "Resource": [
+#                   "arn:aws:s3:::my-elb-logs",
+#                   "arn:aws:s3:::my-elb-logs/*"
+#               ]
+#           }
+#       ]
+#   }
+#
+class LogStash::Inputs::S3SQS < LogStash::Inputs::Threadable
+  include LogStash::PluginMixins::AwsConfig::V2
+
+  BACKOFF_SLEEP_TIME = 1
+  BACKOFF_FACTOR = 2
+  MAX_TIME_BEFORE_GIVING_UP = 60
+  EVENT_SOURCE = 'aws:s3'
+  EVENT_TYPE = 'ObjectCreated'
+
+  config_name "s3sqs"
+
+  default :codec, "plain"
+
+  # Name of the SQS Queue to pull messages from. Note that this is just the name of the queue, not the URL or ARN.
+  config :queue, :validate => :string, :required => true
+
+  attr_reader :poller
+  attr_reader :s3
+
+  def register
+    require "aws-sdk"
+    @logger.info("Registering SQS input", :queue => @queue)
+    setup_queue
+  end
+
+  def setup_queue
+    aws_sqs_client = Aws::SQS::Client.new(aws_options_hash)
+    queue_url = aws_sqs_client.get_queue_url(:queue_name =>  @queue)[:queue_url]
+    @poller = Aws::SQS::QueuePoller.new(queue_url, :client => aws_sqs_client)
+    @s3 = Aws::S3::Client.new(aws_options_hash)
+  rescue Aws::SQS::Errors::ServiceError => e
+    @logger.error("Cannot establish connection to Amazon SQS", :error => e)
+    raise LogStash::ConfigurationError, "Verify the SQS queue name and your credentials"
+  end
+
+  def polling_options
+    {
+      # we will query 1 message at a time, so we can ensure correct error handling if we can't download a single file correctly
+      # (we will throw :skip_delete if download size isn't correct to process the event again later
+      # -> set a reasonable "Default Visibility Timeout" for your queue, so that there's enough time to process the log files)
+      :max_number_of_messages => 1,
+      # we will use the queue's setting, a good value is 10 seconds
+      # (to ensure fast logstash shutdown on the one hand and few api calls on the other hand)
+      :wait_time_seconds => nil,
+    }
+  end
+
+  def handle_message(message, queue)
+    hash = JSON.parse message.body
+    # there may be test events sent from the s3 bucket which won't contain a Records array,
+    # we will skip those events and remove them from queue
+    if hash['Records'] then
+      # typically there will be only 1 record per event, but since it is an array we will
+      # treat it as if there could be more records
+      hash['Records'].each do |record|
+        # in case there are any events with Records that aren't s3 object-created events and can't therefore be
+        # processed by this plugin, we will skip them and remove them from queue
+        if record['eventSource'] == EVENT_SOURCE and record['eventName'].start_with?(EVENT_TYPE) then
+          # try download and :skip_delete if it fails
+          begin
+            response = @s3.get_object(
+              bucket: record['s3']['bucket']['name'],
+              key: record['s3']['object']['key']
+            )
+          rescue => e
+            @logger.warn("issuing :skip_delete on failed download", :bucket => record['s3']['bucket']['name'], :object => record['s3']['object']['key'], :error => e)
+            throw :skip_delete
+          end
+          # verify downloaded content size
+          if response.content_length == record['s3']['object']['size'] then
+            body = response.body
+            # if necessary unzip
+            if response.content_encoding == "gzip" or record['s3']['object']['key'].end_with?(".gz") then
+              begin
+		            temp = Zlib::GzipReader.new(body)
+              rescue => e
+                @logger.warn("content is marked to be gzipped but can't unzip it, assuming plain text", :bucket => record['s3']['bucket']['name'], :object => record['s3']['object']['key'], :error => e)
+                temp = body
+              end
+              body = temp
+            end
+            # process the plain text content
+            begin
+              lines = body.read.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: "\u2370").split(/\n/)
+              lines.each do |line|
+                @codec.decode(line) do |event|
+                  decorate(event)
+
+                  event.set('[@metadata][s3_bucket_name]', record['s3']['bucket']['name'])
+                  event.set('[@metadata][s3_object_key]', record['s3']['object']['key'])
+
+                  queue << event
+                end
+              end
+            rescue => e
+              @logger.warn("issuing :skip_delete on failed plain text processing", :bucket => record['s3']['bucket']['name'], :object => record['s3']['object']['key'], :error => e)
+              throw :skip_delete
+            end
+          # otherwise try again later
+          else
+            @logger.warn("issuing :skip_delete on wrong download content size", :bucket => record['s3']['bucket']['name'], :object => record['s3']['object']['key'],
+              :download_size => response.content_length, :expected => record['s3']['object']['size'])
+            throw :skip_delete
+          end
+        end
+      end
+    end
+  end
+
+  def run(queue)
+    # ensure we can stop logstash correctly
+    poller.before_request do |stats|
+      if stop? then
+        @logger.warn("issuing :stop_polling on stop?", :queue => @queue)
+        # this can take up to "Receive Message Wait Time" (of the sqs queue) seconds to be recognized
+        throw :stop_polling
+      end
+    end
+    # poll a message and process it
+    run_with_backoff do
+      poller.poll(polling_options) do |message|
+        handle_message(message, queue)
+      end
+    end
+  end
+
+  private
+  # Runs an AWS request inside a Ruby block with an exponential backoff in case
+  # we experience a ServiceError.
+  #
+  # @param [Integer] max_time maximum amount of time to sleep before giving up.
+  # @param [Integer] sleep_time the initial amount of time to sleep before retrying.
+  # @param [Block] block Ruby code block to execute.
+  def run_with_backoff(max_time = MAX_TIME_BEFORE_GIVING_UP, sleep_time = BACKOFF_SLEEP_TIME, &block)
+    next_sleep = sleep_time
+    begin
+      block.call
+      next_sleep = sleep_time
+    rescue Aws::SQS::Errors::ServiceError => e
+      @logger.warn("Aws::SQS::Errors::ServiceError ... retrying SQS request with exponential backoff", :queue => @queue, :sleep_time => sleep_time, :error => e)
+      sleep(next_sleep)
+      next_sleep =  next_sleep > max_time ? sleep_time : sleep_time * BACKOFF_FACTOR
+      retry
+    end
+  end
+
+end # class

data/logstash-input-sqs_to_s3.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+  s.name            = 'logstash-input-sqs_to_s3'
+  s.version         = '1.5.0'
+  s.licenses        = ['Apache License (2.0)']
+  s.summary         = "Get logs from AWS s3 buckets as issued by an object-created event via sqs."
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program. This version works with Logstash 5."
+  s.authors         = ["Heiko Finzel"]
+  s.email           = 'hfi@boreus.de'
+  s.homepage        = "https://www.boreus.de"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+
+  s.add_runtime_dependency 'logstash-codec-json'
+  s.add_runtime_dependency "logstash-mixin-aws", ">= 1.0.0"
+
+  s.add_development_dependency 'logstash-devutils'
+end
+

data/spec/inputs/s3sqs_spec.rb

@@ -0,0 +1,9 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/s3sqs"
+
+describe LogStash::Inputs::S3SQS do
+
+  true.should be_true
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,121 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-sqs_to_s3
+version: !ruby/object:Gem::Version
+  version: 1.5.0
+platform: ruby
+authors:
+- Heiko Finzel
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-11-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  name: logstash-mixin-aws
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
+  a stand-alone program. This version works with Logstash 5.
+email: hfi@boreus.de
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/inputs/s3sqs.rb
+- logstash-input-sqs_to_s3.gemspec
+- spec/inputs/s3sqs_spec.rb
+- spec/spec_helper.rb
+homepage: https://www.boreus.de
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.13
+signing_key:
+specification_version: 4
+summary: Get logs from AWS s3 buckets as issued by an object-created event via sqs.
+test_files:
+- spec/inputs/s3sqs_spec.rb
+- spec/spec_helper.rb