fluent-plugin-cloudwatch-ingest-chaeyk 0.3.2

data/README.md

@@ -0,0 +1,132 @@
+# Fluentd Cloudwatch Plugin [![Circle CI](https://circleci.com/gh/sampointer/fluent-plugin-cloudwatch-ingest.svg?style=shield)](https://circleci.com/gh/sampointer/fluent-plugin-cloudwatch-ingest) [![Gem Version](https://badge.fury.io/rb/fluent-plugin-cloudwatch-ingest.svg)](https://badge.fury.io/rb/fluent-plugin-cloudwatch-ingest) ![](http://ruby-gem-downloads-badge.herokuapp.com/fluent-plugin-cloudwatch-ingest?type=total)
+
+## Introduction
+
+This gem was created out of frustration with existing solutions for Cloudwatch log ingestion into a Fluentd pipeline. Specifically, it has been designed to support:
+
+* The 0.14.x fluentd plugin API
+* Native IAM including cross-account authentication via STS
+* Tidy state serialization
+* HA configurations without ingestion duplication
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'fluent-plugin-cloudwatch-ingest'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install fluent-plugin-cloudwatch-ingest
+
+## Usage
+```
+<source>
+  @type cloudwatch_ingest
+  region us-east-1
+  sts_enabled true
+  sts_arn arn:aws:iam::123456789012:role/role_in_another_account
+  sts_session_name fluentd-dev
+  aws_logging_enabled true
+  log_group_name_prefix /aws/lambda
+  log_stream_name_prefix 2017
+  limit_events 10000
+  state_file_name /mnt/nfs/cloudwatch.state
+  interval 60
+  api_interval 5      # Time to wait between API call failures before retry
+  limit_events 10000  # Number of events to fetch in any given iteration
+  event_start_time 0  # Do not fetch events before this time (UNIX epoch, miliseconds)
+  <parse>
+    @type cloudwatch_ingest
+    expression /^(?<message>.+)$/
+    time_format %Y-%m-%d %H:%M:%S.%L
+    event_time true         # take time from the Cloudwatch event, rather than parse it from the body
+    inject_group_name true  # inject the group name into the record
+    inject_stream_name true # inject the stream name into the record
+  </parse>
+</source>
+```
+
+### Authentication
+The plugin will assume an IAM instance role. Without either of the `sts_*` options that role will be used for authentication. With those set the plugin will
+attempt to `sts:AssumeRole` the `sts_arn`. This is useful for fetching logs from many accounts where the fluentd infrastructure lives in one single account.
+
+### Prefixes
+Both the `log_group_name_prefix` and `log_stream_name_prefix` may be omitted, in which case all groups and streams will be ingested. For performance reasons it is often desirable to set the `log_stream_name_prefix` to be today's date, managed by a configuration management system.
+
+### State file
+The state file is a YAML serialization of the current ingestion state. When running in a HA configuration this should be placed on a shared filesystem, such as EFS.
+The state file is opened with an exclusive write call and as such also functions as a lock file in HA configurations. See below.
+
+### HA Setup
+When the state file is located on a shared filesystem an exclusive write lock will attempted each `interval`.
+As such it is safe to run multiple instances of this plugin consuming from the same CloudWatch logging source without fear of duplication, as long as they share a state file.
+In a properly configured auto-scaling group this provides for uninterrupted log ingestion in the event of a failure of any single node.
+
+### Sub-second timestamps
+When using `event_time true` the `@timestamp` field for the record is taken from the time recorded against the event by Cloudwatch. This is the most common mode to run in as it's an easy path to normalization: all of your Lambdas or other AWS service need not have the same, valid, `time_format` nor a regex that matches every case.
+
+If your output plugin supports sub-second precision (and you're running fluentd 0.14.x) you'll "enjoy" sub-second precision.
+
+#### Elasticsearch
+It is a common pattern to use fluentd alongside the [fluentd-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) plugin, either directly or via [fluent-plugin-aws-elasticsearch-service](https://github.com/atomita/fluent-plugin-aws-elasticsearch-service), to ingest logs into Elasticsearch.
+
+At present there is a bug within this plugin that, via an unwise cast, causes records without a named timestamp field to be cast to `DateTime`, losing the precision. This PR: https://github.com/uken/fluent-plugin-elasticsearch/pull/249 fixes that issue. If you need this functionality then I would urge you to comment and express interest over there.
+
+Failing that I maintain my own fork of that repository with the fix in place: https://github.com/sampointer/fluent-plugin-elasticsearch/tree/add_configurable_time_precision_when_timestamp_missing
+
+### IAM
+IAM is a tricky and often bespoke subject. Here's a starter that will ingest all of the logs for all of your Lambdas in the account in which the plugin is running:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "logs:DescribeLogGroups",
+                "logs:DescribeLogStreams",
+                "logs:DescribeMetricFilters",
+                "logs:FilterLogEvents",
+                "logs:GetLogEvents"
+            ],
+            "Resource": [
+                "arn:aws:logs:eu-west-1:123456789012:log-group:/aws/lambda/*:*"
+            ]
+        },
+        {
+            "Effect": "Allow",
+            "Action": [
+                "logs:DescribeLogGroups",
+            ],
+            "Resource": [
+                "arn:aws:logs:eu-west-1:123456789012:log-group:*:*"
+            ]
+        }
+    ]
+}
+```
+
+### Cross-account authentication
+Is a tricky subject that probably cannot be described here. Broadly speaking the IAM instance role of the host on which the plugin is running
+needs to be able to `sts:AssumeRole` the `sts_arn` (and obviously needs `sts_enabled` to be true).
+
+The assumed role should look more-or-less like that above in terms of the actions and resource combinations required.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/sampointer/fluent-plugin-cloudwatch-ingest.
+

data/Rakefile

@@ -0,0 +1,8 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require 'rubocop/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[rubocop spec build]

data/assets/credentials

@@ -0,0 +1,2 @@
+---
+:rubygems_api_key: REPLACEME

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'fluent/plugin/cloudwatch/ingest'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start

data/bin/deploy

@@ -0,0 +1,10 @@
+#!/bin/bash
+#
+# deploy to rubygems.
+
+gem=$(ls pkg/*.gem | tail -1)
+mkdir -p /home/ubuntu/.gem/
+cp assets/credentials /home/ubuntu/.gem/credentials
+sed -i "s/REPLACEME/${rubygems_api_key}/g" /home/ubuntu/.gem/credentials
+chmod 0600 /home/ubuntu/.gem/credentials
+gem push ${gem}

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/circle.yml

@@ -0,0 +1,11 @@
+test:
+  override:
+    - bundle exec rake
+  post:
+    - cp pkg/*.gem ${CIRCLE_ARTIFACTS}
+deployment:
+  release:
+    tag: /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$/
+    owner: sampointer
+    commands:
+      - bin/deploy

data/fluent-plugin-cloudwatch-ingest.gemspec

@@ -0,0 +1,37 @@
+# coding: utf-8
+
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'fluent/plugin/cloudwatch/ingest/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'fluent-plugin-cloudwatch-ingest-chaeyk'
+  spec.version       = Fluent::Plugin::Cloudwatch::Ingest::VERSION
+  spec.authors       = ['Sam Pointer']
+  spec.email         = ['san@outsidethe.net']
+
+  spec.summary       = 'Fluentd plugin to ingest AWS Cloudwatch logs'
+  spec.description   = 'Fluentd plugin to ingest AWS Cloudwatch logs'
+  spec.homepage      = 'https://github.com/sampointer/fluent-plugin-cloudwatch-ingest'
+
+  # Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or
+  # delete this section to allow pushing this gem to any host.
+  if spec.respond_to?(:metadata) # rubocop:disable all
+    spec.metadata['allowed_push_host'] = 'https://rubygems.org'
+  else
+    raise 'RubyGems 2.0 or newer is required to protect against public gem pushes.' # rubocop:disable all
+  end
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) } # rubocop:disable all
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'bundler', '~> 1.10'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'rubocop'
+
+  spec.add_dependency 'fluentd', '~>0.14.13'
+  spec.add_dependency 'aws-sdk', '~>2.8.4'
+end

data/lib/fluent/plugin/cloudwatch/ingest.rb

@@ -0,0 +1,9 @@
+module Fluent
+  module Plugin
+    module Cloudwatch
+      module Ingest
+        # Your code goes here...
+      end
+    end
+  end
+end

data/lib/fluent/plugin/cloudwatch/ingest/version.rb

@@ -0,0 +1,9 @@
+module Fluent
+  module Plugin
+    module Cloudwatch
+      module Ingest
+        VERSION = '0.3.2'.freeze
+      end
+    end
+  end
+end

data/lib/fluent/plugin/in_cloudwatch_ingest.rb

@@ -0,0 +1,299 @@
+require 'aws-sdk'
+require 'fluent/config/error'
+require 'fluent/plugin/input'
+require 'fluent/plugin/parser'
+require 'json'
+require 'pathname'
+require 'psych'
+
+module Fluent::Plugin
+  class CloudwatchIngestInput < Fluent::Plugin::Input
+    Fluent::Plugin.register_input('cloudwatch_ingest', self)
+    helpers :compat_parameters, :parser
+
+    desc 'The region of the source cloudwatch logs'
+    config_param :region, :string, default: 'us-east-1'
+    desc 'Enable STS for cross-account IAM'
+    config_param :sts_enabled, :bool, default: false
+    desc 'The IAM role ARN in the source account to use when STS is enabled'
+    config_param :sts_arn, :string, default: ''
+    desc 'The session name for use with STS'
+    config_param :sts_session_name, :string, default: 'fluentd'
+    desc 'Log group name or prefix. Not setting means "all"'
+    config_param :log_group_name_prefix, :string, default: ''
+    desc 'Log stream name or prefix. Not setting means "all"'
+    config_param :log_stream_name_prefix, :string, default: ''
+    desc 'State file name'
+    config_param :state_file_name, :string, default: '/var/spool/td-agent/cloudwatch.state' # rubocop:disable all
+    desc 'Fetch logs every interval'
+    config_param :interval, :time, default: 60
+    desc 'Time to pause between API call failures and limits'
+    config_param :api_interval, :time, default: 5
+    desc 'Tag to apply to record'
+    config_param :tag, :string, default: 'cloudwatch'
+    desc 'Enabled AWS SDK logging'
+    config_param :aws_logging_enabled, :bool, default: false
+    desc 'Limit the number of events fetched in any iteration'
+    config_param :limit_events, :integer, default: 10_000
+    desc 'Do not fetch events before this time'
+    config_param :event_start_time, :integer, default: 0
+    config_section :parse do
+      config_set_default :@type, 'cloudwatch_ingest'
+      desc 'Regular expression with which to parse the event message'
+      config_param :expression, :string, default: '^(?<message>.+)$'
+      desc 'Take the timestamp from the event rather than the expression'
+      config_param :event_time, :bool, default: true
+      desc 'Time format to use when parsing event message'
+      config_param :time_format, :string, default: '%Y-%m-%d %H:%M:%S.%L'
+      desc 'Inject the log group name into the record'
+      config_param :inject_group_name, :bool, default: true
+      desc 'Inject the log stream name into the record'
+      config_param :inject_stream_name, :bool, default: true
+    end
+
+    def initialize
+      super
+      log.info('Starting fluentd-plugin-cloudwatch-ingest')
+    end
+
+    def configure(conf)
+      super
+      compat_parameters_convert(conf, :parser)
+      parser_config = conf.elements('parse').first
+      unless parser_config
+        raise Fluent::ConfigError, '<parse> section is required.'
+      end
+      unless parser_config['expression']
+        raise Fluent::ConfigError, 'parse/expression is required.'
+      end
+      unless parser_config['event_time']
+        raise Fluent::ConfigError, 'parse/event_time is required.'
+      end
+
+      @parser = parser_create(conf: parser_config)
+      log.info('Configured fluentd-plugin-cloudwatch-ingest')
+    end
+
+    def start
+      super
+      log.info('Started fluentd-plugin-cloudwatch-ingest')
+
+      # Get a handle to Cloudwatch
+      aws_options = {}
+      Aws.config[:region] = @region
+      Aws.config[:logger] = log if @aws_logging
+      log.info("Working in region #{@region}")
+
+      if @sts_enabled
+        aws_options[:credentials] = Aws::AssumeRoleCredentials.new(
+          role_arn: @sts_arn,
+          role_session_name: @sts_session_name
+        )
+
+        log.info("Using STS for authentication with source account ARN: #{@sts_arn}, session name: #{@sts_session_name}") # rubocop:disable all
+      else
+        log.info('Using local instance IAM role for authentication')
+      end
+      @aws = Aws::CloudWatchLogs::Client.new(aws_options)
+      @finished = false
+      @thread = Thread.new(&method(:run))
+    end
+
+    def shutdown
+      @finished = true
+      @thread.join
+    end
+
+    private
+
+    def emit(event, log_group_name, log_stream_name)
+      @parser.parse(event, log_group_name, log_stream_name) do |time, record|
+        router.emit(@tag, time, record)
+      end
+    end
+
+    def log_groups(log_group_prefix)
+      log_groups = []
+
+      # Fetch all log group names
+      next_token = nil
+      loop do
+        begin
+          response = if !log_group_prefix.empty?
+                       @aws.describe_log_groups(
+                         log_group_name_prefix: log_group_prefix,
+                         next_token: next_token
+                       )
+                     else
+                       @aws.describe_log_groups(
+                         next_token: next_token
+                       )
+                     end
+
+          response.log_groups.each { |g| log_groups << g.log_group_name }
+          break unless response.next_token
+          next_token = response.next_token
+        rescue => boom
+          log.error("Unable to retrieve log groups: #{boom}")
+          next_token = nil
+          sleep @api_interval
+          retry
+        end
+      end
+      log.info("Found #{log_groups.size} log groups")
+
+      return log_groups
+    end
+
+    def log_streams(log_group_name, log_stream_name_prefix)
+      log_streams = []
+      next_token = nil
+      loop do
+        begin
+          response = if !log_stream_name_prefix.empty?
+                       @aws.describe_log_streams(
+                         log_group_name: log_group_name,
+                         log_stream_name_prefix: log_stream_name_prefix,
+                         next_token: next_token
+                       )
+                     else
+                       @aws.describe_log_streams(
+                         log_group_name: log_group_name,
+                         next_token: next_token
+                       )
+                     end
+
+          response.log_streams.each { |s| log_streams << s.log_stream_name }
+          break unless response.next_token
+          next_token = response.next_token
+        rescue => boom
+          log.error("Unable to retrieve log streams for group #{log_group_name} with stream prefix #{log_stream_name_prefix}: #{boom}") # rubocop:disable all
+          log_streams = []
+          next_token = nil
+          sleep @api_interval
+          retry
+        end
+      end
+      log.info("Found #{log_streams.size} streams for #{log_group_name}")
+
+      return log_streams
+    end
+
+    def run
+      until @finished
+        begin
+          state = State.new(@state_file_name, log)
+        rescue => boom
+          log.info("Failed lock state. Sleeping for #{@interval}: #{boom}")
+          sleep @interval
+          retry
+        end
+
+        # Fetch the streams for each log group
+        log_groups(@log_group_name_prefix).each do |group|
+          # For each log stream get and emit the events
+          log_streams(group, @log_stream_name_prefix).each do |stream|
+            # See if we have some stored state for this group and stream.
+            # If we have then use the stored forward_token to pick up
+            # from that point. Otherwise start from the start.
+            if state.store[group] && state.store[group][stream]
+              stream_token =
+                (state.store[group][stream] if state.store[group][stream])
+            else
+              stream_token = nil
+            end
+
+            begin
+              response = @aws.get_log_events(
+                log_group_name: group,
+                log_stream_name: stream,
+                next_token: stream_token,
+                limit: @limit_events,
+                start_time: @event_start_time,
+                start_from_head: true
+              )
+
+              response.events.each do |e|
+                begin
+                  emit(e, group, stream)
+                rescue => boom
+                  log.error("Failed to emit event #{e}: #{boom}")
+                end
+              end
+
+              # Once all events for this stream have been processed,
+              # in this iteration, store the forward token
+              state.store[group][stream] = response.next_forward_token
+            rescue => boom
+              log.error("Unable to retrieve events for stream #{stream} in group #{group}: #{boom}") # rubocop:disable all
+              sleep @api_interval
+              retry
+            end
+          end
+        end
+
+        log.info('Pruning and saving state')
+        state.prune(log_groups(@log_group_name_prefix)) # Remove dead streams
+        begin
+          state.save
+          state.close
+        rescue
+          log.error("Unable to save state file: #{boom}")
+        end
+        log.info("Pausing for #{@interval}")
+        sleep @interval
+      end
+    end
+
+    class CloudwatchIngestInput::State
+      class LockFailed < RuntimeError; end
+      attr_accessor :statefile, :store
+
+      def initialize(filepath, log)
+        @filepath = filepath
+        @log = log
+        @store = Hash.new { |h, k| h[k] = {} }
+
+        if File.exist?(filepath)
+          self.statefile = Pathname.new(@filepath).open('r+')
+        else
+          @log.warn("No state file #{statefile} Creating a new one.")
+          begin
+            self.statefile = Pathname.new(@filepath).open('w+')
+            save
+          rescue => boom
+            @log.error("Unable to create new file #{statefile.path}: #{boom}")
+          end
+        end
+
+        # Attempt to obtain an exclusive flock on the file and raise and
+        # exception if we can't
+        @log.info("Obtaining exclusive lock on state file #{statefile.path}")
+        lockstatus = statefile.flock(File::LOCK_EX | File::LOCK_NB)
+        raise CloudwatchIngestInput::State::LockFailed if lockstatus == false
+
+        @store.merge!(Psych.safe_load(statefile.read))
+        @log.info("Loaded #{@store.keys.size} groups from #{statefile.path}")
+      end
+
+      def save
+        statefile.rewind
+        statefile.write(Psych.dump(@store))
+        @log.info("Saved state to #{statefile.path}")
+        statefile.rewind
+      end
+
+      def close
+        statefile.close
+      end
+
+      def prune(log_groups)
+        groups_before = @store.keys.size
+        @store.delete_if { |k, _v| true unless log_groups.include?(k) }
+        @log.info("Pruned #{groups_before - @store.keys.size} keys from store")
+
+        # TODO: also prune streams as these are most likely to be transient
+      end
+    end
+  end
+end