fluent-plugin-kinesis-aggregation 0.2.1 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 2ca1b05d0268c5f19045de4e90129f6018cbeca5
-  data.tar.gz: 434e9f311ebcb99684d1c11620f6ec0e5dce9a55
+SHA256:
+  metadata.gz: eca9c6a9bf3957d2782027f32b7121cb11f777434fa87229dfb0e0aa21f611e4
+  data.tar.gz: 6c482a9463582a0397006a05e4db554b7934f86f2450222847724129c7883432
 SHA512:
-  metadata.gz: 03035ab01ba22533165a108983b940ba58607b874b9e446ecfd012fe70e6128ab651f371d7d6f0fdd6b60befbe360d727a0c9e121976201bfd5adda619ff5228
-  data.tar.gz: 05f031ff1367b86dfce49f178784a02f8f89230a666fc9060752373a0864383dae6ce39db27e0c58e1c47f335890e3c143bde5259e5f0b7d0233c0a9884a3ab6
+  metadata.gz: 612a0af58db3f07e799d925f29148e0816e60c1eb9cd07b0c6a8a8c786506754ac4c737190c4a2fd00e4deb9a574bd79f0b25edba4583970b3b412da4fefb568
+  data.tar.gz: '04698ac254d9930dd2b05c9bb96c3253d5a33d7f65602e30abcae8fab0c58e69d68557954691fd48c35d647812628a5f4bd06edae80ea15d982734aa5c17c433'

data/.travis.yml

@@ -1,8 +1,8 @@
 language: ruby
 
 rvm:
-  - 2.2
-  - 2.1
+  - 2.4
+  - 2.3
 
 os:
   - linux

data/CHANGELOG.md

@@ -1,9 +1,30 @@
 # CHANGELOG
 
-## Next
+## 0.3.2
+
+- Modify aws-sdk usage to require just the API/SDK resources for Kinesis
+- Drop support and testing for deprecated Ruby versions (<2.3)
+
+## 0.3.1
+
+- Change aws-sdk usage to work with both v2 and v3
+  (in particular, makes it possible to use latest td-agent which includes the s3 plugin
+  and pulls in aws-sdk v3)
+
+## 0.3.0
+
+- Update to use fluentd 0.14 API (stick to 0.2.3 if you need support for earlier versions of fluentd)
+  Much thanks to cosmo0920 for doing this.
+
+## 0.2.3
+
+- emit stream name in error
+
+## 0.2.1 - 0.2.2
 
 - update documentation to refer to published gem
 - turn on testing for Ruby 2.1
+- allow running on Ruby 2.1
 
 ## 0.2.0
 

data/README.md

@@ -8,14 +8,26 @@ This is a rewrite of [aws-fluent-plugin-kinesis](https://github.com/awslabs/aws-
 a different shipment method using the
 [KPL aggregation format](https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md).
 
+*Since this plugin was forked, aws-fluent-plugin-kinesis has undergone considerable development (and improvement).
+Most notably, the upcoming 2.0 release supports KPL aggregated records using google-protobuf without
+the overhead of using the KPL:
+https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/107*
+
+*However, it still uses msgpack for internal buffering and only uses protobuf when it ships the records,
+whereas this plugin processes each record as it comes in and ships the result by simple concatenation
+of the encoded records. This may not be faster, of course - could depend on the overhead of calling
+the protobuf methods - but most probably is. The discussion below is also still mostly valid,
+in that the awslabs plugin does not have PutRecord == chunk equivalency, but instead has its
+own internal retry method.*
+
 The basic idea is to have one PutRecord === one chunk. This has a number of advantages:
 
 - much less complexity in plugin (less CPU/memory)
 - by aggregating, we increase the throughput and decrease the cost
 - since a single chunk either succeeds or fails,
   we get to use fluentd's more complex/complete retry mechanism
-  (which is also exposed in the monitor). The existing retry mechanism
-  has [unfortunate issues under heavy load](https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/42)
+  (which is also exposed by the monitor plugin; we view this in datadog). The existing retry mechanism
+  had [unfortunate issues under heavy load](https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/42)
 - we get ordering within a chunk without having to rely on sequence
   numbers (though not overall ordering)
 
@@ -23,7 +35,7 @@ However, there are drawbacks:
 
 - if you're using this as an aggregator, you will need to tune the
   buffer size on your sources fairly low such that it is less
-  that the low buffer_chunk_limit on the aggregator
+  than the low buffer_chunk_limit on the aggregator
 - you have to use a KCL library to ingest
 - you can't use a calculated partition key (based on the record);
   essentially, you need to use a random partition key
@@ -63,7 +75,7 @@ specify the library path via RUBYLIB:
 ## Dependencies
 
  * Ruby 2.1+
- * Fluentd 0.10.43+
+ * Fluentd 0.14.15+ (if you need 0.10 or 0.12 support, use the fluentd-v0.12 branch or version 0.2.x on rubygems)
 
 ## Basic Usage
 
@@ -145,24 +157,28 @@ forces all writes to a specific shard, and if you're using
 a single thread/process will probably keep event ordering
 (not recommended - watch out for hot shards!).
 
-### detach_process
-
-Integer. Optional. This defines the number of parallel processes to start.
-This can be used to increase throughput by allowing multiple processes to
-execute the plugin at once. Setting this option to > 0 will cause the plugin
-to run in a separate process. The default is 0.
-
 ### num_threads
 
 Integer. The number of threads to flush the buffer. This plugin is based on
-Fluentd::BufferedOutput, so we buffer incoming records before emitting them to
+Fluentd::Plugin::Output, so we buffer incoming records before emitting them to
 Amazon Kinesis. You can find the detail about buffering mechanism [here](http://docs.fluentd.org/articles/buffer-plugin-overview).
 Emitting records to Amazon Kinesis via network causes I/O Wait, so parallelizing
 emitting with threads will improve throughput.
 
 This option can be used to parallelize writes into the output(s)
 designated by the output plugin. The default is 1.
-Also you can use this option with *detach_process*.
+Also you can use this option with *multi workers*.
+
+### multi workers
+
+This feature is introduced in Fluentd v0.14.
+Instead of using *detach_process*, this feature can use as the following system directive.
+Note that *detach_process* parameter is removed after using v0.14 Output Plugin API.
+The default is 1.
+
+    <system>
+      workers 5
+    </system>
 
 ### debug
 

data/fluent-plugin-kinesis-aggregation.gemspec

@@ -17,24 +17,25 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-kinesis-aggregation"
-  spec.version       = '0.2.1'
+  spec.version       = '0.3.2'
   spec.author        = 'Atlassian'
+  spec.email         = 'jhaggerty@atlassian.com'
   spec.summary       = %q{Fluentd output plugin that sends KPL style aggregated events to Amazon Kinesis.}
   spec.homepage      = "https://github.com/atlassian/fluent-plugin-kinesis-aggregation"
-  spec.license       = "Apache License, Version 2.0"
+  spec.license       = "Apache-2.0"
 
   spec.files         = `git ls-files`.split($/)
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
-  spec.required_ruby_version = '>= 2.2'
+  spec.required_ruby_version = '>= 2.3'
 
-  spec.add_development_dependency "bundler", "~> 1.3"
-  spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "test-unit-rr", "~> 1.0"
+  spec.add_development_dependency "bundler", ">= 1.10"
+  spec.add_development_dependency "rake", ">= 10.0"
+  spec.add_development_dependency "test-unit", ">= 3.0.8"
+  spec.add_development_dependency "test-unit-rr", ">= 1.0.3"
 
-  spec.add_dependency "fluentd", ">= 0.10.53", "< 0.13"
-  spec.add_dependency "aws-sdk-core", ">= 2.0.12", "< 3.0"
-  spec.add_dependency "msgpack", ">= 0.5.8"
-  spec.add_dependency "google-protobuf", ">= 3.0.0.alpha.4.0"
+  spec.add_dependency "fluentd", [">= 0.14.22", "< 2"]
+  spec.add_dependency "aws-sdk-kinesis", "~> 1", "!= 1.4", "!= 1.5", "!= 1.14"
+  spec.add_dependency "google-protobuf", "~> 3"
 end

data/lib/fluent/plugin/out_kinesis-aggregation.rb

@@ -11,13 +11,14 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 
-require 'aws-sdk-core'
+require 'aws-sdk-kinesis'
 require 'yajl'
 require 'logger'
 require 'securerandom'
 require 'digest'
 
 require 'google/protobuf'
+require 'fluent/plugin/output'
 
 Google::Protobuf::DescriptorPool.generated_pool.build do
   add_message "AggregatedRecord" do
@@ -43,12 +44,11 @@ Record = Google::Protobuf::DescriptorPool.generated_pool.lookup("Record").msgcla
 
 
 module FluentPluginKinesisAggregation
-  class OutputFilter < Fluent::BufferedOutput
+  class OutputFilter < Fluent::Plugin::Output
 
-    include Fluent::DetachMultiProcessMixin
-    include Fluent::SetTimeKeyMixin
-    include Fluent::SetTagKeyMixin
+    helpers :compat_parameters, :inject
 
+    DEFAULT_BUFFER_TYPE = "memory"
     NAME = 'kinesis-aggregation'
     PUT_RECORD_MAX_DATA_SIZE = 1024 * 1024
     # 200 is an arbitrary number more than the envelope overhead
@@ -81,26 +81,31 @@ module FluentPluginKinesisAggregation
 
     config_param :http_proxy, :string, default: nil
 
+    config_section :buffer do
+      config_set_default :@type, DEFAULT_BUFFER_TYPE
+    end
+
     def configure(conf)
+      compat_parameters_convert(conf, :buffer, :inject)
       super
 
-      if @buffer.chunk_limit > FLUENTD_MAX_BUFFER_SIZE
+      if @buffer.chunk_limit_size > FLUENTD_MAX_BUFFER_SIZE
         raise Fluent::ConfigError, "Kinesis buffer_chunk_limit is set to more than the 1mb shard limit (i.e. you won't be able to write your chunks!"
       end
 
-      if @buffer.chunk_limit > FLUENTD_MAX_BUFFER_SIZE / 3
+      if @buffer.chunk_limit_size > FLUENTD_MAX_BUFFER_SIZE / 3
         log.warn 'Kinesis buffer_chunk_limit is set at more than 1/3 of the per second shard limit (1mb). This is not good if you have many producers.'
       end
     end
 
     def start
-      detach_multi_process do
-        super
-        load_client
-      end
+      super
+      load_client
     end
 
     def format(tag, time, record)
+      record = inject_values_to_record(tag, time, record)
+
       return AggregatedRecord.encode(AggregatedRecord.new(
         records: [Record.new(
           partition_key_index: 1,
@@ -112,7 +117,7 @@ module FluentPluginKinesisAggregation
     def write(chunk)
       records = chunk.read
       if records.length > FLUENTD_MAX_BUFFER_SIZE
-        log.error "Can't emit aggregated record of length #{records.length} (more than #{FLUENTD_MAX_BUFFER_SIZE})"
+        log.error "Can't emit aggregated #{@stream_name} stream record of length #{records.length} (more than #{FLUENTD_MAX_BUFFER_SIZE})"
         return # do not throw, since we can't retry
       end
 

data/test/helper.rb

@@ -28,4 +28,7 @@ require 'test/unit/rr'
 $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
 $LOAD_PATH.unshift(File.dirname(__FILE__))
 require 'fluent/test'
+require 'fluent/test/helpers'
+require 'fluent/test/driver/output'
+require 'fluent/process'
 require 'fluent/plugin/out_kinesis-aggregation'

data/test/plugin/test_out_kinesis-aggregation.rb

@@ -14,6 +14,8 @@
 require 'helper'
 
 class KinesisOutputTest < Test::Unit::TestCase
+  include Fluent::Test::Helpers
+
   def setup
     Fluent::Test.setup
   end
@@ -27,14 +29,14 @@ class KinesisOutputTest < Test::Unit::TestCase
     buffer_chunk_limit 100k
   ]
 
-  def create_driver(conf = CONFIG, tag='test')
-    Fluent::Test::BufferedOutputTestDriver
-      .new(FluentPluginKinesisAggregation::OutputFilter, tag).configure(conf)
+  def create_driver(conf = CONFIG)
+    Fluent::Test::Driver::Output
+      .new(FluentPluginKinesisAggregation::OutputFilter).configure(conf)
   end
 
   def create_mock_client
     client = mock(Object.new)
-    mock(Aws::Kinesis::Client).new({}) { client }
+    stub(Aws::Kinesis::Client).new(anything) { client }
     return client
   end
 
@@ -104,7 +106,7 @@ class KinesisOutputTest < Test::Unit::TestCase
     end
 
     d = create_driver
-    d.run
+    d.run(default_tag: "test")
   end
 
   def test_load_client_with_credentials
@@ -132,7 +134,7 @@ class KinesisOutputTest < Test::Unit::TestCase
       buffer_chunk_limit 100k
     EOS
 
-    d.run
+    d.run(default_tag: "test")
   end
 
   def test_load_client_with_role_arn
@@ -160,7 +162,7 @@ class KinesisOutputTest < Test::Unit::TestCase
       fixed_partition_key test_partition_key
       buffer_chunk_limit 100k
     EOS
-    d.run
+    d.run(default_tag: "test")
   end
 
   def test_emitting
@@ -169,18 +171,19 @@ class KinesisOutputTest < Test::Unit::TestCase
     data1 = {"a"=>1,"time"=>"2011-01-02T13:14:15Z","tag"=>"test"}
     data2 = {"a"=>2,"time"=>"2011-01-02T13:14:15Z","tag"=>"test"}
 
-    time = Time.parse("2011-01-02 13:14:15 UTC").to_i
-    d.emit(data1, time)
-    d.emit(data2, time)
+    time = event_time("2011-01-02 13:14:15 UTC")
 
-    client = create_mock_client
-    client.put_record(
-      stream_name: 'test_stream',
-      data: "\xF3\x89\x9A\xC2\n\x01a\n\x12test_partition_key\x1A6\b\x01\x1A2{\"a\":1,\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}\x1A6\b\x01\x1A2{\"a\":2,\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}\xA2\x0E y\x8B\x02\xDF\xAE\xAB\x93\x1C;\xCB\xAD\x1Fx".b,
-      partition_key: 'test_partition_key'
-    ) { {} }
+    d.run(default_tag: "test") do
+      client = create_mock_client
+      stub.instance_of(Aws::Kinesis::Client).put_record(
+        stream_name: 'test_stream',
+        data: "\xF3\x89\x9A\xC2\n\x01a\n\x12test_partition_key\x1A6\b\x01\x1A2{\"a\":1,\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}\x1A6\b\x01\x1A2{\"a\":2,\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}\xA2\x0E y\x8B\x02\xDF\xAE\xAB\x93\x1C;\xCB\xAD\x1Fx".b,
+        partition_key: 'test_partition_key'
+      ) { {} }
 
-    d.run
+      d.feed(time, data1)
+      d.feed(time, data2)
+    end
   end
 
   def test_multibyte
@@ -188,29 +191,32 @@ class KinesisOutputTest < Test::Unit::TestCase
 
     data1 = {"a"=>"\xE3\x82\xA4\xE3\x83\xB3\xE3\x82\xB9\xE3\x83\x88\xE3\x83\xBC\xE3\x83\xAB","time"=>"2011-01-02T13:14:15Z".b,"tag"=>"test"}
 
-    time = Time.parse("2011-01-02 13:14:15 UTC").to_i
-    d.emit(data1, time)
 
-    client = create_mock_client
-    client.put_record(
-      stream_name: 'test_stream',
-      data: "\xF3\x89\x9A\xC2\n\x01a\n\x12test_partition_key\x1AI\b\x01\x1AE{\"a\":\"\xE3\x82\xA4\xE3\x83\xB3\xE3\x82\xB9\xE3\x83\x88\xE3\x83\xBC\xE3\x83\xAB\",\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}_$\x9C\xF9v+pV:g7c\xE3\xF2$\xBA".b,
-      partition_key: 'test_partition_key'
-    ) { {} }
+    time = event_time("2011-01-02 13:14:15 UTC")
+    d.run(default_tag: "test") do
+      client = create_mock_client
+      stub.instance_of(Aws::Kinesis::Client).put_record(
+        stream_name: 'test_stream',
+        data: "\xF3\x89\x9A\xC2\n\x01a\n\x12test_partition_key\x1AI\b\x01\x1AE{\"a\":\"\xE3\x82\xA4\xE3\x83\xB3\xE3\x82\xB9\xE3\x83\x88\xE3\x83\xBC\xE3\x83\xAB\",\"time\":\"2011-01-02T13:14:15Z\",\"tag\":\"test\"}_$\x9C\xF9v+pV:g7c\xE3\xF2$\xBA".b,
+        partition_key: 'test_partition_key'
+      ) { {} }
 
-    d.run
+      d.feed(time, data1)
+    end
   end
 
   def test_fail_on_bigchunk
     d = create_driver
 
-    d.emit(
-      {"msg" => "z" * 1024 * 1024},
-      Time.parse("2011-01-02 13:14:15 UTC").to_i)
-    client = dont_allow(Object.new)
-    client.put_record
-    mock(Aws::Kinesis::Client).new({}) { client }
-
-    d.run
+    assert_raise(Fluent::Plugin::Buffer::BufferChunkOverflowError) do
+      d.run(default_tag: "test") do
+        d.feed(
+          event_time("2011-01-02 13:14:15 UTC"),
+          {"msg" => "z" * 1024 * 1024})
+        client = dont_allow(Object.new)
+        client.put_record
+        mock(Aws::Kinesis::Client).new(anything) { client }
+      end
+    end
   end
 end

metadata

@@ -1,127 +1,139 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-kinesis-aggregation
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.2
 platform: ruby
 authors:
 - Atlassian
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-02-09 00:00:00.000000000 Z
+date: 2020-03-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: '1.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: '1.10'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '10.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '10.0'
 - !ruby/object:Gem::Dependency
-  name: test-unit-rr
+  name: test-unit
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: 3.0.8
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: 3.0.8
 - !ruby/object:Gem::Dependency
-  name: fluentd
+  name: test-unit-rr
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.10.53
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: '0.13'
-  type: :runtime
+        version: 1.0.3
+  type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.10.53
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: '0.13'
+        version: 1.0.3
 - !ruby/object:Gem::Dependency
-  name: aws-sdk-core
+  name: fluentd
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.12
+        version: 0.14.22
     - - "<"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.12
+        version: 0.14.22
     - - "<"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '2'
 - !ruby/object:Gem::Dependency
-  name: msgpack
+  name: aws-sdk-kinesis
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+    - - "!="
+      - !ruby/object:Gem::Version
+        version: '1.4'
+    - - "!="
+      - !ruby/object:Gem::Version
+        version: '1.5'
+    - - "!="
       - !ruby/object:Gem::Version
-        version: 0.5.8
+        version: '1.14'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+    - - "!="
+      - !ruby/object:Gem::Version
+        version: '1.4'
+    - - "!="
+      - !ruby/object:Gem::Version
+        version: '1.5'
+    - - "!="
       - !ruby/object:Gem::Version
-        version: 0.5.8
+        version: '1.14'
 - !ruby/object:Gem::Dependency
   name: google-protobuf
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.0.0.alpha.4.0
+        version: '3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.0.0.alpha.4.0
+        version: '3'
 description: 
-email: 
+email: jhaggerty@atlassian.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -142,7 +154,7 @@ files:
 - test/plugin/test_out_kinesis-aggregation.rb
 homepage: https://github.com/atlassian/fluent-plugin-kinesis-aggregation
 licenses:
-- Apache License, Version 2.0
+- Apache-2.0
 metadata: {}
 post_install_message: 
 rdoc_options: []
@@ -152,15 +164,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.2'
+      version: '2.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.2.2
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: Fluentd output plugin that sends KPL style aggregated events to Amazon Kinesis.