fluent-plugin-kinesis-modified 3.1.3 → 3.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: '089238fcbb0424768cc6162ae3f91ba56fe3e63f'
-  data.tar.gz: a90034b7995ba2100e2cc9691bd60c05605046e2
+  metadata.gz: e5eb2e11af14feafefd700f1cdccbf3e2db10d83
+  data.tar.gz: c439a086a1cd5fdcfab0134c70737c80f4fcc833
 SHA512:
-  metadata.gz: 73e55af9419587fed9e868553a65e84906b4185d14666fe692e90addcd58890fa034f58ec7c589ba87d2022865dfb8b786be9ee39ae291611cf9b70c87838248
-  data.tar.gz: 201112e62f6a66f2f3e4a4f854dec2447b5606e0e8677c637e8a9ff728745b8b881adbac8ee1c143bbb627cb8265485634dce024cf5bf736de3dd4954e3baf8e
+  metadata.gz: f2a8c698e569f3aa524c1b11a9f5b9140340532d06bc94b051474920993b4922f16c124bfb29f1f102e83764bd0d3f880720983ddb4ae6b3806056fe33000bed
+  data.tar.gz: 7c7f1c908d027e9e787e65f5c6c9e921aa76382ad14d2ef36ae86a4e2ae09564af9ed07627cd57b6fa6497182741b50bf54866623f6b61730b82b09000ff71dd

data/README.md

@@ -2,14 +2,15 @@
 
 [![Gitter](https://badges.gitter.im/awslabs/aws-fluent-plugin-kinesis.svg)](https://gitter.im/awslabs/aws-fluent-plugin-kinesis?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 [![Build Status](https://travis-ci.org/awslabs/aws-fluent-plugin-kinesis.svg?branch=master)](https://travis-ci.org/awslabs/aws-fluent-plugin-kinesis)
-[![Gem Version](https://badge.fury.io/rb/fluent-plugin-kinesis.svg)](https://rubygems.org/gems/fluent-plugin-kinesis)
+[![Gem Version](https://badge.fury.io/rb/fluent-plugin-kinesis.svg)](https://rubygems.org/gems/fluent-plugin-kinesis-modified)
 
 [Fluentd][fluentd] output plugin
-that sends events to [Amazon Kinesis Data Streams][streams] and [Amazon Kinesis Data Firehose][firehose]. Also it supports [KPL Aggregated record format][kpl]. This gem includes three output plugins respectively:
+that sends events to [Amazon Kinesis Data Streams][streams] and [Amazon Kinesis Data Firehose][firehose]. Also it supports [KPL Aggregated record format][kpl]. This gem includes four output plugins respectively:
 
 - `kinesis_streams`
 - `kinesis_firehose`
 - `kinesis_streams_aggregated`
+- `kinesis_streams_aggregated_modified`
 
 Also, there is a [documentation on Fluentd official site][fluentd-doc-kinesis].
 
@@ -18,17 +19,17 @@ Also, there is a [documentation on Fluentd official site][fluentd-doc-kinesis].
 ## Installation
 This Fluentd plugin is available as the `fluent-plugin-kinesis` gem from RubyGems.
 
-    gem install fluent-plugin-kinesis
+    gem install fluent-plugin-kinesis-modified
 
 Or you can install this plugin for [td-agent][td-agent] as:
 
-    td-agent-gem install fluent-plugin-kinesis
+    td-agent-gem install fluent-plugin-kinesis-modified
 
 If you would like to build by yourself and install, please see the section below. Your need [bundler][bundler] for this.
 
 In case of using with Fluentd: Fluentd will be also installed via the process below.
 
-    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git
+    git clone -b feature_modified_aggregation --single-branch https://github.com/BlackNight95/aws-fluent-plugin-kinesis.git
     cd aws-fluent-plugin-kinesis
     bundle install
     bundle exec rake build
@@ -36,15 +37,15 @@ In case of using with Fluentd: Fluentd will be also installed via the process be
 
 Also, you can use this plugin with td-agent: You have to install td-agent before installing this plugin.
 
-    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git
+    git clone -b feature_modified_aggregation --single-branch https://github.com/BlackNight95/aws-fluent-plugin-kinesis.git
     cd aws-fluent-plugin-kinesis
     bundle install
     bundle exec rake build
-    fluent-gem install pkg/fluent-plugin-kinesis
+    fluent-gem install pkg/fluent-plugin-kinesis-modified
 
 Or just download specify your Ruby library path. Below is the sample for specifying your library path via RUBYLIB.
 
-    git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git
+    git clone -b feature_modified_aggregation --single-branch https://github.com/BlackNight95/aws-fluent-plugin-kinesis.git
     cd aws-fluent-plugin-kinesis
     bundle install
     export RUBYLIB=$RUBYLIB:/path/to/aws-fluent-plugin-kinesis/lib
@@ -92,15 +93,15 @@ For more details, see [Configuration: kinesis_streams](#configuration-kinesis_st
     </match>
 For more details, see [Configuration: kinesis_firehose](#configuration-kinesis_firehose).
 
-### kinesis_streams_aggregated
+### kinesis_streams_aggregated/kinesis_streams_aggregated_modified
     <match your_tag>
-      @type kinesis_streams_aggregated
+      @type kinesis_streams_aggregated/kinesis_streams_aggregated_modified
       region us-east-1
       stream_name your_stream
       # Unlike kinesis_streams, there is no way to use dynamic partition key.
       # fixed_partition_key or random.
     </match>
-For more details, see [Configuration: kinesis_streams_aggregated](#configuration-kinesis_streams_aggregated).
+For more details, see [Configuration: kinesis_streams_aggregated](#configuration-kinesis_streams_aggregated) and [Configuration: kinesis_streams_aggregated_modified](#configuration-kinesis_streams_aggregated_modified).
 
 ### For better throughput
 Add configurations like below:
@@ -346,8 +347,9 @@ Default:
 - `kinesis_streams`: 500
 - `kinesis_firehose`: 500
 - `kinesis_streams_aggregated`: 100,000
+- `kinesis_streams_aggregated_modified`: 100,000
 
-### batch_request_max_size
+### size_kb_per_record
 Integer. The number of max size of making batch request from record chunk. It can't exceed the default value because it's API limit.
 
 Default:
@@ -355,6 +357,7 @@ Default:
 - `kinesis_streams`: 5 MB
 - `kinesis_firehose`: 4 MB
 - `kinesis_streams_aggregated`: 1 MB
+- `kinesis_streams_aggregated_modified`: 1 MB
 
 ## Configuration: kinesis_streams
 Here are `kinesis_streams` specific configurations.
@@ -386,6 +389,21 @@ A value of fixed partition key. Default `nil`, which means partition key will be
 
 Note: if you specified this option, all records go to a single shard.
 
+## Configuration: kinesis_streams_aggregated_modified
+Here are `kinesis_streams_aggregated_modified` specific configurations.
+
+### stream_name
+Name of the stream to put data.
+
+### fixed_partition_key
+A value of fixed partition key. Default `nil`, which means partition key will be generated randomly.
+
+Note: if you specified this option, all records go to a single shard.
+
+### max_records_per_call
+Integer. Max number of aggregated records sent to kinesis with put_records method in each iteration. max_records_per_call * size_kb_per_record should not exceed 5 MB.
+
+
 ## Development
 
 To launch `fluentd` process with this plugin for development, follow the steps below:

data/lib/fluent/plugin/out_kinesis_streams_aggregated_modified.rb

@@ -21,7 +21,7 @@ module Fluent
       Fluent::Plugin.register_output('kinesis_streams_aggregated_modified', self)
       include KinesisHelper::Aggregator::Mixin
 
-      RequestType = :streams_aggregated
+      RequestType = :streams
       BatchRequestLimitCount = 100_000
       BatchRequestLimitSize  = 1024 * 1024
       include KinesisHelper::API::BatchRequest

data/lib/fluent_plugin_kinesis/version.rb

@@ -13,5 +13,5 @@
 # language governing permissions and limitations under the License.
 
 module FluentPluginKinesis
-  VERSION = '3.1.3'
+  VERSION = '3.1.4'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-kinesis-modified
 version: !ruby/object:Gem::Version
-  version: 3.1.3
+  version: 3.1.4
 platform: ruby
 authors:
 - iHandy
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-07-17 00:00:00.000000000 Z
+date: 2019-08-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -305,7 +305,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.2.3
+rubygems_version: 2.5.2
 signing_key: 
 specification_version: 4
 summary: Fluentd output plugin that sends events to Amazon Kinesis. Forked from fluent-plugin-kinesis