fluent-plugin-kinesis-aggregation 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b75bac816dfcbedd2d634402e10f2e4c00bb9b0c
-  data.tar.gz: d4b59d91022744f5e9a4dcfe7d8f0e4d6ee89878
+  metadata.gz: 2fdb6fdb865ab11d3b344f3560e974b29db2a705
+  data.tar.gz: 7838a02ab5534233c27de0e3b2aac5f7434c6ce1
 SHA512:
-  metadata.gz: 0d06cdd97c38fdfa5a6f6ae66547718a4fd7d5d93453a9d2f3c7d749fd0fc93e6ab2d148c791815ab3e289bce4f8f1c00a4f944af938b4149a9fe6653b44685b
-  data.tar.gz: 8bb20bed570042d2619ec72a808b3a046d50b6a14065618b88294d55ed94d951fd5540646f6ac2321c483cd68ab19dd389e9cc701ec8b0732bd37985d7e358d5
+  metadata.gz: 4f58339950f8502bd0bb4467228dea4d4c734d6f81495187d4f8d5167c5085749464ffda164a3fbd34015d5885ee0346c6979162c62c6be36b9aa7934189cece
+  data.tar.gz: da93990656c28ccde4159cefdf9883af0a2917f5e722ad05673573c85756b7202dd1a3179fb377b8a554766fcd47a44e46b82f305f12aaa0f8d337018d462a53

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 # CHANGELOG
 
+## 0.3.1
+
+- Change aws-sdk usage to work with both v2 and v3
+  (in particular, makes it possible to use latest td-agent which includes the s3 plugin
+  and pulls in aws-sdk v3)
+
 ## 0.3.0
 
 - Update to use fluentd 0.14 API (stick to 0.2.3 if you need support for earlier versions of fluentd)

data/README.md

@@ -8,14 +8,26 @@ This is a rewrite of [aws-fluent-plugin-kinesis](https://github.com/awslabs/aws-
 a different shipment method using the
 [KPL aggregation format](https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md).
 
+*Since this plugin was forked, aws-fluent-plugin-kinesis has undergone considerable development (and improvement).
+Most notably, the upcoming 2.0 release supports KPL aggregated records using google-protobuf without
+the overhead of using the KPL:
+https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/107*
+
+*However, it still uses msgpack for internal buffering and only uses protobuf when it ships the records,
+whereas this plugin processes each record as it comes in and ships the result by simple concatenation
+of the encoded records. This may not be faster, of course - could depend on the overhead of calling
+the protobuf methods - but most probably is. The discussion below is also still mostly valid,
+in that the awslabs plugin does not have PutRecord == chunk equivalency, but instead has its
+own internal retry method.*
+
 The basic idea is to have one PutRecord === one chunk. This has a number of advantages:
 
 - much less complexity in plugin (less CPU/memory)
 - by aggregating, we increase the throughput and decrease the cost
 - since a single chunk either succeeds or fails,
   we get to use fluentd's more complex/complete retry mechanism
-  (which is also exposed in the monitor). The existing retry mechanism
-  has [unfortunate issues under heavy load](https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/42)
+  (which is also exposed by the monitor plugin; we view this in datadog). The existing retry mechanism
+  had [unfortunate issues under heavy load](https://github.com/awslabs/aws-fluent-plugin-kinesis/issues/42)
 - we get ordering within a chunk without having to rely on sequence
   numbers (though not overall ordering)
 
@@ -63,7 +75,7 @@ specify the library path via RUBYLIB:
 ## Dependencies
 
  * Ruby 2.1+
- * Fluentd 0.10.43+
+ * Fluentd 0.14.15+ (if you need 0.10 or 0.12 support, use the fluentd-v0.12 branch or version 0.2.x on rubygems)
 
 ## Basic Usage
 

data/fluent-plugin-kinesis-aggregation.gemspec

@@ -17,7 +17,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-kinesis-aggregation"
-  spec.version       = '0.3.0'
+  spec.version       = '0.3.1'
   spec.author        = 'Atlassian'
   spec.email         = 'jhaggerty@atlassian.com'
   spec.summary       = %q{Fluentd output plugin that sends KPL style aggregated events to Amazon Kinesis.}
@@ -35,7 +35,6 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "test-unit-rr", "~> 1.0"
 
   spec.add_dependency "fluentd", [">= 0.14.15", "< 2"]
-  spec.add_dependency "aws-sdk-core", ">= 2.0.12", "< 3.0"
-  spec.add_dependency "msgpack", "~> 1.0"
+  spec.add_dependency "aws-sdk", ">= 2.0.12", "< 4.0"
   spec.add_dependency "google-protobuf", "~> 3.0"
 end

data/lib/fluent/plugin/out_kinesis-aggregation.rb

@@ -11,7 +11,7 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 
-require 'aws-sdk-core'
+require 'aws-sdk'
 require 'yajl'
 require 'logger'
 require 'securerandom'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-kinesis-aggregation
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Atlassian
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-08-07 00:00:00.000000000 Z
+date: 2018-03-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -73,7 +73,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2'
 - !ruby/object:Gem::Dependency
-  name: aws-sdk-core
+  name: aws-sdk
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -81,7 +81,7 @@ dependencies:
         version: 2.0.12
     - - "<"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -91,21 +91,7 @@ dependencies:
         version: 2.0.12
     - - "<"
       - !ruby/object:Gem::Version
-        version: '3.0'
-- !ruby/object:Gem::Dependency
-  name: msgpack
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '4.0'
 - !ruby/object:Gem::Dependency
   name: google-protobuf
   requirement: !ruby/object:Gem::Requirement
@@ -160,7 +146,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 2.5.2.2
 signing_key: 
 specification_version: 4
 summary: Fluentd output plugin that sends KPL style aggregated events to Amazon Kinesis.