fluent-plugin-elasticsearch 2.0.0 → 2.0.1.rc.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: be60a48d14569fe431d01e4791dad39dca520057
-  data.tar.gz: 5a58c1c390eedcf2c7a45a511af2d5a99c5c44b4
+  metadata.gz: a1eace20ded89e27d5d2ff1b27756dec9d616694
+  data.tar.gz: 662270c8258077cfd76d824c9015fce4c441bde7
 SHA512:
-  metadata.gz: bd02e104796941648c9dee0dcda38f29426a2ad7d8971003a834f1ad304d27f3048c68adbb0d166655580828ef4142a2727b346d578eb3bf9e743a2377fa55e3
-  data.tar.gz: 52168dd54eedf52993ae2a21265ae5b3310404b778e4a4da2aad463d3b37fdc921d7d6176a0f4d47311705cc24d1e16b9d9381c771e77fad1e48bc4cad4668f6
+  metadata.gz: 947cbfbe4cd83dc106befe56408cd8ad954c385bc98742381ed1310e7e641f75c319348098b211848daaeafca72eada16d863132bf85bc4a25902f4bd8ad01c4
+  data.tar.gz: 52a61b0d82ca4c9cc2be40792424a846cf96d521adb184ee6da030ad19949dba0a69e876d6ae82c571b56011fd8317c8cf3a7cc83356aadee7bf5d461771d8dd

data/History.md

@@ -4,6 +4,9 @@
 - Log ES response errors (#230)
 - Use latest elasticsearch-ruby (#240)
 
+### 2.0.1.rc.1
+- Add generating hash id mechanism to avoid records duplication (#318)
+
 ### 2.0.0
 - Release for Fluentd v0.14 stable.
 

data/README.md

@@ -53,6 +53,7 @@ Current maintainers: @cosmo0920
   + [Proxy Support](#proxy-support)
   + [Buffer options](#buffer-options)
   + [Hash flattening](#hash-flattening)
+  + [Generate Hash ID](#generate-hash-id)
   + [Not seeing a config you need?](#not-seeing-a-config-you-need)
   + [Dynamic configuration](#dynamic-configuration)
   + [Placeholders](#placeholders)
@@ -347,7 +348,7 @@ reload_on_failure true # defaults to false
 You can set in the elasticsearch-transport how often dead connections from the elasticsearch-transport's pool will be resurrected.
 
 ```
-resurrect_after 5 # defaults to 60s
+resurrect_after 5s # defaults to 60s
 ```
 
 ### include_tag_key, tag_key
@@ -388,6 +389,29 @@ This following record `{"name": "Johnny", "request_id": "87d89af7daffad6"}` will
 { "name": "Johnny", "request_id": "87d89af7daffad6" }
 ```
 
+Fluentd re-emits events that failed to be indexed/ingested in Elasticsearch with a new and unique `_id` value, this means that congested Elasticsearch clusters that reject events (due to command queue overflow, for example) will cause Fluentd to re-emit the event with a new `_id`, however Elasticsearch may actually process both (or more) attempts (with some delay) and create duplicate events in the index (since each have a unique `_id` value), one possible workaround is to use the [fluent-plugin-genhashvalue](https://github.com/mtakemi/fluent-plugin-genhashvalue) plugin to generate a unique `_hash` key in the record of each event, this `_hash` record can be used as the `id_key` to prevent Elasticsearch from creating deplicate events.
+
+```
+id_key _hash
+```
+
+Example configuration for [fluent-plugin-genhashvalue](https://github.com/mtakemi/fluent-plugin-genhashvalue) (review the documentation of the plugin for more details)
+```
+<filter logs.**>
+  @type genhashvalue
+  keys sessionid,requestid
+  hash_type md5    # md5/sha1/sha256/sha512
+  base64_enc true
+  base91_enc false
+  set_key _hash
+  separator _
+  inc_time_as_key true
+  inc_tag_as_key true
+</filter>
+```
+
+:warning: In order to avoid hash-collisions and loosing data careful consideration is required when choosing the keys in the event record that should be used to calculate the hash 
+
 ### parent_key
 
 ```
@@ -500,7 +524,7 @@ Starting with version 0.8.0, this gem uses excon, which supports proxy with envi
 
 ```
 buffer_type memory
-flush_interval 60
+flush_interval 60s
 retry_limit 17
 retry_wait 1.0
 num_threads 1
@@ -529,6 +553,19 @@ This will produce elasticsearch output that looks like this:
 
 Note that the flattener does not deal with arrays at this time.
 
+### Generate Hash ID
+
+By default, the fluentd elasticsearch plugin does not emit records with a _id field, leaving it to Elasticsearch to generate a unique _id as the record is indexed. When an Elasticsearch cluster is congested and begins to take longer to respond than the configured request_timeout, the fluentd elasticsearch plugin will re-send the same bulk request. Since Elasticsearch can't tell its actually the same request, all documents in the request are indexed again resulting in duplicate data. In certain scenarios, this can result in essentially and infinite loop generating multiple copies of the same data.
+
+Using an _id generated by the fluentd elasticsearch plugin will communicate to Elasticsearch the uniqueness of the requests so that duplicates will be rejected or simply replace the existing records.
+Here is a sample config:
+
+```
+<hash>
+  hash_id_key _id    # storing generated hash id key
+<hash>
+```
+
 ### Not seeing a config you need?
 
 We try to keep the scope of this plugin small and not add too many configuration options. If you think an option would be useful to others, feel free to open an issue or contribute a Pull Request.

data/fluent-plugin-elasticsearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '2.0.0'
+  s.version       = '2.0.1.rc.1'
   s.authors       = ['diogo', 'pitr']
   s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com']
   s.description   = %q{ElasticSearch output plugin for Fluent event collector}

data/lib/fluent/plugin/generate_hash_id_support.rb

@@ -0,0 +1,23 @@
+require 'securerandom'
+require 'base64'
+
+module Fluent
+  module Plugin
+    module GenerateHashIdSupport
+      def self.included(klass)
+        klass.instance_eval {
+          config_section :hash, param_name: :hash_config, required: false, multi: false do
+            config_param :hash_id_key, :string, default: '_id'
+          end
+        }
+      end
+
+      def generate_hash_id_key(record)
+        s = ""
+        s += Base64.strict_encode64(SecureRandom.uuid)
+        record[@hash_config.hash_id_key] = s
+        record
+      end
+    end
+  end
+end

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -11,6 +11,7 @@ end
 
 require 'fluent/plugin/output'
 require_relative 'elasticsearch_index_template'
+require_relative 'generate_hash_id_support'
 
 module Fluent::Plugin
   class ElasticsearchOutput < Output
@@ -79,6 +80,7 @@ module Fluent::Plugin
     end
 
     include Fluent::ElasticsearchIndexTemplate
+    include Fluent::Plugin::GenerateHashIdSupport
 
     def initialize
       super
@@ -340,6 +342,10 @@ module Fluent::Plugin
           record = flatten_record(record)
         end
 
+        if @hash_config
+          record = generate_hash_id_key(record)
+        end
+
         dt = nil
         if @logstash_format || @include_timestamp
           if record.has_key?(TIMESTAMP_FIELD)

data/lib/fluent/plugin/out_elasticsearch_dynamic.rb

@@ -13,6 +13,8 @@ module Fluent::Plugin
     DYNAMIC_PARAM_NAMES = %W[hosts host port include_timestamp logstash_format logstash_prefix logstash_dateformat time_key utc_index index_name tag_key type_name id_key parent_key routing_key write_operation]
     DYNAMIC_PARAM_SYMBOLS = DYNAMIC_PARAM_NAMES.map { |n| "@#{n}".to_sym }
 
+    include Fluent::Plugin::GenerateHashIdSupport
+
     attr_reader :dynamic_config
 
     def configure(conf)
@@ -130,6 +132,10 @@ module Fluent::Plugin
       chunk.msgpack_each do |time, record|
         next unless record.is_a? Hash
 
+        if @hash_config
+          record = generate_hash_id_key(record)
+        end
+
         begin
           # evaluate all configurations here
           DYNAMIC_PARAM_SYMBOLS.each_with_index { |var, i|

data/test/plugin/test_out_elasticsearch.rb

@@ -1,10 +1,12 @@
 require 'helper'
 require 'date'
+require 'fluent/test/helpers'
 require 'fluent/test/driver/output'
 require 'flexmock/test_unit'
 
 class ElasticsearchOutput < Test::Unit::TestCase
   include FlexMock::TestCase
+  include Fluent::Test::Helpers
 
   attr_accessor :index_cmds, :index_command_counts
 
@@ -438,6 +440,34 @@ class ElasticsearchOutput < Test::Unit::TestCase
     assert_equal('myindex', index_cmds.first['index']['_index'])
   end
 
+  class AdditionalHashIdMechanismTest < self
+    data("default"            => {"hash_id_key" => '_id'},
+         "custom hash_id_key" => {"hash_id_key" => '_hash_id'},
+        )
+    def test_writes_with_genrate_hash(data)
+      driver.configure(Fluent::Config::Element.new(
+                         'ROOT', '', {
+                           '@type' => 'elasticsearch',
+                           'id_key' => data["hash_id_key"],
+                         }, [
+                           Fluent::Config::Element.new('hash', '', {
+                                                         'keys' => ['request_id'],
+                                                         'hash_id_key' => data["hash_id_key"],
+                                                       }, [])
+                         ]
+                       ))
+      stub_elastic_ping
+      stub_elastic
+      flexmock(SecureRandom).should_receive(:uuid)
+        .and_return("13a0c028-bf7c-4ae2-ad03-ec09a40006df")
+      time = event_time("2017-10-15 15:00:23.34567890 UTC")
+      driver.run(default_tag: 'test') do
+        driver.feed(time, sample_record.merge('request_id' => 'elastic'))
+      end
+      assert_equal(Base64.strict_encode64(SecureRandom.uuid), index_cmds[1]["#{data["hash_id_key"]}"])
+    end
+  end
+
   class IndexNamePlaceholdersTest < self
     def test_writes_to_speficied_index_with_tag_placeholder
       driver.configure("index_name myindex.${tag}\n")

data/test/plugin/test_out_elasticsearch_dynamic.rb

@@ -1,10 +1,12 @@
 require 'helper'
 require 'date'
+require 'fluent/test/helpers'
 require 'fluent/test/driver/output'
 require 'flexmock/test_unit'
 
 class ElasticsearchOutputDynamic < Test::Unit::TestCase
   include FlexMock::TestCase
+  include Fluent::Test::Helpers
 
   attr_accessor :index_cmds, :index_command_counts
 
@@ -316,6 +318,35 @@ class ElasticsearchOutputDynamic < Test::Unit::TestCase
     assert_equal(2000, total)
   end
 
+  class AdditionalHashIdMechanismTest < self
+    data("default"            => {"hash_id_key" => '_id'},
+         "custom hash_id_key" => {"hash_id_key" => '_hash_id'},
+        )
+    def test_writes_with_genrate_hash(data)
+      driver.configure(Fluent::Config::Element.new(
+                         'ROOT', '', {
+                           '@type' => 'elasticsearch',
+                           'id_key' => data["hash_id_key"],
+                         }, [
+                           Fluent::Config::Element.new('hash', '', {
+                                                         'keys' => ['request_id'],
+                                                         'hash_id_key' => data["hash_id_key"],
+                                                       }, [])
+                         ]
+                       ))
+      stub_elastic_ping
+      stub_elastic
+      stub_elastic
+      flexmock(SecureRandom).should_receive(:uuid)
+        .and_return("82120f33-897a-4d9d-b3d5-14afd18fb412")
+      time = event_time("2017-10-15 15:00:23.34567890 UTC")
+      driver.run(default_tag: 'test') do
+        driver.feed(time, sample_record.merge('request_id' => 'elastic'))
+      end
+      assert_equal(Base64.strict_encode64(SecureRandom.uuid), index_cmds[1]["#{data["hash_id_key"]}"])
+    end
+  end
+
   def test_makes_bulk_request
     stub_elastic_ping
     stub_elastic
@@ -734,7 +765,7 @@ class ElasticsearchOutputDynamic < Test::Unit::TestCase
     stub_request(:post, "http://localhost:9200/_bulk").with do |req|
       raise ZeroDivisionError, "any not host_unreachable_exceptions exception"
     end
-    
+
     driver.configure("reconnect_on_error false\n")
 
     assert_raise(ZeroDivisionError) {

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.0.1.rc.1
 platform: ruby
 authors:
 - diogo
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-11-06 00:00:00.000000000 Z
+date: 2017-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -144,6 +144,7 @@ files:
 - Rakefile
 - fluent-plugin-elasticsearch.gemspec
 - lib/fluent/plugin/elasticsearch_index_template.rb
+- lib/fluent/plugin/generate_hash_id_support.rb
 - lib/fluent/plugin/out_elasticsearch.rb
 - lib/fluent/plugin/out_elasticsearch_dynamic.rb
 - test/helper.rb
@@ -165,9 +166,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
 rubygems_version: 2.6.13