fluent-plugin-elasticsearch 2.12.0 → 2.12.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: daf714ffb7c731c2d2792efd8930165c2ca8baa3c387eae815f41d2ccfd89ead
-  data.tar.gz: df13ef1152b10bb2d63e7a870ab9afcbed6f040e6aee2a671f376690d0e616bb
+  metadata.gz: 9d653e88c7896e738bd345f3f5732c1f41f251b9372bf62fb0da758769b891b5
+  data.tar.gz: 574a4d8d57c0f98a667f853efa0214c6e952b8b30b30197f000037ca7f3044c4
 SHA512:
-  metadata.gz: b33818935a059b0e150bb5dcc0cbc1179a3b7179d4fd0b29c964d8238efcbaa68cc59b3232f3f326ba855914b2d54d1fd9bdf0eb107a75108a1c38dae8c305e0
-  data.tar.gz: 4d8d1cd2a3700e657c763caf5544344e6bfad1c185879539f48b52a65b84e5ba4dbc8bd524a1934e7d29348a20d7763a4c156519f71bb49c296b3192b2185069
+  metadata.gz: 249b1fe3782ec8ce4401c671b24c85d239608c214574b3b692d62e7f201c68d4b4f85d28779e6259c25a5e05b212961eb8ef7c900d541b2c28ceaae3a62d5ead
+  data.tar.gz: 2d1841c7f6d6f71a63daabb78f3b75266be7543e6f6cf03e686beebcaf7aee4a9678f6885aface2fc1c12c6d695ea2185c6f689faef5e01c981793a5ee433cd3

data/History.md

@@ -2,6 +2,12 @@
 
 ### [Unreleased]
 
+### 2.12.1
+- Make configurable unrecoverable types (#501)
+- Add FAQ for TLS enabled nginx proxy TLS version incompatibility trouble (#496)
+- Add FAQs (#492)
+- Remove issuestats.com badges (#489)
+
 ### 2.12.0
 - Decoupling the custom template and rollover index creation #485 (#486)
 

data/ISSUE_TEMPLATE.md

@@ -23,3 +23,4 @@ Provide example config and message
   * paste boot log of fluentd or td-agent
   * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock
 * ES version (optional)
+* Bear Metal or Within Docker or Kubernetes or others? (optional)

data/README.md

@@ -4,8 +4,6 @@
 [![Build Status](https://travis-ci.org/uken/fluent-plugin-elasticsearch.png?branch=master)](https://travis-ci.org/uken/fluent-plugin-elasticsearch)
 [![Coverage Status](https://coveralls.io/repos/uken/fluent-plugin-elasticsearch/badge.png)](https://coveralls.io/r/uken/fluent-plugin-elasticsearch)
 [![Code Climate](https://codeclimate.com/github/uken/fluent-plugin-elasticsearch.png)](https://codeclimate.com/github/uken/fluent-plugin-elasticsearch)
-[![Issue Stats](http://issuestats.com/github/uken/fluent-plugin-elasticsearch/badge/pr)](http://issuestats.com/github/uken/fluent-plugin-elasticsearch)
-[![Issue Stats](http://issuestats.com/github/uken/fluent-plugin-elasticsearch/badge/issue)](http://issuestats.com/github/uken/fluent-plugin-elasticsearch)
 
 Send your logs to Elasticsearch (and search them with Kibana maybe?)
 
@@ -71,10 +69,15 @@ Current maintainers: @cosmo0920
   + [sniffer_class_name](#sniffer_class_name)
   + [reload_after](#reload_after)
   + [validate_client_version](#validate-client-version)
+  + [unrecoverable_error_types](#unrecoverable-error-types)
   + [Not seeing a config you need?](#not-seeing-a-config-you-need)
   + [Dynamic configuration](#dynamic-configuration)
   + [Placeholders](#placeholders)
   + [Multi workers](#multi-workers)
+* [Troubleshooting](#troubleshooting)
+  + [Cannot send events to elasticsearch](#cannot-send-events-to-elasticsearch)
+  + [Cannot see detailed failure log](#cannot-see-detailed-failure-log)
+  + [Cannot connect TLS enabled reverse Proxy](#cannot-connect-tls-enabled-reverse-proxy)
 * [Contact](#contact)
 * [Contributing](#contributing)
 * [Running tests](#running-tests)
@@ -820,6 +823,27 @@ When you use mismatched Elasticsearch server and client libraries, fluent-plugin
 validate_client_version true
 ```
 
+### Unrecoverable Error Types
+
+Default `unrecoverable_error_types` parameter is set up strictly.
+Because `es_rejected_execution_exception` is caused by exceeding Elasticsearch's thread pool capacity.
+Advanced users can increase its capacity, but normal users should follow default behavior.
+
+If you want to increase it and forcibly retrying bulk request, please consider to change `unrecoverable_error_types` parameter from default value.
+
+Change default value of `thread_pool.bulk.queue_size` in elasticsearch.yml:
+e.g.)
+
+```yaml
+thread_pool.bulk.queue_size: 1000
+```
+
+Then, remove `es_rejected_execution_exception` from `unrecoverable_error_types` parameter:
+
+```
+unrecoverable_error_types ["out_of_memory_error"]
+```
+
 ### Not seeing a config you need?
 
 We try to keep the scope of this plugin small and not add too many configuration options. If you think an option would be useful to others, feel free to open an issue or contribute a Pull Request.
@@ -922,6 +946,153 @@ Since Fluentd v0.14, multi workers feature had been implemented to increase thro
 </system>
 ```
 
+## Troubleshooting
+
+### Cannot send events to Elasticsearch
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance with an incompatible version.
+
+For example, td-agent currently bundles the 6.x series of the [elasticsearch-ruby](https://github.com/elastic/elasticsearch-ruby) library. This means that your Elasticsearch server also needs to be 6.x. You can check the actual version of the client library installed on your system by executing the following command.
+
+```
+# For td-agent users
+$ /usr/sbin/td-agent-gem list elasticsearch
+# For standalone Fluentd users
+$ fluent-gem list elasticsearch
+```
+Or, fluent-plugin-elasticsearch v2.11.7 or later, users can inspect version incompatibility with the `validate_client_version` option:
+
+```
+validate_client_version true
+```
+
+If you get the following error message, please consider to install compatibile elasticsearch client gems:
+
+```
+Detected ES 5 but you use ES client 6.1.0.
+Please consider to use 5.x series ES client.
+```
+
+For further details of the version compatibility issue, please read [the official manual](https://github.com/elastic/elasticsearch-ruby#compatibility).
+
+### Cannot see detailed failure log
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance with an incompatible ssl protocol version.
+
+For example, `out_elasticsearch` set up ssl_version to TLSv1 due to historical reason.
+Modern Elasticsearch ecosystem requests to communicate with TLS v1.2 or later.
+But, in this case, `out_elasticsearch` conceals transporter part failure log by default.
+If you want to aquire transporter log, please consider to set the following configuration:
+
+```
+with_transporter_log true
+@log_level debug
+```
+
+Then, the following log is shown in Fluentd log:
+
+```
+2018-10-24 10:00:00 +0900 [error]: #0 [Faraday::ConnectionFailed] SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol (OpenSSL::SSL::SSLError) {:host=>"elasticsearch-host", :port=>80, :scheme=>"https", :user=>"elastic", :password=>"changeme", :protocol=>"https"}
+```
+
+This indicates that inappropriate TLS protocol version is used.
+If you want to use TLS v1.2, please use `ssl_version` parameter like as:
+
+```
+ssl_version TLSv1_2
+```
+
+### Cannot connect TLS enabled reverse Proxy
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance behind nginx reverse proxy which uses an incompatible ssl protocol version.
+
+For example, `out_elasticsearch` set up ssl_version to TLSv1 due to historical reason.
+Nowadays, nginx reverse proxy uses TLS v1.2 or later for security reason.
+But, in this case, `out_elasticsearch` conceals transporter part failure log by default.
+
+If you set up nginx reverse proxy with TLS v1.2:
+
+```
+server {
+    listen <your IP address>:9400;
+    server_name <ES-Host>;
+    ssl on;
+    ssl_certificate /etc/ssl/certs/server-bundle.pem;
+    ssl_certificate_key /etc/ssl/private/server-key.pem;
+    ssl_client_certificate /etc/ssl/certs/ca.pem;
+    ssl_verify_client   on;
+    ssl_verify_depth    2;
+
+    # Reference : https://cipherli.st/
+    ssl_protocols TLSv1.2;
+    ssl_prefer_server_ciphers on;
+    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
+    ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
+    ssl_session_cache shared:SSL:10m;
+    ssl_session_tickets off; # Requires nginx >= 1.5.9
+    ssl_stapling on; # Requires nginx >= 1.3.7
+    ssl_stapling_verify on; # Requires nginx => 1.3.7
+    resolver 127.0.0.1 valid=300s;
+    resolver_timeout 5s;
+    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
+    add_header X-Frame-Options DENY;
+    add_header X-Content-Type-Options nosniff;
+
+    client_max_body_size 64M;
+    keepalive_timeout 5;
+
+    location / {
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_pass http://localhost:9200;
+    }
+}
+```
+
+Then, nginx reverse proxy starts with TLSv1.2.
+
+Fluentd suddenly dies with the following log:
+```
+Oct 31 9:44:45 <ES-Host> fluentd[6442]: log writing failed. execution expired
+Oct 31 9:44:45 <ES-Host> fluentd[6442]: /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/ssl_socket.rb:10:in `initialize': stack level too deep (SystemStackError)
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `new'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `socket'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:111:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/mock.rb:48:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:          ... 9266 levels...
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/bin/fluentd:8:in `<top (required)>'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/bin/fluentd:22:in `load'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/bin/fluentd:22:in `<main>'
+Oct 31 9:44:45 <ES-Host> systemd[1]: fluentd.service: Control process exited, code=exited status=1
+```
+
+If you want to aquire transporter log, please consider to set the following configuration:
+
+```
+with_transporter_log true
+@log_level debug
+```
+
+Then, the following log is shown in Fluentd log:
+
+```
+2018-10-31 10:00:57 +0900 [warn]: #7 [Faraday::ConnectionFailed] Attempt 2 connecting to {:host=>"<ES-Host>", :port=>9400, :scheme=>"https", :protocol=>"https"}
+2018-10-31 10:00:57 +0900 [error]: #7 [Faraday::ConnectionFailed] Connection reset by peer - SSL_connect (Errno::ECONNRESET) {:host=>"<ES-Host>", :port=>9400, :scheme=>"https", :protocol=>"https"}
+```
+
+The above logs indicates that using incompatibile SSL/TLS version between fluent-plugin-elasticsearch and nginx, which is reverse proxy, is root cause of this issue.
+
+If you want to use TLS v1.2, please use `ssl_version` parameter like as:
+
+```
+ssl_version TLSv1_2
+```
+
 ## Contact
 
 If you have a question, [open an Issue](https://github.com/uken/fluent-plugin-elasticsearch/issues).

data/fluent-plugin-elasticsearch.gemspec

@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |s|
   s.name          = 'fluent-plugin-elasticsearch'
-  s.version       = '2.12.0'
+  s.version       = '2.12.1'
   s.authors       = ['diogo', 'pitr']
   s.email         = ['pitr.vern@gmail.com', 'me@diogoterror.com']
   s.description   = %q{Elasticsearch output plugin for Fluent event collector}

data/lib/fluent/plugin/elasticsearch_error_handler.rb

@@ -16,7 +16,7 @@ class Fluent::Plugin::ElasticsearchErrorHandler
   end
 
   def unrecoverable_error_types
-    ["out_of_memory_error", "es_rejected_execution_exception"]
+    @plugin.unrecoverable_error_types
   end
 
   def unrecoverable_error?(type)

data/lib/fluent/plugin/out_elasticsearch.rb

@@ -123,6 +123,7 @@ EOC
     config_param :http_backend, :enum, list: [:excon, :typhoeus], :default => :excon
     config_param :validate_client_version, :bool, :default => false
     config_param :prefer_oj_serializer, :bool, :default => false
+    config_param :unrecoverable_error_types, :array, :default => ["out_of_memory_error", "es_rejected_execution_exception"]
 
     config_section :buffer do
       config_set_default :@type, DEFAULT_BUFFER_TYPE

data/test/plugin/test_elasticsearch_error_handler.rb

@@ -8,10 +8,12 @@ class TestElasticsearchErrorHandler < Test::Unit::TestCase
   class TestPlugin
     attr_reader :log
     attr_reader :write_operation, :error_events
+    attr_accessor :unrecoverable_error_types
     def initialize(log)
       @log = log
       @write_operation = 'index'
       @error_events = []
+      @unrecoverable_error_types = ["out_of_memory_error", "es_rejected_execution_exception"]
     end
 
     def router
@@ -176,6 +178,46 @@ class TestElasticsearchErrorHandler < Test::Unit::TestCase
     end
   end
 
+  def test_es_rejected_execution_exception_responses_as_not_error
+    plugin = TestPlugin.new(@log)
+    plugin.unrecoverable_error_types = ["out_of_memory_error"]
+    handler = Fluent::Plugin::ElasticsearchErrorHandler.new(plugin)
+    records = [{time: 123, record: {"foo" => "bar", '_id' => 'abc'}}]
+    response = parse_response(%({
+      "took" : 0,
+      "errors" : true,
+      "items" : [
+        {
+          "create" : {
+            "_index" : "foo",
+            "status" : 429,
+            "_type"  : "bar",
+            "error" : {
+              "type" : "es_rejected_execution_exception",
+              "reason":"rejected execution of org.elasticsearch.transport.TransportService"
+            }
+          }
+        }
+      ]
+     }))
+
+    begin
+      failed = false
+      chunk = MockChunk.new(records)
+      dummy_extracted_values = []
+      handler.handle_error(response, 'atag', chunk, response['items'].length, dummy_extracted_values)
+    rescue Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchRequestAbortError, Fluent::Plugin::ElasticsearchOutput::RetryStreamError=>e
+      failed = true
+      records = [].tap do |records|
+        next unless e.respond_to?(:retry_stream)
+        e.retry_stream.each {|time, record| records << record}
+      end
+      # should retry chunk when unrecoverable error is not thrown
+      assert_equal 1, records.length
+    end
+    assert_true failed
+  end
+
   def test_retry_error
     records = []
     error_records = Hash.new(false)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-elasticsearch
 version: !ruby/object:Gem::Version
-  version: 2.12.0
+  version: 2.12.1
 platform: ruby
 authors:
 - diogo
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-10-17 00:00:00.000000000 Z
+date: 2018-11-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd