fluent-plugin-elasticsearch 4.3.3 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ed7efe8849b8a7ca1d2cc3c81444e604c0bda824209f9297f0114f01c1ba609a
-  data.tar.gz: f74f8726fa993c3e4bc01b742e97ef4579fb7335809aac2dabce03c2b002e9ad
+  metadata.gz: 5f2f8268d9a8a5acf6d941a915044bd3781a5722cea8bc7ac205d6b7fd6fe580
+  data.tar.gz: 7d9373fb963040efac0ea42f11bf1841f2e0b2a39b5566f5e2794507e77e5f5c
 SHA512:
-  metadata.gz: 6adeee82a86f8217f9e6d259990042330d8a96a3a3ddb03a332a90f4ce90996c99aff7385b05c625eee4c6253b9647ba2bfc7856f51ba306ef7df6917d716fb4
-  data.tar.gz: dec1a1a72c88c2fd4df38e1b57ca883d47c5f28582fb5ba9a9210c882986c5e52a44f4eda03bef708961b461555b1538968b905b346e6782ad1090ca4310747d
+  metadata.gz: 0e432748181717cedfa55239d2da7b0141c3280da7240f6f9643db411f1f3168f00f88c6933d65a5d1b944620dab0d87ce088f4dba76fdf17cec336ca55e83bf
+  data.tar.gz: f44b2a14c5a13e1a59bd7d6926af13a9a9ba7b990594bca45de20071756d9ca88f7ee53d6ec00e09f0ad2ed054510a7803ac11c7b695275d37dbacb1e5d51457

data/.travis.yml

@@ -10,10 +10,6 @@ jobs:
       gemfile: Gemfile
       os: linux
       arch: amd64
-    - rvm: 2.6.3
-      gemfile: gemfiles/Gemfile.without.ilm
-      os: linux
-      arch: amd64
     - rvm: 2.6.3
       gemfile: gemfiles/Gemfile.elasticsearch.v6
       os: linux

data/History.md

@@ -2,6 +2,13 @@
 
 ### [Unreleased]
 
+### 5.0.0
+-  Support #retry_operate on data stream (#863)
+- Support placeholder in @data\_stream\_name for @type elasticsearch\_data\_stream (#862)
+- Extract troubleshooting section (#861)
+- Fix unmatched `<source>` close tag (#860)
+- Initial support for Elasticsearch Data Stream (#859)
+
 ### 4.3.3
 - Handle invalid Elasticsearch::Client#info response (#855)
 

data/README.ElasticsearchInput.md

@@ -37,7 +37,7 @@ In your Fluentd configuration, use `@type elasticsearch` and specify `tag your.a
   index_name fluentd
   type_name fluentd
   tag my.logs
-</match>
+</source>
 ```
 
 ## Configuration

data/README.Troubleshooting.md

@@ -0,0 +1,601 @@
+## Index
+
+* [Troubleshooting](#troubleshooting)
+  + [Cannot send events to elasticsearch](#cannot-send-events-to-elasticsearch)
+  + [Cannot see detailed failure log](#cannot-see-detailed-failure-log)
+  + [Cannot connect TLS enabled reverse Proxy](#cannot-connect-tls-enabled-reverse-proxy)
+  + [Declined logs are resubmitted forever, why?](#declined-logs-are-resubmitted-forever-why)
+  + [Suggested to install typhoeus gem, why?](#suggested-to-install-typhoeus-gem-why)
+  + [Stopped to send events on k8s, why?](#stopped-to-send-events-on-k8s-why)
+  + [Random 400 - Rejected by Elasticsearch is occured, why?](#random-400---rejected-by-elasticsearch-is-occured-why)
+  + [Fluentd seems to hang if it unable to connect Elasticsearch, why?](#fluentd-seems-to-hang-if-it-unable-to-connect-elasticsearch-why)
+  + [Enable Index Lifecycle Management](#enable-index-lifecycle-management)
+  + [How to specify index codec](#how-to-specify-index-codec)
+  + [Cannot push logs to Elasticsearch with connect_write timeout reached, why?](#cannot-push-logs-to-elasticsearch-with-connect_write-timeout-reached-why)
+
+
+## Troubleshooting
+
+### Cannot send events to Elasticsearch
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance with an incompatible version.
+
+For example, td-agent currently bundles the 6.x series of the [elasticsearch-ruby](https://github.com/elastic/elasticsearch-ruby) library. This means that your Elasticsearch server also needs to be 6.x. You can check the actual version of the client library installed on your system by executing the following command.
+
+```
+# For td-agent users
+$ /usr/sbin/td-agent-gem list elasticsearch
+# For standalone Fluentd users
+$ fluent-gem list elasticsearch
+```
+Or, fluent-plugin-elasticsearch v2.11.7 or later, users can inspect version incompatibility with the `validate_client_version` option:
+
+```
+validate_client_version true
+```
+
+If you get the following error message, please consider to install compatible elasticsearch client gems:
+
+```
+Detected ES 5 but you use ES client 6.1.0.
+Please consider to use 5.x series ES client.
+```
+
+For further details of the version compatibility issue, please read [the official manual](https://github.com/elastic/elasticsearch-ruby#compatibility).
+
+### Cannot see detailed failure log
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance with an incompatible ssl protocol version.
+
+For example, `out_elasticsearch` set up ssl_version to TLSv1 due to historical reason.
+Modern Elasticsearch ecosystem requests to communicate with TLS v1.2 or later.
+But, in this case, `out_elasticsearch` conceals transporter part failure log by default.
+If you want to acquire transporter log, please consider to set the following configuration:
+
+```
+with_transporter_log true
+@log_level debug
+```
+
+Then, the following log is shown in Fluentd log:
+
+```
+2018-10-24 10:00:00 +0900 [error]: #0 [Faraday::ConnectionFailed] SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol (OpenSSL::SSL::SSLError) {:host=>"elasticsearch-host", :port=>80, :scheme=>"https", :user=>"elastic", :password=>"changeme", :protocol=>"https"}
+```
+
+This indicates that inappropriate TLS protocol version is used.
+If you want to use TLS v1.2, please use `ssl_version` parameter like as:
+
+```
+ssl_version TLSv1_2
+```
+
+or, in v4.0.2 or later with Ruby 2.5 or later combination, the following congiuration is also valid:
+
+```
+ssl_max_version TLSv1_2
+ssl_min_version TLSv1_2
+```
+
+### Cannot connect TLS enabled reverse Proxy
+
+A common cause of failure is that you are trying to connect to an Elasticsearch instance behind nginx reverse proxy which uses an incompatible ssl protocol version.
+
+For example, `out_elasticsearch` set up ssl_version to TLSv1 due to historical reason.
+Nowadays, nginx reverse proxy uses TLS v1.2 or later for security reason.
+But, in this case, `out_elasticsearch` conceals transporter part failure log by default.
+
+If you set up nginx reverse proxy with TLS v1.2:
+
+```
+server {
+    listen <your IP address>:9400;
+    server_name <ES-Host>;
+    ssl on;
+    ssl_certificate /etc/ssl/certs/server-bundle.pem;
+    ssl_certificate_key /etc/ssl/private/server-key.pem;
+    ssl_client_certificate /etc/ssl/certs/ca.pem;
+    ssl_verify_client   on;
+    ssl_verify_depth    2;
+
+    # Reference : https://cipherli.st/
+    ssl_protocols TLSv1.2;
+    ssl_prefer_server_ciphers on;
+    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
+    ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
+    ssl_session_cache shared:SSL:10m;
+    ssl_session_tickets off; # Requires nginx >= 1.5.9
+    ssl_stapling on; # Requires nginx >= 1.3.7
+    ssl_stapling_verify on; # Requires nginx => 1.3.7
+    resolver 127.0.0.1 valid=300s;
+    resolver_timeout 5s;
+    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
+    add_header X-Frame-Options DENY;
+    add_header X-Content-Type-Options nosniff;
+
+    client_max_body_size 64M;
+    keepalive_timeout 5;
+
+    location / {
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_pass http://localhost:9200;
+    }
+}
+```
+
+Then, nginx reverse proxy starts with TLSv1.2.
+
+Fluentd suddenly dies with the following log:
+```
+Oct 31 9:44:45 <ES-Host> fluentd[6442]: log writing failed. execution expired
+Oct 31 9:44:45 <ES-Host> fluentd[6442]: /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/ssl_socket.rb:10:in `initialize': stack level too deep (SystemStackError)
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `new'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `socket'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:111:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/mock.rb:48:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:          ... 9266 levels...
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/bin/fluentd:8:in `<top (required)>'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/bin/fluentd:22:in `load'
+Oct 31 9:44:45 <ES-Host> fluentd[6442]:         from /opt/fluentd/embedded/bin/fluentd:22:in `<main>'
+Oct 31 9:44:45 <ES-Host> systemd[1]: fluentd.service: Control process exited, code=exited status=1
+```
+
+If you want to acquire transporter log, please consider to set the following configuration:
+
+```
+with_transporter_log true
+@log_level debug
+```
+
+Then, the following log is shown in Fluentd log:
+
+```
+2018-10-31 10:00:57 +0900 [warn]: #7 [Faraday::ConnectionFailed] Attempt 2 connecting to {:host=>"<ES-Host>", :port=>9400, :scheme=>"https", :protocol=>"https"}
+2018-10-31 10:00:57 +0900 [error]: #7 [Faraday::ConnectionFailed] Connection reset by peer - SSL_connect (Errno::ECONNRESET) {:host=>"<ES-Host>", :port=>9400, :scheme=>"https", :protocol=>"https"}
+```
+
+The above logs indicates that using incompatible SSL/TLS version between fluent-plugin-elasticsearch and nginx, which is reverse proxy, is root cause of this issue.
+
+If you want to use TLS v1.2, please use `ssl_version` parameter like as:
+
+```
+ssl_version TLSv1_2
+```
+
+or, in v4.0.2 or later with Ruby 2.5 or later combination, the following congiuration is also valid:
+
+```
+ssl_max_version TLSv1_2
+ssl_min_version TLSv1_2
+```
+
+### Declined logs are resubmitted forever, why?
+
+Sometimes users write Fluentd configuration like this:
+
+```aconf
+<match **>
+  @type elasticsearch
+  host localhost
+  port 9200
+  type_name fluentd
+  logstash_format true
+  time_key @timestamp
+  include_timestamp true
+  reconnect_on_error true
+  reload_on_failure true
+  reload_connections false
+  request_timeout 120s
+</match>
+```
+
+The above configuration does not use [`@label` feature](https://docs.fluentd.org/v1.0/articles/config-file#(5)-group-filter-and-output:-the-%E2%80%9Clabel%E2%80%9D-directive) and use glob(**) pattern.
+It is usually problematic configuration.
+
+In error scenario, error events will be emitted with `@ERROR` label, and `fluent.*` tag.
+The black hole glob pattern resubmits a problematic event into pushing Elasticsearch pipeline.
+
+This situation causes flood of declined log:
+
+```log
+2018-11-13 11:16:27 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="app.fluentcat" time=2018-11-13 11:16:17.492985640 +0000 record={"message"=>"\xFF\xAD"}
+2018-11-13 11:16:38 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="fluent.warn" time=2018-11-13 11:16:27.978851140 +0000 record={"error"=>"#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch>", "location"=>nil, "tag"=>"app.fluentcat", "time"=>2018-11-13 11:16:17.492985640 +0000, "record"=>{"message"=>"\xFF\xAD"}, "message"=>"dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch\" location=nil tag=\"app.fluentcat\" time=2018-11-13 11:16:17.492985640 +0000 record={\"message\"=>\"\\xFF\\xAD\"}"}
+```
+
+Then, user should use more concrete tag route or use `@label`.
+The following sections show two examples how to solve flood of declined log.
+One is using concrete tag routing, the other is using label routing.
+
+#### Using concrete tag routing
+
+The following configuration uses concrete tag route:
+
+```aconf
+<match out.elasticsearch.**>
+  @type elasticsearch
+  host localhost
+  port 9200
+  type_name fluentd
+  logstash_format true
+  time_key @timestamp
+  include_timestamp true
+  reconnect_on_error true
+  reload_on_failure true
+  reload_connections false
+  request_timeout 120s
+</match>
+```
+
+#### Using label feature
+
+The following configuration uses label:
+
+```aconf
+<source>
+  @type forward
+  @label @ES
+</source>
+<label @ES>
+  <match out.elasticsearch.**>
+    @type elasticsearch
+    host localhost
+    port 9200
+    type_name fluentd
+    logstash_format true
+    time_key @timestamp
+    include_timestamp true
+    reconnect_on_error true
+    reload_on_failure true
+    reload_connections false
+    request_timeout 120s
+  </match>
+</label>
+<label @ERROR>
+  <match **>
+    @type stdout
+  </match>
+</label>
+```
+
+### Suggested to install typhoeus gem, why?
+
+fluent-plugin-elasticsearch doesn't depend on typhoeus gem by default.
+If you want to use typhoeus backend, you must install typhoeus gem by your own.
+
+If you use vanilla Fluentd, you can install it by:
+
+```
+gem install typhoeus
+```
+
+But, you use td-agent instead of vanilla Fluentd, you have to use `td-agent-gem`:
+
+```
+td-agent-gem install typhoeus
+```
+
+In more detail, please refer to [the official plugin management document](https://docs.fluentd.org/v1.0/articles/plugin-management).
+
+### Stopped to send events on k8s, why?
+
+fluent-plugin-elasticsearch reloads connection after 10000 requests. (Not correspond to events counts because ES plugin uses bulk API.)
+
+This functionality which is originated from elasticsearch-ruby gem is enabled by default.
+
+Sometimes this reloading functionality bothers users to send events with ES plugin.
+
+On k8s platform, users sometimes shall specify the following settings:
+
+```aconf
+reload_connections false
+reconnect_on_error true
+reload_on_failure true
+```
+
+If you use [fluentd-kubernetes-daemonset](https://github.com/fluent/fluentd-kubernetes-daemonset), you can specify them with environment variables:
+
+* `FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS` as `false`
+* `FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR` as `true`
+* `FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE` as `true`
+
+This issue had been reported at [#525](https://github.com/uken/fluent-plugin-elasticsearch/issues/525).
+
+### Random 400 - Rejected by Elasticsearch is occured, why?
+
+Index templates installed Elasticsearch sometimes generates 400 - Rejected by Elasticsearch errors.
+For example, kubernetes audit log has structure:
+
+```json
+"responseObject":{
+   "kind":"SubjectAccessReview",
+   "apiVersion":"authorization.k8s.io/v1beta1",
+   "metadata":{
+      "creationTimestamp":null
+   },
+   "spec":{
+      "nonResourceAttributes":{
+         "path":"/",
+         "verb":"get"
+      },
+      "user":"system:anonymous",
+      "group":[
+         "system:unauthenticated"
+      ]
+   },
+   "status":{
+      "allowed":true,
+      "reason":"RBAC: allowed by ClusterRoleBinding \"cluster-system-anonymous\" of ClusterRole \"cluster-admin\" to User \"system:anonymous\""
+   }
+},
+```
+
+The last element `status` sometimes becomes `"status":"Success"`.
+This element type glich causes status 400 error.
+
+There are some solutions for fixing this:
+
+#### Solution 1
+
+For a key which causes element type glich case.
+
+Using dymanic mapping with the following template:
+
+```json
+{
+  "template": "YOURINDEXNAME-*",
+  "mappings": {
+    "fluentd": {
+      "dynamic_templates": [
+        {
+          "default_no_index": {
+            "path_match": "^.*$",
+            "path_unmatch": "^(@timestamp|auditID|level|stage|requestURI|sourceIPs|metadata|objectRef|user|verb)(\\..+)?$",
+            "match_pattern": "regex",
+            "mapping": {
+              "index": false,
+              "enabled": false
+            }
+          }
+        }
+      ]
+    }
+  }
+}
+```
+
+Note that `YOURINDEXNAME` should be replaced with your using index prefix.
+
+#### Solution 2
+
+For unstable `responseObject` and `requestObject` key existence case.
+
+```aconf
+<filter YOURROUTETAG>
+  @id kube_api_audit_normalize
+  @type record_transformer
+  auto_typecast false
+  enable_ruby true
+  <record>
+    host "#{ENV['K8S_NODE_NAME']}"
+    responseObject ${record["responseObject"].nil? ? "none": record["responseObject"].to_json}
+    requestObject ${record["requestObject"].nil? ? "none": record["requestObject"].to_json}
+    origin kubernetes-api-audit
+  </record>
+</filter>
+```
+
+Normalize `responseObject` and `requestObject` key with record_transformer and other similiar plugins is needed.
+
+### Fluentd seems to hang if it unable to connect Elasticsearch, why?
+
+On `#configure` phase, ES plugin should wait until ES instance communication is succeeded.
+And ES plugin blocks to launch Fluentd by default.
+Because Fluentd requests to set up configuration correctly on `#configure` phase.
+
+After `#configure` phase, it runs very fast and send events heavily in some heavily using case.
+
+In this scenario, we need to set up configuration correctly until `#configure` phase.
+So, we provide default parameter is too conservative to use advanced users.
+
+To remove too pessimistic behavior, you can use the following configuration:
+
+```aconf
+<match **>
+  @type elasticsearch
+  # Some advanced users know their using ES version.
+  # We can disable startup ES version checking.
+  verify_es_version_at_startup false
+  # If you know that your using ES major version is 7, you can set as 7 here.
+  default_elasticsearch_version 7
+  # If using very stable ES cluster, you can reduce retry operation counts. (minmum is 1)
+  max_retry_get_es_version 1
+  # If using very stable ES cluster, you can reduce retry operation counts. (minmum is 1)
+  max_retry_putting_template 1
+  # ... and some ES plugin configuration
+</match>
+```
+
+### Enable Index Lifecycle Management
+
+Index lifecycle management is template based index management feature.
+
+Main ILM feature parameters are:
+
+* `index_name` (when logstash_format as false)
+* `logstash_prefix` (when logstash_format as true)
+* `enable_ilm`
+* `ilm_policy_id`
+* `ilm_policy`
+
+* Advanced usage parameters
+  * `application_name`
+  * `index_separator`
+
+They are not all mandatory parameters but they are used for ILM feature in effect.
+
+ILM target index alias is created with `index_name` or an index which is calculated from `logstash_prefix`.
+
+From Elasticsearch plugin v4.0.0, ILM target index will be calculated from `index_name` (normal mode) or `logstash_prefix` (using with `logstash_format`as true).
+
+**NOTE:** Before Elasticsearch plugin v4.1.0, using `deflector_alias` parameter when ILM is enabled is permitted and handled, but, in the later releases such that 4.1.1 or later, it cannot use with when ILM is enabled.
+
+And also, ILM feature users should specify their Elasticsearch template for ILM enabled indices.
+Because ILM settings are injected into their Elasticsearch templates.
+
+`application_name` and `index_separator` also affect alias index names.
+
+But this parameter is prepared for advanced usage.
+
+It usually should be used with default value which is `default`.
+
+Then, ILM parameters are used in alias index like as:
+
+##### Simple `index_name` case:
+
+`<index_name><index_separator><application_name>-000001`.
+
+##### `logstash_format` as `true` case:
+
+`<logstash_prefix><logstash_prefix_separator><application_name><logstash_prefix_separator><logstash_dateformat>-000001`.
+
+#### Example ILM settings
+
+```aconf
+index_name fluentd-${tag}
+application_name ${tag}
+index_date_pattern "now/d"
+enable_ilm true
+# Policy configurations
+ilm_policy_id fluentd-policy
+# ilm_policy {} # Use default policy
+template_name your-fluentd-template
+template_file /path/to/fluentd-template.json
+# customize_template {"<<index_prefix>>": "fluentd"}
+```
+
+Note: This plugin only creates rollover-enabled indices, which are aliases pointing to them and index templates, and creates an ILM policy if enabled.
+
+#### Create ILM indices in each day
+
+If you want to create new index in each day, you should use `logstash_format` style configuration:
+
+```aconf
+logstash_prefix fluentd
+application_name default
+index_date_pattern "now/d"
+enable_ilm true
+# Policy configurations
+ilm_policy_id fluentd-policy
+# ilm_policy {} # Use default policy
+template_name your-fluentd-template
+template_file /path/to/fluentd-template.json
+```
+
+Note that if you create a new set of indexes every day, the elasticsearch ILM policy system will treat each day separately and will always
+maintain a separate active write index for each day.
+
+If you have a rollover based on max_age, it will continue to roll the indexes for prior dates even if no new documents are indexed.  If you want
+to delete indexes after a period of time, the ILM policy will never delete the current write index regardless of its age, so you would need a separate
+system, such as curator, to actually delete the old indexes.
+
+For this reason, if you put the date into the index names with ILM you should only rollover based on size or number of documents and may need to use
+curator to actually delete old indexes.
+
+#### Fixed ILM indices
+
+Also, users can use fixed ILM indices configuration.
+If `index_date_pattern` is set as `""`(empty string), Elasticsearch plugin won't attach date pattern in ILM indices:
+
+```aconf
+index_name fluentd
+application_name default
+index_date_pattern ""
+enable_ilm true
+# Policy configurations
+ilm_policy_id fluentd-policy
+# ilm_policy {} # Use default policy
+template_name your-fluentd-template
+template_file /path/to/fluentd-template.json
+```
+
+### How to specify index codec
+
+Elasticsearch can handle compression methods for stored data such as LZ4 and best_compression.
+fluent-plugin-elasticsearch doesn't provide API which specifies compression method.
+
+Users can specify stored data compression method with template:
+
+Create `compression.json` as follows:
+
+```json
+{
+  "order": 100,
+  "index_patterns": [
+    "YOUR-INDEX-PATTERN"
+  ],
+  "settings": {
+    "index": {
+      "codec": "best_compression"
+    }
+  }
+}
+```
+
+Then, specify the above template in your configuration:
+
+```aconf
+template_name best_compression_tmpl
+template_file compression.json
+```
+
+Elasticsearch will store data with `best_compression`:
+
+```
+% curl -XGET 'http://localhost:9200/logstash-2019.12.06/_settings?pretty'
+```
+
+```json
+{
+  "logstash-2019.12.06" : {
+    "settings" : {
+      "index" : {
+        "codec" : "best_compression",
+        "number_of_shards" : "1",
+        "provided_name" : "logstash-2019.12.06",
+        "creation_date" : "1575622843800",
+        "number_of_replicas" : "1",
+        "uuid" : "THE_AWESOMEUUID",
+        "version" : {
+          "created" : "7040100"
+        }
+      }
+    }
+  }
+}
+```
+
+### Cannot push logs to Elasticsearch with connect_write timeout reached, why?
+
+It seems that Elasticsearch cluster is exhausted.
+
+Usually, Fluentd complains like the following log:
+
+```log
+2019-12-29 00:23:33 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=27.283766102716327 slow_flush_log_threshold=15.0 plugin_id="object:aaaffaaaaaff"
+2019-12-29 00:23:33 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=26.161768959928304 slow_flush_log_threshold=15.0 plugin_id="object:aaaffaaaaaff"
+2019-12-29 00:23:33 +0000 [warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=28.713624476008117 slow_flush_log_threshold=15.0 plugin_id="object:aaaffaaaaaff"
+2019-12-29 01:39:18 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. connect_write timeout reached
+2019-12-29 01:39:18 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. connect_write timeout reached
+```
+
+This warnings is usually caused by exhaused Elasticsearch cluster due to resource shortage.
+
+If CPU usage is spiked and Elasticsearch cluster is eating up CPU resource, this issue is caused by CPU resource shortage.
+
+Check your Elasticsearch cluster health status and resource usage.