fluent-plugin-elasticsearch 3.5.6 → 3.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 91343c74b95ee9133fd8323079dbbdd89e539f07942dd936d7107df9dfc5bf8f
4
- data.tar.gz: f51a62e66164d542e7aa2d21cfa6565085e3bf6be1db92cc31adb4ad5709588f
3
+ metadata.gz: 22474d12620f4cc8c2a2224b19e5e98534bd38d86f9489ace710b50139ea1422
4
+ data.tar.gz: 85c5043ca7ab0905efb9e2093c3b04d83dadae645e5aa2e580dbd601db77fe88
5
5
  SHA512:
6
- metadata.gz: 283d651e87233d84bd871df3f1d8b472d91f47dc53953ac47b282cfee654762b0199c7779a398485c24eb89885a01aff5204481a660ed6d7e0fb5243ccbfb2ef
7
- data.tar.gz: 6ae7ab7dac0721dd82082e01ca9de3bc969949159d9b263cf53e43d62d01a543cb84ba6992d46189f16ab4b9b1c63de2ac8dd49b45e9778e7a9ec01fa66395c8
6
+ metadata.gz: f22d3fe7adbb44c8b38f9c61757857937c3293c762dfba6842081e08df65ceddd3c75a0bfbf791c325fa1d82380e6d5b09ed7a3a774f03706a095cb8553323ac
7
+ data.tar.gz: 5f418e39091ac5bdceb2869604480e9b822c7654ba35da44c0ac636b88148168bee352c8bd7c3197fc5642019ab2a145430b4a88a722902c609457f1930eed2d
@@ -1,12 +1,21 @@
1
1
  language: ruby
2
2
 
3
- rvm:
4
- - 2.1
5
- - 2.2
6
- - 2.3
7
- - 2.4.5
8
- - 2.5.3
9
- - 2.6.0
3
+ matrix:
4
+ include:
5
+ - rvm: 2.1
6
+ gemfile: Gemfile
7
+ - rvm: 2.2
8
+ gemfile: Gemfile
9
+ - rvm: 2.3
10
+ gemfile: Gemfile
11
+ - rvm: 2.4.6
12
+ gemfile: Gemfile
13
+ - rvm: 2.5.5
14
+ gemfile: Gemfile
15
+ - rvm: 2.6.3
16
+ gemfile: gemfiles/Gemfile.ilm
17
+ - rvm: 2.6.3
18
+ gemfile: Gemfile
10
19
 
11
20
  gemfile:
12
21
  - Gemfile
data/History.md CHANGED
@@ -1,6 +1,12 @@
1
1
  ## Changelog [[tags]](https://github.com/uken/fluent-plugin-elasticsearch/tags)
2
2
 
3
3
  ### [Unreleased]
4
+ ### 3.6.0
5
+ - Set order in newly created templates (#660)
6
+ - Merge Support index lifecycle management into master (#659)
7
+ - Support template installation with host placeholder (#654)
8
+ - Support index lifecycle management (#651)
9
+
4
10
  ### 3.5.6
5
11
  - Support elasticsearch8 removal of mapping types (#656)
6
12
  - Upgrade webmock to 3 (#652)
data/README.md CHANGED
@@ -87,6 +87,9 @@ Current maintainers: @cosmo0920
87
87
  + [ignore_exceptions](#ignore_exceptions)
88
88
  + [exception_backup](#exception_backup)
89
89
  + [bulk_message_request_threshold](#bulk_message_request_threshold)
90
+ + [enable_ilm](#enable_ilm)
91
+ + [ilm_policy_id](#ilm_policy_id)
92
+ + [ilm_policy](#ilm_policy)
90
93
  * [Troubleshooting](#troubleshooting)
91
94
  + [Cannot send events to elasticsearch](#cannot-send-events-to-elasticsearch)
92
95
  + [Cannot see detailed failure log](#cannot-see-detailed-failure-log)
@@ -97,6 +100,7 @@ Current maintainers: @cosmo0920
97
100
  + [Stopped to send events on k8s, why?](#stopped-to-send-events-on-k8s-why)
98
101
  + [Random 400 - Rejected by Elasticsearch is occured, why?](#random-400---rejected-by-elasticsearch-is-occured-why)
99
102
  + [Fluentd seems to hang if it unable to connect Elasticsearch, why?](#fluentd-seems-to-hang-if-it-unable-to-connect-elasticsearch-why)
103
+ + [Enable Index Lifecycle Management](#enable-index-lifecycle-management)
100
104
  * [Contact](#contact)
101
105
  * [Contributing](#contributing)
102
106
  * [Running tests](#running-tests)
@@ -112,6 +116,8 @@ NOTE: For v0.12 version, you should use 1.x.y version. Please send patch into v0
112
116
 
113
117
  NOTE: This documentation is for fluent-plugin-elasticsearch 2.x or later. For 1.x documentation, please see [v0.12 branch](https://github.com/uken/fluent-plugin-elasticsearch/tree/v0.12).
114
118
 
119
+ NOTE: Using Index Lifecycle management(ILM) feature needs to install elasticsearch-xpack gem v7.4.0 or later.
120
+
115
121
  ## Installation
116
122
 
117
123
  ```sh
@@ -1120,6 +1126,30 @@ Default value is `20MB`. (20 * 1024 * 1024)
1120
1126
 
1121
1127
  If you specify this size as negative number, `bulk_message` request splitting feature will be disabled.
1122
1128
 
1129
+ ## enable_ilm
1130
+
1131
+ Enable Index Lifecycle Management (ILM).
1132
+
1133
+ Default value is `false`.
1134
+
1135
+ **NOTE:** This parameter requests to install elasticsearch-xpack gem.
1136
+
1137
+ ## ilm_policy_id
1138
+
1139
+ Specify ILM policy id.
1140
+
1141
+ Default value is `logstash-policy`.
1142
+
1143
+ **NOTE:** This parameter requests to install elasticsearch-xpack gem.
1144
+
1145
+ ## ilm_policy
1146
+
1147
+ Specify ILM policy contents as Hash.
1148
+
1149
+ Default value is `{}`.
1150
+
1151
+ **NOTE:** This parameter requests to install elasticsearch-xpack gem.
1152
+
1123
1153
  ## Troubleshooting
1124
1154
 
1125
1155
  ### Cannot send events to Elasticsearch
@@ -1537,6 +1567,42 @@ To remove too pessimistic behavior, you can use the following configuration:
1537
1567
  </match>
1538
1568
  ```
1539
1569
 
1570
+ ### Enable Index Lifecycle Management
1571
+
1572
+ Index lifecycle management is template based index management feature.
1573
+
1574
+ Main ILM feature parameters are:
1575
+
1576
+ * `rollover_index`
1577
+ * `deflector_alias`
1578
+ * `enable_ilm`
1579
+ * `ilm_policy_id`
1580
+ * `ilm_policy`
1581
+
1582
+ They are not all mandatory parameters but they are used for ILM feature in effect.
1583
+
1584
+ And also, ILM feature users should specify their Elasticsearch template for ILM enabled indices.
1585
+ Because ILM settings are injected into their Elasticsearch templates.
1586
+
1587
+ ```aconf
1588
+ index_name fluentd-${tag}
1589
+ # Should specify rollover_index as true
1590
+ rollover_index true
1591
+ deflector_alias fluentd-${tag} # Should specify as same index_name
1592
+ index_prefix fluentd
1593
+ application_name ${tag}
1594
+ index_date_pattern "now/d"
1595
+ enable_ilm true
1596
+ # Policy configurations
1597
+ ilm_policy_id fluentd-policy
1598
+ # ilm_policy {} # Use default policy
1599
+ template_name your-fluentd-template
1600
+ template_file /path/to/fluentd-template.json
1601
+ customize_template {"<<index_prefix>>": "fluentd"}
1602
+ ```
1603
+
1604
+ Note: This plugin only creates rollover-enabled indices, which are aliases pointing to them and index templates, and creates an ILM policy if enabled.
1605
+
1540
1606
  ## Contact
1541
1607
 
1542
1608
  If you have a question, [open an Issue](https://github.com/uken/fluent-plugin-elasticsearch/issues).
@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
3
3
 
4
4
  Gem::Specification.new do |s|
5
5
  s.name = 'fluent-plugin-elasticsearch'
6
- s.version = '3.5.6'
6
+ s.version = '3.6.0'
7
7
  s.authors = ['diogo', 'pitr']
8
8
  s.email = ['pitr.vern@gmail.com', 'me@diogoterror.com']
9
9
  s.description = %q{Elasticsearch output plugin for Fluent event collector}
@@ -0,0 +1,10 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in fluent-plugin-elasticsearch.gemspec
4
+ gemspec :path => "../"
5
+
6
+ gem 'simplecov', require: false
7
+ gem 'coveralls', require: false
8
+ gem 'strptime', require: false if RUBY_ENGINE == "ruby" && RUBY_VERSION =~ /^2/
9
+ gem "irb" if RUBY_ENGINE == "ruby" && RUBY_VERSION >= "2.6"
10
+ gem "elasticsearch-xpack"
@@ -0,0 +1,14 @@
1
+ {
2
+ "policy": {
3
+ "phases": {
4
+ "hot": {
5
+ "actions": {
6
+ "rollover": {
7
+ "max_size": "50gb",
8
+ "max_age": "30d"
9
+ }
10
+ }
11
+ }
12
+ }
13
+ }
14
+ }
@@ -0,0 +1,67 @@
1
+ module Fluent::Plugin::ElasticsearchIndexLifecycleManagement
2
+ ILM_DEFAULT_POLICY_PATH = "default-ilm-policy.json"
3
+
4
+ def setup_ilm(enable_ilm, policy_id, ilm_policy = default_policy_payload)
5
+ return unless enable_ilm
6
+
7
+ create_ilm_policy(policy_id, ilm_policy)
8
+ end
9
+
10
+ def verify_ilm_working
11
+ # Check the Elasticsearch instance for ILM readiness - this means that the version has to be a non-OSS release, with ILM feature
12
+ # available and enabled.
13
+ begin
14
+ xpack = xpack_info
15
+ if xpack.nil?
16
+ raise Fluent::ConfigError, "xpack endpoint does not work"
17
+ end
18
+ features = xpack["features"]
19
+ ilm = features.nil? ? nil : features["ilm"]
20
+ raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not installed on your Elasticsearch" if features.nil? || ilm.nil?
21
+ raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not available in your Elasticsearch" unless ilm['available']
22
+ raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not enabled in your Elasticsearch" unless ilm['enabled']
23
+
24
+ rescue Elasticsearch::Transport::Transport::Error => e
25
+ raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not installed on your Elasticsearch", error: e
26
+ end
27
+ end
28
+
29
+ def create_ilm_policy(policy_id, ilm_policy = default_policy_payload)
30
+ if !ilm_policy_exists?(policy_id)
31
+ ilm_policy_put(policy_id, ilm_policy)
32
+ end
33
+ end
34
+
35
+ def xpack_info
36
+ begin
37
+ client.xpack.info
38
+ rescue NoMethodError
39
+ raise RuntimeError, "elasticsearch-xpack gem is not installed."
40
+ rescue
41
+ nil
42
+ end
43
+ end
44
+
45
+ def get_ilm_policy
46
+ client.ilm.get_policy
47
+ end
48
+
49
+ def ilm_policy_exists?(policy_id)
50
+ begin
51
+ client.ilm.get_policy(policy_id: policy_id)
52
+ true
53
+ rescue
54
+ false
55
+ end
56
+ end
57
+
58
+ def ilm_policy_put(policy_id, policy)
59
+ log.info("Installing ILM policy: #{policy}")
60
+ client.ilm.put_policy(policy_id: policy_id, body: policy)
61
+ end
62
+
63
+ def default_policy_payload
64
+ default_policy_path = File.join(__dir__, ILM_DEFAULT_POLICY_PATH)
65
+ Yajl.load(::IO.read(default_policy_path))
66
+ end
67
+ end
@@ -21,8 +21,8 @@ module Fluent::ElasticsearchIndexTemplate
21
21
  JSON.parse(file_contents)
22
22
  end
23
23
 
24
- def template_exists?(name)
25
- client.indices.get_template(:name => name)
24
+ def template_exists?(name, host = nil)
25
+ client(host).indices.get_template(:name => name)
26
26
  return true
27
27
  rescue Elasticsearch::Transport::Transport::Errors::NotFound
28
28
  return false
@@ -51,52 +51,101 @@ module Fluent::ElasticsearchIndexTemplate
51
51
  end
52
52
  end
53
53
 
54
- def template_put(name, template)
55
- client.indices.put_template(:name => name, :body => template)
54
+ def template_put(name, template, host = nil)
55
+ client(host).indices.put_template(:name => name, :body => template)
56
56
  end
57
57
 
58
- def indexcreation(index_name)
59
- client.indices.create(:index => index_name)
58
+ def indexcreation(index_name, host = nil)
59
+ client(host).indices.create(:index => index_name)
60
60
  rescue Elasticsearch::Transport::Transport::Error => e
61
+ if e.message =~ /"already exists"/
62
+ log.debug("Index #{index_name} already exists")
63
+ else
61
64
  log.error("Error while index creation - #{index_name}: #{e.inspect}")
65
+ end
62
66
  end
63
67
 
64
- def template_install(name, template_file, overwrite)
68
+ def template_install(name, template_file, overwrite, enable_ilm = false, deflector_alias_name = nil, ilm_policy_id = nil, host = nil)
69
+ inject_template_name = get_template_name(enable_ilm, name, deflector_alias_name)
65
70
  if overwrite
66
- template_put(name, get_template(template_file))
67
- log.info("Template '#{name}' overwritten with #{template_file}.")
71
+ template_put(inject_template_name,
72
+ enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
73
+ ilm_policy_id,
74
+ get_template(template_file)) :
75
+ get_template(template_file), host)
76
+
77
+ log.info("Template '#{inject_template_name}' overwritten with #{template_file}.")
68
78
  return
69
79
  end
70
- if !template_exists?(name)
71
- template_put(name, get_template(template_file))
72
- log.info("Template configured, but no template installed. Installed '#{name}' from #{template_file}.")
80
+ if !template_exists?(inject_template_name, host)
81
+ template_put(inject_template_name,
82
+ enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
83
+ ilm_policy_id,
84
+ get_template(template_file)) :
85
+ get_template(template_file), host)
86
+ log.info("Template configured, but no template installed. Installed '#{inject_template_name}' from #{template_file}.")
73
87
  else
74
- log.info("Template configured and already installed.")
88
+ log.debug("Template '#{inject_template_name}' configured and already installed.")
75
89
  end
76
90
  end
77
91
 
78
- def template_custom_install(template_name, template_file, overwrite, customize_template, index_prefix, rollover_index, deflector_alias_name, app_name, index_date_pattern)
79
- template_custom_name=template_name.downcase
92
+ def template_custom_install(template_name, template_file, overwrite, customize_template, enable_ilm, deflector_alias_name, ilm_policy_id, host)
93
+ template_custom_name = get_template_name(enable_ilm, template_name, deflector_alias_name)
94
+ custom_template = if enable_ilm
95
+ inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id,
96
+ get_custom_template(template_file,
97
+ customize_template))
98
+ else
99
+ get_custom_template(template_file, customize_template)
100
+ end
80
101
  if overwrite
81
- template_put(template_custom_name, get_custom_template(template_file, customize_template))
102
+ template_put(template_custom_name, custom_template, host)
82
103
  log.info("Template '#{template_custom_name}' overwritten with #{template_file}.")
83
104
  else
84
- if !template_exists?(template_custom_name)
85
- template_put(template_custom_name, get_custom_template(template_file, customize_template))
105
+ if !template_exists?(template_custom_name, host)
106
+ template_put(template_custom_name, custom_template, host)
86
107
  log.info("Template configured, but no template installed. Installed '#{template_custom_name}' from #{template_file}.")
87
108
  else
88
- log.info("Template configured and already installed.")
109
+ log.debug("Template '#{template_custom_name}' configured and already installed.")
89
110
  end
90
111
  end
112
+ end
113
+
114
+ def get_template_name(enable_ilm, template_name, deflector_alias_name)
115
+ enable_ilm ? deflector_alias_name : template_name
116
+ end
117
+
118
+ def inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id, template)
119
+ log.debug("Overwriting index patterns when Index Lifecycle Management is enabled.")
120
+ template.delete('template') if template.include?('template')
121
+ template['index_patterns'] = "#{deflector_alias_name}-*"
122
+ template['order'] = template['order'] ? template['order'] + deflector_alias_name.split('-').length : 50 + deflector_alias_name.split('-').length
123
+ if template['settings'] && (template['settings']['index.lifecycle.name'] || template['settings']['index.lifecycle.rollover_alias'])
124
+ log.debug("Overwriting index lifecycle name and rollover alias when Index Lifecycle Management is enabled.")
125
+ end
126
+ template['settings'].update({ 'index.lifecycle.name' => ilm_policy_id, 'index.lifecycle.rollover_alias' => deflector_alias_name})
127
+ template
128
+ end
91
129
 
130
+ def create_rollover_alias(index_prefix, rollover_index, deflector_alias_name, app_name, index_date_pattern, index_separator, enable_ilm, ilm_policy_id, ilm_policy, host)
92
131
  if rollover_index
93
132
  if !client.indices.exists_alias(:name => deflector_alias_name)
94
- index_name_temp='<'+index_prefix.downcase+'-'+app_name.downcase+'-{'+index_date_pattern+'}-000001>'
95
- indexcreation(index_name_temp)
96
- client.indices.put_alias(:index => index_name_temp, :name => deflector_alias_name)
133
+ index_name_temp='<'+index_prefix.downcase+index_separator+app_name.downcase+'-{'+index_date_pattern+'}-000001>'
134
+ indexcreation(index_name_temp, host)
135
+ body = {}
136
+ body = rollover_alias_payload(deflector_alias_name) if enable_ilm
137
+ client.indices.put_alias(:index => index_name_temp, :name => deflector_alias_name,
138
+ :body => body)
97
139
  log.info("The alias '#{deflector_alias_name}' is created for the index '#{index_name_temp}'")
140
+ if enable_ilm
141
+ if ilm_policy.empty?
142
+ setup_ilm(enable_ilm, ilm_policy_id)
143
+ else
144
+ setup_ilm(enable_ilm, ilm_policy_id, ilm_policy)
145
+ end
146
+ end
98
147
  else
99
- log.info("The alias '#{deflector_alias_name}' is already present")
148
+ log.debug("The alias '#{deflector_alias_name}' is already present")
100
149
  end
101
150
  else
102
151
  log.info("No index and alias creation action performed because rollover_index is set to '#{rollover_index}'")
@@ -109,4 +158,13 @@ module Fluent::ElasticsearchIndexTemplate
109
158
  end
110
159
  end
111
160
 
161
+ def rollover_alias_payload(rollover_alias)
162
+ {
163
+ 'aliases' => {
164
+ rollover_alias => {
165
+ 'is_write_index' => true
166
+ }
167
+ }
168
+ }
169
+ end
112
170
  end
@@ -2,6 +2,10 @@
2
2
  require 'date'
3
3
  require 'excon'
4
4
  require 'elasticsearch'
5
+ begin
6
+ require 'elasticsearch/xpack'
7
+ rescue LoadError
8
+ end
5
9
  require 'json'
6
10
  require 'uri'
7
11
  begin
@@ -18,6 +22,7 @@ require_relative 'elasticsearch_constants'
18
22
  require_relative 'elasticsearch_error'
19
23
  require_relative 'elasticsearch_error_handler'
20
24
  require_relative 'elasticsearch_index_template'
25
+ require_relative 'elasticsearch_index_lifecycle_management'
21
26
  begin
22
27
  require_relative 'oj_serializer'
23
28
  rescue LoadError
@@ -44,6 +49,8 @@ module Fluent::Plugin
44
49
 
45
50
  RequestInfo = Struct.new(:host, :index)
46
51
 
52
+ attr_reader :alias_indexes
53
+
47
54
  helpers :event_emitter, :compat_parameters, :record_accessor
48
55
 
49
56
  Fluent::Plugin.register_output('elasticsearch', self)
@@ -54,6 +61,7 @@ module Fluent::Plugin
54
61
  DEFAULT_TYPE_NAME = "fluentd".freeze
55
62
  DEFAULT_RELOAD_AFTER = -1
56
63
  TARGET_BULK_BYTES = 20 * 1024 * 1024
64
+ DEFAULT_POLICY_ID = "logstash-policy"
57
65
 
58
66
  config_param :host, :string, :default => 'localhost'
59
67
  config_param :port, :integer, :default => 9200
@@ -105,6 +113,7 @@ EOC
105
113
  config_param :customize_template, :hash, :default => nil
106
114
  config_param :rollover_index, :string, :default => false
107
115
  config_param :index_date_pattern, :string, :default => "now/d"
116
+ config_param :index_separator, :string, :default => "-"
108
117
  config_param :deflector_alias, :string, :default => nil
109
118
  config_param :index_prefix, :string, :default => "logstash"
110
119
  config_param :application_name, :string, :default => "default"
@@ -139,6 +148,9 @@ EOC
139
148
  config_param :ignore_exceptions, :array, :default => [], value_type: :string, :desc => "Ignorable exception list"
140
149
  config_param :exception_backup, :bool, :default => true, :desc => "Chunk backup flag when ignore exception occured"
141
150
  config_param :bulk_message_request_threshold, :size, :default => TARGET_BULK_BYTES
151
+ config_param :enable_ilm, :bool, :default => false
152
+ config_param :ilm_policy_id, :string, :default => DEFAULT_POLICY_ID
153
+ config_param :ilm_policy, :hash, :default => {}
142
154
 
143
155
  config_section :buffer do
144
156
  config_set_default :@type, DEFAULT_BUFFER_TYPE
@@ -148,6 +160,7 @@ EOC
148
160
 
149
161
  include Fluent::ElasticsearchIndexTemplate
150
162
  include Fluent::Plugin::ElasticsearchConstants
163
+ include Fluent::Plugin::ElasticsearchIndexLifecycleManagement
151
164
 
152
165
  def initialize
153
166
  super
@@ -181,24 +194,25 @@ EOC
181
194
  raise Fluent::ConfigError, "'max_retry_putting_template' must be greater than or equal to zero." if @max_retry_putting_template < 0
182
195
  raise Fluent::ConfigError, "'max_retry_get_es_version' must be greater than or equal to zero." if @max_retry_get_es_version < 0
183
196
 
184
- # Raise error when using host placeholders and template features at same time.
197
+ # Dump log when using host placeholders and template features at same time.
185
198
  valid_host_placeholder = placeholder?(:host_placeholder, @host)
186
199
  if valid_host_placeholder && (@template_name && @template_file || @templates)
187
- raise Fluent::ConfigError, "host placeholder and template installation are exclusive features."
200
+ if @verify_es_version_at_startup
201
+ raise Fluent::ConfigError, "host placeholder, template installation, and verify Elasticsearch version at startup are exclusive feature at same time. Please specify verify_es_version_at_startup as `false` when host placeholder and template installation are enabled."
202
+ end
203
+ log.info "host placeholder and template installation makes your Elasticsearch cluster a bit slow down(beta)."
188
204
  end
189
205
 
206
+ @alias_indexes = []
190
207
  if !Fluent::Engine.dry_run_mode
191
208
  if @template_name && @template_file
192
- retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
193
- if @customize_template
194
- if @rollover_index
195
- raise Fluent::ConfigError, "'deflector_alias' must be provided if 'rollover_index' is set true ." if not @deflector_alias
196
- end
197
- template_custom_install(@template_name, @template_file, @template_overwrite, @customize_template, @index_prefix, @rollover_index, @deflector_alias, @application_name, @index_date_pattern)
198
- else
199
- template_install(@template_name, @template_file, @template_overwrite)
200
- end
209
+ if @rollover_index
210
+ raise Fluent::ConfigError, "'deflector_alias' must be provided if 'rollover_index' is set true ." if not @deflector_alias
201
211
  end
212
+ if @enable_ilm
213
+ raise Fluent::ConfigError, "'rollover_index' and 'deflector_alias' must be provided if 'enable_ilm' is set true ." if !@deflector_alias &&!@deflector_alias
214
+ end
215
+ verify_ilm_working if @enable_ilm
202
216
  elsif @templates
203
217
  retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
204
218
  templates_hash_install(@templates, @template_overwrite)
@@ -579,7 +593,17 @@ EOC
579
593
  else
580
594
  type_name = nil
581
595
  end
582
- return logstash_prefix, index_name, type_name
596
+ if @deflector_alias
597
+ deflector_alias = extract_placeholders(@deflector_alias, chunk)
598
+ else
599
+ deflector_alias = nil
600
+ end
601
+ if @application_name
602
+ application_name = extract_placeholders(@application_name, chunk)
603
+ else
604
+ application_name = nil
605
+ end
606
+ return logstash_prefix, index_name, type_name, deflector_alias, application_name
583
607
  end
584
608
 
585
609
  def multi_workers_ready?
@@ -653,7 +677,7 @@ EOC
653
677
  end
654
678
 
655
679
  def process_message(tag, meta, header, time, record, extracted_values)
656
- logstash_prefix, index_name, type_name = extracted_values
680
+ logstash_prefix, index_name, type_name, deflector_alias, application_name = extracted_values
657
681
 
658
682
  if @flatten_hashes
659
683
  record = flatten_record(record)
@@ -747,6 +771,23 @@ EOC
747
771
  # send_bulk given a specific bulk request, the original tag,
748
772
  # chunk, and bulk_message_count
749
773
  def send_bulk(data, tag, chunk, bulk_message_count, extracted_values, info)
774
+ logstash_prefix, index_name, type_name, deflector_alias, application_name = extracted_values
775
+ if @template_name && @template_file
776
+ if @alias_indexes.include? deflector_alias
777
+ log.debug("Index alias #{deflector_alias} already exists (cached)")
778
+ else
779
+ retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
780
+ if @customize_template
781
+ template_custom_install(@template_name, @template_file, @template_overwrite, @customize_template, @enable_ilm, deflector_alias, @ilm_policy_id, info.host)
782
+ else
783
+ template_install(@template_name, @template_file, @template_overwrite, @enable_ilm, deflector_alias, @ilm_policy_id, info.host)
784
+ end
785
+ create_rollover_alias(@index_prefix, @rollover_index, deflector_alias, application_name, @index_date_pattern, @index_separator, @enable_ilm, @ilm_policy_id, @ilm_policy, info.host)
786
+ end
787
+ @alias_indexes << deflector_alias unless deflector_alias.nil?
788
+ end
789
+ end
790
+
750
791
  begin
751
792
 
752
793
  log.on_trace { log.trace "bulk request: #{data}" }