fluent-plugin-elasticsearch 3.5.6 → 3.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.travis.yml +16 -7
- data/History.md +6 -0
- data/README.md +66 -0
- data/fluent-plugin-elasticsearch.gemspec +1 -1
- data/gemfiles/Gemfile.ilm +10 -0
- data/lib/fluent/plugin/default-ilm-policy.json +14 -0
- data/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb +67 -0
- data/lib/fluent/plugin/elasticsearch_index_template.rb +81 -23
- data/lib/fluent/plugin/out_elasticsearch.rb +54 -13
- data/test/plugin/test_elasticsearch_index_lifecycle_management.rb +63 -0
- data/test/plugin/test_out_elasticsearch.rb +617 -7
- metadata +7 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 22474d12620f4cc8c2a2224b19e5e98534bd38d86f9489ace710b50139ea1422
|
4
|
+
data.tar.gz: 85c5043ca7ab0905efb9e2093c3b04d83dadae645e5aa2e580dbd601db77fe88
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f22d3fe7adbb44c8b38f9c61757857937c3293c762dfba6842081e08df65ceddd3c75a0bfbf791c325fa1d82380e6d5b09ed7a3a774f03706a095cb8553323ac
|
7
|
+
data.tar.gz: 5f418e39091ac5bdceb2869604480e9b822c7654ba35da44c0ac636b88148168bee352c8bd7c3197fc5642019ab2a145430b4a88a722902c609457f1930eed2d
|
data/.travis.yml
CHANGED
@@ -1,12 +1,21 @@
|
|
1
1
|
language: ruby
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
3
|
+
matrix:
|
4
|
+
include:
|
5
|
+
- rvm: 2.1
|
6
|
+
gemfile: Gemfile
|
7
|
+
- rvm: 2.2
|
8
|
+
gemfile: Gemfile
|
9
|
+
- rvm: 2.3
|
10
|
+
gemfile: Gemfile
|
11
|
+
- rvm: 2.4.6
|
12
|
+
gemfile: Gemfile
|
13
|
+
- rvm: 2.5.5
|
14
|
+
gemfile: Gemfile
|
15
|
+
- rvm: 2.6.3
|
16
|
+
gemfile: gemfiles/Gemfile.ilm
|
17
|
+
- rvm: 2.6.3
|
18
|
+
gemfile: Gemfile
|
10
19
|
|
11
20
|
gemfile:
|
12
21
|
- Gemfile
|
data/History.md
CHANGED
@@ -1,6 +1,12 @@
|
|
1
1
|
## Changelog [[tags]](https://github.com/uken/fluent-plugin-elasticsearch/tags)
|
2
2
|
|
3
3
|
### [Unreleased]
|
4
|
+
### 3.6.0
|
5
|
+
- Set order in newly created templates (#660)
|
6
|
+
- Merge Support index lifecycle management into master (#659)
|
7
|
+
- Support template installation with host placeholder (#654)
|
8
|
+
- Support index lifecycle management (#651)
|
9
|
+
|
4
10
|
### 3.5.6
|
5
11
|
- Support elasticsearch8 removal of mapping types (#656)
|
6
12
|
- Upgrade webmock to 3 (#652)
|
data/README.md
CHANGED
@@ -87,6 +87,9 @@ Current maintainers: @cosmo0920
|
|
87
87
|
+ [ignore_exceptions](#ignore_exceptions)
|
88
88
|
+ [exception_backup](#exception_backup)
|
89
89
|
+ [bulk_message_request_threshold](#bulk_message_request_threshold)
|
90
|
+
+ [enable_ilm](#enable_ilm)
|
91
|
+
+ [ilm_policy_id](#ilm_policy_id)
|
92
|
+
+ [ilm_policy](#ilm_policy)
|
90
93
|
* [Troubleshooting](#troubleshooting)
|
91
94
|
+ [Cannot send events to elasticsearch](#cannot-send-events-to-elasticsearch)
|
92
95
|
+ [Cannot see detailed failure log](#cannot-see-detailed-failure-log)
|
@@ -97,6 +100,7 @@ Current maintainers: @cosmo0920
|
|
97
100
|
+ [Stopped to send events on k8s, why?](#stopped-to-send-events-on-k8s-why)
|
98
101
|
+ [Random 400 - Rejected by Elasticsearch is occured, why?](#random-400---rejected-by-elasticsearch-is-occured-why)
|
99
102
|
+ [Fluentd seems to hang if it unable to connect Elasticsearch, why?](#fluentd-seems-to-hang-if-it-unable-to-connect-elasticsearch-why)
|
103
|
+
+ [Enable Index Lifecycle Management](#enable-index-lifecycle-management)
|
100
104
|
* [Contact](#contact)
|
101
105
|
* [Contributing](#contributing)
|
102
106
|
* [Running tests](#running-tests)
|
@@ -112,6 +116,8 @@ NOTE: For v0.12 version, you should use 1.x.y version. Please send patch into v0
|
|
112
116
|
|
113
117
|
NOTE: This documentation is for fluent-plugin-elasticsearch 2.x or later. For 1.x documentation, please see [v0.12 branch](https://github.com/uken/fluent-plugin-elasticsearch/tree/v0.12).
|
114
118
|
|
119
|
+
NOTE: Using Index Lifecycle management(ILM) feature needs to install elasticsearch-xpack gem v7.4.0 or later.
|
120
|
+
|
115
121
|
## Installation
|
116
122
|
|
117
123
|
```sh
|
@@ -1120,6 +1126,30 @@ Default value is `20MB`. (20 * 1024 * 1024)
|
|
1120
1126
|
|
1121
1127
|
If you specify this size as negative number, `bulk_message` request splitting feature will be disabled.
|
1122
1128
|
|
1129
|
+
## enable_ilm
|
1130
|
+
|
1131
|
+
Enable Index Lifecycle Management (ILM).
|
1132
|
+
|
1133
|
+
Default value is `false`.
|
1134
|
+
|
1135
|
+
**NOTE:** This parameter requests to install elasticsearch-xpack gem.
|
1136
|
+
|
1137
|
+
## ilm_policy_id
|
1138
|
+
|
1139
|
+
Specify ILM policy id.
|
1140
|
+
|
1141
|
+
Default value is `logstash-policy`.
|
1142
|
+
|
1143
|
+
**NOTE:** This parameter requests to install elasticsearch-xpack gem.
|
1144
|
+
|
1145
|
+
## ilm_policy
|
1146
|
+
|
1147
|
+
Specify ILM policy contents as Hash.
|
1148
|
+
|
1149
|
+
Default value is `{}`.
|
1150
|
+
|
1151
|
+
**NOTE:** This parameter requests to install elasticsearch-xpack gem.
|
1152
|
+
|
1123
1153
|
## Troubleshooting
|
1124
1154
|
|
1125
1155
|
### Cannot send events to Elasticsearch
|
@@ -1537,6 +1567,42 @@ To remove too pessimistic behavior, you can use the following configuration:
|
|
1537
1567
|
</match>
|
1538
1568
|
```
|
1539
1569
|
|
1570
|
+
### Enable Index Lifecycle Management
|
1571
|
+
|
1572
|
+
Index lifecycle management is template based index management feature.
|
1573
|
+
|
1574
|
+
Main ILM feature parameters are:
|
1575
|
+
|
1576
|
+
* `rollover_index`
|
1577
|
+
* `deflector_alias`
|
1578
|
+
* `enable_ilm`
|
1579
|
+
* `ilm_policy_id`
|
1580
|
+
* `ilm_policy`
|
1581
|
+
|
1582
|
+
They are not all mandatory parameters but they are used for ILM feature in effect.
|
1583
|
+
|
1584
|
+
And also, ILM feature users should specify their Elasticsearch template for ILM enabled indices.
|
1585
|
+
Because ILM settings are injected into their Elasticsearch templates.
|
1586
|
+
|
1587
|
+
```aconf
|
1588
|
+
index_name fluentd-${tag}
|
1589
|
+
# Should specify rollover_index as true
|
1590
|
+
rollover_index true
|
1591
|
+
deflector_alias fluentd-${tag} # Should specify as same index_name
|
1592
|
+
index_prefix fluentd
|
1593
|
+
application_name ${tag}
|
1594
|
+
index_date_pattern "now/d"
|
1595
|
+
enable_ilm true
|
1596
|
+
# Policy configurations
|
1597
|
+
ilm_policy_id fluentd-policy
|
1598
|
+
# ilm_policy {} # Use default policy
|
1599
|
+
template_name your-fluentd-template
|
1600
|
+
template_file /path/to/fluentd-template.json
|
1601
|
+
customize_template {"<<index_prefix>>": "fluentd"}
|
1602
|
+
```
|
1603
|
+
|
1604
|
+
Note: This plugin only creates rollover-enabled indices, which are aliases pointing to them and index templates, and creates an ILM policy if enabled.
|
1605
|
+
|
1540
1606
|
## Contact
|
1541
1607
|
|
1542
1608
|
If you have a question, [open an Issue](https://github.com/uken/fluent-plugin-elasticsearch/issues).
|
@@ -3,7 +3,7 @@ $:.push File.expand_path('../lib', __FILE__)
|
|
3
3
|
|
4
4
|
Gem::Specification.new do |s|
|
5
5
|
s.name = 'fluent-plugin-elasticsearch'
|
6
|
-
s.version = '3.
|
6
|
+
s.version = '3.6.0'
|
7
7
|
s.authors = ['diogo', 'pitr']
|
8
8
|
s.email = ['pitr.vern@gmail.com', 'me@diogoterror.com']
|
9
9
|
s.description = %q{Elasticsearch output plugin for Fluent event collector}
|
@@ -0,0 +1,10 @@
|
|
1
|
+
source 'https://rubygems.org'
|
2
|
+
|
3
|
+
# Specify your gem's dependencies in fluent-plugin-elasticsearch.gemspec
|
4
|
+
gemspec :path => "../"
|
5
|
+
|
6
|
+
gem 'simplecov', require: false
|
7
|
+
gem 'coveralls', require: false
|
8
|
+
gem 'strptime', require: false if RUBY_ENGINE == "ruby" && RUBY_VERSION =~ /^2/
|
9
|
+
gem "irb" if RUBY_ENGINE == "ruby" && RUBY_VERSION >= "2.6"
|
10
|
+
gem "elasticsearch-xpack"
|
@@ -0,0 +1,67 @@
|
|
1
|
+
module Fluent::Plugin::ElasticsearchIndexLifecycleManagement
|
2
|
+
ILM_DEFAULT_POLICY_PATH = "default-ilm-policy.json"
|
3
|
+
|
4
|
+
def setup_ilm(enable_ilm, policy_id, ilm_policy = default_policy_payload)
|
5
|
+
return unless enable_ilm
|
6
|
+
|
7
|
+
create_ilm_policy(policy_id, ilm_policy)
|
8
|
+
end
|
9
|
+
|
10
|
+
def verify_ilm_working
|
11
|
+
# Check the Elasticsearch instance for ILM readiness - this means that the version has to be a non-OSS release, with ILM feature
|
12
|
+
# available and enabled.
|
13
|
+
begin
|
14
|
+
xpack = xpack_info
|
15
|
+
if xpack.nil?
|
16
|
+
raise Fluent::ConfigError, "xpack endpoint does not work"
|
17
|
+
end
|
18
|
+
features = xpack["features"]
|
19
|
+
ilm = features.nil? ? nil : features["ilm"]
|
20
|
+
raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not installed on your Elasticsearch" if features.nil? || ilm.nil?
|
21
|
+
raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not available in your Elasticsearch" unless ilm['available']
|
22
|
+
raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not enabled in your Elasticsearch" unless ilm['enabled']
|
23
|
+
|
24
|
+
rescue Elasticsearch::Transport::Transport::Error => e
|
25
|
+
raise Fluent::ConfigError, "Index Lifecycle management is enabled in Fluentd, but not installed on your Elasticsearch", error: e
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
def create_ilm_policy(policy_id, ilm_policy = default_policy_payload)
|
30
|
+
if !ilm_policy_exists?(policy_id)
|
31
|
+
ilm_policy_put(policy_id, ilm_policy)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
def xpack_info
|
36
|
+
begin
|
37
|
+
client.xpack.info
|
38
|
+
rescue NoMethodError
|
39
|
+
raise RuntimeError, "elasticsearch-xpack gem is not installed."
|
40
|
+
rescue
|
41
|
+
nil
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def get_ilm_policy
|
46
|
+
client.ilm.get_policy
|
47
|
+
end
|
48
|
+
|
49
|
+
def ilm_policy_exists?(policy_id)
|
50
|
+
begin
|
51
|
+
client.ilm.get_policy(policy_id: policy_id)
|
52
|
+
true
|
53
|
+
rescue
|
54
|
+
false
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
def ilm_policy_put(policy_id, policy)
|
59
|
+
log.info("Installing ILM policy: #{policy}")
|
60
|
+
client.ilm.put_policy(policy_id: policy_id, body: policy)
|
61
|
+
end
|
62
|
+
|
63
|
+
def default_policy_payload
|
64
|
+
default_policy_path = File.join(__dir__, ILM_DEFAULT_POLICY_PATH)
|
65
|
+
Yajl.load(::IO.read(default_policy_path))
|
66
|
+
end
|
67
|
+
end
|
@@ -21,8 +21,8 @@ module Fluent::ElasticsearchIndexTemplate
|
|
21
21
|
JSON.parse(file_contents)
|
22
22
|
end
|
23
23
|
|
24
|
-
def template_exists?(name)
|
25
|
-
client.indices.get_template(:name => name)
|
24
|
+
def template_exists?(name, host = nil)
|
25
|
+
client(host).indices.get_template(:name => name)
|
26
26
|
return true
|
27
27
|
rescue Elasticsearch::Transport::Transport::Errors::NotFound
|
28
28
|
return false
|
@@ -51,52 +51,101 @@ module Fluent::ElasticsearchIndexTemplate
|
|
51
51
|
end
|
52
52
|
end
|
53
53
|
|
54
|
-
def template_put(name, template)
|
55
|
-
client.indices.put_template(:name => name, :body => template)
|
54
|
+
def template_put(name, template, host = nil)
|
55
|
+
client(host).indices.put_template(:name => name, :body => template)
|
56
56
|
end
|
57
57
|
|
58
|
-
def indexcreation(index_name)
|
59
|
-
client.indices.create(:index => index_name)
|
58
|
+
def indexcreation(index_name, host = nil)
|
59
|
+
client(host).indices.create(:index => index_name)
|
60
60
|
rescue Elasticsearch::Transport::Transport::Error => e
|
61
|
+
if e.message =~ /"already exists"/
|
62
|
+
log.debug("Index #{index_name} already exists")
|
63
|
+
else
|
61
64
|
log.error("Error while index creation - #{index_name}: #{e.inspect}")
|
65
|
+
end
|
62
66
|
end
|
63
67
|
|
64
|
-
def template_install(name, template_file, overwrite)
|
68
|
+
def template_install(name, template_file, overwrite, enable_ilm = false, deflector_alias_name = nil, ilm_policy_id = nil, host = nil)
|
69
|
+
inject_template_name = get_template_name(enable_ilm, name, deflector_alias_name)
|
65
70
|
if overwrite
|
66
|
-
template_put(
|
67
|
-
|
71
|
+
template_put(inject_template_name,
|
72
|
+
enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
|
73
|
+
ilm_policy_id,
|
74
|
+
get_template(template_file)) :
|
75
|
+
get_template(template_file), host)
|
76
|
+
|
77
|
+
log.info("Template '#{inject_template_name}' overwritten with #{template_file}.")
|
68
78
|
return
|
69
79
|
end
|
70
|
-
if !template_exists?(
|
71
|
-
template_put(
|
72
|
-
|
80
|
+
if !template_exists?(inject_template_name, host)
|
81
|
+
template_put(inject_template_name,
|
82
|
+
enable_ilm ? inject_ilm_settings_to_template(deflector_alias_name,
|
83
|
+
ilm_policy_id,
|
84
|
+
get_template(template_file)) :
|
85
|
+
get_template(template_file), host)
|
86
|
+
log.info("Template configured, but no template installed. Installed '#{inject_template_name}' from #{template_file}.")
|
73
87
|
else
|
74
|
-
log.
|
88
|
+
log.debug("Template '#{inject_template_name}' configured and already installed.")
|
75
89
|
end
|
76
90
|
end
|
77
91
|
|
78
|
-
def template_custom_install(template_name, template_file, overwrite, customize_template,
|
79
|
-
template_custom_name=template_name
|
92
|
+
def template_custom_install(template_name, template_file, overwrite, customize_template, enable_ilm, deflector_alias_name, ilm_policy_id, host)
|
93
|
+
template_custom_name = get_template_name(enable_ilm, template_name, deflector_alias_name)
|
94
|
+
custom_template = if enable_ilm
|
95
|
+
inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id,
|
96
|
+
get_custom_template(template_file,
|
97
|
+
customize_template))
|
98
|
+
else
|
99
|
+
get_custom_template(template_file, customize_template)
|
100
|
+
end
|
80
101
|
if overwrite
|
81
|
-
template_put(template_custom_name,
|
102
|
+
template_put(template_custom_name, custom_template, host)
|
82
103
|
log.info("Template '#{template_custom_name}' overwritten with #{template_file}.")
|
83
104
|
else
|
84
|
-
if !template_exists?(template_custom_name)
|
85
|
-
template_put(template_custom_name,
|
105
|
+
if !template_exists?(template_custom_name, host)
|
106
|
+
template_put(template_custom_name, custom_template, host)
|
86
107
|
log.info("Template configured, but no template installed. Installed '#{template_custom_name}' from #{template_file}.")
|
87
108
|
else
|
88
|
-
log.
|
109
|
+
log.debug("Template '#{template_custom_name}' configured and already installed.")
|
89
110
|
end
|
90
111
|
end
|
112
|
+
end
|
113
|
+
|
114
|
+
def get_template_name(enable_ilm, template_name, deflector_alias_name)
|
115
|
+
enable_ilm ? deflector_alias_name : template_name
|
116
|
+
end
|
117
|
+
|
118
|
+
def inject_ilm_settings_to_template(deflector_alias_name, ilm_policy_id, template)
|
119
|
+
log.debug("Overwriting index patterns when Index Lifecycle Management is enabled.")
|
120
|
+
template.delete('template') if template.include?('template')
|
121
|
+
template['index_patterns'] = "#{deflector_alias_name}-*"
|
122
|
+
template['order'] = template['order'] ? template['order'] + deflector_alias_name.split('-').length : 50 + deflector_alias_name.split('-').length
|
123
|
+
if template['settings'] && (template['settings']['index.lifecycle.name'] || template['settings']['index.lifecycle.rollover_alias'])
|
124
|
+
log.debug("Overwriting index lifecycle name and rollover alias when Index Lifecycle Management is enabled.")
|
125
|
+
end
|
126
|
+
template['settings'].update({ 'index.lifecycle.name' => ilm_policy_id, 'index.lifecycle.rollover_alias' => deflector_alias_name})
|
127
|
+
template
|
128
|
+
end
|
91
129
|
|
130
|
+
def create_rollover_alias(index_prefix, rollover_index, deflector_alias_name, app_name, index_date_pattern, index_separator, enable_ilm, ilm_policy_id, ilm_policy, host)
|
92
131
|
if rollover_index
|
93
132
|
if !client.indices.exists_alias(:name => deflector_alias_name)
|
94
|
-
index_name_temp='<'+index_prefix.downcase+
|
95
|
-
indexcreation(index_name_temp)
|
96
|
-
|
133
|
+
index_name_temp='<'+index_prefix.downcase+index_separator+app_name.downcase+'-{'+index_date_pattern+'}-000001>'
|
134
|
+
indexcreation(index_name_temp, host)
|
135
|
+
body = {}
|
136
|
+
body = rollover_alias_payload(deflector_alias_name) if enable_ilm
|
137
|
+
client.indices.put_alias(:index => index_name_temp, :name => deflector_alias_name,
|
138
|
+
:body => body)
|
97
139
|
log.info("The alias '#{deflector_alias_name}' is created for the index '#{index_name_temp}'")
|
140
|
+
if enable_ilm
|
141
|
+
if ilm_policy.empty?
|
142
|
+
setup_ilm(enable_ilm, ilm_policy_id)
|
143
|
+
else
|
144
|
+
setup_ilm(enable_ilm, ilm_policy_id, ilm_policy)
|
145
|
+
end
|
146
|
+
end
|
98
147
|
else
|
99
|
-
log.
|
148
|
+
log.debug("The alias '#{deflector_alias_name}' is already present")
|
100
149
|
end
|
101
150
|
else
|
102
151
|
log.info("No index and alias creation action performed because rollover_index is set to '#{rollover_index}'")
|
@@ -109,4 +158,13 @@ module Fluent::ElasticsearchIndexTemplate
|
|
109
158
|
end
|
110
159
|
end
|
111
160
|
|
161
|
+
def rollover_alias_payload(rollover_alias)
|
162
|
+
{
|
163
|
+
'aliases' => {
|
164
|
+
rollover_alias => {
|
165
|
+
'is_write_index' => true
|
166
|
+
}
|
167
|
+
}
|
168
|
+
}
|
169
|
+
end
|
112
170
|
end
|
@@ -2,6 +2,10 @@
|
|
2
2
|
require 'date'
|
3
3
|
require 'excon'
|
4
4
|
require 'elasticsearch'
|
5
|
+
begin
|
6
|
+
require 'elasticsearch/xpack'
|
7
|
+
rescue LoadError
|
8
|
+
end
|
5
9
|
require 'json'
|
6
10
|
require 'uri'
|
7
11
|
begin
|
@@ -18,6 +22,7 @@ require_relative 'elasticsearch_constants'
|
|
18
22
|
require_relative 'elasticsearch_error'
|
19
23
|
require_relative 'elasticsearch_error_handler'
|
20
24
|
require_relative 'elasticsearch_index_template'
|
25
|
+
require_relative 'elasticsearch_index_lifecycle_management'
|
21
26
|
begin
|
22
27
|
require_relative 'oj_serializer'
|
23
28
|
rescue LoadError
|
@@ -44,6 +49,8 @@ module Fluent::Plugin
|
|
44
49
|
|
45
50
|
RequestInfo = Struct.new(:host, :index)
|
46
51
|
|
52
|
+
attr_reader :alias_indexes
|
53
|
+
|
47
54
|
helpers :event_emitter, :compat_parameters, :record_accessor
|
48
55
|
|
49
56
|
Fluent::Plugin.register_output('elasticsearch', self)
|
@@ -54,6 +61,7 @@ module Fluent::Plugin
|
|
54
61
|
DEFAULT_TYPE_NAME = "fluentd".freeze
|
55
62
|
DEFAULT_RELOAD_AFTER = -1
|
56
63
|
TARGET_BULK_BYTES = 20 * 1024 * 1024
|
64
|
+
DEFAULT_POLICY_ID = "logstash-policy"
|
57
65
|
|
58
66
|
config_param :host, :string, :default => 'localhost'
|
59
67
|
config_param :port, :integer, :default => 9200
|
@@ -105,6 +113,7 @@ EOC
|
|
105
113
|
config_param :customize_template, :hash, :default => nil
|
106
114
|
config_param :rollover_index, :string, :default => false
|
107
115
|
config_param :index_date_pattern, :string, :default => "now/d"
|
116
|
+
config_param :index_separator, :string, :default => "-"
|
108
117
|
config_param :deflector_alias, :string, :default => nil
|
109
118
|
config_param :index_prefix, :string, :default => "logstash"
|
110
119
|
config_param :application_name, :string, :default => "default"
|
@@ -139,6 +148,9 @@ EOC
|
|
139
148
|
config_param :ignore_exceptions, :array, :default => [], value_type: :string, :desc => "Ignorable exception list"
|
140
149
|
config_param :exception_backup, :bool, :default => true, :desc => "Chunk backup flag when ignore exception occured"
|
141
150
|
config_param :bulk_message_request_threshold, :size, :default => TARGET_BULK_BYTES
|
151
|
+
config_param :enable_ilm, :bool, :default => false
|
152
|
+
config_param :ilm_policy_id, :string, :default => DEFAULT_POLICY_ID
|
153
|
+
config_param :ilm_policy, :hash, :default => {}
|
142
154
|
|
143
155
|
config_section :buffer do
|
144
156
|
config_set_default :@type, DEFAULT_BUFFER_TYPE
|
@@ -148,6 +160,7 @@ EOC
|
|
148
160
|
|
149
161
|
include Fluent::ElasticsearchIndexTemplate
|
150
162
|
include Fluent::Plugin::ElasticsearchConstants
|
163
|
+
include Fluent::Plugin::ElasticsearchIndexLifecycleManagement
|
151
164
|
|
152
165
|
def initialize
|
153
166
|
super
|
@@ -181,24 +194,25 @@ EOC
|
|
181
194
|
raise Fluent::ConfigError, "'max_retry_putting_template' must be greater than or equal to zero." if @max_retry_putting_template < 0
|
182
195
|
raise Fluent::ConfigError, "'max_retry_get_es_version' must be greater than or equal to zero." if @max_retry_get_es_version < 0
|
183
196
|
|
184
|
-
#
|
197
|
+
# Dump log when using host placeholders and template features at same time.
|
185
198
|
valid_host_placeholder = placeholder?(:host_placeholder, @host)
|
186
199
|
if valid_host_placeholder && (@template_name && @template_file || @templates)
|
187
|
-
|
200
|
+
if @verify_es_version_at_startup
|
201
|
+
raise Fluent::ConfigError, "host placeholder, template installation, and verify Elasticsearch version at startup are exclusive feature at same time. Please specify verify_es_version_at_startup as `false` when host placeholder and template installation are enabled."
|
202
|
+
end
|
203
|
+
log.info "host placeholder and template installation makes your Elasticsearch cluster a bit slow down(beta)."
|
188
204
|
end
|
189
205
|
|
206
|
+
@alias_indexes = []
|
190
207
|
if !Fluent::Engine.dry_run_mode
|
191
208
|
if @template_name && @template_file
|
192
|
-
|
193
|
-
if @
|
194
|
-
if @rollover_index
|
195
|
-
raise Fluent::ConfigError, "'deflector_alias' must be provided if 'rollover_index' is set true ." if not @deflector_alias
|
196
|
-
end
|
197
|
-
template_custom_install(@template_name, @template_file, @template_overwrite, @customize_template, @index_prefix, @rollover_index, @deflector_alias, @application_name, @index_date_pattern)
|
198
|
-
else
|
199
|
-
template_install(@template_name, @template_file, @template_overwrite)
|
200
|
-
end
|
209
|
+
if @rollover_index
|
210
|
+
raise Fluent::ConfigError, "'deflector_alias' must be provided if 'rollover_index' is set true ." if not @deflector_alias
|
201
211
|
end
|
212
|
+
if @enable_ilm
|
213
|
+
raise Fluent::ConfigError, "'rollover_index' and 'deflector_alias' must be provided if 'enable_ilm' is set true ." if !@deflector_alias &&!@deflector_alias
|
214
|
+
end
|
215
|
+
verify_ilm_working if @enable_ilm
|
202
216
|
elsif @templates
|
203
217
|
retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
|
204
218
|
templates_hash_install(@templates, @template_overwrite)
|
@@ -579,7 +593,17 @@ EOC
|
|
579
593
|
else
|
580
594
|
type_name = nil
|
581
595
|
end
|
582
|
-
|
596
|
+
if @deflector_alias
|
597
|
+
deflector_alias = extract_placeholders(@deflector_alias, chunk)
|
598
|
+
else
|
599
|
+
deflector_alias = nil
|
600
|
+
end
|
601
|
+
if @application_name
|
602
|
+
application_name = extract_placeholders(@application_name, chunk)
|
603
|
+
else
|
604
|
+
application_name = nil
|
605
|
+
end
|
606
|
+
return logstash_prefix, index_name, type_name, deflector_alias, application_name
|
583
607
|
end
|
584
608
|
|
585
609
|
def multi_workers_ready?
|
@@ -653,7 +677,7 @@ EOC
|
|
653
677
|
end
|
654
678
|
|
655
679
|
def process_message(tag, meta, header, time, record, extracted_values)
|
656
|
-
logstash_prefix, index_name, type_name = extracted_values
|
680
|
+
logstash_prefix, index_name, type_name, deflector_alias, application_name = extracted_values
|
657
681
|
|
658
682
|
if @flatten_hashes
|
659
683
|
record = flatten_record(record)
|
@@ -747,6 +771,23 @@ EOC
|
|
747
771
|
# send_bulk given a specific bulk request, the original tag,
|
748
772
|
# chunk, and bulk_message_count
|
749
773
|
def send_bulk(data, tag, chunk, bulk_message_count, extracted_values, info)
|
774
|
+
logstash_prefix, index_name, type_name, deflector_alias, application_name = extracted_values
|
775
|
+
if @template_name && @template_file
|
776
|
+
if @alias_indexes.include? deflector_alias
|
777
|
+
log.debug("Index alias #{deflector_alias} already exists (cached)")
|
778
|
+
else
|
779
|
+
retry_operate(@max_retry_putting_template, @fail_on_putting_template_retry_exceed) do
|
780
|
+
if @customize_template
|
781
|
+
template_custom_install(@template_name, @template_file, @template_overwrite, @customize_template, @enable_ilm, deflector_alias, @ilm_policy_id, info.host)
|
782
|
+
else
|
783
|
+
template_install(@template_name, @template_file, @template_overwrite, @enable_ilm, deflector_alias, @ilm_policy_id, info.host)
|
784
|
+
end
|
785
|
+
create_rollover_alias(@index_prefix, @rollover_index, deflector_alias, application_name, @index_date_pattern, @index_separator, @enable_ilm, @ilm_policy_id, @ilm_policy, info.host)
|
786
|
+
end
|
787
|
+
@alias_indexes << deflector_alias unless deflector_alias.nil?
|
788
|
+
end
|
789
|
+
end
|
790
|
+
|
750
791
|
begin
|
751
792
|
|
752
793
|
log.on_trace { log.trace "bulk request: #{data}" }
|