fluentd 1.14.6 → 1.15.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/linux-test.yaml +1 -1
  3. data/.github/workflows/windows-test.yaml +4 -1
  4. data/CHANGELOG.md +53 -1
  5. data/fluentd.gemspec +2 -1
  6. data/lib/fluent/command/ctl.rb +4 -1
  7. data/lib/fluent/command/fluentd.rb +10 -0
  8. data/lib/fluent/config/literal_parser.rb +2 -2
  9. data/lib/fluent/config/yaml_parser/fluent_value.rb +47 -0
  10. data/lib/fluent/config/yaml_parser/loader.rb +91 -0
  11. data/lib/fluent/config/yaml_parser/parser.rb +166 -0
  12. data/lib/fluent/config/yaml_parser/section_builder.rb +107 -0
  13. data/lib/fluent/config/yaml_parser.rb +56 -0
  14. data/lib/fluent/config.rb +14 -1
  15. data/lib/fluent/plugin/file_wrapper.rb +52 -107
  16. data/lib/fluent/plugin/in_tail/group_watch.rb +204 -0
  17. data/lib/fluent/plugin/in_tail/position_file.rb +1 -15
  18. data/lib/fluent/plugin/in_tail.rb +66 -47
  19. data/lib/fluent/plugin/out_forward/socket_cache.rb +2 -0
  20. data/lib/fluent/plugin/output.rb +2 -1
  21. data/lib/fluent/plugin/parser_syslog.rb +1 -1
  22. data/lib/fluent/plugin_helper/server.rb +3 -1
  23. data/lib/fluent/plugin_helper/service_discovery.rb +2 -2
  24. data/lib/fluent/supervisor.rb +109 -25
  25. data/lib/fluent/system_config.rb +2 -1
  26. data/lib/fluent/version.rb +1 -1
  27. data/lib/fluent/winsvc.rb +2 -0
  28. data/test/command/test_ctl.rb +0 -1
  29. data/test/command/test_fluentd.rb +33 -0
  30. data/test/config/test_system_config.rb +3 -1
  31. data/test/config/test_types.rb +1 -1
  32. data/test/plugin/in_tail/test_io_handler.rb +14 -4
  33. data/test/plugin/in_tail/test_position_file.rb +0 -63
  34. data/test/plugin/out_forward/test_socket_cache.rb +26 -1
  35. data/test/plugin/test_file_wrapper.rb +0 -68
  36. data/test/plugin/test_in_object_space.rb +9 -3
  37. data/test/plugin/test_in_syslog.rb +1 -1
  38. data/test/plugin/test_in_tail.rb +629 -353
  39. data/test/plugin/test_out_forward.rb +30 -20
  40. data/test/plugin/test_parser_syslog.rb +1 -1
  41. data/test/plugin_helper/test_cert_option.rb +1 -1
  42. data/test/plugin_helper/test_child_process.rb +16 -4
  43. data/test/test_config.rb +135 -4
  44. data/test/test_supervisor.rb +155 -0
  45. metadata +11 -5
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4c1188a4d037f788a1fbeec731c7117fef237cd4036cd5a5e5b891bba297d184
4
- data.tar.gz: f3a1b383fd8d1ee4e5b431c92c83007b58c26f8d91bf1c9b0ce911662935c4e7
3
+ metadata.gz: 7bc368532203bcfd7b189e51e26a60f500db1306e6f5a4f94a03f333fb9af22d
4
+ data.tar.gz: b87ab437598cb9a41bdf6301e137baadcfc27c709d3e28354c2ba4e8ff8859ed
5
5
  SHA512:
6
- metadata.gz: 9816373aac361da5cbd010e0720982f3f09ab59846e53ade46eccd225b7fc0201f8d02ddb009f3ffb813cfb41d92bd3360edbb7f5ea55ac644f46ad678521d21
7
- data.tar.gz: d92e47b57424f825252d3fcc3204812d2747e634d98762a7843a6d68f1386258fc6302793311d21d82c3801e5f71f0fc30a4af1dc5464e2411a4d65b652b960b
6
+ metadata.gz: 45e167be3ee4b62e444b9f789694f7891590836b7e8888382a6368d3b49ddbece59f614b6eb20eb4b51f62b5e8fd4563e79304163fc78b3aaf4cc57d62a71546
7
+ data.tar.gz: 44112f3c79c48e09fb6c791a0edf6a465815d6ff3f3ccaa500cb4e6b7411c39a3ac7b587c11587b57568777786e3781e2ab68cc7e78088d4af386556b699e247
@@ -13,7 +13,7 @@ jobs:
13
13
  strategy:
14
14
  fail-fast: false
15
15
  matrix:
16
- ruby-version: ['3.1', '3.0', '2.7', '2.6']
16
+ ruby-version: ['3.1', '3.0', '2.7']
17
17
  os: [ubuntu-latest]
18
18
  experimental: [false]
19
19
  include:
@@ -13,11 +13,14 @@ jobs:
13
13
  strategy:
14
14
  fail-fast: false
15
15
  matrix:
16
- ruby-version: ['2.7', '2.6']
16
+ ruby-version: ['3.1', '2.7']
17
17
  os:
18
18
  - windows-latest
19
19
  experimental: [false]
20
20
  include:
21
+ - ruby-version: head
22
+ os: windows-latest
23
+ experimental: true
21
24
  - ruby-version: '3.0.3'
22
25
  os: windows-latest
23
26
  experimental: false
data/CHANGELOG.md CHANGED
@@ -1,3 +1,55 @@
1
+ # v1.15
2
+
3
+ ## Release v1.15.0 - 2022/06/29
4
+
5
+ ### Enhancement
6
+
7
+ * in_tail: Add log throttling in files based on group rules
8
+ https://github.com/fluent/fluentd/pull/3535
9
+ https://github.com/fluent/fluentd/pull/3771
10
+ * Add `dump` command to fluent-ctl
11
+ https://github.com/fluent/fluentd/pull/3680
12
+ * Handle YAML configuration format on configuration file
13
+ https://github.com/fluent/fluentd/pull/3712
14
+ * Add `restart_worker_interval` parameter in `<system>` directive to set
15
+ interval to restart workers that has stopped for some reason.
16
+ https://github.com/fluent/fluentd/pull/3768
17
+
18
+ ### Bug fixes
19
+
20
+ * out_forward: Fix to update timeout of cached sockets
21
+ https://github.com/fluent/fluentd/pull/3711
22
+ * in_tail: Fix a possible crash on file rotation when `follow_inodes true`
23
+ https://github.com/fluent/fluentd/pull/3754
24
+ * output: Fix a possible crash of flush thread
25
+ https://github.com/fluent/fluentd/pull/3755
26
+ * in_tail: Fix crash bugs on Ruby 3.1 on Windows
27
+ https://github.com/fluent/fluentd/pull/3766
28
+ * in_tail: Fix a bug that in_tail cannot open non-ascii path on Windows
29
+ https://github.com/fluent/fluentd/pull/3774
30
+ * Fix a bug that fluentd doesn't release its own log file even after rotated by
31
+ external tools
32
+ https://github.com/fluent/fluentd/pull/3782
33
+
34
+ ### Misc
35
+
36
+ * in_tail: Simplify TargetInfo related code
37
+ https://github.com/fluent/fluentd/pull/3489
38
+ * Fix a wrong issue number in CHANGELOG
39
+ https://github.com/fluent/fluentd/pull/3700
40
+ * server helper: Add comments to linger_timeout behavior about Windows
41
+ https://github.com/fluent/fluentd/pull/3701
42
+ * service_discovery: Fix typo
43
+ https://github.com/fluent/fluentd/pull/3724
44
+ * test: Fix unstable tests and warnings
45
+ https://github.com/fluent/fluentd/pull/3745
46
+ https://github.com/fluent/fluentd/pull/3753
47
+ https://github.com/fluent/fluentd/pull/3767
48
+ https://github.com/fluent/fluentd/pull/3783
49
+ https://github.com/fluent/fluentd/pull/3784
50
+ https://github.com/fluent/fluentd/pull/3785
51
+ https://github.com/fluent/fluentd/pull/3787
52
+
1
53
  # v1.14
2
54
 
3
55
  ## Release v1.14.6 - 2022/03/31
@@ -443,7 +495,7 @@ We recommend to upgrade Fluentd to v1.14.2 or use patched version of
443
495
  ### New feature
444
496
 
445
497
  * in_tail: Add `follow_inode` to support log rotation with wild card
446
- https://github.com/fluent/fluentd/pull/2992
498
+ https://github.com/fluent/fluentd/pull/3182
447
499
  * in_tail: Handle linux capability
448
500
  https://github.com/fluent/fluentd/pull/3155
449
501
  https://github.com/fluent/fluentd/pull/3162
data/fluentd.gemspec CHANGED
@@ -22,7 +22,7 @@ Gem::Specification.new do |gem|
22
22
  gem.add_runtime_dependency("msgpack", [">= 1.3.1", "< 2.0.0"])
23
23
  gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
24
24
  gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
25
- gem.add_runtime_dependency("serverengine", [">= 2.2.5", "< 3.0.0"])
25
+ gem.add_runtime_dependency("serverengine", [">= 2.3.0", "< 3.0.0"])
26
26
  gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.9.0"])
27
27
  gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
28
28
  gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
@@ -38,6 +38,7 @@ Gem::Specification.new do |gem|
38
38
  gem.add_runtime_dependency("win32-service", ["~> 2.3.0"])
39
39
  gem.add_runtime_dependency("win32-ipc", ["~> 0.7.0"])
40
40
  gem.add_runtime_dependency("win32-event", ["~> 0.6.3"])
41
+ gem.add_runtime_dependency("windows-api", ["~> 0.4.5"])
41
42
  gem.add_runtime_dependency("windows-pr", ["~> 1.2.6"])
42
43
  gem.add_runtime_dependency("certstore_c", ["~> 0.1.7"])
43
44
  end
@@ -36,6 +36,7 @@ module Fluent
36
36
  restart: "HUP",
37
37
  flush: "USR1",
38
38
  reload: "USR2",
39
+ dump: "CONT",
39
40
  }
40
41
  WINSVC_CONTROL_CODE_MAP = {
41
42
  shutdown: SERVICE_CONTROL_STOP,
@@ -44,6 +45,7 @@ module Fluent
44
45
  restart: 128,
45
46
  flush: 129,
46
47
  reload: SERVICE_CONTROL_PARAMCHANGE,
48
+ dump: 130,
47
49
  }
48
50
  else
49
51
  COMMAND_MAP = {
@@ -51,6 +53,7 @@ module Fluent
51
53
  restart: :HUP,
52
54
  flush: :USR1,
53
55
  reload: :USR2,
56
+ dump: :CONT,
54
57
  }
55
58
  end
56
59
 
@@ -146,7 +149,7 @@ module Fluent
146
149
  event = Win32::Event.open("#{prefix}#{suffix}")
147
150
  event.set
148
151
  event.close
149
- rescue Errno::ENOENT => e
152
+ rescue Errno::ENOENT
150
153
  puts "Error: Cannot find the fluentd process with the event name: \"#{prefix}\""
151
154
  end
152
155
  end
@@ -126,6 +126,16 @@ op.on('--without-source', "invoke a fluentd without input plugins", TrueClass) {
126
126
  opts[:without_source] = b
127
127
  }
128
128
 
129
+ op.on('--config-file-type VALU', 'guessing file type of fluentd configuration. yaml/yml or guess') { |s|
130
+ if (s == 'yaml') || (s == 'yml')
131
+ opts[:config_file_type] = s.to_sym
132
+ elsif (s == 'guess')
133
+ opts[:config_file_type] = s.to_sym
134
+ else
135
+ usage '--config-file-type accepts yaml/yml or guess'
136
+ end
137
+ }
138
+
129
139
  op.on('--use-v1-config', "Use v1 configuration format (default)", TrueClass) {|b|
130
140
  opts[:use_v1_config] = b
131
141
  }
@@ -191,9 +191,9 @@ worker_id = ENV['SERVERENGINE_WORKER_ID'] || ''
191
191
  EOM
192
192
  begin
193
193
  @eval_context.instance_eval(code)
194
- rescue SetNil => e
194
+ rescue SetNil
195
195
  nil
196
- rescue SetDefault => e
196
+ rescue SetDefault
197
197
  :default
198
198
  end
199
199
  end
@@ -0,0 +1,47 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ module Fluent
18
+ module Config
19
+ module YamlParser
20
+ module FluentValue
21
+ JsonValue = Struct.new(:val) do
22
+ def to_s
23
+ val.to_json
24
+ end
25
+
26
+ def to_element
27
+ to_s
28
+ end
29
+ end
30
+
31
+ StringValue = Struct.new(:val, :context) do
32
+ def to_s
33
+ context.instance_eval("\"#{val}\"")
34
+ rescue Fluent::SetNil => _
35
+ ''
36
+ rescue Fluent::SetDefault => _
37
+ ':default'
38
+ end
39
+
40
+ def to_element
41
+ to_s
42
+ end
43
+ end
44
+ end
45
+ end
46
+ end
47
+ end
@@ -0,0 +1,91 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ require 'psych'
18
+ require 'json'
19
+ require 'fluent/config/error'
20
+ require 'fluent/config/yaml_parser/fluent_value'
21
+
22
+ # Based on https://github.com/eagletmt/hako/blob/34cdde06fe8f3aeafd794be830180c3cedfbb4dc/lib/hako/yaml_loader.rb
23
+
24
+ module Fluent
25
+ module Config
26
+ module YamlParser
27
+ class Loader
28
+ INCLUDE_TAG = 'tag:include'.freeze
29
+ FLUENT_JSON_TAG = 'tag:fluent/json'.freeze
30
+ FLUENT_STR_TAG = 'tag:fluent/s'.freeze
31
+ SHOVEL = '<<'.freeze
32
+
33
+ def initialize(context = Kernel.binding)
34
+ @context = context
35
+ @current_path = nil
36
+ end
37
+
38
+ # @param [String] path
39
+ # @return [Hash]
40
+ def load(path)
41
+ class_loader = Psych::ClassLoader.new
42
+ scanner = Psych::ScalarScanner.new(class_loader)
43
+
44
+ visitor = Visitor.new(scanner, class_loader)
45
+
46
+ visitor._register_domain(INCLUDE_TAG) do |_, val|
47
+ load(path.parent.join(val))
48
+ end
49
+
50
+ visitor._register_domain(FLUENT_JSON_TAG) do |_, val|
51
+ Fluent::Config::YamlParser::FluentValue::JsonValue.new(val)
52
+ end
53
+
54
+ visitor._register_domain(FLUENT_STR_TAG) do |_, val|
55
+ Fluent::Config::YamlParser::FluentValue::StringValue.new(val, @context)
56
+ end
57
+
58
+ path.open do |f|
59
+ visitor.accept(Psych.parse(f))
60
+ end
61
+ end
62
+
63
+ class Visitor < Psych::Visitors::ToRuby
64
+ def initialize(scanner, class_loader)
65
+ super(scanner, class_loader)
66
+ end
67
+
68
+ def _register_domain(name, &block)
69
+ @domain_types.merge!({ name => [name, block] })
70
+ end
71
+
72
+ def revive_hash(hash, o)
73
+ super(hash, o).tap do |r|
74
+ if r[SHOVEL].is_a?(Hash)
75
+ h2 = {}
76
+ r.each do |k, v|
77
+ if k == SHOVEL
78
+ h2.merge!(v)
79
+ else
80
+ h2[k] = v
81
+ end
82
+ end
83
+ r.replace(h2)
84
+ end
85
+ end
86
+ end
87
+ end
88
+ end
89
+ end
90
+ end
91
+ end
@@ -0,0 +1,166 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ require 'fluent/config/yaml_parser/section_builder'
18
+
19
+ module Fluent
20
+ module Config
21
+ module YamlParser
22
+ class Parser
23
+ def initialize(config, indent: 2)
24
+ @base_indent = indent
25
+ @config = config
26
+ end
27
+
28
+ def build
29
+ s = @config['system'] && system_config_build(@config['system'])
30
+ c = @config['config'] && config_build(@config['config'], root: true)
31
+ RootBuilder.new(s, c)
32
+ end
33
+
34
+ private
35
+
36
+ def system_config_build(config)
37
+ section_build('system', config)
38
+ end
39
+
40
+ def config_build(config, indent: 0, root: false)
41
+ sb = SectionBodyBuilder.new(indent, root: root)
42
+ config.each do |c|
43
+ if (lc = c.delete('label'))
44
+ sb.add_section(label_build(lc, indent: indent))
45
+ end
46
+
47
+ if (sc = c.delete('source'))
48
+ sb.add_section(source_build(sc, indent: indent))
49
+ end
50
+
51
+ if (fc = c.delete('filter'))
52
+ sb.add_section(filter_build(fc, indent: indent))
53
+ end
54
+
55
+ if (mc = c.delete('match'))
56
+ sb.add_section(match_build(mc, indent: indent))
57
+ end
58
+
59
+ if (wc = c.delete('worker'))
60
+ sb.add_section(worker_build(wc, indent: indent))
61
+ end
62
+
63
+ included_sections_build(c, sb, indent: indent)
64
+ end
65
+
66
+ sb
67
+ end
68
+
69
+ def label_build(config, indent: 0)
70
+ config = config.dup
71
+ name = config.delete('$name')
72
+ c = config.delete('config')
73
+ SectionBuilder.new('label', config_build(c, indent: indent + @base_indent), indent, name)
74
+ end
75
+
76
+ def worker_build(config, indent: 0)
77
+ config = config.dup
78
+ num = config.delete('$arg')
79
+ c = config.delete('config')
80
+ SectionBuilder.new('worker', config_build(c, indent: indent + @base_indent), indent, num)
81
+ end
82
+
83
+ def source_build(config, indent: 0)
84
+ section_build('source', config, indent: indent)
85
+ end
86
+
87
+ def filter_build(config, indent: 0)
88
+ config = config.dup
89
+ tag = config.delete('$tag')
90
+ if tag.is_a?(Array)
91
+ section_build('filter', config, indent: indent, arg: tag&.join(','))
92
+ else
93
+ section_build('filter', config, indent: indent, arg: tag)
94
+ end
95
+ end
96
+
97
+ def match_build(config, indent: 0)
98
+ config = config.dup
99
+ tag = config.delete('$tag')
100
+ if tag.is_a?(Array)
101
+ section_build('match', config, indent: indent, arg: tag&.join(','))
102
+ else
103
+ section_build('match', config, indent: indent, arg: tag)
104
+ end
105
+ end
106
+
107
+ def included_sections_build(config, section_builder, indent: 0)
108
+ config.each_entry do |e|
109
+ k = e.keys.first
110
+ cc = e.delete(k)
111
+ case k
112
+ when 'label'
113
+ section_builder.add_section(label_build(cc, indent: indent))
114
+ when 'worker'
115
+ section_builder.add_section(worker_build(cc, indent: indent))
116
+ when 'source'
117
+ section_builder.add_section(source_build(cc, indent: indent))
118
+ when 'filter'
119
+ section_builder.add_section(filter_build(cc, indent: indent))
120
+ when 'match'
121
+ section_builder.add_section(match_build(cc, indent: indent))
122
+ end
123
+ end
124
+ end
125
+
126
+ def section_build(name, config, indent: 0, arg: nil)
127
+ sb = SectionBodyBuilder.new(indent + @base_indent)
128
+
129
+ if (v = config.delete('$type'))
130
+ sb.add_line('@type', v)
131
+ end
132
+
133
+ if (v = config.delete('$label'))
134
+ sb.add_line('@label', v)
135
+ end
136
+
137
+ if (v = config.delete('$id'))
138
+ sb.add_line('@id', v)
139
+ end
140
+
141
+ config.each do |key, val|
142
+ if val.is_a?(Array)
143
+ val.each do |v|
144
+ sb.add_section(section_build(key, v, indent: indent + @base_indent))
145
+ end
146
+ elsif val.is_a?(Hash)
147
+ harg = val.delete('$arg')
148
+ if harg.is_a?(Array)
149
+ # To prevent to generate invalid configuration for arg.
150
+ # "arg" should be String object and concatenated by ","
151
+ # when two or more objects are specified there.
152
+ sb.add_section(section_build(key, val, indent: indent + @base_indent, arg: harg&.join(',')))
153
+ else
154
+ sb.add_section(section_build(key, val, indent: indent + @base_indent, arg: harg))
155
+ end
156
+ else
157
+ sb.add_line(key, val)
158
+ end
159
+ end
160
+
161
+ SectionBuilder.new(name, sb, indent, arg)
162
+ end
163
+ end
164
+ end
165
+ end
166
+ end
@@ -0,0 +1,107 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ module Fluent
18
+ module Config
19
+ module YamlParser
20
+ SectionBuilder = Struct.new(:name, :body, :indent_size, :arg) do
21
+ def to_s
22
+ indent = ' ' * indent_size
23
+
24
+ if arg && !arg.to_s.empty?
25
+ "#{indent}<#{name} #{arg}>\n#{body}\n#{indent}</#{name}>"
26
+ else
27
+ "#{indent}<#{name}>\n#{body}\n#{indent}</#{name}>"
28
+ end
29
+ end
30
+
31
+ def to_element
32
+ elem = body.to_element
33
+ elem.name = name
34
+ elem.arg = arg.to_s if arg
35
+ elem.v1_config = true
36
+ elem
37
+ end
38
+ end
39
+
40
+ class RootBuilder
41
+ def initialize(system, conf)
42
+ @system = system
43
+ @conf = conf
44
+ end
45
+
46
+ attr_reader :system, :conf
47
+
48
+ def to_element
49
+ Fluent::Config::Element.new('ROOT', '', {}, [@system, @conf].compact.map(&:to_element).flatten)
50
+ end
51
+
52
+ def to_s
53
+ s = StringIO.new(+'')
54
+ s.puts(@system.to_s) if @system
55
+ s.puts(@conf.to_s) if @conf
56
+
57
+ s.string
58
+ end
59
+ end
60
+
61
+ class SectionBodyBuilder
62
+ Row = Struct.new(:key, :value, :indent) do
63
+ def to_s
64
+ "#{indent}#{key} #{value}"
65
+ end
66
+ end
67
+
68
+ def initialize(indent, root: false)
69
+ @indent = ' ' * indent
70
+ @bodies = []
71
+ @root = root
72
+ end
73
+
74
+ def add_line(k, v)
75
+ @bodies << Row.new(k, v, @indent)
76
+ end
77
+
78
+ def add_section(section)
79
+ @bodies << section
80
+ end
81
+
82
+ def to_element
83
+ if @root
84
+ return @bodies.map(&:to_element)
85
+ end
86
+
87
+ not_section, section = @bodies.partition { |e| e.is_a?(Row) }
88
+ r = {}
89
+ not_section.each do |e|
90
+ v = e.value
91
+ r[e.key] = v.respond_to?(:to_element) ? v.to_element : v
92
+ end
93
+
94
+ if @root
95
+ section.map(&:to_element)
96
+ else
97
+ Fluent::Config::Element.new('', '', r, section.map(&:to_element))
98
+ end
99
+ end
100
+
101
+ def to_s
102
+ @bodies.map(&:to_s).join("\n")
103
+ end
104
+ end
105
+ end
106
+ end
107
+ end
@@ -0,0 +1,56 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ require 'fluent/config/yaml_parser/loader'
18
+ require 'fluent/config/yaml_parser/parser'
19
+ require 'pathname'
20
+
21
+ module Fluent
22
+ module Config
23
+ module YamlParser
24
+ def self.parse(path)
25
+ context = Kernel.binding
26
+
27
+ unless context.respond_to?(:use_nil)
28
+ context.define_singleton_method(:use_nil) do
29
+ raise Fluent::SetNil
30
+ end
31
+ end
32
+
33
+ unless context.respond_to?(:use_default)
34
+ context.define_singleton_method(:use_default) do
35
+ raise Fluent::SetDefault
36
+ end
37
+ end
38
+
39
+ unless context.respond_to?(:hostname)
40
+ context.define_singleton_method(:hostname) do
41
+ Socket.gethostname
42
+ end
43
+ end
44
+
45
+ unless context.respond_to?(:worker_id)
46
+ context.define_singleton_method(:worker_id) do
47
+ ENV['SERVERENGINE_WORKER_ID'] || ''
48
+ end
49
+ end
50
+
51
+ s = Fluent::Config::YamlParser::Loader.new(context).load(Pathname.new(path))
52
+ Fluent::Config::YamlParser::Parser.new(s).build.to_element
53
+ end
54
+ end
55
+ end
56
+ end
data/lib/fluent/config.rb CHANGED
@@ -17,6 +17,7 @@
17
17
  require 'fluent/config/error'
18
18
  require 'fluent/config/element'
19
19
  require 'fluent/configurable'
20
+ require 'fluent/config/yaml_parser'
20
21
 
21
22
  module Fluent
22
23
  module Config
@@ -25,7 +26,18 @@ module Fluent
25
26
  # @param additional_config [String] config which is added to last of config body
26
27
  # @param use_v1_config [Bool] config is formatted with v1 or not
27
28
  # @return [Fluent::Config]
28
- def self.build(config_path:, encoding: 'utf-8', additional_config: nil, use_v1_config: true)
29
+ def self.build(config_path:, encoding: 'utf-8', additional_config: nil, use_v1_config: true, type: nil)
30
+ if type == :guess
31
+ config_file_ext = File.extname(config_path)
32
+ if config_file_ext == '.yaml' || config_file_ext == '.yml'
33
+ type = :yaml
34
+ end
35
+ end
36
+
37
+ if type == :yaml || type == :yml
38
+ return Fluent::Config::YamlParser.parse(config_path)
39
+ end
40
+
29
41
  config_fname = File.basename(config_path)
30
42
  config_basedir = File.dirname(config_path)
31
43
  config_data = File.open(config_path, "r:#{encoding}:utf-8") do |f|
@@ -36,6 +48,7 @@ module Fluent
36
48
  end
37
49
  s
38
50
  end
51
+
39
52
  Fluent::Config.parse(config_data, config_fname, config_basedir, use_v1_config)
40
53
  end
41
54