fluentd 1.3.3 → 1.4.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

Files changed (46) hide show
  1. checksums.yaml +4 -4
  2. data/.github/ISSUE_TEMPLATE.md +3 -1
  3. data/.travis.yml +2 -2
  4. data/CHANGELOG.md +32 -1
  5. data/CONTRIBUTING.md +2 -2
  6. data/MAINTAINERS.md +1 -1
  7. data/README.md +1 -1
  8. data/appveyor.yml +0 -5
  9. data/lib/fluent/config/element.rb +13 -6
  10. data/lib/fluent/config/v1_parser.rb +1 -1
  11. data/lib/fluent/plugin/base.rb +6 -1
  12. data/lib/fluent/plugin/in_http.rb +1 -0
  13. data/lib/fluent/plugin/in_tail.rb +9 -3
  14. data/lib/fluent/plugin/in_tcp.rb +5 -1
  15. data/lib/fluent/plugin/in_udp.rb +5 -1
  16. data/lib/fluent/plugin/out_file.rb +6 -1
  17. data/lib/fluent/plugin/output.rb +5 -3
  18. data/lib/fluent/plugin_helper/retry_state.rb +2 -6
  19. data/lib/fluent/plugin_helper/server.rb +3 -3
  20. data/lib/fluent/root_agent.rb +49 -11
  21. data/lib/fluent/version.rb +1 -1
  22. data/test/command/test_fluentd.rb +2 -2
  23. data/test/command/test_plugin_config_formatter.rb +3 -3
  24. data/test/config/test_config_parser.rb +30 -0
  25. data/test/config/test_configurable.rb +1 -1
  26. data/test/config/test_element.rb +30 -6
  27. data/test/counter/test_store.rb +1 -1
  28. data/test/plugin/test_buffer.rb +2 -2
  29. data/test/plugin/test_filter.rb +2 -2
  30. data/test/plugin/test_filter_record_transformer.rb +1 -1
  31. data/test/plugin/test_in_tail.rb +20 -0
  32. data/test/plugin/test_in_tcp.rb +6 -0
  33. data/test/plugin/test_in_udp.rb +6 -0
  34. data/test/plugin/test_out_file.rb +36 -11
  35. data/test/plugin/test_out_secondary_file.rb +1 -1
  36. data/test/plugin/test_output.rb +1 -1
  37. data/test/plugin/test_output_as_buffered.rb +38 -0
  38. data/test/plugin/test_output_as_buffered_backup.rb +7 -2
  39. data/test/plugin/test_output_as_buffered_retries.rb +8 -4
  40. data/test/plugin/test_parser_syslog.rb +2 -2
  41. data/test/plugin_helper/test_record_accessor.rb +1 -1
  42. data/test/plugin_helper/test_server.rb +1 -1
  43. data/test/test_output.rb +1 -1
  44. data/test/test_plugin.rb +1 -1
  45. data/test/test_root_agent.rb +76 -0
  46. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 4e5a7a4c71fe45e0a65fd941bea4093e8a6c5414
4
- data.tar.gz: 6e4000e2928c6bfda990ea5566e4e32618e031f7
3
+ metadata.gz: ca1f6da9527b76b4421c1507ec0e2702687e196d
4
+ data.tar.gz: 8ae3c1fbac074dbc699bb36cb0975c78e94e3769
5
5
  SHA512:
6
- metadata.gz: 94194178c70c31f9ae8f20f4c85230f1479f53dfbd25e5e8a13e724a5cf8ca7b38a7658a97153838ff486d554d0f44369bf2f622838af324c585594a3748d76d
7
- data.tar.gz: 836807eb180ed4144f8333cb67c11d640b432882497e63d24d760a7a050f4513e137b6ac97bc6503d781e5690e80e26dee25792e6081b99f857f40787256e029
6
+ metadata.gz: 4a2298e9e76277c4ce41c7b43e239643a0b3ad8a953ab47bc242a9e22645e0d517853f73f8e37699ad5bbc57dc36e9d5cfb0504da155fb575c36eebf1893cfb7
7
+ data.tar.gz: 783fb04e180adc4711a6ebf83b91d6fa2d318300ca31ba0c49c8d247c2eceed4992b22be98765f7b30e316cd7815cf9aac7e81873b221c480f4df80f51a3fccb
@@ -1,6 +1,8 @@
1
1
  Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
2
2
 
3
3
  - fluentd or td-agent version.
4
- - Environment information, e.g. OS.
4
+ - Environment information:
5
+ - Operating system: `cat /etc/os-release`
6
+ - Kernel version: `uname -r`
5
7
  - Your configuration
6
8
  - Your problem explanation. If you have an error logs, write it together.
@@ -11,8 +11,6 @@ matrix:
11
11
  os: linux
12
12
  - rvm: 2.2.10
13
13
  os: linux
14
- - rvm: 2.3.7
15
- os: linux
16
14
  - rvm: 2.4.5
17
15
  os: linux
18
16
  - rvm: 2.5.3
@@ -37,6 +35,8 @@ matrix:
37
35
  os: osx
38
36
  osx_image: xcode8.3 # OSX 10.12
39
37
  allow_failures:
38
+ - rvm: 2.3.8
39
+ os: linux
40
40
  - rvm: 2.1.10
41
41
  os: osx
42
42
  osx_image: xcode8.3
@@ -1,6 +1,37 @@
1
1
  # v1.3
2
2
 
3
- ## Release v1.3.3 - 2018/01/06
3
+ ## Release v1.4.0 - 2019/02/24
4
+
5
+ ### New features
6
+
7
+ * multiprocess: Support <worker N-M> syntax
8
+ https://github.com/fluent/fluentd/pull/2292
9
+ * output: Work <secondary> and retry_forever together
10
+ https://github.com/fluent/fluentd/pull/2276
11
+ * out_file: Support placeholders in symlink_path
12
+ https://github.com/fluent/fluentd/pull/2254
13
+
14
+ ### Enhancements
15
+
16
+ * output: Add MessagePack unpacker error to unrecoverable error list
17
+ https://github.com/fluent/fluentd/pull/2301
18
+ * output: Reduce flush deley when large timekey and small timekey_wait are specified
19
+ https://github.com/fluent/fluentd/pull/2291
20
+ * config: Support embedded ruby code in section argument.
21
+ https://github.com/fluent/fluentd/pull/2291
22
+ * in_tail: Improve encoding parameter handling
23
+ https://github.com/fluent/fluentd/pull/2305
24
+ * in_tcp/in_udp: Add <parse> section check
25
+ https://github.com/fluent/fluentd/pull/2267
26
+
27
+ ### Bug fixes
28
+
29
+ * server: Ignore IOError and related errors in UDP
30
+ https://github.com/fluent/fluentd/pull/2310
31
+ * server: Ignore EPIPE in TLS accept to avoid fluentd restart
32
+ https://github.com/fluent/fluentd/pull/2253
33
+
34
+ ## Release v1.3.3 - 2019/01/06
4
35
 
5
36
  ### Enhancements
6
37
 
@@ -27,7 +27,7 @@ submitting an issue to Fluentd. Even better you can submit a Pull Request with a
27
27
  * **Documentation**: Use [fluentd-docs](https://github.com/fluent/fluentd-docs) repository.
28
28
 
29
29
  If you find a bug of 3rd party plugins, please submit an issue to each plugin repository.
30
- And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent releated issues.
30
+ And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent related issues.
31
31
 
32
32
  Note: Before report the issue, check latest version first. Sometimes users report fixed bug with older version.
33
33
 
@@ -37,7 +37,7 @@ Here are some things that would increase a chance that your patch is accepted:
37
37
 
38
38
  * Write tests.
39
39
  * Run tests before send Pull Request by `bundle exec rake test`
40
- * Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
40
+ * Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
41
41
  * Fluentd repositories needs [DCO](https://github.com/apps/dco) on PR. Please add `Signed-off-by` to the commit(See DCO link for more detail).
42
42
 
43
43
  There are some patches which are hard to write tests, e.g. process handling, concurrency issue or etc.
@@ -1,6 +1,6 @@
1
1
  # Fluentd Maintainers
2
2
 
3
- - [Naotoshi Seo](https://github.com/sonots), [DeNA](http://dena.com/intl/)
3
+ - [Naotoshi Seo](https://github.com/sonots), [ZOZO Technologies](https://tech.zozo.com/en/)
4
4
  - [Okkez](https://github.com/okkez), [Clearcode](https://www.clear-code.com/)
5
5
  - [Hiroshi Hatake](https://github.com/cosmo0920), [Clearcode](https://www.clear-code.com/)
6
6
  - [Masahiro Nakagawa](https://github.com/repeatedly), [Treasure Data](https://www.treasuredata.com/)
data/README.md CHANGED
@@ -72,7 +72,7 @@ Many enterprises run Fluentd in production to handle all of their logging needs.
72
72
  - Discussion: https://groups.google.com/group/fluentd
73
73
  - Slack / Community: https://slack.fluentd.org
74
74
  - Newsletters: https://www.fluentd.org/newsletter_signup
75
- - Author: Sadayuki Furuhashi
75
+ - Author: [Sadayuki Furuhashi](https://github.com/frsyuki)
76
76
  - Copyright: 2011-2018 Fluentd Authors
77
77
  - License: Apache License, Version 2.0
78
78
 
@@ -30,14 +30,9 @@ environment:
30
30
  devkit: C:\Ruby23\DevKit
31
31
  - ruby_version: "22-x64"
32
32
  devkit: C:\Ruby23-x64\DevKit
33
- - ruby_version: "21-x64"
34
- devkit: C:\Ruby23-x64\DevKit
35
33
  - ruby_version: "22"
36
34
  devkit: C:\Ruby23\DevKit
37
35
  WIN_RAPID: true
38
- - ruby_version: "21"
39
- devkit: C:\Ruby23\DevKit
40
- WIN_RAPID: true
41
36
  matrix:
42
37
  allow_failures:
43
38
  - ruby_version: "21"
@@ -36,12 +36,12 @@ module Fluent
36
36
  # it's global logger, not plugin logger: deprecated message should be global warning, not plugin level.
37
37
  @logger = defined?($log) ? $log : nil
38
38
 
39
- @target_worker_id = nil
39
+ @target_worker_ids = []
40
40
  end
41
41
 
42
42
  attr_accessor :name, :arg, :unused, :v1_config, :corresponding_proxies, :unused_in
43
43
  attr_writer :elements
44
- attr_reader :target_worker_id
44
+ attr_reader :target_worker_ids
45
45
 
46
46
  RESERVED_PARAMETERS_COMPAT = {
47
47
  '@type' => 'type',
@@ -223,22 +223,29 @@ module Fluent
223
223
  end
224
224
 
225
225
  def set_target_worker_id(worker_id)
226
- @target_worker_id = worker_id
226
+ @target_worker_ids = [worker_id]
227
227
  @elements.each { |e|
228
228
  e.set_target_worker_id(worker_id)
229
229
  }
230
230
  end
231
231
 
232
+ def set_target_worker_ids(worker_ids)
233
+ @target_worker_ids = worker_ids.uniq
234
+ @elements.each { |e|
235
+ e.set_target_worker_ids(worker_ids.uniq)
236
+ }
237
+ end
238
+
232
239
  def for_every_workers?
233
- @target_worker_id == nil
240
+ @target_worker_ids.empty?
234
241
  end
235
242
 
236
243
  def for_this_worker?
237
- @target_worker_id == Fluent::Engine.worker_id
244
+ @target_worker_ids.include?(Fluent::Engine.worker_id)
238
245
  end
239
246
 
240
247
  def for_another_worker?
241
- @target_worker_id != nil && @target_worker_id != Fluent::Engine.worker_id
248
+ !@target_worker_ids.empty? && !@target_worker_ids.include?(Fluent::Engine.worker_id)
242
249
  end
243
250
  end
244
251
  end
@@ -82,7 +82,7 @@ module Fluent
82
82
  elsif skip(/\</)
83
83
  e_name = scan(ELEMENT_NAME)
84
84
  spacing
85
- e_arg = scan_nonquoted_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/)
85
+ e_arg = scan_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/)
86
86
  spacing
87
87
  unless skip(/\>/)
88
88
  parse_error! "expected '>'"
@@ -52,7 +52,12 @@ module Fluent
52
52
 
53
53
  def configure(conf)
54
54
  if conf.respond_to?(:for_this_worker?) && conf.for_this_worker?
55
- system_config_override(workers: 1)
55
+ workers = if conf.target_worker_ids && !conf.target_worker_ids.empty?
56
+ conf.target_worker_ids.size
57
+ else
58
+ 1
59
+ end
60
+ system_config_override(workers: workers)
56
61
  end
57
62
  super
58
63
  @_state ||= State.new(false, false, false, false, false, false, false, false, false)
@@ -163,6 +163,7 @@ module Fluent::Plugin
163
163
 
164
164
  # Skip nil record
165
165
  if record.nil?
166
+ log.debug { "incoming event is invalid: path=#{path_info} params=#{params.to_json}" }
166
167
  if @respond_with_empty_img
167
168
  return ["200 OK", {'Content-Type'=>'image/gif; charset=utf-8'}, EMPTY_GIF_IMAGE]
168
169
  else
@@ -171,6 +171,9 @@ module Fluent::Plugin
171
171
 
172
172
  @encoding = parse_encoding_param(@encoding) if @encoding
173
173
  @from_encoding = parse_encoding_param(@from_encoding) if @from_encoding
174
+ if @encoding == @from_encoding
175
+ log.warn "'encoding' and 'from_encoding' are same encoding. No effect"
176
+ end
174
177
  end
175
178
 
176
179
  def parse_encoding_param(encoding_name)
@@ -657,6 +660,7 @@ module Fluent::Plugin
657
660
  def initialize(from_encoding, encoding)
658
661
  @from_encoding = from_encoding
659
662
  @encoding = encoding
663
+ @need_enc = from_encoding != encoding
660
664
  @buffer = ''.force_encoding(from_encoding)
661
665
  @eol = "\n".encode(from_encoding).freeze
662
666
  end
@@ -682,11 +686,13 @@ module Fluent::Plugin
682
686
  end
683
687
 
684
688
  def convert(s)
685
- if @from_encoding == @encoding
686
- s
689
+ if @need_enc
690
+ s.encode!(@encoding, @from_encoding)
687
691
  else
688
- s.encode(@encoding, @from_encoding)
692
+ s
689
693
  end
694
+ rescue
695
+ s.encode!(@encoding, @from_encoding, :invalid => :replace, :undef => :replace)
690
696
  end
691
697
 
692
698
  def next_line
@@ -41,11 +41,15 @@ module Fluent::Plugin
41
41
 
42
42
  def configure(conf)
43
43
  compat_parameters_convert(conf, :parser)
44
+ parser_config = conf.elements('parse').first
45
+ unless parser_config
46
+ raise Fluent::ConfigError, "<parse> section is required."
47
+ end
44
48
  super
45
49
  @_event_loop_blocking_timeout = @blocking_timeout
46
50
  @source_hostname_key ||= @source_host_key if @source_host_key
47
51
 
48
- @parser = parser_create
52
+ @parser = parser_create(conf: parser_config)
49
53
  end
50
54
 
51
55
  def multi_workers_ready?
@@ -47,12 +47,16 @@ module Fluent::Plugin
47
47
 
48
48
  def configure(conf)
49
49
  compat_parameters_convert(conf, :parser)
50
+ parser_config = conf.elements('parse').first
51
+ unless parser_config
52
+ raise Fluent::ConfigError, "<parse> section is required."
53
+ end
50
54
  super
51
55
  @_event_loop_blocking_timeout = @blocking_timeout
52
56
  @source_hostname_key ||= @source_host_key if @source_host_key
53
57
  @message_length_limit = @body_size_limit if @body_size_limit
54
58
 
55
- @parser = parser_create
59
+ @parser = parser_create(conf: parser_config)
56
60
  end
57
61
 
58
62
  def multi_workers_ready?
@@ -71,6 +71,10 @@ module Fluent::Plugin
71
71
  attr_accessor :last_written_path # for tests
72
72
 
73
73
  module SymlinkBufferMixin
74
+ def output_plugin_for_symlink=(output_plugin)
75
+ @_output_plugin_for_symlink = output_plugin
76
+ end
77
+
74
78
  def symlink_path=(path)
75
79
  @_symlink_path = path
76
80
  end
@@ -83,7 +87,7 @@ module Fluent::Plugin
83
87
  # These chunks will be enqueued immediately, and will be flushed soon.
84
88
  latest_metadata = metadata_list.select{|m| m.timekey }.sort_by(&:timekey).last
85
89
  if chunk.metadata == latest_metadata
86
- FileUtils.ln_sf(chunk.path, @_symlink_path)
90
+ FileUtils.ln_sf(chunk.path, @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk))
87
91
  end
88
92
  chunk
89
93
  end
@@ -161,6 +165,7 @@ module Fluent::Plugin
161
165
  else
162
166
  @buffer.extend SymlinkBufferMixin
163
167
  @buffer.symlink_path = @symlink_path
168
+ @buffer.output_plugin_for_symlink = self
164
169
  end
165
170
  end
166
171
 
@@ -364,7 +364,9 @@ module Fluent
364
364
  raise Fluent::ConfigError, "Invalid <secondary> section for non-buffered plugin" unless @buffering
365
365
  raise Fluent::ConfigError, "<secondary> section cannot have <buffer> section" if @secondary_config.buffer
366
366
  raise Fluent::ConfigError, "<secondary> section cannot have <secondary> section" if @secondary_config.secondary
367
- raise Fluent::ConfigError, "<secondary> section and 'retry_forever' are exclusive" if @buffer_config.retry_forever
367
+ if @buffer_config.retry_forever
368
+ log.warn "<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary"
369
+ end
368
370
 
369
371
  secondary_type = @secondary_config[:@type]
370
372
  unless secondary_type
@@ -1081,7 +1083,7 @@ module Fluent
1081
1083
  end
1082
1084
  end
1083
1085
 
1084
- UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError]
1086
+ UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError, MessagePack::UnpackError]
1085
1087
 
1086
1088
  def try_flush
1087
1089
  chunk = @buffer.dequeue_chunk
@@ -1334,7 +1336,7 @@ module Fluent
1334
1336
  end
1335
1337
  if @chunk_key_time
1336
1338
  if !value_for_interval || @buffer_config.timekey < value_for_interval
1337
- value_for_interval = @buffer_config.timekey
1339
+ value_for_interval = [@buffer_config.timekey, @buffer_config.timekey_wait].min
1338
1340
  end
1339
1341
  end
1340
1342
  unless value_for_interval
@@ -53,10 +53,6 @@ module Fluent
53
53
  @randomize = randomize
54
54
  @randomize_width = randomize_width
55
55
 
56
- if forever && secondary
57
- raise "BUG: forever and secondary are exclusive to each other"
58
- end
59
-
60
56
  @forever = forever
61
57
  @max_steps = max_steps
62
58
 
@@ -118,12 +114,12 @@ module Fluent
118
114
  end
119
115
 
120
116
  def secondary?
121
- @secondary && (@current == :secondary || current_time >= @secondary_transition_at)
117
+ !@forever && @secondary && (@current == :secondary || current_time >= @secondary_transition_at)
122
118
  end
123
119
 
124
120
  def step
125
121
  @steps += 1
126
- if @secondary && @current != :secondary && current_time >= @secondary_transition_at
122
+ if !@forever && @secondary && @current != :secondary && current_time >= @secondary_transition_at
127
123
  @current = :secondary
128
124
  @secondary_transition_steps = @steps
129
125
  end
@@ -523,7 +523,7 @@ module Fluent
523
523
  def on_readable_without_sock
524
524
  begin
525
525
  data = @sock.recv(@max_bytes, @flags)
526
- rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET
526
+ rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF
527
527
  return
528
528
  end
529
529
  @callback.call(data)
@@ -536,7 +536,7 @@ module Fluent
536
536
  def on_readable_with_sock
537
537
  begin
538
538
  data, addr = @sock.recvfrom(@max_bytes)
539
- rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET
539
+ rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF
540
540
  return
541
541
  end
542
542
  @callback.call(data, UDPCallbackSocket.new(@sock, addr, close_socket: @close_socket))
@@ -722,7 +722,7 @@ module Fluent
722
722
 
723
723
  return true
724
724
  end
725
- rescue Errno::ECONNRESET, OpenSSL::SSL::SSLError => e
725
+ rescue Errno::EPIPE, Errno::ECONNRESET, OpenSSL::SSL::SSLError => e
726
726
  @log.trace "unexpected error before accepting TLS connection", error: e
727
727
  close rescue nil
728
728
  end
@@ -64,26 +64,64 @@ module Fluent
64
64
  attr_reader :labels
65
65
 
66
66
  def configure(conf)
67
+ used_worker_ids = []
68
+ available_worker_ids = (0..Fluent::Engine.system_config.workers - 1).to_a
67
69
  # initialize <worker> elements
68
70
  conf.elements(name: 'worker').each do |e|
69
71
  target_worker_id_str = e.arg
70
72
  if target_worker_id_str.empty?
71
- raise ConfigError, "Missing worker id on <worker> directive"
73
+ raise Fluent::ConfigError, "Missing worker id on <worker> directive"
72
74
  end
73
75
 
74
- target_worker_id = target_worker_id_str.to_i
75
- if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
76
- raise ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
77
- end
76
+ target_worker_ids = target_worker_id_str.split("-")
77
+ if target_worker_ids.size == 2
78
+ first_worker_id = target_worker_ids.first.to_i
79
+ last_worker_id = target_worker_ids.last.to_i
80
+ if first_worker_id > last_worker_id
81
+ raise Fluent::ConfigError, "greater first_worker_id<#{first_worker_id}> than last_worker_id<#{last_worker_id}> specified by <worker> directive is not allowed. Available multi worker assign syntax is <smaller_worker_id>-<greater_worker_id>"
82
+ end
83
+ target_worker_ids = []
84
+ first_worker_id.step(last_worker_id, 1) do |worker_id|
85
+ target_worker_id = worker_id.to_i
86
+ target_worker_ids << target_worker_id
87
+
88
+ if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
89
+ raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
90
+ end
91
+ available_worker_ids.delete(target_worker_id) if available_worker_ids.include?(target_worker_id)
92
+ if used_worker_ids.include?(target_worker_id) && !Fluent::Engine.dry_run_mode
93
+ raise Fluent::ConfigError, "specified worker_id<#{worker_id}> collisions is detected on <worker> directive. Available worker id(s): #{available_worker_ids}"
94
+ end
95
+ used_worker_ids << target_worker_id
96
+
97
+ e.elements.each do |elem|
98
+ unless ['source', 'match', 'filter', 'label'].include?(elem.name)
99
+ raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive"
100
+ end
101
+ end
78
102
 
79
- ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
80
- target_worker_id = 0 if Fluent::Engine.dry_run_mode
103
+ # On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
104
+ target_worker_ids = [0] if Fluent::Engine.dry_run_mode
81
105
 
82
- e.elements.each do |elem|
83
- unless ['source', 'match', 'filter', 'label'].include?(elem.name)
84
- raise ConfigError, "<worker> section cannot have <#{elem.name}> directive"
106
+ unless target_worker_ids.empty?
107
+ e.set_target_worker_ids(target_worker_ids.uniq)
108
+ end
109
+ end
110
+ else
111
+ target_worker_id = target_worker_id_str.to_i
112
+ if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1)
113
+ raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}"
114
+ end
115
+
116
+ ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0).
117
+ target_worker_id = 0 if Fluent::Engine.dry_run_mode
118
+
119
+ e.elements.each do |elem|
120
+ unless ['source', 'match', 'filter', 'label'].include?(elem.name)
121
+ raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive"
122
+ end
123
+ elem.set_target_worker_id(target_worker_id)
85
124
  end
86
- elem.set_target_worker_id(target_worker_id)
87
125
  end
88
126
  conf += e
89
127
  end
@@ -16,6 +16,6 @@
16
16
 
17
17
  module Fluent
18
18
 
19
- VERSION = '1.3.3'
19
+ VERSION = '1.4.0'
20
20
 
21
21
  end
@@ -340,7 +340,7 @@ CONF
340
340
  end
341
341
  end
342
342
 
343
- sub_test_case 'configured to suppress configration dump' do
343
+ sub_test_case 'configured to suppress configuration dump' do
344
344
  setup do
345
345
  @basic_conf = <<CONF
346
346
  <source>
@@ -614,7 +614,7 @@ CONF
614
614
  )
615
615
  end
616
616
 
617
- test 'failed to start workers when configured plugins as chidren of MultiOutput do not support multi worker configuration' do
617
+ test 'failed to start workers when configured plugins as children of MultiOutput do not support multi worker configuration' do
618
618
  script = <<-EOC
619
619
  require 'fluent/plugin/output'
620
620
  module Fluent::Plugin
@@ -72,7 +72,7 @@ class TestFluentPluginConfigFormatter < Test::Unit::TestCase
72
72
  desc "username"
73
73
  config_param :username, :string
74
74
  desc "password"
75
- config_param :passowrd, :string, secret: true
75
+ config_param :password, :string, secret: true
76
76
  end
77
77
 
78
78
  config_section :parent do
@@ -162,7 +162,7 @@ slow_flush_log_threshold: float: (20.0)
162
162
  <secondary>: optional, single
163
163
  <authentication>: required, single
164
164
  username: string: (nil)
165
- passowrd: string: (nil)
165
+ password: string: (nil)
166
166
  <parent>: optional, multiple
167
167
  <child>: optional, multiple
168
168
  names: array: (nil)
@@ -217,7 +217,7 @@ TEXT
217
217
 
218
218
  username
219
219
 
220
- #### passowrd (string) (required)
220
+ #### password (string) (required)
221
221
 
222
222
  password
223
223
 
@@ -329,6 +329,36 @@ module Fluent::Config
329
329
  end
330
330
  end
331
331
 
332
+ sub_test_case "Embedded Ruby Code in section attributes" do
333
+ setup do
334
+ ENV["EMBEDDED_VAR"] = "embedded"
335
+ ENV["NESTED_EMBEDDED_VAR"] = "nested-embedded"
336
+ @hostname = Socket.gethostname
337
+ end
338
+
339
+ teardown do
340
+ ENV["EMBEDDED_VAR"] = nil
341
+ ENV["NESTED_EMBEDDED_VAR"] = nil
342
+ end
343
+
344
+ test "embedded Ruby code should be expanded" do
345
+ assert_text_parsed_as(root(
346
+ e("test", 'embedded', {'key'=>'1'}, [
347
+ e('nested1', 'nested-embedded'),
348
+ e('nested2', "#{@hostname}")
349
+ ])), <<-EOF
350
+ <test "#{ENV["EMBEDDED_VAR"]}">
351
+ key 1
352
+ <nested1 "#{ENV["NESTED_EMBEDDED_VAR"]}">
353
+ </nested1>
354
+ <nested2 "#{Socket.gethostname}">
355
+ </nested2>
356
+ </test>
357
+ EOF
358
+ )
359
+ end
360
+ end
361
+
332
362
  # port from test_config.rb
333
363
  sub_test_case '@include parsing' do
334
364
  TMP_DIR = File.dirname(__FILE__) + "/tmp/v1_config#{ENV['TEST_ENV_NUMBER']}"
@@ -733,7 +733,7 @@ module Fluent::Config
733
733
  assert_nothing_raised { b4.configure(config_element('ROOT', '', BASE_ATTRS, [d1.dup, d2.dup, d3.dup, d3.dup])) }
734
734
  end
735
735
 
736
- test 'constructs confuguration object tree for Base3' do
736
+ test 'constructs configuration object tree for Base3' do
737
737
  conf = config_element(
738
738
  'ROOT',
739
739
  '',
@@ -157,7 +157,7 @@ class TestConfigElement < ::Test::Unit::TestCase
157
157
  Fluent::Config::Element.new('ROOT', 'mydata', {}, [])],
158
158
  "differ keys" => [Fluent::Config::Element.new('ROOT', 'mydata', {}, []),
159
159
  Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [])],
160
- "differ elemnts" =>
160
+ "differ elements" =>
161
161
  [Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, []),
162
162
  Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [
163
163
  Fluent::Config::Element.new('test', 'mydata', {'k3' => 'v3'}, [])
@@ -406,11 +406,11 @@ CONF
406
406
  test 'set target_worker_id recursively' do
407
407
  e = element('label', '@mytest', {}, [ element('filter', '**'), element('match', '**', {}, [ element('store'), element('store') ]) ])
408
408
  e.set_target_worker_id(1)
409
- assert_equal 1, e.target_worker_id
410
- assert_equal 1, e.elements[0].target_worker_id
411
- assert_equal 1, e.elements[1].target_worker_id
412
- assert_equal 1, e.elements[1].elements[0].target_worker_id
413
- assert_equal 1, e.elements[1].elements[1].target_worker_id
409
+ assert_equal [1], e.target_worker_ids
410
+ assert_equal [1], e.elements[0].target_worker_ids
411
+ assert_equal [1], e.elements[1].target_worker_ids
412
+ assert_equal [1], e.elements[1].elements[0].target_worker_ids
413
+ assert_equal [1], e.elements[1].elements[1].target_worker_ids
414
414
  end
415
415
  end
416
416
 
@@ -434,12 +434,24 @@ CONF
434
434
  assert e.for_this_worker?
435
435
  end
436
436
 
437
+ test 'target_worker_ids includes current worker_id' do
438
+ e = element()
439
+ e.set_target_worker_ids([0])
440
+ assert e.for_this_worker?
441
+ end
442
+
437
443
  test 'target_worker_id != current worker_id' do
438
444
  e = element()
439
445
  e.set_target_worker_id(1)
440
446
  assert_false e.for_this_worker?
441
447
  end
442
448
 
449
+ test 'target_worker_ids does not includes current worker_id' do
450
+ e = element()
451
+ e.set_target_worker_ids([1, 2])
452
+ assert_false e.for_this_worker?
453
+ end
454
+
443
455
  test "doesn't have target_worker_id" do
444
456
  e = element()
445
457
  assert_false e.for_this_worker?
@@ -453,12 +465,24 @@ CONF
453
465
  assert_false e.for_another_worker?
454
466
  end
455
467
 
468
+ test 'target_worker_ids contains current worker_id' do
469
+ e = element()
470
+ e.set_target_worker_ids([0, 1])
471
+ assert_false e.for_another_worker?
472
+ end
473
+
456
474
  test 'target_worker_id != current worker_id' do
457
475
  e = element()
458
476
  e.set_target_worker_id(1)
459
477
  assert e.for_another_worker?
460
478
  end
461
479
 
480
+ test 'target_worker_ids does not contains current worker_id' do
481
+ e = element()
482
+ e.set_target_worker_ids([1, 2])
483
+ assert e.for_another_worker?
484
+ end
485
+
462
486
  test "doesn't have target_worker_id" do
463
487
  e = element()
464
488
  assert_false e.for_another_worker?
@@ -89,7 +89,7 @@ class CounterStoreTest < ::Test::Unit::TestCase
89
89
  assert_equal nil, @store.get('unknown_key')
90
90
  end
91
91
 
92
- test "raise a error when when a passed key doesn't exist and raise_error option is true" do
92
+ test "raise a error when a passed key doesn't exist and raise_error option is true" do
93
93
  assert_raise Fluent::Counter::UnknownKey do
94
94
  @store.get('unknown_key', raise_error: true)
95
95
  end
@@ -211,7 +211,7 @@ class BufferTest < Test::Unit::TestCase
211
211
  assert_equal 2, @p.queued_num[@dm1]
212
212
  end
213
213
 
214
- test '#close closes all chunks in in dequeued, enqueued and staged' do
214
+ test '#close closes all chunks in dequeued, enqueued and staged' do
215
215
  dmx = create_metadata(Time.parse('2016-04-11 15:50:00 +0000').to_i, nil, nil)
216
216
  cx = create_chunk(dmx, ["x" * 1024])
217
217
  @p.dequeued[cx.unique_id] = cx
@@ -1027,7 +1027,7 @@ class BufferTest < Test::Unit::TestCase
1027
1027
  ##### 900 + 9500 + 9900 * 4 == 5000 + 45000
1028
1028
  end
1029
1029
 
1030
- test '#write raises BufferChunkOverflowError if a record is biggar than chunk limit size' do
1030
+ test '#write raises BufferChunkOverflowError if a record is bigger than chunk limit size' do
1031
1031
  assert_equal [@dm0], @p.stage.keys
1032
1032
  assert_equal [], @p.queue.map(&:metadata)
1033
1033
 
@@ -47,7 +47,7 @@ module FluentPluginFilterTest
47
47
  end
48
48
  end
49
49
  class InvalidPlugin < Fluent::Plugin::Filter
50
- # Because of implemnting `filter_with_time` and `filter` methods
50
+ # Because of implementing `filter_with_time` and `filter` methods
51
51
  def filter_with_time(tag, time, record); end
52
52
  def filter(tag, time, record); end
53
53
  end
@@ -277,7 +277,7 @@ class FilterPluginTest < Test::Unit::TestCase
277
277
  end
278
278
  end
279
279
 
280
- sub_test_case 'filter plugins that is implmented `filter_with_time`' do
280
+ sub_test_case 'filter plugins that is implemented `filter_with_time`' do
281
281
  setup do
282
282
  Fluent::Test.setup
283
283
  @p = FluentPluginFilterTest::NumDoublePluginWithTime.new
@@ -229,7 +229,7 @@ class RecordTransformerFilterTest < Test::Unit::TestCase
229
229
  end
230
230
 
231
231
  %w[yes no].each do |enable_ruby|
232
- test "hostname with enble_ruby #{enable_ruby}" do
232
+ test "hostname with enable_ruby #{enable_ruby}" do
233
233
  config = %[
234
234
  enable_ruby #{enable_ruby}
235
235
  <record>
@@ -649,6 +649,26 @@ class TailInputTest < Test::Unit::TestCase
649
649
  assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding)
650
650
  end
651
651
 
652
+ def test_encoding_with_bad_character
653
+ conf = config_element(
654
+ "", "", {
655
+ "format" => "/(?<message>.*)/",
656
+ "read_from_head" => "true",
657
+ "from_encoding" => "ASCII-8BIT",
658
+ "encoding" => "utf-8"
659
+ })
660
+ d = create_driver(conf)
661
+
662
+ d.run(expect_emits: 1) do
663
+ File.open("#{TMP_DIR}/tail.txt", "w") { |f|
664
+ f.write "te\x86st\n"
665
+ }
666
+ end
667
+
668
+ events = d.events
669
+ assert_equal("te\uFFFDst", events[0][2]['message'])
670
+ assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding)
671
+ end
652
672
 
653
673
  sub_test_case "multiline" do
654
674
  data(flat: MULTILINE_CONFIG,
@@ -48,6 +48,12 @@ class TcpInputTest < Test::Unit::TestCase
48
48
  assert_equal "\n", d.instance.delimiter
49
49
  end
50
50
 
51
+ test ' configure w/o parse section' do
52
+ assert_raise(Fluent::ConfigError.new("<parse> section is required.")) {
53
+ create_driver(BASE_CONFIG)
54
+ }
55
+ end
56
+
51
57
  test_case_data = {
52
58
  'none' => {
53
59
  'format' => 'none',
@@ -59,6 +59,12 @@ class UdpInputTest < Test::Unit::TestCase
59
59
  assert_equal nil, d.instance.receive_buffer_size
60
60
  end
61
61
 
62
+ test ' configure w/o parse section' do
63
+ assert_raise(Fluent::ConfigError.new("<parse> section is required.")) {
64
+ create_driver(BASE_CONFIG)
65
+ }
66
+ end
67
+
62
68
  data(
63
69
  'ipv4' => [CONFIG, '127.0.0.1', :ipv4],
64
70
  'ipv6' => [IPv6_CONFIG, '::1', :ipv6],
@@ -14,7 +14,6 @@ class FileOutputTest < Test::Unit::TestCase
14
14
  end
15
15
 
16
16
  TMP_DIR = File.expand_path(File.dirname(__FILE__) + "/../tmp/out_file#{ENV['TEST_ENV_NUMBER']}")
17
- SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current")
18
17
 
19
18
  CONFIG = %[
20
19
  path #{TMP_DIR}/out_file_test
@@ -663,15 +662,43 @@ class FileOutputTest < Test::Unit::TestCase
663
662
  end
664
663
  end
665
664
 
666
- test 'symlink' do
667
- omit "Windows doesn't support symlink" if Fluent.windows?
668
- conf = CONFIG + %[
669
- symlink_path #{SYMLINK_PATH}
670
- ]
671
- symlink_path = "#{SYMLINK_PATH}"
665
+ SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current")
666
+
667
+ sub_test_case 'symlink' do
668
+ test 'static symlink' do
669
+ omit "Windows doesn't support symlink" if Fluent.windows?
670
+ conf = CONFIG + %[
671
+ symlink_path #{SYMLINK_PATH}
672
+ ]
673
+ symlink_path = "#{SYMLINK_PATH}"
674
+
675
+ d = create_driver(conf)
676
+ begin
677
+ run_and_check(d, symlink_path)
678
+ ensure
679
+ FileUtils.rm_rf(symlink_path)
680
+ end
681
+ end
682
+
683
+ test 'symlink with placeholders' do
684
+ omit "Windows doesn't support symlink" if Fluent.windows?
685
+ conf = %[
686
+ path #{TMP_DIR}/${tag}/out_file_test
687
+ symlink_path #{SYMLINK_PATH}-${tag}
688
+ <buffer tag,time>
689
+ </buffer>
690
+ ]
691
+ symlink_path = "#{SYMLINK_PATH}-tag"
692
+
693
+ d = create_driver(conf)
694
+ begin
695
+ run_and_check(d, symlink_path)
696
+ ensure
697
+ FileUtils.rm_rf(symlink_path)
698
+ end
699
+ end
672
700
 
673
- d = create_driver(conf)
674
- begin
701
+ def run_and_check(d, symlink_path)
675
702
  d.run(default_tag: 'tag') do
676
703
  es = Fluent::OneEventStream.new(event_time("2011-01-02 13:14:15 UTC"), {"a"=>1})
677
704
  d.feed(es)
@@ -688,8 +715,6 @@ class FileOutputTest < Test::Unit::TestCase
688
715
  meta = d.instance.metadata('tag', event_time("2011-01-03 14:15:16 UTC"), {})
689
716
  assert_equal d.instance.buffer.instance_eval{ @stage[meta].path }, File.readlink(symlink_path)
690
717
  end
691
- ensure
692
- FileUtils.rm_rf(symlink_path)
693
718
  end
694
719
  end
695
720
 
@@ -39,7 +39,7 @@ class FileOutputSecondaryTest < Test::Unit::TestCase
39
39
  c.configure(conf)
40
40
  end
41
41
 
42
- sub_test_case 'configture' do
42
+ sub_test_case 'configure' do
43
43
  test 'default configuration' do
44
44
  d = create_driver %[directory #{TMP_DIR}]
45
45
  assert_equal 'dump.bin', d.instance.basename
@@ -547,7 +547,7 @@ class OutputTest < Test::Unit::TestCase
547
547
  assert_equal 86400, s
548
548
  assert_equal :day, t
549
549
  assert_equal '%d', e
550
- s, t, e = @i.get_placeholders_time("my birthiday! at %F")
550
+ s, t, e = @i.get_placeholders_time("my birthday! at %F")
551
551
  assert_equal 86400, s
552
552
  assert_equal :day, t
553
553
  assert_equal '%d', e
@@ -1074,6 +1074,44 @@ class BufferedOutputTest < Test::Unit::TestCase
1074
1074
  end
1075
1075
  end
1076
1076
 
1077
+ sub_test_case 'buffered output with large timekey and small timekey_wait' do
1078
+ setup do
1079
+ chunk_key = 'time'
1080
+ hash = {
1081
+ 'timekey' => 86400, # per 1 day
1082
+ 'timekey_wait' => 10, # 10 seconds delay for flush
1083
+ 'flush_thread_count' => 1,
1084
+ 'flush_thread_burst_interval' => 0.01,
1085
+ }
1086
+ @i = create_output(:buffered)
1087
+ @i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash)]))
1088
+ @i.start
1089
+ @i.after_start
1090
+ end
1091
+
1092
+ test '#configure raises config error if timekey is not specified' do
1093
+ Timecop.freeze( Time.parse('2019-02-08 00:01:00 +0900') )
1094
+ ary = []
1095
+ @i.register(:write){|chunk| ary << chunk.read }
1096
+ @i.thread_wait_until_start
1097
+ events = [
1098
+ [event_time('2019-02-08 00:02:00 +0900'), {"message" => "foobar"}]
1099
+ ]
1100
+ @i.emit_events("test.tag", Fluent::ArrayEventStream.new(events))
1101
+ @i.enqueue_thread_wait
1102
+ assert{ @i.write_count == 0 }
1103
+
1104
+ Timecop.freeze( Time.parse('2019-02-09 00:00:08 +0900') )
1105
+ @i.enqueue_thread_wait
1106
+ assert{ @i.write_count == 0 }
1107
+
1108
+ Timecop.freeze( Time.parse('2019-02-09 00:00:12 +0900') )
1109
+ # wirte should be called in few seconds since
1110
+ # running interval of enque thread is timekey_wait / 11.0.
1111
+ waiting(5){ sleep 0.1 until @i.write_count == 1 }
1112
+ end
1113
+ end
1114
+
1077
1115
  sub_test_case 'buffered output feature with tag key' do
1078
1116
  setup do
1079
1117
  chunk_key = 'tag'
@@ -177,7 +177,12 @@ class BufferedOutputBackupTest < Test::Unit::TestCase
177
177
  }
178
178
  end
179
179
 
180
- test 'backup chunk without secondary' do
180
+ data('unrecoverable error' => Fluent::UnrecoverableError,
181
+ 'type error' => TypeError,
182
+ 'argument error' => ArgumentError,
183
+ 'no method error' => NoMethodError,
184
+ 'msgpack unpack error' => MessagePack::UnpackError)
185
+ test 'backup chunk without secondary' do |error_class|
181
186
  Fluent::SystemConfig.overwrite_system_config('root_dir' => TMP_DIR) do
182
187
  id = 'backup_test'
183
188
  hash = {
@@ -188,7 +193,7 @@ class BufferedOutputBackupTest < Test::Unit::TestCase
188
193
  @i.configure(config_element('ROOT', '', {'@id' => id}, [config_element('buffer', 'tag', hash)]))
189
194
  @i.register(:write) { |chunk|
190
195
  chunk_id = chunk.unique_id;
191
- raise Fluent::UnrecoverableError, "yay, your #write must fail"
196
+ raise error_class, "yay, your #write must fail"
192
197
  }
193
198
 
194
199
  flush_chunks
@@ -706,16 +706,20 @@ class BufferedOutputRetryTest < Test::Unit::TestCase
706
706
  end
707
707
 
708
708
  sub_test_case 'buffered output configured as retry_forever' do
709
- test 'configuration error will be raised if secondary section is configured' do
709
+ setup do
710
+ Fluent::Plugin.register_output('output_retries_secondary_test', FluentPluginOutputAsBufferedRetryTest::DummyFullFeatureOutput2)
711
+ end
712
+
713
+ test 'warning logs are generated if secondary section is configured' do
710
714
  chunk_key = 'tag'
711
715
  hash = {
712
716
  'retry_forever' => true,
713
717
  'retry_randomize' => false,
714
718
  }
715
719
  i = create_output()
716
- assert_raise Fluent::ConfigError do
717
- i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','')]))
718
- end
720
+ i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','', {'@type' => 'output_retries_secondary_test'})]))
721
+ logs = i.log.out.logs
722
+ assert { logs.any? { |l| l.include?("<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary") } }
719
723
  end
720
724
 
721
725
  test 'retry_timeout and retry_max_times will be ignored if retry_forever is true for exponential backoff' do
@@ -276,7 +276,7 @@ class SyslogParserTest < ::Test::Unit::TestCase
276
276
  def test_auto_with_legacy_syslog_message
277
277
  @parser.configure(
278
278
  'time_format' => '%b %d %M:%S:%H',
279
- 'mseeage_format' => 'auto',
279
+ 'message_format' => 'auto',
280
280
  )
281
281
  text = 'Feb 28 00:00:12 192.168.0.1 fluentd[11111]: [error] Syslog test'
282
282
  @parser.instance.parse(text) do |time, record|
@@ -290,7 +290,7 @@ class SyslogParserTest < ::Test::Unit::TestCase
290
290
  @parser.configure(
291
291
  'time_format' => '%b %d %M:%S:%H',
292
292
  'with_priority' => true,
293
- 'mseeage_format' => 'auto',
293
+ 'message_format' => 'auto',
294
294
  )
295
295
  text = '<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test'
296
296
  @parser.instance.parse(text) do |time, record|
@@ -48,7 +48,7 @@ class RecordAccessorHelperTest < Test::Unit::TestCase
48
48
 
49
49
  data("missing ']'" => "$['key1'",
50
50
  "missing array index with dot" => "$.hello[]",
51
- "missing array index with braket" => "$[]",
51
+ "missing array index with bracket" => "$[]",
52
52
  "more chars" => "$.key1[0]foo",
53
53
  "whitespace char included key in dot notation" => "$.key[0].ke y",
54
54
  "empty keys with dot" => "$.",
@@ -767,7 +767,7 @@ class ServerPluginHelperTest < Test::Unit::TestCase
767
767
 
768
768
  def write_cert_and_key(cert_path, cert, key_path, key, passphrase)
769
769
  File.open(cert_path, "w"){|f| f.write(cert.to_pem) }
770
- # Write the secret key (raw or ecnrypted by AES256) in PEM format
770
+ # Write the secret key (raw or encrypted by AES256) in PEM format
771
771
  key_str = passphrase ? key.export(OpenSSL::Cipher.new("AES-256-CBC"), passphrase) : key.export
772
772
  File.open(key_path, "w"){|f| f.write(key_str) }
773
773
  File.chmod(0600, cert_path, key_path)
@@ -129,7 +129,7 @@ module FluentOutputTest
129
129
  end
130
130
 
131
131
  def test_secondary_with_no_warn_log
132
- # ObjectBufferedOutput doesn't implemnt `custom_filter`
132
+ # ObjectBufferedOutput doesn't implement `custom_filter`
133
133
  d = Fluent::Test::BufferedOutputTestDriver.new(Fluent::ObjectBufferedOutput)
134
134
 
135
135
  mock(d.instance.log).warn("secondary type should be same with primary one",
@@ -97,7 +97,7 @@ class PluginTest < Test::Unit::TestCase
97
97
  output1: ['plugin_test_dummy1', Dummy1Output, :new_output],
98
98
  output2: ['plugin_test_dummy2', Dummy2Output, :new_output],
99
99
  )
100
- test 'retruns plugin instances of registered plugin classes' do |(type, klass, m)|
100
+ test 'returns plugin instances of registered plugin classes' do |(type, klass, m)|
101
101
  instance = Fluent::Plugin.__send__(m, type)
102
102
  assert_kind_of klass, instance
103
103
  end
@@ -686,6 +686,54 @@ EOC
686
686
  end
687
687
  end
688
688
 
689
+ test 'raises configuration error for too big worker id on multi workers syntax' do
690
+ errmsg = "worker id 4 specified by <worker> directive is not allowed. Available worker id is between 0 and 3"
691
+ assert_raise Fluent::ConfigError.new(errmsg) do
692
+ conf = <<-EOC
693
+ <worker 1-4>
694
+ </worker>
695
+ EOC
696
+ configure_ra(conf)
697
+ end
698
+ end
699
+
700
+ test 'raises configuration error for worker id collisions on multi workers syntax' do
701
+ errmsg = "specified worker_id<2> collisions is detected on <worker> directive. Available worker id(s): [3]"
702
+ assert_raise Fluent::ConfigError.new(errmsg) do
703
+ conf = <<-EOC
704
+ <worker 0-2>
705
+ </worker>
706
+ <worker 2-4>
707
+ </worker>
708
+ EOC
709
+ configure_ra(conf)
710
+ end
711
+ end
712
+
713
+ test 'raises configuration error for worker id collisions on multi workers syntax when multi avaliable worker_ids are left' do
714
+ errmsg = "specified worker_id<1> collisions is detected on <worker> directive. Available worker id(s): [2, 3]"
715
+ assert_raise Fluent::ConfigError.new(errmsg) do
716
+ conf = <<-EOC
717
+ <worker 0-1>
718
+ </worker>
719
+ <worker 1-3>
720
+ </worker>
721
+ EOC
722
+ configure_ra(conf)
723
+ end
724
+ end
725
+
726
+ test 'raises configuration error for too big worker id on invalid reversed multi workers syntax' do
727
+ errmsg = "greater first_worker_id<3> than last_worker_id<0> specified by <worker> directive is not allowed. Available multi worker assign syntax is <smaller_worker_id>-<greater_worker_id>"
728
+ assert_raise Fluent::ConfigError.new(errmsg) do
729
+ conf = <<-EOC
730
+ <worker 3-0>
731
+ </worker>
732
+ EOC
733
+ configure_ra(conf)
734
+ end
735
+ end
736
+
689
737
  test 'raises configuration error for invalid elements as a child of worker section' do
690
738
  errmsg = '<worker> section cannot have <system> directive'
691
739
  assert_raise Fluent::ConfigError.new(errmsg) do
@@ -844,5 +892,33 @@ EOC
844
892
  assert_equal 0, ra.labels.size
845
893
  refute ra.error_collector
846
894
  end
895
+
896
+ test 'with plugins for workers syntax should match worker_id equals to 2' do
897
+ conf = <<-EOC
898
+ <worker 0-2>
899
+ <source>
900
+ @type forward
901
+ </source>
902
+ <filter **>
903
+ @type test_filter
904
+ @id test_filter
905
+ </filter>
906
+ <match pattern>
907
+ @type stdout
908
+ </match>
909
+ <label @ERROR>
910
+ <match>
911
+ @type null
912
+ </match>
913
+ </label>
914
+ </worker>
915
+ EOC
916
+
917
+ ra = configure_ra(conf)
918
+ assert_kind_of Fluent::Plugin::ForwardInput, ra.inputs.first
919
+ assert_kind_of Fluent::Plugin::StdoutOutput, ra.outputs.first
920
+ assert_kind_of FluentTestFilter, ra.filters.first
921
+ assert ra.error_collector
922
+ end
847
923
  end
848
924
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluentd
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.3.3
4
+ version: 1.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-01-07 00:00:00.000000000 Z
11
+ date: 2019-02-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: msgpack