fluentd 1.4.1 → 1.4.2

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: b5edf62d55c4f1fbdadda5a367479cdf5b875f3b
4
- data.tar.gz: e931a66ae995f73c3309ab9fcf29d2a8e36ef0c8
3
+ metadata.gz: e8855d66a886343a3865b3f224936aa433822027
4
+ data.tar.gz: 96d848425ac12776baf47bb2d999127697a6e7b1
5
5
  SHA512:
6
- metadata.gz: 6c1e49998c72cb8a4d491e355095804167a86dd9a26b391d1d6188c51976ae30c77d28b43c0433e57b83e8a1c4503bc248fce0cbe4b98d079ba795414f6c33ca
7
- data.tar.gz: ec39e3c4c5dafab999a61799a003ab170bf08df076dc1a98ad5c32910e3dc6e4a6aef5d83182b44e7f3cc2621b06a6e3e8333179a62ae466aa5b940ff8ded2d0
6
+ metadata.gz: f9e0ab8e68c24424f1eb75e459211b8195fb4a6ee1b2be99ef1a88da3d5e3e643b5243cadb87f9e68aa0f23eedad0d1c73aa9a53e72f75adfd52d7709dfe47af
7
+ data.tar.gz: 11e5d4293430e2675c70e323111044599be4de9bbf1bcb22045f7514f90788011949ddd27a07ba66b2c9b1ac2ee319d948b0f29113ccbd098ef4b5eb1f7cdebe
data/CHANGELOG.md CHANGED
@@ -1,13 +1,35 @@
1
1
  # v1.4
2
2
 
3
+ ## Release v1.4.2 - 2019/04/02
4
+
5
+ ### Enhancements
6
+
7
+ * in_http: subdomain support in CORS domain
8
+ https://github.com/fluent/fluentd/pull/2337
9
+ * in_monitor_agent: Expose current timekey list as a buffer metrics
10
+ https://github.com/fluent/fluentd/pull/2343
11
+ * in_tcp/in_udp: Add source_address_key parameter
12
+ https://github.com/fluent/fluentd/pull/2347
13
+ * in_forward: Add send_keepalive_packet parameter to check the remote connection is available or not
14
+ https://github.com/fluent/fluentd/pull/2352
15
+
16
+ ### Bug fixes
17
+
18
+ * out_exec_filter: Fix typo of child_respawn description
19
+ https://github.com/fluent/fluentd/pull/2341
20
+ * in_tail: Create parent directories for symlink
21
+ https://github.com/fluent/fluentd/pull/2353
22
+ * in_tail: Fix encoding duplication check for non-specified case
23
+ https://github.com/fluent/fluentd/pull/2361
24
+ * log: Fix time format handling of plugin logger when log format is JSON
25
+ https://github.com/fluent/fluentd/pull/2356
26
+
3
27
  ## Release v1.4.1 - 2019/03/18
4
28
 
5
29
  ### Enhancements
6
30
 
7
31
  * system: Add worker_id to process_name when workers is larger than 1
8
32
  https://github.com/fluent/fluentd/pull/2321
9
- * in_syslog: Check message length when read from buffer in octet counting
10
- https://github.com/fluent/fluentd/pull/2323
11
33
  * parser_regexp: Check named captures. When no named captures, configuration error is raised
12
34
  https://github.com/fluent/fluentd/pull/2331
13
35
 
@@ -15,6 +37,8 @@
15
37
 
16
38
  * out_forward: Make tls_client_private_key_passphrase secret
17
39
  https://github.com/fluent/fluentd/pull/2324
40
+ * in_syslog: Check message length when read from buffer in octet counting
41
+ https://github.com/fluent/fluentd/pull/2323
18
42
 
19
43
  ## Release v1.4.0 - 2019/02/24
20
44
 
data/README.md CHANGED
@@ -53,11 +53,6 @@ You can run specified test via `TEST` environment variable:
53
53
  $ bundle exec rake test TEST=test/test_specified_path.rb
54
54
  $ bundle exec rake test TEST=test/test_*.rb
55
55
 
56
- ## Running in Production
57
-
58
- Many enterprises run Fluentd in production to handle all of their logging needs. For enterprises requiring Security tested binaries, SLA-based support, architectural guidance, and enhanced plugin connectors see [Fluentd Enterprise](https://www.fluentd.org/enterprise).
59
-
60
-
61
56
  ## Fluentd UI: Admin GUI
62
57
 
63
58
  [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd.
data/lib/fluent/log.rb CHANGED
@@ -532,6 +532,7 @@ module Fluent
532
532
  end
533
533
 
534
534
  self.format = @logger.format
535
+ self.time_format = @logger.time_format
535
536
  enable_color @logger.enable_color?
536
537
  end
537
538
 
@@ -540,6 +541,7 @@ module Fluent
540
541
  end
541
542
 
542
543
  alias orig_format= format=
544
+ alias orig_time_format= time_format=
543
545
  alias orig_enable_color enable_color
544
546
 
545
547
  def format=(fmt)
@@ -547,6 +549,11 @@ module Fluent
547
549
  @logger.format = fmt
548
550
  end
549
551
 
552
+ def time_format=(fmt)
553
+ self.orig_time_format = fmt
554
+ @logger.time_format = fmt
555
+ end
556
+
550
557
  def enable_color(b = true)
551
558
  orig_enable_color b
552
559
  @logger.enable_color b
@@ -554,9 +561,9 @@ module Fluent
554
561
 
555
562
  extend Forwardable
556
563
  def_delegators '@logger', :get_worker_id, :enable_color?, :enable_debug, :enable_event,
557
- :disable_events, :log_event_enabled, :log_event_enabled=, :time_format, :time_format=,
558
- :time_formatter, :time_formatter=, :event, :caller_line, :puts, :write, :<<, :flush,
559
- :reset, :out, :out=, :optional_header, :optional_header=, :optional_attrs, :optional_attrs=
564
+ :disable_events, :log_event_enabled, :log_event_enabled=, :event, :caller_line, :puts, :write,
565
+ :<<, :flush, :reset, :out, :out=, :optional_header, :optional_header=, :optional_attrs,
566
+ :optional_attrs=
560
567
  end
561
568
 
562
569
 
@@ -133,6 +133,8 @@ module Fluent
133
133
  Dir.glob(patterns) do |path|
134
134
  next unless File.file?(path)
135
135
 
136
+ log.debug { "restoring buffer file: path = #{path}" }
137
+
136
138
  m = new_metadata() # this metadata will be overwritten by resuming .meta file content
137
139
  # so it should not added into @metadata_list for now
138
140
  mode = Fluent::Plugin::Buffer::FileChunk.assume_chunk_state(path)
@@ -154,6 +154,7 @@ module Fluent
154
154
  @dequeued_num = {} # metadata => int (number of dequeued chunks)
155
155
 
156
156
  @stage_size = @queue_size = 0
157
+ @timekeys = Hash.new(0)
157
158
  @metadata_list = [] # keys of @stage
158
159
  end
159
160
 
@@ -176,12 +177,14 @@ module Fluent
176
177
  @stage.each_pair do |metadata, chunk|
177
178
  @metadata_list << metadata unless @metadata_list.include?(metadata)
178
179
  @stage_size += chunk.bytesize
180
+ add_timekey(metadata)
179
181
  end
180
182
  @queue.each do |chunk|
181
183
  @metadata_list << chunk.metadata unless @metadata_list.include?(chunk.metadata)
182
184
  @queued_num[chunk.metadata] ||= 0
183
185
  @queued_num[chunk.metadata] += 1
184
186
  @queue_size += chunk.bytesize
187
+ add_timekey(chunk.metadata)
185
188
  end
186
189
  log.debug "buffer started", instance: self.object_id, stage_size: @stage_size, queue_size: @queue_size
187
190
  end
@@ -206,6 +209,7 @@ module Fluent
206
209
  super
207
210
  @dequeued = @stage = @queue = @queued_num = @metadata_list = nil
208
211
  @stage_size = @queue_size = 0
212
+ @timekeys.clear
209
213
  end
210
214
 
211
215
  def storable?
@@ -251,6 +255,7 @@ module Fluent
251
255
  @metadata_list[i]
252
256
  else
253
257
  @metadata_list << metadata
258
+ add_timekey(metadata)
254
259
  metadata
255
260
  end
256
261
  end
@@ -261,6 +266,30 @@ module Fluent
261
266
  add_metadata(meta)
262
267
  end
263
268
 
269
+ def add_timekey(metadata)
270
+ if t = metadata.timekey
271
+ @timekeys[t] += 1
272
+ end
273
+ nil
274
+ end
275
+ private :add_timekey
276
+
277
+ def del_timekey(metadata)
278
+ if t = metadata.timekey
279
+ if @timekeys[t] <= 1
280
+ @timekeys.delete(t)
281
+ else
282
+ @timekeys[t] -= 1
283
+ end
284
+ end
285
+ nil
286
+ end
287
+ private :del_timekey
288
+
289
+ def timekeys
290
+ @timekeys.keys
291
+ end
292
+
264
293
  # metadata MUST have consistent object_id for each variation
265
294
  # data MUST be Array of serialized events, or EventStream
266
295
  # metadata_and_data MUST be a hash of { metadata => data }
@@ -506,6 +535,7 @@ module Fluent
506
535
  @metadata_list.delete(metadata)
507
536
  @queued_num.delete(metadata)
508
537
  @dequeued_num.delete(metadata)
538
+ del_timekey(metadata)
509
539
  end
510
540
  log.trace "chunk purged", instance: self.object_id, chunk_id: dump_unique_id_hex(chunk_id), metadata: metadata
511
541
  end
@@ -47,6 +47,8 @@ module Fluent::Plugin
47
47
  config_param :resolve_hostname, :bool, default: nil
48
48
  desc 'Connections will be disconnected right after receiving first message if this value is true.'
49
49
  config_param :deny_keepalive, :bool, default: false
50
+ desc 'Check the remote connection is still available by sending a keepalive packet if this value is true.'
51
+ config_param :send_keepalive_packet, :bool, default: false
50
52
 
51
53
  desc 'Log warning if received chunk size is larger than this value.'
52
54
  config_param :chunk_size_warn_limit, :size, default: nil
@@ -141,6 +143,10 @@ module Fluent::Plugin
141
143
  })
142
144
  end
143
145
  end
146
+
147
+ if @send_keepalive_packet && @deny_keepalive
148
+ raise Fluent::ConfigError, "both 'send_keepalive_packet' and 'deny_keepalive' cannot be set to true"
149
+ end
144
150
  end
145
151
 
146
152
  def multi_workers_ready?
@@ -161,6 +167,7 @@ module Fluent::Plugin
161
167
  shared: shared_socket,
162
168
  resolve_name: @resolve_hostname,
163
169
  linger_timeout: @linger_timeout,
170
+ send_keepalive_packet: @send_keepalive_packet,
164
171
  backlog: @backlog,
165
172
  &method(:handle_connection)
166
173
  )
@@ -394,7 +394,7 @@ module Fluent::Plugin
394
394
  if @cors_allow_origins.include?('*')
395
395
  header["Access-Control-Allow-Origin"] = "*"
396
396
  send_response_and_close("200 OK", header, "")
397
- elsif @cors_allow_origins.include?(@origin)
397
+ elsif include_cors_allow_origin
398
398
  header["Access-Control-Allow-Origin"] = @origin
399
399
  send_response_and_close("200 OK", header, "")
400
400
  else
@@ -414,7 +414,7 @@ module Fluent::Plugin
414
414
  # For every incoming request, we check if we have some CORS
415
415
  # restrictions and white listed origins through @cors_allow_origins.
416
416
  unless @cors_allow_origins.nil?
417
- unless @cors_allow_origins.include?('*') or @cors_allow_origins.include?(@origin)
417
+ unless @cors_allow_origins.include?('*') or include_cors_allow_origin
418
418
  send_response_and_close("403 Forbidden", {'Connection' => 'close'}, "")
419
419
  return
420
420
  end
@@ -464,7 +464,7 @@ module Fluent::Plugin
464
464
  unless @cors_allow_origins.nil?
465
465
  if @cors_allow_origins.include?('*')
466
466
  header['Access-Control-Allow-Origin'] = '*'
467
- elsif @cors_allow_origins.include?(@origin)
467
+ elsif include_cors_allow_origin
468
468
  header['Access-Control-Allow-Origin'] = @origin
469
469
  end
470
470
  end
@@ -512,6 +512,17 @@ module Fluent::Plugin
512
512
  data << "\r\n"
513
513
  write data
514
514
  end
515
+
516
+ def include_cors_allow_origin
517
+ if @cors_allow_origins.include?(@origin)
518
+ return true
519
+ end
520
+ filtered_cors_allow_origins = @cors_allow_origins.select {|origin| origin != ""}
521
+ return filtered_cors_allow_origins.find do |origin|
522
+ (start_str,end_str) = origin.split("*",2)
523
+ @origin.start_with?(start_str) and @origin.end_with?(end_str)
524
+ end != nil
525
+ end
515
526
  end
516
527
  end
517
528
  end
@@ -285,6 +285,7 @@ module Fluent::Plugin
285
285
  MONITOR_INFO = {
286
286
  'output_plugin' => ->(){ is_a?(::Fluent::Plugin::Output) },
287
287
  'buffer_queue_length' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.queue.size },
288
+ 'buffer_timekeys' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.timekeys },
288
289
  'buffer_total_queued_size' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.stage_size + @buffer.queue_size },
289
290
  'retry_count' => ->(){ instance_variable_defined?(:@num_errors) ? @num_errors : nil },
290
291
  }
@@ -171,7 +171,7 @@ module Fluent::Plugin
171
171
 
172
172
  @encoding = parse_encoding_param(@encoding) if @encoding
173
173
  @from_encoding = parse_encoding_param(@from_encoding) if @from_encoding
174
- if @encoding == @from_encoding
174
+ if @encoding && (@encoding == @from_encoding)
175
175
  log.warn "'encoding' and 'from_encoding' are same encoding. No effect"
176
176
  end
177
177
  end
@@ -239,6 +239,7 @@ module Fluent::Plugin
239
239
  false
240
240
  end
241
241
  rescue Errno::ENOENT
242
+ log.debug("#{p} is missing after refresh file list")
242
243
  false
243
244
  end
244
245
  }
@@ -260,6 +261,8 @@ module Fluent::Plugin
260
261
  target_paths = expand_paths
261
262
  existence_paths = @tails.keys
262
263
 
264
+ log.debug { "tailing paths: target = #{target_paths.join(",")} | existing = #{existence_paths.join(",")}" }
265
+
263
266
  unwatched = existence_paths - target_paths
264
267
  added = target_paths - existence_paths
265
268
 
@@ -337,7 +340,7 @@ module Fluent::Plugin
337
340
  def update_watcher(path, pe)
338
341
  if @pf
339
342
  unless pe.read_inode == @pf[path].read_inode
340
- log.trace "Skip update_watcher because watcher has been already updated by other inotify event"
343
+ log.debug "Skip update_watcher because watcher has been already updated by other inotify event"
341
344
  return
342
345
  end
343
346
  end
@@ -33,6 +33,8 @@ module Fluent::Plugin
33
33
  config_param :source_host_key, :string, default: nil, deprecated: "use source_hostname_key instead."
34
34
  desc "The field name of the client's hostname."
35
35
  config_param :source_hostname_key, :string, default: nil
36
+ desc "The field name of the client's address."
37
+ config_param :source_address_key, :string, default: nil
36
38
 
37
39
  config_param :blocking_timeout, :time, default: 0.5
38
40
 
@@ -76,6 +78,7 @@ module Fluent::Plugin
76
78
  tag = extract_tag_from_record(record)
77
79
  tag ||= @tag
78
80
  time ||= extract_time_from_record(record) || Fluent::EventTime.now
81
+ record[@source_address_key] = conn.remote_addr if @source_address_key
79
82
  record[@source_hostname_key] = conn.remote_host if @source_hostname_key
80
83
  router.emit(tag, time, record)
81
84
  end
@@ -33,6 +33,8 @@ module Fluent::Plugin
33
33
  config_param :source_host_key, :string, default: nil, deprecated: "use source_hostname_key instead."
34
34
  desc "The field name of the client's hostname."
35
35
  config_param :source_hostname_key, :string, default: nil
36
+ desc "The field name of the client's address."
37
+ config_param :source_address_key, :string, default: nil
36
38
 
37
39
  desc "Deprecated parameter. Use message_length_limit instead"
38
40
  config_param :body_size_limit, :size, default: nil, deprecated: "use message_length_limit instead."
@@ -79,6 +81,7 @@ module Fluent::Plugin
79
81
  tag = extract_tag_from_record(record)
80
82
  tag ||= @tag
81
83
  time ||= extract_time_from_record(record) || Fluent::EventTime.now
84
+ record[@source_address_key] = sock.remote_addr if @source_address_key
82
85
  record[@source_hostname_key] = sock.remote_host if @source_hostname_key
83
86
  router.emit(tag, time, record)
84
87
  end
@@ -69,7 +69,7 @@ module Fluent::Plugin
69
69
  desc 'The number of spawned process for command.'
70
70
  config_param :num_children, :integer, default: 1
71
71
 
72
- desc 'Respawn command when command exit. ["none", "inf" or positive integer for times to respawn (defaut: none)]'
72
+ desc 'Respawn command when command exit. ["none", "inf" or positive integer for times to respawn (default: none)]'
73
73
  # nil, 'none' or 0: no respawn, 'inf' or -1: infinite times, positive integer: try to respawn specified times only
74
74
  config_param :child_respawn, :string, default: nil
75
75
 
@@ -68,6 +68,7 @@ module Fluent::Plugin
68
68
  config_set_default :timekey, DEFAULT_TIMEKEY
69
69
  end
70
70
 
71
+ attr_reader :dir_perm
71
72
  attr_accessor :last_written_path # for tests
72
73
 
73
74
  module SymlinkBufferMixin
@@ -87,7 +88,9 @@ module Fluent::Plugin
87
88
  # These chunks will be enqueued immediately, and will be flushed soon.
88
89
  latest_metadata = metadata_list.select{|m| m.timekey }.sort_by(&:timekey).last
89
90
  if chunk.metadata == latest_metadata
90
- FileUtils.ln_sf(chunk.path, @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk))
91
+ sym_path = @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk)
92
+ FileUtils.mkdir_p(File.dirname(sym_path), mode: @_output_plugin_for_symlink.dir_perm)
93
+ FileUtils.ln_sf(chunk.path, sym_path)
91
94
  end
92
95
  chunk
93
96
  end
@@ -24,7 +24,7 @@ module Fluent
24
24
  FORMAT_STRUCT_LINGER = 'I!I!' # { int l_onoff; int l_linger; }
25
25
  FORMAT_STRUCT_TIMEVAL = 'L!L!' # { time_t tv_sec; suseconds_t tv_usec; }
26
26
 
27
- def socket_option_validate!(protocol, resolve_name: nil, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil)
27
+ def socket_option_validate!(protocol, resolve_name: nil, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil, send_keepalive_packet: nil)
28
28
  unless resolve_name.nil?
29
29
  if protocol != :tcp && protocol != :udp && protocol != :tls
30
30
  raise ArgumentError, "BUG: resolve_name in available for tcp/udp/tls"
@@ -35,9 +35,14 @@ module Fluent
35
35
  raise ArgumentError, "BUG: linger_timeout is available for tcp/tls"
36
36
  end
37
37
  end
38
+ if send_keepalive_packet
39
+ if protocol != :tcp
40
+ raise ArgumentError, "BUG: send_keepalive_packet is available for tcp"
41
+ end
42
+ end
38
43
  end
39
44
 
40
- def socket_option_set(sock, resolve_name: nil, nonblock: false, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil)
45
+ def socket_option_set(sock, resolve_name: nil, nonblock: false, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil, send_keepalive_packet: nil)
41
46
  unless resolve_name.nil?
42
47
  sock.do_not_reverse_lookup = !resolve_name
43
48
  end
@@ -59,6 +64,9 @@ module Fluent
59
64
  if receive_buffer_size
60
65
  socket_option_set_one(sock, :SO_RCVBUF, receive_buffer_size.to_i)
61
66
  end
67
+ if send_keepalive_packet
68
+ socket_option_set_one(sock, :SO_KEEPALIVE, true)
69
+ end
62
70
  sock
63
71
  end
64
72
 
@@ -16,6 +16,6 @@
16
16
 
17
17
  module Fluent
18
18
 
19
- VERSION = '1.4.1'
19
+ VERSION = '1.4.2'
20
20
 
21
21
  end
@@ -189,6 +189,7 @@ class BufferTest < Test::Unit::TestCase
189
189
 
190
190
  assert_equal 0, plugin.stage_size
191
191
  assert_equal 0, plugin.queue_size
192
+ assert_equal [], plugin.timekeys
192
193
 
193
194
  # @p is started plugin
194
195
 
@@ -242,6 +243,7 @@ class BufferTest < Test::Unit::TestCase
242
243
  assert_nil @p.instance_eval{ @metadata_list } # #metadata_list does #dup for @metadata_list
243
244
  assert_equal 0, @p.stage_size
244
245
  assert_equal 0, @p.queue_size
246
+ assert_equal [], @p.timekeys
245
247
  end
246
248
 
247
249
  test '#metadata_list returns list of metadata on stage or in queue' do
@@ -569,9 +571,12 @@ class BufferTest < Test::Unit::TestCase
569
571
  assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata)
570
572
  assert_equal [@dm2,@dm3], @p.stage.keys
571
573
 
572
- prev_stage_size = @p.stage_size
574
+ timekey = Time.parse('2016-04-11 16:40:00 +0000').to_i
575
+ assert !@p.timekeys.include?(timekey)
573
576
 
574
- m = @p.metadata(timekey: Time.parse('2016-04-11 16:40:00 +0000').to_i)
577
+ prev_stage_size = @p.stage_size
578
+
579
+ m = @p.metadata(timekey: timekey)
575
580
 
576
581
  @p.write({m => ["x" * 256, "y" * 256, "z" * 256]})
577
582
 
@@ -581,6 +586,8 @@ class BufferTest < Test::Unit::TestCase
581
586
 
582
587
  assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata)
583
588
  assert_equal [@dm2,@dm3,m], @p.stage.keys
589
+
590
+ assert @p.timekeys.include?(timekey)
584
591
  end
585
592
 
586
593
  test '#write tries to enqueue and store data into a new chunk if existing chunk is full' do
@@ -688,8 +695,11 @@ class BufferTest < Test::Unit::TestCase
688
695
 
689
696
  assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata)
690
697
  assert_equal [@dm2,@dm3], @p.stage.keys
698
+
699
+ timekey = Time.parse('2016-04-11 16:40:00 +0000').to_i
700
+ assert !@p.timekeys.include?(timekey)
691
701
 
692
- m = @p.metadata(timekey: Time.parse('2016-04-11 16:40:00 +0000').to_i)
702
+ m = @p.metadata(timekey: timekey)
693
703
 
694
704
  es = Fluent::ArrayEventStream.new(
695
705
  [
@@ -708,6 +718,8 @@ class BufferTest < Test::Unit::TestCase
708
718
  assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata)
709
719
  assert_equal [@dm2,@dm3,m], @p.stage.keys
710
720
  assert_equal 1, @p.stage[m].append_count
721
+
722
+ assert @p.timekeys.include?(timekey)
711
723
  end
712
724
 
713
725
  test '#write w/ format tries to enqueue and store data into a new chunk if existing chunk does not have enough space' do
@@ -81,6 +81,27 @@ class ForwardInputTest < Test::Unit::TestCase
81
81
  assert_equal 1, d.instance.security.users.size
82
82
  assert_equal 1, d.instance.security.clients.size
83
83
  end
84
+
85
+ test 'send_keepalive_packet is disabled by default' do
86
+ @d = d = create_driver(CONFIG_AUTH)
87
+ assert_false d.instance.send_keepalive_packet
88
+ end
89
+
90
+ test 'send_keepalive_packet can be enabled' do
91
+ @d = d = create_driver(CONFIG_AUTH + %[
92
+ send_keepalive_packet true
93
+ ])
94
+ assert_true d.instance.send_keepalive_packet
95
+ end
96
+
97
+ test 'both send_keepalive_packet and deny_keepalive cannot be enabled' do
98
+ assert_raise(Fluent::ConfigError.new("both 'send_keepalive_packet' and 'deny_keepalive' cannot be set to true")) do
99
+ create_driver(CONFIG_AUTH + %[
100
+ send_keepalive_packet true
101
+ deny_keepalive true
102
+ ])
103
+ end
104
+ end
84
105
  end
85
106
 
86
107
  sub_test_case 'message' do
@@ -641,6 +641,63 @@ class HttpInputTest < Test::Unit::TestCase
641
641
  end
642
642
  end
643
643
 
644
+ def test_cors_allowed_wildcard_for_subdomain
645
+ d = create_driver(CONFIG + 'cors_allow_origins ["http://*.foo.com"]')
646
+
647
+ time = event_time("2011-01-02 13:14:15 UTC")
648
+ events = [
649
+ ["tag1", time, {"a"=>1}],
650
+ ]
651
+
652
+ d.run do
653
+ events.each do |tag, time, record|
654
+ headers = {"Origin" => "http://subdomain.foo.com"}
655
+
656
+ res = post("/#{tag}", {"json" => record.to_json, "time" => time.to_i}, headers)
657
+
658
+ assert_equal "200", res.code
659
+ assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"]
660
+ end
661
+ end
662
+ end
663
+
664
+ def test_cors_allowed_exclude_empty_string
665
+ d = create_driver(CONFIG + 'cors_allow_origins ["", "http://*.foo.com"]')
666
+
667
+ time = event_time("2011-01-02 13:14:15 UTC")
668
+ events = [
669
+ ["tag1", time, {"a"=>1}],
670
+ ]
671
+
672
+ d.run do
673
+ events.each do |tag, time, record|
674
+ headers = {"Origin" => "http://subdomain.foo.com"}
675
+
676
+ res = post("/#{tag}", {"json" => record.to_json, "time" => time.to_i}, headers)
677
+
678
+ assert_equal "200", res.code
679
+ assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"]
680
+ end
681
+ end
682
+ end
683
+
684
+ def test_cors_allowed_wildcard_preflight_for_subdomain
685
+ d = create_driver(CONFIG + 'cors_allow_origins ["http://*.foo.com"]')
686
+
687
+ d.run do
688
+ header = {
689
+ "Origin" => "http://subdomain.foo.com",
690
+ "Access-Control-Request-Method" => "POST",
691
+ "Access-Control-Request-Headers" => "Content-Type",
692
+ }
693
+ res = options("/cors.test", {}, header)
694
+
695
+ assert_equal "200", res.code
696
+ assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"]
697
+ assert_equal "POST", res["Access-Control-Allow-Methods"]
698
+ end
699
+ end
700
+
644
701
  def test_content_encoding_gzip
645
702
  d = create_driver
646
703
 
@@ -123,6 +123,7 @@ EOC
123
123
  output_info.merge!("config" => {"@id" => "test_out", "@type" => "test_out"}) if with_config
124
124
  error_label_info = {
125
125
  "buffer_queue_length" => 0,
126
+ "buffer_timekeys" => [],
126
127
  "buffer_total_queued_size" => 0,
127
128
  "output_plugin" => true,
128
129
  "plugin_category" => "output",
@@ -296,6 +297,7 @@ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:f
296
297
  expected_test_in_response.merge!("config" => {"@id" => "test_in", "@type" => "test_in"}) if with_config
297
298
  expected_null_response = {
298
299
  "buffer_queue_length" => 0,
300
+ "buffer_timekeys" => [],
299
301
  "buffer_total_queued_size" => 0,
300
302
  "output_plugin" => true,
301
303
  "plugin_category" => "output",
@@ -333,6 +335,7 @@ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:f
333
335
  expected_test_in_response.merge!("config" => {"@id" => "test_in", "@type" => "test_in"}) if with_config
334
336
  expected_null_response = {
335
337
  "buffer_queue_length" => 0,
338
+ "buffer_timekeys" => [],
336
339
  "buffer_total_queued_size" => 0,
337
340
  "output_plugin" => true,
338
341
  "plugin_category" => "output",
@@ -367,6 +370,7 @@ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:f
367
370
  }
368
371
  expected_null_response = {
369
372
  "buffer_queue_length" => 0,
373
+ "buffer_timekeys" => [],
370
374
  "buffer_total_queued_size" => 0,
371
375
  "output_plugin" => true,
372
376
  "plugin_category" => "output",
@@ -434,7 +438,8 @@ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:f
434
438
  <match **>
435
439
  @type test_out_fail_write
436
440
  @id test_out_fail_write
437
- <buffer>
441
+ <buffer time>
442
+ timekey 1m
438
443
  flush_mode immediate
439
444
  </buffer>
440
445
  </match>
@@ -453,17 +458,18 @@ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:f
453
458
  include_config no
454
459
  ")
455
460
  d.instance.start
461
+ output = @ra.outputs[0]
462
+ output.start
463
+ output.after_start
456
464
  expected_test_out_fail_write_response = {
457
465
  "buffer_queue_length" => 1,
466
+ "buffer_timekeys" => [output.calculate_timekey(event_time)],
458
467
  "buffer_total_queued_size" => 40,
459
468
  "output_plugin" => true,
460
469
  "plugin_category" => "output",
461
470
  "plugin_id" => "test_out_fail_write",
462
471
  "type" => "test_out_fail_write",
463
472
  }
464
- output = @ra.outputs[0]
465
- output.start
466
- output.after_start
467
473
  output.emit_events('test.tag', Fluent::ArrayEventStream.new([[event_time, {"message" => "test failed flush 1"}]]))
468
474
  # flush few times to check steps
469
475
  2.times do
@@ -142,4 +142,24 @@ class TcpInputTest < Test::Unit::TestCase
142
142
  assert event[1].is_a?(Fluent::EventTime)
143
143
  assert_equal hostname, event[2]['host']
144
144
  end
145
+
146
+ test 'source_address_key' do
147
+ d = create_driver(BASE_CONFIG + %!
148
+ format none
149
+ source_address_key addr
150
+ !)
151
+ address = nil
152
+ d.run(expect_records: 1) do
153
+ create_tcp_socket('127.0.0.1', PORT) do |sock|
154
+ address = sock.peeraddr[3]
155
+ sock.send("test\n", 0)
156
+ end
157
+ end
158
+
159
+ assert_equal 1, d.events.size
160
+ event = d.events[0]
161
+ assert_equal "tcp", event[0]
162
+ assert event[1].is_a?(Fluent::EventTime)
163
+ assert_equal address, event[2]['addr']
164
+ end
145
165
  end
@@ -201,6 +201,26 @@ class UdpInputTest < Test::Unit::TestCase
201
201
  assert_equal hostname, d.events[0][2]['host']
202
202
  end
203
203
 
204
+ test 'source_address_key' do
205
+ d = create_driver(BASE_CONFIG + %!
206
+ format none
207
+ source_address_key addr
208
+ !)
209
+ address = nil
210
+ d.run(expect_records: 1) do
211
+ create_udp_socket('127.0.0.1', PORT) do |u|
212
+ u.send("test", 0)
213
+ address = u.peeraddr[3]
214
+ end
215
+ end
216
+
217
+ expected = {'message' => 'test'}
218
+ assert_equal 1, d.events.size
219
+ assert_equal "udp", d.events[0][0]
220
+ assert d.events[0][1].is_a?(Fluent::EventTime)
221
+ assert_equal address, d.events[0][2]['addr']
222
+ end
223
+
204
224
  test 'receive_buffer_size' do
205
225
  # doesn't check exact value because it depends on platform and condition
206
226
 
@@ -684,11 +684,11 @@ class FileOutputTest < Test::Unit::TestCase
684
684
  omit "Windows doesn't support symlink" if Fluent.windows?
685
685
  conf = %[
686
686
  path #{TMP_DIR}/${tag}/out_file_test
687
- symlink_path #{SYMLINK_PATH}-${tag}
687
+ symlink_path #{SYMLINK_PATH}/foo/${tag}
688
688
  <buffer tag,time>
689
689
  </buffer>
690
690
  ]
691
- symlink_path = "#{SYMLINK_PATH}-tag"
691
+ symlink_path = "#{SYMLINK_PATH}/foo/tag"
692
692
 
693
693
  d = create_driver(conf)
694
694
  begin
@@ -236,6 +236,15 @@ class ServerPluginHelperTest < Test::Unit::TestCase
236
236
  end
237
237
  end
238
238
 
239
+ data(
240
+ 'server_create udp' => [:server_create, :udp],
241
+ )
242
+ test 'raise error if tcp/tls/unix options specified for udp' do |(m, proto)|
243
+ assert_raise(ArgumentError.new("BUG: send_keepalive_packet is available for tcp")) do
244
+ @d.__send__(m, :myserver, PORT, proto: proto, send_keepalive_packet: true){|x| x }
245
+ end
246
+ end
247
+
239
248
  data(
240
249
  'server_create tcp' => [:server_create, :tcp, {}],
241
250
  'server_create udp' => [:server_create, :udp, {max_bytes: 128}],
@@ -352,7 +361,7 @@ class ServerPluginHelperTest < Test::Unit::TestCase
352
361
  sub_test_case '#server_create_tcp' do
353
362
  test 'can accept all keyword arguments valid for tcp server' do
354
363
  assert_nothing_raised do
355
- @d.server_create_tcp(:s, PORT, bind: '127.0.0.1', shared: false, resolve_name: true, linger_timeout: 10, backlog: 500) do |data, conn|
364
+ @d.server_create_tcp(:s, PORT, bind: '127.0.0.1', shared: false, resolve_name: true, linger_timeout: 10, backlog: 500, send_keepalive_packet: true) do |data, conn|
356
365
  # ...
357
366
  end
358
367
  end
data/test/test_log.rb CHANGED
@@ -744,12 +744,6 @@ class PluginLoggerTest < Test::Unit::TestCase
744
744
  @log.disable_events(Thread.current)
745
745
  end
746
746
 
747
- def test_time_format
748
- assert_equal(@log.time_format, @logger.time_format)
749
- @log.time_format = "time_format"
750
- assert_equal(@log.time_format, @logger.time_format)
751
- end
752
-
753
747
  def test_event
754
748
  mock(@logger).event(Fluent::Log::LEVEL_TRACE, { key: "value" })
755
749
  @log.event(Fluent::Log::LEVEL_TRACE, { key: "value" })
@@ -794,6 +788,36 @@ class PluginLoggerTest < Test::Unit::TestCase
794
788
  assert_equal(@log.optional_attrs, @logger.optional_attrs)
795
789
  end
796
790
  end
791
+
792
+ sub_test_case "partially delegated" do
793
+ def setup
794
+ super
795
+ @log = Fluent::PluginLogger.new(@logger)
796
+ end
797
+
798
+ data(
799
+ text: [:text, "2016-04-21 11:58:41 +0900 [info]: yaaay\n"],
800
+ json: [:json, %Q({"time":"2016-04-21 11:58:41 +0900","level":"info","message":"yaaay"}\n)],
801
+ )
802
+ def test_format(data)
803
+ fmt, expected_log_line = data
804
+ @log.format = fmt
805
+ @log.info "yaaay"
806
+ assert{ @log_device.logs.include? expected_log_line }
807
+ end
808
+
809
+ data(
810
+ text: [:text, "2016 [info]: yaaay\n"],
811
+ json: [:json, %Q({"time":"2016","level":"info","message":"yaaay"}\n)],
812
+ )
813
+ def test_time_format(data)
814
+ fmt, expected_log_line = data
815
+ @log.format = fmt
816
+ @log.time_format = "%Y"
817
+ @log.info "yaaay"
818
+ assert{ @log_device.logs.include? expected_log_line }
819
+ end
820
+ end
797
821
  end
798
822
 
799
823
  class PluginLoggerMixinTest < Test::Unit::TestCase
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluentd
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.4.1
4
+ version: 1.4.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-03-19 00:00:00.000000000 Z
11
+ date: 2019-04-02 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: msgpack