fluentd 1.14.4 → 1.14.5

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: deabdaa85e1adbc3d54e342e2faefaa2c952b492de3167e68fc4be021c605acb
4
- data.tar.gz: f1e08a07176309930295b14c45bebaca86e7a7fd8de1191440d987170c752b96
3
+ metadata.gz: 98b1a3f0960184a4611f6c5139a0e4cb24cec72ec28581ad29d0012d32e1b9aa
4
+ data.tar.gz: 69cd5c67d4171fb9afb6a5c9d2c656b87921cc6e7a96aeed3017a62011010d63
5
5
  SHA512:
6
- metadata.gz: 2f8d7f216549cfd57266bec6a06a42e2def597073aa103bde30bb2bad816f3a25b6b5b942fb7320d69f15164b887ab951e15039f62e1808162e554d11f1c7db6
7
- data.tar.gz: f6a31183c931411e1cd4fae6d3dd8691f06ddfdfed4597bba001415052de1bf0081f1da6c75cf52ff4fd7479bf27894306d6336eb0f320c7adf1451dbcdf579a
6
+ metadata.gz: 32de1e2b6f891d041bac810aba2b1b9614b6cc72d2ffa22bb58cc4c9b61db3c54bac2d1bbfed56bfeae9147bacd840c37f1f43b8e841f15fcb7b07572301db83
7
+ data.tar.gz: 847ad4f2ff4615ee3fed3fe287d58a8734da7476a8ec8f444dcbf7ccec21b16a90e6c34351fc14737fa3ae0a9380e9cb53cbceda2873082cbaaa3e419b56c60a
data/CHANGELOG.md CHANGED
@@ -1,3 +1,27 @@
1
+ # v1.14.5
2
+
3
+ ## Release v1.14.5 - 2022/02/09
4
+
5
+ ### Enhancement
6
+
7
+ * Add support for "application/x-ndjson" to `in_http`
8
+ https://github.com/fluent/fluentd/pull/3616
9
+ * Add support for ucrt binary for Windows
10
+ https://github.com/fluent/fluentd/pull/3613
11
+
12
+ ### Bug fixes
13
+
14
+ * Don't retry when `retry_max_times == 0`
15
+ https://github.com/fluent/fluentd/pull/3608
16
+ * Fix hang-up issue during TLS handshake in `out_forward`
17
+ https://github.com/fluent/fluentd/pull/3601
18
+ * Bump up required ServerEngine to v2.2.5
19
+ https://github.com/fluent/fluentd/pull/3599
20
+ * Fix "invalid byte sequence is replaced" warning on Kubernetes
21
+ https://github.com/fluent/fluentd/pull/3596
22
+ * Fix "ArgumentError: unknown keyword: :logger" on Windows with Ruby 3.1
23
+ https://github.com/fluent/fluentd/pull/3592
24
+
1
25
  # v1.14.4
2
26
 
3
27
  ## Release v1.14.4 - 2022/01/06
data/README.md CHANGED
@@ -1,17 +1,12 @@
1
1
  Fluentd: Open-Source Log Collector
2
2
  ===================================
3
- GitHub Actions:
4
3
 
5
4
  [![Testing on Ubuntu](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml)
6
5
  [![Testing on Windows](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml)
7
6
  [![Testing on macOS](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml)
7
+ [![Testing on Arm64](https://cloud.drone.io/api/badges/fluent/fluentd/status.svg?branch=master)](https://cloud.drone.io/fluent/fluentd)
8
8
  [![Code Climate](https://codeclimate.com/github/fluent/fluentd/badges/gpa.svg)](https://codeclimate.com/github/fluent/fluentd)
9
9
  [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1189/badge)](https://bestpractices.coreinfrastructure.org/projects/1189)
10
- [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd?ref=badge_shield)
11
-
12
- Drone CI for Arm64:
13
-
14
- [![pipeline status](https://cloud.drone.io/api/badges/fluent/fluentd/status.svg?branch=master)](https://cloud.drone.io/fluent/fluentd)
15
10
 
16
11
  [Fluentd](https://www.fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the [Unified Logging Layer](https://www.fluentd.org/blog/unified-logging-layer)).
17
12
 
@@ -19,16 +14,6 @@ Drone CI for Arm64:
19
14
  <img src="https://www.fluentd.org/images/fluentd-architecture.png" width="500px"/>
20
15
  </p>
21
16
 
22
- An event consists of *tag*, *time* and *record*. Tag is a string separated with '.' (e.g. myapp.access). It is used to categorize events. Time is a UNIX time recorded at occurrence of an event. Record is a JSON object.
23
-
24
- ## Example Use Cases
25
-
26
- Use Case | Description | Diagram
27
- -------- | ------------|:---------:
28
- Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/images/recipes/elasticsearch-s3-fluentd.png" height="150"/>
29
- Syslog Alerting | Fluentd can "grep" for events and send out alerts. | <img src="https://www.fluentd.org/images/syslog-fluentd-alert.png" height="100"/>
30
- Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/images/datasources/asynchronous_logging.png" height="150"/>
31
-
32
17
  ## Quick Start
33
18
 
34
19
  $ gem install fluentd
@@ -66,12 +51,6 @@ You can run specified test via `TEST` environment variable:
66
51
  $ bundle exec rake test TEST=test/test_specified_path.rb
67
52
  $ bundle exec rake test TEST=test/test_*.rb
68
53
 
69
- ## Fluentd UI: Admin GUI
70
-
71
- [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd.
72
-
73
- <p align="center"><img width="500" src="https://www.fluentd.org/images/blog/fluentd-ui.gif"/></p>
74
-
75
54
  ## More Information
76
55
 
77
56
  - Website: https://www.fluentd.org/
data/Rakefile CHANGED
@@ -14,7 +14,7 @@ namespace :build do
14
14
  desc 'Build gems for all platforms'
15
15
  task :all do
16
16
  Bundler.with_clean_env do
17
- %w[ruby x86-mingw32 x64-mingw32].each do |name|
17
+ %w[ruby x86-mingw32 x64-mingw32 x64-mingw-ucrt].each do |name|
18
18
  ENV['GEM_BUILD_FAKE_PLATFORM'] = name
19
19
  Rake::Task["build"].execute
20
20
  end
data/fluentd.gemspec CHANGED
@@ -22,7 +22,7 @@ Gem::Specification.new do |gem|
22
22
  gem.add_runtime_dependency("msgpack", [">= 1.3.1", "< 2.0.0"])
23
23
  gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
24
24
  gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
25
- gem.add_runtime_dependency("serverengine", [">= 2.2.2", "< 3.0.0"])
25
+ gem.add_runtime_dependency("serverengine", [">= 2.2.5", "< 3.0.0"])
26
26
  gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.9.0"])
27
27
  gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
28
28
  gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
@@ -51,5 +51,6 @@ Gem::Specification.new do |gem|
51
51
  gem.add_development_dependency("test-unit", ["~> 3.3"])
52
52
  gem.add_development_dependency("test-unit-rr", ["~> 1.0"])
53
53
  gem.add_development_dependency("oj", [">= 2.14", "< 4"])
54
+ gem.add_development_dependency("async", "~> 1.23")
54
55
  gem.add_development_dependency("async-http", ">= 0.50.0")
55
56
  end
@@ -29,4 +29,16 @@ module Fluent
29
29
 
30
30
  class SetDefault < Exception
31
31
  end
32
+
33
+ class NotFoundPluginError < ConfigError
34
+ attr_reader :type, :kind
35
+
36
+ def initialize(msg, type: nil, kind: nil)
37
+ @msg = msg
38
+ @type = type
39
+ @kind = kind
40
+
41
+ super(msg)
42
+ end
43
+ end
32
44
  end
@@ -72,8 +72,8 @@ module Fluent
72
72
 
73
73
  def string_safe_encoding(str)
74
74
  unless str.valid_encoding?
75
- log.info "invalid byte sequence is replaced in `#{str}`" if self.respond_to?(:log)
76
75
  str = str.scrub('?')
76
+ log.info "invalid byte sequence is replaced in `#{str}`" if self.respond_to?(:log)
77
77
  end
78
78
  yield str
79
79
  end
@@ -314,8 +314,16 @@ module Fluent::Plugin
314
314
  @parser_json.parse(js) do |_time, record|
315
315
  return nil, record
316
316
  end
317
+ elsif ndjson = params['ndjson']
318
+ events = []
319
+ ndjson.split(/\r?\n/).each do |js|
320
+ @parser_json.parse(js) do |_time, record|
321
+ events.push(record)
322
+ end
323
+ end
324
+ return nil, events
317
325
  else
318
- raise "'json' or 'msgpack' parameter is required"
326
+ raise "'json', 'ndjson' or 'msgpack' parameter is required"
319
327
  end
320
328
  end
321
329
 
@@ -567,6 +575,8 @@ module Fluent::Plugin
567
575
  params['json'] = @body
568
576
  elsif @content_type =~ /^application\/msgpack/
569
577
  params['msgpack'] = @body
578
+ elsif @content_type =~ /^application\/x-ndjson/
579
+ params['ndjson'] = @body
570
580
  end
571
581
  path_info = uri.path
572
582
 
@@ -1275,11 +1275,15 @@ module Fluent
1275
1275
 
1276
1276
  unless @retry
1277
1277
  @retry = retry_state(@buffer_config.retry_randomize)
1278
- if error
1279
- log.warn "failed to flush the buffer.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, chunk: chunk_id_hex, error: error
1280
- log.warn_backtrace error.backtrace
1278
+ if @retry.limit?
1279
+ # @retry_max_times == 0, fail imediately by the following block
1280
+ else
1281
+ if error
1282
+ log.warn "failed to flush the buffer.", retry_times: @retry.steps, next_retry_time: @retry.next_time.round, chunk: chunk_id_hex, error: error
1283
+ log.warn_backtrace error.backtrace
1284
+ end
1285
+ return
1281
1286
  end
1282
- return
1283
1287
  end
1284
1288
 
1285
1289
  # @retry exists
@@ -96,6 +96,7 @@ module Fluent
96
96
  enable_system_cert_store: true, allow_self_signed_cert: false, cert_paths: nil,
97
97
  cert_path: nil, private_key_path: nil, private_key_passphrase: nil,
98
98
  cert_thumbprint: nil, cert_logical_store_name: nil, cert_use_enterprise_store: true,
99
+ connect_timeout: nil,
99
100
  **kwargs, &block)
100
101
 
101
102
  host_is_ipaddress = IPAddr.new(host) rescue false
@@ -158,13 +159,23 @@ module Fluent
158
159
  end
159
160
  Fluent::TLS.set_version_to_context(context, version, min_version, max_version)
160
161
 
161
- tcpsock = socket_create_tcp(host, port, **kwargs)
162
+ tcpsock = socket_create_tcp(host, port, connect_timeout: connect_timeout, **kwargs)
162
163
  sock = WrappedSocket::TLS.new(tcpsock, context)
163
164
  sock.sync_close = true
164
165
  sock.hostname = fqdn if verify_fqdn && fqdn && sock.respond_to?(:hostname=)
165
166
 
166
167
  log.trace "entering TLS handshake"
167
- sock.connect
168
+ if connect_timeout
169
+ begin
170
+ Timeout.timeout(connect_timeout) { sock.connect }
171
+ rescue Timeout::Error
172
+ log.warn "timeout while connecting tls session", host: host
173
+ sock.close rescue nil
174
+ raise
175
+ end
176
+ else
177
+ sock.connect
178
+ end
168
179
 
169
180
  begin
170
181
  if verify_fqdn
@@ -45,7 +45,8 @@ module Fluent
45
45
  if value = @map[type]
46
46
  return value
47
47
  end
48
- raise ConfigError, "Unknown #{@kind} plugin '#{type}'. Run 'gem search -rd fluent-plugin' to find plugins" # TODO error class
48
+ raise NotFoundPluginError.new("Unknown #{@kind} plugin '#{type}'. Run 'gem search -rd fluent-plugin' to find plugins",
49
+ kind: @kind, type: type)
49
50
  end
50
51
 
51
52
  def reverse_lookup(value)
@@ -16,6 +16,6 @@
16
16
 
17
17
  module Fluent
18
18
 
19
- VERSION = '1.14.4'
19
+ VERSION = '1.14.5'
20
20
 
21
21
  end
@@ -22,7 +22,7 @@ class TextParserTest < ::Test::Unit::TestCase
22
22
  Fluent::TextParser.register_template('multi_event_test', Proc.new { MultiEventTestParser.new })
23
23
 
24
24
  def test_lookup_unknown_format
25
- assert_raise Fluent::ConfigError do
25
+ assert_raise Fluent::NotFoundPluginError do
26
26
  Fluent::Plugin.new_parser('unknown')
27
27
  end
28
28
  end
@@ -46,7 +46,7 @@ class ParserFilterTest < Test::Unit::TestCase
46
46
  assert_raise(Fluent::ConfigError) {
47
47
  create_driver('')
48
48
  }
49
- assert_raise(Fluent::ConfigError) {
49
+ assert_raise(Fluent::NotFoundPluginError) {
50
50
  create_driver %[
51
51
  key_name foo
52
52
  <parse>
@@ -63,7 +63,7 @@ class StdoutFilterTest < Test::Unit::TestCase
63
63
  end
64
64
 
65
65
  def test_invalid_output_type
66
- assert_raise(Fluent::ConfigError) do
66
+ assert_raise(Fluent::NotFoundPluginError) do
67
67
  d = create_driver(CONFIG + config_element("", "", { "output_type" => "foo" }))
68
68
  d.run {}
69
69
  end
@@ -139,7 +139,7 @@ class StdoutFilterTest < Test::Unit::TestCase
139
139
  def test_invalid_output_type
140
140
  conf = config_element
141
141
  conf.elements << config_element("format", "", { "@type" => "stdout", "output_type" => "foo" })
142
- assert_raise(Fluent::ConfigError) do
142
+ assert_raise(Fluent::NotFoundPluginError) do
143
143
  d = create_driver(conf)
144
144
  d.run {}
145
145
  end
@@ -540,6 +540,29 @@ class HttpInputTest < Test::Unit::TestCase
540
540
  assert_equal_event_time time, d.events[1][1]
541
541
  end
542
542
 
543
+ def test_application_ndjson
544
+ d = create_driver
545
+ events = [
546
+ ["tag1", 1643935663, "{\"a\":1}\n{\"b\":2}"],
547
+ ["tag2", 1643935664, "{\"a\":3}\r\n{\"b\":4}"]
548
+ ]
549
+
550
+ expected = [
551
+ ["tag1", 1643935663, {"a"=>1}],
552
+ ["tag1", 1643935663, {"b"=>2}],
553
+ ["tag2", 1643935664, {"a"=>3}],
554
+ ["tag2", 1643935664, {"b"=>4}]
555
+ ]
556
+
557
+ d.run(expect_records: 1) do
558
+ events.each do |tag, time, record|
559
+ res = post("/#{tag}?time=#{time}", record, {"Content-Type"=>"application/x-ndjson"})
560
+ assert_equal("200", res.code)
561
+ end
562
+ end
563
+ assert_equal(expected, d.events)
564
+ end
565
+
543
566
  def test_msgpack
544
567
  d = create_driver
545
568
  time = event_time("2011-01-02 13:14:15 UTC")
@@ -32,7 +32,7 @@ class StdoutOutputTest < Test::Unit::TestCase
32
32
  assert_kind_of Fluent::Plugin::StdoutFormatter, d.instance.formatter
33
33
  assert_equal 'hash', d.instance.formatter.output_type
34
34
 
35
- assert_raise(Fluent::ConfigError) do
35
+ assert_raise(Fluent::NotFoundPluginError) do
36
36
  d = create_driver(CONFIG + "\noutput_type foo")
37
37
  end
38
38
  end
@@ -126,7 +126,7 @@ class StdoutOutputTest < Test::Unit::TestCase
126
126
  assert_kind_of Fluent::Plugin::StdoutFormatter, d.instance.formatter
127
127
  assert_equal 'hash', d.instance.formatter.output_type
128
128
 
129
- assert_raise(Fluent::ConfigError) do
129
+ assert_raise(Fluent::NotFoundPluginError) do
130
130
  create_driver(config_element("ROOT", "", {"output_type" => "foo"}, [config_element("buffer")]))
131
131
  end
132
132
  end
@@ -701,6 +701,53 @@ class BufferedOutputRetryTest < Test::Unit::TestCase
701
701
  assert{ @i.buffer.stage.size == 1 }
702
702
  assert{ chunks.all?{|c| c.empty? } }
703
703
  end
704
+
705
+ test 'Do not retry when retry_max_times is 0' do
706
+ written_tags = []
707
+
708
+ chunk_key = 'tag'
709
+ hash = {
710
+ 'flush_interval' => 1,
711
+ 'flush_thread_burst_interval' => 0.1,
712
+ 'retry_type' => :periodic,
713
+ 'retry_wait' => 1,
714
+ 'retry_randomize' => false,
715
+ 'retry_max_times' => 0,
716
+ 'queued_chunks_limit_size' => 100
717
+ }
718
+ @i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash)]))
719
+ @i.register(:prefer_buffered_processing){ true }
720
+ @i.register(:format){|tag,time,record| [tag,time.to_i,record].to_json + "\n" }
721
+ @i.register(:write){|chunk| written_tags << chunk.metadata.tag; raise "yay, your #write must fail" }
722
+ @i.start
723
+ @i.after_start
724
+
725
+ @i.interrupt_flushes
726
+
727
+ now = Time.parse('2016-04-13 18:33:30 -0700')
728
+ Timecop.freeze( now )
729
+
730
+ @i.emit_events("test.tag.1", dummy_event_stream())
731
+
732
+ now = Time.parse('2016-04-13 18:33:31 -0700')
733
+ Timecop.freeze( now )
734
+
735
+ @i.emit_events("test.tag.2", dummy_event_stream())
736
+
737
+ assert_equal(0, @i.write_count)
738
+ assert_equal(0, @i.num_errors)
739
+
740
+ @i.enqueue_thread_wait
741
+ @i.flush_thread_wakeup
742
+ waiting(2){ Thread.pass until @i.write_count == 1 && @i.num_errors == 1 }
743
+
744
+ assert(@i.write_count == 1)
745
+ assert(@i.num_errors == 1)
746
+ assert(@i.log.out.logs.any?{|l| l.include?("[error]: failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue.") && l.include?("retry_times=0") })
747
+ assert(@i.buffer.queue.size == 0)
748
+ assert(@i.buffer.stage.size == 1)
749
+ assert(@i.buffer.queue.all?{|c| c.empty? })
750
+ end
704
751
  end
705
752
 
706
753
  sub_test_case 'buffered output configured as retry_forever' do
@@ -349,7 +349,7 @@ module FormatterTest
349
349
  include FormatterTest
350
350
 
351
351
  def test_unknown_format
352
- assert_raise ConfigError do
352
+ assert_raise NotFoundPluginError do
353
353
  Fluent::Plugin.new_formatter('unknown')
354
354
  end
355
355
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluentd
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.14.4
4
+ version: 1.14.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-01-06 00:00:00.000000000 Z
11
+ date: 2022-02-09 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -84,7 +84,7 @@ dependencies:
84
84
  requirements:
85
85
  - - ">="
86
86
  - !ruby/object:Gem::Version
87
- version: 2.2.2
87
+ version: 2.2.5
88
88
  - - "<"
89
89
  - !ruby/object:Gem::Version
90
90
  version: 3.0.0
@@ -94,7 +94,7 @@ dependencies:
94
94
  requirements:
95
95
  - - ">="
96
96
  - !ruby/object:Gem::Version
97
- version: 2.2.2
97
+ version: 2.2.5
98
98
  - - "<"
99
99
  - !ruby/object:Gem::Version
100
100
  version: 3.0.0
@@ -338,6 +338,20 @@ dependencies:
338
338
  - - "<"
339
339
  - !ruby/object:Gem::Version
340
340
  version: '4'
341
+ - !ruby/object:Gem::Dependency
342
+ name: async
343
+ requirement: !ruby/object:Gem::Requirement
344
+ requirements:
345
+ - - "~>"
346
+ - !ruby/object:Gem::Version
347
+ version: '1.23'
348
+ type: :development
349
+ prerelease: false
350
+ version_requirements: !ruby/object:Gem::Requirement
351
+ requirements:
352
+ - - "~>"
353
+ - !ruby/object:Gem::Version
354
+ version: '1.23'
341
355
  - !ruby/object:Gem::Dependency
342
356
  name: async-http
343
357
  requirement: !ruby/object:Gem::Requirement
@@ -945,7 +959,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
945
959
  - !ruby/object:Gem::Version
946
960
  version: '0'
947
961
  requirements: []
948
- rubygems_version: 3.1.2
962
+ rubygems_version: 3.3.3
949
963
  signing_key:
950
964
  specification_version: 4
951
965
  summary: Fluentd event collector