fluentd 1.14.1-x86-mingw32 → 1.14.5-x86-mingw32

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

Files changed (47) hide show
  1. checksums.yaml +4 -4
  2. data/.drone.yml +6 -6
  3. data/.github/ISSUE_TEMPLATE/bug_report.yaml +1 -0
  4. data/CHANGELOG.md +94 -0
  5. data/README.md +3 -22
  6. data/Rakefile +1 -1
  7. data/SECURITY.md +18 -0
  8. data/fluentd.gemspec +5 -4
  9. data/lib/fluent/command/cat.rb +13 -3
  10. data/lib/fluent/config/error.rb +12 -0
  11. data/lib/fluent/env.rb +4 -0
  12. data/lib/fluent/plugin/base.rb +1 -1
  13. data/lib/fluent/plugin/buf_file.rb +2 -2
  14. data/lib/fluent/plugin/buffer.rb +62 -8
  15. data/lib/fluent/plugin/in_http.rb +11 -1
  16. data/lib/fluent/plugin/in_tail.rb +21 -4
  17. data/lib/fluent/plugin/out_file.rb +13 -1
  18. data/lib/fluent/plugin/output.rb +11 -6
  19. data/lib/fluent/plugin/parser_apache2.rb +1 -1
  20. data/lib/fluent/plugin/storage_local.rb +3 -5
  21. data/lib/fluent/plugin_helper/socket.rb +13 -2
  22. data/lib/fluent/plugin_id.rb +2 -1
  23. data/lib/fluent/registry.rb +2 -1
  24. data/lib/fluent/supervisor.rb +2 -2
  25. data/lib/fluent/time.rb +21 -20
  26. data/lib/fluent/version.rb +1 -1
  27. data/test/command/test_cat.rb +31 -2
  28. data/test/compat/test_parser.rb +1 -1
  29. data/test/plugin/test_bare_output.rb +1 -1
  30. data/test/plugin/test_buffer.rb +149 -1
  31. data/test/plugin/test_filter.rb +1 -1
  32. data/test/plugin/test_filter_parser.rb +1 -1
  33. data/test/plugin/test_filter_stdout.rb +2 -2
  34. data/test/plugin/test_in_http.rb +23 -0
  35. data/test/plugin/test_in_tail.rb +90 -6
  36. data/test/plugin/test_input.rb +1 -1
  37. data/test/plugin/test_out_exec.rb +6 -4
  38. data/test/plugin/test_out_exec_filter.rb +4 -0
  39. data/test/plugin/test_out_file.rb +29 -13
  40. data/test/plugin/test_out_stdout.rb +2 -2
  41. data/test/plugin/test_output_as_buffered_retries.rb +47 -0
  42. data/test/plugin/test_output_as_buffered_secondary.rb +1 -1
  43. data/test/plugin_helper/test_child_process.rb +9 -9
  44. data/test/plugin_helper/test_timer.rb +2 -2
  45. data/test/test_formatter.rb +1 -1
  46. data/test/test_time_parser.rb +22 -0
  47. metadata +26 -11
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d5f5c1c3b6598d0eb25f9372beda5e5b08932b8facda51aa7e3f9ef9a98849cd
4
- data.tar.gz: 6569c24837680722e497009210e7bf64b1b3a50c9fe4b4ef4d6a89db8cbb0faf
3
+ metadata.gz: 9ab32693bc6090a2f2600b7bd1ac0b34137795a40c9bd9194f15fba1a66ae3f1
4
+ data.tar.gz: d3c4c191b81c61c4506dc72d689d0a990bd3f885f37265dc27bd91db52ff8e68
5
5
  SHA512:
6
- metadata.gz: 3bdfc7de632946e7e0ac8ee65c7ad4334399dee5471021da21fb15484c08471508af2ccd0b28f700850b57869aac29ced58b0e2d60bac1fc6bdc6a8373a4a41b
7
- data.tar.gz: 31dac402718a9e78416f2481704eec0ac35c154006c14f73c60c62149dc83bf53d55d3aad1b94a57f9dd45ae158b4eac979542de4e2d90ff41339881c16175e0
6
+ metadata.gz: 43c53a9a990f9cdc33b266f22c4f927200fa6e3c3fbfbec14698c38c68ec998eee6d5441c3993b3de96988bec8035cb56dffbc4008241034f2f403fbf358937a
7
+ data.tar.gz: 80b0fbe5ffa8ddb7ace7808318d273a549e3ac56d864f2e650cf8c424cef082c7672d15fbdd7d5cdcd77360d94c92a8d24658ede33e55a150bdb9bf6c432f47d
data/.drone.yml CHANGED
@@ -1,13 +1,13 @@
1
1
  kind: pipeline
2
- name: fluentd-test-arm64-2-6-3
2
+ name: fluentd-test-arm64-3-0
3
3
 
4
4
  platform:
5
5
  os: linux
6
6
  arch: arm64
7
7
 
8
8
  steps:
9
- - name: fluentd-test-arm64-2-6-3
10
- image: arm64v8/ruby:2.6.3
9
+ - name: fluentd-test-arm64-3-0
10
+ image: arm64v8/ruby:3.0
11
11
  commands:
12
12
  - apt update
13
13
  - apt -y install libgmp3-dev
@@ -17,15 +17,15 @@ steps:
17
17
  - bundle exec rake test
18
18
  ---
19
19
  kind: pipeline
20
- name: fluentd-test-arm64-latest
20
+ name: fluentd-test-arm64-2-7
21
21
 
22
22
  platform:
23
23
  os: linux
24
24
  arch: arm64
25
25
 
26
26
  steps:
27
- - name: fluentd-test-arm64-latest
28
- image: arm64v8/ruby:latest
27
+ - name: fluentd-test-arm64-2-7
28
+ image: arm64v8/ruby:2.7
29
29
  commands:
30
30
  - apt update
31
31
  - apt -y install libgmp3-dev
@@ -50,6 +50,7 @@ body:
50
50
  label: Your Configuration
51
51
  description: |
52
52
  Write your configuration here. Minimum reproducible fluentd.conf is recommended.
53
+ render: apache
53
54
  validations:
54
55
  required: true
55
56
  - type: textarea
data/CHANGELOG.md CHANGED
@@ -1,3 +1,97 @@
1
+ # v1.14.5
2
+
3
+ ## Release v1.14.5 - 2022/02/09
4
+
5
+ ### Enhancement
6
+
7
+ * Add support for "application/x-ndjson" to `in_http`
8
+ https://github.com/fluent/fluentd/pull/3616
9
+ * Add support for ucrt binary for Windows
10
+ https://github.com/fluent/fluentd/pull/3613
11
+
12
+ ### Bug fixes
13
+
14
+ * Don't retry when `retry_max_times == 0`
15
+ https://github.com/fluent/fluentd/pull/3608
16
+ * Fix hang-up issue during TLS handshake in `out_forward`
17
+ https://github.com/fluent/fluentd/pull/3601
18
+ * Bump up required ServerEngine to v2.2.5
19
+ https://github.com/fluent/fluentd/pull/3599
20
+ * Fix "invalid byte sequence is replaced" warning on Kubernetes
21
+ https://github.com/fluent/fluentd/pull/3596
22
+ * Fix "ArgumentError: unknown keyword: :logger" on Windows with Ruby 3.1
23
+ https://github.com/fluent/fluentd/pull/3592
24
+
25
+ # v1.14.4
26
+
27
+ ## Release v1.14.4 - 2022/01/06
28
+
29
+ ### Enhancement
30
+
31
+ * `in_tail`: Add option to skip long lines (`max_line_size`)
32
+ https://github.com/fluent/fluentd/pull/3565
33
+
34
+ ### Bug fix
35
+
36
+ * Incorrect BufferChunkOverflowError when each event size is < `chunk_limit_size`
37
+ https://github.com/fluent/fluentd/pull/3560
38
+ * On macOS with Ruby 2.7/3.0, `out_file` fails to write events if `append` is true.
39
+ https://github.com/fluent/fluentd/pull/3579
40
+ * test: Fix unstable test cases
41
+ https://github.com/fluent/fluentd/pull/3574
42
+ https://github.com/fluent/fluentd/pull/3577
43
+
44
+ # v1.14.3
45
+
46
+ ## Release v1.14.3 - 2021/11/26
47
+
48
+ ### Enhancement
49
+
50
+ * Changed to accept `http_parser.rb` 0.8.0.
51
+ `http_parser.rb` 0.8.0 is ready for Ractor.
52
+ https://github.com/fluent/fluentd/pull/3544
53
+
54
+ ### Bug fix
55
+
56
+ * in_tail: Fixed a bug that no new logs are read when
57
+ `enable_stat_watcher true` and `enable_watch_timer false` is set.
58
+ https://github.com/fluent/fluentd/pull/3541
59
+ * in_tail: Fixed a bug that the beginning and initial lines are lost
60
+ after startup when `read_from_head false` and path includes wildcard '*'.
61
+ https://github.com/fluent/fluentd/pull/3542
62
+ * Fixed a bug that processing messages were lost when
63
+ BufferChunkOverflowError was thrown even though only a specific
64
+ message size exceeds chunk_limit_size.
65
+ https://github.com/fluent/fluentd/pull/3553
66
+ https://github.com/fluent/fluentd/pull/3562
67
+
68
+ ### Misc
69
+
70
+ * Bump up required version of `win32-service` gem.
71
+ newer version is required to implement additional `fluent-ctl` commands.
72
+ https://github.com/fluent/fluentd/pull/3556
73
+
74
+ # v1.14.2
75
+
76
+ ## Release v1.14.2 - 2021/10/29
77
+
78
+ IMPORTANT: This release contain the fix for CVE-2021-41186 -
79
+ ReDoS vulnerability in `parser_apache2`.
80
+ This vulnerability is affected from Fluentd v0.14.14 to v1.14.1.
81
+ We recommend to upgrade Fluentd to v1.14.2 or use patched version of
82
+ `parser_apache2` plugin.
83
+
84
+ ### Enhancement
85
+
86
+ * fluent-cat: Add `--event-time` option to send specified event time for testing.
87
+
88
+ ### Bug fix
89
+
90
+ * Fixed to generate correct epoch timestamp even after switching Daylight Saving Time
91
+ https://github.com/fluent/fluentd/pull/3524
92
+ * Fixed ReDoS vulnerability in parser_apache2.
93
+ This vulnerability is caused by a certain pattern of a broken apache log.
94
+
1
95
  # v1.14.1
2
96
 
3
97
  ## Release v1.14.1 - 2021/09/29
data/README.md CHANGED
@@ -1,17 +1,12 @@
1
1
  Fluentd: Open-Source Log Collector
2
2
  ===================================
3
- GitHub Actions:
4
3
 
5
4
  [![Testing on Ubuntu](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml)
6
5
  [![Testing on Windows](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml)
7
6
  [![Testing on macOS](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml)
7
+ [![Testing on Arm64](https://cloud.drone.io/api/badges/fluent/fluentd/status.svg?branch=master)](https://cloud.drone.io/fluent/fluentd)
8
8
  [![Code Climate](https://codeclimate.com/github/fluent/fluentd/badges/gpa.svg)](https://codeclimate.com/github/fluent/fluentd)
9
9
  [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1189/badge)](https://bestpractices.coreinfrastructure.org/projects/1189)
10
- [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd?ref=badge_shield)
11
-
12
- Drone CI for Arm64:
13
-
14
- [![pipeline status](https://cloud.drone.io/api/badges/fluent/fluentd/status.svg?branch=master)](https://cloud.drone.io/fluent/fluentd)
15
10
 
16
11
  [Fluentd](https://www.fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the [Unified Logging Layer](https://www.fluentd.org/blog/unified-logging-layer)).
17
12
 
@@ -19,16 +14,6 @@ Drone CI for Arm64:
19
14
  <img src="https://www.fluentd.org/images/fluentd-architecture.png" width="500px"/>
20
15
  </p>
21
16
 
22
- An event consists of *tag*, *time* and *record*. Tag is a string separated with '.' (e.g. myapp.access). It is used to categorize events. Time is a UNIX time recorded at occurrence of an event. Record is a JSON object.
23
-
24
- ## Example Use Cases
25
-
26
- Use Case | Description | Diagram
27
- -------- | ------------|:---------:
28
- Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/images/recipes/elasticsearch-s3-fluentd.png" height="150"/>
29
- Syslog Alerting | Fluentd can "grep" for events and send out alerts. | <img src="https://www.fluentd.org/images/syslog-fluentd-alert.png" height="100"/>
30
- Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/images/datasources/asynchronous_logging.png" height="150"/>
31
-
32
17
  ## Quick Start
33
18
 
34
19
  $ gem install fluentd
@@ -66,12 +51,6 @@ You can run specified test via `TEST` environment variable:
66
51
  $ bundle exec rake test TEST=test/test_specified_path.rb
67
52
  $ bundle exec rake test TEST=test/test_*.rb
68
53
 
69
- ## Fluentd UI: Admin GUI
70
-
71
- [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd.
72
-
73
- <p align="center"><img width="500" src="https://www.fluentd.org/images/blog/fluentd-ui.gif"/></p>
74
-
75
54
  ## More Information
76
55
 
77
56
  - Website: https://www.fluentd.org/
@@ -88,6 +67,8 @@ You can run specified test via `TEST` environment variable:
88
67
 
89
68
  A third party security audit was performed by Cure53, you can see the full report [here](docs/SECURITY_AUDIT.pdf).
90
69
 
70
+ See [SECURITY](SECURITY.md) to contact us about vulnerability.
71
+
91
72
  ## Contributors:
92
73
 
93
74
  Patches contributed by [great developers](https://github.com/fluent/fluentd/contributors).
data/Rakefile CHANGED
@@ -14,7 +14,7 @@ namespace :build do
14
14
  desc 'Build gems for all platforms'
15
15
  task :all do
16
16
  Bundler.with_clean_env do
17
- %w[ruby x86-mingw32 x64-mingw32].each do |name|
17
+ %w[ruby x86-mingw32 x64-mingw32 x64-mingw-ucrt].each do |name|
18
18
  ENV['GEM_BUILD_FAKE_PLATFORM'] = name
19
19
  Rake::Task["build"].execute
20
20
  end
data/SECURITY.md ADDED
@@ -0,0 +1,18 @@
1
+ # Security Policy
2
+
3
+ ## Supported Versions
4
+
5
+ | Version | Supported |
6
+ | ------- | ------------------ |
7
+ | 1.14.x | :white_check_mark: |
8
+ | <= 1.13.x | :x: |
9
+
10
+ ## Reporting a Vulnerability
11
+
12
+ Please contact to current active maintainers. (in alphabetical order)
13
+
14
+ * ashie@clear-code.com
15
+ * fujimoto@clear-code.com
16
+ * hatake@calyptia.com
17
+ * hayashi@clear-code.com
18
+
data/fluentd.gemspec CHANGED
@@ -22,12 +22,12 @@ Gem::Specification.new do |gem|
22
22
  gem.add_runtime_dependency("msgpack", [">= 1.3.1", "< 2.0.0"])
23
23
  gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
24
24
  gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
25
- gem.add_runtime_dependency("serverengine", [">= 2.2.2", "< 3.0.0"])
26
- gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.8.0"])
25
+ gem.add_runtime_dependency("serverengine", [">= 2.2.5", "< 3.0.0"])
26
+ gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.9.0"])
27
27
  gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
28
28
  gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
29
29
  gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])
30
- gem.add_runtime_dependency("strptime", [">= 0.2.2", "< 1.0.0"])
30
+ gem.add_runtime_dependency("strptime", [">= 0.2.4", "< 1.0.0"])
31
31
  gem.add_runtime_dependency("webrick", [">= 1.4.2", "< 1.8.0"])
32
32
 
33
33
  # build gem for a certain platform. see also Rakefile
@@ -35,7 +35,7 @@ Gem::Specification.new do |gem|
35
35
  gem.platform = fake_platform unless fake_platform.empty?
36
36
  if /mswin|mingw/ =~ fake_platform || (/mswin|mingw/ =~ RUBY_PLATFORM && fake_platform.empty?)
37
37
  gem.add_runtime_dependency("win32-api", [">= 1.10", "< 2.0.0"])
38
- gem.add_runtime_dependency("win32-service", ["~> 2.2.0"])
38
+ gem.add_runtime_dependency("win32-service", ["~> 2.3.0"])
39
39
  gem.add_runtime_dependency("win32-ipc", ["~> 0.7.0"])
40
40
  gem.add_runtime_dependency("win32-event", ["~> 0.6.3"])
41
41
  gem.add_runtime_dependency("windows-pr", ["~> 1.2.6"])
@@ -51,5 +51,6 @@ Gem::Specification.new do |gem|
51
51
  gem.add_development_dependency("test-unit", ["~> 3.3"])
52
52
  gem.add_development_dependency("test-unit-rr", ["~> 1.0"])
53
53
  gem.add_development_dependency("oj", [">= 2.14", "< 4"])
54
+ gem.add_development_dependency("async", "~> 1.23")
54
55
  gem.add_development_dependency("async-http", ">= 0.50.0")
55
56
  end
@@ -35,6 +35,7 @@ format = 'json'
35
35
  message_key = 'message'
36
36
  time_as_integer = false
37
37
  retry_limit = 5
38
+ event_time = nil
38
39
 
39
40
  op.on('-p', '--port PORT', "fluent tcp port (default: #{port})", Integer) {|i|
40
41
  port = i
@@ -80,6 +81,10 @@ op.on('--retry-limit N', "Specify the number of retry limit (default: #{retry_li
80
81
  retry_limit = n
81
82
  }
82
83
 
84
+ op.on('--event-time TIME_STRING', "Specify the time expression string (default: nil)") {|v|
85
+ event_time = v
86
+ }
87
+
83
88
  singleton_class.module_eval do
84
89
  define_method(:usage) do |msg|
85
90
  puts op.to_s
@@ -134,7 +139,7 @@ class Writer
134
139
  end
135
140
  end
136
141
 
137
- def initialize(tag, connector, time_as_integer: false, retry_limit: 5)
142
+ def initialize(tag, connector, time_as_integer: false, retry_limit: 5, event_time: nil)
138
143
  @tag = tag
139
144
  @connector = connector
140
145
  @socket = false
@@ -148,6 +153,7 @@ class Writer
148
153
  @retry_wait = 1
149
154
  @retry_limit = retry_limit
150
155
  @time_as_integer = time_as_integer
156
+ @event_time = event_time
151
157
 
152
158
  super()
153
159
  end
@@ -166,7 +172,11 @@ class Writer
166
172
  end
167
173
  end
168
174
 
169
- time = Fluent::EventTime.now
175
+ time = if @event_time
176
+ Fluent::EventTime.parse(@event_time)
177
+ else
178
+ Fluent::EventTime.now
179
+ end
170
180
  time = time.to_i if @time_as_integer
171
181
  entry = if secondary_record?(record)
172
182
  # Even though secondary contains Fluent::EventTime in record,
@@ -309,7 +319,7 @@ else
309
319
  }
310
320
  end
311
321
 
312
- w = Writer.new(tag, connector, time_as_integer: time_as_integer, retry_limit: retry_limit)
322
+ w = Writer.new(tag, connector, time_as_integer: time_as_integer, retry_limit: retry_limit, event_time: event_time)
313
323
  w.start
314
324
 
315
325
  case format
@@ -29,4 +29,16 @@ module Fluent
29
29
 
30
30
  class SetDefault < Exception
31
31
  end
32
+
33
+ class NotFoundPluginError < ConfigError
34
+ attr_reader :type, :kind
35
+
36
+ def initialize(msg, type: nil, kind: nil)
37
+ @msg = msg
38
+ @type = type
39
+ @kind = kind
40
+
41
+ super(msg)
42
+ end
43
+ end
32
44
  end
data/lib/fluent/env.rb CHANGED
@@ -33,4 +33,8 @@ module Fluent
33
33
  def self.linux?
34
34
  /linux/ === RUBY_PLATFORM
35
35
  end
36
+
37
+ def self.macos?
38
+ /darwin/ =~ RUBY_PLATFORM
39
+ end
36
40
  end
@@ -72,8 +72,8 @@ module Fluent
72
72
 
73
73
  def string_safe_encoding(str)
74
74
  unless str.valid_encoding?
75
- log.info "invalid byte sequence is replaced in `#{str}`" if self.respond_to?(:log)
76
75
  str = str.scrub('?')
76
+ log.info "invalid byte sequence is replaced in `#{str}`" if self.respond_to?(:log)
77
77
  end
78
78
  yield str
79
79
  end
@@ -39,8 +39,8 @@ module Fluent
39
39
  config_set_default :chunk_limit_size, DEFAULT_CHUNK_LIMIT_SIZE
40
40
  config_set_default :total_limit_size, DEFAULT_TOTAL_LIMIT_SIZE
41
41
 
42
- config_param :file_permission, :string, default: nil # '0644'
43
- config_param :dir_permission, :string, default: nil # '0755'
42
+ config_param :file_permission, :string, default: nil # '0644' (Fluent::DEFAULT_FILE_PERMISSION)
43
+ config_param :dir_permission, :string, default: nil # '0755' (Fluent::DEFAULT_DIR_PERMISSION)
44
44
 
45
45
  def initialize
46
46
  super
@@ -332,12 +332,14 @@ module Fluent
332
332
  unstaged_chunks = {} # metadata => [chunk, chunk, ...]
333
333
  chunks_to_enqueue = []
334
334
  staged_bytesizes_by_chunk = {}
335
+ # track internal BufferChunkOverflowError in write_step_by_step
336
+ buffer_chunk_overflow_errors = []
335
337
 
336
338
  begin
337
339
  # sort metadata to get lock of chunks in same order with other threads
338
340
  metadata_and_data.keys.sort.each do |metadata|
339
341
  data = metadata_and_data[metadata]
340
- write_once(metadata, data, format: format, size: size) do |chunk, adding_bytesize|
342
+ write_once(metadata, data, format: format, size: size) do |chunk, adding_bytesize, error|
341
343
  chunk.mon_enter # add lock to prevent to be committed/rollbacked from other threads
342
344
  operated_chunks << chunk
343
345
  if chunk.staged?
@@ -352,6 +354,9 @@ module Fluent
352
354
  unstaged_chunks[metadata] ||= []
353
355
  unstaged_chunks[metadata] << chunk
354
356
  end
357
+ if error && !error.empty?
358
+ buffer_chunk_overflow_errors << error
359
+ end
355
360
  end
356
361
  end
357
362
 
@@ -444,6 +449,10 @@ module Fluent
444
449
  end
445
450
  chunk.mon_exit rescue nil # this may raise ThreadError for chunks already committed
446
451
  end
452
+ unless buffer_chunk_overflow_errors.empty?
453
+ # Notify delayed BufferChunkOverflowError here
454
+ raise BufferChunkOverflowError, buffer_chunk_overflow_errors.join(", ")
455
+ end
447
456
  end
448
457
  end
449
458
 
@@ -716,6 +725,7 @@ module Fluent
716
725
 
717
726
  def write_step_by_step(metadata, data, format, splits_count, &block)
718
727
  splits = []
728
+ errors = []
719
729
  if splits_count > data.size
720
730
  splits_count = data.size
721
731
  end
@@ -757,22 +767,65 @@ module Fluent
757
767
  raise ShouldRetry unless chunk.writable?
758
768
  staged_chunk_used = true if chunk.staged?
759
769
 
760
- original_bytesize = chunk.bytesize
770
+ original_bytesize = committed_bytesize = chunk.bytesize
761
771
  begin
762
772
  while writing_splits_index < splits.size
763
773
  split = splits[writing_splits_index]
774
+ formatted_split = format ? format.call(split) : nil
775
+
776
+ if split.size == 1 # Check BufferChunkOverflowError
777
+ determined_bytesize = nil
778
+ if @compress != :text
779
+ determined_bytesize = nil
780
+ elsif formatted_split
781
+ determined_bytesize = formatted_split.bytesize
782
+ elsif split.first.respond_to?(:bytesize)
783
+ determined_bytesize = split.first.bytesize
784
+ end
785
+
786
+ if determined_bytesize && determined_bytesize > @chunk_limit_size
787
+ # It is a obvious case that BufferChunkOverflowError should be raised here.
788
+ # But if it raises here, already processed 'split' or
789
+ # the proceeding 'split' will be lost completely.
790
+ # So it is a last resort to delay raising such a exception
791
+ errors << "a #{determined_bytesize} bytes record (nth: #{writing_splits_index}) is larger than buffer chunk limit size (#{@chunk_limit_size})"
792
+ writing_splits_index += 1
793
+ next
794
+ end
795
+
796
+ if determined_bytesize.nil? || chunk.bytesize + determined_bytesize > @chunk_limit_size
797
+ # The split will (might) cause size over so keep already processed
798
+ # 'split' content here (allow performance regression a bit).
799
+ chunk.commit
800
+ committed_bytesize = chunk.bytesize
801
+ end
802
+ end
803
+
764
804
  if format
765
- chunk.concat(format.call(split), split.size)
805
+ chunk.concat(formatted_split, split.size)
766
806
  else
767
807
  chunk.append(split, compress: @compress)
768
808
  end
809
+ adding_bytes = chunk.bytesize - committed_bytesize
769
810
 
770
811
  if chunk_size_over?(chunk) # split size is larger than difference between size_full? and size_over?
771
812
  chunk.rollback
772
-
773
- if split.size == 1 && original_bytesize == 0
774
- big_record_size = format ? format.call(split).bytesize : split.first.bytesize
775
- raise BufferChunkOverflowError, "a #{big_record_size}bytes record is larger than buffer chunk limit size"
813
+ committed_bytesize = chunk.bytesize
814
+
815
+ if split.size == 1 # Check BufferChunkOverflowError again
816
+ if adding_bytes > @chunk_limit_size
817
+ errors << "concatenated/appended a #{adding_bytes} bytes record (nth: #{writing_splits_index}) is larger than buffer chunk limit size (#{@chunk_limit_size})"
818
+ writing_splits_index += 1
819
+ next
820
+ else
821
+ # As already processed content is kept after rollback, then unstaged chunk should be queued.
822
+ # After that, re-process current split again.
823
+ # New chunk should be allocated, to do it, modify @stage and so on.
824
+ synchronize { @stage.delete(modified_metadata) }
825
+ staged_chunk_used = false
826
+ chunk.unstaged!
827
+ break
828
+ end
776
829
  end
777
830
 
778
831
  if chunk_size_full?(chunk) || split.size == 1
@@ -795,7 +848,8 @@ module Fluent
795
848
  raise
796
849
  end
797
850
 
798
- block.call(chunk, chunk.bytesize - original_bytesize)
851
+ block.call(chunk, chunk.bytesize - original_bytesize, errors)
852
+ errors = []
799
853
  end
800
854
  end
801
855
  rescue ShouldRetry
@@ -314,8 +314,16 @@ module Fluent::Plugin
314
314
  @parser_json.parse(js) do |_time, record|
315
315
  return nil, record
316
316
  end
317
+ elsif ndjson = params['ndjson']
318
+ events = []
319
+ ndjson.split(/\r?\n/).each do |js|
320
+ @parser_json.parse(js) do |_time, record|
321
+ events.push(record)
322
+ end
323
+ end
324
+ return nil, events
317
325
  else
318
- raise "'json' or 'msgpack' parameter is required"
326
+ raise "'json', 'ndjson' or 'msgpack' parameter is required"
319
327
  end
320
328
  end
321
329
 
@@ -567,6 +575,8 @@ module Fluent::Plugin
567
575
  params['json'] = @body
568
576
  elsif @content_type =~ /^application\/msgpack/
569
577
  params['msgpack'] = @body
578
+ elsif @content_type =~ /^application\/x-ndjson/
579
+ params['ndjson'] = @body
570
580
  end
571
581
  path_info = uri.path
572
582
 
@@ -59,6 +59,7 @@ module Fluent::Plugin
59
59
  @ignore_list = []
60
60
  @shutdown_start_time = nil
61
61
  @metrics = nil
62
+ @startup = true
62
63
  end
63
64
 
64
65
  desc 'The paths to read. Multiple paths can be specified, separated by comma.'
@@ -112,6 +113,8 @@ module Fluent::Plugin
112
113
  config_param :path_timezone, :string, default: nil
113
114
  desc 'Follow inodes instead of following file names. Guarantees more stable delivery and allows to use * in path pattern with rotating files'
114
115
  config_param :follow_inodes, :bool, default: false
116
+ desc 'Maximum length of line. The longer line is just skipped.'
117
+ config_param :max_line_size, :size, default: nil
115
118
 
116
119
  config_section :parse, required: false, multi: true, init: true, param_name: :parser_configs do
117
120
  config_argument :usage, :string, default: 'in_tail_parser'
@@ -369,19 +372,25 @@ module Fluent::Plugin
369
372
  def refresh_watchers
370
373
  target_paths_hash = expand_paths
371
374
  existence_paths_hash = existence_path
372
-
373
- log.debug { "tailing paths: target = #{target_paths.join(",")} | existing = #{existence_paths.join(",")}" }
375
+
376
+ log.debug {
377
+ target_paths_str = target_paths_hash.collect { |key, target_info| target_info.path }.join(",")
378
+ existence_paths_str = existence_paths_hash.collect { |key, target_info| target_info.path }.join(",")
379
+ "tailing paths: target = #{target_paths_str} | existing = #{existence_paths_str}"
380
+ }
374
381
 
375
382
  unwatched_hash = existence_paths_hash.reject {|key, value| target_paths_hash.key?(key)}
376
383
  added_hash = target_paths_hash.reject {|key, value| existence_paths_hash.key?(key)}
377
384
 
378
385
  stop_watchers(unwatched_hash, immediate: false, unwatched: true) unless unwatched_hash.empty?
379
386
  start_watchers(added_hash) unless added_hash.empty?
387
+ @startup = false if @startup
380
388
  end
381
389
 
382
390
  def setup_watcher(target_info, pe)
383
391
  line_buffer_timer_flusher = @multiline_mode ? TailWatcher::LineBufferTimerFlusher.new(log, @multiline_flush_interval, &method(:flush_buffer)) : nil
384
- tw = TailWatcher.new(target_info, pe, log, @read_from_head, @follow_inodes, method(:update_watcher), line_buffer_timer_flusher, method(:io_handler), @metrics)
392
+ read_from_head = !@startup || @read_from_head
393
+ tw = TailWatcher.new(target_info, pe, log, read_from_head, @follow_inodes, method(:update_watcher), line_buffer_timer_flusher, method(:io_handler), @metrics)
385
394
 
386
395
  if @enable_watch_timer
387
396
  tt = TimerTrigger.new(1, log) { tw.on_notify }
@@ -389,7 +398,7 @@ module Fluent::Plugin
389
398
  end
390
399
 
391
400
  if @enable_stat_watcher
392
- tt = StatWatcher.new(path, log) { tw.on_notify }
401
+ tt = StatWatcher.new(target_info.path, log) { tw.on_notify }
393
402
  tw.register_watcher(tt)
394
403
  end
395
404
 
@@ -587,6 +596,14 @@ module Fluent::Plugin
587
596
 
588
597
  # @return true if no error or unrecoverable error happens in emit action. false if got BufferOverflowError
589
598
  def receive_lines(lines, tail_watcher)
599
+ lines = lines.reject do |line|
600
+ skip_line = @max_line_size ? line.bytesize > @max_line_size : false
601
+ if skip_line
602
+ log.warn "received line length is longer than #{@max_line_size}"
603
+ log.debug "skipped line: #{line.chomp}"
604
+ end
605
+ skip_line
606
+ end
590
607
  es = @receive_handler.call(lines, tail_watcher)
591
608
  unless es.empty?
592
609
  tag = if @tag_prefix || @tag_suffix
@@ -181,6 +181,13 @@ module Fluent::Plugin
181
181
  @dir_perm = system_config.dir_permission || Fluent::DEFAULT_DIR_PERMISSION
182
182
  @file_perm = system_config.file_permission || Fluent::DEFAULT_FILE_PERMISSION
183
183
  @need_lock = system_config.workers > 1
184
+
185
+ # https://github.com/fluent/fluentd/issues/3569
186
+ @need_ruby_on_macos_workaround = false
187
+ if @append && Fluent.macos?
188
+ condition = Gem::Dependency.new('', [">= 2.7.0", "< 3.1.0"])
189
+ @need_ruby_on_macos_workaround = true if condition.match?('', RUBY_VERSION)
190
+ end
184
191
  end
185
192
 
186
193
  def multi_workers_ready?
@@ -223,7 +230,12 @@ module Fluent::Plugin
223
230
 
224
231
  def write_without_compression(path, chunk)
225
232
  File.open(path, "ab", @file_perm) do |f|
226
- chunk.write_to(f)
233
+ if @need_ruby_on_macos_workaround
234
+ content = chunk.read()
235
+ f.puts content
236
+ else
237
+ chunk.write_to(f)
238
+ end
227
239
  end
228
240
  end
229
241