fluentd 1.13.1-x86-mingw32 → 1.13.2-x86-mingw32

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of fluentd might be problematic. Click here for more details.

checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: dfac89a6bc983358dc19675fddc0c6e28182ff1ee3ac07b5d4dde7d25473571f
4
- data.tar.gz: 693676284c38b07e959db0f571919654feca14bdc3488861d599722ddddd95a0
3
+ metadata.gz: 1c036a9bfd817725bcd7d2224018f35c242500e62e713a6549e9cebb81266662
4
+ data.tar.gz: 103d101ffc4f5be2af12b9e36b4f9ba6019213fd44218758bde892f90ad13551
5
5
  SHA512:
6
- metadata.gz: b16a11e6f23e56c87cea1c4f9ae0916f94988ad562b873e83e6ddfa1b54efb1fda304730d236036f1402e5302d3e1498e27d8616a7fa85b23726cbe187c645d8
7
- data.tar.gz: d0dab564fd15451fbf664818a85cdde1724da2b5677f20ca4d9a4627ee57c5eb8f228bb1b07e954fa3c385978615278ba67e585aa409a8d5ddaaa4884075ca5e
6
+ metadata.gz: e8c51c88bf9b5992afd5095f5c742a23dc024dd4cb3a171f72367c4938d94f53e9a4584dcd71b68009f0ce8b359c450943200c3cb2d567e65f38c99b4db92220
7
+ data.tar.gz: b0917164c636070fb72682c5522e69ebf2dc79c017e9a53362eee6b6c1cb11b7f9b8061c619ef716b38b43598eac858e0805577fd0e108b44920a85c2f8d4cee
@@ -0,0 +1,69 @@
1
+ name: Bug Report
2
+ description: Create a report with a procedure for reproducing the bug
3
+ body:
4
+ - type: markdown
5
+ attributes:
6
+ value: |
7
+ Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
8
+ - type: textarea
9
+ id: description
10
+ attributes:
11
+ label: Describe the bug
12
+ description: A clear and concise description of what the bug is
13
+ validations:
14
+ required: true
15
+ - type: textarea
16
+ id: reproduce
17
+ attributes:
18
+ label: To Reproduce
19
+ description: Steps to reproduce the behavior
20
+ validations:
21
+ required: true
22
+ - type: textarea
23
+ id: expected
24
+ attributes:
25
+ label: Expected behavior
26
+ description: A clear and concise description of what you expected to happen
27
+ validations:
28
+ required: true
29
+ - type: textarea
30
+ id: environment
31
+ attributes:
32
+ label: Your Environment
33
+ description: |
34
+ - Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
35
+ - Operating system: `cat /etc/os-release`
36
+ - Kernel version: `uname -r`
37
+
38
+ Tip: If you hit the problem with older fluentd version, try latest version first.
39
+ value: |
40
+ - Fluentd version:
41
+ - TD Agent version:
42
+ - Operating system:
43
+ - Kernel version:
44
+ render: markdown
45
+ validations:
46
+ required: true
47
+ - type: textarea
48
+ id: configuration
49
+ attributes:
50
+ label: Your Configuration
51
+ description: |
52
+ Write your configuration here. Minimum reproducible fluentd.conf is recommended.
53
+ validations:
54
+ required: true
55
+ - type: textarea
56
+ id: logs
57
+ attributes:
58
+ label: Your Error Log
59
+ description: Write your ALL error log here
60
+ render: shell
61
+ validations:
62
+ required: true
63
+ - type: textarea
64
+ id: addtional-context
65
+ attributes:
66
+ label: Additional context
67
+ description: Add any other context about the problem here.
68
+ validations:
69
+ required: false
@@ -0,0 +1,38 @@
1
+ name: Feature request
2
+ description: Suggest an idea for this project
3
+ body:
4
+ - type: markdown
5
+ attributes:
6
+ value: |
7
+ Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
8
+ - type: textarea
9
+ id: description
10
+ attributes:
11
+ label: Is your feature request related to a problem? Please describe.
12
+ description: |
13
+ A clear and concise description of what the problem is.
14
+ Ex. I'm always frustrated when [...]
15
+ validations:
16
+ required: true
17
+ - type: textarea
18
+ id: solution
19
+ attributes:
20
+ label: Describe the solution you'd like
21
+ description: A clear and concise description of what you want to happen.
22
+ validations:
23
+ required: true
24
+ - type: textarea
25
+ id: alternative
26
+ attributes:
27
+ label: Describe alternatives you've considered
28
+ description: A clear and concise description of any alternative solutions or features you've considered.
29
+ validations:
30
+ required: true
31
+ - type: textarea
32
+ id: addtional-context
33
+ attributes:
34
+ label: Additional context
35
+ description: Add any other context or screenshots about the feature request here.
36
+ validations:
37
+ required: false
38
+
data/CHANGELOG.md CHANGED
@@ -1,3 +1,40 @@
1
+ # v1.13.2
2
+
3
+ ## Release v1.13.2 - 2021/07/12
4
+
5
+ ### Enhancement
6
+
7
+ * fluent-plugin-generate: Storage plugin was supported.
8
+ https://github.com/fluent/fluentd/pull/3426
9
+ * parser_json: Added support to customize configuration of oj options.
10
+ Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`,
11
+ `FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment
12
+ variable to configure it.
13
+ https://github.com/fluent/fluentd/pull/3315
14
+
15
+ ### Bug fix
16
+
17
+ * binlog_reader: Fixed a crash bug by missing "fluent/env" dependency.
18
+ https://github.com/fluent/fluentd/pull/3443
19
+ * Fixed a crash bug on outputting log at the early stage when parsing
20
+ config file. This is a regression since v1.13.0. If you use invalid
21
+ '@' prefix parameter, remove it as a workaround.
22
+ https://github.com/fluent/fluentd/pull/3451
23
+ * in_tail: Fixed a bug that when rotation is occurred, remaining lines
24
+ will be discarded if the throttling feature is enabled.
25
+ https://github.com/fluent/fluentd/pull/3390
26
+ * fluent-plugin-generate: Fixed a crash bug during gemspec generation.
27
+ It was unexpectedly introduced by #3305, thus this bug was a
28
+ regression since 1.12.3.
29
+ https://github.com/fluent/fluentd/pull/3444
30
+
31
+ ### Misc
32
+
33
+ * Fixed the runtime dependency version of http_parse.rb to 0.7.0.
34
+ It was fixed because false positive detection is occurred frequently
35
+ by security scanning tools.
36
+ https://github.com/fluent/fluentd/pull/3450
37
+
1
38
  # v1.13.1
2
39
 
3
40
  ## Release v1.13.1 - 2021/06/25
data/README.md CHANGED
@@ -25,9 +25,9 @@ An event consists of *tag*, *time* and *record*. Tag is a string separated with
25
25
 
26
26
  Use Case | Description | Diagram
27
27
  -------- | ------------|:---------:
28
- Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/assets/img/recipes/elasticsearch-s3-fluentd.png" height="150"/>
28
+ Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/images/recipes/elasticsearch-s3-fluentd.png" height="150"/>
29
29
  Syslog Alerting | Fluentd can "grep" for events and send out alerts. | <img src="https://www.fluentd.org/images/syslog-fluentd-alert.png" height="100"/>
30
- Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/assets/img/datasources/asynchronous_logging.png" height="150"/>
30
+ Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/images/datasources/asynchronous_logging.png" height="150"/>
31
31
 
32
32
  ## Quick Start
33
33
 
data/fluentd.gemspec CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |gem|
23
23
  gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
24
24
  gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
25
25
  gem.add_runtime_dependency("serverengine", [">= 2.2.2", "< 3.0.0"])
26
- gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.7.0"])
26
+ gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.8.0"])
27
27
  gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
28
28
  gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
29
29
  gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])
@@ -27,7 +27,7 @@ class FluentPluginGenerator
27
27
  attr_reader :type, :name
28
28
  attr_reader :license_name
29
29
 
30
- SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter"]
30
+ SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter", "storage"]
31
31
 
32
32
  def initialize(argv = ARGV)
33
33
  @argv = argv
@@ -164,8 +164,13 @@ BANNER
164
164
  end
165
165
 
166
166
  def locked_gem_version(gem_name)
167
- d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
168
- d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
167
+ if File.exist?(lock_file_path)
168
+ d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
169
+ d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
170
+ else
171
+ # fallback even though Fluentd is installed without bundler
172
+ Gem::Specification.find_by_name(gem_name).version.version
173
+ end
169
174
  end
170
175
 
171
176
  def rake_version
@@ -177,8 +182,13 @@ BANNER
177
182
  end
178
183
 
179
184
  def bundler_version
180
- d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
181
- d.locked_gems.bundler_version.version
185
+ if File.exist?(lock_file_path)
186
+ d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
187
+ d.locked_gems.bundler_version.version
188
+ else
189
+ # fallback even though Fluentd is installed without bundler
190
+ Gem::Specification.find_by_name("bundler").version.version
191
+ end
182
192
  end
183
193
 
184
194
  def class_name
data/lib/fluent/config.rb CHANGED
@@ -62,7 +62,7 @@ module Fluent
62
62
  Parser.parse(str, fname, basepath)
63
63
  when :ruby
64
64
  require 'fluent/config/dsl'
65
- $log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file")
65
+ $log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file") if $log
66
66
  Config::DSL::Parser.parse(str, File.join(basepath, fname))
67
67
  else
68
68
  raise "[BUG] unknown configuration parser specification:'#{parser}'"
@@ -20,6 +20,10 @@ require 'fluent/config/error'
20
20
 
21
21
  module Fluent
22
22
  module Config
23
+ def self.reformatted_value(type, val, opts = {}, name = nil)
24
+ REFORMAT_VALUE.call(type, val, opts, name)
25
+ end
26
+
23
27
  def self.size_value(str, opts = {}, name = nil)
24
28
  return nil if str.nil?
25
29
 
@@ -104,6 +108,16 @@ module Fluent
104
108
  Config.string_value(val, opts, name)
105
109
  }
106
110
 
111
+ def self.symbol_value(val, opts = {}, name = nil)
112
+ return nil if val.nil? || val.empty?
113
+
114
+ val.delete_prefix(":").to_sym
115
+ end
116
+
117
+ SYMBOL_TYPE = Proc.new { |val, opts = {}, name = nil|
118
+ Config.symbol_value(val, opts, name)
119
+ }
120
+
107
121
  def self.enum_value(val, opts = {}, name = nil)
108
122
  return nil if val.nil?
109
123
 
@@ -176,6 +190,7 @@ module Fluent
176
190
  when :bool then Config.bool_value(value, opts, name)
177
191
  when :time then Config.time_value(value, opts, name)
178
192
  when :regexp then Config.regexp_value(value, opts, name)
193
+ when :symbol then Config.symbol_value(value, opts, name)
179
194
  else
180
195
  raise "unknown type in REFORMAT: #{type}"
181
196
  end
@@ -37,6 +37,7 @@ module Fluent
37
37
  super(strscan, eval_context)
38
38
  @include_basepath = include_basepath
39
39
  @fname = fname
40
+ @logger = defined?($log) ? $log : nil
40
41
  end
41
42
 
42
43
  def parse!
@@ -99,7 +100,7 @@ module Fluent
99
100
 
100
101
  elsif root_element && skip(/(\@include|include)#{SPACING}/)
101
102
  if !prev_match.start_with?('@')
102
- $log.warn "'include' is deprecated. Use '@include' instead"
103
+ @logger.warn "'include' is deprecated. Use '@include' instead" if @logger
103
104
  end
104
105
  parse_include(attrs, elems)
105
106
 
@@ -123,7 +124,7 @@ module Fluent
123
124
  parse_error! "'@' is the system reserved prefix. Don't use '@' prefix parameter in the configuration: #{k}"
124
125
  else
125
126
  # TODO: This is for backward compatibility. It will throw an error in the future.
126
- $log.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}"
127
+ @logger.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}" if @logger
127
128
  end
128
129
  end
129
130
 
data/lib/fluent/env.rb CHANGED
@@ -15,13 +15,14 @@
15
15
  #
16
16
 
17
17
  require 'serverengine/utils'
18
+ require 'fluent/oj_options'
18
19
 
19
20
  module Fluent
20
21
  DEFAULT_CONFIG_PATH = ENV['FLUENT_CONF'] || '/etc/fluent/fluent.conf'
21
22
  DEFAULT_PLUGIN_DIR = ENV['FLUENT_PLUGIN'] || '/etc/fluent/plugin'
22
23
  DEFAULT_SOCKET_PATH = ENV['FLUENT_SOCKET'] || '/var/run/fluent/fluent.sock'
23
24
  DEFAULT_BACKUP_DIR = ENV['FLUENT_BACKUP_DIR'] || '/tmp/fluent'
24
- DEFAULT_OJ_OPTIONS = {bigdecimal_load: :float, mode: :compat, use_to_json: true}
25
+ DEFAULT_OJ_OPTIONS = Fluent::OjOptions.load_env
25
26
  DEFAULT_DIR_PERMISSION = 0755
26
27
  DEFAULT_FILE_PERMISSION = 0644
27
28
 
@@ -0,0 +1,62 @@
1
+ require 'fluent/config/types'
2
+
3
+ module Fluent
4
+ class OjOptions
5
+ OPTIONS = {
6
+ 'bigdecimal_load': :symbol,
7
+ 'max_nesting': :integer,
8
+ 'mode': :symbol,
9
+ 'use_to_json': :bool
10
+ }
11
+
12
+ ALLOWED_VALUES = {
13
+ 'bigdecimal_load': %i[bigdecimal float auto],
14
+ 'mode': %i[strict null compat json rails object custom]
15
+ }
16
+
17
+ DEFAULTS = {
18
+ 'bigdecimal_load': :float,
19
+ 'mode': :compat,
20
+ 'use_to_json': true
21
+ }
22
+
23
+ @@available = false
24
+
25
+ def self.available?
26
+ @@available
27
+ end
28
+
29
+ def self.load_env
30
+ options = self.get_options
31
+ begin
32
+ require 'oj'
33
+ Oj.default_options = options
34
+ @@available = true
35
+ rescue LoadError
36
+ @@available = false
37
+ end
38
+ options
39
+ end
40
+
41
+ private
42
+
43
+ def self.get_options
44
+ options = {}
45
+ DEFAULTS.each { |key, value| options[key] = value }
46
+
47
+ OPTIONS.each do |key, type|
48
+ env_value = ENV["FLUENT_OJ_OPTION_#{key.upcase}"]
49
+ next if env_value.nil?
50
+
51
+ cast_value = Fluent::Config.reformatted_value(OPTIONS[key], env_value, { strict: true })
52
+ next if cast_value.nil?
53
+
54
+ next if ALLOWED_VALUES[key] && !ALLOWED_VALUES[key].include?(cast_value)
55
+
56
+ options[key.to_sym] = cast_value
57
+ end
58
+
59
+ options
60
+ end
61
+ end
62
+ end
@@ -14,6 +14,7 @@
14
14
  # limitations under the License.
15
15
  #
16
16
 
17
+ require 'fluent/env'
17
18
  require 'fluent/plugin/base'
18
19
  require 'fluent/plugin/owned_by_mixin'
19
20
  require 'fluent/time'
@@ -15,7 +15,7 @@
15
15
  #
16
16
 
17
17
  require 'fluent/plugin/formatter'
18
- require 'fluent/env'
18
+ require 'fluent/oj_options'
19
19
 
20
20
  module Fluent
21
21
  module Plugin
@@ -30,12 +30,14 @@ module Fluent
30
30
  def configure(conf)
31
31
  super
32
32
 
33
- begin
34
- raise LoadError unless @json_parser == 'oj'
35
- require 'oj'
36
- Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
37
- @dump_proc = Oj.method(:dump)
38
- rescue LoadError
33
+ if @json_parser == 'oj'
34
+ if Fluent::OjOptions.available?
35
+ @dump_proc = Oj.method(:dump)
36
+ else
37
+ log.info "Oj isn't installed, fallback to Yajl as json parser"
38
+ @dump_proc = Yajl.method(:dump)
39
+ end
40
+ else
39
41
  @dump_proc = Yajl.method(:dump)
40
42
  end
41
43
 
@@ -522,8 +522,25 @@ module Fluent::Plugin
522
522
  def detach_watcher_after_rotate_wait(tw, ino)
523
523
  # Call event_loop_attach/event_loop_detach is high-cost for short-live object.
524
524
  # If this has a problem with large number of files, use @_event_loop directly instead of timer_execute.
525
- timer_execute(:in_tail_close_watcher, @rotate_wait, repeat: false) do
525
+ if @open_on_every_update
526
+ # Detach now because it's already closed, waiting it doesn't make sense.
526
527
  detach_watcher(tw, ino)
528
+ elsif @read_bytes_limit_per_second < 0
529
+ # throttling isn't enabled, just wait @rotate_wait
530
+ timer_execute(:in_tail_close_watcher, @rotate_wait, repeat: false) do
531
+ detach_watcher(tw, ino)
532
+ end
533
+ else
534
+ # When the throttling feature is enabled, it might not reach EOF yet.
535
+ # Should ensure to read all contents before closing it, with keeping throttling.
536
+ start_time_to_wait = Fluent::Clock.now
537
+ timer = timer_execute(:in_tail_close_watcher, 1, repeat: true) do
538
+ elapsed = Fluent::Clock.now - start_time_to_wait
539
+ if tw.eof? && elapsed >= @rotate_wait
540
+ timer.detach
541
+ detach_watcher(tw, ino)
542
+ end
543
+ end
527
544
  end
528
545
  end
529
546
 
@@ -736,6 +753,10 @@ module Fluent::Plugin
736
753
  end
737
754
  end
738
755
 
756
+ def eof?
757
+ @io_handler.eof?
758
+ end
759
+
739
760
  def on_notify
740
761
  begin
741
762
  stat = Fluent::FileWrapper.stat(@path)
@@ -923,6 +944,7 @@ module Fluent::Plugin
923
944
  @shutdown_start_time = nil
924
945
  @shutdown_timeout = SHUTDOWN_TIMEOUT
925
946
  @shutdown_mutex = Mutex.new
947
+ @eof = false
926
948
 
927
949
  @log.info "following tail of #{@path}"
928
950
  end
@@ -949,6 +971,10 @@ module Fluent::Plugin
949
971
  !!@io
950
972
  end
951
973
 
974
+ def eof?
975
+ @eof
976
+ end
977
+
952
978
  private
953
979
 
954
980
  def limit_bytes_per_second_reached?
@@ -989,6 +1015,7 @@ module Fluent::Plugin
989
1015
  while true
990
1016
  @start_reading_time ||= Fluent::Clock.now
991
1017
  data = io.readpartial(BYTES_TO_READ, @iobuf)
1018
+ @eof = false
992
1019
  @number_bytes_read += data.bytesize
993
1020
  @fifo << data
994
1021
  @fifo.read_lines(@lines)
@@ -1005,6 +1032,7 @@ module Fluent::Plugin
1005
1032
  end
1006
1033
  end
1007
1034
  rescue EOFError
1035
+ @eof = true
1008
1036
  end
1009
1037
  end
1010
1038
 
@@ -1042,14 +1070,17 @@ module Fluent::Plugin
1042
1070
  else
1043
1071
  @io ||= open
1044
1072
  yield @io
1073
+ @eof = true if @io.nil?
1045
1074
  end
1046
1075
  rescue WatcherSetupError => e
1047
1076
  close
1077
+ @eof = true
1048
1078
  raise e
1049
1079
  rescue
1050
1080
  @log.error $!.to_s
1051
1081
  @log.error_backtrace
1052
1082
  close
1083
+ @eof = true
1053
1084
  end
1054
1085
  end
1055
1086
 
@@ -15,8 +15,8 @@
15
15
  #
16
16
 
17
17
  require 'fluent/plugin/parser'
18
- require 'fluent/env'
19
18
  require 'fluent/time'
19
+ require 'fluent/oj_options'
20
20
 
21
21
  require 'yajl'
22
22
  require 'json'
@@ -50,8 +50,7 @@ module Fluent
50
50
  def configure_json_parser(name)
51
51
  case name
52
52
  when :oj
53
- require 'oj'
54
- Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
53
+ raise LoadError unless Fluent::OjOptions.available?
55
54
  [Oj.method(:load), Oj::ParseError]
56
55
  when :json then [JSON.method(:load), JSON::ParserError]
57
56
  when :yajl then [Yajl.method(:load), Yajl::ParseError]
@@ -0,0 +1,30 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ require 'fluent/test/driver/base_owned'
18
+
19
+ module Fluent
20
+ module Test
21
+ module Driver
22
+ class Storage < BaseOwned
23
+ def initialize(klass, **kwargs, &block)
24
+ super
25
+ @section_name = "storage"
26
+ end
27
+ end
28
+ end
29
+ end
30
+ end
@@ -16,6 +16,6 @@
16
16
 
17
17
  module Fluent
18
18
 
19
- VERSION = '1.13.1'
19
+ VERSION = '1.13.2'
20
20
 
21
21
  end
@@ -0,0 +1,40 @@
1
+ <%= preamble %>
2
+
3
+ require "fluent/plugin/storage"
4
+
5
+ module Fluent
6
+ module Plugin
7
+ class <%= class_name %> < Fluent::Plugin::Storage
8
+ Fluent::Plugin.register_storage("<%= plugin_name %>", self)
9
+
10
+ def initialize
11
+ super
12
+ end
13
+
14
+ def configure(conf)
15
+ super
16
+ end
17
+
18
+ def load
19
+ end
20
+
21
+ def save
22
+ end
23
+
24
+ def get(key)
25
+ end
26
+
27
+ def fetch(key, defval)
28
+ end
29
+
30
+ def put(key, value)
31
+ end
32
+
33
+ def delete(key)
34
+ end
35
+
36
+ def update(key, &block)
37
+ end
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,18 @@
1
+ require "helper"
2
+ require "fluent/plugin/<%= plugin_filename %>"
3
+
4
+ class <%= class_name %>Test < Test::Unit::TestCase
5
+ setup do
6
+ Fluent::Test.setup
7
+ end
8
+
9
+ test "failure" do
10
+ flunk
11
+ end
12
+
13
+ private
14
+
15
+ def create_driver(conf)
16
+ Fluent::Test::Driver::Storage.new(Fluent::Plugin::<%= class_name %>).configure(conf)
17
+ end
18
+ end
@@ -29,7 +29,8 @@ class TestFluentPluginGenerator < Test::Unit::TestCase
29
29
  output: ["output", "out"],
30
30
  filter: ["filter", "filter"],
31
31
  parser: ["parser", "parser"],
32
- formatter: ["formatter", "formatter"])
32
+ formatter: ["formatter", "formatter"],
33
+ storage: ["storage", "storage"])
33
34
  test "generate plugin" do |(type, part)|
34
35
  generator = FluentPluginGenerator.new([type, "fake"])
35
36
  stub_git_process(generator)
@@ -180,6 +180,13 @@ class TestConfigTypes < ::Test::Unit::TestCase
180
180
  assert_equal Encoding::UTF_8, actual.encoding
181
181
  end
182
182
 
183
+ data('starts_with_semicolon' => [:conor, ':conor'],
184
+ 'simple_string' => [:conor, 'conor'],
185
+ 'empty_string' => [nil, ''])
186
+ test 'symbol' do |(expected, val)|
187
+ assert_equal Config::SYMBOL_TYPE.call(val, {}), expected
188
+ end
189
+
183
190
  data("val" => [:val, 'val'],
184
191
  "v" => [:v, 'v'],
185
192
  "value" => [:value, 'value'])
@@ -456,6 +456,91 @@ class TailInputTest < Test::Unit::TestCase
456
456
  assert_equal([], d.events)
457
457
  end
458
458
  end
459
+
460
+ sub_test_case "EOF with reads_bytes_per_second" do
461
+ def test_longer_than_rotate_wait
462
+ limit_bytes = 8192
463
+ num_lines = 1024 * 3
464
+ msg = "08bytes"
465
+
466
+ File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
467
+ f.write("#{msg}\n" * num_lines)
468
+ end
469
+
470
+ config = CONFIG_READ_FROM_HEAD +
471
+ SINGLE_LINE_CONFIG +
472
+ config_element("", "", {
473
+ "read_bytes_limit_per_second" => limit_bytes,
474
+ "rotate_wait" => 0.1,
475
+ "refresh_interval" => 0.5,
476
+ })
477
+
478
+ rotated = false
479
+ d = create_driver(config)
480
+ d.run(timeout: 10) do
481
+ while d.events.size < num_lines do
482
+ if d.events.size > 0 && !rotated
483
+ cleanup_file("#{TMP_DIR}/tail.txt")
484
+ FileUtils.touch("#{TMP_DIR}/tail.txt")
485
+ rotated = true
486
+ end
487
+ sleep 0.3
488
+ end
489
+ end
490
+
491
+ assert_equal(num_lines,
492
+ d.events.count do |event|
493
+ event[2]["message"] == msg
494
+ end)
495
+ end
496
+
497
+ def test_shorter_than_rotate_wait
498
+ limit_bytes = 8192
499
+ num_lines = 1024 * 2
500
+ msg = "08bytes"
501
+
502
+ File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
503
+ f.write("#{msg}\n" * num_lines)
504
+ end
505
+
506
+ config = CONFIG_READ_FROM_HEAD +
507
+ SINGLE_LINE_CONFIG +
508
+ config_element("", "", {
509
+ "read_bytes_limit_per_second" => limit_bytes,
510
+ "rotate_wait" => 2,
511
+ "refresh_interval" => 0.5,
512
+ })
513
+
514
+ start_time = Fluent::Clock.now
515
+ rotated = false
516
+ detached = false
517
+ d = create_driver(config)
518
+ mock.proxy(d.instance).setup_watcher(anything, anything) do |tw|
519
+ mock.proxy(tw).detach(anything) do |v|
520
+ detached = true
521
+ v
522
+ end
523
+ tw
524
+ end.twice
525
+
526
+ d.run(timeout: 10) do
527
+ until detached do
528
+ if d.events.size > 0 && !rotated
529
+ cleanup_file("#{TMP_DIR}/tail.txt")
530
+ FileUtils.touch("#{TMP_DIR}/tail.txt")
531
+ rotated = true
532
+ end
533
+ sleep 0.3
534
+ end
535
+ end
536
+
537
+ assert_true(Fluent::Clock.now - start_time > 2)
538
+ assert_equal(num_lines,
539
+ d.events.count do |event|
540
+ event[2]["message"] == msg
541
+ end)
542
+ end
543
+ end
459
544
  end
460
545
 
461
546
  data(flat: CONFIG_READ_FROM_HEAD + SINGLE_LINE_CONFIG,
@@ -64,8 +64,8 @@ class EventTimeTest < Test::Unit::TestCase
64
64
 
65
65
  test 'Oj.dump' do
66
66
  time = Fluent::EventTime.new(100)
67
- require 'fluent/env'
68
- Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
67
+ require 'fluent/oj_options'
68
+ Fluent::OjOptions.load_env
69
69
  assert_equal('{"time":100}', Oj.dump({'time' => time}))
70
70
  assert_equal('["tag",100,{"key":"value"}]', Oj.dump(["tag", time, {"key" => "value"}], mode: :compat))
71
71
  end
@@ -0,0 +1,55 @@
1
+ require_relative 'helper'
2
+ require 'fluent/test'
3
+ require 'fluent/oj_options'
4
+
5
+ class OjOptionsTest < ::Test::Unit::TestCase
6
+ begin
7
+ require 'oj'
8
+ @@oj_is_avaibale = true
9
+ rescue LoadError
10
+ @@oj_is_avaibale = false
11
+ end
12
+
13
+ setup do
14
+ @orig_env = {}
15
+ ENV.each do |key, value|
16
+ @orig_env[key] = value if key.start_with?("FLUENT_OJ_OPTION_")
17
+ end
18
+ end
19
+
20
+ teardown do
21
+ ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
22
+ @orig_env.each { |key, value| ENV[key] = value }
23
+ end
24
+
25
+ test "available?" do
26
+ assert_equal(@@oj_is_avaibale, Fluent::OjOptions.available?)
27
+ end
28
+
29
+ sub_test_case "set by environment variable" do
30
+ test "when no env vars set, returns default options" do
31
+ ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
32
+ defaults = Fluent::OjOptions::DEFAULTS
33
+ assert_equal(defaults, Fluent::OjOptions.load_env)
34
+ assert_equal(defaults, Oj.default_options.slice(*defaults.keys)) if @@oj_is_avaibale
35
+ end
36
+
37
+ test "valid env var passed with valid value, default is overridden" do
38
+ ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":bigdecimal"
39
+ assert_equal(:bigdecimal, Fluent::OjOptions.load_env[:bigdecimal_load])
40
+ assert_equal(:bigdecimal, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
41
+ end
42
+
43
+ test "valid env var passed with invalid value, default is not overriden" do
44
+ ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":conor"
45
+ assert_equal(:float, Fluent::OjOptions.load_env[:bigdecimal_load])
46
+ assert_equal(:float, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
47
+ end
48
+
49
+ test "invalid env var passed, nothing done with it" do
50
+ ENV["FLUENT_OJ_OPTION_CONOR"] = ":conor"
51
+ assert_equal(nil, Fluent::OjOptions.load_env[:conor])
52
+ assert_equal(nil, Oj.default_options[:conor]) if @@oj_is_avaibale
53
+ end
54
+ end
55
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluentd
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.13.1
4
+ version: 1.13.2
5
5
  platform: x86-mingw32
6
6
  authors:
7
7
  - Sadayuki Furuhashi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-06-25 00:00:00.000000000 Z
11
+ date: 2021-07-12 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -107,7 +107,7 @@ dependencies:
107
107
  version: 0.5.1
108
108
  - - "<"
109
109
  - !ruby/object:Gem::Version
110
- version: 0.7.0
110
+ version: 0.8.0
111
111
  type: :runtime
112
112
  prerelease: false
113
113
  version_requirements: !ruby/object:Gem::Requirement
@@ -117,7 +117,7 @@ dependencies:
117
117
  version: 0.5.1
118
118
  - - "<"
119
119
  - !ruby/object:Gem::Version
120
- version: 0.7.0
120
+ version: 0.8.0
121
121
  - !ruby/object:Gem::Dependency
122
122
  name: sigdump
123
123
  requirement: !ruby/object:Gem::Requirement
@@ -463,9 +463,9 @@ files:
463
463
  - ".deepsource.toml"
464
464
  - ".drone.yml"
465
465
  - ".github/ISSUE_TEMPLATE.md"
466
- - ".github/ISSUE_TEMPLATE/bug_report.md"
466
+ - ".github/ISSUE_TEMPLATE/bug_report.yaml"
467
467
  - ".github/ISSUE_TEMPLATE/config.yml"
468
- - ".github/ISSUE_TEMPLATE/feature_request.md"
468
+ - ".github/ISSUE_TEMPLATE/feature_request.yaml"
469
469
  - ".github/PULL_REQUEST_TEMPLATE.md"
470
470
  - ".github/workflows/issue-auto-closer.yml"
471
471
  - ".github/workflows/linux-test.yaml"
@@ -609,6 +609,7 @@ files:
609
609
  - lib/fluent/match.rb
610
610
  - lib/fluent/mixin.rb
611
611
  - lib/fluent/msgpack_factory.rb
612
+ - lib/fluent/oj_options.rb
612
613
  - lib/fluent/output.rb
613
614
  - lib/fluent/output_chain.rb
614
615
  - lib/fluent/parser.rb
@@ -754,6 +755,7 @@ files:
754
755
  - lib/fluent/test/driver/multi_output.rb
755
756
  - lib/fluent/test/driver/output.rb
756
757
  - lib/fluent/test/driver/parser.rb
758
+ - lib/fluent/test/driver/storage.rb
757
759
  - lib/fluent/test/driver/test_event_router.rb
758
760
  - lib/fluent/test/filter_test.rb
759
761
  - lib/fluent/test/formatter_test.rb
@@ -779,12 +781,14 @@ files:
779
781
  - templates/new_gem/lib/fluent/plugin/input.rb.erb
780
782
  - templates/new_gem/lib/fluent/plugin/output.rb.erb
781
783
  - templates/new_gem/lib/fluent/plugin/parser.rb.erb
784
+ - templates/new_gem/lib/fluent/plugin/storage.rb.erb
782
785
  - templates/new_gem/test/helper.rb.erb
783
786
  - templates/new_gem/test/plugin/test_filter.rb.erb
784
787
  - templates/new_gem/test/plugin/test_formatter.rb.erb
785
788
  - templates/new_gem/test/plugin/test_input.rb.erb
786
789
  - templates/new_gem/test/plugin/test_output.rb.erb
787
790
  - templates/new_gem/test/plugin/test_parser.rb.erb
791
+ - templates/new_gem/test/plugin/test_storage.rb.erb
788
792
  - templates/plugin_config_formatter/param.md-compact.erb
789
793
  - templates/plugin_config_formatter/param.md-table.erb
790
794
  - templates/plugin_config_formatter/param.md.erb
@@ -989,6 +993,7 @@ files:
989
993
  - test/test_match.rb
990
994
  - test/test_mixin.rb
991
995
  - test/test_msgpack_factory.rb
996
+ - test/test_oj_options.rb
992
997
  - test/test_output.rb
993
998
  - test/test_plugin.rb
994
999
  - test/test_plugin_classes.rb
@@ -1228,6 +1233,7 @@ test_files:
1228
1233
  - test/test_match.rb
1229
1234
  - test/test_mixin.rb
1230
1235
  - test/test_msgpack_factory.rb
1236
+ - test/test_oj_options.rb
1231
1237
  - test/test_output.rb
1232
1238
  - test/test_plugin.rb
1233
1239
  - test/test_plugin_classes.rb
@@ -1,40 +0,0 @@
1
- ---
2
- name: Bug Report
3
- about: Create a report with a procedure for reproducing the bug
4
-
5
- ---
6
-
7
- Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
8
-
9
- **Describe the bug**
10
- <!-- A clear and concise description of what the bug is. -->
11
-
12
- **To Reproduce**
13
- <!-- Steps to reproduce the behavior: -->
14
-
15
- **Expected behavior**
16
- <!-- A clear and concise description of what you expected to happen. -->
17
-
18
- **Your Environment**
19
-
20
- - Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
21
- - Operating system: `cat /etc/os-release`
22
- - Kernel version: `uname -r`
23
-
24
- If you hit the problem with older fluentd version, try latest version first.
25
-
26
- **Your Configuration**
27
-
28
- ```
29
- <!-- Write your configuration here -->
30
- ```
31
-
32
- **Your Error Log**
33
-
34
- ```
35
- <!-- Write your **ALL** error log here -->
36
- ```
37
-
38
- **Additional context**
39
-
40
- <!-- Add any other context about the problem here. -->
@@ -1,23 +0,0 @@
1
- ---
2
- name: Feature request
3
- about: Suggest an idea for this project
4
-
5
- ---
6
-
7
- Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
8
-
9
- **Is your feature request related to a problem? Please describe.**
10
-
11
- <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
12
-
13
- **Describe the solution you'd like**
14
-
15
- <!-- A clear and concise description of what you want to happen. -->
16
-
17
- **Describe alternatives you've considered**
18
-
19
- <!-- A clear and concise description of any alternative solutions or features you've considered. -->
20
-
21
- **Additional context**
22
-
23
- <!-- Add any other context or screenshots about the feature request here. -->