fluentd 1.13.1-x86-mingw32 → 1.13.2-x86-mingw32
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of fluentd might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.yaml +69 -0
- data/.github/ISSUE_TEMPLATE/feature_request.yaml +38 -0
- data/CHANGELOG.md +37 -0
- data/README.md +2 -2
- data/fluentd.gemspec +1 -1
- data/lib/fluent/command/plugin_generator.rb +15 -5
- data/lib/fluent/config.rb +1 -1
- data/lib/fluent/config/types.rb +15 -0
- data/lib/fluent/config/v1_parser.rb +3 -2
- data/lib/fluent/env.rb +2 -1
- data/lib/fluent/oj_options.rb +62 -0
- data/lib/fluent/plugin/formatter.rb +1 -0
- data/lib/fluent/plugin/formatter_json.rb +9 -7
- data/lib/fluent/plugin/in_tail.rb +32 -1
- data/lib/fluent/plugin/parser_json.rb +2 -3
- data/lib/fluent/test/driver/storage.rb +30 -0
- data/lib/fluent/version.rb +1 -1
- data/templates/new_gem/lib/fluent/plugin/storage.rb.erb +40 -0
- data/templates/new_gem/test/plugin/test_storage.rb.erb +18 -0
- data/test/command/test_plugin_generator.rb +2 -1
- data/test/config/test_types.rb +7 -0
- data/test/plugin/test_in_tail.rb +85 -0
- data/test/test_event_time.rb +2 -2
- data/test/test_oj_options.rb +55 -0
- metadata +12 -6
- data/.github/ISSUE_TEMPLATE/bug_report.md +0 -40
- data/.github/ISSUE_TEMPLATE/feature_request.md +0 -23
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1c036a9bfd817725bcd7d2224018f35c242500e62e713a6549e9cebb81266662
|
4
|
+
data.tar.gz: 103d101ffc4f5be2af12b9e36b4f9ba6019213fd44218758bde892f90ad13551
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e8c51c88bf9b5992afd5095f5c742a23dc024dd4cb3a171f72367c4938d94f53e9a4584dcd71b68009f0ce8b359c450943200c3cb2d567e65f38c99b4db92220
|
7
|
+
data.tar.gz: b0917164c636070fb72682c5522e69ebf2dc79c017e9a53362eee6b6c1cb11b7f9b8061c619ef716b38b43598eac858e0805577fd0e108b44920a85c2f8d4cee
|
@@ -0,0 +1,69 @@
|
|
1
|
+
name: Bug Report
|
2
|
+
description: Create a report with a procedure for reproducing the bug
|
3
|
+
body:
|
4
|
+
- type: markdown
|
5
|
+
attributes:
|
6
|
+
value: |
|
7
|
+
Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
|
8
|
+
- type: textarea
|
9
|
+
id: description
|
10
|
+
attributes:
|
11
|
+
label: Describe the bug
|
12
|
+
description: A clear and concise description of what the bug is
|
13
|
+
validations:
|
14
|
+
required: true
|
15
|
+
- type: textarea
|
16
|
+
id: reproduce
|
17
|
+
attributes:
|
18
|
+
label: To Reproduce
|
19
|
+
description: Steps to reproduce the behavior
|
20
|
+
validations:
|
21
|
+
required: true
|
22
|
+
- type: textarea
|
23
|
+
id: expected
|
24
|
+
attributes:
|
25
|
+
label: Expected behavior
|
26
|
+
description: A clear and concise description of what you expected to happen
|
27
|
+
validations:
|
28
|
+
required: true
|
29
|
+
- type: textarea
|
30
|
+
id: environment
|
31
|
+
attributes:
|
32
|
+
label: Your Environment
|
33
|
+
description: |
|
34
|
+
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
35
|
+
- Operating system: `cat /etc/os-release`
|
36
|
+
- Kernel version: `uname -r`
|
37
|
+
|
38
|
+
Tip: If you hit the problem with older fluentd version, try latest version first.
|
39
|
+
value: |
|
40
|
+
- Fluentd version:
|
41
|
+
- TD Agent version:
|
42
|
+
- Operating system:
|
43
|
+
- Kernel version:
|
44
|
+
render: markdown
|
45
|
+
validations:
|
46
|
+
required: true
|
47
|
+
- type: textarea
|
48
|
+
id: configuration
|
49
|
+
attributes:
|
50
|
+
label: Your Configuration
|
51
|
+
description: |
|
52
|
+
Write your configuration here. Minimum reproducible fluentd.conf is recommended.
|
53
|
+
validations:
|
54
|
+
required: true
|
55
|
+
- type: textarea
|
56
|
+
id: logs
|
57
|
+
attributes:
|
58
|
+
label: Your Error Log
|
59
|
+
description: Write your ALL error log here
|
60
|
+
render: shell
|
61
|
+
validations:
|
62
|
+
required: true
|
63
|
+
- type: textarea
|
64
|
+
id: addtional-context
|
65
|
+
attributes:
|
66
|
+
label: Additional context
|
67
|
+
description: Add any other context about the problem here.
|
68
|
+
validations:
|
69
|
+
required: false
|
@@ -0,0 +1,38 @@
|
|
1
|
+
name: Feature request
|
2
|
+
description: Suggest an idea for this project
|
3
|
+
body:
|
4
|
+
- type: markdown
|
5
|
+
attributes:
|
6
|
+
value: |
|
7
|
+
Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
|
8
|
+
- type: textarea
|
9
|
+
id: description
|
10
|
+
attributes:
|
11
|
+
label: Is your feature request related to a problem? Please describe.
|
12
|
+
description: |
|
13
|
+
A clear and concise description of what the problem is.
|
14
|
+
Ex. I'm always frustrated when [...]
|
15
|
+
validations:
|
16
|
+
required: true
|
17
|
+
- type: textarea
|
18
|
+
id: solution
|
19
|
+
attributes:
|
20
|
+
label: Describe the solution you'd like
|
21
|
+
description: A clear and concise description of what you want to happen.
|
22
|
+
validations:
|
23
|
+
required: true
|
24
|
+
- type: textarea
|
25
|
+
id: alternative
|
26
|
+
attributes:
|
27
|
+
label: Describe alternatives you've considered
|
28
|
+
description: A clear and concise description of any alternative solutions or features you've considered.
|
29
|
+
validations:
|
30
|
+
required: true
|
31
|
+
- type: textarea
|
32
|
+
id: addtional-context
|
33
|
+
attributes:
|
34
|
+
label: Additional context
|
35
|
+
description: Add any other context or screenshots about the feature request here.
|
36
|
+
validations:
|
37
|
+
required: false
|
38
|
+
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,40 @@
|
|
1
|
+
# v1.13.2
|
2
|
+
|
3
|
+
## Release v1.13.2 - 2021/07/12
|
4
|
+
|
5
|
+
### Enhancement
|
6
|
+
|
7
|
+
* fluent-plugin-generate: Storage plugin was supported.
|
8
|
+
https://github.com/fluent/fluentd/pull/3426
|
9
|
+
* parser_json: Added support to customize configuration of oj options.
|
10
|
+
Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`,
|
11
|
+
`FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment
|
12
|
+
variable to configure it.
|
13
|
+
https://github.com/fluent/fluentd/pull/3315
|
14
|
+
|
15
|
+
### Bug fix
|
16
|
+
|
17
|
+
* binlog_reader: Fixed a crash bug by missing "fluent/env" dependency.
|
18
|
+
https://github.com/fluent/fluentd/pull/3443
|
19
|
+
* Fixed a crash bug on outputting log at the early stage when parsing
|
20
|
+
config file. This is a regression since v1.13.0. If you use invalid
|
21
|
+
'@' prefix parameter, remove it as a workaround.
|
22
|
+
https://github.com/fluent/fluentd/pull/3451
|
23
|
+
* in_tail: Fixed a bug that when rotation is occurred, remaining lines
|
24
|
+
will be discarded if the throttling feature is enabled.
|
25
|
+
https://github.com/fluent/fluentd/pull/3390
|
26
|
+
* fluent-plugin-generate: Fixed a crash bug during gemspec generation.
|
27
|
+
It was unexpectedly introduced by #3305, thus this bug was a
|
28
|
+
regression since 1.12.3.
|
29
|
+
https://github.com/fluent/fluentd/pull/3444
|
30
|
+
|
31
|
+
### Misc
|
32
|
+
|
33
|
+
* Fixed the runtime dependency version of http_parse.rb to 0.7.0.
|
34
|
+
It was fixed because false positive detection is occurred frequently
|
35
|
+
by security scanning tools.
|
36
|
+
https://github.com/fluent/fluentd/pull/3450
|
37
|
+
|
1
38
|
# v1.13.1
|
2
39
|
|
3
40
|
## Release v1.13.1 - 2021/06/25
|
data/README.md
CHANGED
@@ -25,9 +25,9 @@ An event consists of *tag*, *time* and *record*. Tag is a string separated with
|
|
25
25
|
|
26
26
|
Use Case | Description | Diagram
|
27
27
|
-------- | ------------|:---------:
|
28
|
-
Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/
|
28
|
+
Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/images/recipes/elasticsearch-s3-fluentd.png" height="150"/>
|
29
29
|
Syslog Alerting | Fluentd can "grep" for events and send out alerts. | <img src="https://www.fluentd.org/images/syslog-fluentd-alert.png" height="100"/>
|
30
|
-
Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/
|
30
|
+
Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/images/datasources/asynchronous_logging.png" height="150"/>
|
31
31
|
|
32
32
|
## Quick Start
|
33
33
|
|
data/fluentd.gemspec
CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |gem|
|
|
23
23
|
gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
|
24
24
|
gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
|
25
25
|
gem.add_runtime_dependency("serverengine", [">= 2.2.2", "< 3.0.0"])
|
26
|
-
gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.
|
26
|
+
gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.8.0"])
|
27
27
|
gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
|
28
28
|
gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
|
29
29
|
gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])
|
@@ -27,7 +27,7 @@ class FluentPluginGenerator
|
|
27
27
|
attr_reader :type, :name
|
28
28
|
attr_reader :license_name
|
29
29
|
|
30
|
-
SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter"]
|
30
|
+
SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter", "storage"]
|
31
31
|
|
32
32
|
def initialize(argv = ARGV)
|
33
33
|
@argv = argv
|
@@ -164,8 +164,13 @@ BANNER
|
|
164
164
|
end
|
165
165
|
|
166
166
|
def locked_gem_version(gem_name)
|
167
|
-
|
168
|
-
|
167
|
+
if File.exist?(lock_file_path)
|
168
|
+
d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
|
169
|
+
d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
|
170
|
+
else
|
171
|
+
# fallback even though Fluentd is installed without bundler
|
172
|
+
Gem::Specification.find_by_name(gem_name).version.version
|
173
|
+
end
|
169
174
|
end
|
170
175
|
|
171
176
|
def rake_version
|
@@ -177,8 +182,13 @@ BANNER
|
|
177
182
|
end
|
178
183
|
|
179
184
|
def bundler_version
|
180
|
-
|
181
|
-
|
185
|
+
if File.exist?(lock_file_path)
|
186
|
+
d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
|
187
|
+
d.locked_gems.bundler_version.version
|
188
|
+
else
|
189
|
+
# fallback even though Fluentd is installed without bundler
|
190
|
+
Gem::Specification.find_by_name("bundler").version.version
|
191
|
+
end
|
182
192
|
end
|
183
193
|
|
184
194
|
def class_name
|
data/lib/fluent/config.rb
CHANGED
@@ -62,7 +62,7 @@ module Fluent
|
|
62
62
|
Parser.parse(str, fname, basepath)
|
63
63
|
when :ruby
|
64
64
|
require 'fluent/config/dsl'
|
65
|
-
$log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file")
|
65
|
+
$log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file") if $log
|
66
66
|
Config::DSL::Parser.parse(str, File.join(basepath, fname))
|
67
67
|
else
|
68
68
|
raise "[BUG] unknown configuration parser specification:'#{parser}'"
|
data/lib/fluent/config/types.rb
CHANGED
@@ -20,6 +20,10 @@ require 'fluent/config/error'
|
|
20
20
|
|
21
21
|
module Fluent
|
22
22
|
module Config
|
23
|
+
def self.reformatted_value(type, val, opts = {}, name = nil)
|
24
|
+
REFORMAT_VALUE.call(type, val, opts, name)
|
25
|
+
end
|
26
|
+
|
23
27
|
def self.size_value(str, opts = {}, name = nil)
|
24
28
|
return nil if str.nil?
|
25
29
|
|
@@ -104,6 +108,16 @@ module Fluent
|
|
104
108
|
Config.string_value(val, opts, name)
|
105
109
|
}
|
106
110
|
|
111
|
+
def self.symbol_value(val, opts = {}, name = nil)
|
112
|
+
return nil if val.nil? || val.empty?
|
113
|
+
|
114
|
+
val.delete_prefix(":").to_sym
|
115
|
+
end
|
116
|
+
|
117
|
+
SYMBOL_TYPE = Proc.new { |val, opts = {}, name = nil|
|
118
|
+
Config.symbol_value(val, opts, name)
|
119
|
+
}
|
120
|
+
|
107
121
|
def self.enum_value(val, opts = {}, name = nil)
|
108
122
|
return nil if val.nil?
|
109
123
|
|
@@ -176,6 +190,7 @@ module Fluent
|
|
176
190
|
when :bool then Config.bool_value(value, opts, name)
|
177
191
|
when :time then Config.time_value(value, opts, name)
|
178
192
|
when :regexp then Config.regexp_value(value, opts, name)
|
193
|
+
when :symbol then Config.symbol_value(value, opts, name)
|
179
194
|
else
|
180
195
|
raise "unknown type in REFORMAT: #{type}"
|
181
196
|
end
|
@@ -37,6 +37,7 @@ module Fluent
|
|
37
37
|
super(strscan, eval_context)
|
38
38
|
@include_basepath = include_basepath
|
39
39
|
@fname = fname
|
40
|
+
@logger = defined?($log) ? $log : nil
|
40
41
|
end
|
41
42
|
|
42
43
|
def parse!
|
@@ -99,7 +100,7 @@ module Fluent
|
|
99
100
|
|
100
101
|
elsif root_element && skip(/(\@include|include)#{SPACING}/)
|
101
102
|
if !prev_match.start_with?('@')
|
102
|
-
|
103
|
+
@logger.warn "'include' is deprecated. Use '@include' instead" if @logger
|
103
104
|
end
|
104
105
|
parse_include(attrs, elems)
|
105
106
|
|
@@ -123,7 +124,7 @@ module Fluent
|
|
123
124
|
parse_error! "'@' is the system reserved prefix. Don't use '@' prefix parameter in the configuration: #{k}"
|
124
125
|
else
|
125
126
|
# TODO: This is for backward compatibility. It will throw an error in the future.
|
126
|
-
|
127
|
+
@logger.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}" if @logger
|
127
128
|
end
|
128
129
|
end
|
129
130
|
|
data/lib/fluent/env.rb
CHANGED
@@ -15,13 +15,14 @@
|
|
15
15
|
#
|
16
16
|
|
17
17
|
require 'serverengine/utils'
|
18
|
+
require 'fluent/oj_options'
|
18
19
|
|
19
20
|
module Fluent
|
20
21
|
DEFAULT_CONFIG_PATH = ENV['FLUENT_CONF'] || '/etc/fluent/fluent.conf'
|
21
22
|
DEFAULT_PLUGIN_DIR = ENV['FLUENT_PLUGIN'] || '/etc/fluent/plugin'
|
22
23
|
DEFAULT_SOCKET_PATH = ENV['FLUENT_SOCKET'] || '/var/run/fluent/fluent.sock'
|
23
24
|
DEFAULT_BACKUP_DIR = ENV['FLUENT_BACKUP_DIR'] || '/tmp/fluent'
|
24
|
-
DEFAULT_OJ_OPTIONS =
|
25
|
+
DEFAULT_OJ_OPTIONS = Fluent::OjOptions.load_env
|
25
26
|
DEFAULT_DIR_PERMISSION = 0755
|
26
27
|
DEFAULT_FILE_PERMISSION = 0644
|
27
28
|
|
@@ -0,0 +1,62 @@
|
|
1
|
+
require 'fluent/config/types'
|
2
|
+
|
3
|
+
module Fluent
|
4
|
+
class OjOptions
|
5
|
+
OPTIONS = {
|
6
|
+
'bigdecimal_load': :symbol,
|
7
|
+
'max_nesting': :integer,
|
8
|
+
'mode': :symbol,
|
9
|
+
'use_to_json': :bool
|
10
|
+
}
|
11
|
+
|
12
|
+
ALLOWED_VALUES = {
|
13
|
+
'bigdecimal_load': %i[bigdecimal float auto],
|
14
|
+
'mode': %i[strict null compat json rails object custom]
|
15
|
+
}
|
16
|
+
|
17
|
+
DEFAULTS = {
|
18
|
+
'bigdecimal_load': :float,
|
19
|
+
'mode': :compat,
|
20
|
+
'use_to_json': true
|
21
|
+
}
|
22
|
+
|
23
|
+
@@available = false
|
24
|
+
|
25
|
+
def self.available?
|
26
|
+
@@available
|
27
|
+
end
|
28
|
+
|
29
|
+
def self.load_env
|
30
|
+
options = self.get_options
|
31
|
+
begin
|
32
|
+
require 'oj'
|
33
|
+
Oj.default_options = options
|
34
|
+
@@available = true
|
35
|
+
rescue LoadError
|
36
|
+
@@available = false
|
37
|
+
end
|
38
|
+
options
|
39
|
+
end
|
40
|
+
|
41
|
+
private
|
42
|
+
|
43
|
+
def self.get_options
|
44
|
+
options = {}
|
45
|
+
DEFAULTS.each { |key, value| options[key] = value }
|
46
|
+
|
47
|
+
OPTIONS.each do |key, type|
|
48
|
+
env_value = ENV["FLUENT_OJ_OPTION_#{key.upcase}"]
|
49
|
+
next if env_value.nil?
|
50
|
+
|
51
|
+
cast_value = Fluent::Config.reformatted_value(OPTIONS[key], env_value, { strict: true })
|
52
|
+
next if cast_value.nil?
|
53
|
+
|
54
|
+
next if ALLOWED_VALUES[key] && !ALLOWED_VALUES[key].include?(cast_value)
|
55
|
+
|
56
|
+
options[key.to_sym] = cast_value
|
57
|
+
end
|
58
|
+
|
59
|
+
options
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
@@ -15,7 +15,7 @@
|
|
15
15
|
#
|
16
16
|
|
17
17
|
require 'fluent/plugin/formatter'
|
18
|
-
require 'fluent/
|
18
|
+
require 'fluent/oj_options'
|
19
19
|
|
20
20
|
module Fluent
|
21
21
|
module Plugin
|
@@ -30,12 +30,14 @@ module Fluent
|
|
30
30
|
def configure(conf)
|
31
31
|
super
|
32
32
|
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
33
|
+
if @json_parser == 'oj'
|
34
|
+
if Fluent::OjOptions.available?
|
35
|
+
@dump_proc = Oj.method(:dump)
|
36
|
+
else
|
37
|
+
log.info "Oj isn't installed, fallback to Yajl as json parser"
|
38
|
+
@dump_proc = Yajl.method(:dump)
|
39
|
+
end
|
40
|
+
else
|
39
41
|
@dump_proc = Yajl.method(:dump)
|
40
42
|
end
|
41
43
|
|
@@ -522,8 +522,25 @@ module Fluent::Plugin
|
|
522
522
|
def detach_watcher_after_rotate_wait(tw, ino)
|
523
523
|
# Call event_loop_attach/event_loop_detach is high-cost for short-live object.
|
524
524
|
# If this has a problem with large number of files, use @_event_loop directly instead of timer_execute.
|
525
|
-
|
525
|
+
if @open_on_every_update
|
526
|
+
# Detach now because it's already closed, waiting it doesn't make sense.
|
526
527
|
detach_watcher(tw, ino)
|
528
|
+
elsif @read_bytes_limit_per_second < 0
|
529
|
+
# throttling isn't enabled, just wait @rotate_wait
|
530
|
+
timer_execute(:in_tail_close_watcher, @rotate_wait, repeat: false) do
|
531
|
+
detach_watcher(tw, ino)
|
532
|
+
end
|
533
|
+
else
|
534
|
+
# When the throttling feature is enabled, it might not reach EOF yet.
|
535
|
+
# Should ensure to read all contents before closing it, with keeping throttling.
|
536
|
+
start_time_to_wait = Fluent::Clock.now
|
537
|
+
timer = timer_execute(:in_tail_close_watcher, 1, repeat: true) do
|
538
|
+
elapsed = Fluent::Clock.now - start_time_to_wait
|
539
|
+
if tw.eof? && elapsed >= @rotate_wait
|
540
|
+
timer.detach
|
541
|
+
detach_watcher(tw, ino)
|
542
|
+
end
|
543
|
+
end
|
527
544
|
end
|
528
545
|
end
|
529
546
|
|
@@ -736,6 +753,10 @@ module Fluent::Plugin
|
|
736
753
|
end
|
737
754
|
end
|
738
755
|
|
756
|
+
def eof?
|
757
|
+
@io_handler.eof?
|
758
|
+
end
|
759
|
+
|
739
760
|
def on_notify
|
740
761
|
begin
|
741
762
|
stat = Fluent::FileWrapper.stat(@path)
|
@@ -923,6 +944,7 @@ module Fluent::Plugin
|
|
923
944
|
@shutdown_start_time = nil
|
924
945
|
@shutdown_timeout = SHUTDOWN_TIMEOUT
|
925
946
|
@shutdown_mutex = Mutex.new
|
947
|
+
@eof = false
|
926
948
|
|
927
949
|
@log.info "following tail of #{@path}"
|
928
950
|
end
|
@@ -949,6 +971,10 @@ module Fluent::Plugin
|
|
949
971
|
!!@io
|
950
972
|
end
|
951
973
|
|
974
|
+
def eof?
|
975
|
+
@eof
|
976
|
+
end
|
977
|
+
|
952
978
|
private
|
953
979
|
|
954
980
|
def limit_bytes_per_second_reached?
|
@@ -989,6 +1015,7 @@ module Fluent::Plugin
|
|
989
1015
|
while true
|
990
1016
|
@start_reading_time ||= Fluent::Clock.now
|
991
1017
|
data = io.readpartial(BYTES_TO_READ, @iobuf)
|
1018
|
+
@eof = false
|
992
1019
|
@number_bytes_read += data.bytesize
|
993
1020
|
@fifo << data
|
994
1021
|
@fifo.read_lines(@lines)
|
@@ -1005,6 +1032,7 @@ module Fluent::Plugin
|
|
1005
1032
|
end
|
1006
1033
|
end
|
1007
1034
|
rescue EOFError
|
1035
|
+
@eof = true
|
1008
1036
|
end
|
1009
1037
|
end
|
1010
1038
|
|
@@ -1042,14 +1070,17 @@ module Fluent::Plugin
|
|
1042
1070
|
else
|
1043
1071
|
@io ||= open
|
1044
1072
|
yield @io
|
1073
|
+
@eof = true if @io.nil?
|
1045
1074
|
end
|
1046
1075
|
rescue WatcherSetupError => e
|
1047
1076
|
close
|
1077
|
+
@eof = true
|
1048
1078
|
raise e
|
1049
1079
|
rescue
|
1050
1080
|
@log.error $!.to_s
|
1051
1081
|
@log.error_backtrace
|
1052
1082
|
close
|
1083
|
+
@eof = true
|
1053
1084
|
end
|
1054
1085
|
end
|
1055
1086
|
|
@@ -15,8 +15,8 @@
|
|
15
15
|
#
|
16
16
|
|
17
17
|
require 'fluent/plugin/parser'
|
18
|
-
require 'fluent/env'
|
19
18
|
require 'fluent/time'
|
19
|
+
require 'fluent/oj_options'
|
20
20
|
|
21
21
|
require 'yajl'
|
22
22
|
require 'json'
|
@@ -50,8 +50,7 @@ module Fluent
|
|
50
50
|
def configure_json_parser(name)
|
51
51
|
case name
|
52
52
|
when :oj
|
53
|
-
|
54
|
-
Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
|
53
|
+
raise LoadError unless Fluent::OjOptions.available?
|
55
54
|
[Oj.method(:load), Oj::ParseError]
|
56
55
|
when :json then [JSON.method(:load), JSON::ParserError]
|
57
56
|
when :yajl then [Yajl.method(:load), Yajl::ParseError]
|
@@ -0,0 +1,30 @@
|
|
1
|
+
#
|
2
|
+
# Fluentd
|
3
|
+
#
|
4
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
5
|
+
# you may not use this file except in compliance with the License.
|
6
|
+
# You may obtain a copy of the License at
|
7
|
+
#
|
8
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
9
|
+
#
|
10
|
+
# Unless required by applicable law or agreed to in writing, software
|
11
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
12
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
13
|
+
# See the License for the specific language governing permissions and
|
14
|
+
# limitations under the License.
|
15
|
+
#
|
16
|
+
|
17
|
+
require 'fluent/test/driver/base_owned'
|
18
|
+
|
19
|
+
module Fluent
|
20
|
+
module Test
|
21
|
+
module Driver
|
22
|
+
class Storage < BaseOwned
|
23
|
+
def initialize(klass, **kwargs, &block)
|
24
|
+
super
|
25
|
+
@section_name = "storage"
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/fluent/version.rb
CHANGED
@@ -0,0 +1,40 @@
|
|
1
|
+
<%= preamble %>
|
2
|
+
|
3
|
+
require "fluent/plugin/storage"
|
4
|
+
|
5
|
+
module Fluent
|
6
|
+
module Plugin
|
7
|
+
class <%= class_name %> < Fluent::Plugin::Storage
|
8
|
+
Fluent::Plugin.register_storage("<%= plugin_name %>", self)
|
9
|
+
|
10
|
+
def initialize
|
11
|
+
super
|
12
|
+
end
|
13
|
+
|
14
|
+
def configure(conf)
|
15
|
+
super
|
16
|
+
end
|
17
|
+
|
18
|
+
def load
|
19
|
+
end
|
20
|
+
|
21
|
+
def save
|
22
|
+
end
|
23
|
+
|
24
|
+
def get(key)
|
25
|
+
end
|
26
|
+
|
27
|
+
def fetch(key, defval)
|
28
|
+
end
|
29
|
+
|
30
|
+
def put(key, value)
|
31
|
+
end
|
32
|
+
|
33
|
+
def delete(key)
|
34
|
+
end
|
35
|
+
|
36
|
+
def update(key, &block)
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
require "helper"
|
2
|
+
require "fluent/plugin/<%= plugin_filename %>"
|
3
|
+
|
4
|
+
class <%= class_name %>Test < Test::Unit::TestCase
|
5
|
+
setup do
|
6
|
+
Fluent::Test.setup
|
7
|
+
end
|
8
|
+
|
9
|
+
test "failure" do
|
10
|
+
flunk
|
11
|
+
end
|
12
|
+
|
13
|
+
private
|
14
|
+
|
15
|
+
def create_driver(conf)
|
16
|
+
Fluent::Test::Driver::Storage.new(Fluent::Plugin::<%= class_name %>).configure(conf)
|
17
|
+
end
|
18
|
+
end
|
@@ -29,7 +29,8 @@ class TestFluentPluginGenerator < Test::Unit::TestCase
|
|
29
29
|
output: ["output", "out"],
|
30
30
|
filter: ["filter", "filter"],
|
31
31
|
parser: ["parser", "parser"],
|
32
|
-
formatter: ["formatter", "formatter"]
|
32
|
+
formatter: ["formatter", "formatter"],
|
33
|
+
storage: ["storage", "storage"])
|
33
34
|
test "generate plugin" do |(type, part)|
|
34
35
|
generator = FluentPluginGenerator.new([type, "fake"])
|
35
36
|
stub_git_process(generator)
|
data/test/config/test_types.rb
CHANGED
@@ -180,6 +180,13 @@ class TestConfigTypes < ::Test::Unit::TestCase
|
|
180
180
|
assert_equal Encoding::UTF_8, actual.encoding
|
181
181
|
end
|
182
182
|
|
183
|
+
data('starts_with_semicolon' => [:conor, ':conor'],
|
184
|
+
'simple_string' => [:conor, 'conor'],
|
185
|
+
'empty_string' => [nil, ''])
|
186
|
+
test 'symbol' do |(expected, val)|
|
187
|
+
assert_equal Config::SYMBOL_TYPE.call(val, {}), expected
|
188
|
+
end
|
189
|
+
|
183
190
|
data("val" => [:val, 'val'],
|
184
191
|
"v" => [:v, 'v'],
|
185
192
|
"value" => [:value, 'value'])
|
data/test/plugin/test_in_tail.rb
CHANGED
@@ -456,6 +456,91 @@ class TailInputTest < Test::Unit::TestCase
|
|
456
456
|
assert_equal([], d.events)
|
457
457
|
end
|
458
458
|
end
|
459
|
+
|
460
|
+
sub_test_case "EOF with reads_bytes_per_second" do
|
461
|
+
def test_longer_than_rotate_wait
|
462
|
+
limit_bytes = 8192
|
463
|
+
num_lines = 1024 * 3
|
464
|
+
msg = "08bytes"
|
465
|
+
|
466
|
+
File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
|
467
|
+
f.write("#{msg}\n" * num_lines)
|
468
|
+
end
|
469
|
+
|
470
|
+
config = CONFIG_READ_FROM_HEAD +
|
471
|
+
SINGLE_LINE_CONFIG +
|
472
|
+
config_element("", "", {
|
473
|
+
"read_bytes_limit_per_second" => limit_bytes,
|
474
|
+
"rotate_wait" => 0.1,
|
475
|
+
"refresh_interval" => 0.5,
|
476
|
+
})
|
477
|
+
|
478
|
+
rotated = false
|
479
|
+
d = create_driver(config)
|
480
|
+
d.run(timeout: 10) do
|
481
|
+
while d.events.size < num_lines do
|
482
|
+
if d.events.size > 0 && !rotated
|
483
|
+
cleanup_file("#{TMP_DIR}/tail.txt")
|
484
|
+
FileUtils.touch("#{TMP_DIR}/tail.txt")
|
485
|
+
rotated = true
|
486
|
+
end
|
487
|
+
sleep 0.3
|
488
|
+
end
|
489
|
+
end
|
490
|
+
|
491
|
+
assert_equal(num_lines,
|
492
|
+
d.events.count do |event|
|
493
|
+
event[2]["message"] == msg
|
494
|
+
end)
|
495
|
+
end
|
496
|
+
|
497
|
+
def test_shorter_than_rotate_wait
|
498
|
+
limit_bytes = 8192
|
499
|
+
num_lines = 1024 * 2
|
500
|
+
msg = "08bytes"
|
501
|
+
|
502
|
+
File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
|
503
|
+
f.write("#{msg}\n" * num_lines)
|
504
|
+
end
|
505
|
+
|
506
|
+
config = CONFIG_READ_FROM_HEAD +
|
507
|
+
SINGLE_LINE_CONFIG +
|
508
|
+
config_element("", "", {
|
509
|
+
"read_bytes_limit_per_second" => limit_bytes,
|
510
|
+
"rotate_wait" => 2,
|
511
|
+
"refresh_interval" => 0.5,
|
512
|
+
})
|
513
|
+
|
514
|
+
start_time = Fluent::Clock.now
|
515
|
+
rotated = false
|
516
|
+
detached = false
|
517
|
+
d = create_driver(config)
|
518
|
+
mock.proxy(d.instance).setup_watcher(anything, anything) do |tw|
|
519
|
+
mock.proxy(tw).detach(anything) do |v|
|
520
|
+
detached = true
|
521
|
+
v
|
522
|
+
end
|
523
|
+
tw
|
524
|
+
end.twice
|
525
|
+
|
526
|
+
d.run(timeout: 10) do
|
527
|
+
until detached do
|
528
|
+
if d.events.size > 0 && !rotated
|
529
|
+
cleanup_file("#{TMP_DIR}/tail.txt")
|
530
|
+
FileUtils.touch("#{TMP_DIR}/tail.txt")
|
531
|
+
rotated = true
|
532
|
+
end
|
533
|
+
sleep 0.3
|
534
|
+
end
|
535
|
+
end
|
536
|
+
|
537
|
+
assert_true(Fluent::Clock.now - start_time > 2)
|
538
|
+
assert_equal(num_lines,
|
539
|
+
d.events.count do |event|
|
540
|
+
event[2]["message"] == msg
|
541
|
+
end)
|
542
|
+
end
|
543
|
+
end
|
459
544
|
end
|
460
545
|
|
461
546
|
data(flat: CONFIG_READ_FROM_HEAD + SINGLE_LINE_CONFIG,
|
data/test/test_event_time.rb
CHANGED
@@ -64,8 +64,8 @@ class EventTimeTest < Test::Unit::TestCase
|
|
64
64
|
|
65
65
|
test 'Oj.dump' do
|
66
66
|
time = Fluent::EventTime.new(100)
|
67
|
-
require 'fluent/
|
68
|
-
|
67
|
+
require 'fluent/oj_options'
|
68
|
+
Fluent::OjOptions.load_env
|
69
69
|
assert_equal('{"time":100}', Oj.dump({'time' => time}))
|
70
70
|
assert_equal('["tag",100,{"key":"value"}]', Oj.dump(["tag", time, {"key" => "value"}], mode: :compat))
|
71
71
|
end
|
@@ -0,0 +1,55 @@
|
|
1
|
+
require_relative 'helper'
|
2
|
+
require 'fluent/test'
|
3
|
+
require 'fluent/oj_options'
|
4
|
+
|
5
|
+
class OjOptionsTest < ::Test::Unit::TestCase
|
6
|
+
begin
|
7
|
+
require 'oj'
|
8
|
+
@@oj_is_avaibale = true
|
9
|
+
rescue LoadError
|
10
|
+
@@oj_is_avaibale = false
|
11
|
+
end
|
12
|
+
|
13
|
+
setup do
|
14
|
+
@orig_env = {}
|
15
|
+
ENV.each do |key, value|
|
16
|
+
@orig_env[key] = value if key.start_with?("FLUENT_OJ_OPTION_")
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
teardown do
|
21
|
+
ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
|
22
|
+
@orig_env.each { |key, value| ENV[key] = value }
|
23
|
+
end
|
24
|
+
|
25
|
+
test "available?" do
|
26
|
+
assert_equal(@@oj_is_avaibale, Fluent::OjOptions.available?)
|
27
|
+
end
|
28
|
+
|
29
|
+
sub_test_case "set by environment variable" do
|
30
|
+
test "when no env vars set, returns default options" do
|
31
|
+
ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
|
32
|
+
defaults = Fluent::OjOptions::DEFAULTS
|
33
|
+
assert_equal(defaults, Fluent::OjOptions.load_env)
|
34
|
+
assert_equal(defaults, Oj.default_options.slice(*defaults.keys)) if @@oj_is_avaibale
|
35
|
+
end
|
36
|
+
|
37
|
+
test "valid env var passed with valid value, default is overridden" do
|
38
|
+
ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":bigdecimal"
|
39
|
+
assert_equal(:bigdecimal, Fluent::OjOptions.load_env[:bigdecimal_load])
|
40
|
+
assert_equal(:bigdecimal, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
|
41
|
+
end
|
42
|
+
|
43
|
+
test "valid env var passed with invalid value, default is not overriden" do
|
44
|
+
ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":conor"
|
45
|
+
assert_equal(:float, Fluent::OjOptions.load_env[:bigdecimal_load])
|
46
|
+
assert_equal(:float, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
|
47
|
+
end
|
48
|
+
|
49
|
+
test "invalid env var passed, nothing done with it" do
|
50
|
+
ENV["FLUENT_OJ_OPTION_CONOR"] = ":conor"
|
51
|
+
assert_equal(nil, Fluent::OjOptions.load_env[:conor])
|
52
|
+
assert_equal(nil, Oj.default_options[:conor]) if @@oj_is_avaibale
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluentd
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.13.
|
4
|
+
version: 1.13.2
|
5
5
|
platform: x86-mingw32
|
6
6
|
authors:
|
7
7
|
- Sadayuki Furuhashi
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-07-12 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -107,7 +107,7 @@ dependencies:
|
|
107
107
|
version: 0.5.1
|
108
108
|
- - "<"
|
109
109
|
- !ruby/object:Gem::Version
|
110
|
-
version: 0.
|
110
|
+
version: 0.8.0
|
111
111
|
type: :runtime
|
112
112
|
prerelease: false
|
113
113
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -117,7 +117,7 @@ dependencies:
|
|
117
117
|
version: 0.5.1
|
118
118
|
- - "<"
|
119
119
|
- !ruby/object:Gem::Version
|
120
|
-
version: 0.
|
120
|
+
version: 0.8.0
|
121
121
|
- !ruby/object:Gem::Dependency
|
122
122
|
name: sigdump
|
123
123
|
requirement: !ruby/object:Gem::Requirement
|
@@ -463,9 +463,9 @@ files:
|
|
463
463
|
- ".deepsource.toml"
|
464
464
|
- ".drone.yml"
|
465
465
|
- ".github/ISSUE_TEMPLATE.md"
|
466
|
-
- ".github/ISSUE_TEMPLATE/bug_report.
|
466
|
+
- ".github/ISSUE_TEMPLATE/bug_report.yaml"
|
467
467
|
- ".github/ISSUE_TEMPLATE/config.yml"
|
468
|
-
- ".github/ISSUE_TEMPLATE/feature_request.
|
468
|
+
- ".github/ISSUE_TEMPLATE/feature_request.yaml"
|
469
469
|
- ".github/PULL_REQUEST_TEMPLATE.md"
|
470
470
|
- ".github/workflows/issue-auto-closer.yml"
|
471
471
|
- ".github/workflows/linux-test.yaml"
|
@@ -609,6 +609,7 @@ files:
|
|
609
609
|
- lib/fluent/match.rb
|
610
610
|
- lib/fluent/mixin.rb
|
611
611
|
- lib/fluent/msgpack_factory.rb
|
612
|
+
- lib/fluent/oj_options.rb
|
612
613
|
- lib/fluent/output.rb
|
613
614
|
- lib/fluent/output_chain.rb
|
614
615
|
- lib/fluent/parser.rb
|
@@ -754,6 +755,7 @@ files:
|
|
754
755
|
- lib/fluent/test/driver/multi_output.rb
|
755
756
|
- lib/fluent/test/driver/output.rb
|
756
757
|
- lib/fluent/test/driver/parser.rb
|
758
|
+
- lib/fluent/test/driver/storage.rb
|
757
759
|
- lib/fluent/test/driver/test_event_router.rb
|
758
760
|
- lib/fluent/test/filter_test.rb
|
759
761
|
- lib/fluent/test/formatter_test.rb
|
@@ -779,12 +781,14 @@ files:
|
|
779
781
|
- templates/new_gem/lib/fluent/plugin/input.rb.erb
|
780
782
|
- templates/new_gem/lib/fluent/plugin/output.rb.erb
|
781
783
|
- templates/new_gem/lib/fluent/plugin/parser.rb.erb
|
784
|
+
- templates/new_gem/lib/fluent/plugin/storage.rb.erb
|
782
785
|
- templates/new_gem/test/helper.rb.erb
|
783
786
|
- templates/new_gem/test/plugin/test_filter.rb.erb
|
784
787
|
- templates/new_gem/test/plugin/test_formatter.rb.erb
|
785
788
|
- templates/new_gem/test/plugin/test_input.rb.erb
|
786
789
|
- templates/new_gem/test/plugin/test_output.rb.erb
|
787
790
|
- templates/new_gem/test/plugin/test_parser.rb.erb
|
791
|
+
- templates/new_gem/test/plugin/test_storage.rb.erb
|
788
792
|
- templates/plugin_config_formatter/param.md-compact.erb
|
789
793
|
- templates/plugin_config_formatter/param.md-table.erb
|
790
794
|
- templates/plugin_config_formatter/param.md.erb
|
@@ -989,6 +993,7 @@ files:
|
|
989
993
|
- test/test_match.rb
|
990
994
|
- test/test_mixin.rb
|
991
995
|
- test/test_msgpack_factory.rb
|
996
|
+
- test/test_oj_options.rb
|
992
997
|
- test/test_output.rb
|
993
998
|
- test/test_plugin.rb
|
994
999
|
- test/test_plugin_classes.rb
|
@@ -1228,6 +1233,7 @@ test_files:
|
|
1228
1233
|
- test/test_match.rb
|
1229
1234
|
- test/test_mixin.rb
|
1230
1235
|
- test/test_msgpack_factory.rb
|
1236
|
+
- test/test_oj_options.rb
|
1231
1237
|
- test/test_output.rb
|
1232
1238
|
- test/test_plugin.rb
|
1233
1239
|
- test/test_plugin_classes.rb
|
@@ -1,40 +0,0 @@
|
|
1
|
-
---
|
2
|
-
name: Bug Report
|
3
|
-
about: Create a report with a procedure for reproducing the bug
|
4
|
-
|
5
|
-
---
|
6
|
-
|
7
|
-
Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
|
8
|
-
|
9
|
-
**Describe the bug**
|
10
|
-
<!-- A clear and concise description of what the bug is. -->
|
11
|
-
|
12
|
-
**To Reproduce**
|
13
|
-
<!-- Steps to reproduce the behavior: -->
|
14
|
-
|
15
|
-
**Expected behavior**
|
16
|
-
<!-- A clear and concise description of what you expected to happen. -->
|
17
|
-
|
18
|
-
**Your Environment**
|
19
|
-
|
20
|
-
- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
|
21
|
-
- Operating system: `cat /etc/os-release`
|
22
|
-
- Kernel version: `uname -r`
|
23
|
-
|
24
|
-
If you hit the problem with older fluentd version, try latest version first.
|
25
|
-
|
26
|
-
**Your Configuration**
|
27
|
-
|
28
|
-
```
|
29
|
-
<!-- Write your configuration here -->
|
30
|
-
```
|
31
|
-
|
32
|
-
**Your Error Log**
|
33
|
-
|
34
|
-
```
|
35
|
-
<!-- Write your **ALL** error log here -->
|
36
|
-
```
|
37
|
-
|
38
|
-
**Additional context**
|
39
|
-
|
40
|
-
<!-- Add any other context about the problem here. -->
|
@@ -1,23 +0,0 @@
|
|
1
|
-
---
|
2
|
-
name: Feature request
|
3
|
-
about: Suggest an idea for this project
|
4
|
-
|
5
|
-
---
|
6
|
-
|
7
|
-
Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
|
8
|
-
|
9
|
-
**Is your feature request related to a problem? Please describe.**
|
10
|
-
|
11
|
-
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
|
12
|
-
|
13
|
-
**Describe the solution you'd like**
|
14
|
-
|
15
|
-
<!-- A clear and concise description of what you want to happen. -->
|
16
|
-
|
17
|
-
**Describe alternatives you've considered**
|
18
|
-
|
19
|
-
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
|
20
|
-
|
21
|
-
**Additional context**
|
22
|
-
|
23
|
-
<!-- Add any other context or screenshots about the feature request here. -->
|