fluentd 1.13.1 → 1.13.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2974ccbbcf3e5273d3561322c9786ca7ad4cdda66817f13e4ed73962f4762b7b
-  data.tar.gz: 5bd0cd30a89e325fd1cbb3868a3f13f4685c00ca557cd38dedf04c478c6dc249
+  metadata.gz: b2f70ac4c7ffec54701aaf1e48416d2806881d39171a63c2eb84f3686a59cd59
+  data.tar.gz: b192b8229744e5fc00816c45b6a6e83bb1748d0eae1773394044907254fa6433
 SHA512:
-  metadata.gz: 4fba66b7426b7818c7e60c6120ce46de4396ce903474b7e3d4e5db0db3f52ed5b324c0d54d312267abcc110ce5010108cac5ceaddb503a2234a973c9f99f879b
-  data.tar.gz: d6d3771be3d7850e4bf265836cc2663c81f97f0c602e2e95457fc2d30ea8c8c7b67e9340ef5db276253c5d15f5538f2712614b9c8b9bfaee4b1e302761ca2052
+  metadata.gz: 338cc5d35a89c042fdfa1a1817479fbfc2b88363619a5cb0cdf08a20a70942c52a561c538e70fb3b737eb7fa2d38bb77a43e993b3693b87bd300b43d64f8ac0c
+  data.tar.gz: aaace3c4450de3c4db7b6875e94e47b30e265f7370fcfcff7e15b9c4568913643fa53e3564e7b61237091e0bbf02fac21bdca93181c4109949e384b73fd072a1

data/.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -0,0 +1,69 @@
+name: Bug Report
+description: Create a report with a procedure for reproducing the bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
+  - type: textarea
+    id: description
+    attributes:
+      label: Describe the bug
+      description: A clear and concise description of what the bug is
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: To Reproduce
+      description: Steps to reproduce the behavior
+    validations:
+      required: true
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected behavior
+      description: A clear and concise description of what you expected to happen
+    validations:
+      required: true
+  - type: textarea
+    id: environment
+    attributes:
+      label: Your Environment
+      description: |
+        - Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
+        - Operating system: `cat /etc/os-release`
+        - Kernel version: `uname -r`
+
+        Tip: If you hit the problem with older fluentd version, try latest version first.
+      value: |
+        - Fluentd version:
+        - TD Agent version:
+        - Operating system:
+        - Kernel version:
+      render: markdown
+    validations:
+      required: true
+  - type: textarea
+    id: configuration
+    attributes:
+      label: Your Configuration
+      description: |
+        Write your configuration here. Minimum reproducible fluentd.conf is recommended.
+    validations:
+      required: true
+  - type: textarea
+    id: logs
+    attributes:
+      label: Your Error Log
+      description: Write your ALL error log here
+      render: shell
+    validations:
+      required: true
+  - type: textarea
+    id: addtional-context
+    attributes:
+      label: Additional context
+      description: Add any other context about the problem here.
+    validations:
+      required: false

data/.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -0,0 +1,38 @@
+name: Feature request
+description: Suggest an idea for this project
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
+  - type: textarea
+    id: description
+    attributes:
+      label: Is your feature request related to a problem? Please describe.
+      description: |
+        A clear and concise description of what the problem is.
+        Ex. I'm always frustrated when [...]
+    validations:
+      required: true
+  - type: textarea
+    id: solution
+    attributes:
+      label: Describe the solution you'd like
+      description: A clear and concise description of what you want to happen.
+    validations:
+      required: true
+  - type: textarea
+    id: alternative
+    attributes:
+      label: Describe alternatives you've considered
+      description: A clear and concise description of any alternative solutions or features you've considered.
+    validations:
+      required: true
+  - type: textarea
+    id: addtional-context
+    attributes:
+      label: Additional context
+      description: Add any other context or screenshots about the feature request here.
+    validations:
+      required: false
+

data/CHANGELOG.md

@@ -1,3 +1,40 @@
+# v1.13.2
+
+## Release v1.13.2 - 2021/07/12
+
+### Enhancement
+
+* fluent-plugin-generate: Storage plugin was supported.
+  https://github.com/fluent/fluentd/pull/3426
+* parser_json: Added support to customize configuration of oj options.
+  Use `FLUENT_OJ_OPTION_BIGDECIMAL_LOAD`, `FLUENT_OJ_OPTION_MAX_NESTING`,
+  `FLUENT_OJ_OPTION_MODE`, and `FLUENT_OJ_OPTION_USE_TO_JSON` environment
+  variable to configure it.
+  https://github.com/fluent/fluentd/pull/3315
+
+### Bug fix
+
+* binlog_reader: Fixed a crash bug by missing "fluent/env" dependency.
+  https://github.com/fluent/fluentd/pull/3443
+* Fixed a crash bug on outputting log at the early stage when parsing
+  config file.  This is a regression since v1.13.0. If you use invalid
+  '@' prefix parameter, remove it as a workaround.
+  https://github.com/fluent/fluentd/pull/3451
+* in_tail: Fixed a bug that when rotation is occurred, remaining lines
+  will be discarded if the throttling feature is enabled.
+  https://github.com/fluent/fluentd/pull/3390
+* fluent-plugin-generate: Fixed a crash bug during gemspec generation.
+  It was unexpectedly introduced by #3305, thus this bug was a
+  regression since 1.12.3.
+  https://github.com/fluent/fluentd/pull/3444
+
+### Misc
+
+* Fixed the runtime dependency version of http_parse.rb to 0.7.0.
+  It was fixed because false positive detection is occurred frequently
+  by security scanning tools.
+  https://github.com/fluent/fluentd/pull/3450
+
 # v1.13.1
 
 ## Release v1.13.1 - 2021/06/25

data/README.md

@@ -25,9 +25,9 @@ An event consists of *tag*, *time* and *record*. Tag is a string separated with
 
 Use Case | Description | Diagram
 -------- | ------------|:---------:
-Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/assets/img/recipes/elasticsearch-s3-fluentd.png" height="150"/>
+Centralizing Apache/Nginx Server Logs | Fluentd can be used to tail access/error logs and transport them reliably to remote systems. | <img src="https://www.fluentd.org/images/recipes/elasticsearch-s3-fluentd.png" height="150"/>
 Syslog Alerting | Fluentd can "grep" for events and send out alerts. | <img src="https://www.fluentd.org/images/syslog-fluentd-alert.png" height="100"/>
-Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/assets/img/datasources/asynchronous_logging.png" height="150"/>
+Mobile/Web Application Logging | Fluentd can function as middleware to enable asynchronous, scalable logging for user action events. | <img src="https://www.fluentd.org/images/datasources/asynchronous_logging.png" height="150"/>
 
 ## Quick Start
 

data/fluentd.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |gem|
   gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
   gem.add_runtime_dependency("cool.io", [">= 1.4.5", "< 2.0.0"])
   gem.add_runtime_dependency("serverengine", [">= 2.2.2", "< 3.0.0"])
-  gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.7.0"])
+  gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.8.0"])
   gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
   gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
   gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])

data/lib/fluent/command/plugin_generator.rb

@@ -27,7 +27,7 @@ class FluentPluginGenerator
   attr_reader :type, :name
   attr_reader :license_name
 
-  SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter"]
+  SUPPORTED_TYPES = ["input", "output", "filter", "parser", "formatter", "storage"]
 
   def initialize(argv = ARGV)
     @argv = argv
@@ -164,8 +164,13 @@ BANNER
   end
 
   def locked_gem_version(gem_name)
-    d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
-    d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
+    if File.exist?(lock_file_path)
+      d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
+      d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
+    else
+      # fallback even though Fluentd is installed without bundler
+      Gem::Specification.find_by_name(gem_name).version.version
+    end
   end
 
   def rake_version
@@ -177,8 +182,13 @@ BANNER
   end
 
   def bundler_version
-    d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
-    d.locked_gems.bundler_version.version
+    if File.exist?(lock_file_path)
+      d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
+      d.locked_gems.bundler_version.version
+    else
+      # fallback even though Fluentd is installed without bundler
+      Gem::Specification.find_by_name("bundler").version.version
+    end
   end
 
   def class_name

data/lib/fluent/config.rb

@@ -62,7 +62,7 @@ module Fluent
         Parser.parse(str, fname, basepath)
       when :ruby
         require 'fluent/config/dsl'
-        $log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file")
+        $log.warn("Ruby DSL configuration format is deprecated. Please use original configuration format. https://docs.fluentd.org/configuration/config-file") if $log
         Config::DSL::Parser.parse(str, File.join(basepath, fname))
       else
         raise "[BUG] unknown configuration parser specification:'#{parser}'"

data/lib/fluent/config/types.rb

@@ -20,6 +20,10 @@ require 'fluent/config/error'
 
 module Fluent
   module Config
+    def self.reformatted_value(type, val, opts = {}, name = nil)
+      REFORMAT_VALUE.call(type, val, opts, name)
+    end
+
     def self.size_value(str, opts = {}, name = nil)
       return nil if str.nil?
 
@@ -104,6 +108,16 @@ module Fluent
       Config.string_value(val, opts, name)
     }
 
+    def self.symbol_value(val, opts = {}, name = nil)
+      return nil if val.nil? || val.empty?
+
+      val.delete_prefix(":").to_sym
+    end
+
+    SYMBOL_TYPE = Proc.new { |val, opts = {}, name = nil|
+      Config.symbol_value(val, opts, name)
+    }
+
     def self.enum_value(val, opts = {}, name = nil)
       return nil if val.nil?
 
@@ -176,6 +190,7 @@ module Fluent
         when :bool then Config.bool_value(value, opts, name)
         when :time then Config.time_value(value, opts, name)
         when :regexp then Config.regexp_value(value, opts, name)
+        when :symbol then Config.symbol_value(value, opts, name)
         else
           raise "unknown type in REFORMAT: #{type}"
         end

data/lib/fluent/config/v1_parser.rb

@@ -37,6 +37,7 @@ module Fluent
         super(strscan, eval_context)
         @include_basepath = include_basepath
         @fname = fname
+        @logger = defined?($log) ? $log : nil
       end
 
       def parse!
@@ -99,7 +100,7 @@ module Fluent
 
           elsif root_element && skip(/(\@include|include)#{SPACING}/)
             if !prev_match.start_with?('@')
-              $log.warn "'include' is deprecated. Use '@include' instead"
+              @logger.warn "'include' is deprecated. Use '@include' instead" if @logger
             end
             parse_include(attrs, elems)
 
@@ -123,7 +124,7 @@ module Fluent
                     parse_error! "'@' is the system reserved prefix. Don't use '@' prefix parameter in the configuration: #{k}"
                   else
                     # TODO: This is for backward compatibility. It will throw an error in the future.
-                    $log.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}"
+                    @logger.warn "'@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: #{k}" if @logger
                   end
                 end
 

data/lib/fluent/env.rb

@@ -15,13 +15,14 @@
 #
 
 require 'serverengine/utils'
+require 'fluent/oj_options'
 
 module Fluent
   DEFAULT_CONFIG_PATH = ENV['FLUENT_CONF'] || '/etc/fluent/fluent.conf'
   DEFAULT_PLUGIN_DIR = ENV['FLUENT_PLUGIN'] || '/etc/fluent/plugin'
   DEFAULT_SOCKET_PATH = ENV['FLUENT_SOCKET'] || '/var/run/fluent/fluent.sock'
   DEFAULT_BACKUP_DIR = ENV['FLUENT_BACKUP_DIR'] || '/tmp/fluent'
-  DEFAULT_OJ_OPTIONS = {bigdecimal_load: :float, mode: :compat, use_to_json: true}
+  DEFAULT_OJ_OPTIONS = Fluent::OjOptions.load_env
   DEFAULT_DIR_PERMISSION = 0755
   DEFAULT_FILE_PERMISSION = 0644
 

data/lib/fluent/oj_options.rb

@@ -0,0 +1,62 @@
+require 'fluent/config/types'
+
+module Fluent
+  class OjOptions
+    OPTIONS = {
+      'bigdecimal_load': :symbol,
+      'max_nesting': :integer,
+      'mode': :symbol,
+      'use_to_json': :bool
+    }
+
+    ALLOWED_VALUES = {
+      'bigdecimal_load': %i[bigdecimal float auto],
+      'mode': %i[strict null compat json rails object custom]
+    }
+
+    DEFAULTS = {
+      'bigdecimal_load': :float,
+      'mode': :compat,
+      'use_to_json': true
+    }
+
+    @@available = false
+
+    def self.available?
+      @@available
+    end
+
+    def self.load_env
+      options = self.get_options
+      begin
+        require 'oj'
+        Oj.default_options = options
+        @@available = true
+      rescue LoadError
+        @@available = false
+      end
+      options
+    end
+
+    private
+
+    def self.get_options
+      options = {}
+      DEFAULTS.each { |key, value| options[key] = value }
+
+      OPTIONS.each do |key, type|
+        env_value = ENV["FLUENT_OJ_OPTION_#{key.upcase}"]
+        next if env_value.nil?
+
+        cast_value = Fluent::Config.reformatted_value(OPTIONS[key], env_value, { strict: true })
+        next if cast_value.nil?
+
+        next if ALLOWED_VALUES[key] && !ALLOWED_VALUES[key].include?(cast_value)
+
+        options[key.to_sym] = cast_value
+      end
+
+      options
+    end
+  end
+end

data/lib/fluent/plugin/formatter.rb

@@ -14,6 +14,7 @@
 #    limitations under the License.
 #
 
+require 'fluent/env'
 require 'fluent/plugin/base'
 require 'fluent/plugin/owned_by_mixin'
 require 'fluent/time'

data/lib/fluent/plugin/formatter_json.rb

@@ -15,7 +15,7 @@
 #
 
 require 'fluent/plugin/formatter'
-require 'fluent/env'
+require 'fluent/oj_options'
 
 module Fluent
   module Plugin
@@ -30,12 +30,14 @@ module Fluent
       def configure(conf)
         super
 
-        begin
-          raise LoadError unless @json_parser == 'oj'
-          require 'oj'
-          Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
-          @dump_proc = Oj.method(:dump)
-        rescue LoadError
+        if @json_parser == 'oj'
+          if Fluent::OjOptions.available?
+            @dump_proc = Oj.method(:dump)
+          else
+            log.info "Oj isn't installed, fallback to Yajl as json parser"
+            @dump_proc = Yajl.method(:dump)
+          end
+        else
           @dump_proc = Yajl.method(:dump)
         end
 

data/lib/fluent/plugin/in_tail.rb

@@ -522,8 +522,25 @@ module Fluent::Plugin
     def detach_watcher_after_rotate_wait(tw, ino)
       # Call event_loop_attach/event_loop_detach is high-cost for short-live object.
       # If this has a problem with large number of files, use @_event_loop directly instead of timer_execute.
-      timer_execute(:in_tail_close_watcher, @rotate_wait, repeat: false) do
+      if @open_on_every_update
+        # Detach now because it's already closed, waiting it doesn't make sense.
         detach_watcher(tw, ino)
+      elsif @read_bytes_limit_per_second < 0
+        # throttling isn't enabled, just wait @rotate_wait
+        timer_execute(:in_tail_close_watcher, @rotate_wait, repeat: false) do
+          detach_watcher(tw, ino)
+        end
+      else
+        # When the throttling feature is enabled, it might not reach EOF yet.
+        # Should ensure to read all contents before closing it, with keeping throttling.
+        start_time_to_wait = Fluent::Clock.now
+        timer = timer_execute(:in_tail_close_watcher, 1, repeat: true) do
+          elapsed = Fluent::Clock.now - start_time_to_wait
+          if tw.eof? && elapsed >= @rotate_wait
+            timer.detach
+            detach_watcher(tw, ino)
+          end
+        end
       end
     end
 
@@ -736,6 +753,10 @@ module Fluent::Plugin
         end
       end
 
+      def eof?
+        @io_handler.eof?
+      end
+
       def on_notify
         begin
           stat = Fluent::FileWrapper.stat(@path)
@@ -923,6 +944,7 @@ module Fluent::Plugin
           @shutdown_start_time = nil
           @shutdown_timeout = SHUTDOWN_TIMEOUT
           @shutdown_mutex = Mutex.new
+          @eof = false
 
           @log.info "following tail of #{@path}"
         end
@@ -949,6 +971,10 @@ module Fluent::Plugin
           !!@io
         end
 
+        def eof?
+          @eof
+        end
+
         private
 
         def limit_bytes_per_second_reached?
@@ -989,6 +1015,7 @@ module Fluent::Plugin
                   while true
                     @start_reading_time ||= Fluent::Clock.now
                     data = io.readpartial(BYTES_TO_READ, @iobuf)
+                    @eof = false
                     @number_bytes_read += data.bytesize
                     @fifo << data
                     @fifo.read_lines(@lines)
@@ -1005,6 +1032,7 @@ module Fluent::Plugin
                     end
                   end
                 rescue EOFError
+                  @eof = true
                 end
               end
 
@@ -1042,14 +1070,17 @@ module Fluent::Plugin
           else
             @io ||= open
             yield @io
+            @eof = true if @io.nil?
           end
         rescue WatcherSetupError => e
           close
+          @eof = true
           raise e
         rescue
           @log.error $!.to_s
           @log.error_backtrace
           close
+          @eof = true
         end
       end
 

data/lib/fluent/plugin/parser_json.rb

@@ -15,8 +15,8 @@
 #
 
 require 'fluent/plugin/parser'
-require 'fluent/env'
 require 'fluent/time'
+require 'fluent/oj_options'
 
 require 'yajl'
 require 'json'
@@ -50,8 +50,7 @@ module Fluent
       def configure_json_parser(name)
         case name
         when :oj
-          require 'oj'
-          Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
+          raise LoadError unless Fluent::OjOptions.available?
           [Oj.method(:load), Oj::ParseError]
         when :json then [JSON.method(:load), JSON::ParserError]
         when :yajl then [Yajl.method(:load), Yajl::ParseError]

data/lib/fluent/test/driver/storage.rb

@@ -0,0 +1,30 @@
+#
+# Fluentd
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+require 'fluent/test/driver/base_owned'
+
+module Fluent
+  module Test
+    module Driver
+      class Storage < BaseOwned
+        def initialize(klass, **kwargs, &block)
+          super
+          @section_name = "storage"
+        end
+      end
+    end
+  end
+end

data/lib/fluent/version.rb

@@ -16,6 +16,6 @@
 
 module Fluent
 
-  VERSION = '1.13.1'
+  VERSION = '1.13.2'
 
 end

data/templates/new_gem/lib/fluent/plugin/storage.rb.erb

@@ -0,0 +1,40 @@
+<%= preamble %>
+
+require "fluent/plugin/storage"
+
+module Fluent
+  module Plugin
+    class <%= class_name %> < Fluent::Plugin::Storage
+      Fluent::Plugin.register_storage("<%= plugin_name %>", self)
+
+      def initialize
+        super
+      end
+
+      def configure(conf)
+        super
+      end
+
+      def load
+      end
+
+      def save
+      end
+
+      def get(key)
+      end
+
+      def fetch(key, defval)
+      end
+
+      def put(key, value)
+      end
+
+      def delete(key)
+      end
+
+      def update(key, &block)
+      end
+    end
+  end
+end

data/templates/new_gem/test/plugin/test_storage.rb.erb

@@ -0,0 +1,18 @@
+require "helper"
+require "fluent/plugin/<%= plugin_filename %>"
+
+class <%= class_name %>Test < Test::Unit::TestCase
+  setup do
+    Fluent::Test.setup
+  end
+
+  test "failure" do
+    flunk
+  end
+
+  private
+
+  def create_driver(conf)
+    Fluent::Test::Driver::Storage.new(Fluent::Plugin::<%= class_name %>).configure(conf)
+  end
+end

data/test/command/test_plugin_generator.rb

@@ -29,7 +29,8 @@ class TestFluentPluginGenerator < Test::Unit::TestCase
          output: ["output", "out"],
          filter: ["filter", "filter"],
          parser: ["parser", "parser"],
-         formatter: ["formatter", "formatter"])
+         formatter: ["formatter", "formatter"],
+         storage: ["storage", "storage"])
     test "generate plugin" do |(type, part)|
       generator = FluentPluginGenerator.new([type, "fake"])
       stub_git_process(generator)

data/test/config/test_types.rb

@@ -180,6 +180,13 @@ class TestConfigTypes < ::Test::Unit::TestCase
       assert_equal Encoding::UTF_8, actual.encoding
     end
 
+    data('starts_with_semicolon' => [:conor, ':conor'],
+         'simple_string' => [:conor, 'conor'],
+         'empty_string' => [nil, ''])
+    test 'symbol' do |(expected, val)|
+      assert_equal Config::SYMBOL_TYPE.call(val, {}), expected
+    end
+
     data("val" => [:val, 'val'],
          "v" => [:v, 'v'],
          "value" => [:value, 'value'])

data/test/plugin/test_in_tail.rb

@@ -456,6 +456,91 @@ class TailInputTest < Test::Unit::TestCase
           assert_equal([], d.events)
         end
       end
+
+      sub_test_case "EOF with reads_bytes_per_second" do
+        def test_longer_than_rotate_wait
+          limit_bytes = 8192
+          num_lines = 1024 * 3
+          msg = "08bytes"
+
+          File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
+            f.write("#{msg}\n" * num_lines)
+          end
+
+          config = CONFIG_READ_FROM_HEAD +
+                   SINGLE_LINE_CONFIG +
+                   config_element("", "", {
+                                    "read_bytes_limit_per_second" => limit_bytes,
+                                    "rotate_wait" => 0.1,
+                                    "refresh_interval" => 0.5,
+                                  })
+
+          rotated = false
+          d = create_driver(config)
+          d.run(timeout: 10) do
+            while d.events.size < num_lines do
+              if d.events.size > 0 && !rotated
+                cleanup_file("#{TMP_DIR}/tail.txt")
+                FileUtils.touch("#{TMP_DIR}/tail.txt")
+                rotated = true
+              end
+              sleep 0.3
+            end
+          end
+
+          assert_equal(num_lines,
+                       d.events.count do |event|
+                         event[2]["message"] == msg
+                       end)
+        end
+
+        def test_shorter_than_rotate_wait
+          limit_bytes = 8192
+          num_lines = 1024 * 2
+          msg = "08bytes"
+
+          File.open("#{TMP_DIR}/tail.txt", "wb") do |f|
+            f.write("#{msg}\n" * num_lines)
+          end
+
+          config = CONFIG_READ_FROM_HEAD +
+                   SINGLE_LINE_CONFIG +
+                   config_element("", "", {
+                                    "read_bytes_limit_per_second" => limit_bytes,
+                                    "rotate_wait" => 2,
+                                    "refresh_interval" => 0.5,
+                                  })
+
+          start_time = Fluent::Clock.now
+          rotated = false
+          detached = false
+          d = create_driver(config)
+          mock.proxy(d.instance).setup_watcher(anything, anything) do |tw|
+            mock.proxy(tw).detach(anything) do |v|
+              detached = true
+              v
+            end
+            tw
+          end.twice
+
+          d.run(timeout: 10) do
+            until detached do
+              if d.events.size > 0 && !rotated
+                cleanup_file("#{TMP_DIR}/tail.txt")
+                FileUtils.touch("#{TMP_DIR}/tail.txt")
+                rotated = true
+              end
+              sleep 0.3
+            end
+          end
+
+          assert_true(Fluent::Clock.now - start_time > 2)
+          assert_equal(num_lines,
+                       d.events.count do |event|
+                         event[2]["message"] == msg
+                       end)
+        end
+      end
     end
 
     data(flat: CONFIG_READ_FROM_HEAD + SINGLE_LINE_CONFIG,

data/test/test_event_time.rb

@@ -64,8 +64,8 @@ class EventTimeTest < Test::Unit::TestCase
 
   test 'Oj.dump' do
     time = Fluent::EventTime.new(100)
-    require 'fluent/env'
-    Oj.default_options = Fluent::DEFAULT_OJ_OPTIONS
+    require 'fluent/oj_options'
+    Fluent::OjOptions.load_env
     assert_equal('{"time":100}', Oj.dump({'time' => time}))
     assert_equal('["tag",100,{"key":"value"}]', Oj.dump(["tag", time, {"key" => "value"}], mode: :compat))
   end

data/test/test_oj_options.rb

@@ -0,0 +1,55 @@
+require_relative 'helper'
+require 'fluent/test'
+require 'fluent/oj_options'
+
+class OjOptionsTest < ::Test::Unit::TestCase
+  begin
+    require 'oj'
+    @@oj_is_avaibale = true
+  rescue LoadError
+    @@oj_is_avaibale = false
+  end
+
+  setup do
+    @orig_env = {}
+    ENV.each do |key, value|
+      @orig_env[key] = value if key.start_with?("FLUENT_OJ_OPTION_")
+    end
+  end
+
+  teardown do
+    ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
+    @orig_env.each { |key, value| ENV[key] = value }
+  end
+
+  test "available?" do
+    assert_equal(@@oj_is_avaibale, Fluent::OjOptions.available?)
+  end
+
+  sub_test_case "set by environment variable" do
+    test "when no env vars set, returns default options" do
+      ENV.delete_if { |key| key.start_with?("FLUENT_OJ_OPTION_") }
+      defaults = Fluent::OjOptions::DEFAULTS
+      assert_equal(defaults, Fluent::OjOptions.load_env)
+      assert_equal(defaults, Oj.default_options.slice(*defaults.keys)) if @@oj_is_avaibale
+    end
+
+    test "valid env var passed with valid value, default is overridden" do
+      ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":bigdecimal"
+      assert_equal(:bigdecimal, Fluent::OjOptions.load_env[:bigdecimal_load])
+      assert_equal(:bigdecimal, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
+    end
+
+    test "valid env var passed with invalid value, default is not overriden" do
+      ENV["FLUENT_OJ_OPTION_BIGDECIMAL_LOAD"] = ":conor"
+      assert_equal(:float, Fluent::OjOptions.load_env[:bigdecimal_load])
+      assert_equal(:float, Oj.default_options[:bigdecimal_load]) if @@oj_is_avaibale
+    end
+
+    test "invalid env var passed, nothing done with it" do
+      ENV["FLUENT_OJ_OPTION_CONOR"] = ":conor"
+      assert_equal(nil, Fluent::OjOptions.load_env[:conor])
+      assert_equal(nil, Oj.default_options[:conor]) if @@oj_is_avaibale
+    end
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluentd
 version: !ruby/object:Gem::Version
-  version: 1.13.1
+  version: 1.13.2
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-06-25 00:00:00.000000000 Z
+date: 2021-07-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -107,7 +107,7 @@ dependencies:
         version: 0.5.1
     - - "<"
       - !ruby/object:Gem::Version
-        version: 0.7.0
+        version: 0.8.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -117,7 +117,7 @@ dependencies:
         version: 0.5.1
     - - "<"
       - !ruby/object:Gem::Version
-        version: 0.7.0
+        version: 0.8.0
 - !ruby/object:Gem::Dependency
   name: sigdump
   requirement: !ruby/object:Gem::Requirement
@@ -373,9 +373,9 @@ files:
 - ".deepsource.toml"
 - ".drone.yml"
 - ".github/ISSUE_TEMPLATE.md"
-- ".github/ISSUE_TEMPLATE/bug_report.md"
+- ".github/ISSUE_TEMPLATE/bug_report.yaml"
 - ".github/ISSUE_TEMPLATE/config.yml"
-- ".github/ISSUE_TEMPLATE/feature_request.md"
+- ".github/ISSUE_TEMPLATE/feature_request.yaml"
 - ".github/PULL_REQUEST_TEMPLATE.md"
 - ".github/workflows/issue-auto-closer.yml"
 - ".github/workflows/linux-test.yaml"
@@ -519,6 +519,7 @@ files:
 - lib/fluent/match.rb
 - lib/fluent/mixin.rb
 - lib/fluent/msgpack_factory.rb
+- lib/fluent/oj_options.rb
 - lib/fluent/output.rb
 - lib/fluent/output_chain.rb
 - lib/fluent/parser.rb
@@ -664,6 +665,7 @@ files:
 - lib/fluent/test/driver/multi_output.rb
 - lib/fluent/test/driver/output.rb
 - lib/fluent/test/driver/parser.rb
+- lib/fluent/test/driver/storage.rb
 - lib/fluent/test/driver/test_event_router.rb
 - lib/fluent/test/filter_test.rb
 - lib/fluent/test/formatter_test.rb
@@ -689,12 +691,14 @@ files:
 - templates/new_gem/lib/fluent/plugin/input.rb.erb
 - templates/new_gem/lib/fluent/plugin/output.rb.erb
 - templates/new_gem/lib/fluent/plugin/parser.rb.erb
+- templates/new_gem/lib/fluent/plugin/storage.rb.erb
 - templates/new_gem/test/helper.rb.erb
 - templates/new_gem/test/plugin/test_filter.rb.erb
 - templates/new_gem/test/plugin/test_formatter.rb.erb
 - templates/new_gem/test/plugin/test_input.rb.erb
 - templates/new_gem/test/plugin/test_output.rb.erb
 - templates/new_gem/test/plugin/test_parser.rb.erb
+- templates/new_gem/test/plugin/test_storage.rb.erb
 - templates/plugin_config_formatter/param.md-compact.erb
 - templates/plugin_config_formatter/param.md-table.erb
 - templates/plugin_config_formatter/param.md.erb
@@ -899,6 +903,7 @@ files:
 - test/test_match.rb
 - test/test_mixin.rb
 - test/test_msgpack_factory.rb
+- test/test_oj_options.rb
 - test/test_output.rb
 - test/test_plugin.rb
 - test/test_plugin_classes.rb
@@ -1138,6 +1143,7 @@ test_files:
 - test/test_match.rb
 - test/test_mixin.rb
 - test/test_msgpack_factory.rb
+- test/test_oj_options.rb
 - test/test_output.rb
 - test/test_plugin.rb
 - test/test_plugin_classes.rb

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,40 +0,0 @@
----
-name:  Bug Report
-about: Create a report with a procedure for reproducing the bug
-
----
-
-Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
-
-**Describe the bug**
-<!-- A clear and concise description of what the bug is. -->
-
-**To Reproduce**
-<!-- Steps to reproduce the behavior: -->
-
-**Expected behavior**
-<!-- A clear and concise description of what you expected to happen. -->
-
-**Your Environment**
-
-- Fluentd or td-agent version: `fluentd --version` or `td-agent --version`
-- Operating system: `cat /etc/os-release`
-- Kernel version: `uname -r`
-
-If you hit the problem with older fluentd version, try latest version first.
-
-**Your Configuration**
-
-```
-<!-- Write your configuration here -->
-```
-
-**Your Error Log**
-
-```
-<!-- Write your **ALL** error log here -->
-```
-
-**Additional context**
-
-<!-- Add any other context about the problem here. -->

data/.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,23 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for this project
-
----
-
-Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem.
-
-**Is your feature request related to a problem? Please describe.**
-
-<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
-
-**Describe the solution you'd like**
-
-<!-- A clear and concise description of what you want to happen. -->
-
-**Describe alternatives you've considered**
-
-<!-- A clear and concise description of any alternative solutions or features you've considered. -->
-
-**Additional context**
-
-<!-- Add any other context or screenshots about the feature request here. -->