fluentd 1.12.0-x64-mingw32 → 1.13.0-x64-mingw32

data/CONTRIBUTING.md

@@ -6,7 +6,7 @@ We'd love your contribution. Here are the guidelines!
 
 RESOURCES of [Official site](https://www.fluentd.org/) and [Fluentd documentation](https://docs.fluentd.org/) may help you.
 
-If you have further questions about Fluentd and plugins, please direct these to [Mailing List](https://groups.google.com/forum/#!forum/fluentd).
+If you have further questions about Fluentd and plugins, please direct these to [Community Forum](https://discuss.fluentd.org/).
 Don't use Github issue for asking questions. Here are examples:
 
 - I installed xxx plugin but it doesn't work. Why?
@@ -16,7 +16,7 @@ Don't use Github issue for asking questions. Here are examples:
 We may close such questions to keep clear repository for developers and users.
 Github issue is mainly for submitting a bug report or feature request. See below.
 
-If you can't judge your case is a bug or not, use mailing list or slack first.
+If you can't judge your case is a bug or not, use community forum or slack first.
 
 ## Found a bug?
 

data/MAINTAINERS.md

@@ -1,8 +1,11 @@
 # Fluentd Maintainers
 
 - [Naotoshi Seo](https://github.com/sonots), [ZOZO Technologies](https://tech.zozo.com/en/)
-- [Okkez](https://github.com/okkez), [Clearcode](https://www.clear-code.com/)
-- [Hiroshi Hatake](https://github.com/cosmo0920), [Clearcode](https://www.clear-code.com/)
+- [Okkez](https://github.com/okkez)
+- [Hiroshi Hatake](https://github.com/cosmo0920), [Calyptia](https://www.calyptia.com/)
 - [Masahiro Nakagawa](https://github.com/repeatedly), [Treasure Data](https://www.treasuredata.com/)
 - [Satoshi Tagomori](https://github.com/tagomoris), [Treasure Data](https://www.treasuredata.com/)
 - [Eduardo Silva](https://github.com/edsiper), [Arm Treasure Data](https://www.treasuredata.com/)
+- [Fujimoto Seiji](https://github.com/fujimots), [ClearCode](https://www.clear-code.com/)
+- [Takuro Ashie](https://github.com/ashie), [ClearCode](https://www.clear-code.com/)
+- [Kentaro Hayashi](https://github.com/kenhys), [ClearCode](https://www.clear-code.com/)

data/README.md

@@ -1,8 +1,11 @@
 Fluentd: Open-Source Log Collector
 ===================================
-Travis CI:
+GitHub Actions:
 
-[<img src="https://travis-ci.org/fluent/fluentd.svg" />](https://travis-ci.org/fluent/fluentd) [![Code Climate](https://codeclimate.com/github/fluent/fluentd/badges/gpa.svg)](https://codeclimate.com/github/fluent/fluentd)
+[![Testing on Ubuntu](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/linux-test.yaml)
+[![Testing on Windows](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/windows-test.yaml)
+[![Testing on macOS](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml/badge.svg?branch=master)](https://github.com/fluent/fluentd/actions/workflows/macos-test.yaml)
+[![Code Climate](https://codeclimate.com/github/fluent/fluentd/badges/gpa.svg)](https://codeclimate.com/github/fluent/fluentd)
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1189/badge)](https://bestpractices.coreinfrastructure.org/projects/1189)
 [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd?ref=badge_shield)
 
@@ -74,11 +77,11 @@ You can run specified test via `TEST` environment variable:
 - Website: https://www.fluentd.org/
 - Documentation: https://docs.fluentd.org/
 - Project repository: https://github.com/fluent
-- Discussion: https://groups.google.com/group/fluentd
+- Discussion: https://discuss.fluentd.org/
 - Slack / Community: https://slack.fluentd.org
 - Newsletters: https://www.fluentd.org/newsletter
 - Author: [Sadayuki Furuhashi](https://github.com/frsyuki)
-- Copyright: 2011-2019 Fluentd Authors
+- Copyright: 2011-2021 Fluentd Authors
 - License: Apache License, Version 2.0
 
 ## Security

data/example/counter.conf

@@ -3,7 +3,7 @@
     scope server1
     bind 127.0.0.1
     port 24321
-    path tmp/back
+    backup_path tmp/back
   </counter_server>
 </system>
 

data/fluentd.gemspec

@@ -28,12 +28,14 @@ Gem::Specification.new do |gem|
   gem.add_runtime_dependency("tzinfo", [">= 1.0", "< 3.0"])
   gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])
   gem.add_runtime_dependency("strptime", [">= 0.2.2", "< 1.0.0"])
+  gem.add_runtime_dependency("webrick", [">= 1.4.2", "< 1.8.0"])
 
   # build gem for a certain platform. see also Rakefile
   fake_platform = ENV['GEM_BUILD_FAKE_PLATFORM'].to_s
   gem.platform = fake_platform unless fake_platform.empty?
   if /mswin|mingw/ =~ fake_platform || (/mswin|mingw/ =~ RUBY_PLATFORM && fake_platform.empty?)
-    gem.add_runtime_dependency("win32-service", ["~> 2.1.5"])
+    gem.add_runtime_dependency("win32-api", [">= 1.10", "< 2.0.0"])
+    gem.add_runtime_dependency("win32-service", ["~> 2.2.0"])
     gem.add_runtime_dependency("win32-ipc", ["~> 0.7.0"])
     gem.add_runtime_dependency("win32-event", ["~> 0.6.3"])
     gem.add_runtime_dependency("windows-pr", ["~> 1.2.6"])
@@ -44,11 +46,10 @@ Gem::Specification.new do |gem|
   gem.add_development_dependency("flexmock", ["~> 2.0"])
   gem.add_development_dependency("parallel_tests", ["~> 0.15.3"])
   gem.add_development_dependency("simplecov", ["~> 0.7"])
-  gem.add_development_dependency("rr", ["~> 1.0"])
+  gem.add_development_dependency("rr", ["~> 3.0"])
   gem.add_development_dependency("timecop", ["~> 0.9"])
   gem.add_development_dependency("test-unit", ["~> 3.3"])
   gem.add_development_dependency("test-unit-rr", ["~> 1.0"])
   gem.add_development_dependency("oj", [">= 2.14", "< 4"])
-  gem.add_development_dependency("ext_monitor", [">= 0.1.2", "< 0.2"])
-  gem.add_development_dependency("async-http")
+  gem.add_development_dependency("async-http", ">= 0.50.0")
 end

data/lib/fluent/command/bundler_injection.rb

@@ -40,6 +40,6 @@ else
     File.expand_path(File.join(File.dirname(__FILE__), 'fluentd.rb')),
   ] + ARGV
 
-  exec *cmdline
+  exec(*cmdline)
   exit! 127
 end

data/lib/fluent/command/ca_generate.rb

@@ -75,6 +75,8 @@ HELP
 
       digest = OpenSSL::Digest::SHA256.new
 
+      factory = OpenSSL::X509::ExtensionFactory.new
+
       cert = OpenSSL::X509::Certificate.new
       cert.not_before = Time.at(0)
       cert.not_after = Time.now + 5 * 365 * 86400 # 5 years after
@@ -82,7 +84,7 @@ HELP
       cert.serial = 1
       cert.issuer = issuer
       cert.subject = subject
-      cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(true)]))
+      cert.add_extension(factory.create_extension('basicConstraints', 'CA:TRUE'))
       cert.sign(key, digest)
 
       return cert, key
@@ -111,8 +113,9 @@ HELP
       cert.issuer = issuer
       cert.subject = subject
 
-      cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(false)]))
-      cert.add_extension OpenSSL::X509::Extension.new('nsCertType', 'server')
+      factory = OpenSSL::X509::ExtensionFactory.new
+      server_cert.add_extension(factory.create_extension('basicConstraints', 'CA:FALSE'))
+      server_cert.add_extension(factory.create_extension('nsCertType', 'server'))
 
       cert.sign ca_key, digest
 

data/lib/fluent/command/cat.rb

@@ -101,7 +101,6 @@ rescue
   usage $!.to_s
 end
 
-require 'thread'
 require 'socket'
 require 'yajl'
 require 'msgpack'
@@ -153,14 +152,30 @@ class Writer
     super()
   end
 
+  def secondary_record?(record)
+    record.class != Hash &&
+      record.size == 2 &&
+      record.first.class == Fluent::EventTime &&
+      record.last.class == Hash
+  end
+
   def write(record)
-    if record.class != Hash
-      raise ArgumentError, "Input must be a map (got #{record.class})"
+    unless secondary_record?(record)
+      if record.class != Hash
+        raise ArgumentError, "Input must be a map (got #{record.class})"
+      end
     end
 
     time = Fluent::EventTime.now
     time = time.to_i if @time_as_integer
-    entry = [time, record]
+    entry = if secondary_record?(record)
+              # Even though secondary contains Fluent::EventTime in record,
+              # fluent-cat just ignore it and set Fluent::EventTime.now instead.
+              # This specification is adopted to keep consistency.
+              [time, record.last]
+            else
+              [time, record]
+            end
     synchronize {
       unless write_impl([entry])
         # write failed

data/lib/fluent/command/fluentd.rb

@@ -85,9 +85,8 @@ op.on('-o', '--log PATH', "log file path") {|s|
   opts[:log_path] = s
 }
 
-ROTATE_AGE = %w(daily weekly monthly)
 op.on('--log-rotate-age AGE', 'generations to keep rotated log files') {|age|
-  if ROTATE_AGE.include?(age)
+  if Fluent::Log::LOG_ROTATE_AGE.include?(age)
     opts[:log_rotate_age] = age
   else
     begin
@@ -163,6 +162,10 @@ op.on('--conf-encoding ENCODING', "specify configuration file encoding") { |s|
   opts[:conf_encoding] = s
 }
 
+op.on('--disable-shared-socket', "Don't open shared socket for multiple workers") { |b|
+  opts[:disable_shared_socket] = b
+}
+
 if Fluent.windows?
   require 'windows/library'
   include Windows::Library

data/lib/fluent/command/plugin_config_formatter.rb

@@ -44,6 +44,7 @@ class FluentPluginConfigFormatter
     @verbose = false
     @libs = []
     @plugin_dirs = []
+    @table = false
     @options = {}
 
     prepare_option_parser
@@ -162,9 +163,20 @@ class FluentPluginConfigFormatter
     else
       sections, params = base_section.partition {|_name, value| value[:section] }
     end
+    if @table && (not params.empty?)
+      dumped << "### Configuration\n\n"
+      dumped << "|parameter|type|description|default|\n"
+      dumped << "|---|---|---|---|\n"
+    end
     params.each do |name, config|
       next if name == :section
-      template_name = @compact ? "param.md-compact.erb" : "param.md.erb"
+      template_name = if @compact
+                        "param.md-compact.erb"
+                      elsif @table
+                        "param.md-table.erb"
+                      else
+                        "param.md.erb"
+                      end
       template = template_path(template_name).read
       dumped <<
         if ERB.instance_method(:initialize).parameters.assoc(:key) # Ruby 2.6+
@@ -257,6 +269,9 @@ BANNER
     @parser.on("-p", "--plugin=DIR", "Add plugin directory") do |s|
       @plugin_dirs << s
     end
+    @parser.on("-t", "--table", "Use table syntax to dump parameters") do
+      @table = true
+    end
   end
 
   def parse_options!

data/lib/fluent/command/plugin_generator.rb

@@ -105,7 +105,7 @@ Generate a project skeleton for creating a Fluentd plugin
 
 Arguments:
 \ttype: #{SUPPORTED_TYPES.join(",")}
-\tname: Your plugin name
+\tname: Your plugin name (fluent-plugin- prefix will be added to <name>)
 
 Options:
 BANNER
@@ -151,6 +151,36 @@ BANNER
     underscore_name
   end
 
+  def gem_file_path
+    File.expand_path(File.join(File.dirname(__FILE__),
+                               "../../../",
+                               "Gemfile"))
+  end
+
+  def lock_file_path
+    File.expand_path(File.join(File.dirname(__FILE__),
+                               "../../../",
+                               "Gemfile.lock"))
+  end
+
+  def locked_gem_version(gem_name)
+    d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
+    d.locked_gems.dependencies[gem_name].requirement.requirements.first.last.version
+  end
+
+  def rake_version
+    locked_gem_version("rake")
+  end
+
+  def test_unit_version
+    locked_gem_version("test-unit")
+  end
+
+  def bundler_version
+    d = Bundler::Definition.build(gem_file_path, lock_file_path, false)
+    d.locked_gems.bundler_version.version
+  end
+
   def class_name
     "#{capitalized_name}#{type.capitalize}"
   end

data/lib/fluent/compat/parser.rb

@@ -244,10 +244,10 @@ module Fluent
         end
 
         def convert_value_to_nil(value)
-          if value and @null_empty_string
+          if value && @null_empty_string
             value = (value == '') ? nil : value
           end
-          if value and @null_value_pattern
+          if value && @null_value_pattern
             value = ::Fluent::StringUtil.match_regexp(@null_value_pattern, value) ? nil : value
           end
           value

data/lib/fluent/config/section.rb

@@ -179,7 +179,7 @@ module Fluent
             end
 
             if section_params[varname].nil?
-              unless proxy.defaults.has_key?(varname) and proxy.defaults[varname].nil?
+              unless proxy.defaults.has_key?(varname) && proxy.defaults[varname].nil?
                 logger.error "config error in:\n#{conf}" if logger
                 raise ConfigError, "'#{name}' parameter is required but nil is specified"
               end
@@ -247,7 +247,7 @@ module Fluent
       def self.check_unused_section(proxy, conf, plugin_class)
         elems = conf.respond_to?(:elements) ? conf.elements : []
         elems.each { |e|
-          next if plugin_class.nil? && Fluent::Config::V1Parser::ELEM_SYMBOLS.include?(e.name) # skip pre-defined non-plugin elements because it doens't have proxy section
+          next if plugin_class.nil? && Fluent::Config::V1Parser::ELEM_SYMBOLS.include?(e.name) # skip pre-defined non-plugin elements because it doesn't have proxy section
           next if e.unused_in && e.unused_in.empty? # the section is used at least once
 
           if proxy.sections.any? { |name, subproxy| e.name == subproxy.name.to_s || e.name == subproxy.alias.to_s }

data/lib/fluent/config/types.rb

@@ -186,7 +186,7 @@ module Fluent
       return nil if val.nil?
 
       param = if val.is_a?(String)
-                val.start_with?('{') ? JSON.load(val) : Hash[val.strip.split(/\s*,\s*/).map{|v| v.split(':', 2)}]
+                val.start_with?('{') ? JSON.parse(val) : Hash[val.strip.split(/\s*,\s*/).map{|v| v.split(':', 2)}]
               else
                 val
               end
@@ -213,7 +213,7 @@ module Fluent
       return nil if val.nil?
 
       param = if val.is_a?(String)
-                val.start_with?('[') ? JSON.load(val) : val.strip.split(/\s*,\s*/)
+                val.start_with?('[') ? JSON.parse(val) : val.strip.split(/\s*,\s*/)
               else
                 val
               end

data/lib/fluent/event.rb

@@ -254,19 +254,9 @@ module Fluent
     end
 
     def each(unpacker: nil, &block)
-      if @unpacked_times
-        @unpacked_times.each_with_index do |time, i|
-          block.call(time, @unpacked_records[i])
-        end
-      else
-        @unpacked_times = []
-        @unpacked_records = []
-        (unpacker || Fluent::MessagePackFactory.msgpack_unpacker).feed_each(@data) do |time, record|
-          @unpacked_times << time
-          @unpacked_records << record
-          block.call(time, record)
-        end
-        @size = @unpacked_times.size
+      ensure_unpacked!(unpacker: unpacker)
+      @unpacked_times.each_with_index do |time, i|
+        block.call(time, @unpacked_records[i])
       end
       nil
     end

data/lib/fluent/load.rb

@@ -1,4 +1,3 @@
-require 'thread'
 require 'socket'
 require 'fcntl'
 require 'time'

data/lib/fluent/log.rb

@@ -49,6 +49,7 @@ module Fluent
     LOG_TYPE_DEFAULT = :default # show logs in all supervisor/workers, with worker id in workers (default)
 
     LOG_TYPES = [LOG_TYPE_SUPERVISOR, LOG_TYPE_WORKER0, LOG_TYPE_DEFAULT].freeze
+    LOG_ROTATE_AGE = %w(daily weekly monthly)
 
     def self.str_to_level(log_level_str)
       case log_level_str.downcase

data/lib/fluent/plugin/file_wrapper.rb

@@ -46,6 +46,48 @@ module Fluent
     end
   end
 
+  class Win32Error < StandardError
+    require 'windows/error'
+    include Windows::Error
+
+    attr_reader :errcode, :msg
+
+    WSABASEERR = 10000
+
+    def initialize(errcode, msg = nil)
+      @errcode = errcode
+      @msg = msg
+    end
+
+    def format_english_message(errcode)
+      buf = 0.chr * 260
+      flags = FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_ARGUMENT_ARRAY
+      english_lang_id = 1033 # The result of MAKELANGID(LANG_ENGLISH, SUBLANG_ENGLISH_US)
+      FormatMessageA.call(flags, 0, errcode, english_lang_id, buf, buf.size, 0)
+      buf.force_encoding(Encoding.default_external).strip
+    end
+
+    def to_s
+      msg = super
+      msg << ": code: #{@errcode}, #{format_english_message(@errcode)}"
+      msg << " - #{@msg}" if @msg
+      msg
+    end
+
+    def inspect
+      "#<#{to_s}>"
+    end
+
+    def ==(other)
+      return false if other.class != Win32Error
+      @errcode == other.errcode && @msg == other.msg
+    end
+
+    def wsaerr?
+      @errcode >= WSABASEERR
+    end
+  end
+
   # To open and get stat with setting FILE_SHARE_DELETE
   class WindowsFile
     require 'windows/file'
@@ -77,11 +119,14 @@ module Fluent
       @file_handle = CreateFile.call(@path, access, sharemode,
                      0, creationdisposition, FILE_ATTRIBUTE_NORMAL, 0)
       if @file_handle == INVALID_HANDLE_VALUE
-        err = GetLastError.call
-        if err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND || err == ERROR_ACCESS_DENIED
-          raise SystemCallError.new(2)
+        win32err = Win32Error.new(Win32::API.last_error, path)
+        errno = ServerEngine::RbWinSock.rb_w32_map_errno(win32err.errcode)
+        if errno == Errno::EINVAL::Errno || win32err.wsaerr?
+          # maybe failed to map
+          raise win32err
+        else
+          raise SystemCallError.new(win32err.message, errno)
         end
-        raise SystemCallError.new(err)
       end
     end