fluentd 1.1.1 → 1.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: '08745174bdefc09fdc359003c6acb65f63d58bac'
-  data.tar.gz: 45e36b62fe808b05273d98eab2ca69c0e46b8c93
+  metadata.gz: e20dab78cc15d81c32ba9cd70fd5a806c6234a73
+  data.tar.gz: 2c133e7665781dcc0d97da8459489ce3edf46f8f
 SHA512:
-  metadata.gz: b855e7e95bd3d6b29b50a6b4ba3321a4fd4166c4e0ef8bd4ab104252bd1b6b738a16eb8a93eae35cbf4b1daf7a8ad209d7ef1cb3be508633071d9490ccb15ffb
-  data.tar.gz: 90754b2ba1765da45e47906866dbb3cfd77bd5373134e9f160b58f3bab3832e1db144124a2665cec19a7e731c70620edf7f14a9d453cd452646263dcc4b14b74
+  metadata.gz: 3d3775f9459c4c5ee8747c39596d2750e51ae6e0bab04958c6162d1faad4a6afd7deb2bcce27ed14b2f1eb925666554d444506cc30408a5db898f29aab3c6258
+  data.tar.gz: c67fcc90f8f3717b73f4d3d78d5899881c688d52d3a39d4b0e112412092973a65d86ca7e9930326f894f3e896013afa42971532db895d67ecbc6e48c5b12180b

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 # v1.1
 
+## Release v1.1.2 - 2018/03/18
+
+### Enhancements
+
+* filter_grep: Support pattern starts with character classes with //
+  https://github.com/fluent/fluentd/pull/1887
+
+### Bug fixes
+
+* in_tail: Handle records in the correct order on file rotation
+  https://github.com/fluent/fluentd/pull/1880
+* out_forward: Fix race condition with <security> on multi thread environment
+  https://github.com/fluent/fluentd/pull/1893
+* output: Prevent flushing threads consume too much CPU when retry happens
+  https://github.com/fluent/fluentd/pull/1901
+* config: Fix boolean param handling for comment without value
+  https://github.com/fluent/fluentd/pull/1883
+* test: Fix random test failures in test/plugin/test_out_forward.rb
+  https://github.com/fluent/fluentd/pull/1881
+  https://github.com/fluent/fluentd/pull/1890
+* command: Fix typo in binlog_reader
+  https://github.com/fluent/fluentd/pull/1898
+
 ## Release v1.1.1 - 2018/03/05
 
 ### Enhancements
@@ -926,4 +949,4 @@ This list includes changes of 0.14.0.pre.1 and release candidates.
   https://github.com/fluent/fluentd/pull/955
   https://github.com/fluent/fluentd/pull/966
 
-See https://github.com/fluent/fluentd/blob/v0.12/ChangeLog for v0.12 changelog
+See https://github.com/fluent/fluentd/blob/v0.12/CHANGELOG.md for v0.12 changelog

data/lib/fluent/command/binlog_reader.rb

@@ -32,7 +32,7 @@ Usage: fluent-binlog-reader <command> [<args>]
 Commands of fluent-binlog-reader:
    cat     :     Read files sequentially, writing them to standard output.
    head    :     Display the beginning of a text file.
-   format  :     Display plugins that you can use.
+   formats :     Display plugins that you can use.
 
 See 'fluent-binlog-reader <command> --help' for more information on a specific command.
 HELP

data/lib/fluent/config/types.rb

@@ -60,7 +60,13 @@ module Fluent
       when ''
         true
       else
-        nil
+        # Current parser passes comment without actual values, e.g. "param #foo".
+        # parser should pass empty string in this case but changing behaviour may break existing environment so keep parser behaviour. Just ignore comment value in boolean handling for now.
+        if str.respond_to?('start_with?') && str.start_with?('#')
+          true
+        else
+          nil
+        end
       end
     end
 

data/lib/fluent/plugin/filter_grep.rb

@@ -41,7 +41,11 @@ module Fluent::Plugin
       config_param :key, :string
       desc "The regular expression."
       config_param :pattern do |value|
-        Regexp.compile(value)
+        if value.start_with?("/") and value.end_with?("/")
+          Regexp.compile(value[1..-2])
+        else
+          Regexp.compile(value)
+        end
       end
     end
 
@@ -50,7 +54,11 @@ module Fluent::Plugin
       config_param :key, :string
       desc "The regular expression."
       config_param :pattern do |value|
-        Regexp.compile(value)
+        if value.start_with?("/") and value.end_with?("/")
+          Regexp.compile(value[1..-2])
+        else
+          Regexp.compile(value)
+        end
       end
     end
 

data/lib/fluent/plugin/in_tail.rb

@@ -580,6 +580,8 @@ module Fluent::Plugin
               @pe.update(inode, 0)
             else # file is rotated and new file found
               watcher_needs_update = true
+              # Handle the old log file before renewing TailWatcher [fluentd#1055]
+              @io_handler.on_notify
             end
           else # file is rotated and new file not found
             # Clear RotateHandler to avoid duplicated file watch in same path.

data/lib/fluent/plugin/out_forward.rb

@@ -521,7 +521,6 @@ module Fluent::Plugin
         @standby = server.standby
         @failure = failure
         @available = true
-        @state = nil
 
         # @hostname is used for certificate verification & TLS SNI
         host_is_hostname = !(IPAddr.new(@host) rescue false)
@@ -537,7 +536,6 @@ module Fluent::Plugin
         @password = server.password
         @shared_key = server.shared_key || (sender.security && sender.security.shared_key) || ""
         @shared_key_salt = generate_salt
-        @shared_key_nonce = ""
 
         @unpacker = Fluent::Engine.msgpack_unpacker
 
@@ -552,6 +550,8 @@ module Fluent::Plugin
       attr_reader :sockaddr  # used by on_heartbeat
       attr_reader :failure, :available # for test
 
+      RequestInfo = Struct.new(:state, :shared_key_nonce, :auth)
+
       def validate_host_resolution!
         resolved_host
       end
@@ -568,8 +568,8 @@ module Fluent::Plugin
         @standby
       end
 
-      def establish_connection(sock)
-        while available? && @state != :established
+      def establish_connection(sock, ri)
+        while available? && ri.state != :established
           begin
             # TODO: On Ruby 2.2 or earlier, read_nonblock doesn't work expectedly.
             # We need rewrite around here using new socket/server plugin helper.
@@ -579,13 +579,13 @@ module Fluent::Plugin
               next
             end
             @unpacker.feed_each(buf) do |data|
-              on_read(sock, data)
+              on_read(sock, ri, data)
             end
           rescue IO::WaitReadable
             # If the exception is Errno::EWOULDBLOCK or Errno::EAGAIN, it is extended by IO::WaitReadable.
             # So IO::WaitReadable can be used to rescue the exceptions for retrying read_nonblock.
             # https//docs.ruby-lang.org/en/2.3.0/IO.html#method-i-read_nonblock
-            sleep @sender.read_interval unless @state == :established
+            sleep @sender.read_interval unless ri.state == :established
           rescue SystemCallError => e
             @log.warn "disconnected by error", host: @host, port: @port, error: e
             disable!
@@ -599,9 +599,9 @@ module Fluent::Plugin
       end
 
       def send_data_actual(sock, tag, chunk)
-        @state = @sender.security ? :helo : :established
-        if @state != :established
-          establish_connection(sock)
+        ri = RequestInfo.new(@sender.security ? :helo : :established)
+        if ri.state != :established
+          establish_connection(sock, ri)
         end
 
         unless available?
@@ -756,7 +756,7 @@ module Fluent::Plugin
         SecureRandom.hex(16)
       end
 
-      def check_helo(message)
+      def check_helo(ri, message)
         @log.debug "checking helo"
         # ['HELO', options(hash)]
         unless message.size == 2 && message[0] == 'HELO'
@@ -764,23 +764,23 @@ module Fluent::Plugin
         end
         opts = message[1] || {}
         # make shared_key_check failed (instead of error) if protocol version mismatch exist
-        @shared_key_nonce = opts['nonce'] || ''
-        @authentication = opts['auth'] || ''
+        ri.shared_key_nonce = opts['nonce'] || ''
+        ri.auth = opts['auth'] || ''
         true
       end
 
-      def generate_ping
+      def generate_ping(ri)
         @log.debug "generating ping"
         # ['PING', self_hostname, sharedkey\_salt, sha512\_hex(sharedkey\_salt + self_hostname + nonce + shared_key),
         #  username || '', sha512\_hex(auth\_salt + username + password) || '']
         shared_key_hexdigest = Digest::SHA512.new.update(@shared_key_salt)
           .update(@sender.security.self_hostname)
-          .update(@shared_key_nonce)
+          .update(ri.shared_key_nonce)
           .update(@shared_key)
           .hexdigest
         ping = ['PING', @sender.security.self_hostname, @shared_key_salt, shared_key_hexdigest]
-        if !@authentication.empty?
-          password_hexdigest = Digest::SHA512.new.update(@authentication).update(@username).update(@password).hexdigest
+        if !ri.auth.empty?
+          password_hexdigest = Digest::SHA512.new.update(ri.auth).update(@username).update(@password).hexdigest
           ping.push(@username, password_hexdigest)
         else
           ping.push('','')
@@ -788,7 +788,7 @@ module Fluent::Plugin
         ping
       end
 
-      def check_pong(message)
+      def check_pong(ri, message)
         @log.debug "checking pong"
         # ['PONG', bool(authentication result), 'reason if authentication failed',
         #  self_hostname, sha512\_hex(salt + self_hostname + nonce + sharedkey)]
@@ -805,7 +805,7 @@ module Fluent::Plugin
           return false, 'same hostname between input and output: invalid configuration'
         end
 
-        clientside = Digest::SHA512.new.update(@shared_key_salt).update(hostname).update(@shared_key_nonce).update(@shared_key).hexdigest
+        clientside = Digest::SHA512.new.update(@shared_key_salt).update(hostname).update(ri.shared_key_nonce).update(@shared_key).hexdigest
         unless shared_key_hexdigest == clientside
           return false, 'shared key mismatch'
         end
@@ -813,29 +813,29 @@ module Fluent::Plugin
         return true, nil
       end
 
-      def on_read(sock, data)
+      def on_read(sock, ri, data)
         @log.trace __callee__
 
-        case @state
+        case ri.state
         when :helo
-          unless check_helo(data)
+          unless check_helo(ri, data)
             @log.warn "received invalid helo message from #{@name}"
             disable! # shutdown
             return
           end
-          sock.write(generate_ping.to_msgpack)
-          @state = :pingpong
+          sock.write(generate_ping(ri).to_msgpack)
+          ri.state = :pingpong
         when :pingpong
-          succeeded, reason = check_pong(data)
+          succeeded, reason = check_pong(ri, data)
           unless succeeded
             @log.warn "connection refused to #{@name}: #{reason}"
             disable! # shutdown
             return
           end
-          @state = :established
+          ri.state = :established
           @log.debug "connection established", host: @host, port: @port
         else
-          raise "BUG: unknown session state: #{@state}"
+          raise "BUG: unknown session state: #{ri.state}"
         end
       end
     end

data/lib/fluent/plugin/output.rb

@@ -1313,9 +1313,17 @@ module Fluent
           # This thread don't use `thread_current_running?` because this thread should run in `before_shutdown` phase
           while @output_flush_threads_running
             current_clock = Fluent::Clock.now
-            interval = state.next_clock - current_clock
+            next_retry_time = nil
 
-            if state.next_clock <= current_clock && @retry_mutex.synchronize { @retry ? @retry.next_time <= Time.now : true }
+            @retry_mutex.synchronize do
+              next_retry_time = @retry ? @retry.next_time : nil
+            end
+
+            if state.next_clock > current_clock
+              interval = state.next_clock - current_clock
+            elsif next_retry_time && next_retry_time > Time.now
+              interval = next_retry_time.to_f - Time.now.to_f
+            else
               try_flush
 
               # next_flush_time uses flush_thread_interval or flush_thread_burst_interval (or retrying)

data/lib/fluent/version.rb

@@ -16,6 +16,6 @@
 
 module Fluent
 
-  VERSION = '1.1.1'
+  VERSION = '1.1.2'
 
 end

data/test/config/test_plugin_configuration.rb

@@ -0,0 +1,56 @@
+require_relative '../helper'
+require 'fluent/plugin/input'
+require 'fluent/test/driver/input'
+
+module ConfigurationForPlugins
+  class AllBooleanParams < Fluent::Plugin::Input
+    config_param :flag1, :bool, default: true
+    config_param :flag2, :bool, default: true
+    config_param :flag3, :bool, default: false
+    config_param :flag4, :bool, default: false
+
+    config_section :child, param_name: :children, multi: true, required: true do
+      config_param :flag1, :bool, default: true
+      config_param :flag2, :bool, default: true
+      config_param :flag3, :bool, default: false
+      config_param :flag4, :bool, default: false
+    end
+  end
+
+  class BooleanParamsWithoutValue < ::Test::Unit::TestCase
+    CONFIG = <<CONFIG
+    flag1 
+    flag2 # yaaaaaaaaaay
+    flag3 
+    flag4 # yaaaaaaaaaay
+    <child>
+      flag1
+      flag2 # yaaaaaaaaaay
+      flag3
+      flag4 # yaaaaaaaaaay
+    </child>
+    <child>
+      flag1 # yaaaaaaaaaay
+      flag2
+      flag3 # yaaaaaaaaaay
+      flag4
+    </child>
+    # with following whitespace
+    <child>
+      flag1 
+      flag2 
+      flag3 
+      flag4 
+    </child>
+CONFIG
+
+    test 'create plugin via driver' do
+      d = Fluent::Test::Driver::Input.new(AllBooleanParams)
+      d.configure(CONFIG)
+      assert_equal([true] * 4, [d.instance.flag1, d.instance.flag2, d.instance.flag3, d.instance.flag4])
+      num_of_sections = 3
+      assert_equal num_of_sections, d.instance.children.size
+      assert_equal([true] * (num_of_sections * 4), d.instance.children.map{|c| [c.flag1, c.flag2, c.flag3, c.flag4]}.flatten)
+    end
+  end
+end

data/test/plugin/test_filter_grep.rb

@@ -73,6 +73,24 @@ class GrepFilterTest < Test::Unit::TestCase
         end
       end
     end
+
+    sub_test_case "pattern with slashes" do
+      test "start with character classes" do
+        conf = %[
+          <regexp>
+            key message
+            pattern /[a-z]test/
+          </regexp>
+          <exclude>
+            key message
+            pattern /[A-Z]test/
+          </exclude>
+        ]
+        d = create_driver(conf)
+        assert_equal(/[a-z]test/, d.instance.regexps.first.pattern)
+        assert_equal(/[A-Z]test/, d.instance.excludes.first.pattern)
+      end
+    end
   end
 
   sub_test_case 'filter_stream' do

data/test/plugin/test_out_forward.rb

@@ -486,6 +486,7 @@ EOL
         retry_type periodic
         retry_wait 30s
         flush_at_shutdown false # suppress errors in d.instance_shutdown
+        flush_thread_interval 30s
       </buffer>
     ])
 
@@ -501,7 +502,7 @@ EOL
     target_input_driver.end_if{ d.instance.rollback_count > 0 }
     target_input_driver.end_if{ !node.available }
     target_input_driver.run(expect_records: 2, timeout: 25) do
-      d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false) do
+      d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false, flush: false) do
         delayed_commit_timeout_value = d.instance.delayed_commit_timeout
         d.feed([[time, records[0]], [time,records[1]]])
       end
@@ -530,6 +531,7 @@ EOL
         retry_type periodic
         retry_wait 30s
         flush_at_shutdown false # suppress errors in d.instance_shutdown
+        flush_thread_interval 30s
       </buffer>
     ])
 
@@ -545,7 +547,7 @@ EOL
     target_input_driver.end_if{ d.instance.rollback_count > 0 }
     target_input_driver.end_if{ !node.available }
     target_input_driver.run(expect_records: 2, timeout: 25) do
-      d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false) do
+      d.run(default_tag: 'test', timeout: 20, wait_flush_completion: false, shutdown: false, flush: false) do
         delayed_commit_timeout_value = d.instance.delayed_commit_timeout
         d.feed([[time, records[0]], [time,records[1]]])
       end
@@ -664,6 +666,51 @@ EOL
     assert_equal(['test', time, records[1]], events[1])
   end
 
+  # This test is not 100% but test failed with previous Node implementation which has race condition
+  test 'Node with security is thread-safe on multi threads' do
+    input_conf = TARGET_CONFIG + %[
+                   <security>
+                     self_hostname in.localhost
+                     shared_key fluentd-sharedkey
+                     <client>
+                       host 127.0.0.1
+                     </client>
+                   </security>
+                 ]
+    target_input_driver = create_target_input_driver(conf: input_conf)
+    output_conf = %[
+      send_timeout 51
+      <security>
+        self_hostname localhost
+        shared_key fluentd-sharedkey
+      </security>
+      <server>
+        name test
+        host #{TARGET_HOST}
+        port #{TARGET_PORT}
+        shared_key fluentd-sharedkey
+      </server>
+    ]
+    @d = d = create_driver(output_conf)
+
+    chunk = Fluent::Plugin::Buffer::MemoryChunk.new(Fluent::Plugin::Buffer::Metadata.new(nil, nil, nil))
+    target_input_driver.run(timeout: 15) do
+      d.run(shutdown: false) do
+        node = d.instance.nodes.first
+        arr = []
+        4.times {
+          arr << Thread.new {
+            node.send_data('test', chunk) rescue nil
+          }
+        }
+        arr.each { |a| a.join }
+      end
+    end
+
+    logs = d.logs
+    assert_false(logs.any? { |log| log.include?("invalid format for PONG message") || log.include?("shared key mismatch") }, "'#{logs.last.strip}' happens")
+  end
+
   def create_target_input_driver(response_stub: nil, disconnect: false, conf: TARGET_CONFIG)
     require 'fluent/plugin/in_forward'
 

data/test/plugin/test_output.rb

@@ -907,7 +907,8 @@ class OutputTest < Test::Unit::TestCase
 
   sub_test_case 'slow_flush_log_threshold' do
     def invoke_slow_flush_log_threshold_test(i)
-      i.configure(config_element('ROOT', '', {'slow_flush_log_threshold' => 0.5}, [config_element('buffer', '', {"flush_mode" => "immediate"})]))
+      i.configure(config_element('ROOT', '', {'slow_flush_log_threshold' => 0.5},
+                                 [config_element('buffer', '', {"flush_mode" => "immediate", "flush_thread_interval" => 30})]))
       i.start
       i.after_start
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluentd
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.1.2
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-03-05 00:00:00.000000000 Z
+date: 2018-03-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -330,7 +330,6 @@ files:
 - CHANGELOG.md
 - CONTRIBUTING.md
 - COPYING
-- ENTERPRISE_PROVIDERS.md
 - GOVERNANCE.md
 - Gemfile
 - LICENSE
@@ -603,6 +602,7 @@ files:
 - test/config/test_dsl.rb
 - test/config/test_element.rb
 - test/config/test_literal_parser.rb
+- test/config/test_plugin_configuration.rb
 - test/config/test_section.rb
 - test/config/test_system_config.rb
 - test/config/test_types.rb
@@ -772,6 +772,7 @@ test_files:
 - test/config/test_dsl.rb
 - test/config/test_element.rb
 - test/config/test_literal_parser.rb
+- test/config/test_plugin_configuration.rb
 - test/config/test_section.rb
 - test/config/test_system_config.rb
 - test/config/test_types.rb

data/ENTERPRISE_PROVIDERS.md

@@ -1,43 +0,0 @@
-# Fluentd Enterprise Providers
-
-[Fluentd](https://www.fluentd.org) is widely adopted in the enterprise and in most of scenarios, production environments requires special support and services. The following document describe the certified  service providers and further guidelines to apply to become one.
-
-## Fluentd Providers
-
-The following section lists the companies that provides Support, Consulting Services and Products around Fluentd for the Enterprise needs.
-
-- [Treasure Data](https://fluentd.treasuredata.com)
-
-### Treasure Data
-
-[Treasure Data](https://www.treasuredata.com) is one of the principal sponsors of Fluentd, it provides technical support, consulting services and [Fluentd Enterprise](https://fluentd.treasuredata.com/), a Fluentd-on steroids with enhanced security and connectors for enterprise backends such as [Apache Kafka](https://kafka.apache.org) and [Splunk](https://www.splunk.com) within others.
-
-For more details about Fluentd Enterprise provided by Treasure Data, click [here](https://fluentd.treasuredata.com/).
-
-## Apply to become a Fluentd Service Provider
-
-In order to keep a transparent involvement of companies in Fluentd growth, to be listed as a service provider your company must fulfil the following criteria:
-
-- The company offering the services or products must be fully incorporated.
-- At least two former employees must be contributors of Fluentd ecosystem (core, plugins, documentation, etc).
-- Company participate from Fluentd community activities: meetups, conferences (or webinars) or online communication channels (Slack)
-
-In order to be listed as a service provider, please send an email using your corporate account to the maintainers (listed at bottom) with the following information:
-
-- Company Information
-  - Company Name
-  - Company URL
-  - Country / Location
-- Employees
-  - Name, email and github handle of employees who contribute to Fluentd ecosystem
-  - List of projects or contributions where the company is involved
-- Services
-  - What kind of services around Fluentd do you provide ?
-- Community
-  - List of activities of your company within the Fluentd community in the last 12 months
-  - Are you a member of the [Cloud Native Computing Foundation](https://cncf.io) ?
-
-### Maintainers / Recipients
-
-- Kiyoto Tamura <kiyoto@treasure-data.com>
-- Eduardo Silva <eduardo@treasure-data.com>