fluentd 0.14.7-x86-mingw32 → 0.14.10-x86-mingw32

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: beb3dc3b05f93e01bcfbd3bd5bba851613e7a56a
-  data.tar.gz: 6e618a140882db9be00c10fb76b47e5eec9feced
+  metadata.gz: cc9fd1115013c31c251063f81dc0d91be3b5624c
+  data.tar.gz: 95d7e3af0d15244336c42b8d1bfe83f08afd8f36
 SHA512:
-  metadata.gz: 91175a11ce3c0307b21ec2f12932038c5e626b10287de29a75a65c702c62c915f749132d87e959761bedb2b2dd1bb283c160c4e375a7091a61f690fd6ddb51d7
-  data.tar.gz: 57d912a874804209343d0dbe3c0fbc856cf3cdd6f085d19d7cfdf3c0112a9e739001027ce2ee69f21cb511d085f2e8098709b912cc0eccc063430f48e47738e1
+  metadata.gz: 92f075a8834ebbcd7b5c1c43433afc8f9d62a53f68e26ef8e82519515dc7abf8b4a9ad03a43307c3a232a84f281a1f3968e2dda160659c54a01da164fe348224
+  data.tar.gz: 5d0f4bc6df7f9bfe63d7c5f9c4b8e8117111bcaa93f60a32a884e90e69523200a458732e452c964edf590af43da71e51789d01a2dff7a9e28c653413909f834a

data/.gitignore

@@ -24,3 +24,5 @@ Gemfile.local
 *.swp
 coverage/*
 .vagrant/
+cov-int/
+cov-fluentd.tar.gz

data/.travis.yml

@@ -1,6 +1,8 @@
 language: ruby
 cache: bundler
 
+# script: bundle exec rake test TESTOPTS=-v
+
 # http://rubies.travis-ci.org/
 # See here for osx_image -> OSX versions: https://docs.travis-ci.com/user/languages/objective-c
 matrix:

data/CONTRIBUTING.md

@@ -7,8 +7,13 @@ We'd love your contribution. Here are the guidelines!
 RESOURCES of [Official site](http://www.fluentd.org/) and [Fluentd documentation](http://docs.fluentd.org/) may help you.
 
 If you have further questions about Fluentd and plugins, please direct these to [Mailing List](https://groups.google.com/forum/#!forum/fluentd).
+Don't use Github issue for asking questions. Here are examples:
 
-Don't use Github issue for asking questions.
+- I installed xxx plugin but it doesn't work. Why?
+- Fluentd starts but logs are not sent to xxx. Am I wrong?
+- I want to do xxx. How to realize it with plugins?
+
+We may close such questions to keep clear repository for developers and users.
 Github issue is mainly for submitting a bug report or feature request. See below.
 
 ## Found a bug?

data/ChangeLog

@@ -1,15 +1,108 @@
 # v0.14
 
+## Release v0.14.10 - 2016/12/14
+
+### New features / Enhancement
+
+* Add socket/server plugin helper to write TCP/UDP clients/servers as Fluentd plugin
+  https://github.com/fluent/fluentd/pull/1312
+  https://github.com/fluent/fluentd/pull/1350
+  https://github.com/fluent/fluentd/pull/1356
+  https://github.com/fluent/fluentd/pull/1362
+* Fix to raise errors when injected hostname is also specified as chunk key
+  https://github.com/fluent/fluentd/pull/1357
+* in_tail: Optimize to read lines from file
+  https://github.com/fluent/fluentd/pull/1325
+* in_monitor_agent: Add new parameter "include_config"(default: true)
+  https://github.com/fluent/fluentd/pull/1317
+* in_syslog: Add "priority_key" and "facility_key" options
+  https://github.com/fluent/fluentd/pull/1351
+* filter_record_transformer: Remove obsoleted syntax like "${message}" and not to dump records in logs
+  https://github.com/fluent/fluentd/pull/1328
+* Add an option "--time-as-integer" to fluent-cat command to send events from v0.14 fluent-cat to v0.12 fluentd
+  https://github.com/fluent/fluentd/pull/1349
+
+### Bug fixes
+
+* Specify correct Oj options for newer versions (Oj 2.18.0 or later)
+  https://github.com/fluent/fluentd/pull/1331
+* TimeSlice output plugins (in v0.12 style) raise errors when "utc" parameter is specified
+  https://github.com/fluent/fluentd/pull/1319
+* Parser plugins cannot use options for regular expressions
+  https://github.com/fluent/fluentd/pull/1326/files
+* Fix bugs not to raise errors to use logger in v0.12 plugins
+  https://github.com/fluent/fluentd/pull/1344
+  https://github.com/fluent/fluentd/pull/1332
+* Fix bug about shutting down Fluentd in Windows
+  https://github.com/fluent/fluentd/pull/1367
+* in_tail: Close files explicitly in tests
+  https://github.com/fluent/fluentd/pull/1327
+* in_forward: Fix bug not to convert buffer configurations into v0.14 parameters
+  https://github.com/fluent/fluentd/pull/1337
+* out_forward: Fix bug to raise error when "expire_dns_cache" is specified
+  https://github.com/fluent/fluentd/pull/1346
+* out_file: Fix bug to raise error about buffer chunking when it's configured as secondary
+  https://github.com/fluent/fluentd/pull/1338
+
+## Release v0.14.9 - 2016/11/15
+
+### New features / Enhancement
+
+* filter_parser: Port fluent-plugin-parser into built-in plugin
+  https://github.com/fluent/fluentd/pull/1191
+* parser/formatter plugin helpers with default @type in plugin side
+  https://github.com/fluent/fluentd/pull/1267
+* parser: Reconstruct Parser related classes
+  https://github.com/fluent/fluentd/pull/1286
+* filter_record_transformer: Remove old behaviours
+  https://github.com/fluent/fluentd/pull/1311
+* Migrate some built-in plugins into v0.14 API
+  https://github.com/fluent/fluentd/pull/1257 (out_file)
+  https://github.com/fluent/fluentd/pull/1297 (out_exec, out_exec_filter)
+  https://github.com/fluent/fluentd/pull/1306 (in_forward, out_forward)
+  https://github.com/fluent/fluentd/pull/1308 (in_http)
+* test: Improve test drivers
+  https://github.com/fluent/fluentd/pull/1302
+  https://github.com/fluent/fluentd/pull/1305
+
+### Bug fixes
+
+* log: Avoid name conflict between Fluent::Logger
+  https://github.com/fluent/fluentd/pull/1274
+* fluent-cat: Fix fluent-cat command to send sub-second precision time
+  https://github.com/fluent/fluentd/pull/1277
+* config: Fix a bug not to overwrite default value with nil
+  https://github.com/fluent/fluentd/pull/1296
+* output: Fix timezone for compat timesliced output plugins
+  https://github.com/fluent/fluentd/pull/1307
+* out_forward: fix not to raise error when out_forward is initialized as secondary
+  https://github.com/fluent/fluentd/pull/1313
+* output: Event router for secondary output
+  https://github.com/fluent/fluentd/pull/1283
+* test: fix to return the block value as expected by many rubyists
+  https://github.com/fluent/fluentd/pull/1284
+
+## Release v0.14.8 - 2016/10/13
+
+### Bug fixes
+
+* Add msgpack_each to buffer chunks in compat-layer output plugins
+  https://github.com/fluent/fluentd/pull/1273
+
 ## Release v0.14.7 - 2016/10/07
 
 ### New features / Enhancement
 
+* Support data compression in buffer plugins
+  https://github.com/fluent/fluentd/pull/1172
 * in_forward: support to transfer compressed data
   https://github.com/fluent/fluentd/pull/1179
 * out_stdout: fix to show nanosecond resolution time
   https://github.com/fluent/fluentd/pull/1249
 * Add option to rotate Fluentd daemon's log
   https://github.com/fluent/fluentd/pull/1235
+* Add extract plugin helper, with symmetric time parameter support in parser/formatter and inject/extract
+  https://github.com/fluent/fluentd/pull/1207
 * Add a feature to parse/format numeric time (unix time [+ subsecond value])
   https://github.com/fluent/fluentd/pull/1254
 * Raise configuration errors for inconsistent <label> configurations
@@ -22,6 +115,8 @@
   https://github.com/fluent/fluentd/pull/1227
 * Add some features to write plugins more easily
   https://github.com/fluent/fluentd/pull/1256
+* Add a tool to load dumped events from file
+  https://github.com/fluent/fluentd/pull/1165
 
 ### Bug fixes
 

data/Rakefile

@@ -24,6 +24,9 @@ end
 
 desc 'Run test_unit based test'
 Rake::TestTask.new(:base_test) do |t|
+  # To run test with dumping all test case names (to find never ending test case)
+  #  $ bundle exec rake test TESTOPTS=-v
+  #
   # To run test for only one file (or file path pattern)
   #  $ bundle exec rake base_test TEST=test/test_specified_path.rb
   #  $ bundle exec rake base_test TEST=test/test_*.rb
@@ -50,4 +53,22 @@ task :coverage do |t|
   Rake::Task["test"].invoke
 end
 
+desc 'Build Coverity tarball & upload it'
+task :coverity do
+  # https://scan.coverity.com/projects/fluentd?tab=overview
+  # See "View Defects" after sign-in.
+  #
+  # Setup steps:
+  # 1. get coverity build tool and set PATH to bin/: https://scan.coverity.com/download
+  # 2. set environment variables:
+  #    * $COVERITY_USER (your email address)
+  #    * $COVERITY_TOKEN (token for Fluentd project: https://scan.coverity.com/projects/fluentd?tab=project_settings)
+  sh "cov-build --dir cov-int --no-command --fs-capture-search ./"
+  sh "tar czf cov-fluentd.tar.gz cov-int"
+  user = ENV['COVERITY_USER']
+  token = ENV['COVERITY_TOKEN']
+  sh "curl --form token=#{token} --form email=#{user} --form file=@cov-fluentd.tar.gz --form version=\"Master\" --form description=\"GIT Master\" https://scan.coverity.com/builds?project=Fluentd"
+  FileUtils.rm_rf(['./cov-int', 'cov-fluentd.tar.gz'])
+end
+
 task default: [:test, :build]

data/appveyor.yml

@@ -12,6 +12,7 @@ install:
 build: off
 test_script:
   - bundle exec rake test
+#  - bundle exec rake test TESTOPTS=-v
 
 branches:
   only:

data/code-of-conduct.md

@@ -0,0 +1,3 @@
+## Fluentd Community Code of Conduct
+
+Fluentd follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

data/example/out_exec_filter.conf

@@ -0,0 +1,42 @@
+<source>
+  @type dummy
+  @label @exec
+  tag exec_input
+  rate 10
+  auto_increment_key num
+  dummy {"data":"mydata"}
+</source>
+
+<label @exec>
+  <match exec_input>
+    @type exec_filter
+    @label @stdout
+    tag result
+    command ruby -e 'STDOUT.sync = true; proc = ->(){line = STDIN.readline.chomp; puts line + "\t" + Process.pid.to_s}; 1000.times{ proc.call }'
+    num_children 3
+    child_respawn -1
+    <inject>
+      time_key time
+      time_type float
+    </inject>
+    <format>
+      @type tsv
+      keys data, num, time
+    </format>
+    <parse>
+      @type tsv
+      keys data, num, time, pid
+    </parse>
+    <extract>
+      time_key time
+      time_type float
+    </extract>
+  </match>
+</label>
+
+<label @stdout>
+  <match result>
+    @type stdout
+  </match>
+</label>
+

data/fluentd.gemspec

@@ -22,7 +22,7 @@ Gem::Specification.new do |gem|
   gem.add_runtime_dependency("msgpack", [">= 0.7.0", "< 2.0.0"])
   gem.add_runtime_dependency("yajl-ruby", ["~> 1.0"])
   gem.add_runtime_dependency("cool.io", ["~> 1.4.5"])
-  gem.add_runtime_dependency("serverengine", ["~> 2.0"])
+  gem.add_runtime_dependency("serverengine", [">= 2.0.4", "< 3.0.0"])
   gem.add_runtime_dependency("http_parser.rb", [">= 0.5.1", "< 0.7.0"])
   gem.add_runtime_dependency("sigdump", ["~> 0.2.2"])
   gem.add_runtime_dependency("tzinfo", ["~> 1.0"])

data/lib/fluent/agent.rb

@@ -123,7 +123,7 @@ module Fluent
       log.info "adding match#{@context.nil? ? '' : " in #{@context}"}", pattern: pattern, type: type
 
       output = Plugin.new_output(type)
-      output.router = @event_router if output.respond_to?(:router=)
+      output.context_router = @event_router
       output.configure(conf)
       @outputs << output
       if output.respond_to?(:outputs) && output.respond_to?(:multi_output?) && output.multi_output?
@@ -144,7 +144,7 @@ module Fluent
       log.info "adding filter#{@context.nil? ? '' : " in #{@context}"}", pattern: pattern, type: type
 
       filter = Plugin.new_filter(type)
-      filter.router = @event_router
+      filter.context_router = @event_router
       filter.configure(conf)
       @filters << filter
       @event_router.add_rule(pattern, filter)

data/lib/fluent/command/binlog_reader.rb

@@ -164,7 +164,7 @@ module BinlogReaderCommand
     def call
       @formatter = lookup_formatter(@options[:format], @options[:config_params])
 
-      File.open(@path, 'r') do |io|
+      File.open(@path, 'rb') do |io|
         i = 1
         Fluent::MessagePackFactory.unpacker(io).each do |(time, record)|
           print @formatter.format(@path, time, record) # path is used for tag

data/lib/fluent/command/cat.rb

@@ -16,6 +16,8 @@
 
 require 'optparse'
 require 'fluent/env'
+require 'fluent/time'
+require 'fluent/msgpack_factory'
 
 op = OptionParser.new
 
@@ -29,6 +31,7 @@ socket_path = Fluent::DEFAULT_SOCKET_PATH
 config_path = Fluent::DEFAULT_CONFIG_PATH
 format = 'json'
 message_key = 'message'
+time_as_integer = false
 
 op.on('-p', '--port PORT', "fluent tcp port (default: #{port})", Integer) {|i|
   port = i
@@ -66,6 +69,10 @@ op.on('--message-key KEY', "key field for none format (default: #{message_key})"
   message_key = s
 }
 
+op.on('--time-as-integer', "Send time as integer for v0.12 or earlier", TrueClass) { |b|
+  time_as_integer = true
+}
+
 (class << self; self; end).module_eval do
   define_method(:usage) do |msg|
     puts op.to_s
@@ -121,7 +128,7 @@ class Writer
     end
   end
 
-  def initialize(tag, connector)
+  def initialize(tag, connector, time_as_integer: false)
     @tag = tag
     @connector = connector
     @socket = false
@@ -134,6 +141,7 @@ class Writer
     @pending_limit = 1024  # TODO
     @retry_wait = 1
     @retry_limit = 5  # TODO
+    @time_as_integer = time_as_integer
 
     super()
   end
@@ -143,7 +151,9 @@ class Writer
       raise ArgumentError, "Input must be a map (got #{record.class})"
     end
 
-    entry = [Time.now.to_i, record]
+    time = Fluent::EventTime.now
+    time = time.to_i if @time_as_integer
+    entry = [time, record]
     synchronize {
       unless write_impl([entry])
         # write failed
@@ -200,7 +210,8 @@ class Writer
     end
 
     begin
-      socket.write [@tag, array].to_msgpack
+      packer = Fluent::MessagePackFactory.packer
+      socket.write packer.pack([@tag, array])
       socket.flush
     rescue
       $stderr.puts "write failed: #{$!}"
@@ -272,7 +283,7 @@ else
   }
 end
 
-w = Writer.new(tag, connector)
+w = Writer.new(tag, connector, time_as_integer: time_as_integer)
 w.start
 
 case format

data/lib/fluent/compat/output.rb

@@ -26,6 +26,7 @@ require 'fluent/compat/record_filter_mixin'
 require 'fluent/compat/output_chain'
 require 'fluent/timezone'
 require 'fluent/mixin'
+require 'fluent/event'
 require 'fluent/process' # to load Fluent::DetachProcessMixin
 
 require 'fluent/plugin_helper/compat_parameters'
@@ -80,7 +81,7 @@ module Fluent
       end
 
       def assume_timekey!
-        @_formatter = Fluent::Timezone.formatter(@_timezone, @_time_slice_format)
+        @_formatter = Fluent::TimeFormatter.new(@_time_slice_format, nil, @_timezone)
 
         return if self.metadata.timekey
         if self.respond_to?(:path) && self.path =~ /\.(\d+)\.(?:b|q)(?:[a-z0-9]+)/
@@ -123,6 +124,7 @@ module Fluent
       # prepend this module to BufferedOutput (including ObjectBufferedOutput) plugin singleton class
       def write(chunk)
         chunk.extend(ChunkSizeCompatMixin)
+        chunk.extend(ChunkMessagePackEventStreamer)
         chunk.extend(AddKeyToChunkMixin) if chunk.metadata.variables && chunk.metadata.variables.has_key?(:key)
         super
       end
@@ -132,6 +134,7 @@ module Fluent
       # prepend this module to TimeSlicedOutput plugin singleton class
       def write(chunk)
         chunk.extend(ChunkSizeCompatMixin)
+        chunk.extend(ChunkMessagePackEventStreamer)
         chunk.extend(AddTimeSliceKeyToChunkMixin)
         chunk.time_slice_format = @time_slice_format
         chunk.timekey = @_timekey
@@ -614,17 +617,19 @@ module Fluent
           end
 
           if conf['timezone']
-            @timezone = conf['timezone']
-            Fluent::Timezone.validate!(@timezone)
+            Fluent::Timezone.validate!(conf['timezone'])
           elsif conf['utc']
-            @timezone = "+0000"
-            @localtime = false
+            # v0.12 assumes UTC without any configuration
+            # 'localtime=false && no timezone key' means UTC
+            conf['localtime'] = "false"
+            conf.delete('utc')
           elsif conf['localtime']
-            @timezone = Time.now.strftime('%z')
-            @localtime = true
+            conf['timezone'] = Time.now.strftime('%z')
+            conf['localtime'] = "true"
           else
-            @timezone = "+0000" # v0.12 assumes UTC without any configuration
-            @localtime = false
+            # v0.12 assumes UTC without any configuration
+            # 'localtime=false && no timezone key' means UTC
+            conf['localtime'] = "false"
           end
 
           @_timekey = case conf['time_slice_format']

data/lib/fluent/compat/parser.rb

@@ -18,6 +18,9 @@ require 'fluent/plugin'
 require 'fluent/plugin/parser'
 require 'fluent/mixin'
 
+require 'fluent/config'
+require 'fluent/compat/type_converter'
+
 require 'fluent/plugin/parser_regexp'
 require 'fluent/plugin/parser_json'
 require 'fluent/plugin/parser_tsv'
@@ -56,13 +59,14 @@ module Fluent
         format = conf['format']
 
         @parser = TextParser.lookup(format)
-        if ! @estimate_current_event.nil? && @parser.respond_to?(:'estimate_current_event=')
-          @parser.estimate_current_event = @estimate_current_event
-        end
 
         if @parser.respond_to?(:configure)
           @parser.configure(conf)
         end
+        if !@estimate_current_event.nil? && @parser.respond_to?(:'estimate_current_event=')
+          # external code sets parser.estimate_current_event = false
+          @parser.estimate_current_event = @estimate_current_event
+        end
 
         return true
       end
@@ -116,23 +120,59 @@ module Fluent
         end
       end
 
-      class TimeParser < Fluent::Plugin::Parser::TimeParser
+      module TypeConverterCompatParameters
+        def convert_type_converter_parameters!(conf)
+          if conf["types"]
+            delimiter = conf["types_delimiter"] || ','
+            label_delimiter = conf["types_label_delimiter"] || ':'
+            types = {}
+            conf['types'].split(delimiter).each do |pair|
+              key, value = pair.split(label_delimiter, 2)
+              if value.start_with?("time#{label_delimiter}")
+                value = value.split(label_delimiter, 2).join(':')
+              elsif value.start_with?("array#{label_delimiter}")
+                value = value.split(label_delimiter, 2).join(':')
+              end
+              types[key] = value
+            end
+            conf["types"] = JSON.dump(types)
+          end
+        end
+      end
+
+      class TimeParser < Fluent::TimeParser
         # TODO: warn when deprecated
       end
 
       class RegexpParser < Fluent::Plugin::RegexpParser
+        include TypeConverterCompatParameters
+
         # TODO: warn when deprecated
         def initialize(regexp, conf = {})
           super()
 
+          @stored_regexp = regexp
+          @manually_configured = false
           unless conf.empty?
-            unless conf.is_a?(Config::Element)
-              conf = Config::Element.new('default_regexp_conf', '', conf, [])
-            end
-            configure(conf)
+            conf_init = if conf.is_a?(Fluent::Config::Element)
+                          conf
+                        else
+                          Fluent::Config::Element.new('parse', '', conf, [])
+                        end
+            self.configure(conf_init)
+            @manually_configured = true
           end
+        end
+
+        def configure(conf)
+          return if @manually_configured # not to run twice
+
+          conf['expression'] ||= @stored_regexp.source
+          conf['ignorecase'] ||= @stored_regexp.options & Regexp::IGNORECASE != 0
+          conf['multiline'] ||= @stored_regexp.options & Regexp::MULTILINE != 0
+          convert_type_converter_parameters!(conf)
 
-          @regexp = regexp
+          super
         end
 
         def patterns
@@ -140,24 +180,114 @@ module Fluent
         end
       end
 
-      class ValuesParser < Fluent::Plugin::ValuesParser
-        # TODO: warn when deprecated
+      class ValuesParser < Parser
+        include Fluent::Compat::TypeConverter
+
+        config_param :keys, :array, default: []
+        config_param :time_key, :string, default: nil
+        config_param :null_value_pattern, :string, default: nil
+        config_param :null_empty_string, :bool, default: false
+
+        def configure(conf)
+          super
+
+          if @time_key && !@keys.include?(@time_key) && @estimate_current_event
+            raise Fluent::ConfigError, "time_key (#{@time_key.inspect}) is not included in keys (#{@keys.inspect})"
+          end
+
+          if @time_format && !@time_key
+            raise Fluent::ConfigError, "time_format parameter is ignored because time_key parameter is not set. at #{conf.inspect}"
+          end
+
+          @time_parser = time_parser_create
+
+          if @null_value_pattern
+            @null_value_pattern = Regexp.new(@null_value_pattern)
+          end
+
+          @mutex = Mutex.new
+        end
+
+        def values_map(values)
+          record = Hash[keys.zip(values.map { |value| convert_value_to_nil(value) })]
+
+          if @time_key
+            value = @keep_time_key ? record[@time_key] : record.delete(@time_key)
+            time = if value.nil?
+                     if @estimate_current_event
+                       Fluent::EventTime.now
+                     else
+                       nil
+                     end
+                   else
+                     @mutex.synchronize { @time_parser.parse(value) }
+                   end
+          elsif @estimate_current_event
+            time = Fluent::EventTime.now
+          else
+            time = nil
+          end
+
+          convert_field_type!(record) if @type_converters
+
+          return time, record
+        end
+
+        private
+
+        def convert_field_type!(record)
+          @type_converters.each_key { |key|
+            if value = record[key]
+              record[key] = convert_type(key, value)
+            end
+          }
+        end
+
+        def convert_value_to_nil(value)
+          if value and @null_empty_string
+            value = (value == '') ? nil : value
+          end
+          if value and @null_value_pattern
+            value = ::Fluent::StringUtil.match_regexp(@null_value_pattern, value) ? nil : value
+          end
+          value
+        end
       end
 
       class JSONParser < Fluent::Plugin::JSONParser
+        include TypeConverterCompatParameters
         # TODO: warn when deprecated
+        def configure(conf)
+          convert_type_converter_parameters!(conf)
+          super
+        end
       end
 
       class TSVParser < Fluent::Plugin::TSVParser
+        include TypeConverterCompatParameters
         # TODO: warn when deprecated
+        def configure(conf)
+          convert_type_converter_parameters!(conf)
+          super
+        end
       end
 
       class LabeledTSVParser < Fluent::Plugin::LabeledTSVParser
+        include TypeConverterCompatParameters
         # TODO: warn when deprecated
+        def configure(conf)
+          convert_type_converter_parameters!(conf)
+          super
+        end
       end
 
       class CSVParser < Fluent::Plugin::CSVParser
+        include TypeConverterCompatParameters
         # TODO: warn when deprecated
+        def configure(conf)
+          convert_type_converter_parameters!(conf)
+          super
+        end
       end
 
       class NoneParser < Fluent::Plugin::NoneParser