fluentd 0.14.21 → 0.14.22.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e0c04447f174a283ac32276dd0dbde42bfabac9b
-  data.tar.gz: dc5da1036653db4373345239dc4486918423e479
+  metadata.gz: cabaaf075c80b44f6b8f77efb057ad809212f871
+  data.tar.gz: 7092d208bb92fe4992764dbe418b117976a7ba66
 SHA512:
-  metadata.gz: 7015711b0401b911a3242e96bf90f5bd56693e1ccc62565d2cee241403354599a3c29bf487f10989297bfcf41a7d0c90c274396f0a07dc9c338fc26a66769323
-  data.tar.gz: 9cd584791a19c95166e34e325f104cb38918769c836f1779008e77e7d36367ccb786d3fa9b16381d8d9ce680ec22b903b60e9a8e0244f635954d188c6a5e2a6c
+  metadata.gz: 235273ae1d58e4e3c3be98642805cb9d62ac3df2c75cc94622ee78bf62b65e5927aec1ad9ba2043f4c06b65abed6a490340243fb1b82d42e6e0e8e9bd8d4a500
+  data.tar.gz: dcab5670899cb22e7914cffcb2a31f590d054f31ec44b500720cbbc02e59c4f22a9da02b2078035dc5dbf7eea625c34f60f8002eb9c83271c60f0b02883a0a9c

data/ADOPTERS.md

@@ -0,0 +1,5 @@
+# Fluentd Adopters
+
+Fluentd is widely used by hundred of companies, please refer to the testimonial section of our project website  to learn more about it:
+
+https://www.fluentd.org/testimonials

data/CONTRIBUTING.md

@@ -4,7 +4,7 @@ We'd love your contribution. Here are the guidelines!
 
 ## Got a question or problem?
 
-RESOURCES of [Official site](http://www.fluentd.org/) and [Fluentd documentation](http://docs.fluentd.org/) may help you.
+RESOURCES of [Official site](https://www.fluentd.org/) and [Fluentd documentation](https://docs.fluentd.org/) may help you.
 
 If you have further questions about Fluentd and plugins, please direct these to [Mailing List](https://groups.google.com/forum/#!forum/fluentd).
 Don't use Github issue for asking questions. Here are examples:

data/ENTERPRISE_PROVIDERS.md

@@ -1,6 +1,6 @@
 # Fluentd Enterprise Providers
 
-[Fluentd](http://www.fluentd.org) is widely adopted in the enterprise and in most of scenarios, production environments requires special support and services. The following document describe the certified  service providers and further guidelines to apply to become one.
+[Fluentd](https://www.fluentd.org) is widely adopted in the enterprise and in most of scenarios, production environments requires special support and services. The following document describe the certified  service providers and further guidelines to apply to become one.
 
 ## Fluentd Providers
 
@@ -10,7 +10,7 @@ The following section lists the companies that provides Support, Consulting Serv
 
 ### Treasure Data
 
-[Treasure Data](https://www.treasuredata.com) is one of the principal sponsors of Fluentd, it provides technical support, consulting services and [Fluentd Enterprise](https://fluentd.treasuredata.com/), a Fluentd-on steroids with enhanced security and connectors for enterprise backends such as [Apache Kafka](http://kafka.apache.org) and [Splunk](http://www.splunk.com) within others.
+[Treasure Data](https://www.treasuredata.com) is one of the principal sponsors of Fluentd, it provides technical support, consulting services and [Fluentd Enterprise](https://fluentd.treasuredata.com/), a Fluentd-on steroids with enhanced security and connectors for enterprise backends such as [Apache Kafka](https://kafka.apache.org) and [Splunk](https://www.splunk.com) within others.
 
 For more details about Fluentd Enterprise provided by Treasure Data, click [here](https://fluentd.treasuredata.com/).
 
@@ -35,7 +35,7 @@ In order to be listed as a service provider, please send an email using your cor
   - What kind of services around Fluentd do you provide ?
 - Community
   - List of activities of your company within the Fluentd community in the last 12 months
-  - Are you a member of the [Cloud Native Computing Foundation](http://cncf.io) ?
+  - Are you a member of the [Cloud Native Computing Foundation](https://cncf.io) ?
 
 ### Maintainers / Recipients
 

data/README.md

@@ -5,10 +5,10 @@ Fluentd: Open-Source Log Collector
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1189/badge)](https://bestpractices.coreinfrastructure.org/projects/1189)
 [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd.svg?type=shield)](https://app.fossa.io/projects/git%2Bhttps%3A%2F%2Fgithub.com%2Ffluent%2Ffluentd?ref=badge_shield)
 
-[Fluentd](http://fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the [Unified Logging Layer](http://www.fluentd.org/blog/unified-logging-layer)).
+[Fluentd](https://fluentd.org/) collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the [Unified Logging Layer](https://www.fluentd.org/blog/unified-logging-layer)).
 
 <p align="center">
-<img src="http://docs.fluentd.org/images/fluentd-architecture.png" width="500px"/>
+<img src="https://docs.fluentd.org/images/fluentd-architecture.png" width="500px"/>
 </p>
 
 An event consists of *tag*, *time* and *record*. Tag is a string separated with '.' (e.g. myapp.access). It is used to categorize events. Time is a UNIX time recorded at occurrence of an event. Record is a JSON object.
@@ -62,16 +62,16 @@ Many enterprises run Fluentd in production to handle all of their logging needs.
 
 [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd.
 
-<p align="center"><img width="500" src="http://www.fluentd.org/images/blog/fluentd-ui.gif"/></p>
+<p align="center"><img width="500" src="https://www.fluentd.org/images/blog/fluentd-ui.gif"/></p>
 
 ## More Information
 
 - Website: https://www.fluentd.org/
-- Documentation: http://docs.fluentd.org/
-- Project repository: http://github.com/fluent
-- Discussion: http://groups.google.com/group/fluentd
-- Slack / Community: http://slack.fluentd.org
-- Newsletters: http://get.treasuredata.com/Fluentd_education
+- Documentation: https://docs.fluentd.org/
+- Project repository: https://github.com/fluent
+- Discussion: https://groups.google.com/group/fluentd
+- Slack / Community: https://slack.fluentd.org
+- Newsletters: https://www.fluentd.org/newsletter_signup
 - Author: Sadayuki Furuhashi
 - Copyright: (c) 2011 FURUHASHI Sadayuki
 - License: Apache License, Version 2.0

data/fluentd.gemspec

@@ -8,7 +8,7 @@ Gem::Specification.new do |gem|
   gem.email         = ["frsyuki@gmail.com"]
   gem.description   = %q{Fluentd is an open source data collector designed to scale and simplify log management. It can collect, process and ship many kinds of data in near real-time.}
   gem.summary       = %q{Fluentd event collector}
-  gem.homepage      = "http://fluentd.org/"
+  gem.homepage      = "https://fluentd.org/"
 
   gem.files         = `git ls-files`.split($\)
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }

data/lib/fluent/plugin/formatter_tsv.rb

@@ -25,9 +25,13 @@ module Fluent
       config_param :keys, :array, value_type: :string
       desc 'The delimiter character (or string) of TSV values'
       config_param :delimiter, :string, default: "\t"
+      desc 'The parameter to enable writing to new lines'
+      config_param :add_newline, :bool, default: true
 
       def format(tag, time, record)
-        @keys.map{|k| record[k].to_s }.join(@delimiter)
+        formatted = @keys.map{|k| record[k].to_s }.join(@delimiter)
+        formatted << "\n".freeze if @add_newline
+        formatted
       end
     end
   end

data/lib/fluent/plugin/out_file.rb

@@ -141,8 +141,8 @@ module Fluent::Plugin
         dummy_record_keys = get_placeholders_keys(@path_template) || ['message']
         dummy_record = Hash[dummy_record_keys.zip(['data'] * dummy_record_keys.size)]
 
-        test_meta1 = metadata_for_test(dummy_tag, Fluent::Engine.now, dummy_record)
-        test_path = extract_placeholders(@path_template, test_meta1)
+        test_chunk1 = chunk_for_test(dummy_tag, Fluent::Engine.now, dummy_record)
+        test_path = extract_placeholders(@path_template, test_chunk1)
         unless ::Fluent::FileUtil.writable_p?(test_path)
           raise Fluent::ConfigError, "out_file: `#{test_path}` is not writable"
         end
@@ -178,7 +178,7 @@ module Fluent::Plugin
     end
 
     def write(chunk)
-      path = extract_placeholders(@path_template, chunk.metadata)
+      path = extract_placeholders(@path_template, chunk)
       FileUtils.mkdir_p File.dirname(path), mode: @dir_perm
 
       writer = case

data/lib/fluent/plugin/out_forward.rb

@@ -582,7 +582,7 @@ module Fluent::Plugin
           rescue IO::WaitReadable
             # If the exception is Errno::EWOULDBLOCK or Errno::EAGAIN, it is extended by IO::WaitReadable.
             # So IO::WaitReadable can be used to rescue the exceptions for retrying read_nonblock.
-            # http://docs.ruby-lang.org/en/2.3.0/IO.html#method-i-read_nonblock
+            # https//docs.ruby-lang.org/en/2.3.0/IO.html#method-i-read_nonblock
             sleep @sender.read_interval unless @state == :established
           rescue SystemCallError => e
             @log.warn "disconnected by error", host: @host, port: @port, error: e

data/lib/fluent/plugin/out_secondary_file.rb

@@ -71,7 +71,7 @@ module Fluent::Plugin
     end
 
     def write(chunk)
-      path_without_suffix = extract_placeholders(@path_without_suffix, chunk.metadata)
+      path_without_suffix = extract_placeholders(@path_without_suffix, chunk)
       path = generate_path(path_without_suffix)
       FileUtils.mkdir_p File.dirname(path), mode: @dir_perm
 
@@ -106,7 +106,7 @@ module Fluent::Plugin
         raise Fluent::ConfigError, "out_secondary_file: basename or directory has an incompatible placeholder #{ph}, remove tag placeholder, like `${tag}`, from basename or directory"
       end
 
-      vars = placeholders.reject { |placeholder| placeholder.match(/tag(\[\d+\])?/) }
+      vars = placeholders.reject { |placeholder| placeholder.match(/tag(\[\d+\])?/) || (placeholder == 'chunk_id') }
 
       if ph = vars.find { |v| !@chunk_keys.include?(v) }
         raise Fluent::ConfigError, "out_secondary_file: basename or directory has an incompatible placeholder #{ph}, remove variable placeholder, like `${varname}`, from basename or directory"

data/lib/fluent/plugin/out_stream.rb

@@ -93,7 +93,7 @@ module Fluent
     def initialize
       super
       $log.warn "'tcp' output is obsoleted and will be removed. Use 'forward' instead."
-      $log.warn "see 'forward' section in http://docs.fluentd.org/ for the high-availability configuration."
+      $log.warn "see 'forward' section in https://docs.fluentd.org/ for the high-availability configuration."
     end
 
     config_param :port, :integer, default: LISTEN_PORT

data/lib/fluent/plugin/output.rb

@@ -39,6 +39,7 @@ module Fluent
       CHUNK_KEY_PATTERN = /^[-_.@a-zA-Z0-9]+$/
       CHUNK_KEY_PLACEHOLDER_PATTERN = /\$\{[-_.@$a-zA-Z0-9]+\}/
       CHUNK_TAG_PLACEHOLDER_PATTERN = /\$\{(tag(?:\[\d+\])?)\}/
+      CHUNK_ID_PLACEHOLDER_PATTERN = /\$\{chunk_id\}/
 
       CHUNKING_FIELD_WARN_NUM = 4
 
@@ -679,13 +680,27 @@ module Fluent
       end
 
       def get_placeholders_keys(str)
-        str.scan(CHUNK_KEY_PLACEHOLDER_PATTERN).map{|ph| ph[2..-2]}.reject{|s| s == "tag"}.sort
+        str.scan(CHUNK_KEY_PLACEHOLDER_PATTERN).map{|ph| ph[2..-2]}.reject{|s| (s == "tag") || (s == 'chunk_id') }.sort
       end
 
       # TODO: optimize this code
-      def extract_placeholders(str, metadata)
+      def extract_placeholders(str, chunk)
+        metadata = if chunk.is_a?(Fluent::Plugin::Buffer::Chunk)
+                     chunk_passed = true
+                     chunk.metadata
+                   else
+                     chunk_passed = false
+                     # For existing plugins. Old plugin passes Chunk.metadata instead of Chunk
+                     chunk
+                   end
         if metadata.empty?
-          str
+          str.sub(CHUNK_ID_PLACEHOLDER_PATTERN) {
+            if chunk_passed
+              dump_unique_id_hex(chunk.unique_id)
+            else
+              log.warn "${chunk_id} is not allowed in this plugin. Pass Chunk instead of metadata in extract_placeholders's 2nd argument"
+            end
+          }
         else
           rvalue = str.dup
           # strftime formatting
@@ -720,7 +735,13 @@ module Fluent
           if rvalue =~ CHUNK_KEY_PLACEHOLDER_PATTERN
             log.warn "chunk key placeholder '#{$1}' not replaced. template:#{str}"
           end
-          rvalue
+          rvalue.sub(CHUNK_ID_PLACEHOLDER_PATTERN) {
+            if chunk_passed
+              dump_unique_id_hex(chunk.unique_id)
+            else
+              log.warn "${chunk_id} is not allowed in this plugin. Pass Chunk instead of metadata in extract_placeholders's 2nd argument"
+            end
+          }
         end
       end
 
@@ -801,6 +822,13 @@ module Fluent
         end
       end
 
+      def chunk_for_test(tag, time, record)
+        require 'fluent/plugin/buffer/memory_chunk'
+
+        m = metadata_for_test(tag, time, record)
+        Fluent::Plugin::Buffer::MemoryChunk.new(m)
+      end
+
       def metadata_for_test(tag, time, record)
         raise "BUG: #metadata_for_test is available only when no actual metadata exists" unless @buffer.metadata_list.empty?
         m = metadata(tag, time, record)

data/lib/fluent/plugin/parser_apache2.rb

@@ -26,10 +26,14 @@ module Fluent
 
       def initialize
         super
-        @time_parser = time_parser_create(format: TIME_FORMAT)
         @mutex = Mutex.new
       end
 
+      def configure(conf)
+        super
+        @time_parser = time_parser_create(format: TIME_FORMAT)
+      end
+
       def patterns
         {'format' => REGEXP, 'time_format' => TIME_FORMAT}
       end

data/lib/fluent/plugin_helper/cert_option.rb

@@ -104,18 +104,28 @@ module Fluent
         cert.not_before = Time.at(0)
         cert.not_after = Time.now + opts[:expiration]
         cert.public_key = key
-        cert.serial = 1
+        cert.version = 2
+        cert.serial = rand(2**(8*10))
         cert.issuer = issuer
         cert.subject  = subject
 
         return cert, key
       end
 
+      def cert_option_add_extensions(cert, extensions)
+        ef = OpenSSL::X509::ExtensionFactory.new
+        extensions.each do |ext|
+          oid, value = ext
+          cert.add_extension ef.create_extension(oid, value)
+        end
+      end
+
       def cert_option_generate_ca_pair_self_signed(generate_opts)
         cert, key = cert_option_generate_pair(generate_opts)
 
-        # basicConstraints: this cert is for CA or not
-        cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(true)]))
+        cert_option_add_extensions(cert, [
+          ['basicConstraints', 'CA:TRUE']
+        ])
 
         cert.sign(key, generate_opts[:digest].to_s)
         return cert, key
@@ -127,9 +137,12 @@ module Fluent
         cert, key = cert_option_generate_pair(generate_opts, ca_cert.subject)
         raise "BUG: certificate digest algorithm not set" unless generate_opts[:digest]
 
-        # basicConstraints: this cert is for CA or not
-        cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(false)]))
-        cert.add_extension OpenSSL::X509::Extension.new('nsCertType', 'server')
+        cert_option_add_extensions(cert, [
+          ['basicConstraints', 'CA:FALSE'],
+          ['nsCertType', 'server'],
+          ['keyUsage', 'digitalSignature,keyEncipherment'],
+          ['extendedKeyUsage', 'serverAuth']
+        ])
 
         cert.sign(ca_key, generate_opts[:digest].to_s)
         return cert, key, nil
@@ -139,9 +152,10 @@ module Fluent
         cert, key = cert_option_generate_pair(generate_opts)
         raise "BUG: certificate digest algorithm not set" unless generate_opts[:digest]
 
-        # basicConstraints: this cert is for CA or not
-        cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(false)]))
-        cert.add_extension OpenSSL::X509::Extension.new('nsCertType', 'server')
+        cert_option_add_extensions(cert, [
+          ['basicConstraints', 'CA:FALSE'],
+          ['nsCertType', 'server']
+        ])
 
         cert.sign(key, generate_opts[:digest].to_s)
         return cert, key, nil

data/lib/fluent/plugin_helper/record_accessor.rb

@@ -133,8 +133,9 @@ module Fluent
 
           until param.empty?
             if in_bracket
-              if param[0] == "'"
-                if i = param.index("']")
+              if param[0] == "'" || param[0] == '"'
+                if i = param.index("']") || param.index('"]')
+                  raise Fluent::ConfigError, "Mismatched quotes. Invalid syntax: #{orig_param}" unless param[0] == param[i]
                   result << param[1..i - 1]
                   param = param[i + 2..-1]
                   in_bracket = false

data/lib/fluent/test/helpers.rb

@@ -21,7 +21,7 @@ require 'fluent/time'
 module Fluent
   module Test
     module Helpers
-      # See "Example Custom Assertion: http://test-unit.github.io/test-unit/en/Test/Unit/Assertions.html
+      # See "Example Custom Assertion: https://test-unit.github.io/test-unit/en/Test/Unit/Assertions.html
       def assert_equal_event_time(expected, actual, message = nil)
         expected_s = "#{Time.at(expected.sec)} (nsec #{expected.nsec})"
         actual_s   = "#{Time.at(actual.sec)  } (nsec #{actual.nsec})"

data/lib/fluent/version.rb

@@ -16,6 +16,6 @@
 
 module Fluent
 
-  VERSION = '0.14.21'
+  VERSION = '0.14.22.rc1'
 
 end

data/templates/new_gem/README.md.erb

@@ -1,6 +1,6 @@
 # <%= gem_name %>
 
-[Fluentd](http://fluentd.org/) <%= type %> plugin to do something.
+[Fluentd](https://fluentd.org/) <%= type %> plugin to do something.
 
 TODO: write description for you plugin.
 

data/test/plugin/test_formatter_ltsv.rb

@@ -23,14 +23,17 @@ class LabeledTSVFormatterTest < ::Test::Unit::TestCase
     d = create_driver
     assert_equal "\t", d.instance.delimiter
     assert_equal  ":", d.instance.label_delimiter
+    assert_equal  true, d.instance.add_newline
 
     d = create_driver(
       'delimiter'       => ',',
       'label_delimiter' => '=',
+      'add_newline' => false,
     )
 
     assert_equal ",", d.instance.delimiter
     assert_equal "=", d.instance.label_delimiter
+    assert_equal  false, d.instance.add_newline
   end
 
   def test_format

data/test/plugin/test_formatter_tsv.rb

@@ -0,0 +1,68 @@
+require_relative '../helper'
+require 'fluent/test/driver/formatter'
+require 'fluent/plugin/formatter_tsv'
+
+class TSVFormatterTest < ::Test::Unit::TestCase
+  def setup
+    @time = event_time
+  end
+
+  def create_driver(conf = "")
+    Fluent::Test::Driver::Formatter.new(Fluent::Plugin::TSVFormatter).configure(conf)
+  end
+
+  def tag
+    "tag"
+  end
+
+  def record
+    {'message' => 'awesome', 'greeting' => 'hello'}
+  end
+
+  def test_config_params
+    d = create_driver(
+      'keys' => 'message,greeting',
+    )
+    assert_equal ["message", "greeting"], d.instance.keys
+    assert_equal "\t", d.instance.delimiter
+    assert_equal true, d.instance.add_newline
+
+    d = create_driver(
+      'keys' => 'message,greeting',
+      'delimiter' => ',',
+      'add_newline' => false,
+    )
+    assert_equal ["message", "greeting"], d.instance.keys
+    assert_equal ",", d.instance.delimiter
+    assert_equal false, d.instance.add_newline
+  end
+
+  def test_format
+    d = create_driver(
+      'keys' => 'message,greeting',
+    )
+    formatted = d.instance.format(tag, @time, record)
+
+    assert_equal("awesome\thello\n", formatted)
+  end
+
+  def test_format_without_newline
+    d = create_driver(
+      'keys' => 'message,greeting',
+      'add_newline' => false,
+    )
+    formatted = d.instance.format(tag, @time, record)
+
+    assert_equal("awesome\thello", formatted)
+  end
+
+  def test_format_with_customized_delimiters
+    d = create_driver(
+      'keys' => 'message,greeting',
+      'delimiter' => ',',
+    )
+    formatted = d.instance.format(tag, @time, record)
+
+    assert_equal("awesome,hello\n", formatted)
+  end
+end

data/test/plugin/test_out_file.rb

@@ -557,6 +557,35 @@ class FileOutputTest < Test::Unit::TestCase
     check_gzipped_result(path, formatted_lines * 3)
   end
 
+  test '${chunk_id}' do
+    time = event_time("2011-01-02 13:14:15 UTC")
+    formatted_lines = %[2011-01-02T13:14:15Z\ttest\t{"a":1}\n] + %[2011-01-02T13:14:15Z\ttest\t{"a":2}\n]
+
+    write_once = ->(){
+      d = create_driver %[
+        path #{TMP_DIR}/out_file_chunk_id_${chunk_id}
+        utc
+        append true
+        <buffer>
+          timekey_use_utc true
+        </buffer>
+      ]
+      d.run(default_tag: 'test'){
+        d.feed(time, {"a"=>1})
+        d.feed(time, {"a"=>2})
+      }
+      d.instance.last_written_path
+    }
+
+    path = write_once.call
+    if File.basename(path) =~ /out_file_chunk_id_([-_.@a-zA-Z0-9].*).20110102.log/
+      unique_id = Fluent::UniqueId.hex(Fluent::UniqueId.generate)
+      assert_equal unique_id.size, $1.size, "chunk_id size is mismatched"
+    else
+      flunk "chunk_id is not included in the path"
+    end
+  end
+
   test 'symlink' do
     omit "Windows doesn't support symlink" if Fluent.windows?
     conf = CONFIG + %[

data/test/plugin/test_out_secondary_file.rb

@@ -260,6 +260,20 @@ class FileOutputSecondaryTest < Test::Unit::TestCase
       assert_equal "#{TMP_DIR}/dump.bin", path
     end
 
+    test 'path with ${chunk_id}' do
+      d = create_driver %[
+        directory #{TMP_DIR}
+        basename out_file_chunk_id_${chunk_id}
+      ]
+      path = d.instance.write(@c)
+      if File.basename(path) =~ /out_file_chunk_id_([-_.@a-zA-Z0-9].*).0/
+        unique_id = Fluent::UniqueId.hex(Fluent::UniqueId.generate)
+        assert_equal unique_id.size, $1.size, "chunk_id size is mismatched"
+      else
+        flunk "chunk_id is not included in the path"
+      end
+    end
+
     data(
       invalid_tag: [/tag/, '${tag}'],
       invalid_tag0: [/tag\[0\]/, '${tag[0]}'],

data/test/plugin/test_output.rb

@@ -125,6 +125,10 @@ class OutputTest < Test::Unit::TestCase
   def create_metadata(timekey: nil, tag: nil, variables: nil)
     Fluent::Plugin::Buffer::Metadata.new(timekey, tag, variables)
   end
+  def create_chunk(timekey: nil, tag: nil, variables: nil)
+    m = Fluent::Plugin::Buffer::Metadata.new(timekey, tag, variables)
+    Fluent::Plugin::Buffer::MemoryChunk.new(m)
+  end
   def waiting(seconds)
     begin
       Timeout.timeout(seconds) do
@@ -219,7 +223,9 @@ class OutputTest < Test::Unit::TestCase
       assert @i.terminated?
     end
 
-    test '#extract_placeholders does nothing if chunk key is not specified' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders does nothing if chunk key is not specified' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', '')]))
       assert !@i.chunk_key_time
       assert !@i.chunk_key_tag
@@ -227,11 +233,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal tmpl, @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal tmpl, @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders can extract time if time key and range are configured' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders can extract time if time key and range are configured' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'time', {'timekey' => 60*30, 'timekey_zone' => "+0900"})]))
       assert @i.chunk_key_time
       assert !@i.chunk_key_tag
@@ -239,11 +251,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/2016/04/11/20-30/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail", @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/2016/04/11/20-30/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail", @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders can extract tag and parts of tag if tag is configured' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders can extract tag and parts of tag if tag is configured' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'tag', {})]))
       assert !@i.chunk_key_time
       assert @i.chunk_key_tag
@@ -251,11 +269,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/%Y/%m/%d/%H-%M/fluentd.test.output/test/output/${key1}/${key2}/tail", @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/%Y/%m/%d/%H-%M/fluentd.test.output/test/output/${key1}/${key2}/tail", @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders can extract variables if variables are configured' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders can extract variables if variables are configured' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'key1,key2', {})]))
       assert !@i.chunk_key_time
       assert !@i.chunk_key_tag
@@ -263,11 +287,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/value1/value2/tail", @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/value1/value2/tail", @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders can extract nested variables if variables are configured with dot notation' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders can extract nested variables if variables are configured with dot notation' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'key,$.nest.key', {})]))
       assert !@i.chunk_key_time
       assert !@i.chunk_key_tag
@@ -275,11 +305,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/${key}/${$.nest.key}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {:key => "value1", :"$.nest.key" => "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/value1/value2/tail", @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/value1/value2/tail", @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders can extract all chunk keys if configured' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders can extract all chunk keys if configured' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'time,tag,key1,key2', {'timekey' => 60*30, 'timekey_zone' => "+0900"})]))
       assert @i.chunk_key_time
       assert @i.chunk_key_tag
@@ -287,11 +323,17 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[1]}/${tag[2]}/${key1}/${key2}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
-      m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/2016/04/11/20-30/fluentd.test.output/test/output/value1/value2/tail", @i.extract_placeholders(tmpl, m)
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/2016/04/11/20-30/fluentd.test.output/test/output/value1/value2/tail", @i.extract_placeholders(tmpl, c)
     end
 
-    test '#extract_placeholders removes out-of-range tag part and unknown variable placeholders' do
+    data(:new_api => :chunk,
+         :old_api => :metadata)
+    test '#extract_placeholders removes out-of-range tag part and unknown variable placeholders' do |api|
       @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'time,tag,key1,key2', {'timekey' => 60*30, 'timekey_zone' => "+0900"})]))
       assert @i.chunk_key_time
       assert @i.chunk_key_tag
@@ -299,8 +341,23 @@ class OutputTest < Test::Unit::TestCase
       tmpl = "/mypath/%Y/%m/%d/%H-%M/${tag}/${tag[3]}/${tag[4]}/${key3}/${key4}/tail"
       t = event_time('2016-04-11 20:30:00 +0900')
       v = {key1: "value1", key2: "value2"}
+      c = if api == :chunk
+            create_chunk(timekey: t, tag: 'fluentd.test.output', variables: v)
+          else
+            create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
+          end
+      assert_equal "/mypath/2016/04/11/20-30/fluentd.test.output/////tail", @i.extract_placeholders(tmpl, c)
+    end
+
+    test '#extract_placeholders logs warn message if metadata is passed for ${chunk_id} placeholder' do
+      @i.configure(config_element('ROOT', '', {}, [config_element('buffer', '')]))
+      tmpl = "/mypath/${chunk_id}/tail"
+      t = event_time('2016-04-11 20:30:00 +0900')
+      v = {key1: "value1", key2: "value2"}
       m = create_metadata(timekey: t, tag: 'fluentd.test.output', variables: v)
-      assert_equal "/mypath/2016/04/11/20-30/fluentd.test.output/////tail", @i.extract_placeholders(tmpl, m)
+      @i.extract_placeholders(tmpl, m)
+      logs = @i.log.out.logs
+      assert { logs.any? { |log| log.include?("${chunk_id} is not allowed in this plugin") } }
     end
 
     sub_test_case '#placeholder_validators' do

data/test/plugin/test_parser_apache2.rb

@@ -16,6 +16,7 @@ class Apache2ParserTest < ::Test::Unit::TestCase
       'referer' => nil,
       'agent'   => 'Opera/12.0'
     }
+    @parser.configure({})
   end
 
   def test_parse

data/test/plugin_helper/test_record_accessor.rb

@@ -24,20 +24,23 @@ class RecordAccessorHelperTest < Test::Unit::TestCase
     end
 
     data('dot' => '$.key1.key2[0]',
-         'bracket' => "$['key1']['key2'][0]")
+         'bracket' => "$['key1']['key2'][0]",
+         'bracket w/ double quotes' => '$["key1"]["key2"][0]')
     test "nested keys ['key1', 'key2', 0]" do |param|
       result = Fluent::PluginHelper::RecordAccessor::Accessor.parse_parameter(param)
       assert_equal ['key1', 'key2', 0], result
     end
 
-    data('bracket' => "$['key1'][0]['ke y2']")
+    data('bracket' => "$['key1'][0]['ke y2']",
+         'bracket w/ double quotes' => '$["key1"][0]["ke y2"]')
     test "nested keys ['key1', 0, 'ke y2']" do |param|
       result = Fluent::PluginHelper::RecordAccessor::Accessor.parse_parameter(param)
       assert_equal ['key1', 0, 'ke y2'], result
     end
 
     data('dot' => '$.[0].key1.[1].key2',
-         'bracket' => "$[0]['key1'][1]['key2']")
+         'bracket' => "$[0]['key1'][1]['key2']",
+         'bracket w/ double quotes' => '$[0]["key1"][1]["key2"]')
     test "nested keys [0, 'key1', 1, 'key2']" do |param|
       result = Fluent::PluginHelper::RecordAccessor::Accessor.parse_parameter(param)
       assert_equal [0, 'key1', 1, 'key2'], result
@@ -49,7 +52,9 @@ class RecordAccessorHelperTest < Test::Unit::TestCase
          "more chars" => "$.key1[0]foo",
          "whitespace char included key in dot notation" => "$.key[0].ke y",
          "empty keys with dot" => "$.",
-         "empty keys with bracket" => "$[")
+         "empty keys with bracket" => "$[",
+         "mismatched quotes1" => "$['key1']['key2\"]",
+         "mismatched quotes2" => '$["key1"]["key2\']')
     test 'invalid syntax' do |param|
       assert_raise Fluent::ConfigError do
         Fluent::PluginHelper::RecordAccessor::Accessor.parse_parameter(param)
@@ -78,14 +83,16 @@ class RecordAccessorHelperTest < Test::Unit::TestCase
     end
 
     data('dot' => '$.key1.key2[0]',
-         'bracket' => "$['key1']['key2'][0]")
+         'bracket' => "$['key1']['key2'][0]",
+         'bracket w/ double quotes' => '$["key1"]["key2"][0]')
     test "nested keys ['key1', 'key2', 0]" do |param|
       r = {'key1' => {'key2' => [1, 2, 3]}}
       accessor = @d.record_accessor_create(param)
       assert_equal 1, accessor.call(r)
     end
 
-    data('bracket' => "$['key1'][0]['ke y2']")
+    data('bracket' => "$['key1'][0]['ke y2']",
+         'bracket w/ double quotes' => '$["key1"][0]["ke y2"]')
     test "nested keys ['key1', 0, 'ke y2']" do |param|
       r = {'key1' => [{'ke y2' => "value"}]}
       accessor = @d.record_accessor_create(param)
@@ -98,7 +105,9 @@ class RecordAccessorHelperTest < Test::Unit::TestCase
          "whitespace char included key in dot notation" => "$.key[0].ke y",
          "more chars" => "$.key1[0]foo",
          "empty keys with dot" => "$.",
-         "empty keys with bracket" => "$[")
+         "empty keys with bracket" => "$[",
+         "mismatched quotes1" => "$['key1']['key2\"]",
+         "mismatched quotes2" => '$["key1"]["key2\']')
     test 'invalid syntax' do |param|
       assert_raise Fluent::ConfigError do
         @d.record_accessor_create(param)

data/test/plugin_helper/test_server.rb

@@ -776,6 +776,7 @@ class ServerPluginHelperTest < Test::Unit::TestCase
   def create_server_pair_signed_by_self(cert_path, private_key_path, passphrase)
     cert, key, _ = CertUtil.cert_option_generate_server_pair_self_signed(create_server_options)
     write_cert_and_key(cert_path, cert, private_key_path, key, passphrase)
+    return cert
   end
 
   def create_ca_pair_signed_by_self(cert_path, private_key_path, passphrase)
@@ -786,6 +787,7 @@ class ServerPluginHelperTest < Test::Unit::TestCase
   def create_server_pair_signed_by_ca(ca_cert_path, ca_key_path, ca_key_passphrase, cert_path, private_key_path, passphrase)
     cert, key, _ = CertUtil.cert_option_generate_server_pair_by_ca(ca_cert_path, ca_key_path, ca_key_passphrase, create_server_options)
     write_cert_and_key(cert_path, cert, private_key_path, key, passphrase)
+    return cert
   end
 
   def create_server_pair_chained_with_root_ca(ca_cert_path, ca_key_path, ca_key_passphrase, cert_path, private_key_path, passphrase)
@@ -842,6 +844,20 @@ class ServerPluginHelperTest < Test::Unit::TestCase
     sock.close rescue nil
   end
 
+  def assert_certificate(cert, expected_extensions)
+    get_extension = lambda do |oid|
+      cert.extensions.detect { |e| e.oid == oid }
+    end
+
+    assert_true cert.serial > 1
+    assert_equal 2, cert.version
+
+    expected_extensions.each do |ext|
+      expected_oid, expected_value = ext
+      assert_equal expected_value, get_extension.call(expected_oid).value
+    end
+  end
+
   sub_test_case '#server_create_tls with various certificate options' do
     setup do
       @d = Dummy.new # to get plugin not configured/started yet
@@ -896,7 +912,12 @@ class ServerPluginHelperTest < Test::Unit::TestCase
       test 'load self-signed cert/key pair (files), verified from clients using cert files' do |private_key_passphrase|
         cert_path = File.join(@server_cert_dir, "cert.pem")
         private_key_path = File.join(@certs_dir, "server.key.pem")
-        create_server_pair_signed_by_self(cert_path, private_key_path, private_key_passphrase)
+        cert = create_server_pair_signed_by_self(cert_path, private_key_path, private_key_passphrase)
+
+        assert_certificate(cert,[
+          ['basicConstraints', 'CA:FALSE'],
+          ['nsCertType', 'SSL Server']
+        ])
 
         tls_options = {
           protocol: :tls,
@@ -963,7 +984,14 @@ class ServerPluginHelperTest < Test::Unit::TestCase
 
         cert_path = File.join(@server_cert_dir, "cert.pem")
         private_key_path = File.join(@certs_dir, "server.key.pem")
-        create_server_pair_signed_by_ca(ca_cert_path, ca_key_path, ca_key_passphrase, cert_path, private_key_path, private_key_passphrase)
+        cert = create_server_pair_signed_by_ca(ca_cert_path, ca_key_path, ca_key_passphrase, cert_path, private_key_path, private_key_passphrase)
+
+        assert_certificate(cert,[
+            ['basicConstraints', 'CA:FALSE'],
+            ['nsCertType', 'SSL Server'],
+            ['keyUsage', 'Digital Signature, Key Encipherment'],
+            ['extendedKeyUsage', 'TLS Web Server Authentication']
+        ])
 
         tls_options = {
           protocol: :tls,

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluentd
 version: !ruby/object:Gem::Version
-  version: 0.14.21
+  version: 0.14.22.rc1
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-07 00:00:00.000000000 Z
+date: 2017-10-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -318,10 +318,11 @@ files:
 - ".github/ISSUE_TEMPLATE.md"
 - ".gitignore"
 - ".travis.yml"
+- ADOPTERS.md
 - AUTHORS
+- CHANGELOG.md
 - CONTRIBUTING.md
 - COPYING
-- ChangeLog
 - ENTERPRISE_PROVIDERS.md
 - Gemfile
 - MAINTAINERS.md
@@ -623,6 +624,7 @@ files:
 - test/plugin/test_formatter_msgpack.rb
 - test/plugin/test_formatter_out_file.rb
 - test/plugin/test_formatter_single_value.rb
+- test/plugin/test_formatter_tsv.rb
 - test/plugin/test_in_debug_agent.rb
 - test/plugin/test_in_dummy.rb
 - test/plugin/test_in_exec.rb
@@ -718,7 +720,7 @@ files:
 - test/test_time_formatter.rb
 - test/test_time_parser.rb
 - test/test_unique_id.rb
-homepage: http://fluentd.org/
+homepage: https://fluentd.org/
 licenses:
 - Apache-2.0
 metadata: {}
@@ -733,12 +735,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.1'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.11
+rubygems_version: 2.6.12
 signing_key: 
 specification_version: 4
 summary: Fluentd event collector
@@ -789,6 +791,7 @@ test_files:
 - test/plugin/test_formatter_msgpack.rb
 - test/plugin/test_formatter_out_file.rb
 - test/plugin/test_formatter_single_value.rb
+- test/plugin/test_formatter_tsv.rb
 - test/plugin/test_in_debug_agent.rb
 - test/plugin/test_in_dummy.rb
 - test/plugin/test_in_exec.rb