fluent-plugin-sql 1.1.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 23ecce81f2ec2830716c45445688e709d0b16cf4
-  data.tar.gz: 0fd2ea1be49a7a67428727138840905a932db1d0
+SHA256:
+  metadata.gz: 15313a249bf6706d547ed4af5baa6de995f256204b17a57d9569c0efbab59bd9
+  data.tar.gz: 1909f37a4b7e5f3cfa281212a6e66c462e9cbc82f79930713f79e0cb9cc1761e
 SHA512:
-  metadata.gz: e033466e88f576493744f89e4153ca6fe9258ca4ef70372892aaaefa212c354965577308b8c4efa43b8489c571c51ac9ad9cf1a28496d03892c90c710a22cf79
-  data.tar.gz: 4f527cb05b629a9a0d9558ee9940ecd7b127a45b91f9d5f3276e74de6f7cee0617bcc13e3d4c716f101f0463787fb90d75c238309fd4180bb494ac9ddb86cae1
+  metadata.gz: 5f5664d4a54f8cd0a383baec2610e7e9d80e752873d2235a198734805fd0290b64b6588d26c9893cb9d98f99f0204f1f56bbe11fed825269e5b1dc9b6d5d0699
+  data.tar.gz: 67a5e652889d5e36dcf8e4b5b8419ef62d46b706873779822fab0c03a4bc25ad466a22817ef89e59b2056649c90a16e613cd278785dc469f6ef5c0b2d34614cd

data/.travis.yml

@@ -5,17 +5,17 @@ addons:
   postgresql: "9.4"
 
 rvm:
-  - 2.3.6
-  - 2.4.6
-  - 2.5.5
-  - 2.6.3
+  - 2.4.10
+  - 2.5.8
+  - 2.6.6
+  - 2.7.1
   - ruby-head
 
 gemfile:
  - Gemfile
 
 before_install:
-  - gem update --system
+  - gem update bundler
 
 before_script:
   - psql -U postgres -c "CREATE ROLE fluentd WITH LOGIN ENCRYPTED PASSWORD 'fluentd';"

data/ChangeLog

@@ -0,0 +1,16 @@
+Release 2.0.0 - 2020/04/22
+
+  * out_sql: Remove v0.12 API code and use full v1 API. Change buffer format.
+
+Release 1.1.1 - 2019/05/10
+
+  * out_sql: Support schema_search_path option of PostgreSQL
+
+Release 1.1.0 - 2018/10/04
+
+  * Upgrade ActiveRecord to 5.1
+
+Release 1.0.0 - 2018/04/06
+
+  * Support v1 API
+

data/README.md

@@ -5,7 +5,7 @@
 This SQL plugin has two parts:
 
 1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
-2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
+2. SQL **output** plugin that writes records into RDBMSes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
 
 ## Requirements
 
@@ -14,19 +14,18 @@ This SQL plugin has two parts:
 | >= 1.0.0          | >= v0.14.4 | >= 2.1 |
 | <  1.0.0          | <  v0.14.0 | >= 1.9 |
 
-NOTE: fluent-plugin-sql v1.0.0 is now RC. We will release stable v1.0.0 soon.
+NOTE: fluent-plugin-sql v2's buffer format is different from v1. If you update the plugin to v2, don't reuse v1's buffer.
 
 ## Installation
 
     $ fluent-gem install fluent-plugin-sql --no-document
-    $ fluent-gem install pg -v 0.21.0 --no-document # for postgresql
+    $ fluent-gem install pg --no-document # for postgresql
 
 You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
 
 We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
 
-We must use pg gem 0.21.0 (< 1.0.0) because ActiveRecord 5.0 or later doesn't support Ruby 2.1.
-NOTE: ActiveRecord has supported pg 1.0.0 since ActiveRecord 5.1.5.
+If you use ruby 2.1, use pg gem 0.21.0 (< 1.0.0) because ActiveRecord 5.1.4 or earlier doesn't support Ruby 2.1.
 
 ## Input: How It Works
 
@@ -151,10 +150,12 @@ This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `dat
 * **username** RDBMS login user name
 * **password** RDBMS login password
 * **socket** RDBMS socket path
+* **pool** A connection pool synchronizes thread access to a limited number of database connections
+* **timeout** RDBMS connection timeout
 * **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
 
 \<table\> sections:
 
 * **table** RDBM table name
 * **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
-* **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
+* **\<table pattern\>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of \<match\>](https://docs.fluentd.org/configuration/config-file#how-match-patterns-work). **Exactly one \<table\> element must NOT have this parameter so that it becomes the default table to store data**.

data/VERSION

@@ -1 +1 @@
-1.1.1
+2.0.0

data/fluent-plugin-sql.gemspec

@@ -9,7 +9,6 @@ Gem::Specification.new do |gem|
   gem.version     = File.read("VERSION").strip
   gem.authors     = ["Sadayuki Furuhashi"]
   gem.email       = "frsyuki@gmail.com"
-  gem.has_rdoc    = false
   #gem.platform    = Gem::Platform::RUBY
   gem.files       = `git ls-files`.split("\n")
   gem.test_files  = `git ls-files -- {test,spec,features}/*`.split("\n")

data/lib/fluent/plugin/out_sql.rb

@@ -1,11 +1,12 @@
 require "fluent/plugin/output"
 
+require 'active_record'
+require 'activerecord-import'
+
 module Fluent::Plugin
   class SQLOutput < Output
     Fluent::Plugin.register_output('sql', self)
 
-    DEFAULT_BUFFER_TYPE = "memory"
-
     helpers :inject, :compat_parameters, :event_emitter
 
     desc 'RDBMS host'
@@ -28,9 +29,13 @@ module Fluent::Plugin
     config_param :remove_tag_prefix, :string, default: nil
     desc 'enable fallback'
     config_param :enable_fallback, :bool, default: true
+    desc "size of ActiveRecord's connection pool"
+    config_param :pool, :integer, default: 5
+    desc "specifies the timeout to establish a new connection to the database before failing"
+    config_param :timeout, :integer, default: 5000
 
     config_section :buffer do
-      config_set_default :@type, DEFAULT_BUFFER_TYPE
+      config_set_default :chunk_keys, ["tag"]
     end
 
     attr_accessor :tables
@@ -83,11 +88,12 @@ module Fluent::Plugin
         # @model.column_names
       end
 
-      def import(chunk)
+      def import(chunk, output)
+        tag = chunk.metadata.tag
         records = []
-        chunk.msgpack_each { |tag, time, data|
+        chunk.msgpack_each { |time, data|
           begin
-            # format process should be moved to emit / format after supports error stream.
+            data = output.inject_values_to_record(tag, time, data)
             records << @model.new(@format_proc.call(data))
           rescue => e
             args = {error: e, table: @table, record: Yajl.dump(data)}
@@ -102,7 +108,7 @@ module Fluent::Plugin
             @log.warn "Got deterministic error. Fallback to one-by-one import", error: e
             one_by_one_import(records)
           else
-            $log.warn "Got deterministic error. Fallback is disabled", error: e
+            @log.warn "Got deterministic error. Fallback is disabled", error: e
             raise e
           end
         end
@@ -144,8 +150,6 @@ module Fluent::Plugin
 
     def initialize
       super
-      require 'active_record'
-      require 'activerecord-import'
     end
 
     def configure(conf)
@@ -171,7 +175,10 @@ module Fluent::Plugin
           @tables << te
         end
       }
-      @only_default = @tables.empty?
+
+      if @pool < @buffer_config.flush_thread_count
+        log.warn "connection pool size is smaller than buffer's flush_thread_count. Recommend to increase pool value", :pool => @pool, :flush_thread_count => @buffer_config.flush_thread_count
+      end
 
       if @default_table.nil?
         raise Fluent::ConfigError, "There is no default table. <table> is required in sql output"
@@ -190,6 +197,8 @@ module Fluent::Plugin
         password: @password,
         socket: @socket,
         schema_search_path: @schema_search_path,
+        pool: @pool,
+        timeout: @timeout,
       }
 
       @base_model = Class.new(ActiveRecord::Base) do
@@ -210,19 +219,6 @@ module Fluent::Plugin
       super
     end
 
-    def emit(tag, es, chain)
-      if @only_default
-        super(tag, es, chain)
-      else
-        super(tag, es, chain, format_tag(tag))
-      end
-    end
-
-    def format(tag, time, record)
-      record = inject_values_to_record(tag, time, record)
-      [tag, time, record].to_msgpack
-    end
-
     def formatted_to_msgpack_binary
       true
     end
@@ -231,11 +227,12 @@ module Fluent::Plugin
       ActiveRecord::Base.connection_pool.with_connection do
 
         @tables.each { |table|
-          if table.pattern.match(chunk.key)
-            return table.import(chunk)
+          tag = format_tag(chunk.metadata.tag)
+          if table.pattern.match(tag)
+            return table.import(chunk, self)
           end
         }
-        @default_table.import(chunk)
+        @default_table.import(chunk, self)
       end
     end
 
@@ -254,7 +251,7 @@ module Fluent::Plugin
     end
 
     def format_tag(tag)
-      if @remove_tag_prefix
+      if tag && @remove_tag_prefix
         tag.gsub(@remove_tag_prefix, '')
       else
         tag

data/test/plugin/test_out_sql.rb

@@ -43,7 +43,8 @@ class SqlOutputTest < Test::Unit::TestCase
       password: "fluentd",
       schema_search_path: 'public',
       remove_tag_suffix: /^db/,
-      enable_fallback: true
+      enable_fallback: true,
+      pool: 5
     }
     actual = {
       host: d.instance.host,
@@ -54,7 +55,8 @@ class SqlOutputTest < Test::Unit::TestCase
       password: d.instance.password,
       schema_search_path: d.instance.schema_search_path,
       remove_tag_suffix: d.instance.remove_tag_prefix,
-      enable_fallback: d.instance.enable_fallback
+      enable_fallback: d.instance.enable_fallback,
+      pool: d.instance.pool
     }
     assert_equal(expected, actual)
     assert_empty(d.instance.tables)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-sql
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 2.0.0
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-05-09 00:00:00.000000000 Z
+date: 2020-04-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -135,6 +135,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".travis.yml"
+- ChangeLog
 - Gemfile
 - Gemfile.v0.12
 - README.md
@@ -166,8 +167,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.14.1
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: SQL input/output plugin for Fluentd event collector