fluent-plugin-sql 1.1.1 → 2.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 23ecce81f2ec2830716c45445688e709d0b16cf4
4
- data.tar.gz: 0fd2ea1be49a7a67428727138840905a932db1d0
2
+ SHA256:
3
+ metadata.gz: 15313a249bf6706d547ed4af5baa6de995f256204b17a57d9569c0efbab59bd9
4
+ data.tar.gz: 1909f37a4b7e5f3cfa281212a6e66c462e9cbc82f79930713f79e0cb9cc1761e
5
5
  SHA512:
6
- metadata.gz: e033466e88f576493744f89e4153ca6fe9258ca4ef70372892aaaefa212c354965577308b8c4efa43b8489c571c51ac9ad9cf1a28496d03892c90c710a22cf79
7
- data.tar.gz: 4f527cb05b629a9a0d9558ee9940ecd7b127a45b91f9d5f3276e74de6f7cee0617bcc13e3d4c716f101f0463787fb90d75c238309fd4180bb494ac9ddb86cae1
6
+ metadata.gz: 5f5664d4a54f8cd0a383baec2610e7e9d80e752873d2235a198734805fd0290b64b6588d26c9893cb9d98f99f0204f1f56bbe11fed825269e5b1dc9b6d5d0699
7
+ data.tar.gz: 67a5e652889d5e36dcf8e4b5b8419ef62d46b706873779822fab0c03a4bc25ad466a22817ef89e59b2056649c90a16e613cd278785dc469f6ef5c0b2d34614cd
@@ -5,17 +5,17 @@ addons:
5
5
  postgresql: "9.4"
6
6
 
7
7
  rvm:
8
- - 2.3.6
9
- - 2.4.6
10
- - 2.5.5
11
- - 2.6.3
8
+ - 2.4.10
9
+ - 2.5.8
10
+ - 2.6.6
11
+ - 2.7.1
12
12
  - ruby-head
13
13
 
14
14
  gemfile:
15
15
  - Gemfile
16
16
 
17
17
  before_install:
18
- - gem update --system
18
+ - gem update bundler
19
19
 
20
20
  before_script:
21
21
  - psql -U postgres -c "CREATE ROLE fluentd WITH LOGIN ENCRYPTED PASSWORD 'fluentd';"
@@ -0,0 +1,16 @@
1
+ Release 2.0.0 - 2020/04/22
2
+
3
+ * out_sql: Remove v0.12 API code and use full v1 API. Change buffer format.
4
+
5
+ Release 1.1.1 - 2019/05/10
6
+
7
+ * out_sql: Support schema_search_path option of PostgreSQL
8
+
9
+ Release 1.1.0 - 2018/10/04
10
+
11
+ * Upgrade ActiveRecord to 5.1
12
+
13
+ Release 1.0.0 - 2018/04/06
14
+
15
+ * Support v1 API
16
+
data/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
  This SQL plugin has two parts:
6
6
 
7
7
  1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
8
- 2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
8
+ 2. SQL **output** plugin that writes records into RDBMSes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
9
9
 
10
10
  ## Requirements
11
11
 
@@ -14,19 +14,18 @@ This SQL plugin has two parts:
14
14
  | >= 1.0.0 | >= v0.14.4 | >= 2.1 |
15
15
  | < 1.0.0 | < v0.14.0 | >= 1.9 |
16
16
 
17
- NOTE: fluent-plugin-sql v1.0.0 is now RC. We will release stable v1.0.0 soon.
17
+ NOTE: fluent-plugin-sql v2's buffer format is different from v1. If you update the plugin to v2, don't reuse v1's buffer.
18
18
 
19
19
  ## Installation
20
20
 
21
21
  $ fluent-gem install fluent-plugin-sql --no-document
22
- $ fluent-gem install pg -v 0.21.0 --no-document # for postgresql
22
+ $ fluent-gem install pg --no-document # for postgresql
23
23
 
24
24
  You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
25
25
 
26
26
  We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
27
27
 
28
- We must use pg gem 0.21.0 (< 1.0.0) because ActiveRecord 5.0 or later doesn't support Ruby 2.1.
29
- NOTE: ActiveRecord has supported pg 1.0.0 since ActiveRecord 5.1.5.
28
+ If you use ruby 2.1, use pg gem 0.21.0 (< 1.0.0) because ActiveRecord 5.1.4 or earlier doesn't support Ruby 2.1.
30
29
 
31
30
  ## Input: How It Works
32
31
 
@@ -151,10 +150,12 @@ This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `dat
151
150
  * **username** RDBMS login user name
152
151
  * **password** RDBMS login password
153
152
  * **socket** RDBMS socket path
153
+ * **pool** A connection pool synchronizes thread access to a limited number of database connections
154
+ * **timeout** RDBMS connection timeout
154
155
  * **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
155
156
 
156
157
  \<table\> sections:
157
158
 
158
159
  * **table** RDBM table name
159
160
  * **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
160
- * **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
161
+ * **\<table pattern\>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of \<match\>](https://docs.fluentd.org/configuration/config-file#how-match-patterns-work). **Exactly one \<table\> element must NOT have this parameter so that it becomes the default table to store data**.
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.1.1
1
+ 2.0.0
@@ -9,7 +9,6 @@ Gem::Specification.new do |gem|
9
9
  gem.version = File.read("VERSION").strip
10
10
  gem.authors = ["Sadayuki Furuhashi"]
11
11
  gem.email = "frsyuki@gmail.com"
12
- gem.has_rdoc = false
13
12
  #gem.platform = Gem::Platform::RUBY
14
13
  gem.files = `git ls-files`.split("\n")
15
14
  gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
@@ -1,11 +1,12 @@
1
1
  require "fluent/plugin/output"
2
2
 
3
+ require 'active_record'
4
+ require 'activerecord-import'
5
+
3
6
  module Fluent::Plugin
4
7
  class SQLOutput < Output
5
8
  Fluent::Plugin.register_output('sql', self)
6
9
 
7
- DEFAULT_BUFFER_TYPE = "memory"
8
-
9
10
  helpers :inject, :compat_parameters, :event_emitter
10
11
 
11
12
  desc 'RDBMS host'
@@ -28,9 +29,13 @@ module Fluent::Plugin
28
29
  config_param :remove_tag_prefix, :string, default: nil
29
30
  desc 'enable fallback'
30
31
  config_param :enable_fallback, :bool, default: true
32
+ desc "size of ActiveRecord's connection pool"
33
+ config_param :pool, :integer, default: 5
34
+ desc "specifies the timeout to establish a new connection to the database before failing"
35
+ config_param :timeout, :integer, default: 5000
31
36
 
32
37
  config_section :buffer do
33
- config_set_default :@type, DEFAULT_BUFFER_TYPE
38
+ config_set_default :chunk_keys, ["tag"]
34
39
  end
35
40
 
36
41
  attr_accessor :tables
@@ -83,11 +88,12 @@ module Fluent::Plugin
83
88
  # @model.column_names
84
89
  end
85
90
 
86
- def import(chunk)
91
+ def import(chunk, output)
92
+ tag = chunk.metadata.tag
87
93
  records = []
88
- chunk.msgpack_each { |tag, time, data|
94
+ chunk.msgpack_each { |time, data|
89
95
  begin
90
- # format process should be moved to emit / format after supports error stream.
96
+ data = output.inject_values_to_record(tag, time, data)
91
97
  records << @model.new(@format_proc.call(data))
92
98
  rescue => e
93
99
  args = {error: e, table: @table, record: Yajl.dump(data)}
@@ -102,7 +108,7 @@ module Fluent::Plugin
102
108
  @log.warn "Got deterministic error. Fallback to one-by-one import", error: e
103
109
  one_by_one_import(records)
104
110
  else
105
- $log.warn "Got deterministic error. Fallback is disabled", error: e
111
+ @log.warn "Got deterministic error. Fallback is disabled", error: e
106
112
  raise e
107
113
  end
108
114
  end
@@ -144,8 +150,6 @@ module Fluent::Plugin
144
150
 
145
151
  def initialize
146
152
  super
147
- require 'active_record'
148
- require 'activerecord-import'
149
153
  end
150
154
 
151
155
  def configure(conf)
@@ -171,7 +175,10 @@ module Fluent::Plugin
171
175
  @tables << te
172
176
  end
173
177
  }
174
- @only_default = @tables.empty?
178
+
179
+ if @pool < @buffer_config.flush_thread_count
180
+ log.warn "connection pool size is smaller than buffer's flush_thread_count. Recommend to increase pool value", :pool => @pool, :flush_thread_count => @buffer_config.flush_thread_count
181
+ end
175
182
 
176
183
  if @default_table.nil?
177
184
  raise Fluent::ConfigError, "There is no default table. <table> is required in sql output"
@@ -190,6 +197,8 @@ module Fluent::Plugin
190
197
  password: @password,
191
198
  socket: @socket,
192
199
  schema_search_path: @schema_search_path,
200
+ pool: @pool,
201
+ timeout: @timeout,
193
202
  }
194
203
 
195
204
  @base_model = Class.new(ActiveRecord::Base) do
@@ -210,19 +219,6 @@ module Fluent::Plugin
210
219
  super
211
220
  end
212
221
 
213
- def emit(tag, es, chain)
214
- if @only_default
215
- super(tag, es, chain)
216
- else
217
- super(tag, es, chain, format_tag(tag))
218
- end
219
- end
220
-
221
- def format(tag, time, record)
222
- record = inject_values_to_record(tag, time, record)
223
- [tag, time, record].to_msgpack
224
- end
225
-
226
222
  def formatted_to_msgpack_binary
227
223
  true
228
224
  end
@@ -231,11 +227,12 @@ module Fluent::Plugin
231
227
  ActiveRecord::Base.connection_pool.with_connection do
232
228
 
233
229
  @tables.each { |table|
234
- if table.pattern.match(chunk.key)
235
- return table.import(chunk)
230
+ tag = format_tag(chunk.metadata.tag)
231
+ if table.pattern.match(tag)
232
+ return table.import(chunk, self)
236
233
  end
237
234
  }
238
- @default_table.import(chunk)
235
+ @default_table.import(chunk, self)
239
236
  end
240
237
  end
241
238
 
@@ -254,7 +251,7 @@ module Fluent::Plugin
254
251
  end
255
252
 
256
253
  def format_tag(tag)
257
- if @remove_tag_prefix
254
+ if tag && @remove_tag_prefix
258
255
  tag.gsub(@remove_tag_prefix, '')
259
256
  else
260
257
  tag
@@ -43,7 +43,8 @@ class SqlOutputTest < Test::Unit::TestCase
43
43
  password: "fluentd",
44
44
  schema_search_path: 'public',
45
45
  remove_tag_suffix: /^db/,
46
- enable_fallback: true
46
+ enable_fallback: true,
47
+ pool: 5
47
48
  }
48
49
  actual = {
49
50
  host: d.instance.host,
@@ -54,7 +55,8 @@ class SqlOutputTest < Test::Unit::TestCase
54
55
  password: d.instance.password,
55
56
  schema_search_path: d.instance.schema_search_path,
56
57
  remove_tag_suffix: d.instance.remove_tag_prefix,
57
- enable_fallback: d.instance.enable_fallback
58
+ enable_fallback: d.instance.enable_fallback,
59
+ pool: d.instance.pool
58
60
  }
59
61
  assert_equal(expected, actual)
60
62
  assert_empty(d.instance.tables)
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-sql
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.1.1
4
+ version: 2.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-05-09 00:00:00.000000000 Z
11
+ date: 2020-04-22 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -135,6 +135,7 @@ extensions: []
135
135
  extra_rdoc_files: []
136
136
  files:
137
137
  - ".travis.yml"
138
+ - ChangeLog
138
139
  - Gemfile
139
140
  - Gemfile.v0.12
140
141
  - README.md
@@ -166,8 +167,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
166
167
  - !ruby/object:Gem::Version
167
168
  version: '0'
168
169
  requirements: []
169
- rubyforge_project:
170
- rubygems_version: 2.6.14.1
170
+ rubygems_version: 3.1.2
171
171
  signing_key:
172
172
  specification_version: 4
173
173
  summary: SQL input/output plugin for Fluentd event collector