sm-fluent-plugin-sql 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 2fc5710a69b577a42c8c49f8b37584d41686ed33
4
+ data.tar.gz: 2b647c14bdafa9428ec5a90cc3e70bc9118ea74b
5
+ SHA512:
6
+ metadata.gz: 6c8ff37a4cc11bba7a6955d4f6cb93aa0c40cbf158b1590083ccbf45ffc5b0eb964eacbbb919cc9498545ed8401711f0c7e39a0f2d95999fd25a7335be806243
7
+ data.tar.gz: ffc73333259e9a7a941c7b321327ac9f8a1d8716b2b9480bcced6630526b922c70d791e288ca023bd2a885f1382a6922b24dc129347499865b4a42945bb210c8
@@ -0,0 +1,4 @@
1
+ sudo: false
2
+
3
+ before_install:
4
+ - gem install bundler
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source "http://rubygems.org"
2
+ gemspec
@@ -0,0 +1,148 @@
1
+ # SQL input plugin for [Fluentd](http://fluentd.org) event collector
2
+
3
+ ## Overview
4
+
5
+ This SQL plugin has two parts:
6
+
7
+ 1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
8
+ 2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
9
+
10
+ ## Installation
11
+
12
+ $ fluent-gem install fluent-plugin-sql --no-document
13
+ $ fluent-gem install pg --no-document # for postgresql
14
+
15
+ You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
16
+
17
+ We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
18
+
19
+ ## Input: How It Works
20
+
21
+ This plugin runs following SQL periodically:
22
+
23
+ SELECT * FROM *table* WHERE *update\_column* > *last\_update\_column\_value* ORDER BY *update_column* ASC LIMIT 500
24
+
25
+ What you need to configure is *update\_column*. The column should be an incremental column (such as AUTO\_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as `last_updated_at` column) so that this plugin reads the UPDATEd rows as well.
26
+ If you omit to set *update\_column* parameter, it uses primary key.
27
+
28
+ It stores last selected rows to a file (named *state\_file*) to not forget the last row when Fluentd restarts.
29
+
30
+ ## Input: Configuration
31
+
32
+ <source>
33
+ @type sql
34
+
35
+ host rdb_host
36
+ database rdb_database
37
+ adapter mysql2_or_postgresql_or_etc
38
+ username myusername
39
+ password mypassword
40
+
41
+ tag_prefix my.rdb # optional, but recommended
42
+
43
+ select_interval 60s # optional
44
+ select_limit 500 # optional
45
+
46
+ state_file /var/run/fluentd/sql_state
47
+
48
+ <table>
49
+ table table1
50
+ tag table1 # optional
51
+ update_column update_col1
52
+ time_column time_col2 # optional
53
+ </table>
54
+
55
+ <table>
56
+ table table2
57
+ tag table2 # optional
58
+ update_column updated_at
59
+ time_column updated_at # optional
60
+ </table>
61
+
62
+ # detects all tables instead of <table> sections
63
+ #all_tables
64
+ </source>
65
+
66
+ * **host** RDBMS host
67
+ * **port** RDBMS port
68
+ * **database** RDBMS database name
69
+ * **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
70
+ * **username** RDBMS login user name
71
+ * **password** RDBMS login password
72
+ * **tag_prefix** prefix of tags of events. actual tag will be this\_tag\_prefix.tables\_tag (optional)
73
+ * **select_interval** interval to run SQLs (optional)
74
+ * **select_limit** LIMIT of number of rows for each SQL (optional)
75
+ * **state_file** path to a file to store last rows
76
+ * **all_tables** reads all tables instead of configuring each tables in \<table\> sections
77
+
78
+ \<table\> sections:
79
+
80
+ * **tag** tag name of events (optional; default value is table name)
81
+ * **table** RDBM table name
82
+ * **update_column**: see above description
83
+ * **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
84
+ * **primary_key** (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.
85
+
86
+ ## Input: Limitation
87
+
88
+ You should make sure target tables have index (and/or partitions) on the *update\_column*. Otherwise SELECT causes full table scan and serious performance problem.
89
+
90
+ You can't replicate DELETEd rows.
91
+
92
+ ## Output: How It Works
93
+
94
+ This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.
95
+
96
+ ## Output: Configuration
97
+
98
+ <match my.rdb.*>
99
+ @type sql
100
+ host rdb_host
101
+ port 3306
102
+ database rdb_database
103
+ adapter mysql2_or_postgresql_or_etc
104
+ username myusername
105
+ password mypassword
106
+ socket path_to_socket
107
+ remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql
108
+
109
+ <table>
110
+ table table1
111
+ column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
112
+ # This is the default table because it has no "pattern" argument in <table>
113
+ # The logic is such that if all non-default <table> blocks
114
+ # do not match, the default one is chosen.
115
+ # The default table is required.
116
+ </table>
117
+
118
+ <table hello.*> # You can pass the same pattern you use in match statements.
119
+ table table2
120
+ # This is the non-default table. It is chosen if the tag matches the pattern
121
+ # AFTER remove_tag_prefix is applied to the incoming event. For example, if
122
+ # the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
123
+ # makes it "hello.world", which gets matched here because of "pattern hello.*".
124
+ </table>
125
+
126
+ <table hello.world>
127
+ table table3
128
+ # This is the second non-default table. You can have as many non-default tables
129
+ # as you wish. One caveat: non-default tables are matched top-to-bottom and
130
+ # the events go into the first table it matches to. Hence, this particular table
131
+ # never gets any data, since the above "hello.*" subsumes "hello.world".
132
+ </table>
133
+ </match>
134
+
135
+ * **host** RDBMS host
136
+ * **port** RDBMS port
137
+ * **database** RDBMS database name
138
+ * **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
139
+ * **username** RDBMS login user name
140
+ * **password** RDBMS login password
141
+ * **socket** RDBMS socket path
142
+ * **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
143
+
144
+ \<table\> sections:
145
+
146
+ * **table** RDBM table name
147
+ * **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
148
+ * **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
@@ -0,0 +1,14 @@
1
+
2
+ require 'bundler'
3
+ Bundler::GemHelper.install_tasks
4
+
5
+ #require 'rake/testtask'
6
+ #
7
+ #Rake::TestTask.new(:test) do |test|
8
+ # test.libs << 'lib' << 'test'
9
+ # test.test_files = FileList['test/*.rb']
10
+ # test.verbose = true
11
+ #end
12
+
13
+ task :default => [:build]
14
+
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 0.5.1
@@ -0,0 +1,24 @@
1
+ # encoding: utf-8
2
+ $:.push File.expand_path('../lib', __FILE__)
3
+
4
+ Gem::Specification.new do |gem|
5
+ gem.name = "sm-fluent-plugin-sql"
6
+ gem.description = "SQL input/output plugin for Fluentd event collector"
7
+ gem.homepage = "https://github.com/frsyuki/fluent-plugin-sql"
8
+ gem.summary = gem.description
9
+ gem.version = File.read("VERSION").strip
10
+ gem.authors = ["Sadayuki Furuhashi"]
11
+ gem.email = "frsyuki@gmail.com"
12
+ gem.has_rdoc = false
13
+ #gem.platform = Gem::Platform::RUBY
14
+ gem.files = `git ls-files`.split("\n")
15
+ gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
16
+ gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
17
+ gem.require_paths = ['lib']
18
+ gem.license = "Apache-2.0"
19
+
20
+ gem.add_dependency "fluentd", [">= 0.12.17", "< 2"]
21
+ gem.add_dependency 'activerecord', "~> 4.2"
22
+ gem.add_dependency 'activerecord-import', "~> 0.7"
23
+ gem.add_development_dependency "rake", ">= 0.9.2"
24
+ end
@@ -0,0 +1,312 @@
1
+ #
2
+ # Fluent
3
+ #
4
+ # Copyright (C) 2013 FURUHASHI Sadayuki
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+ #
18
+ module Fluent
19
+
20
+ require 'active_record'
21
+
22
+ class SQLInput < Input
23
+ Plugin.register_input('sql', self)
24
+
25
+ # For fluentd v0.12.16 or earlier
26
+ class << self
27
+ unless method_defined?(:desc)
28
+ def desc(description)
29
+ end
30
+ end
31
+ end
32
+
33
+ desc 'RDBMS host'
34
+ config_param :host, :string
35
+ desc 'RDBMS port'
36
+ config_param :port, :integer, :default => nil
37
+ desc 'RDBMS driver name.'
38
+ config_param :adapter, :string
39
+ desc 'RDBMS database name'
40
+ config_param :database, :string
41
+ desc 'RDBMS login user name'
42
+ config_param :username, :string, :default => nil
43
+ desc 'RDBMS login password'
44
+ config_param :password, :string, :default => nil, :secret => true
45
+ desc 'RDBMS socket path'
46
+ config_param :socket, :string, :default => nil
47
+
48
+ desc 'path to a file to store last rows'
49
+ config_param :state_file, :string, :default => nil
50
+ desc 'prefix of tags of events. actual tag will be this_tag_prefix.tables_tag (optional)'
51
+ config_param :tag_prefix, :string, :default => nil
52
+ desc 'interval to run SQLs (optional)'
53
+ config_param :select_interval, :time, :default => 60
54
+ desc 'limit of number of rows for each SQL(optional)'
55
+ config_param :select_limit, :time, :default => 500
56
+
57
+ unless method_defined?(:log)
58
+ define_method(:log) { $log }
59
+ end
60
+
61
+ class TableElement
62
+ include Configurable
63
+
64
+ config_param :table, :string
65
+ config_param :tag, :string, :default => nil
66
+ config_param :update_column, :string, :default => nil
67
+ config_param :time_column, :string, :default => nil
68
+ config_param :primary_key, :string, :default => nil
69
+
70
+ def configure(conf)
71
+ super
72
+ end
73
+
74
+ def init(tag_prefix, base_model, router)
75
+ @router = router
76
+ @tag = "#{tag_prefix}.#{@tag}" if tag_prefix
77
+
78
+ # creates a model for this table
79
+ table_name = @table
80
+ primary_key = @primary_key
81
+ @model = Class.new(base_model) do
82
+ self.table_name = table_name
83
+ self.inheritance_column = '_never_use_'
84
+ self.primary_key = primary_key if primary_key
85
+
86
+ #self.include_root_in_json = false
87
+
88
+ def read_attribute_for_serialization(n)
89
+ v = send(n)
90
+ if v.respond_to?(:to_msgpack)
91
+ v
92
+ else
93
+ v.to_s
94
+ end
95
+ end
96
+ end
97
+
98
+ # ActiveRecord requires model class to have a name.
99
+ class_name = table_name.singularize.camelize
100
+ base_model.const_set(class_name, @model)
101
+
102
+ # Sets model_name otherwise ActiveRecord causes errors
103
+ model_name = ActiveModel::Name.new(@model, nil, class_name)
104
+ @model.define_singleton_method(:model_name) { model_name }
105
+
106
+ # if update_column is not set, here uses primary key
107
+ unless @update_column
108
+ pk = @model.columns_hash[@model.primary_key]
109
+ unless pk
110
+ raise "Composite primary key is not supported. Set update_column parameter to <table> section."
111
+ end
112
+ @update_column = pk.name
113
+ end
114
+ end
115
+
116
+ # emits next records and returns the last record of emitted records
117
+ def emit_next_records(last_record, limit)
118
+ relation = @model
119
+ if last_record && last_update_value = last_record[@update_column]
120
+ relation = relation.where("#{@update_column} > ?", last_update_value)
121
+ end
122
+ relation = relation.order("#{@update_column} ASC")
123
+ relation = relation.limit(limit) if limit > 0
124
+
125
+ now = Engine.now
126
+
127
+ me = MultiEventStream.new
128
+ relation.each do |obj|
129
+ record = obj.serializable_hash rescue nil
130
+ if record
131
+ if @time_column && tv = obj.read_attribute(@time_column)
132
+ if tv.is_a?(Time)
133
+ time = tv.to_i
134
+ else
135
+ time = Time.parse(tv.to_s).to_i rescue now
136
+ end
137
+ else
138
+ time = now
139
+ end
140
+ me.add(time, record)
141
+ last_record = record
142
+ end
143
+ end
144
+
145
+ last_record = last_record.dup if last_record # some plugin rewrites record :(
146
+ @router.emit_stream(@tag, me)
147
+
148
+ return last_record
149
+ end
150
+ end
151
+
152
+ def configure(conf)
153
+ super
154
+
155
+ unless @state_file
156
+ $log.warn "'state_file PATH' parameter is not set to a 'sql' source."
157
+ $log.warn "this parameter is highly recommended to save the last rows to resume tailing."
158
+ end
159
+
160
+ @tables = conf.elements.select {|e|
161
+ e.name == 'table'
162
+ }.map {|e|
163
+ te = TableElement.new
164
+ te.configure(e)
165
+ te
166
+ }
167
+
168
+ if config['all_tables']
169
+ @all_tables = true
170
+ end
171
+ end
172
+
173
+ SKIP_TABLE_REGEXP = /\Aschema_migrations\Z/i
174
+
175
+ def start
176
+ @state_store = @state_file.nil? ? MemoryStateStore.new : StateStore.new(@state_file)
177
+
178
+ config = {
179
+ :adapter => @adapter,
180
+ :host => @host,
181
+ :port => @port,
182
+ :database => @database,
183
+ :username => @username,
184
+ :password => @password,
185
+ :socket => @socket,
186
+ }
187
+
188
+ # creates subclass of ActiveRecord::Base so that it can have different
189
+ # database configuration from ActiveRecord::Base.
190
+ @base_model = Class.new(ActiveRecord::Base) do
191
+ # base model doesn't have corresponding phisical table
192
+ self.abstract_class = true
193
+ end
194
+
195
+ # ActiveRecord requires the base_model to have a name. Here sets name
196
+ # of an anonymous class by assigning it to a constant. In Ruby, class has
197
+ # a name of a constant assigned first
198
+ SQLInput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
199
+
200
+ # Now base_model can have independent configuration from ActiveRecord::Base
201
+ @base_model.establish_connection(config)
202
+
203
+ if @all_tables
204
+ # get list of tables from the database
205
+ @tables = @base_model.connection.tables.map do |table_name|
206
+ if table_name.match(SKIP_TABLE_REGEXP)
207
+ # some tables such as "schema_migrations" should be ignored
208
+ nil
209
+ else
210
+ te = TableElement.new
211
+ te.configure({
212
+ 'table' => table_name,
213
+ 'tag' => table_name,
214
+ 'update_column' => nil,
215
+ })
216
+ te
217
+ end
218
+ end.compact
219
+ end
220
+
221
+ # ignore tables if TableElement#init failed
222
+ @tables.reject! do |te|
223
+ begin
224
+ te.init(@tag_prefix, @base_model, router)
225
+ log.info "Selecting '#{te.table}' table"
226
+ false
227
+ rescue => e
228
+ log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
229
+ log.warn_backtrace e.backtrace
230
+ true
231
+ end
232
+ end
233
+
234
+ @stop_flag = false
235
+ @thread = Thread.new(&method(:thread_main))
236
+ end
237
+
238
+ def shutdown
239
+ @stop_flag = true
240
+ $log.debug "Waiting for thread to finish"
241
+ @thread.join
242
+ end
243
+
244
+ def thread_main
245
+ until @stop_flag
246
+ sleep @select_interval
247
+
248
+ begin
249
+ conn = @base_model.connection
250
+ conn.active? || conn.reconnect!
251
+ rescue => e
252
+ log.warn "can't connect to database. Reconnect at next try"
253
+ next
254
+ end
255
+
256
+ @tables.each do |t|
257
+ begin
258
+ last_record = @state_store.last_records[t.table]
259
+ @state_store.last_records[t.table] = t.emit_next_records(last_record, @select_limit)
260
+ @state_store.update!
261
+ rescue => e
262
+ log.error "unexpected error", :error => e.message, :error_class => e.class
263
+ log.error_backtrace e.backtrace
264
+ end
265
+ end
266
+ end
267
+ end
268
+
269
+ class StateStore
270
+ def initialize(path)
271
+ require 'yaml'
272
+
273
+ @path = path
274
+ if File.exists?(@path)
275
+ @data = YAML.load_file(@path)
276
+ if @data == false || @data == []
277
+ # this happens if an users created an empty file accidentally
278
+ @data = {}
279
+ elsif !@data.is_a?(Hash)
280
+ raise "state_file on #{@path.inspect} is invalid"
281
+ end
282
+ else
283
+ @data = {}
284
+ end
285
+ end
286
+
287
+ def last_records
288
+ @data['last_records'] ||= {}
289
+ end
290
+
291
+ def update!
292
+ File.open(@path, 'w') {|f|
293
+ f.write YAML.dump(@data)
294
+ }
295
+ end
296
+ end
297
+
298
+ class MemoryStateStore
299
+ def initialize
300
+ @data = {}
301
+ end
302
+
303
+ def last_records
304
+ @data['last_records'] ||= {}
305
+ end
306
+
307
+ def update!
308
+ end
309
+ end
310
+ end
311
+
312
+ end
@@ -0,0 +1,251 @@
1
+ module Fluent
2
+ class SQLOutput < BufferedOutput
3
+ Plugin.register_output('sql', self)
4
+
5
+ include SetTimeKeyMixin
6
+ include SetTagKeyMixin
7
+
8
+ # For fluentd v0.12.16 or earlier
9
+ class << self
10
+ unless method_defined?(:desc)
11
+ def desc(description)
12
+ end
13
+ end
14
+ end
15
+
16
+ desc 'RDBMS host'
17
+ config_param :host, :string
18
+ desc 'RDBMS port'
19
+ config_param :port, :integer, :default => nil
20
+ desc 'RDBMS driver name.'
21
+ config_param :adapter, :string
22
+ desc 'RDBMS login user name'
23
+ config_param :username, :string, :default => nil
24
+ desc 'RDBMS login password'
25
+ config_param :password, :string, :default => nil, :secret => true
26
+ desc 'RDBMS database name'
27
+ config_param :database, :string
28
+ desc 'RDBMS socket path'
29
+ config_param :socket, :string, :default => nil
30
+ desc 'remove the given prefix from the events'
31
+ config_param :remove_tag_prefix, :string, :default => nil
32
+
33
+ attr_accessor :tables
34
+
35
+ unless method_defined?(:log)
36
+ define_method(:log) { $log }
37
+ end
38
+
39
+ # TODO: Merge SQLInput's TableElement
40
+ class TableElement
41
+ include Configurable
42
+
43
+ config_param :table, :string
44
+ config_param :column_mapping, :string
45
+ config_param :num_retries, :integer, :default => 5
46
+
47
+ attr_reader :model
48
+ attr_reader :pattern
49
+
50
+ def initialize(pattern, log)
51
+ super()
52
+ @pattern = MatchPattern.create(pattern)
53
+ @log = log
54
+ end
55
+
56
+ def configure(conf)
57
+ super
58
+
59
+ @mapping = parse_column_mapping(@column_mapping)
60
+ @format_proc = Proc.new { |record|
61
+ new_record = {}
62
+ @mapping.each { |k, c|
63
+ new_record[c] = record[k]
64
+ }
65
+ new_record
66
+ }
67
+ end
68
+
69
+ def init(base_model)
70
+ # See SQLInput for more details of following code
71
+ table_name = @table
72
+ @model = Class.new(base_model) do
73
+ self.table_name = table_name
74
+ self.inheritance_column = '_never_use_output_'
75
+ end
76
+
77
+ class_name = table_name.singularize.camelize
78
+ base_model.const_set(class_name, @model)
79
+ model_name = ActiveModel::Name.new(@model, nil, class_name)
80
+ @model.define_singleton_method(:model_name) { model_name }
81
+
82
+ # TODO: check column_names and table schema
83
+ # @model.column_names
84
+ end
85
+
86
+ def import(chunk)
87
+ records = []
88
+ chunk.msgpack_each { |tag, time, data|
89
+ begin
90
+ # format process should be moved to emit / format after supports error stream.
91
+ records << @model.new(@format_proc.call(data))
92
+ rescue => e
93
+ args = {:error => e.message, :error_class => e.class, :table => @table, :record => Yajl.dump(data)}
94
+ @log.warn "Failed to create the model. Ignore a record:", args
95
+ end
96
+ }
97
+ begin
98
+ @model.import(records)
99
+ rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
100
+ # ignore other exceptions to use Fluentd retry mechanizm
101
+ @log.warn "Got deterministic error. Fallback to one-by-one import", :error => e.message, :error_class => e.class
102
+ one_by_one_import(records)
103
+ end
104
+ end
105
+
106
+ def one_by_one_import(records)
107
+ records.each { |record|
108
+ retries = 0
109
+ begin
110
+ @model.import([record])
111
+ rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
112
+ @log.error "Got deterministic error again. Dump a record", :error => e.message, :error_class => e.class, :record => record
113
+ rescue => e
114
+ retries += 1
115
+ if retries > @num_retries
116
+ @log.error "Can't recover undeterministic error. Dump a record", :error => e.message, :error_class => e.class, :record => record
117
+ next
118
+ end
119
+
120
+ @log.warn "Failed to import a record: retry number = #{retries}", :error => e.message, :error_class => e.class
121
+ sleep 0.5
122
+ retry
123
+ end
124
+ }
125
+ end
126
+
127
+ private
128
+
129
+ def parse_column_mapping(column_mapping_conf)
130
+ mapping = {}
131
+ column_mapping_conf.split(',').each { |column_map|
132
+ key, column = column_map.strip.split(':', 2)
133
+ column = key if column.nil?
134
+ mapping[key] = column
135
+ }
136
+ mapping
137
+ end
138
+ end
139
+
140
+ def initialize
141
+ super
142
+ require 'active_record'
143
+ require 'activerecord-import'
144
+ end
145
+
146
+ def configure(conf)
147
+ super
148
+
149
+ if remove_tag_prefix = conf['remove_tag_prefix']
150
+ @remove_tag_prefix = Regexp.new('^' + Regexp.escape(remove_tag_prefix))
151
+ end
152
+
153
+ @tables = []
154
+ @default_table = nil
155
+ conf.elements.select { |e|
156
+ e.name == 'table'
157
+ }.each { |e|
158
+ te = TableElement.new(e.arg, log)
159
+ te.configure(e)
160
+ if e.arg.empty?
161
+ $log.warn "Detect duplicate default table definition" if @default_table
162
+ @default_table = te
163
+ else
164
+ @tables << te
165
+ end
166
+ }
167
+ @only_default = @tables.empty?
168
+
169
+ if @default_table.nil?
170
+ raise ConfigError, "There is no default table. <table> is required in sql output"
171
+ end
172
+ end
173
+
174
+ def start
175
+ super
176
+
177
+ config = {
178
+ :adapter => @adapter,
179
+ :host => @host,
180
+ :port => @port,
181
+ :database => @database,
182
+ :username => @username,
183
+ :password => @password,
184
+ :socket => @socket,
185
+ }
186
+
187
+ @base_model = Class.new(ActiveRecord::Base) do
188
+ self.abstract_class = true
189
+ end
190
+
191
+ SQLOutput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
192
+ @base_model.establish_connection(config)
193
+
194
+ # ignore tables if TableElement#init failed
195
+ @tables.reject! do |te|
196
+ init_table(te, @base_model)
197
+ end
198
+ init_table(@default_table, @base_model)
199
+ end
200
+
201
+ def shutdown
202
+ super
203
+ end
204
+
205
+ def emit(tag, es, chain)
206
+ if @only_default
207
+ super(tag, es, chain)
208
+ else
209
+ super(tag, es, chain, format_tag(tag))
210
+ end
211
+ end
212
+
213
+ def format(tag, time, record)
214
+ [tag, time, record].to_msgpack
215
+ end
216
+
217
+ def write(chunk)
218
+ conn = @base_model.connection
219
+ conn.active? || conn.reconnect!
220
+
221
+ @tables.each { |table|
222
+ if table.pattern.match(chunk.key)
223
+ return table.import(chunk)
224
+ end
225
+ }
226
+ @default_table.import(chunk)
227
+ end
228
+
229
+ private
230
+
231
+ def init_table(te, base_model)
232
+ begin
233
+ te.init(base_model)
234
+ log.info "Selecting '#{te.table}' table"
235
+ false
236
+ rescue => e
237
+ log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
238
+ log.warn_backtrace e.backtrace
239
+ true
240
+ end
241
+ end
242
+
243
+ def format_tag(tag)
244
+ if @remove_tag_prefix
245
+ tag.gsub(@remove_tag_prefix, '')
246
+ else
247
+ tag
248
+ end
249
+ end
250
+ end
251
+ end
metadata ADDED
@@ -0,0 +1,113 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: sm-fluent-plugin-sql
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.5.1
5
+ platform: ruby
6
+ authors:
7
+ - Sadayuki Furuhashi
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-03-16 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: fluentd
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: 0.12.17
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '2'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: 0.12.17
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '2'
33
+ - !ruby/object:Gem::Dependency
34
+ name: activerecord
35
+ requirement: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: '4.2'
40
+ type: :runtime
41
+ prerelease: false
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - "~>"
45
+ - !ruby/object:Gem::Version
46
+ version: '4.2'
47
+ - !ruby/object:Gem::Dependency
48
+ name: activerecord-import
49
+ requirement: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - "~>"
52
+ - !ruby/object:Gem::Version
53
+ version: '0.7'
54
+ type: :runtime
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - "~>"
59
+ - !ruby/object:Gem::Version
60
+ version: '0.7'
61
+ - !ruby/object:Gem::Dependency
62
+ name: rake
63
+ requirement: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - ">="
66
+ - !ruby/object:Gem::Version
67
+ version: 0.9.2
68
+ type: :development
69
+ prerelease: false
70
+ version_requirements: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ">="
73
+ - !ruby/object:Gem::Version
74
+ version: 0.9.2
75
+ description: SQL input/output plugin for Fluentd event collector
76
+ email: frsyuki@gmail.com
77
+ executables: []
78
+ extensions: []
79
+ extra_rdoc_files: []
80
+ files:
81
+ - ".travis.yml"
82
+ - Gemfile
83
+ - README.md
84
+ - Rakefile
85
+ - VERSION
86
+ - fluent-plugin-sql.gemspec
87
+ - lib/fluent/plugin/in_sql.rb
88
+ - lib/fluent/plugin/out_sql.rb
89
+ homepage: https://github.com/frsyuki/fluent-plugin-sql
90
+ licenses:
91
+ - Apache-2.0
92
+ metadata: {}
93
+ post_install_message:
94
+ rdoc_options: []
95
+ require_paths:
96
+ - lib
97
+ required_ruby_version: !ruby/object:Gem::Requirement
98
+ requirements:
99
+ - - ">="
100
+ - !ruby/object:Gem::Version
101
+ version: '0'
102
+ required_rubygems_version: !ruby/object:Gem::Requirement
103
+ requirements:
104
+ - - ">="
105
+ - !ruby/object:Gem::Version
106
+ version: '0'
107
+ requirements: []
108
+ rubyforge_project:
109
+ rubygems_version: 2.4.8
110
+ signing_key:
111
+ specification_version: 4
112
+ summary: SQL input/output plugin for Fluentd event collector
113
+ test_files: []