sm-fluent-plugin-sql 0.5.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 2fc5710a69b577a42c8c49f8b37584d41686ed33
4
+ data.tar.gz: 2b647c14bdafa9428ec5a90cc3e70bc9118ea74b
5
+ SHA512:
6
+ metadata.gz: 6c8ff37a4cc11bba7a6955d4f6cb93aa0c40cbf158b1590083ccbf45ffc5b0eb964eacbbb919cc9498545ed8401711f0c7e39a0f2d95999fd25a7335be806243
7
+ data.tar.gz: ffc73333259e9a7a941c7b321327ac9f8a1d8716b2b9480bcced6630526b922c70d791e288ca023bd2a885f1382a6922b24dc129347499865b4a42945bb210c8
@@ -0,0 +1,4 @@
1
+ sudo: false
2
+
3
+ before_install:
4
+ - gem install bundler
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source "http://rubygems.org"
2
+ gemspec
@@ -0,0 +1,148 @@
1
+ # SQL input plugin for [Fluentd](http://fluentd.org) event collector
2
+
3
+ ## Overview
4
+
5
+ This SQL plugin has two parts:
6
+
7
+ 1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
8
+ 2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
9
+
10
+ ## Installation
11
+
12
+ $ fluent-gem install fluent-plugin-sql --no-document
13
+ $ fluent-gem install pg --no-document # for postgresql
14
+
15
+ You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
16
+
17
+ We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
18
+
19
+ ## Input: How It Works
20
+
21
+ This plugin runs following SQL periodically:
22
+
23
+ SELECT * FROM *table* WHERE *update\_column* > *last\_update\_column\_value* ORDER BY *update_column* ASC LIMIT 500
24
+
25
+ What you need to configure is *update\_column*. The column should be an incremental column (such as AUTO\_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as `last_updated_at` column) so that this plugin reads the UPDATEd rows as well.
26
+ If you omit to set *update\_column* parameter, it uses primary key.
27
+
28
+ It stores last selected rows to a file (named *state\_file*) to not forget the last row when Fluentd restarts.
29
+
30
+ ## Input: Configuration
31
+
32
+ <source>
33
+ @type sql
34
+
35
+ host rdb_host
36
+ database rdb_database
37
+ adapter mysql2_or_postgresql_or_etc
38
+ username myusername
39
+ password mypassword
40
+
41
+ tag_prefix my.rdb # optional, but recommended
42
+
43
+ select_interval 60s # optional
44
+ select_limit 500 # optional
45
+
46
+ state_file /var/run/fluentd/sql_state
47
+
48
+ <table>
49
+ table table1
50
+ tag table1 # optional
51
+ update_column update_col1
52
+ time_column time_col2 # optional
53
+ </table>
54
+
55
+ <table>
56
+ table table2
57
+ tag table2 # optional
58
+ update_column updated_at
59
+ time_column updated_at # optional
60
+ </table>
61
+
62
+ # detects all tables instead of <table> sections
63
+ #all_tables
64
+ </source>
65
+
66
+ * **host** RDBMS host
67
+ * **port** RDBMS port
68
+ * **database** RDBMS database name
69
+ * **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
70
+ * **username** RDBMS login user name
71
+ * **password** RDBMS login password
72
+ * **tag_prefix** prefix of tags of events. actual tag will be this\_tag\_prefix.tables\_tag (optional)
73
+ * **select_interval** interval to run SQLs (optional)
74
+ * **select_limit** LIMIT of number of rows for each SQL (optional)
75
+ * **state_file** path to a file to store last rows
76
+ * **all_tables** reads all tables instead of configuring each tables in \<table\> sections
77
+
78
+ \<table\> sections:
79
+
80
+ * **tag** tag name of events (optional; default value is table name)
81
+ * **table** RDBM table name
82
+ * **update_column**: see above description
83
+ * **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
84
+ * **primary_key** (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.
85
+
86
+ ## Input: Limitation
87
+
88
+ You should make sure target tables have index (and/or partitions) on the *update\_column*. Otherwise SELECT causes full table scan and serious performance problem.
89
+
90
+ You can't replicate DELETEd rows.
91
+
92
+ ## Output: How It Works
93
+
94
+ This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.
95
+
96
+ ## Output: Configuration
97
+
98
+ <match my.rdb.*>
99
+ @type sql
100
+ host rdb_host
101
+ port 3306
102
+ database rdb_database
103
+ adapter mysql2_or_postgresql_or_etc
104
+ username myusername
105
+ password mypassword
106
+ socket path_to_socket
107
+ remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql
108
+
109
+ <table>
110
+ table table1
111
+ column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
112
+ # This is the default table because it has no "pattern" argument in <table>
113
+ # The logic is such that if all non-default <table> blocks
114
+ # do not match, the default one is chosen.
115
+ # The default table is required.
116
+ </table>
117
+
118
+ <table hello.*> # You can pass the same pattern you use in match statements.
119
+ table table2
120
+ # This is the non-default table. It is chosen if the tag matches the pattern
121
+ # AFTER remove_tag_prefix is applied to the incoming event. For example, if
122
+ # the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
123
+ # makes it "hello.world", which gets matched here because of "pattern hello.*".
124
+ </table>
125
+
126
+ <table hello.world>
127
+ table table3
128
+ # This is the second non-default table. You can have as many non-default tables
129
+ # as you wish. One caveat: non-default tables are matched top-to-bottom and
130
+ # the events go into the first table it matches to. Hence, this particular table
131
+ # never gets any data, since the above "hello.*" subsumes "hello.world".
132
+ </table>
133
+ </match>
134
+
135
+ * **host** RDBMS host
136
+ * **port** RDBMS port
137
+ * **database** RDBMS database name
138
+ * **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
139
+ * **username** RDBMS login user name
140
+ * **password** RDBMS login password
141
+ * **socket** RDBMS socket path
142
+ * **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
143
+
144
+ \<table\> sections:
145
+
146
+ * **table** RDBM table name
147
+ * **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
148
+ * **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
@@ -0,0 +1,14 @@
1
+
2
+ require 'bundler'
3
+ Bundler::GemHelper.install_tasks
4
+
5
+ #require 'rake/testtask'
6
+ #
7
+ #Rake::TestTask.new(:test) do |test|
8
+ # test.libs << 'lib' << 'test'
9
+ # test.test_files = FileList['test/*.rb']
10
+ # test.verbose = true
11
+ #end
12
+
13
+ task :default => [:build]
14
+
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 0.5.1
@@ -0,0 +1,24 @@
1
+ # encoding: utf-8
2
+ $:.push File.expand_path('../lib', __FILE__)
3
+
4
+ Gem::Specification.new do |gem|
5
+ gem.name = "sm-fluent-plugin-sql"
6
+ gem.description = "SQL input/output plugin for Fluentd event collector"
7
+ gem.homepage = "https://github.com/frsyuki/fluent-plugin-sql"
8
+ gem.summary = gem.description
9
+ gem.version = File.read("VERSION").strip
10
+ gem.authors = ["Sadayuki Furuhashi"]
11
+ gem.email = "frsyuki@gmail.com"
12
+ gem.has_rdoc = false
13
+ #gem.platform = Gem::Platform::RUBY
14
+ gem.files = `git ls-files`.split("\n")
15
+ gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
16
+ gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
17
+ gem.require_paths = ['lib']
18
+ gem.license = "Apache-2.0"
19
+
20
+ gem.add_dependency "fluentd", [">= 0.12.17", "< 2"]
21
+ gem.add_dependency 'activerecord', "~> 4.2"
22
+ gem.add_dependency 'activerecord-import', "~> 0.7"
23
+ gem.add_development_dependency "rake", ">= 0.9.2"
24
+ end
@@ -0,0 +1,312 @@
1
+ #
2
+ # Fluent
3
+ #
4
+ # Copyright (C) 2013 FURUHASHI Sadayuki
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+ #
18
+ module Fluent
19
+
20
+ require 'active_record'
21
+
22
+ class SQLInput < Input
23
+ Plugin.register_input('sql', self)
24
+
25
+ # For fluentd v0.12.16 or earlier
26
+ class << self
27
+ unless method_defined?(:desc)
28
+ def desc(description)
29
+ end
30
+ end
31
+ end
32
+
33
+ desc 'RDBMS host'
34
+ config_param :host, :string
35
+ desc 'RDBMS port'
36
+ config_param :port, :integer, :default => nil
37
+ desc 'RDBMS driver name.'
38
+ config_param :adapter, :string
39
+ desc 'RDBMS database name'
40
+ config_param :database, :string
41
+ desc 'RDBMS login user name'
42
+ config_param :username, :string, :default => nil
43
+ desc 'RDBMS login password'
44
+ config_param :password, :string, :default => nil, :secret => true
45
+ desc 'RDBMS socket path'
46
+ config_param :socket, :string, :default => nil
47
+
48
+ desc 'path to a file to store last rows'
49
+ config_param :state_file, :string, :default => nil
50
+ desc 'prefix of tags of events. actual tag will be this_tag_prefix.tables_tag (optional)'
51
+ config_param :tag_prefix, :string, :default => nil
52
+ desc 'interval to run SQLs (optional)'
53
+ config_param :select_interval, :time, :default => 60
54
+ desc 'limit of number of rows for each SQL(optional)'
55
+ config_param :select_limit, :time, :default => 500
56
+
57
+ unless method_defined?(:log)
58
+ define_method(:log) { $log }
59
+ end
60
+
61
+ class TableElement
62
+ include Configurable
63
+
64
+ config_param :table, :string
65
+ config_param :tag, :string, :default => nil
66
+ config_param :update_column, :string, :default => nil
67
+ config_param :time_column, :string, :default => nil
68
+ config_param :primary_key, :string, :default => nil
69
+
70
+ def configure(conf)
71
+ super
72
+ end
73
+
74
+ def init(tag_prefix, base_model, router)
75
+ @router = router
76
+ @tag = "#{tag_prefix}.#{@tag}" if tag_prefix
77
+
78
+ # creates a model for this table
79
+ table_name = @table
80
+ primary_key = @primary_key
81
+ @model = Class.new(base_model) do
82
+ self.table_name = table_name
83
+ self.inheritance_column = '_never_use_'
84
+ self.primary_key = primary_key if primary_key
85
+
86
+ #self.include_root_in_json = false
87
+
88
+ def read_attribute_for_serialization(n)
89
+ v = send(n)
90
+ if v.respond_to?(:to_msgpack)
91
+ v
92
+ else
93
+ v.to_s
94
+ end
95
+ end
96
+ end
97
+
98
+ # ActiveRecord requires model class to have a name.
99
+ class_name = table_name.singularize.camelize
100
+ base_model.const_set(class_name, @model)
101
+
102
+ # Sets model_name otherwise ActiveRecord causes errors
103
+ model_name = ActiveModel::Name.new(@model, nil, class_name)
104
+ @model.define_singleton_method(:model_name) { model_name }
105
+
106
+ # if update_column is not set, here uses primary key
107
+ unless @update_column
108
+ pk = @model.columns_hash[@model.primary_key]
109
+ unless pk
110
+ raise "Composite primary key is not supported. Set update_column parameter to <table> section."
111
+ end
112
+ @update_column = pk.name
113
+ end
114
+ end
115
+
116
+ # emits next records and returns the last record of emitted records
117
+ def emit_next_records(last_record, limit)
118
+ relation = @model
119
+ if last_record && last_update_value = last_record[@update_column]
120
+ relation = relation.where("#{@update_column} > ?", last_update_value)
121
+ end
122
+ relation = relation.order("#{@update_column} ASC")
123
+ relation = relation.limit(limit) if limit > 0
124
+
125
+ now = Engine.now
126
+
127
+ me = MultiEventStream.new
128
+ relation.each do |obj|
129
+ record = obj.serializable_hash rescue nil
130
+ if record
131
+ if @time_column && tv = obj.read_attribute(@time_column)
132
+ if tv.is_a?(Time)
133
+ time = tv.to_i
134
+ else
135
+ time = Time.parse(tv.to_s).to_i rescue now
136
+ end
137
+ else
138
+ time = now
139
+ end
140
+ me.add(time, record)
141
+ last_record = record
142
+ end
143
+ end
144
+
145
+ last_record = last_record.dup if last_record # some plugin rewrites record :(
146
+ @router.emit_stream(@tag, me)
147
+
148
+ return last_record
149
+ end
150
+ end
151
+
152
+ def configure(conf)
153
+ super
154
+
155
+ unless @state_file
156
+ $log.warn "'state_file PATH' parameter is not set to a 'sql' source."
157
+ $log.warn "this parameter is highly recommended to save the last rows to resume tailing."
158
+ end
159
+
160
+ @tables = conf.elements.select {|e|
161
+ e.name == 'table'
162
+ }.map {|e|
163
+ te = TableElement.new
164
+ te.configure(e)
165
+ te
166
+ }
167
+
168
+ if config['all_tables']
169
+ @all_tables = true
170
+ end
171
+ end
172
+
173
+ SKIP_TABLE_REGEXP = /\Aschema_migrations\Z/i
174
+
175
+ def start
176
+ @state_store = @state_file.nil? ? MemoryStateStore.new : StateStore.new(@state_file)
177
+
178
+ config = {
179
+ :adapter => @adapter,
180
+ :host => @host,
181
+ :port => @port,
182
+ :database => @database,
183
+ :username => @username,
184
+ :password => @password,
185
+ :socket => @socket,
186
+ }
187
+
188
+ # creates subclass of ActiveRecord::Base so that it can have different
189
+ # database configuration from ActiveRecord::Base.
190
+ @base_model = Class.new(ActiveRecord::Base) do
191
+ # base model doesn't have corresponding phisical table
192
+ self.abstract_class = true
193
+ end
194
+
195
+ # ActiveRecord requires the base_model to have a name. Here sets name
196
+ # of an anonymous class by assigning it to a constant. In Ruby, class has
197
+ # a name of a constant assigned first
198
+ SQLInput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
199
+
200
+ # Now base_model can have independent configuration from ActiveRecord::Base
201
+ @base_model.establish_connection(config)
202
+
203
+ if @all_tables
204
+ # get list of tables from the database
205
+ @tables = @base_model.connection.tables.map do |table_name|
206
+ if table_name.match(SKIP_TABLE_REGEXP)
207
+ # some tables such as "schema_migrations" should be ignored
208
+ nil
209
+ else
210
+ te = TableElement.new
211
+ te.configure({
212
+ 'table' => table_name,
213
+ 'tag' => table_name,
214
+ 'update_column' => nil,
215
+ })
216
+ te
217
+ end
218
+ end.compact
219
+ end
220
+
221
+ # ignore tables if TableElement#init failed
222
+ @tables.reject! do |te|
223
+ begin
224
+ te.init(@tag_prefix, @base_model, router)
225
+ log.info "Selecting '#{te.table}' table"
226
+ false
227
+ rescue => e
228
+ log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
229
+ log.warn_backtrace e.backtrace
230
+ true
231
+ end
232
+ end
233
+
234
+ @stop_flag = false
235
+ @thread = Thread.new(&method(:thread_main))
236
+ end
237
+
238
+ def shutdown
239
+ @stop_flag = true
240
+ $log.debug "Waiting for thread to finish"
241
+ @thread.join
242
+ end
243
+
244
+ def thread_main
245
+ until @stop_flag
246
+ sleep @select_interval
247
+
248
+ begin
249
+ conn = @base_model.connection
250
+ conn.active? || conn.reconnect!
251
+ rescue => e
252
+ log.warn "can't connect to database. Reconnect at next try"
253
+ next
254
+ end
255
+
256
+ @tables.each do |t|
257
+ begin
258
+ last_record = @state_store.last_records[t.table]
259
+ @state_store.last_records[t.table] = t.emit_next_records(last_record, @select_limit)
260
+ @state_store.update!
261
+ rescue => e
262
+ log.error "unexpected error", :error => e.message, :error_class => e.class
263
+ log.error_backtrace e.backtrace
264
+ end
265
+ end
266
+ end
267
+ end
268
+
269
+ class StateStore
270
+ def initialize(path)
271
+ require 'yaml'
272
+
273
+ @path = path
274
+ if File.exists?(@path)
275
+ @data = YAML.load_file(@path)
276
+ if @data == false || @data == []
277
+ # this happens if an users created an empty file accidentally
278
+ @data = {}
279
+ elsif !@data.is_a?(Hash)
280
+ raise "state_file on #{@path.inspect} is invalid"
281
+ end
282
+ else
283
+ @data = {}
284
+ end
285
+ end
286
+
287
+ def last_records
288
+ @data['last_records'] ||= {}
289
+ end
290
+
291
+ def update!
292
+ File.open(@path, 'w') {|f|
293
+ f.write YAML.dump(@data)
294
+ }
295
+ end
296
+ end
297
+
298
+ class MemoryStateStore
299
+ def initialize
300
+ @data = {}
301
+ end
302
+
303
+ def last_records
304
+ @data['last_records'] ||= {}
305
+ end
306
+
307
+ def update!
308
+ end
309
+ end
310
+ end
311
+
312
+ end
@@ -0,0 +1,251 @@
1
+ module Fluent
2
+ class SQLOutput < BufferedOutput
3
+ Plugin.register_output('sql', self)
4
+
5
+ include SetTimeKeyMixin
6
+ include SetTagKeyMixin
7
+
8
+ # For fluentd v0.12.16 or earlier
9
+ class << self
10
+ unless method_defined?(:desc)
11
+ def desc(description)
12
+ end
13
+ end
14
+ end
15
+
16
+ desc 'RDBMS host'
17
+ config_param :host, :string
18
+ desc 'RDBMS port'
19
+ config_param :port, :integer, :default => nil
20
+ desc 'RDBMS driver name.'
21
+ config_param :adapter, :string
22
+ desc 'RDBMS login user name'
23
+ config_param :username, :string, :default => nil
24
+ desc 'RDBMS login password'
25
+ config_param :password, :string, :default => nil, :secret => true
26
+ desc 'RDBMS database name'
27
+ config_param :database, :string
28
+ desc 'RDBMS socket path'
29
+ config_param :socket, :string, :default => nil
30
+ desc 'remove the given prefix from the events'
31
+ config_param :remove_tag_prefix, :string, :default => nil
32
+
33
+ attr_accessor :tables
34
+
35
+ unless method_defined?(:log)
36
+ define_method(:log) { $log }
37
+ end
38
+
39
+ # TODO: Merge SQLInput's TableElement
40
+ class TableElement
41
+ include Configurable
42
+
43
+ config_param :table, :string
44
+ config_param :column_mapping, :string
45
+ config_param :num_retries, :integer, :default => 5
46
+
47
+ attr_reader :model
48
+ attr_reader :pattern
49
+
50
+ def initialize(pattern, log)
51
+ super()
52
+ @pattern = MatchPattern.create(pattern)
53
+ @log = log
54
+ end
55
+
56
+ def configure(conf)
57
+ super
58
+
59
+ @mapping = parse_column_mapping(@column_mapping)
60
+ @format_proc = Proc.new { |record|
61
+ new_record = {}
62
+ @mapping.each { |k, c|
63
+ new_record[c] = record[k]
64
+ }
65
+ new_record
66
+ }
67
+ end
68
+
69
+ def init(base_model)
70
+ # See SQLInput for more details of following code
71
+ table_name = @table
72
+ @model = Class.new(base_model) do
73
+ self.table_name = table_name
74
+ self.inheritance_column = '_never_use_output_'
75
+ end
76
+
77
+ class_name = table_name.singularize.camelize
78
+ base_model.const_set(class_name, @model)
79
+ model_name = ActiveModel::Name.new(@model, nil, class_name)
80
+ @model.define_singleton_method(:model_name) { model_name }
81
+
82
+ # TODO: check column_names and table schema
83
+ # @model.column_names
84
+ end
85
+
86
+ def import(chunk)
87
+ records = []
88
+ chunk.msgpack_each { |tag, time, data|
89
+ begin
90
+ # format process should be moved to emit / format after supports error stream.
91
+ records << @model.new(@format_proc.call(data))
92
+ rescue => e
93
+ args = {:error => e.message, :error_class => e.class, :table => @table, :record => Yajl.dump(data)}
94
+ @log.warn "Failed to create the model. Ignore a record:", args
95
+ end
96
+ }
97
+ begin
98
+ @model.import(records)
99
+ rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
100
+ # ignore other exceptions to use Fluentd retry mechanizm
101
+ @log.warn "Got deterministic error. Fallback to one-by-one import", :error => e.message, :error_class => e.class
102
+ one_by_one_import(records)
103
+ end
104
+ end
105
+
106
+ def one_by_one_import(records)
107
+ records.each { |record|
108
+ retries = 0
109
+ begin
110
+ @model.import([record])
111
+ rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
112
+ @log.error "Got deterministic error again. Dump a record", :error => e.message, :error_class => e.class, :record => record
113
+ rescue => e
114
+ retries += 1
115
+ if retries > @num_retries
116
+ @log.error "Can't recover undeterministic error. Dump a record", :error => e.message, :error_class => e.class, :record => record
117
+ next
118
+ end
119
+
120
+ @log.warn "Failed to import a record: retry number = #{retries}", :error => e.message, :error_class => e.class
121
+ sleep 0.5
122
+ retry
123
+ end
124
+ }
125
+ end
126
+
127
+ private
128
+
129
+ def parse_column_mapping(column_mapping_conf)
130
+ mapping = {}
131
+ column_mapping_conf.split(',').each { |column_map|
132
+ key, column = column_map.strip.split(':', 2)
133
+ column = key if column.nil?
134
+ mapping[key] = column
135
+ }
136
+ mapping
137
+ end
138
+ end
139
+
140
+ def initialize
141
+ super
142
+ require 'active_record'
143
+ require 'activerecord-import'
144
+ end
145
+
146
+ def configure(conf)
147
+ super
148
+
149
+ if remove_tag_prefix = conf['remove_tag_prefix']
150
+ @remove_tag_prefix = Regexp.new('^' + Regexp.escape(remove_tag_prefix))
151
+ end
152
+
153
+ @tables = []
154
+ @default_table = nil
155
+ conf.elements.select { |e|
156
+ e.name == 'table'
157
+ }.each { |e|
158
+ te = TableElement.new(e.arg, log)
159
+ te.configure(e)
160
+ if e.arg.empty?
161
+ $log.warn "Detect duplicate default table definition" if @default_table
162
+ @default_table = te
163
+ else
164
+ @tables << te
165
+ end
166
+ }
167
+ @only_default = @tables.empty?
168
+
169
+ if @default_table.nil?
170
+ raise ConfigError, "There is no default table. <table> is required in sql output"
171
+ end
172
+ end
173
+
174
+ def start
175
+ super
176
+
177
+ config = {
178
+ :adapter => @adapter,
179
+ :host => @host,
180
+ :port => @port,
181
+ :database => @database,
182
+ :username => @username,
183
+ :password => @password,
184
+ :socket => @socket,
185
+ }
186
+
187
+ @base_model = Class.new(ActiveRecord::Base) do
188
+ self.abstract_class = true
189
+ end
190
+
191
+ SQLOutput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
192
+ @base_model.establish_connection(config)
193
+
194
+ # ignore tables if TableElement#init failed
195
+ @tables.reject! do |te|
196
+ init_table(te, @base_model)
197
+ end
198
+ init_table(@default_table, @base_model)
199
+ end
200
+
201
+ def shutdown
202
+ super
203
+ end
204
+
205
+ def emit(tag, es, chain)
206
+ if @only_default
207
+ super(tag, es, chain)
208
+ else
209
+ super(tag, es, chain, format_tag(tag))
210
+ end
211
+ end
212
+
213
+ def format(tag, time, record)
214
+ [tag, time, record].to_msgpack
215
+ end
216
+
217
+ def write(chunk)
218
+ conn = @base_model.connection
219
+ conn.active? || conn.reconnect!
220
+
221
+ @tables.each { |table|
222
+ if table.pattern.match(chunk.key)
223
+ return table.import(chunk)
224
+ end
225
+ }
226
+ @default_table.import(chunk)
227
+ end
228
+
229
+ private
230
+
231
+ def init_table(te, base_model)
232
+ begin
233
+ te.init(base_model)
234
+ log.info "Selecting '#{te.table}' table"
235
+ false
236
+ rescue => e
237
+ log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
238
+ log.warn_backtrace e.backtrace
239
+ true
240
+ end
241
+ end
242
+
243
+ def format_tag(tag)
244
+ if @remove_tag_prefix
245
+ tag.gsub(@remove_tag_prefix, '')
246
+ else
247
+ tag
248
+ end
249
+ end
250
+ end
251
+ end
metadata ADDED
@@ -0,0 +1,113 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: sm-fluent-plugin-sql
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.5.1
5
+ platform: ruby
6
+ authors:
7
+ - Sadayuki Furuhashi
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2016-03-16 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: fluentd
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: 0.12.17
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '2'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: 0.12.17
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '2'
33
+ - !ruby/object:Gem::Dependency
34
+ name: activerecord
35
+ requirement: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: '4.2'
40
+ type: :runtime
41
+ prerelease: false
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - "~>"
45
+ - !ruby/object:Gem::Version
46
+ version: '4.2'
47
+ - !ruby/object:Gem::Dependency
48
+ name: activerecord-import
49
+ requirement: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - "~>"
52
+ - !ruby/object:Gem::Version
53
+ version: '0.7'
54
+ type: :runtime
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - "~>"
59
+ - !ruby/object:Gem::Version
60
+ version: '0.7'
61
+ - !ruby/object:Gem::Dependency
62
+ name: rake
63
+ requirement: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - ">="
66
+ - !ruby/object:Gem::Version
67
+ version: 0.9.2
68
+ type: :development
69
+ prerelease: false
70
+ version_requirements: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ">="
73
+ - !ruby/object:Gem::Version
74
+ version: 0.9.2
75
+ description: SQL input/output plugin for Fluentd event collector
76
+ email: frsyuki@gmail.com
77
+ executables: []
78
+ extensions: []
79
+ extra_rdoc_files: []
80
+ files:
81
+ - ".travis.yml"
82
+ - Gemfile
83
+ - README.md
84
+ - Rakefile
85
+ - VERSION
86
+ - fluent-plugin-sql.gemspec
87
+ - lib/fluent/plugin/in_sql.rb
88
+ - lib/fluent/plugin/out_sql.rb
89
+ homepage: https://github.com/frsyuki/fluent-plugin-sql
90
+ licenses:
91
+ - Apache-2.0
92
+ metadata: {}
93
+ post_install_message:
94
+ rdoc_options: []
95
+ require_paths:
96
+ - lib
97
+ required_ruby_version: !ruby/object:Gem::Requirement
98
+ requirements:
99
+ - - ">="
100
+ - !ruby/object:Gem::Version
101
+ version: '0'
102
+ required_rubygems_version: !ruby/object:Gem::Requirement
103
+ requirements:
104
+ - - ">="
105
+ - !ruby/object:Gem::Version
106
+ version: '0'
107
+ requirements: []
108
+ rubyforge_project:
109
+ rubygems_version: 2.4.8
110
+ signing_key:
111
+ specification_version: 4
112
+ summary: SQL input/output plugin for Fluentd event collector
113
+ test_files: []