sm-fluent-plugin-sql 0.5.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.travis.yml +4 -0
- data/Gemfile +2 -0
- data/README.md +148 -0
- data/Rakefile +14 -0
- data/VERSION +1 -0
- data/fluent-plugin-sql.gemspec +24 -0
- data/lib/fluent/plugin/in_sql.rb +312 -0
- data/lib/fluent/plugin/out_sql.rb +251 -0
- metadata +113 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 2fc5710a69b577a42c8c49f8b37584d41686ed33
|
4
|
+
data.tar.gz: 2b647c14bdafa9428ec5a90cc3e70bc9118ea74b
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 6c8ff37a4cc11bba7a6955d4f6cb93aa0c40cbf158b1590083ccbf45ffc5b0eb964eacbbb919cc9498545ed8401711f0c7e39a0f2d95999fd25a7335be806243
|
7
|
+
data.tar.gz: ffc73333259e9a7a941c7b321327ac9f8a1d8716b2b9480bcced6630526b922c70d791e288ca023bd2a885f1382a6922b24dc129347499865b4a42945bb210c8
|
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,148 @@
|
|
1
|
+
# SQL input plugin for [Fluentd](http://fluentd.org) event collector
|
2
|
+
|
3
|
+
## Overview
|
4
|
+
|
5
|
+
This SQL plugin has two parts:
|
6
|
+
|
7
|
+
1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
|
8
|
+
2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
|
9
|
+
|
10
|
+
## Installation
|
11
|
+
|
12
|
+
$ fluent-gem install fluent-plugin-sql --no-document
|
13
|
+
$ fluent-gem install pg --no-document # for postgresql
|
14
|
+
|
15
|
+
You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
|
16
|
+
|
17
|
+
We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
|
18
|
+
|
19
|
+
## Input: How It Works
|
20
|
+
|
21
|
+
This plugin runs following SQL periodically:
|
22
|
+
|
23
|
+
SELECT * FROM *table* WHERE *update\_column* > *last\_update\_column\_value* ORDER BY *update_column* ASC LIMIT 500
|
24
|
+
|
25
|
+
What you need to configure is *update\_column*. The column should be an incremental column (such as AUTO\_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as `last_updated_at` column) so that this plugin reads the UPDATEd rows as well.
|
26
|
+
If you omit to set *update\_column* parameter, it uses primary key.
|
27
|
+
|
28
|
+
It stores last selected rows to a file (named *state\_file*) to not forget the last row when Fluentd restarts.
|
29
|
+
|
30
|
+
## Input: Configuration
|
31
|
+
|
32
|
+
<source>
|
33
|
+
@type sql
|
34
|
+
|
35
|
+
host rdb_host
|
36
|
+
database rdb_database
|
37
|
+
adapter mysql2_or_postgresql_or_etc
|
38
|
+
username myusername
|
39
|
+
password mypassword
|
40
|
+
|
41
|
+
tag_prefix my.rdb # optional, but recommended
|
42
|
+
|
43
|
+
select_interval 60s # optional
|
44
|
+
select_limit 500 # optional
|
45
|
+
|
46
|
+
state_file /var/run/fluentd/sql_state
|
47
|
+
|
48
|
+
<table>
|
49
|
+
table table1
|
50
|
+
tag table1 # optional
|
51
|
+
update_column update_col1
|
52
|
+
time_column time_col2 # optional
|
53
|
+
</table>
|
54
|
+
|
55
|
+
<table>
|
56
|
+
table table2
|
57
|
+
tag table2 # optional
|
58
|
+
update_column updated_at
|
59
|
+
time_column updated_at # optional
|
60
|
+
</table>
|
61
|
+
|
62
|
+
# detects all tables instead of <table> sections
|
63
|
+
#all_tables
|
64
|
+
</source>
|
65
|
+
|
66
|
+
* **host** RDBMS host
|
67
|
+
* **port** RDBMS port
|
68
|
+
* **database** RDBMS database name
|
69
|
+
* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
|
70
|
+
* **username** RDBMS login user name
|
71
|
+
* **password** RDBMS login password
|
72
|
+
* **tag_prefix** prefix of tags of events. actual tag will be this\_tag\_prefix.tables\_tag (optional)
|
73
|
+
* **select_interval** interval to run SQLs (optional)
|
74
|
+
* **select_limit** LIMIT of number of rows for each SQL (optional)
|
75
|
+
* **state_file** path to a file to store last rows
|
76
|
+
* **all_tables** reads all tables instead of configuring each tables in \<table\> sections
|
77
|
+
|
78
|
+
\<table\> sections:
|
79
|
+
|
80
|
+
* **tag** tag name of events (optional; default value is table name)
|
81
|
+
* **table** RDBM table name
|
82
|
+
* **update_column**: see above description
|
83
|
+
* **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
|
84
|
+
* **primary_key** (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.
|
85
|
+
|
86
|
+
## Input: Limitation
|
87
|
+
|
88
|
+
You should make sure target tables have index (and/or partitions) on the *update\_column*. Otherwise SELECT causes full table scan and serious performance problem.
|
89
|
+
|
90
|
+
You can't replicate DELETEd rows.
|
91
|
+
|
92
|
+
## Output: How It Works
|
93
|
+
|
94
|
+
This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.
|
95
|
+
|
96
|
+
## Output: Configuration
|
97
|
+
|
98
|
+
<match my.rdb.*>
|
99
|
+
@type sql
|
100
|
+
host rdb_host
|
101
|
+
port 3306
|
102
|
+
database rdb_database
|
103
|
+
adapter mysql2_or_postgresql_or_etc
|
104
|
+
username myusername
|
105
|
+
password mypassword
|
106
|
+
socket path_to_socket
|
107
|
+
remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql
|
108
|
+
|
109
|
+
<table>
|
110
|
+
table table1
|
111
|
+
column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
|
112
|
+
# This is the default table because it has no "pattern" argument in <table>
|
113
|
+
# The logic is such that if all non-default <table> blocks
|
114
|
+
# do not match, the default one is chosen.
|
115
|
+
# The default table is required.
|
116
|
+
</table>
|
117
|
+
|
118
|
+
<table hello.*> # You can pass the same pattern you use in match statements.
|
119
|
+
table table2
|
120
|
+
# This is the non-default table. It is chosen if the tag matches the pattern
|
121
|
+
# AFTER remove_tag_prefix is applied to the incoming event. For example, if
|
122
|
+
# the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
|
123
|
+
# makes it "hello.world", which gets matched here because of "pattern hello.*".
|
124
|
+
</table>
|
125
|
+
|
126
|
+
<table hello.world>
|
127
|
+
table table3
|
128
|
+
# This is the second non-default table. You can have as many non-default tables
|
129
|
+
# as you wish. One caveat: non-default tables are matched top-to-bottom and
|
130
|
+
# the events go into the first table it matches to. Hence, this particular table
|
131
|
+
# never gets any data, since the above "hello.*" subsumes "hello.world".
|
132
|
+
</table>
|
133
|
+
</match>
|
134
|
+
|
135
|
+
* **host** RDBMS host
|
136
|
+
* **port** RDBMS port
|
137
|
+
* **database** RDBMS database name
|
138
|
+
* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
|
139
|
+
* **username** RDBMS login user name
|
140
|
+
* **password** RDBMS login password
|
141
|
+
* **socket** RDBMS socket path
|
142
|
+
* **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
|
143
|
+
|
144
|
+
\<table\> sections:
|
145
|
+
|
146
|
+
* **table** RDBM table name
|
147
|
+
* **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
|
148
|
+
* **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
|
data/Rakefile
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
|
2
|
+
require 'bundler'
|
3
|
+
Bundler::GemHelper.install_tasks
|
4
|
+
|
5
|
+
#require 'rake/testtask'
|
6
|
+
#
|
7
|
+
#Rake::TestTask.new(:test) do |test|
|
8
|
+
# test.libs << 'lib' << 'test'
|
9
|
+
# test.test_files = FileList['test/*.rb']
|
10
|
+
# test.verbose = true
|
11
|
+
#end
|
12
|
+
|
13
|
+
task :default => [:build]
|
14
|
+
|
data/VERSION
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
0.5.1
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
$:.push File.expand_path('../lib', __FILE__)
|
3
|
+
|
4
|
+
Gem::Specification.new do |gem|
|
5
|
+
gem.name = "sm-fluent-plugin-sql"
|
6
|
+
gem.description = "SQL input/output plugin for Fluentd event collector"
|
7
|
+
gem.homepage = "https://github.com/frsyuki/fluent-plugin-sql"
|
8
|
+
gem.summary = gem.description
|
9
|
+
gem.version = File.read("VERSION").strip
|
10
|
+
gem.authors = ["Sadayuki Furuhashi"]
|
11
|
+
gem.email = "frsyuki@gmail.com"
|
12
|
+
gem.has_rdoc = false
|
13
|
+
#gem.platform = Gem::Platform::RUBY
|
14
|
+
gem.files = `git ls-files`.split("\n")
|
15
|
+
gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
16
|
+
gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
17
|
+
gem.require_paths = ['lib']
|
18
|
+
gem.license = "Apache-2.0"
|
19
|
+
|
20
|
+
gem.add_dependency "fluentd", [">= 0.12.17", "< 2"]
|
21
|
+
gem.add_dependency 'activerecord', "~> 4.2"
|
22
|
+
gem.add_dependency 'activerecord-import', "~> 0.7"
|
23
|
+
gem.add_development_dependency "rake", ">= 0.9.2"
|
24
|
+
end
|
@@ -0,0 +1,312 @@
|
|
1
|
+
#
|
2
|
+
# Fluent
|
3
|
+
#
|
4
|
+
# Copyright (C) 2013 FURUHASHI Sadayuki
|
5
|
+
#
|
6
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
7
|
+
# you may not use this file except in compliance with the License.
|
8
|
+
# You may obtain a copy of the License at
|
9
|
+
#
|
10
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
11
|
+
#
|
12
|
+
# Unless required by applicable law or agreed to in writing, software
|
13
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
14
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
15
|
+
# See the License for the specific language governing permissions and
|
16
|
+
# limitations under the License.
|
17
|
+
#
|
18
|
+
module Fluent
|
19
|
+
|
20
|
+
require 'active_record'
|
21
|
+
|
22
|
+
class SQLInput < Input
|
23
|
+
Plugin.register_input('sql', self)
|
24
|
+
|
25
|
+
# For fluentd v0.12.16 or earlier
|
26
|
+
class << self
|
27
|
+
unless method_defined?(:desc)
|
28
|
+
def desc(description)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
desc 'RDBMS host'
|
34
|
+
config_param :host, :string
|
35
|
+
desc 'RDBMS port'
|
36
|
+
config_param :port, :integer, :default => nil
|
37
|
+
desc 'RDBMS driver name.'
|
38
|
+
config_param :adapter, :string
|
39
|
+
desc 'RDBMS database name'
|
40
|
+
config_param :database, :string
|
41
|
+
desc 'RDBMS login user name'
|
42
|
+
config_param :username, :string, :default => nil
|
43
|
+
desc 'RDBMS login password'
|
44
|
+
config_param :password, :string, :default => nil, :secret => true
|
45
|
+
desc 'RDBMS socket path'
|
46
|
+
config_param :socket, :string, :default => nil
|
47
|
+
|
48
|
+
desc 'path to a file to store last rows'
|
49
|
+
config_param :state_file, :string, :default => nil
|
50
|
+
desc 'prefix of tags of events. actual tag will be this_tag_prefix.tables_tag (optional)'
|
51
|
+
config_param :tag_prefix, :string, :default => nil
|
52
|
+
desc 'interval to run SQLs (optional)'
|
53
|
+
config_param :select_interval, :time, :default => 60
|
54
|
+
desc 'limit of number of rows for each SQL(optional)'
|
55
|
+
config_param :select_limit, :time, :default => 500
|
56
|
+
|
57
|
+
unless method_defined?(:log)
|
58
|
+
define_method(:log) { $log }
|
59
|
+
end
|
60
|
+
|
61
|
+
class TableElement
|
62
|
+
include Configurable
|
63
|
+
|
64
|
+
config_param :table, :string
|
65
|
+
config_param :tag, :string, :default => nil
|
66
|
+
config_param :update_column, :string, :default => nil
|
67
|
+
config_param :time_column, :string, :default => nil
|
68
|
+
config_param :primary_key, :string, :default => nil
|
69
|
+
|
70
|
+
def configure(conf)
|
71
|
+
super
|
72
|
+
end
|
73
|
+
|
74
|
+
def init(tag_prefix, base_model, router)
|
75
|
+
@router = router
|
76
|
+
@tag = "#{tag_prefix}.#{@tag}" if tag_prefix
|
77
|
+
|
78
|
+
# creates a model for this table
|
79
|
+
table_name = @table
|
80
|
+
primary_key = @primary_key
|
81
|
+
@model = Class.new(base_model) do
|
82
|
+
self.table_name = table_name
|
83
|
+
self.inheritance_column = '_never_use_'
|
84
|
+
self.primary_key = primary_key if primary_key
|
85
|
+
|
86
|
+
#self.include_root_in_json = false
|
87
|
+
|
88
|
+
def read_attribute_for_serialization(n)
|
89
|
+
v = send(n)
|
90
|
+
if v.respond_to?(:to_msgpack)
|
91
|
+
v
|
92
|
+
else
|
93
|
+
v.to_s
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
97
|
+
|
98
|
+
# ActiveRecord requires model class to have a name.
|
99
|
+
class_name = table_name.singularize.camelize
|
100
|
+
base_model.const_set(class_name, @model)
|
101
|
+
|
102
|
+
# Sets model_name otherwise ActiveRecord causes errors
|
103
|
+
model_name = ActiveModel::Name.new(@model, nil, class_name)
|
104
|
+
@model.define_singleton_method(:model_name) { model_name }
|
105
|
+
|
106
|
+
# if update_column is not set, here uses primary key
|
107
|
+
unless @update_column
|
108
|
+
pk = @model.columns_hash[@model.primary_key]
|
109
|
+
unless pk
|
110
|
+
raise "Composite primary key is not supported. Set update_column parameter to <table> section."
|
111
|
+
end
|
112
|
+
@update_column = pk.name
|
113
|
+
end
|
114
|
+
end
|
115
|
+
|
116
|
+
# emits next records and returns the last record of emitted records
|
117
|
+
def emit_next_records(last_record, limit)
|
118
|
+
relation = @model
|
119
|
+
if last_record && last_update_value = last_record[@update_column]
|
120
|
+
relation = relation.where("#{@update_column} > ?", last_update_value)
|
121
|
+
end
|
122
|
+
relation = relation.order("#{@update_column} ASC")
|
123
|
+
relation = relation.limit(limit) if limit > 0
|
124
|
+
|
125
|
+
now = Engine.now
|
126
|
+
|
127
|
+
me = MultiEventStream.new
|
128
|
+
relation.each do |obj|
|
129
|
+
record = obj.serializable_hash rescue nil
|
130
|
+
if record
|
131
|
+
if @time_column && tv = obj.read_attribute(@time_column)
|
132
|
+
if tv.is_a?(Time)
|
133
|
+
time = tv.to_i
|
134
|
+
else
|
135
|
+
time = Time.parse(tv.to_s).to_i rescue now
|
136
|
+
end
|
137
|
+
else
|
138
|
+
time = now
|
139
|
+
end
|
140
|
+
me.add(time, record)
|
141
|
+
last_record = record
|
142
|
+
end
|
143
|
+
end
|
144
|
+
|
145
|
+
last_record = last_record.dup if last_record # some plugin rewrites record :(
|
146
|
+
@router.emit_stream(@tag, me)
|
147
|
+
|
148
|
+
return last_record
|
149
|
+
end
|
150
|
+
end
|
151
|
+
|
152
|
+
def configure(conf)
|
153
|
+
super
|
154
|
+
|
155
|
+
unless @state_file
|
156
|
+
$log.warn "'state_file PATH' parameter is not set to a 'sql' source."
|
157
|
+
$log.warn "this parameter is highly recommended to save the last rows to resume tailing."
|
158
|
+
end
|
159
|
+
|
160
|
+
@tables = conf.elements.select {|e|
|
161
|
+
e.name == 'table'
|
162
|
+
}.map {|e|
|
163
|
+
te = TableElement.new
|
164
|
+
te.configure(e)
|
165
|
+
te
|
166
|
+
}
|
167
|
+
|
168
|
+
if config['all_tables']
|
169
|
+
@all_tables = true
|
170
|
+
end
|
171
|
+
end
|
172
|
+
|
173
|
+
SKIP_TABLE_REGEXP = /\Aschema_migrations\Z/i
|
174
|
+
|
175
|
+
def start
|
176
|
+
@state_store = @state_file.nil? ? MemoryStateStore.new : StateStore.new(@state_file)
|
177
|
+
|
178
|
+
config = {
|
179
|
+
:adapter => @adapter,
|
180
|
+
:host => @host,
|
181
|
+
:port => @port,
|
182
|
+
:database => @database,
|
183
|
+
:username => @username,
|
184
|
+
:password => @password,
|
185
|
+
:socket => @socket,
|
186
|
+
}
|
187
|
+
|
188
|
+
# creates subclass of ActiveRecord::Base so that it can have different
|
189
|
+
# database configuration from ActiveRecord::Base.
|
190
|
+
@base_model = Class.new(ActiveRecord::Base) do
|
191
|
+
# base model doesn't have corresponding phisical table
|
192
|
+
self.abstract_class = true
|
193
|
+
end
|
194
|
+
|
195
|
+
# ActiveRecord requires the base_model to have a name. Here sets name
|
196
|
+
# of an anonymous class by assigning it to a constant. In Ruby, class has
|
197
|
+
# a name of a constant assigned first
|
198
|
+
SQLInput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
|
199
|
+
|
200
|
+
# Now base_model can have independent configuration from ActiveRecord::Base
|
201
|
+
@base_model.establish_connection(config)
|
202
|
+
|
203
|
+
if @all_tables
|
204
|
+
# get list of tables from the database
|
205
|
+
@tables = @base_model.connection.tables.map do |table_name|
|
206
|
+
if table_name.match(SKIP_TABLE_REGEXP)
|
207
|
+
# some tables such as "schema_migrations" should be ignored
|
208
|
+
nil
|
209
|
+
else
|
210
|
+
te = TableElement.new
|
211
|
+
te.configure({
|
212
|
+
'table' => table_name,
|
213
|
+
'tag' => table_name,
|
214
|
+
'update_column' => nil,
|
215
|
+
})
|
216
|
+
te
|
217
|
+
end
|
218
|
+
end.compact
|
219
|
+
end
|
220
|
+
|
221
|
+
# ignore tables if TableElement#init failed
|
222
|
+
@tables.reject! do |te|
|
223
|
+
begin
|
224
|
+
te.init(@tag_prefix, @base_model, router)
|
225
|
+
log.info "Selecting '#{te.table}' table"
|
226
|
+
false
|
227
|
+
rescue => e
|
228
|
+
log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
|
229
|
+
log.warn_backtrace e.backtrace
|
230
|
+
true
|
231
|
+
end
|
232
|
+
end
|
233
|
+
|
234
|
+
@stop_flag = false
|
235
|
+
@thread = Thread.new(&method(:thread_main))
|
236
|
+
end
|
237
|
+
|
238
|
+
def shutdown
|
239
|
+
@stop_flag = true
|
240
|
+
$log.debug "Waiting for thread to finish"
|
241
|
+
@thread.join
|
242
|
+
end
|
243
|
+
|
244
|
+
def thread_main
|
245
|
+
until @stop_flag
|
246
|
+
sleep @select_interval
|
247
|
+
|
248
|
+
begin
|
249
|
+
conn = @base_model.connection
|
250
|
+
conn.active? || conn.reconnect!
|
251
|
+
rescue => e
|
252
|
+
log.warn "can't connect to database. Reconnect at next try"
|
253
|
+
next
|
254
|
+
end
|
255
|
+
|
256
|
+
@tables.each do |t|
|
257
|
+
begin
|
258
|
+
last_record = @state_store.last_records[t.table]
|
259
|
+
@state_store.last_records[t.table] = t.emit_next_records(last_record, @select_limit)
|
260
|
+
@state_store.update!
|
261
|
+
rescue => e
|
262
|
+
log.error "unexpected error", :error => e.message, :error_class => e.class
|
263
|
+
log.error_backtrace e.backtrace
|
264
|
+
end
|
265
|
+
end
|
266
|
+
end
|
267
|
+
end
|
268
|
+
|
269
|
+
class StateStore
|
270
|
+
def initialize(path)
|
271
|
+
require 'yaml'
|
272
|
+
|
273
|
+
@path = path
|
274
|
+
if File.exists?(@path)
|
275
|
+
@data = YAML.load_file(@path)
|
276
|
+
if @data == false || @data == []
|
277
|
+
# this happens if an users created an empty file accidentally
|
278
|
+
@data = {}
|
279
|
+
elsif !@data.is_a?(Hash)
|
280
|
+
raise "state_file on #{@path.inspect} is invalid"
|
281
|
+
end
|
282
|
+
else
|
283
|
+
@data = {}
|
284
|
+
end
|
285
|
+
end
|
286
|
+
|
287
|
+
def last_records
|
288
|
+
@data['last_records'] ||= {}
|
289
|
+
end
|
290
|
+
|
291
|
+
def update!
|
292
|
+
File.open(@path, 'w') {|f|
|
293
|
+
f.write YAML.dump(@data)
|
294
|
+
}
|
295
|
+
end
|
296
|
+
end
|
297
|
+
|
298
|
+
class MemoryStateStore
|
299
|
+
def initialize
|
300
|
+
@data = {}
|
301
|
+
end
|
302
|
+
|
303
|
+
def last_records
|
304
|
+
@data['last_records'] ||= {}
|
305
|
+
end
|
306
|
+
|
307
|
+
def update!
|
308
|
+
end
|
309
|
+
end
|
310
|
+
end
|
311
|
+
|
312
|
+
end
|
@@ -0,0 +1,251 @@
|
|
1
|
+
module Fluent
|
2
|
+
class SQLOutput < BufferedOutput
|
3
|
+
Plugin.register_output('sql', self)
|
4
|
+
|
5
|
+
include SetTimeKeyMixin
|
6
|
+
include SetTagKeyMixin
|
7
|
+
|
8
|
+
# For fluentd v0.12.16 or earlier
|
9
|
+
class << self
|
10
|
+
unless method_defined?(:desc)
|
11
|
+
def desc(description)
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
desc 'RDBMS host'
|
17
|
+
config_param :host, :string
|
18
|
+
desc 'RDBMS port'
|
19
|
+
config_param :port, :integer, :default => nil
|
20
|
+
desc 'RDBMS driver name.'
|
21
|
+
config_param :adapter, :string
|
22
|
+
desc 'RDBMS login user name'
|
23
|
+
config_param :username, :string, :default => nil
|
24
|
+
desc 'RDBMS login password'
|
25
|
+
config_param :password, :string, :default => nil, :secret => true
|
26
|
+
desc 'RDBMS database name'
|
27
|
+
config_param :database, :string
|
28
|
+
desc 'RDBMS socket path'
|
29
|
+
config_param :socket, :string, :default => nil
|
30
|
+
desc 'remove the given prefix from the events'
|
31
|
+
config_param :remove_tag_prefix, :string, :default => nil
|
32
|
+
|
33
|
+
attr_accessor :tables
|
34
|
+
|
35
|
+
unless method_defined?(:log)
|
36
|
+
define_method(:log) { $log }
|
37
|
+
end
|
38
|
+
|
39
|
+
# TODO: Merge SQLInput's TableElement
|
40
|
+
class TableElement
|
41
|
+
include Configurable
|
42
|
+
|
43
|
+
config_param :table, :string
|
44
|
+
config_param :column_mapping, :string
|
45
|
+
config_param :num_retries, :integer, :default => 5
|
46
|
+
|
47
|
+
attr_reader :model
|
48
|
+
attr_reader :pattern
|
49
|
+
|
50
|
+
def initialize(pattern, log)
|
51
|
+
super()
|
52
|
+
@pattern = MatchPattern.create(pattern)
|
53
|
+
@log = log
|
54
|
+
end
|
55
|
+
|
56
|
+
def configure(conf)
|
57
|
+
super
|
58
|
+
|
59
|
+
@mapping = parse_column_mapping(@column_mapping)
|
60
|
+
@format_proc = Proc.new { |record|
|
61
|
+
new_record = {}
|
62
|
+
@mapping.each { |k, c|
|
63
|
+
new_record[c] = record[k]
|
64
|
+
}
|
65
|
+
new_record
|
66
|
+
}
|
67
|
+
end
|
68
|
+
|
69
|
+
def init(base_model)
|
70
|
+
# See SQLInput for more details of following code
|
71
|
+
table_name = @table
|
72
|
+
@model = Class.new(base_model) do
|
73
|
+
self.table_name = table_name
|
74
|
+
self.inheritance_column = '_never_use_output_'
|
75
|
+
end
|
76
|
+
|
77
|
+
class_name = table_name.singularize.camelize
|
78
|
+
base_model.const_set(class_name, @model)
|
79
|
+
model_name = ActiveModel::Name.new(@model, nil, class_name)
|
80
|
+
@model.define_singleton_method(:model_name) { model_name }
|
81
|
+
|
82
|
+
# TODO: check column_names and table schema
|
83
|
+
# @model.column_names
|
84
|
+
end
|
85
|
+
|
86
|
+
def import(chunk)
|
87
|
+
records = []
|
88
|
+
chunk.msgpack_each { |tag, time, data|
|
89
|
+
begin
|
90
|
+
# format process should be moved to emit / format after supports error stream.
|
91
|
+
records << @model.new(@format_proc.call(data))
|
92
|
+
rescue => e
|
93
|
+
args = {:error => e.message, :error_class => e.class, :table => @table, :record => Yajl.dump(data)}
|
94
|
+
@log.warn "Failed to create the model. Ignore a record:", args
|
95
|
+
end
|
96
|
+
}
|
97
|
+
begin
|
98
|
+
@model.import(records)
|
99
|
+
rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
|
100
|
+
# ignore other exceptions to use Fluentd retry mechanizm
|
101
|
+
@log.warn "Got deterministic error. Fallback to one-by-one import", :error => e.message, :error_class => e.class
|
102
|
+
one_by_one_import(records)
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
def one_by_one_import(records)
|
107
|
+
records.each { |record|
|
108
|
+
retries = 0
|
109
|
+
begin
|
110
|
+
@model.import([record])
|
111
|
+
rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
|
112
|
+
@log.error "Got deterministic error again. Dump a record", :error => e.message, :error_class => e.class, :record => record
|
113
|
+
rescue => e
|
114
|
+
retries += 1
|
115
|
+
if retries > @num_retries
|
116
|
+
@log.error "Can't recover undeterministic error. Dump a record", :error => e.message, :error_class => e.class, :record => record
|
117
|
+
next
|
118
|
+
end
|
119
|
+
|
120
|
+
@log.warn "Failed to import a record: retry number = #{retries}", :error => e.message, :error_class => e.class
|
121
|
+
sleep 0.5
|
122
|
+
retry
|
123
|
+
end
|
124
|
+
}
|
125
|
+
end
|
126
|
+
|
127
|
+
private
|
128
|
+
|
129
|
+
def parse_column_mapping(column_mapping_conf)
|
130
|
+
mapping = {}
|
131
|
+
column_mapping_conf.split(',').each { |column_map|
|
132
|
+
key, column = column_map.strip.split(':', 2)
|
133
|
+
column = key if column.nil?
|
134
|
+
mapping[key] = column
|
135
|
+
}
|
136
|
+
mapping
|
137
|
+
end
|
138
|
+
end
|
139
|
+
|
140
|
+
def initialize
|
141
|
+
super
|
142
|
+
require 'active_record'
|
143
|
+
require 'activerecord-import'
|
144
|
+
end
|
145
|
+
|
146
|
+
def configure(conf)
|
147
|
+
super
|
148
|
+
|
149
|
+
if remove_tag_prefix = conf['remove_tag_prefix']
|
150
|
+
@remove_tag_prefix = Regexp.new('^' + Regexp.escape(remove_tag_prefix))
|
151
|
+
end
|
152
|
+
|
153
|
+
@tables = []
|
154
|
+
@default_table = nil
|
155
|
+
conf.elements.select { |e|
|
156
|
+
e.name == 'table'
|
157
|
+
}.each { |e|
|
158
|
+
te = TableElement.new(e.arg, log)
|
159
|
+
te.configure(e)
|
160
|
+
if e.arg.empty?
|
161
|
+
$log.warn "Detect duplicate default table definition" if @default_table
|
162
|
+
@default_table = te
|
163
|
+
else
|
164
|
+
@tables << te
|
165
|
+
end
|
166
|
+
}
|
167
|
+
@only_default = @tables.empty?
|
168
|
+
|
169
|
+
if @default_table.nil?
|
170
|
+
raise ConfigError, "There is no default table. <table> is required in sql output"
|
171
|
+
end
|
172
|
+
end
|
173
|
+
|
174
|
+
def start
|
175
|
+
super
|
176
|
+
|
177
|
+
config = {
|
178
|
+
:adapter => @adapter,
|
179
|
+
:host => @host,
|
180
|
+
:port => @port,
|
181
|
+
:database => @database,
|
182
|
+
:username => @username,
|
183
|
+
:password => @password,
|
184
|
+
:socket => @socket,
|
185
|
+
}
|
186
|
+
|
187
|
+
@base_model = Class.new(ActiveRecord::Base) do
|
188
|
+
self.abstract_class = true
|
189
|
+
end
|
190
|
+
|
191
|
+
SQLOutput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
|
192
|
+
@base_model.establish_connection(config)
|
193
|
+
|
194
|
+
# ignore tables if TableElement#init failed
|
195
|
+
@tables.reject! do |te|
|
196
|
+
init_table(te, @base_model)
|
197
|
+
end
|
198
|
+
init_table(@default_table, @base_model)
|
199
|
+
end
|
200
|
+
|
201
|
+
def shutdown
|
202
|
+
super
|
203
|
+
end
|
204
|
+
|
205
|
+
def emit(tag, es, chain)
|
206
|
+
if @only_default
|
207
|
+
super(tag, es, chain)
|
208
|
+
else
|
209
|
+
super(tag, es, chain, format_tag(tag))
|
210
|
+
end
|
211
|
+
end
|
212
|
+
|
213
|
+
def format(tag, time, record)
|
214
|
+
[tag, time, record].to_msgpack
|
215
|
+
end
|
216
|
+
|
217
|
+
def write(chunk)
|
218
|
+
conn = @base_model.connection
|
219
|
+
conn.active? || conn.reconnect!
|
220
|
+
|
221
|
+
@tables.each { |table|
|
222
|
+
if table.pattern.match(chunk.key)
|
223
|
+
return table.import(chunk)
|
224
|
+
end
|
225
|
+
}
|
226
|
+
@default_table.import(chunk)
|
227
|
+
end
|
228
|
+
|
229
|
+
private
|
230
|
+
|
231
|
+
def init_table(te, base_model)
|
232
|
+
begin
|
233
|
+
te.init(base_model)
|
234
|
+
log.info "Selecting '#{te.table}' table"
|
235
|
+
false
|
236
|
+
rescue => e
|
237
|
+
log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
|
238
|
+
log.warn_backtrace e.backtrace
|
239
|
+
true
|
240
|
+
end
|
241
|
+
end
|
242
|
+
|
243
|
+
def format_tag(tag)
|
244
|
+
if @remove_tag_prefix
|
245
|
+
tag.gsub(@remove_tag_prefix, '')
|
246
|
+
else
|
247
|
+
tag
|
248
|
+
end
|
249
|
+
end
|
250
|
+
end
|
251
|
+
end
|
metadata
ADDED
@@ -0,0 +1,113 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: sm-fluent-plugin-sql
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.5.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Sadayuki Furuhashi
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2016-03-16 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: fluentd
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 0.12.17
|
20
|
+
- - "<"
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '2'
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.12.17
|
30
|
+
- - "<"
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '2'
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: activerecord
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - "~>"
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '4.2'
|
40
|
+
type: :runtime
|
41
|
+
prerelease: false
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - "~>"
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '4.2'
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
name: activerecord-import
|
49
|
+
requirement: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - "~>"
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0.7'
|
54
|
+
type: :runtime
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - "~>"
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: '0.7'
|
61
|
+
- !ruby/object:Gem::Dependency
|
62
|
+
name: rake
|
63
|
+
requirement: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: 0.9.2
|
68
|
+
type: :development
|
69
|
+
prerelease: false
|
70
|
+
version_requirements: !ruby/object:Gem::Requirement
|
71
|
+
requirements:
|
72
|
+
- - ">="
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: 0.9.2
|
75
|
+
description: SQL input/output plugin for Fluentd event collector
|
76
|
+
email: frsyuki@gmail.com
|
77
|
+
executables: []
|
78
|
+
extensions: []
|
79
|
+
extra_rdoc_files: []
|
80
|
+
files:
|
81
|
+
- ".travis.yml"
|
82
|
+
- Gemfile
|
83
|
+
- README.md
|
84
|
+
- Rakefile
|
85
|
+
- VERSION
|
86
|
+
- fluent-plugin-sql.gemspec
|
87
|
+
- lib/fluent/plugin/in_sql.rb
|
88
|
+
- lib/fluent/plugin/out_sql.rb
|
89
|
+
homepage: https://github.com/frsyuki/fluent-plugin-sql
|
90
|
+
licenses:
|
91
|
+
- Apache-2.0
|
92
|
+
metadata: {}
|
93
|
+
post_install_message:
|
94
|
+
rdoc_options: []
|
95
|
+
require_paths:
|
96
|
+
- lib
|
97
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
98
|
+
requirements:
|
99
|
+
- - ">="
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
103
|
+
requirements:
|
104
|
+
- - ">="
|
105
|
+
- !ruby/object:Gem::Version
|
106
|
+
version: '0'
|
107
|
+
requirements: []
|
108
|
+
rubyforge_project:
|
109
|
+
rubygems_version: 2.4.8
|
110
|
+
signing_key:
|
111
|
+
specification_version: 4
|
112
|
+
summary: SQL input/output plugin for Fluentd event collector
|
113
|
+
test_files: []
|