sm-fluent-plugin-sql 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.travis.yml +4 -0
- data/Gemfile +2 -0
- data/README.md +148 -0
- data/Rakefile +14 -0
- data/VERSION +1 -0
- data/fluent-plugin-sql.gemspec +24 -0
- data/lib/fluent/plugin/in_sql.rb +312 -0
- data/lib/fluent/plugin/out_sql.rb +251 -0
- metadata +113 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 2fc5710a69b577a42c8c49f8b37584d41686ed33
|
4
|
+
data.tar.gz: 2b647c14bdafa9428ec5a90cc3e70bc9118ea74b
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 6c8ff37a4cc11bba7a6955d4f6cb93aa0c40cbf158b1590083ccbf45ffc5b0eb964eacbbb919cc9498545ed8401711f0c7e39a0f2d95999fd25a7335be806243
|
7
|
+
data.tar.gz: ffc73333259e9a7a941c7b321327ac9f8a1d8716b2b9480bcced6630526b922c70d791e288ca023bd2a885f1382a6922b24dc129347499865b4a42945bb210c8
|
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,148 @@
|
|
1
|
+
# SQL input plugin for [Fluentd](http://fluentd.org) event collector
|
2
|
+
|
3
|
+
## Overview
|
4
|
+
|
5
|
+
This SQL plugin has two parts:
|
6
|
+
|
7
|
+
1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
|
8
|
+
2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
|
9
|
+
|
10
|
+
## Installation
|
11
|
+
|
12
|
+
$ fluent-gem install fluent-plugin-sql --no-document
|
13
|
+
$ fluent-gem install pg --no-document # for postgresql
|
14
|
+
|
15
|
+
You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
|
16
|
+
|
17
|
+
We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
|
18
|
+
|
19
|
+
## Input: How It Works
|
20
|
+
|
21
|
+
This plugin runs following SQL periodically:
|
22
|
+
|
23
|
+
SELECT * FROM *table* WHERE *update\_column* > *last\_update\_column\_value* ORDER BY *update_column* ASC LIMIT 500
|
24
|
+
|
25
|
+
What you need to configure is *update\_column*. The column should be an incremental column (such as AUTO\_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as `last_updated_at` column) so that this plugin reads the UPDATEd rows as well.
|
26
|
+
If you omit to set *update\_column* parameter, it uses primary key.
|
27
|
+
|
28
|
+
It stores last selected rows to a file (named *state\_file*) to not forget the last row when Fluentd restarts.
|
29
|
+
|
30
|
+
## Input: Configuration
|
31
|
+
|
32
|
+
<source>
|
33
|
+
@type sql
|
34
|
+
|
35
|
+
host rdb_host
|
36
|
+
database rdb_database
|
37
|
+
adapter mysql2_or_postgresql_or_etc
|
38
|
+
username myusername
|
39
|
+
password mypassword
|
40
|
+
|
41
|
+
tag_prefix my.rdb # optional, but recommended
|
42
|
+
|
43
|
+
select_interval 60s # optional
|
44
|
+
select_limit 500 # optional
|
45
|
+
|
46
|
+
state_file /var/run/fluentd/sql_state
|
47
|
+
|
48
|
+
<table>
|
49
|
+
table table1
|
50
|
+
tag table1 # optional
|
51
|
+
update_column update_col1
|
52
|
+
time_column time_col2 # optional
|
53
|
+
</table>
|
54
|
+
|
55
|
+
<table>
|
56
|
+
table table2
|
57
|
+
tag table2 # optional
|
58
|
+
update_column updated_at
|
59
|
+
time_column updated_at # optional
|
60
|
+
</table>
|
61
|
+
|
62
|
+
# detects all tables instead of <table> sections
|
63
|
+
#all_tables
|
64
|
+
</source>
|
65
|
+
|
66
|
+
* **host** RDBMS host
|
67
|
+
* **port** RDBMS port
|
68
|
+
* **database** RDBMS database name
|
69
|
+
* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
|
70
|
+
* **username** RDBMS login user name
|
71
|
+
* **password** RDBMS login password
|
72
|
+
* **tag_prefix** prefix of tags of events. actual tag will be this\_tag\_prefix.tables\_tag (optional)
|
73
|
+
* **select_interval** interval to run SQLs (optional)
|
74
|
+
* **select_limit** LIMIT of number of rows for each SQL (optional)
|
75
|
+
* **state_file** path to a file to store last rows
|
76
|
+
* **all_tables** reads all tables instead of configuring each tables in \<table\> sections
|
77
|
+
|
78
|
+
\<table\> sections:
|
79
|
+
|
80
|
+
* **tag** tag name of events (optional; default value is table name)
|
81
|
+
* **table** RDBM table name
|
82
|
+
* **update_column**: see above description
|
83
|
+
* **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
|
84
|
+
* **primary_key** (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.
|
85
|
+
|
86
|
+
## Input: Limitation
|
87
|
+
|
88
|
+
You should make sure target tables have index (and/or partitions) on the *update\_column*. Otherwise SELECT causes full table scan and serious performance problem.
|
89
|
+
|
90
|
+
You can't replicate DELETEd rows.
|
91
|
+
|
92
|
+
## Output: How It Works
|
93
|
+
|
94
|
+
This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.
|
95
|
+
|
96
|
+
## Output: Configuration
|
97
|
+
|
98
|
+
<match my.rdb.*>
|
99
|
+
@type sql
|
100
|
+
host rdb_host
|
101
|
+
port 3306
|
102
|
+
database rdb_database
|
103
|
+
adapter mysql2_or_postgresql_or_etc
|
104
|
+
username myusername
|
105
|
+
password mypassword
|
106
|
+
socket path_to_socket
|
107
|
+
remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql
|
108
|
+
|
109
|
+
<table>
|
110
|
+
table table1
|
111
|
+
column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
|
112
|
+
# This is the default table because it has no "pattern" argument in <table>
|
113
|
+
# The logic is such that if all non-default <table> blocks
|
114
|
+
# do not match, the default one is chosen.
|
115
|
+
# The default table is required.
|
116
|
+
</table>
|
117
|
+
|
118
|
+
<table hello.*> # You can pass the same pattern you use in match statements.
|
119
|
+
table table2
|
120
|
+
# This is the non-default table. It is chosen if the tag matches the pattern
|
121
|
+
# AFTER remove_tag_prefix is applied to the incoming event. For example, if
|
122
|
+
# the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
|
123
|
+
# makes it "hello.world", which gets matched here because of "pattern hello.*".
|
124
|
+
</table>
|
125
|
+
|
126
|
+
<table hello.world>
|
127
|
+
table table3
|
128
|
+
# This is the second non-default table. You can have as many non-default tables
|
129
|
+
# as you wish. One caveat: non-default tables are matched top-to-bottom and
|
130
|
+
# the events go into the first table it matches to. Hence, this particular table
|
131
|
+
# never gets any data, since the above "hello.*" subsumes "hello.world".
|
132
|
+
</table>
|
133
|
+
</match>
|
134
|
+
|
135
|
+
* **host** RDBMS host
|
136
|
+
* **port** RDBMS port
|
137
|
+
* **database** RDBMS database name
|
138
|
+
* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
|
139
|
+
* **username** RDBMS login user name
|
140
|
+
* **password** RDBMS login password
|
141
|
+
* **socket** RDBMS socket path
|
142
|
+
* **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
|
143
|
+
|
144
|
+
\<table\> sections:
|
145
|
+
|
146
|
+
* **table** RDBM table name
|
147
|
+
* **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
|
148
|
+
* **<table pattern>**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.
|
data/Rakefile
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
|
2
|
+
require 'bundler'
|
3
|
+
Bundler::GemHelper.install_tasks
|
4
|
+
|
5
|
+
#require 'rake/testtask'
|
6
|
+
#
|
7
|
+
#Rake::TestTask.new(:test) do |test|
|
8
|
+
# test.libs << 'lib' << 'test'
|
9
|
+
# test.test_files = FileList['test/*.rb']
|
10
|
+
# test.verbose = true
|
11
|
+
#end
|
12
|
+
|
13
|
+
task :default => [:build]
|
14
|
+
|
data/VERSION
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
0.5.1
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
$:.push File.expand_path('../lib', __FILE__)
|
3
|
+
|
4
|
+
Gem::Specification.new do |gem|
|
5
|
+
gem.name = "sm-fluent-plugin-sql"
|
6
|
+
gem.description = "SQL input/output plugin for Fluentd event collector"
|
7
|
+
gem.homepage = "https://github.com/frsyuki/fluent-plugin-sql"
|
8
|
+
gem.summary = gem.description
|
9
|
+
gem.version = File.read("VERSION").strip
|
10
|
+
gem.authors = ["Sadayuki Furuhashi"]
|
11
|
+
gem.email = "frsyuki@gmail.com"
|
12
|
+
gem.has_rdoc = false
|
13
|
+
#gem.platform = Gem::Platform::RUBY
|
14
|
+
gem.files = `git ls-files`.split("\n")
|
15
|
+
gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
16
|
+
gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
17
|
+
gem.require_paths = ['lib']
|
18
|
+
gem.license = "Apache-2.0"
|
19
|
+
|
20
|
+
gem.add_dependency "fluentd", [">= 0.12.17", "< 2"]
|
21
|
+
gem.add_dependency 'activerecord', "~> 4.2"
|
22
|
+
gem.add_dependency 'activerecord-import', "~> 0.7"
|
23
|
+
gem.add_development_dependency "rake", ">= 0.9.2"
|
24
|
+
end
|
@@ -0,0 +1,312 @@
|
|
1
|
+
#
|
2
|
+
# Fluent
|
3
|
+
#
|
4
|
+
# Copyright (C) 2013 FURUHASHI Sadayuki
|
5
|
+
#
|
6
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
7
|
+
# you may not use this file except in compliance with the License.
|
8
|
+
# You may obtain a copy of the License at
|
9
|
+
#
|
10
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
11
|
+
#
|
12
|
+
# Unless required by applicable law or agreed to in writing, software
|
13
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
14
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
15
|
+
# See the License for the specific language governing permissions and
|
16
|
+
# limitations under the License.
|
17
|
+
#
|
18
|
+
module Fluent
|
19
|
+
|
20
|
+
require 'active_record'
|
21
|
+
|
22
|
+
class SQLInput < Input
|
23
|
+
Plugin.register_input('sql', self)
|
24
|
+
|
25
|
+
# For fluentd v0.12.16 or earlier
|
26
|
+
class << self
|
27
|
+
unless method_defined?(:desc)
|
28
|
+
def desc(description)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
desc 'RDBMS host'
|
34
|
+
config_param :host, :string
|
35
|
+
desc 'RDBMS port'
|
36
|
+
config_param :port, :integer, :default => nil
|
37
|
+
desc 'RDBMS driver name.'
|
38
|
+
config_param :adapter, :string
|
39
|
+
desc 'RDBMS database name'
|
40
|
+
config_param :database, :string
|
41
|
+
desc 'RDBMS login user name'
|
42
|
+
config_param :username, :string, :default => nil
|
43
|
+
desc 'RDBMS login password'
|
44
|
+
config_param :password, :string, :default => nil, :secret => true
|
45
|
+
desc 'RDBMS socket path'
|
46
|
+
config_param :socket, :string, :default => nil
|
47
|
+
|
48
|
+
desc 'path to a file to store last rows'
|
49
|
+
config_param :state_file, :string, :default => nil
|
50
|
+
desc 'prefix of tags of events. actual tag will be this_tag_prefix.tables_tag (optional)'
|
51
|
+
config_param :tag_prefix, :string, :default => nil
|
52
|
+
desc 'interval to run SQLs (optional)'
|
53
|
+
config_param :select_interval, :time, :default => 60
|
54
|
+
desc 'limit of number of rows for each SQL(optional)'
|
55
|
+
config_param :select_limit, :time, :default => 500
|
56
|
+
|
57
|
+
unless method_defined?(:log)
|
58
|
+
define_method(:log) { $log }
|
59
|
+
end
|
60
|
+
|
61
|
+
class TableElement
|
62
|
+
include Configurable
|
63
|
+
|
64
|
+
config_param :table, :string
|
65
|
+
config_param :tag, :string, :default => nil
|
66
|
+
config_param :update_column, :string, :default => nil
|
67
|
+
config_param :time_column, :string, :default => nil
|
68
|
+
config_param :primary_key, :string, :default => nil
|
69
|
+
|
70
|
+
def configure(conf)
|
71
|
+
super
|
72
|
+
end
|
73
|
+
|
74
|
+
def init(tag_prefix, base_model, router)
|
75
|
+
@router = router
|
76
|
+
@tag = "#{tag_prefix}.#{@tag}" if tag_prefix
|
77
|
+
|
78
|
+
# creates a model for this table
|
79
|
+
table_name = @table
|
80
|
+
primary_key = @primary_key
|
81
|
+
@model = Class.new(base_model) do
|
82
|
+
self.table_name = table_name
|
83
|
+
self.inheritance_column = '_never_use_'
|
84
|
+
self.primary_key = primary_key if primary_key
|
85
|
+
|
86
|
+
#self.include_root_in_json = false
|
87
|
+
|
88
|
+
def read_attribute_for_serialization(n)
|
89
|
+
v = send(n)
|
90
|
+
if v.respond_to?(:to_msgpack)
|
91
|
+
v
|
92
|
+
else
|
93
|
+
v.to_s
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
97
|
+
|
98
|
+
# ActiveRecord requires model class to have a name.
|
99
|
+
class_name = table_name.singularize.camelize
|
100
|
+
base_model.const_set(class_name, @model)
|
101
|
+
|
102
|
+
# Sets model_name otherwise ActiveRecord causes errors
|
103
|
+
model_name = ActiveModel::Name.new(@model, nil, class_name)
|
104
|
+
@model.define_singleton_method(:model_name) { model_name }
|
105
|
+
|
106
|
+
# if update_column is not set, here uses primary key
|
107
|
+
unless @update_column
|
108
|
+
pk = @model.columns_hash[@model.primary_key]
|
109
|
+
unless pk
|
110
|
+
raise "Composite primary key is not supported. Set update_column parameter to <table> section."
|
111
|
+
end
|
112
|
+
@update_column = pk.name
|
113
|
+
end
|
114
|
+
end
|
115
|
+
|
116
|
+
# emits next records and returns the last record of emitted records
|
117
|
+
def emit_next_records(last_record, limit)
|
118
|
+
relation = @model
|
119
|
+
if last_record && last_update_value = last_record[@update_column]
|
120
|
+
relation = relation.where("#{@update_column} > ?", last_update_value)
|
121
|
+
end
|
122
|
+
relation = relation.order("#{@update_column} ASC")
|
123
|
+
relation = relation.limit(limit) if limit > 0
|
124
|
+
|
125
|
+
now = Engine.now
|
126
|
+
|
127
|
+
me = MultiEventStream.new
|
128
|
+
relation.each do |obj|
|
129
|
+
record = obj.serializable_hash rescue nil
|
130
|
+
if record
|
131
|
+
if @time_column && tv = obj.read_attribute(@time_column)
|
132
|
+
if tv.is_a?(Time)
|
133
|
+
time = tv.to_i
|
134
|
+
else
|
135
|
+
time = Time.parse(tv.to_s).to_i rescue now
|
136
|
+
end
|
137
|
+
else
|
138
|
+
time = now
|
139
|
+
end
|
140
|
+
me.add(time, record)
|
141
|
+
last_record = record
|
142
|
+
end
|
143
|
+
end
|
144
|
+
|
145
|
+
last_record = last_record.dup if last_record # some plugin rewrites record :(
|
146
|
+
@router.emit_stream(@tag, me)
|
147
|
+
|
148
|
+
return last_record
|
149
|
+
end
|
150
|
+
end
|
151
|
+
|
152
|
+
def configure(conf)
|
153
|
+
super
|
154
|
+
|
155
|
+
unless @state_file
|
156
|
+
$log.warn "'state_file PATH' parameter is not set to a 'sql' source."
|
157
|
+
$log.warn "this parameter is highly recommended to save the last rows to resume tailing."
|
158
|
+
end
|
159
|
+
|
160
|
+
@tables = conf.elements.select {|e|
|
161
|
+
e.name == 'table'
|
162
|
+
}.map {|e|
|
163
|
+
te = TableElement.new
|
164
|
+
te.configure(e)
|
165
|
+
te
|
166
|
+
}
|
167
|
+
|
168
|
+
if config['all_tables']
|
169
|
+
@all_tables = true
|
170
|
+
end
|
171
|
+
end
|
172
|
+
|
173
|
+
SKIP_TABLE_REGEXP = /\Aschema_migrations\Z/i
|
174
|
+
|
175
|
+
def start
|
176
|
+
@state_store = @state_file.nil? ? MemoryStateStore.new : StateStore.new(@state_file)
|
177
|
+
|
178
|
+
config = {
|
179
|
+
:adapter => @adapter,
|
180
|
+
:host => @host,
|
181
|
+
:port => @port,
|
182
|
+
:database => @database,
|
183
|
+
:username => @username,
|
184
|
+
:password => @password,
|
185
|
+
:socket => @socket,
|
186
|
+
}
|
187
|
+
|
188
|
+
# creates subclass of ActiveRecord::Base so that it can have different
|
189
|
+
# database configuration from ActiveRecord::Base.
|
190
|
+
@base_model = Class.new(ActiveRecord::Base) do
|
191
|
+
# base model doesn't have corresponding phisical table
|
192
|
+
self.abstract_class = true
|
193
|
+
end
|
194
|
+
|
195
|
+
# ActiveRecord requires the base_model to have a name. Here sets name
|
196
|
+
# of an anonymous class by assigning it to a constant. In Ruby, class has
|
197
|
+
# a name of a constant assigned first
|
198
|
+
SQLInput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
|
199
|
+
|
200
|
+
# Now base_model can have independent configuration from ActiveRecord::Base
|
201
|
+
@base_model.establish_connection(config)
|
202
|
+
|
203
|
+
if @all_tables
|
204
|
+
# get list of tables from the database
|
205
|
+
@tables = @base_model.connection.tables.map do |table_name|
|
206
|
+
if table_name.match(SKIP_TABLE_REGEXP)
|
207
|
+
# some tables such as "schema_migrations" should be ignored
|
208
|
+
nil
|
209
|
+
else
|
210
|
+
te = TableElement.new
|
211
|
+
te.configure({
|
212
|
+
'table' => table_name,
|
213
|
+
'tag' => table_name,
|
214
|
+
'update_column' => nil,
|
215
|
+
})
|
216
|
+
te
|
217
|
+
end
|
218
|
+
end.compact
|
219
|
+
end
|
220
|
+
|
221
|
+
# ignore tables if TableElement#init failed
|
222
|
+
@tables.reject! do |te|
|
223
|
+
begin
|
224
|
+
te.init(@tag_prefix, @base_model, router)
|
225
|
+
log.info "Selecting '#{te.table}' table"
|
226
|
+
false
|
227
|
+
rescue => e
|
228
|
+
log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
|
229
|
+
log.warn_backtrace e.backtrace
|
230
|
+
true
|
231
|
+
end
|
232
|
+
end
|
233
|
+
|
234
|
+
@stop_flag = false
|
235
|
+
@thread = Thread.new(&method(:thread_main))
|
236
|
+
end
|
237
|
+
|
238
|
+
def shutdown
|
239
|
+
@stop_flag = true
|
240
|
+
$log.debug "Waiting for thread to finish"
|
241
|
+
@thread.join
|
242
|
+
end
|
243
|
+
|
244
|
+
def thread_main
|
245
|
+
until @stop_flag
|
246
|
+
sleep @select_interval
|
247
|
+
|
248
|
+
begin
|
249
|
+
conn = @base_model.connection
|
250
|
+
conn.active? || conn.reconnect!
|
251
|
+
rescue => e
|
252
|
+
log.warn "can't connect to database. Reconnect at next try"
|
253
|
+
next
|
254
|
+
end
|
255
|
+
|
256
|
+
@tables.each do |t|
|
257
|
+
begin
|
258
|
+
last_record = @state_store.last_records[t.table]
|
259
|
+
@state_store.last_records[t.table] = t.emit_next_records(last_record, @select_limit)
|
260
|
+
@state_store.update!
|
261
|
+
rescue => e
|
262
|
+
log.error "unexpected error", :error => e.message, :error_class => e.class
|
263
|
+
log.error_backtrace e.backtrace
|
264
|
+
end
|
265
|
+
end
|
266
|
+
end
|
267
|
+
end
|
268
|
+
|
269
|
+
class StateStore
|
270
|
+
def initialize(path)
|
271
|
+
require 'yaml'
|
272
|
+
|
273
|
+
@path = path
|
274
|
+
if File.exists?(@path)
|
275
|
+
@data = YAML.load_file(@path)
|
276
|
+
if @data == false || @data == []
|
277
|
+
# this happens if an users created an empty file accidentally
|
278
|
+
@data = {}
|
279
|
+
elsif !@data.is_a?(Hash)
|
280
|
+
raise "state_file on #{@path.inspect} is invalid"
|
281
|
+
end
|
282
|
+
else
|
283
|
+
@data = {}
|
284
|
+
end
|
285
|
+
end
|
286
|
+
|
287
|
+
def last_records
|
288
|
+
@data['last_records'] ||= {}
|
289
|
+
end
|
290
|
+
|
291
|
+
def update!
|
292
|
+
File.open(@path, 'w') {|f|
|
293
|
+
f.write YAML.dump(@data)
|
294
|
+
}
|
295
|
+
end
|
296
|
+
end
|
297
|
+
|
298
|
+
class MemoryStateStore
|
299
|
+
def initialize
|
300
|
+
@data = {}
|
301
|
+
end
|
302
|
+
|
303
|
+
def last_records
|
304
|
+
@data['last_records'] ||= {}
|
305
|
+
end
|
306
|
+
|
307
|
+
def update!
|
308
|
+
end
|
309
|
+
end
|
310
|
+
end
|
311
|
+
|
312
|
+
end
|
@@ -0,0 +1,251 @@
|
|
1
|
+
module Fluent
|
2
|
+
class SQLOutput < BufferedOutput
|
3
|
+
Plugin.register_output('sql', self)
|
4
|
+
|
5
|
+
include SetTimeKeyMixin
|
6
|
+
include SetTagKeyMixin
|
7
|
+
|
8
|
+
# For fluentd v0.12.16 or earlier
|
9
|
+
class << self
|
10
|
+
unless method_defined?(:desc)
|
11
|
+
def desc(description)
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
desc 'RDBMS host'
|
17
|
+
config_param :host, :string
|
18
|
+
desc 'RDBMS port'
|
19
|
+
config_param :port, :integer, :default => nil
|
20
|
+
desc 'RDBMS driver name.'
|
21
|
+
config_param :adapter, :string
|
22
|
+
desc 'RDBMS login user name'
|
23
|
+
config_param :username, :string, :default => nil
|
24
|
+
desc 'RDBMS login password'
|
25
|
+
config_param :password, :string, :default => nil, :secret => true
|
26
|
+
desc 'RDBMS database name'
|
27
|
+
config_param :database, :string
|
28
|
+
desc 'RDBMS socket path'
|
29
|
+
config_param :socket, :string, :default => nil
|
30
|
+
desc 'remove the given prefix from the events'
|
31
|
+
config_param :remove_tag_prefix, :string, :default => nil
|
32
|
+
|
33
|
+
attr_accessor :tables
|
34
|
+
|
35
|
+
unless method_defined?(:log)
|
36
|
+
define_method(:log) { $log }
|
37
|
+
end
|
38
|
+
|
39
|
+
# TODO: Merge SQLInput's TableElement
|
40
|
+
class TableElement
|
41
|
+
include Configurable
|
42
|
+
|
43
|
+
config_param :table, :string
|
44
|
+
config_param :column_mapping, :string
|
45
|
+
config_param :num_retries, :integer, :default => 5
|
46
|
+
|
47
|
+
attr_reader :model
|
48
|
+
attr_reader :pattern
|
49
|
+
|
50
|
+
def initialize(pattern, log)
|
51
|
+
super()
|
52
|
+
@pattern = MatchPattern.create(pattern)
|
53
|
+
@log = log
|
54
|
+
end
|
55
|
+
|
56
|
+
def configure(conf)
|
57
|
+
super
|
58
|
+
|
59
|
+
@mapping = parse_column_mapping(@column_mapping)
|
60
|
+
@format_proc = Proc.new { |record|
|
61
|
+
new_record = {}
|
62
|
+
@mapping.each { |k, c|
|
63
|
+
new_record[c] = record[k]
|
64
|
+
}
|
65
|
+
new_record
|
66
|
+
}
|
67
|
+
end
|
68
|
+
|
69
|
+
def init(base_model)
|
70
|
+
# See SQLInput for more details of following code
|
71
|
+
table_name = @table
|
72
|
+
@model = Class.new(base_model) do
|
73
|
+
self.table_name = table_name
|
74
|
+
self.inheritance_column = '_never_use_output_'
|
75
|
+
end
|
76
|
+
|
77
|
+
class_name = table_name.singularize.camelize
|
78
|
+
base_model.const_set(class_name, @model)
|
79
|
+
model_name = ActiveModel::Name.new(@model, nil, class_name)
|
80
|
+
@model.define_singleton_method(:model_name) { model_name }
|
81
|
+
|
82
|
+
# TODO: check column_names and table schema
|
83
|
+
# @model.column_names
|
84
|
+
end
|
85
|
+
|
86
|
+
def import(chunk)
|
87
|
+
records = []
|
88
|
+
chunk.msgpack_each { |tag, time, data|
|
89
|
+
begin
|
90
|
+
# format process should be moved to emit / format after supports error stream.
|
91
|
+
records << @model.new(@format_proc.call(data))
|
92
|
+
rescue => e
|
93
|
+
args = {:error => e.message, :error_class => e.class, :table => @table, :record => Yajl.dump(data)}
|
94
|
+
@log.warn "Failed to create the model. Ignore a record:", args
|
95
|
+
end
|
96
|
+
}
|
97
|
+
begin
|
98
|
+
@model.import(records)
|
99
|
+
rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
|
100
|
+
# ignore other exceptions to use Fluentd retry mechanizm
|
101
|
+
@log.warn "Got deterministic error. Fallback to one-by-one import", :error => e.message, :error_class => e.class
|
102
|
+
one_by_one_import(records)
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
def one_by_one_import(records)
|
107
|
+
records.each { |record|
|
108
|
+
retries = 0
|
109
|
+
begin
|
110
|
+
@model.import([record])
|
111
|
+
rescue ActiveRecord::StatementInvalid, ActiveRecord::ThrowResult, ActiveRecord::Import::MissingColumnError => e
|
112
|
+
@log.error "Got deterministic error again. Dump a record", :error => e.message, :error_class => e.class, :record => record
|
113
|
+
rescue => e
|
114
|
+
retries += 1
|
115
|
+
if retries > @num_retries
|
116
|
+
@log.error "Can't recover undeterministic error. Dump a record", :error => e.message, :error_class => e.class, :record => record
|
117
|
+
next
|
118
|
+
end
|
119
|
+
|
120
|
+
@log.warn "Failed to import a record: retry number = #{retries}", :error => e.message, :error_class => e.class
|
121
|
+
sleep 0.5
|
122
|
+
retry
|
123
|
+
end
|
124
|
+
}
|
125
|
+
end
|
126
|
+
|
127
|
+
private
|
128
|
+
|
129
|
+
def parse_column_mapping(column_mapping_conf)
|
130
|
+
mapping = {}
|
131
|
+
column_mapping_conf.split(',').each { |column_map|
|
132
|
+
key, column = column_map.strip.split(':', 2)
|
133
|
+
column = key if column.nil?
|
134
|
+
mapping[key] = column
|
135
|
+
}
|
136
|
+
mapping
|
137
|
+
end
|
138
|
+
end
|
139
|
+
|
140
|
+
def initialize
|
141
|
+
super
|
142
|
+
require 'active_record'
|
143
|
+
require 'activerecord-import'
|
144
|
+
end
|
145
|
+
|
146
|
+
def configure(conf)
|
147
|
+
super
|
148
|
+
|
149
|
+
if remove_tag_prefix = conf['remove_tag_prefix']
|
150
|
+
@remove_tag_prefix = Regexp.new('^' + Regexp.escape(remove_tag_prefix))
|
151
|
+
end
|
152
|
+
|
153
|
+
@tables = []
|
154
|
+
@default_table = nil
|
155
|
+
conf.elements.select { |e|
|
156
|
+
e.name == 'table'
|
157
|
+
}.each { |e|
|
158
|
+
te = TableElement.new(e.arg, log)
|
159
|
+
te.configure(e)
|
160
|
+
if e.arg.empty?
|
161
|
+
$log.warn "Detect duplicate default table definition" if @default_table
|
162
|
+
@default_table = te
|
163
|
+
else
|
164
|
+
@tables << te
|
165
|
+
end
|
166
|
+
}
|
167
|
+
@only_default = @tables.empty?
|
168
|
+
|
169
|
+
if @default_table.nil?
|
170
|
+
raise ConfigError, "There is no default table. <table> is required in sql output"
|
171
|
+
end
|
172
|
+
end
|
173
|
+
|
174
|
+
def start
|
175
|
+
super
|
176
|
+
|
177
|
+
config = {
|
178
|
+
:adapter => @adapter,
|
179
|
+
:host => @host,
|
180
|
+
:port => @port,
|
181
|
+
:database => @database,
|
182
|
+
:username => @username,
|
183
|
+
:password => @password,
|
184
|
+
:socket => @socket,
|
185
|
+
}
|
186
|
+
|
187
|
+
@base_model = Class.new(ActiveRecord::Base) do
|
188
|
+
self.abstract_class = true
|
189
|
+
end
|
190
|
+
|
191
|
+
SQLOutput.const_set("BaseModel_#{rand(1 << 31)}", @base_model)
|
192
|
+
@base_model.establish_connection(config)
|
193
|
+
|
194
|
+
# ignore tables if TableElement#init failed
|
195
|
+
@tables.reject! do |te|
|
196
|
+
init_table(te, @base_model)
|
197
|
+
end
|
198
|
+
init_table(@default_table, @base_model)
|
199
|
+
end
|
200
|
+
|
201
|
+
def shutdown
|
202
|
+
super
|
203
|
+
end
|
204
|
+
|
205
|
+
def emit(tag, es, chain)
|
206
|
+
if @only_default
|
207
|
+
super(tag, es, chain)
|
208
|
+
else
|
209
|
+
super(tag, es, chain, format_tag(tag))
|
210
|
+
end
|
211
|
+
end
|
212
|
+
|
213
|
+
def format(tag, time, record)
|
214
|
+
[tag, time, record].to_msgpack
|
215
|
+
end
|
216
|
+
|
217
|
+
def write(chunk)
|
218
|
+
conn = @base_model.connection
|
219
|
+
conn.active? || conn.reconnect!
|
220
|
+
|
221
|
+
@tables.each { |table|
|
222
|
+
if table.pattern.match(chunk.key)
|
223
|
+
return table.import(chunk)
|
224
|
+
end
|
225
|
+
}
|
226
|
+
@default_table.import(chunk)
|
227
|
+
end
|
228
|
+
|
229
|
+
private
|
230
|
+
|
231
|
+
def init_table(te, base_model)
|
232
|
+
begin
|
233
|
+
te.init(base_model)
|
234
|
+
log.info "Selecting '#{te.table}' table"
|
235
|
+
false
|
236
|
+
rescue => e
|
237
|
+
log.warn "Can't handle '#{te.table}' table. Ignoring.", :error => e.message, :error_class => e.class
|
238
|
+
log.warn_backtrace e.backtrace
|
239
|
+
true
|
240
|
+
end
|
241
|
+
end
|
242
|
+
|
243
|
+
def format_tag(tag)
|
244
|
+
if @remove_tag_prefix
|
245
|
+
tag.gsub(@remove_tag_prefix, '')
|
246
|
+
else
|
247
|
+
tag
|
248
|
+
end
|
249
|
+
end
|
250
|
+
end
|
251
|
+
end
|
metadata
ADDED
@@ -0,0 +1,113 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: sm-fluent-plugin-sql
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.5.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Sadayuki Furuhashi
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2016-03-16 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: fluentd
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 0.12.17
|
20
|
+
- - "<"
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '2'
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.12.17
|
30
|
+
- - "<"
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '2'
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: activerecord
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - "~>"
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '4.2'
|
40
|
+
type: :runtime
|
41
|
+
prerelease: false
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - "~>"
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '4.2'
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
name: activerecord-import
|
49
|
+
requirement: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - "~>"
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0.7'
|
54
|
+
type: :runtime
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - "~>"
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: '0.7'
|
61
|
+
- !ruby/object:Gem::Dependency
|
62
|
+
name: rake
|
63
|
+
requirement: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: 0.9.2
|
68
|
+
type: :development
|
69
|
+
prerelease: false
|
70
|
+
version_requirements: !ruby/object:Gem::Requirement
|
71
|
+
requirements:
|
72
|
+
- - ">="
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: 0.9.2
|
75
|
+
description: SQL input/output plugin for Fluentd event collector
|
76
|
+
email: frsyuki@gmail.com
|
77
|
+
executables: []
|
78
|
+
extensions: []
|
79
|
+
extra_rdoc_files: []
|
80
|
+
files:
|
81
|
+
- ".travis.yml"
|
82
|
+
- Gemfile
|
83
|
+
- README.md
|
84
|
+
- Rakefile
|
85
|
+
- VERSION
|
86
|
+
- fluent-plugin-sql.gemspec
|
87
|
+
- lib/fluent/plugin/in_sql.rb
|
88
|
+
- lib/fluent/plugin/out_sql.rb
|
89
|
+
homepage: https://github.com/frsyuki/fluent-plugin-sql
|
90
|
+
licenses:
|
91
|
+
- Apache-2.0
|
92
|
+
metadata: {}
|
93
|
+
post_install_message:
|
94
|
+
rdoc_options: []
|
95
|
+
require_paths:
|
96
|
+
- lib
|
97
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
98
|
+
requirements:
|
99
|
+
- - ">="
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
103
|
+
requirements:
|
104
|
+
- - ">="
|
105
|
+
- !ruby/object:Gem::Version
|
106
|
+
version: '0'
|
107
|
+
requirements: []
|
108
|
+
rubyforge_project:
|
109
|
+
rubygems_version: 2.4.8
|
110
|
+
signing_key:
|
111
|
+
specification_version: 4
|
112
|
+
summary: SQL input/output plugin for Fluentd event collector
|
113
|
+
test_files: []
|