fluent-plugin-mysql 0.0.8 → 0.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (9) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +166 -137
  3. data/README_mysql.md +140 -0
  4. data/fluent-plugin-mysql.gemspec +3 -3
  5. data/lib/fluent/plugin/out_mysql.rb +2 -0
  6. data/lib/fluent/plugin/out_mysql_bulk.rb +127 -0
  7. data/test/helper.rb +1 -0
  8. data/test/plugin/test_out_mysql_bulk.rb +157 -0
  9. metadata +9 -4
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: aef912f01a0f6ca0a2907c5f9c5408c4c4ff2159
4
- data.tar.gz: b262b8918909b2e9c333b1e24c0aaf4241c5cf31
3
+ metadata.gz: 4aeb921f12fcfa4b99d1c1b75e54ec1ca13b81fb
4
+ data.tar.gz: 20bc5df93a4ab66f7896ea1870167f764962dfff
5
5
  SHA512:
6
- metadata.gz: 4ce9e3cffa7340e9d082c732fb711f7a7e5ce098f6281cac16bc13d6bc1901935dce67a6059b9103c1ffb8ba1471108e8889a19a996dd65340b25dca8ce0bf68
7
- data.tar.gz: 219b142587cab14566b49793741884fc35a50d7962fb7e474d5e81f609271bcd6cb745024e5d83bc4250aea1a93c2db7d84d8d2ade49e9b22cd68363531e6cba
6
+ metadata.gz: 0918d0a678fe2b122a4cbb4b732798eebed884dfa280281f626b8d3c2d60b62aa8535710bac78247d5a25b3fc98dea1e7a15826198d84088ca8b8ad252961514
7
+ data.tar.gz: ba434b5bc888fe1fe16f12ec3c2cf830730f7e05c2d5037c225a473c68b6a316c6e538f63bfaf5c8fbb0615f778db71cd4db5e1f2ec138664b6ee4450cb3bdc9
data/README.md CHANGED
@@ -1,140 +1,169 @@
1
- # fluent-plugin-mysql
2
-
3
- ## Component
4
-
5
- ### MysqlOutput
6
-
7
- [Fluentd](http://fluentd.org) plugin to store mysql tables over SQL, to each columns per values, or to single column as json.
8
-
9
- ## Configuration
10
-
11
- ### MysqlOutput
12
-
13
- MysqlOutput needs MySQL server's host/port/database/username/password, and INSERT format as SQL, or as table name and columns.
14
-
15
- <match output.by.sql.*>
16
- type mysql
17
- host master.db.service.local
18
- # port 3306 # default
19
- database application_logs
20
- username myuser
21
- password mypass
22
- key_names status,bytes,vhost,path,rhost,agent,referer
23
- sql INSERT INTO accesslog (status,bytes,vhost,path,rhost,agent,referer) VALUES (?,?,?,?,?,?,?)
24
- flush_interval 5s
25
- </match>
26
-
27
- <match output.by.names.*>
28
- type mysql
29
- host master.db.service.local
30
- database application_logs
31
- username myuser
32
- password mypass
33
- key_names status,bytes,vhost,path,rhost,agent,referer
34
- table accesslog
35
- # 'columns' names order must be same with 'key_names'
36
- columns status,bytes,vhost,path,rhost,agent,referer
37
- flush_interval 5s
38
- </match>
39
-
40
- Or, insert json into single column.
41
-
42
- <match output.as.json.*>
43
- type mysql
44
- host master.db.service.local
45
- database application_logs
46
- username root
47
- table accesslog
48
- columns jsondata
49
- format json
50
- flush_interval 5s
51
- </match>
52
-
53
- To include time/tag into output, use `include_time_key` and `include_tag_key`, like this:
54
-
55
- <match output.with.tag.and.time.*>
56
- type mysql
57
- host my.mysql.local
58
- database anydatabase
59
- username yourusername
60
- password secret
61
-
62
- include_time_key yes
63
- ### default `time_format` is ISO-8601
64
- # time_format %Y%m%d-%H%M%S
65
- ### default `time_key` is 'time'
66
- # time_key timekey
67
-
68
- include_tag_key yes
69
- ### default `tag_key` is 'tag'
70
- # tag_key tagkey
71
-
72
- table anydata
73
- key_names time,tag,field1,field2,field3,field4
74
- sql INSERT INTO baz (coltime,coltag,col1,col2,col3,col4) VALUES (?,?,?,?,?,?)
75
- </match>
76
-
77
- Or, for json:
78
-
79
- <match output.with.tag.and.time.as.json.*>
80
- type mysql
81
- host database.local
82
- database foo
83
- username root
84
-
85
- include_time_key yes
86
- utc # with UTC timezone output (default: localtime)
87
- time_format %Y%m%d-%H%M%S
88
- time_key timeattr
89
-
90
- include_tag_key yes
91
- tag_key tagattr
92
- table accesslog
93
- columns jsondata
94
- format json
95
- </match>
96
- #=> inserted json data into column 'jsondata' with addtional attribute 'timeattr' and 'tagattr'
97
-
98
- ### JsonPath format
99
-
100
- You can use [JsonPath](http://goessner.net/articles/JsonPath/) selectors as key_names, such as:
101
-
102
- <match output.with.jsonpath.format.*>
103
- type mysql
104
- host database.local
105
- database foo
106
- username bar
107
-
108
- include_time_key yes
109
- utc
110
- include_tag_key yes
111
- table baz
112
-
113
- format jsonpath
114
- key_names time, tag, id, data.name, tags[0]
115
- sql INSERT INTO baz (coltime,coltag,id,name,tag1) VALUES (?,?,?,?,?)
116
- </match>
117
-
118
- Which for a record like:
119
-
120
- `{ 'id' => 15, 'data'=> {'name' => 'jsonpath' }, 'tags' => ['unit', 'simple'] }`
121
-
122
- will generate the following insert values:
123
-
124
- `('2012-12-17T01:23:45Z','test',15,'jsonpath','unit')`
125
-
126
- ## Prerequisites
127
-
128
- `fluent-plugin-mysql` uses `mysql2` gem, and `mysql2` links against `libmysqlclient`. See [Installing](https://github.com/brianmario/mysql2#installing) for its installation.
129
-
130
- ## TODO
131
-
132
- * implement 'tag_mapped'
133
- * dynamic tag based table selection
1
+
2
+ # fluent-plugin-mysql, a plugin for [Fluentd](http://fluentd.org)
3
+
4
+ fluent plugin mysql bulk insert is high performance and on duplicate key update respond.
5
+
6
+ ## Note
7
+ fluent-plugin-mysql-bulk merged this repository.
8
+
9
+ [mysql plugin](README_mysql.md) is deprecated. You should use mysql_bulk.
10
+
11
+ ## Parameters
12
+
13
+ param|value
14
+ --------|------
15
+ host|database host(default: 127.0.0.1)
16
+ database|database name(require)
17
+ username|user(require)
18
+ password|password(default: blank)
19
+ column_names|bulk insert column (require)
20
+ key_names|value key names, ${time} is placeholder Time.at(time).strftime("%Y-%m-%d %H:%M:%S") (default : column_names)
21
+ table|bulk insert table (require)
22
+ on_duplicate_key_update|on duplicate key update enable (true:false)
23
+ on_duplicate_update_keys|on duplicate key update column, comma separator
24
+
25
+ ## Configuration Example(bulk insert)
26
+
27
+ ```
28
+ <match mysql.input>
29
+ type mysql_bulk
30
+ host localhost
31
+ database test_app_development
32
+ username root
33
+ password hogehoge
34
+ column_names id,user_name,created_at,updated_at
35
+ table users
36
+ flush_interval 10s
37
+ </match>
38
+ ```
39
+
40
+ Assume following input is coming:
41
+
42
+ ```js
43
+ mysql.input: {"user_name":"toyama","created_at":"2014/01/03 21:35:15","updated_at":"2014/01/03 21:35:15","dummy":"hogehoge"}
44
+ mysql.input: {"user_name":"toyama2","created_at":"2014/01/03 21:35:21","updated_at":"2014/01/03 21:35:21","dummy":"hogehoge"}
45
+ mysql.input: {"user_name":"toyama3","created_at":"2014/01/03 21:35:27","updated_at":"2014/01/03 21:35:27","dummy":"hogehoge"}
46
+ ```
47
+
48
+ then result becomes as below (indented):
49
+
50
+ ```sql
51
+ +-----+-----------+---------------------+---------------------+
52
+ | id | user_name | created_at | updated_at |
53
+ +-----+-----------+---------------------+---------------------+
54
+ | 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
55
+ | 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
56
+ | 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
57
+ +-----+-----------+---------------------+---------------------+
58
+ ```
59
+
60
+ running query
61
+
62
+ ```sql
63
+ INSERT INTO users (id,user_name,created_at,updated_at) VALUES (NULL,'toyama','2014/01/03 21:35:15','2014/01/03 21:35:15'),(NULL,'toyama2','2014/01/03 21:35:21','2014/01/03 21:35:21')
64
+ ```
65
+
66
+ ## Configuration Example(bulk insert , if duplicate error record update)
67
+
68
+ ```
69
+ <match mysql.input>
70
+ type mysql_bulk
71
+ host localhost
72
+ database test_app_development
73
+ username root
74
+ password hogehoge
75
+ column_names id,user_name,created_at,updated_at
76
+ table users
77
+ on_duplicate_key_update true
78
+ on_duplicate_update_keys user_name,updated_at
79
+ flush_interval 60s
80
+ </match>
81
+ ```
82
+
83
+ Assume following input is coming:
84
+
85
+ ```js
86
+ mysql.input: {"id":"1" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:03","updated_at":"2014/01/03 21:58:03"}
87
+ mysql.input: {"id":"2" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:06","updated_at":"2014/01/03 21:58:06"}
88
+ mysql.input: {"id":"3" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:08","updated_at":"2014/01/03 21:58:08"}
89
+ mysql.input: {"id":"10","user_name":"toyama7","created_at":"2014/01/03 21:58:18","updated_at":"2014/01/03 21:58:18"}
90
+ ```
91
+
92
+ then result becomes as below (indented):
93
+
94
+ ```sql
95
+ +-----+-----------+---------------------+---------------------+
96
+ | id | user_name | created_at | updated_at |
97
+ +-----+-----------+---------------------+---------------------+
98
+ | 1 | toyama7 | 2014-01-03 21:35:15 | 2014-01-03 21:58:03 |
99
+ | 2 | toyama7 | 2014-01-03 21:35:21 | 2014-01-03 21:58:06 |
100
+ | 3 | toyama7 | 2014-01-03 21:35:27 | 2014-01-03 21:58:08 |
101
+ | 10 | toyama7 | 2014-01-03 21:58:18 | 2014-01-03 21:58:18 |
102
+ +-----+-----------+---------------------+---------------------+
103
+ ```
104
+
105
+ if duplicate id , update username and updated_at
106
+
107
+
108
+ ## Configuration Example(bulk insert,fluentd key different column name)
109
+
110
+ ```
111
+ <match mysql.input>
112
+ type mysql_bulk
113
+ host localhost
114
+ database test_app_development
115
+ username root
116
+ password hogehoge
117
+ column_names id,user_name,created_at,updated_at
118
+ key_names id,user,created_date,updated_date
119
+ table users
120
+ flush_interval 10s
121
+ </match>
122
+ ```
123
+
124
+ Assume following input is coming:
125
+
126
+ ```js
127
+ mysql.input: {"user":"toyama","created_date":"2014/01/03 21:35:15","updated_date":"2014/01/03 21:35:15","dummy":"hogehoge"}
128
+ mysql.input: {"user":"toyama2","created_date":"2014/01/03 21:35:21","updated_date":"2014/01/03 21:35:21","dummy":"hogehoge"}
129
+ mysql.input: {"user":"toyama3","created_date":"2014/01/03 21:35:27","updated_date":"2014/01/03 21:35:27","dummy":"hogehoge"}
130
+ ```
131
+
132
+ then result becomes as below (indented):
133
+
134
+ ```sql
135
+ +-----+-----------+---------------------+---------------------+
136
+ | id | user_name | created_at | updated_at |
137
+ +-----+-----------+---------------------+---------------------+
138
+ | 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
139
+ | 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
140
+ | 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
141
+ +-----+-----------+---------------------+---------------------+
142
+ ```
143
+
144
+
145
+
146
+
147
+ ## spec
148
+
149
+ ```
150
+ bundle install
151
+ rake test
152
+ ```
153
+
154
+ ## todo
155
+
156
+ divide bulk insert(exsample 1000 per)
157
+
158
+
159
+ ## Contributing
160
+
161
+ 1. Fork it
162
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
163
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
164
+ 4. Push to the branch (`git push origin my-new-feature`)
165
+ 5. Create new [Pull Request](../../pull/new/master)
134
166
 
135
167
  ## Copyright
136
168
 
137
- * Copyright
138
- * Copyright(C) 2012- TAGOMORI Satoshi (tagomoris)
139
- * License
140
- * Apache License, Version 2.0
169
+ Copyright (c) 2016 Hiroshi Toyama. See [LICENSE](LICENSE) for details.
data/README_mysql.md ADDED
@@ -0,0 +1,140 @@
1
+ # fluent-plugin-mysql
2
+
3
+ ## Component
4
+
5
+ ### MysqlOutput
6
+
7
+ [Fluentd](http://fluentd.org) plugin to store mysql tables over SQL, to each columns per values, or to single column as json.
8
+
9
+ ## Configuration
10
+
11
+ ### MysqlOutput
12
+
13
+ MysqlOutput needs MySQL server's host/port/database/username/password, and INSERT format as SQL, or as table name and columns.
14
+
15
+ <match output.by.sql.*>
16
+ type mysql
17
+ host master.db.service.local
18
+ # port 3306 # default
19
+ database application_logs
20
+ username myuser
21
+ password mypass
22
+ key_names status,bytes,vhost,path,rhost,agent,referer
23
+ sql INSERT INTO accesslog (status,bytes,vhost,path,rhost,agent,referer) VALUES (?,?,?,?,?,?,?)
24
+ flush_interval 5s
25
+ </match>
26
+
27
+ <match output.by.names.*>
28
+ type mysql
29
+ host master.db.service.local
30
+ database application_logs
31
+ username myuser
32
+ password mypass
33
+ key_names status,bytes,vhost,path,rhost,agent,referer
34
+ table accesslog
35
+ # 'columns' names order must be same with 'key_names'
36
+ columns status,bytes,vhost,path,rhost,agent,referer
37
+ flush_interval 5s
38
+ </match>
39
+
40
+ Or, insert json into single column.
41
+
42
+ <match output.as.json.*>
43
+ type mysql
44
+ host master.db.service.local
45
+ database application_logs
46
+ username root
47
+ table accesslog
48
+ columns jsondata
49
+ format json
50
+ flush_interval 5s
51
+ </match>
52
+
53
+ To include time/tag into output, use `include_time_key` and `include_tag_key`, like this:
54
+
55
+ <match output.with.tag.and.time.*>
56
+ type mysql
57
+ host my.mysql.local
58
+ database anydatabase
59
+ username yourusername
60
+ password secret
61
+
62
+ include_time_key yes
63
+ ### default `time_format` is ISO-8601
64
+ # time_format %Y%m%d-%H%M%S
65
+ ### default `time_key` is 'time'
66
+ # time_key timekey
67
+
68
+ include_tag_key yes
69
+ ### default `tag_key` is 'tag'
70
+ # tag_key tagkey
71
+
72
+ table anydata
73
+ key_names time,tag,field1,field2,field3,field4
74
+ sql INSERT INTO baz (coltime,coltag,col1,col2,col3,col4) VALUES (?,?,?,?,?,?)
75
+ </match>
76
+
77
+ Or, for json:
78
+
79
+ <match output.with.tag.and.time.as.json.*>
80
+ type mysql
81
+ host database.local
82
+ database foo
83
+ username root
84
+
85
+ include_time_key yes
86
+ utc # with UTC timezone output (default: localtime)
87
+ time_format %Y%m%d-%H%M%S
88
+ time_key timeattr
89
+
90
+ include_tag_key yes
91
+ tag_key tagattr
92
+ table accesslog
93
+ columns jsondata
94
+ format json
95
+ </match>
96
+ #=> inserted json data into column 'jsondata' with addtional attribute 'timeattr' and 'tagattr'
97
+
98
+ ### JsonPath format
99
+
100
+ You can use [JsonPath](http://goessner.net/articles/JsonPath/) selectors as key_names, such as:
101
+
102
+ <match output.with.jsonpath.format.*>
103
+ type mysql
104
+ host database.local
105
+ database foo
106
+ username bar
107
+
108
+ include_time_key yes
109
+ utc
110
+ include_tag_key yes
111
+ table baz
112
+
113
+ format jsonpath
114
+ key_names time, tag, id, data.name, tags[0]
115
+ sql INSERT INTO baz (coltime,coltag,id,name,tag1) VALUES (?,?,?,?,?)
116
+ </match>
117
+
118
+ Which for a record like:
119
+
120
+ `{ 'id' => 15, 'data'=> {'name' => 'jsonpath' }, 'tags' => ['unit', 'simple'] }`
121
+
122
+ will generate the following insert values:
123
+
124
+ `('2012-12-17T01:23:45Z','test',15,'jsonpath','unit')`
125
+
126
+ ## Prerequisites
127
+
128
+ `fluent-plugin-mysql` uses `mysql2` gem, and `mysql2` links against `libmysqlclient`. See [Installing](https://github.com/brianmario/mysql2#installing) for its installation.
129
+
130
+ ## TODO
131
+
132
+ * implement 'tag_mapped'
133
+ * dynamic tag based table selection
134
+
135
+ ## Copyright
136
+
137
+ * Copyright
138
+ * Copyright(C) 2016- TAGOMORI Satoshi (tagomoris)
139
+ * License
140
+ * Apache License, Version 2.0
data/fluent-plugin-mysql.gemspec CHANGED
@@ -1,9 +1,9 @@
1
1
  # -*- encoding: utf-8 -*-
2
2
  Gem::Specification.new do |gem|
3
3
  gem.name = "fluent-plugin-mysql"
4
- gem.version = "0.0.8"
5
- gem.authors = ["TAGOMORI Satoshi"]
6
- gem.email = ["tagomoris@gmail.com"]
4
+ gem.version = "0.0.9"
5
+ gem.authors = ["TAGOMORI Satoshi", "Toyama Hiroshi"]
6
+ gem.email = ["tagomoris@gmail.com", "toyama0919@gmail.com"]
7
7
  gem.description = %q{fluent plugin to insert mysql as json(single column) or insert statement}
8
8
  gem.summary = %q{fluent plugin to insert mysql}
9
9
  gem.homepage = "https://github.com/tagomoris/fluent-plugin-mysql"
data/lib/fluent/plugin/out_mysql.rb CHANGED
@@ -33,6 +33,8 @@ class Fluent::MysqlOutput < Fluent::BufferedOutput
33
33
  def configure(conf)
34
34
  super
35
35
 
36
+ log.warn "[mysql] This plugin deprecated. You should use mysql_bulk."
37
+
36
38
  # TODO tag_mapped
37
39
 
38
40
  case @format
data/lib/fluent/plugin/out_mysql_bulk.rb ADDED
@@ -0,0 +1,127 @@
1
+ # -*- encoding : utf-8 -*-
2
+ module Fluent
3
+ class Fluent::MysqlBulkOutput < Fluent::BufferedOutput
4
+ Fluent::Plugin.register_output('mysql_bulk', self)
5
+
6
+ config_param :host, :string, default: '127.0.0.1'
7
+ config_param :port, :integer, default: 3306
8
+ config_param :database, :string
9
+ config_param :username, :string
10
+ config_param :password, :string, default: '', secret: true
11
+
12
+ config_param :column_names, :string
13
+ config_param :key_names, :string, default: nil
14
+ config_param :table, :string
15
+
16
+ config_param :on_duplicate_key_update, :bool, default: false
17
+ config_param :on_duplicate_update_keys, :string, default: nil
18
+
19
+ attr_accessor :handler
20
+
21
+ def initialize
22
+ super
23
+ require 'mysql2-cs-bind'
24
+ end
25
+
26
+ # Define `log` method for v0.10.42 or earlier
27
+ unless method_defined?(:log)
28
+ define_method("log") { $log }
29
+ end
30
+
31
+ def configure(conf)
32
+ super
33
+
34
+ if @column_names.nil?
35
+ fail Fluent::ConfigError, 'column_names MUST specified, but missing'
36
+ end
37
+
38
+ if @on_duplicate_key_update
39
+ if @on_duplicate_update_keys.nil?
40
+ fail Fluent::ConfigError, 'on_duplicate_key_update = true , on_duplicate_update_keys nil!'
41
+ end
42
+ @on_duplicate_update_keys = @on_duplicate_update_keys.split(',')
43
+
44
+ @on_duplicate_key_update_sql = ' ON DUPLICATE KEY UPDATE '
45
+ updates = []
46
+ @on_duplicate_update_keys.each do |update_column|
47
+ updates << "#{update_column} = VALUES(#{update_column})"
48
+ end
49
+ @on_duplicate_key_update_sql += updates.join(',')
50
+ end
51
+
52
+ @column_names = @column_names.split(',')
53
+ @key_names = @key_names.nil? ? @column_names : @key_names.split(',')
54
+ end
55
+
56
+ def start
57
+ super
58
+ result = client.xquery("SHOW COLUMNS FROM #{@table}")
59
+ @max_lengths = []
60
+ @column_names.each do |column|
61
+ info = result.select { |x| x['Field'] == column }.first
62
+ r = /(char|varchar)\(([\d]+)\)/
63
+ begin
64
+ max_length = info['Type'].scan(r)[0][1].to_i
65
+ rescue
66
+ max_length = nil
67
+ end
68
+ @max_lengths << max_length
69
+ end
70
+ end
71
+
72
+ def shutdown
73
+ super
74
+ end
75
+
76
+ def format(tag, time, record)
77
+ [tag, time, format_proc.call(tag, time, record)].to_msgpack
78
+ end
79
+
80
+ def client
81
+ Mysql2::Client.new(
82
+ host: @host,
83
+ port: @port,
84
+ username: @username,
85
+ password: @password,
86
+ database: @database,
87
+ flags: Mysql2::Client::MULTI_STATEMENTS
88
+ )
89
+ end
90
+
91
+ def write(chunk)
92
+ @handler = client
93
+ values = []
94
+ values_template = "(#{ @column_names.map { |key| '?' }.join(',') })"
95
+ chunk.msgpack_each do |tag, time, data|
96
+ values << Mysql2::Client.pseudo_bind(values_template, data)
97
+ end
98
+ sql = "INSERT INTO #{@table} (#{@column_names.join(',')}) VALUES #{values.join(',')}"
99
+ sql += @on_duplicate_key_update_sql if @on_duplicate_key_update
100
+
101
+ log.info "bulk insert values size => #{values.size}"
102
+ @handler.xquery(sql)
103
+ @handler.close
104
+ end
105
+
106
+ private
107
+
108
+ def format_proc
109
+ proc do |tag, time, record|
110
+ values = []
111
+ @key_names.each_with_index do |key, i|
112
+ if key == '${time}'
113
+ value = Time.at(time).strftime('%Y-%m-%d %H:%M:%S')
114
+ else
115
+ if @max_lengths[i].nil? || record[key].nil?
116
+ value = record[key]
117
+ else
118
+ value = record[key].slice(0, @max_lengths[i])
119
+ end
120
+ end
121
+ values << value
122
+ end
123
+ values
124
+ end
125
+ end
126
+ end
127
+ end
data/test/helper.rb CHANGED
@@ -23,6 +23,7 @@ unless ENV.has_key?('VERBOSE')
23
23
  end
24
24
 
25
25
  require 'fluent/plugin/out_mysql'
26
+ require 'fluent/plugin/out_mysql_bulk'
26
27
 
27
28
  class Test::Unit::TestCase
28
29
  end
data/test/plugin/test_out_mysql_bulk.rb ADDED
@@ -0,0 +1,157 @@
1
+ require 'helper'
2
+ require 'mysql2-cs-bind'
3
+
4
+ class MysqlBulkOutputTest < Test::Unit::TestCase
5
+ def setup
6
+ Fluent::Test.setup
7
+ end
8
+
9
+ def create_driver(conf = CONFIG, tag = 'test')
10
+ d = Fluent::Test::BufferedOutputTestDriver.new(Fluent::MysqlBulkOutput, tag).configure(conf)
11
+ d.instance.instance_eval {
12
+ def client
13
+ obj = Object.new
14
+ obj.instance_eval {
15
+ def xquery(*args); [1]; end
16
+ def close; true; end
17
+ }
18
+ obj
19
+ end
20
+ }
21
+ d
22
+ end
23
+
24
+ def test_configure_error
25
+ assert_raise(Fluent::ConfigError) do
26
+ d = create_driver %[
27
+ host localhost
28
+ database test_app_development
29
+ username root
30
+ password hogehoge
31
+ table users
32
+ on_duplicate_key_update true
33
+ on_duplicate_update_keys user_name,updated_at
34
+ flush_interval 10s
35
+ ]
36
+ end
37
+
38
+ assert_raise(Fluent::ConfigError) do
39
+ d = create_driver %[
40
+ host localhost
41
+ database test_app_development
42
+ username root
43
+ password hogehoge
44
+ column_names id,user_name,created_at,updated_at
45
+ table users
46
+ on_duplicate_key_update true
47
+ flush_interval 10s
48
+ ]
49
+ end
50
+
51
+ assert_raise(Fluent::ConfigError) do
52
+ d = create_driver %[
53
+ host localhost
54
+ username root
55
+ password hogehoge
56
+ column_names id,user_name,created_at,updated_at
57
+ table users
58
+ on_duplicate_key_update true
59
+ on_duplicate_update_keys user_name,updated_at
60
+ flush_interval 10s
61
+ ]
62
+ end
63
+ end
64
+
65
+ def test_configure
66
+ # not define format(default csv)
67
+ assert_nothing_raised(Fluent::ConfigError) do
68
+ d = create_driver %[
69
+ host localhost
70
+ database test_app_development
71
+ username root
72
+ password hogehoge
73
+ column_names id,user_name,created_at,updated_at
74
+ table users
75
+ on_duplicate_key_update true
76
+ on_duplicate_update_keys user_name,updated_at
77
+ flush_interval 10s
78
+ ]
79
+ end
80
+
81
+ assert_nothing_raised(Fluent::ConfigError) do
82
+ d = create_driver %[
83
+ database test_app_development
84
+ username root
85
+ password hogehoge
86
+ column_names id,user_name,created_at,updated_at
87
+ table users
88
+ ]
89
+ end
90
+
91
+ assert_nothing_raised(Fluent::ConfigError) do
92
+ d = create_driver %[
93
+ database test_app_development
94
+ username root
95
+ password hogehoge
96
+ column_names id,user_name,created_at,updated_at
97
+ table users
98
+ on_duplicate_key_update true
99
+ on_duplicate_update_keys user_name,updated_at
100
+ ]
101
+ end
102
+
103
+ assert_nothing_raised(Fluent::ConfigError) do
104
+ d = create_driver %[
105
+ database test_app_development
106
+ username root
107
+ password hogehoge
108
+ column_names id,user_name,created_at,updated_at
109
+ key_names id,user,created_date,updated_date
110
+ table users
111
+ on_duplicate_key_update true
112
+ on_duplicate_update_keys user_name,updated_at
113
+ ]
114
+ end
115
+ end
116
+
117
+ def test_variables
118
+ d = create_driver %[
119
+ database test_app_development
120
+ username root
121
+ password hogehoge
122
+ column_names id,user_name,created_at,updated_at
123
+ table users
124
+ on_duplicate_key_update true
125
+ on_duplicate_update_keys user_name,updated_at
126
+ ]
127
+
128
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
129
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.column_names
130
+ assert_equal " ON DUPLICATE KEY UPDATE user_name = VALUES(user_name),updated_at = VALUES(updated_at)", d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
131
+
132
+ d = create_driver %[
133
+ database test_app_development
134
+ username root
135
+ password hogehoge
136
+ column_names id,user_name,created_at,updated_at
137
+ table users
138
+ ]
139
+
140
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
141
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.column_names
142
+ assert_nil d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
143
+
144
+ d = create_driver %[
145
+ database test_app_development
146
+ username root
147
+ password hogehoge
148
+ key_names id,user_name,created_at,updated_at
149
+ column_names id,user,created_date,updated_date
150
+ table users
151
+ ]
152
+
153
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
154
+ assert_equal ['id','user','created_date','updated_date'], d.instance.column_names
155
+ assert_nil d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
156
+ end
157
+ end
metadata CHANGED
@@ -1,14 +1,15 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-mysql
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.8
4
+ version: 0.0.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - TAGOMORI Satoshi
8
+ - Toyama Hiroshi
8
9
  autorequire:
9
10
  bindir: bin
10
11
  cert_chain: []
11
- date: 2015-07-31 00:00:00.000000000 Z
12
+ date: 2016-01-08 00:00:00.000000000 Z
12
13
  dependencies:
13
14
  - !ruby/object:Gem::Dependency
14
15
  name: fluentd
@@ -83,6 +84,7 @@ dependencies:
83
84
  description: fluent plugin to insert mysql as json(single column) or insert statement
84
85
  email:
85
86
  - tagomoris@gmail.com
87
+ - toyama0919@gmail.com
86
88
  executables: []
87
89
  extensions: []
88
90
  extra_rdoc_files: []
@@ -91,11 +93,14 @@ files:
91
93
  - Gemfile
92
94
  - LICENSE.txt
93
95
  - README.md
96
+ - README_mysql.md
94
97
  - Rakefile
95
98
  - fluent-plugin-mysql.gemspec
96
99
  - lib/fluent/plugin/out_mysql.rb
100
+ - lib/fluent/plugin/out_mysql_bulk.rb
97
101
  - test/helper.rb
98
102
  - test/plugin/test_out_mysql.rb
103
+ - test/plugin/test_out_mysql_bulk.rb
99
104
  homepage: https://github.com/tagomoris/fluent-plugin-mysql
100
105
  licenses:
101
106
  - APLv2
@@ -116,11 +121,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
116
121
  version: '0'
117
122
  requirements: []
118
123
  rubyforge_project:
119
- rubygems_version: 2.4.5
124
+ rubygems_version: 2.4.2
120
125
  signing_key:
121
126
  specification_version: 4
122
127
  summary: fluent plugin to insert mysql
123
128
  test_files:
124
129
  - test/helper.rb
125
130
  - test/plugin/test_out_mysql.rb
126
- has_rdoc:
131
+ - test/plugin/test_out_mysql_bulk.rb