fluent-plugin-mysql 0.0.8 → 0.0.9

Sign up to get free protection for your applications and to get access to all the features.
Files changed (9) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +166 -137
  3. data/README_mysql.md +140 -0
  4. data/fluent-plugin-mysql.gemspec +3 -3
  5. data/lib/fluent/plugin/out_mysql.rb +2 -0
  6. data/lib/fluent/plugin/out_mysql_bulk.rb +127 -0
  7. data/test/helper.rb +1 -0
  8. data/test/plugin/test_out_mysql_bulk.rb +157 -0
  9. metadata +9 -4
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: aef912f01a0f6ca0a2907c5f9c5408c4c4ff2159
4
- data.tar.gz: b262b8918909b2e9c333b1e24c0aaf4241c5cf31
3
+ metadata.gz: 4aeb921f12fcfa4b99d1c1b75e54ec1ca13b81fb
4
+ data.tar.gz: 20bc5df93a4ab66f7896ea1870167f764962dfff
5
5
  SHA512:
6
- metadata.gz: 4ce9e3cffa7340e9d082c732fb711f7a7e5ce098f6281cac16bc13d6bc1901935dce67a6059b9103c1ffb8ba1471108e8889a19a996dd65340b25dca8ce0bf68
7
- data.tar.gz: 219b142587cab14566b49793741884fc35a50d7962fb7e474d5e81f609271bcd6cb745024e5d83bc4250aea1a93c2db7d84d8d2ade49e9b22cd68363531e6cba
6
+ metadata.gz: 0918d0a678fe2b122a4cbb4b732798eebed884dfa280281f626b8d3c2d60b62aa8535710bac78247d5a25b3fc98dea1e7a15826198d84088ca8b8ad252961514
7
+ data.tar.gz: ba434b5bc888fe1fe16f12ec3c2cf830730f7e05c2d5037c225a473c68b6a316c6e538f63bfaf5c8fbb0615f778db71cd4db5e1f2ec138664b6ee4450cb3bdc9
data/README.md CHANGED
@@ -1,140 +1,169 @@
1
- # fluent-plugin-mysql
2
-
3
- ## Component
4
-
5
- ### MysqlOutput
6
-
7
- [Fluentd](http://fluentd.org) plugin to store mysql tables over SQL, to each columns per values, or to single column as json.
8
-
9
- ## Configuration
10
-
11
- ### MysqlOutput
12
-
13
- MysqlOutput needs MySQL server's host/port/database/username/password, and INSERT format as SQL, or as table name and columns.
14
-
15
- <match output.by.sql.*>
16
- type mysql
17
- host master.db.service.local
18
- # port 3306 # default
19
- database application_logs
20
- username myuser
21
- password mypass
22
- key_names status,bytes,vhost,path,rhost,agent,referer
23
- sql INSERT INTO accesslog (status,bytes,vhost,path,rhost,agent,referer) VALUES (?,?,?,?,?,?,?)
24
- flush_interval 5s
25
- </match>
26
-
27
- <match output.by.names.*>
28
- type mysql
29
- host master.db.service.local
30
- database application_logs
31
- username myuser
32
- password mypass
33
- key_names status,bytes,vhost,path,rhost,agent,referer
34
- table accesslog
35
- # 'columns' names order must be same with 'key_names'
36
- columns status,bytes,vhost,path,rhost,agent,referer
37
- flush_interval 5s
38
- </match>
39
-
40
- Or, insert json into single column.
41
-
42
- <match output.as.json.*>
43
- type mysql
44
- host master.db.service.local
45
- database application_logs
46
- username root
47
- table accesslog
48
- columns jsondata
49
- format json
50
- flush_interval 5s
51
- </match>
52
-
53
- To include time/tag into output, use `include_time_key` and `include_tag_key`, like this:
54
-
55
- <match output.with.tag.and.time.*>
56
- type mysql
57
- host my.mysql.local
58
- database anydatabase
59
- username yourusername
60
- password secret
61
-
62
- include_time_key yes
63
- ### default `time_format` is ISO-8601
64
- # time_format %Y%m%d-%H%M%S
65
- ### default `time_key` is 'time'
66
- # time_key timekey
67
-
68
- include_tag_key yes
69
- ### default `tag_key` is 'tag'
70
- # tag_key tagkey
71
-
72
- table anydata
73
- key_names time,tag,field1,field2,field3,field4
74
- sql INSERT INTO baz (coltime,coltag,col1,col2,col3,col4) VALUES (?,?,?,?,?,?)
75
- </match>
76
-
77
- Or, for json:
78
-
79
- <match output.with.tag.and.time.as.json.*>
80
- type mysql
81
- host database.local
82
- database foo
83
- username root
84
-
85
- include_time_key yes
86
- utc # with UTC timezone output (default: localtime)
87
- time_format %Y%m%d-%H%M%S
88
- time_key timeattr
89
-
90
- include_tag_key yes
91
- tag_key tagattr
92
- table accesslog
93
- columns jsondata
94
- format json
95
- </match>
96
- #=> inserted json data into column 'jsondata' with addtional attribute 'timeattr' and 'tagattr'
97
-
98
- ### JsonPath format
99
-
100
- You can use [JsonPath](http://goessner.net/articles/JsonPath/) selectors as key_names, such as:
101
-
102
- <match output.with.jsonpath.format.*>
103
- type mysql
104
- host database.local
105
- database foo
106
- username bar
107
-
108
- include_time_key yes
109
- utc
110
- include_tag_key yes
111
- table baz
112
-
113
- format jsonpath
114
- key_names time, tag, id, data.name, tags[0]
115
- sql INSERT INTO baz (coltime,coltag,id,name,tag1) VALUES (?,?,?,?,?)
116
- </match>
117
-
118
- Which for a record like:
119
-
120
- `{ 'id' => 15, 'data'=> {'name' => 'jsonpath' }, 'tags' => ['unit', 'simple'] }`
121
-
122
- will generate the following insert values:
123
-
124
- `('2012-12-17T01:23:45Z','test',15,'jsonpath','unit')`
125
-
126
- ## Prerequisites
127
-
128
- `fluent-plugin-mysql` uses `mysql2` gem, and `mysql2` links against `libmysqlclient`. See [Installing](https://github.com/brianmario/mysql2#installing) for its installation.
129
-
130
- ## TODO
131
-
132
- * implement 'tag_mapped'
133
- * dynamic tag based table selection
1
+
2
+ # fluent-plugin-mysql, a plugin for [Fluentd](http://fluentd.org)
3
+
4
+ fluent plugin mysql bulk insert is high performance and on duplicate key update respond.
5
+
6
+ ## Note
7
+ fluent-plugin-mysql-bulk merged this repository.
8
+
9
+ [mysql plugin](README_mysql.md) is deprecated. You should use mysql_bulk.
10
+
11
+ ## Parameters
12
+
13
+ param|value
14
+ --------|------
15
+ host|database host(default: 127.0.0.1)
16
+ database|database name(require)
17
+ username|user(require)
18
+ password|password(default: blank)
19
+ column_names|bulk insert column (require)
20
+ key_names|value key names, ${time} is placeholder Time.at(time).strftime("%Y-%m-%d %H:%M:%S") (default : column_names)
21
+ table|bulk insert table (require)
22
+ on_duplicate_key_update|on duplicate key update enable (true:false)
23
+ on_duplicate_update_keys|on duplicate key update column, comma separator
24
+
25
+ ## Configuration Example(bulk insert)
26
+
27
+ ```
28
+ <match mysql.input>
29
+ type mysql_bulk
30
+ host localhost
31
+ database test_app_development
32
+ username root
33
+ password hogehoge
34
+ column_names id,user_name,created_at,updated_at
35
+ table users
36
+ flush_interval 10s
37
+ </match>
38
+ ```
39
+
40
+ Assume following input is coming:
41
+
42
+ ```js
43
+ mysql.input: {"user_name":"toyama","created_at":"2014/01/03 21:35:15","updated_at":"2014/01/03 21:35:15","dummy":"hogehoge"}
44
+ mysql.input: {"user_name":"toyama2","created_at":"2014/01/03 21:35:21","updated_at":"2014/01/03 21:35:21","dummy":"hogehoge"}
45
+ mysql.input: {"user_name":"toyama3","created_at":"2014/01/03 21:35:27","updated_at":"2014/01/03 21:35:27","dummy":"hogehoge"}
46
+ ```
47
+
48
+ then result becomes as below (indented):
49
+
50
+ ```sql
51
+ +-----+-----------+---------------------+---------------------+
52
+ | id | user_name | created_at | updated_at |
53
+ +-----+-----------+---------------------+---------------------+
54
+ | 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
55
+ | 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
56
+ | 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
57
+ +-----+-----------+---------------------+---------------------+
58
+ ```
59
+
60
+ running query
61
+
62
+ ```sql
63
+ INSERT INTO users (id,user_name,created_at,updated_at) VALUES (NULL,'toyama','2014/01/03 21:35:15','2014/01/03 21:35:15'),(NULL,'toyama2','2014/01/03 21:35:21','2014/01/03 21:35:21')
64
+ ```
65
+
66
+ ## Configuration Example(bulk insert , if duplicate error record update)
67
+
68
+ ```
69
+ <match mysql.input>
70
+ type mysql_bulk
71
+ host localhost
72
+ database test_app_development
73
+ username root
74
+ password hogehoge
75
+ column_names id,user_name,created_at,updated_at
76
+ table users
77
+ on_duplicate_key_update true
78
+ on_duplicate_update_keys user_name,updated_at
79
+ flush_interval 60s
80
+ </match>
81
+ ```
82
+
83
+ Assume following input is coming:
84
+
85
+ ```js
86
+ mysql.input: {"id":"1" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:03","updated_at":"2014/01/03 21:58:03"}
87
+ mysql.input: {"id":"2" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:06","updated_at":"2014/01/03 21:58:06"}
88
+ mysql.input: {"id":"3" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:08","updated_at":"2014/01/03 21:58:08"}
89
+ mysql.input: {"id":"10","user_name":"toyama7","created_at":"2014/01/03 21:58:18","updated_at":"2014/01/03 21:58:18"}
90
+ ```
91
+
92
+ then result becomes as below (indented):
93
+
94
+ ```sql
95
+ +-----+-----------+---------------------+---------------------+
96
+ | id | user_name | created_at | updated_at |
97
+ +-----+-----------+---------------------+---------------------+
98
+ | 1 | toyama7 | 2014-01-03 21:35:15 | 2014-01-03 21:58:03 |
99
+ | 2 | toyama7 | 2014-01-03 21:35:21 | 2014-01-03 21:58:06 |
100
+ | 3 | toyama7 | 2014-01-03 21:35:27 | 2014-01-03 21:58:08 |
101
+ | 10 | toyama7 | 2014-01-03 21:58:18 | 2014-01-03 21:58:18 |
102
+ +-----+-----------+---------------------+---------------------+
103
+ ```
104
+
105
+ if duplicate id , update username and updated_at
106
+
107
+
108
+ ## Configuration Example(bulk insert,fluentd key different column name)
109
+
110
+ ```
111
+ <match mysql.input>
112
+ type mysql_bulk
113
+ host localhost
114
+ database test_app_development
115
+ username root
116
+ password hogehoge
117
+ column_names id,user_name,created_at,updated_at
118
+ key_names id,user,created_date,updated_date
119
+ table users
120
+ flush_interval 10s
121
+ </match>
122
+ ```
123
+
124
+ Assume following input is coming:
125
+
126
+ ```js
127
+ mysql.input: {"user":"toyama","created_date":"2014/01/03 21:35:15","updated_date":"2014/01/03 21:35:15","dummy":"hogehoge"}
128
+ mysql.input: {"user":"toyama2","created_date":"2014/01/03 21:35:21","updated_date":"2014/01/03 21:35:21","dummy":"hogehoge"}
129
+ mysql.input: {"user":"toyama3","created_date":"2014/01/03 21:35:27","updated_date":"2014/01/03 21:35:27","dummy":"hogehoge"}
130
+ ```
131
+
132
+ then result becomes as below (indented):
133
+
134
+ ```sql
135
+ +-----+-----------+---------------------+---------------------+
136
+ | id | user_name | created_at | updated_at |
137
+ +-----+-----------+---------------------+---------------------+
138
+ | 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
139
+ | 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
140
+ | 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
141
+ +-----+-----------+---------------------+---------------------+
142
+ ```
143
+
144
+
145
+
146
+
147
+ ## spec
148
+
149
+ ```
150
+ bundle install
151
+ rake test
152
+ ```
153
+
154
+ ## todo
155
+
156
+ divide bulk insert(exsample 1000 per)
157
+
158
+
159
+ ## Contributing
160
+
161
+ 1. Fork it
162
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
163
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
164
+ 4. Push to the branch (`git push origin my-new-feature`)
165
+ 5. Create new [Pull Request](../../pull/new/master)
134
166
 
135
167
  ## Copyright
136
168
 
137
- * Copyright
138
- * Copyright(C) 2012- TAGOMORI Satoshi (tagomoris)
139
- * License
140
- * Apache License, Version 2.0
169
+ Copyright (c) 2016 Hiroshi Toyama. See [LICENSE](LICENSE) for details.
data/README_mysql.md ADDED
@@ -0,0 +1,140 @@
1
+ # fluent-plugin-mysql
2
+
3
+ ## Component
4
+
5
+ ### MysqlOutput
6
+
7
+ [Fluentd](http://fluentd.org) plugin to store mysql tables over SQL, to each columns per values, or to single column as json.
8
+
9
+ ## Configuration
10
+
11
+ ### MysqlOutput
12
+
13
+ MysqlOutput needs MySQL server's host/port/database/username/password, and INSERT format as SQL, or as table name and columns.
14
+
15
+ <match output.by.sql.*>
16
+ type mysql
17
+ host master.db.service.local
18
+ # port 3306 # default
19
+ database application_logs
20
+ username myuser
21
+ password mypass
22
+ key_names status,bytes,vhost,path,rhost,agent,referer
23
+ sql INSERT INTO accesslog (status,bytes,vhost,path,rhost,agent,referer) VALUES (?,?,?,?,?,?,?)
24
+ flush_interval 5s
25
+ </match>
26
+
27
+ <match output.by.names.*>
28
+ type mysql
29
+ host master.db.service.local
30
+ database application_logs
31
+ username myuser
32
+ password mypass
33
+ key_names status,bytes,vhost,path,rhost,agent,referer
34
+ table accesslog
35
+ # 'columns' names order must be same with 'key_names'
36
+ columns status,bytes,vhost,path,rhost,agent,referer
37
+ flush_interval 5s
38
+ </match>
39
+
40
+ Or, insert json into single column.
41
+
42
+ <match output.as.json.*>
43
+ type mysql
44
+ host master.db.service.local
45
+ database application_logs
46
+ username root
47
+ table accesslog
48
+ columns jsondata
49
+ format json
50
+ flush_interval 5s
51
+ </match>
52
+
53
+ To include time/tag into output, use `include_time_key` and `include_tag_key`, like this:
54
+
55
+ <match output.with.tag.and.time.*>
56
+ type mysql
57
+ host my.mysql.local
58
+ database anydatabase
59
+ username yourusername
60
+ password secret
61
+
62
+ include_time_key yes
63
+ ### default `time_format` is ISO-8601
64
+ # time_format %Y%m%d-%H%M%S
65
+ ### default `time_key` is 'time'
66
+ # time_key timekey
67
+
68
+ include_tag_key yes
69
+ ### default `tag_key` is 'tag'
70
+ # tag_key tagkey
71
+
72
+ table anydata
73
+ key_names time,tag,field1,field2,field3,field4
74
+ sql INSERT INTO baz (coltime,coltag,col1,col2,col3,col4) VALUES (?,?,?,?,?,?)
75
+ </match>
76
+
77
+ Or, for json:
78
+
79
+ <match output.with.tag.and.time.as.json.*>
80
+ type mysql
81
+ host database.local
82
+ database foo
83
+ username root
84
+
85
+ include_time_key yes
86
+ utc # with UTC timezone output (default: localtime)
87
+ time_format %Y%m%d-%H%M%S
88
+ time_key timeattr
89
+
90
+ include_tag_key yes
91
+ tag_key tagattr
92
+ table accesslog
93
+ columns jsondata
94
+ format json
95
+ </match>
96
+ #=> inserted json data into column 'jsondata' with addtional attribute 'timeattr' and 'tagattr'
97
+
98
+ ### JsonPath format
99
+
100
+ You can use [JsonPath](http://goessner.net/articles/JsonPath/) selectors as key_names, such as:
101
+
102
+ <match output.with.jsonpath.format.*>
103
+ type mysql
104
+ host database.local
105
+ database foo
106
+ username bar
107
+
108
+ include_time_key yes
109
+ utc
110
+ include_tag_key yes
111
+ table baz
112
+
113
+ format jsonpath
114
+ key_names time, tag, id, data.name, tags[0]
115
+ sql INSERT INTO baz (coltime,coltag,id,name,tag1) VALUES (?,?,?,?,?)
116
+ </match>
117
+
118
+ Which for a record like:
119
+
120
+ `{ 'id' => 15, 'data'=> {'name' => 'jsonpath' }, 'tags' => ['unit', 'simple'] }`
121
+
122
+ will generate the following insert values:
123
+
124
+ `('2012-12-17T01:23:45Z','test',15,'jsonpath','unit')`
125
+
126
+ ## Prerequisites
127
+
128
+ `fluent-plugin-mysql` uses `mysql2` gem, and `mysql2` links against `libmysqlclient`. See [Installing](https://github.com/brianmario/mysql2#installing) for its installation.
129
+
130
+ ## TODO
131
+
132
+ * implement 'tag_mapped'
133
+ * dynamic tag based table selection
134
+
135
+ ## Copyright
136
+
137
+ * Copyright
138
+ * Copyright(C) 2016- TAGOMORI Satoshi (tagomoris)
139
+ * License
140
+ * Apache License, Version 2.0
data/fluent-plugin-mysql.gemspec CHANGED
@@ -1,9 +1,9 @@
1
1
  # -*- encoding: utf-8 -*-
2
2
  Gem::Specification.new do |gem|
3
3
  gem.name = "fluent-plugin-mysql"
4
- gem.version = "0.0.8"
5
- gem.authors = ["TAGOMORI Satoshi"]
6
- gem.email = ["tagomoris@gmail.com"]
4
+ gem.version = "0.0.9"
5
+ gem.authors = ["TAGOMORI Satoshi", "Toyama Hiroshi"]
6
+ gem.email = ["tagomoris@gmail.com", "toyama0919@gmail.com"]
7
7
  gem.description = %q{fluent plugin to insert mysql as json(single column) or insert statement}
8
8
  gem.summary = %q{fluent plugin to insert mysql}
9
9
  gem.homepage = "https://github.com/tagomoris/fluent-plugin-mysql"
data/lib/fluent/plugin/out_mysql.rb CHANGED
@@ -33,6 +33,8 @@ class Fluent::MysqlOutput < Fluent::BufferedOutput
33
33
  def configure(conf)
34
34
  super
35
35
 
36
+ log.warn "[mysql] This plugin deprecated. You should use mysql_bulk."
37
+
36
38
  # TODO tag_mapped
37
39
 
38
40
  case @format
data/lib/fluent/plugin/out_mysql_bulk.rb ADDED
@@ -0,0 +1,127 @@
1
+ # -*- encoding : utf-8 -*-
2
+ module Fluent
3
+ class Fluent::MysqlBulkOutput < Fluent::BufferedOutput
4
+ Fluent::Plugin.register_output('mysql_bulk', self)
5
+
6
+ config_param :host, :string, default: '127.0.0.1'
7
+ config_param :port, :integer, default: 3306
8
+ config_param :database, :string
9
+ config_param :username, :string
10
+ config_param :password, :string, default: '', secret: true
11
+
12
+ config_param :column_names, :string
13
+ config_param :key_names, :string, default: nil
14
+ config_param :table, :string
15
+
16
+ config_param :on_duplicate_key_update, :bool, default: false
17
+ config_param :on_duplicate_update_keys, :string, default: nil
18
+
19
+ attr_accessor :handler
20
+
21
+ def initialize
22
+ super
23
+ require 'mysql2-cs-bind'
24
+ end
25
+
26
+ # Define `log` method for v0.10.42 or earlier
27
+ unless method_defined?(:log)
28
+ define_method("log") { $log }
29
+ end
30
+
31
+ def configure(conf)
32
+ super
33
+
34
+ if @column_names.nil?
35
+ fail Fluent::ConfigError, 'column_names MUST specified, but missing'
36
+ end
37
+
38
+ if @on_duplicate_key_update
39
+ if @on_duplicate_update_keys.nil?
40
+ fail Fluent::ConfigError, 'on_duplicate_key_update = true , on_duplicate_update_keys nil!'
41
+ end
42
+ @on_duplicate_update_keys = @on_duplicate_update_keys.split(',')
43
+
44
+ @on_duplicate_key_update_sql = ' ON DUPLICATE KEY UPDATE '
45
+ updates = []
46
+ @on_duplicate_update_keys.each do |update_column|
47
+ updates << "#{update_column} = VALUES(#{update_column})"
48
+ end
49
+ @on_duplicate_key_update_sql += updates.join(',')
50
+ end
51
+
52
+ @column_names = @column_names.split(',')
53
+ @key_names = @key_names.nil? ? @column_names : @key_names.split(',')
54
+ end
55
+
56
+ def start
57
+ super
58
+ result = client.xquery("SHOW COLUMNS FROM #{@table}")
59
+ @max_lengths = []
60
+ @column_names.each do |column|
61
+ info = result.select { |x| x['Field'] == column }.first
62
+ r = /(char|varchar)\(([\d]+)\)/
63
+ begin
64
+ max_length = info['Type'].scan(r)[0][1].to_i
65
+ rescue
66
+ max_length = nil
67
+ end
68
+ @max_lengths << max_length
69
+ end
70
+ end
71
+
72
+ def shutdown
73
+ super
74
+ end
75
+
76
+ def format(tag, time, record)
77
+ [tag, time, format_proc.call(tag, time, record)].to_msgpack
78
+ end
79
+
80
+ def client
81
+ Mysql2::Client.new(
82
+ host: @host,
83
+ port: @port,
84
+ username: @username,
85
+ password: @password,
86
+ database: @database,
87
+ flags: Mysql2::Client::MULTI_STATEMENTS
88
+ )
89
+ end
90
+
91
+ def write(chunk)
92
+ @handler = client
93
+ values = []
94
+ values_template = "(#{ @column_names.map { |key| '?' }.join(',') })"
95
+ chunk.msgpack_each do |tag, time, data|
96
+ values << Mysql2::Client.pseudo_bind(values_template, data)
97
+ end
98
+ sql = "INSERT INTO #{@table} (#{@column_names.join(',')}) VALUES #{values.join(',')}"
99
+ sql += @on_duplicate_key_update_sql if @on_duplicate_key_update
100
+
101
+ log.info "bulk insert values size => #{values.size}"
102
+ @handler.xquery(sql)
103
+ @handler.close
104
+ end
105
+
106
+ private
107
+
108
+ def format_proc
109
+ proc do |tag, time, record|
110
+ values = []
111
+ @key_names.each_with_index do |key, i|
112
+ if key == '${time}'
113
+ value = Time.at(time).strftime('%Y-%m-%d %H:%M:%S')
114
+ else
115
+ if @max_lengths[i].nil? || record[key].nil?
116
+ value = record[key]
117
+ else
118
+ value = record[key].slice(0, @max_lengths[i])
119
+ end
120
+ end
121
+ values << value
122
+ end
123
+ values
124
+ end
125
+ end
126
+ end
127
+ end
data/test/helper.rb CHANGED
@@ -23,6 +23,7 @@ unless ENV.has_key?('VERBOSE')
23
23
  end
24
24
 
25
25
  require 'fluent/plugin/out_mysql'
26
+ require 'fluent/plugin/out_mysql_bulk'
26
27
 
27
28
  class Test::Unit::TestCase
28
29
  end
data/test/plugin/test_out_mysql_bulk.rb ADDED
@@ -0,0 +1,157 @@
1
+ require 'helper'
2
+ require 'mysql2-cs-bind'
3
+
4
+ class MysqlBulkOutputTest < Test::Unit::TestCase
5
+ def setup
6
+ Fluent::Test.setup
7
+ end
8
+
9
+ def create_driver(conf = CONFIG, tag = 'test')
10
+ d = Fluent::Test::BufferedOutputTestDriver.new(Fluent::MysqlBulkOutput, tag).configure(conf)
11
+ d.instance.instance_eval {
12
+ def client
13
+ obj = Object.new
14
+ obj.instance_eval {
15
+ def xquery(*args); [1]; end
16
+ def close; true; end
17
+ }
18
+ obj
19
+ end
20
+ }
21
+ d
22
+ end
23
+
24
+ def test_configure_error
25
+ assert_raise(Fluent::ConfigError) do
26
+ d = create_driver %[
27
+ host localhost
28
+ database test_app_development
29
+ username root
30
+ password hogehoge
31
+ table users
32
+ on_duplicate_key_update true
33
+ on_duplicate_update_keys user_name,updated_at
34
+ flush_interval 10s
35
+ ]
36
+ end
37
+
38
+ assert_raise(Fluent::ConfigError) do
39
+ d = create_driver %[
40
+ host localhost
41
+ database test_app_development
42
+ username root
43
+ password hogehoge
44
+ column_names id,user_name,created_at,updated_at
45
+ table users
46
+ on_duplicate_key_update true
47
+ flush_interval 10s
48
+ ]
49
+ end
50
+
51
+ assert_raise(Fluent::ConfigError) do
52
+ d = create_driver %[
53
+ host localhost
54
+ username root
55
+ password hogehoge
56
+ column_names id,user_name,created_at,updated_at
57
+ table users
58
+ on_duplicate_key_update true
59
+ on_duplicate_update_keys user_name,updated_at
60
+ flush_interval 10s
61
+ ]
62
+ end
63
+ end
64
+
65
+ def test_configure
66
+ # not define format(default csv)
67
+ assert_nothing_raised(Fluent::ConfigError) do
68
+ d = create_driver %[
69
+ host localhost
70
+ database test_app_development
71
+ username root
72
+ password hogehoge
73
+ column_names id,user_name,created_at,updated_at
74
+ table users
75
+ on_duplicate_key_update true
76
+ on_duplicate_update_keys user_name,updated_at
77
+ flush_interval 10s
78
+ ]
79
+ end
80
+
81
+ assert_nothing_raised(Fluent::ConfigError) do
82
+ d = create_driver %[
83
+ database test_app_development
84
+ username root
85
+ password hogehoge
86
+ column_names id,user_name,created_at,updated_at
87
+ table users
88
+ ]
89
+ end
90
+
91
+ assert_nothing_raised(Fluent::ConfigError) do
92
+ d = create_driver %[
93
+ database test_app_development
94
+ username root
95
+ password hogehoge
96
+ column_names id,user_name,created_at,updated_at
97
+ table users
98
+ on_duplicate_key_update true
99
+ on_duplicate_update_keys user_name,updated_at
100
+ ]
101
+ end
102
+
103
+ assert_nothing_raised(Fluent::ConfigError) do
104
+ d = create_driver %[
105
+ database test_app_development
106
+ username root
107
+ password hogehoge
108
+ column_names id,user_name,created_at,updated_at
109
+ key_names id,user,created_date,updated_date
110
+ table users
111
+ on_duplicate_key_update true
112
+ on_duplicate_update_keys user_name,updated_at
113
+ ]
114
+ end
115
+ end
116
+
117
+ def test_variables
118
+ d = create_driver %[
119
+ database test_app_development
120
+ username root
121
+ password hogehoge
122
+ column_names id,user_name,created_at,updated_at
123
+ table users
124
+ on_duplicate_key_update true
125
+ on_duplicate_update_keys user_name,updated_at
126
+ ]
127
+
128
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
129
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.column_names
130
+ assert_equal " ON DUPLICATE KEY UPDATE user_name = VALUES(user_name),updated_at = VALUES(updated_at)", d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
131
+
132
+ d = create_driver %[
133
+ database test_app_development
134
+ username root
135
+ password hogehoge
136
+ column_names id,user_name,created_at,updated_at
137
+ table users
138
+ ]
139
+
140
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
141
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.column_names
142
+ assert_nil d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
143
+
144
+ d = create_driver %[
145
+ database test_app_development
146
+ username root
147
+ password hogehoge
148
+ key_names id,user_name,created_at,updated_at
149
+ column_names id,user,created_date,updated_date
150
+ table users
151
+ ]
152
+
153
+ assert_equal ['id','user_name','created_at','updated_at'], d.instance.key_names
154
+ assert_equal ['id','user','created_date','updated_date'], d.instance.column_names
155
+ assert_nil d.instance.instance_variable_get(:@on_duplicate_key_update_sql)
156
+ end
157
+ end
metadata CHANGED
@@ -1,14 +1,15 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-mysql
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.8
4
+ version: 0.0.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - TAGOMORI Satoshi
8
+ - Toyama Hiroshi
8
9
  autorequire:
9
10
  bindir: bin
10
11
  cert_chain: []
11
- date: 2015-07-31 00:00:00.000000000 Z
12
+ date: 2016-01-08 00:00:00.000000000 Z
12
13
  dependencies:
13
14
  - !ruby/object:Gem::Dependency
14
15
  name: fluentd
@@ -83,6 +84,7 @@ dependencies:
83
84
  description: fluent plugin to insert mysql as json(single column) or insert statement
84
85
  email:
85
86
  - tagomoris@gmail.com
87
+ - toyama0919@gmail.com
86
88
  executables: []
87
89
  extensions: []
88
90
  extra_rdoc_files: []
@@ -91,11 +93,14 @@ files:
91
93
  - Gemfile
92
94
  - LICENSE.txt
93
95
  - README.md
96
+ - README_mysql.md
94
97
  - Rakefile
95
98
  - fluent-plugin-mysql.gemspec
96
99
  - lib/fluent/plugin/out_mysql.rb
100
+ - lib/fluent/plugin/out_mysql_bulk.rb
97
101
  - test/helper.rb
98
102
  - test/plugin/test_out_mysql.rb
103
+ - test/plugin/test_out_mysql_bulk.rb
99
104
  homepage: https://github.com/tagomoris/fluent-plugin-mysql
100
105
  licenses:
101
106
  - APLv2
@@ -116,11 +121,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
116
121
  version: '0'
117
122
  requirements: []
118
123
  rubyforge_project:
119
- rubygems_version: 2.4.5
124
+ rubygems_version: 2.4.2
120
125
  signing_key:
121
126
  specification_version: 4
122
127
  summary: fluent plugin to insert mysql
123
128
  test_files:
124
129
  - test/helper.rb
125
130
  - test/plugin/test_out_mysql.rb
126
- has_rdoc:
131
+ - test/plugin/test_out_mysql_bulk.rb