fluent-plugin-mysql-select-insert 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: a43d623e098edb37e7977ad86d9dfdfdc7748c129f0dc6127f773d755d19f8e0
4
+ data.tar.gz: 89c6745c98fc33f0961b476cfa668129a4a07c4bf6e5fe2d21540e64950774fb
5
+ SHA512:
6
+ metadata.gz: d0dab5f92c67d336c6ec60505d4755e21ec941775a2cac4d8cbfbd7aeb3e068652ad9bbdf52393ebee3b5a106758b01a5eeccc62ecb189155b4ea609f9360e92
7
+ data.tar.gz: 995963147d81ef09a96111e9b112d9177b127ebd3459a97b71c187ddd71392670db9974e6cb2126afafadfeb4755a14339811da8ff125bfaba323b4f66321260
@@ -0,0 +1,8 @@
1
+ sudo: false
2
+ language: ruby
3
+ cache: bundler
4
+ rvm:
5
+ - 2.5.0
6
+ before_install:
7
+ - gem install bundler -v 1.16.1
8
+ - mysql -e 'CREATE DATABASE IF NOT EXISTS fluent_plugin_mysql_select_insert;'
data/Gemfile ADDED
@@ -0,0 +1,5 @@
1
+ source "https://rubygems.org"
2
+
3
+ git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
4
+
5
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,202 @@
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
@@ -0,0 +1,108 @@
1
+ # fluent-plugin-mysql-select-insert
2
+
3
+ [Fluentd](https://fluentd.org/) output plugin to insert records by SELECT query.
4
+ You can select records using events data and join multiple tables.
5
+
6
+ ## Installation
7
+
8
+ ### RubyGems
9
+
10
+ ```
11
+ $ gem install fluent-plugin-mysql-select-insert
12
+ ```
13
+
14
+ ### Bundler
15
+
16
+ Add following line to your Gemfile:
17
+
18
+ ```ruby
19
+ gem "fluent-plugin-mysql-select-insert"
20
+ ```
21
+
22
+ And then execute:
23
+
24
+ ```
25
+ $ bundle
26
+ ```
27
+
28
+ ## Configuration
29
+
30
+ Parameter | Description | Default
31
+ ----------|----------------|-------------
32
+ host | Database host | 127.0.0.1
33
+ port | Database port | 3306
34
+ database | Database name | (required)
35
+ username | Database user | (required)
36
+ password | Database password | (empty string)
37
+ table | Table into which records are inserted | (required)
38
+ select_query | SELECT query without WHERE clause to insert records | (required)
39
+ condition_column | Column used in WHERE clause to compare with values of 'condition_key' in events | (required)
40
+ condition_key | Key whose value is used in WHERE clause to compare with 'condition_column' | (required)
41
+ extra_condition | Extra condition used in WHERE clause. You can specify placeholders like 'col1 = ? AND col2 = ?, ${tag[0]}, ${tag[1]}'. The metadata is extracted in the second or following arguments. | (empty array)
42
+ ignore | Add 'IGNORE' modifier to INSERT statement | false
43
+
44
+ ## Example Configuration
45
+
46
+ ```
47
+ <match pattern>
48
+ @type mysql_select_insert
49
+ database fluent_plugin_mysql_select_insert
50
+ username root
51
+
52
+ table accessed_users
53
+
54
+ select_query "SELECT
55
+ users.id
56
+ FROM
57
+ users
58
+ INNER JOIN
59
+ devices
60
+ ON
61
+ devices.id = users.device_id"
62
+
63
+ condition_key uuid
64
+ condition_column devices.uuid
65
+ extra_condition "app_id = ?, ${app_id}"
66
+
67
+ ignore true
68
+
69
+ <buffer app_id>
70
+ </buffer>
71
+ </match>
72
+ ```
73
+
74
+ Assuming that the following data comes in:
75
+
76
+ ```
77
+ pattern {"uuid":"03449258-29ce-403c-900a-a2c6ea1d09a2","app_id":1}
78
+ pattern {"uuid":"11aebc82-b661-44ab-951d-d1618058699a","app_id":1}
79
+ ```
80
+
81
+ The INSERT statement will be like below:
82
+
83
+ ```
84
+ INSERT IGNORE INTO
85
+ accessed_users
86
+ SELECT
87
+ users.id
88
+ FROM
89
+ users
90
+ INNER JOIN
91
+ devices
92
+ ON
93
+ devices.id = users.device_id
94
+ WHERE
95
+ uuid IN (
96
+ '03449258-29ce-403c-900a-a2c6ea1d09a2',
97
+ '11aebc82-b661-44ab-951d-d1618058699a'
98
+ )
99
+ AND app_id = 1
100
+ ;
101
+ ```
102
+
103
+
104
+ ## Copyright
105
+
106
+ * Copyright(c) 2018- abicky
107
+ * License
108
+ * Apache License, Version 2.0
@@ -0,0 +1,13 @@
1
+ require "bundler"
2
+ Bundler::GemHelper.install_tasks
3
+
4
+ require "rake/testtask"
5
+
6
+ Rake::TestTask.new(:test) do |t|
7
+ t.libs.push("lib", "test")
8
+ t.test_files = FileList["test/**/test_*.rb"]
9
+ t.verbose = true
10
+ t.warning = true
11
+ end
12
+
13
+ task default: [:test]
@@ -0,0 +1,28 @@
1
+ lib = File.expand_path("../lib", __FILE__)
2
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
3
+
4
+ Gem::Specification.new do |spec|
5
+ spec.name = "fluent-plugin-mysql-select-insert"
6
+ spec.version = "0.1.0"
7
+ spec.authors = ["abicky"]
8
+ spec.email = ["takeshi.arabiki@gmail.com"]
9
+
10
+ spec.summary = %q{Fluentd output plugin to insert records by SELECT query.}
11
+ spec.description = %q{You can select records using events data and join multiple tables.}
12
+ spec.homepage = "https://github.com/abicky/fluent-plugin-mysql-select-insert"
13
+ spec.license = "Apache-2.0"
14
+
15
+ test_files, files = `git ls-files -z`.split("\x0").partition do |f|
16
+ f.match(%r{^(test|spec|features)/})
17
+ end
18
+ spec.files = files
19
+ spec.executables = files.grep(%r{^bin/}) { |f| File.basename(f) }
20
+ spec.test_files = test_files
21
+ spec.require_paths = ["lib"]
22
+
23
+ spec.add_development_dependency "bundler", "~> 1.16"
24
+ spec.add_development_dependency "rake", "~> 12.0"
25
+ spec.add_development_dependency "test-unit", "~> 3.0"
26
+ spec.add_runtime_dependency "fluentd", [">= 0.14.10", "< 2"]
27
+ spec.add_runtime_dependency "mysql2-cs-bind"
28
+ end
@@ -0,0 +1,100 @@
1
+ #
2
+ # Copyright 2018- abicky
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+
16
+ require "fluent/plugin/output"
17
+ require "mysql2"
18
+
19
+ module Fluent
20
+ module Plugin
21
+ class MysqlSelectInsertOutput < Output
22
+ Fluent::Plugin.register_output("mysql_select_insert", self)
23
+
24
+ config_param :host, :string, default: "127.0.0.1",
25
+ desc: "Database host"
26
+ config_param :port, :integer, default: 3306,
27
+ desc: "Database port"
28
+ config_param :database, :string,
29
+ desc: "Database name"
30
+ config_param :username, :string,
31
+ desc: "Database user"
32
+ config_param :password, :string, default: "", secret: true,
33
+ desc: "Database password"
34
+ config_param :table, :string,
35
+ desc: "Table into which records are inserted"
36
+
37
+ config_param :select_query, :string,
38
+ desc: "SELECT query without WHERE clause to insert records"
39
+ config_param :condition_column, :string,
40
+ desc: "Column used in WHERE clause to compare with values of 'condition_key' in events"
41
+ config_param :condition_key, :string,
42
+ desc: "Key whose value is used in WHERE clause to compare with 'condition_column'"
43
+ config_param :extra_condition, :array, value_type: :string, default: [],
44
+ desc: "Extra condition used in WHERE clause. You can specify placeholders like 'col1 = ? AND col2 = ?, ${tag[0]}, ${tag[1]}'. The metadata is extracted in the second or following arguments."
45
+
46
+ config_param :ignore, :bool, default: false,
47
+ desc: "Add 'IGNORE' modifier to INSERT statement"
48
+
49
+ config_section :buffer do
50
+ config_set_default :chunk_limit_records, 1000
51
+ end
52
+
53
+ def configure(conf)
54
+ super
55
+
56
+ if select_query =~ /\bwhere\b/i
57
+ fail Fluent::ConfigError, "You can't specify WHERE clause in 'select_query'"
58
+ end
59
+ end
60
+
61
+ def write(chunk)
62
+ client = new_client
63
+ condition_values = []
64
+ chunk.msgpack_each do |time, record|
65
+ condition_values << "'#{client.escape(record[condition_key])}'"
66
+ end
67
+ sql = <<~SQL
68
+ INSERT #{"IGNORE" if ignore} INTO `#{table}` #{select_query}
69
+ WHERE #{condition_column} IN (#{condition_values.join(",")})
70
+ SQL
71
+
72
+ sql << " AND (#{extra_condition[0]})" unless extra_condition.empty?
73
+ if extra_condition.size < 2
74
+ client.query(sql)
75
+ else
76
+ require "mysql2-cs-bind"
77
+ bound_params = extra_condition[1..-1].map { |c| extract_placeholders(c, chunk.metadata) }
78
+ client.xquery(sql, bound_params)
79
+ end
80
+ client.close
81
+ end
82
+
83
+ def multi_workers_ready?
84
+ true
85
+ end
86
+
87
+ private
88
+
89
+ def new_client
90
+ Mysql2::Client.new(
91
+ host: @host,
92
+ port: @port,
93
+ username: @username,
94
+ password: @password,
95
+ database: @database,
96
+ )
97
+ end
98
+ end
99
+ end
100
+ end
@@ -0,0 +1,8 @@
1
+ $LOAD_PATH.unshift(File.expand_path("../../", __FILE__))
2
+ require "test-unit"
3
+ require "fluent/test"
4
+ require "fluent/test/driver/output"
5
+ require "fluent/test/helpers"
6
+
7
+ Test::Unit::TestCase.include(Fluent::Test::Helpers)
8
+ Test::Unit::TestCase.extend(Fluent::Test::Helpers)
@@ -0,0 +1,256 @@
1
+ require "helper"
2
+ require "fluent/plugin/out_mysql_select_insert"
3
+
4
+ class MysqlSelectInsertOutputTest < Test::Unit::TestCase
5
+ class << self
6
+ def startup
7
+ new_client.query(<<~SQL)
8
+ CREATE TABLE IF NOT EXISTS `devices` (
9
+ `id` int(10) unsigned NOT NULL,
10
+ `app_id` int(10) unsigned NOT NULL,
11
+ `uuid` char(36) NOT NULL,
12
+ PRIMARY KEY (`id`),
13
+ UNIQUE KEY (`app_id`, `uuid`)
14
+ );
15
+ CREATE TABLE IF NOT EXISTS `users` (
16
+ `id` int(10) unsigned NOT NULL,
17
+ `device_id` int(10) unsigned NOT NULL,
18
+ PRIMARY KEY (`id`)
19
+ );
20
+ CREATE TABLE IF NOT EXISTS `accessed_users` (
21
+ `user_id` int(10) unsigned NOT NULL,
22
+ PRIMARY KEY (`user_id`)
23
+ );
24
+ SQL
25
+ end
26
+
27
+ def shutdown
28
+ new_client.query(<<~SQL)
29
+ DROP TABLE IF EXISTS `devices`;
30
+ DROP TABLE IF EXISTS `users`;
31
+ DROP TABLE IF EXISTS `accessed_users`;
32
+ SQL
33
+ end
34
+
35
+ def new_client
36
+ Mysql2::Client.new(
37
+ username: "root",
38
+ database: "fluent_plugin_mysql_select_insert",
39
+ flags: Mysql2::Client::MULTI_STATEMENTS,
40
+ )
41
+ end
42
+ end
43
+
44
+ setup do
45
+ Fluent::Test.setup
46
+
47
+ self.class.new_client.query(<<~SQL)
48
+ INSERT INTO `devices` (`id`, `app_id`, `uuid`) VALUES
49
+ (1, 1, '03449258-29ce-403c-900a-a2c6ea1d09a2'),
50
+ (2, 1, '11aebc82-b661-44ab-951d-d1618058699a'),
51
+ (3, 2, '2c9a7bb7-aec2-441c-ade0-edf31fdacb43'),
52
+ (4, 2, '3eae0b4b-bdd1-4c82-b04a-5335535f5b7b');
53
+ INSERT INTO `users` (`id`, `device_id`) VALUES
54
+ (1001, 1),
55
+ (1002, 2),
56
+ (1003, 3),
57
+ (1004, 4);
58
+ SQL
59
+ end
60
+
61
+ teardown do
62
+ self.class.new_client.query(<<~SQL)
63
+ TRUNCATE TABLE `devices`;
64
+ TRUNCATE TABLE `users`;
65
+ TRUNCATE TABLE `accessed_users`;
66
+ SQL
67
+ end
68
+
69
+ CONFIG = <<~CONF
70
+ database fluent_plugin_mysql_select_insert
71
+ username root
72
+
73
+ table accessed_users
74
+
75
+ select_query "SELECT
76
+ users.id
77
+ FROM
78
+ users
79
+ INNER JOIN
80
+ devices
81
+ ON
82
+ devices.id = users.device_id"
83
+
84
+ condition_key uuid
85
+ condition_column devices.uuid
86
+ CONF
87
+
88
+ test "write" do
89
+ d = create_driver
90
+ d.run(default_tag: "tag") do
91
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 1 })
92
+ d.feed(event_time, { "uuid" => "11aebc82-b661-44ab-951d-d1618058699a", "app_id" => 1 })
93
+ d.feed(event_time, { "uuid" => "2c9a7bb7-aec2-441c-ade0-edf31fdacb43", "app_id" => 2 })
94
+ end
95
+
96
+ result = self.class.new_client.query('SELECT * FROM `accessed_users`')
97
+ assert_equal [{ "user_id" => 1001 }, { "user_id" => 1002 }, { "user_id" => 1003 }], result.to_a
98
+ end
99
+
100
+ sub_test_case "'extra_condition' is specified" do
101
+ test "write" do
102
+ d = create_driver(<<~CONF)
103
+ database fluent_plugin_mysql_select_insert
104
+ username root
105
+
106
+ table accessed_users
107
+
108
+ select_query "SELECT
109
+ users.id
110
+ FROM
111
+ users
112
+ INNER JOIN
113
+ devices
114
+ ON
115
+ devices.id = users.device_id"
116
+
117
+ condition_key uuid
118
+ condition_column devices.uuid
119
+ extra_condition "app_id = 1"
120
+
121
+ <buffer>
122
+ </buffer>
123
+ CONF
124
+ d.run(default_tag: "tag") do
125
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 1 })
126
+ d.feed(event_time, { "uuid" => "2c9a7bb7-aec2-441c-ade0-edf31fdacb43", "app_id" => 2 })
127
+ end
128
+
129
+ result = self.class.new_client.query('SELECT * FROM `accessed_users`')
130
+ assert_equal [{ "user_id" => 1001 }], result.to_a
131
+ end
132
+ end
133
+
134
+ sub_test_case "'extra_condition' is specified with placeholders" do
135
+ config = <<~CONF
136
+ database fluent_plugin_mysql_select_insert
137
+ username root
138
+
139
+ table accessed_users
140
+
141
+ select_query "SELECT
142
+ users.id
143
+ FROM
144
+ users
145
+ INNER JOIN
146
+ devices
147
+ ON
148
+ devices.id = users.device_id"
149
+
150
+ condition_key uuid
151
+ condition_column devices.uuid
152
+ extra_condition "app_id = ?, ${app_id}"
153
+
154
+ <buffer app_id>
155
+ </buffer>
156
+ CONF
157
+
158
+ test "write with valid data" do
159
+ d = create_driver(config)
160
+ d.run(default_tag: "tag") do
161
+ # Correct app_id
162
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 1 })
163
+ end
164
+ result = self.class.new_client.query('SELECT * FROM `accessed_users`')
165
+ assert_equal [{ "user_id" => 1001 }], result.to_a
166
+ end
167
+
168
+ test "write with invalid data" do
169
+ d = create_driver(config)
170
+ d.run(default_tag: "tag") do
171
+ # Different app_id
172
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 2 })
173
+ end
174
+ result = self.class.new_client.query('SELECT * FROM `accessed_users`')
175
+ assert_equal [], result.to_a
176
+ end
177
+ end
178
+
179
+ sub_test_case "'ignore' is specified" do
180
+ base_config = <<~CONF
181
+ database fluent_plugin_mysql_select_insert
182
+ username root
183
+
184
+ table accessed_users
185
+
186
+ select_query "SELECT
187
+ users.id
188
+ FROM
189
+ users
190
+ INNER JOIN
191
+ devices
192
+ ON
193
+ devices.id = users.device_id"
194
+
195
+ condition_key uuid
196
+ condition_column devices.uuid
197
+ CONF
198
+
199
+ test "write with ignore false" do
200
+ # XXX: "Mysql2::Error: Table 'fluent_plugin_mysql_select_insert.accessed_users' doesn't exist" occurs without sleep
201
+ sleep 1
202
+ self.class.new_client.query("INSERT INTO `accessed_users` VALUES (1001)")
203
+
204
+ d = create_driver("#{base_config}\n ignore false")
205
+ assert_raise Mysql2::Error do
206
+ d.run(default_tag: "tag") do
207
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 1 })
208
+ end
209
+ end
210
+ end
211
+
212
+ test "write with ignore true" do
213
+ self.class.new_client.query("INSERT INTO `accessed_users` VALUES (1001)")
214
+
215
+ d = create_driver("#{base_config}\n ignore true")
216
+ assert_nothing_raised do
217
+ d.run(default_tag: "tag") do
218
+ d.feed(event_time, { "uuid" => "03449258-29ce-403c-900a-a2c6ea1d09a2", "app_id" => 1 })
219
+ end
220
+ end
221
+ end
222
+ end
223
+
224
+ sub_test_case "WHERE clause is specified in 'select_query'" do
225
+ test "configure" do
226
+ assert_raise Fluent::ConfigError do
227
+ create_driver(<<~CONF)
228
+ database fluent_plugin_mysql_select_insert
229
+ username root
230
+
231
+ table accessed_users
232
+
233
+ select_query "SELECT
234
+ users.id
235
+ FROM
236
+ users
237
+ INNER JOIN
238
+ devices
239
+ ON
240
+ devices.id = users.device_id
241
+ WHERE
242
+ app_id = 1"
243
+
244
+ condition_key uuid
245
+ condition_column devices.uuid
246
+ CONF
247
+ end
248
+ end
249
+ end
250
+
251
+ private
252
+
253
+ def create_driver(conf = CONFIG)
254
+ Fluent::Test::Driver::Output.new(Fluent::Plugin::MysqlSelectInsertOutput).configure(conf)
255
+ end
256
+ end
metadata ADDED
@@ -0,0 +1,131 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: fluent-plugin-mysql-select-insert
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - abicky
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2018-04-27 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: bundler
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.16'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.16'
27
+ - !ruby/object:Gem::Dependency
28
+ name: rake
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '12.0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '12.0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: test-unit
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '3.0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '3.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: fluentd
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: 0.14.10
62
+ - - "<"
63
+ - !ruby/object:Gem::Version
64
+ version: '2'
65
+ type: :runtime
66
+ prerelease: false
67
+ version_requirements: !ruby/object:Gem::Requirement
68
+ requirements:
69
+ - - ">="
70
+ - !ruby/object:Gem::Version
71
+ version: 0.14.10
72
+ - - "<"
73
+ - !ruby/object:Gem::Version
74
+ version: '2'
75
+ - !ruby/object:Gem::Dependency
76
+ name: mysql2-cs-bind
77
+ requirement: !ruby/object:Gem::Requirement
78
+ requirements:
79
+ - - ">="
80
+ - !ruby/object:Gem::Version
81
+ version: '0'
82
+ type: :runtime
83
+ prerelease: false
84
+ version_requirements: !ruby/object:Gem::Requirement
85
+ requirements:
86
+ - - ">="
87
+ - !ruby/object:Gem::Version
88
+ version: '0'
89
+ description: You can select records using events data and join multiple tables.
90
+ email:
91
+ - takeshi.arabiki@gmail.com
92
+ executables: []
93
+ extensions: []
94
+ extra_rdoc_files: []
95
+ files:
96
+ - ".travis.yml"
97
+ - Gemfile
98
+ - LICENSE
99
+ - README.md
100
+ - Rakefile
101
+ - fluent-plugin-mysql-select-insert.gemspec
102
+ - lib/fluent/plugin/out_mysql_select_insert.rb
103
+ - test/helper.rb
104
+ - test/plugin/test_out_mysql_select_insert.rb
105
+ homepage: https://github.com/abicky/fluent-plugin-mysql-select-insert
106
+ licenses:
107
+ - Apache-2.0
108
+ metadata: {}
109
+ post_install_message:
110
+ rdoc_options: []
111
+ require_paths:
112
+ - lib
113
+ required_ruby_version: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ required_rubygems_version: !ruby/object:Gem::Requirement
119
+ requirements:
120
+ - - ">="
121
+ - !ruby/object:Gem::Version
122
+ version: '0'
123
+ requirements: []
124
+ rubyforge_project:
125
+ rubygems_version: 2.7.3
126
+ signing_key:
127
+ specification_version: 4
128
+ summary: Fluentd output plugin to insert records by SELECT query.
129
+ test_files:
130
+ - test/helper.rb
131
+ - test/plugin/test_out_mysql_select_insert.rb