fluent-plugin-mysql-2 0.3.7

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b08ae8a41a2889d1785aec8b5441392ffbcd249e5c38d78e257dd27e64465d2b
+  data.tar.gz: 6853750e68fffe251f60737fc2f8773201d976faafa71d13db0b9cec6eee87e2
+SHA512:
+  metadata.gz: 68c4733f8c8ac2f1f585f09316c911611722f7b7c9486f658de414f04dc4be35a0bf17a6574048aa4334d58285c7f707a466680f421a9206ee9e54f4f9165818
+  data.tar.gz: 928424531e1218fc9be1c001d65faabc8d2fff84c3694b9fe9b35a4fce803682af344390c579b3016f41a2c5cb2edee6319e688439bf277c6f482a355da56691

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+test.conf

data/.travis.yml

@@ -0,0 +1,14 @@
+language: ruby
+
+rvm:
+  - 2.1
+  - 2.2.3
+  - 2.3.0
+
+gemfile:
+  - Gemfile
+
+before_install:
+  - gem update bundler
+
+script: 'bundle exec rake test'

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-mysql.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2012- TAGOMORI Satoshi
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,350 @@
+
+# fluent-plugin-mysql, a plugin for [Fluentd](http://fluentd.org) [![Build Status](https://secure.travis-ci.org/tagomoris/fluent-plugin-mysql.png?branch=master)](http://travis-ci.org/tagomoris/fluent-plugin-mysql)
+
+fluent plugin mysql bulk insert is high performance and on duplicate key update respond.
+
+## Note
+fluent-plugin-mysql-bulk merged this repository.
+
+[mysql plugin](README_mysql.md) is deprecated. You should use mysql_bulk.
+
+v0.1.5 only supports fluentd-0.12.X and v0.2.0 only supports fluentd-0.14.X.
+
+## Parameters
+
+param|value
+--------|------
+host|database host(default: 127.0.0.1)
+port|database port(default: 3306)
+database|database name(require)
+username|user(require)
+password|password(default: blank)
+sslkey|path to client key(default: nil)
+sslcert|path to client cert(default: nil)
+sslca|path to ca cert(default: nil)
+sslcapath|path to ca certs(default: nil)
+sslcipher|ssl cipher(default: nil)
+sslverify|verify server certificate(default: nil)
+column_names|bulk insert column (require)
+key_names|value key names, ${time} is placeholder Time.at(time).strftime("%Y-%m-%d %H:%M:%S") (default : column_names)
+json_key_names|Key names which store data as json, comma separator.
+unixtimestamp_key_names|Key names which store data as datetime from unix time stamp
+table|bulk insert table (require)
+on_duplicate_key_update|on duplicate key update enable (true:false)
+on_duplicate_update_keys|on duplicate key update column, comma separator
+transaction_isolation_level|set transaction isolation level(default: nil)
+
+## Configuration Example(bulk insert)
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at,updated_at
+  table users
+  flush_interval 10s
+</match>
+```
+
+Assume following input is coming:
+
+```js
+mysql.input: {"user_name":"toyama","created_at":"2014/01/03 21:35:15","updated_at":"2014/01/03 21:35:15","dummy":"hogehoge"}
+mysql.input: {"user_name":"toyama2","created_at":"2014/01/03 21:35:21","updated_at":"2014/01/03 21:35:21","dummy":"hogehoge"}
+mysql.input: {"user_name":"toyama3","created_at":"2014/01/03 21:35:27","updated_at":"2014/01/03 21:35:27","dummy":"hogehoge"}
+```
+
+then result becomes as below (indented):
+
+```sql
++-----+-----------+---------------------+---------------------+
+| id  | user_name | created_at          | updated_at          |
++-----+-----------+---------------------+---------------------+
+| 1   | toyama    | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
+| 2   | toyama2   | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
+| 3   | toyama3   | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
++-----+-----------+---------------------+---------------------+
+```
+
+running query
+
+```sql
+INSERT INTO users (id,user_name,created_at,updated_at) VALUES (NULL,'toyama','2014/01/03 21:35:15','2014/01/03 21:35:15'),(NULL,'toyama2','2014/01/03 21:35:21','2014/01/03 21:35:21')
+```
+
+## Configuration Example(bulk insert , if duplicate error record update)
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at,updated_at
+  table users
+  on_duplicate_key_update true
+  on_duplicate_update_keys user_name,updated_at
+  flush_interval 60s
+</match>
+```
+
+Assume following input is coming:
+
+```js
+mysql.input: {"id":"1" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:03","updated_at":"2014/01/03 21:58:03"}
+mysql.input: {"id":"2" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:06","updated_at":"2014/01/03 21:58:06"}
+mysql.input: {"id":"3" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:08","updated_at":"2014/01/03 21:58:08"}
+mysql.input: {"id":"10","user_name":"toyama7","created_at":"2014/01/03 21:58:18","updated_at":"2014/01/03 21:58:18"}
+```
+
+then result becomes as below (indented):
+
+```sql
++-----+-----------+---------------------+---------------------+
+| id  | user_name | created_at          | updated_at          |
++-----+-----------+---------------------+---------------------+
+|   1 | toyama7   | 2014-01-03 21:35:15 | 2014-01-03 21:58:03 |
+|   2 | toyama7   | 2014-01-03 21:35:21 | 2014-01-03 21:58:06 |
+|   3 | toyama7   | 2014-01-03 21:35:27 | 2014-01-03 21:58:08 |
+|  10 | toyama7   | 2014-01-03 21:58:18 | 2014-01-03 21:58:18 |
++-----+-----------+---------------------+---------------------+
+```
+
+if duplicate id , update username and updated_at
+
+
+## Configuration Example(bulk insert,fluentd key different column name)
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at,updated_at
+  key_names id,user,created_date,updated_date
+  table users
+  flush_interval 10s
+</match>
+```
+
+Assume following input is coming:
+
+```js
+mysql.input: {"user":"toyama","created_date":"2014/01/03 21:35:15","updated_date":"2014/01/03 21:35:15","dummy":"hogehoge"}
+mysql.input: {"user":"toyama2","created_date":"2014/01/03 21:35:21","updated_date":"2014/01/03 21:35:21","dummy":"hogehoge"}
+mysql.input: {"user":"toyama3","created_date":"2014/01/03 21:35:27","updated_date":"2014/01/03 21:35:27","dummy":"hogehoge"}
+```
+
+then result becomes as below (indented):
+
+```sql
++-----+-----------+---------------------+---------------------+
+| id  | user_name | created_at          | updated_at          |
++-----+-----------+---------------------+---------------------+
+| 1   | toyama    | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
+| 2   | toyama2   | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
+| 3   | toyama3   | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
++-----+-----------+---------------------+---------------------+
+```
+
+## Configuration Example(bulk insert, time complement)
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at
+  key_names id,user,${time}
+  table users
+  flush_interval 10s
+</match>
+```
+
+Assume following input is coming:
+
+```js
+2014-01-03 21:35:15+09:00: mysql.input: {"user":"toyama","dummy":"hogehoge"}
+2014-01-03 21:35:21+09:00: mysql.input: {"user":"toyama2","dummy":"hogehoge"}
+2014-01-03 21:35:27+09:00: mysql.input: {"user":"toyama3","dummy":"hogehoge"}
+```
+
+then `created_at` column is set from time attribute in a fluentd packet:
+
+```sql
++-----+-----------+---------------------+
+| id  | user_name | created_at          |
++-----+-----------+---------------------+
+| 1   | toyama    | 2014-01-03 21:35:15 |
+| 2   | toyama2   | 2014-01-03 21:35:21 |
+| 3   | toyama3   | 2014-01-03 21:35:27 |
++-----+-----------+---------------------+
+```
+
+## Configuration Example(bulk insert, time complement with specific timezone)
+
+As described above, `${time}` placeholder sets time with `Time.at(time).strftime("%Y-%m-%d %H:%M:%S")`.
+This handles the time with fluentd server default timezone.
+If you want to use the specific timezone, you can use the include_time_key feature.
+This is useful in case fluentd server and mysql have different timezone.
+You can use various timezone format. See below.
+http://docs.fluentd.org/articles/formatter-plugin-overview
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+
+  include_time_key yes
+  timezone +00
+  time_format %Y-%m-%d %H:%M:%S
+  time_key created_at
+
+  column_names id,user_name,created_at
+  key_names id,user,created_at
+  table users
+  flush_interval 10s
+</match>
+```
+
+Assume following input is coming(fluentd server is using JST +09 timezone):
+
+```js
+2014-01-03 21:35:15+09:00: mysql.input: {"user":"toyama","dummy":"hogehoge"}
+2014-01-03 21:35:21+09:00: mysql.input: {"user":"toyama2","dummy":"hogehoge"}
+2014-01-03 21:35:27+09:00: mysql.input: {"user":"toyama3","dummy":"hogehoge"}
+```
+
+then `created_at` column is set from time attribute in a fluentd packet with timezone converted to +00 UTC:
+
+```sql
++-----+-----------+---------------------+
+| id  | user_name | created_at          |
++-----+-----------+---------------------+
+| 1   | toyama    | 2014-01-03 12:35:15 |
+| 2   | toyama2   | 2014-01-03 12:35:21 |
+| 3   | toyama3   | 2014-01-03 12:35:27 |
++-----+-----------+---------------------+
+```
+
+## Configuration Example(bulk insert with tag placeholder for table name)
+
+This description is for v0.14.X users.
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at
+  key_names id,user,${time}
+  table users_${tag}
+  <buffer tag>
+    @type memory
+    flush_interval 60s
+  </buffer>
+</match>
+```
+
+Assume following input is coming:
+
+```js
+2016-09-26 18:42:13+09:00: mysql.input: {"user":"toyama","dummy":"hogehoge"}
+2016-09-26 18:42:16+09:00: mysql.input: {"user":"toyama2","dummy":"hogehoge"}
+2016-09-26 18:42:19+09:00: mysql.input: {"user":"toyama3","dummy":"hogehoge"}
+```
+
+then `created_at` column is set from time attribute in a fluentd packet:
+
+```sql
+mysql> select * from users_mysql_input;
++----+-----------+---------------------+
+| id | user_name | created_at          |
++----+-----------+---------------------+
+|  1 | toyama    | 2016-09-26 18:42:13 |
+|  2 | toyama2   | 2016-09-26 18:42:16 |
+|  3 | toyama3   | 2016-09-26 18:42:19 |
++----+-----------+---------------------+
+3 rows in set (0.00 sec)
+```
+
+## Configuration Example(bulk insert with time format placeholder for table name)
+
+This description is for v0.14.X users.
+
+```
+<match mysql.input>
+  @type mysql_bulk
+  host localhost
+  database test_app_development
+  username root
+  password hogehoge
+  column_names id,user_name,created_at
+  key_names id,user,${time}
+  table users_%Y%m%d
+  <buffer time>
+    @type memory
+    timekey 60s
+    timekey_wait 60s
+  </buffer>
+</match>
+```
+
+Assume following input is coming:
+
+```js
+2016-09-26 18:37:06+09:00: mysql.input: {"user":"toyama","dummy":"hogehoge"}
+2016-09-26 18:37:08+09:00: mysql.input: {"user":"toyama2","dummy":"hogehoge"}
+2016-09-26 18:37:11+09:00: mysql.input: {"user":"toyama3","dummy":"hogehoge"}
+```
+
+then `created_at` column is set from time attribute in a fluentd packet:
+
+```sql
+mysql> select * from users_20160926;
++----+-----------+---------------------+
+| id | user_name | created_at          |
++----+-----------+---------------------+
+|  1 | toyama    | 2016-09-26 18:37:06 |
+|  2 | toyama2   | 2016-09-26 18:37:08 |
+|  3 | toyama3   | 2016-09-26 18:37:11 |
++----+-----------+---------------------+
+3 rows in set (0.00 sec)
+```
+
+## spec
+
+```
+bundle install
+rake test
+```
+
+## todo
+
+divide bulk insert(exsample 1000 per)
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new [Pull Request](../../pull/new/master)
+
+## Copyright
+
+Copyright (c) 2016 Hiroshi Toyama. See [LICENSE](LICENSE.txt) for details.

data/README_mysql.md

@@ -0,0 +1,140 @@
+# fluent-plugin-mysql
+
+## Component
+
+### MysqlOutput
+
+[Fluentd](http://fluentd.org) plugin to store mysql tables over SQL, to each columns per values, or to single column as json.
+
+## Configuration
+
+### MysqlOutput
+
+MysqlOutput needs MySQL server's host/port/database/username/password, and INSERT format as SQL, or as table name and columns.
+
+    <match output.by.sql.*>
+      type mysql
+      host master.db.service.local
+      # port 3306 # default
+      database application_logs
+      username myuser
+      password mypass
+      key_names status,bytes,vhost,path,rhost,agent,referer
+      sql INSERT INTO accesslog (status,bytes,vhost,path,rhost,agent,referer) VALUES (?,?,?,?,?,?,?)
+      flush_interval 5s
+    </match>
+    
+    <match output.by.names.*>
+      type mysql
+      host master.db.service.local
+      database application_logs
+      username myuser
+      password mypass
+      key_names status,bytes,vhost,path,rhost,agent,referer
+      table accesslog
+      # 'columns' names order must be same with 'key_names'
+      columns status,bytes,vhost,path,rhost,agent,referer
+      flush_interval 5s
+    </match>
+
+Or, insert json into single column.
+
+    <match output.as.json.*>
+      type mysql
+      host master.db.service.local
+      database application_logs
+      username root
+      table accesslog
+      columns jsondata
+      format json
+      flush_interval 5s
+    </match>
+
+To include time/tag into output, use `include_time_key` and `include_tag_key`, like this:
+
+    <match output.with.tag.and.time.*>
+      type mysql
+      host my.mysql.local
+      database anydatabase
+      username yourusername
+      password secret
+      
+      include_time_key yes
+      ### default `time_format` is ISO-8601
+      # time_format %Y%m%d-%H%M%S
+      ### default `time_key` is 'time'
+      # time_key timekey
+      
+      include_tag_key yes
+      ### default `tag_key` is 'tag'
+      # tag_key tagkey
+      
+      table anydata
+      key_names time,tag,field1,field2,field3,field4
+      sql INSERT INTO baz (coltime,coltag,col1,col2,col3,col4) VALUES (?,?,?,?,?,?)
+    </match>
+
+Or, for json:
+
+    <match output.with.tag.and.time.as.json.*>
+      type mysql
+      host database.local
+      database foo
+      username root
+      
+      include_time_key yes
+      utc   # with UTC timezone output (default: localtime)
+      time_format %Y%m%d-%H%M%S
+      time_key timeattr
+      
+      include_tag_key yes
+      tag_key tagattr
+      table accesslog
+      columns jsondata
+      format json
+    </match>
+    #=> inserted json data into column 'jsondata' with addtional attribute 'timeattr' and 'tagattr'
+
+### JsonPath format
+
+You can use [JsonPath](http://goessner.net/articles/JsonPath/) selectors as key_names, such as:
+
+	<match output.with.jsonpath.format.*>
+	   type mysql
+	   host database.local
+       database foo
+       username bar
+
+       include_time_key yes
+       utc
+       include_tag_key yes
+       table baz
+
+       format jsonpath
+       key_names time, tag, id, data.name, tags[0]
+       sql INSERT INTO baz (coltime,coltag,id,name,tag1) VALUES (?,?,?,?,?)
+	</match>
+
+Which for a record like:
+
+`{ 'id' => 15, 'data'=> {'name' => 'jsonpath' }, 'tags' => ['unit', 'simple'] }`
+
+will generate the following insert values:
+
+`('2012-12-17T01:23:45Z','test',15,'jsonpath','unit')`
+
+## Prerequisites
+
+`fluent-plugin-mysql` uses `mysql2` gem, and `mysql2` links against `libmysqlclient`. See [Installing](https://github.com/brianmario/mysql2#installing) for its installation.
+
+## TODO
+
+* implement 'tag_mapped'
+  * dynamic tag based table selection
+
+## Copyright
+
+* Copyright
+  * Copyright(C) 2016- TAGOMORI Satoshi (tagomoris)
+* License
+  * Apache License, Version 2.0

data/Rakefile

@@ -0,0 +1,11 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test

data/fluent-plugin-mysql.gemspec

@@ -0,0 +1,24 @@
+# -*- encoding: utf-8 -*-
+Gem::Specification.new do |gem|
+  gem.name          = "fluent-plugin-mysql-2"
+  gem.version       = "0.3.7"
+  gem.authors       = ["TAGOMORI Satoshi", "Toyama Hiroshi", "Alex Scarborough"]
+  gem.email         = ["tagomoris@gmail.com", "toyama0919@gmail.com", "alex@teak.io"]
+  gem.description   = %q{fluent plugin to insert mysql as json(single column) or insert statement}
+  gem.summary       = %q{fluent plugin to insert mysql}
+  gem.homepage      = "https://github.com/GoCarrot/fluent-plugin-mysql"
+  gem.license       = "Apache-2.0"
+
+  gem.files         = `git ls-files`.split($\)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_runtime_dependency "fluentd", ['>= 0.14.8', '< 2']
+  gem.add_runtime_dependency "mysql2-cs-bind"
+  gem.add_runtime_dependency "jsonpath"
+  gem.add_runtime_dependency "oj"
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "test-unit"
+  gem.add_development_dependency "timecop", "~> 0.8.0"
+end