fluent-plugin-sql 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9ca23328ec98228710390c94c0f3f3a1d762e3c4
-  data.tar.gz: 0ff6ef9512db112b7938b2af612a2ffe59ddd59e
+  metadata.gz: f22da5fcea57030f5a458f011a543329737057a0
+  data.tar.gz: 9c334fb30a6ad62cc505b60006b91fbd3e2ea200
 SHA512:
-  metadata.gz: 302054c46157cc5438085b75e612a8353d6859b5f997ca66984b2064231e25587cdaee438050e645c345ad33c3a2377a21a85f3317f6d25c1fdceaf181e8f29f
-  data.tar.gz: d567b461ca80f21adbd7c07e33a4deb801a20ea54ccc955e6091e84230ce51c7925b41520aa8640788079921d197bf1f33727a9b4f736dad2158a921cefc8537
+  metadata.gz: d82f9b07974099025f990131f6d22337f71e1ef67cba5a61f634f0a476878a9294b1f87ac58124770b52aec57e9a30c03081ae8630a9f5d342f2c93005146b99
+  data.tar.gz: 738ea7e30cbb0ed9beecca8fa332388d2ae3b8750aa437aaa8bf63e522bf96d014f40eeb1afb99fdf38f81b52bcc596328348e1adc816e2011f0b5dd559cda77

data/README.md

@@ -2,9 +2,21 @@
 
 ## Overview
 
-This sql input plugin reads records from a RDBMS periodically. Thus you can copy tables to other storages through Fluentd.
+This SQL plugin has two parts:
 
-## How does it work?
+1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
+2. SQL **output** plugin that writes records into RDBMes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.
+
+## Installation
+
+    $ fluent-gem install fluent-plugin-sql --no-document
+    $ fluent-gem install pg --no-document # for postgresql
+
+You should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.
+
+We recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.
+
+## Input: How It Works
 
 This plugin runs following SQL periodically:
 
@@ -15,14 +27,14 @@ If you omit to set *update\_column* parameter, it uses primary key.
 
 It stores last selected rows to a file (named *state\_file*) to not forget the last row when Fluentd restarts.
 
-## Configuration
+## Input: Configuration
 
     <source>
       type sql
 
       host rdb_host
       database rdb_database
-      adapter mysql2_or_postgresql_etc
+      adapter mysql2_or_postgresql_or_etc
       username myusername
       password mypassword
 
@@ -54,7 +66,7 @@ It stores last selected rows to a file (named *state\_file*) to not forget the l
 * **host** RDBMS host
 * **port** RDBMS port
 * **database** RDBMS database name
-* **adapter** RDBMS driver name (mysql2 for MySQL, postgresql for PostgreSQL, etc.)
+* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
 * **user** RDBMS login user name
 * **password** RDBMS login password
 * **tag_prefix** prefix of tags of events. actual tag will be this\_tag\_prefix.tables\_tag (optional)
@@ -70,8 +82,67 @@ It stores last selected rows to a file (named *state\_file*) to not forget the l
 * **update_column**: see above description
 * **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
 
-## Limitation
+## Input: Limitation
 
 You should make sure target tables have index (and/or partitions) on the *update\_column*. Otherwise SELECT causes full table scan and serious performance problem.
 
 You can't replicate DELETEd rows.
+
+## Output: How It Works
+
+This plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.
+
+## Output: Configuration
+
+    <match my.rdb.*>
+      type sql
+      host rdb_host
+      port 3306
+      database rdb_database
+      adapter mysql2_or_postgresql_or_etc
+      username myusername
+      password mypassword
+      socket path_to_socket
+      remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql
+
+      <table>
+        table table1
+        # This is the default table because it has no "pattern" field
+        # The logic is such that if all non-default <table> blocks
+        # do not match, the default one is chosen.
+        # The default table is required.
+      </table>
+
+      <table>
+        table table2
+        pattern hello.* # You can pass the same pattern you use in match statements.
+        # This is the non-default table. It is chosen if the tag matches the pattern
+        # AFTER remove_tag_prefix is applied to the incoming event. For example, if
+        # the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
+        # makes it "hello.world", which gets matched here because of "pattern hello.*".
+      </table>
+      
+      <table>
+        table table3
+        pattern hello.world
+        # This is the second non-default table. You can have as many non-default tables
+        # as you wish. One caveat: non-default tables are matched top-to-bottom and
+        # the events go into the first table it matches to. Hence, this particular table
+        # never gets any data, since the above "hello.*" subsumes "hello.world".
+      </table>
+    </match>
+
+* **host** RDBMS host
+* **port** RDBMS port
+* **database** RDBMS database name
+* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
+* **user** RDBMS login user name
+* **password** RDBMS login password
+* **socket** RDBMS socket path
+* **remove_tag_prefix** remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)
+
+\<table\> sections:
+
+* **table** RDBM table name
+* **column_mapping**: Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.
+* **pattern**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of <match>](http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd). **Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data**.

data/VERSION

@@ -1 +1 @@
-0.3.0
+0.4.0

data/fluent-plugin-sql.gemspec

@@ -17,10 +17,8 @@ Gem::Specification.new do |gem|
   gem.require_paths = ['lib']
   gem.license = "Apache 2.0"
 
-  gem.add_dependency "fluentd", "~> 0.10.0"
-  gem.add_dependency 'activerecord', "~> 4.0.2"
-  gem.add_dependency 'activerecord-import', "~> 0.4.1"
-  gem.add_dependency 'mysql2', ['~> 0.3.12']
-  gem.add_dependency 'pg', ['~> 0.16.0']
+  gem.add_dependency "fluentd", [">= 0.10.0", "< 2"]
+  gem.add_dependency 'activerecord', "~> 4.2"
+  gem.add_dependency 'activerecord-import', "~> 0.7"
   gem.add_development_dependency "rake", ">= 0.9.2"
 end

data/lib/fluent/plugin/in_sql.rb

@@ -150,7 +150,7 @@ module Fluent
     SKIP_TABLE_REGEXP = /\Aschema_migrations\Z/i
 
     def start
-      @state_store = StateStore.new(@state_file)
+      @state_store = @state_file.nil? ? MemoryStateStore.new : StateStore.new(@state_file)
 
       config = {
         :adapter => @adapter,
@@ -235,6 +235,8 @@ module Fluent
 
     class StateStore
       def initialize(path)
+        require 'yaml'
+
         @path = path
         if File.exists?(@path)
           @data = YAML.load_file(@path)
@@ -259,6 +261,19 @@ module Fluent
         }
       end
     end
+
+    class MemoryStateStore
+      def initialize
+        @data = {}
+      end
+
+      def last_records
+        @data['last_records'] ||= {}
+      end
+
+      def update!
+      end
+    end
   end
 
 end

metadata

@@ -1,85 +1,63 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-sql
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-08-28 00:00:00.000000000 Z
+date: 2015-02-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 0.10.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 0.10.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.0.2
+        version: '4.2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.0.2
+        version: '4.2'
 - !ruby/object:Gem::Dependency
   name: activerecord-import
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.4.1
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 0.4.1
-- !ruby/object:Gem::Dependency
-  name: mysql2
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 0.3.12
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 0.3.12
-- !ruby/object:Gem::Dependency
-  name: pg
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 0.16.0
+        version: '0.7'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.16.0
+        version: '0.7'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement