fluent-plugin-norikra 0.0.9 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5290c7b55ef87189a7ff88bce1f1e490f72ed2cc
-  data.tar.gz: 0c5dfe962ed60e8fc45baec125237349f78607e7
+  metadata.gz: fb3d07d2f2997aa968bdeb9a2c9b367d11b8402d
+  data.tar.gz: ba678530347bc93801c557d70eaf5bda2aec33e5
 SHA512:
-  metadata.gz: 4596d0b75141328f0ded539b44f1281e4a719786ce848231a261d0f1555710d9d72c589131ab078a15f71fb3d2f8e42d2c4e7aac4a22f91075fcdb49a78df5ea
-  data.tar.gz: 400f0cf1365c998e7a3c7d81a2ba3f58b9406662ddb4b87b23dda5cff7d4f80b06b92589fa10516b152b930930ef9867b5acbedd46fc3da8a7f6817fed196dbd
+  metadata.gz: 06681dafb3386c4773dd557b801103c0d73f9880607d9521abc860b17272b129f612a83217fa305254fa19de569b426e02fa49056fac962525ab9035b1b32ab9
+  data.tar.gz: bab9693ba1db4ca31416bde2dc4d6b1ca91b388562c7364d362adf8916902aa509cb5f42f4729f935813f9f1a22882f28d5a77102a954c257ceb53309167f159

data/README.md

@@ -1,31 +1,35 @@
 # fluent-plugin-norikra
 
-Fluentd output plugin to send events to norikra server, and to fetch events (and re-send on fluentd network) from norikra server.
+Fluentd plugins to send/receive events to/from Norikra server.
 
-With NorikraOutput, we can:
+Norikra is an open source server software provides "Stream Processing" with SQL, written in JRuby, runs on JVM, licensed under GPLv2.
+For more details, see: http://norikra.github.io/ .
 
- * execute Norikra server as built-in process dynamically
- * generate Norikra's target automatically with Fluentd's tags
- * register queries automatically with Fluentd's tags and messages
- * get all events on Norikra and emit on Fluentd network automatically
+fluent-plugin-norikra has 3 plugins: in\_norikra, out\_norikra and out\_norikra\_filter.
+ * in\_norikra
+   * fetch events of query results from Norikra server
+ * out\_norikra
+   * send events to Norikra server
+ * out\_norikra\_filter
+   * launch Norikra server as child process dynamically, as needed
+   * use Norikra server as event filter (like out\_exec\_filter)
+   * register/execute queries for targets newly incoming
 
 # Setup
 
-At first, install JRuby and Norikra on your host if you are not using stand-alone Norikra servers.
-
-1. install latest jruby
-  * (rbenv) `rbenv install jruby-1.7.4`
-  * (rvm) `rvm install jruby-1.7.4`
-  * or other tools you want.
-2. swith to jruby, and install Norikra
-  * `gem install norikra`
-3. check and note `jruby` and `norikra-server`
-  * `which jruby`
-  * `which norikra-server`
-4. switch CRuby (with Fluentd), and install this plugin
-  * `gem install fluent-plugin-norikra` (or use `fluent-gem`)
-5. configure Fluentd, and execute.
-  * and write `path` configuration of `<server>` section (if you want)
+`fluent-plugin-norikra` works with Norikra server, on same server with Fluentd, or anywhere reachable over network from Fluentd.
+For Norikra server setup, see: http://norikra.github.io/ .
+
+NOTES:
+ * Fluentd and fluent-plugin-norikra requires CRuby (MatzRuby).
+ * Norikra requires JRuby.
+
+To use out\_norikra\_filter with dynamic Norikra server launching, check actual path of command `norikra` under installed JRuby tree. (ex: `$HOME/.rbenv/versions/jruby-1.7.8/bin/norikra`)
+
+To use this plugin:
+  1. run `gem install fluent-plugin-norikra` or `fluent-gem install fluent-plugin-norikra` to install plugin
+  1. edit configuration files
+  1. execute fluentd
 
 # Configuration
 
@@ -33,87 +37,156 @@ For variations, see `example` directory.
 
 ## NorikraOutput
 
-With built-in Norikra server, to receive tags like `event.foo` and send norikra's target `foo`, and get count of its records per minute, and per hour.
-
-    <match event.*>
-      type norikra
-      norikra localhost:26571 # this is default
-      <server>
-        execute yes
-		jruby   /home/user/.rbenv/versions/jruby-1.7.4/bin/jruby
-        path    /home/user/.rbenv/versions/jruby-1.7.4/bin/norikra
-		opts    -Xmx2g
-      </server>
-      
-      remove_tag_prefix event
-      target_map_tag    yes
-      
-      <default>
-	    <query>
-		  name       count_min_${target}
-		  expression SELECT count(*) AS cnt FROM ${target}.win:time_batch(1 minute)
-		  tag        count.min.${target}
-		</query>
-	    <query>
-		  name       count_hour_${target}
-		  expression SELECT count(*) AS cnt FROM ${target}.win:time_batch(1 hour)
-		  tag        count.hour.${target}
-		</query>
-      </default>
-    </match>
-
-With default setting, all fields are defined as 'string', so you must use `cast` for numerical processing in query (For more details, see Norikra and Esper's documents).
-
-If you know some field's types of records, you can define types of these fields. This plugin will define field types before it send records into Norikra server.
-
-    <match event.*>
-      type norikra
-      norikra localhost:26571 # this is default
-      <server>
-        execute yes
-		jruby   /home/user/.rbenv/versions/jruby-1.7.4/bin/jruby
-        path    /home/user/.rbenv/versions/jruby-1.7.4/bin/norikra
-		opts    -Xmx2g
-      </server>
-      
-      remove_tag_prefix event
-      target_map_tag    yes
-      
-      <default>
-        field_int    amount
-        field_long   size
-        field_double price
-        
-	    <query>
-		  name       sales_${target}
-		  expression SELECT price * amount AS  FROM ${target}.win:time_batch(1 minute) WHERE size > 0
-		  tag        sales.min.${target}
-		</query>
-      </default>
-    </match>
-
-Additional field definitions and query registrations should be written in `<target TARGET_NAME>` sections. In each sections (or also in 'default'), you can specify `auto_field false` to suppress field number which increasing infinitely.
-
-    <default>
-      ... # for all of access logs
-    </default>
-    <target login>
-      field_string protocol # like 'oauth', 'openid', ...
-	  field_int    proto_num # integer means internal id of protocols
-	  <query>
-	    name       protocol
-		expression SELECT protocol, count(*) AS cnt FROM ${target}.win:time_batch(1 hour) WHERE proto_num != 0 GROUP BY protocol
-		tag        login.counts
-	  </query>
-    </target>
-    <target other_action>
-      auto_field false
-	  ...
-    </target>
-	# ...
+Sends events to remote Norikra server. Minimal configurations are:
+```apache
+<match data.*>
+  type    norikra
+  norikra norikra.server.local:26571
+  
+  remove_tag_prefix data
+  target_map_tag    true  # fluentd's tag 'data.event' -> norikra's target 'event'
+</match>
+```
+
+NorikraOutput plugin opens Norikra's target for newly incoming tags. You can specify fields to include/exclude, and specify types of each fields, for each targets (and all targets by `default`). Definitions in `<target TARGET_NAME>` overwrites `<default>` specifications.
+```apache
+<match data.*>
+  type    norikra
+  norikra norikra.server.local:26571
+  
+  target_map_tag    true  # fluentd's tag -> norikra's target
+  remove_tag_prefix data
+  # other options:
+  #   target_map_key KEY_NAME  # use specified key's value as target in fluentd event
+  #   target_string  STRING    # use fixed target name specified
+  
+  <default>
+    include *     # send all fields values to norikra
+    exclude time  # exclude 'time' field from sending event values
+       # AND/OR 'include_regexp' and 'exclude_regexp' available
+    field_integer seq  # field 'seq' defined as integer for all targets
+  </default>
+  
+  <target users>
+    field_string  name,address
+    field_integer age
+    field_float   height,weight
+    field_boolean superuser
+  </target>
+</match>
+```
+
+With default setting, all fields are defined as 'string', so you must use `field_xxxx` parameters for numerical processing in query (For more details, see Norikra and Esper's documents).
+
+If fluentd's events has so many variations of sets of fields, you can specify not to include fields automatically, with `auto_field` option:
+```apache
+<match data.*>
+  type    norikra
+  norikra norikra.server.local:26571
+  
+  target_map_tag    true  # fluentd's tag 'data.event' -> norikra's target 'event'
+  remove_tag_prefix data
+  
+  <default>
+    auto_field false  # norikra includes fields only used in queries.
+  </default>
+</match>
+```
 
 Fields which are referred in queries are automatically registered on norikra server in spite of `auto_field false`.
 
+** NOTE: <default> and <target> sections in NorikraOutput ignores <query> sections. see NorikraFilterOutput **
+
+## NorikraInput
+
+Fetch events from Norikra server, and emits these into Fluentd itself. NorikraInput uses Norikra's API `event` (for queries), and `sweep` (for query groups).
+
+Minimal configurations:
+```apache
+<source>
+  type    norikra
+  norikra norikra.server.local:26571
+  <fetch>
+    method     sweep
+    # target QUERY_GROUP_NAME  # not specified => default query group
+    tag        query_name
+    tag_prefix norikra.query
+    # other options:
+    #  tag field FIELDNAME : tag by value with specified field name in output event
+    #  tag string STRING   : fixed string specified
+    interval 3s  # interval to call api
+  </fetch>
+</source>
+```
+
+Available `<fetch>` methods are `event` and `sweep`. `target` parameter is handled as query name for `event`, and as query group name for `sweep`.
+```apache
+<source>
+  type    norikra
+  norikra norikra.server.local:26571
+  <fetch>
+    method   event
+    target   data_count_1hour
+    tag      string data.count.1hour
+    interval 60m
+  </fetch>
+  <fetch>
+    method   event
+    target   data_count_5min
+    tag      string data.count.5min
+    interval 5m
+  </fetch>
+  <fetch>
+    method     sweep
+    target     count_queries
+    tag  field target_name
+    tag_prefix data.count.all
+    interval 15s
+  </fetch>
+</source>
+```
+
+## NorikraFilterOutput
+
+NorikraFilterOutput has all features of both of NorikraInput and NorikraOutput, and also has additional features:
+  * execute Norikra server
+  * runs queries for newly incoming targets.
+
+If you runs Norikra as standalone process, better configurations are to use NorikraInput and NorikraOutput separately. NorikraFilterOutput is for simple aggregations and filterings.
+
+Configuration example to receive tags like `event.foo` and send norikra's target `foo`, and get count of its records per minute, and per hour with built-in Norikra server:
+```apache
+<match event.*>
+  type    norikra_filter
+  <server>
+    path    /home/username/.rbenv/versions/jruby-1.7.4/bin/norikra
+    # opts  -Xmx2g  # options of 'norikra start'
+  </server>
+  
+  remove_tag_prefix event
+  target_map_tag    yes
+  
+  <default>
+    <query>
+	  name       count_min_${target}
+      group      count_query_group # or default when omitted
+	  expression SELECT count(*) AS cnt FROM ${target}.win:time_batch(1 minute)
+	  tag        count.min.${target}
+	</query>
+    <query>
+	  name       count_hour_${target}
+      group      count_query_group
+	  expression SELECT count(*) AS cnt FROM ${target}.win:time_batch(1 hour)
+	  tag        count.hour.${target}
+	</query>
+  </default>
+</match>
+```
+
+Results of queries automatically registered by NorikraFilterOutput with `tag` parameter, will be fetched automatically by this plugin, and re-emitted into Fluentd itself.
+
+Other all options are available as same as NorikraInput and NorikraOutput. `<default>`, `<target>` and `<fetch>` sections, `auto_field`, `include|exclude` and `field_xxxx` specifiers for targets and parameters for `<fetch>` sections.
+
 ### Input event data filtering
 
 If you want send known fields only, specify `exclude *` and `include` or `include_regexp` like this:
@@ -152,34 +225,10 @@ Norikra's target (like table name) can be generated from:
    * all records are sent in single target
    * `target_string from_fluentd`
 
-### Event sweeping
-
-Norikra server accepts queries and events from everywhere other than Fluentd. This plugin can get events from these queries/events.
-
-To gather all events of Norikra server, including queries from outside of Fluentd configurations, write `<event>` section.
-
-    <events>
-      method sweep
-      tag    query_name
-      # tag    field FIELDNAME
-      # tag    string FIXED_STRING
-      tag_prefix norikra.event     # actual tag: norikra.event.QUERYNAME
-      sweep_interval 5s
-    </events>
-
-NOTE: 'sweep' get all events from Norikra, and other clients cannot get these events. Take care for other clients.
-
-# FAQ
-
-* TODO: write this section
-  * `fetch_interval`
-  * error logs for new target, success logs of retry
-* input/output plugin
-  * output plugin with active-standby servers
-
 # TODO
 
-* TODO: write this section
+* write abou these topics
+  * error logs for new target, success logs of retry
 
 # Copyright
 

data/example/blank.conf

@@ -3,11 +3,10 @@
 </source>
 
 <match event.*>
-  type norikra
+  type norikra_filter
   norikra localhost:26571 # this is default
   <server>
-    execute yes
-    path    /Users/tagomoris/.rbenv/versions/jruby-1.7.3/bin/norikra
+    path    /Users/tagomoris/.rbenv/versions/jruby-1.7.8/bin/norikra # $HOME
   </server>
   
   remove_tag_prefix event

data/example/blank.rb

@@ -0,0 +1,15 @@
+source {
+  type "forward"
+}
+
+home_dir = ::Object::ENV['HOME']
+
+match('event.*') {
+  type "norikra_filter"
+  norikra "localhost:26571"
+  server {
+    path "#{home_dir}/.rbenv/versions/jruby-1.7.8/bin/norikra"
+  }
+  remove_tag_prefix "event"
+  target_map_tag true
+}

data/example/blank2.conf

@@ -3,7 +3,7 @@
 </source>
 
 <match event.*>
-  type norikra
+  type norikra_filter
   norikra localhost:26571 # this is default
 
   remove_tag_prefix event

data/example/example1.conf

@@ -1,9 +1,8 @@
 <match event.*>
-  type norikra
+  type norikra_filter
   norikra localhost:26571
 
   <server>
-    execute yes # (default)no
     path    /home/user/.rbenv/versions/jruby-1.7.4/bin/norikra
   </server>
 
@@ -24,13 +23,13 @@
     # include foo,bar,baz
     # include_regexp status.*
     field_boolean flag
-    field_int status
-    field_long    duration,bytes
+    field_integer status,duration,bytes
 
     <query>
       name pv_${target}
       expression SELECT count(*) AS cnt FROM ${target}.win:time_batch(1 minutes) WHERE not flag
       tag pv.${target}
+      # group pv_query_group # default: nil (default group)
       fetch_interval 15s # default -> time_batch / 4 ? -> (none) -> 60s
       # fetch_interval is ignored when <events> section specified
     </query>
@@ -57,12 +56,12 @@
     </query>
   </target>
 
-  <events>
+  <fetch>
     method sweep # listen(not implemented)
     tag query_name
     # tag field FIELDNAME
     # tag string TAG_STRING
     tag_prefix cep
-    sweep_interval 5s
-  </events>
+    interval 5s
+  </fetch>
 </match>

data/example/test1.conf

@@ -3,11 +3,10 @@
 </source>
 
 <match test.*>
-  type norikra
+  type norikra_filter
   norikra localhost:26571
   <server>
-    execute yes
-    path /Users/tagomoris/.rbenv/versions/jruby-1.7.3/bin/norikra
+    path /Users/tagomoris/.rbenv/versions/jruby-1.7.8/bin/norikra # $HOME
   </server>
 
   remove_tag_prefix test
@@ -19,11 +18,11 @@
       expression SELECT '${target}' as target,count(*) AS cnt FROM ${target}.win:time_batch(30 sec)
     </query>
   </default>
-  <event>
+  <fetch>
     method sweep
     tag field target
     tag_prefix count
-    sweep_interval 5s
+    interval 5s
   </event>
 </match>
 

data/example/test1.rb

@@ -0,0 +1,40 @@
+source {
+  type :forward
+}
+
+home_dir = ::Object::ENV['HOME']
+
+match('test.*') {
+  type :norikra_filter
+  norikra 'localhost:26571'
+  server {
+    path "#{home_dir}/.rbenv/versions/jruby-1.7.8/bin/norikra"
+  }
+
+  remove_tag_prefix 'test'
+  target_map_tag true
+
+  default {
+    query {
+      name "count_${target}"
+      expression "SELECT '${target}' as target,count(*) AS cnt FROM ${target}.win:time_batch(30 sec)"
+      group "testing"
+      tag "count.x.${target}"
+    }
+  }
+
+  fetch {
+    method :sweep
+    tag 'field target'
+    tag_prefix 'count'
+    interval 5
+  }
+}
+
+match('fluent.*') {
+  type :null
+}
+
+match('**') {
+  type :stdout
+}

data/example/test2.conf

@@ -3,7 +3,7 @@
 </source>
 
 <match {accesslog,applog}.**>
-  type norikra
+  type norikra_filter
   norikra localhost:26571
   target_map_tag yes
 
@@ -14,16 +14,15 @@
 
   <target accesslog.**>
     field_string  vhost,path,method,referer,rhost,userlabel,agent,ua_name,ua_category,ua_os,ua_version,ua_vendor
-    field_int     status
-    field_long    bytes,duration
+    field_integer status,bytes,duration
     field_boolean FLAG,status_redirection,status_errors,rhost_internal,suffix_miscfile,suffix_imagefile,agent_bot
   </target>
 
-  <events>
+  <fetch>
     method sweep
     tag query_name
     tag_prefix norikra.event
-    sweep_interval 5s
-  </events>
+    interval 5s
+  </fetch>
 </match>
 

data/example/test_in_out.rb

@@ -0,0 +1,48 @@
+source {
+  type :forward
+}
+
+match('test.*') {
+  type :norikra
+  norikra 'localhost:26571'
+
+  remove_tag_prefix 'test'
+  target_map_tag true
+
+  default {
+    include '*'
+    exclude 'hhmmss'
+  }
+
+  target('data') {
+    field_string 'name'
+    field_integer 'age'
+  }
+}
+
+source {
+  type :norikra
+
+  fetch {
+    method :sweep
+    # target => nil (group: default)
+    tag 'field target'
+    tag_prefix 'norikra.query'
+    interval 3
+  }
+
+  fetch {
+    method :event
+    target 'data_count'
+    tag 'string norikra.count.data'
+    interval 5
+  }
+}
+
+match('fluent.**') {
+  type :null
+}
+
+match('**') {
+  type :stdout
+}

data/fluent-plugin-norikra.gemspec

@@ -2,7 +2,7 @@
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-norikra"
-  spec.version       = "0.0.9"
+  spec.version       = "0.1.0"
   spec.authors       = ["TAGOMORI Satoshi"]
   spec.email         = ["tagomoris@gmail.com"]
   spec.description   = %q{process events on fluentd with SQL like query, with built-in Norikra server if needed.}

data/lib/fluent/plugin/in_norikra.rb

@@ -0,0 +1,75 @@
+require_relative 'norikra/input'
+
+require 'norikra-client'
+
+module Fluent
+  class NorikraInput < Fluent::Input
+    include Fluent::NorikraPlugin::InputMixin
+
+    Fluent::Plugin.register_input('norikra', self)
+
+    config_param :norikra, :string, :default => 'localhost:26571'
+
+    config_param :connect_timeout, :integer, :default => nil
+    config_param :send_timeout, :integer, :default => nil
+    config_param :receive_timeout, :integer, :default => nil
+
+    # <fetch> tags
+    # <fetch>
+    #   method event
+    #   target QUERY_NAME
+    #   interval 5s
+    #   tag    query_name
+    #   # tag    field FIELDNAME
+    #   # tag    string FIXED_STRING
+    #   tag_prefix norikra.event     # actual tag: norikra.event.QUERYNAME
+    # </fetch>
+    # <fetch>
+    #   method sweep
+    #   target QUERY_GROUP # or unspecified => default
+    #   interval 60s
+    #   tag field group_by_key
+    #   tag_prefix norikra.query
+    # </fetch>
+
+    def configure(conf)
+      super
+
+      @host,@port = @norikra.split(':', 2)
+      @port = @port.to_i
+
+      conf.elements.each do |element|
+        case element.name
+        when 'fetch'
+          # ignore: processed in InputMixin, and set @fetch_queue
+        else
+          raise Fluent::ConfigError, "unknown configuration section name for this plugin: #{element.name}"
+        end
+      end
+
+      setup_input(conf)
+    end
+
+    def client(opts={})
+      Norikra::Client.new(@host, @port, {
+          :connect_timeout => opts[:connect_timeout] || @connect_timeout,
+          :send_timeout    => opts[:send_timeout]    || @send_timeout,
+          :receive_timeout => opts[:receive_timeout] || @receive_timeout,
+        })
+    end
+
+    def start
+      super
+      start_input
+    end
+
+    def shutdown
+      stop_input
+      shutdown_input
+    end
+
+    def fetchable?
+      true
+    end
+  end
+end