norikra 0.1.2-java → 0.1.3-java

data/README.md

@@ -22,7 +22,6 @@ For JRuby installation, you can use `rbenv`, `rvm` and `xbuild`, or install JRub
 
 * JRuby: http://jruby.org/
 * rbenv: https://github.com/sstephenson/rbenv/
-* rvm: https://rvm.io/rvm/install/
 * xbuild: https://github.com/tagomoris/xbuild
 
 ### Command line options
@@ -55,6 +54,27 @@ For other options, see help:
 
     norikra help start
 
+### How to execute norikra server and tests in development
+
+Fix code and tests:
+
+1. clone this repository
+1. run `bundle install` w/ jruby
+1. add/fix spec in `spec/*_spec.rb`
+1. fix code in `lib`
+1. run `bundle exec rake`
+
+Run tests faster than 2 or more times:
+
+1. execute `spork`
+1. execute `script/spec_server_pry` in another terminal
+1. run `rspec` in pry console (executed fastly after second times)
+
+Execute norikra server with target/query continuation:
+
+1. `bundle exec rake devserver`
+1. `Ctrl-C` and re-execute for updated code
+
 ## Clients
 
 Use `Norikra::Client` and `norikra-client` cli command. These are available on both of JRuby and CRuby.
@@ -104,24 +124,71 @@ And send more events:
 
 Query 'www.search' matches the last event automatically.
 
+## Performance
+
+Threads option available with `norikra start`. Simple specifiers for performance with threadings:
+
+    norikra start --micro     # or --small, --middle, --large (default: 'micro')
+
+Norikra server has 3 types of threads:
+
+* engine: 4 query engine thread types on Esper
+  * inbound: input data handler for queries
+  * outbound: output data handler for queries
+  * router: event handler which decides which query needs that events
+  * timer: executer for queries with time_batches and other timing events
+* rpc: data input/output rpc handler threads on Jetty
+* web: web ui request handler threads on Jetty
+
+In many cases, norikra server handling high rate events needs large number of rpc threads to handle input/output rpc requests. WebUI don't need threads rather than default in almost all of cases.
+
+Engine threads depends on queries running on norikra, input/output event data rate and target numbers. For more details, see Esper's API Documents: http://esper.codehaus.org/esper-4.10.0/doc/reference/en-US/html/api.html#api-threading
+
+Norikra's simple specifiers details of threadings are:
+
+* micro: development and testing
+  * engine: all processings on single threads
+  * rpc: 2 threads
+  * web: 2 threads
+* small: low rate events on virtual servers
+  * engine: inbound 1, outbound 1, route 1, timer 1 threads
+  * rpc: 2 threads
+  * web: 2 threads
+* middle: high rate events on physical servers
+  * engine: inbound 4, outbound 2, route 2, timer 2 threads
+  * rpc: 4 threads
+  * web: 2 threads
+* large: inbound heavy traffic and huge amount of queries
+  * engine: inbound 6, outbound 6, route 4, timer 4 threads
+  * rpc: 8 threads
+  * web: 2 threads
+
+To specify sizes of each threads, use `--*-threads=NUM` options. For more details, see: `norikra help start`.
+
 ## User Defined Functions
 
-UDFs/UDAFs can be loaded as plugin gems over rubygems or as private plugins. For example, see 'norikra-udf-mock' repository.
+UDFs/UDAFs can be loaded as plugin gems over rubygems or as private plugins.
+In fact, Norikra's UDFs/UDAFs are Esper's plugin with a JRuby class to indicate plugin metadata.
 
-TBD
+For details how to write your own UDF/UDAF for norikra and to release it as gem, see README of `norikra-udf-mock`.
+https://github.com/norikra/norikra-udf-mock
 
-## Performance
+## Tips
 
-Threads option available with command line options.
+We need to write this document on http://norikra.github.io
 
-TBD
+### Esper query syntax
 
-## Versions
+* String concatination operator: `||`
 
-* v0.1.0:
- * First release for production
+## Changes
+
+* v0.1.2:
+ * Fix CLI start command to detect jruby path collectly (behind rbenv/rvm and others)
 * v0.1.1:
  * Fix types more explicitly for users ('int/long' -> 'integer', 'float/double' -> 'float')
+* v0.1.0:
+ * First release for production
 
 ## Copyright
 

data/lib/norikra/cli.rb

@@ -42,6 +42,9 @@ module Norikra
         option :'outbound-threads', :type => :numeric, :default => nil, :desc => 'number of threads for outbound data'
         option :'route-threads',    :type => :numeric, :default => nil, :desc => 'number of threads for events routing for query execution'
         option :'timer-threads',    :type => :numeric, :default => nil, :desc => 'number of threads for internal timers for query execution'
+        ### about capacity options of esper's capacity-bound queue processing, see Esper's thread reference.
+        # http://esper.codehaus.org/esper-4.10.0/doc/reference/en-US/html/configuration.html#config-engine-threading-advanced
+        # default nil: unbound queueing
         option :'inbound-thread-capacity',  :type => :numeric, :default => nil
         option :'outbound-thread-capacity', :type => :numeric, :default => nil
         option :'route-thread-capacity',    :type => :numeric, :default => nil

data/lib/norikra/fieldset.rb

@@ -4,10 +4,10 @@ require 'norikra/field'
 module Norikra
   class FieldSet
     attr_accessor :summary, :fields
-    attr_accessor :target, :level
+    attr_accessor :target, :level, :query_unique_keys
 
     # fieldset doesn't have container fields
-    def initialize(fields, default_optional=nil, rebounds=0)
+    def initialize(fields, default_optional=nil, rebounds=0, query_unique_keys=[])
       @fields = {}
       # fields.keys are raw key for container access chains
       fields.keys.each do |key|
@@ -29,12 +29,13 @@ module Norikra
       @target = nil
       @level = nil
       @rebounds = rebounds
+      @query_unique_keys = query_unique_keys
       @event_type_name = nil
     end
 
     def dup
       fields = Hash[@fields.map{|key,field| [key, {:type => field.type, :optional => field.optional}]}]
-      self.class.new(fields, nil, @rebounds)
+      self.class.new(fields, nil, @rebounds, @query_unique_keys)
     end
 
     def self.leaves(container)
@@ -127,7 +128,7 @@ module Norikra
     #TODO: have a bug?
     def ==(other)
       return false if self.class != other.class
-      self.summary == other.summary
+      self.summary == other.summary && self.query_unique_keys == other.query_unique_keys
     end
 
     def definition
@@ -157,8 +158,9 @@ module Norikra
                  raise ArgumentError, "unknown fieldset bind level: #{level}, for target #{target}"
                end
       @rebounds += 1 if update_type_name
+      query_unique_key = @query_unique_keys ? @query_unique_keys.join("\t") : ''
 
-      @event_type_name = prefix + Digest::MD5.hexdigest([target, level.to_s, @rebounds.to_s, @summary].join("\t"))
+      @event_type_name = prefix + Digest::MD5.hexdigest([target, level.to_s, @rebounds.to_s, query_unique_key, @summary].join("\t"))
       self
     end
 

data/lib/norikra/typedef_manager.rb

@@ -82,7 +82,7 @@ module Norikra
 
       mapping = {}
       fields_set.each do |target,fields|
-        mapping[target] = generate_query_fieldset(target, fields.sort.uniq)
+        mapping[target] = generate_query_fieldset(target, fields.sort.uniq, query.name, query.group)
       end
       mapping
     end
@@ -106,7 +106,7 @@ module Norikra
       guessed
     end
 
-    def generate_query_fieldset(target, field_name_list)
+    def generate_query_fieldset(target, field_name_list, query_name, query_group)
       # all fields of field_name_list should exists in definitions of typedef fields
       # for this premise, call 'bind_fieldset' for data fieldset before this method.
       required_fields = {}
@@ -117,7 +117,7 @@ module Norikra
           end
         end
       end
-      Norikra::FieldSet.new(required_fields)
+      Norikra::FieldSet.new(required_fields, nil, 0, [query_name, (query_group || '')])
     end
 
     def base_fieldset(target)

data/lib/norikra/version.rb

@@ -1,3 +1,3 @@
 module Norikra
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

data/spec/fieldset_spec.rb

@@ -42,12 +42,27 @@ describe Norikra::FieldSet do
     set = Norikra::FieldSet.new({'x' => 'string', 'y' => 'long', 'a' => 'Boolean'})
     set2 = Norikra::FieldSet.new({'a' => 'string', 'b' => 'int', 'c' => 'float', 'd' => 'bool', 'e' => 'integer'})
 
+    q_set1 = Norikra::FieldSet.new({'x' => 'string', 'y' => 'long'}, nil, 0, ['set1', 'g1'])
+    q_set2 = Norikra::FieldSet.new({'x' => 'string', 'y' => 'long'}, nil, 0, ['set2', nil])
+    q_set3 = Norikra::FieldSet.new({'x' => 'string', 'y' => 'long'}, nil, 0, ['set3', nil])
+
     describe '#dup' do
       it 'make duplicated object with different internal instance' do
         x = set.dup
         expect(x.fields.object_id).not_to eql(set.fields.object_id)
         expect(x.fields).to eq(set.fields)
       end
+
+      it 'make duplicated object with query name and group' do
+        expect(q_set1.dup.instance_eval{ @query_unique_keys }).to eql(['set1', 'g1'])
+        expect(q_set2.dup.instance_eval{ @query_unique_keys }).to eql(['set2', nil])
+
+        expect(q_set2 == q_set3).to be_false
+
+        expect(q_set1 == q_set1.dup).to be_true
+        expect(q_set2 == q_set2.dup).to be_true
+        expect(q_set3 == q_set3.dup).to be_true
+      end
     end
 
     describe '.leaves' do
@@ -252,6 +267,18 @@ describe Norikra::FieldSet do
         expect(x.instance_eval{ @event_type_name }).not_to be_nil
         expect(x.instance_eval{ @event_type_name }).to match(/q_[0-9a-f]{32}/) # MD5 hexdump
       end
+
+      it 'sets different event_type_name for different query names and groups for query fieldsets w/ same field set' do
+        x = q_set1.dup
+        y = q_set2.dup
+        z = q_set3.dup
+        x.bind('TargetExample', :query)
+        y.bind('TargetExample', :query)
+        z.bind('TargetExample', :query)
+
+        expect(x.instance_eval{ @event_type_name }).not_to eql(y.instance_eval{ @event_type_name })
+        expect(x.instance_eval{ @event_type_name }).not_to eql(z.instance_eval{ @event_type_name })
+      end
     end
 
     describe '#rebind' do

data/spec/typedef_manager_spec.rb

@@ -218,7 +218,7 @@ describe Norikra::TypedefManager do
 
     describe '#generate_query_fieldset' do
       it 'returns fieldset instance with all required(non-optional) fields of target, and fields of query requires' do
-        r = manager.generate_query_fieldset('sample', ['a', 'b','f'])
+        r = manager.generate_query_fieldset('sample', ['a', 'b','f'], 'qname1', nil)
         expect(r.fields.size).to eql(4) # a,b,c,f
         expect(r.summary).to eql('a:string,b:string,c:float,f:boolean')
       end

metadata

@@ -1,14 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: norikra
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  prerelease:
+  version: 0.1.3
 platform: java
 authors:
 - TAGOMORI Satoshi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2013-12-06 00:00:00.000000000 Z
+date: 2013-12-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mizuno
@@ -17,11 +18,13 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.6'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.6'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -31,11 +34,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -45,11 +50,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -59,11 +66,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: 0.0.5
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: 0.0.5
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -73,11 +82,13 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.1.0
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.1.0
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -87,11 +98,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -101,11 +114,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -115,11 +130,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -129,11 +146,13 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.3'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.3'
+    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -143,11 +162,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -157,11 +178,13 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '2.0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '2.0'
+    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -171,11 +194,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -185,11 +210,13 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+    none: false
   prerelease: false
   type: :development
 description: 'CEP: Complex Event Processor with Esper EPL qeury language, messagepack
@@ -334,7 +361,6 @@ files:
 homepage: http://norikra.github.io/
 licenses:
 - GPLv2
-metadata: {}
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -344,17 +370,25 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - '>='
     - !ruby/object:Gem::Version
+      segments:
+      - 0
+      hash: 2
       version: '0'
+  none: false
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - '>='
     - !ruby/object:Gem::Version
+      segments:
+      - 0
+      hash: 2
       version: '0'
+  none: false
 requirements: []
 rubyforge_project:
-rubygems_version: 2.1.9
+rubygems_version: 1.8.24
 signing_key:
-specification_version: 4
+specification_version: 3
 summary: CEP engine/server with Esper and JRuby
 test_files:
 - spec/field_spec.rb

checksums.yaml

@@ -1,7 +0,0 @@
----
-SHA1:
-  metadata.gz: e528e20d062b04aa923fc369e3848b79af3988a5
-  data.tar.gz: 6b630871d14d48df16ca71521c0fbf1d9927e9cc
-SHA512:
-  metadata.gz: bb30da37af4e23c1a8c1f121afe314e88f31123547d4c6a588cbadca36134c543b3a34edfda9197e5ee1afdbe4883f38ed0096789de4331889363f0b73d06530
-  data.tar.gz: 3bc139cc9b58f567dcf340aa256275445df156dc524923edce8a99100d6a52f356938407f4436aaa931b6419e897ac85340fa9d7900bb9a71b27e23888bbcf22