norikra 0.1.3-java → 0.1.4-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6d5ba6d34ed7bba46947b8434b0ac32b8cb7ba2d
+  data.tar.gz: 2c6a5434cfc175aab9de2f89d3ab10198a75a06f
+SHA512:
+  metadata.gz: 48a696a0f5611dc7657c82e1e119a10355da055eadfd4417b0456508a345ce4a18bc8617bd4aa46435cf49c85c7e0863ef42e01747923de89a0695e9ea7a86db
+  data.tar.gz: 9b04d5197dd10a38c6d66dd09ae580509f2469c1d6727f774e203e61400796accc5c3de164df2262b0d0521959c49bb80bd8241648e339f234d75881b8994a71

data/.ruby-version

@@ -1 +1 @@
-jruby-1.7.4
+jruby-1.7.11

data/README.md

@@ -24,35 +24,9 @@ For JRuby installation, you can use `rbenv`, `rvm` and `xbuild`, or install JRub
 * rbenv: https://github.com/sstephenson/rbenv/
 * xbuild: https://github.com/tagomoris/xbuild
 
-### Command line options
+### Install and launch
 
-To start norikra server in foreground:
-
-    norikra start
-
-Norikra server doesn't save targets/queries in default.
-Specify `--stats STATS_FILE_PATH` option to save these runtime configuration automatically.
-
-    norikra start --stats /path/to/data/norikra.stats.json
-
-JVM options like `-Xmx` are available:
-
-    norikra start -Xmx2g
-
-To daemonize:
-
-    norikra start -Xmx2g --daemonize --logdir=/var/log/norikra
-    norikra start -Xmx2g --daemonize --pidfile /var/run/norikra.pid --logdir=/var/log/norikra
-    # To stop
-    norikra stop
-
-Performance options about threadings:
-
-    norikra start --micro     # or --small, --middle, --large
-
-For other options, see help:
-
-    norikra help start
+See: http://norikra.github.io/
 
 ### How to execute norikra server and tests in development
 
@@ -77,7 +51,12 @@ Execute norikra server with target/query continuation:
 
 ## Clients
 
-Use `Norikra::Client` and `norikra-client` cli command. These are available on both of JRuby and CRuby.
+Use `norikra-client` cli command. These are available on both of JRuby and CRuby.
+
+https://rubygems.org/gems/norikra-client
+https://rubygems.org/gems/norikra-client-jruby
+
+And the client library for application developers are also included in these gems.
 
 https://github.com/norikra/norikra-client-ruby
 
@@ -86,103 +65,12 @@ For other languages:
 
 ## Events and Queries
 
-For example, think about event streams related with one web service (ex: 'www'). At first, define `target` with mandantory fields (in other words, minimal fields set for variations of 'www' events).
-
-    norikra-client target open www path:string status:integer referer:string agent:string userid:integer
-    norikra-client target list
-
-Supported types are `string`, `boolean`, `integer`, `float` and `hash`, `array`.
-
-You can register queries when you want.
-
-    # norikra-client query add QUERY_NAME  QUERY_EXPRESSION
-    norikra-client query add www.toppageviews 'SELECT count(*) AS cnt FROM www.win:time_batch(10 sec) WHERE path="/" AND status=200'
-
-And send events into norikra (multi line events [json-per-line] and LTSV events are also allowed).
-
-    echo '{"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}' | norikra-client event send www
-    echo '{"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}' | norikra-client event send www
-    echo '{"path":"/content", "status":200, "referer":"/login", "agent":"MSIE", "userid":3}' | norikra-client event send www
-    echo '{"path":"/page/1", "status":200, "referer":"/content", "agent":"MSIE", "userid":3}' | norikra-client event send www
-
-Finally, you can get query outputs:
-
-    norikra-client event fetch www.toppageviews
-	{"time":"2013/05/15 15:10:35","cnt":1}
-	{"time":"2013/05/15 15:10:45","cnt":0}
-
-You can just add queries with optional fields:
-
-    norikra-client query add www.search 'SELECT count(*) AS cnt FROM www.win:time_batch(10 sec) WHERE path="/content" AND search_param.length() > 0'
-
-And send more events:
-
-    echo '{"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}' | norikra-client event send www
-    echo '{"path":"/", "status":200, "referer":"", "agent":"Firefox", "userid":4}' | norikra-client event send www
-    echo '{"path":"/content", "status":200, "referer":"/login", "agent":"MSIE", "userid":3}' | norikra-client event send www
-    echo '{"path":"/content", "status":200, "referer":"/login", "agent":"Firefox", "userid":4, "search_param":"news worldwide"}' | norikra-client event send www
-
-Query 'www.search' matches the last event automatically.
-
-## Performance
-
-Threads option available with `norikra start`. Simple specifiers for performance with threadings:
-
-    norikra start --micro     # or --small, --middle, --large (default: 'micro')
-
-Norikra server has 3 types of threads:
-
-* engine: 4 query engine thread types on Esper
-  * inbound: input data handler for queries
-  * outbound: output data handler for queries
-  * router: event handler which decides which query needs that events
-  * timer: executer for queries with time_batches and other timing events
-* rpc: data input/output rpc handler threads on Jetty
-* web: web ui request handler threads on Jetty
-
-In many cases, norikra server handling high rate events needs large number of rpc threads to handle input/output rpc requests. WebUI don't need threads rather than default in almost all of cases.
-
-Engine threads depends on queries running on norikra, input/output event data rate and target numbers. For more details, see Esper's API Documents: http://esper.codehaus.org/esper-4.10.0/doc/reference/en-US/html/api.html#api-threading
-
-Norikra's simple specifiers details of threadings are:
-
-* micro: development and testing
-  * engine: all processings on single threads
-  * rpc: 2 threads
-  * web: 2 threads
-* small: low rate events on virtual servers
-  * engine: inbound 1, outbound 1, route 1, timer 1 threads
-  * rpc: 2 threads
-  * web: 2 threads
-* middle: high rate events on physical servers
-  * engine: inbound 4, outbound 2, route 2, timer 2 threads
-  * rpc: 4 threads
-  * web: 2 threads
-* large: inbound heavy traffic and huge amount of queries
-  * engine: inbound 6, outbound 6, route 4, timer 4 threads
-  * rpc: 8 threads
-  * web: 2 threads
-
-To specify sizes of each threads, use `--*-threads=NUM` options. For more details, see: `norikra help start`.
-
-## User Defined Functions
-
-UDFs/UDAFs can be loaded as plugin gems over rubygems or as private plugins.
-In fact, Norikra's UDFs/UDAFs are Esper's plugin with a JRuby class to indicate plugin metadata.
-
-For details how to write your own UDF/UDAF for norikra and to release it as gem, see README of `norikra-udf-mock`.
-https://github.com/norikra/norikra-udf-mock
-
-## Tips
-
-We need to write this document on http://norikra.github.io
-
-### Esper query syntax
-
-* String concatination operator: `||`
+See: http://norikra.github.io/
 
 ## Changes
 
+* v0.1.3:
+ * Fix critical bug about query de-registration
 * v0.1.2:
  * Fix CLI start command to detect jruby path collectly (behind rbenv/rvm and others)
 * v0.1.1:

data/lib/norikra/cli.rb

@@ -18,6 +18,8 @@ module Norikra
                        :desc => 'status file path to load/dump targets, queries and server configurations [none]'
         option :'suppress-dump-stat', :type => :boolean, :default => false, \
                        :desc => 'specify not to update stat file with updated targets/queries/configurations on runtime [false]'
+        option :'dump-stat-interval', :type => :numeric, :default => nil, \
+                       :desc => 'interval(seconds) of status file dumps on runtime [none (on shutdown only)]'
 
         ### Daemonize options
         option :daemonize, :type => :boolean, :default => false, :aliases => "-d", \
@@ -135,7 +137,7 @@ module Norikra
 
       ### stat file
       conf[:stats] = {
-        path: options[:stats], suppress: options[:'suppress-dump-stat'],
+        path: options[:stats], suppress: options[:'suppress-dump-stat'], interval: options[:'dump-stat-interval'],
       }
 
       ### threads

data/lib/norikra/engine.rb

@@ -230,6 +230,7 @@ module Norikra
         trace "sendEvent", :data => formed
         @runtime.sendEvent(formed.to_java, fieldset.event_type_name)
       end
+      target.update!
       @statistics[:events][:processed] += events.size
       nil
     end

data/lib/norikra/rpc/handler.rb

@@ -103,6 +103,12 @@ class Norikra::RPC::Handler
     }
   end
 
+  def see(query_name)
+    logging(:show, :see, [query_name]){
+      @engine.output_pool.fetch(query_name)
+    }
+  end
+
   def sweep(query_group=nil)
     logging(:show, :sweep){
       @engine.output_pool.sweep(query_group)

data/lib/norikra/server.rb

@@ -82,6 +82,7 @@ module Norikra
 
       @stats_path = conf[:stats][:path]
       @stats_suppress_dump = conf[:stats][:suppress]
+      @stats_dump_interval = conf[:stats][:interval]
       @stats = if @stats_path && test(?r, @stats_path)
                  Norikra::Stats.load(@stats_path)
                else
@@ -147,11 +148,42 @@ module Norikra
       [:INT, :TERM].each do |s|
         Signal.trap(s, shutdown_proc)
       end
-      #TODO: SIGHUP? SIGUSR1? SIGUSR2? (dumps of query/fields? or other handler?)
+
+      @dump_stats = false
+      @dump_next_time = if @stats_dump_interval
+                          Time.now + @stats_dump_interval
+                        else
+                          nil
+                        end
+      Signal.trap(:USR1, ->{ @dump_stats = true })
+
+      #TODO: SIGHUP?(dynamic plugin loading?) SIGUSR2?
 
       while @running
         sleep 0.3
+
+        if @stats && !@stats_suppress_dump
+          if @dump_stats || (@dump_next_time && Time.now > @dump_next_time)
+            dump_stats
+            @dump_stats = false
+            @dump_next_time = Time.now + @stats_dump_interval if @dump_next_time
+          end
+        end
+      end
+    end
+
+    def shutdown
+      info "Norikra server shutting down."
+      @webserver.stop
+      @rpcserver.stop
+      @engine.stop
+      info "Norikra server stopped."
+
+      if @stats_path && !@stats_suppress_dump
+        dump_stats
       end
+
+      info "Norikra server shutdown complete."
     end
 
     def load_plugins
@@ -170,33 +202,23 @@ module Norikra
       end
     end
 
-    def shutdown
-      info "Norikra server shutting down."
-      @webserver.stop
-      @rpcserver.stop
-      @engine.stop
-      info "Norikra server stopped."
-
-      if @stats_path && !@stats_suppress_dump
-        stats = Norikra::Stats.new(
-          host: @host,
-          port: @port,
-          threads: @thread_conf,
-          log: @log_conf,
-          targets: @engine.targets.map{|t|
-            {
-              :name => t.name,
-              :fields => @engine.typedef_manager.dump_target(t.name),
-              :auto_field => t.auto_field
-            }
-          },
-          queries: @engine.queries.map(&:dump)
-        )
-        stats.dump(@stats_path)
-        info "Current status saved", :path => @stats_path
-      end
-
-      info "Norikra server shutdown complete."
+    def dump_stats
+      stats = Norikra::Stats.new(
+        host: @host,
+        port: @port,
+        threads: @thread_conf,
+        log: @log_conf,
+        targets: @engine.targets.map{|t|
+          {
+            :name => t.name,
+            :fields => @engine.typedef_manager.dump_target(t.name),
+            :auto_field => t.auto_field
+          }
+        },
+        queries: @engine.queries.map(&:dump)
+      )
+      stats.dump(@stats_path)
+      info "Current status saved", :path => @stats_path
     end
   end
 end

data/lib/norikra/target.rb

@@ -1,6 +1,6 @@
 module Norikra
   class Target
-    attr_accessor :name, :fields, :auto_field
+    attr_accessor :name, :fields, :auto_field, :last_modified
 
     def self.valid?(target_name)
       target_name =~ /^[a-zA-Z]([_a-zA-Z0-9]*[a-zA-Z0-9])?$/
@@ -10,6 +10,7 @@ module Norikra
       @name = name
       @fields = fields
       @auto_field = !!auto_field
+      @last_modified = nil
     end
 
     def <=>(other)
@@ -27,5 +28,9 @@ module Norikra
     def auto_field?
       @auto_field
     end
+
+    def update!
+      @last_modified = Time.now
+    end
   end
 end

data/lib/norikra/version.rb

@@ -1,3 +1,3 @@
 module Norikra
-  VERSION = "0.1.3"
+  VERSION = "0.1.4"
 end

data/lib/norikra/webui/handler.rb

@@ -68,7 +68,14 @@ class Norikra::WebUI::Handler < Sinatra::Base
       pooled_events = Hash[* queries.map{|q| [q.name, engine.output_pool.pool.fetch(q.name, []).size.to_s]}.flatten]
 
       engine_targets = engine.targets.sort
-      targets = engine_targets.map{|t| {name: t.name, auto_field: t.auto_field, fields: engine.typedef_manager.field_list(t.name) }}
+      targets = engine_targets.map{|t|
+        {
+          name: t.name,
+          auto_field: t.auto_field,
+          fields: engine.typedef_manager.field_list(t.name),
+          modified: t.last_modified.to_s,
+        }
+      }
 
       erb :index, :layout => :base, :locals => {
         input_data: input_data,

data/norikra.gemspec

@@ -22,8 +22,8 @@ Gem::Specification.new do |spec|
   spec.add_runtime_dependency "mizuno", "~> 0.6"
   spec.add_runtime_dependency "rack"
   spec.add_runtime_dependency "thor"
-  spec.add_runtime_dependency "msgpack-rpc-over-http-jruby", ">= 0.0.5"
-  spec.add_runtime_dependency "norikra-client-jruby", "~> 0.1.0"
+  spec.add_runtime_dependency "msgpack-rpc-over-http-jruby", ">= 0.0.6"
+  spec.add_runtime_dependency "norikra-client-jruby", ">= 0.1.4"
   spec.add_runtime_dependency "sinatra"
   spec.add_runtime_dependency "sinatra-contrib"
   spec.add_runtime_dependency "erubis"

data/views/base.erb

@@ -45,7 +45,7 @@
             <li <%= @page == "queries" ? 'class="active"' : '' %>><a href="#targets">Targets</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
-            <li><a href="https://github.com/tagomoris/norikra">Webpage</a></li>
+            <li><a href="https://github.com/norikra/norikra">Webpage</a></li>
           </ul>
         </div><!--/.nav-collapse -->
       </div>

data/views/index.erb

@@ -66,7 +66,7 @@
 <% if queries.size > 0 %>
 <table class="table">
   <tr><th>Group</th><th>Query name</th><th>Targets</th><th>Query</th><th style="text-align:right;">Events</th><th></th><th></th></tr>
-  <% queries.each do |query| %>
+  <% queries.each_with_index do |query, index| %>
   <tr>
     <td><%= query.group || "(default)" %></td>
     <td><%= query.name %></td>
@@ -83,12 +83,12 @@
       <% end %>
     </td>
     <td>
-      <a class="btn btn-danger btn-xs" data-toggle="modal" href="#removeQuery<%= query.name %>">
+      <a class="btn btn-danger btn-xs" data-toggle="modal" href="#removeQuery<%= index %>">
         <span class="glyphicon glyphicon-trash"></span>
       </a>
       <div class="modal fade"
-        id="removeQuery<%= query.name %>"
-        tabindex="-1" role="dialog" aria-labelledby="removeQueryLabel<%= query.name %>" aria-hidden="true">
+        id="removeQuery<%= index %>"
+        tabindex="-1" role="dialog" aria-labelledby="removeQueryLabel<%= index %>" aria-hidden="true">
         <div class="modal-dialog">
           <div class="modal-content">
             <div class="modal-header">
@@ -151,7 +151,7 @@
 
 <% if targets.size > 0 %>
 <table class="table">
-  <tr><th>Target</th><th>Auto field</th><th>Fields</th><th></th></tr>
+  <tr><th>Target</th><th>Auto field</th><th>Fields</th><th></th><th>modified</th></tr>
   <% targets.each do |target| %>
   <tr>
     <td><%= target[:name] %></td>
@@ -164,6 +164,7 @@
       <% end %>
     </td>
     <td style="font-size: small; color: #c0c0c0;">TODO: change auto_fields, reserve fields, remove targets</td>
+    <td style="text-align: right;"><%= target[:modified] %></td>
   </tr>
   <% end %>
 </table>

metadata

@@ -1,15 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: norikra
 version: !ruby/object:Gem::Version
-  prerelease:
-  version: 0.1.3
+  version: 0.1.4
 platform: java
 authors:
 - TAGOMORI Satoshi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2013-12-31 00:00:00.000000000 Z
+date: 2014-03-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mizuno
@@ -18,13 +17,11 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.6'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.6'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -34,13 +31,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -50,13 +45,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -65,30 +58,26 @@ dependencies:
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 0.0.5
-    none: false
+        version: 0.0.6
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 0.0.5
-    none: false
+        version: 0.0.6
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
   name: norikra-client-jruby
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 0.1.0
-    none: false
+        version: 0.1.4
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 0.1.0
-    none: false
+        version: 0.1.4
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -98,13 +87,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -114,13 +101,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -130,13 +115,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
@@ -146,13 +129,11 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.3'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.3'
-    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -162,13 +143,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -178,13 +157,11 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '2.0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '2.0'
-    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -194,13 +171,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
@@ -210,13 +185,11 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-    none: false
   prerelease: false
   type: :development
 description: 'CEP: Complex Event Processor with Esper EPL qeury language, messagepack
@@ -361,6 +334,7 @@ files:
 homepage: http://norikra.github.io/
 licenses:
 - GPLv2
+metadata: {}
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -370,25 +344,17 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - '>='
     - !ruby/object:Gem::Version
-      segments:
-      - 0
-      hash: 2
       version: '0'
-  none: false
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - '>='
     - !ruby/object:Gem::Version
-      segments:
-      - 0
-      hash: 2
       version: '0'
-  none: false
 requirements: []
 rubyforge_project:
-rubygems_version: 1.8.24
+rubygems_version: 2.1.9
 signing_key:
-specification_version: 3
+specification_version: 4
 summary: CEP engine/server with Esper and JRuby
 test_files:
 - spec/field_spec.rb