norikra 0.1.7-java → 1.0.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c60623a71ba50774fd30d435b33ddd5abddb5a32
-  data.tar.gz: 4540758d58b47d144feaa504087a393a7853d380
+  metadata.gz: 538c722175146fe4406fa69a0d9c98a2d8c560e3
+  data.tar.gz: 3ac5dc22cd77d7b6faa7f1e5f9e025834a57dfe7
 SHA512:
-  metadata.gz: fba4984d9d6c96573f169d5d7fe1f62e6bf711666c0546886d11fce7723beaaf64dce7a30d884b1c05870c7edce49603345e18d660c6d67824d408e22ef768ba
-  data.tar.gz: 3359da4c4bd851e4f2191ab859b902843619d17124da18b5fe6ab7f3c9fe1d07afed9d12b0e4579fc5b81c40ebba7db0a8035fe1e831dd0dea3ec7367c7d820f
+  metadata.gz: b6c74589514fc80a08f122e3422acc5c4ca6a00077111d98cefc44e75c0e190d9cf3600d7e7135864b2b2045547cd801952522396f8a581756295475108ac5c4
+  data.tar.gz: 610063d16fb5f1ca4c88dfeb61b6119985952c391fa4698a3d2957e5207453e61a796e0e3cfae6d8180656d06c61496867e740a319ca13159446a56f96422740

data/README.md

@@ -62,6 +62,7 @@ https://github.com/norikra/norikra-client-ruby
 
 For other languages:
  * Perl: https://github.com/norikra/norikra-client-perl
+ * Python: https://github.com/norikra/norikra-client-python
 
 ## Events and Queries
 
@@ -69,6 +70,22 @@ See: http://norikra.github.io/
 
 ## Changes
 
+* v1.0.0:
+  * Update esper version (4.9.0 -> 5.0.0)
+    * Support Group By ROLLUP, Grouping Sets and CUBE
+    * Support Group By clause in subqueries
+    * Support subqueries that select multiple columns to provide input to enumeration methods
+    * Support Having clause to have subquery
+  * Add loopback query group to connect query output into targets directly
+  * RPC:
+    * Add JSON RPC API (/api on port 26578)
+    * Add API to fetch norikra server logs
+  * Add `--stats-secondary` option to store versioned stats files
+  * Fix bug:
+    * to return `[]` for NULL query output (fixed to `nil`)
+    * to handle container value for simple value field (exception -> NULL)
+* v0.1.7:
+  * Fix `Pattern` support bug
 * v0.1.6:
   * Fix bug: Wrong escape for java instance method calls
 * v0.1.5:

data/esper/changelog.txt

@@ -1,6 +1,109 @@
 Esper Changelog
 ===================
 
+Changes in version 5.0.0
+------------------------
+  - The following dependent jar files are upgraded:
+       - ANTLR runtime library version is now 4.1 (antlr-runtime-4.1.jar) (previously antlr-runtime-3.2.jar)
+       - CGLIB version is now 3.1 (cglib-nodep-3.1.jar) (previously cglib-nodep-2.2.jar)
+       - Commons-logging version is now 1.1.3 (commons-logging-1.1.3.jar) (previously commons-logging-1.1.1.jar)
+       - (optionally) Log4j version is now 1.2.17 (log4j-1.2.17.jar) (previously log4j-1.2.16.jar)
+  - Support for rollup and cube aggregation output. Support for SQL standard output and SQL keywords namely "rollup", "cube", "grouping sets", "grouping", "grouping_id".
+  - Support for fire-and-forget insert-into queries using the "values" keyword i.e. "insert into ... values (...)"
+  - Support for Java 8-style lambda syntax using the "->" operator the same way as "=>" as equivalent syntax
+  - Support for context partitioned variables. Use "context ... create variable ..." to create a contex partitioned variable.
+  - Support for method invocation joins to receive object-array events from application code
+  - Support for query planner expression analysis hints that provide additional control over query planning
+  - Support for the query planner for joins, subqueries, fire-and-forget and on-action queries to consider in-keyword expressions and "or"
+  - Support for filter execution planning to consider "or"
+  - Statements on unbound streams by default no longer retain the last event for use with the iterator API, and instead an application can specify @IterableUnbound or provide an engine configuration to have statements retain the last event
+  - Statement names are now always trimmed
+  - Applications that use isolated service providers must now set an engine-wide configuration flag to enable this feature
+  - The "as" keyword in the select-clause is now optional 
+  - Improved rendering of expressions by rendering parenthesis according to precedence and by removing white space in expressions. 
+    This may affect statements where the select-clause does not name columns using "as" since they may receive a different column name compared to version 4.
+  - We have changed time period expression month and year interval computation to become relative to the current time. This change disallows float-type values for year and month in a time interval.
+  - Improved formatting and content of validation messages
+  - Many performance related enhancements
+  - Removed version 4 deprecated APIs
+
+ 
+Changes in version 4.11.0
+------------------------
+  - Support for patterns to suppress and discard overlapping matches
+  - Support for match-recognize define-clause and measures-clause to use enumeration methods and array-index access for grouped variables
+  - Support for match-recognize matching upon pattern state termination when used with interval
+  - Support for overlapping context to specify "distinct"
+  - Support for group-by clause in subqueries
+  - Support for subqueries that select multiple columns to provide input to enumeration methods
+  - Support for built-in annotations to be case-insensitive
+
+Minor EPL changes
+  - When in a select-clause the "as" column name is surrounded with backticks (e.g. "select name as `somename` ...") the property name of the output event does no longer carry backticks (i.e. is "somename").
+
+Minor API changes
+  - SODA class PatternMatchUntilExpr now requires to set single-bounds as a separate expression
+
+Bug fixes and other improvements
+  - Fixed issue XIN-243-46130 Window(*) returning scalars throws NPE when used with null values
+  - Fixed issue FOT-686-75760 Context partition using for-clause with keyed-segmented context when for-clause utilizes the same key the for-clause is not grouping
+  - Fixed issue NUO-642-65913 When JMS unmarshaller receives an exception log error and continue
+  - Fixed issue LTU-232-62589 Map/oa type with nested pojo with generic-typed list not producing type fragments
+  - Fixed issue ZXU-154-59144 Context partitioned join against virtual data window throws NPE
+  - Fixed issue IOO-666-18256 EPL module with script that has brackets and semicolon throws exception on deployment
+  - Fixed issue UHA-316-24638 Fully-aggregated subquery without filter throws NPE; subquery against context-partitioned VDW throws NPE
+  - Fixed issue ESPER-754 Class that extends class triggers two events when named window declared for both child and parent class
+  - Fixed issue ESPER-756 NPE during destroy() when using inbound thread pool and when not waiting for pool to empty
+  - Fixed issue ESPER-757 VDW callback on consuming rstream events
+  - Fixed issue ESPER-758 Thread safety issue with contained-event selection
+  - Fixed issue ESPER-759 Pattern with not-operator controlled by repeat ([n]) does not turn repeat to false
+  - Fixed issue ESPER-761 On-Demand query inserting row into named window that contains an enumeration throws NPE
+  - Fixed issue ESPER-762 Keyed segmented context with match recognize and "prev" in define-clause causes first event to not evaluate
+  - Fixed issue ESPER-763 Allow on-merge assignment of event-typed property
+  - Fixed issue ESPER-765 Context declaration with @Priority causes first event to arrive twice
+  - Fixed issue ESPER-766 Allow enumeration method in pattern filter to use pattern tagged events as input
+  - Fixed issue ESPER-767 Support for tick-escape in select-clause column name
+  - Fixed issue ESPER-768 Improve standard deviation to use Knuth's
+  - Fixed issue ESPER-769 Issue evaluating nested enumeration
+  - Fixed issue ESPER-770 Improve compatibility checking when named window event gets inserted into stream of same-as type (Map and ObjectArray only)
+  - Fixed issue ESPER-771 Allow subqueries that select a single column that it itself an event to provide enumerable events and insert-into compatible prop
+  - Fixed issue ESPER-772 ClassCastException for method invocation on a property that is itself an event
+
+Changes in version 4.10.0
+------------------------
+Major API or EPL changes
+  - Support for context declarations to perform initiate or start immediately
+  - Support for mutation expressions for merge and update
+  - Support for parameterized fire-and-forget queries
+  - Support for fire-and-forget queries to consider the "in" opertator for index lookup
+  - Support for having-clause to have subquery
+  - Support for explicitly selecting eventbean instances of expressions (ESPER-732)
+  - Support for fire-and-forget queries that insert-into a named window
+
+Bug fixes
+  - Fixed Issue ESPER-733 Passing aggregate scalars to global script
+  - Fixed Issue ESPER-734 Unclear exception for invalid match_recognize
+  - Fixed Issue ESPER-735 Esper 4.9.0 OSGi bundle execution environment restricted to J2SE 1.6
+  - Fixed Issue ESPER-737 NullPointerException with grouped data window intersecting time window and last/first event
+  - Fixed Issue ESPER-738 Performance issue with 8 threads, keyed context partitioned, synchronized listener, event sequence for same partition
+  - Fixed Issue ESPER-739 All calls methods named "get" assumed to be Date-Time method
+  - Fixed Issue ESPER-740 POJO event populated by selectclause and via POJO constructor and select clause selects same type twice is created using wrong parameters
+  - Fixed Issue ESPER-741 Named window on-action with relational operator (<, <=) where-clause incorrect behavior when reversing comparison
+  - Fixed Issue ESPER-742 Static method lookup selects incorrect classstatic method lookup
+  - Fixed Issue ESPER-743 Join accross ObjectArrayEvent and MapEvent leaks data from MapEvent
+  - Fixed Issue ESPER-744 parse error in prepared statement for count(*,filter_expr)
+  - Fixed Issue ESPER-746 Self-Join between derived value streams provides incorrect result
+  - Fixed Issue ESPER-747 Cron timer is skipping months
+  - Fixed Issue ESPER-748 create schema fails with fully qualified property type name
+  - Fixed Issue ESPER-749 split stream with "output all" not working with prepared statement
+  - Fixed Issue ESPER-750 @audit miscounts pattern instances with "not"
+  - Fixed Issue ESPER-751 StackOverflow when routing new event during high volume processing
+  - Fixed Issue ESPER-752 @Drop annotation with context: First event not dropped
+  - Fixed Issue KPH-656-86796 EPL module with javascript incorrectly parsed
+  - Fixed Issue PJM-181-84952 Possible deadlock for patterns that have filters that have subqueries
+  - Fixed Issue IGC-717-55951 efficiency of group reclaim with grouped data windows
+  - Fixed Issue TBC-634-19891 use thread context class loader for default deserialize
+
 Changes in version 4.9.0
 ------------------------
 Major API or EPL changes

data/esper/esper/lib/esper_3rdparties.license

@@ -1,21 +1,15 @@
-**** ANTLR 3 License
+**** ANTLR 4 License (see http://www.antlr.org/license.html)
 
 [The BSD License]
-Copyright (c) 2005, Terence Parr
-All rights reserved. 
-Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 
-
-Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 
-Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 
-Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 
+Copyright (c) 2012 Terence Parr and Sam Harwell
+All rights reserved.
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
 
 
-ANTLR 2 License
-
-We reserve no legal rights to the ANTLR--it is fully in the public domain. An individual or company may do whatever they wish with source code distributed with ANTLR or the code generated by ANTLR, including the incorporation of ANTLR, or its output, into commerical software. 
-We encourage users to develop software with ANTLR. However, we do ask that credit is given to us for developing ANTLR. By "credit", we mean that if you use ANTLR or incorporate any source code into one of your programs (commercial product, research project, or otherwise) that you acknowledge this fact somewhere in the documentation, research report, etc... If you like ANTLR and have developed a nice tool with the output, please mention that you developed it using ANTLR. In addition, we ask that the headers remain intact in our source code. As long as these guidelines are kept, we expect to continue enhancing this system and expect to make other tools available as they are completed. 
-
 **** Apache Commons Logging, CGLIB bytecode generator, Apache Log4j Library, Apache Commons Collection
 
 

data/esper/esper/lib/readme.txt

@@ -1,15 +1,15 @@
 Required runtime libraries
 ==========================
 
-antlr-runtime-3.2
+antlr-runtime-4.1
 - runtime for ANTLR parser generator
 - required for runtime
 
-commons-logging-1.1.1.jar
+commons-logging-1.1.3.jar
 - Apache Commons Logging, for logging
 - required for runtime
 
-cglib-nodep-2.2.jar
+cglib-nodep-3.1.jar
 - CGLIB bytecode generator
 - required for runtime
 
@@ -17,7 +17,7 @@ cglib-nodep-2.2.jar
 Optional runtime libraries
 ==========================
 
-log4j-1.2.16.jar
+log4j-1.2.17.jar
 - Log4j Library for logging
 - runtime, optional
 
@@ -25,14 +25,14 @@ log4j-1.2.16.jar
 Compile and Build-time libraries - Not part of the binary distribution and not required for runtime
 ================================
 
-junit-4.8.1.jar
+junit-4.11.jar
 - JUnit test framework
 - buildtime, not required for runtime
 
-antlr-3.2.jar
+antlr-4.1-complete.jar
 - ANTLR parser generator, for compile-time generation of parser and tree walker from grammars
 - buildtime, not required for runtime
 
-mysql-connector-java-5.1.13-bin.jar
+mysql-connector-java-5.1.28-bin.jar
 - MySQL JDBC driver, for unit testing of database connectivity
 - buildtime, not required for runtime

data/lib/norikra/cli.rb

@@ -17,6 +17,8 @@ module Norikra
 
         option :stats, :type => :string, :default => nil, :aliases => "-s", \
                        :desc => 'status file path to load/dump targets and queries [none]'
+        option :'stats-secondary', :type => :string, :default => nil, \
+                       :desc => 'status file secondary path, to dump stats with date/time, like "stats.%Y%m%d.json" [none]'
         option :'suppress-dump-stat', :type => :boolean, :default => false, \
                        :desc => 'specify not to update stat file with updated targets/queries on runtime [false]'
         option :'dump-stat-interval', :type => :numeric, :default => nil, \
@@ -65,6 +67,7 @@ module Norikra
                :desc => "directory path of logfiles when daemonized [nil (console for foreground)]"
         option :'log-filesize', :type => :string, :default => nil, :desc => 'log rotation size [10MB]'
         option :'log-backups' , :type => :numeric, :default => nil, :desc => 'log rotation backups [10]'
+        option :'log-buffer-lines', :type => :numeric, :default => nil, :desc => 'log lines to fetch from API [1000]'
 
         ### Loglevel options
         option :'more-quiet',   :type => :boolean, :default => false,                   :desc => 'set loglevel as ERROR'
@@ -187,7 +190,8 @@ module Norikra
 
       ### stat file
       conf[:stats] = {
-        path: options[:stats], suppress: options[:'suppress-dump-stat'], interval: options[:'dump-stat-interval'],
+        path: options[:stats], secondary_path: options[:'stats-secondary'],
+        suppress: options[:'suppress-dump-stat'], interval: options[:'dump-stat-interval'],
       }
 
       ### threads
@@ -218,7 +222,9 @@ module Norikra
                  else nil # for default (assumed as 'INFO')
                  end
       conf[:log] = {
-        level: loglevel, dir: options[:logdir], filesize: options[:'log-filesize'], backups: options[:'log-backups'],
+        level: loglevel, dir: options[:logdir],
+        filesize: options[:'log-filesize'], backups: options[:'log-backups'],
+        bufferlines: options[:'log-buffer-lines'],
       }
 
       server_options = {

data/lib/norikra/engine.rb

@@ -6,10 +6,10 @@ require 'norikra/target'
 require 'norikra/logger'
 include Norikra::Log
 
-require 'esper-4.9.0.jar'
-require 'esper/lib/commons-logging-1.1.1.jar'
-require 'esper/lib/antlr-runtime-3.2.jar'
-require 'esper/lib/cglib-nodep-2.2.jar'
+require 'esper-5.0.0.jar'
+require 'esper/lib/commons-logging-1.1.3.jar'
+require 'esper/lib/antlr-runtime-4.1.jar'
+require 'esper/lib/cglib-nodep-3.1.jar'
 
 require 'norikra/typedef_manager'
 
@@ -255,7 +255,9 @@ module Norikra
         event.keys.each do |key|
           trace "event content key:#{key}, value:#{event[key]}, value class:#{event[key].class}"
           unescaped_key = Norikra::Field.unescape_name(key)
-          if event[key].respond_to?(:to_hash)
+          if event[key].nil?
+            converted[unescaped_key] = nil
+          elsif event[key].respond_to?(:to_hash)
             converted[unescaped_key] = event[key].to_hash
           elsif event[key].respond_to?(:to_a)
             converted[unescaped_key] = event[key].to_a
@@ -274,6 +276,30 @@ module Norikra
         @events_statistics[:output] += events.size
       end
     end
+
+    class LoopbackListener < Listener
+      def initialize(engine, query_name, query_group, events_statistics)
+        @engine = engine
+        @query_name = query_name
+        @query_group = query_group
+        @events_statistics = events_statistics
+        @loopback_target = Norikra::Query.loopback(query_group)
+      end
+
+      def update(new_events, old_events)
+        t = Time.now.to_i
+        event_list = new_events.map{|e| type_convert(e) }
+        trace "loopback event", :query => @query_name, :group => @query_group, :event => event_list
+        @events_statistics[:output] += event_list.size
+        #
+        # We does NOT convert 'container.$0' into container['field'].
+        # Use escaped names like 'container__0'. That is NOT so confused.
+        #
+        #TODO: check performance and consider about loopback event threads
+        @engine.send(@loopback_target, event_list)
+      end
+    end
+
     ##### Unmatched events are simply ignored
     # class UnmatchedListener
     #   include com.espertech.esper.client.UnmatchedListener
@@ -339,8 +365,22 @@ module Norikra
     end
 
     def register_query(query)
+
+      if lo_target_name = Norikra::Query.loopback(query.group)
+        raise "Invalid loopback target name should be checked before. THIS IS BUG." unless Norikra::Target.valid?(lo_target_name)
+
+        target = Norikra::Target.new(lo_target_name)
+        unless @targets.include?(target)
+          info "opening loopback target", :target => lo_target_name
+          open_target(target)
+        end
+      end
+
       @mutex.synchronize do
         raise Norikra::ClientError, "query '#{query.name}' already exists" unless @queries.select{|q| q.name == query.name }.empty?
+        if lo_target_name = Norikra::Query.loopback(query.group)
+          raise Norikra::ClientError, "loopback target '#{lo_target_name}'" unless Norikra::Target.valid?(lo_target_name)
+        end
 
         unless @typedef_manager.ready?(query)
           @waiting_queries.push(query)
@@ -446,11 +486,20 @@ module Norikra
       statement_model = administrator.compileEPL(query.expression)
       Norikra::Query.rewrite_query(statement_model, event_type_name_map)
 
+      listener = if Norikra::Query.loopback(query.group)
+                   LoopbackListener.new(self, query.name, query.group, @statistics[:events])
+                 else
+                   Listener.new(query.name, query.group, @output_pool, @statistics[:events])
+                 end
+
       epl = administrator.create(statement_model)
-      epl.java_send :addListener, [com.espertech.esper.client.UpdateListener.java_class], Listener.new(query.name, query.group, @output_pool, @statistics[:events])
+      epl.java_send :addListener, [com.espertech.esper.client.UpdateListener.java_class], listener
+
       query.statement_name = epl.getName
+
       # epl is automatically started.
       # epl.isStarted #=> true
+      true
     end
 
     # this method should be protected with @mutex lock

data/lib/norikra/field.rb

@@ -175,9 +175,22 @@ module Norikra
     # def value(event) # by define_value_accessor
 
     def format(value, element_path=nil) #element_path ex: 'fname.fchild', 'fname.$0', 'f.fchild.$2'
+      return nil if value.nil? || value.is_a?(Hash) || value.is_a?(Array)
+
       case @type
       when 'string'  then value.to_s
-      when 'boolean' then value =~ /^(true|false)$/i ? ($1.downcase == 'true') : (!!value)
+      when 'boolean'
+        if value.is_a?(TrueClass) || value.is_a?(FalseClass)
+          value
+        elsif value.is_a?(String)
+          if value =~ /^(true|false)$/i
+            value.downcase == 'true'
+          else
+            !!value
+          end
+        else
+          !!value
+        end
       when 'integer' then value.to_i
       when 'float' then value.to_f
       when 'hash', 'array'

data/lib/norikra/logger.rb

@@ -1,5 +1,5 @@
 require 'java'
-require 'esper/lib/commons-logging-1.1.1.jar'
+require 'esper/lib/commons-logging-1.1.3.jar'
 
 require 'monitor'
 
@@ -96,7 +96,7 @@ module Norikra
         p.setProperty('log4j.rootLogger', "#{@@level},default")
         org.apache.log4j.PropertyConfigurator.configure(p)
 
-        @@logger = Logger.new('norikra.log')
+        @@logger = Logger.new('norikra.log', opts)
 
       else # for test(rspec)
         p.setProperty('log4j.appender.default', 'org.apache.log4j.varia.NullAppender')
@@ -156,32 +156,59 @@ module Norikra
   end
 
   class Logger
-    def initialize(name)
+    attr_reader :buffer
+
+    DEFAULT_MEMORY_BUFFER_LINES = 1000
+    TIME_FORMAT = '%Y-%m-%d %H:%M:%S %z'
+
+    def initialize(name, opts={})
+      p opts
       @log4j = org.apache.commons.logging.LogFactory.getLog(name)
+      @buffer = Array.new
+      @buffer_lines = opts[:bufferlines] || DEFAULT_MEMORY_BUFFER_LINES
+    end
+
+    def push(level, line)
+      if @buffer.size == @buffer_lines
+        @buffer.shift
+      end
+      @buffer << [Time.now.strftime(TIME_FORMAT), level, line]
     end
 
     def trace(message, data=nil, from=nil)
-      @log4j.trace(format(from, message, data))
+      line = format(from, message, data)
+      push('trace', line)
+      @log4j.trace(line)
     end
 
     def debug(message, data=nil, from=nil)
-      @log4j.debug(format(from, message, data))
+      line = format(from, message, data)
+      push('debug', line)
+      @log4j.debug(line)
     end
 
     def info(message, data=nil, from=nil)
-      @log4j.info(format(from, message, data))
+      line = format(from, message, data)
+      push('info', line)
+      @log4j.info(line)
     end
 
     def warn(message, data=nil, from=nil)
-      @log4j.warn(format(from, message, data))
+      line = format(from, message, data)
+      push('warn', line)
+      @log4j.warn(line)
     end
 
     def error(message, data=nil, from=nil)
-      @log4j.error(format(from, message, data))
+      line = format(from, message, data)
+      push('error', line)
+      @log4j.error(line)
     end
 
     def fatal(message, data=nil, from=nil)
-      @log4j.fatal(format(from, message, data))
+      line = format(from, message, data)
+      push('fatal', line)
+      @log4j.fatal(line)
     end
 
     def format_location(locations)
@@ -213,7 +240,6 @@ module Norikra
     # LOG_LOG4J_FORMAT = '%d{yyyy-MM-dd HH:mm:ss Z} [%p] %m%n'
     # LOG_FORMAT = '%s: %s%s'
     FORMAT_SIMULATED = "%s [%s] %s\n"
-    FORMAT_SIMULATED_TIME = '%Y-%m-%d %H:%M:%S %z'
     attr_accessor :logs, :output
     def initialize
       @logs = { :TRACE => [], :DEBUG => [], :INFO => [], :WARN => [], :ERROR => [], :FATAL => [] }
@@ -221,7 +247,7 @@ module Norikra
     end
     def log(level, message, data, from)
       @logs[level].push({:message => message, :data => data, :from => from})
-      formatted = sprintf(FORMAT_SIMULATED, Time.now.strftime(FORMAT_SIMULATED_TIME), level.to_s, format(from, message, data))
+      formatted = sprintf(FORMAT_SIMULATED, Time.now.strftime(TIME_FORMAT), level.to_s, format(from, message, data))
       @output.push(formatted)
     end
     def trace(m,d,f); self.log(:TRACE,m,d,f); end