processor 1.0.0 → 2.0.0

data/.coveralls.yml

@@ -0,0 +1 @@
+service_name: travis-ci

data/.gitignore

@@ -7,6 +7,7 @@ Gemfile.lock
 /tmp
 /log
 /tags
+/coverage
 
 ## generic files to ignore
 *~

data/README.md

@@ -1,15 +1,31 @@
 Processor
 ==========
 [![Build Status](https://travis-ci.org/AlexParamonov/processor.png?branch=master)](http://travis-ci.org/AlexParamonov/processor)
-[![Gemnasium Build Status](https://gemnasium.com/AlexParamonov/processor.png)](http://gemnasium.com/AlexParamonov/processor)  
+[![Gemnasium Build Status](https://gemnasium.com/AlexParamonov/processor.png)](http://gemnasium.com/AlexParamonov/processor)
+[![Coverage Status](https://coveralls.io/repos/AlexParamonov/processor/badge.png?branch=master)](https://coveralls.io/r/AlexParamonov/processor?branch=master)
 
-Processor could execute any `DataProcessor` you specify and log entire
-process using any number of loggers you need.
 
-You may add own observers for monitoring background tasks on even send
-an email to bussiness with generated report.
+Processor is a tool that helps to iterate over collection and
+perform complex actions on a result. It is extremely useful in data
+migrations, report generation, etc.
 
-Processor provide customisation for almost every part of it.  
+Collection could be iteratively fetched by parts but for processor
+it will looks like an endless collection. There are a lot such tiny
+goodness that makes usage of processor pleasant. Need logging,
+exception processing, post/pre processing a result - no problems, all
+included and easily extended.
+
+Use the processor to DRY your migrations, report and stop mess with
+logging and post processing.
+
+Did I mentioned you can run in threads as easy as say
+`processor.run_in_threads 10`?
+
+Processor could execute any `DataProcessor` you specify and log
+entire process using any number of loggers you need. You may add own
+observers for monitoring background tasks on even send an email to
+business with generated report. Processor provide customisation for
+almost every part of it.
 
 
 Contents
@@ -21,6 +37,7 @@ Contents
     1. Run modes
     1. Processor Thread
     1. Observers
+1. Contacts
 1. Compatibility
 1. Contributing
 1. Copyright
@@ -51,20 +68,29 @@ Usage
 ------------
 
 ### Data processors
-Actual processing is done by a Data Processor, provided by end user.  
-This processor should implement in general 2 methods:
+Working with a data is responsibility of a `DataProcessor`.
 
-1. `process(record)`
-1. `records`
+`DataProcessor` should obtain records to process by its `records`
+method and process a record by `process` method. If some post/pre
+action is needed, it could be performed inside `start` and `finish`
+methods. In case of exceptions there are `error(exception)` and
+`record_error(record, exception)` methods. `error` method is called if
+unprocessed errors happened during processing and `record_error` if
+processing current record raised. `finalize` method will run in any
+case allowing you to gracefully finalize processing.
 
-But it is recomended to implement a `name` method also, because it
-is required by several observers. Inherit your Data Processor from
-NullProcessor to get default behavior out of the box.
+To add new `DataProcessor` it is recommended to inherit from
+`NullProcessor` and implement methods that are needed only.
 
-See `Processor::Example::Migration` for example (`example/migration.rb`).
+`Processor` provides several data processors:
 
-There are several predefined data processors you can reuse:
+1. NullProcessor [[code](https://github.com/AlexParamonov/processor/blob/master/lib/processor/data/null_processor.rb), [specs](https://github.com/AlexParamonov/processor/blob/master/spec/processor/data/null_processor_spec.rb)]
+1. ArrayProcessor [[code](https://github.com/AlexParamonov/processor/blob/master/lib/processor/data/array_processor.rb), [specs](https://github.com/AlexParamonov/processor/blob/master/spec/processor/data/array_processor_spec.rb)]
+1. BatchProcessor [[code](https://github.com/AlexParamonov/processor/blob/master/lib/processor/data/batch_processor.rb), [specs](https://github.com/AlexParamonov/processor/blob/master/spec/processor/data/batch_processor_spec.rb)]
+1. CsvProcessor
+1. SolrPagesProcessor
 
+The last two are more as example, your probably would change them.
 
 #### ArrayProcessor
 The simplest one: `process` and `records` methods should be implemented.
@@ -73,13 +99,12 @@ The simplest one: `process` and `records` methods should be implemented.
 #### BatchProcessor
 Allows to fetch records by batches of defined size.
 
-It is based on `query` method that suppose to run a query method on
-database.
+It is based on `query` method that is supposed to run a query method on
+a database.
 
-Recomended to override `fetch_batch` method to get real reason to
-use batch processing. `fetch_batch` could be `query.first(10)` or
-`query.page(next_page)`. See `data/solr_pages_processor.rb` and
-`data/solr_processor.rb` for example.
+Recommended to override `fetch_batch` method to get real reason to use
+batch processing. `fetch_batch` could be `query.page(next_page)` for
+example. See `data/solr_pages_processor.rb`.
 
 
 #### Other
@@ -94,7 +119,7 @@ Currently 2 run modes are supported:
 It runs `process` one by one for each found record returned by
 `records` method.
 
-Recomended to call it using a `Processor::Thread`:
+Call it using a `Processor::Thread`:
 ``` ruby
 Processor::Thread.new(migration).run_successive
 ```
@@ -109,19 +134,20 @@ constructor:
 Processor::ProcessRunner::Threads.new 5
 ```
 
-Recomended to call it using a Processor::Thread :
+Call it using a `Processor::Thread`:
 ``` ruby
 Processor::Thread.new(migration).run_in_threads 5
 ```
 
 
 ### Observers
-Processor support unlimited number of observers, watching processing.
+Processor support unlimited number of observers that are watching
+processing.
 
-Thay could monitor running migrations and output to logs, console or
-file usefull information. Or thay can show a progress bar to your
-console. Or pack a generated report to archive and send by email to
-bussiness on success or notify developers on failure.
+They could monitor `Data Processor`s and output to logs, console
+or file. Or they can show a progress bar on the console. Or pack a
+generated report to archive and send it by email to the business on
+success or notify developers on failure.
 
 
 This observers should respond to `update` method. But if you inherit
@@ -134,7 +160,7 @@ Read below section Processor Thread to see how to use observers in runner.
 
 ### Processor Thread
 `Processor::Thread` is a Facade pattern. It simplifies access to all
-Processor classes and provide __stable__ interface.
+Processor classes and provides __stable__ interface.
 
 Creating a new Thread:
 ``` ruby
@@ -149,20 +175,13 @@ Processor::Thread.new data_processor, observer1, observer2, ...
 Instance have a `run_as` method that accepts a block:
 ``` ruby
 thread = Processor::Thread.new @migration
-thread.run_as do |processor, *|
+thread.run_as do |processor|
   processor.records.each do |record|
     processor.process record
   end
 end
 ```
 
-Block could accept next arguments: `processor`, `events`,
-`recursion_preventer` method. Last one could be called to prevent
-recurtion:
-``` ruby
-recursion_preventer.call
-```
-
 Instance have a `run_successive` method: 
 ``` ruby
 data_processor = UserLocationMigration.new
@@ -180,7 +199,7 @@ thread.run_in_threads 10
 See `spec/processor/thread_spec.rb` and `spec/example_spec.rb` and
 `example` directory for other usage examples.
 
-It is recomended to wrap Processor::Thread by classes named like:
+It is recommended to wrap Processor::Thread by classes named like:
 ``` ruby
 WeeklyReport
 TaxonomyMigration
@@ -191,11 +210,11 @@ The point is to hide configuration of observers and use (if you wish)
 your own API to run reports or migrations:
 ``` ruby
 weekly_report.create_and_deliver
-user_data_import.import_from_csv(file)
+user_data_import.from_csv(file)
 etc.
 ```
 
-It is possible to use it raw, but please dont fear to add a wrapper
+It is possible to use it raw, but please don't fear to add a wrapper
 class like `CsvUserImport` for this:
 ``` ruby
 csv_data_processor = Processor::Data::CsvProcessor.new file
@@ -214,6 +233,14 @@ More documentation could be found by running
 rspec
 ```
 
+Find more examples under [example directory](https://github.com/AlexParamonov/processor/tree/master/example)
+
+Contacts
+-------------
+Have questions or recommendations? Contact me via `alexander.n.paramonov@gmail.com`
+Found a bug or have enhancement request? You are welcome at [Github bugtracker](https://github.com/AlexParamonov/processor/issues)
+
+
 Compatibility
 -------------
 tested with Ruby
@@ -222,7 +249,7 @@ tested with Ruby
 * rbx-19mode
 * ruby-head
 
-see [build history](http://travis-ci.org/#!/AlexParamonov/processor/builds)
+See [build history](http://travis-ci.org/#!/AlexParamonov/processor/builds)
 
 Contributing
 -------------

data/example/migrator.rb

@@ -43,6 +43,7 @@ module Processor
 
         stdout_logger_debug = Processor::Observer::Logger.new(logger1, messenger: messenger)
         stdout_logger_info = Processor::Observer::Logger.new(logger2)
+        # file_logger_info = Processor::Observer::Logger.new
         your_custom_observer1 = Observer::NullObserver.new
         your_custom_observer2 = Observer::NullObserver.new
 
@@ -50,6 +51,7 @@ module Processor
           migration,
           stdout_logger_debug,
           stdout_logger_info,
+          # file_logger_info,
           your_custom_observer1,
           your_custom_observer2,
         )

data/lib/processor/data/array_processor.rb

@@ -6,6 +6,10 @@ module Processor
       def records
         raise NotImplementedError
       end
+
+      def total_records
+        @total_records ||= records.count
+      end
     end
   end
 end

data/lib/processor/data/null_processor.rb

@@ -1,6 +1,12 @@
 module Processor
   module Data
     class NullProcessor
+      def start; end
+      def finish; end
+      def finalize; end
+      def error(exception); end
+      def record_error(record, exception); end
+
       def process(record)
         # do nothing
       end
@@ -10,16 +16,7 @@ module Processor
       end
 
       def total_records
-        @total_records ||= records.count
-      end
-
-      def name
-        # underscore a class name
-        self.class.name.to_s.
-          gsub(/::/, '_').
-          gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
-          gsub(/([a-z\d])([A-Z])/,'\1_\2').
-          downcase
+        0
       end
     end
   end

data/lib/processor/environment.rb

@@ -0,0 +1,3 @@
+module Processor
+  RUNNING_ON_CI = !!ENV['CI']
+end

data/lib/processor/event_processor.rb

@@ -0,0 +1,25 @@
+module Processor
+  class EventProcessor
+    def initialize(processor, observers = [])
+      @observers = observers
+      @processor = processor
+    end
+
+    def register(event, *data)
+      observers.each do |observer|
+        observer.update event.to_sym, processor, *data
+      end
+    end
+
+    def method_missing(method, *args)
+      register "before_#{method}", *args
+      result = processor.public_send method, *args
+      register "after_#{method}", result, *args
+
+      result
+    end
+
+    private
+    attr_reader :observers, :processor
+  end
+end

data/lib/processor/logger_messages.rb

@@ -0,0 +1,56 @@
+require 'logger'
+require 'delegate'
+
+module Processor
+  class LoggerMessages < SimpleDelegator
+    def initialize(logger)
+      log_device = fetch_log_device logger
+      messages = case log_device
+                 when File::NULL then NullMessages.new
+                 when String then FileMessages.new log_device
+                 when File then FileMessages.new log_device.path
+                 when IO then IoMessages.new
+                 else NullMessages.new
+                 end
+
+      super messages
+    end
+
+    class NullMessages
+      def initialize(filename = "")
+        @filename = filename
+      end
+      def initialized; "" end
+      def finished; "" end
+      private
+      attr_reader :filename
+    end
+
+    class IoMessages < NullMessages
+      def initialized
+        "Proggress will be streaming to provided IO object"
+      end
+    end
+
+    class FileMessages < NullMessages
+      def initialized
+        <<-MESSAGE
+          Proggress will be saved to the log file. Run
+          tail -f #{filename}
+          to see log in realtime
+          MESSAGE
+      end
+
+      def finished
+        "Log file saved to #{filename}"
+      end
+    end
+
+    private
+    def fetch_log_device logger
+      return unless logger.is_a? ::Logger
+      log_dev = logger.instance_variable_get "@logdev"
+      log_dev.filename or log_dev.dev
+    end
+  end
+end

data/lib/processor/messenger.rb

@@ -1,8 +1,10 @@
-require "logger"
+require 'logger'
+require 'delegate'
 
 module Processor
   class Messenger < SimpleDelegator
-    def initialize(level = :info, file = STDOUT)
+    def initialize(level = :info, file = STDOUT, sender = nil)
+      @sender = sender
       if level == :null
         file = "/dev/null"
         level = :fatal
@@ -18,17 +20,28 @@ module Processor
       super logger
     end
 
+    %w[debug info warn error fatal unknown].each do |level|
+      define_method level do |message, &block|
+        return if message.nil? || message.empty?
+        add(Logger.const_get(level.upcase), nil, message, &block)
+      end
+    end
+
     def message(*args)
       self.info *args
     end
 
     private
+    attr_reader :sender
     def format_message(severity, datetime, progname, message)
       lines = message.split("\n").map do |line|
         "> %s" % line.gsub(/^\s+/, '')
       end.join("\n")
 
-      "\n#{severity} message on #{datetime}:\n#{lines}\n\n"
+      message = "\n#{severity} message"
+      message << " from #{sender}" if sender
+      message << " on #{datetime}"
+      message << ":\n#{lines}\n"
     end
   end
 end

data/lib/processor/observer/logger.rb

@@ -1,65 +1,78 @@
-require_relative 'null_observer'
 require 'logger'
+require 'ostruct'
+require_relative 'null_observer'
+require 'processor/subroutine/name'
+require 'processor/logger_messages'
 
 module Processor
   module Observer
     class Logger < NullObserver
       def initialize(logger = nil, options = {})
         @logger_source = logger
+        @messages = options.fetch :messages, nil
+        @messages = OpenStruct.new @messages if @messages.is_a? Hash
+
         super options
       end
 
-      def processing_started(processor)
-        initialize_logger(processor)
-        logger.info "Processing of #{processor.name} started."
-
-        message = <<-MESSAGE
-          Proggress will be saved to the log file. Run
-          tail -f #{log_file_name}
-          to see log in realtime
-        MESSAGE
-
-        messenger.info message if use_log_file?
+      def after_start(result)
+        logger.info "Processing of #{processor_name} started."
+        messenger.info messages.initialized
       end
 
-      def before_record_processing(record)
+      def before_process(record)
         logger.debug "Record #{id_for record} is going to be processed"
       end
 
-      def after_record_processing(record, result)
+      def after_process(result, record)
         logger.info "Processed #{id_for record}: #{result}"
       end
 
-      def processing_finished(processor)
-        logger.info "Processing of #{processor.name} finished."
-        messenger.info "Log file saved to #{log_file_name}" if use_log_file?
+      def after_finalize(result)
+        logger.info "Processing of #{processor_name} finished."
+        messenger.message messages.finished
       end
 
-      def record_processing_error(record, exception)
+      def after_record_error(result, record, exception)
         logger.error "Error processing #{id_for record}: #{exception}"
       end
 
-      def processing_error(processor, exception)
-        logger.fatal "Processing #{processor.name} FAILED: #{exception.backtrace}"
+      def after_error(result, exception)
+        logger.fatal "Processing #{processor_name} FAILED: #{exception.backtrace}"
       end
 
-      private
-      attr_reader :logger, :log_file_name
-
-      def initialize_logger(processor)
-        @logger =
-          if @logger_source.is_a? Proc
-            @logger_source.call processor.name
+      def logger
+        @logger ||= begin
+        if @logger_source.is_a? Proc
+            @logger_source.call processor_name
           else
-            @logger_source or ::Logger.new(create_log_filename(processor.name)).tap do |logger|
+            @logger_source or ::Logger.new(create_log_filename(processor_name)).tap do |logger|
               logger.level = ::Logger::INFO
             end
           end
-        messenger.debug "Observer initialized with logger #{@logger}"
+        end
+      end
+
+      private
+
+      def messages
+        @messages ||= LoggerMessages.new logger
+      end
+
+      def processor_name
+        @processor_name ||=
+          begin
+            @processor = Subroutine::Name.new processor unless processor.respond_to? :name
+            processor.name
+          end
       end
 
       def create_log_filename(processor_name)
-        FileUtils.mkdir log_directory unless File.directory? log_directory
+        unless File.directory? log_directory
+          FileUtils.mkdir log_directory
+          messenger.warn "Created new directory for logs: #{File.absolute_path log_directory}"
+        end
+
         @log_file_name = "#{log_directory}/#{processor_name}_on_#{current_time_string}.log"
       end
 
@@ -67,10 +80,6 @@ module Processor
         "log"
       end
 
-      def use_log_file?
-        not log_file_name.nil?
-      end
-
       def current_time_string
         Time.now.gmtime.strftime "%Y-%m-%d_%H%M%S_UTC"
       end

data/lib/processor/observer/null_observer.rb

@@ -4,12 +4,17 @@ require 'processor/messenger'
 module Processor
   module Observer
     class NullObserver
+      attr_reader :processor
+
       def initialize(options = {})
-        @messenger = options.fetch :messenger, Processor::Messenger.new(:info)
+        @messenger = options.fetch :messenger, Processor::Messenger.new(:info, STDOUT, self.class.name)
+        @processor = options.fetch :processor, nil
       end
 
-      def method_missing(*); end
-      alias_method :update, :send
+      def update(method_name, processor = nil, *args)
+        @processor ||= processor
+        send method_name, *args if respond_to? method_name
+      end
 
       private
       attr_reader :messenger

data/lib/processor/process_runner/threads.rb

@@ -6,23 +6,18 @@ module Processor
         @threads = []
       end
 
-      def call(processor, events, recursion_preventer)
+      def call(processor)
         join_threads
 
         begin
           processor.records.each do |record|
-            recursion_preventer.call
             if threads_created >= number_of_threads then join_threads end
 
             new_thread(processor, record) do |thread_data_processor, thread_record|
               begin
-                events.register :before_record_processing, thread_record
-
-                result = thread_data_processor.process(thread_record)
-
-                events.register :after_record_processing, thread_record, result
+                thread_data_processor.process(thread_record)
               rescue StandardError => exception
-                events.register :record_processing_error, thread_record, exception
+                thread_data_processor.record_error thread_record, exception
               end
             end
 

data/lib/processor/runner.rb

@@ -1,38 +1,23 @@
-require_relative "events_registrator"
-
 module Processor
   class Runner
-    def initialize(processor, events_registrator)
+    def initialize(processor)
       @processor = processor
-      @events = events_registrator
     end
 
     def run(process_runner)
-      events.register :processing_started, processor
-
-      process_runner.call processor, events, method(:recursion_preventer)
+      processor.start
+      process_runner.call processor
+      processor.finish
 
-      events.register :processing_finished, processor
     rescue Exception => exception
-      events.register :processing_error, processor, exception
+      processor.error exception
       raise exception
-    end
 
-    protected
-    attr_writer :counter
-    def counter
-      @counter ||= 0
+    ensure
+      processor.finalize
     end
 
     private
-    attr_reader :events, :processor
-    def recursion_preventer
-      self.counter += 1
-      raise Exception, "Processing fall into recursion. Check logs." if self.counter > max_records_to_process
-    end
-
-    def max_records_to_process
-      @max_records_to_process ||= (processor.total_records * 1.1).round + 10
-    end
+    attr_reader :processor
   end
 end

data/lib/processor/subroutine/counter.rb

@@ -0,0 +1,25 @@
+require 'delegate'
+
+module Processor
+  module Subroutine
+    class Counter < ::SimpleDelegator
+      def process(*)
+        super
+        record_processed
+      end
+
+      def remaining_records_count
+        [ total_records - processed_records_count, 0 ].max
+      end
+
+      def processed_records_count
+        @processed_records_count ||= 0
+      end
+
+      private
+      def record_processed
+        @processed_records_count = processed_records_count + 1
+      end
+    end
+  end
+end