processor 1.0.0.alpha → 1.0.0

data/.gitignore

@@ -5,6 +5,7 @@
 Gemfile.lock
 /pkg
 /tmp
+/log
 /tags
 
 ## generic files to ignore

data/README.md

@@ -3,8 +3,12 @@ Processor
 [![Build Status](https://travis-ci.org/AlexParamonov/processor.png?branch=master)](http://travis-ci.org/AlexParamonov/processor)
 [![Gemnasium Build Status](https://gemnasium.com/AlexParamonov/processor.png)](http://gemnasium.com/AlexParamonov/processor)  
 
-Processor could execute any `DataProcessor` you specify and log entire process using any number of loggers you need.  
-You may add own observers for monitoring background tasks on even send an email to bussiness with generated report.  
+Processor could execute any `DataProcessor` you specify and log entire
+process using any number of loggers you need.
+
+You may add own observers for monitoring background tasks on even send
+an email to bussiness with generated report.
+
 Processor provide customisation for almost every part of it.  
 
 
@@ -13,6 +17,10 @@ Contents
 1. Installation
 1. Requirements
 1. Usage
+    1. Data processors
+    1. Run modes
+    1. Processor Thread
+    1. Observers
 1. Compatibility
 1. Contributing
 1. Copyright
@@ -20,19 +28,16 @@ Contents
 Installation
 ------------
 Add this line to your application's Gemfile:
-
 ``` ruby
 gem 'processor'
 ```
 
 And then execute:
-
 ``` sh
 bundle
 ```
 
 Or install it yourself as:
-
 ``` sh
 gem install processor
 ```
@@ -44,40 +49,155 @@ Requirements
 
 Usage
 ------------
-1. Implement a `DataProcessor`. See `Processor::Example::Migration`
-and `processor/data` directory. CSV and Solr data processors are usabe
-but not yet finished and tested.
-1. Run your `DataProcessor`:
 
+### Data processors
+Actual processing is done by a Data Processor, provided by end user.  
+This processor should implement in general 2 methods:
+
+1. `process(record)`
+1. `records`
+
+But it is recomended to implement a `name` method also, because it
+is required by several observers. Inherit your Data Processor from
+NullProcessor to get default behavior out of the box.
+
+See `Processor::Example::Migration` for example (`example/migration.rb`).
+
+There are several predefined data processors you can reuse:
+
+
+#### ArrayProcessor
+The simplest one: `process` and `records` methods should be implemented.
+
+
+#### BatchProcessor
+Allows to fetch records by batches of defined size.
+
+It is based on `query` method that suppose to run a query method on
+database.
+
+Recomended to override `fetch_batch` method to get real reason to
+use batch processing. `fetch_batch` could be `query.first(10)` or
+`query.page(next_page)`. See `data/solr_pages_processor.rb` and
+`data/solr_processor.rb` for example.
+
+
+#### Other
+see `data/csv_processor.rb` for running migration from CSV files.
+
+
+### Run modes
+Currently 2 run modes are supported:
+
+
+#### Successive
+It runs `process` one by one for each found record returned by
+`records` method.
+
+Recomended to call it using a `Processor::Thread`:
+``` ruby
+Processor::Thread.new(migration).run_successive
+```
+
+#### Threads
+It runs `process` for each found record returned by `records` method
+not waiting for previous `process` to finish.
+
+Possible to specify number of threads used by passing a number to
+constructor:
+``` ruby
+Processor::ProcessRunner::Threads.new 5
+```
+
+Recomended to call it using a Processor::Thread :
+``` ruby
+Processor::Thread.new(migration).run_in_threads 5
+```
+
+
+### Observers
+Processor support unlimited number of observers, watching processing.
+
+Thay could monitor running migrations and output to logs, console or
+file usefull information. Or thay can show a progress bar to your
+console. Or pack a generated report to archive and send by email to
+bussiness on success or notify developers on failure.
+
+
+This observers should respond to `update` method. But if you inherit
+from `Processor::Observers::NullObserver` you'll get a bunch of
+methods, such as before_ and after_ processing, error handling methods
+to use. See `Processor::Observers::Logger` for example.
+
+Read below section Processor Thread to see how to use observers in runner.
+
+
+### Processor Thread
+`Processor::Thread` is a Facade pattern. It simplifies access to all
+Processor classes and provide __stable__ interface.
+
+Creating a new Thread:
+``` ruby
+Processor::Thread.new data_processor
+```
+
+You may provide optional observers:
+``` ruby
+Processor::Thread.new data_processor, observer1, observer2, ...
+```
+
+Instance have a `run_as` method that accepts a block:
+``` ruby
+thread = Processor::Thread.new @migration
+thread.run_as do |processor, *|
+  processor.records.each do |record|
+    processor.process record
+  end
+end
+```
+
+Block could accept next arguments: `processor`, `events`,
+`recursion_preventer` method. Last one could be called to prevent
+recurtion:
+``` ruby
+recursion_preventer.call
+```
+
+Instance have a `run_successive` method: 
 ``` ruby
 data_processor = UserLocationMigration.new
 thread = Processor::Thread.new data_processor
 thread.run_successive
 ```
-See `spec/processor/thread_spec.rb` and `spec/example_spec.rb` and
-`example` directory for other usage examples.  
 
+And `run_in_threads` method:
+``` ruby
+data_processor = UserCsvImport.new csv_file
+thread = Processor::Thread.new data_processor
+thread.run_in_threads 10
+```
 
-It is recomended to wrap Processor::Thread in your classes like:
+See `spec/processor/thread_spec.rb` and `spec/example_spec.rb` and
+`example` directory for other usage examples.
 
-```
+It is recomended to wrap Processor::Thread by classes named like:
+``` ruby
 WeeklyReport
 TaxonomyMigration
 UserDataImport
 ```
-to hide configuration of observers or use your own API to run
-migrations:
 
-```
+The point is to hide configuration of observers and use (if you wish)
+your own API to run reports or migrations:
+``` ruby
 weekly_report.create_and_deliver
 user_data_import.import_from_csv(file)
 etc.
 ```
 
-Sure, it is possible to use it raw, but please dont fear to add a
-wrapper class for it:
-
-```
+It is possible to use it raw, but please dont fear to add a wrapper
+class like `CsvUserImport` for this:
+``` ruby
 csv_data_processor = Processor::Data::CsvProcessor.new file
 stdout_notifier = Processor::Observer::Logger.new(Logger.new(STDOUT))
 logger_observer = Processor::Observer::Logger.new
@@ -89,10 +209,10 @@ Processor::Thread.new(
 ).run_in_threads 5
 ```
 
-### Observers
-Observers should respond to `update` method but if you inherit from
-`Processor::Observers::NullObserver` you'll get a bunch of methods to
-use. See `Processor::Observers::Logger` for example.
+More documentation could be found by running
+``` sh
+rspec
+```
 
 Compatibility
 -------------

data/example/migrator.rb

@@ -13,7 +13,7 @@ module Processor
         # logger could be an instance of Ruby Logger
         logger1 = ::Logger.new(STDOUT).tap do |logger|
           logger.formatter =  -> _, _, _, msg do
-            "debug\t< #{msg}\n"
+            "log < debug logger: #{msg}\n"
           end
           logger.level = ::Logger::DEBUG
         end
@@ -23,7 +23,7 @@ module Processor
         logger2 = -> name do
           ::Logger.new(STDOUT).tap do |logger|
             logger.formatter =  -> _, _, _, msg do
-              "info\t< #{msg}\n"
+              "log <  info logger: #{msg}\n"
             end
             logger.level = ::Logger::INFO
           end
@@ -36,11 +36,9 @@ module Processor
         # in this case logger will be initialized as Ruby Logger and write to log/name_of_processor_time_stamp.log
 
         # You may customize a messenger:
-        messenger = ::Logger.new(STDOUT).tap do |logger|
-          logger.formatter =  -> _, _, _, msg do
-            "message\t> #{msg}\n"
-          end
-          logger.level = ::Logger::DEBUG
+        messenger = Processor::Messenger.new :debug
+        messenger.formatter = -> _, _, _, msg do
+          "message\t> #{msg}\n"
         end
 
         stdout_logger_debug = Processor::Observer::Logger.new(logger1, messenger: messenger)

data/lib/processor/data/batch_processor.rb

@@ -10,7 +10,9 @@ module Processor
       def records
         Enumerator.new do |result|
           loop do
-            fetch_batch.each do |record|
+            batch = fetch_batch
+            break if batch.count < 1
+            batch.each do |record|
               result << record
             end
           end

data/lib/processor/data/csv_processor.rb

@@ -1,8 +1,8 @@
-require_relative 'batch_processor'
+require_relative 'null_processor'
 
 module Processor
   module Data
-    class CsvProcessor < BatchProcessor
+    class CsvProcessor < NullProcessor
       def initialize(file, csv_options = {})
         @file = file
         @separator = separator

data/lib/processor/messenger.rb

@@ -0,0 +1,34 @@
+require "logger"
+
+module Processor
+  class Messenger < SimpleDelegator
+    def initialize(level = :info, file = STDOUT)
+      if level == :null
+        file = "/dev/null"
+        level = :fatal
+      end
+
+      log_level = Logger.const_get(level.upcase);
+
+      logger = ::Logger.new(file).tap do |logger|
+        logger.formatter = method(:format_message)
+        logger.level = log_level
+      end
+
+      super logger
+    end
+
+    def message(*args)
+      self.info *args
+    end
+
+    private
+    def format_message(severity, datetime, progname, message)
+      lines = message.split("\n").map do |line|
+        "> %s" % line.gsub(/^\s+/, '')
+      end.join("\n")
+
+      "\n#{severity} message on #{datetime}:\n#{lines}\n\n"
+    end
+  end
+end

data/lib/processor/observer/logger.rb

@@ -13,12 +13,13 @@ module Processor
         initialize_logger(processor)
         logger.info "Processing of #{processor.name} started."
 
-        message = <<-MESSAGE.gsub(/^\s+/, '')
+        message = <<-MESSAGE
           Proggress will be saved to the log file. Run
           tail -f #{log_file_name}
           to see log in realtime
-          MESSAGE
-          messenger.info message if use_log_file?
+        MESSAGE
+
+        messenger.info message if use_log_file?
       end
 
       def before_record_processing(record)
@@ -58,7 +59,12 @@ module Processor
       end
 
       def create_log_filename(processor_name)
-        @log_file_name = "log/#{processor_name}_on_#{current_time_string}.log"
+        FileUtils.mkdir log_directory unless File.directory? log_directory
+        @log_file_name = "#{log_directory}/#{processor_name}_on_#{current_time_string}.log"
+      end
+
+      def log_directory
+        "log"
       end
 
       def use_log_file?

data/lib/processor/observer/null_observer.rb

@@ -1,17 +1,11 @@
 require 'logger'
+require 'processor/messenger'
 
 module Processor
   module Observer
     class NullObserver
       def initialize(options = {})
-        @messenger = options.fetch :messenger do
-          ::Logger.new(STDOUT).tap do |logger|
-            logger.formatter =  -> _, _, _, msg do
-              "> #{msg}\n"
-            end
-            logger.level = ::Logger::INFO
-          end
-        end
+        @messenger = options.fetch :messenger, Processor::Messenger.new(:info)
       end
 
       def method_missing(*); end

data/lib/processor/version.rb

@@ -1,3 +1,3 @@
 module Processor
-  VERSION = "1.0.0.alpha"
+  VERSION = "1.0.0"
 end

data/lib/processor.rb

@@ -3,6 +3,8 @@ require "processor/version"
 require "processor/data/array_processor"
 require "processor/data/batch_processor"
 require "processor/data/null_processor"
+require "processor/data/solr_processor"
+require "processor/data/solr_pages_processor"
 
 require "processor/observer/logger"
 require "processor/observer/null_observer"

data/spec/processor/data/batch_processor_spec.rb

@@ -16,8 +16,7 @@ describe Processor::Data::BatchProcessor do
 
     query = Enumerator.new do |y|
       a = 1
-      loop do
-        break if a > 10
+      until a > 10
         watcher.created
         y << a
         a += 1
@@ -35,4 +34,9 @@ describe Processor::Data::BatchProcessor do
     subject.stub(query: query)
     subject.total_records.should eq 5
   end
+
+  it "should stop iteration if fetch_records returned empty result set" do
+    subject.stub(fetch_batch: [])
+    subject.records.to_a.should eq []
+  end
 end

data/spec/processor/messenger_spec.rb

@@ -0,0 +1,53 @@
+require 'spec_helper_lite'
+require 'processor/messenger'
+
+describe Processor::Messenger do
+  let(:io) { STDOUT }
+
+  it "should delegate to logger" do
+    io.should_receive(:write)
+    messenger = Processor::Messenger.new :debug, io
+    messenger.debug "debug message"
+  end
+
+  it "should accept symbol as messaging level" do
+    messenger = Processor::Messenger.new :info
+    messenger.level.should eq Logger::INFO
+  end
+
+  it "should not output anything if :null level provided" do
+    io.should_not_receive(:write)
+    messenger = Processor::Messenger.new :null, io
+    messenger.fatal "fatal error"
+  end
+
+  describe "messages" do
+    let(:messenger) { Processor::Messenger.new :debug, io }
+    %w[debug info error fatal].each do |message_level|
+      it "should send #{message_level} message" do
+        message = "#{message_level} message"
+        io.should_receive(:write).with /#{message}/
+        messenger.send message_level, message
+      end
+    end
+  end
+
+  it "should send info message by .message method" do
+    messenger = Processor::Messenger.new :info, io
+    messenger.should_receive(:info).with("test message")
+    messenger.message "test message"
+  end
+
+  describe "formatter" do
+    let(:messenger) { Processor::Messenger.new :info, io }
+
+    it "should accept formatter proc" do
+      io.should_receive(:write).with("test message")
+      messenger.formatter = -> _,_,_,message do
+        message
+      end
+
+      messenger.message "test message"
+    end
+  end
+end

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: processor
 version: !ruby/object:Gem::Version
-  version: 1.0.0.alpha
-  prerelease: 6
+  version: 1.0.0
+  prerelease: 
 platform: ruby
 authors:
 - Alexander Paramonov
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-15 00:00:00.000000000 Z
+date: 2013-05-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -88,6 +88,7 @@ files:
 - lib/processor/data/solr_pages_processor.rb
 - lib/processor/data/solr_processor.rb
 - lib/processor/events_registrator.rb
+- lib/processor/messenger.rb
 - lib/processor/observer/logger.rb
 - lib/processor/observer/null_observer.rb
 - lib/processor/process_runner/successive.rb
@@ -101,6 +102,7 @@ files:
 - spec/processor/data/batch_processor_spec.rb
 - spec/processor/data/null_processor_spec.rb
 - spec/processor/events_registrator_spec.rb
+- spec/processor/messenger_spec.rb
 - spec/processor/observer/logger_spec.rb
 - spec/processor/process_runner/specs.rb
 - spec/processor/process_runner/successive_spec.rb
@@ -123,13 +125,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1752503773457190967
+      hash: 204713070122044710
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>'
+  - - ! '>='
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
+      segments:
+      - 0
+      hash: 204713070122044710
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.25
@@ -142,6 +147,7 @@ test_files:
 - spec/processor/data/batch_processor_spec.rb
 - spec/processor/data/null_processor_spec.rb
 - spec/processor/events_registrator_spec.rb
+- spec/processor/messenger_spec.rb
 - spec/processor/observer/logger_spec.rb
 - spec/processor/process_runner/specs.rb
 - spec/processor/process_runner/successive_spec.rb