procrastinator 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 46f0ec23b44a55033b241f8f1eea274a99dcc481
-  data.tar.gz: 84476d93786e0e63f53e11cbf913c8df9f641134
+  metadata.gz: 585203be626fab88b3799c54a970386cf8500995
+  data.tar.gz: 25b86adaa256bd8e64559237689849a3e1051e19
 SHA512:
-  metadata.gz: 0f54aba2184d65a6dba03a756688565bfc6d45520ea521a0ce488353ea81c02ff1946d072145b176bc5ddc643f01670256bfb9b73819f70a7961d10f63ce1712
-  data.tar.gz: 8181179a9437191dc43b34e61e67c7f846a83633962d7f3f41b1a69696b22769e85b5a15a1f8672c03af7c1f3fb8d0f3290fea6cef908e3bcf8b7da2128589b8
+  metadata.gz: 758e964a9edf01df9ae7e5d845614384c81cf76f053ac7939a5822bb2ca80bb088d3023df58025ece12ddd2f9a040711e324c6e7b084be8f07c48f4800732e0f
+  data.tar.gz: 0ade5c71cab428f01c0e8080e2fb5d8e867ab809b8c96c7cf81df8bd3fa36e390d066c7918a42bc4d7de6278b942264d3af1d9c2a7b1ab896fa8fff3cf27cd6d

data/README.md

@@ -13,13 +13,9 @@ Add this line to your application's Gemfile:
 gem 'procrastinator'
 ```
 
-And then execute:
+And then run:
 
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install procrastinator
+    bundle install
 
 ## Usage
 Setup a procrastination environment:
@@ -42,7 +38,7 @@ Read on for more details on each step.
 
 ### Setup Phase
 The setup phase first defines which queues are available and the persistence strategy to use for reading 
-and writing tasks. It then starts up a sub process for working on each queue within that environment.
+and writing tasks. It then spins off a sub process for working on each queue within that environment. 
 
 
 #### Declaring a Persistence Strategy
@@ -98,6 +94,14 @@ Each queue is worked in a separate process.
 
 The sub-processes checks that the parent process is still alive every 5 seconds. If there is no process with the parent's PID, the sub-process will self-exit. 
 
+Sub-processes can be given a name prefix with the process_prefix method: 
+
+```ruby
+procrastinator = Procrastinator.setup(task_persister) do |env|
+   env.process_prefix('myapp')
+end
+```
+
 ###Scheduling Tasks For Later
 Procrastinator will let you be lazy: 
 
@@ -156,9 +160,9 @@ your task **must provide** a `#run` method:
 
 You may also optionally provide these hook methods, which are run during different points in the process:
 
- * `#success` - run after the task has completed successfully 
- * `#fail` - run after the task has failed due to `#run` producing a `StandardError` or subclass.
- * `#final_fail` - run after the task has failed for the last time because either:
+ * `#success(logger)` - run after the task has completed successfully 
+ * `#fail(logger, error)` - run after the task has failed due to `#run` producing a `StandardError` or subclass.
+ * `#final_fail(logger, error)` - run after the task has failed for the last time because either:
     1. the number of attempts is >= the `max_attempts` defined for the queue; or
     2. the time reported by `Time.now` is past the task's `expire_at` time.
 
@@ -166,8 +170,10 @@ If a task reaches `#final_fail` it will be marked to never be run again.
 
 ***Task Failure & Rescheduling***
 
-Tasks that fail have their `run_at` rescheduled on an increasing delay according to this formula: 
- * 30 + attempts<sup>4</sup> **(in seconds)**
+Tasks that fail have their `run_at` rescheduled on an increasing delay **(in seconds)** according to this formula: 
+ * 30 + n<sup>4</sup>
+  
+n = the number of attempts 
 
 Both failing and final_failing will cause the error timestamp and reason to be stored in `:last_fail_at` and `:last_error`.
 
@@ -200,6 +206,35 @@ env.act
 env.act(:cleanup, :email)
 ```
 
+### Logging
+Logging is crucial to knowing what went wrong in an application after the fact, and because Procrastinator runs workers
+in separate processes, providing a logger instance isn't really an option. 
+ 
+Instead, provide a directory that your Procrastinator instance should write log entries into:
+
+```ruby
+procrastinator = Procrastinator.setup do |env|
+   env.log_dir('log/')
+end
+```
+ 
+Each worker creates its own log named after the queue it is working on (eg. `log/email-queue-worker.log`). The default 
+directory is `./log/`, relative to wherever the application is running. Logging will not occur at all if `log_dir` is 
+assigned a falsey value. 
+
+The logging level can be set using `log_level` and a value from the Ruby standard library 
+[Logger class](https://ruby-doc.org/stdlib-2.2.3/libdoc/logger/rdoc/Logger.html) (eg. `Logger::WARN`, `Logger::DEBUG`, etc.). 
+
+It logs process start at level `INFO`, process termination due to parent disppearance at level `ERROR` and task hooks 
+`#success`, `#fail`, and `#final_fail` are at a level `DEBUG`. 
+
+```ruby
+procrastinator = Procrastinator.setup do |env|
+   env.log_dir('log/')
+   env.log_level(Logger::INFO) # setting the default explicity
+end
+```
+
 ## Contributing
 Bug reports and pull requests are welcome on GitHub at 
 [https://github.com/TenjinInc/procrastinator](https://github.com/TenjinInc/procrastinator).

data/lib/procrastinator.rb

@@ -2,6 +2,7 @@ require 'procrastinator/version'
 require 'procrastinator/queue_worker'
 require 'procrastinator/task_worker'
 require 'procrastinator/environment'
+require 'logger'
 
 
 module Procrastinator

data/lib/procrastinator/environment.rb

@@ -2,6 +2,8 @@ module Procrastinator
    class Environment
       attr_reader :persister, :queue_definitions, :queue_workers, :processes, :test_mode
 
+      DEFAULT_LOG_DIRECTORY = 'log/'
+
       def initialize(persister:, test_mode: false)
          raise ArgumentError.new('persister cannot be nil') if persister.nil?
 
@@ -14,6 +16,8 @@ module Procrastinator
          @queue_definitions = {}
          @queue_workers     = []
          @processes         = []
+         @log_dir           = DEFAULT_LOG_DIRECTORY
+         @log_level         = Logger::INFO
       end
 
       def define_queue(name, properties={})
@@ -25,16 +29,23 @@ module Procrastinator
       def spawn_workers
          if @test_mode
             @queue_definitions.each do |name, props|
-               @queue_workers << QueueWorker.new(props.merge(name: name, persister: @persister))
+               @queue_workers << QueueWorker.new(props.merge(name:      name,
+                                                             persister: @persister,
+                                                             log_dir:   @log_dir))
             end
          else
             @queue_definitions.each do |name, props|
                pid = fork do
-                  Process.setproctitle("#{name}-queue-worker")
+                  worker = QueueWorker.new(props.merge(name:      name,
+                                                       persister: @persister,
+                                                       log_dir:   @log_dir,
+                                                       log_level: @log_level))
+
+                  worker.start_log
 
-                  worker = QueueWorker.new(props.merge(name: name, persister: @persister))
+                  Process.setproctitle("#{@process_prefix ? "#{@process_prefix}-" : ''}#{worker.long_name}") # tODO: add an app name prefix
 
-                  monitor_parent
+                  monitor_parent(worker)
 
                   worker.work
                end
@@ -63,7 +74,17 @@ module Procrastinator
 
       def delay(queue: nil, run_at: Time.now.to_i, expire_at: nil, task:)
          raise ArgumentError.new('task may not be nil') if task.nil?
-         raise MalformedTaskError.new('given task does not support #run method') unless task.respond_to? :run
+         raise MalformedTaskError.new('the provided task does not support #run method') unless task.respond_to? :run
+
+         # We're checking these on init because it's one of those extremely rare cases where you'd want to know
+         # incredibly early, because of the sub-processing. It's a bit belt-and suspenders, but UX is important for
+         # devs, too.
+         [:success, :fail, :final_fail].each do |method_name|
+            if task.respond_to?(method_name) && task.method(method_name).arity <= 0
+               raise MalformedTaskError.new("the provided task must accept a parameter to its ##{method_name} method")
+            end
+         end
+
          if queue.nil? && @queue_definitions.size > 1
             raise ArgumentError.new("queue must be specified when more than one is registered. Defined queues are: #{queue_definitions.keys.map { |k| ':' + k.to_s }.join(', ')}")
          else
@@ -82,11 +103,30 @@ module Procrastinator
          @test_mode = true
       end
 
+      def log_dir(path)
+         @log_dir = path
+      end
+
+      def log_level(lvl)
+         @log_level = lvl
+      end
+
+      def process_prefix(prefix)
+         @process_prefix = prefix
+      end
+
       private
-      def monitor_parent
-         heartbeat_thread = Thread.new(Process.ppid) do |ppid|
+      def monitor_parent(worker)
+         parent_pid = Process.ppid
+
+         heartbeat_thread = Thread.new(parent_pid) do |pid|
             loop do
-               Process.kill 0, ppid
+               begin
+                  Process.kill(0, pid) # kill(0) will check if the process exists
+               rescue Errno::ESRCH
+                  worker.log_parent_exit
+                  exit
+               end
 
                sleep(5)
             end

data/lib/procrastinator/queue_worker.rb

@@ -10,6 +10,8 @@ module Procrastinator
       # Timeout is in seconds
       def initialize(name:,
                      persister:,
+                     log_dir: nil,
+                     log_level: Logger::INFO,
                      max_attempts: DEFAULT_MAX_ATTEMPTS,
                      timeout: DEFAULT_TIMEOUT,
                      update_period: DEFAULT_UPDATE_PERIOD,
@@ -21,13 +23,14 @@ module Procrastinator
          raise(MalformedTaskPersisterError.new('The supplied IO object must respond to #update_task')) unless persister.respond_to? :update_task
          raise(MalformedTaskPersisterError.new('The supplied IO object must respond to #delete_task')) unless persister.respond_to? :delete_task
 
-
          @name          = name.to_s.gsub(/\s/, '_').to_sym
          @timeout       = timeout
          @max_attempts  = max_attempts
          @update_period = update_period
          @max_tasks     = max_tasks
          @persister     = persister
+         @log_dir       = log_dir
+         @log_level     = log_level
       end
 
       def work
@@ -45,6 +48,8 @@ module Procrastinator
 
          tasks.first(@max_tasks).each do |task_data|
             if Time.now.to_i >= task_data[:run_at].to_i
+               task_data.merge!(logger: @logger) if @logger
+
                tw = TaskWorker.new(task_data)
 
                tw.work
@@ -57,6 +62,40 @@ module Procrastinator
             end
          end
       end
+
+      def long_name
+         "#{@name}-queue-worker"
+      end
+
+      # Starts a log file and stores the logger within this queue worker.
+      #
+      # Separate from init because logging is context-dependent
+      def start_log
+         if @log_dir
+            log_path = Pathname.new("#{@log_dir}/#{long_name}.log")
+
+            log_path.dirname.mkpath
+            File.open(log_path.to_path, 'a+') do |f|
+               f.write ''
+            end
+
+            @logger = Logger.new(log_path.to_path)
+
+            @logger.level = @log_level
+
+            @logger.info(['',
+                          '===================================',
+                          "Started worker process, #{long_name}, to work off queue #{@name}.",
+                          "Worker pid=#{Process.pid}; parent pid=#{Process.ppid}.",
+                          '==================================='].join("\n"))
+         end
+      end
+
+      def log_parent_exit
+         raise RuntimeError.new('Cannot log when logger not defined. Call #start_log first.') unless @logger
+
+         @logger.error("Terminated worker process (#{Process.pid}) due to main process (#{Process.ppid}) disappearing.")
+      end
    end
 
    class MalformedTaskPersisterError < StandardError

data/lib/procrastinator/task_worker.rb

@@ -13,7 +13,8 @@ module Procrastinator
                      max_attempts: nil,
                      last_fail_at: nil,
                      last_error: nil,
-                     task:)
+                     task:,
+                     logger: Logger.new(StringIO.new))
          @id             = id
          @run_at         = run_at.to_i
          @initial_run_at = initial_run_at.to_i
@@ -24,6 +25,7 @@ module Procrastinator
          @timeout        = timeout
          @last_fail_at   = last_fail_at
          @last_error     = last_error
+         @logger         = logger
 
          raise(MalformedTaskError.new('given task does not support #run method')) unless @task.respond_to? :run
          raise(ArgumentError.new('timeout cannot be negative')) if timeout && timeout < 0
@@ -39,21 +41,27 @@ module Procrastinator
                @task.run
             end
 
-            try_hook(:success)
+            try_hook(:success, @logger)
+
+            @logger.debug("Task completed: #{YAML.dump(@task)}")
+
             @last_error   = nil
             @last_fail_at = nil
          rescue StandardError => e
             @last_fail_at = Time.now.to_i
 
             if too_many_fails? || expired?
-               try_hook(:final_fail, e)
+               try_hook(:final_fail, @logger, e)
 
                @run_at     = nil
                @last_error = "#{expired? ? 'Task expired' : 'Task failed too many times'}: #{e.backtrace.join("\n")}"
+
+               @logger.debug("Task failed permanently: #{YAML.dump(@task)}")
             else
-               try_hook(:fail, e)
+               try_hook(:fail, @logger, e)
 
                @last_error = %Q[Task failed: #{e.message}\n#{e.backtrace.join("\n")}]
+               @logger.debug("Task failed: #{YAML.dump(@task)}")
 
                reschedule
             end

data/lib/procrastinator/version.rb

@@ -1,3 +1,3 @@
 module Procrastinator
-   VERSION = '0.3.0'
+   VERSION = '0.4.0'
 end

data/procrastinator.gemspec

@@ -26,4 +26,5 @@ Gem::Specification.new do |spec|
    spec.add_development_dependency 'rspec', '~> 3.0'
    spec.add_development_dependency 'timecop', '~> 0.8'
    spec.add_development_dependency 'simplecov', '~> 0.11'
+   spec.add_development_dependency 'fakefs', '~> 0.10'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: procrastinator
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Robin Miller
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-06-12 00:00:00.000000000 Z
+date: 2016-11-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -80,6 +80,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.11'
+- !ruby/object:Gem::Dependency
+  name: fakefs
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
 description: A strightforward job queue that is not dependent on Rails or any particular
   database or persistence mechanism.
 email: