procrastinator 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b4fce4f790799f4157d84b40960ce894e71562b9
-  data.tar.gz: 0384cc924d66ebe90af1585dc05ab2133d750355
+  metadata.gz: a9d61dca76d919b7ece43fd8b7b59c9256cb8334
+  data.tar.gz: 70251bb840acfa728d0fbb1cf5b457364dd655ac
 SHA512:
-  metadata.gz: 7ae26a1c7f3bf0962b64e6598103c91f6ad4b31eeb8b8e553b6859ff73d7231385555853573162c4d5351414cc39c7ba5fc80e4037dfa12c915f4f42db2934db
-  data.tar.gz: 8555135727bbbd608c58c5ddb1d0f0c877ce086301135426a958e134aec8717d2244e40529966959b46dff19af7b21ae6a2dfef94747b91b4d4fad474d7af1ed
+  metadata.gz: ffbb42494ee062144f658b11a9c936072432c411e0aae2f43ca7a7b561b8272f445d8a8017b4b708c06463fdd2bc276a8f869ee4506576f5d521fa2a17280d4f
+  data.tar.gz: 3afb4512571a8e49530bb05726cd4628636f10925791967fe789b11a536a2c48df87d9d0311af7e2a8066ccb41698c0d8b50f15cb9080525b9a7081de130bb00

data/.ruby-version

@@ -0,0 +1 @@
+ruby-2.1.2

data/README.md

@@ -1,8 +1,7 @@
 # Procrastinator
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/procrastinator`. To experiment with that code, run `bin/console` for an interactive prompt.
-
-TODO: Delete this and the text above, and describe your gem
+Procrastinator is a framework-independent job scheduling gem. It creates a subprocess for each queue that performs 
+tasks at the appropriate times. 
 
 ## Installation
 
@@ -21,21 +20,166 @@ Or install it yourself as:
     $ gem install procrastinator
 
 ## Usage
+Setup a procrastination environment:
 
-TODO: Write usage instructions here
+```ruby
+procrastinator = Procrastinator.setup(TaskPersister.new) do |env|   # MyTaskPersister is a class defined by you 
+  env.define_queue(:email)
+  env.define_queue(:cleanup, max_attempts: 3)
+end
+```
 
-## Development
+And then delay some tasks:
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+```ruby
+procrastinator.delay(queue: :email, task: EmailGreeting.new('bob@example.com'))
+procrastinator.delay(queue: :cleanup, run_at: Time.now + 3600, task: ClearTempData.new)
+```
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+Read on for more details on each step. 
 
-## Contributing
+###`Procrastinator.setup`
+The setup phase first defines which queues are available and the persistence strategy to use for reading 
+and writing tasks. It then starts up a sub process for working on each queue within that environment.
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/procrastinator. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
+#### Persister Strategy
+The persister instance is the last step between Procrastinator and your data storage (eg. database). As core Procrastinator is framework-agnostic, it needs you to provide an object that knows how to read and write task data. 
 
-## License
+Your [strategy](https://en.wikipedia.org/wiki/Strategy_pattern) class is required to provide the following methods: 
+
+* `#read_tasks(queue_name)` - Returns a list of hashes from your data storage. Each hash must contains the properites of one task, as seen in the *Attributes Hash* 
+* `#create_task(data)` - Creates a task in your datastore. Receives a hash with keys `:queue`, `:run_at`, `:initial_run_at`, `:expire_at`, and `:task` as described in *Attributes Hash* 
+* `#update_task(attributes)` - Receives the Attributes Hash as the data to be saved
+* `#delete_task(id)` - Deletes the task with the given id. 
+
+If the strategy does not have all of these methods, Procrastinator will explode with a `MalformedPersisterError`  and you will be sad. 
+
+***Attributes Hash***
+
+|  Hash Key         | Type   | Description                                                                           |
+|-------------------|--------| --------------------------------------------------------------------------------------|
+| `:id`             | int    | Unique identifier for this exact task                                                 |
+| `:queue`          | symbol | Name of the queue the task is inside                                                      | 
+| `:run_at`         | int    | Unix timestamp of when to next attempt running the task                               |
+| `:initial_run_at` | int    | Unix timestamp of the original run_at; before the first attempt, this is equal to run_at |
+| `:expire_at`      | int    | Unix timestamp of when to permanently fail the task because it is too late to be useful |
+| `:attempts`       | int    | Number of times the task has tried to run; this should only be > 0 if the task fails  |
+| `:last_fail_at`   | int    | Unix timestamp of when the most recent failure happened                               |
+| `:last_error`     | string | Error message + bracktrace of the most recent failure. May be very long.              |
+| `:task`           | string | YAML-dumped ruby object definition of the task.                                       |
+
+Notice that the times are all given as unix epoch timestamps. This is to avoid any confusion with timezones, and it is recommended that you store times in this manner for the same reason. 
+
+####Defining Queues
+`Procrastinator.setup` requires a block be provided, and that in the block call `#define_queue` be called on the provided environment. Define queue takes a queue name symbol and these properies as a hash
 
-The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+ * :timeout - Time, in seconds, after which it should fail tasks in this queue for taking too long to execute.
+ * :max_attempts - Maximum number of attempts for tasks in this queue. If attempts is >= max_attempts, the task will be final_failed and marked to never run again
+ * :update_period - Delay, in seconds, between refreshing the task list from the persister
+ * :max_tasks - The maximum number of tasks to run concurrently with multi-threading. 
+
+**Examples**
+```ruby 
+Procrastinator.setup(some_persister) do |env|
+   env.define_queue(:email)
+   
+   # with all defaults set explicitly
+   env.define_queue(:email, timeout: 3600, max_attempts: 20, update_period: 10, max_tasks: 10)
+end
+```
+  
+#### Queue Sub-Processes
+Each queue is worked in a separate process, and each process is multi-threaded to handle more than one task at a time. This should help prevent a single task from clogging up the whole queue, or a single queue clogging up the entire system. Tasks are chosen by their `:run_at` property. 
+
+The sub-processes check ervey 5 seconds that the parent process is still alive. If there is no process with the parent's process ID, the sub-process will self-exit. 
+
+###`Environment#delay`
+When there are multiple queues defined, you are required to provide a queue name:
+
+```ruby
+procrastinator = Procrastinator.setup(task_persister) do |env|
+   env.define_queue(:email)
+   env.|env|define_queue(:cleanup)
+end
+
+procrastinator.delay(:email, task: EmailReminder.new('bob@example.com'))
+```
+
+Otherwise, Procrastinator will let you be lazy: 
+
+```ruby
+procrastinator = Procrastinator.setup(task_persister) do |env|
+   env.define_queue(:email)
+end
+
+procrastinator.delay(task: EmailReminder.new('bob@example.com'))
+```
 
+You can set when the particular task is to be run and/or when it should expire. Be aware that neither is guaranteed to be real-time; a task will run as soon as it possible after `run_at` is passed, and will expire once it attempts to run after `expire_at` has passed. 
+
+```ruby
+procrastinator = Procrastinator.setup(task_persister) do |env|
+   env.define_queue(:email)
+end
+
+# run on or after 1 January 3000
+procrastinator.delay(run_at: Time.new(3000, 1, 1), task: EmailGreeting.new('philip_j_fry@example.com'))
+
+# explicitly setting default run_at
+procrastinator.delay(run_at: Time.now, task: EmailReminder.new('bob@example.com'))
+```
+
+You can also set an `expire_at` deadline on when to run a task. If the task has not been run before `expire_at` is passed, then it will be final-failed the next time it is attempted. Setting `expire_at` to `nil` will mean it will never expire (but may still fail permanently if, say, `max_attempts` is reached).
+
+```ruby
+procrastinator = Procrastinator.setup(task_persister) do |env|
+   env.define_queue(:email)
+end
+
+procrastinator.delay(expire_at: , task: EmailGreeting.new('bob@example.com'))
+
+# explicitly setting default
+procrastinator.delay(expire_at: nil, task: EmailGreeting.new('bob@example.com'))
+```
+
+#### Task
+Like the persister provided to `.setup`, your task is a strategy object that fills in the details of what to do. For this, 
+your task **must provide** a `#run` method:
+
+ * `#run` - Performs the core work of the task.
+
+You may also optionally provide these hook methods, which are run during different points in the process:
+
+ * `#success` - run after the task has completed successfully 
+ * `#fail` - run after the task has failed due to `#run` producing a `StandardError` or subclass.
+ * `#final_fail` - run after the task has failed for the last time because either:
+    1. the number of attempts is >= the `max_attempts` defined for the queue; or
+    2. the time reported by `Time.now` is past the task's `expire_at` time.
+
+If a task reaches `#final_fail` it will be marked to never be run again.
+
+***Task Failure & Rescheduling***
+
+Tasks that fail have their `run_at` rescheduled on an increasing delay according to this formula: 
+ * 30 + attempts<sup>4</sup> **(in seconds)**
+
+Both failing and final_failing will cause the error timestamp and reason to be stored in `:last_fail_at` and `:last_error`.
+
+## Contributing
+Bug reports and pull requests are welcome on GitHub at 
+[https://github.com/TenjinInc/procrastinator](https://github.com/TenjinInc/procrastinator).
+ 
+This project is intended to be a friendly space for collaboration, and contributors are expected to adhere to the 
+[Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+### Core Developers
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can 
+also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the 
+version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, 
+push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## License
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/lib/procrastinator/environment.rb

@@ -0,0 +1,73 @@
+module Procrastinator
+   class Environment
+      attr_reader :persister, :queues, :processes
+
+      def initialize(persister)
+         raise ArgumentError.new('persister cannot be nil') if persister.nil?
+
+         [:read_tasks, :create_task, :update_task, :delete_task].each do |method|
+            raise MalformedPersisterError.new("persister must repond to ##{method}") unless persister.respond_to? method
+         end
+
+         @persister = persister
+         @queues    = {}
+         @processes = []
+      end
+
+      def define_queue(name, properties={})
+         raise ArgumentError.new('queue name cannot be nil') if name.nil?
+
+         @queues[name] = properties
+      end
+
+      def spawn_workers
+         @queues.each do |name, props|
+            pid = fork do
+               Process.setproctitle("#{name}-queue-worker")
+
+               worker = QueueWorker.new(props.merge(name: name, persister: @persister))
+
+               monitor_parent
+
+               worker.work
+            end
+
+            Process.detach(pid) unless pid.nil?
+            @processes << pid
+         end
+      end
+
+      def delay(queue: nil, run_at: Time.now.to_i, expire_at: nil, task:)
+         raise ArgumentError.new('task may not be nil') if task.nil?
+         raise MalformedTaskError.new('given task does not support #run method') unless task.respond_to? :run
+         if queue.nil? && @queues.size > 1
+            raise ArgumentError.new('queue must be specified when more than one is registered')
+         else
+            queue ||= @queues.keys.first
+            raise ArgumentError.new(%Q{there is no "#{queue}" queue registered in this environment}) if @queues[queue].nil?
+         end
+
+         @persister.create_task(queue:          queue,
+                                run_at:         run_at.to_i,
+                                initial_run_at: run_at.to_i,
+                                expire_at:      expire_at.nil? ? nil : expire_at.to_i,
+                                task:           YAML.dump(task))
+      end
+
+      private
+      def monitor_parent
+         heartbeat_thread = Thread.new(Process.ppid) do |ppid|
+            loop do
+               Process.kill 0, ppid
+
+               sleep(5)
+            end
+         end
+
+         heartbeat_thread.abort_on_exception = true
+      end
+   end
+
+   class MalformedPersisterError < StandardError
+   end
+end

data/lib/procrastinator/queue_worker.rb

@@ -0,0 +1,60 @@
+module Procrastinator
+   class QueueWorker
+      DEFAULT_TIMEOUT       = 3600 # seconds = one hour
+      DEFAULT_MAX_ATTEMPTS  = 20
+      DEFAULT_UPDATE_PERIOD = 10 # seconds
+      DEFAULT_MAX_TASKS     = 10
+
+      attr_reader :name, :timeout, :max_attempts, :update_period, :max_tasks
+
+      # Timeout is in seconds
+      def initialize(name:,
+                     persister:,
+                     max_attempts: DEFAULT_MAX_ATTEMPTS,
+                     timeout: DEFAULT_TIMEOUT,
+                     update_period: DEFAULT_UPDATE_PERIOD,
+                     max_tasks: DEFAULT_MAX_TASKS)
+         raise ArgumentError.new('Queue name may not be nil') unless name
+         raise ArgumentError.new('Persister may not be nil') unless persister
+
+         raise(MalformedTaskPersisterError.new('The supplied IO object must respond to #read_tasks')) unless persister.respond_to? :read_tasks
+         raise(MalformedTaskPersisterError.new('The supplied IO object must respond to #update_task')) unless persister.respond_to? :update_task
+         raise(MalformedTaskPersisterError.new('The supplied IO object must respond to #delete_task')) unless persister.respond_to? :delete_task
+
+
+         @name          = name.to_s.gsub(/\s/, '_').to_sym
+         @timeout       = timeout
+         @max_attempts  = max_attempts
+         @update_period = update_period
+         @max_tasks     = max_tasks
+         @persister     = persister
+      end
+
+      def work
+         loop do
+            sleep(@update_period)
+
+            # shuffling and re-sorting to avoid worst case O(n^2) on quicksort
+            # when receiving already sorted data. Ideally, we'd use a better algo, but this will do for now
+            tasks = @persister.read_tasks(@name).shuffle.sort_by { |t| t[:run_at] }
+
+            tasks.first(@max_tasks).each do |task_data|
+               if Time.now.to_i >= task_data[:run_at].to_i
+                  tw = TaskWorker.new(task_data)
+
+                  tw.work
+
+                  if tw.successful?
+                     @persister.delete_task(task_data[:id])
+                  else
+                     @persister.update_task(tw.to_hash.merge(queue: @name))
+                  end
+               end
+            end
+         end
+      end
+   end
+
+   class MalformedTaskPersisterError < StandardError
+   end
+end

data/lib/procrastinator/task_worker.rb

@@ -0,0 +1,112 @@
+require 'yaml'
+
+module Procrastinator
+   class TaskWorker
+      attr_reader :id, :run_at, :initial_run_at, :expire_at, :task, :attempts, :last_fail_at, :last_error
+
+      def initialize(id: nil,
+                     run_at: nil,
+                     initial_run_at: nil,
+                     expire_at: nil,
+                     attempts: 0,
+                     timeout: nil,
+                     max_attempts: nil,
+                     last_fail_at: nil,
+                     last_error: nil,
+                     task:)
+         @id             = id
+         @run_at         = run_at.to_i
+         @initial_run_at = initial_run_at.to_i
+         @expire_at      = expire_at.nil? ? nil : expire_at.to_i
+         @task           = YAML.load(task)
+         @attempts       = attempts
+         @max_attempts   = max_attempts
+         @timeout        = timeout
+         @last_fail_at   = last_fail_at
+         @last_error     = last_error
+
+         raise(MalformedTaskError.new('given task does not support #run method')) unless @task.respond_to? :run
+         raise(ArgumentError.new('timeout cannot be negative')) if timeout && timeout < 0
+      end
+
+      def work
+         @attempts += 1
+
+         begin
+            raise(TaskExpiredError.new("task is over its expiry time of #{@expire_at}")) if expired?
+
+            Timeout::timeout(@timeout) do
+               @task.run
+            end
+
+            try_hook(:success)
+            @last_error   = nil
+            @last_fail_at = nil
+         rescue StandardError => e
+            @last_fail_at = Time.now.to_i
+
+            if too_many_fails? || expired?
+               try_hook(:final_fail, e)
+
+               @run_at     = nil
+               @last_error = "#{expired? ? 'Task expired' : 'Task failed too many times'}: #{e.backtrace.join("\n")}"
+            else
+               try_hook(:fail, e)
+
+               @last_error = 'Task failed: ' + e.backtrace.join("\n")
+
+               reschedule
+            end
+         end
+      end
+
+      def successful?
+         if !expired? && @attempts <= 0
+            raise(RuntimeError, 'you cannot check for success before running #work')
+         end
+
+         @last_error.nil? && @last_fail_at.nil?
+      end
+
+      def too_many_fails?
+         !@max_attempts.nil? && @attempts >= @max_attempts
+      end
+
+      def expired?
+         !@expire_at.nil? && Time.now.to_i > @expire_at
+      end
+
+      def to_hash
+         {id:             @id,
+          run_at:         @run_at,
+          initial_run_at: @initial_run_at,
+          expire_at:      @expire_at,
+          attempts:       @attempts,
+          last_fail_at:   @last_fail_at,
+          last_error:     @last_error,
+          task:           YAML.dump(@task)}
+      end
+
+      private
+      def try_hook(method, *params)
+         begin
+            @task.send(method, *params) if @task.respond_to? method
+         rescue StandardError => e
+            $stderr.puts "#{method.to_s.capitalize} hook error: #{e.message}"
+         end
+      end
+
+      def reschedule
+         # (30 + n_attempts^4) seconds is chosen to rapidly expand
+         # but with the baseline of 30s to avoid hitting the disc too frequently.
+
+         @run_at += 30 + (@attempts**4)
+      end
+   end
+
+   class TaskExpiredError < StandardError
+   end
+
+   class MalformedTaskError < StandardError
+   end
+end

data/lib/procrastinator/version.rb

@@ -1,3 +1,3 @@
 module Procrastinator
-  VERSION = "0.1.0"
+   VERSION = '0.2.1'
 end

data/lib/procrastinator.rb

@@ -1,5 +1,21 @@
-require "procrastinator/version"
+require 'procrastinator/version'
+require 'procrastinator/queue_worker'
+require 'procrastinator/task_worker'
+require 'procrastinator/environment'
+
 
 module Procrastinator
-  # Your code goes here...
+   def self.setup(persister, &block)
+      raise ArgumentError.new('Procrastinator.setup must be given a block') if block.nil?
+
+      env = Environment.new(persister)
+
+      yield(env)
+
+      raise RuntimeError.new('setup block did not define any queues') if env.queues.empty?
+
+      env.spawn_workers
+
+      env
+   end
 end

data/procrastinator.gemspec

@@ -4,22 +4,26 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'procrastinator/version'
 
 Gem::Specification.new do |spec|
-  spec.name          = "procrastinator"
-  spec.version       = Procrastinator::VERSION
-  spec.authors       = ["Robin Miller"]
-  spec.email         = ["robin@tenjin.ca"]
+   spec.name    = 'procrastinator'
+   spec.version = Procrastinator::VERSION
+   spec.authors = ['Robin Miller']
+   spec.email   = ['robin@tenjin.ca']
 
-  spec.summary       = %q{Simple generalized job queues.}
-  spec.description   = %q{A strightforward job queue that is not dependent on Rails or any particular database or persistence mechanism.}
-  spec.homepage      = "https://github.com/TenjinInc/procrastinator"
-  spec.license       = "MIT"
+   spec.summary     = %q{Delayed task queues made simple.}
+   spec.description = %q{A strightforward job queue that is not dependent on Rails or any particular database or persistence mechanism.}
+   spec.homepage    = 'https://github.com/TenjinInc/procrastinator'
+   spec.license     = 'MIT'
 
-  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
-  spec.bindir        = "exe"
-  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = ["lib"]
+   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+   spec.bindir        = 'exe'
+   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+   spec.require_paths = ['lib']
 
-  spec.add_development_dependency "bundler", "~> 1.11"
-  spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "rspec", "~> 3.0"
+   spec.required_ruby_version = '> 2.0'
+
+   spec.add_development_dependency 'bundler', '~> 1.11'
+   spec.add_development_dependency 'rake', '~> 10.0'
+   spec.add_development_dependency 'rspec', '~> 3.0'
+   spec.add_development_dependency 'timecop', '~> 0.8'
+   spec.add_development_dependency 'simplecov', '~> 0.11'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: procrastinator
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.1
 platform: ruby
 authors:
 - Robin Miller
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-12-20 00:00:00.000000000 Z
+date: 2016-02-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -52,6 +52,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.8'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.8'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.11'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.11'
 description: A strightforward job queue that is not dependent on Rails or any particular
   database or persistence mechanism.
 email:
@@ -62,6 +90,7 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - ".rspec"
+- ".ruby-version"
 - ".travis.yml"
 - CODE_OF_CONDUCT.md
 - Gemfile
@@ -71,6 +100,9 @@ files:
 - bin/console
 - bin/setup
 - lib/procrastinator.rb
+- lib/procrastinator/environment.rb
+- lib/procrastinator/queue_worker.rb
+- lib/procrastinator/task_worker.rb
 - lib/procrastinator/version.rb
 - procrastinator.gemspec
 homepage: https://github.com/TenjinInc/procrastinator
@@ -83,9 +115,9 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
@@ -93,8 +125,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.6
+rubygems_version: 2.2.2
 signing_key: 
 specification_version: 4
-summary: Simple generalized job queues.
+summary: Delayed task queues made simple.
 test_files: []