procrastinator 0.9.0 → 1.0.0.pre.rc2

data/README.md

@@ -1,31 +1,36 @@
 # Procrastinator
-Procrastinator is a pure ruby job scheduling gem to allow your app to put off work for later. 
-Tasks are scheduled in queues and those queues are monitored by separate worker subprocesses. 
-Once the scheduled time arrives, the queue worker performs that task. 
 
-If the task fails to complete or takes too long, it delays it and tries again later.
+Procrastinator is a pure Ruby job scheduling gem. Put off tasks until later or for another process to handle.
+
+Tasks are can be rescheduled and retried after failures, and you can use whatever storage mechanism is needed.
 
 ## Big Picture
+
 If you have tasks like this:
 
 ```ruby
+# Sends a welcome email
 class SendWelcomeEmail
    def run
-      # ... email stuff ...
+      # ... etc
    end
 end
 ```
 
-Setup a procrastination environment:
+Setup a procrastination environment like:
+
 ```ruby
 scheduler = Procrastinator.setup do |env|
-   env.define_queue :greeting, SendWelcomeEmail
-   env.define_queue :thumbnail, GenerateThumbnail, timeout: 60
-   env.define_queue :birthday, SendBirthdayEmail, max_attempts: 3
+   env.with_store some_email_task_database do
+      env.define_queue :greeting, SendWelcomeEmail
+      env.define_queue :birthday, SendBirthdayEmail, max_attempts: 3
+   end
+
+   env.define_queue :thumbnail, GenerateThumbnail, store: 'imgtasks.csv', timeout: 60
 end
 ```
 
-And then get your lazy on:
+Put jobs off until later:
 
 ```ruby
 scheduler.delay(:greeting, data: 'bob@example.com')
@@ -35,31 +40,39 @@ scheduler.delay(:thumbnail, data: {file: 'full_image.png', width: 100, height: 1
 scheduler.delay(:send_birthday_email, run_at: Time.now + 3600, data: {user_id: 5})
 ```
 
+And tell a process to actually do them:
+
+```ruby
+# Starts a daemon with a thread for each queue. 
+# Other options are direct control (for testing) or threaded (for terminals)
+scheduler.work.daemonized!
+```
+
 ## Contents
-  * [Installation](#installation)
-  * [Setup](#setup)
-    + [Defining Queues: `#define_queue`](#defining-queues----define-queue-)
-    + [The Task Loader: `#load_with`](#the-task-loader----load-with-)
-      - [Task Data](#task-data)
-    + [The Task Context: `#provide_context`](#the-task-context----provide-context-)
-    + [Controlling Queue Processes: `#each_process`](#controlling-queue-processes----each-process-)
-      - [The Subprocess Hook](#the-subprocess-hook)
-      - [Naming Processes](#naming-processes)
-      - [Tracking Process IDs](#tracking-process-ids)
-  * [Tasks](#tasks)
-    + [Accessing Task Attributes](#accessing-task-attributes)
-    + [Retries](#retries)
-  * [Scheduling Tasks](#scheduling-tasks)
-    + [Providing Data](#providing-data)
-    + [Controlling Timing](#controlling-timing)
-    + [Rescheduling](#rescheduling)
-    + [Cancelling](#cancelling)
-  * [Preventing Queue Workers](#preventing-queue-workers)
-  * [Test Mode](#test-mode)
-  * [Errors & Logging](#errors---logging)
-  * [Contributing](#contributing)
-    + [Developers](#developers)
-  * [License](#license)
+
+- [Installation](#installation)
+- [Setup](#setup)
+    * [Defining Queues](#defining-queues)
+    * [Task Store](#task-store)
+        + [Data Fields](#data-fields)
+    * [Task Container](#task-container)
+- [Tasks](#tasks)
+    * [Accessing Task Attributes](#accessing-task-attributes)
+    * [Errors & Logging](#errors-logging)
+- [Scheduling Tasks](#scheduling-tasks)
+    * [Providing Data](#providing-data)
+    * [Scheduling](#scheduling)
+    * [Rescheduling](#rescheduling)
+    * [Retries](#retries)
+    * [Cancelling](#cancelling)
+- [Working on Tasks](#working-on-tasks)
+    * [Stepwise Working](#stepwise-working)
+    * [Threaded Working](#threaded-working)
+    * [Daemonized Working](#daemonized-working)
+        + [PID Files](#pid-files)
+- [Contributing](#contributing)
+    * [Developers](#developers)
+- [License](#license)
 
 <!-- ToC generated with http://ecotrust-canada.github.io/markdown-toc/ -->
 
@@ -76,243 +89,185 @@ And then run:
     bundle install
 
 ## Setup
-`Procrastinator.setup` allows you to define which queues are available. You can also optionally
-specify a task loader IO object, task context, and other settings.
 
-```ruby
-Procrastinator.setup do |env|
-   # call methods on env to set configurations
-end
-```
-
-When setup is complete, Procrastinator spins off a sub process to work on each queue and 
-returns the configured scheduler, which is used to `#delay` tasks. 
-
-If there are old queue processes, Procrastinator will kill them before spawning a replacement. 
-
-### Defining Queues: `#define_queue`
-In the setup block, you can call `#define_queue` on the environment: 
+`Procrastinator.setup` allows you to define which queues are available and other settings.
 
 ```ruby
-Procrastinator.setup do |env|
-   env.define_queue :greeting, SendWelcomeEmail
+require 'procrastinator'
+
+scheduler = Procrastinator.setup do |config|
+   # ...
 end
 ```
 
-The first two parameters are the queue name symbol and the task class to run on that queue. 
-
-You can also provide these keyword arguments:
-
- * `:timeout`
- 
-   Duration (seconds) after which tasks in this queue should fail for taking too long.
-    
- * `:max_attempts` 
- 
-   Maximum number of attempts for tasks in this queue. Once `attempts` meets or exceeds `max_attempts`, the task will 
-   be permanently failed.
-    
- * `:update_period`
-  
-   Delay (seconds) between reloads of all tasks from the task loader.
-   
- * `:max_tasks`
- 
-   The maximum number of tasks to run concurrently within a queue worker process.
-
-
-```ruby 
-# all defaults set explicitly:
-env.define_queue :queue_name, YourTaskClass, timeout: 3600, max_attempts: 20, update_period: 10, max_tasks: 10
-```
+It then returns a `Scheduler` that your code can use to schedule tasks or tell to start working.
 
-### The Task Loader: `#load_with`
-The task loader is a [strategy](https://en.wikipedia.org/wiki/Strategy_pattern) pattern object
-that knows how to read and write tasks in your data storage (eg. file, database, etc).
+* See [Scheduling Tasks](#scheduling-tasks)
+* See [Start Working](#start-working)
 
-Procrastinator comes with a simple CSV file task loader by default, but you are encouraged to build one that suits
-your situation. 
+### Defining Queues
 
-In setup, the environment's `#load_with` method expects an task loader instance: 
+In setup, call `#define_queue` with a symbol name and the class that performs those jobs:
 
 ```ruby
-loader = MyTaskLoader.new('tasks.csv')
+# You must provide a queue name and the class that handles those jobs
+config.define_queue :greeting, SendWelcomeEmail
 
-scheduler = Procrastinator.setup do |env|
-   # ... other setup stuff ...
+# but queues have some optional settings, too
+config.define_queue :greeting, SendWelcomeEmail, store: 'tasks.csv', timeout: 60, max_attempts: 2, update_period: 1
 
-   env.load_with loader
-   
-   # if you're using the default CSV loader and want to set where it saves the CSV,
-   # provide the keyword argument :location with your path or filename
-   env.load_with location: '/var/myapp/'
-end
+# all defaults set explicitly
+config.define_queue :greeting, SendWelcomeEmail, store: 'procrastinator.csv', timeout: 3600, max_attempts: 20, update_period: 10
 ```
 
-For per-process resource management (eg. independent database connections), you can build and assign a 
-task loader in the `#each_process` block.
+Description of keyword options:
+
+| Option           | Description   |
+|------------------| ------------- |
+| `:store`         | Storage IO object for tasks. See [Task Store](#task-store)     |
+| `:timeout`       | Max duration (seconds) before tasks are failed for taking too long  |
+| `:max_attempts`  | Once a task has been attempted `max_attempts` times, it will be permanently failed. |
+| `:update_period` | Delay (seconds) between reloads of all tasks from the task store  |
+
+### Task Store
+
+A task store is a [strategy](https://en.wikipedia.org/wiki/Strategy_pattern) pattern object that knows how to read and
+write tasks in your data storage (eg. database, CSV file, etc).
 
 ```ruby
-connection = SomeDatabaseLibrary.connect('my_app_development')
+task_store = ReminderStore.new # eg. some SQL task storage class you wrote
 
-scheduler = Procrastinator.setup do |env|
-   env.load_with MyTaskLoader.new(connection)
-   
-   env.each_process do
-      # make a fresh connection
-      connection.reconnect
-      env.load_with MyTaskLoader.new(connection)
-   end
-   
-   # .. other setup stuff ...
+Procrastinator.setup do |config|
+   config.define_queue(:reminder, ReminderTask, store: task_store)
+
+   # to use the default CSV storage, provide :store with a string or Pathname
+   config.define_queue(:reminder, ReminderTask, store: '/var/myapp/tasks.csv')
 end
 ```
 
-A task loader class is required to implement *all* of the following four methods: 
+A task store is required to implement *all* of the following methods or else it will raise a
+`MalformedPersisterError`:
 
 1. `#read(attributes)`
 
-   Returns a list of hashes from your datastore that match the given attributes hash. The search
-   attributes will be in their final form (eg. `:data` will already be serialized).
-   Each hash must contain the properties listed in [Task Data](#task-data) below.
-     
+   Returns a list of hashes from your datastore that match the given attributes hash. The search attributes will be in
+   their final form (eg. `:data` will already be serialized). Each hash must contain the properties listed
+   in [Task Data](#task-data) below.
+
 2. `#create(queue:, run_at:, initial_run_at:, expire_at:, data:)`
 
-   Creates a task in your datastore. Receives a hash with [Task Data](#task-data) keys: 
+   Saves a task in your storage. Receives a hash with [Task Data](#task-data) keys:
    `:queue`, `:run_at`, `:initial_run_at`, `:expire_at`, and `:data`.
-    
+
 3. `#update(id, new_data)`
- 
+
    Saves the provided full [Task Data](#task-data) hash to your datastore.
-   
+
 4. `#delete(id)`
- 
-   Deletes the task with the given identifier in your datastore.
 
-<!-- This paragraph is here to allow people to google for the error keyword -->
-If your loader is missing any of the above methods, Procrastinator will explode 
-with a `MalformedPersisterError` and you will be sad. 
+   Deletes the task with the given identifier from storage
 
-#### Task Data
-These are the data fields for each individual scheduled task. When using the built-in task loader,
-these are the field names. If you have a database, use this to inform your table schema. 
+Procrastinator comes with a simple CSV file task store by default, but you are encouraged to build one that suits your
+situation.
 
-|  Hash Key         | Type   | Description                                                                             |
-|-------------------|--------| ----------------------------------------------------------------------------------------|
-| `:id`             | int    | Unique identifier for this exact task                                                   |
-| `:queue`          | symbol | Name of the queue the task is inside                                                    | 
-| `:run_at`         | int    | Unix timestamp of when to next attempt running the task. ¹                              |
-| `:initial_run_at` | int    | Unix timestamp of the originally requested run                                          |
-| `:expire_at`      | int    | Unix timestamp of when to permanently fail the task because it is too late to be useful |
-| `:attempts`       | int    | Number of times the task has tried to run; this should only be > 0 if the task fails    |
-| `:last_fail_at`   | int    | Unix timestamp of when the most recent failure happened                                 |
-| `:last_error`     | string | Error message + bracktrace of the most recent failure. May be very long.                |
-| `:data`           | string | Serialized data accessible in the task instance.²                                       |
+_Warning_: Task stores shared between queues **must** be thread-safe if using threaded or daemonized work modes.
 
-> ¹ If `nil`, that indicates that it is permanently failed and will never run, either due to expiry or too many attempts.
+#### Data Fields
 
-> ² Serialized using YAML.dump and unserialized with YAML.safe_load. Keep to simple data types (eg. id numbers) to 
-> reduce storage space, eliminate redundancy, and reduce the chance of a serialization error.
+These are the data fields for each individual scheduled task. When using the built-in task store, these are the field
+names. If you have a database, use this to inform your table schema.
 
-Notice that the times are all stored as unix epoch integer timestamps. This is to avoid confusion or conversion
-errors with timezones or daylight savings.
+|  Hash Key         | Type    | Description                                                                             |
+|-------------------|---------| ----------------------------------------------------------------------------------------|
+| `:id`             | int     | Unique identifier for this exact task                                                   |
+| `:queue`          | symbol  | Name of the queue the task is inside                                                    | 
+| `:run_at`         | iso8601 | Time to attempt running the task next. ¹                                      |
+| `:initial_run_at` | iso8601 | Originally requested run_at. Reset when rescheduled.                                    |
+| `:expire_at`      | iso8601 | Time to permanently fail the task because it is too late to be useful |
+| `:attempts`       | int     | Number of times the task has tried to run; this should only be > 0 if the task fails    |
+| `:last_fail_at`   | iso8601 | Unix timestamp of when the most recent failure happened                                 |
+| `:last_error`     | string  | Error message + bracktrace of the most recent failure. May be very long.                |
+| `:data`           | string  | Serialized data accessible in the task instance.                                      |
 
-### The Task Context: `#provide_context`
-Whatever you give to `#provide_context` will be made available to your Task through the task attribute `:context`. 
+> ¹ `nil` indicates that it is permanently failed and will never run, either due to expiry or too many attempts.
 
-This can be useful for things like app containers, but you can use it for whatever you like.  
+Data is serialized using `JSON.dump` and `JSON.parse` with **symbolized keys**. It is strongly recommended to only
+supply simple data types (eg. id numbers) to reduce storage space, eliminate redundancy, and reduce the chance of a
+serialization error.
 
-```ruby
-Procrastinator.setup do |env|
-   # .. other setup stuff ...
- 
-   env.provide_context some_key: "This hash will be passed into your task's methods"
-end
+Times are all handled as [ISO8601](https://en.wikipedia.org/wiki/ISO_8601) formatted strings.
 
-# ... and in your task ...
-class MyTask
-   include Procrastinator::Task
-   
-   task_attr :context
-   
-   def run
-      puts "My context is: #{context}"
-   end
+#### Default Task Store
+
+Specifying no storage will cause Procrastinator to save tasks using the very basic built-in CSV storage. It is not
+designed for heavy loads, so you should replace it in a production environment.
+
+The file path is defined in `Procrastinator::Store::SimpleCommaStore::DEFAULT_FILE`.
+
+```ruby
+Procrastinator.setup do |config|
+   # this will use the default CSV task store. 
+   config.define_queue(:reminder, ReminderTask)
 end
 ```
 
-### Controlling Queue Processes: `#each_process`
-In the setup block, use `#each_process` to configure details about the queue subprocesses. 
+#### Shared Task Stores
 
-#### The Subprocess Hook
-If you pass a block to `#each_process`, it will be run after the process is forked, 
-but before the queue worker starts working on tasks.
+When there are tasks use the same storage, you can wrap them in a `with_store` block.
 
 ```ruby
-Procrastinator.setup do |env|
-   # ... other setup stuff ...
-
-   env.each_process do 
-      # create process-specific resources here, like database connections 
-      # (the parent process's connection could disappear, because they're asychnronous)
-      connection = SomeDatabase.connect('bob@mainframe/my_database')
-      
-      # these two are the configuration methods you're most likely to use in #each_process
-      env.provide_context MyApp.build_task_package
-      env.load_with MyDatabase.new(connection)
+email_task_store = EmailTaskStore.new # eg. some SQL task storage class you wrote
+
+Procrastinator.setup do |config|
+   with_store(email_task_store) do
+      # queues defined inside this block will use the email task store
+      config.define_queue(:welcome, WelcomeTask)
+      config.define_queue(:reminder, ReminderTask)
    end
+
+   # and this will not use it
+   config.define_queue(:thumbnails, ThumbnailTask)
 end
 ```
 
-NB: The `each_process` block **does not run in Test Mode**. 
+### Task Container
 
-#### Naming Processes
-Each queue subprocess is named after the queue it's working on, eg. `greeting-queue-worker` or 
-`thumbnails-queue-worker`.
+Whatever is given to `#provide_container` will available to Tasks through the task attribute `:container`.
 
-If you're running multiple apps on the same machine, then you may want to distinguish which app the queue worker 
-was spawned for. You can provide a `prefix:` argument to `#each_process` with a string that will be prepended to the 
-process command name.  
+This can be useful for things like app containers, but you can use it for whatever you like.
 
 ```ruby
-scheduler = Procrastinator.setup do |env|
-   # ... other setup stuff ...
-   
-   env.each_process(prefix: 'myapp')
+Procrastinator.setup do |env|
+   env.provide_container lunch: 'Lasagna'
+
+   # .. other setup stuff ...
 end
-```
 
-#### Tracking Process IDs
-The sub processes are tracked by saving their PIDs in files. You can set the directory where those files are saved: 
+# ... and in your task ...
+class LunchTask
+   attr_accessor :logger, :scheduler, :container
 
-```ruby
-scheduler = Procrastinator.setup do |env|
-   # ... other setup stuff ...
-   
-   env.each_process(pid_dir: '/var/run/myapp/')
+   def run
+      logger.info("Today's Lunch is: #{ container[:lunch] }")
+   end
 end
 ```
 
-Any PIDs found in those files are killed right before Procrastinator spawns queues, 
-so that it can replace them with new code.  
-
 ## Tasks
-Your task class is what actually gets run on the task queue. They will look something like this: 
 
+Your task class is what actually gets run on the task queue. They'll look like this:
 
 ```ruby
+
 class MyTask
-   include Procrastinator::Task
-   
-   # Give any of these symbols to task_attr and they will become available as methods
-   # task_attr :data, :logger, :context, :scheduler
-   
+   # These attributes will be assigned by Procrastinator when the task is run.
+   attr_accessor :logger, :scheduler, :container, :data
+
    # Performs the core work of the task. 
    def run
       # ... perform your task ...
    end
-   
-   
+
    # ========================================
    #             OPTIONAL HOOKS
    #
@@ -320,19 +275,19 @@ class MyTask
    # below. Only #run is mandatory.
    #
    # ========================================
-   
+
    # Called after the task has completed successfully. 
    # Receives the result of #run.
    def success(run_result)
       # ...
    end
-   
+
    # Called after #run raises any StandardError (or subclass).
    # Receives the raised error.
    def fail(error)
       # ...
    end
-   
+
    # Called after either is true: 
    #   1. the time reported by Time.now is past the task's expire_at time.
    #   2. the task has failed and the number of attempts is equal to or greater than the queue's `max_attempts`. 
@@ -346,55 +301,67 @@ class MyTask
 end
 ```
 
-### Accessing Task Attributes
-Include `Procrastinator::Task` in your task class and then use `task_attr` to register which task attributes your 
-task wants access to. 
+### Attribute Accessors
+
+Tasks must provide accessors for `:logger`, `:container`, and `:scheduler`, while the `:data` accessor is semi-optional
+(see below). This is to prevent the tasks from referencing unknown variables once they actually get run.
+
+* `:data`  The data you provided in the call to `#delay`. Any task with a `:data` accessor will require data be passed
+  to `#delay`, and vice versa. See [Task Data](#task-data) for more.
+
+* `:container` The container you've provided in your setup. See [Task Container](#task-container) for more.
+
+* `:logger` The queue's Logger object. See [Logging](#logging) for more.
+
+* `:scheduler` A scheduler object that you can use to schedule new tasks (eg. with `#delay`).
+
+### Errors & Logging
+
+Errors that trigger `#fail` or `#final_fail` are saved to the task storage under columns `last_error` and
+`last_error_at`.
+
+Each queue worker also keeps a logfile log using the Ruby
+[Logger class](https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html). Log files are named after the queue (
+eg. `log/welcome-queue-worker.log`).
 
 ```ruby
-class MyTask
-   include Procrastinator::Task
-   
-   # declare the task attributes you care about by calling task_attr. 
-   # You can use any of these symbols: 
-   #             :data, :logger, :context, :scheduler
-   task_attr :data, :logger
-   
-   def run
-      # the attributes listed in task_attr become methods like attr_accessor
-      logger.info("The data for this task is #{data}")
-   end
+scheduler = Procrastinator.setup do |env|
+   # you can set custom log location and level:
+   env.log_with(directory: '/var/log/myapp/', level: Logger::DEBUG)
+
+   # you can also set the log rotation age or size (see Logger docs for details)
+   env.log_with(shift: 1024, age: 5)
+
+   # use a falsey log level to disable logging entirely:
+   env.log_with(level: false)
 end
 ```
 
- * `:data`
-   This is the data that you provided in the call to `#delay`. Any task that registers `:data` as a task 
-   attribute will require data be passed to `#delay`.
-   See [Task Data](#task-data) for more.
-   
- * `:context`
-  
-   The context you've provided in your setup. See [Task Context](#task-context-provide_context) for more.
- 
- * `:logger`
-    
-   The queue's Logger object. See [Logging](#logging) for more.
-    
- * `:scheduler`
- 
-   A scheduler object that you can use to schedule new tasks (eg. with `#delay`).
+The logger can be accessed in your tasks by calling `logger` or `@logger`.
 
+```ruby
 
-### Retries
-Failed tasks have their `run_at` rescheduled on an increasing delay (in seconds) according to this formula: 
+class MyTask
+   attr_accessor :logger, :scheduler, :container
 
-> 30 + (number_of_attempts)<sup>4</sup>
+   def run
+      logger.info('This task got run. Hooray!')
+   end
+end
+```
 
-Situations that call `#fail` or `#final_fail` will cause the error timestamp and reason to be stored in `:last_fail_at` 
-and `:last_error`.
+Some events are always logged by default:
 
+|event               |level  |
+|--------------------|-------|
+|process started     | INFO  |
+|#success called     | DEBUG |
+|#fail called        | DEBUG |
+|#final_fail called  | DEBUG |
 
 ## Scheduling Tasks
-To schedule tasks, just call `#delay` on the environment returned from `Procrastinator.setup`: 
+
+To schedule tasks, just call `#delay` on the environment returned from `Procrastinator.setup`:
 
 ```ruby
 scheduler = Procrastinator.setup do |env|
@@ -406,7 +373,7 @@ end
 scheduler.delay(:reminder, data: 'bob@example.com')
 ``` 
 
-If you have only one queue, you can omit the queue name: 
+If there is only one queue, you may omit the queue name:
 
 ```ruby
 scheduler = Procrastinator.setup do |env|
@@ -417,14 +384,16 @@ scheduler.delay(data: 'bob@example.com')
 ```
 
 ### Providing Data
-Most tasks need some additional information to complete their work, like id numbers, 
 
-The `:data` parameter is serialized to string as YAML, so it's better to keep it as simple as possible. For example, if 
-you have a database instead of passing in a complex Ruby object, pass in just the primary key and reload it in the 
-task's `#run`. This will require less space in your database and avoids obsolete or duplicated information.    
+Most tasks need some additional information to complete their work, like id numbers,
+
+The `:data` parameter is serialized to string as YAML, so it's better to keep it as simple as possible. For example, if
+you have a database instead of passing in a complex Ruby object, pass in just the primary key and reload it in the
+task's `#run`. This will require less space in your database and avoids obsolete or duplicated information.
 
-### Controlling Timing
-You can set when the particular task is to be run and/or when it should expire. Be aware that the task is not guaranteed 
+### Scheduling
+
+You can set when the particular task is to be run and/or when it should expire. Be aware that the task is not guaranteed
 to run at a precise time; the only promise is that the task will be attempted *after* `run_at` and before `expire_at`.
 
 ```ruby
@@ -435,22 +404,22 @@ scheduler.delay(:greeting, run_at: Time.new(3000, 1, 1), data: 'philip_j_fry@exa
 scheduler.delay(:thumbnail, run_at: Time.now, data: 'shut_up_and_take_my_money.gif')
 ```
 
-You can also set an `expire_at` deadline. If the task has not been run before `expire_at` is passed, then it will be 
-final-failed the next time it would be attempted.
-Setting `expire_at` to `nil` means it will never expire (but may still fail permanently if, 
-say, `max_attempts` is reached).
+You can also set an `expire_at` deadline. If the task has not been run before `expire_at` is passed, then it will be
+final-failed the next time it would be attempted. Setting `expire_at` to `nil` means it will never expire (but may still
+fail permanently if, say, `max_attempts` is reached).
 
 ```ruby
 # will not run at or after 
-scheduler.delay(:happy_birthday, expire_at: Time.new(2018, 03, 17, 12, 00, '-06:00'),  data: 'contact@tenjin.ca'))
+scheduler.delay(:happy_birthday, expire_at: Time.new(2018, 03, 17, 12, 00, '-06:00'), data: 'contact@tenjin.ca')
 
 # expire_at defaults to nil:
 scheduler.delay(:greeting, expire_at: nil, data: 'bob@example.com')
 ```
 
 ### Rescheduling
-Call `#reschedule` with the queue name and some identifying 
-information, and then calling #to on that to provide the new time.
+
+Call `#reschedule` with the queue name and some identifying information, and then calling #to on that to provide the new
+time.
 
 ```ruby
 scheduler = Procrastinator.setup do |env|
@@ -466,18 +435,30 @@ scheduler.reschedule(:reminder, data: 'bob@example.com').to(run_at: Time.parse('
 scheduler.reschedule(:reminder, data: 'bob@example.com').to(expire_at: Time.parse('June 23 12:00'))
 
 # or both
-scheduler.reschedule(:reminder, data: 'bob@example.com').to(run_at:    Time.parse('June 20 12:00'), 
+scheduler.reschedule(:reminder, data: 'bob@example.com').to(run_at:    Time.parse('June 20 12:00'),
                                                             expire_at: Time.parse('June 23 12:00'))
 ```
 
-Rescheduling updates the task's `:run_at` and `:initial_run_at` to a new value, if provided and/or 
-`:expire_at` to a new value if provided. A `RuntimeError` is raised if the resulting runtime is after the expiry. 
+Rescheduling sets the task's:
+
+* `:run_at` and `:initial_run_at` to a new value, if provided
+* `:expire_at` to a new value if provided.
+* `:attempts` to `0`
+* `:last_error` and `:last_error_at` to `nil`.
+
+Rescheduling will not change `:id`, `:queue` or `:data`. A `RuntimeError` is raised if the runtime is after the expiry.
 
-It also resets `:attempts` to `0` and clears both `:last_error` and `:last_error_at` to `nil`.
+### Retries
+
+Failed tasks have their `run_at` rescheduled on an increasing delay (in seconds) according to this formula:
 
-Rescheduling will not change `:id`, `:queue` or `:data`.
+> 30 + (number_of_attempts)<sup>4</sup>
+
+Situations that call `#fail` or `#final_fail` will cause the error timestamp and reason to be stored in `:last_fail_at`
+and `:last_error`.
 
 ### Cancelling
+
 Call `#cancel` with the queue name and some identifying information to narrow the search to a single task.
 
 ```ruby
@@ -497,112 +478,125 @@ scheduler.reschedule(:reminder, run_at: Time.parse('June 1'), data: 'bob@example
 scheduler.reschedule(:reminder, id: 137)
 ```
 
-## Preventing Queue Workers
-Sometimes you want to be able to put your app into a maintenance mode temporarily. 
-If Procrastinator sees the environment variable `PROCRASTINATOR_STOP` is set, it
-will not spawn child process queue workers at all. 
+## Working on Tasks
 
-To kill workers, set `PROCRASTINATOR_STOP` to 1 and then restart your app. It will
-find all the old subprocesses and halt them, but not create new ones.  
+Use the scheduler object returned by setup to `#work` queues **serially**, **threaded**, or **daemonized**.
 
-This is distinct from Test Mode, because Test Mode will still create workers, but on the
-same process as the original caller to setup. 
+### Serial Working
 
-## Test Mode
-Procrastinator uses multi-threading and multi-processing internally, which is a nightmare for automated testing. 
-Test Mode will disable all of that and rely on your tests to tell it when to act. 
+Working serially performs a task from each queue directly. There is no multithreading or daemonizing.
 
-Set `Procrastinator.test_mode = true` before setup, or call `#enable_test_mode` on 
-the procrastination environment:
+Work serially for TDD tests or other situations you need close direct control.
 
 ```ruby
-# all further calls to `Procrastinator.setup` will produce a procrastination environment where Test Mode is enabled
-Procrastinator.test_mode = true
- 
-# or you can also enable it in the setup
-scheduler = Procrastinator.setup do |env|
-   env.enable_test_mode
-    
-   # ... other settings...
-end
-```
+# work just one task, no threading
+scheduler.work.serially
 
-Then in your tests, tell the procrastinator environment to work off one item: 
+# work the first five tasks
+scheduler.work.serially(steps: 5)
 
+# only work tasks on greeting and reminder queues
+scheduler.work(:greeting, :reminders).serially(steps: 2)
 ```
-# execute one task on all queues
-env.act
 
-# or provide queue names to execute one task on those specific queues
-scheduler.act(:cleanup, :email)
-```
+### Threaded Working
 
-## Errors & Logging
-Errors that trigger #fail or #final_fail are saved in the task persistence (database, file, etc) under `last_error` and 
-`last_error_at`.
+Threaded working will spawn a worker thread per queue.
 
-Each queue worker also writes its own log using the Ruby 
-[Logger class](https://ruby-doc.org/stdlib-2.5.1/libdoc/logger/rdoc/Logger.html).
-The log files are named after its queue process name (eg. `log/welcome-queue-worker.log`) and
-they are saved in the log directory defined in setup. 
+Use threaded working for task queues that should only run while the main application is running. This includes the usual
+caveats around multithreading, so proceed with caution.
 
 ```ruby
-scheduler = Procrastinator.setup do |env|
-   # ... other setup stuff ... 
-
-   # you can set custom log directory and level:
-   env.log_inside '/var/log/myapp/'
-   env.log_at_level Logger::DEBUG
-   
-   # these are the defaults:
-   env.log_inside 'log/' # relative to the running directory
-   env.log_at_level Logger::INFO
-   
-   # use nil to disable logging entirely:
-   env.log_inside nil
-end
+# work tasks until the application exits
+scheduler.work.threaded
+
+# work tasks for 5 seconds
+scheduler.work.threaded(timeout: 5)
+
+# only work tasks on greeting and reminder queues
+scheduler.work(:greeting, :reminders).threaded
 ```
 
-The logger can be accessed in your tasks by including Procrastinator::Task in your task class and then calling
-`task_attr :logger`. 
+### Daemonized Working
+
+Daemonized working **consumes the current process** and then proceeds with threaded working in the new daemon.
+
+Use daemonized working for production environments, especially in conjunction with daemon monitors
+like [Monit](https://mmonit.com/monit/). Provide a block to daemonized! to get
 
 ```ruby
-class MyTask
-   include Procrastinator::Task
-   
-   task_attr :logger
+# work tasks forever as a headless daemon process.
+scheduler.work.daemonized!
 
-   def run
-      logger.info('This task got run. Hooray!')
-   end
+# you can specify the new process name and the directory to save the procrastinator.pid file 
+scheduler.work.daemonized!(name: 'myapp-queue', pid_path: '/var/run')
+
+# ... or set the pid file name precisely by giving a .pid path
+scheduler.work.daemonized!(pid_path: '/var/run/myapp.pid')
+
+# only work tasks in the 'greeting' and 'reminder' queues
+scheduler.work(:greeting, :reminders).daemonized!
+
+# supply a block to run code after the daemon subprocess has forked off
+scheduler.work.daemonized! do
+   # this gets run after the daemon is spawned
+   task_store.reconnect_mysql
 end
 ```
 
-**Default Log Messages**
+Procrastinator endeavours to be thread-safe and support concurrency, but this flexibility allows for many possible
+combinations.
 
-|event               |level  |
-|--------------------|-------|
-|process started     | INFO  |
-|#success called     | DEBUG |
-|#fail called        | DEBUG |
-|#final_fail called  | DEBUG |
+Expected use is a single process with one thread per queue. More complex use is possible but Procrastinator can't
+guarantee concurrency in your Task Store.
+
+#### PID Files
+
+Process ID files are a single-line file that saves the daemon's process ID number. It's saved to the directory given
+by `:pid_dir`. The default location is `pids/` relative to the file that called `#daemonized!`.
+
+## Similar Tools
+
+Procrastinator is a library that exists to enable job queues with flexibility in storage mechanism and minimal
+dependencies. It's neat but it is specifically intended for smaller datasets. Some other approaches include:
+
+### Linux etc: Cron and At
+
+Consider [Cron](https://en.wikipedia.org/wiki/Cron) for tasks that run on a regular schedule.
+
+Consider [At](https://en.wikipedia.org/wiki/At_(command)) for tasks that run once at a particular time.
+
+While neither tool natively supports retry, they can be great solutions for simple situations.
+
+### Gem: Resque
+
+Consider [Resque](https://rubygems.org/gems/resque) for larger datasets (eg. 10,000+ jobs) where performance
+optimization becomes relevant.
+
+### Gem: Rails ActiveJob / DelayedJob
+
+Consider [DelayedJob](https://rubygems.org/gems/delayed_job) for projects that are tightly integrated with Rails and
+fully commit to live in that ecosystem.
 
 ## Contributing
-Bug reports and pull requests are welcome on GitHub at 
+
+Bug reports and pull requests are welcome on GitHub at
 [https://github.com/TenjinInc/procrastinator](https://github.com/TenjinInc/procrastinator).
- 
-This project is intended to be a friendly space for collaboration, and contributors are expected to adhere to the 
+
+This project is intended to be a friendly space for collaboration, and contributors are expected to adhere to the
 [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 Play nice.
 
 ### Developers
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can 
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can
 also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the 
-version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, 
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the
+version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version,
 push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## License
+
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).