que 1.0.0.beta3 → 1.0.0.beta4

data/docs/job_helper_methods.md

@@ -1,27 +0,0 @@
-## Job Helper Methods
-
-There are a number of instance methods on Que::Job that you can use in your jobs, preferably in transactions. See [Writing Reliable Jobs](/writing_reliable_jobs.md) for more information on where to use these methods.
-
-### destroy
-
-This method deletes the job from the queue table, ensuring that it won't be worked a second time.
-
-### finish
-
-This method marks the current job as finished, ensuring that it won't be worked a second time. This is like destroy, in that it finalizes a job, but this method leaves the job in the table, in case you want to query it later.
-
-### expire
-
-This method marks the current job as expired. It will be left in the table and won't be retried, but it will be easy to query for expired jobs. This method is called if the job exceeds its maximum_retry_count.
-
-### retry_in
-
-This method marks the current job to be retried later. You can pass a numeric to this method, in which case that is the number of seconds after which it can be retried (`retry_in(10)`, `retry_in(0.5)`), or, if you're using ActiveSupport, you can pass in a duration object (`retry_in(10.minutes)`). This automatically happens, with an exponentially-increasing interval, when the job encounters an error.
-
-### error_count
-
-This method returns the total number of times the job has errored, in case you want to modify the job's behavior after it has failed a given number of times.
-
-### default_resolve_action
-
-If you don't perform a resolve action (destroy, finish, expire, retry_in) while the job is worked, Que will call this method for you. By default it simply calls `destroy`, but you can override it in your Job subclasses if you wish - for example, to call `finish`, or to invoke some more complicated logic.

data/docs/logging.md

@@ -1,62 +0,0 @@
-## Logging
-
-By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running via the `que` executable). So, your logs will look something like:
-
-```
-I, [2017-08-12T05:07:31.094201 #4687]  INFO -- : {"lib":"que","hostname":"lovelace","pid":21626,"thread":21471100,"event":"job_worked","job_id":6157665,"elapsed":0.531411}
-```
-
-Of course you can have it log wherever you like:
-
-```ruby
-Que.logger = Logger.new(...)
-```
-
-If you don't like logging in JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
-
-```ruby
-Que.log_formatter = proc do |data|
-  "Thread number #{data[:thread]} experienced a #{data[:event]}"
-end
-```
-
-If the log formatter returns nil or false, nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
-
-```ruby
-Que.log_formatter = proc do |data|
-  if [:job_worked, :job_unavailable].include?(data[:event])
-    JSON.dump(data)
-  end
-end
-```
-
-## Logging Job Completion
-
-Que logs a `job_worked` event whenever a job completes, though by default this event is logged at the `DEBUG` level. Since people often run their applications at the `INFO` level or above, this can make the logs too silent for some use cases. Similarly, you may want to log at a higher level if a time-sensitive job begins taking too long to run.
-
-You can solve these problems by configuring the level at which a job is logged on a per-job basis. Simply define a `log_level` method in your job class - it will be called with a float representing the number of seconds it took for the job to run, and it should return a symbol indicating what level to log the job at:
-
-```ruby
-class TimeSensitiveJob < Que::Job
-  def run(*args)
-    RemoteAPI.execute_important_request
-  end
-
-  def log_level(elapsed)
-    if elapsed > 60
-      # This job took over a minute! We should complain about it!
-      :warn
-    elsif elapsed > 30
-      # A little long, but no big deal!
-      :info
-    else
-      # This is fine, don't bother logging at all.
-      false
-    end
-  end
-end
-```
-
-This method should return a symbol that is a valid logging level (one of `[:debug, :info, :warn, :error, :fatal, :unknown]`). If the method returns anything other than one of these symbols, the job won't be logged.
-
-If a job errors, a `job_errored` event will be emitted at the `ERROR` log level. This is not currently configurable.

data/docs/managing_workers.md

@@ -1,25 +0,0 @@
-## Managing Workers
-
-Que uses a multithreaded pool of workers to run jobs in parallel - this allows you to save memory by working many jobs simultaneously in the same process. The `que` executable starts up a pool of 6 workers by default. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
-
-Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will cause you to lose efficiency to context-switching. So, you'll want to choose the appropriate number of workers for your use case.
-
-### Working Jobs Via Executable
-
-```shell
-# Run a pool of 6 workers:
-que
-
-# Or configure the number of workers:
-que --worker-count 10
-```
-
-See `que -h` for a complete list of command-line options.
-
-### Thread-Unsafe Application Code
-
-If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to run a single worker at a time, like so:
-
-```shell
-que --worker-count 1
-```

data/docs/middleware.md

@@ -1,36 +0,0 @@
-## Middleware
-
-A new feature in 1.0 is support for custom middleware around various actions.
-
-This API is experimental for the 1.0 beta and may change.
-
-### Defining Middleware For Jobs
-
-You can define middleware to wrap worked jobs. You can use this to add custom instrumentation around jobs, log how long they take to complete, etc.
-
-``` ruby
-Que.job_middleware.push(
-  -> (job, &block) {
-    # Do stuff with the job object - report on it, count time elapsed, etc.
-    block.call
-    nil # Doesn't matter what's returned.
-  }
-)
-```
-
-### Defining Middleware For SQL statements
-
-SQL middleware wraps queries that Que executes, or which you might decide to execute via Que.execute(). You can use hook this into NewRelic or a similar service to instrument how long SQL queries take, for example.
-
-``` ruby
-Que.sql_middleware.push(
-  -> (sql, params, &block) {
-    Service.instrument(sql: sql, params: params) do
-      block.call
-    end
-    nil # Still doesn't matter what's returned.
-  }
-)
-```
-
-Please be careful with what you do inside an SQL middleware - this code will execute inside Que's locking thread, which runs in a fairly tight loop that is optimized for performance. If you do something inside this block that incurs blocking I/O (like synchronously touching an external service) you may find Que being less able to pick up jobs quickly.

data/docs/migrating.md

@@ -1,27 +0,0 @@
-## Migrating
-
-Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
-
-```ruby
-class UpdateQue < ActiveRecord::Migration[5.0]
-  def self.up
-    Que.migrate! version: 3
-  end
-
-  def self.down
-    Que.migrate! version: 2
-  end
-end
-```
-
-This will make sure that your database schema stays consistent with your codebase. If you're looking for something quicker and dirtier, you can always manually migrate in a console session:
-
-```ruby
-# Change schema to version 3.
-Que.migrate! version: 3
-
-# Check your current schema version.
-Que.db_version #=> 3
-```
-
-Note that you can remove Que from your database completely by migrating to version 0.

data/docs/multiple_queues.md

@@ -1,31 +0,0 @@
-## Multiple Queues
-
-Que supports the use of multiple queues in a single job table. Please note that this feature is intended to support the case where multiple codebases are sharing the same job queue - if you want to support jobs of differing priorities, the numeric priority system offers better flexibility and performance.
-
-For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
-
-```shell
-que --queue-name credit_cards
-# The -q flag is equivalent, and either can be passed multiple times.
-que -q default -q credit_cards
-```
-
-Then you can set jobs to be enqueued in that queue specifically:
-
-```ruby
-ProcessCreditCard.enqueue current_user.id, queue: 'credit_cards'
-
-# Or:
-
-class ProcessCreditCard < Que::Job
-  # Set a default queue for this job class; this can be overridden by
-  # passing the :queue parameter to enqueue like above.
-  self.queue = 'credit_cards'
-end
-```
-
-In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
-
-```ruby
-Que.enqueue current_user.id, job_class: 'ProcessCreditCard', queue: 'credit_cards'
-```

data/docs/shutting_down_safely.md

@@ -1,7 +0,0 @@
-## Shutting Down Safely
-
-To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
-
-To prevent this, Que will block the worker process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
-
-So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.

data/docs/using_plain_connections.md

@@ -1,65 +0,0 @@
-## Using Plain Postgres Connections
-
-If you're not using an ORM like ActiveRecord or Sequel, you can use a distinct connection pool to manage your Postgres connections. Please be aware that if you **are** using ActiveRecord or Sequel, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.
-
-## Using ConnectionPool or Pond
-
-Support for two connection pool gems is included in Que. The first is the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
-
-```ruby
-require 'uri'
-require 'pg'
-require 'connection_pool'
-
-uri = URI.parse(ENV['DATABASE_URL'])
-
-Que.connection = ConnectionPool.new(size: 10) do
-  PG::Connection.open(
-    host:     uri.host,
-    user:     uri.user,
-    password: uri.password,
-    port:     uri.port || 5432,
-    dbname:   uri.path[1..-1]
-  )end
-```
-
-Be sure to pick your pool size carefully - if you use 10 for the size, you'll incur the overhead of having 10 connections open to Postgres even if you never use more than a couple of them.
-
-The Pond gem doesn't have this drawback - it is very similar to ConnectionPool, but establishes connections lazily (add `gem 'pond'` to your Gemfile):
-
-```ruby
-require 'uri'
-require 'pg'
-require 'pond'
-
-uri = URI.parse(ENV['DATABASE_URL'])
-
-Que.connection = Pond.new(maximum_size: 10) do
-  PG::Connection.open(
-    host:     uri.host,
-    user:     uri.user,
-    password: uri.password,
-    port:     uri.port || 5432,
-    dbname:   uri.path[1..-1]
-  )
-end
-```
-
-## Using Any Other Connection Pool
-
-You can use any other in-process connection pool by defining access to it in a proc that's passed to `Que.connection_proc = proc`. The proc you pass should accept a block and call it with a connection object. For instance, Que's built-in interface to Sequel's connection pool is basically implemented like:
-
-```ruby
-Que.connection_proc = proc do |&block|
-  DB.synchronize do |connection|
-    block.call(connection)
-  end
-end
-```
-
-This proc must meet a few requirements:
-- The yielded object must be an instance of `PG::Connection`.
-- It must be reentrant - if it is called with a block, and then called again inside that block, it must return the same object. For example, in `proc.call{|conn1| proc.call{|conn2| conn1.object_id == conn2.object_id}}` the innermost condition must be true.
-- It must lock the connection object and prevent any other thread from accessing it for the duration of the block.
-
-If any of these conditions aren't met, Que will raise an error.

data/docs/using_sequel.md

@@ -1,49 +0,0 @@
-## Using Sequel
-
-If you're using Sequel, with or without Rails, you'll need to give Que a specific database instance to use:
-
-```ruby
-DB = Sequel.connect(ENV['DATABASE_URL'])
-Que.connection = DB
-```
-
-If you are using Sequel's migrator, your app initialization won't happen, so you may need to tweak your migrations to `require 'que'` and set its connection:
-
-```ruby
-require 'que'
-Sequel.migration do
-  up do
-    Que.connection = self
-    Que.migrate! :version => 3
-  end
-  down do
-    Que.connection = self
-    Que.migrate! :version => 0
-  end
-end
-```
-
-Then you can safely use the same database object to transactionally protect your jobs:
-
-```ruby
-class MyJob < Que::Job
-  def run(user_id:)
-    # Do stuff.
-
-    DB.transaction do
-      # Make changes to the database.
-
-      # Destroying this job will be protected by the same transaction.
-      destroy
-    end
-  end
-end
-
-# Or, in your controller action:
-DB.transaction do
-  @user = User.create(params[:user])
-  MyJob.enqueue user_id: @user.id
-end
-```
-
-Sequel automatically wraps model persistance actions (create, update, destroy) in transactions, so you can simply call #enqueue methods from your models' callbacks, if you wish.

data/docs/writing_reliable_jobs.md

@@ -1,108 +0,0 @@
-## Writing Reliable Jobs
-
-Que does everything it can to ensure that jobs are worked exactly once, but if something bad happens when a job is halfway completed, there's no way around it - the job will need be repeated over again from the beginning, probably by a different worker. When you're writing jobs, you need to be prepared for this to happen.
-
-The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also destroy the job inside that transaction, like so:
-
-```ruby
-class UpdateWidgetPrice < Que::Job
-  def run(widget_id)
-    widget = Widget[widget_id]
-    price  = ExternalService.get_widget_price(widget_id)
-
-    ActiveRecord::Base.transaction do
-      # Make changes to the database.
-      widget.update price: price
-
-      # Mark the job as destroyed, so it doesn't run again.
-      destroy
-    end
-  end
-end
-```
-
-Here, you're taking advantage of the guarantees of an [ACID](https://en.wikipedia.org/wiki/ACID) database. The job is destroyed along with the other changes, so either the write will succeed and the job will be run only once, or it will fail and the database will be left untouched. But even if it fails, the job can simply be retried, and there are no lingering effects from the first attempt, so no big deal.
-
-The more difficult type of job is one that makes changes that can't be controlled transactionally. For example, writing to an external service:
-
-```ruby
-class ChargeCreditCard < Que::Job
-  def run(user_id, credit_card_id)
-    CreditCardService.charge(credit_card_id, amount: "$10.00")
-
-    ActiveRecord::Base.transaction do
-      User.where(id: user_id).update_all charged_at: Time.now
-      destroy
-    end
-  end
-end
-```
-
-What if the process abruptly dies after we tell the provider to charge the credit card, but before we finish the transaction? Que will retry the job, but there's no way to tell where (or even if) it failed the first time. The credit card will be charged a second time, and then you've got an angry customer. The ideal solution in this case is to make the job [idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning that it will have the same effect no matter how many times it is run:
-
-```ruby
-class ChargeCreditCard < Que::Job
-  def run(user_id, credit_card_id)
-    unless CreditCardService.check_for_previous_charge(credit_card_id)
-      CreditCardService.charge(credit_card_id, amount: "$10.00")
-    end
-
-    ActiveRecord::Base.transaction do
-      User.where(id: user_id).update_all charged_at: Time.now
-      destroy
-    end
-  end
-end
-```
-
-This makes the job slightly more complex, but reliable (or, at least, as reliable as your credit card service).
-
-Finally, there are some jobs where you won't want to write to the database at all:
-
-```ruby
-class SendVerificationEmail < Que::Job
-  def run(email_address)
-    Mailer.verification_email(email_address).deliver
-  end
-end
-```
-
-In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.
-
-### Timeouts
-
-Long-running jobs aren't necessarily a problem for the database, since the overhead of an individual job is very small (just an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
-
-If there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can (and should) timeout those parts of your job. For example, consider a job that needs to make an HTTP request and then write to the database:
-
-```ruby
-class ScrapeStuff < Que::Job
-  def run(url_to_scrape)
-    result = YourHTTPLibrary.get(url_to_scrape)
-
-    ActiveRecord::Base.transaction do
-      # Insert result...
-
-      destroy
-    end
-  end
-end
-```
-
-That request could take a very long time, or never return at all. Let's use the timeout feature that almost all HTTP libraries offer some version of:
-
-```ruby
-class ScrapeStuff < Que::Job
-  def run(url_to_scrape)
-    result = YourHTTPLibrary.get(url_to_scrape, timeout: 5)
-
-    ActiveRecord::Base.transaction do
-      # Insert result...
-
-      destroy
-    end
-  end
-end
-```
-
-Now, if the request takes more than five seconds, an error will be raised (probably - check your library's documentation) and Que will just retry the job later.