que 1.0.0.beta → 1.0.0.beta2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9bbd70fe386be02949156458e095b91fc7af2001
-  data.tar.gz: d7029485d2eb220cad37d8b41ca8021a6089e934
+SHA256:
+  metadata.gz: 8ad84e5065bf9a35a8ed97f1b905b62935a67bf3713d537c4601c1bb6bc48e7c
+  data.tar.gz: 6ced1d272b59a81854f6794147856a1f21bfe048316d767b27ee984935fa8300
 SHA512:
-  metadata.gz: 94f36a7474bf38f0a06839f4ce7afc904bb855e828aff90891ccdfbf87f77e017bd823f3b92ed0bbce3ade75b931d4158c70ee1df3fd4d52d48554a8f7e2c74e
-  data.tar.gz: e90625eb0de12115e779cd2f497db6b4348e26736959f6c4d7b8bd40b58b51c9b086c6f8acaf4569f31b120733648e6bb84cfe6a52dc1628167efead9a016250
+  metadata.gz: a103cc5cd43608bc66023a16b32fe71f87b5adbaae7ba8d544f388d704d12d18fddbf2528f98d5aaddaf4e93d4c112d27f2921c3f33c4fd9dedb23410e8ff24e
+  data.tar.gz: befd6ca6e2ecd9f7f454e8a7c1789d5effb1638b4b9fb74ed5c83b07e413e7f5d54ea76338621e6a696464194a6118fe0ffcd46acc0c094b0fd57a4e8a244b85

data/CHANGELOG.1.0.beta.md

@@ -0,0 +1,127 @@
+### 1.0.0.beta2 (2018-04-13)
+
+*   Fixed an incompatibility that caused the new locker to hang when using Rails in development mode (#213).
+
+*   Fixed a bug with setting the log level via the CLI when the configured logger was based on a callable (#210).
+
+*   Renamed Que.middleware to Que.job_middleware.
+
+*   Added Que.sql_middleware.
+
+*   Officially added support for Ruby 2.5.
+
+*   Internal cleanup and renamings.
+
+### 1.0.0.beta (2017-10-25)
+
+*   **A schema upgrade to version 4 will be required for this release.** See [the migration doc](https://github.com/chanks/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
+
+    *   Please note that this migration requires a rewrite of the jobs table, which makes it O(n) with the size of the table. If you have a very large backlog of jobs you may want to schedule downtime for this migration.
+
+*   Que's implementation has been changed from one in which worker threads hold their own PG connections and lock their own jobs to one in which a single thread (and PG connection) locks jobs through LISTEN/NOTIFY and batch polling, and passes jobs along to worker threads. This has many benefits, including:
+
+    *   Jobs queued for immediate processing can be actively distributed to workers with LISTEN/NOTIFY, which is more efficient than having workers repeatedly poll for new jobs.
+
+    *   When polling is necessary (to pick up jobs that are scheduled for the future or that need to be retried due to errors), jobs can be locked and fetched in batches, rather than one at a time.
+
+    *   Individual workers no longer need to monopolize their own (usually idle) connections while working jobs, so Ruby processes will require fewer Postgres connections.
+
+    *   PgBouncer or another external connection pool can be used for workers' connections (though not for the connection used to lock and listen for jobs).
+
+*   Other features introduced in this version include:
+
+    *   Much better support for all versions of ActiveJob.
+
+        *   In particular, you may (optionally) include `Que::ActiveJob::JobExtensions` into `ApplicationJob` to get support for all of Que's job helper methods.
+
+    *   Custom middleware that wrap running jobs are now supported.
+
+    *   Support for categorizing jobs with tags.
+
+    *   Support for configuring a `maximum_retry_count` on individual job classes.
+
+    *   Job configuration options are now inheritable, so job class hierarchies are more useful.
+
+    *   There are now built-in models for ActiveRecord and Sequel to allow inspecting the queue easily.
+
+    *   Jobs that have finished working may optionally be retained in the database indefinitely.
+
+        *   To keep a job record, replace the `destroy` calls in your jobs with `finish`. `destroy` will still delete records entirely, for jobs that you don't want to keep.
+
+        *   If you don't resolve a job yourself one way or another, Que will still `destroy` the job for you by default.
+
+        *   Finished jobs have a timestamp set in the finished_at column.
+
+    *   Jobs that have errored too many times will now be marked as expired, and won't be retried again.
+
+        *   You can configure a maximum_retry_count in your job classes, to set the threshold at which a job will be marked expired. The default is 15.
+
+        *   To manually mark a job as expired (and keep it in the database but not try to run it again) you can call `expire` helper in your job.
+
+    *   You can now set job priority thresholds for individual workers, to ensure that there will always be space available for high-priority jobs.
+
+    *   `Que.job_states` returns a list of locked jobs and the hostname/pid of the Ruby processes that have locked them.
+
+    *   `Que.connection_proc=` has been added, to allow for the easy integration of custom connection pools.
+
+*   In keeping with semantic versioning, the major version is being bumped since the new implementation requires some backwards-incompatible changes. These changes include:
+
+    *   Support for MRI Rubies before 2.2 has been dropped.
+
+    *   Support for Postgres versions before 9.5 has been dropped (JSONB and upsert support is required).
+
+    *   JRuby support has been dropped. It will be reintroduced whenever the jruby-pg gem is production-ready.
+
+    *   The `que:work` rake task has been removed. Use the `que` executable instead.
+
+        *   Therefore, configuring workers using QUE_* environment variables is no longer supported. Please pass the appropriate options to the `que` executable instead.
+
+    *   The `mode` setter has been removed.
+
+        *   To run jobs synchronously when they are enqueued (the old `:sync` behavior) you can set `Que.run_synchronously = true`.
+
+        *   To start up the worker pool (the old :async behavior) you should use the `que` executable to start up a worker process. There's no longer a supported API for running workers outside of the `que` executable.
+
+    *   The following methods are not meaningful under the new implementation and have been removed:
+
+        *   The `Que.wake_interval` getter and setter.
+
+        *   The `Que.worker_count` getter and setter.
+
+        *   `Que.wake!`
+
+        *   `Que.wake_all!`
+
+    *   Since Que needs a dedicated Postgres connection to manage job locks, running Que through a single PG connection is no longer supported.
+
+        *   It's not clear that anyone ever actually did this.
+
+    *   `Que.worker_states` has been removed, as the connection that locks a job is no longer the one that the job is using to run. Its functionality has been partially replaced with `Que.job_states`.
+
+    *   When using Rails, for simplicity, job attributes and keys in argument hashes are now converted to symbols when retrieved from the database, rather than being converted to instances of HashWithIndifferentAccess.
+
+    *   Arguments passed to jobs are now deep-frozen, to prevent unexpected behavior when the args are mutated and the job is reenqueued.
+
+    *   Since JSONB is now used to store arguments, the order of argument hashes is no longer maintained.
+
+        *   It wouldn't have been a good idea to rely on this anyway.
+
+    *   Calling Que.log() directly is no longer supported/recommended.
+
+    *   Features marked as deprecated in the final 0.x releases have been removed.
+
+*   Finally, if you've built up your own tooling and customizations around Que, you may need to be aware of some DB schema changes made in the migration to schema version #4.
+
+    *   The `job_id` column has been renamed `id` and is now the primary key. This makes it easier to manage the queue using an ActiveRecord model.
+
+    *   Finished jobs are now kept in the DB, unless you explicitly call `destroy`. If you want to query the DB for only jobs that haven't finished yet, add a `WHERE finished_at IS NULL` condition to your query, or use the not_finished scope on one of the provided ORM models.
+
+    *   There is now an `expired_at` timestamp column, which is set when a job reaches its maximum number of retries and will not be attempted again.
+
+    *   Due to popular demand, the default queue name is now "default" rather than an empty string. The migration will move pending jobs under the "" queue to the "default" queue.
+
+    *   The `last_error` column has been split in two, to `last_error_message` and `last_error_backtrace`. These two columns are now limited to 500 and 10,000 characters, respectively. The migration will split old error data correctly, and truncate it if necessary.
+
+    *   Names for queues and job classes are now limited to 500 characters, which is still far longer than either of these values should reasonably be.
+
+    *   There is now a `data` JSONB column which is used to support various ways of organizing jobs (setting tags on them, etc).

data/CHANGELOG.md

@@ -1,4 +1,4 @@
-### 1.0.0.beta (2017-10-25)
+### 1.0.0.beta2 (2018-04-13)
 
 *   **A schema upgrade to version 4 will be required for this release.** See [the migration doc](https://github.com/chanks/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
 
@@ -20,7 +20,7 @@
 
         *   In particular, you may (optionally) include `Que::ActiveJob::JobExtensions` into `ApplicationJob` to get support for all of Que's job helper methods.
 
-    *   Custom middleware that wrap running jobs are now supported.
+    *   Custom middleware that wrap running jobs and executing SQL statements are now supported.
 
     *   Support for categorizing jobs with tags.
 
@@ -68,11 +68,7 @@
 
         *   To start up the worker pool (the old :async behavior) you should use the `que` executable to start up a worker process. There's no longer a supported API for running workers outside of the `que` executable.
 
-    *   The way Que uses prepared statements internally has changed. This shouldn't affect anyone's use of Que, except that the `disable_prepared_statements` configuration option is no longer necessary and has been removed.
-
-        *   Specifically, while Que previously used prepared statements for most of its built-in queries, now only the polling query uses it, due to its complexity. Since the polling query is only run through a dedicated connection, it's no longer possible for prepared statements to conflict with external connection pools, which was the reason that `disable_prepared_statements` was supported in the first place.
-
-    *   In addition to `Que.disable_prepared_statements=`, the following methods are not meaningful under the new implementation and have been removed:
+    *   The following methods are not meaningful under the new implementation and have been removed:
 
         *   The `Que.wake_interval` getter and setter.
 
@@ -116,6 +112,32 @@
 
     *   There is now a `data` JSONB column which is used to support various ways of organizing jobs (setting tags on them, etc).
 
+For a detailed list of the changes between each beta release of 1.0.0, see [the beta Changelog](CHANGELOG.1.0.beta.md).
+
+### 0.14.3 (2018-03-02)
+
+*   Recorded errors now always include the error class, so that empty error messages can still be helpful. (  joehorsnell)
+
+*   Recorded error messages are now truncated to the first 500 characters.
+
+### 0.14.2 (2018-01-05)
+
+*   Deprecate the Que.disable_prepared_statements= accessors.
+
+*   Add Que.use_prepared_statements= configuration accessors.
+
+*   Update the generated Rails migration to declare a version. (NARKOZ)
+
+### 0.14.1 (2017-12-14)
+
+*   Fix a bug with typecasting boolean values on Rails 5+.
+
+### 0.14.0 (2017-08-11)
+
+*   Fix incompatibility with Rails 5.1.
+
+*   Drop support for waking an in-process worker when an ActiveRecord transaction commits.
+
 ### 0.13.1 (2017-07-05)
 
 *   Fix issue that caused error stacktraces to not be persisted in most cases.

data/README.md

@@ -1,6 +1,8 @@
 # Que
 
-**TL;DR: Que is a high-performance job queue that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.**
+**This README and the rest of the docs on the master branch all refer to Que 1.0, which is currently in beta. If you're using version 0.x, please refer to the docs on [the 0.x branch](https://github.com/chanks/que/tree/0.x).**
+
+*TL;DR: Que is a high-performance job queue that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.*
 
 Que ("keɪ", or "kay") is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
 
@@ -113,6 +115,17 @@ You can also add options to run the job after a specific time, or with a specifi
 ChargeCreditCard.enqueue card.id, user_id: current_user.id, run_at: 1.day.from_now, priority: 5
 ```
 
+Finally, you can work jobs using the included `que` CLI. Try running `que -h` to get a list of runtime options:
+```
+$ que -h
+usage: que [options] [file/to/require] ...
+    -h, --help                       Show this help text.
+    -i, --poll-interval [INTERVAL]   Set maximum interval between polls for available jobs, in seconds (default: 5)
+    ...
+```
+
+You may need to pass que a file path to require so that it can load your app. Que will automatically load `config/environment.rb` if it exists, so you shouldn't need an argument if you're using Rails.
+
 ## Testing
 
 There are a couple ways to do testing. You may want to set `Que::Job.run_synchronously = true`, which will cause JobClass.enqueue to simply execute the job's logic synchronously, as if you'd run JobClass.run(*your_args). Or, you may want to leave it disabled so you can assert on the job state once they are stored in the database.

data/bin/command_line_interface.rb

@@ -93,7 +93,7 @@ module Que
               String,
               "Set a custom database url to connect to for locking purposes.",
             ) do |url|
-              connection_url = url
+              options[:connection_url] = url
             end
 
             opts.on(
@@ -107,19 +107,19 @@ module Que
             opts.on(
               '--maximum-buffer-size [SIZE]',
               Integer,
-              "Set maximum number of jobs to be cached in this process " \
-                "awaiting a worker (default: 8)",
+              "Set maximum number of jobs to be locked and held in this " \
+                "process awaiting a worker (default: 8)",
             ) do |s|
-              options[:maximum_queue_size] = s
+              options[:maximum_buffer_size] = s
             end
 
             opts.on(
               '--minimum-buffer-size [SIZE]',
               Integer,
-              "Set minimum number of jobs to be cached in this process " \
-                "awaiting a worker (default: 2)",
+              "Set minimum number of jobs to be locked and held in this " \
+                "process awaiting a worker (default: 2)",
             ) do |s|
-              options[:minimum_queue_size] = s
+              options[:minimum_buffer_size] = s
             end
 
             opts.on(
@@ -172,7 +172,7 @@ OUTPUT
         end
 
         begin
-          Que.logger.level = Logger.const_get(log_level.upcase)
+          Que.get_logger.level = Logger.const_get(log_level.upcase)
         rescue NameError
           output.puts "Unsupported logging level: #{log_level} (try debug, info, warn, error, or fatal)"
           return 1
@@ -195,19 +195,6 @@ OUTPUT
 
         options[:poll_interval] = poll_interval
 
-        if connection_url
-          uri = URI.parse(connection_url)
-
-          options[:connection] =
-            PG::Connection.open(
-              host:     uri.host,
-              user:     uri.user,
-              password: uri.password,
-              port:     uri.port || 5432,
-              dbname:   uri.path[1..-1],
-            )
-        end
-
         locker =
           begin
             Que::Locker.new(options)
@@ -221,13 +208,14 @@ OUTPUT
         $stop_que_executable = false
         %w[INT TERM].each { |signal| trap(signal) { $stop_que_executable = true } }
 
+        output.puts "Que waiting for jobs..."
+
         loop do
           sleep 0.01
           break if $stop_que_executable || locker.stopping?
         end
 
-        output.puts ''
-        output.puts "Finishing Que's current jobs before exiting..."
+        output.puts "\nFinishing Que's current jobs before exiting..."
 
         locker.stop!
 

data/docs/active_job.md

@@ -1,6 +1,6 @@
 ## Using Que With ActiveJob
 
 You can include `Que::ActiveJob::JobExtensions` into your `ApplicationJob` subclass to get support for all of Que's 
-[helper methods](/job_helper_methods.md). These methods will become no-ops if you use a queue adapter that isn't Que, so if you like to use a different adapter in development they shouldn't interfere.
+[helper methods](/docs/job_helper_methods.md). These methods will become no-ops if you use a queue adapter that isn't Que, so if you like to use a different adapter in development they shouldn't interfere.
 
 Additionally, including `Que::ActiveJob::JobExtensions` lets you define a run() method that supports keyword arguments.

data/docs/command_line_interface.md

@@ -10,8 +10,8 @@ usage: que [options] [file/to/require] ...
     -w, --worker-count [COUNT]       Set number of workers in process (default: 6)
         --connection-url [URL]       Set a custom database url to connect to for locking purposes.
         --log-internals              Log verbosely about Que's internal state. Only recommended for debugging issues
-        --maximum-buffer-size [SIZE] Set maximum number of jobs to be cached in this process awaiting a worker (default: 8)
-        --minimum-buffer-size [SIZE] Set minimum number of jobs to be cached in this process awaiting a worker (default: 2)
+        --maximum-buffer-size [SIZE] Set maximum number of jobs to be locked and held in this process awaiting a worker (default: 8)
+        --minimum-buffer-size [SIZE] Set minimum number of jobs to be locked and held in this process awaiting a worker (default: 2)
         --wait-period [PERIOD]       Set maximum interval between checks of the in-memory job queue, in milliseconds (default: 50)
         --worker-priorities [LIST]   List of priorities to assign to workers, unspecified workers take jobs of any priority (default: 10,30,50)
 ```
@@ -30,7 +30,7 @@ This option sets the number of seconds the process will wait between polls of th
 
 ### minimum-buffer-size and maximum-buffer-size
 
-These options set the size of the internal buffer that Que uses to cache job information until it's ready for workers. The default minimum is 2 and the maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked, and will only resort to polling if the buffer dips below 2. If you don't want jobs to be buffered at all, you can set both of these values to zero.
+These options set the size of the internal buffer that Que uses to hold jobs until they're ready for workers. The default minimum is 2 and the maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked, and will only resort to polling if the buffer dips below 2. If you don't want jobs to be buffered at all, you can set both of these values to zero.
 
 ### connection-url
 

data/docs/middleware.md

@@ -1,9 +1,15 @@
-## Defining Middleware For Jobs
+## Middleware
 
-You can define middleware to wrap jobs. For example:
+A new feature in 1.0 is support for custom middleware around various actions.
+
+This API is experimental for the 1.0 beta and may change.
+
+### Defining Middleware For Jobs
+
+You can define middleware to wrap worked jobs. You can use this to add custom instrumentation around jobs, log how long they take to complete, etc.
 
 ``` ruby
-Que.middleware.push(
+Que.job_middleware.push(
   -> (job, &block) {
     # Do stuff with the job object - report on it, count time elapsed, etc.
     block.call
@@ -12,4 +18,19 @@ Que.middleware.push(
 )
 ```
 
-This API is experimental for the 1.0 beta and may change.
+### Defining Middleware For SQL statements
+
+SQL middleware wraps queries that Que executes, or which you might decide to execute via Que.execute(). You can use hook this into NewRelic or a similar service to instrument how long SQL queries take, for example.
+
+``` ruby
+Que.sql_middleware.push(
+  -> (sql, params, &block) {
+    Service.instrument(sql: sql, params: params) do
+      block.call
+    end
+    nil # Still doesn't matter what's returned.
+  }
+)
+```
+
+Please be careful with what you do inside an SQL middleware - this code will execute inside Que's locking thread, which runs in a fairly tight loop that is optimized for performance. If you do something inside this block that incurs blocking I/O (like synchronously touching an external service) you may find Que being less able to pick up jobs quickly.

data/lib/que.rb

@@ -35,7 +35,7 @@ module Que
   require_relative 'que/connection_pool'
   require_relative 'que/job_methods'
   require_relative 'que/job'
-  require_relative 'que/job_cache'
+  require_relative 'que/job_buffer'
   require_relative 'que/locker'
   require_relative 'que/metajob'
   require_relative 'que/migrations'
@@ -44,6 +44,12 @@ module Que
   require_relative 'que/version'
   require_relative 'que/worker'
 
+  class << self
+    attr_writer :default_queue
+  end
+
+  self.default_queue = nil
+
   class << self
     include Utils::Assertions
     include Utils::Constantization
@@ -65,7 +71,6 @@ module Que
 
     # Global configuration logic.
     attr_accessor :use_prepared_statements
-    attr_writer :default_queue
 
     def default_queue
       @default_queue || DEFAULT_QUEUE
@@ -77,8 +82,8 @@ module Que
         if conn.to_s == 'ActiveRecord'
           # Load and setup AR compatibility.
           require_relative 'que/active_record/connection'
-          m = Que::ActiveRecord::Connection::Middleware
-          middleware << m unless middleware.include?(m)
+          m = Que::ActiveRecord::Connection::JobMiddleware
+          job_middleware << m unless job_middleware.include?(m)
           Que::ActiveRecord::Connection.method(:checkout)
         else
           case conn.class.to_s

data/lib/que/active_record/connection.rb

@@ -18,16 +18,16 @@ module Que
         # Use Rails' executor (if present) to make sure that the connection
         # we're using isn't taken from us while the block runs. See
         # https://github.com/chanks/que/issues/166#issuecomment-274218910
-        def wrap_in_rails_executor
+        def wrap_in_rails_executor(&block)
           if defined?(::Rails.application.executor)
-            ::Rails.application.executor.wrap { yield }
+            ::Rails.application.executor.wrap(&block)
           else
             yield
           end
         end
       end
 
-      module Middleware
+      module JobMiddleware
         class << self
           def call(job)
             yield

data/lib/que/active_record/model.rb

@@ -16,11 +16,11 @@ module Que
       scope :finished,     -> { where(t[:finished_at].not_eq(nil)) }
       scope :not_finished, -> { where(t[:finished_at].eq(nil)) }
 
-      scope :scheduled,     -> { where(t[:run_at].gt("now()")) }
-      scope :not_scheduled, -> { where(t[:run_at].lteq("now()")) }
+      scope :scheduled,     -> { where(t[:run_at].gt  (Arel.sql("now()"))) }
+      scope :not_scheduled, -> { where(t[:run_at].lteq(Arel.sql("now()"))) }
 
       scope :ready,     -> { not_errored.not_expired.not_finished.not_scheduled }
-      scope :not_ready, -> { where(t[:error_count].gt(0).or(t[:expired_at].not_eq(nil)).or(t[:finished_at].not_eq(nil)).or(t[:run_at].gt("now()"))) }
+      scope :not_ready, -> { where(t[:error_count].gt(0).or(t[:expired_at].not_eq(nil)).or(t[:finished_at].not_eq(nil)).or(t[:run_at].gt(Arel.sql("now()")))) }
 
       class << self
         def by_job_class(job_class)