backburner-allq 1.0.8 → 1.0.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 79a15bcd72ad936420aa8eacf4d856b0f9816a8bb3444f275876926f55f0d6b1
-  data.tar.gz: 476609e40b08501233c4b621a91444678d22e51ae08e6f6793ea9b35cf194a9c
+  metadata.gz: 27888373123e4db33adc4349646b3313ff962e98a18a3ff26918e7efd701fa92
+  data.tar.gz: b71459c2a3d38a986c86b7a51a238ada776f451ceb9a242dc74b58a9ca873149
 SHA512:
-  metadata.gz: 786cd380f45ee131447351c511e636363557b1bff5da0a27480d93ac2821ce4efb01bf7f19ab779edde15edc97c6460225db3be6b9286564240c6faf202e042b
-  data.tar.gz: 3caa971a79ae76725a7052717f8d803db8fd01c0bd8e08284f22ba08b7a0d2234dcebd1d06b6fd2a2c3146303388f102909b448e03df5ff19902c114c6236adf
+  metadata.gz: 064e65cfb8c739d502b833007fb6b1c9d2c6ffd433c20b8f5501200c44047d596fbbf7eb0a0b55def1503193d0bedb4793c003d814b2986b164562d20f1f7ad0
+  data.tar.gz: a6d06ecca8a34f24e1ab7b667a3334220dbfbb65e7e0da4b4884e8b40e491261316484133ec1ee14b144607c3e8adbc42f2fdbe9e3fc3fba880c63e50ae13cdf

data/CHANGELOG.md

@@ -1,133 +1,4 @@
 # CHANGELOG
 
-## Version 1.5.0 (September 10 2018)
-
-* TBD
-
-## Version 1.4.1 (June 10 2017)
-
-* Fix warning for constant ::Fixnum is deprecated (@amatsuda)
-
-## Version 1.4.0 (May 13 2017)
-
-* Fix unit tests to be more consistent (@eltone)
-* Ensure job supports body hash with symbol keys (@eltone)
-* Add support for custom serialization formats (@eltone)
-* Log the params when a job timeout occurs (@nathantsoi)
-
-## Version 1.3.1 (April 21 2016)
-
-* Addition of thread-pool-based concurrency (@contentfree)
-
-## Version 1.3.0 (February 05 2016)
-
-* Enqueue command now responds with beanstalk response details
-
-## Version 1.2.0 (November 01 2015)
-
-* FIX Made connections to beanstalkd more resilient (@contentfree)
-
-## Version 1.2.0.pre (October 24 2015)
-
- * FIX Replace static Beaneater connection with individual connections per worker instance/thread (@contentfree)
- * FIX Beaneater connections try really hard to repair themselves if disconnected accidentally (@contentfree)
- * NEW Event hook for workers: on_reconnect (@contentfree)
-
-## Version 1.1.0 (September 14 2015)
-
- * NEW Ability to configure namespace separator (@bfolkens)
- * NEW Avoid timeouts altogether by setting queue_respond_timeout to 0 (@zacviandier)
- * NEW Event hooks for on_retry and on_bury (@contentfree)
- * NEW Support lambdas for queue names (@contentfree)
- * NEW Allow for control of delay calculation (@contentfree)
- * NEW Ability to specify environment when running the CLI (@contentfree)
- * NEW Control default async behavior of methods (@contentfree)
-
-## Version 1.0.0 (April 26 2015)
-
- * NEW Updating to Beaneater 1.0 (@alup)
-
-## Version 0.4.6 (October 26 2014)
-
- * NEW Add job to on_error handler if the handler has a 4th argument (@Nitrodist)
- * NEW Use a timeout when looking for a job to reserve (@EasyPost)
- * NEW Support configuring settings on threads on fork class (@silentshade)
- * FIX queue override by existing queues (@silentshade)
- * FIX Use thread to log exit message (@silentshade)
-
-## Version 0.4.5 (December 16 2013)
-
- * FIX #47 Create a backburner connection per thread (Thanks @thcrock)
-
-## Version 0.4.4 (October 27 2013)
-
- * NEW #51 Added ability to set per-queue default ttr's (Thanks @ryanjohns)
-
-## Version 0.4.3 (July 19 2013)
-
- * FIX #44 Additional fix to issue introduced in 0.4.2
- * FIX #45 More graceful shutdown using Kernel.exit and rescuing SystemExit. (Thanks @ryanjohns)
-
-## Version 0.4.2 (July 3 2013)
-
- * FIX #44 Properly retry to connect to beanstalkd when connection fails.
-
-## Version 0.4.1 (June 28 2013)
-
- * FIX #43 Properly support CLI options and smart load the app environment.
-
-## Version 0.4.0 (June 28 2013)
-
-NOTE: This is the start of working with @bradgessler to improve backburner and merge with quebert
-
- * NEW #26 #27 Remove need for Queue mixin, allow plain ruby objects
- * NEW Default all jobs to a single general queue rather than separate queues
- * NEW Add support for named priorities, allowing shorthand names for priority values
-
-## Version 0.3.4 (April 23 2013)
-
- * FIX #22 Adds signal handlers for worker to manage proper shutdown (Thanks @tkiley)
-
-## Version 0.3.3 (April 19 2013)
-
- * Fix naming conflict rename 'config' to 'queue_config'
-
-## Version 0.3.2 (Jan 23 2013)
-
- * Bump version of beaneater to 0.3.0 (better socket handling)
-
-## Version 0.3.1 (Dec 28 2012)
-
- * Adds basic forking processing strategy and rake tasks (Thanks @danielfarrell)
-
-## Version 0.3.0 (Nov 14 2012)
-
- * Major update with support for a 'threads_on_fork' processing strategy (Thanks @ShadowBelmolve)
- * Different workers have different rake tasks (Thanks @ShadowBelmolve)
- * Added processing strategy specific examples i.e stress.rb and adds new unit tests. (Thanks @ShadowBelmolve)
-
-## Version 0.2.6 (Nov 12 2012)
-
- * Upgrade to beaneater 0.2.0
-
-## Version 0.2.5 (Nov 9 2012)
-
- * Add support for multiple worker processing strategies through subclassing.
-
-## Version 0.2.0 (Nov 7 2012)
-
- * Add new plugin hooks feature (see HOOKS.md)
-
-## Version 0.1.2 (Nov 7 2012)
-
- * Adds ability to specify a custom logger.
- * Adds job retry configuration and worker support.
-
-## Version 0.1.1 (Nov 6 2012)
-
- * Fix issue with timed out reserves
-
-## Version 0.1.0 (Nov 4 2012)
-
- * Switch to beaneater as new ruby beanstalkd client
- * Add support for array of connections in `beanstalk_url`
+## Version 1.0.9 (September 9 2020)
+Branched from backburner to use Allq instead

data/HOOKS.md

@@ -6,7 +6,7 @@ In many cases you can use a hook rather than mess around with Backburner's inter
 ## Job Hooks
 
 Hooks are transparently adapted from [Resque](https://github.com/resque/resque/blob/master/docs/HOOKS.md), so
-if you are familiar with their hook API, now you can use nearly the same ones with beanstalkd and backburner!
+if you are familiar with their hook API, now you can use nearly the same ones with allq and backburner!
 
 There are a variety of hooks available that are triggered during the lifecycle of a job:
 

data/README.md

@@ -1,14 +1,14 @@
 # Backburner [![Build Status](https://travis-ci.org/nesquena/backburner.svg?branch=master)](https://travis-ci.org/nesquena/backburner)
 
-Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue that can handle a very high volume of jobs.
+Backburner-allq is a [allq](http://www.allqueue.com)-powered job queue that can handle a very high volume of jobs.
 You create background jobs and place them on multiple work queues to be processed later.
 
-Processing background jobs reliably has never been easier than with beanstalkd and Backburner. This gem works with any ruby-based
+Processing background jobs reliably has never been easier than with allq and Backburner. This gem works with any ruby-based
 web framework, but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails.
 
-If you want to use beanstalk for your job processing, consider using Backburner.
+If you want to use allq for your job processing, consider using Backburner.
 Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads.
-Persistent queues are supported when beanstalkd persistence mode is enabled.
+Persistent queues are supported when allq persistence mode is enabled.
 
 Backburner supports multiple queues, job priorities, delays, and timeouts. In addition,
 Backburner has robust support for retrying failed jobs, handling error cases,
@@ -26,28 +26,14 @@ libraries under the hood. Every job queue requires a queue store that jobs are p
 In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through
 **ActiveRecord** and a database such as PostgreSQL.
 
-The work queue underlying these gems tells you infinitely more about the differences than anything else.
-Beanstalk is probably the best solution for job queues available today for many reasons.
-The real question then is... "Why Beanstalk?".
 
-## Why Beanstalk?
+You will quickly see that **allq** is an underrated but incredible project that is extremely well-suited as a job queue.
+Significantly better suited for this task than Redis or a database. 
 
-Illya has an excellent blog post
-[Scalable Work Queues with Beanstalk](http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/) and
-Adam Wiggins posted [an excellent comparison](http://adam.herokuapp.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/).
+A single instance of Allq is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
+because it is an in-memory, event-driven system. 
 
-You will quickly see that **beanstalkd** is an underrated but incredible project that is extremely well-suited as a job queue.
-Significantly better suited for this task than Redis or a database. Beanstalk is a simple,
-and a very fast work queue service rolled into a single binary - it is the memcached of work queues.
-Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project.
-[PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day.
-
-A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
-because it is an in-memory, event-driven system. Powered by libevent under the hood,
-it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol,
-and a rich set of tools for job management that go well beyond a simple FIFO work queue.
-
-Beanstalkd supports the following features out of the box:
+Allq supports the following features out of the box:
 
 | Feature | Description                     |
 | ------- | ------------------------------- |
@@ -59,27 +45,14 @@ Beanstalkd supports the following features out of the box:
 | **Persistence**     | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
 | **Federation**      | Horizontal scalability provided through federation by the client. |
 | **Error Handling**  | Bury any job which causes an error for later debugging and inspection.|
+| **Fair Queueing**   | Allows you to shard work across queues for fair client queuing.|
+| **Workflows**       | Allows you tell Job C to wait for Job A and Job B to finish first.|
 
-Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
-In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup.
+Keep in mind that these features are supported out of the box with allq and require no special code within this gem to support.
+In the end, **allq is the ideal job queue** while also being ridiculously easy to install and setup.
 
 ## Installation
 
-First, you probably want to [install beanstalkd](http://kr.github.com/beanstalkd/download.html), which powers the job queues.
-Depending on your platform, this should be as simple as (for Ubuntu):
-
-    $ sudo apt-get install beanstalkd
-
-Add this line to your application's Gemfile:
-
-    gem 'backburner'
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
     $ gem install backburner
 
 ## Configuration ##
@@ -88,7 +61,7 @@ Backburner is extremely simple to setup. Just configure basic settings for backb
 
 ```ruby
 Backburner.configure do |config|
-  config.beanstalk_url       = "beanstalk://127.0.0.1"
+  config.beanstalk_url       = "allq://127.0.0.1:8090"
   config.tube_namespace      = "some.app.production"
   config.namespace_separator = "."
   config.on_error            = lambda { |e| puts e }
@@ -112,7 +85,7 @@ The key options available are:
 
 | Option                | Description                                                          |
 | -----------------     | -------------------------------                                      |
-| `beanstalk_url`       | Address for beanstalkd connection i.e 'beanstalk://127.0.0.1'        |
+| `allq_url`            | Address for allq connection i.e 'allq://127.0.0.1:8090'              |
 | `tube_namespace`      | Prefix used for all tubes related to this backburner queue.          |
 | `namespace_separator` | Separator used for namespace and queue name                          |
 | `on_error`            | Lambda invoked with the error whenever any job in the system fails.  |
@@ -129,15 +102,10 @@ The key options available are:
 | `job_serializer_proc` | Lambda serializes a job body to a string to write to the task        |
 | `job_parser_proc`     | Lambda parses a task body string to a hash                           |
 
-## Breaking Changes
-
-Before **v0.4.0**: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would
-be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
-unless otherwise specified.
 
 ## Usage
 
-Backburner allows you to create jobs and place them onto any number of beanstalk tubes, and later pull those jobs off the tubes and
+Backburner allows you to create jobs and place them onto any number of allq tubes, and later pull those jobs off the tubes and
 process them asynchronously with a worker.
 
 ### Enqueuing Jobs ###
@@ -165,7 +133,7 @@ class NewsletterJob
 
   # optional, defaults to respond_timeout in config
   def self.queue_respond_timeout
-    300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and beanstalkd and should be avoided
+    300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and allq and should be avoided
   end
 
   # optional, defaults to retry_delay_proc in config
@@ -387,15 +355,15 @@ This is why our examples use object IDs instead of passing around objects.
 
 ### Named Priorities
 
-As of v0.4.0, Backburner has support for named priorities. beanstalkd priorities are numerical but
+As of v0.4.0, Backburner has support for named priorities. allq priorities are numerical but
 backburner supports a mapping between a word and a numerical value. The following priorities are
-available by default: `high` is 0, `medium` is 100, and `low` is 200.
+available by default: `high` is 0, `medium` is 4, and `low` is 9.
 
 Priorities can be customized with:
 
 ```ruby
 Backburner.configure do |config|
-  config.priority_labels = { :custom => 50, :useful => 5 }
+  config.priority_labels = { :custom => 1, :useful => 5 }
   # or append to default priorities with
   # config.priority_labels  = Backburner::Configuration::PRIORITY_LABELS.merge(:foo => 5)
 end
@@ -557,8 +525,7 @@ Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md
 
 ### Workers in Production
 
-Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html)
-is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
+Once you have Backburner setup in your application, starting workers is really easy. Simply add the task to your Rakefile:
 
 ```ruby
 # Rakefile
@@ -609,8 +576,7 @@ backburner -e $environment
 
 #### Reconnecting
 
-In Backburner, if the beanstalkd connection is temporarily severed, several retries to establish the connection will be attempted.
-After several retries, if the connection is still not able to be made, a `Beaneater::NotConnected` exception will be raised.
+In Backburner, if the allq connection is temporarily severed, several retries to establish the connection will be attempted.
 You can manually catch this exception, and attempt another manual retry using `Backburner::Worker.retry_connection!`.
 
 ### Web Front-end

data/TODO

@@ -1,4 +0,0 @@
-- Custom front-end in sinatra for viewing beanstalk jobs
-  - Refer to https://github.com/denniskuczynski/beanstalkd_view
-- Fork jobs to control memory
-  - https://github.com/michaeldwan/stalker/commit/386267690a7c03e11d1a8b7b6f08b7c9c7cd2c0d

data/backburner-allq.gemspec

@@ -4,9 +4,9 @@ require File.expand_path('../lib/backburner/version', __FILE__)
 Gem::Specification.new do |s|
   s.authors       = ["Jason Malcolm"]
   s.email         = ["jason@blitline.com"]
-  s.description   = %q{Beanstalk background job processing made easy}
+  s.description   = %q{Allq background job processing made easy}
   s.summary       = %q{Reliable allq background job processing made easy for Ruby and Sinatra}
-  s.homepage      = "http://github.com/nesquena/backburner"
+  s.homepage      = "http://github.com/blitline-dev/backburner-allq"
 
   s.files         = `git ls-files`.split($\)
   s.executables   = s.files.grep(%r{^bin/}).map{ |f| File.basename(f) }

data/deploy.sh

@@ -1,3 +1,3 @@
 echo "Did you update the version?"
 gem build backburner-allq.gemspec
-gem push backburner-allq-1.0.8.gem
+gem push backburner-allq-1.0.14.gem

data/lib/backburner/allq_wrapper.rb

@@ -114,13 +114,19 @@ module Backburner
       @client.release_put(job.id)
     end
 
-    def bury(job); end
+    def bury(job)
+     @client.bury_put(job.id)
+    end
 
     def tube_names
       stats_hash = stats
       stats_hash.keys
     end
 
+    def tubes
+      tube_names
+    end
+
     def peek_buried(tube_name = 'default')
       job = nil
       job = @client.peek_get(tube_name, buried: true)
@@ -161,15 +167,8 @@ module Backburner
 
       default = 5
 
-      return default if app_priority == 32_000
-
-      return 6 if app_priority >= 33_000
-
-      return 7 if app_priority >= 35_000
-
-      return 8 if app_priority >= 37_000
-
-      default
+      # return app_priority unless larger than 10
+      app_priority > 10 ? 5 : app_priority
     end
 
     def log_result(job_result)
@@ -191,6 +190,7 @@ module Backburner
                                  ttl: ttl,
                                  delay: delay,
                                  priority: adjusted_priority,
+                                 shard_key: options[:shard_key],
                                  parent_id: parent_id)
       new_job
     end
@@ -212,21 +212,11 @@ module Backburner
                                               priority: adjusted_priority,
                                               timeout: timeout,
                                               run_on_timeout: run_on_timeout,
+                                              shard_key: options[:shard_key],
                                               limit: limit)
       new_parent_job
     end
 
-    def put2(body, pri = 5, ttl = 600, tube_name = "default", delay = 0)
-      # Old school way
-      options = {
-        pri: pri,
-        ttl: ttl,
-        tube_name: tube_name,
-        delay: delay
-      }
-      put(body, options)
-    end
-
     def put(body, options)
       # New school put
       retry_count = 0
@@ -236,7 +226,6 @@ module Backburner
       begin
         Timeout.timeout(10) do
           if body && body.to_s.include?('["default"]')
-            puts "PUTTING DEFAULT! #{caller.inspect}"
           end
 
           if is_parent
@@ -265,6 +254,11 @@ module Backburner
       result
     end
 
+    def stats(tube)
+      final_stats = stats
+      final_stats[tube]
+    end
+
     def stats
       raw_stats = @admin.stats_get
       final_stats = {}

data/lib/backburner/connection.rb

@@ -96,9 +96,17 @@ module Backburner
 
     def put(tube_name, data, opt)
       pri = (opt[:pri] || 5).to_i
-      delay = opt[:delay].to_i
       ttr = (opt[:ttr] || 600).to_i
-      @allq_wrapper.put2(data, pri, ttr, tube_name, delay)
+
+      options = {
+        tube_name: tube_name,
+        pri: pri,
+        delay: opt[:delay].to_i,
+        ttr: ttr
+      }
+
+      options.merge!(opt)
+      @allq_wrapper.put(data, options)
     end
 
     def get(tube_name)

data/lib/backburner/version.rb

@@ -1,3 +1,3 @@
 module Backburner
-  VERSION = "1.0.8"
+  VERSION = "1.0.14"
 end

data/lib/backburner/worker.rb

@@ -40,8 +40,7 @@ module Backburner
         connection.retryable do
           tube_name = expand_tube_name(queue || job_class)
           serialized_data = Backburner.configuration.job_serializer_proc.call(data)
-          puts "Tube name #{tube_name}"
-          response = connection.put(tube_name, serialized_data, :pri => pri, :delay => delay, :ttr => ttr)
+          response = connection.put(tube_name, serialized_data, :pri => pri, :delay => delay, :ttr => ttr, :shard_key => opts[:shard_key])
         end
         return nil unless Backburner::Hooks.invoke_hook_events(job_class, :after_enqueue, *args)
       ensure
@@ -149,14 +148,18 @@ module Backburner
         return
       end
 
-      # NB: There's a slight chance here that the connection to beanstalkd has
+      # NB: There's a slight chance here that the connection to allq has
       # gone down between the time we reserved / processed the job and here.
       num_retries = job.releases
       max_job_retries = resolve_max_job_retries(job.job_class)
       retry_status = "failed: attempt #{num_retries+1} of #{max_job_retries+1}"
       retry_delay = resolve_retry_delay(job.job_class)
       delay = resolve_retry_delay_proc(job.job_class).call(retry_delay, num_retries) rescue retry_delay
-      job.release(delay)
+      if num_retries + 1 > max_job_retries
+        job.bury
+      else
+        job.release(delay)
+      end
       self.log_job_end(job.name, "#{retry_status}, retrying in #{delay}s") if job_started_at
 
       handle_error(e, job.name, job.args, job)

data/lib/backburner/workers/threading.rb

@@ -105,7 +105,7 @@ module Backburner
       # Process the tube settings
       # This overrides @tubes_data set by process_tube_names method. So a tube has name 'super_job:5'
       # and the tube class has setting queue_jobs_limit 10, the result limit will be 10
-      # If the tube is known by existing beanstalkd queue, but not by class - skip it
+      # If the tube is known by existing allq queue, but not by class - skip it
       #
       def process_tube_options
         Backburner::Worker.known_queue_classes.each do |queue|

data/lib/backburner/workers/threads_on_fork.rb

@@ -105,7 +105,7 @@ module Backburner
       # Process the tube settings
       # This overrides @tubes_data set by process_tube_names method. So a tube has name 'super_job:5:20:10'
       # and the tube class has setting queue_jobs_limit 10, the result limit will be 10
-      # If the tube is known by existing beanstalkd queue, but not by class - skip it
+      # If the tube is known by existing allq queue, but not by class - skip it
       #
       def process_tube_options
         Backburner::Worker.known_queue_classes.each do |queue|

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: backburner-allq
 version: !ruby/object:Gem::Version
-  version: 1.0.8
+  version: 1.0.14
 platform: ruby
 authors:
 - Jason Malcolm
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-09-09 00:00:00.000000000 Z
+date: 2021-02-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: allq_rest
@@ -100,7 +100,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: Beanstalk background job processing made easy
+description: Allq background job processing made easy
 email:
 - jason@blitline.com
 executables:
@@ -169,7 +169,7 @@ files:
 - test/workers/simple_worker_test.rb
 - test/workers/threading_worker_test.rb
 - test/workers/threads_on_fork_worker_test.rb
-homepage: http://github.com/nesquena/backburner
+homepage: http://github.com/blitline-dev/backburner-allq
 licenses:
 - MIT
 metadata: {}