qmore 0.5.0

data/.gitignore

@@ -0,0 +1,9 @@
+*.gem
+*.DS_Store
+.bundle
+Gemfile.lock
+pkg/*
+.idea
+spec/dump.rdb
+spec/redis-server.log
+spec/redis.pid

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+script: bundle exec rspec spec

data/CHANGELOG

@@ -0,0 +1,4 @@
+0.5.0
+-----
+
+Initial version ported from resque-dynamic-queues 0.8.1

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in qmore.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Matt Conway (matt@conwaysplace.com)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,118 @@
+Qmore is a qless plugin that gives one more control over how queues are processed.  Qmore allows one to specify the queues a worker processes by the use of wildcards, negations, or dynamic look up from redis.  It also allows one to specify the relative priority between queues (rather than within a single queue).  It plugs into the Qless webapp to make it easy to manage the queues.
+
+Authored against Qless 0.9.3, so it at least works with that - try running the tests if you use a different version of qless
+
+[![Build Status](https://secure.travis-ci.org/wr0ngway/qmore.png)](http://travis-ci.org/wr0ngway/qmore)
+
+Usage
+-----
+
+To use the rake tasks built into qless, just adding qless to your Gemfile should cause it to get required before the task executes.  If you aren't using a gemfile, then you'll need to require qmore directly so that it sets up ENV['JOB_RESERVER'] to use Qmore::JobReserver.
+
+Alternatively, if you have some other way of launching workers (e.g. qless-pool), you can assign the reserver explicitly in the setup rake task or some other initializer:
+
+    Qless::Pool.pool_factory.reserver_class = Qmore::JobReserver
+    Qmore.client == Qless::Pool.pool_factory.client
+
+To enable the web UI, use a config.ru similar to the following depending on your environment:
+
+    require 'qless/server'
+    require 'qmore-server'
+    
+    Qless::Server.client = Qless::Client.new(:host => "some-host", :port => 7000)
+    Qmore.client = Qless::Server.client
+    run Qless::Server.new(Qmore.client)
+
+Dynamic Queues
+--------------
+
+Start your workers with a QUEUES that can contain '\*' for zero-or more of any character, '!' to exclude the following pattern, or @key to look up the patterns from redis.  Some examples help:
+
+    QUEUES='foo' rake qless:work
+
+Pulls jobs from the queue 'foo'
+
+    QUEUES='*' rake qless:work
+
+Pulls jobs from any queue
+
+    QUEUES='*foo' rake qless:work
+
+Pulls jobs from queues that end in foo
+
+    QUEUES='*foo*' rake qless:work
+
+Pulls jobs from queues whose names contain foo
+
+    QUEUES='*foo*,!foobar' rake qless:work
+
+Pulls jobs from queues whose names contain foo except the foobar queue
+
+    QUEUES='*foo*,!*bar' rake qless:work
+
+Pulls jobs from queues whose names contain foo except queues whose names end in bar
+
+    QUEUES='@key' rake qless:work
+
+Pulls jobs from queue names stored in redis (use Qless.set\_dynamic\_queue("key", ["queuename1", "queuename2"]) to set them)
+
+    QUEUES='*,!@key' rake qless:work
+
+Pulls jobs from any queue execept ones stored in redis
+
+    QUEUES='@' rake qless:work
+
+Pulls jobs from queue names stored in redis using the hostname of the worker
+
+    Qless.set_dynamic_queue("key", ["*foo*", "!*bar"])
+    QUEUES='@key' rake qless:work
+
+Pulls jobs from queue names stored in redis, with wildcards/negations
+
+    task :custom_worker do
+      ENV['QUEUES'] = "*foo*,!*bar"
+      Rake::Task['qless:work'].invoke
+    end
+
+From a custom rake script
+
+Queue Priority
+--------------
+
+Start your workers with a QUEUES that contains many queue names - the priority is most useful when using wildcards.
+
+The qmore priority web ui is shown as a tab in the qless web UI, and allows you to define the queue priorities.  To activate it, you need to require 'qmore-server' in whatever initializer you use to bring up qless-web.
+
+Then you should set use the web ui to determine the order a worker will pick a queue for processing.  The "Fairly" checkbox makes all queues that match that pattern get ordered in a random fashion.
+
+For example, say my qless system has the queues:
+
+low_foo, low_bar, low_baz, high_foo, high_bar, high_baz, otherqueue, somequeue, myqueue
+
+And I run my worker with QUEUES=\*
+
+If I set my patterns like:
+
+high\_\* (fairly unchecked)  
+default (fairly unchecked)  
+low\_\* (fairly unchecked)  
+
+Then, the worker will scan the queues for work in this order:
+high_bar, high_baz, high_foo, myqueue, otherqueue, somequeue, low_bar, low_baz, low_foo
+
+If I set my patterns like:
+
+high\_\* (fairly checked)  
+default (fairly checked)  
+low\_\* (fairly checked)  
+
+Then, the worker will scan the queues for work in this order:
+
+\*[high_bar, high_baz, high_foo].shuffle, \*[myqueue, otherqueue, somequeue].shuffle, \*[low_bar, low_baz, low_foo].shuffle
+
+
+Contributors
+------------
+
+Matt Conway ( https://github.com/wr0ngway )
+Bert Goethals ( https://github.com/Bertg )

data/Rakefile

@@ -0,0 +1,2 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks

data/config.ru

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+require 'logger'
+
+$LOAD_PATH.unshift ::File.expand_path(::File.dirname(__FILE__) + '/lib')
+require 'qmore-server'
+
+use Rack::ShowExceptions
+run Qless::Server.new(Qmore.client)

data/lib/qmore-server.rb

@@ -0,0 +1,5 @@
+require 'qmore'
+require 'qless/server'
+require 'qmore/server'
+
+Qless::Server.register Qmore::Server

data/lib/qmore.rb

@@ -0,0 +1,22 @@
+require 'qless'
+require 'qless/worker'
+require 'qmore/attributes'
+require 'qmore/job_reserver'
+
+module Qmore
+  
+  def self.client=(client)
+    @client = client
+  end
+  
+  def self.client
+    @client ||= Qless::Client.new 
+  end
+end
+
+module Qless
+  module JobReservers
+    QmoreReserver = Qmore::JobReserver
+  end
+end
+ENV['JOB_RESERVER'] ||= 'QmoreReserver'

data/lib/qmore/attributes.rb

@@ -0,0 +1,191 @@
+require 'multi_json'
+
+module Qmore
+  DYNAMIC_QUEUE_KEY = "qmore:dynamic"
+  PRIORITY_KEY = "qmore:priority"
+  DYNAMIC_FALLBACK_KEY = "default"
+
+  module Attributes
+    extend self
+
+    def redis
+      Qmore.client.redis
+    end
+    
+    def decode(data)
+      MultiJson.load(data) if data
+    end
+    
+    def encode(data)
+      MultiJson.dump(data)
+    end
+    
+    def get_dynamic_queue(key, fallback=['*'])
+      data = redis.hget(DYNAMIC_QUEUE_KEY, key)
+      queue_names = decode(data)
+
+      if queue_names.nil? || queue_names.size == 0
+        data = redis.hget(DYNAMIC_QUEUE_KEY, DYNAMIC_FALLBACK_KEY)
+        queue_names = decode(data)
+      end
+      
+      if queue_names.nil? || queue_names.size == 0
+        queue_names = fallback
+      end
+
+      return queue_names
+    end
+
+    def set_dynamic_queue(key, values)
+      if values.nil? or values.size == 0
+        redis.hdel(DYNAMIC_QUEUE_KEY, key)
+      else
+        redis.hset(DYNAMIC_QUEUE_KEY, key, encode(values))
+      end
+    end
+    
+    def set_dynamic_queues(dynamic_queues)
+      redis.multi do
+        redis.del(DYNAMIC_QUEUE_KEY)
+        dynamic_queues.each do |k, v|
+          set_dynamic_queue(k, v)
+        end
+      end
+    end
+
+    def get_dynamic_queues
+      result = {}
+      queues = redis.hgetall(DYNAMIC_QUEUE_KEY)
+      queues.each {|k, v| result[k] = decode(v) }
+      result[DYNAMIC_FALLBACK_KEY] ||= ['*']
+      return result
+    end
+    
+    def get_priority_buckets
+      priorities = Array(redis.lrange(PRIORITY_KEY, 0, -1))
+      priorities = priorities.collect {|p| decode(p) }
+      priorities << {'pattern' => 'default'} unless priorities.find {|b| b['pattern'] == 'default' }
+      return priorities
+    end
+
+    def set_priority_buckets(data)
+      redis.multi do
+        redis.del(PRIORITY_KEY)
+        Array(data).each do |v|
+           redis.rpush(PRIORITY_KEY, encode(v))
+        end
+      end
+    end
+
+    # Returns a list of queues to use when searching for a job.
+    #
+    # A splat ("*") means you want every queue (in alpha order) - this
+    # can be useful for dynamically adding new queues.
+    #
+    # The splat can also be used as a wildcard within a queue name,
+    # e.g. "*high*", and negation can be indicated with a prefix of "!"
+    #
+    # An @key can be used to dynamically look up the queue list for key from redis.
+    # If no key is supplied, it defaults to the worker's hostname, and wildcards
+    # and negations can be used inside this dynamic queue list.   Set the queue
+    # list for a key with set_dynamic_queue(key, ["q1", "q2"]
+    #
+    def expand_queues(queue_patterns, real_queues)
+      queue_patterns = queue_patterns.dup
+      real_queues = real_queues.dup
+      
+      matched_queues = []
+
+      while q = queue_patterns.shift
+        q = q.to_s
+
+        if q =~ /^(!)?@(.*)/
+          key = $2.strip
+          key = Socket.gethostname if key.size == 0
+
+          add_queues = get_dynamic_queue(key)
+          add_queues.map! { |q| q.gsub!(/^!/, '') || q.gsub!(/^/, '!') } if $1
+
+          queue_patterns.concat(add_queues)
+          next
+        end
+
+        if q =~ /^!/
+          negated = true
+          q = q[1..-1]
+        end
+
+        patstr = q.gsub(/\*/, ".*")
+        pattern = /^#{patstr}$/
+        if negated
+          matched_queues -= matched_queues.grep(pattern)
+        else
+          matches = real_queues.grep(/^#{pattern}$/)
+          matches = [q] if matches.size == 0 && q == patstr
+          matched_queues.concat(matches)
+        end
+      end
+
+      return matched_queues.uniq.sort
+    end
+
+    def prioritize_queues(priority_buckets, real_queues)
+      real_queues = real_queues.dup
+      priority_buckets = priority_buckets.dup
+
+      result = []
+      default_idx = -1, default_fairly = false;
+
+      # Walk the priority patterns, extract each into its own bucket
+      priority_buckets.each do |bucket|
+        bucket_pattern = bucket['pattern']
+        fairly = bucket['fairly']
+
+        # note the position of the default bucket for inserting the remaining queues at that location
+        if bucket_pattern == 'default'
+          default_idx = result.size
+          default_fairly = fairly
+          next
+        end
+
+        bucket_queues, remaining = [], []
+        
+        patterns = bucket_pattern.split(',')
+        patterns.each do |pattern|
+          pattern = pattern.strip
+          
+          if pattern =~ /^!/
+            negated = true
+            pattern = pattern[1..-1]
+          end
+          
+          patstr = pattern.gsub(/\*/, ".*")
+          pattern = /^#{patstr}$/
+        
+        
+          if negated
+            bucket_queues -= bucket_queues.grep(pattern)
+          else
+            bucket_queues.concat(real_queues.grep(pattern))
+          end
+        
+        end
+        
+        bucket_queues.uniq!
+        bucket_queues.shuffle! if fairly
+        real_queues = real_queues - bucket_queues
+        
+        result << bucket_queues
+        
+      end
+
+      # insert the remaining queues at the position the default item was at (or last)
+      real_queues.shuffle! if default_fairly
+      result.insert(default_idx, real_queues)
+      result.flatten!
+
+      return result
+    end
+    
+  end
+end

data/lib/qmore/job_reserver.rb

@@ -0,0 +1,37 @@
+module Qmore
+  class JobReserver
+    include Qmore::Attributes
+    attr_reader :queues
+
+    def initialize(queues)
+      @queues = queues
+    end
+
+    def description
+      @description ||= @queues.map(&:name).join(', ') + " (qmore)"
+    end
+    
+    def reserve
+      realize_queues.each do |q|
+        job = q.pop
+        return job if job
+      end
+      
+      nil
+    end
+    
+    private
+    
+    def realize_queues
+      queue_names = @queues.collect(&:name)
+      real_queues = Qmore.client.queues.counts.collect {|h| h['name'] }
+      
+      realized_queues = expand_queues(queue_names, real_queues)
+      realized_queues = prioritize_queues(get_priority_buckets, realized_queues)
+      realized_queues = realized_queues.collect {|q| Qmore.client.queues[q] }
+      realized_queues
+    end
+          
+  end
+
+end

data/lib/qmore/server.rb

@@ -0,0 +1,93 @@
+require 'qmore'
+
+module Qmore
+  module Server
+
+    Attr = Qmore::Attributes
+
+    def self.registered(app)
+
+      app.helpers do
+        
+        def qmore_view(filename, options = {}, locals = {})
+          options = {:layout => true, :locals => { :title => filename.to_s.capitalize }}.merge(options)
+          dir = File.expand_path("../server/views/", __FILE__)
+          erb(File.read(File.join(dir, "#{filename}.erb")), options, locals)
+        end
+
+        alias :original_tabs :tabs
+        def tabs
+          qmore_tabs = [
+              {:name => 'DynamicQueues', :path => '/dynamicqueues'},
+              {:name => 'QueuePriority', :path => '/queuepriority'}
+          ]
+          queue_tab_index = original_tabs.index {|t| t[:name] == 'Queues' }
+          original_tabs.insert(queue_tab_index + 1, *qmore_tabs)
+        end
+        
+      end
+
+      #
+      # Dynamic queues
+      #
+
+      app.get "/dynamicqueues" do
+        @queues = []
+        real_queues = Qmore.client.queues.counts.collect {|q| q['name'] }
+        dqueues = Attr.get_dynamic_queues
+        dqueues.each do |k, v|
+          expanded = Attr.expand_queues(["@#{k}"], real_queues)
+          expanded = expanded.collect { |q| q.split(":").last }
+          view_data = {
+              'name' => k,
+              'value' => Array(v).join(", "),
+              'expanded' => expanded.join(", ")
+          }
+          @queues << view_data
+        end
+
+        @queues.sort! do |a, b|
+          an = a['name']
+          bn = b['name']
+          if an == 'default'
+            1
+          elsif bn == 'default'
+            -1
+          else
+            an <=> bn
+          end
+        end
+
+        qmore_view :dynamicqueues
+      end
+
+      app.post "/dynamicqueues" do
+        dynamic_queues = Array(params['queues'])
+        queues = {}
+        dynamic_queues.each do |queue|
+          key = queue['name']
+          values = queue['value'].to_s.split(',').collect { |q| q.gsub(/\s/, '') }
+          queues[key] = values
+        end
+        Attr.set_dynamic_queues(queues)
+        redirect "/dynamicqueues"
+      end
+
+      #
+      # Queue priorities
+      #
+
+      app.get "/queuepriority" do
+        @priorities = Attr.get_priority_buckets
+        qmore_view :priorities
+      end
+
+      app.post "/queuepriority" do
+        priorities = params['priorities']
+        Attr.set_priority_buckets priorities
+        redirect "/queuepriority"
+      end
+
+    end
+  end
+end