sqew 0.1.0

data/.gitignore

@@ -0,0 +1,19 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+tmp
+test/tmp
+test/version_tmp
+spec/tmp/*
+!spec/tmp/.gitkeep

data/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in smallq.gemspec
+gemspec
+
+gem "rake"

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Zach Moazeni
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,117 @@
+# sqew (pronounced "skew")
+
+sqew is a lightweight background processor. You start a single process that will act as a queue manager and will work multiple jobs concurrently. sqew is short for "small queue" and is not meant to be an all-in-one scalable solution. sqew adopts a similiar API to make migrating to other background processors easy.
+
+## When would sqew be a good fit for my project?
+
+* You don't need to split workers across multiple machines.
+* You don't want to manage multiple background worker processes, but you do want multiple jobs to run concurrently.
+* You don't want to worry about threading issues.
+* You don't want to worry about long running processes memory leaking.
+* You don't care about the enqueueing or job forking performance.
+* You don't need multiple queues (this may change soon).
+
+If these don't fit the bill or you need more power, I recommend you try the great other great gems such as [Resque](https://github.com/defunkt/resque), [Sidekiq](https://github.com/mperham/sidekiq), and [Qu](https://github.com/bkeepers/qu).
+
+## Is it any good?
+
+[Yes.](http://news.ycombinator.com/item?id=3067434)
+
+## Rails Installation
+
+Add this line to your application's Gemfile:
+
+    gem sqew, :require "sqew/rails"
+
+Add an initializer in `config/initializers/sqew.rb`
+
+    Sqew.configure do |config|
+      config.db = "#{Rails.root}/tmp/"
+      config.server = "http://0.0.0.0:8884"
+    end
+
+The `db` config will be a directory where sqew will manage its databases, and the `server` config is what what the worker will connect at as well as where the application will post jobs to.
+
+Once you have sqew configured, you can start the queue manager by running `rake sqew:work`. This will manage the queues, it'll act as a server where the application post jobs, and it will work the jobs as the arrive.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem sqew
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install sqew
+    
+## The Sqew Manager
+
+The Sqew manager is a JSON API for inspecting the queue, pushing work onto the queue, and manipulating the queue and workers. Actions you can perform are:
+
+    # enqueue a job
+    # Sqew.push(TheJobClass, 1, 2, 3)
+    POST /enqueue
+      {"job":"TheJobClass", "args":[1, 2, 3]}
+    
+    # ping the server to programatically see if it is alive
+    # Sqew.ping
+    GET /ping
+    
+    # get the status of the queue, running jobs, failed jobs, and how many workers the server will use
+    # Sqew.status
+    GET /status
+    
+    # dynamically change the number of workers (processes) the manager will use
+    # Sqew.workers = 10 (the default is 3)
+    PUT /workers
+      "10"
+    
+    # clear the entire queue
+    # Sqew.clear
+    DELETE /clear
+    
+    # clear just the failed jobs
+    # Sqew.clear("failed")
+    DELETE /clear
+      failed
+    
+    # delete a specific job by id
+    # Sqew.delete(11)
+    DELETE /11
+  
+## Enqueuing jobs
+
+From your application you will create jobs just like [Resque](https://github.com/defunkt/resque), [Sidekiq](https://github.com/mperham/sidekiq), and [Qu](https://github.com/bkeepers/qu) in the form of
+
+    class MyJob
+      def perform(arg1, arg2)
+        # .. the job code
+      end
+    end
+
+And the manager will receive the job and start working on it when it can. You can enqueue the job from Sqew:
+
+    Sqew.push(MyJob, 1, 2)
+    
+If you're using Rails 4 you can enqueue the job by using the Rails queuing API:
+
+    Rails.queue.push(MyJob, 1, 2)
+
+
+## TODO Soon
+
+* Reusable God/Bluepil/Monit config for managing the worker
+* Javascript browser front-end for the manager
+* Multiple queues with weight (possibly)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,7 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new('spec')
+
+task :default => :spec

data/ext/qu.rb

@@ -0,0 +1,36 @@
+require 'qu/version'
+require 'qu/logger'
+require 'qu/failure'
+require 'qu/payload'
+require 'qu/backend/base'
+
+require 'forwardable'
+require 'logger'
+
+module Qu
+  autoload :Worker, 'qu/worker'
+
+  extend SingleForwardable
+  extend self
+
+  attr_accessor :backend, :failure, :logger
+
+  def_delegators :backend, :length, :queues, :reserve, :clear, :connection=
+
+  def backend
+    @backend || raise("Qu backend not configured. Install one of the backend gems like qu-redis.")
+  end
+
+  def configure(&block)
+    block.call(self)
+  end
+
+  def enqueue(klass, *args)
+    backend.enqueue Payload.new(:klass => klass, :args => args)
+  end
+end
+
+Qu.configure do |c|
+  c.logger = Logger.new(STDOUT)
+  c.logger.level = Logger::INFO
+end

data/ext/slave.rb

@@ -0,0 +1,14 @@
+class Slave
+  def shutdown opts = {}
+    quiet = getopts(opts)['quiet']
+    raise "already shutdown" if @shutdown unless quiet
+    begin; @lifeline.cut; rescue Exception; end
+    @shutdown = true
+  end
+  
+  class LifeLine
+    def cling &b
+      on_cut{ b.call if b }.join
+    end
+  end
+end

data/lib/sqew.rb

@@ -0,0 +1,95 @@
+require "slave"
+require File.expand_path("../ext/slave", File.dirname(__FILE__))
+require File.expand_path("../ext/qu", File.dirname(__FILE__))
+
+require "leveldb"
+require "sinatra/base"
+require "thin"
+require "multi_json"
+
+require "sqew/version"
+require "sqew/worker"
+require "sqew/manager"
+require "sqew/server"
+require "sqew/payload"
+require "sqew/backend/leveldb"
+
+require "forwardable"
+require "net/http"
+
+module Sqew
+  module ClassMethods
+    extend Forwardable
+
+    attr_accessor :server
+
+    def qu
+      Qu
+    end
+
+    def server=(raw)
+      URI.parse(raw) # verify it's parsable
+      @server = raw
+    end
+    
+    def_delegators :qu, :backend, :backend=, :length, :queues, :reserve, :logger, :logger=, :failure, :failure=
+  end
+  extend ClassMethods
+
+  class << self
+    def configure(*args, &block)
+      self.backend = Sqew::Backend::LevelDB.new
+      block.call(self)
+      self.server ||= "http://0.0.0.0:9962"
+      self.db     ||= "/tmp/"
+    end
+
+    def http
+      uri = URI.parse(server)
+      @http ||= Net::HTTP.new(uri.host, uri.port)
+    end
+
+    def push(job, *args)
+      request = Net::HTTP::Post.new("/enqueue")
+      request.body = MultiJson.encode("job" => job.to_s, "args" => args)
+      http.request(request)
+    end
+    alias_method :enqueue, :push
+
+    def ping
+      request = Net::HTTP::Get.new("/ping")
+      http.request(request)
+    end
+
+    def status
+      request = Net::HTTP::Get.new("/status")
+      response = http.request(request)
+      if response.code == "200"
+        MultiJson.decode(http.request(request).body)
+      else
+        raise "Error connecting to server #{response.code}:#{response.body}"
+      end
+    end
+
+    def set_workers(count)
+      request = Net::HTTP::Put.new("/workers")
+      request.body = count.to_s
+      http.request(request)
+    end
+    alias_method :workers=, :set_workers
+
+    def clear(*queues)
+      request = Net::HTTP::Delete.new("/clear")
+      request.body = queues.join(",")
+      http.request(request)
+    end
+
+    def delete(id)
+      request = Net::HTTP::Delete.new("/#{id}")
+      http.request(request)
+    end
+  end
+
+  extend SingleForwardable
+  def_delegators :backend, :failed_jobs, :running_jobs, :queued_jobs, :db, :db=
+end

data/lib/sqew/backend/immediate.rb

@@ -0,0 +1,17 @@
+require "qu/backend/immediate"
+
+module Sqew
+  module Backend
+    class Immediate < Qu::Backend::Immediate
+      def queued_jobs
+        []
+      end
+      alias_method :running_jobs, :queued_jobs
+      alias_method :failed_jobs, :queued_jobs
+    end
+
+    def close
+    end
+  end
+end
+

data/lib/sqew/backend/leveldb.rb

@@ -0,0 +1,104 @@
+module Sqew
+  module Backend
+    class LevelDB < Qu::Backend::Base
+      attr_accessor :db
+
+      def enqueue(payload)
+        id = Time.now.to_f.to_s
+        queue.put(id, MultiJson.encode(klass:payload.klass.to_s, args:payload.args), :sync => true)
+      end
+
+      def length(*)
+        queue.keys.length
+      end
+
+      def clear(*queues)
+        drop_all(queue) if queues.include?("queue") || queues.empty?
+        drop_all(errors) if queues.include?("failed") || queues.empty?
+      end
+
+      def clear_running
+        drop_all(running)
+      end
+
+      def delete(id)
+        if queue.exists?(id)
+          queue.delete(id)
+        elsif errors.exists?(id)
+          errors.delete(id)
+        end
+      end
+      
+      def reserve(_, options = {block:false})
+        loop do
+          if raw = queue.first
+            id, job = raw
+            queue.delete(id, :sync => true)
+            running.put(id, job, :sync => true)
+            return Sqew::Payload.new(MultiJson.decode(job).update(id:id))
+          end
+          
+          if options[:block]
+            sleep 3
+          else
+            break
+          end
+        end
+      end
+
+      def completed(payload)
+        running.delete(payload.id, :sync => true)
+      end
+
+      def failed(payload, error)
+        running.delete(payload.id, :sync => true)
+        errors.put(payload.id, MultiJson.encode("klass" => payload.klass.to_s, "args" => payload.args, "error" => error.message, "backtrace" => error.backtrace.join("\n")), :sync => true)
+      end
+
+      def failed_jobs
+        errors.to_a.map {|k,v| MultiJson.decode(v).update("id" => k) }
+      end
+
+      def running_jobs
+        running.to_a.map {|k,v| MultiJson.decode(v).update("id" => k) }
+      end
+
+      def queued_jobs
+        queue.to_a.map {|k,v| MultiJson.decode(v).update("id" => k) }
+      end
+
+      def release(*)
+      end
+
+      def register_worker(*)
+      end
+
+      def unregister_worker(*)
+      end
+
+      def close
+        queue.close
+        running.close
+        errors.close
+        @queue, @running, @errors = nil
+      end
+
+      private
+      def queue
+        @queue ||= ::LevelDB::DB.new("#{db}/queue.ldb")
+      end
+
+      def running
+        @running ||= ::LevelDB::DB.new("#{db}/running.ldb")
+      end
+
+      def errors
+        @errors ||= ::LevelDB::DB.new("#{db}/errors.ldb")
+      end
+
+      def drop_all(db)
+        db.each {|k,_| db.delete(k) }
+      end
+    end
+  end
+end

data/lib/sqew/manager.rb

@@ -0,0 +1,124 @@
+module Sqew
+  class Manager < Qu::Worker
+    attr_accessor :max_workers
+    
+    def initialize(max_workers = 3)
+      raise "Configure sqew before starting the manager" unless Sqew.server
+      super([])
+      @max_workers = max_workers
+      @uri = URI.parse(Sqew.server)
+      @poll = 1
+      @thin_server = nil
+
+      @group = ThreadGroup.new
+    end
+
+    def pause_workers
+      @paused_workers ||= @max_workers
+      @max_workers = 0
+    end
+
+    def resume_workers
+      if @paused_workers
+        @max_workers = @paused_workers
+      end
+    end
+
+    def max_workers=(count)
+      @paused_workers = nil
+      @max_workers = count
+    end
+
+    def start_server
+      logger.info "Starting server on #{@uri}"
+      Thread.new do
+        Thin::Logging.silent = true
+        @thin_server = Thin::Server.new(@uri.host, @uri.port, Server.new(self), {signals:false})
+        @thin_server.start
+      end
+    end
+    
+    def stop_server
+      @thin_server.stop
+    end
+
+    def handle_signals
+      logger.debug "Worker #{id} registering traps for INT and TERM signals"
+      %W(INT TERM).each do |sig|
+        trap(sig) do
+          logger.info "Worker #{id} received #{sig}, will wait for workers to finish then quit"
+          @exiting = true
+        end
+      end
+    end
+
+    def work_off
+      Qu.backend.clear_running
+      super
+    end
+
+    def start
+      logger.warn "Worker #{id} starting"
+      start_slave
+      handle_signals
+      start_server
+      Qu.backend.clear_running
+      loop do
+        work
+        sleep @poll
+        
+        if @exiting
+          stop_server
+          @group.list.map {|t| t.join }
+          Qu.backend.close
+          stop_slave
+          break
+        end
+      end
+    ensure
+      logger.debug "Worker #{id} done"
+    end
+
+    private
+    def work
+      if available?
+        job = Qu.reserve(self)
+        if job
+          thread = Thread.new do
+            begin
+              logger.debug "Worker #{id}:#{Process.pid} reserved job #{job}"
+              remote_thread = @worker.fork_job(job)
+              success, error = remote_thread.value
+              logger.debug "Worker #{id}:#{Process.pid} completed job #{job}"
+              
+              if success
+                Qu.backend.completed(job)
+              else
+                Qu.failure.create(job, error) if Qu.failure
+                Qu.backend.failed(job, error)
+              end
+            rescue Exception => e
+              logger.error "Thread Error"
+              log_exception(e)
+            end              
+          end
+          @group.add(thread)
+        end
+      end
+    end
+
+    def available?
+      @group.list.size < @max_workers
+    end
+
+    def start_slave
+      trap("INT") { }
+      @worker_server = Slave.new(:threadsafe => true) { Sqew::Worker.new }
+      @worker = @worker_server.object
+    end
+
+    def stop_slave
+      @worker_server.shutdown
+    end
+  end
+end

data/lib/sqew/payload.rb

@@ -0,0 +1,14 @@
+module Sqew
+  class Payload < Qu::Payload
+    def perform_forked(pipe)
+      klass.perform(*args)
+      pipe.write(Marshal.dump([true, nil]))
+    rescue Exception => e
+      pipe.write(Marshal.dump([false, e]))
+      raise e
+      # raise special exception
+    ensure
+      pipe.close
+    end
+  end
+end

data/lib/sqew/rails.rb

@@ -0,0 +1,14 @@
+require "sqew"
+
+Sqew.configure do |c|
+  c.logger = Logger.new(STDOUT)
+  c.logger.level = Logger::INFO
+end
+
+if defined?(Rails)
+  if defined?(Rails::Railtie)
+    require 'sqew/railtie'
+  else
+    Sqew.logger = Rails.logger
+  end
+end

data/lib/sqew/railtie.rb

@@ -0,0 +1,12 @@
+module Sqew
+  class Railtie < Rails::Railtie
+    rake_tasks do
+      load "sqew/tasks.rb"
+    end
+
+    initializer "sqew.logger" do |app|
+      Sqew.logger = Rails.logger
+      config.queue = Sqew if config.respond_to?(:queue)
+    end
+  end
+end

data/lib/sqew/server.rb

@@ -0,0 +1,73 @@
+module Sqew
+  class Server < Sinatra::Base
+
+    def initialize(manager)
+      super
+      @manager = manager
+    end
+
+    def route_missing
+      raise Sinatra::NotFound
+    end
+    
+    set :show_exceptions, false
+
+    not_found do
+      [404, {}, ""]
+    end
+    
+    post "/enqueue" do
+      begin
+        request.body.rewind
+        json = MultiJson.decode(request.body)
+        request.body.rewind
+
+        Qu.enqueue(json["job"], *json["args"])
+        [202, {"Content-Type" => "application/json"}, ""]
+      rescue Exception => e
+        json = MultiJson.encode({"error" => "#{e.message}\n#{e.backtrace}"})
+        [500, {"Content-Type" => "application/json"}, json]
+      end
+    end
+
+    get "/status" do
+      json = MultiJson.encode({
+                                "queued" => Sqew.queued_jobs,
+                                "failed" => Sqew.failed_jobs,
+                                "running" => Sqew.running_jobs,
+                                "workers" => @manager.max_workers
+                              })
+      [200, {"Content-Type" => "application/json"}, json]
+    end
+
+    get "/ping" do
+      [200, {}, ""]
+    end
+
+    put "/workers" do
+      b = request.body.read
+      request.body.rewind
+      case b
+      when "pause"
+        @manager.pause_workers
+      when "resume"
+        @manager.resume_workers
+      else
+        @manager.max_workers = b.to_i
+      end
+      [200, {}, ""]
+    end
+
+    delete "/clear" do
+      b = request.body.read
+      request.body.rewind
+      Qu.clear(*b.split(","))
+      [200, {}, ""]
+    end
+
+    delete "/:id" do
+      Qu.backend.delete(params[:id])
+      [200, {}, ""]
+    end
+  end
+end

data/lib/sqew/tasks.rb

@@ -0,0 +1,13 @@
+namespace :sqew do
+  desc "Start a worker"
+  task :work  => :environment do
+    Sqew::Manager.new.start
+  end
+end
+
+# Convenience tasks compatibility
+task 'jobs:work'   => 'sqew:work'
+task 'resque:work' => 'sqew:work'
+
+# No-op task in case it doesn't already exist
+task :environment

data/lib/sqew/version.rb

@@ -0,0 +1,3 @@
+module Sqew
+  VERSION = "0.1.0"
+end

data/lib/sqew/worker.rb

@@ -0,0 +1,26 @@
+module Sqew
+  class Worker
+    include Qu::Logger
+
+    def fork_job(job)
+      rd, wr = IO.pipe
+      pid = fork do
+        srand
+        rd.close
+        job.perform_forked(wr)
+        # catch special exception and regular/internalXS ones
+        # add logging to regular (internal)
+      end
+
+      Thread.new do
+        begin
+          Process.wait(pid)
+          wr.close
+          Marshal.load(rd.read)
+        ensure
+          rd.close
+        end
+      end
+    end
+  end
+end

data/lib/tasks.rb

@@ -0,0 +1,13 @@
+namespace :sqew do
+  desc "Start a worker"
+  task :work => :environment do
+    Sqew::Manager.new.start
+  end
+end
+
+# Convenience tasks compatibility
+task 'jobs:work'   => 'sqew:work'
+task 'resque:work' => 'sqew:work'
+
+# No-op task in case it doesn't already exist
+task :environment

data/spec/leveldb_spec.rb

@@ -0,0 +1,57 @@
+require "spec_helper"
+
+describe Sqew::Backend::LevelDB do
+  before do
+    @backend = Sqew::Backend::LevelDB.new()
+    @backend.db = DB_PATH
+  end
+
+  it "should report queued jobs" do
+    Timecop.freeze(Time.utc(2012, 5, 2)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 1)) }
+    Timecop.freeze(Time.utc(2012, 5, 3)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 2)) }
+    Timecop.freeze(Time.utc(2012, 5, 1)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 3)) }
+    @backend.queued_jobs.should == [
+                                    {"klass"=>"TestJob", "args"=>3, "id" => "1335830400.0"},
+                                    {"klass"=>"TestJob", "args"=>1, "id" => "1335916800.0"},
+                                    {"klass"=>"TestJob", "args"=>2, "id" => "1336003200.0"}
+                                   ]
+  end
+
+  it "should report the failed jobs" do
+    error1 = double("error1", :message => "some error1", :backtrace => ["backtrace1"])
+    error2 = double("error2", :message => "some error2", :backtrace => ["backtrace2"])
+    error3 = double("error3", :message => "some error3", :backtrace => ["backtrace3"])
+    
+    @backend.failed(Qu::Payload.new(:klass => TestJob, :args => 1, :id => "2"), error2)
+    @backend.failed(Qu::Payload.new(:klass => TestJob, :args => 1, :id => "3"), error3)
+    @backend.failed(Qu::Payload.new(:klass => TestJob, :args => 1, :id => "1"), error1)
+
+    @backend.failed_jobs.should == [
+                                    {"klass"=>"TestJob", "args"=>1, "id" => "1", "error" => "some error1", "backtrace" => "backtrace1"},
+                                    {"klass"=>"TestJob", "args"=>1, "id" => "2", "error" => "some error2", "backtrace" => "backtrace2"},
+                                    {"klass"=>"TestJob", "args"=>1, "id" => "3", "error" => "some error3", "backtrace" => "backtrace3"}
+    ]
+    
+  end
+
+  it "should report running jobs" do
+    Timecop.freeze(Time.utc(2012, 5, 2)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 1)) }
+    Timecop.freeze(Time.utc(2012, 5, 3)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 2)) }
+
+    job = @backend.reserve(nil)
+    @backend.queued_jobs.should == [{"klass"=>"TestJob", "args"=>2, "id" => "1336003200.0"}]
+    @backend.running_jobs.should == [{"klass"=>"TestJob", "args"=>1, "id" => "1335916800.0"}]
+  end
+  
+  it "should allow jobs to be deleted" do
+    error1 = double("error1", :message => "some error1", :backtrace => ["backtrace1"])
+    @backend.failed(Qu::Payload.new(:klass => TestJob, :args => 1, :id => "1"), error1)
+    Timecop.freeze(Time.utc(2012, 5, 2)) { @backend.enqueue(Qu::Payload.new(:klass => TestJob, :args => 1)) }
+
+    @backend.delete("1")
+    @backend.failed_jobs.should == []
+
+    @backend.delete("1335916800.0")
+    @backend.queued_jobs.should == []
+  end
+end

data/spec/manager_spec.rb

@@ -0,0 +1,35 @@
+require "spec_helper"
+
+describe Sqew::Manager do
+  it "allows the server to be started and stopped" do
+    manager = Sqew::Manager.new(9962)
+    manager.start_server
+    sleep 1
+    response = Net::HTTP.get_response(URI.parse("http://0.0.0.0:9962/ping"))
+    response.code.should == "200"
+
+    manager.stop_server
+    sleep 1
+    expect { Net::HTTP.get_response(URI.parse("http://0.0.0.0:9962/ping")) }.to raise_error(SystemCallError) 
+  end
+
+  it "allows queues to be cleared" do
+    Qu.enqueue(FailJob, -1)
+    manager = Sqew::Manager.new
+    manager.work_off
+    Qu.enqueue(TestJob, 5)
+    Qu.clear
+    Sqew.queued_jobs.should == []
+    Sqew.failed_jobs.should == []
+  end
+
+  it "allows queues to be cleared by name" do
+    Qu.enqueue(FailJob, -1)
+    manager = Sqew::Manager.new
+    manager.work_off
+    Qu.enqueue(TestJob, 5)
+    Qu.clear("failed")
+    Sqew.queued_jobs.should_not == []
+    Sqew.failed_jobs.should == []
+  end
+end

data/spec/server_spec.rb

@@ -0,0 +1,71 @@
+require "spec_helper"
+
+describe Sqew::Server do
+  before do
+    @old_backend = Sqew.backend
+    Sqew.backend = Sqew::Backend::Immediate.new
+    
+    @manager = Sqew::Manager.new(3)
+    Artifice.activate_with(Sqew::Server.new(@manager))
+  end
+
+  after do
+    Artifice.deactivate
+    Sqew.backend = @old_backend
+  end
+  
+  it "enqueues jobs" do
+    TestJob.testing.should == 0
+    Sqew.push("TestJob", 15)
+    TestJob.testing.should == 15
+  end
+
+  it "allows pings" do
+    response = Sqew.ping
+    response.code.should == "200"
+  end
+
+  it "returns the status" do
+    status = Sqew.status
+    status.should == {"queued" => [], "running" => [], "failed" => [], "workers" => 3}
+  end
+
+  it "allows workers to be configured" do
+    response = Sqew.set_workers(5)
+    response.code.should == "200"
+    @manager.max_workers.should == 5
+
+    Sqew.workers = 2
+    @manager.max_workers.should == 2
+  end
+
+  it "allows workers to be paused" do
+    Sqew.workers = :pause
+    @manager.max_workers.should == 0
+    
+    Sqew.workers = :resume
+    @manager.max_workers.should == 3
+  end
+
+  it "should remember the number of workers, even if paused multiple times" do
+    Sqew.workers = :pause
+    Sqew.workers = :pause
+    Sqew.workers = :resume
+    @manager.max_workers.should == 3
+  end
+
+  it "should allow queues to be cleared" do
+    Qu.should_receive(:clear).with("queued", "failed")
+    Sqew.clear("queued", "failed")
+  end
+  
+  it "should allow all queues to be cleared" do
+    Qu.should_receive(:clear).with()
+    Sqew.clear
+  end
+
+  it "should allow jobs to be deleted" do
+    Qu.backend.should_receive(:delete).with("10")
+    Sqew.delete("10")
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,36 @@
+require "rspec"
+require "sqew"
+require "fileutils"
+require "timeout"
+require "artifice"
+require "timecop"
+require "sqew/backend/immediate"
+
+Dir[File.expand_path("support/*.rb", File.dirname(__FILE__))].each {|r| require r}
+
+DB_PATH = File.expand_path("./tmp/db", File.dirname(__FILE__))
+
+Sqew.configure do |c|
+  c.db = DB_PATH
+  c.logger.level = Logger::UNKNOWN
+end
+
+RSpec.configure do |c|
+  c.debug = true
+
+  c.before do
+    Sqew.backend.close
+    FileUtils.rm_rf(DB_PATH)
+    FileUtils.mkdir_p(DB_PATH)
+    TestJob.reset
+    Timecop.return
+  end
+end
+
+def save_and_open_page(response)
+  filename = File.expand_path("tmp/#{Digest::SHA1.hexdigest(Time.now.to_f.to_s)}.html", File.dirname(__FILE__)) 
+  File.open(filename, "w") do |file|
+    file << response.body
+  end
+  `open #{filename}`
+end

data/spec/sqew_spec.rb

@@ -0,0 +1,41 @@
+require "spec_helper"
+
+describe Sqew do
+  it "allows jobs to be enqueued and worked" do
+    TestJob.testing.should == 0
+    Qu.enqueue(TestJob, 20)
+    manager = Sqew::Manager.new
+    manager.work_off
+    TestJob.testing.should == 20
+    Sqew.running_jobs.should == []
+  end
+
+  it "reports the size of the queue" do
+    Sqew.length.should == 0
+    Qu.enqueue(TestJob, 20)
+    Sqew.length.should == 1
+  end
+
+  it "provides failed jobs" do
+    Qu.enqueue(FailJob, -1)
+    manager = Sqew::Manager.new
+    manager.work_off
+    failed = Sqew.failed_jobs
+    failed.size.should == 1
+    failed[0]["klass"].should == "FailJob"
+  end
+
+  it "provides running jobs" do
+    Qu.enqueue(SlowJob, 10)
+    manager = Sqew::Manager.new
+    begin
+      thread = Thread.new { manager.work_off }
+      sleep 1
+      running = Sqew.running_jobs
+      running.size.should == 1
+      running[0]["klass"].should == "SlowJob"
+    ensure
+      thread.terminate
+    end
+  end
+end

data/spec/support/jobs.rb

@@ -0,0 +1,29 @@
+class TestJob
+  @testing = 0
+
+  class << self
+    def perform(arg)
+      @testing = arg
+    end
+
+    def testing
+      @testing
+    end
+
+    def reset
+      @testing = 0
+    end
+  end
+end
+
+class FailJob
+  def self.perform(*)
+    raise "failed in FailJob"
+  end
+end
+
+class SlowJob
+  def self.perform(secs)
+    sleep secs
+  end
+end

data/sqew.gemspec

@@ -0,0 +1,32 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/sqew/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.authors       = ["Zach Moazeni"]
+  gem.email         = ["zach.moazeni@gmail.com"]
+  gem.description   = "a lightweight background processor"
+  gem.summary       = "sqew is a lightweight background processor. You start a single process that will act as a queue manager and will work multiple jobs concurrently.
+
+sqew is not meant to be an all encompassing scalable solution. If you need more management over worker processes or need to split it among multiple machines, it's recommend to use other background processors such as resque, sidekiq, and qu"
+  
+  gem.homepage      = "https://github.com/zmoazeni/sqew"
+
+  gem.files         = `git ls-files`.split($\)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = "sqew"
+  gem.require_paths = ["lib"]
+  gem.version       = Sqew::VERSION
+
+  gem.add_dependency "leveldb-ruby", "~> 0.14"
+  gem.add_dependency "qu", "~> 0.1"
+  gem.add_dependency "multi_json", "~> 1.0"
+  gem.add_dependency "sinatra", "~> 1.0"
+  gem.add_dependency "thin", "~> 1.0"
+  gem.add_dependency "slave", "1.3"
+
+  gem.add_development_dependency "rspec"
+  gem.add_development_dependency "debugger"
+  gem.add_development_dependency "artifice"
+  gem.add_development_dependency "timecop"
+end

metadata

@@ -0,0 +1,243 @@
+--- !ruby/object:Gem::Specification
+name: sqew
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Zach Moazeni
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-05-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: leveldb-ruby
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.14'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.14'
+- !ruby/object:Gem::Dependency
+  name: qu
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: multi_json
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: sinatra
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: thin
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: slave
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: debugger
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: artifice
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: a lightweight background processor
+email:
+- zach.moazeni@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- ext/qu.rb
+- ext/slave.rb
+- lib/sqew.rb
+- lib/sqew/backend/immediate.rb
+- lib/sqew/backend/leveldb.rb
+- lib/sqew/manager.rb
+- lib/sqew/payload.rb
+- lib/sqew/rails.rb
+- lib/sqew/railtie.rb
+- lib/sqew/server.rb
+- lib/sqew/tasks.rb
+- lib/sqew/version.rb
+- lib/sqew/worker.rb
+- lib/tasks.rb
+- spec/leveldb_spec.rb
+- spec/manager_spec.rb
+- spec/server_spec.rb
+- spec/spec_helper.rb
+- spec/sqew_spec.rb
+- spec/support/jobs.rb
+- spec/tmp/.gitkeep
+- sqew.gemspec
+homepage: https://github.com/zmoazeni/sqew
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.21
+signing_key: 
+specification_version: 3
+summary: sqew is a lightweight background processor. You start a single process that
+  will act as a queue manager and will work multiple jobs concurrently.  sqew is not
+  meant to be an all encompassing scalable solution. If you need more management over
+  worker processes or need to split it among multiple machines, it's recommend to
+  use other background processors such as resque, sidekiq, and qu
+test_files:
+- spec/leveldb_spec.rb
+- spec/manager_spec.rb
+- spec/server_spec.rb
+- spec/spec_helper.rb
+- spec/sqew_spec.rb
+- spec/support/jobs.rb
+- spec/tmp/.gitkeep