localjob 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f1b3b73b7b6d29292e899fbc6e3dd8ec230bfd8c
-  data.tar.gz: 18b94b58722de37178649a63ec0ce04e269295a2
+  metadata.gz: 0d66cdf9e9ef8c26230cc76e00a40964249c62db
+  data.tar.gz: cdb1b11bacb52f1d9cdc75dec508b4fbe61ee743
 SHA512:
-  metadata.gz: 1fed832f36a4f04b3f4ce3c606090c88868d5c9d3556e5db0276033bc2819be697c947f5454d750c8aab59f91b6c039af809feffb9383c98533a29a3dc6dd91c
-  data.tar.gz: 62861fd26776063bedd58f77593c15ec75ff76ed818f87b703493a8d5dd0c51f3e9991a22c663f609df380e91ced6598a892d780c95c753fce052900f1c35f62
+  metadata.gz: 9fbe02975d57dd69a328050f4f04a1f3e07312714f6bd5ac8a6fdcbddf02ba55905a11a4a82e677a49524e2b1552f0da0a16d0090f9fc2adf433d8a547f45c5d
+  data.tar.gz: 201974c4584b2c9c3c549ef07be74569b85500cdf7a307d1efd8d8079de9c63a79830a59a8c85a1bb3aa4f86264c23db1d6e5135311824a276401d839198e426

data/README.md

@@ -1,28 +1,22 @@
 # Localjob
 
-Localjob is a simple, self-contained background queue built on top of [POSIX
-message queues][pmq]. Workers and the app pushing to the queue must reside on
-the same machine. It's the sqlite of background queues. Here's a post about [how
-it works][blog].
+Localjob is a simple, self-contained background queue built on top of [System V
+message queues][sysv] (SysV Message Queue => SysV MQ for short). Workers and the
+app pushing to the queue must reside on the same machine. It's the sqlite of
+background queues. Here's a post about [how it works][blog].
 
 Localjob is for early-development situations where you don't need a
 full-featured background queue, but just want to get started with something
-simple that does not rely on any external services. A bigger goal with the
-project is to be able to migrate to another background queue system by switching
-adapter: `Localjob.adapter = Resque` to switch to Resque, without changes to
-your own code.
+simple that does not rely on any external services. The advantage of the SysV
+queues is that your Rails app or worker can restart at any time, without loosing
+any events.
 
-The POSIX message queue is persistent till reboot. You **will need to tune system
-parameters for your application**, please consult [posix-mqueue][pmq-gem]'s
-documentation.
-
-Localjob works on Ruby >= 2.0.0. On Linux, it will use the POSIX message queue.
-On OS X it will use SysV message queues.
+Localjob works on Ruby >= 2.0.0 on Linux and OS X.
 
 Add it to your Gemfile:
 
 ```ruby
-gem 'localjob', "0.1.0"
+gem 'localjob', "~> 0.2"
 ```
 
 ## Usage
@@ -48,10 +42,11 @@ queue = Localjob.new
 queue << EmailJob.new(current_user.id, welcome_email)
 ```
 
-A job is serialized with YAML and pushed onto a persistent POSIX message queue.
+A job is serialized with YAML and pushed onto a persistent SysV Message Queue.
 This means a worker does not have to listen on the queue to push things to it.
-Workers will pop off the message queue, but only one will receive the job.
-Deserialize the message to create an instance of your object, and call
+Pops off the message queue are atomic, so only one will receive the queue. This
+means you can run multiple workers on the same machine if you wish. The workers
+will deserialize the message to create an instance of your object, and call
 `#perform` on the object.
 
 ### Rails initializer
@@ -61,7 +56,7 @@ constant referencing each of your queues. This allows easy access anywhere in
 your app. In `config/initializers/localjob.rb`:
 
 ```ruby
-BackgroundQueue = Localjob.new("main-queue")
+BackgroundQueue = Localjob.new
 ```
 
 Then in your app you can simply reference the constant to push to the queue:
@@ -83,19 +78,6 @@ Gracefully shut down workers by sending `SIGQUIT` to them. This will make sure
 the worker completes its current job before shutting down. Jobs can be sent to
 the queue meanwhile, and the worker will process them once it starts again.
 
-### Queues
-
-Localjobs supports multiple queues, and workers can be assigned to queues. By
-default everything is on a single queue. To push to a named queue:
-
-```ruby
-email = Localjob.new("email")
-email << EmailJob.new(current_user.id, welcome_email)
-```
-
-The worker spawn command `localjob work` takes a `--queues` argument which is a
-comma seperated list of queues to listen on, e.g. `localjob work --queues email,webhooks`.
-
 ### Testing
 
 Create your instance of the queue as normal in your setup:
@@ -103,7 +85,7 @@ Create your instance of the queue as normal in your setup:
 ```ruby
 def setup
   @queue  = Localjob.new("test-queue")
-  @worker = Localjob::Worker.new("test-queue") 
+  @worker = Localjob::Worker.new("test-queue")
 end
 ```
 
@@ -132,6 +114,5 @@ def test_pop_and_send_to_worker
 end
 ```
 
-[pmq]: http://linux.die.net/man/7/mq_overview
-[pmq-gem]: https://github.com/Sirupsen/posix-mqueue
+[sysv]: http://man7.org/linux/man-pages/man7/svipc.7.html
 [blog]: http://sirupsen.com/unix-background-queue/

data/lib/localjob/sysv_adapter.rb

@@ -1,47 +1,34 @@
-require 'SysVIPC'
+require 'sysvmq'
 
 class Localjob
-  class Channel
-    def shift
-      raise "SysV adapter does not support multiple queues" if @queues.size > 1
-      @queues.first.shift
-    end
-  end
-
   class SysvAdapter
-    include SysVIPC
+    RECEIVE_ALL_TYPES = 0
 
-    def initialize(name)
-      @filename = "/tmp/#{name}"
-    end
-    
-    def mq
-      unless @mq
-        File.open(@filename, "w") { }
-        key = ftok(@filename, 0)
-  
-        @mq = MessageQueue.new(key, IPC_CREAT | 0600)
-      end
-      
-      @mq
+    attr_reader :queue
+
+    def initialize(key, size: 8192, flags: SysVMQ::IPC_CREAT | 0660)
+      @key = key
+      @queue = SysVMQ.new(key, size, flags)
     end
 
     def receive
-      mq.receive(0, 8024)
+      queue.receive(RECEIVE_ALL_TYPES)
     end
 
     def send(message)
-      mq.send(1, message)
+      queue.send(message, 1)
     end
 
     def size
-      mq.ipc_stat.msg_qnum
+      queue.stats[:count]
+    end
+
+    def stats
+      queue.stats
     end
 
     def destroy
-      File.delete(@filename)
-      @mq.rm
-      @mq = nil
+      queue.destroy
     end
   end
 end

data/lib/localjob/version.rb

@@ -1,3 +1,3 @@
 class Localjob
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

data/lib/localjob/worker.rb

@@ -1,10 +1,12 @@
 class Localjob
   class Worker
-    attr_accessor :logger, :channel
-    attr_reader :options
+    TERMINATION_MESSAGE = "__TERMINATE__"
 
-    def initialize(queues, logger: Logger.new(STDOUT), **options)
-      @channel, @logger = Channel.new(queues), logger
+    attr_accessor :logger
+    attr_reader :options, :queue
+
+    def initialize(queue, logger: Logger.new(STDOUT), **options)
+      @queue, @logger = queue, logger
       @options = options
       @shutdown = false
     end
@@ -18,35 +20,67 @@ class Localjob
       Process.pid
     end
 
-    def work
+    def work(thread: false)
       logger.info "Worker #{pid} now listening!"
       trap_signals
+
+      return work_thread if thread
+
       create_pid_file(@options[:pid_file])
       deamonize if @options[:deamon]
-      loop { shift_and_process }
+
+      loop { break unless shift_and_process }
+    end
+
+    def kill
+      if @thread
+        Thread.kill(@thread)
+        @thread.join
+      else
+        shutdown
+      end
     end
 
     private
 
-    def shift_and_process
-      exit! if @shutdown
+    def work_thread
+      @thread = Thread.new do
+        begin
+          loop do
+            shift_and_process
+          end
+        # Respond to Thread.kill by sending termination message
+        ensure
+          shutdown
+          work
+        end
+      end
+    end
 
-      job = wait { @channel.shift }
-      process job
-    rescue Object => e
+    def shutdown!
+      logger.info "Worker #{pid} shutting down.."
+      File.rm(@options[:pid_file]) if @options[:pid_file]
+      return false if @thread
+      exit!
+    end
+
+    def shutdown
+      @queue << TERMINATION_MESSAGE
+    end
+
+    def shift_and_process
+      job = queue.shift
+      return shutdown! if job == TERMINATION_MESSAGE || !job
+      process(job)
+      return true # Explicit return of true, job#perform may return nil
+    rescue Object
       logger.error "Worker #{pid} job failed: #{job}"
       logger.error "#{$!}\n#{$@.join("\n")}"
     end
 
     def trap_signals
-      Signal.trap("QUIT") { graceful_shutdown }
-    end
-
-    def wait
-      @waiting = true
-      job = yield
-      @waiting = false
-      job
+      Signal.trap("QUIT") { shutdown }
+      Signal.trap("INT") { shutdown }
     end
 
     def deamonize
@@ -56,10 +90,5 @@ class Localjob
     def create_pid_file(path)
       File.open(path, 'w') { |f| f << self.pid } if path
     end
-
-    def graceful_shutdown
-      exit! if @waiting
-      @shutdown = true
-    end
   end
 end

data/lib/localjob.rb

@@ -3,15 +3,8 @@ require 'logger'
 require 'forwardable'
 
 require "localjob/version"
-require 'localjob/channel'
 require 'localjob/worker'
-
-case RUBY_PLATFORM
-when /linux/
-  require 'localjob/linux_adapter'
-else
-  require 'localjob/sysv_adapter'
-end
+require 'localjob/sysv_adapter'
 
 class Localjob
   extend Forwardable
@@ -21,7 +14,8 @@ class Localjob
 
   def_delegators :queue, :to_io, :destroy, :size
 
-  def initialize(name = "localjob")
+  # LOCALJOB in 1337speak
+  def initialize(name = 0x10CA110B)
     @name = name
   end
 
@@ -30,14 +24,7 @@ class Localjob
   end
 
   def queue
-    return @queue if @queue
-
-    case RUBY_PLATFORM
-    when /linux/
-      @queue = LinuxAdapter.new(@name)
-    else
-      @queue = SysvAdapter.new(@name)
-    end
+    @queue ||= SysvAdapter.new(@name)
   end
 
   def <<(object)
@@ -46,5 +33,8 @@ class Localjob
 
   def shift
     serializer.load queue.receive
+  # Bad serialization
+  rescue Object
+    nil
   end
 end

data/localjob.gemspec

@@ -17,12 +17,12 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
-  spec.extensions    = ["ext/mkrf_conf.rb"]
 
   spec.add_dependency "thor", "0.18.1"
-  spec.add_dependency "SysVIPC"
+  spec.add_dependency "sysvmq", "~> 0.1"
 
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "mocha"
+  spec.add_development_dependency "stackprof"
 end

data/test/localjob_test.rb

@@ -21,18 +21,10 @@ class LocaljobTest < LocaljobTestCase
   def test_handles_multiple_queues
     @localjob << WalrusJob.new("move")
 
-    other = queue("other-queue")
+    other = queue(0xDEADCAFE)
     other << WalrusJob.new("dance")
 
     assert_equal 1, @localjob.size
     assert_equal 1, other.size
   end
-
-  on_platform 'linux' do
-    def test_throws_error_if_message_is_too_large
-      assert_raises Errno::EMSGSIZE do
-        @localjob << AngryWalrusJob.new("f" * @localjob.queue.mqueue.msgsize)
-      end
-    end
-  end
 end

data/test/mock_adapter.rb

@@ -1,4 +1,5 @@
 require 'test_helper'
+require 'localjob/mock_adapter'
 
 class MockAdapterTest < LocaljobTestCase
   def setup

data/test/sysv_adapter_test.rb

@@ -0,0 +1,23 @@
+require 'test_helper'
+
+class SysvAdapterTest < LocaljobTestCase
+  def setup
+    @localjob = queue
+    @localjob.queue = Localjob::SysvAdapter.new(0xDEADC0DE)
+  end
+
+  def teardown
+    @localjob.destroy
+  end
+
+  def test_send_and_receive
+    msg = "Hello World"
+    @localjob << msg
+    assert_equal msg, @localjob.shift
+  end
+
+  def test_size
+    @localjob << "Hello World"
+    assert_equal 1, @localjob.size
+  end
+end

data/test/test_helper.rb

@@ -8,13 +8,13 @@ require 'jobs'
 class LocaljobTestCase < MiniTest::Unit::TestCase
   protected
   # This is a method to make sure the logger is set right.
-  def worker(queues = ["/localjob-test"])
-    Localjob::Worker.new(queues, logger: logger)
+  def worker(queue)
+    Localjob::Worker.new(queue, logger: logger)
   end
 
   # This is a method to make sure all queues are registred and destroyed after
   # each teach run.
-  def queue(name = "/localjob-test")
+  def queue(name = 0x10CA110B)
     @queues ||= []
     queue = Localjob.new(name)
     @queues << queue
@@ -22,7 +22,7 @@ class LocaljobTestCase < MiniTest::Unit::TestCase
   end
 
   def teardown
-    clear_queue
+    clear_queues
   end
 
   def logger
@@ -32,19 +32,7 @@ class LocaljobTestCase < MiniTest::Unit::TestCase
     @logger = Logger.new(output_file)
   end
 
-  def clear_queue
+  def clear_queues
     @queues.each(&:destroy) if @queues
-
-    # This forces the GC to garbage collect, and thus close file descriptioners
-    # in POSIX::Mqueue. Otherwise we'll get flooded with warnings. This is to
-    # ensure a clean state everytime with a new message queue for each test.
-    # It's slower. But safe.
-    GC.start
-  end
-
-  def self.on_platform(platform)
-    if RUBY_PLATFORM =~ /#{platform}/
-      yield
-    end
   end
 end

data/test/worker_test.rb

@@ -3,7 +3,7 @@ require 'test_helper'
 class WorkerTest < LocaljobTestCase
   def setup
     @localjob = queue
-    @worker   = worker
+    @worker   = worker(@localjob)
   end
 
   def test_pop_and_send_to_worker
@@ -33,58 +33,53 @@ class WorkerTest < LocaljobTestCase
     @localjob << AngryWalrusJob.new(100)
     @localjob << WalrusJob.new("be happy")
 
-    a = Thread.start { @worker.work }
+    @worker.work(thread: true)
 
-    # Hack to account for race condition, 0.01s should be plenty
-    sleep 0.01
-    a.kill
+    sleep 0.1
+    @worker.kill
 
     assert_equal 0, @localjob.size
   end
 
   def test_worker_doesnt_die_on_bad_serialization
-    @localjob.queue.send "--- !ruby/object:Whatever {}\n"
+    @localjob << "--- !ruby/object:Whatever {}\n"
 
-    a = Thread.start { @worker.work }
-
-    sleep 0.01
-    a.kill
+    @worker.work(thread: true)
+    sleep 0.1
+    @worker.kill
   end
 
-  on_platform 'linux' do
-    # This won't work on OS X because the SysV IPC gem apparently doesnt give us
-    # interrupt syscalls to give us signals.
-    def test_sigquit_terminates_the_worker
-      @localjob << WalrusJob.new("move")
+  def test_sigquit_terminates_the_worker
+    @localjob << WalrusJob.new("move")
 
-      assert_equal 1, @localjob.size
+    assert_equal 1, @localjob.size
 
-      pid = fork { @worker.work }
+    pid = fork { @worker.work }
+    sleep 0.1
 
-      # Hack to account for race condition, 0.01s should be plenty
-      sleep 0.1
+    Process.kill("QUIT", pid)
 
-      Process.kill("QUIT", pid)
-      Process.wait
+    assert_equal 0, @localjob.size
+  end
 
-      assert_equal 0, @localjob.size
-    end
+  def test_sigquit_terminates_the_worker
+    @localjob << WalrusJob.new("move")
 
-    def test_workers_listen_on_multiple_queues
-      @localjob << WalrusJob.new("move")
+    pid = fork { @worker.work }
+    sleep 0.1
 
-      other = queue("other-queue")
-      other << WalrusJob.new("dance")
+    Process.kill("INT", pid)
+  end
 
-      @worker.channel << 'other-queue'
+  def test_thread_worker
+    @localjob << WalrusJob.new("move")
 
-      a = Thread.start { @worker.work }
+    assert_equal 1, @localjob.size
 
-      sleep 0.01
-      a.kill
+    @worker.work(thread: true)
+    sleep 0.1
+    @worker.kill
 
-      assert_equal 0, @localjob.size
-      assert_equal 0, other.size
-    end
+    assert_equal 0, @localjob.size
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: localjob
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Simon Eskildsen
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-02 00:00:00.000000000 Z
+date: 2014-02-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -25,59 +25,73 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.18.1
 - !ruby/object:Gem::Dependency
-  name: SysVIPC
+  name: sysvmq
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.1'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.3'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: mocha
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: stackprof
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: Simple, self-contained background queue built on top of POSIX message
@@ -86,21 +100,17 @@ email:
 - sirup@sirupsen.com
 executables:
 - localjob
-extensions:
-- ext/mkrf_conf.rb
+extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
+- ".gitignore"
 - Gemfile
 - LICENSE.txt
 - README.md
 - Rakefile
 - bin/localjob
-- ext/mkrf_conf.rb
 - lib/localjob.rb
-- lib/localjob/channel.rb
 - lib/localjob/cli.rb
-- lib/localjob/linux_adapter.rb
 - lib/localjob/mock_adapter.rb
 - lib/localjob/sysv_adapter.rb
 - lib/localjob/version.rb
@@ -109,6 +119,7 @@ files:
 - test/jobs.rb
 - test/localjob_test.rb
 - test/mock_adapter.rb
+- test/sysv_adapter_test.rb
 - test/test_helper.rb
 - test/worker_test.rb
 homepage: ''
@@ -121,17 +132,17 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.0.3
+rubygems_version: 2.2.0
 signing_key: 
 specification_version: 4
 summary: Simple, self-contained background queue built on top of POSIX message queues.
@@ -140,5 +151,6 @@ test_files:
 - test/jobs.rb
 - test/localjob_test.rb
 - test/mock_adapter.rb
+- test/sysv_adapter_test.rb
 - test/test_helper.rb
 - test/worker_test.rb

data/ext/mkrf_conf.rb

@@ -1,4 +0,0 @@
-require 'rubygems/dependency_installer' 
-
-gem = Gem::DependencyInstaller.new
-gem.install "posix-mqueue", "0.0.7" if RUBY_PLATFORM =~ /linux/

data/lib/localjob/channel.rb

@@ -1,18 +0,0 @@
-class Localjob
-  class Channel
-    attr_accessor :queues
-
-    def initialize(queues)
-      @queues = [queues].flatten.map { |q| queue_from_name(q) }
-    end
-
-    def <<(queue)
-      @queues << queue_from_name(queue)
-    end
-
-    private
-    def queue_from_name(queue)
-      queue.instance_of?(Localjob) ? queue : Localjob.new(queue)
-    end
-  end
-end

data/lib/localjob/linux_adapter.rb

@@ -1,45 +0,0 @@
-require 'posix/mqueue'
-
-class Localjob
-  class Channel
-    def shift
-      (queue,), = IO.select(@queues)
-      queue.shift
-    rescue POSIX::Mqueue::QueueEmpty
-      retry
-    end
-  end
-
-  class LinuxAdapter
-    attr_reader :mqueue
-
-    def initialize(name)
-      @mqueue = POSIX::Mqueue.new(fix_queue_name(name))
-    end
-
-    def receive
-      @mqueue.timedreceive
-    end
-
-    def send(message)
-      @mqueue.timedsend message
-    end
-
-    def size
-      @mqueue.size
-    end
-
-    def destroy
-      @mqueue.unlink
-    end
-
-    def to_io
-      @mqueue.to_io
-    end
-
-    private
-    def fix_queue_name(name)
-      name.start_with?('/') ? name : "/#{name}"
-    end
-  end
-end