localjob 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: becc6c53c0de0692e654c0e4fd8b220b13baa424
-  data.tar.gz: 770a92796d0fedf97c3d66728e039b3d966e7f92
+  metadata.gz: 6c33dc84596d653899831a3697428d950028e0c5
+  data.tar.gz: 8547e6ca4022b618956578e8dc3f11fe65ae8b00
 SHA512:
-  metadata.gz: f56f0b1ae32cda8650cd0e6375278ad34bf16c00db257a180d1070c980c7eee422f7e604afcbbe013c7da7c0ae403d2f0a7c795987fd52330f101fd19f3a985e
-  data.tar.gz: 3ecf92a571bd824625ea348b5f0ccdf8e4f5b7e3ba2fc3ab468a1915c90a768f5a250803d71a8a1caf41387c21d237320f605c250e6edcf4bdbcc035d4b1c90f
+  metadata.gz: 22eab445b06c1011c53a244a2677feec403a4f501e0dc35ca663b405d4d754f3f891914420fb811a5a840076f1fbd6af71f7afc31d5f1e9c76a297730fbf281c
+  data.tar.gz: d7ede95238c734c9ab043b2861152203197fb7606a86ee6b81083ce304c9f479cbaf50842515301518f6566733b7978a1b77dff828827df476e9dd5505191199

data/README.md

@@ -12,18 +12,18 @@ project is to be able to migrate to another background queue system by switching
 adapter: `Localjob.adapter = Resque` to switch to Resque, without changes to
 your own code.
 
-The POSIX message queue is persistent till reboot. You will need to tune system
-parameters for your application, please consult [posix-mqueue][pmq-gem]'s
+The POSIX message queue is persistent till reboot. You **will need to tune system
+parameters for your application**, please consult [posix-mqueue][pmq-gem]'s
 documentation.
 
-Localjob works on Ruby >= 2.0.0 and Linux.
+Localjob works on Ruby >= 2.0.0. On Linux, it will use the POSIX message queue.
+On OS X (and Windows, not tested) it will use a mock class instead of the
+message queue, to aid you in testing and running Localjob in development.
 
-WIP not everything works as advertised, but try it out and report anything that
-doesn't work! It's not released as a gem yet, so add the following to your
-Gemfile to use it:
+Add it to your Gemfile:
 
 ```ruby
-gem 'localjob', git: "git@github.com:Sirupsen/localjob.git"
+gem 'localjob'
 ```
 
 ## Usage
@@ -49,8 +49,88 @@ queue = Localjob.new
 queue << EmailJob.new(current_user.id, welcome_email)
 ```
 
-Then spawn a worker with `localjob work`. It takes a few arguments, like `-d` to
-deamonize itself. `localjob help work` to list all options.
+A job is serialized with YAML and pushed onto a persistent POSIX message queue.
+This means a worker does not have to listen on the queue to push things to it.
+Workers will pop off the message queue, but only one will receive the job.
+Deserialize the message to create an instance of your object, and call
+`#perform` on the object.
+
+### Rails initializer
+
+For easy access to your queues in Rails, you can add an initializer to set up a
+constant referencing each of your queues. This allows easy access anywhere in
+your app. In `config/initializers/localjob.rb`:
+
+```ruby
+BackgroundQueue = Localjob.new("main-queue")
+```
+
+Then in your app you can simply reference the constant to push to the queue:
+
+```ruby
+BackgroundQueue << EmailJob.new(current_user.id, welcome_email)
+```
+
+### Managing workers
+
+Spawning workers can be done with `localjob`. Run `localjob work` to spawn a
+single worker. It takes a few arguments. The most important being `--require`
+which takes a path the worker will require before processing jobs. For Rails,
+you can run `localjob work` without any arguments. `localjob(2)` has a few other
+commands such as `list` to list all queues and `size` to list the size of all
+queues. `localjob help` to list all commands.
+
+Gracefully shut down workers by sending `SIGQUIT` to them. This will make sure
+the worker completes its current job before shutting down. Jobs can be sent to
+the queue meanwhile, and the worker will process them once it starts again.
+
+### Queues
+
+Localjobs supports multiple queues, and workers can be assigned to queues. By
+default everything is on a single queue. To push to a named queue:
+
+```ruby
+email = Localjob.new("email")
+email << EmailJob.new(current_user.id, welcome_email)
+```
+
+The worker spawn command `localjob work` takes a `--queues` argument which is a
+comma seperated list of queues to listen on, e.g. `localjob work --queues email,webhooks`.
+
+### Testing
+
+Create your instance of the queue as normal in your setup:
+
+```ruby
+def setup
+  @queue = LocalJob.new("test-queue")
+end
+```
+
+In your `teardown` you'll want to destroy your queue:
+
+```ruby
+def teardown
+  @queue.destroy
+end
+```
+
+You can get the size of your queue by calling `@queue.size`. You pop off the
+queue with `@queue.shift`. Other than that, just use the normal API. You can
+also read the tests for Localjob to get an idea of how to test. Sample test:
+
+```ruby
+def test_pop_and_send_to_worker
+  WalrusJob.any_instance.expects(:perform)
+
+  @localjob << WalrusJob.new("move")
+
+  job = @localjob.shift
+  @worker.process(job)
+
+  assert_equal 0, @localjob.size
+end
+```
 
 [pmq]: http://linux.die.net/man/7/mq_overview
 [pmq-gem]: https://github.com/Sirupsen/posix-mqueue

data/ext/mkrf_conf.rb

@@ -0,0 +1,17 @@
+require 'rubygems'
+require 'rubygems/command.rb'
+require 'rubygems/dependency_installer.rb' 
+begin
+  Gem::Command.build_args = ARGV
+rescue NoMethodError
+end 
+
+inst = Gem::DependencyInstaller.new
+
+begin
+  if RUBY_PLATFORM =~ /linux/
+    inst.install "posix-mqueue", "0.0.7"
+  end
+rescue
+  exit(1)
+end 

data/lib/localjob/channel.rb

@@ -11,10 +11,12 @@ class Localjob
     end
 
     def shift
-      (queue,), = IO.select(@queues)
-      queue.shift
-    rescue POSIX::Mqueue::QueueEmpty
-      retry
+      begin
+        (queue,), = IO.select(@queues)
+        queue.shift
+      rescue POSIX::Mqueue::QueueEmpty
+        retry
+      end
     end
 
     private

data/lib/localjob/linux_adapter.rb

@@ -0,0 +1,34 @@
+class Localjob
+  class LinuxAdapter
+    attr_reader :mqueue
+
+    def initialize(name)
+      @mqueue = POSIX::Mqueue.new(fix_queue_name(name))
+    end
+
+    def receive
+      @mqueue.timedreceive
+    end
+
+    def send(message)
+      @mqueue.timedsend message
+    end
+
+    def size
+      @mqueue.size
+    end
+
+    def destroy
+      @mqueue.unlink
+    end
+
+    def to_io
+      @mqueue.to_io
+    end
+
+    private
+    def fix_queue_name(name)
+      name.start_with?('/') ? name : "/#{name}"
+    end
+  end
+end

data/lib/localjob/mock_adapter.rb

@@ -0,0 +1,32 @@
+class Localjob
+  class Channel
+    def shift
+      queue = @queues.find { |q| q.size > 0 }
+      return queue.shift
+    end
+  end
+
+  class MockAdapter
+    def initialize(name = 'default')
+      @@queues ||= {}
+      @name = name 
+      @@queues[@name] ||= []
+    end
+
+    def receive
+      @@queues[@name].shift
+    end
+
+    def send(message)
+      @@queues[@name] << message
+    end
+
+    def size
+      @@queues[@name].size
+    end
+
+    def destroy
+      @@queues[@name] = nil
+    end
+  end
+end

data/lib/localjob/version.rb

@@ -1,3 +1,3 @@
 class Localjob
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/lib/localjob.rb

@@ -1,16 +1,26 @@
-require 'posix/mqueue'
+begin
+  require 'posix/mqueue'
+rescue LoadError
+end
+
 require 'yaml'
 require 'logger'
+require 'forwardable'
 
 require "localjob/version"
 require 'localjob/channel'
 require 'localjob/worker'
 
 class Localjob
-  attr_reader :queue_name
+  extend Forwardable
+
+  attr_reader :name
+  attr_accessor :queue
+
+  def_delegators :queue, :to_io, :destroy, :size
 
-  def initialize(queue = "localjob")
-    @queue_name = fix_queue_name(queue)
+  def initialize(name = "localjob")
+    @name = name
   end
 
   def serializer
@@ -18,31 +28,23 @@ class Localjob
   end
 
   def queue
-    @queue ||= POSIX::Mqueue.new(@queue_name)
-  end
+    return @queue if @queue
 
-  def <<(object)
-    queue.timedsend serializer.dump(object)
+    case RUBY_PLATFORM
+    when /linux/
+      require 'localjob/linux_adapter'
+      @queue = LinuxAdapter.new(@name)
+    else
+      require 'localjob/mock_adapter'
+      @queue = MockAdapter.new(@name)
+    end
   end
 
-  def size
-    queue.size
+  def <<(object)
+    queue.send serializer.dump(object)
   end
 
   def shift
-    serializer.load queue.timedreceive
-  end
-
-  def destroy
-    queue.unlink
-  end
-
-  def to_io
-    queue.to_io
-  end
-
-  private
-  def fix_queue_name(queue)
-    queue.start_with?('/') ? queue : "/#{queue}"
+    serializer.load queue.receive
   end
 end

data/localjob.gemspec

@@ -17,8 +17,8 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
+  spec.extensions    = ["ext/mkrf_conf.rb"]
 
-  spec.add_dependency "posix-mqueue", "0.0.9"
   spec.add_dependency "thor", "0.18.1"
 
   spec.add_development_dependency "bundler", "~> 1.3"

data/test/localjob_test.rb

@@ -18,12 +18,6 @@ class LocaljobTest < LocaljobTestCase
     assert_equal "move", job.action
   end
 
-  def test_throws_error_if_message_is_too_large
-    assert_raises Errno::EMSGSIZE do
-      @localjob << AngryWalrusJob.new("f" * @localjob.queue.msgsize)
-    end
-  end
-
   def test_handles_multiple_queues
     @localjob << WalrusJob.new("move")
 
@@ -33,4 +27,12 @@ class LocaljobTest < LocaljobTestCase
     assert_equal 1, @localjob.size
     assert_equal 1, other.size
   end
+
+  on_platform 'linux' do
+    def test_throws_error_if_message_is_too_large
+      assert_raises Errno::EMSGSIZE do
+        @localjob << AngryWalrusJob.new("f" * @localjob.queue.mqueue.msgsize)
+      end
+    end
+  end
 end

data/test/mock_adapter.rb

@@ -0,0 +1,22 @@
+require 'test_helper'
+
+class MockAdapterTest < LocaljobTestCase
+  def setup
+    @localjob = queue
+    @localjob.queue = Localjob::MockAdapter.new("localjob")
+  end
+
+  def test_push_to_queue
+    @localjob << "hello world"
+    assert_equal 1, @localjob.size
+  end
+
+  def test_push_and_pop_from_queue
+    @localjob << "hello world"
+    assert_equal "hello world", @localjob.shift
+  end
+
+  def test_destroy_queue
+    @localjob << "hello world"
+  end
+end

data/test/test_helper.rb

@@ -41,4 +41,10 @@ class LocaljobTestCase < MiniTest::Unit::TestCase
     # It's slower. But safe.
     GC.start
   end
+
+  def self.on_platform(platform)
+    if RUBY_PLATFORM =~ /#{platform}/
+      yield
+    end
+  end
 end

data/test/worker_test.rb

@@ -18,38 +18,26 @@ class WorkerTest < LocaljobTestCase
   def test_working_off_queue_in_child
     @localjob << WalrusJob.new("move")
 
-    fork do
+    a = Thread.start {
       job = @localjob.shift
       @worker.process(job)
-    end
-
-    Process.wait
-    assert_equal 0, @localjob.size
-  end
-
-  def test_sigquit_terminates_the_worker
-    @localjob << WalrusJob.new("move")
-
-    assert_equal 1, @localjob.size
-
-    pid = fork { @worker.work }
+    }
 
-    Process.kill("QUIT", pid)
-    Process.wait
+    a.join
 
     assert_equal 0, @localjob.size
   end
 
+
   def test_doesnt_stop_on_error
     @localjob << AngryWalrusJob.new(100)
     @localjob << WalrusJob.new("be happy")
 
-    pid = fork { @worker.work }
+    a = Thread.start { @worker.work }
 
     # Hack to account for race condition, 0.01s should be plenty
     sleep 0.01
-    Process.kill("QUIT", pid)
-    Process.wait
+    a.kill
 
     assert_equal 0, @localjob.size
   end
@@ -62,24 +50,36 @@ class WorkerTest < LocaljobTestCase
 
     @worker.channel << 'other-queue'
 
-    pid = fork { @worker.work }
+    a = Thread.start { @worker.work }
 
-    # Hack to account for race condition, 0.01s should be plenty
     sleep 0.01
-    Process.kill("QUIT", pid)
-    Process.wait
+    a.kill
 
     assert_equal 0, @localjob.size
     assert_equal 0, other.size
   end
 
   def test_worker_doesnt_die_on_bad_serialization
-    @localjob.queue.timedsend "--- !ruby/object:Whatever {}\n"
+    @localjob.queue.send "--- !ruby/object:Whatever {}\n"
 
-    pid = fork { @worker.work }
+    a = Thread.start { @worker.work }
 
     sleep 0.01
-    Process.kill("QUIT", pid)
-    Process.wait
+    a.kill
+  end
+ 
+  on_platform 'linux' do
+    def test_sigquit_terminates_the_worker
+      @localjob << WalrusJob.new("move")
+
+      assert_equal 1, @localjob.size
+
+      pid = fork { @worker.work }
+
+      Process.kill("QUIT", pid)
+      Process.wait
+
+      assert_equal 0, @localjob.size
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: localjob
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Simon Eskildsen
@@ -10,20 +10,6 @@ bindir: bin
 cert_chain: []
 date: 2013-09-03 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: posix-mqueue
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 0.0.9
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 0.0.9
 - !ruby/object:Gem::Dependency
   name: thor
   requirement: !ruby/object:Gem::Requirement
@@ -86,7 +72,8 @@ email:
 - sirup@sirupsen.com
 executables:
 - localjob
-extensions: []
+extensions:
+- ext/mkrf_conf.rb
 extra_rdoc_files: []
 files:
 - .gitignore
@@ -95,14 +82,18 @@ files:
 - README.md
 - Rakefile
 - bin/localjob
+- ext/mkrf_conf.rb
 - lib/localjob.rb
 - lib/localjob/channel.rb
 - lib/localjob/cli.rb
+- lib/localjob/linux_adapter.rb
+- lib/localjob/mock_adapter.rb
 - lib/localjob/version.rb
 - lib/localjob/worker.rb
 - localjob.gemspec
 - test/jobs.rb
 - test/localjob_test.rb
+- test/mock_adapter.rb
 - test/test_helper.rb
 - test/worker_test.rb
 homepage: ''
@@ -133,5 +124,6 @@ summary: Simple, self-contained background queue built on top of POSIX message q
 test_files:
 - test/jobs.rb
 - test/localjob_test.rb
+- test/mock_adapter.rb
 - test/test_helper.rb
 - test/worker_test.rb