employer 0.0.1 → 0.1

data/.gitignore

@@ -15,3 +15,4 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+config/

data/.pryrc

@@ -0,0 +1,2 @@
+require "bundler"
+Bundler.require

data/.rspec

@@ -0,0 +1 @@
+--color --format documentation -I. -rspec_helper

data/README.md

@@ -1,6 +1,9 @@
 # Employer
 
-TODO: Write a gem description
+There comes a time in the life of an application that async job processing
+becomes a requirement. If you want something flexible that you can easily adapt
+to fit in with your application's infrastucture, then Employer may be what you
+are looking for.
 
 ## Installation
 
@@ -18,7 +21,135 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+To use Employer to run jobs you need to do the following:
+
+- Define your jobs as classes that include the Employer::Job module
+- Hook up Employer with a backend to manage the jobs
+
+### Defining your own jobs
+
+Implementing your own jobs is simple, here's a silly example:
+
+```ruby
+class NamePutsJob
+  include Employer::Job
+
+  attribute :first_name
+  attribute :last_name
+  attribute :tries
+
+  def initialize
+    tries ||= 0 
+  end
+
+  def try_again?
+    true if tries < 3
+  end
+
+  def perform
+    puts "#{first_name} #{last_name}"
+  end
+end
+```
+
+The attribute class method will define an attr_accessor and will ensure that the
+attribute is part of the serialized data that is sent to the backend when a job
+is enqueued.
+
+The perform method is what will get executed when Employer picks up a job for
+processing.
+
+If a job fails the try_again? method will determine whether or not the job gets
+tried again, if this method returns false the job will be marked as failed and
+won't be attempted again.
+
+### Hooking up a backend
+
+Employer manages its jobs through its pipeline, in order to feed jobs into the
+pipeline and to get jobs out of the pipeline you need to connect a backend to
+it. You can either use a backend that someone has built already (if you're using
+Mongoid 3 you can use the employer-mongoid gem), or implement your own. A valid
+pipeline backend must implement the methods shown in the below code snippet:
+
+```ruby
+class CustomPipelineBackend
+  # job_hash is a Hash with the following keys:
+  # - class: The class of the Job object
+  # - attributes: A Hash with attribute values set on the Job object
+  def enqueue(job_hash)
+  end
+
+  # dequeue must return a job_hash in the same format as is passed into 
+  # enqueue, except that it must add the key id with the Job's unique 
+  # identifier (such as a record id)
+  def dequeue
+  end
+
+  # complete accepts a Job object, using its id the pipeline backend should
+  # mark the job as complete
+  def complete(job)
+  end
+
+  # fail accepts a Job object, using its id the pipeline backend should
+  # mark the job as failed
+  def fail(job)
+  end
+
+  # reset accepts a Job object, using its id the pipeline backend should
+  # reset the job by marking it as free
+  def reset(job)
+  end
+end
+```
+
+To hook up the backend to Employer you must generate and edit a config file by
+running `employer config` (or more likely `bundle exec employer config`). If you
+don't specify a custom path (with -c /path/to/employer\_config.rb) this will
+generate config/employer.rb, the file will look something like this:
+
+```ruby
+# If you're using Rails the below line requires config/environment to setup the
+# Rails environment. If you're not using Rails you'll want to require something
+# here that sets up Employer's environment appropriately (making available the
+# classes that your jobs need to do their work, providing the connection to
+# your database, etc.)
+# require "./config/environment"
+
+require "employer-mongoid"
+
+# Setup the backend for the pipeline, this is where the boss gets the jobs to
+# process. See the documentation for details on writing your own pipeline
+# backend.
+pipeline_backend Employer::Mongoid::Pipeline.new
+
+# Use employees that fork subprocesses to perform jobs. You cannot use these
+# with JRuby, because JRuby doesn't support Process#fork.
+forking_employees 4
+
+# Use employees that run their jobs in threads, you can use these when using
+# JRuby. While threaded employees also work with MRI they are limited by the
+# GIL (this may or may not be a problem depending on the type of work your jobs
+# need to do).
+# threading_employees 4
+```
+
+The comments in the file pretty much explain how you should edit it.
+
+When setup properly you can start processing jobs by running `employer` (or
+`employer -c /path/to/employer\_config.rb`, likely prepended with `bundle exec`)
+
+In your application code you can obtain a pipeline to enqueue jobs with like so:
+
+```ruby
+# Obtain the pipeline
+pipeline = Employer::Workshop.enqueue("/path/to/employer\_config.rb")
+
+# Enqueue a job
+job = NamePutsJob.new
+job.first_name = "Mark"
+job.last_name = "Kremer"
+pipeline.enqueue(job)
+```
 
 ## Contributing
 

data/bin/employer

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "employer"
+require "employer/cli"
+
+Employer::CLI.start(ARGV)

data/employer.gemspec

@@ -9,10 +9,16 @@ Gem::Specification.new do |gem|
   gem.authors       = ["Mark Kremer"]
   gem.email         = ["mark@without-brains.net"]
   gem.summary       = %q{Job processing made easy}
-  gem.homepage      = ""
+  gem.homepage      = "https://github.com/mkremer/employer"
+  gem.license       = "MIT"
 
   gem.files         = `git ls-files`.split($/)
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
+
+  gem.add_runtime_dependency "thor", "~> 0.17"
+
+  gem.add_development_dependency "rspec"
+  gem.add_development_dependency "pry"
 end

data/lib/employer.rb

@@ -1,5 +1,7 @@
 require "employer/version"
-
-module Employer
-  # Your code goes here...
-end
+require "employer/errors"
+require "employer/pipeline"
+require "employer/job"
+require "employer/employees"
+require "employer/boss"
+require "employer/workshop"

data/lib/employer/boss.rb

@@ -0,0 +1,110 @@
+require_relative "errors"
+
+module Employer
+  class Boss
+    attr_reader :pipeline, :employees, :keep_going, :sleep_time
+
+    def initialize
+      @pipeline = nil
+      @employees = []
+      @sleep_time_index = 0
+    end
+
+    def pipeline=(pipeline)
+      @pipeline = pipeline
+    end
+
+    def allocate_employee(employee)
+      employees << employee
+    end
+
+    def stop_managing
+      @keep_going = false
+    end
+
+    def manage
+      @keep_going = true
+
+      while keep_going
+        delegate_work
+        progress_update
+      end
+
+      wait_on_employees
+    end
+
+    def delegate_work
+      while free_employee? && job = get_work
+        delegate_job(job)
+      end
+    end
+
+    def get_work
+      sleep_times = [0.1, 0.5, 1, 2.5, 5]
+      if job = pipeline.dequeue
+        @sleep_time_index = 0
+      else
+        @sleep_time_index += 1 unless @sleep_time_index == (sleep_times.count - 1)
+      end
+      @sleep_time = sleep_times[@sleep_time_index]
+      sleep(sleep_time)
+      job
+    end
+
+    def progress_update
+      busy_employees.each do |employee|
+        update_job_status(employee)
+      end
+    end
+
+    def update_job_status(employee)
+      return if employee.work_in_progress?
+
+      job = employee.job
+
+      if employee.work_completed?
+        pipeline.complete(job)
+      elsif employee.work_failed?
+        if job.try_again?
+          pipeline.reset(job)
+        else
+          pipeline.fail(job)
+        end
+      end
+
+      employee.free
+    end
+
+    def wait_on_employees
+      busy_employees.each do |employee|
+        employee.wait_for_completion
+        update_job_status(employee)
+      end
+    end
+
+    def stop_employees
+      busy_employees.each do |employee|
+        employee.stop_working
+        update_job_status(employee)
+        employee.free
+      end
+    end
+
+    def delegate_job(job)
+      raise Employer::Errors::NoEmployeeFree unless employee = free_employee
+      employee.work(job)
+    end
+
+    def busy_employees
+      employees.select { |employee| !employee.free? }
+    end
+
+    def free_employee
+      employees.find(&:free?)
+    end
+
+    def free_employee?
+      free_employee
+    end
+  end
+end

data/lib/employer/cli.rb

@@ -0,0 +1,70 @@
+require "thor"
+require "fileutils"
+
+module Employer
+  class CLI < Thor
+    default_task :work
+
+    desc "work", "Process jobs"
+    option :config, default: "config/employer.rb", desc: "Config file to use"
+    def work
+      unless File.exists?(options[:config])
+        STDERR.puts "#{options[:config]} does not exist."
+        exit 1
+      end
+
+      int_count = 0
+      workshop = Employer::Workshop.setup(File.read(options[:config]))
+
+      Signal.trap("INT") do
+        int_count += 1
+        if int_count == 1
+          workshop.stop
+        else
+          workshop.stop_now
+        end
+      end
+
+      workshop.run
+    end
+
+    desc "config", "Generate config file"
+    option :config, default: "config/employer.rb", desc: "Path to config file"
+    def config
+      if File.exists?(options[:config])
+        STDERR.puts "#{options[:config]} already exists."
+        exit 1
+      end
+
+      FileUtils.mkdir("config") unless File.directory?("config")
+
+      File.open(options[:config], "w") do |file|
+        file.write <<CONFIG
+# If you're using Rails the below line requires config/environment to setup the
+# Rails environment. If you're not using Rails you'll want to require something
+# here that sets up Employer's environment appropriately (making available the
+# classes that your jobs need to do their work, providing the connection to
+# your database, etc.)
+# require "./config/environment"
+
+require "employer-mongoid"
+
+# Setup the backend for the pipeline, this is where the boss gets the jobs to
+# process. See the documentation for details on writing your own pipeline
+# backend.
+pipeline_backend Employer::Mongoid::Pipeline.new
+
+# Use employees that fork subprocesses to perform jobs. You cannot use these
+# with JRuby, because JRuby doesn't support Process#fork.
+forking_employees 4
+
+# Use employees that run their jobs in threads, you can use these when using
+# JRuby. While threaded employees also work with MRI they are limited by the
+# GIL (this may or may not be a problem depending on the type of work your jobs
+# need to do).
+# threading_employees 4
+CONFIG
+      end
+    end
+  end
+end

data/lib/employer/employees.rb

@@ -0,0 +1,3 @@
+require_relative "employees/abstract_employee"
+require_relative "employees/forking_employee"
+require_relative "employees/threading_employee"

data/lib/employer/employees/abstract_employee.rb

@@ -0,0 +1,55 @@
+require_relative "../errors"
+
+module Employer
+  module Employees
+    class AbstractEmployee
+      attr_reader :job
+
+      def work(job)
+        raise Employer::Errors::EmployeeBusy unless free?
+        @job = job
+      end
+
+      def perform_job
+        job.perform
+      end
+
+      def wait_for_completion
+        work_state(true)
+      end
+
+      def stop_working
+        return if work_completed? || work_failed?
+        force_work_stop
+      end
+
+      def free?
+        job.nil?
+      end
+
+      def work_in_progress?
+        true if work_state == :busy
+      end
+
+      def work_completed?
+        true if work_state == :complete
+      end
+
+      def work_failed?
+        true if work_state == :failed
+      end
+
+      def free
+        return unless work_completed? || work_failed?
+        @work_state = nil
+        @job = nil
+      end
+
+      def work_state(wait = false)
+      end
+
+      def force_work_stop
+      end
+    end
+  end
+end

data/lib/employer/employees/forking_employee.rb

@@ -0,0 +1,52 @@
+require_relative "abstract_employee"
+
+module Employer
+  module Employees
+    class ForkingEmployee < AbstractEmployee
+      def work(job)
+        super
+
+        @job_pid = fork do
+          state = nil
+
+          begin
+            perform_job
+            state = 0
+          ensure
+            state = 1 if state.nil?
+            exit(state)
+          end
+        end
+      end
+
+      def free
+        super
+        @job_pid = nil
+      end
+
+      def work_state(wait = false)
+        return @work_state if [:complete, :failed].include?(@work_state)
+
+        @work_state = :busy
+
+        flags = wait == false ? Process::WNOHANG : 0
+        pid, status = Process.waitpid2(@job_pid, flags)
+        if pid
+          if status.exitstatus == 0
+            @work_state = :complete
+          else
+            @work_state = :failed
+          end
+        end
+
+        @work_state
+      end
+
+      def force_work_stop
+        return if free?
+        Process.kill("KILL", @job_pid)
+        work_state(true)
+      end
+    end
+  end
+end