rocketjob 0.7.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d784e40e75e2aca697e1258fb6d823be8e03c6f1
+  data.tar.gz: 69008d9f1ae0396be4e1838f3d931299af226005
+SHA512:
+  metadata.gz: 614602a9f849b27bfbd4e2fda0985da5ae798e4a95e9ccbfe84a49d620fc1fedd9423df5395e7a1acefc4f42f61008d467d136a2c122ef19038e1dfd4b7dd555
+  data.tar.gz: be35af2fafca63f647ebb485cfcdbcb76399556ac86693a48ec22d98cbfca512925b7c07ac9ca191a1fb7ce37ff0e2235124ddc7f79c7bf313647797e14e35f1

data/README.md

@@ -0,0 +1,160 @@
+# rocketjob
+
+High volume, priority based, background job processing solution for Ruby.
+
+## Status
+
+Alpha - Feedback on the API is welcome. API will change.
+
+Already in use in production internally processing large files with millions
+of records, as well as large jobs to walk though large databases.
+
+## Why?
+
+We have tried for years to make both `resque` and more recently `sidekiq`
+work for large high performance batch processing.
+Even `sidekiq-pro` was purchased and used in an attempt to process large batches.
+
+Unfortunately, after all the pain and suffering with the existing asynchronous
+worker solutions none of them have worked in our production environment without
+significant hand-holding and constant support. Mysteriously the odd record/job
+was disappearing when processing 100's of millions of jobs with no indication
+where those lost jobs went.
+
+In our environment we cannot lose even a single job or record, as all data is
+business critical. The existing batch processing solution do not supply any way
+to collect the output from batch processing and as a result every job has custom
+code to collect it's output. rocketjob has built in support to collect the results
+of any batch job.
+
+High availability and high throughput were being limited by how much we could get
+through `redis`. Being a single-threaded process it is constrained to a single
+CPU. Putting `redis` on a large multi-core box does not help since it will not
+use more than one CPU at a time.
+Additionally, `redis` is constrained to the amount of physical memory is available
+on the server.
+`redis` worked very well when processing was below around 100,000 jobs a day,
+when our workload suddenly increased to over 100,000,000 a day it could not keep
+up. Its single CPU would often hit 100% CPU utilization when running many `sidekiq-pro`
+servers. We also had to store actual job data in a separate MySQL database since
+it would not fit in memory on the `redis` server.
+
+`rocketjob` was created out of necessity due to constant support. End-users were
+constantly contacting the development team to ask on the status of "hung" or
+"in-complete" jobs, as part of our DevOps role.
+
+Another significant production support challenge is trying to get `resque` or `sidekiq`
+to process the batch jobs in a very specific order. Switching from queue-based
+to priority-based job processing means that all jobs are processed in the order of
+their priority and not what queues are defined on what servers and in what quantity.
+This approach has allowed us to significantly increase the CPU and IO utilization
+across all worker machines. The traditional queue based approach required constant
+tweaking in the production environment to try and balance workload without overwhelming
+any one server.
+
+End-users are now able to modify the priority of their various jobs at runtime
+so that they can get that business critical job out first, instead of having to
+wait for other jobs of the same type/priority to finish first.
+
+Since `rocketjob` uploads the entire file, or all data for processing it does not
+require jobs to store the data in other databases.
+Additionally, `rocketjob` supports encryption and compression of any data uploaded
+into Sliced Jobs to ensure PCI compliance and to prevent sensitive from being exposed
+either at rest in the data store, or in flight as it is being read or written to the
+backend data store.
+Often large files received for processing contain sensitive data that must not be exposed
+in the backend job store. Having this capability built-in ensures all our jobs
+are properly securing sensitive data.
+
+Since moving to `rocketjob` our production support has diminished and now we can
+focus on writing code again. :)
+
+## Introduction
+
+`rocketjob` is a global "priority based queue" (https://en.wikipedia.org/wiki/Priority_queue)
+All jobs are placed in a single global queue and the job with the highest priority
+is processed first. Jobs with the same priority are processed on a first-in
+first-out (FIFO) basis.
+
+This differs from the traditional approach of separate queues for jobs which
+quickly becomes cumbersome when there are for example over a hundred different
+types of jobs.
+
+The global priority based queue ensures that the servers are utilized to their
+capacity without requiring constant manual intervention.
+
+`rocketjob` is designed to handle hundreds of millions of concurrent jobs
+that are often encountered in high volume batch processing environments.
+It is designed from the ground up to support large batch file processing.
+For example a single file that contains millions of records to be processed
+as quickly as possible without impacting other jobs with a higher priority.
+
+## Management
+
+The companion project [rocketjob mission control](https://github.com/lambcr/rocket_job_mission_control)
+contains the Rails Engine that can be loaded into your Rails project to add
+a web interface for viewing and managing `rocketjob` jobs.
+
+`rocketjob mission control` can also be run stand-alone in a shell Rails application.
+
+By separating `rocketjob mission control` into a separate gem means it does not
+have to be loaded where `rocketjob` jobs are defined or run.
+
+## Jobs
+
+Simple single task jobs:
+
+Example job to run in a separate worker process
+
+```ruby
+class MyJob < RocketJob::Job
+  # Method to call asynchronously by the worker
+  def perform(email_address, message)
+    # For example send an email to the supplied address with the supplied message
+    send_email(email_address, message)
+  end
+end
+```
+
+To queue the above job for processing:
+
+```ruby
+MyJob.perform_later('jack@blah.com', 'lets meet')
+```
+
+## Configuration
+
+MongoMapper will already configure itself in Rails environments. Sometimes we want
+to use a different Mongo Database instance for the records and results.
+
+For example, the RocketJob::Job can be stored in a Mongo Database that is replicated
+across data centers, whereas we may not want to replicate record and result data
+due to it's sheer volume.
+
+```ruby
+config.before_initialize do
+  # If this environment has a separate Work server
+  # Share the common mongo configuration file
+  config_file = root.join('config', 'mongo.yml')
+  if config_file.file?
+    if config = YAML.load(ERB.new(config_file.read).result)["#{Rails.env}_work]
+      options = (config['options']||{}).symbolize_keys
+      # In the development environment the Mongo driver generates a lot of
+      # network trace log data, move its debug logging to :trace
+      options[:logger] = SemanticLogger::DebugAsTraceLogger.new('Mongo:Work')
+      RocketJob::Config.mongo_work_connection = Mongo::MongoClient.from_uri(config['uri'], options)
+
+      # It is also possible to store the jobs themselves in a separate MongoDB database
+      # RocketJob::Config.mongo_connection = Mongo::MongoClient.from_uri(config['uri'], options)
+    end
+  else
+    puts "\nmongo.yml config file not found: #{config_file}"
+  end
+end
+```
+
+## Requirements
+
+MongoDB V2.6 or greater. V3 is recommended
+
+* V2.6 includes a feature to allow lookups using the `$or` clause to use an index

data/Rakefile

@@ -0,0 +1,28 @@
+require 'rake/clean'
+require 'rake/testtask'
+
+$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
+require 'rocket_job/version'
+
+task :gem do
+  system "gem build rocketjob.gemspec"
+end
+
+task :publish => :gem do
+  system "git tag -a v#{RocketJob::VERSION} -m 'Tagging #{RocketJob::VERSION}'"
+  system "git push --tags"
+  system "gem push rocketjob-#{RocketJob::VERSION}.gem"
+  system "rm rocketjob-#{RocketJob::VERSION}.gem"
+end
+
+desc "Run Test Suite"
+task :test do
+  Rake::TestTask.new(:functional) do |t|
+    t.test_files = FileList['test/**/*_test.rb']
+    t.verbose    = true
+  end
+
+  Rake::Task['functional'].invoke
+end
+
+task :default => :test

data/bin/rocketjob

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+require 'rocketjob'
+
+# Start a rocketjob server instance from the command line
+begin
+  RocketJob::CLI.new(ARGV).run
+rescue => exc
+  # Failsafe logger that writes to STDERR
+  SemanticLogger.add_appender(STDERR, :error, &SemanticLogger::Appender::Base.colorized_formatter)
+  SemanticLogger['RocketJob'].error('Rocket Job shutting down due to exception', exc)
+  SemanticLogger.flush
+  exit 1
+end

data/lib/rocket_job/cli.rb

@@ -0,0 +1,76 @@
+require 'optparse'
+module RocketJob
+  # Command Line Interface parser for RocketJob
+  class CLI
+    attr_reader :name, :threads, :environment, :pidfile, :directory, :quiet
+
+    def initialize(argv)
+      @name             = nil
+      @threads          = nil
+
+      @quiet            = false
+      @environment      = ENV['RAILS_ENV'] || ENV['RACK_ENV'] || 'development'
+      @pidfile          = nil
+      @directory        = '.'
+      parse(argv)
+    end
+
+    # Run a RocketJob::Server from the command line
+    def run
+      SemanticLogger.add_appender(STDOUT,  &SemanticLogger::Appender::Base.colorized_formatter) unless quiet
+      boot_rails
+      write_pidfile
+
+      opts = {}
+      opts[:name]             = name if name
+      opts[:max_threads]      = threads if threads
+      Server.run(opts)
+    end
+
+    # Initialize the Rails environment
+    def boot_rails
+      require File.expand_path("#{directory}/config/environment.rb")
+      if Rails.configuration.eager_load
+        RocketJob::Server.logger.benchmark_info('Eager loaded Rails and all Engines') do
+          Rails.application.eager_load!
+          Rails::Engine.subclasses.each { |engine| engine.eager_load! }
+        end
+      end
+    end
+
+    # Create a PID file if requested
+    def write_pidfile
+      return unless pidfile
+      pid = $$
+      File.open(pidfile, 'w') { |f| f.puts(pid) }
+
+      # Remove pidfile on exit
+      at_exit do
+        File.delete(pidfile) if pid == $$
+      end
+    end
+
+    # Parse command line options placing results in the corresponding instance variables
+    def parse(argv)
+      parser = OptionParser.new do |o|
+        o.on('-n', '--name NAME', 'Unique Name of this server instance (Default: hostname:PID)') { |arg| @name = arg }
+        o.on('-t', '--threads COUNT', 'Number of worker threads to start') { |arg| @threads = arg.to_i }
+        o.on('-q', '--quiet', 'Do not write to stdout, only to logfile. Necessary when running as a daemon') { @quiet = true }
+        o.on('-d', '--dir DIR', 'Directory containing Rails app, if not current directory') { |arg| @directory = arg }
+        o.on('-e', '--environment ENVIRONMENT', 'The environment to run the app on (Default: RAILS_ENV || RACK_ENV || development)') { |arg| @environment = arg }
+        o.on('--pidfile PATH', 'Use PATH as a pidfile') { |arg| @pidfile = arg }
+        o.on('-v', '--version', 'Print the version information') do
+          puts "Rocket Job v#{RocketJob::VERSION}"
+          exit 1
+        end
+      end
+      parser.banner = 'rocketjob <options>'
+      parser.on_tail '-h', '--help', 'Show help' do
+        puts parser
+        exit 1
+      end
+      parser.parse! argv
+    end
+
+  end
+end

data/lib/rocket_job/concerns/worker.rb

@@ -0,0 +1,157 @@
+# encoding: UTF-8
+
+# Worker behavior for a job
+module RocketJob
+  module Concerns
+    module Worker
+      def self.included(base)
+        base.extend ClassMethods
+        base.class_eval do
+          # While working on a slice, the current slice is available via this reader
+          attr_reader :rocket_job_slice
+
+          @rocket_job_defaults = nil
+        end
+      end
+
+      module ClassMethods
+        # Returns [Job] after queue-ing it for processing
+        def later(method, *args, &block)
+          if RocketJob::Config.inline_mode
+            now(method, *args, &block)
+          else
+            job = build(method, *args, &block)
+            job.save!
+            job
+          end
+        end
+
+        # Create a job and process it immediately in-line by this thread
+        def now(method, *args, &block)
+          job = build(method, *args, &block)
+          server = Server.new(name: 'inline')
+          server.started
+          job.start
+          while job.running? && !job.work(server)
+          end
+          job
+        end
+
+        # Build a Rocket Job instance
+        #
+        # Note:
+        #  - #save! must be called on the return job instance if it needs to be
+        #    queued for processing.
+        #  - If data is uploaded into the job instance before saving, and is then
+        #    discarded, call #cleanup! to clear out any partially uploaded data
+        def build(method, *args, &block)
+          job = new(arguments: args, perform_method: method.to_sym)
+          @rocket_job_defaults.call(job) if @rocket_job_defaults
+          block.call(job) if block
+          job
+        end
+
+        # Method to be performed later
+        def perform_later(*args, &block)
+          later(:perform, *args, &block)
+        end
+
+        # Method to be performed later
+        def perform_build(*args, &block)
+          build(:perform, *args, &block)
+        end
+
+        # Method to be performed now
+        def perform_now(*args, &block)
+          now(:perform, *args, &block)
+        end
+
+        # Define job defaults
+        def rocket_job(&block)
+          @rocket_job_defaults = block
+          self
+        end
+      end
+
+      def rocket_job_csv_parser
+        # TODO Change into an instance variable once CSV handling has been re-worked
+        RocketJob::Utility::CSVRow.new
+      end
+
+      # Works on this job
+      #
+      # Returns [true|false] whether this job should be excluded from the next lookup
+      #
+      # If an exception is thrown the job is marked as failed and the exception
+      # is set in the job itself.
+      #
+      # Thread-safe, can be called by multiple threads at the same time
+      def work(server)
+        raise 'Job must be started before calling #work' unless running?
+        begin
+          # before_perform
+          call_method(perform_method, arguments, event: :before, log_level: log_level)
+
+          # perform
+          call_method(perform_method, arguments, log_level: log_level)
+          if self.collect_output?
+            self.output = (result.is_a?(Hash) || result.is_a?(BSON::OrderedHash)) ? result : { result: result }
+          end
+
+          # after_perform
+          call_method(perform_method, arguments, event: :after, log_level: log_level)
+          complete!
+        rescue Exception => exc
+          set_exception(server.name, exc)
+          raise exc if RocketJob::Config.inline_mode
+        end
+        false
+      end
+
+      protected
+
+      # Calls a method on this job, if it is defined
+      # Adds the event name to the method call if supplied
+      #
+      # Returns [Object] the result of calling the method
+      #
+      # Parameters
+      #   method [Symbol]
+      #     The method to call on this job
+      #
+      #   arguments [Array]
+      #     Arguments to pass to the method call
+      #
+      #   Options:
+      #     event: [Symbol]
+      #       Any one of: :before, :after
+      #       Default: None, just calls the method itself
+      #
+      #     log_level: [Symbol]
+      #       Log level to apply to silence logging during the call
+      #       Default: nil ( no change )
+      #
+      def call_method(method, arguments, options={})
+        options               = options.dup
+        event                 = options.delete(:event)
+        log_level             = options.delete(:log_level)
+        raise(ArgumentError, "Unknown #{self.class.name}#call_method options: #{options.inspect}") if options.size > 0
+
+        the_method = event.nil? ? method : "#{event}_#{method}".to_sym
+        if respond_to?(the_method)
+          method_name = "#{self.class.name}##{the_method}"
+          logger.info "Start #{method_name}"
+          logger.benchmark_info("Completed #{method_name}",
+            metric:             "rocketjob/#{self.class.name.underscore}/#{the_method}",
+            log_exception:      :full,
+            on_exception_level: :error,
+            silence:            log_level
+          ) do
+            self.send(the_method, *arguments)
+          end
+        end
+      end
+
+    end
+  end
+end