rocketjob 0.9.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b2d5bd2e258623a70cb2ac81bb559db1302d2c40
-  data.tar.gz: 5e478b33ac8e404496ca112536dcd57e24d3b369
+  metadata.gz: 4f0d0679cdb6e1b6fb284b1dfd2c90b5b201ffe7
+  data.tar.gz: ea135d888befde2dd286242a8c5bb1c338a45128
 SHA512:
-  metadata.gz: 6c39b919588cd3f58490294e92ef2233f397a867b4e3bf41073e16374238348f6a2ec892a2dfbcc73ed838ef62c3344a6a3d7ec9289e70ba4ee575cf5ab096a7
-  data.tar.gz: 07603275a4346907ea05613012582cbf746571011d25bb4bb42d3bfa022932c8867030025a338e83aa21f8de9eb597e9cee83d9cb34e3eb9175054da8c2101d9
+  metadata.gz: c8640b98d22541f3739a4a61fd726981a3f349c0b57ecdaf88e006839e5ac98d3505144723df65a7f98505951a9f971c0d393669fa655e63b445221a64d8e485
+  data.tar.gz: 88665fc325e161c4cb3d4dbdf506250c3891742199c6e3d9c0b5f74064f245faf08b21213222ff0679fff24ee2cea9bef4b2af3ae7da1f146dac57e472800d30

data/README.md

@@ -1,330 +1,20 @@
-# rocketjob[![Build Status](https://secure.travis-ci.org/rocketjob/rocketjob.png?branch=master)](http://travis-ci.org/rocketjob/rocketjob) ![](http://ruby-gem-downloads-badge.herokuapp.com/rocketjob?type=total)
+# rocketjob [![Gem Version](https://badge.fury.io/rb/rocketjob.svg)](http://badge.fury.io/rb/rocketjob) [![Build Status](https://secure.travis-ci.org/rocketjob/rocketjob.png?branch=master)](http://travis-ci.org/rocketjob/rocketjob) ![](http://ruby-gem-downloads-badge.herokuapp.com/rocketjob?type=total)
 
-High volume, priority based, background job processing solution for Ruby.
+High volume, priority based, distributed, background job processing solution for Ruby.
 
 ## Status
 
-Beta - Feedback on the API is welcome. API may change.
+Production Ready
 
-Already in use in production internally processing large files with millions
+Already in use in production processing large files with millions
 of records, as well as large jobs to walk though large databases.
 
-## Why?
+## Documentation
 
-We have tried for years to make both `resque` and more recently `sidekiq`
-work for large high performance batch processing.
-Even `sidekiq-pro` was purchased and used in an attempt to process large batches.
+* [Guide](http://rocketjob.io/)
+* [API Reference](http://www.rubydoc.info/gems/rocketjob/)
 
-Unfortunately, after all the pain and suffering with the existing asynchronous
-worker solutions none of them have worked in our production environment without
-significant hand-holding and constant support. Mysteriously the odd record/job
-was disappearing when processing 100's of millions of jobs with no indication
-where those lost jobs went.
-
-In our environment we cannot lose even a single job or record, as all data is
-business critical. The existing batch processing solution do not supply any way
-to collect the output from batch processing and as a result every job has custom
-code to collect it's output. rocketjob has built in support to collect the results
-of any batch job.
-
-High availability and high throughput were being limited by how much we could get
-through `redis`. Being a single-threaded process it is constrained to a single
-CPU. Putting `redis` on a large multi-core box does not help since it will not
-use more than one CPU at a time.
-Additionally, `redis` is constrained to the amount of physical memory is available
-on the server.
-`redis` worked very well when processing was below around 100,000 jobs a day,
-when our workload suddenly increased to over 100,000,000 a day it could not keep
-up. Its single CPU would often hit 100% CPU utilization when running many `sidekiq-pro`
-servers. We also had to store actual job data in a separate MySQL database since
-it would not fit in memory on the `redis` server.
-
-`rocketjob` was created out of necessity due to constant support. End-users were
-constantly contacting the development team to ask on the status of "hung" or
-"in-complete" jobs, as part of our DevOps role.
-
-Another significant production support challenge is trying to get `resque` or `sidekiq`
-to process the batch jobs in a very specific order. Switching from queue-based
-to priority-based job processing means that all jobs are processed in the order of
-their priority and not what queues are defined on what servers and in what quantity.
-This approach has allowed us to significantly increase the CPU and IO utilization
-across all worker machines. The traditional queue based approach required constant
-tweaking in the production environment to try and balance workload without overwhelming
-any one server.
-
-End-users are now able to modify the priority of their various jobs at runtime
-so that they can get that business critical job out first, instead of having to
-wait for other jobs of the same type/priority to finish first.
-
-Since `rocketjob` uploads the entire file, or all data for processing it does not
-require jobs to store the data in other databases.
-Additionally, `rocketjob` supports encryption and compression of any data uploaded
-into Sliced Jobs to ensure PCI compliance and to prevent sensitive from being exposed
-either at rest in the data store, or in flight as it is being read or written to the
-backend data store.
-Often large files received for processing contain sensitive data that must not be exposed
-in the backend job store. Having this capability built-in ensures all our jobs
-are properly securing sensitive data.
-
-Since moving to `rocketjob` our production support has diminished and now we can
-focus on writing code again. :)
-
-## Introduction
-
-`rocketjob` is a global "priority based queue" (https://en.wikipedia.org/wiki/Priority_queue)
-All jobs are placed in a single global queue and the job with the highest priority
-is processed first. Jobs with the same priority are processed on a first-in
-first-out (FIFO) basis.
-
-This differs from the traditional approach of separate queues for jobs which
-quickly becomes cumbersome when there are for example over a hundred different
-types of jobs.
-
-The global priority based queue ensures that the workers are utilized to their
-capacity without requiring constant manual intervention.
-
-`rocketjob` is designed to handle hundreds of millions of concurrent jobs
-that are often encountered in high volume batch processing environments.
-It is designed from the ground up to support large batch file processing.
-For example a single file that contains millions of records to be processed
-as quickly as possible without impacting other jobs with a higher priority.
-
-## Management
-
-The companion project [rocketjob mission control](https://github.com/rocketjob/rocket_job_mission_control)
-contains the Rails Engine that can be loaded into your Rails project to add
-a web interface for viewing and managing `rocketjob` jobs.
-
-`rocketjob mission control` can also be run stand-alone in a shell Rails application.
-
-By separating `rocketjob mission control` into a separate gem means it does not
-have to be loaded where `rocketjob` jobs are defined or run.
-
-## Jobs
-
-Simple single task jobs:
-
-Example job to run in a separate worker process
-
-```ruby
-class MyJob < RocketJob::Job
-  # Method to call asynchronously by the worker
-  def perform(email_address, message)
-    # For example send an email to the supplied address with the supplied message
-    send_email(email_address, message)
-  end
-end
-```
-
-To queue the above job for processing:
-
-```ruby
-MyJob.perform_later('jack@blah.com', 'lets meet')
-```
-
-## Directory Monitoring
-
-A common task with many batch processing systems is to look for the appearance of
-new files and kick off jobs to process them. `DirmonJob` is a job designed to do
-this task.
-
-`DirmonJob` runs every 5 minutes by default, looking for new files that have appeared
-based on configured entries called `DirmonEntry`. Ultimately these entries will be
-configurable via `rocketjob_mission_control`, the web management interface for `rocketjob`.
-
-Example, creating a `DirmonEntry`
-
-```ruby
-RocketJob::DirmonEntry.new(
-  path:         'path_to_monitor/*',
-  job:          'Jobs::TestJob',
-  arguments:    [ { input: 'yes' } ],
-  properties:   { priority: 23, perform_method: :event },
-  archive_directory: '/exports/archive'
-)
-```
-
-The attributes of DirmonEntry:
-
-* path <String>
-
-Wildcard path to search for files in.
-For details on valid path values, see: http://ruby-doc.org/core-2.2.2/Dir.html#method-c-glob
-
-Example:
-
-    * input_files/process1/*.csv*
-    * input_files/process2/**/*
-
-* job <String>
-
-Name of the job to start
-
-* arguments <Array>
-
-Any user supplied arguments for the method invocation
-All keys must be UTF-8 strings. The values can be any valid BSON type:
-
-    * Integer
-    * Float
-    * Time    (UTC)
-    * String  (UTF-8)
-    * Array
-    * Hash
-    * True
-    * False
-    * Symbol
-    * nil
-    * Regular Expression
-
-_Note_: Date is not supported, convert it to a UTC time
-
-* properties <Hash>
-
-Any job properties to set.
-
-Example, override the default job priority:
-
-```ruby
-{ priority: 45 }
-```
-
-* archive_directory
-
-Archive directory to move the file to before the job is started. It is important to
-move the file before it is processed so that it is not picked up again for processing.
-If no archive_directory is supplied the file will be moved to a folder called '_archive'
-in the same folder as the file itself.
-
-If the `path` above is a relative path the relative path structure will be
-maintained when the file is moved to the archive path.
-
-* enabled <Boolean>
-
-Allow a monitoring entry to be disabled so that it is ignored by `DirmonJob`.
-This feature is useful for operations to temporarily stop processing files
-from a particular source, without having to completely delete the `DirmonEntry`.
-It can also be used to create a `DirmonEntry` without it becoming immediately
-active.
-```
-
-### Starting the directory monitor
-
-The directory monitor job only needs to be started once per installation by running
-the following code:
-
-```ruby
-RocketJob::Jobs::DirmonJob.perform_later
-```
-
-The polling interval to check for new files can be modified when starting the job
-for the first time by adding:
-```ruby
-RocketJob::Jobs::DirmonJob.perform_later do |job|
-  job.check_seconds = 180
-end
-```
-
-The default priority for `DirmonJob` is 40, to increase it's priority:
-```ruby
-RocketJob::Jobs::DirmonJob.perform_later do |job|
-  job.check_seconds = 300
-  job.priority      = 25
-end
-```
-
-Once `DirmonJob` has been started it's priority and check interval can be
-changed at any time as follows:
-
-```ruby
-RocketJob::Jobs::DirmonJob.first.set(check_seconds: 180, priority: 20)
-```
-
-The `DirmonJob` will automatically re-schedule a new instance of itself to run in
-the future after it completes a each scan/run. If successful the current job instance
-will destroy itself.
-
-In this way it avoids having a single Directory Monitor process that constantly
-sits there monitoring folders for changes. More importantly it avoids a "single
-point of failure" that is typical for earlier directory monitoring solutions.
-Every time `DirmonJob` runs and scans the paths for new files it could be running
-on a new worker. If any server/worker is removed or shutdown it will not stop
-`DirmonJob` since it will just run on another worker instance.
-
-There can only be one `DirmonJob` instance `queued` or `running` at a time. Any
-attempt to start a second instance will result in an exception.
-
-If an exception occurs while running `DirmonJob`, a failed job instance will remain
-in the job list for problem determination. The failed job cannot be restarted and
-should be destroyed if no longer needed.
-
-## Rails Configuration
-
-MongoMapper will already configure itself in Rails environments. `rocketjob` can
-be configured to use a separate MongoDB instance from the Rails application as follows:
-
-For example, we may want `RocketJob::Job` to be stored in a Mongo Database that
-is replicated across data centers, whereas we may not want to replicate the
-`RocketJob::SlicedJob`** slices due to it's sheer volume.
-
-```ruby
-config.before_initialize do
-  # Share the common mongo configuration file
-  config_file = root.join('config', 'mongo.yml')
-  if config_file.file?
-    config = YAML.load(ERB.new(config_file.read).result)
-    if config["#{Rails.env}_rocketjob]
-      options = (config['options']||{}).symbolize_keys
-      options[:logger] = SemanticLogger::DebugAsTraceLogger.new('Mongo:rocketjob')
-      RocketJob::Config.mongo_connection = Mongo::MongoClient.from_uri(config['uri'], options)
-    end
-    # It is also possible to store the jobs themselves in a separate MongoDB database
-    if config["#{Rails.env}_rocketjob_work]
-      options = (config['options']||{}).symbolize_keys
-      options[:logger] = SemanticLogger::DebugAsTraceLogger.new('Mongo:rocketjob_work')
-      RocketJob::Config.mongo_work_connection = Mongo::MongoClient.from_uri(config['uri'], options)
-    end
-  else
-    puts "\nmongo.yml config file not found: #{config_file}"
-  end
-end
-```
-
-For an example config file, `config/mongo.yml`, see [mongo.yml](https://github.com/rocketjob/rocketjob/blob/master/test/config/mongo.yml)
-
-## Standalone Configuration
-
-When running `rocketjob` in a standalone environment without Rails, the MongoDB
-connections will need to be setup as follows:
-
-```ruby
-options = {
-  pool_size:    50,
-  pool_timeout: 5,
-  logger:       SemanticLogger::DebugAsTraceLogger.new('Mongo:Work'),
-}
-
-# For example when using a replica-set for high availability
-uri = 'mongodb://mongo1.site.com:27017,mongo2.site.com:27017/production_rocketjob'
-RocketJob::Config.mongo_connection = Mongo::MongoClient.from_uri(uri, options)
-
-# Use a separate database, or even server for `RocketJob::SlicedJob` slices
-uri = 'mongodb://mongo1.site.com:27017,mongo2.site.com:27017/production_rocketjob_slices'
-RocketJob::Config.mongo_work_connection = Mongo::MongoClient.from_uri(uri, options)
-```
-
-## Requirements
-
-MongoDB V2.6 or greater. V3 is recommended
-
-* V2.6 includes a feature to allow lookups using the `$or` clause to use an index
-
-## Meta
-
-* Code: `git clone git://github.com/rocketjob/rocketjob.git`
-* Home: <https://github.com/rocketjob/rocketjob>
-* Bugs: <http://github.com/rocketjob/rocketjob/issues>
-* Gems: <http://rubygems.org/gems/rocketjob>
+## Versioning
 
 This project uses [Semantic Versioning](http://semver.org/).
 

data/lib/rocket_job/dirmon_entry.rb

@@ -2,6 +2,10 @@ module RocketJob
   class DirmonEntry
     include MongoMapper::Document
 
+    # Name for this path entry used to identify this DirmonEntry
+    # in the user interface
+    key :name,               String
+
     # Wildcard path to search for files in
     #
     # Example:
@@ -12,13 +16,13 @@ module RocketJob
     #
     # Note
     # - If there are no '*' in the path then an exact filename match is expected
-    key :path,          String
+    key :path,               String
 
     # Job to start
     #
     # Example:
     #   "ProcessItJob"
-    key :job,           String
+    key :job_name,          String
 
     # Any user supplied arguments for the method invocation
     # All keys must be UTF-8 strings. The values can be any valid BSON type:
@@ -35,13 +39,13 @@ module RocketJob
     #   Regular Expression
     #
     # Note: Date is not supported, convert it to a UTC time
-    key :arguments,     Array, default: []
+    key :arguments,          Array, default: []
 
     # Any job properties to set
     #
     # Example, override the default job priority:
     #   { priority: 45 }
-    key :properties,    Hash, default: {}
+    key :properties,         Hash, default: {}
 
     # Archive directory to move files to when processed to prevent processing the
     # file again.
@@ -53,8 +57,41 @@ module RocketJob
     key :archive_directory,  String
 
     # Allow a monitoring path to be temporarily disabled
-    key :enabled,       Boolean, default: true
+    key :enabled,            Boolean, default: true
+
+    # Method to perform on the job, usually :perform
+    key :perform_method,     Symbol, default: :perform
+
+    # Returns the Job to be queued
+    def job_class
+      job_name.nil? ? nil : job_name.constantize
+    end
+
+    validates_presence_of :path, :job_name
+
+    validates_each :job_name do |record, attr, value|
+      exists = false
+      begin
+        exists = value.nil? ? false : value.constantize.ancestors.include?(RocketJob::Job)
+      rescue NameError => exc
+      end
+      record.errors.add(attr, 'job_name must be defined and must be derived from RocketJob::Job') unless exists
+    end
+
+    validates_each :arguments do |record, attr, value|
+      if klass = record.job_class
+        count = klass.argument_count(record.perform_method)
+        record.errors.add(attr, "There must be #{count} argument(s)") if  value.size != count
+      end
+    end
+
+    validates_each :properties do |record, attr, value|
+      if record.job_name && (methods = record.job_class.instance_methods)
+        value.each_pair do |key, value|
+          record.errors.add(attr, "Unknown property: #{key.inspect} with value: #{value}") unless methods.include?("#{key}=".to_sym)
+        end
+      end
+    end
 
-    validates_presence_of :path, :job
   end
 end

data/lib/rocket_job/job.rb

@@ -215,6 +215,11 @@ module RocketJob
       where(state: 'paused').each { |job| job.resume! }
     end
 
+    # Returns the number of required arguments for this job
+    def self.argument_count(method=:perform)
+      instance_method(method).arity
+    end
+
     # Returns [true|false] whether to collect the results from running this batch
     def collect_output?
       collect_output == true

data/lib/rocket_job/jobs/dirmon_job.rb

@@ -96,7 +96,7 @@ module RocketJob
       def check_file(entry, file_name, previous_size)
         size = File.size(file_name)
         if previous_size && (previous_size == size)
-          logger.info("File stabilized: #{file_name}. Starting: #{entry.job}")
+          logger.info("File stabilized: #{file_name}. Starting: #{entry.job_name}")
           start_job(entry, file_name)
           nil
         else
@@ -111,8 +111,9 @@ module RocketJob
 
       # Starts the job for the supplied entry
       def start_job(entry, file_name)
-        entry.job.constantize.perform_later(*entry.arguments) do |job|
-          # Set properties, also allows :perform_method to be overridden
+        entry.job_class.perform_later(*entry.arguments) do |job|
+          job.perform_method = entry.perform_method
+          # Set properties
           entry.properties.each_pair { |k, v| job.send("#{k}=".to_sym, v) }
 
           upload_file(job, file_name, entry.archive_directory)

data/lib/rocket_job/version.rb

@@ -1,4 +1,4 @@
 # encoding: UTF-8
 module RocketJob #:nodoc
-  VERSION = "0.9.1"
+  VERSION = "1.0.0"
 end

data/test/dirmon_entry_test.rb

@@ -0,0 +1,70 @@
+require_relative 'test_helper'
+require_relative 'jobs/test_job'
+
+# Unit Test for RocketJob::Job
+class DirmonEntryTest < Minitest::Test
+  context RocketJob::DirmonEntry do
+    teardown do
+      @dirmon_entry.destroy if @dirmon_entry && @dirmon_entry.new_record?
+    end
+
+    context '.config' do
+      should 'support multiple databases' do
+        assert_equal 'test_rocketjob', RocketJob::DirmonEntry.collection.db.name
+      end
+    end
+
+    context '#validate' do
+      should 'existance' do
+        assert entry = RocketJob::DirmonEntry.new(job_name: 'Jobs::TestJob')
+        assert_equal false, entry.valid?
+        assert_equal [ "can't be blank" ], entry.errors[:path], entry.errors.inspect
+      end
+
+      should 'job_name' do
+        assert entry = RocketJob::DirmonEntry.new(path: '/abc/**')
+        assert_equal false, entry.valid?
+        assert_equal ["can't be blank", "job_name must be defined and must be derived from RocketJob::Job"], entry.errors[:job_name], entry.errors.inspect
+      end
+
+      should 'arguments' do
+        assert entry = RocketJob::DirmonEntry.new(
+          job_name: 'Jobs::TestJob',
+          path:     '/abc/**'
+        )
+        assert_equal false, entry.valid?
+        assert_equal ["There must be 1 argument(s)"], entry.errors[:arguments], entry.errors.inspect
+      end
+
+      should 'arguments with perform_method' do
+        assert entry = RocketJob::DirmonEntry.new(
+          job_name:       'Jobs::TestJob',
+          path:           '/abc/**',
+          perform_method: :sum
+        )
+        assert_equal false, entry.valid?
+        assert_equal ["There must be 2 argument(s)"], entry.errors[:arguments], entry.errors.inspect
+      end
+
+      should 'valid' do
+        assert entry = RocketJob::DirmonEntry.new(
+          job_name:  'Jobs::TestJob',
+          path:      '/abc/**',
+          arguments: [1]
+        )
+        assert entry.valid?, entry.errors.inspect
+      end
+
+      should 'valid with perform_method' do
+        assert entry = RocketJob::DirmonEntry.new(
+          job_name:       'Jobs::TestJob',
+          path:           '/abc/**',
+          perform_method: :sum,
+          arguments:      [1,2]
+        )
+        assert entry.valid?, entry.errors.inspect
+      end
+    end
+
+  end
+end

data/test/dirmon_job_test.rb

@@ -11,7 +11,7 @@ class DirmonJobTest < Minitest::Test
       @archive_directory = '/tmp/archive_directory'
       @entry        = RocketJob::DirmonEntry.new(
         path:         'abc/*',
-        job:          'Jobs::TestJob',
+        job_name:     'Jobs::TestJob',
         arguments:    [ { input: 'yes' } ],
         properties:   { priority: 23, perform_method: :event },
         archive_directory: @archive_directory
@@ -27,12 +27,6 @@ class DirmonJobTest < Minitest::Test
       FileUtils.remove_dir(@archive_directory, true) if Dir.exist?(@archive_directory)
     end
 
-    context '.config' do
-      should 'support multiple databases' do
-        assert_equal 'test_rocketjob', RocketJob::DirmonEntry.collection.db.name
-      end
-    end
-
     context '#archive_file' do
       should 'archive absolute path file' do
         begin
@@ -123,7 +117,7 @@ class DirmonJobTest < Minitest::Test
         job = @dirmon_job.stub(:upload_file, -> j, fn, sp { assert_equal [file_name, @archive_directory], [fn, sp] }) do
           @dirmon_job.start_job(@entry, file_name)
         end
-        assert_equal @entry.job, job.class.name
+        assert_equal @entry.job_name, job.class.name
         assert_equal 23, job.priority
         assert_equal [ {:input=>"yes", "before_event"=>true, "event"=>true, "after_event"=>true} ], job.arguments
       end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rocketjob
 version: !ruby/object:Gem::Version
-  version: 0.9.1
+  version: 1.0.0
 platform: ruby
 authors:
 - Reid Morrison
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-07-22 00:00:00.000000000 Z
+date: 2015-07-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aasm
@@ -108,8 +108,8 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.0'
-description: Designed for batch processing from single records to millions of records
-  in a single batch. Uses threading instead of process forking for greater throughtput.
+description: High volume, priority based, distributed, background job processing solution
+  for Ruby
 email:
 - reidmo@gmail.com
 executables:
@@ -132,13 +132,14 @@ files:
 - lib/rocket_job/worker.rb
 - lib/rocketjob.rb
 - test/config/mongo.yml
+- test/dirmon_entry_test.rb
 - test/dirmon_job_test.rb
 - test/job_test.rb
 - test/job_worker_test.rb
 - test/jobs/test_job.rb
 - test/test_helper.rb
 - test/worker_test.rb
-homepage: https://github.com/rocketjob/rocketjob
+homepage: http://rocketjob.io
 licenses:
 - GPL-3.0
 metadata: {}
@@ -161,9 +162,10 @@ rubyforge_project:
 rubygems_version: 2.4.8
 signing_key: 
 specification_version: 4
-summary: High volume, priority based, Enterprise Batch Processing solution for Ruby
+summary: Background job processing system for Ruby
 test_files:
 - test/config/mongo.yml
+- test/dirmon_entry_test.rb
 - test/dirmon_job_test.rb
 - test/job_test.rb
 - test/job_worker_test.rb