bj 1.0.0 → 1.0.1

data/HISTORY

@@ -1,19 +1,24 @@
-0.0.5:
-  - use full path to ruby for plugin mode
-  - plugin correctly installs bin -->> script
-  - plugin install uses --force
-  - properly quote paths in windows (spaces)
-  - swith win signal to ABRT (was INT)
-  - background job regrestration now uses ppid to pin the subprocess to a
-    parent
-  - use ppid to detect parent death and exit in event loop 
-  - don't use gem dependanices in plugin as they are broken when loading from
-    muliple gem repos
-  - added a small amount of drb magic that allows signals to work across
-    processes even on windows (see http://drawohara.com/post/22540307)
+  1.0.1:
+    - fixed name collision with 'record.attributes = hash' ar mass
+      assignment method (thx jon guymon)
+    - added new sponsor: http://igicom.com/
+
+  0.0.5:
+    - use full path to ruby for plugin mode
+    - plugin correctly installs bin -->> script
+    - plugin install uses --force
+    - properly quote paths in windows (spaces)
+    - switch win signal to ABRT (was INT)
+    - background job regrestration now uses ppid to pin the subprocess to a
+      parent
+    - use ppid to detect parent death and exit in event loop 
+    - don't use gem dependanices in plugin as they are broken when loading from
+      muliple gem repos
+    - added a small amount of drb magic that allows signals to work across
+      processes even on windows (see http://drawohara.com/post/22540307)
   
 0.0.4:
-  - basic functionality in widows
+  - basic functionality in windows
   - several small bug fixes
 
 0.0.3:

data/README

@@ -2,94 +2,257 @@ NAME
   bj
 
 SYNOPSIS
-  bj (migration_code|generate_migration|migrate|setup|run|submit|list|set|config|pid) [options]+
+  bj (migration_code|generate_migration|migrate|setup|plugin|run|submit|list|set|config|pid) [options]+
 
 DESCRIPTION
   ________________________________
   Overview
   --------------------------------
   
-    Backgroundjob (Bj) is a simple to use background priority queue for rails.
-    Although not yet tested on windows, the design of bj is such that operation
-    should be possible on any operating system, including M$.
+  Backgroundjob (Bj) is a brain dead simple zero admin background priority queue
+  for Rails.  Bj is robust, platform independent (including windows), and
+  supports internal or external manangement of the background runner process.
   
-    Jobs can be submitted to the queue directly using the api or from the
-    commandline using the 'bj' script.  For example
+  Jobs can be submitted to the queue directly using the api or from the command
+  line using the ./script/bj:
   
-    code:
-        Bj.submit 'cat /etc/password'
+    api:
+      Bj.submit 'cat /etc/password'
   
-      cli:
-        bj submit cat /etc/password
+    command line:
+      bj submit cat /etc/password
   
-    When used from inside a rails application bj arranges that another process
-    will always be running in the background to process the jobs that you submit.
-    By using a separate process to run jobs bj does not impact the resource
-    utilization of your rails application at all and enables several very cool
-    features:
+  Bj's priority queue lives in the database and is therefore durable - your jobs
+  will live across an app crash or machine reboot.  The job management is
+  comprehensive capturing stdout, stderr, exit_status, and temporal statistics
+  about each job:
   
-      1) Bj allows you to sumbit jobs to any of your configured databases and,
-      in each case, spawns a separate background process to run jobs from that
-      queue
+    jobs = Bj.submit array_of_commands, :priority => 42
   
-        Bj.in :production do
-          Bj.submit 'production_job.exe'
-        end
+    ...
   
-        Bj.in :development do
-          Bj.submit 'development_job.exe'
-        end
+    jobs.each do |job|
+      if job.finished?
+        p job.stdout
+        p job.stderr
+        p job.exit_status
+        p job.started_at
+        p job.finished_at
+      end
+    end
+  
+  In addition the background runner process logs all commands run and their
+  exit_status to a log named using the following convention:
+  
+    rails_root/log/bj.#{ HOSTNAME }.#{ RAILS_ENV }.log
+  
+  Bj allows you to submit jobs to multiple databases; for instance, if your
+  application is running in development mode you may do:
+  
+    Bj.in :production do
+      Bj.submit 'my_job.exe'
+    end
+  
+  Bj manages the ever growing list of jobs ran by automatically archiving them
+  into another table (by default jobs > 24 hrs old are archived) to prevent the
+  jobs table from becoming bloated and huge.
+  
+  All Bj's tables are namespaced and accessible via the Bj module:
+  
+    Bj.table.job.find(:all)         # jobs table
+    Bj.table.job_archive.find(:all) # archived jobs
+    Bj.table.config.find(:all)      # configuration and runner state
+  
+  Bj always arranges for submitted jobs to run with a current working directory
+  of RAILS_ROOT and with the correct RAILS_ENV setting.  For example, if you
+  submit a job in production it will have ENV['RAILS_ENV'] == 'production'.
+  
+  When Bj manages the background runner it will never outlive the rails
+  application - it is started and stopped on demand as the rails app is started
+  and stopped.  This is also true for ./script/console - Bj will automatically
+  fire off the background runner to process jobs submitted using the console.
+  
+  Bj ensures that only one background process is running for your application -
+  firing up three mongrels or fcgi processes will result in only one background
+  runner being started.  Note that the number of background runners does not
+  determine throughput - that is determined primarily by the nature of the jobs
+  themselves and how much work they perform per process.
   
-      2) Although bj ensures that a process is always running to process
-      your jobs, you can start a proces manually.  This means that any machine
-      capable of seeing your RAILS_ROOT can run jobs for your application, allowing
-      one to setup a cluster of machines doing the work of a single front end rails
-      applicaiton.
   
   ________________________________
-  Install 
+  Architecture
   --------------------------------
+  
+  If one ignores platform specific details the design of Bj is quite simple: the
+  main Rails application submits jobs to table, stored in the database.  The act
+  of submitting triggers exactly one of two things to occur:
+  
+    1) a new long running background runner to be started
+  
+    2) an existing background runner to be signaled
+  
+  The background runner refuses to run two copies of itself for a given
+  hostname/rails_env combination.  For example you may only have one background
+  runner processing jobs on localhost in development mode.
+  
+  The background runner, under normal circumstances, is managed by Bj itself -
+  you need do nothing to start, monitor, or stop it - it just works.  However,
+  some people will prefer manage their own background process, see 'External
+  Runner' section below for more on this.
+  
+  The runner simply processes each job in a highest priority oldest-in fashion,
+  capturing stdout, stderr, exit_status, etc. and storing the information back
+  into the database while logging it's actions.  When there are no jobs to run
+  the runner goes to sleep for 42 seconds; however this sleep is interuptable,
+  such as when the runner is signaled that a new job has been submitted so,
+  under normal circumstances there will be zero lag between job submission and
+  job running for an empty queue.
+  
+  
+  ________________________________
+  External Runner / Clustering
+  --------------------------------
+  
+  For the paranoid control freaks out there (myself included) it is quite
+  possible to manage and monitor the runner process manually.  This can be
+  desirable in production setups where monitoring software may kill leaking
+  rails apps periodically.
+  
+  Recalling that Bj will only allow one copy of itself to process jobs per
+  hostname/rails_env pair we can simply do something like this in cron
+  
+    cmd = bj run --forever \
+                 --rails_env=development \
+                 --rails_root=/Users/ahoward/rails_root
+  
+    */15 * * * * $cmd 
+  
+  this will simply attempt the start the background runner every 15 minutes if,
+  and only if, it's not *already* running.
+  
+  In addtion to this you'll want to tell Bj not to manage the runner itself
+  using
+  
+    Bj.config["production.no_tickle"] = true
+  
+  Note that, for clusting setups, it's as simple as adding a crontab and config
+  entry like this for each host.  Because Bj throttles background runners per
+  hostname this will allow one runner per hostname - making it quite simple to
+  cluster three nodes behind a besieged rails application.
+  
+  
+  ________________________________
+  Designing Jobs
+  --------------------------------
+  
+  Bj runs it's jobs as command line applications.  It ensures that all jobs run
+  in RAILS_ROOT so it's quite natural to apply a pattern such as
+  
+    mkdir ./jobs
+    edit ./jobs/background_job_to_run
+  
+    ...
+  
+    Bj.submit "./jobs/background_job_to_run"
+  
+  If you need to run you jobs under an entire rails environment you'll need to
+  do this:
+  
+    Bj.submit "./script/runner ./jobs/background_job_to_run"
+  
+  Obviously "./script/runner" loads the rails environment for you.  It's worth
+  noting that this happens for each job and that this is by design: the reason
+  is that most rails applications leak memory like a sieve so, if one were to
+  spawn a long running process that used the application code base you'd have a
+  lovely doubling of memory usage on you app servers.  Although loading the
+  rails environment for each background job requires a little time, a little
+  cpu, and a lot less memory.  A future version of Bj will provide a way to load
+  the rails environment once and to process background jobs in this environment,
+  but anyone wanting to use this in production will be required to duct tape
+  their entire chest and have a team of oxen rip off the tape without screaming
+  to prove steelyness of spirit and profound understanding of the other side.
+  
+  Don't forget that you can submit jobs with command line arguments:
+  
+    Bj.submit "./jobs/a.rb 1 foobar --force" 
+  
+  and that you can do powerful things by passing stdin to a job that powers
+  through a list of work.  For instance, assume a "./jobs/bulkmail" job
+  resembling
+  
+    STDIN.each do |line|
+      address = line.strip
+      mail_message_to address
+    end
     
-    Bj can be installed two ways: as a gem or as a plugin.
+  then you could
   
-      gem:
+    stdin = [
+      "foo@bar.com",
+      "bar@foo.com",
+      "ara.t.howard@codeforpeople.com",
+    ]
   
-        1) $sudo gem install bj
-        2) bj setup ./rails_root/
+    Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
+  
+  and all those emails would be sent in the background.
+  
+  Bj's power is putting jobs in the background in a simple and robust fashion.
+  It's your task to build intelligent jobs that leverage batch processing, and
+  other, possibilities.  The upshot of building tasks this way is that they are
+  quite easy to test before submitting them from inside your application.
   
-      plugin:
   
+  ________________________________
+  Install 
+  --------------------------------
+    
+    Bj can be installed two ways: as a plugin or via rubygems
+  
+      plugin:
         1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
         2) ./script/bj setup
   
+      gem:
+        1) $sudo gem install bj
+        2) add "require 'bj'" to config/environment.rb
+        3) bj setup
+  
   ________________________________
   Api
   --------------------------------
   
-    submit jobs for background processing.  'jobs' can be a string or array of
-    strings.  options are applied to each job in the 'jobs', and the list of
-    submitted jobs is always returned.  options (string or symbol) can be
-    
-      :rails_env => production|development|key_in_database_yml 
-                    when given this keyword causes bj to submit jobs to the
-                    specified database.  default is RAILS_ENV.
-    
-      :priority => any number, including negative ones.  default is zero.
-    
-      :tag => a tag added to the job.  simply makes searching easier.
+  submit jobs for background processing.  'jobs' can be a string or array of
+  strings.  options are applied to each job in the 'jobs', and the list of
+  submitted jobs is always returned.  options (string or symbol) can be
     
-      :env => a hash specifying any additional environment vars the background
-              process should have.
-    
-      :stdin => any stdin the background process should have.
-    
-    eg:
-    
-      jobs = Bj.submit 'echo foobar', :tag => 'simple job'
-    
-      jobs = Bj.submit '/bin/cat', :stdin => 'in the hat'
+    :rails_env => production|development|key_in_database_yml 
+                  when given this keyword causes bj to submit jobs to the
+                  specified database.  default is RAILS_ENV.
+  
+    :priority => any number, including negative ones.  default is zero.
+  
+    :tag => a tag added to the job.  simply makes searching easier.
+  
+    :env => a hash specifying any additional environment vars the background
+            process should have.
+  
+    :stdin => any stdin the background process should have.  must respond_to
+              to_s
     
-      jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+  eg:
+  
+    jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+  
+    jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+  
+    jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+  
+    jobs = Bj.submit './script/runner /dev/stdin',
+                     :stdin => 'p RAILS_ENV',
+                     :tag => 'dynamic ruby code'
+  
+    jobs Bj.submit array_of_commands, :priority => 451 
     
   when jobs are run, they are run in RAILS_ROOT.  various attributes are
   available *only* once the job has finished.  you can check whether or not a
@@ -110,6 +273,21 @@ DESCRIPTION
       end
   
   See lib/bj/api.rb for more details.
+  
+  ________________________________
+  Sponsors 
+  --------------------------------
+    http://quintess.com/
+    http://www.engineyard.com/
+    http://igicom.com/
+    http://eparklabs.com/
+  
+    http://your_company.com/      <<-- (targeted marketing aimed at *you*)
+  
+  ________________________________
+  Version
+  --------------------------------
+  1.0.1
 
 PARAMETERS
   --rails_root=rails_root, -R (0 ~> rails_root=) 

data/bin/bj

@@ -277,8 +277,9 @@ Main {
     ________________________________
     Sponsors 
     --------------------------------
-      http://www.engineyard.com/
       http://quintess.com/
+      http://www.engineyard.com/
+      http://igicom.com/
       http://eparklabs.com/
 
       http://your_company.com/      <<-- (targeted marketing aimed at *you*)

data/gemspec.rb

@@ -12,11 +12,11 @@ Gem::Specification::new do |spec|
   spec.executables = Dir::glob("bin/*").map{|exe| File::basename exe}
   
   spec.require_path = "lib" 
-  spec.autorequire = lib
+  #spec.autorequire = lib
 
   spec.has_rdoc = File::exist? "doc" 
   spec.test_suite_file = "test/#{ lib }.rb" if File::directory? "test"
-  spec.add_dependency 'attributes', '>= 5.0.0'
+  #spec.add_dependency 'attributes', '>= 5.0.0'
   spec.add_dependency 'main', '>= 2.6.0'
   spec.add_dependency 'systemu', '>= 1.2.0'
   spec.add_dependency 'orderedhash', '>= 0.0.3'
@@ -26,4 +26,5 @@ Gem::Specification::new do |spec|
   spec.author = "Ara T. Howard"
   spec.email = "ara.t.howard@gmail.com"
   spec.homepage = "http://codeforpeople.com/lib/ruby/#{ lib }/"
+  spec.rubyforge_project = "codeforpeople"
 end

data/lib/bj.rb

@@ -3,7 +3,7 @@ unless defined? Bj
   #
   # constants and associated attrs
   #
-    Bj::VERSION = "1.0.0" #unless defined? Bj::VERSION
+    Bj::VERSION = "1.0.1" #unless defined? Bj::VERSION
     def self.version() Bj::VERSION end
 
     Bj::LIBDIR = File.expand_path(File::join(File.dirname(__FILE__), "bj")) + File::SEPARATOR unless
@@ -46,7 +46,8 @@ unless defined? Bj
   #
   # rubyforge/remote or local/lib 
   #
-    %w[ attributes systemu orderedhash ].each do |lib|
+    #%w[ attributes systemu orderedhash ].each do |lib|
+    %w[ systemu orderedhash ].each do |lib|
       begin
         require lib
       rescue
@@ -56,6 +57,7 @@ unless defined? Bj
   #
   # local 
   #
+    load libdir("attributes.rb")
     load libdir("stdext.rb")
     load libdir("util.rb")
     load libdir("errors.rb")

data/lib/bj/attributes.rb

@@ -0,0 +1,120 @@
+module Attributes
+  Attributes::VERSION = '5.0.0' unless defined? Attributes::VERSION
+  def self.version() Attributes::VERSION end
+
+  class List < ::Array
+    def << element
+      super
+      self
+    ensure
+      uniq!
+      index!
+    end
+    def index!
+      @index ||= Hash.new
+      each{|element| @index[element] = true}
+    end
+    def include? element
+      @index ||= Hash.new
+      @index[element] ? true : false
+    end
+    def initializers
+      @initializers ||= Hash.new
+    end
+  end
+
+  def attributes *a, &b
+    unless a.empty?
+      returned = Hash.new
+
+      hashes, names = a.partition{|x| Hash === x}
+      names_and_defaults = {}
+      hashes.each{|h| names_and_defaults.update h}
+      names.flatten.compact.each{|name| names_and_defaults.update name => nil}
+
+      initializers = __attributes__.initializers
+
+      names_and_defaults.each do |name, default|
+        raise NameError, "bad instance variable name '@#{ name }'" if "@#{ name }" =~ %r/[!?=]$/o
+        name = name.to_s
+
+        initialize = b || lambda { default }
+        initializer = lambda do |this|
+          Object.instance_method('instance_eval').bind(this).call &initialize
+        end
+        initializer_id = initializer.object_id
+        __attributes__.initializers[name] = initializer
+
+        module_eval <<-code
+          def #{ name }=(*value, &block)
+            value.unshift block if block
+            @#{ name } = value.first
+          end
+        code
+
+        module_eval <<-code
+          def #{ name }(*value, &block)
+            value.unshift block if block
+            return self.send('#{ name }=', value.first) unless value.empty?
+            #{ name }! unless defined? @#{ name }
+            @#{ name }
+          end
+        code
+
+        module_eval <<-code
+          def #{ name }!
+            initializer = ObjectSpace._id2ref #{ initializer_id }
+            self.#{ name } = initializer.call(self)
+            @#{ name }
+          end
+        code
+
+        module_eval <<-code
+          def #{ name }?
+            #{ name }
+          end
+        code
+
+        attributes << name
+        returned[name] = initializer 
+      end
+
+      returned
+    else
+      begin
+        __attribute_list__
+      rescue NameError
+        singleton_class =
+          class << self
+            self
+          end
+        klass = self
+        singleton_class.module_eval do
+          attribute_list = List.new 
+          define_method('attribute_list'){ klass == self ? attribute_list : raise(NameError) }
+          alias_method '__attribute_list__', 'attribute_list'
+        end
+        __attribute_list__
+      end
+    end
+  end
+
+  %w( __attributes__ __attribute__ attribute ).each{|dst| alias_method dst, 'attributes'}
+end
+
+=begin
+class Object
+  def attributes *a, &b
+    sc = 
+      class << self
+        self
+      end
+    sc.attributes *a, &b
+  end
+  %w( __attributes__ __attribute__ attribute ).each{|dst| alias_method dst, 'attributes'}
+end
+=end
+
+class Module
+  include Attributes
+end

metadata

@@ -1,80 +1,21 @@
 --- !ruby/object:Gem::Specification 
-rubygems_version: 0.9.4
-specification_version: 1
 name: bj
 version: !ruby/object:Gem::Version 
-  version: 1.0.0
-date: 2007-12-28 00:00:00 -07:00
-summary: bj
-require_paths: 
-- lib
-email: ara.t.howard@gmail.com
-homepage: http://codeforpeople.com/lib/ruby/bj/
-rubyforge_project: 
-description: 
-autorequire: bj
-default_executable: 
-bindir: bin
-has_rdoc: false
-required_ruby_version: !ruby/object:Gem::Version::Requirement 
-  requirements: 
-  - - ">"
-    - !ruby/object:Gem::Version 
-      version: 0.0.0
-  version: 
+  version: 1.0.1
 platform: ruby
-signing_key: 
-cert_chain: 
-post_install_message: 
 authors: 
 - Ara T. Howard
-files: 
-- bin
-- bin/bj
-- gemspec.rb
-- HISTORY
-- install.rb
-- lib
-- lib/bj
-- lib/bj/api.rb
-- lib/bj/bj.rb
-- lib/bj/errors.rb
-- lib/bj/joblist.rb
-- lib/bj/logger.rb
-- lib/bj/runner.rb
-- lib/bj/stdext.rb
-- lib/bj/table.rb
-- lib/bj/util.rb
-- lib/bj.rb
-- rakefile
-- README
-- TODO
-test_files: []
-
-rdoc_options: []
-
-extra_rdoc_files: []
-
-executables: 
-- bj
-extensions: []
-
-requirements: []
+autorequire: 
+bindir: bin
+cert_chain: []
 
+date: 2008-01-08 00:00:00 -07:00
+default_executable: 
 dependencies: 
-- !ruby/object:Gem::Dependency 
-  name: attributes
-  version_requirement: 
-  version_requirements: !ruby/object:Gem::Version::Requirement 
-    requirements: 
-    - - ">="
-      - !ruby/object:Gem::Version 
-        version: 5.0.0
-    version: 
 - !ruby/object:Gem::Dependency 
   name: main
   version_requirement: 
-  version_requirements: !ruby/object:Gem::Version::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
@@ -83,7 +24,7 @@ dependencies:
 - !ruby/object:Gem::Dependency 
   name: systemu
   version_requirement: 
-  version_requirements: !ruby/object:Gem::Version::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
@@ -92,9 +33,67 @@ dependencies:
 - !ruby/object:Gem::Dependency 
   name: orderedhash
   version_requirement: 
-  version_requirements: !ruby/object:Gem::Version::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
         version: 0.0.3
     version: 
+description: 
+email: ara.t.howard@gmail.com
+executables: 
+- bj
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- bin
+- bin/bj
+- gemspec.rb
+- HISTORY
+- install.rb
+- lib
+- lib/bj
+- lib/bj/api.rb
+- lib/bj/attributes.rb
+- lib/bj/bj.rb
+- lib/bj/errors.rb
+- lib/bj/joblist.rb
+- lib/bj/logger.rb
+- lib/bj/runner.rb
+- lib/bj/stdext.rb
+- lib/bj/table.rb
+- lib/bj/util.rb
+- lib/bj.rb
+- rakefile
+- README
+- TODO
+has_rdoc: false
+homepage: http://codeforpeople.com/lib/ruby/bj/
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: codeforpeople
+rubygems_version: 1.0.1
+signing_key: 
+specification_version: 2
+summary: bj
+test_files: []
+