bj_fixed_for_rails3 1.0.2

data/HISTORY

@@ -0,0 +1,64 @@
+  1.0.1:
+    - fixed name collision with 'record.attributes = hash' ar mass
+      assignment method (thx jon guymon)
+    - added new sponsor: http://igicom.com/
+
+  0.0.5:
+    - use full path to ruby for plugin mode
+    - plugin correctly installs bin -->> script
+    - plugin install uses --force
+    - properly quote paths in windows (spaces)
+    - switch win signal to ABRT (was INT)
+    - background job regrestration now uses ppid to pin the subprocess to a
+      parent
+    - use ppid to detect parent death and exit in event loop 
+    - don't use gem dependanices in plugin as they are broken when loading from
+      muliple gem repos
+    - added a small amount of drb magic that allows signals to work across
+      processes even on windows (see http://drawohara.com/post/22540307)
+  
+0.0.4:
+  - basic functionality in windows
+  - several small bug fixes
+
+0.0.3:
+  - *many* small bug fixes
+
+  - plugin install should now pick up dependancies from plugin dir, last
+    release had LOAD_PATH/gem issues and was picking up globally installed
+    gems
+
+  - automatic management of the background processing can be turned off if you
+    want to manage your own processes 
+
+  - all jobs are automatically restartable unless submitted with 
+
+       :restartable => false
+
+    this means that, should a runner ever die, upon restart any jobs that were
+    mid-process will automatically be restarted
+
+  - signal based parent lifeline move out of thread and into even loop
+
+  - :lock => true added to a few AR finds to support true write serializable
+    transaction isolation when the db supports it
+
+  - all migrations now use :force => true and
+
+  - running 'bj setup' will always generate a new migration, even if you've
+    already run it before.  this allows easy version upgrades.
+
+  - a few command would blow up on windows because they weren't prefixed with
+    'ruby'.  gotta love the lack of #shebang line on windoze...
+
+  - ./script/bj is searched first before system path env var
+
+  - a default PATH is provided for whacky systems without one
+
+  - database.yml is filtered through ERB ala rails
+  
+0.0.2:
+  - path bug fixes
+
+0.0.1:
+  - initial release

data/README

@@ -0,0 +1,308 @@
+NAME
+  bj
+
+SYNOPSIS
+  bj (migration_code|generate_migration|migrate|setup|plugin|run|submit|list|set|config|pid) [options]+
+
+DESCRIPTION
+  ________________________________
+  Overview
+  --------------------------------
+  
+  Backgroundjob (Bj) is a brain dead simple zero admin background priority queue
+  for Rails.  Bj is robust, platform independent (including windows), and
+  supports internal or external manangement of the background runner process.
+  
+  Jobs can be submitted to the queue directly using the api or from the command
+  line using the ./script/bj:
+  
+    api:
+      Bj.submit 'cat /etc/password'
+  
+    command line:
+      bj submit cat /etc/password
+  
+  Bj's priority queue lives in the database and is therefore durable - your jobs
+  will live across an app crash or machine reboot.  The job management is
+  comprehensive capturing stdout, stderr, exit_status, and temporal statistics
+  about each job:
+  
+    jobs = Bj.submit array_of_commands, :priority => 42
+  
+    ...
+  
+    jobs.each do |job|
+      if job.finished?
+        p job.stdout
+        p job.stderr
+        p job.exit_status
+        p job.started_at
+        p job.finished_at
+      end
+    end
+  
+  In addition the background runner process logs all commands run and their
+  exit_status to a log named using the following convention:
+  
+    rails_root/log/bj.#{ HOSTNAME }.#{ RAILS_ENV }.log
+  
+  Bj allows you to submit jobs to multiple databases; for instance, if your
+  application is running in development mode you may do:
+  
+    Bj.in :production do
+      Bj.submit 'my_job.exe'
+    end
+  
+  Bj manages the ever growing list of jobs ran by automatically archiving them
+  into another table (by default jobs > 24 hrs old are archived) to prevent the
+  jobs table from becoming bloated and huge.
+  
+  All Bj's tables are namespaced and accessible via the Bj module:
+  
+    Bj.table.job.find(:all)         # jobs table
+    Bj.table.job_archive.find(:all) # archived jobs
+    Bj.table.config.find(:all)      # configuration and runner state
+  
+  Bj always arranges for submitted jobs to run with a current working directory
+  of RAILS_ROOT and with the correct RAILS_ENV setting.  For example, if you
+  submit a job in production it will have ENV['RAILS_ENV'] == 'production'.
+  
+  When Bj manages the background runner it will never outlive the rails
+  application - it is started and stopped on demand as the rails app is started
+  and stopped.  This is also true for ./script/console - Bj will automatically
+  fire off the background runner to process jobs submitted using the console.
+  
+  Bj ensures that only one background process is running for your application -
+  firing up three mongrels or fcgi processes will result in only one background
+  runner being started.  Note that the number of background runners does not
+  determine throughput - that is determined primarily by the nature of the jobs
+  themselves and how much work they perform per process.
+  
+  
+  ________________________________
+  Architecture
+  --------------------------------
+  
+  If one ignores platform specific details the design of Bj is quite simple: the
+  main Rails application submits jobs to table, stored in the database.  The act
+  of submitting triggers exactly one of two things to occur:
+  
+    1) a new long running background runner to be started
+  
+    2) an existing background runner to be signaled
+  
+  The background runner refuses to run two copies of itself for a given
+  hostname/rails_env combination.  For example you may only have one background
+  runner processing jobs on localhost in development mode.
+  
+  The background runner, under normal circumstances, is managed by Bj itself -
+  you need do nothing to start, monitor, or stop it - it just works.  However,
+  some people will prefer manage their own background process, see 'External
+  Runner' section below for more on this.
+  
+  The runner simply processes each job in a highest priority oldest-in fashion,
+  capturing stdout, stderr, exit_status, etc. and storing the information back
+  into the database while logging it's actions.  When there are no jobs to run
+  the runner goes to sleep for 42 seconds; however this sleep is interuptable,
+  such as when the runner is signaled that a new job has been submitted so,
+  under normal circumstances there will be zero lag between job submission and
+  job running for an empty queue.
+  
+  
+  ________________________________
+  External Runner / Clustering
+  --------------------------------
+  
+  For the paranoid control freaks out there (myself included) it is quite
+  possible to manage and monitor the runner process manually.  This can be
+  desirable in production setups where monitoring software may kill leaking
+  rails apps periodically.
+  
+  Recalling that Bj will only allow one copy of itself to process jobs per
+  hostname/rails_env pair we can simply do something like this in cron
+  
+    cmd = bj run --forever \
+                 --rails_env=development \
+                 --rails_root=/Users/ahoward/rails_root
+  
+    */15 * * * * $cmd 
+  
+  this will simply attempt the start the background runner every 15 minutes if,
+  and only if, it's not *already* running.
+  
+  In addtion to this you'll want to tell Bj not to manage the runner itself
+  using
+  
+    Bj.config["production.no_tickle"] = true
+  
+  Note that, for clusting setups, it's as simple as adding a crontab and config
+  entry like this for each host.  Because Bj throttles background runners per
+  hostname this will allow one runner per hostname - making it quite simple to
+  cluster three nodes behind a besieged rails application.
+  
+  
+  ________________________________
+  Designing Jobs
+  --------------------------------
+  
+  Bj runs it's jobs as command line applications.  It ensures that all jobs run
+  in RAILS_ROOT so it's quite natural to apply a pattern such as
+  
+    mkdir ./jobs
+    edit ./jobs/background_job_to_run
+  
+    ...
+  
+    Bj.submit "./jobs/background_job_to_run"
+  
+  If you need to run you jobs under an entire rails environment you'll need to
+  do this:
+  
+    Bj.submit "./script/runner ./jobs/background_job_to_run"
+  
+  Obviously "./script/runner" loads the rails environment for you.  It's worth
+  noting that this happens for each job and that this is by design: the reason
+  is that most rails applications leak memory like a sieve so, if one were to
+  spawn a long running process that used the application code base you'd have a
+  lovely doubling of memory usage on you app servers.  Although loading the
+  rails environment for each background job requires a little time, a little
+  cpu, and a lot less memory.  A future version of Bj will provide a way to load
+  the rails environment once and to process background jobs in this environment,
+  but anyone wanting to use this in production will be required to duct tape
+  their entire chest and have a team of oxen rip off the tape without screaming
+  to prove steelyness of spirit and profound understanding of the other side.
+  
+  Don't forget that you can submit jobs with command line arguments:
+  
+    Bj.submit "./jobs/a.rb 1 foobar --force" 
+  
+  and that you can do powerful things by passing stdin to a job that powers
+  through a list of work.  For instance, assume a "./jobs/bulkmail" job
+  resembling
+  
+    STDIN.each do |line|
+      address = line.strip
+      mail_message_to address
+    end
+    
+  then you could
+  
+    stdin = [
+      "foo@bar.com",
+      "bar@foo.com",
+      "ara.t.howard@codeforpeople.com",
+    ]
+  
+    Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
+  
+  and all those emails would be sent in the background.
+  
+  Bj's power is putting jobs in the background in a simple and robust fashion.
+  It's your task to build intelligent jobs that leverage batch processing, and
+  other, possibilities.  The upshot of building tasks this way is that they are
+  quite easy to test before submitting them from inside your application.
+  
+  
+  ________________________________
+  Install 
+  --------------------------------
+    
+    Bj can be installed two ways: as a plugin or via rubygems
+  
+      plugin:
+        1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
+        2) ./script/bj setup
+  
+      gem:
+        1) $sudo gem install bj
+        2) add "require 'bj'" to config/environment.rb
+        3) bj setup
+  
+  ________________________________
+  Api
+  --------------------------------
+  
+  submit jobs for background processing.  'jobs' can be a string or array of
+  strings.  options are applied to each job in the 'jobs', and the list of
+  submitted jobs is always returned.  options (string or symbol) can be
+    
+    :rails_env => production|development|key_in_database_yml 
+                  when given this keyword causes bj to submit jobs to the
+                  specified database.  default is RAILS_ENV.
+  
+    :priority => any number, including negative ones.  default is zero.
+  
+    :tag => a tag added to the job.  simply makes searching easier.
+  
+    :env => a hash specifying any additional environment vars the background
+            process should have.
+  
+    :stdin => any stdin the background process should have.  must respond_to
+              to_s
+    
+  eg:
+  
+    jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+  
+    jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+  
+    jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+  
+    jobs = Bj.submit './script/runner /dev/stdin',
+                     :stdin => 'p RAILS_ENV',
+                     :tag => 'dynamic ruby code'
+  
+    jobs Bj.submit array_of_commands, :priority => 451 
+    
+  when jobs are run, they are run in RAILS_ROOT.  various attributes are
+  available *only* once the job has finished.  you can check whether or not a
+  job is finished by using the #finished method, which simple does a reload and
+  checks to see if the exit_status is non-nil.
+    
+    eg:
+    
+      jobs = Bj.submit list_of_jobs, :tag => 'important'
+      ...
+      
+      jobs.each do |job|
+        if job.finished?
+          p job.exit_status
+          p job.stdout
+          p job.stderr
+        end
+      end
+  
+  See lib/bj/api.rb for more details.
+  
+  ________________________________
+  Sponsors 
+  --------------------------------
+    http://quintess.com/
+    http://www.engineyard.com/
+    http://igicom.com/
+    http://eparklabs.com/
+  
+    http://your_company.com/      <<-- (targeted marketing aimed at *you*)
+  
+  ________________________________
+  Version
+  --------------------------------
+  1.0.1
+
+PARAMETERS
+  --rails_root=rails_root, -R (0 ~> rails_root=) 
+      the rails_root will be guessed unless you set this 
+  --rails_env=rails_env, -E (0 ~> rails_env=development) 
+      set the rails_env 
+  --log=log, -l (0 ~> log=STDERR) 
+      set the logfile 
+  --help, -h 
+
+AUTHOR
+  ara.t.howard@gmail.com
+
+URIS
+  http://codeforpeople.com/lib/ruby/
+  http://rubyforge.org/projects/codeforpeople/
+  http://codeforpeople.rubyforge.org/svn/rails/plugins/
+

data/TODO

@@ -0,0 +1,40 @@
+
+? the whole gem_path thing is still fubar
+
+? commands need quoting, esp for windows, "c:\Documents And..." etc
+
+- signals not operating properly on windows , non critical error tho...
+
+- need to figure out how to cache connections for Bj.in(...)
+
+- ttl will be added. maxing it out will cause auto-resubmission (Steve Midgley)
+
+- is having the runner thread try forever to start the process the best thing?
+
+- allow easy way to run ruby code.  perhaps ./script/runner 'eval STDIN.read'
+  is good enough
+
+- allow easy way to run ruby code that persists
+
+- allow specification of runner on submit (--runner)
+
+- allow specification of tags a runner will consume (--tag)
+
+- flesh out the cli interface - it's a test only at this point
+
+- test in windows
+
+================================================================================
+
+X ./script/console submission hangs on windows
+X default PATH setting
+X install issues for dave? - gem_path...
+X main only loaded for (bin|script)/bj
+X make it possible to declare externally managed runners
+X restartable will be added. true by default (Steve Midgley)
+X do the lifeline inline with the loop
+X need to address the serialzable writer issue (:lock => true ??)
+X migrations use --force
+X i forget to add "#{ Bj.ruby } ... " to the generate command
+X ./script/bj must be found in path before c:/.....bin/bj
+X make sure database.yml is loaded via YAML::load(ERB.new(File.read * "config/database.yml").result)