SciMed-bj 1.2.4

data/TODO

@@ -0,0 +1,53 @@
+
+- fix TZ bug
+
+- fix doc bug regarding runners
+
+- crontab option
+
+- make tickle easier
+
+- eval option
+
+- flesh out the cli interface - it's a test only at this point
+
+- commands need quoting, esp for windows, "c:\Documents And..." etc
+
+
+
+
+- the whole gem_path thing is still fubar
+
+- signals not operating properly on windows , non critical error tho...
+
+- need to figure out how to cache connections for Bj.in(...)
+
+- ttl will be added. maxing it out will cause auto-resubmission (Steve Midgley)
+
+- is having the runner thread try forever to start the process the best thing?
+
+- allow easy way to run ruby code.  perhaps ./script/runner 'eval STDIN.read'
+  is good enough
+
+- allow easy way to run ruby code that persists
+
+- allow specification of runner on submit (--runner)
+
+- allow specification of tags a runner will consume (--tag)
+
+- test in windows
+
+================================================================================
+
+X ./script/console submission hangs on windows
+X default PATH setting
+X install issues for dave? - gem_path...
+X main only loaded for (bin|script)/bj
+X make it possible to declare externally managed runners
+X restartable will be added. true by default (Steve Midgley)
+X do the lifeline inline with the loop
+X need to address the serialzable writer issue (:lock => true ??)
+X migrations use --force
+X i forget to add "#{ Bj.ruby } ... " to the generate command
+X ./script/bj must be found in path before c:/.....bin/bj
+X make sure database.yml is loaded via YAML::load(ERB.new(File.read * "config/database.yml").result)

data/VERSION

@@ -0,0 +1 @@
+1.2.4

data/bin/bj

@@ -0,0 +1,692 @@
+#! /usr/bin/env ruby
+
+require "bj"
+require "main"
+
+Main {
+  usage["description"] = <<-txt
+    ________________________________
+    Overview
+    --------------------------------
+    
+    Backgroundjob (Bj) is a brain dead simple, zero admin, background priority
+    queue for Rails.  Bj is robust, platform independent (including winblows),
+    and supports internal or external manangement of the background runner
+    process.
+    
+    Jobs can be submitted to the queue directly using the api or from the
+    command line using the ./script/bj:
+    
+      api:
+        Bj.submit 'cat /etc/password'
+    
+      command line:
+        bj submit cat /etc/password
+    
+    Bj's priority queue lives in the database and is therefore durable - your
+    jobs will live across an app crash or machine reboot.  The job management is
+    comprehensive - capturing stdout, stderr, exit_status, and temporal
+    statistics about each job:
+    
+      jobs =
+        Bj.submit command_or_array_of_commands, :priority => 42
+    
+      ...
+    
+      jobs.each do |job|
+        if job.finished?
+          p job.stdout
+          p job.stderr
+          p job.exit_status
+          p job.started_at
+          p job.finished_at
+        end
+      end
+    
+    In addition the background runner process logs all commands run and their
+    exit_status to a log named using the following convention:
+    
+      rails_root/log/bj.\#{ HOSTNAME }.\#{ RAILS_ENV }.log
+    
+    Bj allows you to submit jobs to multiple databases; for instance, if your
+    application is running in development mode you may do:
+    
+      Bj.in :production do
+        Bj.submit 'my_job.exe'
+      end
+    
+    Bj manages the ever growing list of jobs ran by automatically archiving them
+    into another table (by default jobs > 24 hrs old are archived) to prevent the
+    jobs table from becoming bloated and huge.
+    
+    All Bj's tables are namespaced and accessible via the Bj module:
+    
+      Bj.table.job.find(:all)         # jobs table
+      Bj.table.job_archive.find(:all) # archived jobs
+      Bj.table.config.find(:all)      # configuration and runner state
+    
+    Bj always arranges for submitted jobs to run with a current working
+    directory of RAILS_ROOT and with the correct RAILS_ENV setting.  For
+    example, if you submit a job in production it will have ENV['RAILS_ENV'] ==
+    'production' when the job is ran.
+    
+    When Bj manages the background runner it will never outlive the rails
+    application - it is started and stopped on demand in parallel with the rails
+    app parent process.  This is also true for ./script/console - Bj will
+    automatically fire off the background runner to process jobs submitted using
+    the console iff needed.
+    
+    Bj starts *one process per server*.  For instance, if you are running 3
+    mongrels via mongrel cluster Bj will automatically fire up *one background
+    process* per mongrel.  This helps you scale background processes in a
+    relatively POLS fashion, but note that the number of background runners does
+    not *really* determine throughput - that is determined primarily by the
+    nature of the jobs themselves and how much work they perform per process,
+    though having more background processes can also help.
+    
+    
+    ________________________________
+    Architecture
+    --------------------------------
+    
+    If one ignores platform specific details the design of Bj is quite simple: the
+    main Rails application submits jobs to table, stored in the database.  The act
+    of submitting triggers exactly one of two things to occur:
+    
+      1) a new long running background runner to be started
+    
+      2) an existing background runner to be signaled
+    
+    The background runner refuses to run two copies of itself for a given
+    hostname/rails_env combination.  For example you may only have one background
+    runner processing jobs on localhost in development mode.
+    
+    The background runner, under normal circumstances, is managed by Bj itself -
+    you need do nothing to start, monitor, or stop it - it just works.  However,
+    some people will prefer manage their own background process, see 'External
+    Runner' section below for more on this.
+    
+    The runner simply processes each job in a highest priority oldest-in fashion,
+    capturing stdout, stderr, exit_status, etc. and storing the information back
+    into the database while logging it's actions.  When there are no jobs to run
+    the runner goes to sleep for 42 seconds; however this sleep is interuptable,
+    such as when the runner is signaled that a new job has been submitted so,
+    under normal circumstances there will be zero lag between job submission and
+    job running for an empty queue.
+    
+    
+    ________________________________
+    External Runner / Clustering
+    --------------------------------
+    
+    For the paranoid control freaks out there (myself included) it is quite
+    possible to manage and monitor the runner process manually.  This can be
+    desirable in production setups where monitoring software may kill leaking
+    rails apps periodically or in any case where you want to montior and control
+    the background process directly.
+    
+    Recalling that Bj will only allow one copy of itself to process jobs per
+    hostname/rails_env pair we can simply do something like this in cron
+    
+      cmd = bj run --forever \
+                   --rails_env=development \
+                   --rails_root=/Users/ahoward/rails_root
+    
+      */15 * * * * $cmd 
+    
+    this will simply attempt the start the background runner every 15 minutes if,
+    and only if, it's not *already* running.
+    
+    In addtion to this you'll want to tell Bj not to manage the runner itself
+    using
+    
+      Bj.config["production.no_tickle"] = true
+
+    For instance in ./config/initializer/bj.rb or ./config/environment.rb.
+    
+    Note that, for clusting setups, it's as simple as adding a crontab and
+    config entry like this for each host you app is running on.  Because Bj
+    throttles background runners per hostname this will allow one runner per
+    hostname - making it quite simple to cluster three nodes behind a besieged
+    rails application.
+    
+    
+    ________________________________
+    Designing Jobs
+    --------------------------------
+    
+    Bj runs it's jobs as command line applications.  It ensures that all jobs run
+    in RAILS_ROOT so it's quite natural to apply a pattern such as
+    
+      mkdir ./jobs
+      edit ./jobs/background_job_to_run
+    
+      ...
+    
+      Bj.submit "./jobs/background_job_to_run"
+    
+    If you need to run you jobs under an entire rails environment you'll need to
+    do this:
+    
+      Bj.submit "./script/runner ./jobs/background_job_to_run"
+    
+    Obviously "./script/runner" loads the rails environment for you.  It's worth
+    noting that this happens for each job and that this is by design: the reason
+    is that most rails applications leak memory like a sieve so, if one were to
+    spawn a long running process that used the application code base you'd have a
+    lovely doubling of memory usage on you app servers.  Although loading the
+    rails environment for each background job requires a little time, a little
+    cpu, and a lot less memory.  A future version of Bj will provide a way to load
+    the rails environment once and to process background jobs in this environment,
+    but anyone wanting to use this in production will be required to duct tape
+    their entire chest and have a team of oxen rip off the tape without screaming
+    to prove steelyness of spirit and profound understanding of the other side.
+    
+    Don't forget that you can submit jobs with command line arguments:
+    
+      Bj.submit "./jobs/a.rb 1 foobar --force" 
+    
+    and that you can do powerful things by passing stdin to a job that powers
+    through a list of work.  For instance, assume a "./jobs/bulkmail" job
+    resembling
+    
+      STDIN.each do |line|
+        address = line.strip
+        mail_message_to address
+      end
+      
+    then you could
+    
+      stdin = [
+        "foo@bar.com",
+        "bar@foo.com",
+        "ara.t.howard@codeforpeople.com",
+      ]
+    
+      Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
+    
+    and all those emails would be sent in the background.
+    
+    Bj's power is putting jobs in the background in a simple and robust fashion.
+    It's your task to build intelligent jobs that leverage batch processing, and
+    other, possibilities.  The upshot of building tasks this way is that they are
+    quite easy to test before submitting them from inside your application.
+    
+    
+    ________________________________
+    Install 
+    --------------------------------
+      
+      Bj can be installed two ways: as a plugin or via rubygems
+    
+        plugin:
+          1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
+          2) ./script/bj setup
+    
+        gem:
+          1) $sudo gem install bj
+          2) add "require 'bj'" to config/environment.rb
+          3) bj setup
+    
+    ________________________________
+    Api
+    --------------------------------
+    
+    submit jobs for background processing.  'jobs' can be a string or array of
+    strings.  options are applied to each job in the 'jobs', and the list of
+    submitted jobs is always returned.  options (string or symbol) can be
+      
+      :rails_env => production|development|key_in_database_yml 
+                    when given this keyword causes bj to submit jobs to the
+                    specified database.  default is RAILS_ENV.
+    
+      :priority => any number, including negative ones.  default is zero.
+    
+      :tag => a tag added to the job.  simply makes searching easier.
+    
+      :env => a hash specifying any additional environment vars the background
+              process should have.
+    
+      :stdin => any stdin the background process should have.  must respond_to
+                to_s
+      
+    eg:
+    
+      jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+    
+      jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+    
+      jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+    
+      jobs = Bj.submit './script/runner /dev/stdin',
+                       :stdin => 'p RAILS_ENV',
+                       :tag => 'dynamic ruby code'
+    
+      jobs Bj.submit array_of_commands, :priority => 451 
+      
+    when jobs are run, they are run in RAILS_ROOT.  various attributes are
+    available *only* once the job has finished.  you can check whether or not a
+    job is finished by using the #finished method, which simple does a reload and
+    checks to see if the exit_status is non-nil.
+      
+      eg:
+      
+        jobs = Bj.submit list_of_jobs, :tag => 'important'
+        ...
+        
+        jobs.each do |job|
+          if job.finished?
+            p job.exit_status
+            p job.stdout
+            p job.stderr
+          end
+        end
+    
+    See lib/bj/api.rb for more details.
+    
+    ________________________________
+    Sponsors 
+    --------------------------------
+      http://quintess.com/
+      http://www.engineyard.com/
+      http://igicom.com/
+      http://eparklabs.com/
+    
+      http://your_company.com/      <<-- (targeted marketing aimed at *you*)
+    
+    ________________________________
+    Version
+    --------------------------------
+    1.0.1
+  txt
+
+  usage["uris"] = <<-txt
+    http://codeforpeople.com/lib/ruby/
+    http://rubyforge.org/projects/codeforpeople/
+    http://codeforpeople.rubyforge.org/svn/rails/plugins/
+  txt
+
+  author "ara.t.howard@gmail.com"
+
+  option("rails_root", "R"){
+    description "the rails_root will be guessed unless you set this"
+    argument_required
+    default RAILS_ROOT
+  }
+
+  option("rails_env", "E"){
+    description "set the rails_env"
+    argument_required
+    default RAILS_ENV
+  }
+
+  option("log", "l"){
+    description "set the logfile"
+    argument_required
+    default STDERR
+  }
+
+
+  mode "migration_code" do
+    description "dump migration code on stdout"
+
+    def run
+      puts Bj.table.migration_code
+    end
+  end
+
+  mode "generate_migration" do
+    description "generate a migration"
+
+    def run
+      Bj.generate_migration
+    end
+  end
+
+  mode "migrate" do
+    description "migrate the db"
+
+    def run
+      Bj.migrate
+    end
+  end
+
+  mode "setup" do
+    description "generate a migration and migrate"
+
+    def run
+      set_rails_env(argv.first) if argv.first
+      Bj.setup
+    end
+  end
+
+  mode "plugin" do
+    description "dump the plugin into rails_root"
+
+    def run
+      Bj.plugin
+    end
+  end
+
+  mode "run" do
+    description "start a job runnner, possibly as a daemon"
+
+    option("--forever"){}
+    option("--ppid"){
+      argument :required
+      cast :integer
+    }
+    option("--wait"){
+      argument :required
+      cast :integer
+    }
+    option("--limit"){
+      argument :required
+      cast :integer
+    }
+    option("--redirect"){
+      argument :required
+    }
+    option("--daemon"){}
+
+    def run
+      options = {}
+
+=begin
+      %w[ forever ].each do |key|
+        options[key.to_sym] = true if param[key].given?
+      end
+=end
+
+      %w[ forever ppid wait limit ].each do |key|
+        options[key.to_sym] = param[key].value if param[key].given?
+      end
+
+#p options
+#exit
+      if param["redirect"].given?
+        open(param["redirect"].value, "a+") do |fd|
+          STDERR.reopen fd 
+          STDOUT.reopen fd 
+        end
+        STDERR.sync = true
+        STDOUT.sync = true
+      end
+
+      trap("SIGTERM"){
+        info{ "SIGTERM" }
+        exit
+      }
+
+      if param["daemon"].given?
+        daemon{ Bj.run options }
+      else
+        Bj.run options 
+      end
+    end
+  end
+
+  mode "submit" do
+    keyword("file"){
+      argument :required
+      attr
+    }
+
+    def run
+      joblist = Bj.joblist.for argv.join(' ') 
+
+      case file
+        when "-"
+          joblist.push(Bj.joblist.jobs_from_io(STDIN))
+        when "--", "---"
+          joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
+        else
+          open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
+      end
+
+      jobs = Bj.submit joblist, :no_tickle => true
+
+      oh = lambda{|job| OrderedHash["id", job.id, "command", job.command]}
+
+      y jobs.map{|job| oh[job]} 
+    end
+  end
+
+  mode "list" do
+    def run
+      Bj.transaction do
+        y Bj::Table::Job.find(:all).map(&:to_hash)
+      end
+    end
+  end
+
+  mode "set" do
+    argument("key"){ attr }
+
+    argument("value"){ attr }
+
+    option("hostname", "H"){
+      argument :required
+      default Bj.hostname
+      attr
+    }
+
+    option("cast", "c"){
+      argument :required
+      default "to_s" 
+      attr
+    }
+
+    def run
+      Bj.transaction do
+        Bj.config.set(key, value, :hostname => hostname, :cast => cast)
+        y Bj.table.config.for(:hostname => hostname)
+      end
+    end
+  end
+
+  mode "config" do
+    option("hostname", "H"){
+      argument :required
+      default Bj.hostname
+    }
+
+    def run
+      Bj.transaction do
+        y Bj.table.config.for(:hostname => param["hostname"].value)
+      end
+    end
+  end
+
+  mode "pid" do
+    option("hostname", "H"){
+      argument :required
+      default Bj.hostname
+    }
+
+    def run
+      Bj.transaction do
+        config = Bj.table.config.for(:hostname => param["hostname"].value)
+        puts config[ "#{ RAILS_ENV }.pid" ] if config
+      end
+    end
+  end
+
+
+  def run 
+    help!
+  end
+
+  def before_run 
+    self.logger = param["log"].value if param["log"].given?
+    Bj.logger = logger
+    set_rails_root(param["rails_root"].value) if param["rails_root"].given?
+    set_rails_env(param["rails_env"].value) if param["rails_env"].given?
+  end
+
+  def set_rails_root rails_root
+    ENV["RAILS_ROOT"] = rails_root 
+    ::Object.instance_eval do
+      remove_const :RAILS_ROOT
+      const_set :RAILS_ROOT, rails_root
+    end
+  end
+
+  def set_rails_env rails_env
+    ENV["RAILS_ENV"] = rails_env 
+    ::Object.instance_eval do
+      remove_const :RAILS_ENV
+      const_set :RAILS_ENV, rails_env
+    end
+  end
+
+  def daemon
+    ra, wa = IO.pipe
+    rb, wb = IO.pipe
+    if fork
+      at_exit{ exit! }
+      wa.close
+      r = ra
+      rb.close
+      w = wb
+      pid = r.gets
+      w.puts pid
+      Integer pid.strip
+    else
+      ra.close
+      w = wa
+      wb.close
+      r = rb
+      open("/dev/null", "r+") do |fd|
+        STDIN.reopen fd 
+        STDOUT.reopen fd 
+        STDERR.reopen fd 
+      end
+      Process::setsid rescue nil
+      pid =
+        fork do
+          Dir::chdir RAILS_ROOT
+          File::umask 0
+          $DAEMON = true
+          yield
+          exit!
+        end
+      w.puts pid
+      r.gets
+      exit!
+    end
+  end
+}
+
+
+
+
+
+#
+# we setup a few things so the script works regardless of whether it was
+# called out of /usr/local/bin, ./script, or wherever.  note that the script
+# does *not* require the entire rails application to be loaded into memory!
+# we could just load boot.rb and environment.rb, but this method let's
+# submitting and running jobs be infinitely more lightweight.
+#
+
+BEGIN {
+#
+# see if we're running out of RAILS_ROOT/script/
+#
+unless defined?(BJ_SCRIPT)
+  BJ_SCRIPT =
+    if %w[ script config app ].map{|d| test ?d, "#{ File.dirname __FILE__ }/../#{ d }"}.all?
+      __FILE__
+    else
+      nil
+    end
+end
+#
+# setup RAILS_ROOT
+#
+unless defined?(RAILS_ROOT)
+  ### grab env var first
+  rails_root = ENV["RAILS_ROOT"]
+
+  ### commandline usage clobbers
+  kv = nil
+  ARGV.delete_if{|arg| arg =~ %r/^RAILS_ROOT=/ and kv = arg}
+  rails_root = kv.split(%r/=/,2).last if kv
+
+  ### we know the rails_root if we are in RAILS_ROOT/script/ 
+  unless rails_root 
+    if BJ_SCRIPT
+      rails_root = File.expand_path "#{ File.dirname __FILE__ }/.."
+    end
+  end
+
+  ### perhaps the current directory is a rails_root?
+  unless rails_root 
+    if %w[ script config app ].map{|d| test(?d, d)}.all?
+      rails_root = File.expand_path "."
+    end
+  end
+
+  ### bootstrap 
+  RAILS_ROOT = rails_root
+end
+#
+# setup RAILS_ENV
+#
+unless defined?(RAILS_ENV)
+  ### grab env var first
+  rails_env = ENV["RAILS_ENV"]
+
+  ### commandline usage clobbers
+  kv = nil
+  ARGV.delete_if{|arg| arg =~ %r/^RAILS_ENV=/ and kv = arg}
+  rails_env = kv.split(%r/=/,2).last if kv
+
+  ### fallback to development
+  unless rails_env
+    rails_env = "development"
+  end 
+
+  ### bootstrap 
+  RAILS_ENV = rails_env
+end
+#
+# ensure that rubygems is loaded
+#
+  begin
+    require "rubygems"
+  rescue
+    42
+  end
+#
+# load gems from plugin dir iff installed as plugin - otherwise load normally
+#
+if RAILS_ROOT and BJ_SCRIPT
+=begin
+  dir = Gem.dir
+  path = Gem.path
+  gem_home = File.join RAILS_ROOT, "vendor", "plugins", "bj", "gem_home"
+  gem_path = [gem_home]
+  Gem.send :use_paths, gem_home, gem_path
+  gem "bj"
+  require "bj"
+  gem "main"
+  require "main"
+=end
+  libdir = File.join(RAILS_ROOT, "vendor", "plugins", "bj", "lib")
+  $LOAD_PATH.unshift libdir
+end
+#
+# hack of #to_s of STDERR/STDOUT for nice help messages
+#
+  class << STDERR
+    def to_s() 'STDERR' end
+  end
+  class << STDOUT
+    def to_s() 'STDOUT' end
+  end
+}
+
+
+__END__