SciMed-bj 1.2.4

Sign up to get free protection for your applications and to get access to all the features.
data/TODO ADDED
@@ -0,0 +1,53 @@
1
+
2
+ - fix TZ bug
3
+
4
+ - fix doc bug regarding runners
5
+
6
+ - crontab option
7
+
8
+ - make tickle easier
9
+
10
+ - eval option
11
+
12
+ - flesh out the cli interface - it's a test only at this point
13
+
14
+ - commands need quoting, esp for windows, "c:\Documents And..." etc
15
+
16
+
17
+
18
+
19
+ - the whole gem_path thing is still fubar
20
+
21
+ - signals not operating properly on windows , non critical error tho...
22
+
23
+ - need to figure out how to cache connections for Bj.in(...)
24
+
25
+ - ttl will be added. maxing it out will cause auto-resubmission (Steve Midgley)
26
+
27
+ - is having the runner thread try forever to start the process the best thing?
28
+
29
+ - allow easy way to run ruby code. perhaps ./script/runner 'eval STDIN.read'
30
+ is good enough
31
+
32
+ - allow easy way to run ruby code that persists
33
+
34
+ - allow specification of runner on submit (--runner)
35
+
36
+ - allow specification of tags a runner will consume (--tag)
37
+
38
+ - test in windows
39
+
40
+ ================================================================================
41
+
42
+ X ./script/console submission hangs on windows
43
+ X default PATH setting
44
+ X install issues for dave? - gem_path...
45
+ X main only loaded for (bin|script)/bj
46
+ X make it possible to declare externally managed runners
47
+ X restartable will be added. true by default (Steve Midgley)
48
+ X do the lifeline inline with the loop
49
+ X need to address the serialzable writer issue (:lock => true ??)
50
+ X migrations use --force
51
+ X i forget to add "#{ Bj.ruby } ... " to the generate command
52
+ X ./script/bj must be found in path before c:/.....bin/bj
53
+ X make sure database.yml is loaded via YAML::load(ERB.new(File.read * "config/database.yml").result)
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 1.2.4
data/bin/bj ADDED
@@ -0,0 +1,692 @@
1
+ #! /usr/bin/env ruby
2
+
3
+ require "bj"
4
+ require "main"
5
+
6
+ Main {
7
+ usage["description"] = <<-txt
8
+ ________________________________
9
+ Overview
10
+ --------------------------------
11
+
12
+ Backgroundjob (Bj) is a brain dead simple, zero admin, background priority
13
+ queue for Rails. Bj is robust, platform independent (including winblows),
14
+ and supports internal or external manangement of the background runner
15
+ process.
16
+
17
+ Jobs can be submitted to the queue directly using the api or from the
18
+ command line using the ./script/bj:
19
+
20
+ api:
21
+ Bj.submit 'cat /etc/password'
22
+
23
+ command line:
24
+ bj submit cat /etc/password
25
+
26
+ Bj's priority queue lives in the database and is therefore durable - your
27
+ jobs will live across an app crash or machine reboot. The job management is
28
+ comprehensive - capturing stdout, stderr, exit_status, and temporal
29
+ statistics about each job:
30
+
31
+ jobs =
32
+ Bj.submit command_or_array_of_commands, :priority => 42
33
+
34
+ ...
35
+
36
+ jobs.each do |job|
37
+ if job.finished?
38
+ p job.stdout
39
+ p job.stderr
40
+ p job.exit_status
41
+ p job.started_at
42
+ p job.finished_at
43
+ end
44
+ end
45
+
46
+ In addition the background runner process logs all commands run and their
47
+ exit_status to a log named using the following convention:
48
+
49
+ rails_root/log/bj.\#{ HOSTNAME }.\#{ RAILS_ENV }.log
50
+
51
+ Bj allows you to submit jobs to multiple databases; for instance, if your
52
+ application is running in development mode you may do:
53
+
54
+ Bj.in :production do
55
+ Bj.submit 'my_job.exe'
56
+ end
57
+
58
+ Bj manages the ever growing list of jobs ran by automatically archiving them
59
+ into another table (by default jobs > 24 hrs old are archived) to prevent the
60
+ jobs table from becoming bloated and huge.
61
+
62
+ All Bj's tables are namespaced and accessible via the Bj module:
63
+
64
+ Bj.table.job.find(:all) # jobs table
65
+ Bj.table.job_archive.find(:all) # archived jobs
66
+ Bj.table.config.find(:all) # configuration and runner state
67
+
68
+ Bj always arranges for submitted jobs to run with a current working
69
+ directory of RAILS_ROOT and with the correct RAILS_ENV setting. For
70
+ example, if you submit a job in production it will have ENV['RAILS_ENV'] ==
71
+ 'production' when the job is ran.
72
+
73
+ When Bj manages the background runner it will never outlive the rails
74
+ application - it is started and stopped on demand in parallel with the rails
75
+ app parent process. This is also true for ./script/console - Bj will
76
+ automatically fire off the background runner to process jobs submitted using
77
+ the console iff needed.
78
+
79
+ Bj starts *one process per server*. For instance, if you are running 3
80
+ mongrels via mongrel cluster Bj will automatically fire up *one background
81
+ process* per mongrel. This helps you scale background processes in a
82
+ relatively POLS fashion, but note that the number of background runners does
83
+ not *really* determine throughput - that is determined primarily by the
84
+ nature of the jobs themselves and how much work they perform per process,
85
+ though having more background processes can also help.
86
+
87
+
88
+ ________________________________
89
+ Architecture
90
+ --------------------------------
91
+
92
+ If one ignores platform specific details the design of Bj is quite simple: the
93
+ main Rails application submits jobs to table, stored in the database. The act
94
+ of submitting triggers exactly one of two things to occur:
95
+
96
+ 1) a new long running background runner to be started
97
+
98
+ 2) an existing background runner to be signaled
99
+
100
+ The background runner refuses to run two copies of itself for a given
101
+ hostname/rails_env combination. For example you may only have one background
102
+ runner processing jobs on localhost in development mode.
103
+
104
+ The background runner, under normal circumstances, is managed by Bj itself -
105
+ you need do nothing to start, monitor, or stop it - it just works. However,
106
+ some people will prefer manage their own background process, see 'External
107
+ Runner' section below for more on this.
108
+
109
+ The runner simply processes each job in a highest priority oldest-in fashion,
110
+ capturing stdout, stderr, exit_status, etc. and storing the information back
111
+ into the database while logging it's actions. When there are no jobs to run
112
+ the runner goes to sleep for 42 seconds; however this sleep is interuptable,
113
+ such as when the runner is signaled that a new job has been submitted so,
114
+ under normal circumstances there will be zero lag between job submission and
115
+ job running for an empty queue.
116
+
117
+
118
+ ________________________________
119
+ External Runner / Clustering
120
+ --------------------------------
121
+
122
+ For the paranoid control freaks out there (myself included) it is quite
123
+ possible to manage and monitor the runner process manually. This can be
124
+ desirable in production setups where monitoring software may kill leaking
125
+ rails apps periodically or in any case where you want to montior and control
126
+ the background process directly.
127
+
128
+ Recalling that Bj will only allow one copy of itself to process jobs per
129
+ hostname/rails_env pair we can simply do something like this in cron
130
+
131
+ cmd = bj run --forever \
132
+ --rails_env=development \
133
+ --rails_root=/Users/ahoward/rails_root
134
+
135
+ */15 * * * * $cmd
136
+
137
+ this will simply attempt the start the background runner every 15 minutes if,
138
+ and only if, it's not *already* running.
139
+
140
+ In addtion to this you'll want to tell Bj not to manage the runner itself
141
+ using
142
+
143
+ Bj.config["production.no_tickle"] = true
144
+
145
+ For instance in ./config/initializer/bj.rb or ./config/environment.rb.
146
+
147
+ Note that, for clusting setups, it's as simple as adding a crontab and
148
+ config entry like this for each host you app is running on. Because Bj
149
+ throttles background runners per hostname this will allow one runner per
150
+ hostname - making it quite simple to cluster three nodes behind a besieged
151
+ rails application.
152
+
153
+
154
+ ________________________________
155
+ Designing Jobs
156
+ --------------------------------
157
+
158
+ Bj runs it's jobs as command line applications. It ensures that all jobs run
159
+ in RAILS_ROOT so it's quite natural to apply a pattern such as
160
+
161
+ mkdir ./jobs
162
+ edit ./jobs/background_job_to_run
163
+
164
+ ...
165
+
166
+ Bj.submit "./jobs/background_job_to_run"
167
+
168
+ If you need to run you jobs under an entire rails environment you'll need to
169
+ do this:
170
+
171
+ Bj.submit "./script/runner ./jobs/background_job_to_run"
172
+
173
+ Obviously "./script/runner" loads the rails environment for you. It's worth
174
+ noting that this happens for each job and that this is by design: the reason
175
+ is that most rails applications leak memory like a sieve so, if one were to
176
+ spawn a long running process that used the application code base you'd have a
177
+ lovely doubling of memory usage on you app servers. Although loading the
178
+ rails environment for each background job requires a little time, a little
179
+ cpu, and a lot less memory. A future version of Bj will provide a way to load
180
+ the rails environment once and to process background jobs in this environment,
181
+ but anyone wanting to use this in production will be required to duct tape
182
+ their entire chest and have a team of oxen rip off the tape without screaming
183
+ to prove steelyness of spirit and profound understanding of the other side.
184
+
185
+ Don't forget that you can submit jobs with command line arguments:
186
+
187
+ Bj.submit "./jobs/a.rb 1 foobar --force"
188
+
189
+ and that you can do powerful things by passing stdin to a job that powers
190
+ through a list of work. For instance, assume a "./jobs/bulkmail" job
191
+ resembling
192
+
193
+ STDIN.each do |line|
194
+ address = line.strip
195
+ mail_message_to address
196
+ end
197
+
198
+ then you could
199
+
200
+ stdin = [
201
+ "foo@bar.com",
202
+ "bar@foo.com",
203
+ "ara.t.howard@codeforpeople.com",
204
+ ]
205
+
206
+ Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
207
+
208
+ and all those emails would be sent in the background.
209
+
210
+ Bj's power is putting jobs in the background in a simple and robust fashion.
211
+ It's your task to build intelligent jobs that leverage batch processing, and
212
+ other, possibilities. The upshot of building tasks this way is that they are
213
+ quite easy to test before submitting them from inside your application.
214
+
215
+
216
+ ________________________________
217
+ Install
218
+ --------------------------------
219
+
220
+ Bj can be installed two ways: as a plugin or via rubygems
221
+
222
+ plugin:
223
+ 1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
224
+ 2) ./script/bj setup
225
+
226
+ gem:
227
+ 1) $sudo gem install bj
228
+ 2) add "require 'bj'" to config/environment.rb
229
+ 3) bj setup
230
+
231
+ ________________________________
232
+ Api
233
+ --------------------------------
234
+
235
+ submit jobs for background processing. 'jobs' can be a string or array of
236
+ strings. options are applied to each job in the 'jobs', and the list of
237
+ submitted jobs is always returned. options (string or symbol) can be
238
+
239
+ :rails_env => production|development|key_in_database_yml
240
+ when given this keyword causes bj to submit jobs to the
241
+ specified database. default is RAILS_ENV.
242
+
243
+ :priority => any number, including negative ones. default is zero.
244
+
245
+ :tag => a tag added to the job. simply makes searching easier.
246
+
247
+ :env => a hash specifying any additional environment vars the background
248
+ process should have.
249
+
250
+ :stdin => any stdin the background process should have. must respond_to
251
+ to_s
252
+
253
+ eg:
254
+
255
+ jobs = Bj.submit 'echo foobar', :tag => 'simple job'
256
+
257
+ jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
258
+
259
+ jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
260
+
261
+ jobs = Bj.submit './script/runner /dev/stdin',
262
+ :stdin => 'p RAILS_ENV',
263
+ :tag => 'dynamic ruby code'
264
+
265
+ jobs Bj.submit array_of_commands, :priority => 451
266
+
267
+ when jobs are run, they are run in RAILS_ROOT. various attributes are
268
+ available *only* once the job has finished. you can check whether or not a
269
+ job is finished by using the #finished method, which simple does a reload and
270
+ checks to see if the exit_status is non-nil.
271
+
272
+ eg:
273
+
274
+ jobs = Bj.submit list_of_jobs, :tag => 'important'
275
+ ...
276
+
277
+ jobs.each do |job|
278
+ if job.finished?
279
+ p job.exit_status
280
+ p job.stdout
281
+ p job.stderr
282
+ end
283
+ end
284
+
285
+ See lib/bj/api.rb for more details.
286
+
287
+ ________________________________
288
+ Sponsors
289
+ --------------------------------
290
+ http://quintess.com/
291
+ http://www.engineyard.com/
292
+ http://igicom.com/
293
+ http://eparklabs.com/
294
+
295
+ http://your_company.com/ <<-- (targeted marketing aimed at *you*)
296
+
297
+ ________________________________
298
+ Version
299
+ --------------------------------
300
+ 1.0.1
301
+ txt
302
+
303
+ usage["uris"] = <<-txt
304
+ http://codeforpeople.com/lib/ruby/
305
+ http://rubyforge.org/projects/codeforpeople/
306
+ http://codeforpeople.rubyforge.org/svn/rails/plugins/
307
+ txt
308
+
309
+ author "ara.t.howard@gmail.com"
310
+
311
+ option("rails_root", "R"){
312
+ description "the rails_root will be guessed unless you set this"
313
+ argument_required
314
+ default RAILS_ROOT
315
+ }
316
+
317
+ option("rails_env", "E"){
318
+ description "set the rails_env"
319
+ argument_required
320
+ default RAILS_ENV
321
+ }
322
+
323
+ option("log", "l"){
324
+ description "set the logfile"
325
+ argument_required
326
+ default STDERR
327
+ }
328
+
329
+
330
+ mode "migration_code" do
331
+ description "dump migration code on stdout"
332
+
333
+ def run
334
+ puts Bj.table.migration_code
335
+ end
336
+ end
337
+
338
+ mode "generate_migration" do
339
+ description "generate a migration"
340
+
341
+ def run
342
+ Bj.generate_migration
343
+ end
344
+ end
345
+
346
+ mode "migrate" do
347
+ description "migrate the db"
348
+
349
+ def run
350
+ Bj.migrate
351
+ end
352
+ end
353
+
354
+ mode "setup" do
355
+ description "generate a migration and migrate"
356
+
357
+ def run
358
+ set_rails_env(argv.first) if argv.first
359
+ Bj.setup
360
+ end
361
+ end
362
+
363
+ mode "plugin" do
364
+ description "dump the plugin into rails_root"
365
+
366
+ def run
367
+ Bj.plugin
368
+ end
369
+ end
370
+
371
+ mode "run" do
372
+ description "start a job runnner, possibly as a daemon"
373
+
374
+ option("--forever"){}
375
+ option("--ppid"){
376
+ argument :required
377
+ cast :integer
378
+ }
379
+ option("--wait"){
380
+ argument :required
381
+ cast :integer
382
+ }
383
+ option("--limit"){
384
+ argument :required
385
+ cast :integer
386
+ }
387
+ option("--redirect"){
388
+ argument :required
389
+ }
390
+ option("--daemon"){}
391
+
392
+ def run
393
+ options = {}
394
+
395
+ =begin
396
+ %w[ forever ].each do |key|
397
+ options[key.to_sym] = true if param[key].given?
398
+ end
399
+ =end
400
+
401
+ %w[ forever ppid wait limit ].each do |key|
402
+ options[key.to_sym] = param[key].value if param[key].given?
403
+ end
404
+
405
+ #p options
406
+ #exit
407
+ if param["redirect"].given?
408
+ open(param["redirect"].value, "a+") do |fd|
409
+ STDERR.reopen fd
410
+ STDOUT.reopen fd
411
+ end
412
+ STDERR.sync = true
413
+ STDOUT.sync = true
414
+ end
415
+
416
+ trap("SIGTERM"){
417
+ info{ "SIGTERM" }
418
+ exit
419
+ }
420
+
421
+ if param["daemon"].given?
422
+ daemon{ Bj.run options }
423
+ else
424
+ Bj.run options
425
+ end
426
+ end
427
+ end
428
+
429
+ mode "submit" do
430
+ keyword("file"){
431
+ argument :required
432
+ attr
433
+ }
434
+
435
+ def run
436
+ joblist = Bj.joblist.for argv.join(' ')
437
+
438
+ case file
439
+ when "-"
440
+ joblist.push(Bj.joblist.jobs_from_io(STDIN))
441
+ when "--", "---"
442
+ joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
443
+ else
444
+ open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
445
+ end
446
+
447
+ jobs = Bj.submit joblist, :no_tickle => true
448
+
449
+ oh = lambda{|job| OrderedHash["id", job.id, "command", job.command]}
450
+
451
+ y jobs.map{|job| oh[job]}
452
+ end
453
+ end
454
+
455
+ mode "list" do
456
+ def run
457
+ Bj.transaction do
458
+ y Bj::Table::Job.find(:all).map(&:to_hash)
459
+ end
460
+ end
461
+ end
462
+
463
+ mode "set" do
464
+ argument("key"){ attr }
465
+
466
+ argument("value"){ attr }
467
+
468
+ option("hostname", "H"){
469
+ argument :required
470
+ default Bj.hostname
471
+ attr
472
+ }
473
+
474
+ option("cast", "c"){
475
+ argument :required
476
+ default "to_s"
477
+ attr
478
+ }
479
+
480
+ def run
481
+ Bj.transaction do
482
+ Bj.config.set(key, value, :hostname => hostname, :cast => cast)
483
+ y Bj.table.config.for(:hostname => hostname)
484
+ end
485
+ end
486
+ end
487
+
488
+ mode "config" do
489
+ option("hostname", "H"){
490
+ argument :required
491
+ default Bj.hostname
492
+ }
493
+
494
+ def run
495
+ Bj.transaction do
496
+ y Bj.table.config.for(:hostname => param["hostname"].value)
497
+ end
498
+ end
499
+ end
500
+
501
+ mode "pid" do
502
+ option("hostname", "H"){
503
+ argument :required
504
+ default Bj.hostname
505
+ }
506
+
507
+ def run
508
+ Bj.transaction do
509
+ config = Bj.table.config.for(:hostname => param["hostname"].value)
510
+ puts config[ "#{ RAILS_ENV }.pid" ] if config
511
+ end
512
+ end
513
+ end
514
+
515
+
516
+ def run
517
+ help!
518
+ end
519
+
520
+ def before_run
521
+ self.logger = param["log"].value if param["log"].given?
522
+ Bj.logger = logger
523
+ set_rails_root(param["rails_root"].value) if param["rails_root"].given?
524
+ set_rails_env(param["rails_env"].value) if param["rails_env"].given?
525
+ end
526
+
527
+ def set_rails_root rails_root
528
+ ENV["RAILS_ROOT"] = rails_root
529
+ ::Object.instance_eval do
530
+ remove_const :RAILS_ROOT
531
+ const_set :RAILS_ROOT, rails_root
532
+ end
533
+ end
534
+
535
+ def set_rails_env rails_env
536
+ ENV["RAILS_ENV"] = rails_env
537
+ ::Object.instance_eval do
538
+ remove_const :RAILS_ENV
539
+ const_set :RAILS_ENV, rails_env
540
+ end
541
+ end
542
+
543
+ def daemon
544
+ ra, wa = IO.pipe
545
+ rb, wb = IO.pipe
546
+ if fork
547
+ at_exit{ exit! }
548
+ wa.close
549
+ r = ra
550
+ rb.close
551
+ w = wb
552
+ pid = r.gets
553
+ w.puts pid
554
+ Integer pid.strip
555
+ else
556
+ ra.close
557
+ w = wa
558
+ wb.close
559
+ r = rb
560
+ open("/dev/null", "r+") do |fd|
561
+ STDIN.reopen fd
562
+ STDOUT.reopen fd
563
+ STDERR.reopen fd
564
+ end
565
+ Process::setsid rescue nil
566
+ pid =
567
+ fork do
568
+ Dir::chdir RAILS_ROOT
569
+ File::umask 0
570
+ $DAEMON = true
571
+ yield
572
+ exit!
573
+ end
574
+ w.puts pid
575
+ r.gets
576
+ exit!
577
+ end
578
+ end
579
+ }
580
+
581
+
582
+
583
+
584
+
585
+ #
586
+ # we setup a few things so the script works regardless of whether it was
587
+ # called out of /usr/local/bin, ./script, or wherever. note that the script
588
+ # does *not* require the entire rails application to be loaded into memory!
589
+ # we could just load boot.rb and environment.rb, but this method let's
590
+ # submitting and running jobs be infinitely more lightweight.
591
+ #
592
+
593
+ BEGIN {
594
+ #
595
+ # see if we're running out of RAILS_ROOT/script/
596
+ #
597
+ unless defined?(BJ_SCRIPT)
598
+ BJ_SCRIPT =
599
+ if %w[ script config app ].map{|d| test ?d, "#{ File.dirname __FILE__ }/../#{ d }"}.all?
600
+ __FILE__
601
+ else
602
+ nil
603
+ end
604
+ end
605
+ #
606
+ # setup RAILS_ROOT
607
+ #
608
+ unless defined?(RAILS_ROOT)
609
+ ### grab env var first
610
+ rails_root = ENV["RAILS_ROOT"]
611
+
612
+ ### commandline usage clobbers
613
+ kv = nil
614
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ROOT=/ and kv = arg}
615
+ rails_root = kv.split(%r/=/,2).last if kv
616
+
617
+ ### we know the rails_root if we are in RAILS_ROOT/script/
618
+ unless rails_root
619
+ if BJ_SCRIPT
620
+ rails_root = File.expand_path "#{ File.dirname __FILE__ }/.."
621
+ end
622
+ end
623
+
624
+ ### perhaps the current directory is a rails_root?
625
+ unless rails_root
626
+ if %w[ script config app ].map{|d| test(?d, d)}.all?
627
+ rails_root = File.expand_path "."
628
+ end
629
+ end
630
+
631
+ ### bootstrap
632
+ RAILS_ROOT = rails_root
633
+ end
634
+ #
635
+ # setup RAILS_ENV
636
+ #
637
+ unless defined?(RAILS_ENV)
638
+ ### grab env var first
639
+ rails_env = ENV["RAILS_ENV"]
640
+
641
+ ### commandline usage clobbers
642
+ kv = nil
643
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ENV=/ and kv = arg}
644
+ rails_env = kv.split(%r/=/,2).last if kv
645
+
646
+ ### fallback to development
647
+ unless rails_env
648
+ rails_env = "development"
649
+ end
650
+
651
+ ### bootstrap
652
+ RAILS_ENV = rails_env
653
+ end
654
+ #
655
+ # ensure that rubygems is loaded
656
+ #
657
+ begin
658
+ require "rubygems"
659
+ rescue
660
+ 42
661
+ end
662
+ #
663
+ # load gems from plugin dir iff installed as plugin - otherwise load normally
664
+ #
665
+ if RAILS_ROOT and BJ_SCRIPT
666
+ =begin
667
+ dir = Gem.dir
668
+ path = Gem.path
669
+ gem_home = File.join RAILS_ROOT, "vendor", "plugins", "bj", "gem_home"
670
+ gem_path = [gem_home]
671
+ Gem.send :use_paths, gem_home, gem_path
672
+ gem "bj"
673
+ require "bj"
674
+ gem "main"
675
+ require "main"
676
+ =end
677
+ libdir = File.join(RAILS_ROOT, "vendor", "plugins", "bj", "lib")
678
+ $LOAD_PATH.unshift libdir
679
+ end
680
+ #
681
+ # hack of #to_s of STDERR/STDOUT for nice help messages
682
+ #
683
+ class << STDERR
684
+ def to_s() 'STDERR' end
685
+ end
686
+ class << STDOUT
687
+ def to_s() 'STDOUT' end
688
+ end
689
+ }
690
+
691
+
692
+ __END__