backgroundrb-rails3 1.1

data/doc/content/rails/rails.yaml

@@ -0,0 +1,5 @@
+# Built-in
+filters_pre: [ "redcloth" ]
+# Custom
+title: "Rails Integration of BackgrounDRb"
+sidebar_items: [["Introduction", "#introduction"], ["Async Task", "#async_task"], ["Sync Task", "#sync_task"], ["Worker Results", "#worker_results"], ["Persistent Tasks", "#persistent_task"], ["New Worker", "#new_worker"], ["Worker Info", "#worker_info"], ["Clustering", "#clustering"]]

data/doc/content/scheduling/scheduling.txt

@@ -0,0 +1,166 @@
+<div id="content">
+
+%(entry-title)<a name="timer_scheduling"> Timer Based Scheduling </a>%
+
+Simple tasks in the workers can be scheduled using @add_timer@ or @add_periodic_timer@ methods.
+For example:
+
+<pre class="multiline">class HelloWorker < BackgrounDRb::MetaWorker
+  set_worker_name :hello_worker
+
+  def create(args = nil)
+    # time argument is in seconds
+    add_periodic_timer(10) { expire_sessions }
+  end
+
+  def expire_sessions
+    # expire user sessions
+  end
+end </pre>
+
+Similarly one can use @add_timer@ to fire oneshot task execution.
+
+%(entry-title)<a name="unix_scheduling"> Unix Scheduler </a>%
+
+_BackgrounDRb_ supports normal unix styled schedules which can be configured
+from @backgroundrb.yml@ file. A sample configuration looks like:
+
+<pre class="multiline">:backgroundrb:
+  :ip: 0.0.0.0
+  :port: 11006
+:schedules:
+  :foo_worker:
+    :foobar:
+      :trigger_args:
+        :start: <%= Time.now + 5.seconds %>
+        :end: <%= Time.now + 10.minutes %>
+        :repeat_interval: <%= 1.minute %> </pre>
+
+Above scheduler option schedules method @foobar@ defined inside @foo_worker@ to start
+executing by 5 seconds delay and stop after 10 minutes. Method should periodically execute
+every 1 minute between that time period. *Never in any scheduling option, you should schedule @create@
+method/task*
+
+%(entry-title)<a name="cron_scheduling"> Cron Scheduling </a>%
+
+_BackgrounDRb_ also supports Cron based ccheduling.
+You can use a configuration file for cron scheduling of workers. The method specified in the configuration
+file would be called periodically. You should accommodate for the fact that the time gap between periodic
+invocation of a method should be more than the time that is actually required to execute the method.
+If a method takes longer time than the time window specified, your method invocations will lag
+perpetually.
+
+A Sample Configuration file for Cron based Scheduling looks like:
+
+<pre class="multiline">
+:backgroundrb:
+  :ip: 0.0.0.0
+  :port: 11006
+:schedules:
+  :foo_worker:
+    :barbar:
+      :trigger_args: */10 * * * * *
+      :data: Hello World </pre>
+
+
+Above scheduler will schedule invocation of @barbar@ method inside @foo_worker@ at every 10 seconds.
+You can also schedule invocation of multiple methods in same worker at different intervals, just use
+following as an example configuration file.
+
+<pre class="multiline">
+:backgroundrb:
+  :ip: 0.0.0.0
+  :port: 11006
+:schedules:
+  :foo_worker:
+    :barbar:
+      :trigger_args: */10 * * * * *
+      :data: Hello World
+    :some_task: # execute some_method in foo_worker every 2nd hour
+      :trigger_args: 0 * */2 * * *
+      :data: Hello World </pre>
+
+
+p(sub-title). A Word about Cron Scheduler
+
+Note that the initial field in the BackgrounDRb cron trigger specifies
+seconds, not minutes as with Unix-cron.
+
+The fields (which can be an asterisk, meaning all valid patterns) are:
+
+<pre class="boxed">sec[0,59] min[0,59], hour[0,23], day[1,31], month[1,12], weekday[0,6], year</pre>
+
+The syntax pretty much follows Unix-cron. The following will trigger
+on the first hour and the thirtieth minute every day:
+
+<pre class="boxed">0 30 1 * * * *</pre>
+
+The following will trigger the specified method every 10 seconds:
+
+<pre class="boxed">*/10 * * * * * *</pre>
+
+The following will trigger the specified method every 1 hour:
+
+<pre class="boxed">0 0 * * * * *</pre>
+
+For each field you can use a comma-separated list. The following would
+trigger on the 5th, 16th and 23rd minute every hour:
+
+<pre class="boxed"> 5,16,23 * * * * *</pre>
+
+Fields also support ranges, using a dash between values. The following
+triggers from 8th through the 17th hour, at five past the hour:
+
+<pre class="boxed"> 5 8-17 * * * *</pre>
+
+Finally, fields support repeat interval syntax. The following triggers
+every five minutes, every other hour after the sixth hour:
+
+<pre class="boxed"> */5 6/2 * * * *</pre>
+
+Here is a more complex example: months 0,2,4,5,6,8,10,12, every day
+and hour, minutes 1,2,3,4,6,20, seconds: every 5th second counting
+from the 28th second plus the 59th second:
+
+
+<pre class="boxed">28/5,59 1-4,6,20 */1 * 5,0/2 * *</pre>
+
+Note that if you specify an asterisk in the first field (seconds)
+it will trigger every second for the subsequent match.
+
+%(entry-title)<a name="restart_on_schedule"> Restart worker on schedule </a>%
+
+Usually when your worker is scheduled to execute at longer intervals, it
+doesn't make sense to have worker around, when its doing nothing. Since, scheduling
+via configuration file requires that your worker must be loaded when _BackgrounDRb_ starts,
+your worker is always around, even when doing nothing.
+
+You can reuse worker in processing requests from rails, but if its not possible
+and you rather want worker to start afresh each time, scheduler detects a firetime, you can use
+following syntax to autostart workers on scheduled time:
+
+<pre class="multiline">class HelloWorker < BackgrounDRb::MetaWorker
+  set_worker_name :hello_worker
+  reload_on_schedule true
+
+  def create(args = nil)
+    # this method is called, when worker is loaded for the first time
+  end
+end </pre>
+
+In above worker @reload_on_schedule true@ makes sure that your worker is restarted on
+scheduled time. This feature is only available in version 1.0.3 onwards.
+
+%(entry-title)<a name="schedule_at"> Schedule one shot execution of task at specified time </a>%
+
+<p><b> Only available for tasks persisted to database table </b></p>
+
+If you are using job queue table and want one shot execution of a task scheduled at a particular time. You can use:
+
+<pre class="multiline">MiddleMan(:hello_worker).enq_some_task(:arg => "hello_world",
+   :job_key => "boy",:scheduled_at => Time.now + 30.minutes)</pre>
+
+Which will schedule specified task to be executed after 30 minutes from now.
+
+
+</div>

data/doc/content/scheduling/scheduling.yaml

@@ -0,0 +1,5 @@
+# Built-in
+filters_pre: [ "redcloth" ]
+# Custom
+title: "Scheduling Tasks using BackgrounDRb"
+sidebar_items: [["Timers", "#timer_scheduling"], ["Unix Scheduler", "#unix_scheduling"], ["Cron Scheduler", "#cron_scheduling"], ["Restart on Schedule", "#restart_on_schedule"], ["Schedule At", "#schedule_at"]]

data/doc/content/workers/workers.txt

@@ -0,0 +1,178 @@
+<div id="content">
+
+%(entry-title)<a name="introduction"> Workers </a>%
+
+Workers are your building blocks of Asynchronous Task Processing. An empty auto generated worker looks like this:
+
+<pre class="multiline">class BillingWorker < BackgrounDRb::MetaWorker
+  set_worker_name :billing_worker
+  def create(args = nil)
+    # method gets called, when new instance of worker is created.
+  end
+end </pre>
+
+@set_worker_name@ will set the worker name which can be later used while invoking tasks on the worker.
+@create@ method gets called when worker is loaded for the first time. If you are starting your worker
+from rails, you can pass arguments to @create@ method using:
+
+<pre class="multiline">MiddleMan.new_worker(:worker => :billing_worker,\
+     :worker_key => user_session,:data => current_user.id) </pre>
+
+p(sub-title). Using Workers
+
+You can invoke random tasks on workers from rails or you can schedule them using config file. Look into
+"Scheduling":/scheduling/index.html section for scheduling and "Rails Integration":/rails/index.html section
+for invoking worker tasks from rails.
+
+p(sub-title). Inbuilt instance methods available in your workers:
+
+*(content_list) @cache@ : Can be used to store random results from worker which can be later retrieved from rails. For example:
+  <pre class="boxed"> cache[key] = some_data </pre>
+* @add_timer@ : Look in scheduler section
+* @add_periodic_timer@ : Look in scheduler section
+* @thread_pool@ : Look below
+* @connect@: Look in "Advanced":/advanced/index.html section. Used to connect to external TCP/IP servers.
+* @start_server@: Look in "Advanced":/advanced/index.html section. Used to start TCP/IP server from worker.
+* @send_data@: Can be used to send objects to master process. You can ignore this method.
+* @job_key@: When you invoke a task from rails by passing a job_key, that job_key can be accessed in workers with @job_key@. For example:
+  From rails:
+  <pre class="boxed"> MiddleMan.worker(:foo_worker).async_some_task(:arg => urls,
+    :job_key => current_user[:id])</pre>
+  Now this @job_key@ can be accessed inside workers with:
+  <pre class="multiline">class FooWorker < BackgrounDRb::MetaWorker
+    def some_task urls
+      .. do some work with urls ..
+      cache[job_key] = result
+    end
+  end</pre>
+
+p(sub-title). Options via class methods :
+
+Following class methods are available for further tuning of workers:
+
+*(content_list) @pool_size@ : Can be used to control thread pool size. Accepts pool size as integer value.
+* @set_no_auto_load@ : Can be used to disable auto loading of workers when _BackgrounDRb_ starts. Accepts true or false.
+* @reload_on_schedule@ : Can be used to enable reloading of worker at scheduled execution time. Accepts true or false.
+* @set_worker_name@ : Can be used to set worker name. Accepts symbol as worker name.
+
+Following snippet demonstrates their usages:
+
+<pre class="multiline">class HelloWorker < BackgrounDRb::MetaWorker
+  set_worker_name :hello_worker
+  reload_on_schedule true
+  pool_size 10
+end</pre>
+
+When @reload_on_schedule@ is true, worker won't be loaded while _BackgrounDRb_ starts and hence you don't need
+@set_no_auto_load@ option there.
+
+%(entry-title)<a name="thread_pool"> Using Thread Pool </a>%
+
+Remember _BackgrounDRb_ follows event model of network programming, but sad truth of life is not all networking
+libraries follow this model and hence they make use of blocking IO and threads. BackgrounDRb allows you to run
+all such tasks concurrently in threads which are internally managed by BackgrounDRb thread pool.
+
+Each worker has access to object @thread_pool@ which can be used to run task in a thread concurrently.
+
+<pre class="boxed">thread_pool.defer(:scrap_wikipedia,scrap_url) </pre>
+
+So whatever code you write within @scrap_wikipedia@ method is going to run concurrently.
+
+*WARNING*: Many of the Ruby libraries out there aren't thread safe and they may not
+work as advertised when used from threads(example: Mechanize,Scrubyt)
+
+%(entry-title)<a name="result_caching"> Result Caching </a>%
+
+<b>Update : </b> <em>Using MemCache to store result objects is strongly recommended. Inbuilt cache works, but may give
+unpredictable results. Also, using Memcache serves as an out of process cache, which can be queried at any time.
+If your worker is doing some processing, inbuilt cache may not return result until worker picks up that request.</em>
+
+
+All workers can cache results using @cache@ attribute. This result object can be then
+queried from rails using @ask_result@. For example:
+
+<pre class="multiline">class ProgressWorker < BackgrounDRb::MetaWorker
+  set_worker_name :progress_worker
+  def create
+    @counter = 0
+    add_periodic_timer(2) { increment_counter }
+  end
+  def increment_counter
+    @counter += 1
+    cache[some_key] = counter
+  end
+end</pre>
+
+And using @MiddleMan@ proxy, you can keep querying the status of progress bar :
+
+<pre class="boxed">MiddleMan.worker(:progress_worker).ask_result(some_key)</pre>
+
+By default, @cache@ is a worker local hash like object, which is used for storing results.
+But if you plan to store lots of objects in cache from your worker, it may not be an
+optimal solution. You can easily replace in-worker cache with memcache.
+
+You need to change @backgroundrb.yml@ file like this, for using memcache for object caching:
+
+<pre class="multiline">:backgroundrb:
+  :ip: 0.0.0.0
+  :port: 11006
+  :result_storage: memcache
+
+:memcache: "10.0.0.1:11211,10.0.0.2:11211" </pre>
+
+Everything else remains the same.
+
+%(entry-title)<a name="persistent_job">Persistent Task Queue </a>%
+
+BackgrounDRb now have out of box support for persistent job queues which are persisted to the
+database. API to add a task in the job_queue is pretty simple:
+
+<pre class="boxed">MiddleMan(:hello_worker).enq_some_task(:arg => "hello_world",:job_key => "boy")</pre>
+
+So in your hello worker:
+
+<pre class="multiline">
+class HelloWorker
+  def some_task args
+    .. do some work ..
+    persistent_job.finish! #=> marks the job as finished. totally thread safe
+  end
+end</pre>
+
+@persistent_job@ is a thread local variable and will refer to currently
+running queued task can be used from thread pool as well. For example:
+
+<pre class="multiline">
+class HelloWorker
+  def some_task args
+    thread_pool.defer(:fetch_url,args)
+  end
+   def fetch_url tags
+    .. runs in thread ..
+    .. fetch tasks ..
+    persistent_job.finish!
+  end
+end</pre>
+
+
+%(entry-title)<a name="worker_testing">Testing Workers </a>%
+
+_BackgrounDRb_ comes with a baked in mechanism to write test cases. First make sure that you
+have bdrb_test_helper.rb in the test directory of your rails app (run @rake backgroundrb:setup@, if you dont have one).
+
+Just put your worker test cases in test/unit directory of your rails application and require the helper. Now, you should be good to go.
+
+<pre class="multiline">require File.join(File.dirname(__FILE__) + "/../bdrb_test_helper")
+require "god_worker"
+ context "When god worker starts" do
+  setup do
+    god_worker = GodWorker.new
+  end
+end </pre>
+
+All above helper file does is that it stubs out, relevant worker methods, which really need network IO.
+There can be methods added, which aren't stubbed, for all such methods you are encouraged to stub them and send
+the patch to the backgroundrb mailing list.
+
+
+</div>

data/doc/content/workers/workers.yaml

@@ -0,0 +1,5 @@
+# Built-in
+filters_pre: [ "redcloth" ]
+# Custom
+title: "Using BackgrounDRb workers"
+sidebar_items: [["Introduction", "#introduction"], ["Thread Pool", "#thread_pool"], ["Result Caching", "#result_caching"], ["Persistent Jobs", "#persistent_job"], ["Worker Testing", "#worker_testing"]]

data/doc/layouts/default/default.erb

@@ -0,0 +1,56 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+    <title>BackgrounDRb - <%= @page.title %></title>
+    <link rel="stylesheet" href="/style.css" type="text/css" media="screen" />
+  </head>
+
+  <body>
+
+    <div id="wrapper">
+      <div id="header">
+	<h1> BackgrounDRb </h1>
+      </div>
+
+      <div id="menu">
+	<ul>
+	  <li><a href="/"> Intro </a></li>
+          <li><a href="/scheduling/"> Scheduling </a></li>
+          <li><a href="/workers/"> Workers </a></li>
+          <li><a href="/rails/"> Rails Integration </a></li>
+          <li><a href="/advanced/"> Advanced </a></li>
+	  <li><a href="/manual/index.html"> Manual </a></li>
+	  <li><a href="/community/"> Community </a></li>
+          <li><a href="/faq"> FAQs </a></li>
+	</ul>
+      </div>
+
+      <div id="sidebar">
+	<div id="feed">
+	  <!-- <a class="feed-button" href="#">&nbsp;</a> -->
+	</div>
+	<ul>
+	  <% if @page.sidebar_items %>
+	  <% for menu_item in @page.sidebar_items %>
+	  <li><a href="<%= menu_item[1] %>"> <%= menu_item[0] %> </a></li>
+	  <% end %>
+	  <% end %>
+	</ul>
+
+	<div id="sidebar-bottom"> &nbsp; </div>
+      </div>
+
+	<%= @page.content %>
+
+      <div id="footer">
+	<div id="footer-valid">
+	  <a href="http://validator.w3.org/check/referer">xhtml</a> :: 
+          <span align="right"> Created with <a href="http://nanoc.stoneship.org/"> Nanoc </a> </span>
+	</div>
+        
+      </div>
+
+    </div> <!-- end of wrapper div tag -->
+  </body>
+</html>

data/doc/layouts/default/default.yaml

@@ -0,0 +1,4 @@
+# Built-in
+
+# Custom 
+filter: erb

data/doc/lib/default.rb

@@ -0,0 +1,7 @@
+# All files in the 'lib' directory will be loaded
+# before nanoc starts compiling.
+
+def html_escape(str)
+  str.gsub('&', '&amp;').str('<', '&lt;').str('>', '&gt;').str('"', '&quot;')
+end
+alias h html_escape