starfish 1.0.0 → 1.1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (54) hide show
  1. data/README +33 -2
  2. data/doc/classes/MapReduce.src/M000001.html +1 -1
  3. data/doc/classes/MapReduce.src/M000002.html +1 -1
  4. data/doc/classes/MapReduce.src/M000003.html +1 -1
  5. data/doc/classes/MapReduce.src/M000004.html +1 -1
  6. data/doc/classes/MapReduce.src/M000005.html +1 -1
  7. data/doc/classes/MapReduce.src/M000006.html +1 -1
  8. data/doc/classes/MapReduce.src/M000007.html +1 -1
  9. data/doc/classes/MapReduce.src/M000008.html +1 -1
  10. data/doc/classes/MapReduce.src/M000009.html +1 -1
  11. data/doc/classes/MapReduce.src/M000010.html +1 -1
  12. data/doc/classes/MapReduce.src/M000011.html +1 -1
  13. data/doc/classes/MapReduce.src/M000012.html +1 -1
  14. data/doc/classes/MapReduce.src/M000013.html +1 -1
  15. data/doc/classes/MapReduce.src/M000014.html +1 -1
  16. data/doc/classes/MapReduce/ActiveRecord/Base.src/M000015.html +6 -14
  17. data/doc/classes/MapReduce/File.html +70 -0
  18. data/doc/classes/MapReduce/File.src/M000019.html +42 -0
  19. data/doc/classes/MapReduce/File/Client.html +177 -0
  20. data/doc/classes/MapReduce/File/Client.src/M000020.html +18 -0
  21. data/doc/classes/MapReduce/File/Client.src/M000021.html +20 -0
  22. data/doc/classes/MapReduce/File/Client.src/M000022.html +18 -0
  23. data/doc/classes/Starfish.html +61 -62
  24. data/doc/classes/Starfish.src/M000023.html +5 -9
  25. data/doc/classes/Starfish.src/M000024.html +4 -22
  26. data/doc/classes/Starfish.src/M000025.html +29 -19
  27. data/doc/classes/Starfish.src/M000026.html +64 -4
  28. data/doc/classes/Starfish.src/M000027.html +9 -8
  29. data/doc/classes/Starfish.src/M000028.html +23 -5
  30. data/doc/classes/Starfish.src/M000029.html +22 -5
  31. data/doc/classes/Starfish.src/M000030.html +5 -5
  32. data/doc/classes/Starfish.src/M000031.html +22 -0
  33. data/doc/classes/Starfish.src/{M000020.html → M000032.html} +5 -5
  34. data/doc/classes/Starfish.src/M000033.html +18 -0
  35. data/doc/classes/Starfish.src/M000034.html +18 -0
  36. data/doc/classes/Starfish/RingFinger.html +6 -6
  37. data/doc/classes/Starfish/RingFinger.src/{M000031.html → M000035.html} +1 -1
  38. data/doc/created.rid +1 -1
  39. data/doc/files/README.html +52 -6
  40. data/doc/files/lib/map_reduce/active_record_rb.html +1 -2
  41. data/doc/files/lib/map_reduce/file_rb.html +1 -1
  42. data/doc/files/lib/map_reduce_rb.html +1 -1
  43. data/doc/files/lib/starfish_rb.html +1 -1
  44. data/doc/fr_class_index.html +1 -0
  45. data/doc/fr_method_index.html +17 -13
  46. data/examples/map_reduce/file.rb +2 -3
  47. data/lib/map_reduce.rb +2 -1
  48. data/lib/map_reduce/active_record.rb +25 -19
  49. data/lib/map_reduce/file.rb +93 -0
  50. data/lib/starfish.rb +9 -2
  51. metadata +15 -7
  52. data/doc/classes/Starfish.src/M000019.html +0 -19
  53. data/doc/classes/Starfish.src/M000021.html +0 -45
  54. data/doc/classes/Starfish.src/M000022.html +0 -71
data/README CHANGED
@@ -1,6 +1,6 @@
1
1
  == Welcome to Starfish
2
2
 
3
- Starfish is a utility born from abstracting code from a highly popular website (http://mog.com) to make distributed programming ridiculously easy.
3
+ Starfish is a utility to make distributed programming ridiculously easy.
4
4
 
5
5
  == Download
6
6
 
@@ -8,6 +8,10 @@ Starfish is a utility born from abstracting code from a highly popular website (
8
8
  * http://rubyforge.org/projects/starfish
9
9
  * svn co svn://rubyforge.org//var/svn/starfish
10
10
 
11
+ == Background
12
+
13
+ Starfish was born by abstracting code from a highly popular website (http://mog.com). I had written distributed programming code for various tasks including database corruption cleanup as well as real-time indexing requirements and decided that nobody should have to write as much code as I did to do those tasks. The general idea of distributed programming is vastly underused by most programmers and Starfish hides everything you don't have to worry about.
14
+
11
15
  == Usage
12
16
 
13
17
  Make a file and define the server and client. What follows is a simple example:
@@ -48,7 +52,7 @@ By default, the server will start a log file at /tmp/foo.rb.log. You can overrid
48
52
 
49
53
  The :log parameter is intelligent and can take various other formats including nil for no logging or instances of any kind of logger class you prefer to use.
50
54
 
51
- == MapReduce
55
+ == MapReduce with ActiveRecord
52
56
 
53
57
  The crown jewel of Starfish is actually a library built on Starfish called MapReduce. Inspired by Google's MapReduce (http://en.wikipedia.org/wiki/MapReduce), it automates the process of divide and conquer for large data sets. Currently, Starfish's implementation only works with the ActiveRecord ORM from Ruby on Rails. The idea is that you have a very large data set that would be unmanageable to process otherwise. For example, let's say your database table has 30GB of information in it. It would be very difficult to keep 30GB of records in RAM at one time; even if you processed them serially, it would take a very long time. The idea is to have many clients grabbing chunks of the database at a time making much faster work, even on a single processor machine, than would otherwise be feasible.
54
58
 
@@ -125,6 +129,33 @@ And you will be returned a YAML hash of statistics.
125
129
  time_spent_grabbing_queues: 0.090328
126
130
  num_objects_grabbed: 10
127
131
 
132
+ == MapReduce with Files
133
+
134
+ You can also divide and conquer with files. The idea is that you have a large file and want to process line-by-line as quickly as possible. It is simple to accomplish this with Starfish.
135
+
136
+ server do |map_reduce|
137
+ map_reduce.type = File
138
+ map_reduce.input = "/tmp/big_log_file"
139
+ end
140
+
141
+ client do |line|
142
+ if line =~ /some_regex/
143
+ logger.info(line)
144
+ end
145
+ end
146
+
147
+ Simply replacing the type and input lets you process your file in a distributed way. Like when using ActiveRecord, you also have options:
148
+
149
+ server do |map_reduce|
150
+ map_reduce.type = File
151
+ map_reduce.input = "/tmp/big_log_file"
152
+ map_reduce.queue_size = 1000 # how many lines of the file to buffer at a time
153
+ map_reduce.lines_per_client = 100 # how many lines each client will process at a time
154
+ map_reduce.rescan_when_complete = true
155
+ end
156
+
157
+ For a file, rescan_when_complete goes back to the beginning of the file when you have finished processing it. You can also use the vigilant option to wait for new data to be added to the file.
158
+
128
159
  == Examples
129
160
 
130
161
  See the examples/ directory.
data/doc/classes/MapReduce.src/M000001.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 17</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 18</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">initialize</span>
15
15
  <span class="ruby-ivar">@lock</span> = <span class="ruby-keyword kw">false</span>
16
16
  <span class="ruby-ivar">@offset</span> = <span class="ruby-value">0</span>
data/doc/classes/MapReduce.src/M000002.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 34</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 35</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">add_time_spent_processing_objects</span>(<span class="ruby-identifier">time</span>)
15
15
  <span class="ruby-ivar">@time_spent_processing_objects</span> <span class="ruby-operator">+=</span> <span class="ruby-identifier">time</span>
16
16
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce.src/M000003.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 38</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 39</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">stats</span>
15
15
  {
16
16
  <span class="ruby-identifier">:time_began</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-ivar">@time_began</span>,
data/doc/classes/MapReduce.src/M000004.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 49</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 50</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">type=</span>(<span class="ruby-identifier">type</span>)
15
15
  <span class="ruby-ivar">@type</span> = <span class="ruby-identifier">type</span>
16
16
  <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">valid_type?</span>
data/doc/classes/MapReduce.src/M000005.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 56</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 57</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">base_type_to_s</span>
15
15
  <span class="ruby-identifier">base_type</span>.<span class="ruby-identifier">to_s</span> <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">valid_type?</span>
16
16
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce.src/M000006.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 60</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 61</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">type_to_s</span>
15
15
  <span class="ruby-identifier">type</span>.<span class="ruby-identifier">to_s</span> <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">valid_type?</span>
16
16
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce.src/M000007.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 64</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 65</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">spool=</span>(<span class="ruby-identifier">type</span>, <span class="ruby-identifier">input</span>)
15
15
  <span class="ruby-keyword kw">self</span>.<span class="ruby-identifier">type</span> = <span class="ruby-identifier">type</span>
16
16
  <span class="ruby-keyword kw">self</span>.<span class="ruby-identifier">input</span> = <span class="ruby-identifier">input</span>
data/doc/classes/MapReduce.src/M000008.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 69</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 70</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">valid?</span>
15
15
  <span class="ruby-identifier">valid_type?</span> <span class="ruby-operator">&amp;&amp;</span> <span class="ruby-identifier">valid_input?</span>
16
16
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce.src/M000009.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 73</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 74</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">valid_type?</span>
15
15
  <span class="ruby-ivar">@@types</span>.<span class="ruby-identifier">keys</span>.<span class="ruby-identifier">each</span> {<span class="ruby-operator">|</span><span class="ruby-identifier">type</span><span class="ruby-operator">|</span> <span class="ruby-keyword kw">return</span> <span class="ruby-keyword kw">true</span> <span class="ruby-keyword kw">if</span> <span class="ruby-ivar">@type</span> <span class="ruby-operator">&amp;&amp;</span> (<span class="ruby-ivar">@type</span> <span class="ruby-operator">&lt;</span> <span class="ruby-identifier">type</span> <span class="ruby-operator">||</span> <span class="ruby-ivar">@type</span> <span class="ruby-operator">==</span> <span class="ruby-identifier">type</span>)}
16
16
  <span class="ruby-keyword kw">return</span> <span class="ruby-keyword kw">false</span>
data/doc/classes/MapReduce.src/M000010.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 78</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 79</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">valid_input?</span>
15
15
  <span class="ruby-keyword kw">if</span> <span class="ruby-keyword kw">not</span> <span class="ruby-identifier">valid_type?</span>
16
16
  <span class="ruby-keyword kw">return</span> <span class="ruby-keyword kw">false</span>
data/doc/classes/MapReduce.src/M000011.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 87</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 88</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">method_missing</span>(<span class="ruby-identifier">name</span>, <span class="ruby-operator">*</span><span class="ruby-identifier">args</span>)
15
15
  <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">name</span>.<span class="ruby-identifier">to_s</span> <span class="ruby-operator">=~</span> <span class="ruby-regexp re">/(.*)=$/</span> <span class="ruby-operator">&amp;&amp;</span> <span class="ruby-identifier">args</span>[<span class="ruby-value">0</span>].<span class="ruby-identifier">is_a?</span>(<span class="ruby-constant">Proc</span>)
16
16
  <span class="ruby-keyword kw">self</span>.<span class="ruby-identifier">class</span>.<span class="ruby-identifier">instance_eval</span> <span class="ruby-keyword kw">do</span>
data/doc/classes/MapReduce.src/M000012.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 97</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 98</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">raise_if_invalid!</span>
15
15
  <span class="ruby-keyword kw">if</span> <span class="ruby-keyword kw">not</span> <span class="ruby-identifier">valid_type?</span>
16
16
  <span class="ruby-identifier">raise</span> <span class="ruby-constant">MapReduceError</span>, <span class="ruby-value str">&quot;invalid type, please make sure you provide one of the following classes or sub-classes thereof: ActiveRecord::Base, File, or Array&quot;</span>
data/doc/classes/MapReduce.src/M000013.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 106</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 107</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">map_reduce?</span>
15
15
  <span class="ruby-keyword kw">true</span>
16
16
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce.src/M000014.html CHANGED
@@ -10,7 +10,7 @@
10
10
  <link rel="stylesheet" href="../.././rdoc-style.css" type="text/css" media="screen" />
11
11
  </head>
12
12
  <body class="standalone-code">
13
- <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 110</span>
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce.rb, line 111</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">base_type</span>
15
15
  <span class="ruby-identifier">check_type</span> = <span class="ruby-ivar">@type</span>
16
16
  <span class="ruby-identifier">type_found</span> = <span class="ruby-keyword kw">false</span>
data/doc/classes/MapReduce/ActiveRecord/Base.src/M000015.html CHANGED
@@ -12,26 +12,18 @@
12
12
  <body class="standalone-code">
13
13
  <pre><span class="ruby-comment cmt"># File lib/map_reduce/active_record.rb, line 45</span>
14
14
  <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">get_id</span>
15
- <span class="ruby-keyword kw">if</span> <span class="ruby-ivar">@queue</span>.<span class="ruby-identifier">empty?</span> <span class="ruby-operator">&amp;&amp;</span> <span class="ruby-ivar">@offset</span> <span class="ruby-operator">==</span> <span class="ruby-value">0</span>
16
- <span class="ruby-ivar">@total</span> = <span class="ruby-identifier">type</span>.<span class="ruby-identifier">count</span>(<span class="ruby-identifier">input</span>)
17
- <span class="ruby-keyword kw">if</span> <span class="ruby-keyword kw">not</span> <span class="ruby-ivar">@rescan_when_complete</span>
18
- <span class="ruby-ivar">@queue_size</span> <span class="ruby-operator">||=</span> <span class="ruby-ivar">@total</span>
19
- <span class="ruby-keyword kw">end</span>
20
- <span class="ruby-keyword kw">end</span>
21
-
22
15
  <span class="ruby-identifier">t</span> = <span class="ruby-constant">Time</span>.<span class="ruby-identifier">now</span>
23
16
 
24
- <span class="ruby-identifier">object_id</span> = <span class="ruby-keyword kw">if</span> <span class="ruby-ivar">@lock</span>
25
- <span class="ruby-identifier">:locked_queue_wait</span>
17
+ <span class="ruby-keyword kw">if</span> <span class="ruby-ivar">@lock</span>
18
+ <span class="ruby-keyword kw">return</span> <span class="ruby-identifier">:locked_queue_wait</span>
26
19
  <span class="ruby-keyword kw">else</span>
27
- <span class="ruby-identifier">queue</span>.<span class="ruby-identifier">shift</span> <span class="ruby-operator">||</span> <span class="ruby-identifier">:empty_queue_wait</span>
20
+ <span class="ruby-keyword kw">unless</span> <span class="ruby-identifier">object_id</span> = <span class="ruby-identifier">queue</span>.<span class="ruby-identifier">shift</span>
21
+ <span class="ruby-keyword kw">return</span> <span class="ruby-identifier">:empty_queue_wait</span>
22
+ <span class="ruby-keyword kw">end</span>
28
23
  <span class="ruby-keyword kw">end</span>
29
24
 
30
25
  <span class="ruby-ivar">@time_spent_grabbing_objects</span> <span class="ruby-operator">+=</span> (<span class="ruby-constant">Time</span>.<span class="ruby-identifier">now</span> <span class="ruby-operator">-</span> <span class="ruby-identifier">t</span>)
31
-
32
- <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">object_id</span>
33
- <span class="ruby-ivar">@num_objects_grabbed</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
34
- <span class="ruby-keyword kw">end</span>
26
+ <span class="ruby-ivar">@num_objects_grabbed</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
35
27
 
36
28
  <span class="ruby-keyword kw">return</span> <span class="ruby-identifier">object_id</span>
37
29
  <span class="ruby-keyword kw">end</span></pre>
data/doc/classes/MapReduce/File.html CHANGED
@@ -76,6 +76,13 @@
76
76
 
77
77
  </div>
78
78
 
79
+ <div id="method-list">
80
+ <h3 class="section-bar">Methods</h3>
81
+
82
+ <div class="name-list">
83
+ <a href="#M000019">get_lines</a>&nbsp;&nbsp;
84
+ </div>
85
+ </div>
79
86
 
80
87
  </div>
81
88
 
@@ -84,14 +91,77 @@
84
91
 
85
92
  <div id="section">
86
93
 
94
+ <div id="class-list">
95
+ <h3 class="section-bar">Classes and Modules</h3>
96
+
97
+ Class <a href="File/Client.html" class="link">MapReduce::File::Client</a><br />
98
+
99
+ </div>
87
100
 
88
101
 
89
102
 
90
103
 
104
+ <div id="attribute-list">
105
+ <h3 class="section-bar">Attributes</h3>
106
+
107
+ <div class="name-list">
108
+ <table>
109
+ <tr class="top-aligned-row context-row">
110
+ <td class="context-item-name">empty_queue_wait</td>
111
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
112
+ <td class="context-item-desc"></td>
113
+ </tr>
114
+ <tr class="top-aligned-row context-row">
115
+ <td class="context-item-name">lines_per_client</td>
116
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
117
+ <td class="context-item-desc"></td>
118
+ </tr>
119
+ <tr class="top-aligned-row context-row">
120
+ <td class="context-item-name">locked_queue_wait</td>
121
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
122
+ <td class="context-item-desc"></td>
123
+ </tr>
124
+ <tr class="top-aligned-row context-row">
125
+ <td class="context-item-name">queue_size</td>
126
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
127
+ <td class="context-item-desc"></td>
128
+ </tr>
129
+ <tr class="top-aligned-row context-row">
130
+ <td class="context-item-name">rescan_when_complete</td>
131
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
132
+ <td class="context-item-desc"></td>
133
+ </tr>
134
+ <tr class="top-aligned-row context-row">
135
+ <td class="context-item-name">vigilant</td>
136
+ <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
137
+ <td class="context-item-desc"></td>
138
+ </tr>
139
+ </table>
140
+ </div>
141
+ </div>
91
142
 
92
143
 
93
144
 
94
145
  <!-- if method_list -->
146
+ <div id="methods">
147
+ <h3 class="section-bar">Public Instance methods</h3>
148
+
149
+ <div id="method-M000019" class="method-detail">
150
+ <a name="M000019"></a>
151
+
152
+ <div class="method-heading">
153
+ <a href="File.src/M000019.html" target="Code" class="method-signature"
154
+ onclick="popupCode('File.src/M000019.html');return false;">
155
+ <span class="method-name">get_lines</span><span class="method-args">()</span>
156
+ </a>
157
+ </div>
158
+
159
+ <div class="method-description">
160
+ </div>
161
+ </div>
162
+
163
+
164
+ </div>
95
165
 
96
166
 
97
167
  </div>
data/doc/classes/MapReduce/File.src/M000019.html ADDED
@@ -0,0 +1,42 @@
1
+ <?xml version="1.0" encoding="iso-8859-1"?>
2
+ <!DOCTYPE html
3
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
4
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
+
6
+ <html>
7
+ <head>
8
+ <title>get_lines (MapReduce::File)</title>
9
+ <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
10
+ <link rel="stylesheet" href="../../.././rdoc-style.css" type="text/css" media="screen" />
11
+ </head>
12
+ <body class="standalone-code">
13
+ <pre><span class="ruby-comment cmt"># File lib/map_reduce/file.rb, line 25</span>
14
+ <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">get_lines</span>
15
+ <span class="ruby-identifier">t</span> = <span class="ruby-constant">Time</span>.<span class="ruby-identifier">now</span>
16
+
17
+ <span class="ruby-identifier">lines</span> = []
18
+ (<span class="ruby-ivar">@lines_per_client</span> <span class="ruby-operator">||</span> <span class="ruby-value">1</span>).<span class="ruby-identifier">times</span> <span class="ruby-keyword kw">do</span>
19
+ <span class="ruby-keyword kw">begin</span>
20
+ <span class="ruby-keyword kw">if</span> <span class="ruby-ivar">@lock</span>
21
+ <span class="ruby-identifier">sleep</span> <span class="ruby-identifier">locked_queue_wait</span> <span class="ruby-operator">||</span> <span class="ruby-value">1</span>
22
+ <span class="ruby-keyword kw">retry</span>
23
+ <span class="ruby-keyword kw">else</span>
24
+ <span class="ruby-keyword kw">if</span> <span class="ruby-identifier">line</span> = <span class="ruby-identifier">queue</span>.<span class="ruby-identifier">shift</span>
25
+ <span class="ruby-identifier">lines</span> <span class="ruby-operator">&lt;&lt;</span> <span class="ruby-identifier">line</span>
26
+ <span class="ruby-keyword kw">else</span>
27
+ <span class="ruby-identifier">sleep</span> <span class="ruby-identifier">empty_queue_wait</span> <span class="ruby-operator">||</span> <span class="ruby-value">30</span>
28
+ <span class="ruby-keyword kw">retry</span>
29
+ <span class="ruby-keyword kw">end</span>
30
+ <span class="ruby-keyword kw">end</span>
31
+ <span class="ruby-keyword kw">rescue</span> <span class="ruby-constant">MapReduceError</span>
32
+ <span class="ruby-keyword kw">retry</span>
33
+ <span class="ruby-keyword kw">end</span>
34
+ <span class="ruby-keyword kw">end</span>
35
+
36
+ <span class="ruby-ivar">@time_spent_grabbing_objects</span> <span class="ruby-operator">+=</span> (<span class="ruby-constant">Time</span>.<span class="ruby-identifier">now</span> <span class="ruby-operator">-</span> <span class="ruby-identifier">t</span>)
37
+ <span class="ruby-ivar">@num_objects_grabbed</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
38
+
39
+ <span class="ruby-keyword kw">return</span> <span class="ruby-identifier">lines</span>.<span class="ruby-identifier">join</span>
40
+ <span class="ruby-keyword kw">end</span></pre>
41
+ </body>
42
+ </html>
data/doc/classes/MapReduce/File/Client.html ADDED
@@ -0,0 +1,177 @@
1
+ <?xml version="1.0" encoding="iso-8859-1"?>
2
+ <!DOCTYPE html
3
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
4
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
+
6
+ <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7
+ <head>
8
+ <title>Class: MapReduce::File::Client</title>
9
+ <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
10
+ <meta http-equiv="Content-Script-Type" content="text/javascript" />
11
+ <link rel="stylesheet" href="../../.././rdoc-style.css" type="text/css" media="screen" />
12
+ <script type="text/javascript">
13
+ // <![CDATA[
14
+
15
+ function popupCode( url ) {
16
+ window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
17
+ }
18
+
19
+ function toggleCode( id ) {
20
+ if ( document.getElementById )
21
+ elem = document.getElementById( id );
22
+ else if ( document.all )
23
+ elem = eval( "document.all." + id );
24
+ else
25
+ return false;
26
+
27
+ elemStyle = elem.style;
28
+
29
+ if ( elemStyle.display != "block" ) {
30
+ elemStyle.display = "block"
31
+ } else {
32
+ elemStyle.display = "none"
33
+ }
34
+
35
+ return true;
36
+ }
37
+
38
+ // Make codeblocks hidden by default
39
+ document.writeln( "<style type=\"text/css\">div.method-source-code { display: none }</style>" )
40
+
41
+ // ]]>
42
+ </script>
43
+
44
+ </head>
45
+ <body>
46
+
47
+
48
+
49
+ <div id="classHeader">
50
+ <table class="header-table">
51
+ <tr class="top-aligned-row">
52
+ <td><strong>Class</strong></td>
53
+ <td class="class-name-in-header">MapReduce::File::Client</td>
54
+ </tr>
55
+ <tr class="top-aligned-row">
56
+ <td><strong>In:</strong></td>
57
+ <td>
58
+ <a href="../../../files/lib/map_reduce/file_rb.html">
59
+ lib/map_reduce/file.rb
60
+ </a>
61
+ <br />
62
+ </td>
63
+ </tr>
64
+
65
+ <tr class="top-aligned-row">
66
+ <td><strong>Parent:</strong></td>
67
+ <td>
68
+ Object
69
+ </td>
70
+ </tr>
71
+ </table>
72
+ </div>
73
+ <!-- banner header -->
74
+
75
+ <div id="bodyContent">
76
+
77
+
78
+
79
+ <div id="contextContent">
80
+
81
+
82
+
83
+ </div>
84
+
85
+ <div id="method-list">
86
+ <h3 class="section-bar">Methods</h3>
87
+
88
+ <div class="name-list">
89
+ <a href="#M000021">each</a>&nbsp;&nbsp;
90
+ <a href="#M000022">logger</a>&nbsp;&nbsp;
91
+ <a href="#M000020">new</a>&nbsp;&nbsp;
92
+ </div>
93
+ </div>
94
+
95
+ </div>
96
+
97
+
98
+ <!-- if includes -->
99
+ <div id="includes">
100
+ <h3 class="section-bar">Included Modules</h3>
101
+
102
+ <div id="includes-list">
103
+ <span class="include-name">DRbUndumped</span>
104
+ <span class="include-name">Enumerable</span>
105
+ </div>
106
+ </div>
107
+
108
+ <div id="section">
109
+
110
+
111
+
112
+
113
+
114
+
115
+
116
+
117
+ <!-- if method_list -->
118
+ <div id="methods">
119
+ <h3 class="section-bar">Public Class methods</h3>
120
+
121
+ <div id="method-M000020" class="method-detail">
122
+ <a name="M000020"></a>
123
+
124
+ <div class="method-heading">
125
+ <a href="Client.src/M000020.html" target="Code" class="method-signature"
126
+ onclick="popupCode('Client.src/M000020.html');return false;">
127
+ <span class="method-name">new</span><span class="method-args">(server_object)</span>
128
+ </a>
129
+ </div>
130
+
131
+ <div class="method-description">
132
+ </div>
133
+ </div>
134
+
135
+ <h3 class="section-bar">Public Instance methods</h3>
136
+
137
+ <div id="method-M000021" class="method-detail">
138
+ <a name="M000021"></a>
139
+
140
+ <div class="method-heading">
141
+ <a href="Client.src/M000021.html" target="Code" class="method-signature"
142
+ onclick="popupCode('Client.src/M000021.html');return false;">
143
+ <span class="method-name">each</span><span class="method-args">() {|@server_object.get_lines| ...}</span>
144
+ </a>
145
+ </div>
146
+
147
+ <div class="method-description">
148
+ </div>
149
+ </div>
150
+
151
+ <div id="method-M000022" class="method-detail">
152
+ <a name="M000022"></a>
153
+
154
+ <div class="method-heading">
155
+ <a href="Client.src/M000022.html" target="Code" class="method-signature"
156
+ onclick="popupCode('Client.src/M000022.html');return false;">
157
+ <span class="method-name">logger</span><span class="method-args">(*args)</span>
158
+ </a>
159
+ </div>
160
+
161
+ <div class="method-description">
162
+ </div>
163
+ </div>
164
+
165
+
166
+ </div>
167
+
168
+
169
+ </div>
170
+
171
+
172
+ <div id="validator-badges">
173
+ <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
174
+ </div>
175
+
176
+ </body>
177
+ </html>