simple-future 1.0.0.pre1 → 1.0.0.pre2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (26) hide show
  1. checksums.yaml +4 -4
  2. data/LICENSE.txt +21 -0
  3. data/README.md +175 -0
  4. data/Rakefile +27 -0
  5. data/doc/SimpleFuture/ChildError.html +411 -0
  6. data/doc/SimpleFuture/Error.html +140 -0
  7. data/doc/SimpleFuture/ResultTypeError.html +147 -0
  8. data/doc/SimpleFuture.html +998 -0
  9. data/doc/_index.html +152 -0
  10. data/doc/class_list.html +51 -0
  11. data/doc/css/common.css +1 -0
  12. data/doc/css/full_list.css +58 -0
  13. data/doc/css/style.css +499 -0
  14. data/doc/file.README.html +255 -0
  15. data/doc/file_list.html +56 -0
  16. data/doc/frames.html +17 -0
  17. data/doc/index.html +255 -0
  18. data/doc/js/app.js +248 -0
  19. data/doc/js/full_list.js +216 -0
  20. data/doc/js/jquery.js +4 -0
  21. data/doc/method_list.html +147 -0
  22. data/doc/top-level-namespace.html +110 -0
  23. data/simple-future.gemspec +31 -0
  24. data/spec/simple-future_spec.rb +180 -0
  25. data/spec/spec_helper.rb +6 -0
  26. metadata +25 -1
data/doc/file.README.html ADDED
@@ -0,0 +1,255 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>
7
+ File: README
8
+
9
+ &mdash; Documentation by YARD 0.9.12
10
+
11
+ </title>
12
+
13
+ <link rel="stylesheet" href="css/style.css" type="text/css" charset="utf-8" />
14
+
15
+ <link rel="stylesheet" href="css/common.css" type="text/css" charset="utf-8" />
16
+
17
+ <script type="text/javascript" charset="utf-8">
18
+ pathId = "README";
19
+ relpath = '';
20
+ </script>
21
+
22
+
23
+ <script type="text/javascript" charset="utf-8" src="js/jquery.js"></script>
24
+
25
+ <script type="text/javascript" charset="utf-8" src="js/app.js"></script>
26
+
27
+
28
+ </head>
29
+ <body>
30
+ <div class="nav_wrap">
31
+ <iframe id="nav" src="file_list.html?1"></iframe>
32
+ <div id="resizer"></div>
33
+ </div>
34
+
35
+ <div id="main" tabindex="-1">
36
+ <div id="header">
37
+ <div id="menu">
38
+
39
+ <a href="_index.html">Index</a> &raquo;
40
+ <span class="title">File: README</span>
41
+
42
+ </div>
43
+
44
+ <div id="search">
45
+
46
+ <a class="full_list_link" id="class_list_link"
47
+ href="class_list.html">
48
+
49
+ <svg width="24" height="24">
50
+ <rect x="0" y="4" width="24" height="4" rx="1" ry="1"></rect>
51
+ <rect x="0" y="12" width="24" height="4" rx="1" ry="1"></rect>
52
+ <rect x="0" y="20" width="24" height="4" rx="1" ry="1"></rect>
53
+ </svg>
54
+ </a>
55
+
56
+ </div>
57
+ <div class="clear"></div>
58
+ </div>
59
+
60
+ <div id="content"><div id='filecontents'>
61
+ <h1 id="label-SimpleFuture-3A+Simple-2C+Process-based+Concurrency">SimpleFuture: Simple, Process-based Concurrency</h1>
62
+
63
+ <p><code>SimpleFuture</code> is a Ruby module that gives you pretty good
64
+ concurrency without a lot of work. It doesn&#39;t use threads and so will
65
+ avoid their common pitfalls as well working on Rubies that don&#39;t
66
+ support them.</p>
67
+
68
+ <p>It is a simple implementation of <a
69
+ href="https://en.wikipedia.org/wiki/Futures_and_promises">Future</a>
70
+ construct.</p>
71
+
72
+ <h2 id="label-Resources">Resources</h2>
73
+ <ul><li>
74
+ <p><a href="https://github.com/suetanvil/simple-future/">Home Page</a></p>
75
+ </li><li>
76
+ <p><a href="https://github.com/suetanvil/simple-future/issues">Issues</a></p>
77
+ </li><li>
78
+ <p><a href="http://www.rubydoc.info/gems/simple-future/1.0.0/">Reference
79
+ Docs</a></p>
80
+ </li></ul>
81
+
82
+ <h2 id="label-Basic+idea">Basic idea</h2>
83
+
84
+ <p>Suppose you need to do a bunch of long-running things concurrently (e.g.
85
+ transcode a bunch of videos) and you don&#39;t want to have to deal with
86
+ threads.</p>
87
+
88
+ <p>The easy thing to do is fork a subprocess for each item and then use
89
+ <code>Marshal</code> and <code>IO.pipe</code> to retrieve the result(s).</p>
90
+
91
+ <p>That&#39;s pretty much what <code>SimpleFuture</code> does for you. You
92
+ pass it a block and it runs it in a forked child and gives you back the
93
+ result when it&#39;s ready. And as a bonus, it will limit the number of
94
+ children it creates at one time, so you don&#39;t have to.</p>
95
+
96
+ <h2 id="label-Those+videos-2C+for+example">Those videos, for example</h2>
97
+
98
+ <p>So let&#39;s say you&#39;re transcoding a bunch of (legally obtained) video
99
+ files:</p>
100
+
101
+ <pre class="code ruby"><code class="ruby"><span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
102
+ <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='kw'>or</span>
103
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Error transcoding </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_vid'>vid</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span>
104
+ <span class='kw'>end</span>
105
+ </code></pre>
106
+
107
+ <p>This is nice and simple, but it isn&#39;t taking advantage of the zillions
108
+ of cores you have on your fancy modern workstation. So you use
109
+ <code>SimpleFuture</code>:</p>
110
+
111
+ <pre class="code ruby"><code class="ruby"><span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
112
+ <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span>
113
+ <span class='kw'>end</span>
114
+
115
+ <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_wait_for_all'><span class='object_link'><a href="SimpleFuture.html#wait_for_all-class_method" title="SimpleFuture.wait_for_all (method)">wait_for_all</a></span></span>
116
+ </code></pre>
117
+
118
+ <p>And this will do what you want. In particular, it limits itself to one
119
+ process per CPU core so you don&#39;t thrash your system. (You can change
120
+ this limit via <code>SimpleFuture.max_tasks</code>.)</p>
121
+
122
+ <h2 id="label-Wait+for+it">Wait for it</h2>
123
+
124
+ <p>Notice that we end the script with <code>wait_for_all</code>. This is
125
+ because each <code>SimpleFuture</code> needs to have its <code>wait</code>
126
+ method called at some point. This is what <code>wait_for_all</code> does.
127
+ (<code>SimpleFuture.new</code> will also sometimes call an existing
128
+ instance&#39;s <code>wait</code> but you can&#39;t depend on that.)</p>
129
+
130
+ <p>Since <code>wait</code> blocks until the child process finishes, so will
131
+ <code>wait_for_all</code>. If you don&#39;t like that, you can also use
132
+ <code>all_done?</code>; this won&#39;t block so you can do other stuff
133
+ while waiting:</p>
134
+
135
+ <pre class="code ruby"><code class="ruby"><span class='kw'>while</span> <span class='op'>!</span><span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_all_done?'><span class='object_link'><a href="SimpleFuture.html#all_done%3F-class_method" title="SimpleFuture.all_done? (method)">all_done?</a></span></span>
136
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Waiting for child processes to finish.</span><span class='tstring_end'>&quot;</span></span>
137
+ <span class='id identifier rubyid_sleep'>sleep</span> <span class='int'>1</span>
138
+ <span class='kw'>end</span>
139
+ </code></pre>
140
+
141
+ <p>By the way, it&#39;s harmless to call <code>wait_for_all</code> if nothing
142
+ is running so it&#39;s often good practice to just call it before quitting.</p>
143
+
144
+ <h2 id="label-I+need+answers">I need answers</h2>
145
+
146
+ <p>But let&#39;s say that <code>transcode()</code> also returns an important
147
+ result such as the compression ratio and you want to know the average. This
148
+ means that you need to get a result back from the child process.
149
+ Fortunately, <code>SimpleFuture</code> handles this.</p>
150
+
151
+ <p>First, we need to keep all of the <code>SimpleFuture</code> objects:</p>
152
+
153
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_futures'>futures</span> <span class='op'>=</span> <span class='lbracket'>[</span><span class='rbracket'>]</span>
154
+ <span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
155
+ <span class='id identifier rubyid_futures'>futures</span><span class='period'>.</span><span class='id identifier rubyid_push'>push</span><span class='lparen'>(</span><span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span><span class='rparen'>)</span>
156
+ <span class='kw'>end</span>
157
+ </code></pre>
158
+
159
+ <p>Next, we use <code>map</code> to extract the results:</p>
160
+
161
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_ratios'>ratios</span> <span class='op'>=</span> <span class='id identifier rubyid_futures'>futures</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span><span class='lbrace'>{</span><span class='op'>|</span><span class='id identifier rubyid_f'>f</span><span class='op'>|</span> <span class='id identifier rubyid_f'>f</span><span class='period'>.</span><span class='id identifier rubyid_value'>value</span><span class='rbrace'>}</span>
162
+ </code></pre>
163
+
164
+ <p>And then compute the average:</p>
165
+
166
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_sum'>sum</span> <span class='op'>=</span> <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_inject'>inject</span><span class='lparen'>(</span><span class='float'>0.0</span><span class='comma'>,</span> <span class='symbol'>:+</span><span class='rparen'>)</span>
167
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Average compression ratio: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_sum'>sum</span> <span class='op'>/</span> <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span> <span class='kw'>if</span>
168
+ <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span> <span class='op'>&gt;</span> <span class='int'>0</span>
169
+ </code></pre>
170
+
171
+ <p>Note that we&#39;re not calling <code>wait_for_all</code> here. This is
172
+ because <code>value</code> already calls <code>wait</code> and since it
173
+ gets called on each <code>SimpleFuture</code>, we know for sure that all
174
+ child processes have been cleaned up.</p>
175
+
176
+ <p>Also, it may be tempting to merge the loop and the map above but this is a
177
+ <strong>bad idea</strong>:</p>
178
+
179
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_ratios'>ratios</span> <span class='op'>=</span> <span class='lbracket'>[</span><span class='rbracket'>]</span>
180
+ <span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
181
+ <span class='id identifier rubyid_f'>f</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span>
182
+ <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_push'>push</span><span class='lparen'>(</span><span class='id identifier rubyid_f'>f</span><span class='period'>.</span><span class='id identifier rubyid_wait'>wait</span><span class='rparen'>)</span> <span class='comment'># BAD IDEA! DON&#39;T DO THIS!
183
+ </span><span class='kw'>end</span>
184
+ </code></pre>
185
+
186
+ <p>Because <code>wait</code> stops until the child process finishes,
187
+ you&#39;re effectively going back to single-threaded processing. You need
188
+ to create a collection of <code>SimpleFuture</code>s first and
189
+ <em>then</em> <code>wait</code> on them.</p>
190
+
191
+ <h2 id="label-Oooopsie-21">Oooopsie!</h2>
192
+
193
+ <p><code>SimpleFuture</code> tries to only throw exceptions in two cases:
194
+ something is wrong with your code or something has gone wrong beyond its
195
+ control (e.g. Ruby crashed). In either case, the right thing at this point
196
+ is usually to quit. (If that&#39;s not an option, the rubydocs will give
197
+ you the gory details on what throws what.)</p>
198
+
199
+ <p>Generally, you should:</p>
200
+ <ol><li>
201
+ <p>Never quit the child process; always exit the block with a (simple,
202
+ Marshal-compatible) value.</p>
203
+ </li><li>
204
+ <p>Avoid throwing exceptions out of the child block unless it&#39;s to avoid
205
+ breaking rule 1.</p>
206
+ </li></ol>
207
+
208
+ <p>Also, you can probably use low-level systems methods to trick
209
+ <code>SimpleFuture</code> into doing the wrong thing. Don&#39;t do that.</p>
210
+
211
+ <h2 id="label-Installation">Installation</h2>
212
+
213
+ <p>SimpleFuture is available as a gem:</p>
214
+
215
+ <pre class="code ruby"><code class="ruby">$ [sudo] gem install simple-future</code></pre>
216
+
217
+ <p>Source code is available at GitHub:</p>
218
+
219
+ <pre class="code ruby"><code class="ruby">$ git clone https://github.com/suetanvil/simple-future.git
220
+ $ cd simple-future
221
+ $ rake</code></pre>
222
+
223
+ <p>To build, you need to install <code>rake</code>, <code>rspec</code> and
224
+ <code>yard</code>.</p>
225
+
226
+ <p>It should work on Ruby 2.2.0 or later, provided your Ruby/OS combination
227
+ supports <code>Process.fork()</code>. To confirm this, evaluate</p>
228
+
229
+ <pre class="code ruby"><code class="ruby"><span class='const'>Process</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span><span class='lparen'>(</span><span class='symbol'>:fork</span><span class='rparen'>)</span> <span class='comment'># true
230
+ </span></code></pre>
231
+
232
+ <p>If it returns true, you&#39;re good to go. If not, the gem will noisily
233
+ fail.</p>
234
+
235
+ <h2 id="label-Bugs+and+Quirks">Bugs and Quirks</h2>
236
+
237
+ <p><code>SimpleFuture.new</code> will block if there are too many child
238
+ processes. In this case, it waits for the <strong>oldest</strong> process
239
+ to finish, even if other processes have finished in the meantime.</p>
240
+
241
+ <p>The unit tests use <code>sleep</code> to give child processes a relatively
242
+ predictable execution times and test based on that. This means there&#39;s
243
+ a chance that the tests can fail on a sufficiently slow or overloaded
244
+ system. (It&#39;s unlikely that this can happen in real life, though.)</p>
245
+ </div></div>
246
+
247
+ <div id="footer">
248
+ Generated on Wed Jan 17 18:36:16 2018 by
249
+ <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
250
+ 0.9.12 (ruby-2.4.3).
251
+ </div>
252
+
253
+ </div>
254
+ </body>
255
+ </html>
data/doc/file_list.html ADDED
@@ -0,0 +1,56 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
5
+ <meta charset="utf-8" />
6
+
7
+ <link rel="stylesheet" href="css/full_list.css" type="text/css" media="screen" charset="utf-8" />
8
+
9
+ <link rel="stylesheet" href="css/common.css" type="text/css" media="screen" charset="utf-8" />
10
+
11
+
12
+
13
+ <script type="text/javascript" charset="utf-8" src="js/jquery.js"></script>
14
+
15
+ <script type="text/javascript" charset="utf-8" src="js/full_list.js"></script>
16
+
17
+
18
+ <title>File List</title>
19
+ <base id="base_target" target="_parent" />
20
+ </head>
21
+ <body>
22
+ <div id="content">
23
+ <div class="fixed_header">
24
+ <h1 id="full_list_header">File List</h1>
25
+ <div id="full_list_nav">
26
+
27
+ <span><a target="_self" href="class_list.html">
28
+ Classes
29
+ </a></span>
30
+
31
+ <span><a target="_self" href="method_list.html">
32
+ Methods
33
+ </a></span>
34
+
35
+ <span><a target="_self" href="file_list.html">
36
+ Files
37
+ </a></span>
38
+
39
+ </div>
40
+
41
+ <div id="search">Search: <input type="text" /></div>
42
+ </div>
43
+
44
+ <ul id="full_list" class="file">
45
+
46
+
47
+ <li id="object_README" class="odd">
48
+ <div class="item"><span class="object_link"><a href="index.html" title="README">README</a></span></div>
49
+ </li>
50
+
51
+
52
+
53
+ </ul>
54
+ </div>
55
+ </body>
56
+ </html>
data/doc/frames.html ADDED
@@ -0,0 +1,17 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <title>Documentation by YARD 0.9.12</title>
6
+ </head>
7
+ <script type="text/javascript" charset="utf-8">
8
+ var match = unescape(window.location.hash).match(/^#!(.+)/);
9
+ var name = match ? match[1] : 'index.html';
10
+ name = name.replace(/^(\w+):\/\//, '').replace(/^\/\//, '');
11
+ window.top.location = name;
12
+ </script>
13
+ <noscript>
14
+ <h1>Oops!</h1>
15
+ <h2>YARD requires JavaScript!</h2>
16
+ </noscript>
17
+ </html>
data/doc/index.html ADDED
@@ -0,0 +1,255 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>
7
+ File: README
8
+
9
+ &mdash; Documentation by YARD 0.9.12
10
+
11
+ </title>
12
+
13
+ <link rel="stylesheet" href="css/style.css" type="text/css" charset="utf-8" />
14
+
15
+ <link rel="stylesheet" href="css/common.css" type="text/css" charset="utf-8" />
16
+
17
+ <script type="text/javascript" charset="utf-8">
18
+ pathId = "README";
19
+ relpath = '';
20
+ </script>
21
+
22
+
23
+ <script type="text/javascript" charset="utf-8" src="js/jquery.js"></script>
24
+
25
+ <script type="text/javascript" charset="utf-8" src="js/app.js"></script>
26
+
27
+
28
+ </head>
29
+ <body>
30
+ <div class="nav_wrap">
31
+ <iframe id="nav" src="class_list.html?1"></iframe>
32
+ <div id="resizer"></div>
33
+ </div>
34
+
35
+ <div id="main" tabindex="-1">
36
+ <div id="header">
37
+ <div id="menu">
38
+
39
+ <a href="_index.html">Index</a> &raquo;
40
+ <span class="title">File: README</span>
41
+
42
+ </div>
43
+
44
+ <div id="search">
45
+
46
+ <a class="full_list_link" id="class_list_link"
47
+ href="class_list.html">
48
+
49
+ <svg width="24" height="24">
50
+ <rect x="0" y="4" width="24" height="4" rx="1" ry="1"></rect>
51
+ <rect x="0" y="12" width="24" height="4" rx="1" ry="1"></rect>
52
+ <rect x="0" y="20" width="24" height="4" rx="1" ry="1"></rect>
53
+ </svg>
54
+ </a>
55
+
56
+ </div>
57
+ <div class="clear"></div>
58
+ </div>
59
+
60
+ <div id="content"><div id='filecontents'>
61
+ <h1 id="label-SimpleFuture-3A+Simple-2C+Process-based+Concurrency">SimpleFuture: Simple, Process-based Concurrency</h1>
62
+
63
+ <p><code>SimpleFuture</code> is a Ruby module that gives you pretty good
64
+ concurrency without a lot of work. It doesn&#39;t use threads and so will
65
+ avoid their common pitfalls as well working on Rubies that don&#39;t
66
+ support them.</p>
67
+
68
+ <p>It is a simple implementation of <a
69
+ href="https://en.wikipedia.org/wiki/Futures_and_promises">Future</a>
70
+ construct.</p>
71
+
72
+ <h2 id="label-Resources">Resources</h2>
73
+ <ul><li>
74
+ <p><a href="https://github.com/suetanvil/simple-future/">Home Page</a></p>
75
+ </li><li>
76
+ <p><a href="https://github.com/suetanvil/simple-future/issues">Issues</a></p>
77
+ </li><li>
78
+ <p><a href="http://www.rubydoc.info/gems/simple-future/1.0.0/">Reference
79
+ Docs</a></p>
80
+ </li></ul>
81
+
82
+ <h2 id="label-Basic+idea">Basic idea</h2>
83
+
84
+ <p>Suppose you need to do a bunch of long-running things concurrently (e.g.
85
+ transcode a bunch of videos) and you don&#39;t want to have to deal with
86
+ threads.</p>
87
+
88
+ <p>The easy thing to do is fork a subprocess for each item and then use
89
+ <code>Marshal</code> and <code>IO.pipe</code> to retrieve the result(s).</p>
90
+
91
+ <p>That&#39;s pretty much what <code>SimpleFuture</code> does for you. You
92
+ pass it a block and it runs it in a forked child and gives you back the
93
+ result when it&#39;s ready. And as a bonus, it will limit the number of
94
+ children it creates at one time, so you don&#39;t have to.</p>
95
+
96
+ <h2 id="label-Those+videos-2C+for+example">Those videos, for example</h2>
97
+
98
+ <p>So let&#39;s say you&#39;re transcoding a bunch of (legally obtained) video
99
+ files:</p>
100
+
101
+ <pre class="code ruby"><code class="ruby"><span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
102
+ <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='kw'>or</span>
103
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Error transcoding </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_vid'>vid</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span>
104
+ <span class='kw'>end</span>
105
+ </code></pre>
106
+
107
+ <p>This is nice and simple, but it isn&#39;t taking advantage of the zillions
108
+ of cores you have on your fancy modern workstation. So you use
109
+ <code>SimpleFuture</code>:</p>
110
+
111
+ <pre class="code ruby"><code class="ruby"><span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
112
+ <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span>
113
+ <span class='kw'>end</span>
114
+
115
+ <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_wait_for_all'><span class='object_link'><a href="SimpleFuture.html#wait_for_all-class_method" title="SimpleFuture.wait_for_all (method)">wait_for_all</a></span></span>
116
+ </code></pre>
117
+
118
+ <p>And this will do what you want. In particular, it limits itself to one
119
+ process per CPU core so you don&#39;t thrash your system. (You can change
120
+ this limit via <code>SimpleFuture.max_tasks</code>.)</p>
121
+
122
+ <h2 id="label-Wait+for+it">Wait for it</h2>
123
+
124
+ <p>Notice that we end the script with <code>wait_for_all</code>. This is
125
+ because each <code>SimpleFuture</code> needs to have its <code>wait</code>
126
+ method called at some point. This is what <code>wait_for_all</code> does.
127
+ (<code>SimpleFuture.new</code> will also sometimes call an existing
128
+ instance&#39;s <code>wait</code> but you can&#39;t depend on that.)</p>
129
+
130
+ <p>Since <code>wait</code> blocks until the child process finishes, so will
131
+ <code>wait_for_all</code>. If you don&#39;t like that, you can also use
132
+ <code>all_done?</code>; this won&#39;t block so you can do other stuff
133
+ while waiting:</p>
134
+
135
+ <pre class="code ruby"><code class="ruby"><span class='kw'>while</span> <span class='op'>!</span><span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_all_done?'><span class='object_link'><a href="SimpleFuture.html#all_done%3F-class_method" title="SimpleFuture.all_done? (method)">all_done?</a></span></span>
136
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Waiting for child processes to finish.</span><span class='tstring_end'>&quot;</span></span>
137
+ <span class='id identifier rubyid_sleep'>sleep</span> <span class='int'>1</span>
138
+ <span class='kw'>end</span>
139
+ </code></pre>
140
+
141
+ <p>By the way, it&#39;s harmless to call <code>wait_for_all</code> if nothing
142
+ is running so it&#39;s often good practice to just call it before quitting.</p>
143
+
144
+ <h2 id="label-I+need+answers">I need answers</h2>
145
+
146
+ <p>But let&#39;s say that <code>transcode()</code> also returns an important
147
+ result such as the compression ratio and you want to know the average. This
148
+ means that you need to get a result back from the child process.
149
+ Fortunately, <code>SimpleFuture</code> handles this.</p>
150
+
151
+ <p>First, we need to keep all of the <code>SimpleFuture</code> objects:</p>
152
+
153
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_futures'>futures</span> <span class='op'>=</span> <span class='lbracket'>[</span><span class='rbracket'>]</span>
154
+ <span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
155
+ <span class='id identifier rubyid_futures'>futures</span><span class='period'>.</span><span class='id identifier rubyid_push'>push</span><span class='lparen'>(</span><span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span><span class='rparen'>)</span>
156
+ <span class='kw'>end</span>
157
+ </code></pre>
158
+
159
+ <p>Next, we use <code>map</code> to extract the results:</p>
160
+
161
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_ratios'>ratios</span> <span class='op'>=</span> <span class='id identifier rubyid_futures'>futures</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span><span class='lbrace'>{</span><span class='op'>|</span><span class='id identifier rubyid_f'>f</span><span class='op'>|</span> <span class='id identifier rubyid_f'>f</span><span class='period'>.</span><span class='id identifier rubyid_value'>value</span><span class='rbrace'>}</span>
162
+ </code></pre>
163
+
164
+ <p>And then compute the average:</p>
165
+
166
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_sum'>sum</span> <span class='op'>=</span> <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_inject'>inject</span><span class='lparen'>(</span><span class='float'>0.0</span><span class='comma'>,</span> <span class='symbol'>:+</span><span class='rparen'>)</span>
167
+ <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Average compression ratio: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_sum'>sum</span> <span class='op'>/</span> <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span> <span class='kw'>if</span>
168
+ <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span> <span class='op'>&gt;</span> <span class='int'>0</span>
169
+ </code></pre>
170
+
171
+ <p>Note that we&#39;re not calling <code>wait_for_all</code> here. This is
172
+ because <code>value</code> already calls <code>wait</code> and since it
173
+ gets called on each <code>SimpleFuture</code>, we know for sure that all
174
+ child processes have been cleaned up.</p>
175
+
176
+ <p>Also, it may be tempting to merge the loop and the map above but this is a
177
+ <strong>bad idea</strong>:</p>
178
+
179
+ <pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_ratios'>ratios</span> <span class='op'>=</span> <span class='lbracket'>[</span><span class='rbracket'>]</span>
180
+ <span class='kw'>for</span> <span class='id identifier rubyid_vid'>vid</span> <span class='kw'>in</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_glob'>glob</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>*.mp4</span><span class='tstring_end'>&#39;</span></span><span class='rparen'>)</span>
181
+ <span class='id identifier rubyid_f'>f</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="SimpleFuture.html" title="SimpleFuture (class)">SimpleFuture</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="SimpleFuture.html#initialize-instance_method" title="SimpleFuture#initialize (method)">new</a></span></span> <span class='lbrace'>{</span> <span class='id identifier rubyid_run_transcode'>run_transcode</span><span class='lparen'>(</span><span class='id identifier rubyid_vid'>vid</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>../transcoded/</span><span class='tstring_end'>&quot;</span></span><span class='rparen'>)</span> <span class='rbrace'>}</span>
182
+ <span class='id identifier rubyid_ratios'>ratios</span><span class='period'>.</span><span class='id identifier rubyid_push'>push</span><span class='lparen'>(</span><span class='id identifier rubyid_f'>f</span><span class='period'>.</span><span class='id identifier rubyid_wait'>wait</span><span class='rparen'>)</span> <span class='comment'># BAD IDEA! DON&#39;T DO THIS!
183
+ </span><span class='kw'>end</span>
184
+ </code></pre>
185
+
186
+ <p>Because <code>wait</code> stops until the child process finishes,
187
+ you&#39;re effectively going back to single-threaded processing. You need
188
+ to create a collection of <code>SimpleFuture</code>s first and
189
+ <em>then</em> <code>wait</code> on them.</p>
190
+
191
+ <h2 id="label-Oooopsie-21">Oooopsie!</h2>
192
+
193
+ <p><code>SimpleFuture</code> tries to only throw exceptions in two cases:
194
+ something is wrong with your code or something has gone wrong beyond its
195
+ control (e.g. Ruby crashed). In either case, the right thing at this point
196
+ is usually to quit. (If that&#39;s not an option, the rubydocs will give
197
+ you the gory details on what throws what.)</p>
198
+
199
+ <p>Generally, you should:</p>
200
+ <ol><li>
201
+ <p>Never quit the child process; always exit the block with a (simple,
202
+ Marshal-compatible) value.</p>
203
+ </li><li>
204
+ <p>Avoid throwing exceptions out of the child block unless it&#39;s to avoid
205
+ breaking rule 1.</p>
206
+ </li></ol>
207
+
208
+ <p>Also, you can probably use low-level systems methods to trick
209
+ <code>SimpleFuture</code> into doing the wrong thing. Don&#39;t do that.</p>
210
+
211
+ <h2 id="label-Installation">Installation</h2>
212
+
213
+ <p>SimpleFuture is available as a gem:</p>
214
+
215
+ <pre class="code ruby"><code class="ruby">$ [sudo] gem install simple-future</code></pre>
216
+
217
+ <p>Source code is available at GitHub:</p>
218
+
219
+ <pre class="code ruby"><code class="ruby">$ git clone https://github.com/suetanvil/simple-future.git
220
+ $ cd simple-future
221
+ $ rake</code></pre>
222
+
223
+ <p>To build, you need to install <code>rake</code>, <code>rspec</code> and
224
+ <code>yard</code>.</p>
225
+
226
+ <p>It should work on Ruby 2.2.0 or later, provided your Ruby/OS combination
227
+ supports <code>Process.fork()</code>. To confirm this, evaluate</p>
228
+
229
+ <pre class="code ruby"><code class="ruby"><span class='const'>Process</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span><span class='lparen'>(</span><span class='symbol'>:fork</span><span class='rparen'>)</span> <span class='comment'># true
230
+ </span></code></pre>
231
+
232
+ <p>If it returns true, you&#39;re good to go. If not, the gem will noisily
233
+ fail.</p>
234
+
235
+ <h2 id="label-Bugs+and+Quirks">Bugs and Quirks</h2>
236
+
237
+ <p><code>SimpleFuture.new</code> will block if there are too many child
238
+ processes. In this case, it waits for the <strong>oldest</strong> process
239
+ to finish, even if other processes have finished in the meantime.</p>
240
+
241
+ <p>The unit tests use <code>sleep</code> to give child processes a relatively
242
+ predictable execution times and test based on that. This means there&#39;s
243
+ a chance that the tests can fail on a sufficiently slow or overloaded
244
+ system. (It&#39;s unlikely that this can happen in real life, though.)</p>
245
+ </div></div>
246
+
247
+ <div id="footer">
248
+ Generated on Wed Jan 17 18:36:16 2018 by
249
+ <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
250
+ 0.9.12 (ruby-2.4.3).
251
+ </div>
252
+
253
+ </div>
254
+ </body>
255
+ </html>