rightscale-nanite 0.4.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (8) hide show
  1. data/LICENSE +201 -0
  2. data/README.rdoc +431 -0
  3. data/Rakefile +73 -0
  4. data/TODO +24 -0
  5. data/bin/nanite-admin +65 -0
  6. data/bin/nanite-agent +74 -0
  7. data/bin/nanite-mapper +50 -0
  8. metadata +69 -0
data/LICENSE ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
@@ -0,0 +1,431 @@
1
+ = Nanite : A self-assembling fabric of Ruby daemons
2
+
3
+ Google Group: http://groups.google.com/group/nanite
4
+ irc.freenode.net: #nanite
5
+
6
+ == Intro
7
+
8
+ Nanite is a new way of thinking about building cloud ready web applications. Having
9
+ a scalable message queueing back-end with all the discovery and dynamic load based
10
+ dispatch that Nanite has is a very scalable way to construct web application back-ends.
11
+
12
+ A Nanite system has two types of components. There are nanite agents, these are the
13
+ daemons where you implement your code functionality as actors. And then there are
14
+ mappers.
15
+
16
+ Mappers are the control nodes of the system. There can be any number of mappers, these
17
+ typically run inside of your merb or rails app running on the thin webserver
18
+ (eventmachine is needed) but you can also run command line mappers from the shell.
19
+
20
+ Each Nanite agent sends a ping to the mapper exchange every @ping_time seconds. All of
21
+ the mappers are subscribed to this exchange and they all get a copy of the ping with your
22
+ status update. If the mappers do not get a ping from a certain agent within a timeout
23
+ @ping_time the mappers will remove that agent from any dispatch. When the agent comes
24
+ back online or gets less busy it will re-advertise itself to the mapper exchange therefore
25
+ adding itself back to the dispatch. This makes for a very nice self-healing cluster of
26
+ worker processes and a cluster of front-end mapper processes.
27
+
28
+ In your Nanites you can have any number of actor classes. These actors are like controllers
29
+ in Rails or Merb and this is where you implement your own custom functionality. An actor looks
30
+ like:
31
+
32
+ class Foo
33
+ include Nanite::Actor
34
+ expose :bar
35
+ def bar(payload)
36
+ "got payload: #{payload}"
37
+ end
38
+ end
39
+ Nanite::Dispatcher.register(Foo.new)
40
+
41
+ The methods that you 'expose' on the actors are advertised to the mappers like:
42
+
43
+ Foo#bar => /foo/bar
44
+ Mock#list => /mock/list
45
+
46
+ Every agent advertises its status every time it pings the mapper cluster.
47
+ The default status that is advertised is the load average as a float. This
48
+ is used for the default request dispatching based on least loaded server.
49
+
50
+ You can change what is advertised as status to anything you want that is
51
+ comparable(<=>) by doing something like this in your agent's init.rb file:
52
+
53
+ status_proc = lambda { MyApp.some_statistic_indicating_load }
54
+
55
+ This proc will be recalled every @ping_time and sent to the mappers.
56
+
57
+ This is the 'fitness function' for selecting the least loaded Nanite. You can
58
+ dream up any scheme of populating this function so the mappers can select the
59
+ right Nanites based on their status.
60
+
61
+
62
+ == A quick note about security:
63
+
64
+ Nanite security is based upon RabbitMQ vhosts so anything attached to one vhost
65
+ can talk to anything else on the same vhost. So you generally want one vhost
66
+ per app space.
67
+
68
+
69
+ == Installation
70
+
71
+ Nanite has a lot of moving parts. Follow the directions below and we'll get you up
72
+ and running in no-time.
73
+
74
+ === Install Erlang (OS X)
75
+
76
+ See the Erlang website for the latest info : http://www.erlang.org/download.html
77
+
78
+ In your chosen source dir which we'll refer to as <SRC>:
79
+
80
+ cd <SRC>
81
+ wget http://www.erlang.org/download/otp_src_R12B-5.tar.gz
82
+ tar -zxvf otp_src_R12B-5.tar.gz
83
+ cd otp_src_R12B-5
84
+ ./configure --enable-hipe --enable-darwin-universal
85
+ make
86
+ sudo make install
87
+
88
+ === Install Erlang (Linux .deb)
89
+
90
+ sudo apt-get install erlang-nox
91
+
92
+ === Install Nanite Ruby Gem
93
+
94
+ Installing the gem gives us access to the various Nanite commands in the default binary path.
95
+
96
+ cd <SRC>
97
+ git clone git://github.com/ezmobius/nanite.git
98
+ cd nanite
99
+ rake gem
100
+ sudo gem install pkg/nanite-<VERSION>.gem
101
+
102
+
103
+ === Install EventMachine Ruby Gem
104
+
105
+ sudo gem install eventmachine
106
+
107
+ === Install AMQP Ruby Gem
108
+
109
+ # install version >= 0.6.0 from RubyForge
110
+ sudo gem install amqp
111
+
112
+ # or install from source
113
+ cd <SRC>
114
+ git clone git://github.com/tmm1/amqp.git
115
+ cd amqp && rake gem && sudo gem install amqp-<VERSION>.gem
116
+
117
+ === Install RabbitMQ from source tarball (OS X and generic Linux)
118
+
119
+ In short, you'll need a working erlang installation, python, and simplejson. If your python ("python -V") is 2.6, simplejson is included
120
+ in your distribution. Otherwise, install simplejson:
121
+
122
+ easy_install simplejson
123
+
124
+ If you don't have easy_install, install setuptools from http://pypi.python.org/pypi/setuptools:
125
+
126
+ # Get the latest .egg file for your version of python ("python -V"), then run it:
127
+ wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
128
+ sh setuptools-0.6c9-py2.5.egg
129
+
130
+ and install simplejson, which is needed for the "make" step below.
131
+
132
+ These instructions assume the latest RabbitMQ release 1.5.3:
133
+
134
+ # Download somewhere
135
+ cd /root
136
+ wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.3/rabbitmq-server-1.5.3.tar.gz
137
+
138
+ # Go to your erlang lib directory, usually /usr/lib/erlang/lib or:
139
+ cd /usr/local/lib/erlang/lib
140
+
141
+ tar -zxf ~/rabbitmq-server-1.5.3.tar.gz
142
+ cd rabbitmq-server-1.5.3
143
+ make
144
+
145
+ # There is no "make install" phase.
146
+
147
+ Be sure to add the /usr/local/lib/erlang/lib/rabbitmq-server-1.5.3/scripts to your $PATH.
148
+
149
+ The following websites may also be useful:
150
+ * RabbitMQ website for the latest info : http://www.rabbitmq.com/download.html
151
+ * RabbitMQ server source download : http://www.rabbitmq.com/server.html
152
+ * RabbitMQ build instructions : http://www.rabbitmq.com/build-server.html
153
+ * RabbitMQ install instructions : http://www.rabbitmq.com/install.html
154
+
155
+
156
+ === Install RabbitMQ (Linux .deb)
157
+
158
+ wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.3/rabbitmq-server_1.5.3-1_all.deb
159
+ sudo apt-get install logrotate
160
+ sudo dpkg -i rabbitmq-server_1.5.3-1_all.deb
161
+
162
+ == Test your installation
163
+
164
+ === Test your Erlang install
165
+
166
+ Start an Erlang shell
167
+
168
+ erl
169
+
170
+ Enter the following commands in the Erlang shell (When installed correctly each should print a great deal of erlang config info):
171
+
172
+ rabbit:module_info().
173
+
174
+ Exit the Erlang shell with the following command (or Ctrl-c):
175
+
176
+ q().
177
+
178
+
179
+ === Start RabbitMQ
180
+
181
+ All directories will be automatically set up on first run.
182
+
183
+ To run RabbitMQ in the foreground:
184
+ sudo rabbitmq-server
185
+
186
+ To run RabbitMQ in the background:
187
+ sudo rabbitmq-server -detached
188
+
189
+ To check status:
190
+ rabbitmqctl status
191
+
192
+ To stop server:
193
+ rabbitmqctl stop
194
+
195
+ You can learn more about RabbitMQ admin here: http://www.rabbitmq.com/admin-guide.html
196
+
197
+
198
+ === Test Ruby/EventMachine/AMQP end-to-end with a running RabbitMQ instance
199
+
200
+ You can test that all the moving parts are working end-to-end by running one of the AMQP example programs. There are a number of example tests provided with AMQP but the one we will try simply queues and prints two messages immediately, and another one five seconds later.
201
+
202
+ # this test can only be run if you made a local clone
203
+ # of the amqp repository during the amqp installation above
204
+ # (it does not matter however if the gem was installed from src or rubyforge)
205
+ cd <SRC>/amqp
206
+ ruby examples/mq/simple-get.rb
207
+
208
+
209
+ === Test Nanite (finally)
210
+
211
+ First we'll do a little setup and run a script which adds an agent account (nanite), a mapper account (mapper)
212
+ and a '/nanite' vhost to RabbitMQ. RabbitMQ broker must of course be running before you run this script.
213
+
214
+ cd nanite
215
+ ./examples/rabbitconf.rb
216
+
217
+ Now lets run a few agents. Each of these is a long running process and needs to run in its own shell. Each agent also needs its own unique identity, or token to differentiate it from the other running agents. You can try opening a couple of them now if you like. The agents will generally provide no output when running. You can always get more help about the nanite command with 'nanite --help'.
218
+
219
+ First shell
220
+
221
+ cd examples/simpleagent
222
+ nanite-agent --token fred
223
+
224
+ Second shell
225
+
226
+ cd examples/simpleagent
227
+ nanite-agent --token bob
228
+
229
+ Now run a mapper. Mappers can be run from within your Merb or Rails app, from an interactive irb shell, or from the command line. For this example we'll run it from the command line so open a third shell window and run the following:
230
+
231
+ cd examples/simpleagent
232
+ ./cli.rb
233
+
234
+ Which should soon return something like the following.
235
+
236
+ {"bob"=>"hello nanite"} # where the '--token bob' parameter was passed
237
+ {"55a7f300c454203eacc218b6fbd2edc6"=>"hello nanite"} # where no '--token' was passed. auto-generated identity.
238
+
239
+ Now if you want to make this interesting, you can issue a Ctrl-c in one of the agent's windows to kill it. And then run cli.rb again. You should see that you still get a result back since the mapper is finding the remaining agent to do its work.
240
+
241
+ If you want to try requesting work to be done by an agent from an interactive command shell try the following. This assumes that you have the agents running as indicated in the example above ('>>' is the nanite shell prompt).
242
+
243
+ cd nanite
244
+ ./bin/nanite-mapper -i -u mapper -p testing -v /nanite
245
+ Starting mapper console
246
+ >> request('/simple/echo') {|res| p res }
247
+
248
+ By default this will dispatch to the agent with the lowest reported load average.
249
+
250
+ There are a few other selectors as well:
251
+
252
+ # run this request on *all* agents that expose the /foo/bar Foo#bar actor
253
+ >> request('/foo/bar', 'hi', :selector => :all) {|res| p res }
254
+
255
+ # run this request on one random agent that expose the /whatever/hello Whatever#hello actor
256
+ >> request('/whatever/hello', 42, :selector => :random) {|res| p res }
257
+
258
+ You can create your own selectors based on arbitrary stuff you put in status from your agents see cluster.rb for examples of how least_loaded, all and random are implemented.
259
+
260
+ You can run as many mappers as you want, they will all be hot masters.
261
+
262
+ The calls are asynchronous. This means the block you pass to Nanite::Agent#request is not run until the response from the agent(s) have returned. So keep that in mind. Should you need to poll from an ajax web app for results you should have your block stuff the results in the database for any web front end to pick up with the next poll.
263
+
264
+ Another option to test your agents is to use nanite-admin
265
+
266
+ $ nanite-admin
267
+ starting nanite-admin
268
+ >> Thin web server (v1.0.1 codename ?)
269
+ >> Maximum connections set to 1024
270
+ >> Listening on 0.0.0.0:4000, CTRL+C to stop
271
+
272
+ and bring up the Nanite Control Tower at http://localhost:4000 .
273
+
274
+ Have fun!
275
+
276
+ == Web Framework Integration
277
+
278
+ The integration of Nanite into web frameworks depends on the web server running our application and, more importantly, if it uses EventMachine or not.
279
+
280
+ Thin is EventMachine-based, so we only need to make sure that the EventMachine reactor is already 'heated' up
281
+
282
+ Thread.new do
283
+ until EM.reactor_running?
284
+ sleep 1
285
+ end
286
+ Nanite.start_mapper(:host => 'localhost', :user => 'mapper', :pass => 'testing', :vhost => '/nanite', :log_level => 'info')
287
+ end
288
+
289
+ Mongrel on the other hand does not use EventMachine and therefore requires to wrap the start of our mapper
290
+
291
+ Thread.new do
292
+ EM.run do
293
+ Nanite.start_mapper(:host => 'localhost', :user => 'mapper', :pass => 'testing', :vhost => '/nanite', :log_level => 'info')
294
+ end
295
+ end
296
+
297
+ Using nanite with Passenger:
298
+
299
+ current = Thread.current
300
+ Thread.new do
301
+ AMQP.start(:host => AMQP_HOST) do
302
+ current.wakeup
303
+ end
304
+ end
305
+ Thread.stop
306
+
307
+ # catch these, stop AMQP, stop eventmachine, and re-throw to
308
+ Mongrel/Passenger's signal traps
309
+ EM.run do
310
+ ['INT', 'TERM'].each do |sig|
311
+ old = trap(sig) do
312
+ AMQP.stop do
313
+ EM.stop
314
+ old.call
315
+ end
316
+ end
317
+ end
318
+ end
319
+
320
+ =======
321
+ Where to put the mapper initialization code depends on the framework and our preference.
322
+ For Rails the canonical place to start our mapper is within nanite.rb (or any other filename you prefer) in config/initalizers.
323
+ In Merb we can use init.rb in config.
324
+
325
+ == Security
326
+
327
+ Nanite implements a secure serializer which can be used in place of the other serializers to encrypt the
328
+ AMQP messages exchanged between the mappers and the agents.
329
+
330
+ The secure serializer uses X.509 certificates and cryptographic keys to sign and encrypt the messages.
331
+
332
+ It is important to understand that:
333
+ 1. A certificate only includes the public key component of a cryptographic key
334
+ pair.
335
+ 2. Signing requires the use of a certificate and its private key, checking the
336
+ signature then only requires the certificate (the idea is that only the
337
+ signer has the secret private key and thus can sign but anyone can check the
338
+ signature).
339
+ 3. Encrypting only requires the certificate but decrypting also requires the
340
+ private key (anyone can encrypt the data but only the intended recipient can
341
+ decrypt it).
342
+
343
+ A signing serializer thus needs access to the signer certificate and private
344
+ key. An encrypting serializer *also* needs access to the intended recipients
345
+ certificates. There needs to be a way to dynamically retrieve the corresponding
346
+ certificates. This is done using certificate stores.
347
+
348
+ Certificate stores associate identities with certificates. The identity is
349
+ associated when the data is serialized and can be keyed off to retrieve the
350
+ right certificate upon deserialization.
351
+
352
+ Nanite provides a static store implementation which can be used when the
353
+ certificates used for serialization are always the same and can be kept in
354
+ memory. Nanite also provides a certificate store proxy cache which can be
355
+ associated with any store implementation and will cache the most used
356
+ certificates.
357
+
358
+ The serializer should be initialized prior to being used by calling the 'init'
359
+ method:
360
+
361
+ # Initialize serializer, must be called prior to using it.
362
+ #
363
+ # - 'identity': Identity associated with serialized messages
364
+ # - 'cert': Certificate used to sign serialized messages and
365
+ # decrypt encrypted messages
366
+ # - 'key': Private key corresponding to 'cert'
367
+ # - 'store': Certificate store. Exposes certificates used for
368
+ # encryption and signature validation.
369
+ # - 'encrypt': Whether data should be signed and encrypted ('true')
370
+ # or just signed ('false'), 'true' by default.
371
+ #
372
+ def SecureSerializer.init(identity, cert, key, store, encrypt = true)
373
+
374
+ The 'secure' agent example (examples\secure) shows how the mappers and agents should be
375
+ configured to use the secure serializer.
376
+
377
+ == Troubleshooting
378
+
379
+ ** IMPORTANT **
380
+ If you are using Apple's built in Ruby that comes with Leopard (10.5.x) or Tiger (10.4.x) then your
381
+ READLINE lib is hosed and the interactive shell will not work. As soon as you drop into a shell
382
+ Apple's fakey READLINE will halt the event machine loop so you won't see any nanites
383
+ registering. I don't have a good workaround except to tell you not to use Apple's Ruby,
384
+ build your own or use ports.
385
+
386
+
387
+ ** What to do if rabbitconf.rb dies with: {badrpc,nodedown} and nothing you do seems to matter **
388
+
389
+ If rabbitconf.rb dies saying "{badrpc,nodedown}" it means that for some reason,
390
+ the rabbitmqctl program rabbitconf.rb is using to setup the agent accounts for nanite is
391
+ not able to connect to your RabbitMQ server. Assuming RabbitMQ is running and is
392
+ known to work (try the examples that come with the amqp library), then there's a chance
393
+ something is going wrong with using short node names in your Erlang install.
394
+
395
+ The easiest way to verify it is by starting two separate Erlang shells like this (note that "odin" is my hostname):
396
+
397
+ $ erl -sname fred
398
+ (fred@odin)1>
399
+
400
+ $ erl -sname bob
401
+ (bob@odin)1>
402
+
403
+ And then trying to 'ping' one from the other. In the 'fred' node you can do that like this:
404
+
405
+ (fred@odin)1> net_adm:ping(bob@odin).
406
+ pang
407
+
408
+ Note : If your hostname has a hyphen in it (e.g. macbook-pro) you should include the user@hostname in quotes like:
409
+
410
+ net_adm:ping('bob@macbook-pro').
411
+
412
+ If you see 'pang' (which is apparently Swedish for something like "crap! it's broken.") then short name distribution isn't working for you, and you need to fall back to using a full name. If you see 'pong', then that's not actually your problem.
413
+
414
+ First, verify that your system's hostname is set to a fully qualified domain name. On OS X it can end in '.local':
415
+
416
+ hostname
417
+ odin.local
418
+
419
+ Then test that *this* will work by starting a node with the full name of 'fred@<your hostname>':
420
+
421
+ erl -name fred@odin.local
422
+ (fred@odin.local)1>
423
+
424
+ then bob@<yourhostname>, and then finally try to ping fred:
425
+
426
+ erl -name bob@odin.local
427
+ (bob@odin.local)1> net_adm:ping(fred@odin.local).
428
+ pong
429
+
430
+ In my case, it looks like that worked. Now... on to getting rabbitconf.rb to run! To do that, you need to edit the 'rabbitmq-server' and 'rabbitmqctl' scripts in your RabbitMQ distribution and edit the -sname arguments to use -name and a full name.
431
+
@@ -0,0 +1,73 @@
1
+ require 'rubygems'
2
+ require 'rake/gempackagetask'
3
+ require "spec/rake/spectask"
4
+ begin; require 'rubygems'; rescue LoadError; end
5
+ begin
6
+ require 'hanna/rdoctask'
7
+ rescue LoadError
8
+ require 'rake/rdoctask'
9
+ end
10
+ require 'rake/clean'
11
+
12
+ GEM = "nanite"
13
+ VER = "0.4.1"
14
+ AUTHOR = "Ezra Zygmuntowicz"
15
+ EMAIL = "ezra@engineyard.com"
16
+ HOMEPAGE = "http://github.com/ezmobius/nanite"
17
+ SUMMARY = "self assembling fabric of ruby daemons"
18
+
19
+ Dir.glob('tasks/*.rake').each { |r| Rake.application.add_import r }
20
+
21
+ spec = Gem::Specification.new do |s|
22
+ s.name = GEM
23
+ s.version = ::VER
24
+ s.platform = Gem::Platform::RUBY
25
+ s.has_rdoc = true
26
+ s.extra_rdoc_files = ["README.rdoc", "LICENSE", 'TODO']
27
+ s.summary = SUMMARY
28
+ s.description = s.summary
29
+ s.author = AUTHOR
30
+ s.email = EMAIL
31
+ s.homepage = HOMEPAGE
32
+
33
+ s.bindir = "bin"
34
+ s.executables = %w( nanite-agent nanite-mapper nanite-admin )
35
+
36
+ s.add_dependency('amqp', '>= 0.6.0')
37
+
38
+ s.require_path = 'lib'
39
+ s.files = %w(LICENSE README.rdoc Rakefile TODO) + Dir.glob("{lib,bin,specs}/**/*")
40
+ end
41
+
42
+ Rake::GemPackageTask.new(spec) do |pkg|
43
+ pkg.gem_spec = spec
44
+ end
45
+
46
+ task :install => [:package] do
47
+ sh %{sudo gem install pkg/#{GEM}-#{VER}}
48
+ end
49
+
50
+ desc "Run unit specs"
51
+ Spec::Rake::SpecTask.new do |t|
52
+ t.spec_opts = ["--format", "specdoc", "--colour"]
53
+ t.spec_files = FileList["spec/**/*_spec.rb"]
54
+ end
55
+
56
+ desc 'Generate RDoc documentation'
57
+ Rake::RDocTask.new do |rd|
58
+ rd.title = spec.name
59
+ rd.rdoc_dir = 'rdoc'
60
+ rd.main = "README.rdoc"
61
+ rd.rdoc_files.include("lib/**/*.rb", *spec.extra_rdoc_files)
62
+ end
63
+ CLOBBER.include(:clobber_rdoc)
64
+
65
+ desc 'Generate and open documentation'
66
+ task :docs => :rdoc do
67
+ case RUBY_PLATFORM
68
+ when /darwin/ ; sh 'open rdoc/index.html'
69
+ when /mswin|mingw/ ; sh 'start rdoc\index.html'
70
+ else
71
+ sh 'firefox rdoc/index.html'
72
+ end
73
+ end
data/TODO ADDED
@@ -0,0 +1,24 @@
1
+ TODO:
2
+
3
+ - The examples/crew.rb file is pointing towards a hard coded user dir. Needs to be
4
+ documented as part of a working example.
5
+ - examples/async_rack_front/async_rack_front.ru needs to be documented and verified working.
6
+
7
+ Ian:
8
+ - Sync Mapper/Agent#start with nanite-mapper/agent
9
+ - Update docs for Agent#start and Mapper#start
10
+ - Update docs in nanite-agent and nanite-mapper
11
+ - Ensure file transfer works
12
+ - Check secure stuff still works
13
+ - Check custom status_proc works
14
+ - Check documentation, only document public methods
15
+ - ensure the removal of threaded_actors option doesn't cause shit to block
16
+
17
+ - Look into using EM deferables for actors dispatch.
18
+ - Integration specs that spawn a small cluster of nanites
19
+ - Rename Ping to Status
20
+ - request/push should take *args for payload?
21
+
22
+ Maybe:
23
+ - Make mapper queue durable and Results respect :persistent flag on the request
24
+ - Add a global result received callback
@@ -0,0 +1,65 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ # To work without being installed as a gem:
4
+ libdir = File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
5
+ $:.unshift libdir unless $:.include? libdir
6
+
7
+ require 'nanite'
8
+ require 'nanite/admin'
9
+ require 'eventmachine'
10
+ require 'thin'
11
+ # IMPORTANT!
12
+ # You need raggi's patched async version of Thin at the moment to use
13
+ # the nanite-admin tool.
14
+ #
15
+ # raggi's Git repo contains a branch called 'async_for_rack' which contains the
16
+ # version of Thin you want to install. raggi has apparently removed the 'master'
17
+ # branch of his Git repo so you may see a warning like that shown below.
18
+ #
19
+ # git clone git://github.com/raggi/thin.git thin-raggi-async
20
+ # ...
21
+ # warning: remote HEAD refers to nonexistent ref, unable to checkout. <<== IGNORE THIS
22
+ #
23
+ # cd thin-raggi-async/
24
+ # git checkout --track -b async_for_rack origin/async_for_rack
25
+ # warning: You appear to be on a branch yet to be born. <<== IGNORE THIS
26
+ # warning: Forcing checkout of origin/async_for_rack. <<== IGNORE THIS
27
+ # Branch async_for_rack set up to track remote branch refs/remotes/origin/async_for_rack.
28
+ # Switched to a new branch "async_for_rack"
29
+
30
+ # run : 'rake install' to build and install the Thin gem
31
+ # cd <NANITE>
32
+ # ./bin/nanite-admin
33
+
34
+ # When you need to update this Thin install you should be able to do a 'git pull' on the
35
+ # "async_for_rack" branch.
36
+
37
+ require File.dirname(__FILE__) + '/../lib/nanite'
38
+ require 'yaml'
39
+ require "optparse"
40
+
41
+ include Nanite::CommonConfig
42
+
43
+ options = {}
44
+
45
+ opts = OptionParser.new do |opts|
46
+ opts.banner = "Usage: nanite-admin [-flags] [argument]"
47
+ opts.define_head "Nanite Admin: a basic control interface for your nanite cluster."
48
+ opts.separator '*'*80
49
+
50
+ setup_mapper_options(opts, options)
51
+
52
+ opts.on("--thin-debug", "Set the equivalent of the '--debug' flag on the Thin webserver.") do
53
+ options[:thin_debug] = true
54
+ end
55
+ end
56
+
57
+ opts.parse!
58
+
59
+ EM.run do
60
+ Nanite.start_mapper(options)
61
+ Nanite::Log.info "starting nanite-admin"
62
+ Rack::Handler::Thin.run(Nanite::Admin.new(Nanite.mapper), :Port => 4000) do
63
+ Thin::Logging.debug = options[:thin_debug]
64
+ end
65
+ end
@@ -0,0 +1,74 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + '/../lib/nanite'
4
+ require 'optparse'
5
+
6
+ include Nanite::CommonConfig
7
+
8
+ options = {}
9
+
10
+ opts = OptionParser.new do |opts|
11
+ opts.banner = "Usage: nanite-agent [-flag] [argument]"
12
+ opts.define_head "Nanite Agent: ruby process that acts upon messages passed to it by a mapper."
13
+ opts.separator '*'*80
14
+
15
+ setup_common_options(opts, options, 'agent')
16
+
17
+ opts.on("-n", "--nanite NANITE_ROOT", "Specify the root of your nanite agent project.") do |nanite|
18
+ options[:root] = nanite
19
+ end
20
+
21
+ opts.on("--ping-time PINGTIME", "Specify how often the agents contacts the mapper") do |ping|
22
+ options[:ping_time] = ping
23
+ end
24
+
25
+ opts.on("--actors-dir DIR", "Path to directory containing actors (NANITE_ROOT/actors by default)") do |dir|
26
+ options[:actors_dir] = dir
27
+ end
28
+
29
+ opts.on("--actors ACTORS", "Comma separated list of actors to load (all ruby files in actors directory by default)") do |a|
30
+ options[:actors] = a.split(',')
31
+ end
32
+
33
+ opts.on("--initrb FILE", "Path to agent initialization file (NANITE_ROOT/init.rb by default)") do |initrb|
34
+ options[:initrb] = initrb
35
+ end
36
+
37
+ opts.on("--single-threaded", "Run all operations in one thread") do
38
+ options[:single_threaded] = true
39
+ end
40
+ end
41
+
42
+ opts.parse!
43
+
44
+ if ARGV[0] == 'stop' || ARGV[0] == 'status'
45
+ agent = Nanite::Agent.new(options)
46
+ pid_file = Nanite::PidFile.new(agent.identity, agent.options)
47
+ unless pid = pid_file.read_pid
48
+ puts "#{pid_file} not found"
49
+ exit
50
+ end
51
+ if ARGV[0] == 'stop'
52
+ puts "Stopping nanite agent #{agent.identity} (pid #{pid})"
53
+ begin
54
+ Process.kill('TERM', pid)
55
+ rescue Errno::ESRCH
56
+ puts "Process does not exist (pid #{pid})"
57
+ exit
58
+ end
59
+ puts 'Done.'
60
+ else
61
+ if Process.getpgid(pid) != -1
62
+ psdata = `ps up #{pid}`.split("\n").last.split
63
+ memory = (psdata[5].to_i / 1024)
64
+ puts "The agent is alive, using #{memory}MB of memory"
65
+ else
66
+ puts "The agent is not running but has a stale pid file at #{pid_file}"
67
+ end
68
+ end
69
+ exit
70
+ end
71
+
72
+ EM.run do
73
+ Nanite.start_agent(options)
74
+ end
@@ -0,0 +1,50 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + '/../lib/nanite'
4
+ require 'optparse'
5
+
6
+ include Nanite::CommonConfig
7
+
8
+ options = {}
9
+
10
+ opts = OptionParser.new do |opts|
11
+ opts.banner = "Usage: nanite-mapper [-flags] [argument]"
12
+ opts.define_head "Nanite Mapper: clustered head unit for self assembling cluster of ruby processes."
13
+ opts.separator '*'*80
14
+
15
+ setup_mapper_options(opts, options)
16
+ end
17
+
18
+ opts.parse!
19
+
20
+ if ARGV[0] == 'stop' || ARGV[0] == 'status'
21
+ mapper = Nanite::Mapper.new(options)
22
+ pid_file = Nanite::PidFile.new(mapper.identity, mapper.options)
23
+ unless pid = pid_file.read_pid
24
+ puts "#{pid_file} not found"
25
+ exit
26
+ end
27
+ if ARGV[0] == 'stop'
28
+ puts "Stopping nanite mapper #{mapper.identity} (pid #{pid})"
29
+ begin
30
+ Process.kill('TERM', pid)
31
+ rescue Errno::ESRCH
32
+ puts "Process does not exist (pid #{pid})"
33
+ exit
34
+ end
35
+ puts 'Done.'
36
+ else
37
+ if Process.getpgid(pid) != -1
38
+ psdata = `ps up #{pid}`.split("\n").last.split
39
+ memory = (psdata[5].to_i / 1024)
40
+ puts "The mapper is alive, using #{memory}MB of memory"
41
+ else
42
+ puts "The mapper is not running but has a stale pid file at #{pid_file}"
43
+ end
44
+ end
45
+ exit
46
+ end
47
+
48
+ EM.run do
49
+ Nanite.start_mapper(options)
50
+ end
metadata ADDED
@@ -0,0 +1,69 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: rightscale-nanite
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.4.1
5
+ platform: ruby
6
+ authors:
7
+ - Ezra Zygmuntowicz
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+
12
+ date: 2009-05-16 00:00:00 -07:00
13
+ default_executable:
14
+ dependencies:
15
+ - !ruby/object:Gem::Dependency
16
+ name: amqp
17
+ type: :runtime
18
+ version_requirement:
19
+ version_requirements: !ruby/object:Gem::Requirement
20
+ requirements:
21
+ - - ">="
22
+ - !ruby/object:Gem::Version
23
+ version: 0.6.0
24
+ version:
25
+ description: self assembling fabric of ruby daemons
26
+ email: ezra@engineyard.com
27
+ executables:
28
+ - nanite-agent
29
+ - nanite-mapper
30
+ - nanite-admin
31
+ extensions: []
32
+
33
+ extra_rdoc_files:
34
+ - README.rdoc
35
+ - LICENSE
36
+ - TODO
37
+ files:
38
+ - LICENSE
39
+ - README.rdoc
40
+ - Rakefile
41
+ - TODO
42
+ has_rdoc: true
43
+ homepage: http://github.com/ezmobius/nanite
44
+ post_install_message:
45
+ rdoc_options: []
46
+
47
+ require_paths:
48
+ - lib
49
+ required_ruby_version: !ruby/object:Gem::Requirement
50
+ requirements:
51
+ - - ">="
52
+ - !ruby/object:Gem::Version
53
+ version: "0"
54
+ version:
55
+ required_rubygems_version: !ruby/object:Gem::Requirement
56
+ requirements:
57
+ - - ">="
58
+ - !ruby/object:Gem::Version
59
+ version: "0"
60
+ version:
61
+ requirements: []
62
+
63
+ rubyforge_project:
64
+ rubygems_version: 1.2.0
65
+ signing_key:
66
+ specification_version: 2
67
+ summary: self assembling fabric of ruby daemons
68
+ test_files: []
69
+