fare 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 5c0019fbf8a276f7c14c283ce4464352e0f7615d
4
+ data.tar.gz: a7aa3871867ab7a7baead1c33541a23273996dd8
5
+ SHA512:
6
+ metadata.gz: 593d28b21e3ea505860e1c8b712a88c8ef464879fb332a943f0c7d4d7c4d65e163da8aab7f57780829b1ea5d977e55089fd498550b50919d994c771a3d28f7cf
7
+ data.tar.gz: 38e798e47e8b446fc0eb3386e9ddc768022186d261b8e48ebbaf32ebc99f540fabf0a47b81d9635f9f3ebddb32b2b445fcbb5f2e7252f83cc975e46b7b51c813
@@ -0,0 +1,20 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
18
+ log/
19
+ config/fare.test.lock
20
+ spec/**/*.test.lock
data/.rspec ADDED
@@ -0,0 +1,4 @@
1
+ --color
2
+ --format documentation
3
+ --order random
4
+ --require spec_helper
@@ -0,0 +1,10 @@
1
+ language: ruby
2
+ rvm:
3
+ - 2.0.0
4
+ - 2.1.2
5
+ cache: bundler
6
+ script:
7
+ - DEBUG=true bundle exec rake
8
+ notifications:
9
+ email: robots@yourkarma.com
10
+ slack: karmahq:UEjh8w4cTLqavAf30RZUqFrW
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in fare.gemspec
4
+ gemspec
@@ -0,0 +1,470 @@
1
+ # Fare
2
+
3
+ > Get on the message bus with Fare!
4
+
5
+ Fare is an event system built around Amazon's SNS and SQS services. You publish
6
+ an event to SNS, which will automatically be distributed to multiple different
7
+ queues, from where they can be picked up for processing.
8
+
9
+ Fare is built around a couple of conventions. These conventions make it possible
10
+ to remove knowledge between the services in your architecture. The publisher
11
+ doesn't know about the subscribers and the subscribers don't know the publisher.
12
+ This leaves you with the freedom to replace any part without impacting the rest.
13
+
14
+ Note: This system is not intended for real time events.
15
+
16
+ ## In a nutshell
17
+
18
+ Let's say you want to send new users a welcome email. The problem is that user
19
+ accounts are managed in one service, while email is managed via another. And you
20
+ don't want to burden the accounts service to know about the million things that
21
+ need to happen after a user signs up.
22
+
23
+ Fare lets you publish the event "a user has just signed up" in one service, and
24
+ let you have a bunch of different services react to it, asynchronously.
25
+
26
+ First, the accounts service needs to declare which events it can publish. This
27
+ is done in a separate file, `config/fare.rb`.
28
+
29
+ ``` ruby
30
+ app_name "accounts"
31
+
32
+ publishes subject: "user", action: "signup"
33
+ publishes subject: "user", action: "login"
34
+ ```
35
+
36
+ To make sure that the proper topics exist, you run `fare update`. A lockfile
37
+ is created, similar to how Bundler creates a lockfile containing all the
38
+ information it figured out before the service needs to start.
39
+
40
+ Then in that application, you eventually publish the event:
41
+
42
+ ``` ruby
43
+ def create
44
+ @user = User.create!(params[:user])
45
+ Fare.publish(subject: "user", action: "signup", payload: @user.attributes)
46
+ redirect_to some_path
47
+ end
48
+ ```
49
+
50
+ Meanwhile, in your email service, you declare that you are interested when users
51
+ sign up, so you can send them an email. This is done in the `config/fare.rb` of
52
+ the email service.
53
+
54
+ For each type of event you need to specify a middleware stack, similar to Rack
55
+ middleware.
56
+
57
+ ``` ruby
58
+ app_name "postman_pat" # yes, our mailer is called Postman Pat
59
+
60
+ subscriber do
61
+
62
+ setup do
63
+ require "postman_pat"
64
+ end
65
+
66
+ stack do
67
+ listen_to subject: "user", action: "signup"
68
+ run do
69
+ use FilterUnsubscribed
70
+ use SendMail
71
+ end
72
+ end
73
+ end
74
+ ```
75
+
76
+ To make sure that the queues are subscribed to the topics, you run `fare update`.
77
+
78
+ Those middleware might look something like:
79
+
80
+ ``` ruby
81
+ class FilterUnsubscribed
82
+ def initialize(app, options = {})
83
+ @app = app
84
+ @unsubscribers = options.fetch(:unsubscribers) { Unsubscribers }
85
+ end
86
+
87
+ def call(env)
88
+ event = env.fetch(:event)
89
+ email = event.payload.fetch("email")
90
+
91
+ # don't continue the middleware chain if users have unsubscribed
92
+ unless @unsubscribers.include?(email)
93
+ @app.call(env)
94
+ end
95
+ end
96
+ end
97
+ ```
98
+
99
+ Then run `fare subscriber start` and it will start polling for user signup events.
100
+
101
+ ## How it works
102
+
103
+ An event with subject "user" and action "signup" will be published to an SNS
104
+ topic "production-user-signup". The publisher of the event doesn't need to know
105
+ anything else.
106
+
107
+ A subscriber called "postman_pat", that is interested in that event will create
108
+ an SNS queue called "production-postman_pat", which is subscribed to the proper
109
+ SNS topic(s). The subscriber doesn't need to know what the origin of the event
110
+ is.
111
+
112
+ The events themselves are formatted with to a convention. They are in the
113
+ following JSON format, here is an example:
114
+
115
+ ``` json
116
+ {
117
+ "id": "c51856e0-48bc-4653-8fe2-82c1f549c490",
118
+ "subject": "user",
119
+ "action": "signup",
120
+ "source": "accounts",
121
+ "version": "0.1.0",
122
+ "sent_at": "2014-01-01 13:48:01",
123
+ "payload": "whatever you put in as payload",
124
+ }
125
+ ```
126
+
127
+ The "id" field is a UUID that is generated when the event is published. The
128
+ "source" is the name of the app that published the event. The "version" field is
129
+ an optional field that you can use to version the structure of your payloads.
130
+ The payload is whatever you supplied, but serialized to JSON.
131
+
132
+ This JSON is Base64-encoded because of a limitation of SQS (see Limitations below).
133
+
134
+ SQS has a neat trick when it comes to polling. If one subscriber picks up the
135
+ item, it will become invisible and unavailable for others to pick up. After Fare
136
+ has processed the event, it will be deleted for good. If for some reason you
137
+ cannot finish processing the message, due to an error or because the timeout
138
+ (default: 30 seconds) expires, it will become available again. This means that
139
+ it is really straight forward to run the subscriber multiple times to scale up.
140
+ That being said, the subscriber is threaded, which means you can already process
141
+ multiple messages in parallel in one process.
142
+
143
+ ## Usage
144
+
145
+ Similar to Bundler, you'll need to create a configuration file that holds all
146
+ the topics you want to use in your app. You then need to run a command that
147
+ will generate a lock version of that configuration file. This is done so that
148
+ your actions will never have to create a topic or queue when the application is
149
+ running, slowing your application down in the process.
150
+
151
+ ### Publishers
152
+
153
+ Make a file called `config/fare.rb` that includes configuration like this:
154
+
155
+ ``` ruby
156
+ # Commands like `fare update` don't load your application, you need to make sure
157
+ # it knows how to talk to AWS here.
158
+ environment :test do
159
+ AWS.config(
160
+ secret_access_key: "xxx",
161
+ access_key_id: "yyy",
162
+ )
163
+ end
164
+
165
+ app_name "my_publisher"
166
+
167
+ # make a line for each topic you want to publish to:
168
+ publishes subject: "user", action: "signup", version: "0.1"
169
+ ```
170
+
171
+ Then run the command to create all the topics and queues:
172
+
173
+ ```
174
+ $ fare update
175
+ ```
176
+
177
+ This will create a file called `config/fare.production.lock`. When deploying
178
+ add the `fare update` command to your deployment scripts after you do bundle
179
+ install.
180
+
181
+ You can only publish events listed.
182
+
183
+ ### Subscribers
184
+
185
+ To create a subscriber, you need the same start of `config/fare.rb` as you would
186
+ normally have. If you publish events while subscribing you need to list them as
187
+ a normal publisher.
188
+
189
+ In addition to the normal configuration, you need to add a subscriber. A
190
+ subscriber has one "setup" block and one or more "stacks".
191
+
192
+ A "stack" lists which events it is interested in with one or more "listen_to"
193
+ commands.
194
+
195
+ The subscriber is one queue and subscribes to all topics mentioned in all
196
+ stacks. By combining all these events into a single queue, you can lighten
197
+ the load a bit for events that don't happen all that often and don't need a
198
+ dedicated queue.
199
+
200
+ Example:
201
+
202
+ ``` ruby
203
+ environment :test do
204
+ # ....
205
+ end
206
+
207
+ app_name "my_subscriber"
208
+
209
+ # if this app has a subscriber, configure it like this:
210
+ subscriber do
211
+
212
+ # stuff to do when the subscriber starts
213
+ setup do
214
+ require "your_app"
215
+ end
216
+
217
+ stack do
218
+ # have at least one of these lines for each topic you want to subscribe to:
219
+ listen_to subject: "user", action: "signup"
220
+ listen_to subject: "user", action: "something_else"
221
+ run do
222
+ # process the events like Rack middleware
223
+ use SomeMiddleware
224
+ use SomeOtherMiddleware
225
+ end
226
+ end
227
+
228
+ stack do
229
+ listen_to subject: "user", action: "stub_toe"
230
+ run do
231
+ use SomeOtherMiddleware
232
+ use SomeMiddleware
233
+ end
234
+ end
235
+
236
+ end
237
+ ```
238
+
239
+ Then run the subscriber:
240
+
241
+ ```
242
+ $ fare subscriber start
243
+ ```
244
+
245
+ However, you can also create multiple queues within one code base, if you want to.
246
+
247
+ To do that, you have to give your subscribers a name:
248
+
249
+ ``` ruby
250
+ app_name "my_subscriber"
251
+
252
+ subscriber do
253
+ setup do
254
+ # ...
255
+ end
256
+
257
+ stack do
258
+ listen_to ...
259
+ end
260
+ end
261
+
262
+ subscriber :additional_queue_name do
263
+ setup do
264
+ # ...
265
+ end
266
+ stack do
267
+ listen_to ...
268
+ end
269
+ end
270
+ ```
271
+
272
+ Then you need to start each subscriber individually:
273
+
274
+ ```
275
+ $ fare subscriber start --daemonize --name additional_queue_name
276
+ $ fare subscriber start --daemonize # only runs the unnamed queue
277
+ ```
278
+
279
+ ### Processing events
280
+
281
+ As you saw in the configuration file, Fare works with a middleware like stack.
282
+ You probably already know how to create middleware from Rack, and middleware for
283
+ Fare is no different.
284
+
285
+ ``` ruby
286
+ class SomeMiddleware
287
+
288
+ def initialize(app, options = {})
289
+ @app = app
290
+ end
291
+
292
+ def call(env)
293
+ # do some work before
294
+ event = env.fetch(:event)
295
+
296
+ @app.call(env)
297
+
298
+ # do some work after
299
+ end
300
+
301
+ end
302
+ ```
303
+
304
+ There is no endpoint, because you don't need a special return state as with HTTP
305
+ requests.
306
+
307
+ Here are some things you can access on the event you are passed in:
308
+
309
+ ``` ruby
310
+ event.subject # => "user"
311
+ event.action # => "signup"
312
+ event.payload # => { "email" => "test@example.org" } # etc
313
+ event.sent_at # => a datetime object
314
+ event.version # => "0.1"
315
+ event.source # => "name_of_publisher"
316
+ event.id # => an event id automatically added by Fare
317
+ ```
318
+
319
+ To run a subscriber that processes the events:
320
+
321
+ ```
322
+ $ fare subscriber start
323
+ ```
324
+
325
+ Run with the `--help` option to see the options available.
326
+
327
+ ### Backups
328
+
329
+ You can also let Fare automatically create a backup queue. This special queue
330
+ will be subscribed to all topics. This might come in handy if you want to process statistics.
331
+
332
+ To enable, add `backup!` to the toplevel of `config/fare.rb`
333
+
334
+ You can create a specialized subscriber to listen to those backups, by calling
335
+ the subscriber "backup".
336
+
337
+ ``` ruby
338
+ subscriber :backup do
339
+
340
+ setup do
341
+ # ...
342
+ end
343
+
344
+ stack do
345
+ # ...
346
+ end
347
+
348
+ end
349
+ ```
350
+
351
+ And you can run it as normal with `$ fare subscriber start --name backup`.
352
+
353
+ ### Testing
354
+
355
+ You can stub publishing of messages when you're testing. Simply add this to your
356
+ test suite:
357
+
358
+ ``` ruby
359
+ Fare.test_mode!
360
+ ```
361
+
362
+ You can find the messages that were published:
363
+
364
+ ``` ruby
365
+ Fare.stubbed_messages
366
+ ```
367
+
368
+ Don't forget to clear this list between tests:
369
+
370
+ ``` ruby
371
+ before :each do
372
+ Fare.stubbed_messages.clear
373
+ end
374
+ ```
375
+
376
+ There is also a matcher available for RSpec:
377
+
378
+ ``` ruby
379
+ require "fare/rspec"
380
+
381
+ RSpec.describe "Creating a user" do
382
+
383
+ it "publishes an event" do
384
+ expect {
385
+ # ...
386
+ }.to publish :user, :signup
387
+ end
388
+
389
+ end
390
+ ```
391
+
392
+ Fare provides a fake in memory queue for testing purposes, that you can put
393
+ events into, and then drain with the Middleware stack you specified in the
394
+ configuration file. It will not run forever, so you can easily test that events
395
+ are being handled correctly.
396
+
397
+ To test your subscriber, make sure you are in stubbing mode:
398
+
399
+ ``` ruby
400
+ Fare.test_mode!
401
+ ```
402
+
403
+ Then to simulate an event in the queue:
404
+
405
+ ``` ruby
406
+ # given an event:
407
+ Fare.given_event(subject: "X", action: "Y", payload: "Z")
408
+
409
+ # when the event is handled:
410
+ Fare.run
411
+
412
+ # then do the assertions on the expected outcome
413
+ ```
414
+
415
+ ### Provided Middleware
416
+
417
+ There are a couple of middleware provided.
418
+
419
+ #### Logging
420
+
421
+ Logs events, just provide a logger instance.
422
+
423
+ Example:
424
+
425
+ ``` ruby
426
+ use Fare::Middleware::Logging, logger: Logger.new($stdout)
427
+ ```
428
+
429
+ #### NewRelic
430
+
431
+ Instruments the the event processing. You need to require this middleware
432
+ manually, because we don't want to be loading NewRelic when doing `fare update`.
433
+
434
+ Example:
435
+
436
+ ``` ruby
437
+ subscriber do
438
+ setup do
439
+ require "fare/middleware/newrelic"
440
+ end
441
+ always_run do
442
+ use Fare::Middleware::NewRelic
443
+ end
444
+ end
445
+ ```
446
+
447
+
448
+ #### Raven
449
+
450
+ Sends errors to Sentry.
451
+
452
+ Example:
453
+
454
+ ``` ruby
455
+ use Fare::Middleware::Raven, dsn: "http://...", logger: logger, environment: "production"
456
+ ```
457
+
458
+ ## Limitations
459
+
460
+ Before using make sure you know the limitations of Amazon SNS and SQS. For
461
+ instance, there is a maximum of 256KB for messages in SQS, but since Fare adds
462
+ some metadata, the limit for payloads is a bit less.
463
+
464
+ Events are serialized with Base64, because SQS doesn't like special characters.
465
+
466
+ ## TODO
467
+
468
+ * Let the middleware know how to revert itself
469
+ * Ask for extra handling time from SQS when handling takes too long
470
+ * Configuration for SQS Dead Letter queues.