fare 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.gitignore +20 -0
- data/.rspec +4 -0
- data/.travis.yml +10 -0
- data/Gemfile +4 -0
- data/README.md +470 -0
- data/Rakefile +28 -0
- data/bin/fare +48 -0
- data/fare.gemspec +35 -0
- data/features/multiqueue.feature +60 -0
- data/features/multistack.feature +65 -0
- data/features/step_definitions/aruba.rb +1 -0
- data/features/step_definitions/fare_steps.rb +40 -0
- data/features/subscriber.feature +95 -0
- data/features/support/env.rb +34 -0
- data/lib/fare.rb +96 -0
- data/lib/fare/configuration.rb +57 -0
- data/lib/fare/configuration_dsl.rb +134 -0
- data/lib/fare/configuration_when_locked.rb +82 -0
- data/lib/fare/event.rb +26 -0
- data/lib/fare/generate_lock_file.rb +131 -0
- data/lib/fare/load_configuration_file.rb +45 -0
- data/lib/fare/middleware/logging.rb +46 -0
- data/lib/fare/middleware/newrelic.rb +35 -0
- data/lib/fare/middleware/raven.rb +47 -0
- data/lib/fare/publisher.rb +65 -0
- data/lib/fare/queue_adapter.rb +30 -0
- data/lib/fare/rspec.rb +85 -0
- data/lib/fare/subscriber.rb +35 -0
- data/lib/fare/subscriber_cli.rb +270 -0
- data/lib/fare/subscriber_stack.rb +39 -0
- data/lib/fare/test_mode.rb +204 -0
- data/lib/fare/topic.rb +25 -0
- data/lib/fare/topic_adapter.rb +13 -0
- data/lib/fare/update_cli.rb +41 -0
- data/lib/fare/version.rb +3 -0
- data/spec/logger_spec.rb +45 -0
- data/spec/raven_spec.rb +52 -0
- data/spec/rspec_integration_spec.rb +45 -0
- data/spec/spec_helper.rb +30 -0
- data/spec/stubbed_subscribing_spec.rb +66 -0
- metadata +264 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 5c0019fbf8a276f7c14c283ce4464352e0f7615d
|
4
|
+
data.tar.gz: a7aa3871867ab7a7baead1c33541a23273996dd8
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 593d28b21e3ea505860e1c8b712a88c8ef464879fb332a943f0c7d4d7c4d65e163da8aab7f57780829b1ea5d977e55089fd498550b50919d994c771a3d28f7cf
|
7
|
+
data.tar.gz: 38e798e47e8b446fc0eb3386e9ddc768022186d261b8e48ebbaf32ebc99f540fabf0a47b81d9635f9f3ebddb32b2b445fcbb5f2e7252f83cc975e46b7b51c813
|
data/.gitignore
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
*.gem
|
2
|
+
*.rbc
|
3
|
+
.bundle
|
4
|
+
.config
|
5
|
+
.yardoc
|
6
|
+
Gemfile.lock
|
7
|
+
InstalledFiles
|
8
|
+
_yardoc
|
9
|
+
coverage
|
10
|
+
doc/
|
11
|
+
lib/bundler/man
|
12
|
+
pkg
|
13
|
+
rdoc
|
14
|
+
spec/reports
|
15
|
+
test/tmp
|
16
|
+
test/version_tmp
|
17
|
+
tmp
|
18
|
+
log/
|
19
|
+
config/fare.test.lock
|
20
|
+
spec/**/*.test.lock
|
data/.rspec
ADDED
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,470 @@
|
|
1
|
+
# Fare
|
2
|
+
|
3
|
+
> Get on the message bus with Fare!
|
4
|
+
|
5
|
+
Fare is an event system built around Amazon's SNS and SQS services. You publish
|
6
|
+
an event to SNS, which will automatically be distributed to multiple different
|
7
|
+
queues, from where they can be picked up for processing.
|
8
|
+
|
9
|
+
Fare is built around a couple of conventions. These conventions make it possible
|
10
|
+
to remove knowledge between the services in your architecture. The publisher
|
11
|
+
doesn't know about the subscribers and the subscribers don't know the publisher.
|
12
|
+
This leaves you with the freedom to replace any part without impacting the rest.
|
13
|
+
|
14
|
+
Note: This system is not intended for real time events.
|
15
|
+
|
16
|
+
## In a nutshell
|
17
|
+
|
18
|
+
Let's say you want to send new users a welcome email. The problem is that user
|
19
|
+
accounts are managed in one service, while email is managed via another. And you
|
20
|
+
don't want to burden the accounts service to know about the million things that
|
21
|
+
need to happen after a user signs up.
|
22
|
+
|
23
|
+
Fare lets you publish the event "a user has just signed up" in one service, and
|
24
|
+
let you have a bunch of different services react to it, asynchronously.
|
25
|
+
|
26
|
+
First, the accounts service needs to declare which events it can publish. This
|
27
|
+
is done in a separate file, `config/fare.rb`.
|
28
|
+
|
29
|
+
``` ruby
|
30
|
+
app_name "accounts"
|
31
|
+
|
32
|
+
publishes subject: "user", action: "signup"
|
33
|
+
publishes subject: "user", action: "login"
|
34
|
+
```
|
35
|
+
|
36
|
+
To make sure that the proper topics exist, you run `fare update`. A lockfile
|
37
|
+
is created, similar to how Bundler creates a lockfile containing all the
|
38
|
+
information it figured out before the service needs to start.
|
39
|
+
|
40
|
+
Then in that application, you eventually publish the event:
|
41
|
+
|
42
|
+
``` ruby
|
43
|
+
def create
|
44
|
+
@user = User.create!(params[:user])
|
45
|
+
Fare.publish(subject: "user", action: "signup", payload: @user.attributes)
|
46
|
+
redirect_to some_path
|
47
|
+
end
|
48
|
+
```
|
49
|
+
|
50
|
+
Meanwhile, in your email service, you declare that you are interested when users
|
51
|
+
sign up, so you can send them an email. This is done in the `config/fare.rb` of
|
52
|
+
the email service.
|
53
|
+
|
54
|
+
For each type of event you need to specify a middleware stack, similar to Rack
|
55
|
+
middleware.
|
56
|
+
|
57
|
+
``` ruby
|
58
|
+
app_name "postman_pat" # yes, our mailer is called Postman Pat
|
59
|
+
|
60
|
+
subscriber do
|
61
|
+
|
62
|
+
setup do
|
63
|
+
require "postman_pat"
|
64
|
+
end
|
65
|
+
|
66
|
+
stack do
|
67
|
+
listen_to subject: "user", action: "signup"
|
68
|
+
run do
|
69
|
+
use FilterUnsubscribed
|
70
|
+
use SendMail
|
71
|
+
end
|
72
|
+
end
|
73
|
+
end
|
74
|
+
```
|
75
|
+
|
76
|
+
To make sure that the queues are subscribed to the topics, you run `fare update`.
|
77
|
+
|
78
|
+
Those middleware might look something like:
|
79
|
+
|
80
|
+
``` ruby
|
81
|
+
class FilterUnsubscribed
|
82
|
+
def initialize(app, options = {})
|
83
|
+
@app = app
|
84
|
+
@unsubscribers = options.fetch(:unsubscribers) { Unsubscribers }
|
85
|
+
end
|
86
|
+
|
87
|
+
def call(env)
|
88
|
+
event = env.fetch(:event)
|
89
|
+
email = event.payload.fetch("email")
|
90
|
+
|
91
|
+
# don't continue the middleware chain if users have unsubscribed
|
92
|
+
unless @unsubscribers.include?(email)
|
93
|
+
@app.call(env)
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
99
|
+
Then run `fare subscriber start` and it will start polling for user signup events.
|
100
|
+
|
101
|
+
## How it works
|
102
|
+
|
103
|
+
An event with subject "user" and action "signup" will be published to an SNS
|
104
|
+
topic "production-user-signup". The publisher of the event doesn't need to know
|
105
|
+
anything else.
|
106
|
+
|
107
|
+
A subscriber called "postman_pat", that is interested in that event will create
|
108
|
+
an SNS queue called "production-postman_pat", which is subscribed to the proper
|
109
|
+
SNS topic(s). The subscriber doesn't need to know what the origin of the event
|
110
|
+
is.
|
111
|
+
|
112
|
+
The events themselves are formatted with to a convention. They are in the
|
113
|
+
following JSON format, here is an example:
|
114
|
+
|
115
|
+
``` json
|
116
|
+
{
|
117
|
+
"id": "c51856e0-48bc-4653-8fe2-82c1f549c490",
|
118
|
+
"subject": "user",
|
119
|
+
"action": "signup",
|
120
|
+
"source": "accounts",
|
121
|
+
"version": "0.1.0",
|
122
|
+
"sent_at": "2014-01-01 13:48:01",
|
123
|
+
"payload": "whatever you put in as payload",
|
124
|
+
}
|
125
|
+
```
|
126
|
+
|
127
|
+
The "id" field is a UUID that is generated when the event is published. The
|
128
|
+
"source" is the name of the app that published the event. The "version" field is
|
129
|
+
an optional field that you can use to version the structure of your payloads.
|
130
|
+
The payload is whatever you supplied, but serialized to JSON.
|
131
|
+
|
132
|
+
This JSON is Base64-encoded because of a limitation of SQS (see Limitations below).
|
133
|
+
|
134
|
+
SQS has a neat trick when it comes to polling. If one subscriber picks up the
|
135
|
+
item, it will become invisible and unavailable for others to pick up. After Fare
|
136
|
+
has processed the event, it will be deleted for good. If for some reason you
|
137
|
+
cannot finish processing the message, due to an error or because the timeout
|
138
|
+
(default: 30 seconds) expires, it will become available again. This means that
|
139
|
+
it is really straight forward to run the subscriber multiple times to scale up.
|
140
|
+
That being said, the subscriber is threaded, which means you can already process
|
141
|
+
multiple messages in parallel in one process.
|
142
|
+
|
143
|
+
## Usage
|
144
|
+
|
145
|
+
Similar to Bundler, you'll need to create a configuration file that holds all
|
146
|
+
the topics you want to use in your app. You then need to run a command that
|
147
|
+
will generate a lock version of that configuration file. This is done so that
|
148
|
+
your actions will never have to create a topic or queue when the application is
|
149
|
+
running, slowing your application down in the process.
|
150
|
+
|
151
|
+
### Publishers
|
152
|
+
|
153
|
+
Make a file called `config/fare.rb` that includes configuration like this:
|
154
|
+
|
155
|
+
``` ruby
|
156
|
+
# Commands like `fare update` don't load your application, you need to make sure
|
157
|
+
# it knows how to talk to AWS here.
|
158
|
+
environment :test do
|
159
|
+
AWS.config(
|
160
|
+
secret_access_key: "xxx",
|
161
|
+
access_key_id: "yyy",
|
162
|
+
)
|
163
|
+
end
|
164
|
+
|
165
|
+
app_name "my_publisher"
|
166
|
+
|
167
|
+
# make a line for each topic you want to publish to:
|
168
|
+
publishes subject: "user", action: "signup", version: "0.1"
|
169
|
+
```
|
170
|
+
|
171
|
+
Then run the command to create all the topics and queues:
|
172
|
+
|
173
|
+
```
|
174
|
+
$ fare update
|
175
|
+
```
|
176
|
+
|
177
|
+
This will create a file called `config/fare.production.lock`. When deploying
|
178
|
+
add the `fare update` command to your deployment scripts after you do bundle
|
179
|
+
install.
|
180
|
+
|
181
|
+
You can only publish events listed.
|
182
|
+
|
183
|
+
### Subscribers
|
184
|
+
|
185
|
+
To create a subscriber, you need the same start of `config/fare.rb` as you would
|
186
|
+
normally have. If you publish events while subscribing you need to list them as
|
187
|
+
a normal publisher.
|
188
|
+
|
189
|
+
In addition to the normal configuration, you need to add a subscriber. A
|
190
|
+
subscriber has one "setup" block and one or more "stacks".
|
191
|
+
|
192
|
+
A "stack" lists which events it is interested in with one or more "listen_to"
|
193
|
+
commands.
|
194
|
+
|
195
|
+
The subscriber is one queue and subscribes to all topics mentioned in all
|
196
|
+
stacks. By combining all these events into a single queue, you can lighten
|
197
|
+
the load a bit for events that don't happen all that often and don't need a
|
198
|
+
dedicated queue.
|
199
|
+
|
200
|
+
Example:
|
201
|
+
|
202
|
+
``` ruby
|
203
|
+
environment :test do
|
204
|
+
# ....
|
205
|
+
end
|
206
|
+
|
207
|
+
app_name "my_subscriber"
|
208
|
+
|
209
|
+
# if this app has a subscriber, configure it like this:
|
210
|
+
subscriber do
|
211
|
+
|
212
|
+
# stuff to do when the subscriber starts
|
213
|
+
setup do
|
214
|
+
require "your_app"
|
215
|
+
end
|
216
|
+
|
217
|
+
stack do
|
218
|
+
# have at least one of these lines for each topic you want to subscribe to:
|
219
|
+
listen_to subject: "user", action: "signup"
|
220
|
+
listen_to subject: "user", action: "something_else"
|
221
|
+
run do
|
222
|
+
# process the events like Rack middleware
|
223
|
+
use SomeMiddleware
|
224
|
+
use SomeOtherMiddleware
|
225
|
+
end
|
226
|
+
end
|
227
|
+
|
228
|
+
stack do
|
229
|
+
listen_to subject: "user", action: "stub_toe"
|
230
|
+
run do
|
231
|
+
use SomeOtherMiddleware
|
232
|
+
use SomeMiddleware
|
233
|
+
end
|
234
|
+
end
|
235
|
+
|
236
|
+
end
|
237
|
+
```
|
238
|
+
|
239
|
+
Then run the subscriber:
|
240
|
+
|
241
|
+
```
|
242
|
+
$ fare subscriber start
|
243
|
+
```
|
244
|
+
|
245
|
+
However, you can also create multiple queues within one code base, if you want to.
|
246
|
+
|
247
|
+
To do that, you have to give your subscribers a name:
|
248
|
+
|
249
|
+
``` ruby
|
250
|
+
app_name "my_subscriber"
|
251
|
+
|
252
|
+
subscriber do
|
253
|
+
setup do
|
254
|
+
# ...
|
255
|
+
end
|
256
|
+
|
257
|
+
stack do
|
258
|
+
listen_to ...
|
259
|
+
end
|
260
|
+
end
|
261
|
+
|
262
|
+
subscriber :additional_queue_name do
|
263
|
+
setup do
|
264
|
+
# ...
|
265
|
+
end
|
266
|
+
stack do
|
267
|
+
listen_to ...
|
268
|
+
end
|
269
|
+
end
|
270
|
+
```
|
271
|
+
|
272
|
+
Then you need to start each subscriber individually:
|
273
|
+
|
274
|
+
```
|
275
|
+
$ fare subscriber start --daemonize --name additional_queue_name
|
276
|
+
$ fare subscriber start --daemonize # only runs the unnamed queue
|
277
|
+
```
|
278
|
+
|
279
|
+
### Processing events
|
280
|
+
|
281
|
+
As you saw in the configuration file, Fare works with a middleware like stack.
|
282
|
+
You probably already know how to create middleware from Rack, and middleware for
|
283
|
+
Fare is no different.
|
284
|
+
|
285
|
+
``` ruby
|
286
|
+
class SomeMiddleware
|
287
|
+
|
288
|
+
def initialize(app, options = {})
|
289
|
+
@app = app
|
290
|
+
end
|
291
|
+
|
292
|
+
def call(env)
|
293
|
+
# do some work before
|
294
|
+
event = env.fetch(:event)
|
295
|
+
|
296
|
+
@app.call(env)
|
297
|
+
|
298
|
+
# do some work after
|
299
|
+
end
|
300
|
+
|
301
|
+
end
|
302
|
+
```
|
303
|
+
|
304
|
+
There is no endpoint, because you don't need a special return state as with HTTP
|
305
|
+
requests.
|
306
|
+
|
307
|
+
Here are some things you can access on the event you are passed in:
|
308
|
+
|
309
|
+
``` ruby
|
310
|
+
event.subject # => "user"
|
311
|
+
event.action # => "signup"
|
312
|
+
event.payload # => { "email" => "test@example.org" } # etc
|
313
|
+
event.sent_at # => a datetime object
|
314
|
+
event.version # => "0.1"
|
315
|
+
event.source # => "name_of_publisher"
|
316
|
+
event.id # => an event id automatically added by Fare
|
317
|
+
```
|
318
|
+
|
319
|
+
To run a subscriber that processes the events:
|
320
|
+
|
321
|
+
```
|
322
|
+
$ fare subscriber start
|
323
|
+
```
|
324
|
+
|
325
|
+
Run with the `--help` option to see the options available.
|
326
|
+
|
327
|
+
### Backups
|
328
|
+
|
329
|
+
You can also let Fare automatically create a backup queue. This special queue
|
330
|
+
will be subscribed to all topics. This might come in handy if you want to process statistics.
|
331
|
+
|
332
|
+
To enable, add `backup!` to the toplevel of `config/fare.rb`
|
333
|
+
|
334
|
+
You can create a specialized subscriber to listen to those backups, by calling
|
335
|
+
the subscriber "backup".
|
336
|
+
|
337
|
+
``` ruby
|
338
|
+
subscriber :backup do
|
339
|
+
|
340
|
+
setup do
|
341
|
+
# ...
|
342
|
+
end
|
343
|
+
|
344
|
+
stack do
|
345
|
+
# ...
|
346
|
+
end
|
347
|
+
|
348
|
+
end
|
349
|
+
```
|
350
|
+
|
351
|
+
And you can run it as normal with `$ fare subscriber start --name backup`.
|
352
|
+
|
353
|
+
### Testing
|
354
|
+
|
355
|
+
You can stub publishing of messages when you're testing. Simply add this to your
|
356
|
+
test suite:
|
357
|
+
|
358
|
+
``` ruby
|
359
|
+
Fare.test_mode!
|
360
|
+
```
|
361
|
+
|
362
|
+
You can find the messages that were published:
|
363
|
+
|
364
|
+
``` ruby
|
365
|
+
Fare.stubbed_messages
|
366
|
+
```
|
367
|
+
|
368
|
+
Don't forget to clear this list between tests:
|
369
|
+
|
370
|
+
``` ruby
|
371
|
+
before :each do
|
372
|
+
Fare.stubbed_messages.clear
|
373
|
+
end
|
374
|
+
```
|
375
|
+
|
376
|
+
There is also a matcher available for RSpec:
|
377
|
+
|
378
|
+
``` ruby
|
379
|
+
require "fare/rspec"
|
380
|
+
|
381
|
+
RSpec.describe "Creating a user" do
|
382
|
+
|
383
|
+
it "publishes an event" do
|
384
|
+
expect {
|
385
|
+
# ...
|
386
|
+
}.to publish :user, :signup
|
387
|
+
end
|
388
|
+
|
389
|
+
end
|
390
|
+
```
|
391
|
+
|
392
|
+
Fare provides a fake in memory queue for testing purposes, that you can put
|
393
|
+
events into, and then drain with the Middleware stack you specified in the
|
394
|
+
configuration file. It will not run forever, so you can easily test that events
|
395
|
+
are being handled correctly.
|
396
|
+
|
397
|
+
To test your subscriber, make sure you are in stubbing mode:
|
398
|
+
|
399
|
+
``` ruby
|
400
|
+
Fare.test_mode!
|
401
|
+
```
|
402
|
+
|
403
|
+
Then to simulate an event in the queue:
|
404
|
+
|
405
|
+
``` ruby
|
406
|
+
# given an event:
|
407
|
+
Fare.given_event(subject: "X", action: "Y", payload: "Z")
|
408
|
+
|
409
|
+
# when the event is handled:
|
410
|
+
Fare.run
|
411
|
+
|
412
|
+
# then do the assertions on the expected outcome
|
413
|
+
```
|
414
|
+
|
415
|
+
### Provided Middleware
|
416
|
+
|
417
|
+
There are a couple of middleware provided.
|
418
|
+
|
419
|
+
#### Logging
|
420
|
+
|
421
|
+
Logs events, just provide a logger instance.
|
422
|
+
|
423
|
+
Example:
|
424
|
+
|
425
|
+
``` ruby
|
426
|
+
use Fare::Middleware::Logging, logger: Logger.new($stdout)
|
427
|
+
```
|
428
|
+
|
429
|
+
#### NewRelic
|
430
|
+
|
431
|
+
Instruments the the event processing. You need to require this middleware
|
432
|
+
manually, because we don't want to be loading NewRelic when doing `fare update`.
|
433
|
+
|
434
|
+
Example:
|
435
|
+
|
436
|
+
``` ruby
|
437
|
+
subscriber do
|
438
|
+
setup do
|
439
|
+
require "fare/middleware/newrelic"
|
440
|
+
end
|
441
|
+
always_run do
|
442
|
+
use Fare::Middleware::NewRelic
|
443
|
+
end
|
444
|
+
end
|
445
|
+
```
|
446
|
+
|
447
|
+
|
448
|
+
#### Raven
|
449
|
+
|
450
|
+
Sends errors to Sentry.
|
451
|
+
|
452
|
+
Example:
|
453
|
+
|
454
|
+
``` ruby
|
455
|
+
use Fare::Middleware::Raven, dsn: "http://...", logger: logger, environment: "production"
|
456
|
+
```
|
457
|
+
|
458
|
+
## Limitations
|
459
|
+
|
460
|
+
Before using make sure you know the limitations of Amazon SNS and SQS. For
|
461
|
+
instance, there is a maximum of 256KB for messages in SQS, but since Fare adds
|
462
|
+
some metadata, the limit for payloads is a bit less.
|
463
|
+
|
464
|
+
Events are serialized with Base64, because SQS doesn't like special characters.
|
465
|
+
|
466
|
+
## TODO
|
467
|
+
|
468
|
+
* Let the middleware know how to revert itself
|
469
|
+
* Ask for extra handling time from SQS when handling takes too long
|
470
|
+
* Configuration for SQS Dead Letter queues.
|