langis 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/Gemfile +14 -0
- data/LICENSE +202 -0
- data/NOTICE +4 -0
- data/README.md +533 -0
- data/Rakefile +57 -0
- data/VERSION +1 -0
- data/generators/langis_config_generator.rb +7 -0
- data/generators/templates/langis_config.rb +44 -0
- data/lib/langis.rb +60 -0
- data/lib/langis/dsl.rb +346 -0
- data/lib/langis/engine.rb +146 -0
- data/lib/langis/middleware.rb +135 -0
- data/lib/langis/rackish.rb +118 -0
- data/lib/langis/sinks.rb +138 -0
- data/spec/langis/dsl_spec.rb +301 -0
- data/spec/langis/engine_spec.rb +168 -0
- data/spec/langis/middleware_spec.rb +196 -0
- data/spec/langis/rackish_spec.rb +33 -0
- data/spec/langis/sinks/delayed_job_sink_spec.rb +227 -0
- data/spec/langis/sinks/redis_resque_sink_spec.rb +232 -0
- data/spec/redis.conf +132 -0
- data/spec/spec_helper.rb +8 -0
- metadata +171 -0
data/Gemfile
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
source 'http://rubygems.org'
|
2
|
+
|
3
|
+
# Dependencies for base library
|
4
|
+
gem 'blockenspiel'
|
5
|
+
gem 'eventmachine'
|
6
|
+
|
7
|
+
group :spec do
|
8
|
+
gem 'delayed_job'
|
9
|
+
gem 'redis'
|
10
|
+
gem 'resque'
|
11
|
+
gem 'rspec', '2.0.0.beta.10'
|
12
|
+
gem 'sqlite3-ruby', :require => 'sqlite3'
|
13
|
+
gem 'temping'
|
14
|
+
end
|
data/LICENSE
ADDED
@@ -0,0 +1,202 @@
|
|
1
|
+
|
2
|
+
Apache License
|
3
|
+
Version 2.0, January 2004
|
4
|
+
http://www.apache.org/licenses/
|
5
|
+
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
7
|
+
|
8
|
+
1. Definitions.
|
9
|
+
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
12
|
+
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
14
|
+
the copyright owner that is granting the License.
|
15
|
+
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
17
|
+
other entities that control, are controlled by, or are under common
|
18
|
+
control with that entity. For the purposes of this definition,
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
20
|
+
direction or management of such entity, whether by contract or
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
23
|
+
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
25
|
+
exercising permissions granted by this License.
|
26
|
+
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
28
|
+
including but not limited to software source code, documentation
|
29
|
+
source, and configuration files.
|
30
|
+
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
32
|
+
transformation or translation of a Source form, including but
|
33
|
+
not limited to compiled object code, generated documentation,
|
34
|
+
and conversions to other media types.
|
35
|
+
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
37
|
+
Object form, made available under the License, as indicated by a
|
38
|
+
copyright notice that is included in or attached to the work
|
39
|
+
(an example is provided in the Appendix below).
|
40
|
+
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
47
|
+
the Work and Derivative Works thereof.
|
48
|
+
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
50
|
+
the original version of the Work and any modifications or additions
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
62
|
+
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
65
|
+
subsequently incorporated within the Work.
|
66
|
+
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
73
|
+
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
79
|
+
where such license applies only to those patent claims licensable
|
80
|
+
by such Contributor that are necessarily infringed by their
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
83
|
+
institute patent litigation against any entity (including a
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
86
|
+
or contributory patent infringement, then any patent licenses
|
87
|
+
granted to You under this License for that Work shall terminate
|
88
|
+
as of the date such litigation is filed.
|
89
|
+
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
92
|
+
modifications, and in Source or Object form, provided that You
|
93
|
+
meet the following conditions:
|
94
|
+
|
95
|
+
(a) You must give any other recipients of the Work or
|
96
|
+
Derivative Works a copy of this License; and
|
97
|
+
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
99
|
+
stating that You changed the files; and
|
100
|
+
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
103
|
+
attribution notices from the Source form of the Work,
|
104
|
+
excluding those notices that do not pertain to any part of
|
105
|
+
the Derivative Works; and
|
106
|
+
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
109
|
+
include a readable copy of the attribution notices contained
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
112
|
+
of the following places: within a NOTICE text file distributed
|
113
|
+
as part of the Derivative Works; within the Source form or
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
115
|
+
within a display generated by the Derivative Works, if and
|
116
|
+
wherever such third-party notices normally appear. The contents
|
117
|
+
of the NOTICE file are for informational purposes only and
|
118
|
+
do not modify the License. You may add Your own attribution
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
121
|
+
that such additional attribution notices cannot be construed
|
122
|
+
as modifying the License.
|
123
|
+
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
125
|
+
may provide additional or different license terms and conditions
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
129
|
+
the conditions stated in this License.
|
130
|
+
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
134
|
+
this License, without any additional terms or conditions.
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
136
|
+
the terms of any separate license agreement you may have executed
|
137
|
+
with Licensor regarding such Contributions.
|
138
|
+
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
141
|
+
except as required for reasonable and customary use in describing the
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
143
|
+
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
153
|
+
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
159
|
+
incidental, or consequential damages of any character arising as a
|
160
|
+
result of this License or out of the use or inability to use the
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
163
|
+
other commercial damages or losses), even if such Contributor
|
164
|
+
has been advised of the possibility of such damages.
|
165
|
+
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
169
|
+
or other liability obligations and/or rights consistent with this
|
170
|
+
License. However, in accepting such obligations, You may act only
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
175
|
+
of your accepting any such warranty or additional liability.
|
176
|
+
|
177
|
+
END OF TERMS AND CONDITIONS
|
178
|
+
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
180
|
+
|
181
|
+
To apply the Apache License to your work, attach the following
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
183
|
+
replaced with your own identifying information. (Don't include
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
185
|
+
comment syntax for the file format. We also recommend that a
|
186
|
+
file or class name and description of purpose be included on the
|
187
|
+
same "printed page" as the copyright notice for easier
|
188
|
+
identification within third-party archives.
|
189
|
+
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
191
|
+
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
193
|
+
you may not use this file except in compliance with the License.
|
194
|
+
You may obtain a copy of the License at
|
195
|
+
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
197
|
+
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
201
|
+
See the License for the specific language governing permissions and
|
202
|
+
limitations under the License.
|
data/NOTICE
ADDED
data/README.md
ADDED
@@ -0,0 +1,533 @@
|
|
1
|
+
Langis
|
2
|
+
======
|
3
|
+
|
4
|
+
Langis is a Rack inspired publish-subscribe system for Ruby.
|
5
|
+
|
6
|
+
It has flexible message routing and message handling using a custom
|
7
|
+
Domain Specific Language and a Rack-inspired message handler framework.
|
8
|
+
This can give Rails applications better (and decoupled) visibility
|
9
|
+
and management of the background processes it creates (or executes)
|
10
|
+
in response to its actions (in controllers or models).
|
11
|
+
|
12
|
+
Links
|
13
|
+
-----
|
14
|
+
* Repository - <http://github.com/byu/langis>
|
15
|
+
* Yard/RDocs - <http://rdoc.info/projects/byu/langis>
|
16
|
+
* Issues - <http://github.com/byu/langis/issues>
|
17
|
+
|
18
|
+
*Questions?* Message one of the Authors listed below.
|
19
|
+
|
20
|
+
A Brief and Incomplete Overview of Why and How Langis
|
21
|
+
-----------------------------------------------------
|
22
|
+
|
23
|
+
Our main problems:
|
24
|
+
|
25
|
+
* We have long running jobs that get queued up in different controllers,
|
26
|
+
models and model observers. Jobs may even queue up other jobs.
|
27
|
+
Application business process becomes increasingly difficult to maintain
|
28
|
+
as every new change may touch (or add to) different parts of the code.
|
29
|
+
* Higher latency response times because the model observer callbacks are run
|
30
|
+
in the same thread as the Rails request. Clients won't get responses
|
31
|
+
until we finish queuing up all the jobs, or handle the job queuing failures.
|
32
|
+
Example: A queuing failure could be due to a hung queue server that receives
|
33
|
+
the job, but hangs and doesn't return a response there by blocking our
|
34
|
+
Rails thread.
|
35
|
+
* Sometimes we want to execute light-weight tasks like pregenerating
|
36
|
+
(and caching) some content for a user's next page view, but needs to be
|
37
|
+
done with more immediacy than what can be guaranteed by our job libraries.
|
38
|
+
|
39
|
+
How can Langis (Signal spelled backwards) solve this?
|
40
|
+
|
41
|
+
* Langis first postulates that job creation is a response to Events
|
42
|
+
(a type of Message) in the system.
|
43
|
+
* Secondly, we centralize the configuration of channels and their subscribers.
|
44
|
+
This is done using a Domain Specific Language and central configuration in
|
45
|
+
a Rails initializer.
|
46
|
+
* Finally, we have Rack-inspired middleware and applications that is executed
|
47
|
+
in EventMachine deferred thread pools to respond to such Events.
|
48
|
+
|
49
|
+
For example, an ActiveRecord observer model will just publish a "ModelEvent"
|
50
|
+
message (e.g. - to represent Article Created) into Langis instead of directly
|
51
|
+
(tigher coupling) creating respective DelayedJob jobs. Langis will be
|
52
|
+
configured to route the "ModelEvent" to listening Rack-based applications
|
53
|
+
that will then create the jobs (looser coupling).
|
54
|
+
|
55
|
+
A Quick Note on Nomenclature
|
56
|
+
----------------------------
|
57
|
+
|
58
|
+
Langis is inspired by **Rack**, but does not explicitly implement the Rack
|
59
|
+
Specification.
|
60
|
+
|
61
|
+
**Rackish** is used to describe things that are based in Rack, but not
|
62
|
+
actually Rack Specification conformant.
|
63
|
+
|
64
|
+
For example, we use the term *Rackish Application* to talk about an
|
65
|
+
actual Rack Application that doesn't actually require a fully conformant
|
66
|
+
*Rack Environment* as input. To be more clear, Langis does not provide
|
67
|
+
environment variables such as SCRIPT_NAME, rack.version, etc.
|
68
|
+
|
69
|
+
However, it is possible to run real Rack Applications from Langis if the
|
70
|
+
Rack Environment is set up properly by prepending custom middleware to
|
71
|
+
the Rackish Application stack.
|
72
|
+
|
73
|
+
Installation and Usage
|
74
|
+
======================
|
75
|
+
|
76
|
+
Install the gem from gemcutter:
|
77
|
+
|
78
|
+
> sudo gem install 'langis'
|
79
|
+
|
80
|
+
As a plugin:
|
81
|
+
|
82
|
+
> script/plugin install git://github.com/byu/langis.git
|
83
|
+
|
84
|
+
Then add it to the project `Gemfile`.
|
85
|
+
|
86
|
+
> gem 'langis'
|
87
|
+
|
88
|
+
Or add it into the `config/environment.rb` file (only for Rails):
|
89
|
+
|
90
|
+
> config.gem 'langis'
|
91
|
+
|
92
|
+
Dependencies
|
93
|
+
------------
|
94
|
+
Be aware of the dependencies of our dependencies that have been omitted
|
95
|
+
from this list.
|
96
|
+
|
97
|
+
* blockenspiel (Dsl) - Our Domain Specific Language engine
|
98
|
+
* set (Dsl) - Ruby stdlib
|
99
|
+
* eventmachine (Engine)
|
100
|
+
* <http://rubyeventmachine.com/>
|
101
|
+
* <http://github.com/eventmachine/eventmachine>
|
102
|
+
|
103
|
+
Optional Dependencies
|
104
|
+
---------------------
|
105
|
+
* DelayedJob - <http://github.com/tobi/delayed_job>
|
106
|
+
* For Langis::Sinks.delayed_job
|
107
|
+
* Redis - <http://code.google.com/p/redis/>
|
108
|
+
* For Langis::Sinks.redis
|
109
|
+
* Redis-rb - <http://github.com/ezmobius/redis-rb>
|
110
|
+
* Resque - <http://github.com/defunkt/resque>
|
111
|
+
* For Langis::Sinks.resque
|
112
|
+
* ActiveModel - <http://github.com/rails/rails>
|
113
|
+
* To help model your messages
|
114
|
+
|
115
|
+
Configuration
|
116
|
+
-------------
|
117
|
+
|
118
|
+
To use in rails, we provide a generator to create a simple initializer.
|
119
|
+
|
120
|
+
> script/generate langis_config
|
121
|
+
|
122
|
+
It generates the following file:
|
123
|
+
|
124
|
+
> config/initializers/langis_config.rb
|
125
|
+
|
126
|
+
By default, it initializes a LangisEngine that pretty much does nothing.
|
127
|
+
|
128
|
+
LangisEngine = (lambda {
|
129
|
+
# Define the routes
|
130
|
+
config = Langis::Dsl.langis_plumbing do
|
131
|
+
intake :default do
|
132
|
+
flow_to :default
|
133
|
+
end
|
134
|
+
|
135
|
+
for_sink :default do
|
136
|
+
run lambda { |env| puts Rails.logger.info(env.inspect) }
|
137
|
+
end
|
138
|
+
|
139
|
+
check_valve do
|
140
|
+
end
|
141
|
+
end
|
142
|
+
|
143
|
+
# Create an example success callback channel.
|
144
|
+
success_channel = EM::Channel.new
|
145
|
+
success_channel.subscribe(proc do |msg|
|
146
|
+
# TODO: Implement your own success handler.
|
147
|
+
# Rails.logger.info "Success: #{msg.inspect}"
|
148
|
+
end)
|
149
|
+
|
150
|
+
# Create an example error callback channel.
|
151
|
+
error_channel = EM::Channel.new
|
152
|
+
error_channel.subscribe(proc do |msg|
|
153
|
+
# TODO: Implement your own error handler.
|
154
|
+
# Rails.logger.warn "Error: #{msg.inspect}"
|
155
|
+
end)
|
156
|
+
|
157
|
+
# Create and return the actual EventMachine based Langis Engine.
|
158
|
+
return Langis::Engine::EventMachineEngine.new(
|
159
|
+
config.build_pipes,
|
160
|
+
:success_channel => success_channel,
|
161
|
+
:error_channel => error_channel)
|
162
|
+
}).call
|
163
|
+
|
164
|
+
Usage: An Event Model and DelayedJob
|
165
|
+
------------------------------------
|
166
|
+
|
167
|
+
Now one can pump arbitrary messages through the engine to the default intake.
|
168
|
+
|
169
|
+
LangisEngine.pump 'Hello World'
|
170
|
+
|
171
|
+
Or one can target the intake specifically.
|
172
|
+
|
173
|
+
LangisEngine.pump 'Hello World', :default
|
174
|
+
|
175
|
+
It would be more useful to pump messages that are meaningful and routable.
|
176
|
+
In the following example, we use ActiveModel modules to help implement
|
177
|
+
such a message.
|
178
|
+
|
179
|
+
# A generic class used to describe ActiveRecord observable events.
|
180
|
+
class ModelEvent
|
181
|
+
extend ActiveModel::Naming
|
182
|
+
include ActiveModel::Serializers::JSON
|
183
|
+
|
184
|
+
attr_accessor :model_name
|
185
|
+
attr_accessor :model_id
|
186
|
+
attr_accessor :event_type
|
187
|
+
attr_accessor :uuid
|
188
|
+
attr_accessor :timestamp
|
189
|
+
|
190
|
+
def initialize(attributes={})
|
191
|
+
self.model_name = attributes[:model_name]
|
192
|
+
self.model_id = attributes[:model_id]
|
193
|
+
self.event_type = attributes[:event_type]
|
194
|
+
self.uuid = UUID.new
|
195
|
+
self.timestamp = DateTime.now
|
196
|
+
end
|
197
|
+
|
198
|
+
# required by the serializer
|
199
|
+
def attributes
|
200
|
+
{
|
201
|
+
'model_name' => model_name,
|
202
|
+
'model_id' => model_id,
|
203
|
+
'event_type' => event_type,
|
204
|
+
'uuid' => uuid,
|
205
|
+
'timestamp' => timestamp
|
206
|
+
}
|
207
|
+
end
|
208
|
+
|
209
|
+
# Langis introspects the message_type to help route messages.
|
210
|
+
def message_type
|
211
|
+
"#{model_name}_#{event_type}"
|
212
|
+
end
|
213
|
+
end
|
214
|
+
|
215
|
+
Assuming that we have an ActiveModel record for our Rails app:
|
216
|
+
|
217
|
+
class Article < ActiveRecord::Base
|
218
|
+
end
|
219
|
+
|
220
|
+
The ModelEvent object is created in Article's create observer.
|
221
|
+
|
222
|
+
def after_create(article)
|
223
|
+
LangisEngine.pump ModelEvent.new(
|
224
|
+
:model_id => article.id,
|
225
|
+
:model_name => article.class.model_name,
|
226
|
+
:event_type => 'created')
|
227
|
+
end
|
228
|
+
|
229
|
+
The LangisEngine's routes may be configure using the following DSL:
|
230
|
+
|
231
|
+
intake :default do
|
232
|
+
flow_to :xmpp_article, :webhook_article, :when => 'Article_created'
|
233
|
+
end
|
234
|
+
|
235
|
+
for_sink :xmpp_article do
|
236
|
+
run Langis::Sinks.delayed_job XmppArticle, :transform => :model_id
|
237
|
+
end
|
238
|
+
|
239
|
+
for_sink :webhook_article do
|
240
|
+
run Langis::Sinks.delayed_job WebhookArticle, :transform => :model_id
|
241
|
+
end
|
242
|
+
|
243
|
+
The above DSL describes the default intake that accepts messages, which is
|
244
|
+
configured to send messages of message_type "Article_created" to the
|
245
|
+
:xmpp_article and :webhook_article sinks. Also note that a transform is
|
246
|
+
declared for these sinks. The declared transforms execute the :model_id
|
247
|
+
method on each received ModelEvent, which then takes that method's return
|
248
|
+
value an uses it as the DelayedJob's job #new parameters. For Resque sinks,
|
249
|
+
those said return values would be the parameters for the Resque job's perform
|
250
|
+
method. These transforms are used to accommodate the different serialization
|
251
|
+
techniques for different background processing libraries-- DelayedJob's
|
252
|
+
Yaml deserialization isn't so good with ActiveModel based objects.
|
253
|
+
|
254
|
+
class XmppArticle < Struct.new(:article_id)
|
255
|
+
def perform
|
256
|
+
# Load model, create text message, and send Xmpp message
|
257
|
+
end
|
258
|
+
end
|
259
|
+
|
260
|
+
class WebhookArticle < Struct.new(:article_id)
|
261
|
+
def perform
|
262
|
+
# Load model, create xml message, and post to Webhook
|
263
|
+
end
|
264
|
+
end
|
265
|
+
|
266
|
+
Note that DelayedJob 2.0+ requires additional initialization to declare
|
267
|
+
the type of DelayedJob Backend to use. Example:
|
268
|
+
|
269
|
+
Delayed::Worker.backend = :active_record
|
270
|
+
|
271
|
+
Usage: Resque and Json
|
272
|
+
----------------------
|
273
|
+
|
274
|
+
The marshalling for Resque jobs is Json based. So, it is possible to pass
|
275
|
+
in the ModelEvent without using the :transform option. It will be serialized
|
276
|
+
to_json automatically, but deserialized into a Hash object in the Resque job
|
277
|
+
perform. To actually get it back into an actual ModelEvent object, one will
|
278
|
+
have to implement that Hash-to-ModelEvent code.
|
279
|
+
|
280
|
+
# A different job implementation
|
281
|
+
class ArticleResqueWebhook
|
282
|
+
def self.perform(model_event)
|
283
|
+
# This model_event will be a Hash map, the deserialized object
|
284
|
+
# from the ModelEvent#to_json
|
285
|
+
end
|
286
|
+
end
|
287
|
+
|
288
|
+
# Using the same observer
|
289
|
+
def after_create(article)
|
290
|
+
LangisEngine.pump ModelEvent.new(
|
291
|
+
:model_id => article.id,
|
292
|
+
:model_name => article.class.model_name,
|
293
|
+
:event_type => 'created')
|
294
|
+
end
|
295
|
+
|
296
|
+
# In the Langis Dsl
|
297
|
+
intake :default do
|
298
|
+
flow_to :article_resque_webhook, :when => 'Article_created'
|
299
|
+
end
|
300
|
+
for_sink :article_resque_webhook do
|
301
|
+
run Langis::Sinks.resque ArticleResqueWebhook
|
302
|
+
end
|
303
|
+
|
304
|
+
Usage: Route by Intakes
|
305
|
+
-----------------------
|
306
|
+
|
307
|
+
Langis is flexible in the ability to handle different types of messages and
|
308
|
+
routing. For example, we could just pass on the actual ActiveRecord objects to
|
309
|
+
different intakes:
|
310
|
+
|
311
|
+
# In the Article observer
|
312
|
+
def after_create(article)
|
313
|
+
LangisEngine.pump article, :article_created
|
314
|
+
end
|
315
|
+
|
316
|
+
# In the Dsl, assuming all messages to this intake are Article objects.
|
317
|
+
# NOTE: If that can't be guaranteed, then implement a middleware
|
318
|
+
# filter for the alternate_xmpp_article sink.
|
319
|
+
intake :article_created do
|
320
|
+
flow_to :alternate_xmpp_article
|
321
|
+
end
|
322
|
+
|
323
|
+
# Gets the article's id as the input to the job
|
324
|
+
for_sink :alternate_xmpp_article do
|
325
|
+
run Langis::Sinks.delayed_job XmppArticle, :transform => :id
|
326
|
+
end
|
327
|
+
|
328
|
+
Usage: Dump to Redis
|
329
|
+
--------------------
|
330
|
+
|
331
|
+
But now, I want to log each message published into a Redis log. The following
|
332
|
+
takes every message in and RPUSHes its #to_json representation into a
|
333
|
+
Redis key. Implementation note: Redis calls #to_s to serialize objects
|
334
|
+
before saving to the database. So even if the message does not respond to
|
335
|
+
to_json, its to_s (for the following example) will be used.
|
336
|
+
|
337
|
+
REDIS_DB = Redis.new
|
338
|
+
|
339
|
+
intake :default do
|
340
|
+
# This captures all messages
|
341
|
+
flow_to :log_to_redis
|
342
|
+
end
|
343
|
+
|
344
|
+
for_sink :log_to_redis do
|
345
|
+
run Langis::Sinks.redis(REDIS_DB, 'myapp:event_logs',
|
346
|
+
:transform => :to_json)
|
347
|
+
end
|
348
|
+
|
349
|
+
Note that one can reuse the same Redis connection between the Redis
|
350
|
+
sink and the Resque sink.
|
351
|
+
|
352
|
+
# Do this in the initialization before Langis Dsl configuration.
|
353
|
+
REDIS_DB = Redis.new
|
354
|
+
Resque.redis = REDIS_DB
|
355
|
+
|
356
|
+
# And in the Langis Dsl:
|
357
|
+
for_sink :log_to_redis do
|
358
|
+
run Langis::Sinks.redis REDIS_DB, 'myapp:event_logs'
|
359
|
+
end
|
360
|
+
|
361
|
+
Usage: Running Rackish Apps in Background Jobs
|
362
|
+
-----------------------------------------------
|
363
|
+
|
364
|
+
Langis also provides a simple driver class to run Rackish applications as
|
365
|
+
DelayedJob or Resque background jobs. What this means is that a developer
|
366
|
+
can create a Langis Sink (Rackish Application) and have it run either from
|
367
|
+
the thread pool in the main process (Rails) or in background worker processes.
|
368
|
+
This assumes that the env (including the pumped message) can be marshalled
|
369
|
+
by Yaml or Json (as used by DelayedJob and Resque).
|
370
|
+
|
371
|
+
For example, we may want to post data to a webhook.
|
372
|
+
|
373
|
+
# A super simple Rack app that posts data to a uri.
|
374
|
+
class JsonWebhookOutlet
|
375
|
+
def call(env)
|
376
|
+
# Make HTTP POST to uri with json data here.
|
377
|
+
uri = env['uri']
|
378
|
+
data = env['data']
|
379
|
+
|
380
|
+
# Then return the success response.
|
381
|
+
[200, {}, 'OK']
|
382
|
+
end
|
383
|
+
end
|
384
|
+
|
385
|
+
Based on when new articles are created.
|
386
|
+
|
387
|
+
# In the Article observer
|
388
|
+
def after_create(article)
|
389
|
+
LangisEngine.pump article, :article_created
|
390
|
+
end
|
391
|
+
|
392
|
+
We use Langis to handle the observed events.
|
393
|
+
|
394
|
+
# In the Langis Dsl, the following intake is defined
|
395
|
+
intake :article_created do
|
396
|
+
flow_to :webhook_article
|
397
|
+
end
|
398
|
+
|
399
|
+
The following is a sink that will post to the webhook in the background
|
400
|
+
thread of the same process.
|
401
|
+
|
402
|
+
# This Langis Dsl sink definition executes the JsonWebhookOutlet
|
403
|
+
# Rackish application using the thread pool in the main Rails process.
|
404
|
+
# The uri and json data are obtained using Langis middleware transforms;
|
405
|
+
# it assumes that the actual Article instance has the following to_methods.
|
406
|
+
for_sink :webhook_article do
|
407
|
+
use EnvFieldTransform, :to_method => :to_json, :key => 'data'
|
408
|
+
use EnvFieldTransform, :to_method => :get_owner_webhook, :key => 'uri'
|
409
|
+
run JsonWebhookOutlet.new
|
410
|
+
end
|
411
|
+
|
412
|
+
But we really would like to use the background jobs such as the following.
|
413
|
+
This is the alternative Langis sink definition that queues up the
|
414
|
+
work as a background job. It has the same to_method transforms as above.
|
415
|
+
But this sink definition also uses the Parameterizer to create the
|
416
|
+
proper arguments so the RackishJob job will run the json webhook
|
417
|
+
Rackish Application. The Parameterizer is defined to do the following:
|
418
|
+
|
419
|
+
1. Create an Array of 2 items.
|
420
|
+
a. The first item is a fixed string: 'post_to_webhook'.
|
421
|
+
b. The second item is a new hash containing the uri and data elements
|
422
|
+
from the prior EnvFieldTransforms.
|
423
|
+
2. Save the new Array to the the input enviromentment under the key
|
424
|
+
named 'save.to.this.key'.
|
425
|
+
|
426
|
+
The delayed job sink finally queues up the Rackish job with the
|
427
|
+
arguments listed in 'save.to.this.key'.
|
428
|
+
|
429
|
+
for_sink :webhook_article do
|
430
|
+
use EnvFieldTransform, :to_method => :to_json, :key => 'data'
|
431
|
+
use EnvFieldTransform, :to_method => :get_webhook, :key => 'uri'
|
432
|
+
use Langis::Middleware::Parameterizer,
|
433
|
+
'post_to_webhook',
|
434
|
+
lambda { |env|
|
435
|
+
{
|
436
|
+
'uri' => env['uri'],
|
437
|
+
'data' => env['data']
|
438
|
+
}
|
439
|
+
},
|
440
|
+
:env_key => 'save.to.this.key'
|
441
|
+
run Langis::Sinks.delayed_job(
|
442
|
+
Langis::Rackish::RackishJob,
|
443
|
+
:env_key => 'save.to.this.key')
|
444
|
+
end
|
445
|
+
|
446
|
+
And in the background process, we need to wire up the 'post_to_webhook'
|
447
|
+
name to the actual code.
|
448
|
+
|
449
|
+
# This initializer code is run by the background worker process on startup.
|
450
|
+
# It is not needed in the main Rails process.
|
451
|
+
Langis::Rackish::RackishJob.register_rackish_app(
|
452
|
+
'post_to_webhook',
|
453
|
+
Rack::Builder.app do
|
454
|
+
run JsonWebhookOutlet.new
|
455
|
+
end)
|
456
|
+
|
457
|
+
Running EventMachine in Webservers
|
458
|
+
==================================
|
459
|
+
|
460
|
+
The main Langis Engine is built using EventMachine. And one must take care
|
461
|
+
about how to start up EventMachine depending on the web server used.
|
462
|
+
|
463
|
+
Mongrel
|
464
|
+
-------
|
465
|
+
|
466
|
+
Mongrel is simple and single threaded, so you need to run the following
|
467
|
+
somewhere in the initializer code.
|
468
|
+
|
469
|
+
Thread.new do
|
470
|
+
EM.run
|
471
|
+
end
|
472
|
+
|
473
|
+
There shouldn't be any problem if a Message is published to a Langis
|
474
|
+
Intake before EventMachine fully comes up. That message will stay in
|
475
|
+
the EventMachine Channel queue, waiting to be processed once EventMachine
|
476
|
+
does start.
|
477
|
+
|
478
|
+
Thin
|
479
|
+
----
|
480
|
+
|
481
|
+
Thin also uses EventMachine. So you don't need to do anything in particular.
|
482
|
+
|
483
|
+
Passenger and Unicorn
|
484
|
+
---------------------
|
485
|
+
|
486
|
+
TODO: Needs investigating
|
487
|
+
|
488
|
+
Testing
|
489
|
+
=======
|
490
|
+
This library uses [Bundler](http://gembundler.com/) instead
|
491
|
+
of the base system's rubygems to pull in the requirements for tests.
|
492
|
+
|
493
|
+
> bundle install
|
494
|
+
>
|
495
|
+
> rake spec
|
496
|
+
>
|
497
|
+
> rake features
|
498
|
+
>
|
499
|
+
> rake rcov
|
500
|
+
|
501
|
+
However, `rake rcov` requires rcov to be installed in the base system.
|
502
|
+
|
503
|
+
Note on Patches/Pull Requests
|
504
|
+
=============================
|
505
|
+
* Fork the project.
|
506
|
+
* Make your feature addition or bug fix.
|
507
|
+
* Add tests for it. This is important so I don't break it in a
|
508
|
+
future version unintentionally.
|
509
|
+
* Commit, do not mess with rakefile, version, or history.
|
510
|
+
(if you want to have your own version, that is fine but bump version in a
|
511
|
+
commit by itself I can ignore when I pull)
|
512
|
+
* Send me a pull request. Bonus points for topic branches.
|
513
|
+
|
514
|
+
Authors
|
515
|
+
=======
|
516
|
+
* Benjamin Yu - <http://benjaminyu.org/>, <http://github.com/byu>
|
517
|
+
|
518
|
+
Copyright
|
519
|
+
=========
|
520
|
+
|
521
|
+
> Copyright 2009 Benjamin Yu
|
522
|
+
>
|
523
|
+
> Licensed under the Apache License, Version 2.0 (the "License");
|
524
|
+
> you may not use this file except in compliance with the License.
|
525
|
+
> You may obtain a copy of the License at
|
526
|
+
>
|
527
|
+
> http://www.apache.org/licenses/LICENSE-2.0
|
528
|
+
>
|
529
|
+
> Unless required by applicable law or agreed to in writing, software
|
530
|
+
> distributed under the License is distributed on an "AS IS" BASIS,
|
531
|
+
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
532
|
+
> See the License for the specific language governing permissions and
|
533
|
+
> limitations under the License.
|