kthxbye 1.3.0 → 1.3.2
Sign up to get free protection for your applications and to get access to all the features.
- data/{README.rdoc → README.md} +74 -62
- data/lib/kthxbye/job.rb +1 -0
- data/lib/kthxbye/version.rb +1 -1
- data/lib/kthxbye/worker.rb +0 -1
- metadata +32 -35
data/{README.rdoc → README.md}
RENAMED
@@ -1,16 +1,20 @@
|
|
1
|
-
|
1
|
+
# kthxbye
|
2
2
|
|
3
|
-
|
3
|
+
Delayed job processing with async job completion and results notificiation*
|
4
4
|
|
5
|
-
|
5
|
+
_* depends on [Node.js][1] and [Socket.io][2]_
|
6
6
|
|
7
|
-
|
7
|
+
## Quick Links
|
8
8
|
|
9
|
-
|
9
|
+
code: <http://github.com/plukevdh/kthxbye>
|
10
10
|
|
11
|
-
|
11
|
+
docs: <http://rdoc.info/github/plukevdh/kthxbye/>
|
12
12
|
|
13
|
-
|
13
|
+
gem: <https://rubygems.org/gems/kthxbye>
|
14
|
+
|
15
|
+
issues: <http://github.com/plukevdh/kthxbye/issues>
|
16
|
+
|
17
|
+
## History
|
14
18
|
|
15
19
|
Kthxbye is the answer to a fairly unique-yet-common problem: Background job
|
16
20
|
processing when we care about the result.
|
@@ -22,13 +26,13 @@ operation to be returned to the user.
|
|
22
26
|
|
23
27
|
Here's a real-world example. I work with a set of legacy Oracle databases that
|
24
28
|
stores much of our business logic as PLSQL procedures. Yes, this is not "The
|
25
|
-
Rails Way
|
29
|
+
Rails Way ™" but it's the only way for the company I work for right now.
|
26
30
|
Many of the procedures that I run as part of several of the applications I
|
27
31
|
support can take on average one minute or more with a standard deviation of
|
28
32
|
almost 2 minutes (with a forced timeout of 5 minutes). That's kinda a long time
|
29
33
|
to sit and wait on a web app.
|
30
34
|
|
31
|
-
http://img.skitch.com/20100901-gadna641fj4wdeswgj74y2pssq.png
|
35
|
+
![Stats](http://img.skitch.com/20100901-gadna641fj4wdeswgj74y2pssq.png)
|
32
36
|
|
33
37
|
We don't really want users sitting waiting for up to 5 minutes (when it forces
|
34
38
|
failure) unable to do anything or (even worse) hitting refresh or the action
|
@@ -36,22 +40,23 @@ again. Especially bad when this can mean the HTTP server is getting backed up
|
|
36
40
|
as more and more people run these long running processes.
|
37
41
|
|
38
42
|
Moreover, the users need to get response from the completed job before moving
|
39
|
-
on. Most job processors (DJ, Resque) are setup for running jobs that do not
|
43
|
+
on. Most job processors ([DJ][4], [Resque][3]) are setup for running jobs that do not
|
40
44
|
require the result to be returned to the web app (think mass-mailers, queue
|
41
45
|
population, image resizing). They just run and the output goes to a database,
|
42
46
|
an inbox or a file server.
|
43
47
|
|
44
|
-
|
48
|
+
## Enter Kthxbye
|
45
49
|
|
46
50
|
Kthxbye is an attempt to solve this problem. It is based heavily off of
|
47
|
-
|
51
|
+
[Resque][3] and why not an addition to Resque?
|
48
52
|
I needed some hack time with Redis on my own as I've never used it before...
|
49
53
|
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
54
|
+
> I can learn any language or tool in a matter of days if you give me
|
55
|
+
>
|
56
|
+
> 1. a good manual
|
57
|
+
> 2. an even better project to work on.
|
58
|
+
>
|
59
|
+
> _-Prof. Shumacher_
|
55
60
|
|
56
61
|
This project accomplishes both those goals. This is an attempt to learn
|
57
62
|
something, using Resque and Redis docs as a manual, while at the same time
|
@@ -59,48 +64,48 @@ creating a much needed solution to a problem.
|
|
59
64
|
|
60
65
|
The idea is to be able to do the following:
|
61
66
|
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
+
# dummy job class
|
68
|
+
class MyJob
|
69
|
+
def self.perform(data)
|
70
|
+
puts "Do something with #{data}"
|
71
|
+
data.gsub(/hello/i, "Goodbye")
|
72
|
+
end
|
67
73
|
end
|
68
|
-
end
|
69
74
|
|
70
|
-
|
71
|
-
|
75
|
+
# setup options, then connect
|
76
|
+
Kthxbye::Config.setup(:redis_server => 'localhost', :redis_port => 8080)
|
72
77
|
|
73
|
-
|
74
|
-
|
78
|
+
# each enqueued job returns a unique id to poll with
|
79
|
+
unique_id = Kthxbye.enqueue("jobs", MyJob, "Hello World")
|
75
80
|
|
76
|
-
|
81
|
+
# ... code code code ...
|
77
82
|
|
78
|
-
|
79
|
-
|
83
|
+
# polls queue every 5 seconds
|
84
|
+
computed_value = Kthxbye.poll("jobs", unique_id, 5)
|
80
85
|
|
81
86
|
and then in some other world, on some other machine, (that still has knowledge of MyJob)
|
82
87
|
|
83
|
-
|
84
|
-
|
88
|
+
# inits with queue
|
89
|
+
worker = Kthxbye::Worker.new("jobs")
|
85
90
|
|
86
|
-
|
87
|
-
|
91
|
+
# connects to queue and runs jobs found there
|
92
|
+
worker.run
|
88
93
|
|
89
|
-
|
94
|
+
**Pretty... damn... simple.™**
|
90
95
|
|
91
|
-
|
96
|
+
## Installation
|
92
97
|
|
93
98
|
Installation isn't hard. Simply:
|
94
99
|
|
95
|
-
|
100
|
+
gem install kthxbye
|
96
101
|
|
97
102
|
or in Rails
|
98
103
|
|
99
|
-
|
104
|
+
gem "kthxbye"
|
100
105
|
|
101
|
-
and then in your project directory run
|
106
|
+
and then in your project directory (if using Rails) run
|
102
107
|
|
103
|
-
|
108
|
+
rails g kthxbye
|
104
109
|
|
105
110
|
which will install all of the necessary assets, of which there are three:
|
106
111
|
|
@@ -112,30 +117,32 @@ These can be ignored if you really don't want to have an asynchronus widget to
|
|
112
117
|
display the job's status to the user in real-time. You can simply use the gem
|
113
118
|
and get the results on your own time.
|
114
119
|
|
115
|
-
If are using the widget however, we depend on
|
116
|
-
handle the Redis PUBSUB listening and updating and
|
120
|
+
If are using the widget however, we depend on [node.js][1] to
|
121
|
+
handle the Redis PUBSUB listening and updating and [Socket.io][2]
|
117
122
|
to stream the results back to the clients.
|
118
123
|
|
119
124
|
In order to install Socket.io on your application side, run
|
120
125
|
|
121
|
-
|
126
|
+
git clone http://github.com/LearnBoost/Socket.IO.git socket-io --recursive
|
122
127
|
|
123
|
-
|
128
|
+
**Note:** due to the way Rails tries to serve these files and the seemingly
|
124
129
|
conflicting way that Socket.io has hardcoded some paths into the code, you may
|
125
130
|
need to create a symlink to the public/javascripts/socket.io directory in you
|
126
131
|
public directory. This was the easiest solution I found.
|
127
132
|
|
128
|
-
There is a separate node.js client at
|
133
|
+
There is a separate node.js client at <http://github.com/plukevdh/kthxbye-node>.
|
129
134
|
Simply clone the repo or download the poll.js file and set it up to your hearts
|
130
|
-
content.
|
135
|
+
content.
|
136
|
+
|
137
|
+
*NOTE:* You will need to configure your ports/redis server in the
|
131
138
|
poll.js file as well as installing socket.io in the same dir as the poll.js file
|
132
|
-
via
|
133
|
-
(from the socket.io install instructions http://github.com/LearnBoost/Socket.IO-node)
|
139
|
+
via `git clone git://github.com/LearnBoost/Socket.IO-node.git socket.io-node --recursive`
|
140
|
+
(from the [socket.io][2] install instructions <http://github.com/LearnBoost/Socket.IO-node>)
|
134
141
|
|
135
142
|
|
136
|
-
|
143
|
+
## Troubleshooting
|
137
144
|
|
138
|
-
|
145
|
+
### The part of the show where I try to save you some trouble
|
139
146
|
|
140
147
|
There are a few things I found along the way that were necessary to make this
|
141
148
|
integration run super smoothly:
|
@@ -147,16 +154,21 @@ You may need to create a link in your public/ dir to socket.io to the install
|
|
147
154
|
in your public/javascripts/ dir.
|
148
155
|
|
149
156
|
|
150
|
-
|
157
|
+
## Want to Help? Rock on.
|
151
158
|
|
152
|
-
* Fork
|
153
|
-
*
|
154
|
-
*
|
155
|
-
|
156
|
-
*
|
157
|
-
|
158
|
-
*
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
Copyright
|
159
|
+
* Fork.
|
160
|
+
* Add or fix stuff.
|
161
|
+
* Prove it works with everything else.
|
162
|
+
* Commit changes.
|
163
|
+
* Send me a pull request!
|
164
|
+
* Glory in the fact that your name is now in the annals of open-sourcedom.
|
165
|
+
* Repeat.
|
166
|
+
|
167
|
+
## Copyright
|
168
|
+
|
169
|
+
Copyright © 2010 Luke van der Hoeven. See LICENSE for details.
|
170
|
+
|
171
|
+
[1]: http://nodejs.org
|
172
|
+
[2]: http://socket.io
|
173
|
+
[3]: http://github.com/defunkt/resque
|
174
|
+
[4]: http://github.com/tobi/delayed_job
|
data/lib/kthxbye/job.rb
CHANGED
data/lib/kthxbye/version.rb
CHANGED
data/lib/kthxbye/worker.rb
CHANGED
@@ -179,7 +179,6 @@ module Kthxbye
|
|
179
179
|
redis.srem( :working, self )
|
180
180
|
redis.del( "worker:#{self}" )
|
181
181
|
log "Completed job #{@current_job}"
|
182
|
-
redis.publish("job.completed", @current_job.id)
|
183
182
|
Stats.incr("processed")
|
184
183
|
Stats.incr("processed:#{self}")
|
185
184
|
@current_job = nil
|
metadata
CHANGED
@@ -1,13 +1,12 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: kthxbye
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash: 27
|
5
4
|
prerelease: false
|
6
5
|
segments:
|
7
6
|
- 1
|
8
7
|
- 3
|
9
|
-
-
|
10
|
-
version: 1.3.
|
8
|
+
- 2
|
9
|
+
version: 1.3.2
|
11
10
|
platform: ruby
|
12
11
|
authors:
|
13
12
|
- Luke van der Hoeven
|
@@ -15,7 +14,7 @@ autorequire:
|
|
15
14
|
bindir: bin
|
16
15
|
cert_chain: []
|
17
16
|
|
18
|
-
date: 2010-
|
17
|
+
date: 2010-12-01 00:00:00 -05:00
|
19
18
|
default_executable: kthxbye-monitor
|
20
19
|
dependencies: []
|
21
20
|
|
@@ -27,53 +26,53 @@ extensions: []
|
|
27
26
|
|
28
27
|
extra_rdoc_files:
|
29
28
|
- LICENSE
|
30
|
-
- README.
|
29
|
+
- README.md
|
31
30
|
files:
|
32
31
|
- bin/kthxbye-monitor
|
33
|
-
- lib/kthxbye/
|
34
|
-
- lib/kthxbye/
|
32
|
+
- lib/generators/kthxbye/kthxbye_generator.rb
|
33
|
+
- lib/generators/kthxbye/templates/kthxbye.css
|
34
|
+
- lib/generators/kthxbye/templates/kthxbye.js
|
35
|
+
- lib/generators/kthxbye/templates/kthxbye_widget.png
|
36
|
+
- lib/generators/kthxbye/USAGE
|
37
|
+
- lib/kthxbye/config.rb
|
35
38
|
- lib/kthxbye/exceptions.rb
|
36
|
-
- lib/kthxbye/
|
37
|
-
- lib/kthxbye/
|
39
|
+
- lib/kthxbye/failure.rb
|
40
|
+
- lib/kthxbye/helper.rb
|
41
|
+
- lib/kthxbye/job.rb
|
42
|
+
- lib/kthxbye/railtie.rb
|
43
|
+
- lib/kthxbye/railties/kthxbye.rake
|
44
|
+
- lib/kthxbye/stats.rb
|
45
|
+
- lib/kthxbye/version.rb
|
38
46
|
- lib/kthxbye/web_interface/public/application.js
|
39
47
|
- lib/kthxbye/web_interface/public/awesome-buttons.css
|
48
|
+
- lib/kthxbye/web_interface/public/jquery.js
|
40
49
|
- lib/kthxbye/web_interface/public/style.css
|
41
|
-
- lib/kthxbye/web_interface/views/
|
50
|
+
- lib/kthxbye/web_interface/views/error.haml
|
51
|
+
- lib/kthxbye/web_interface/views/failed.haml
|
42
52
|
- lib/kthxbye/web_interface/views/hash.haml
|
53
|
+
- lib/kthxbye/web_interface/views/layout.haml
|
43
54
|
- lib/kthxbye/web_interface/views/overview.haml
|
44
|
-
- lib/kthxbye/web_interface/views/failed.haml
|
45
|
-
- lib/kthxbye/web_interface/views/set.haml
|
46
55
|
- lib/kthxbye/web_interface/views/queue.haml
|
47
|
-
- lib/kthxbye/web_interface/views/view_backtrace.haml
|
48
|
-
- lib/kthxbye/web_interface/views/layout.haml
|
49
56
|
- lib/kthxbye/web_interface/views/queues.haml
|
50
|
-
- lib/kthxbye/web_interface/views/
|
57
|
+
- lib/kthxbye/web_interface/views/set.haml
|
51
58
|
- lib/kthxbye/web_interface/views/stats.haml
|
59
|
+
- lib/kthxbye/web_interface/views/view_backtrace.haml
|
60
|
+
- lib/kthxbye/web_interface/views/workers.haml
|
52
61
|
- lib/kthxbye/web_interface/views/working.haml
|
53
|
-
- lib/kthxbye/
|
54
|
-
- lib/kthxbye/config.rb
|
55
|
-
- lib/kthxbye/railtie.rb
|
56
|
-
- lib/kthxbye/stats.rb
|
57
|
-
- lib/kthxbye/helper.rb
|
62
|
+
- lib/kthxbye/web_interface.rb
|
58
63
|
- lib/kthxbye/worker.rb
|
59
|
-
- lib/kthxbye/railties/kthxbye.rake
|
60
|
-
- lib/generators/kthxbye/templates/kthxbye.css
|
61
|
-
- lib/generators/kthxbye/templates/kthxbye_widget.png
|
62
|
-
- lib/generators/kthxbye/templates/kthxbye.js
|
63
|
-
- lib/generators/kthxbye/USAGE
|
64
|
-
- lib/generators/kthxbye/kthxbye_generator.rb
|
65
64
|
- lib/kthxbye.rb
|
66
65
|
- LICENSE
|
67
|
-
- README.
|
66
|
+
- README.md
|
68
67
|
- DESIGN.textile
|
69
68
|
- config.ru
|
70
69
|
- Gemfile
|
71
|
-
- test/
|
70
|
+
- test/helper.rb
|
72
71
|
- test/redis-test.conf
|
73
|
-
- test/test_helper.rb
|
74
72
|
- test/test_failure.rb
|
73
|
+
- test/test_helper.rb
|
74
|
+
- test/test_kthxbye.rb
|
75
75
|
- test/test_worker.rb
|
76
|
-
- test/helper.rb
|
77
76
|
has_rdoc: true
|
78
77
|
homepage: http://github.com/plukevdh/kthxbye
|
79
78
|
licenses: []
|
@@ -88,7 +87,6 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
88
87
|
requirements:
|
89
88
|
- - ">="
|
90
89
|
- !ruby/object:Gem::Version
|
91
|
-
hash: 3
|
92
90
|
segments:
|
93
91
|
- 0
|
94
92
|
version: "0"
|
@@ -97,7 +95,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
97
95
|
requirements:
|
98
96
|
- - ">="
|
99
97
|
- !ruby/object:Gem::Version
|
100
|
-
hash: 23
|
101
98
|
segments:
|
102
99
|
- 1
|
103
100
|
- 3
|
@@ -111,9 +108,9 @@ signing_key:
|
|
111
108
|
specification_version: 3
|
112
109
|
summary: Async processing + results notification
|
113
110
|
test_files:
|
114
|
-
- test/
|
111
|
+
- test/helper.rb
|
115
112
|
- test/redis-test.conf
|
116
|
-
- test/test_helper.rb
|
117
113
|
- test/test_failure.rb
|
114
|
+
- test/test_helper.rb
|
115
|
+
- test/test_kthxbye.rb
|
118
116
|
- test/test_worker.rb
|
119
|
-
- test/helper.rb
|