q4m 0.0.6 → 0.0.7
Sign up to get free protection for your applications and to get access to all the features.
- data/README.rdoc +248 -26
- data/VERSION +1 -1
- data/lib/q4m.rb +11 -1
- metadata +4 -4
data/README.rdoc
CHANGED
@@ -2,32 +2,83 @@
|
|
2
2
|
|
3
3
|
Interface to pragmatically setup a queue job for Q4M!
|
4
4
|
|
5
|
+
= So... What's Q4M
|
6
|
+
|
7
|
+
Q4M is a open-sourced MySQL Engine. that attempts to take away some
|
8
|
+
logic from your application by encapsulating it inside of mysql.
|
9
|
+
|
10
|
+
The documentation is kinda bad, but the magic that it does in the
|
11
|
+
background is sweeeeeet!
|
12
|
+
|
13
|
+
You basically store all of your background jobs inside a MySQL table.
|
14
|
+
that table's engine should be a *queue* (q4m) engine!
|
15
|
+
|
16
|
+
Exampe:
|
17
|
+
|
18
|
+
drop table if exists shell_jobs, ruby_jobs;
|
19
|
+
create table shell_jobs (command text) engine=queue;
|
20
|
+
create table ruby_jobs (command text) engine=queue;
|
21
|
+
|
22
|
+
That engine will work under some kind of transaction. When you connect
|
23
|
+
to the database the engine will give you a list of all the pending
|
24
|
+
jobs. When you start a transaction you'll receive *ONE* job to work
|
25
|
+
with. In the mean time the whole table gets locked..
|
26
|
+
And if you do a 'select *' all results will return emtpy!
|
27
|
+
|
28
|
+
Don't panic!
|
29
|
+
|
30
|
+
This is great, you can never access two jobs at the same time.
|
31
|
+
if another worker/machine connects they'll get a list of jobs except
|
32
|
+
the one is wrapped in the transaction!
|
33
|
+
|
34
|
+
So that means that the locking only happens on the connection that is
|
35
|
+
perfoming the transaction but any other workers will get the full list
|
36
|
+
of jobs except the ones that are being processed by other workers.
|
37
|
+
|
38
|
+
When the first transaction gets terminated the engine will
|
39
|
+
automatically unlock the table and delete the record.
|
40
|
+
|
41
|
+
This is great, because you can have any number of workers working
|
42
|
+
concurrently without making a mess!
|
43
|
+
|
44
|
+
Meaning that a job can never be called in twice by anyone, and that
|
45
|
+
you can grow horizontally, the limit is the sky!
|
46
|
+
|
47
|
+
What's next?
|
48
|
+
|
49
|
+
The job is done here!
|
50
|
+
|
51
|
+
Go do something more interesting like clustering! ;)
|
52
|
+
|
53
|
+
Seek & destroy, Seek & destroy!
|
54
|
+
|
55
|
+
= So what exactly does this gem do?
|
56
|
+
|
57
|
+
Gives you an interface to quickly setup how specificly a job is
|
58
|
+
supposed to run!
|
59
|
+
|
60
|
+
Basically, you define a class create an method called execute, which
|
61
|
+
receives a *job* argument. truly a MySQL record in the form of an
|
62
|
+
array, what you do from there is up to you!
|
63
|
+
|
64
|
+
if something goes wrong just call the *queue_abort* method and
|
65
|
+
everything gets rolled back!
|
66
|
+
|
5
67
|
= Installation
|
6
68
|
|
7
69
|
kazu@utopia:~$ gem install q4m
|
8
70
|
|
9
|
-
=
|
71
|
+
= Simple example
|
10
72
|
|
11
73
|
require 'rubygems'
|
12
74
|
require 'q4m'
|
13
75
|
|
14
|
-
|
76
|
+
# Specify how shell jobs will run
|
15
77
|
|
16
|
-
|
17
|
-
def execute job
|
18
|
-
# let the log know if the job was executed correctly!
|
19
|
-
job = job.first
|
20
|
-
return @log.info("Executed: #{job.inspect}") if system job
|
21
|
-
|
22
|
-
# If failed then log the job that failed and rollback the job so
|
23
|
-
# it stays in the queue!
|
24
|
-
@log.error "Aborted job: #{job.inspect}"
|
25
|
-
queue_abort
|
26
|
-
end
|
78
|
+
class ShellJob < ::Q4m::Queue::Job
|
27
79
|
|
28
|
-
|
29
|
-
|
30
|
-
'my_queue'
|
80
|
+
def execute job
|
81
|
+
queue_abort unless system(job.first)
|
31
82
|
end
|
32
83
|
|
33
84
|
end
|
@@ -37,23 +88,194 @@ Interface to pragmatically setup a queue job for Q4M!
|
|
37
88
|
pass = 'secret'
|
38
89
|
database = 'my_db'
|
39
90
|
|
91
|
+
# Instantiate and run a job!
|
40
92
|
mysql = Mysql.new host, user, pass, database
|
41
93
|
job = ShellJob.new mysql, Logger.new('q4m.log')
|
42
94
|
|
43
95
|
job.run
|
44
96
|
|
97
|
+
= Quick tip
|
98
|
+
|
99
|
+
Note that: The *ShellJob* class will try to connect to the
|
100
|
+
'shell_jobs' q4m table by default, if you want to use a different table
|
101
|
+
override the *table_name* method and specify one of your liking!
|
102
|
+
|
103
|
+
class ShellJob < ::Q4m::Queue::Job
|
104
|
+
|
105
|
+
def execute job
|
106
|
+
queue_abort unless system(job.first)
|
107
|
+
end
|
108
|
+
|
109
|
+
def table_name
|
110
|
+
'my_cool_table'
|
111
|
+
end
|
112
|
+
|
113
|
+
end
|
114
|
+
|
115
|
+
= Run on a specific environment like rails?
|
116
|
+
|
117
|
+
Would you like to have an entire rails-app as your worker?
|
118
|
+
but dont't want to load the whole env with rake tasks everytime?
|
119
|
+
|
120
|
+
That would be sweet right? and you can even reuse/test that code.
|
121
|
+
|
122
|
+
No problem!
|
123
|
+
|
124
|
+
You can leave only-one instance of rails running
|
125
|
+
and tell it to pull jobs from the queue whenever it can?
|
126
|
+
|
127
|
+
Yes!!
|
128
|
+
|
129
|
+
But let's keep it flexibe so let's run two kinds of jobs, :ruby & :shell
|
130
|
+
|
131
|
+
module Background
|
132
|
+
|
133
|
+
# Specify a shell job. isn't this short & sweet?
|
134
|
+
class ShellJob < ::Q4m::Queue::Job
|
135
|
+
|
136
|
+
def execute job
|
137
|
+
queue_abort unless system(job.first)
|
138
|
+
end
|
139
|
+
|
140
|
+
end
|
141
|
+
|
142
|
+
# Now do the same thing for ruby
|
143
|
+
# and let's encapsulate the whole thing, and say that all ruby
|
144
|
+
# code that goes into the queue will be methods and they must
|
145
|
+
# always return booleans, if true complete the job successfully!
|
146
|
+
# rollback the transaction otherwise!
|
147
|
+
class RubyJob < ::Q4m::Queue::Job
|
148
|
+
|
149
|
+
def execute job
|
150
|
+
queue_abort unless (eval(job.first) == true)
|
151
|
+
end
|
152
|
+
|
153
|
+
end
|
154
|
+
|
155
|
+
# This hack will mimic a very simple deamon that will run in the
|
156
|
+
# background stay alert for any new jobs that get appended to
|
157
|
+
# the queue
|
158
|
+
class Deamon
|
159
|
+
|
160
|
+
def self.run type = :ruby
|
161
|
+
log = Rails.root + 'log' + 'q4m.log'
|
162
|
+
path = Rails.root + 'config' + 'database.yml'
|
163
|
+
conf = YAML::load_file(path)['queues']
|
164
|
+
mysql = Mysql.new conf['host'], conf['username'], conf['password'], conf['database']
|
165
|
+
job = if type == :ruby
|
166
|
+
Background::RubyJob.new mysql, Logger.new(log)
|
167
|
+
else
|
168
|
+
Background::ShellJob.new mysql, Logger.new(log)
|
169
|
+
end
|
170
|
+
if File.exist? '/tmp/run_queue.txt'
|
171
|
+
job.run
|
172
|
+
sleep 1
|
173
|
+
print '.'
|
174
|
+
mysql.close
|
175
|
+
run
|
176
|
+
end
|
177
|
+
end
|
178
|
+
|
179
|
+
end
|
180
|
+
|
181
|
+
end
|
182
|
+
|
183
|
+
# Now let's create a model that will help us create jobs for ruby
|
184
|
+
class RubyJob < ActiveRecord::Base
|
185
|
+
|
186
|
+
establish_connection :queues
|
187
|
+
|
188
|
+
def self.append command
|
189
|
+
create! :command => command
|
190
|
+
end
|
191
|
+
|
192
|
+
end
|
193
|
+
|
194
|
+
# And for shell
|
195
|
+
# this can be encapsulated into a parent class but whatever!
|
196
|
+
class ShellJob < ActiveRecord::Base
|
197
|
+
|
198
|
+
establish_connection :queues
|
199
|
+
|
200
|
+
def self.append command
|
201
|
+
create! :command => command
|
202
|
+
end
|
203
|
+
|
204
|
+
end
|
205
|
+
|
206
|
+
# take it one step farther and build a helper, after all we are
|
207
|
+
# going to be working with strings!
|
208
|
+
class String
|
209
|
+
|
210
|
+
def jobify type = :ruby
|
211
|
+
return RubyJob.append self if type == :ruby
|
212
|
+
return ShellJob.append self if type == :shell
|
213
|
+
nil
|
214
|
+
end
|
215
|
+
|
216
|
+
end
|
217
|
+
|
218
|
+
= How to actually run it (on rails)!
|
219
|
+
|
220
|
+
# With the code above you can do something like this!
|
221
|
+
|
222
|
+
$ touch /tmp/run_queue.txt
|
223
|
+
|
224
|
+
# Run the deamon in a rails console!
|
225
|
+
>> Background::Deamon.run
|
226
|
+
|
227
|
+
# Let's create some background jobs on another console..
|
228
|
+
|
229
|
+
# Create a shell job, send it to the queue for later processing!
|
230
|
+
'RAILS_ENV=production rake -T'.jobify :shell
|
231
|
+
|
232
|
+
# Do the same but with ruby!
|
233
|
+
# NOTE: method inside should return boolean!!
|
234
|
+
'User.create_thumbnails(1)'.jobify
|
235
|
+
|
236
|
+
# Things should start running on the background, look at your logs!
|
237
|
+
|
238
|
+
# To stop the deamon cleanly just delete the file '/tmp/run_queue.txt'
|
239
|
+
|
240
|
+
= This is great!
|
241
|
+
|
242
|
+
Basically any machine running rails with this code, can pull jobs and
|
243
|
+
do some hard work for you! even if they are not part of your cluster!
|
244
|
+
|
245
|
+
The only requirement is that they can access the mysql-database.
|
246
|
+
|
247
|
+
and no, you don't need to install that engine locally!
|
248
|
+
|
249
|
+
= Extensibility!
|
250
|
+
|
251
|
+
Go farther just create a new job class and run whatever you like!
|
252
|
+
|
253
|
+
= Q4M lib?
|
254
|
+
|
255
|
+
This is how I installed it.
|
256
|
+
|
257
|
+
wget http://q4m.kazuhooku.com/dist/old/mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar.gz
|
258
|
+
unzip *.gz
|
259
|
+
tar -xvf mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar
|
260
|
+
cd q4m-0.8.9-linux-x86_64
|
261
|
+
sudo cp libqueue_engine.so /usr/lib/mysql/plugin
|
262
|
+
sudo chmod 777 /usr/lib/mysql/plugin/libqueue_engine.so
|
263
|
+
mysql -u root -p -f mysql < support-files/install.sql
|
264
|
+
|
265
|
+
You might wanna go to:
|
266
|
+
|
267
|
+
http://q4m.github.com/
|
45
268
|
|
269
|
+
And see tha latest on how to get it installed.
|
46
270
|
|
47
|
-
|
271
|
+
just thought of posting it, so you'll get an idea.
|
48
272
|
|
49
|
-
|
50
|
-
* Make your feature addition or bug fix.
|
51
|
-
* Add tests for it. This is important so I don't break it in a
|
52
|
-
future version unintentionally.
|
53
|
-
* Commit, do not mess with rakefile, version, or history.
|
54
|
-
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
|
55
|
-
* Send me a pull request. Bonus points for topic branches.
|
273
|
+
Yeah, but what's your distro?
|
56
274
|
|
57
|
-
|
275
|
+
it's right here!
|
58
276
|
|
59
|
-
|
277
|
+
deploy@staging> cat /etc/*-release
|
278
|
+
DISTRIB_ID=Ubuntu
|
279
|
+
DISTRIB_RELEASE=10.04
|
280
|
+
DISTRIB_CODENAME=lucid
|
281
|
+
DISTRIB_DESCRIPTION="Ubuntu 10.04 LTS"
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.7
|
data/lib/q4m.rb
CHANGED
@@ -24,12 +24,22 @@ module Q4m
|
|
24
24
|
# - Executes the latest job.
|
25
25
|
# - Restores the queue so other worker can execute a process.
|
26
26
|
#
|
27
|
-
def run
|
27
|
+
def run terminate = false
|
28
28
|
return unless @has_jobs
|
29
29
|
queue_wait
|
30
30
|
@log.info("Executing: #{latest_job.inspect}")
|
31
31
|
execute latest_job
|
32
32
|
queue_end
|
33
|
+
shutdown if terminate
|
34
|
+
nil
|
35
|
+
end
|
36
|
+
|
37
|
+
# Shuts down the worker
|
38
|
+
#
|
39
|
+
def shutdown
|
40
|
+
@mysql.close
|
41
|
+
@log.close
|
42
|
+
nil
|
33
43
|
end
|
34
44
|
|
35
45
|
private
|
metadata
CHANGED
@@ -1,13 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: q4m
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
hash:
|
4
|
+
hash: 17
|
5
5
|
prerelease: false
|
6
6
|
segments:
|
7
7
|
- 0
|
8
8
|
- 0
|
9
|
-
-
|
10
|
-
version: 0.0.
|
9
|
+
- 7
|
10
|
+
version: 0.0.7
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- kazuyoshi tlacaelel
|
@@ -15,7 +15,7 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date: 2010-11-
|
18
|
+
date: 2010-11-12 00:00:00 -06:00
|
19
19
|
default_executable:
|
20
20
|
dependencies:
|
21
21
|
- !ruby/object:Gem::Dependency
|