updater 0.9.1 → 0.9.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.markdown +188 -0
- data/Rakefile +6 -4
- data/VERSION +1 -1
- data/lib/updater/update.rb +1 -1
- metadata +48 -25
- data/README +0 -7
data/README.markdown
ADDED
@@ -0,0 +1,188 @@
|
|
1
|
+
Updater
|
2
|
+
=======
|
3
|
+
|
4
|
+
Updater is a background job queue processor.
|
5
|
+
It works a bit like delayed_job or rescue,
|
6
|
+
processing jobs in the background to allow user facing processes to stay responsive.
|
7
|
+
It also allow jobs to be scheduled and run at a particular time.
|
8
|
+
It is intended to work with a number of different ORM layers,
|
9
|
+
but at the moment only DataMapper is implemented.
|
10
|
+
Native support for MongoDB and ActiveRecord are planed.
|
11
|
+
Get it on GemCutter with `gem install updater`.
|
12
|
+
|
13
|
+
Main Features
|
14
|
+
-------------
|
15
|
+
|
16
|
+
* Handles both immediate and time delayed jobs
|
17
|
+
* Intelligent Automatic Scaling
|
18
|
+
* Does not poll the database
|
19
|
+
* Uses minimal resources when not under load
|
20
|
+
* Flexible Job Chaining allows for intelligent error handling and rescheduling
|
21
|
+
* Powerful configuration with intelligent defaults
|
22
|
+
* Comes with `rake` tasks
|
23
|
+
|
24
|
+
These feature are as of ver 0.9.1. See the change log for addational features
|
25
|
+
|
26
|
+
Use Cases
|
27
|
+
---------
|
28
|
+
|
29
|
+
Web based applications face two restrictions on their functionality
|
30
|
+
as it pertains to data processing tasks.
|
31
|
+
First, the application will only run code in response to a user request.
|
32
|
+
Updater handles the case of actions or events that need to be triggered
|
33
|
+
without a request coming in to the web application.
|
34
|
+
Second, web applications, particularly those under heavy load,
|
35
|
+
need to handle as many request as possible in a given time frame.
|
36
|
+
Updater allows data processing and communication tasks
|
37
|
+
to happen outside of the request response cycle
|
38
|
+
and makes it possible to move these tasks onto dedicated hardware
|
39
|
+
|
40
|
+
Updater is also useful for circumstances where code needs to be run
|
41
|
+
in a time dependent fashion,
|
42
|
+
particularly when these events need to be closely controlled by the code.
|
43
|
+
|
44
|
+
Updater is not a replacement for `cron`.
|
45
|
+
Jobs that are regular and repeating can be run
|
46
|
+
more consistently and with fewer resources with `cron`.
|
47
|
+
Updater should be considered when the application generates
|
48
|
+
a large number of one time events,
|
49
|
+
and/or the events need to be regularly manipulated bu the application.
|
50
|
+
|
51
|
+
Updater is also not the optimal solution if the only goal
|
52
|
+
is to offload large numbers of immediate tasks.
|
53
|
+
For this the author recommends
|
54
|
+
[resque](http://github.com/defunkt/resque)
|
55
|
+
by Chris Wanstrath.
|
56
|
+
|
57
|
+
Resque lacks a number of Updater's more powerful features,
|
58
|
+
and as of this writing we are not aware of any ability in resque
|
59
|
+
to set the time the job is run.
|
60
|
+
But rescue does offer much higher potential throughput, and
|
61
|
+
a more robust queue structure backed by the Redis key-value store.
|
62
|
+
|
63
|
+
Using Updater
|
64
|
+
=============
|
65
|
+
|
66
|
+
Initial Installation
|
67
|
+
--------------------
|
68
|
+
|
69
|
+
Updater comes packaged as a gem and is published on GemCutter.
|
70
|
+
Get the latest version with `gem install updater`
|
71
|
+
|
72
|
+
Consepts
|
73
|
+
--------
|
74
|
+
|
75
|
+
Updater is not complex to use but it does, of necessity, have a number of *moving parts*.
|
76
|
+
In order to use updater successfully two tasks must be accomplished:
|
77
|
+
your application (referred to as the client) must be able to add jobs to the queue,
|
78
|
+
and a separate process (the server) must be setup and run
|
79
|
+
which will preform the actions specified in those jobs.
|
80
|
+
|
81
|
+
Jobs are stored in a data store that is shared between client and server.
|
82
|
+
Usually this data store will be a table in your database.
|
83
|
+
Other data stores are possible, but require significantly configuration.
|
84
|
+
Updater is designed to have a minimal impact on the load of the data store,
|
85
|
+
and it therefore should be a reasonable solution for most applications.
|
86
|
+
(For a discussion of when this is not a reasonable solution see
|
87
|
+
[the Rescue Blog Post](http://github.com/blog/542-introducing-resque))
|
88
|
+
|
89
|
+
Updater is very flexible about what can be run in as a background job.
|
90
|
+
The Job will call a single method on either a Class or an instance.
|
91
|
+
The method must be public.
|
92
|
+
Any arguments can be passed to the method so long as the can be marshaled.
|
93
|
+
It is important to keep in mind that the job will be run in a completely separate process.
|
94
|
+
Any global state will have to be recreated,
|
95
|
+
and data must be persisted in some form in order to be seen by the client.
|
96
|
+
For web applications this is usually not an issue.
|
97
|
+
|
98
|
+
Calling a class method is fairly strait forward,
|
99
|
+
but calling a method on in instance take a little more work.
|
100
|
+
Instances must be somehow persisted in the client
|
101
|
+
then reinstantiated on the worker process.
|
102
|
+
The assumption is that this will be don through the ORM and data store.
|
103
|
+
Each ORM adapter in Updater lists a defaults methods
|
104
|
+
for retrieving particular instances from the data store.
|
105
|
+
When an instance is scheduled as the target of a job,
|
106
|
+
its class and id will be stored in the `updates` table.
|
107
|
+
When the job is run it will first use the it to pull the instance out of the data store,
|
108
|
+
then call the appropriate method on that instance.
|
109
|
+
|
110
|
+
Client Setup
|
111
|
+
------------
|
112
|
+
|
113
|
+
The client application needs to do two things to be able to effectively schedule jobs:
|
114
|
+
1) require the `updater` library, and
|
115
|
+
2) call the `Updater::Setup.client_setup` method during initialization.
|
116
|
+
The `client_setup` method is responsible for establishing interprocess communication with the server,
|
117
|
+
and selecting the correct ORM adapter for Updater.
|
118
|
+
It does this using the configuration file discussed later in this document.
|
119
|
+
This method can take an optional hash that will override the options in the configuration file.
|
120
|
+
It can also be passes a `:logger` option which will use the passed in logger instance for updater logging.
|
121
|
+
|
122
|
+
Scheduling Jobs
|
123
|
+
---------------
|
124
|
+
|
125
|
+
The core of Updater is placing Jobs on the queue to be run.
|
126
|
+
Jobs are scheduled using methods on the `Updater::Update` class.
|
127
|
+
Please see the rdoc entry for `Updater::Update#at` for more details
|
128
|
+
The following methods can be used to schedule jobs:
|
129
|
+
|
130
|
+
* `at(time, target, method, args, options)`: Schedules a job to run at a given time
|
131
|
+
* `in(delay, target, method, args, options)`: Schedules a job to run after a given interval
|
132
|
+
* `immidiate(target, method, args, options)`: Schedules a job to run immidiatly.
|
133
|
+
|
134
|
+
**target**: is the object (class or instance) that the method will be called on.
|
135
|
+
|
136
|
+
**method**: the method name as a string or symbol.
|
137
|
+
If this is unspesified `:preform` is asumed (a la Resque)
|
138
|
+
|
139
|
+
**args**: an array of objects that will be passed as arguments to the method.
|
140
|
+
Either leave this blank, or set to `[]` to call without arguments.
|
141
|
+
All members of the array must be marshalable.
|
142
|
+
|
143
|
+
**options**: a hash of extra information, details can be found in the Options section.
|
144
|
+
|
145
|
+
We intend to add a module that can be included into a target class
|
146
|
+
that will allow scheduling in the same general manner as delayed_job.
|
147
|
+
This addation is planned for version 1.2.
|
148
|
+
|
149
|
+
The Configuration File
|
150
|
+
----------------------
|
151
|
+
|
152
|
+
In updater both client and server use a single configuration file.
|
153
|
+
The location of this file can be set using the `UPDATE_CONFIG` environment variable.
|
154
|
+
When this is not set Updater will instead look for some intelligently chosen defaults.
|
155
|
+
These defaults are based on the assumption
|
156
|
+
that the client is one of a number of popular web frameworks
|
157
|
+
that use the rails directory structure.
|
158
|
+
It will look in either the current working directory or a subdirectory called `config`
|
159
|
+
(with preference for the latter) for a file called `updater.config`.
|
160
|
+
Failing that it will look for a .updater file in the current working directory.
|
161
|
+
Rake files should endeavor to set an appropriate working directory
|
162
|
+
before invoking the setup tasks.
|
163
|
+
|
164
|
+
The configuration itself is a ERb interpreted YAML file.
|
165
|
+
This is of use in limiting repetition,
|
166
|
+
and in changing options based on the environment (test/development/production)
|
167
|
+
|
168
|
+
Please see the options section for details about the various options in this file.
|
169
|
+
|
170
|
+
Starting Workers (Server)
|
171
|
+
-------------------------
|
172
|
+
|
173
|
+
In the parlance of background job processing,
|
174
|
+
a process that executes jobs is known as a worker.
|
175
|
+
The recommended way to start workers is through a rake task.
|
176
|
+
First, include `updater/tasks` in your application's Rakefile.
|
177
|
+
This will add start, stop and monitor tasks into the `updater` namespace.
|
178
|
+
`start` will use the options in your configuration file to start a worker process.
|
179
|
+
Likewise, `stop` will shut that process down.
|
180
|
+
The monitor task will start an http server
|
181
|
+
that you can use to monitor and control the job queue and workers.
|
182
|
+
(This feature is not currently implemented)
|
183
|
+
|
184
|
+
Individual workers are initialized and shutdown by a master process
|
185
|
+
which monitors the work load and starts or stops individual workers as needed
|
186
|
+
within the limits established in the configuration file.
|
187
|
+
You should, therefore, only need to use `start` once.
|
188
|
+
|
data/Rakefile
CHANGED
@@ -1,7 +1,9 @@
|
|
1
1
|
require 'rubygems'
|
2
2
|
require 'rake/gempackagetask'
|
3
|
+
gem 'rspec', '=1.3.0'
|
3
4
|
require 'spec/rake/spectask'
|
4
5
|
|
6
|
+
|
5
7
|
VERSION_FILE = File.join(File.dirname(__FILE__), 'VERSION')
|
6
8
|
|
7
9
|
GEM_NAME = "updater"
|
@@ -17,19 +19,19 @@ spec = Gem::Specification.new do |s|
|
|
17
19
|
s.date = File.ctime(VERSION_FILE)
|
18
20
|
s.platform = Gem::Platform::RUBY
|
19
21
|
s.has_rdoc = true
|
20
|
-
s.extra_rdoc_files = ["README", "LICENSE", "VERSION"]
|
22
|
+
s.extra_rdoc_files = ["README.markdown", "LICENSE", "VERSION"]
|
21
23
|
s.summary = SUMMARY
|
22
24
|
s.description = s.summary
|
23
25
|
s.author = AUTHOR
|
24
26
|
s.email = EMAIL
|
25
27
|
s.homepage = HOMEPAGE
|
26
|
-
s.
|
27
|
-
s.add_development_dependency('rspec', '
|
28
|
+
s.add_development_dependency('datamapper', '>= 0.10.2')
|
29
|
+
s.add_development_dependency('rspec', '=1.3.0')
|
28
30
|
s.add_development_dependency('timecop', '>= 0.2.1')
|
29
31
|
s.add_development_dependency('chronic', '>= 0.2.3')
|
30
32
|
s.require_path = 'lib'
|
31
33
|
s.bindir = 'bin'
|
32
|
-
s.files = %w(LICENSE README Rakefile VERSION) + Dir.glob("{lib,spec,bin}/**/*")
|
34
|
+
s.files = %w(LICENSE README.markdown Rakefile VERSION) + Dir.glob("{lib,spec,bin}/**/*")
|
33
35
|
|
34
36
|
end
|
35
37
|
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.9.
|
1
|
+
0.9.2
|
data/lib/updater/update.rb
CHANGED
@@ -202,7 +202,7 @@ module Updater
|
|
202
202
|
hash[:finder] = options[:finder] || hash[:finder]
|
203
203
|
hash[:finder_args] = options[:finder_args] || hash[:finder_args]
|
204
204
|
|
205
|
-
hash[:method] = method || :
|
205
|
+
hash[:method] = method || :perform
|
206
206
|
hash[:method_args] = args
|
207
207
|
|
208
208
|
[:name,:failure,:success,:ensure].each do |opt|
|
metadata
CHANGED
@@ -1,7 +1,12 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: updater
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
|
4
|
+
prerelease: false
|
5
|
+
segments:
|
6
|
+
- 0
|
7
|
+
- 9
|
8
|
+
- 2
|
9
|
+
version: 0.9.2
|
5
10
|
platform: ruby
|
6
11
|
authors:
|
7
12
|
- John F. Miller
|
@@ -9,49 +14,65 @@ autorequire:
|
|
9
14
|
bindir: bin
|
10
15
|
cert_chain: []
|
11
16
|
|
12
|
-
date: 2010-
|
17
|
+
date: 2010-05-07 00:00:00 -07:00
|
13
18
|
default_executable:
|
14
19
|
dependencies:
|
15
20
|
- !ruby/object:Gem::Dependency
|
16
21
|
name: datamapper
|
17
|
-
|
18
|
-
|
19
|
-
version_requirements: !ruby/object:Gem::Requirement
|
22
|
+
prerelease: false
|
23
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
20
24
|
requirements:
|
21
25
|
- - ">="
|
22
26
|
- !ruby/object:Gem::Version
|
27
|
+
segments:
|
28
|
+
- 0
|
29
|
+
- 10
|
30
|
+
- 2
|
23
31
|
version: 0.10.2
|
24
|
-
|
32
|
+
type: :development
|
33
|
+
version_requirements: *id001
|
25
34
|
- !ruby/object:Gem::Dependency
|
26
35
|
name: rspec
|
27
|
-
|
28
|
-
|
29
|
-
version_requirements: !ruby/object:Gem::Requirement
|
36
|
+
prerelease: false
|
37
|
+
requirement: &id002 !ruby/object:Gem::Requirement
|
30
38
|
requirements:
|
31
|
-
- - "
|
39
|
+
- - "="
|
32
40
|
- !ruby/object:Gem::Version
|
33
|
-
|
34
|
-
|
41
|
+
segments:
|
42
|
+
- 1
|
43
|
+
- 3
|
44
|
+
- 0
|
45
|
+
version: 1.3.0
|
46
|
+
type: :development
|
47
|
+
version_requirements: *id002
|
35
48
|
- !ruby/object:Gem::Dependency
|
36
49
|
name: timecop
|
37
|
-
|
38
|
-
|
39
|
-
version_requirements: !ruby/object:Gem::Requirement
|
50
|
+
prerelease: false
|
51
|
+
requirement: &id003 !ruby/object:Gem::Requirement
|
40
52
|
requirements:
|
41
53
|
- - ">="
|
42
54
|
- !ruby/object:Gem::Version
|
55
|
+
segments:
|
56
|
+
- 0
|
57
|
+
- 2
|
58
|
+
- 1
|
43
59
|
version: 0.2.1
|
44
|
-
|
60
|
+
type: :development
|
61
|
+
version_requirements: *id003
|
45
62
|
- !ruby/object:Gem::Dependency
|
46
63
|
name: chronic
|
47
|
-
|
48
|
-
|
49
|
-
version_requirements: !ruby/object:Gem::Requirement
|
64
|
+
prerelease: false
|
65
|
+
requirement: &id004 !ruby/object:Gem::Requirement
|
50
66
|
requirements:
|
51
67
|
- - ">="
|
52
68
|
- !ruby/object:Gem::Version
|
69
|
+
segments:
|
70
|
+
- 0
|
71
|
+
- 2
|
72
|
+
- 3
|
53
73
|
version: 0.2.3
|
54
|
-
|
74
|
+
type: :development
|
75
|
+
version_requirements: *id004
|
55
76
|
description: A Gem for queuing methods for later calling which is ORM Agnostic, and has advanced Error Handling
|
56
77
|
email: emperor@antarestrader.com
|
57
78
|
executables: []
|
@@ -59,12 +80,12 @@ executables: []
|
|
59
80
|
extensions: []
|
60
81
|
|
61
82
|
extra_rdoc_files:
|
62
|
-
- README
|
83
|
+
- README.markdown
|
63
84
|
- LICENSE
|
64
85
|
- VERSION
|
65
86
|
files:
|
66
87
|
- LICENSE
|
67
|
-
- README
|
88
|
+
- README.markdown
|
68
89
|
- Rakefile
|
69
90
|
- VERSION
|
70
91
|
- lib/updater.rb
|
@@ -106,18 +127,20 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
106
127
|
requirements:
|
107
128
|
- - ">="
|
108
129
|
- !ruby/object:Gem::Version
|
130
|
+
segments:
|
131
|
+
- 0
|
109
132
|
version: "0"
|
110
|
-
version:
|
111
133
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
112
134
|
requirements:
|
113
135
|
- - ">="
|
114
136
|
- !ruby/object:Gem::Version
|
137
|
+
segments:
|
138
|
+
- 0
|
115
139
|
version: "0"
|
116
|
-
version:
|
117
140
|
requirements: []
|
118
141
|
|
119
142
|
rubyforge_project:
|
120
|
-
rubygems_version: 1.3.
|
143
|
+
rubygems_version: 1.3.6
|
121
144
|
signing_key:
|
122
145
|
specification_version: 3
|
123
146
|
summary: A Gem for queuing methods for later calling which is ORM Agnostic, and has advanced Error Handling
|
data/README
DELETED
@@ -1,7 +0,0 @@
|
|
1
|
-
updater
|
2
|
-
=======
|
3
|
-
|
4
|
-
This plugin provides database driven delayed exicution of DataMapper model
|
5
|
-
classes. It can call class method on any class availible class or it can find
|
6
|
-
an instance of a class from a database and exicute any method on that instance.
|
7
|
-
Error handleing is also provided.
|