resque-sliders 0.1.2 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +65 -33
- data/bin/kewatcher +18 -8
- data/lib/resque-sliders/helpers.rb +4 -4
- data/lib/resque-sliders/kewatcher.rb +94 -54
- data/lib/resque-sliders/server.rb +3 -3
- data/lib/resque-sliders/server/public/css/jqueryui-1.8.16/blitzer/jquery-ui.custom.css +17 -17
- data/lib/resque-sliders/server/public/js/sliders.js +5 -3
- data/lib/resque-sliders/server/views/index.erb +5 -5
- data/lib/resque-sliders/version.rb +1 -1
- data/test/kewatcher_test.rb +44 -0
- data/test/redis-test.conf +1 -1
- data/test/test_helper.rb +1 -1
- metadata +10 -5
- data/Rakefile +0 -41
- data/test/resque-sliders_test.rb +0 -12
data/README.md
CHANGED
@@ -1,8 +1,9 @@
|
|
1
|
-
Resque Sliders
|
1
|
+
Resque Sliders [](http://travis-ci.org/kmullin/resque-sliders)
|
2
2
|
==============
|
3
3
|
|
4
4
|
[github.com/kmullin/resque-sliders](https://github.com/kmullin/resque-sliders)
|
5
5
|
|
6
|
+
|
6
7
|
Description
|
7
8
|
-----------
|
8
9
|
|
@@ -14,8 +15,7 @@ From the Resque-Web UI, you can:
|
|
14
15
|
* Start workers with any queue, or combination of queues on any host, and specify how many of each should be running
|
15
16
|
* Pause / Stop / Restart ALL running workers
|
16
17
|
|
17
|
-
|
18
|
-
ResqueSliders comes with two parts:
|
18
|
+
ResqueSliders comes in two parts:
|
19
19
|
|
20
20
|
* `KEWatcher`: A daemon that runs on any machine that needs to run Resque workers, watches over the workers and controls which ones are running
|
21
21
|
* `Resque-Web Plugin`: A bunch of slider bars, with text-input box to specify what queues to run on the workers
|
@@ -25,13 +25,11 @@ Installation
|
|
25
25
|
|
26
26
|
Install as a gem:
|
27
27
|
|
28
|
-
|
29
|
-
$ gem install resque-sliders
|
30
|
-
```
|
28
|
+
$ gem install resque-sliders
|
31
29
|
|
32
30
|
KEWatcher
|
33
31
|
---------
|
34
|
-
This is the daemon component that runs on any host that you want to run Resque workers on. The daemon's job is to manage how many Resque workers should be running, and what they should be running. It also provides an easy way to stop all workers during maintenance or deploys.
|
32
|
+
This is the daemon component that runs on any host that you want to run Resque workers on. The daemon's job is to manage **how many** Resque workers should be running, and **what** they should be running. It also provides an easy way to stop all workers during maintenance or deploys.
|
35
33
|
|
36
34
|
When the daemon first runs, it will register itself, by hostname with Redis:
|
37
35
|
|
@@ -48,37 +46,61 @@ Options:
|
|
48
46
|
-f, --force FORCE KILL ANY OTHER RUNNING KEWATCHERS
|
49
47
|
-v, --verbose Verbosity (Can be specified more than once, -vv)
|
50
48
|
-m, --max MAX Max Children (default: 10)
|
51
|
-
-
|
52
|
-
-
|
53
|
-
|
49
|
+
-w, --wait WAIT_TIME Time (in seconds) to wait for worker to die before sending TERM signal (default: 20 seconds)
|
50
|
+
-t, --time MAX_TIME Max Time (in seconds) to wait for worker to die before sending KILL (-9) signal (FORCE QUIT) (default: 60)
|
51
|
+
NOTE: With Resque >= 1.22.0 force quit is handled for you so by default this is the same as:
|
52
|
+
RESQUE_TERM_TIMEOUT=40 or the difference of MAX_TIME and WAIT_TIME
|
53
|
+
more info: http://hone.heroku.com/resque/2012/08/21/resque-signals.html
|
54
|
+
-a, --async Do NOT wait for Resque workers to die completely before spawning new workers (default: false)
|
55
|
+
-V, --version Prints Version
|
56
|
+
```
|
57
|
+
|
58
|
+
### Important Options
|
59
|
+
|
60
|
+
```
|
61
|
+
-m|--max MAX (Max Children): Maximum number of workers to run on host (default: 10)
|
62
|
+
-w|--wait WAIT_TIME (Wait Time): How long to wait before sending TERM to zombies (default: 20 seconds)
|
63
|
+
-t|--time TIME (Total Time): How long to wait before sending KILL to zombies (default: 60 seconds)
|
64
|
+
NOTE: Resque >= 1.22.0 includes signal handling of its own to force quit, so we use it if its there, and override with our own timeout here
|
65
|
+
-a|--async (Async): Should we spawn new workers before old ones have fully terminated (default: false)
|
66
|
+
-r|--rakefile RAKEFILE (Rakefile): Pass along a rakefile to use when calling rake ... resque:work - shouldn't be needed if run from project directory
|
67
|
+
-f|--force (Force): Force any currently running KEWatcher processes to quit, waiting for it to do so, and starting in its place
|
68
|
+
RAILS_ENV: If you're using rails, you need to set your RAILS_ENV variable
|
54
69
|
```
|
55
70
|
|
56
|
-
|
71
|
+
### Controlling the Daemon
|
72
|
+
|
73
|
+
Once the daemon is running on each host that is going to run Resque workers, you'll need to tell them which queues to run.
|
74
|
+
|
75
|
+
The queue configuration is done via Resque-Web interface
|
57
76
|
|
58
|
-
|
59
|
-
* `Total Time (-t|--time TIME)`: How long you want to wait before sending `TERM` to resque (like `kill -9`) (default: 2 minutes)
|
60
|
-
* `Wait Time (-w|--wait WAIT_TIME)`: How many seconds **MINIMUM** we spend in blocking wait() call per worker when cleaning up zombies (default: 25 seconds)
|
61
|
-
* `Rakefile (-r|--rakefile RAKEFILE)`: Pass along a rakefile to use when calling `rake ... resque:work` - shouldn't be needed if run from project directory
|
62
|
-
* `Force (-f|--force)`: Force any currently running KEWatcher processes to QUIT, waiting for it to do so, and starting in its place
|
63
|
-
* `RAILS_ENV`: If you're using rails, you need to set your RAILS_ENV variable
|
77
|
+
#### Resque-Web
|
64
78
|
|
65
|
-
|
79
|
+
See below for screenshots
|
66
80
|
|
67
|
-
|
81
|
+
Buttons:
|
68
82
|
|
69
|
-
* `
|
70
|
-
* `
|
71
|
-
* `
|
72
|
-
|
73
|
-
|
83
|
+
* `Play` / `Pause` - Start or Pause
|
84
|
+
* `Stop` - Stop all workers
|
85
|
+
* `Reload` - Sends HUP signal to running KEWatcher
|
86
|
+
|
87
|
+
#### Signals
|
88
|
+
|
89
|
+
KEWatcher supports all the [same signals as Resque](https://github.com/defunkt/resque#signals):
|
90
|
+
|
91
|
+
* `TERM` / `INT` / `QUIT` - Shutdown. Gracefully kill all child Resque workers, and wait for them to finish before exiting
|
92
|
+
* `HUP` - Restart all Resque workers by gracefully killing them, and starting new ones in their place
|
93
|
+
* `USR1` - Stop all Resque workers, and don't start any more
|
94
|
+
* `USR2` - Pause spawning of new queues, but leave current ones running
|
95
|
+
* `CONT` - Unpause. Continue spawning/managing child Resque workers
|
74
96
|
|
75
97
|
|
76
98
|
Resque-Web Integration
|
77
99
|
----------------------
|
78
|
-
**Main Screen:** showing 3 hosts
|
100
|
+
**Main Screen:** showing 3 hosts, and showing that one of the nodes is not running KEWatcher
|
79
101
|

|
80
102
|
|
81
|
-
**Host Screen:** showing
|
103
|
+
**Host Screen:** showing different `QUEUE` combinations (comma separated) and slider bars indicating how many of each of them should run
|
82
104
|

|
83
105
|
|
84
106
|
To enable the Resque-Web Integration you'll need to load ResqueSliders to enable the Sliders tab. Just add:
|
@@ -88,15 +110,25 @@ require 'resque-sliders'
|
|
88
110
|
```
|
89
111
|
to a file, like resque-web_init.rb, and run resque-web:
|
90
112
|
|
91
|
-
|
92
|
-
resque-web resque-web_init.rb
|
93
|
-
```
|
113
|
+
resque-web resque-web_init.rb
|
94
114
|
|
95
115
|
|
96
|
-
|
97
|
-
|
116
|
+
Works on
|
117
|
+
--------
|
98
118
|
|
99
119
|
`resque-sliders` has been tested on the following platforms:
|
100
120
|
|
101
|
-
|
102
|
-
|
121
|
+
#### Ruby
|
122
|
+
|
123
|
+
* 1.9.3
|
124
|
+
* 1.8.7 (ree)
|
125
|
+
* probabaly more...
|
126
|
+
|
127
|
+
Contributing
|
128
|
+
------------
|
129
|
+
|
130
|
+
Want to fix a bug? See a new feature?
|
131
|
+
|
132
|
+
1. [Fork](https://github.com/kmullin/resque-sliders/fork_select) me
|
133
|
+
2. Create a new branch
|
134
|
+
3. Open a [Pull Request](https://github.com/kmullin/resque-sliders/pull/new)
|
data/bin/kewatcher
CHANGED
@@ -6,6 +6,7 @@ require 'rubygems'
|
|
6
6
|
require 'yaml'
|
7
7
|
require 'optparse'
|
8
8
|
require 'resque-sliders/kewatcher'
|
9
|
+
require 'resque-sliders/version'
|
9
10
|
|
10
11
|
options = {
|
11
12
|
:verbosity => 0,
|
@@ -14,6 +15,7 @@ options = {
|
|
14
15
|
|
15
16
|
OptionParser.new do |opt|
|
16
17
|
opt.banner = "Usage: #{File.basename($0)} [options]"
|
18
|
+
opt.version = Resque::Plugins::ResqueSliders::Version
|
17
19
|
|
18
20
|
opt.separator ""
|
19
21
|
opt.separator "Options:"
|
@@ -39,20 +41,28 @@ OptionParser.new do |opt|
|
|
39
41
|
options[:verbosity] += 1
|
40
42
|
end
|
41
43
|
|
42
|
-
opt.on("-m", "--max MAX", "Max Children (default: 10)") do |max|
|
43
|
-
options[:max_children] = max
|
44
|
+
opt.on("-m", "--max MAX", Integer, "Max Children (default: 10)") do |max|
|
45
|
+
options[:max_children] = max
|
44
46
|
end
|
45
47
|
|
46
|
-
opt.on("-
|
47
|
-
options[:
|
48
|
+
opt.on("-w", "--wait WAIT_TIME", Float, "Time (in seconds) to wait for worker to die before sending TERM signal (default: 20 seconds)") do |wait|
|
49
|
+
options[:zombie_term_wait] = wait
|
48
50
|
end
|
49
51
|
|
50
|
-
opt.on("-
|
51
|
-
|
52
|
+
opt.on("-t", "--time MAX_TIME", Float,
|
53
|
+
"Max Time (in seconds) to wait for worker to die before sending KILL (-9) signal (FORCE QUIT) (default: 60)",
|
54
|
+
"NOTE: With Resque >= 1.22.0 force quit is handled for you so by default this is the same as:",
|
55
|
+
" RESQUE_TERM_TIMEOUT=40 or the difference of MAX_TIME and WAIT_TIME",
|
56
|
+
" more info: http://hone.heroku.com/resque/2012/08/21/resque-signals.html") do |wait|
|
57
|
+
options[:zombie_kill_wait] = wait
|
52
58
|
end
|
53
59
|
|
54
|
-
opt.on("-
|
55
|
-
|
60
|
+
opt.on("-a", "--async", "Do NOT wait for Resque workers to die completely before spawning new workers (default: false)") do
|
61
|
+
options[:async] = true
|
62
|
+
end
|
63
|
+
|
64
|
+
opt.on("-V", "--version", "Prints Version") do
|
65
|
+
puts opt.version
|
56
66
|
exit
|
57
67
|
end
|
58
68
|
|
@@ -20,11 +20,11 @@ module Resque
|
|
20
20
|
end
|
21
21
|
|
22
22
|
def redis_set_hash(key, field, fvalue)
|
23
|
-
Resque.redis.hset(key, field, fvalue) == 1
|
23
|
+
Resque.redis.hset(key, field, fvalue) == 1
|
24
24
|
end
|
25
25
|
|
26
26
|
def redis_del_hash(key, field)
|
27
|
-
Resque.redis.hdel(key, field) == 1
|
27
|
+
Resque.redis.hdel(key, field) == 1
|
28
28
|
end
|
29
29
|
|
30
30
|
# Return Hash: { queue => # }
|
@@ -48,13 +48,13 @@ module Resque
|
|
48
48
|
raise 'Dont call me that' unless %w(reload pause stop).include?(sig)
|
49
49
|
if @hostname
|
50
50
|
# if instance variable set from running daemon, make a freshy
|
51
|
-
redis_get_hash_field(host_config_key, "#{@hostname}:#{sig}").to_i == 1
|
51
|
+
redis_get_hash_field(host_config_key, "#{@hostname}:#{sig}").to_i == 1
|
52
52
|
else
|
53
53
|
# otherwise cache call in a Hash
|
54
54
|
@host_signal_map ||= {}
|
55
55
|
@host_signal_map[host] ||= {}
|
56
56
|
unless @host_signal_map[host].has_key?(sig)
|
57
|
-
@host_signal_map[host] = {sig => redis_get_hash_field(host_config_key, "#{host}:#{sig}").to_i == 1
|
57
|
+
@host_signal_map[host] = {sig => redis_get_hash_field(host_config_key, "#{host}:#{sig}").to_i == 1}.update(@host_signal_map[host])
|
58
58
|
end
|
59
59
|
@host_signal_map[host][sig]
|
60
60
|
end
|
@@ -14,24 +14,29 @@ module Resque
|
|
14
14
|
# Verbosity level (Integer)
|
15
15
|
attr_accessor :verbosity
|
16
16
|
|
17
|
+
attr_reader :pidfile, :zombie_term_wait, :zombie_kill_wait, :max_children
|
18
|
+
|
17
19
|
# Initialize daemon with options from command-line.
|
18
20
|
def initialize(options={})
|
19
|
-
@verbosity = (options[:verbosity] || 0).to_i
|
20
|
-
@
|
21
|
-
@
|
22
|
-
@
|
21
|
+
@verbosity = (options[:verbosity] || 0).to_i # verbosity level
|
22
|
+
@zombie_term_wait = options[:zombie_term_wait] || 20 # time to wait before TERM
|
23
|
+
@zombie_kill_wait = ENV['RESQUE_TERM_TIMEOUT'].to_i + @zombie_term_wait unless ENV['RESQUE_TERM_TIMEOUT'].nil?
|
24
|
+
@zombie_kill_wait ||= options[:zombie_kill_wait] || 60 # time to wait before -9
|
25
|
+
@hostile_takeover = options[:force] # kill running kewatcher?
|
23
26
|
@rakefile = File.expand_path(options[:rakefile]) rescue nil
|
24
27
|
@rakefile = File.exists?(@rakefile) ? @rakefile : nil if @rakefile
|
25
28
|
@pidfile = File.expand_path(options[:pidfile]) rescue nil
|
26
|
-
@pidfile = @pidfile =~ /\.pid
|
29
|
+
@pidfile = @pidfile =~ /\.pid$/ ? @pidfile : @pidfile + '.pid' if @pidfile
|
27
30
|
save_pid!
|
28
31
|
|
29
|
-
@max_children =
|
32
|
+
@max_children = options[:max_children] || 10
|
30
33
|
@hostname = `hostname -s`.chomp.downcase
|
31
34
|
@pids = Hash.new # init pids array to track running children
|
32
35
|
@need_queues = Array.new # keep track of pids that are needed
|
33
36
|
@dead_queues = Array.new # keep track of pids that are dead
|
34
37
|
@zombie_pids = Hash.new # keep track of zombie's we kill and dont watch(), with elapsed time we've waited for it to die
|
38
|
+
@async = options[:async] || false # sync and wait by default
|
39
|
+
@hupped = 0
|
35
40
|
|
36
41
|
Resque.redis = case options[:config]
|
37
42
|
when Hash
|
@@ -45,7 +50,10 @@ module Resque
|
|
45
50
|
def run!(interval=0.1)
|
46
51
|
interval = Float(interval)
|
47
52
|
if running?
|
48
|
-
|
53
|
+
unless @hostile_takeover
|
54
|
+
puts "Already running. Restart Not Forced exiting..."
|
55
|
+
exit
|
56
|
+
end
|
49
57
|
restart_running!
|
50
58
|
end
|
51
59
|
$0 = "KEWatcher: Starting"
|
@@ -58,22 +66,37 @@ module Resque
|
|
58
66
|
count += 1
|
59
67
|
log! ["watching:", @pids.keys.join(', '), "(#{@pids.keys.length})"].delete_if { |x| x == (nil || '') }.join(' ') if count % (10 / interval) == 1
|
60
68
|
|
61
|
-
tick = count % (20 / interval) == 1
|
69
|
+
tick = count % (20 / interval) == 1
|
62
70
|
(log! "checking signals..."; check_signals) if tick
|
63
71
|
if not (paused? || shutdown?)
|
64
72
|
queue_diff! if tick # do first and also about every 20 seconds so we can throttle calls to redis
|
65
73
|
|
66
74
|
while @pids.keys.length < @max_children && (@need_queues.length > 0 || @dead_queues.length > 0)
|
67
75
|
queue = @dead_queues.shift || @need_queues.shift
|
76
|
+
exec_string = ""
|
77
|
+
exec_string << 'rake'
|
78
|
+
exec_string << " -f #{@rakefile}" if @rakefile
|
79
|
+
exec_string << ' environment' if ENV['RAILS_ENV']
|
80
|
+
exec_string << ' resque:work'
|
81
|
+
env_opts = {"QUEUE" => queue}
|
82
|
+
if Resque::Version >= '1.22.0' # when API changed for signals
|
83
|
+
term_timeout = @zombie_kill_wait - @zombie_term_wait
|
84
|
+
term_timeout = term_timeout > 0 ? term_timeout : 1
|
85
|
+
env_opts.merge!({
|
86
|
+
'TERM_CHILD' => '1',
|
87
|
+
'RESQUE_TERM_TIMEOUT' => term_timeout.to_s # use new signal handling
|
88
|
+
})
|
89
|
+
end
|
90
|
+
exec_args = if RUBY_VERSION < '1.9'
|
91
|
+
[exec_string, env_opts.map {|k,v| "#{k}=#{v}"}].flatten.join(' ')
|
92
|
+
else
|
93
|
+
[env_opts, exec_string] # 1.9.x exec
|
94
|
+
end
|
68
95
|
pid = fork do
|
69
|
-
|
70
|
-
|
71
|
-
exec(exec_string + " QUEUE=#{queue}") # 1.8.x exec
|
72
|
-
else
|
73
|
-
exec({"QUEUE"=>queue}, exec_string) # 1.9.x exec
|
74
|
-
end
|
96
|
+
srand # seed
|
97
|
+
exec(*exec_args)
|
75
98
|
end
|
76
|
-
@pids.store(pid, queue) # store
|
99
|
+
@pids.store(pid, queue) # store pid and queue its running if fork() ?
|
77
100
|
procline
|
78
101
|
end
|
79
102
|
end
|
@@ -85,6 +108,12 @@ module Resque
|
|
85
108
|
|
86
109
|
sleep(interval) # microsleep
|
87
110
|
kill_zombies! unless shutdown? # need to cleanup ones we've killed
|
111
|
+
if @hupped > 0
|
112
|
+
log "HUP received; purging children..."
|
113
|
+
signal_hup
|
114
|
+
do_reload!
|
115
|
+
@hupped -= 1
|
116
|
+
end
|
88
117
|
|
89
118
|
@pids.keys.each do |pid|
|
90
119
|
begin
|
@@ -120,7 +149,7 @@ module Resque
|
|
120
149
|
count += 1
|
121
150
|
if count % 5 == 1
|
122
151
|
puts "Killing running KEWatcher: #{pid}"
|
123
|
-
Process.kill(
|
152
|
+
Process.kill(:TERM, pid)
|
124
153
|
end
|
125
154
|
s = 3 * count
|
126
155
|
puts "Waiting #{s}s for it to die..."
|
@@ -150,7 +179,7 @@ module Resque
|
|
150
179
|
|
151
180
|
begin
|
152
181
|
trap('QUIT') { shutdown! }
|
153
|
-
trap('HUP') {
|
182
|
+
trap('HUP') { @hupped += 1 }
|
154
183
|
trap('USR1') { log "USR1 received; killing little children..."; set_signal_flag('stop'); signal_usr1 }
|
155
184
|
trap('USR2') { log "USR2 received; not making babies"; set_signal_flag('pause'); signal_usr2 }
|
156
185
|
trap('CONT') { log "CONT received; making babies..."; set_signal_flag('play'); signal_cont }
|
@@ -170,7 +199,7 @@ module Resque
|
|
170
199
|
if reload?(@hostname)
|
171
200
|
log ' -> RELOAD from web-ui'
|
172
201
|
signal_hup
|
173
|
-
|
202
|
+
do_reload!
|
174
203
|
elsif stop?(@hostname)
|
175
204
|
log ' -> STOPPED from web-ui' if not paused? or @pids.keys.length > 0
|
176
205
|
signal_usr1
|
@@ -179,27 +208,29 @@ module Resque
|
|
179
208
|
signal_usr2
|
180
209
|
else
|
181
210
|
log! ' -> Continuing; no signal found'
|
182
|
-
@dead_queues = Array.new if paused? # clear killed queues when entering out of pause automatically, in same tick will refresh
|
183
211
|
signal_cont
|
184
212
|
end
|
185
213
|
end
|
186
214
|
|
187
|
-
def procline
|
188
|
-
status ||= 'stopped' if paused? and @pids.keys.empty?
|
215
|
+
def procline
|
216
|
+
status ||= 'stopped' if paused? and (@pids.keys.empty? and @zombie_pids.keys.empty?)
|
189
217
|
status ||= 'paused' if paused?
|
190
|
-
status = "#{[@pids.keys.length,status].compact.join('-')}" unless status == 'stopped'
|
191
|
-
|
218
|
+
status = "#{[@pids.keys.length, @zombie_pids.keys.length, status].compact.join('-')}" unless status == 'stopped'
|
219
|
+
name = "KEWatcher"
|
220
|
+
pid_str = []
|
221
|
+
pid_str << "R:#{@pids.keys.join(',')}" unless @pids.keys.empty?
|
222
|
+
pid_str << "Z:#{@zombie_pids.keys.join(',')}" unless @zombie_pids.keys.empty?
|
223
|
+
$0 = "#{name} (#{status}): #{pid_str.join(' ')}"
|
192
224
|
log! $0
|
193
225
|
end
|
194
226
|
|
195
227
|
def queue_diff!
|
196
228
|
# Forces queue diff
|
197
229
|
# Overrides what needs to start from Redis
|
198
|
-
|
199
|
-
to_start = diff.first
|
200
|
-
to_kill = diff.last
|
230
|
+
to_start, to_kill = queue_diff
|
201
231
|
to_kill.each { |pid| remove! pid }
|
202
232
|
@need_queues = to_start # authoritative answer from redis of what needs to be running
|
233
|
+
@dead_queues = Array.new
|
203
234
|
end
|
204
235
|
|
205
236
|
def queue_diff
|
@@ -209,8 +240,6 @@ module Resque
|
|
209
240
|
|
210
241
|
goal, to_start, to_kill = [], [], []
|
211
242
|
queue_values(@hostname).each_pair { |queue,count| goal += [queue] * count.to_i }
|
212
|
-
# to sort or not to sort?
|
213
|
-
# goal.sort!
|
214
243
|
|
215
244
|
running_queues = @pids.values # check list
|
216
245
|
goal.each do |q|
|
@@ -257,10 +286,19 @@ module Resque
|
|
257
286
|
procline
|
258
287
|
end
|
259
288
|
|
289
|
+
def do_reload!
|
290
|
+
while not @async and @zombie_pids.length > 0
|
291
|
+
kill_zombies!
|
292
|
+
end
|
293
|
+
end
|
294
|
+
|
260
295
|
def shutdown!
|
261
296
|
log "Exiting..."
|
262
297
|
@shutdown = true
|
263
298
|
kill_children
|
299
|
+
while @zombie_pids.keys.length > 0
|
300
|
+
kill_zombies!
|
301
|
+
end
|
264
302
|
%w(current max).each { |x| unregister_setting("#{x}_children") }
|
265
303
|
log! "Unregistered Max Children"
|
266
304
|
Process.waitall()
|
@@ -300,47 +338,49 @@ module Resque
|
|
300
338
|
|
301
339
|
def kill_zombies!
|
302
340
|
return if @zombie_pids.empty?
|
303
|
-
|
304
|
-
|
305
|
-
|
306
|
-
zombies.each do |pid,elapsed|
|
307
|
-
elapsed += Time.now - mark
|
308
|
-
mark = Time.now # mark that we updated elapsed; time becomes relative
|
309
|
-
time_left = (@ttime * 60.0) - elapsed # seconds
|
310
|
-
wait = time_left < wait ? time_left : wait
|
311
|
-
wait = @zombie_wait if wait < @zombie_wait && wait > 0
|
312
|
-
wait = 0.1 if wait <= 0
|
313
|
-
log! "Waiting for Zombie: #{pid} (#{'%.2f' % wait} seconds) => #{'%.2f' % elapsed} elapsed"
|
341
|
+
local_zombies = @zombie_pids.dup
|
342
|
+
to_delete = []
|
343
|
+
local_zombies.each do |pid,kill_data|
|
314
344
|
begin
|
345
|
+
when_killed, times_killed = kill_data
|
346
|
+
elapsed = Time.now - when_killed
|
347
|
+
sig = if elapsed >= @zombie_term_wait and times_killed == 1
|
348
|
+
:TERM
|
349
|
+
elsif elapsed >= @zombie_kill_wait and not Resque::Version >= '1.22.0'
|
350
|
+
:KILL
|
351
|
+
else
|
352
|
+
nil
|
353
|
+
end
|
354
|
+
unless sig.nil?
|
355
|
+
log "Waited more than #{@zombie_term_wait} seconds for #{pid}. Sending #{sig}..."
|
356
|
+
Process.kill(sig, pid)
|
357
|
+
@zombie_pids.merge!({pid => [when_killed, times_killed + 1]})
|
358
|
+
end
|
359
|
+
wait = !@async ? (@zombie_term_wait - elapsed) / @zombie_pids.length : 0.01
|
360
|
+
wait = wait > 0 ? wait : 0.01
|
315
361
|
# Issue wait() to make sure pid isn't forgotten
|
316
362
|
Timeout::timeout(wait) { Process.wait(pid) }
|
363
|
+
to_delete << pid
|
364
|
+
next
|
317
365
|
rescue Timeout::Error
|
318
|
-
|
319
|
-
|
320
|
-
|
321
|
-
(log "Waited more than #{@ttime} minutes for #{pid}. Force quitting..."; Process.kill('TERM',pid)) if elapsed / 60.0 >= @ttime
|
366
|
+
# waited too long so just catch and ignore, and continue
|
367
|
+
rescue Errno::ESRCH, Errno::ECHILD # child is gone
|
368
|
+
to_delete << pid
|
322
369
|
next
|
323
|
-
rescue Errno::ECHILD # child is gone
|
324
|
-
ensure
|
325
|
-
elapsed += Time.now - mark
|
326
|
-
mark = Time.now # mark that we updated elapsed
|
327
|
-
log! "Elapsed incr: #{elapsed}"
|
328
|
-
zombies.each_key { |x| zombies[x] = elapsed } # reset all to current elapsed
|
329
|
-
@zombie_pids = zombies # make sure to update root Hash
|
330
370
|
end
|
331
|
-
@zombie_pids.delete(pid)
|
332
371
|
end
|
372
|
+
to_delete.each { |pid| @zombie_pids.delete(pid) }
|
333
373
|
end
|
334
374
|
|
335
375
|
def kill_child(pid)
|
336
376
|
begin
|
337
|
-
Process.kill(
|
377
|
+
Process.kill(:QUIT, pid) # try graceful shutdown
|
338
378
|
log! "Child #{pid} killed. (#{@pids.keys.length-1})"
|
339
379
|
rescue Object => e # dunno what this does but it works; dont know exception
|
340
380
|
log! "Child #{pid} already dead, sad day. (#{@pids.keys.length-1}) #{e}"
|
341
381
|
ensure
|
342
|
-
# Keep track of ones we
|
343
|
-
@zombie_pids[pid] =
|
382
|
+
# Keep track of ones we've killed
|
383
|
+
@zombie_pids[pid] = [Time.now, 1] # set to current time, killed #
|
344
384
|
end
|
345
385
|
end
|
346
386
|
|
@@ -15,7 +15,7 @@ module Resque
|
|
15
15
|
public_view(params[key], key == 'img' ? 'images' : key)
|
16
16
|
else
|
17
17
|
@sliders = Commander.new
|
18
|
-
redirect "/sliders/#{@sliders.all_hosts.first}" if @sliders.all_hosts.length == 1
|
18
|
+
redirect url_path("/sliders/#{@sliders.all_hosts.first}") if @sliders.all_hosts.length == 1
|
19
19
|
slider_view :index
|
20
20
|
end
|
21
21
|
end
|
@@ -51,10 +51,10 @@ module Resque
|
|
51
51
|
end
|
52
52
|
|
53
53
|
def public_view(filename, dir='')
|
54
|
+
file = File.join(PUBLIC_PATH, dir, filename)
|
54
55
|
begin
|
55
56
|
cache_control :public, :max_age => 1800
|
56
|
-
file
|
57
|
-
send_file file, :last_modified => File.mtime(file)
|
57
|
+
send_file file
|
58
58
|
rescue Errno::ENOENT
|
59
59
|
404
|
60
60
|
end
|
@@ -59,26 +59,26 @@
|
|
59
59
|
.ui-widget { font-family: Arial,sans-serif; font-size: 1.1em; }
|
60
60
|
.ui-widget .ui-widget { font-size: 1em; }
|
61
61
|
.ui-widget input, .ui-widget select, .ui-widget textarea, .ui-widget button { font-family: Arial,sans-serif; font-size: 1em; }
|
62
|
-
.ui-widget-content { border: 1px solid #eeeeee; background: #ffffff url(
|
62
|
+
.ui-widget-content { border: 1px solid #eeeeee; background: #ffffff url(?img=ui-bg_flat_75_ffffff_40x100.png) 50% 50% repeat-x; color: #333333; }
|
63
63
|
.ui-widget-content a { color: #333333; }
|
64
|
-
.ui-widget-header { border: 1px solid #e3a1a1; background: #cc0000 url(
|
64
|
+
.ui-widget-header { border: 1px solid #e3a1a1; background: #cc0000 url(?img=ui-bg_highlight-soft_15_cc0000_1x100.png) 50% 50% repeat-x; color: #ffffff; font-weight: bold; }
|
65
65
|
.ui-widget-header a { color: #ffffff; }
|
66
66
|
|
67
67
|
/* Interaction states
|
68
68
|
----------------------------------*/
|
69
|
-
.ui-state-default, .ui-widget-content .ui-state-default, .ui-widget-header .ui-state-default { border: 1px solid #d8dcdf; background: #eeeeee url(
|
69
|
+
.ui-state-default, .ui-widget-content .ui-state-default, .ui-widget-header .ui-state-default { border: 1px solid #d8dcdf; background: #eeeeee url(?img=ui-bg_highlight-hard_100_eeeeee_1x100.png) 50% 50% repeat-x; font-weight: bold; color: #004276; }
|
70
70
|
.ui-state-default a, .ui-state-default a:link, .ui-state-default a:visited { color: #004276; text-decoration: none; }
|
71
|
-
.ui-state-hover, .ui-widget-content .ui-state-hover, .ui-widget-header .ui-state-hover, .ui-state-focus, .ui-widget-content .ui-state-focus, .ui-widget-header .ui-state-focus { border: 1px solid #cdd5da; background: #f6f6f6 url(
|
71
|
+
.ui-state-hover, .ui-widget-content .ui-state-hover, .ui-widget-header .ui-state-hover, .ui-state-focus, .ui-widget-content .ui-state-focus, .ui-widget-header .ui-state-focus { border: 1px solid #cdd5da; background: #f6f6f6 url(?img=ui-bg_highlight-hard_100_f6f6f6_1x100.png) 50% 50% repeat-x; font-weight: bold; color: #111111; }
|
72
72
|
.ui-state-hover a, .ui-state-hover a:hover { color: #111111; text-decoration: none; }
|
73
|
-
.ui-state-active, .ui-widget-content .ui-state-active, .ui-widget-header .ui-state-active { border: 1px solid #eeeeee; background: #ffffff url(
|
73
|
+
.ui-state-active, .ui-widget-content .ui-state-active, .ui-widget-header .ui-state-active { border: 1px solid #eeeeee; background: #ffffff url(?img=ui-bg_flat_65_ffffff_40x100.png) 50% 50% repeat-x; font-weight: bold; color: #cc0000; }
|
74
74
|
.ui-state-active a, .ui-state-active a:link, .ui-state-active a:visited { color: #cc0000; text-decoration: none; }
|
75
75
|
.ui-widget :active { outline: none; }
|
76
76
|
|
77
77
|
/* Interaction Cues
|
78
78
|
----------------------------------*/
|
79
|
-
.ui-state-highlight, .ui-widget-content .ui-state-highlight, .ui-widget-header .ui-state-highlight {border: 1px solid #fcd3a1; background: #fbf8ee url(
|
79
|
+
.ui-state-highlight, .ui-widget-content .ui-state-highlight, .ui-widget-header .ui-state-highlight {border: 1px solid #fcd3a1; background: #fbf8ee url(?img=ui-bg_glass_55_fbf8ee_1x400.png) 50% 50% repeat-x; color: #444444; }
|
80
80
|
.ui-state-highlight a, .ui-widget-content .ui-state-highlight a,.ui-widget-header .ui-state-highlight a { color: #444444; }
|
81
|
-
.ui-state-error, .ui-widget-content .ui-state-error, .ui-widget-header .ui-state-error {border: 1px solid #cc0000; background: #f3d8d8 url(
|
81
|
+
.ui-state-error, .ui-widget-content .ui-state-error, .ui-widget-header .ui-state-error {border: 1px solid #cc0000; background: #f3d8d8 url(?img=ui-bg_diagonals-thick_75_f3d8d8_40x40.png) 50% 50% repeat; color: #2e2e2e; }
|
82
82
|
.ui-state-error a, .ui-widget-content .ui-state-error a, .ui-widget-header .ui-state-error a { color: #2e2e2e; }
|
83
83
|
.ui-state-error-text, .ui-widget-content .ui-state-error-text, .ui-widget-header .ui-state-error-text { color: #2e2e2e; }
|
84
84
|
.ui-priority-primary, .ui-widget-content .ui-priority-primary, .ui-widget-header .ui-priority-primary { font-weight: bold; }
|
@@ -89,14 +89,14 @@
|
|
89
89
|
----------------------------------*/
|
90
90
|
|
91
91
|
/* states and images */
|
92
|
-
.ui-icon { width: 16px; height: 16px; background-image: url(
|
93
|
-
.ui-widget-content .ui-icon {background-image: url(
|
94
|
-
.ui-widget-header .ui-icon {background-image: url(
|
95
|
-
.ui-state-default .ui-icon { background-image: url(
|
96
|
-
.ui-state-hover .ui-icon, .ui-state-focus .ui-icon {background-image: url(
|
97
|
-
.ui-state-active .ui-icon {background-image: url(
|
98
|
-
.ui-state-highlight .ui-icon {background-image: url(
|
99
|
-
.ui-state-error .ui-icon, .ui-state-error-text .ui-icon {background-image: url(
|
92
|
+
.ui-icon { width: 16px; height: 16px; background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
93
|
+
.ui-widget-content .ui-icon {background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
94
|
+
.ui-widget-header .ui-icon {background-image: url(?img=ui-icons_ffffff_256x240.png); }
|
95
|
+
.ui-state-default .ui-icon { background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
96
|
+
.ui-state-hover .ui-icon, .ui-state-focus .ui-icon {background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
97
|
+
.ui-state-active .ui-icon {background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
98
|
+
.ui-state-highlight .ui-icon {background-image: url(?img=ui-icons_004276_256x240.png); }
|
99
|
+
.ui-state-error .ui-icon, .ui-state-error-text .ui-icon {background-image: url(?img=ui-icons_cc0000_256x240.png); }
|
100
100
|
|
101
101
|
/* positioning */
|
102
102
|
.ui-icon-carat-1-n { background-position: 0 0; }
|
@@ -286,8 +286,8 @@
|
|
286
286
|
.ui-corner-all, .ui-corner-bottom, .ui-corner-right, .ui-corner-br { -moz-border-radius-bottomright: 6px; -webkit-border-bottom-right-radius: 6px; -khtml-border-bottom-right-radius: 6px; border-bottom-right-radius: 6px; }
|
287
287
|
|
288
288
|
/* Overlays */
|
289
|
-
.ui-widget-overlay { background: #a6a6a6 url(
|
290
|
-
.ui-widget-shadow { margin: -8px 0 0 -8px; padding: 8px; background: #333333 url(
|
289
|
+
.ui-widget-overlay { background: #a6a6a6 url(?img=ui-bg_dots-small_65_a6a6a6_2x2.png) 50% 50% repeat; opacity: .40;filter:Alpha(Opacity=40); }
|
290
|
+
.ui-widget-shadow { margin: -8px 0 0 -8px; padding: 8px; background: #333333 url(?img=ui-bg_flat_0_333333_40x100.png) 50% 50% repeat-x; opacity: .10;filter:Alpha(Opacity=10); -moz-border-radius: 8px; -khtml-border-radius: 8px; -webkit-border-radius: 8px; border-radius: 8px; }/*
|
291
291
|
* jQuery UI Resizable 1.8.16
|
292
292
|
*
|
293
293
|
* Copyright 2011, AUTHORS.txt (http://jqueryui.com/about)
|
@@ -34,7 +34,9 @@ $(document).ready(function() {
|
|
34
34
|
signal[sig] = true;
|
35
35
|
|
36
36
|
var span = $(this);
|
37
|
-
|
37
|
+
var re = new RegExp('/' + host + '$');
|
38
|
+
var url = window.location.pathname.replace(re, '') + '/' + host;
|
39
|
+
$.post(url, signal, function(data) {
|
38
40
|
switch(data.signal) {
|
39
41
|
case 'reload':
|
40
42
|
span.removeClass('ui-icon-refresh').addClass('ui-icon-alert').attr('id', [host, 'ALERT'].join(':'));
|
@@ -55,7 +57,7 @@ $(document).ready(function() {
|
|
55
57
|
e.preventDefault();
|
56
58
|
var queue = sanitize_input($("#new_queue").val());
|
57
59
|
var host = $(this).attr("id");
|
58
|
-
$.post(
|
60
|
+
$.post(host, { quantity: 1, queue: queue }, function(data) {
|
59
61
|
// reload window
|
60
62
|
window.location.reload(true);
|
61
63
|
});
|
@@ -83,7 +85,7 @@ $(document).ready(function() {
|
|
83
85
|
set_total();
|
84
86
|
},
|
85
87
|
change: function( event, ui ) {
|
86
|
-
$.post('
|
88
|
+
$.post('', { quantity: ui.value, queue: queue }, function(data) {
|
87
89
|
});
|
88
90
|
},
|
89
91
|
});
|
@@ -3,10 +3,10 @@
|
|
3
3
|
<h1>Sliders</h1>
|
4
4
|
|
5
5
|
<div id="sliders">
|
6
|
-
<link href="/sliders?css=sliders.css" rel="stylesheet" type="text/css" />
|
7
|
-
<link href="/sliders?css=jqueryui-1.8.16/blitzer/jquery-ui.custom.css" rel="stylesheet" type="text/css" />
|
8
|
-
<script src="/sliders?js=jquery-ui-1.8.16.custom.min.js" type="text/javascript"></script>
|
9
|
-
<script src="/sliders?js=sliders.js" type="text/javascript"></script>
|
6
|
+
<link href="<%= url_path("/sliders?css=sliders.css") %>" rel="stylesheet" type="text/css" />
|
7
|
+
<link href="<%= url_path("/sliders?css=jqueryui-1.8.16/blitzer/jquery-ui.custom.css") %>" rel="stylesheet" type="text/css" />
|
8
|
+
<script src="<%= url_path("/sliders?js=jquery-ui-1.8.16.custom.min.js") %>" type="text/javascript"></script>
|
9
|
+
<script src="<%= url_path("/sliders?js=sliders.js") %>" type="text/javascript"></script>
|
10
10
|
<% if params[:host] || @sliders.all_hosts.length == 1 %>
|
11
11
|
<% host = params[:host] || @sliders.all_hosts.first %>
|
12
12
|
<% max = @sliders.max_children(host) || 'Not running' %>
|
@@ -36,7 +36,7 @@
|
|
36
36
|
</tr>
|
37
37
|
<% @sliders.all_hosts.each do |host| %>
|
38
38
|
<tr>
|
39
|
-
<td><a href="/sliders
|
39
|
+
<td><a href="<%= url_path("/sliders/#{host}") %>"><%= host %></a></td>
|
40
40
|
<td><% if @sliders.stale_hosts.include?(host) %>
|
41
41
|
Not Checked in (Stale)
|
42
42
|
<% else %>
|
@@ -0,0 +1,44 @@
|
|
1
|
+
require File.expand_path(File.dirname(__FILE__) + '/test_helper')
|
2
|
+
require 'tempfile'
|
3
|
+
|
4
|
+
context "kewatcher" do
|
5
|
+
setup do
|
6
|
+
Resque.redis.flushall
|
7
|
+
pidfile = Tempfile.new('pidfile')
|
8
|
+
@options = {
|
9
|
+
:pidfile => pidfile.path,
|
10
|
+
:config => '127.0.0.1:9736'
|
11
|
+
}
|
12
|
+
@kewatcher = Resque::Plugins::ResqueSliders::KEWatcher.new(@options)
|
13
|
+
end
|
14
|
+
|
15
|
+
teardown do
|
16
|
+
pid = @kewatcher.running?
|
17
|
+
Process.kill(:TERM, pid) if pid
|
18
|
+
end
|
19
|
+
|
20
|
+
test "kewatcher" do
|
21
|
+
assert_instance_of Resque::Plugins::ResqueSliders::KEWatcher, @kewatcher
|
22
|
+
end
|
23
|
+
|
24
|
+
test "saves pidfile" do
|
25
|
+
assert File.exists?(@kewatcher.pidfile)
|
26
|
+
end
|
27
|
+
|
28
|
+
test "kewatcher runs" do
|
29
|
+
`bundle exec kewatcher --config #{@options[:config]} >/dev/null 2>&1 & sleep 3`
|
30
|
+
assert @kewatcher.running?
|
31
|
+
end
|
32
|
+
|
33
|
+
test "kewatcher wont run twice" do
|
34
|
+
`bundle exec kewatcher --config #{@options[:config]} >/dev/null 2>&1 &`
|
35
|
+
sleep 3
|
36
|
+
output = `bundle exec kewatcher --config #{@options[:config]}`
|
37
|
+
assert_match %r{Already running}, output
|
38
|
+
end
|
39
|
+
|
40
|
+
test "system forks" do
|
41
|
+
assert fork { exit }
|
42
|
+
end
|
43
|
+
|
44
|
+
end
|
data/test/redis-test.conf
CHANGED
@@ -14,7 +14,7 @@ port 9736
|
|
14
14
|
# If you want you can bind a single interface, if the bind option is not
|
15
15
|
# specified all the interfaces will listen for connections.
|
16
16
|
#
|
17
|
-
|
17
|
+
bind 127.0.0.1
|
18
18
|
|
19
19
|
# Close the connection after a client is idle for N seconds (0 to disable)
|
20
20
|
timeout 300
|
data/test/test_helper.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-sliders
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-10-
|
12
|
+
date: 2012-10-23 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: resque
|
@@ -48,7 +48,6 @@ extra_rdoc_files:
|
|
48
48
|
files:
|
49
49
|
- MIT-LICENSE
|
50
50
|
- README.md
|
51
|
-
- Rakefile
|
52
51
|
- bin/kewatcher
|
53
52
|
- config/config_example.yml
|
54
53
|
- lib/resque-sliders.rb
|
@@ -74,8 +73,8 @@ files:
|
|
74
73
|
- lib/resque-sliders/server/public/js/sliders.js
|
75
74
|
- lib/resque-sliders/server/views/index.erb
|
76
75
|
- lib/resque-sliders/version.rb
|
76
|
+
- test/kewatcher_test.rb
|
77
77
|
- test/redis-test.conf
|
78
|
-
- test/resque-sliders_test.rb
|
79
78
|
- test/test_helper.rb
|
80
79
|
homepage: https://github.com/kmullin/resque-sliders
|
81
80
|
licenses: []
|
@@ -89,12 +88,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
89
88
|
- - ! '>='
|
90
89
|
- !ruby/object:Gem::Version
|
91
90
|
version: '0'
|
91
|
+
segments:
|
92
|
+
- 0
|
93
|
+
hash: 430698583574743090
|
92
94
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
93
95
|
none: false
|
94
96
|
requirements:
|
95
97
|
- - ! '>='
|
96
98
|
- !ruby/object:Gem::Version
|
97
99
|
version: '0'
|
100
|
+
segments:
|
101
|
+
- 0
|
102
|
+
hash: 430698583574743090
|
98
103
|
requirements: []
|
99
104
|
rubyforge_project:
|
100
105
|
rubygems_version: 1.8.24
|
@@ -103,6 +108,6 @@ specification_version: 3
|
|
103
108
|
summary: ! 'Resque-Sliders: a plugin for resque that controls which resque workers
|
104
109
|
are running on each host, from within Resque-web'
|
105
110
|
test_files:
|
111
|
+
- test/kewatcher_test.rb
|
106
112
|
- test/redis-test.conf
|
107
|
-
- test/resque-sliders_test.rb
|
108
113
|
- test/test_helper.rb
|
data/Rakefile
DELETED
@@ -1,41 +0,0 @@
|
|
1
|
-
$LOAD_PATH.unshift 'helpers'
|
2
|
-
$LOAD_PATH.unshift 'lib'
|
3
|
-
#
|
4
|
-
# Setup
|
5
|
-
#
|
6
|
-
|
7
|
-
|
8
|
-
require 'bundler/gem_tasks'
|
9
|
-
require 'resque/tasks'
|
10
|
-
require 'resque_job'
|
11
|
-
|
12
|
-
def command?(command)
|
13
|
-
system("type #{command} > /dev/null 2>&1")
|
14
|
-
end
|
15
|
-
|
16
|
-
|
17
|
-
#
|
18
|
-
# Tests
|
19
|
-
#
|
20
|
-
|
21
|
-
task :default => :test
|
22
|
-
|
23
|
-
desc "Run the test suite"
|
24
|
-
task :test do
|
25
|
-
rg = command?(:rg)
|
26
|
-
Dir['test/**/*_test.rb'].each do |f|
|
27
|
-
rg ? sh("rg #{f}") : ruby(f)
|
28
|
-
end
|
29
|
-
end
|
30
|
-
|
31
|
-
desc "Bump version"
|
32
|
-
task :git_tag_version do
|
33
|
-
require 'resque-sliders/version'
|
34
|
-
git_tags = `git tag -l`.split.map { |x| x.gsub(/v/, '') }
|
35
|
-
version = Resque::Plugins::ResqueSliders::Version
|
36
|
-
commit_sha = `git log -1 HEAD|head -n1|awk '{print $2}'`
|
37
|
-
unless git_tags.include?(version)
|
38
|
-
(puts version, commit_sha; `git tag v#{version} #{commit_sha}`)
|
39
|
-
`rake build`
|
40
|
-
end
|
41
|
-
end
|