racecar 0.3.1 → 0.3.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +31 -5
- data/exe/racecar +7 -1
- data/exe/racecarctl +0 -1
- data/lib/racecar/cli.rb +119 -28
- data/lib/racecar/config.rb +93 -123
- data/lib/racecar/ctl.rb +62 -5
- data/lib/racecar/daemon.rb +97 -0
- data/lib/racecar/rails_config_file_loader.rb +7 -3
- data/lib/racecar/runner.rb +6 -2
- data/lib/racecar/version.rb +1 -1
- data/racecar.gemspec +1 -0
- metadata +19 -4
- data/lib/racecar/env_loader.rb +0 -49
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 88312452dcd53ac7b8097149e4b2f07231fdcd57
|
4
|
+
data.tar.gz: 9206d8f5584cd93d366f9d41f15314a9a8e89abf
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3bc5f769117603f47314259f3d6b639a370fac11b17b66883f4fe9e30087cc9d14be1c63e48603e9c69ad5a51059dcb846a666499b447915c81f454d5e0b197d
|
7
|
+
data.tar.gz: 5fcdd199a4f3c047341aa96ccf458ff7ba9ff211e61513453cd5ed1a57dc7b183e9bfa5c933deeb4956b823b1045f9732fd1e7b4b74145a62738c48134602a75
|
data/CHANGELOG.md
ADDED
data/README.md
CHANGED
@@ -18,7 +18,8 @@ Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a chal
|
|
18
18
|
8. [Operations](#operations)
|
19
19
|
3. [Development](#development)
|
20
20
|
4. [Contributing](#contributing)
|
21
|
-
5. [
|
21
|
+
5. [Support and Discussion](#support-and-discussion)
|
22
|
+
6. [Copyright and license](#copyright-and-license)
|
22
23
|
|
23
24
|
|
24
25
|
## Installation
|
@@ -173,6 +174,7 @@ Racecar is first and foremost an executable _consumer runner_. The `racecar` exe
|
|
173
174
|
|
174
175
|
The first time you execute `racecar` with a consumer class a _consumer group_ will be created with a group id derived from the class name (this can be configured). If you start `racecar` with the same consumer class argument multiple times, the processes will join the existing group – even if you start them on other nodes. You will typically want to have at least two consumers in each of your groups – preferably on separate nodes – in order to deal with failures.
|
175
176
|
|
177
|
+
|
176
178
|
### Configuration
|
177
179
|
|
178
180
|
Racecar provides a flexible way to configure your consumer in a way that feels at home in a Rails application. If you haven't already, run `bundle exec rails generate racecar:install` in order to generate a config file. You'll get a separate section for each Rails environment, with the common configuration values in a shared `common` section.
|
@@ -219,7 +221,7 @@ Racecar has support for using SASL to authenticate clients using either the GSSA
|
|
219
221
|
|
220
222
|
If using GSSAPI:
|
221
223
|
|
222
|
-
* `sasl_gssapi_principal` – The GSSAPI principal
|
224
|
+
* `sasl_gssapi_principal` – The GSSAPI principal.
|
223
225
|
* `sasl_gssapi_keytab` – Optional GSSAPI keytab.
|
224
226
|
|
225
227
|
If using PLAIN:
|
@@ -280,6 +282,23 @@ If you've ever used Heroku you'll recognize the format – indeed, deploying to
|
|
280
282
|
With Foreman, you can easily run these processes locally by executing `foreman run`; in production you'll want to _export_ to another process management format such as Upstart or Runit. [capistrano-foreman](https://github.com/hyperoslo/capistrano-foreman) allows you to do this with Capistrano.
|
281
283
|
|
282
284
|
|
285
|
+
#### Running consumers in the background
|
286
|
+
|
287
|
+
While it is recommended that you use a process supervisor to manage the Racecar consumer processes, it is possible to _daemonize_ the Racecar processes themselves if that is more to your liking. Note that this support is currently in alpha, as it hasn't been tested extensively in production settings.
|
288
|
+
|
289
|
+
In order to daemonize Racecar, simply pass in `--daemonize` when executing the command:
|
290
|
+
|
291
|
+
$ bundle exec racecar --daemonize ResizeImagesConsumer
|
292
|
+
|
293
|
+
This will start the consumer process in the background. A file containing the process id (the "pidfile") will be created, with the file name being constructed from the consumer class name. If you want to specify the name of the pidfile yourself, pass in `--pidfile=some-file.pid`.
|
294
|
+
|
295
|
+
Since the process is daemonized, you need to know the process id (PID) in order to be able to stop it. Use the `racecarctl` command to do this:
|
296
|
+
|
297
|
+
$ bundle exec racecarctl stop --pidfile=some-file.pid
|
298
|
+
|
299
|
+
Again, the recommended approach is to manage the processes using process managers. Only do this if you have to.
|
300
|
+
|
301
|
+
|
283
302
|
### Handling errors
|
284
303
|
|
285
304
|
When processing messages from a Kafka topic, your code may encounter an error and raise an exception. The cause is typically on of two things:
|
@@ -334,13 +353,20 @@ In order to introspect the configuration of a consumer process, send it the `SIG
|
|
334
353
|
|
335
354
|
## Development
|
336
355
|
|
337
|
-
After checking out the repo, run `bin/setup` to install dependencies. Then, run `
|
356
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rspec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
338
357
|
|
339
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
340
358
|
|
341
359
|
## Contributing
|
342
360
|
|
343
|
-
Bug reports and pull requests are welcome on GitHub
|
361
|
+
Bug reports and pull requests are welcome on [GitHub](https://github.com/zendesk/racecar). Feel free to [join our Slack team](https://ruby-kafka-slack.herokuapp.com/) and ask how best to contribute!
|
362
|
+
|
363
|
+
|
364
|
+
## Support and Discussion
|
365
|
+
|
366
|
+
If you've discovered a bug, please file a [Github issue](https://github.com/zendesk/racecar/issues/new), and make sure to include all the relevant information, including the version of Racecar, ruby-kafka, and Kafka that you're using.
|
367
|
+
|
368
|
+
If you have other questions, or would like to discuss best practises, how to contribute to the project, or any other ruby-kafka related topic, [join our Slack team](https://ruby-kafka-slack.herokuapp.com/)!
|
369
|
+
|
344
370
|
|
345
371
|
## Copyright and license
|
346
372
|
|
data/exe/racecar
CHANGED
data/exe/racecarctl
CHANGED
data/lib/racecar/cli.rb
CHANGED
@@ -1,32 +1,26 @@
|
|
1
1
|
require "optparse"
|
2
2
|
require "logger"
|
3
|
+
require "fileutils"
|
3
4
|
require "racecar/rails_config_file_loader"
|
5
|
+
require "racecar/daemon"
|
4
6
|
|
5
7
|
module Racecar
|
6
|
-
|
8
|
+
class Cli
|
7
9
|
def self.main(args)
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
opts.on("-r", "--require LIBRARY", "Require the LIBRARY before starting the consumer") do |lib|
|
12
|
-
require lib
|
13
|
-
end
|
14
|
-
|
15
|
-
opts.on("-l", "--log LOGFILE", "Log to the specified file") do |logfile|
|
16
|
-
Racecar.config.logfile = logfile
|
17
|
-
end
|
18
|
-
|
19
|
-
opts.on_tail("--version", "Show Racecar version") do
|
20
|
-
require "racecar/version"
|
21
|
-
$stderr.puts "Racecar #{Racecar::VERSION}"
|
22
|
-
exit
|
23
|
-
end
|
24
|
-
end
|
10
|
+
new(args).run
|
11
|
+
end
|
25
12
|
|
26
|
-
|
13
|
+
def initialize(args)
|
14
|
+
@parser = build_parser
|
15
|
+
@parser.parse!(args)
|
16
|
+
@consumer_name = args.first or raise Racecar::Error, "no consumer specified"
|
17
|
+
end
|
27
18
|
|
28
|
-
|
19
|
+
def config
|
20
|
+
Racecar.config
|
21
|
+
end
|
29
22
|
|
23
|
+
def run
|
30
24
|
$stderr.puts "=> Starting Racecar consumer #{consumer_name}..."
|
31
25
|
|
32
26
|
RailsConfigFileLoader.load!
|
@@ -35,23 +29,120 @@ module Racecar
|
|
35
29
|
consumer_class = Kernel.const_get(consumer_name)
|
36
30
|
|
37
31
|
# Load config defined by the consumer class itself.
|
38
|
-
|
32
|
+
config.load_consumer_class(consumer_class)
|
39
33
|
|
40
|
-
|
34
|
+
config.validate!
|
35
|
+
|
36
|
+
if config.logfile
|
37
|
+
$stderr.puts "=> Logging to #{config.logfile}"
|
38
|
+
Racecar.logger = Logger.new(config.logfile)
|
39
|
+
end
|
41
40
|
|
42
|
-
if
|
43
|
-
|
44
|
-
|
41
|
+
if config.log_level
|
42
|
+
Racecar.logger.level = config.log_level
|
43
|
+
end
|
44
|
+
|
45
|
+
if config.datadog_enabled
|
46
|
+
configure_datadog
|
45
47
|
end
|
46
48
|
|
47
49
|
$stderr.puts "=> Wrooooom!"
|
48
|
-
|
50
|
+
|
51
|
+
if config.daemonize
|
52
|
+
daemonize!
|
53
|
+
else
|
54
|
+
$stderr.puts "=> Ctrl-C to shutdown consumer"
|
55
|
+
end
|
49
56
|
|
50
57
|
processor = consumer_class.new
|
51
58
|
|
52
|
-
|
59
|
+
begin
|
60
|
+
Racecar.run(processor)
|
61
|
+
rescue => e
|
62
|
+
$stderr.puts "=> Crashed: #{e}"
|
63
|
+
|
64
|
+
raise
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
private
|
69
|
+
|
70
|
+
attr_reader :consumer_name
|
71
|
+
|
72
|
+
def daemonize!
|
73
|
+
daemon = Daemon.new(File.expand_path(config.pidfile))
|
74
|
+
|
75
|
+
daemon.check_pid
|
76
|
+
|
77
|
+
$stderr.puts "=> Starting background process"
|
78
|
+
$stderr.puts "=> Writing PID to #{daemon.pidfile}"
|
79
|
+
|
80
|
+
daemon.suppress_input
|
81
|
+
|
82
|
+
if config.logfile.nil?
|
83
|
+
daemon.suppress_output
|
84
|
+
else
|
85
|
+
daemon.redirect_output(config.logfile)
|
86
|
+
end
|
87
|
+
|
88
|
+
daemon.daemonize!
|
89
|
+
daemon.write_pid
|
90
|
+
end
|
91
|
+
|
92
|
+
def build_parser
|
93
|
+
OptionParser.new do |opts|
|
94
|
+
opts.banner = "Usage: racecar MyConsumer [options]"
|
95
|
+
|
96
|
+
opts.on("-r", "--require STRING", "Require a library before starting the consumer") do |lib|
|
97
|
+
require lib
|
98
|
+
end
|
99
|
+
|
100
|
+
opts.on("-l", "--log STRING", "Log to the specified file") do |logfile|
|
101
|
+
config.logfile = logfile
|
102
|
+
end
|
103
|
+
|
104
|
+
Racecar::Config.variables.each do |variable|
|
105
|
+
opt_name = "--" << variable.name.to_s.gsub("_", "-")
|
106
|
+
opt_name << " #{variable.type.upcase}" unless variable.boolean?
|
107
|
+
|
108
|
+
desc = variable.description || "N/A"
|
109
|
+
|
110
|
+
if variable.default
|
111
|
+
desc << " (default: #{variable.default.inspect})"
|
112
|
+
end
|
113
|
+
|
114
|
+
opts.on(opt_name, desc) do |value|
|
115
|
+
if variable.boolean?
|
116
|
+
# Boolean switches are automatically mapped to true/false.
|
117
|
+
config.set(variable.name, value)
|
118
|
+
else
|
119
|
+
# Other CLI params need to be decoded into values of the correct type.
|
120
|
+
config.decode(variable.name, value)
|
121
|
+
end
|
122
|
+
end
|
123
|
+
end
|
124
|
+
|
125
|
+
opts.on_tail("--version", "Show Racecar version") do
|
126
|
+
require "racecar/version"
|
127
|
+
$stderr.puts "Racecar #{Racecar::VERSION}"
|
128
|
+
exit
|
129
|
+
end
|
130
|
+
|
131
|
+
opts.on_tail("-h", "--help", "Show this message") do
|
132
|
+
puts opts
|
133
|
+
exit
|
134
|
+
end
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
|
+
def configure_datadog
|
139
|
+
require "kafka/datadog"
|
53
140
|
|
54
|
-
|
141
|
+
datadog = Kafka::Datadog
|
142
|
+
datadog.host = config.datadog_host if config.datadog_host.present?
|
143
|
+
datadog.port = config.datadog_port if config.datadog_port.present?
|
144
|
+
datadog.namespace = config.datadog_namespace if config.datadog_namespace.present?
|
145
|
+
datadog.tags = config.datadog_tags if config.datadog_tags.present?
|
55
146
|
end
|
56
147
|
end
|
57
148
|
end
|
data/lib/racecar/config.rb
CHANGED
@@ -1,100 +1,120 @@
|
|
1
|
-
require "
|
2
|
-
require "yaml"
|
3
|
-
require "racecar/env_loader"
|
1
|
+
require "king_konf"
|
4
2
|
|
5
3
|
module Racecar
|
6
|
-
class Config
|
7
|
-
|
8
|
-
brokers
|
9
|
-
client_id
|
4
|
+
class Config < KingKonf::Config
|
5
|
+
prefix :racecar
|
10
6
|
|
11
|
-
|
12
|
-
|
7
|
+
desc "A list of Kafka brokers in the cluster that you're consuming from"
|
8
|
+
list :brokers, default: ["localhost:9092"]
|
13
9
|
|
14
|
-
|
15
|
-
|
16
|
-
pause_timeout
|
17
|
-
connect_timeout
|
18
|
-
socket_timeout
|
19
|
-
group_id_prefix
|
20
|
-
group_id
|
21
|
-
subscriptions
|
22
|
-
max_wait_time
|
10
|
+
desc "A string used to identify the client in logs and metrics"
|
11
|
+
string :client_id, default: "racecar"
|
23
12
|
|
24
|
-
|
25
|
-
|
13
|
+
desc "How frequently to commit offset positions"
|
14
|
+
integer :offset_commit_interval, default: 10
|
26
15
|
|
27
|
-
|
28
|
-
|
29
|
-
ssl_client_cert
|
30
|
-
ssl_client_cert_key
|
16
|
+
desc "How many messages to process before forcing a checkpoint"
|
17
|
+
integer :offset_commit_threshold, default: 0
|
31
18
|
|
32
|
-
|
33
|
-
|
34
|
-
sasl_plain_authzid
|
35
|
-
sasl_plain_username
|
36
|
-
sasl_plain_password
|
37
|
-
)
|
19
|
+
desc "How often to send a heartbeat message to Kafka"
|
20
|
+
integer :heartbeat_interval, default: 10
|
38
21
|
|
39
|
-
|
40
|
-
|
41
|
-
client_id
|
42
|
-
)
|
22
|
+
desc "How long to pause a partition for if the consumer raises an exception while processing a message"
|
23
|
+
integer :pause_timeout, default: 10
|
43
24
|
|
44
|
-
|
45
|
-
|
46
|
-
client_id: "racecar",
|
25
|
+
desc "The idle timeout after which a consumer is kicked out of the group"
|
26
|
+
integer :session_timeout, default: 30
|
47
27
|
|
48
|
-
|
28
|
+
desc "How long to wait when trying to connect to a Kafka broker"
|
29
|
+
integer :connect_timeout, default: 10
|
49
30
|
|
50
|
-
|
51
|
-
|
31
|
+
desc "How long to wait when trying to communicate with a Kafka broker"
|
32
|
+
integer :socket_timeout, default: 30
|
52
33
|
|
53
|
-
|
54
|
-
|
34
|
+
desc "How long to allow the Kafka brokers to wait before returning messages"
|
35
|
+
integer :max_wait_time, default: 5
|
55
36
|
|
56
|
-
|
57
|
-
|
37
|
+
desc "A prefix used when generating consumer group names"
|
38
|
+
string :group_id_prefix
|
58
39
|
|
59
|
-
|
60
|
-
|
40
|
+
desc "The group id to use for a given group of consumers"
|
41
|
+
string :group_id
|
61
42
|
|
62
|
-
|
63
|
-
|
43
|
+
desc "A filename that log messages should be written to"
|
44
|
+
string :logfile
|
64
45
|
|
65
|
-
|
66
|
-
|
46
|
+
desc "The log level for the Racecar logs"
|
47
|
+
string :log_level
|
67
48
|
|
68
|
-
|
69
|
-
|
70
|
-
socket_timeout: 30,
|
49
|
+
desc "A valid SSL certificate authority"
|
50
|
+
string :ssl_ca_cert
|
71
51
|
|
72
|
-
|
73
|
-
|
74
|
-
max_wait_time: 5,
|
52
|
+
desc "The path to a valid SSL certificate authority file"
|
53
|
+
string :ssl_ca_cert_file_path
|
75
54
|
|
76
|
-
|
77
|
-
|
78
|
-
}
|
55
|
+
desc "A valid SSL client certificate"
|
56
|
+
string :ssl_client_cert
|
79
57
|
|
80
|
-
|
58
|
+
desc "A valid SSL client certificate key"
|
59
|
+
string :ssl_client_cert_key
|
81
60
|
|
82
|
-
|
83
|
-
|
84
|
-
|
61
|
+
desc "The GSSAPI principal"
|
62
|
+
string :sasl_gssapi_principal
|
63
|
+
|
64
|
+
desc "Optional GSSAPI keytab"
|
65
|
+
string :sasl_gssapi_keytab
|
66
|
+
|
67
|
+
desc "The authorization identity to use"
|
68
|
+
string :sasl_plain_authzid
|
69
|
+
|
70
|
+
desc "The username used to authenticate"
|
71
|
+
string :sasl_plain_username
|
72
|
+
|
73
|
+
desc "The password used to authenticate"
|
74
|
+
string :sasl_plain_password
|
75
|
+
|
76
|
+
desc "The file in which to store the Racecar process' PID when daemonized"
|
77
|
+
string :pidfile
|
78
|
+
|
79
|
+
desc "Run the Racecar process in the background as a daemon"
|
80
|
+
boolean :daemonize, default: false
|
81
|
+
|
82
|
+
desc "Enable Datadog metrics"
|
83
|
+
boolean :datadog_enabled, default: false
|
84
|
+
|
85
|
+
desc "The host running the Datadog agent"
|
86
|
+
string :datadog_host
|
87
|
+
|
88
|
+
desc "The port of the Datadog agent"
|
89
|
+
integer :datadog_port
|
90
|
+
|
91
|
+
desc "The namespace to use for Datadog metrics"
|
92
|
+
string :datadog_namespace
|
93
|
+
|
94
|
+
desc "Tags that should always be set on Datadog metrics"
|
95
|
+
list :datadog_tags
|
96
|
+
|
97
|
+
# The error handler must be set directly on the object.
|
98
|
+
attr_reader :error_handler
|
99
|
+
|
100
|
+
attr_accessor :subscriptions
|
101
|
+
|
102
|
+
def initialize(env: ENV)
|
103
|
+
super(env: env)
|
104
|
+
@error_handler = proc {}
|
105
|
+
@subscriptions = []
|
85
106
|
end
|
86
107
|
|
87
108
|
def inspect
|
88
|
-
|
109
|
+
self.class.variables
|
110
|
+
.map(&:name)
|
89
111
|
.map {|key| [key, get(key).inspect].join(" = ") }
|
90
112
|
.join("\n")
|
91
113
|
end
|
92
114
|
|
93
115
|
def validate!
|
94
|
-
|
95
|
-
|
96
|
-
raise ConfigError, "required configuration key `#{key}` not defined"
|
97
|
-
end
|
116
|
+
if brokers.empty?
|
117
|
+
raise ConfigError, "`brokers` must not be empty"
|
98
118
|
end
|
99
119
|
|
100
120
|
if socket_timeout <= max_wait_time
|
@@ -106,41 +126,10 @@ module Racecar
|
|
106
126
|
end
|
107
127
|
end
|
108
128
|
|
109
|
-
def load_file(path, environment)
|
110
|
-
# First, load the ERB template from disk.
|
111
|
-
template = ERB.new(File.new(path).read)
|
112
|
-
|
113
|
-
# The last argument to `safe_load` allows us to use aliasing to share
|
114
|
-
# configuration between environments.
|
115
|
-
processed = YAML.safe_load(template.result(binding), [], [], true)
|
116
|
-
|
117
|
-
data = processed.fetch(environment)
|
118
|
-
|
119
|
-
load(data)
|
120
|
-
end
|
121
|
-
|
122
|
-
def get(key)
|
123
|
-
public_send(key)
|
124
|
-
end
|
125
|
-
|
126
|
-
def set(key, value)
|
127
|
-
unless ALLOWED_KEYS.include?(key.to_s)
|
128
|
-
raise ConfigError, "unknown configuration key `#{key}`"
|
129
|
-
end
|
130
|
-
|
131
|
-
instance_variable_set("@#{key}", value)
|
132
|
-
end
|
133
|
-
|
134
|
-
def load(data)
|
135
|
-
data.each do |key, value|
|
136
|
-
set(key, value)
|
137
|
-
end
|
138
|
-
end
|
139
|
-
|
140
129
|
def load_consumer_class(consumer_class)
|
141
|
-
|
130
|
+
self.group_id = consumer_class.group_id || self.group_id
|
142
131
|
|
143
|
-
|
132
|
+
self.group_id ||= [
|
144
133
|
# Configurable and optional prefix:
|
145
134
|
group_id_prefix,
|
146
135
|
|
@@ -148,32 +137,13 @@ module Racecar
|
|
148
137
|
consumer_class.name.gsub(/[a-z][A-Z]/) {|str| str[0] << "-" << str[1] }.downcase,
|
149
138
|
].compact.join("")
|
150
139
|
|
151
|
-
|
152
|
-
|
140
|
+
self.subscriptions = consumer_class.subscriptions
|
141
|
+
self.max_wait_time = consumer_class.max_wait_time || self.max_wait_time
|
142
|
+
self.pidfile ||= "#{group_id}.pid"
|
153
143
|
end
|
154
144
|
|
155
145
|
def on_error(&handler)
|
156
146
|
@error_handler = handler
|
157
147
|
end
|
158
|
-
|
159
|
-
private
|
160
|
-
|
161
|
-
def load_env!
|
162
|
-
loader = EnvLoader.new(ENV, self)
|
163
|
-
|
164
|
-
loader.string_list(:brokers)
|
165
|
-
loader.string(:client_id)
|
166
|
-
loader.string(:group_id_prefix)
|
167
|
-
loader.string(:group_id)
|
168
|
-
loader.integer(:offset_commit_interval)
|
169
|
-
loader.integer(:offset_commit_threshold)
|
170
|
-
loader.integer(:heartbeat_interval)
|
171
|
-
loader.integer(:pause_timeout)
|
172
|
-
loader.integer(:connect_timeout)
|
173
|
-
loader.integer(:socket_timeout)
|
174
|
-
loader.integer(:max_wait_time)
|
175
|
-
|
176
|
-
loader.validate!
|
177
|
-
end
|
178
148
|
end
|
179
149
|
end
|
data/lib/racecar/ctl.rb
CHANGED
@@ -1,19 +1,62 @@
|
|
1
1
|
require "optparse"
|
2
2
|
require "racecar/rails_config_file_loader"
|
3
|
+
require "racecar/daemon"
|
3
4
|
|
4
5
|
module Racecar
|
5
6
|
class Ctl
|
6
7
|
ProduceMessage = Struct.new(:value, :key, :topic)
|
7
8
|
|
8
9
|
def self.main(args)
|
9
|
-
command = args.shift
|
10
|
+
command = args.shift
|
10
11
|
|
11
|
-
|
12
|
+
if command.nil?
|
13
|
+
puts "No command specified. Commands:"
|
14
|
+
puts " - status"
|
15
|
+
puts " - stop"
|
16
|
+
puts " - produce"
|
17
|
+
else
|
18
|
+
ctl = new(command)
|
19
|
+
|
20
|
+
if ctl.respond_to?(command)
|
21
|
+
ctl.send(command, args)
|
22
|
+
else
|
23
|
+
raise Racecar::Error, "invalid command: #{command}"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
def initialize(command)
|
29
|
+
@command = command
|
30
|
+
end
|
12
31
|
|
13
|
-
|
14
|
-
|
32
|
+
def status(args)
|
33
|
+
parse_options!(args)
|
34
|
+
|
35
|
+
pidfile = Racecar.config.pidfile
|
36
|
+
daemon = Daemon.new(pidfile)
|
37
|
+
|
38
|
+
if daemon.running?
|
39
|
+
puts "running (PID = #{daemon.pid})"
|
15
40
|
else
|
16
|
-
|
41
|
+
puts daemon.pid_status
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def stop(args)
|
46
|
+
parse_options!(args)
|
47
|
+
|
48
|
+
pidfile = Racecar.config.pidfile
|
49
|
+
daemon = Daemon.new(pidfile)
|
50
|
+
|
51
|
+
if daemon.running?
|
52
|
+
daemon.stop!
|
53
|
+
while daemon.running?
|
54
|
+
puts "Waiting for Racecar process to stop..."
|
55
|
+
sleep 5
|
56
|
+
end
|
57
|
+
puts "Racecar stopped"
|
58
|
+
else
|
59
|
+
puts "Racecar is not currently running"
|
17
60
|
end
|
18
61
|
end
|
19
62
|
|
@@ -62,5 +105,19 @@ module Racecar
|
|
62
105
|
|
63
106
|
$stderr.puts "=> Delivered message to Kafka cluster"
|
64
107
|
end
|
108
|
+
|
109
|
+
private
|
110
|
+
|
111
|
+
def parse_options!(args)
|
112
|
+
parser = OptionParser.new do |opts|
|
113
|
+
opts.banner = "Usage: racecarctl #{@command} [options]"
|
114
|
+
|
115
|
+
opts.on("--pidfile PATH", "Use the PID stored in the specified file") do |path|
|
116
|
+
Racecar.config.pidfile = File.expand_path(path)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
|
120
|
+
parser.parse!(args)
|
121
|
+
end
|
65
122
|
end
|
66
123
|
end
|
@@ -0,0 +1,97 @@
|
|
1
|
+
module Racecar
|
2
|
+
class Daemon
|
3
|
+
attr_reader :pidfile
|
4
|
+
|
5
|
+
def initialize(pidfile)
|
6
|
+
raise Racecar::Error, "No pidfile specified" if pidfile.nil?
|
7
|
+
|
8
|
+
@pidfile = pidfile
|
9
|
+
end
|
10
|
+
|
11
|
+
def pidfile?
|
12
|
+
!pidfile.nil?
|
13
|
+
end
|
14
|
+
|
15
|
+
def daemonize!
|
16
|
+
exit if fork
|
17
|
+
Process.setsid
|
18
|
+
exit if fork
|
19
|
+
Dir.chdir "/"
|
20
|
+
end
|
21
|
+
|
22
|
+
def redirect_output(logfile)
|
23
|
+
FileUtils.mkdir_p(File.dirname(logfile), mode: 0755)
|
24
|
+
FileUtils.touch(logfile)
|
25
|
+
|
26
|
+
File.chmod(0644, logfile)
|
27
|
+
|
28
|
+
$stderr.reopen(logfile, 'a')
|
29
|
+
$stdout.reopen($stderr)
|
30
|
+
$stdout.sync = $stderr.sync = true
|
31
|
+
end
|
32
|
+
|
33
|
+
def suppress_input
|
34
|
+
$stdin.reopen('/dev/null')
|
35
|
+
end
|
36
|
+
|
37
|
+
def suppress_output
|
38
|
+
$stderr.reopen('/dev/null', 'a')
|
39
|
+
$stdout.reopen($stderr)
|
40
|
+
end
|
41
|
+
|
42
|
+
def check_pid
|
43
|
+
if pidfile?
|
44
|
+
case pid_status
|
45
|
+
when :running, :not_owned
|
46
|
+
$stderr.puts "=> Racecar is already running with that PID file (#{pidfile})"
|
47
|
+
exit(1)
|
48
|
+
when :dead
|
49
|
+
File.delete(pidfile)
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
def pid
|
55
|
+
if File.exists?(pidfile)
|
56
|
+
File.read(pidfile).to_i
|
57
|
+
else
|
58
|
+
nil
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def running?
|
63
|
+
pid_status == :running || pid_status == :not_owned
|
64
|
+
end
|
65
|
+
|
66
|
+
def stop!
|
67
|
+
Process.kill("TERM", pid)
|
68
|
+
end
|
69
|
+
|
70
|
+
def pid_status
|
71
|
+
return :exited if pid.nil?
|
72
|
+
return :dead if pid == 0
|
73
|
+
|
74
|
+
# This will raise Errno::ESRCH if the process doesn't exist.
|
75
|
+
Process.kill(0, pid)
|
76
|
+
|
77
|
+
:running
|
78
|
+
rescue Errno::ESRCH
|
79
|
+
:dead
|
80
|
+
rescue Errno::EPERM
|
81
|
+
:not_owned
|
82
|
+
end
|
83
|
+
|
84
|
+
def write_pid
|
85
|
+
File.open(pidfile, ::File::CREAT | ::File::EXCL | ::File::WRONLY) do |f|
|
86
|
+
f.write(Process.pid.to_s)
|
87
|
+
end
|
88
|
+
|
89
|
+
at_exit do
|
90
|
+
File.delete(pidfile) if File.exists?(pidfile)
|
91
|
+
end
|
92
|
+
rescue Errno::EEXIST
|
93
|
+
check_pid
|
94
|
+
retry
|
95
|
+
end
|
96
|
+
end
|
97
|
+
end
|
@@ -5,12 +5,18 @@ module Racecar
|
|
5
5
|
|
6
6
|
begin
|
7
7
|
require "rails"
|
8
|
+
rescue LoadError
|
9
|
+
# Not a Rails application.
|
10
|
+
end
|
8
11
|
|
12
|
+
if defined?(Rails)
|
9
13
|
$stderr.puts "=> Detected Rails, booting application..."
|
10
14
|
|
11
15
|
require "./config/environment"
|
12
16
|
|
13
|
-
|
17
|
+
if (Rails.root + config_file).readable?
|
18
|
+
Racecar.config.load_file(config_file, Rails.env)
|
19
|
+
end
|
14
20
|
|
15
21
|
# In development, write Rails logs to STDOUT. This mirrors what e.g.
|
16
22
|
# Unicorn does.
|
@@ -20,8 +26,6 @@ module Racecar
|
|
20
26
|
console.level = Rails.logger.level
|
21
27
|
Rails.logger.extend(ActiveSupport::Logger.broadcast(console))
|
22
28
|
end
|
23
|
-
rescue LoadError
|
24
|
-
# Not a Rails application.
|
25
29
|
end
|
26
30
|
end
|
27
31
|
end
|
data/lib/racecar/runner.rb
CHANGED
@@ -80,14 +80,18 @@ module Racecar
|
|
80
80
|
|
81
81
|
# Restart the consumer loop.
|
82
82
|
retry
|
83
|
+
rescue Kafka::InvalidSessionTimeout
|
84
|
+
raise ConfigError, "`session_timeout` is set either too high or too low"
|
83
85
|
rescue Kafka::Error => e
|
84
86
|
error = "#{e.class}: #{e.message}\n" + e.backtrace.join("\n")
|
85
87
|
@logger.error "Consumer thread crashed: #{error}"
|
86
88
|
|
87
89
|
config.error_handler.call(e)
|
88
|
-
end
|
89
90
|
|
90
|
-
|
91
|
+
raise
|
92
|
+
else
|
93
|
+
@logger.info "Gracefully shutting down"
|
94
|
+
end
|
91
95
|
end
|
92
96
|
end
|
93
97
|
end
|
data/lib/racecar/version.rb
CHANGED
data/racecar.gemspec
CHANGED
@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
|
|
20
20
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
21
21
|
spec.require_paths = ["lib"]
|
22
22
|
|
23
|
+
spec.add_runtime_dependency "king_konf", "~> 0.1.6"
|
23
24
|
spec.add_runtime_dependency "ruby-kafka", "~> 0.4"
|
24
25
|
|
25
26
|
spec.add_development_dependency "bundler", "~> 1.13"
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: racecar
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel Schierbeck
|
@@ -9,8 +9,22 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: exe
|
11
11
|
cert_chain: []
|
12
|
-
date: 2017-
|
12
|
+
date: 2017-10-16 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: king_konf
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
requirements:
|
18
|
+
- - "~>"
|
19
|
+
- !ruby/object:Gem::Version
|
20
|
+
version: 0.1.6
|
21
|
+
type: :runtime
|
22
|
+
prerelease: false
|
23
|
+
version_requirements: !ruby/object:Gem::Requirement
|
24
|
+
requirements:
|
25
|
+
- - "~>"
|
26
|
+
- !ruby/object:Gem::Version
|
27
|
+
version: 0.1.6
|
14
28
|
- !ruby/object:Gem::Dependency
|
15
29
|
name: ruby-kafka
|
16
30
|
requirement: !ruby/object:Gem::Requirement
|
@@ -79,6 +93,7 @@ extra_rdoc_files: []
|
|
79
93
|
files:
|
80
94
|
- ".gitignore"
|
81
95
|
- ".rspec"
|
96
|
+
- CHANGELOG.md
|
82
97
|
- Gemfile
|
83
98
|
- LICENSE.txt
|
84
99
|
- Procfile
|
@@ -99,7 +114,7 @@ files:
|
|
99
114
|
- lib/racecar/config.rb
|
100
115
|
- lib/racecar/consumer.rb
|
101
116
|
- lib/racecar/ctl.rb
|
102
|
-
- lib/racecar/
|
117
|
+
- lib/racecar/daemon.rb
|
103
118
|
- lib/racecar/rails_config_file_loader.rb
|
104
119
|
- lib/racecar/runner.rb
|
105
120
|
- lib/racecar/version.rb
|
@@ -124,7 +139,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
124
139
|
version: '0'
|
125
140
|
requirements: []
|
126
141
|
rubyforge_project:
|
127
|
-
rubygems_version: 2.
|
142
|
+
rubygems_version: 2.6.11
|
128
143
|
signing_key:
|
129
144
|
specification_version: 4
|
130
145
|
summary: A framework for running Kafka consumers
|
data/lib/racecar/env_loader.rb
DELETED
@@ -1,49 +0,0 @@
|
|
1
|
-
module Racecar
|
2
|
-
class EnvLoader
|
3
|
-
def initialize(env, config)
|
4
|
-
@env = env
|
5
|
-
@config = config
|
6
|
-
@loaded_keys = []
|
7
|
-
end
|
8
|
-
|
9
|
-
def string(name)
|
10
|
-
set(name) {|value| value }
|
11
|
-
end
|
12
|
-
|
13
|
-
def integer(name)
|
14
|
-
set(name) do |value|
|
15
|
-
begin
|
16
|
-
Integer(value)
|
17
|
-
rescue ArgumentError
|
18
|
-
raise ConfigError, "#{value.inspect} is not an integer"
|
19
|
-
end
|
20
|
-
end
|
21
|
-
end
|
22
|
-
|
23
|
-
def string_list(name)
|
24
|
-
set(name) {|value| value.split(",") }
|
25
|
-
end
|
26
|
-
|
27
|
-
def validate!
|
28
|
-
# Make sure the user hasn't made a typo and added a key we don't know
|
29
|
-
# about.
|
30
|
-
@env.keys.grep(/^RACECAR_/).each do |key|
|
31
|
-
unless @loaded_keys.include?(key)
|
32
|
-
raise ConfigError, "unknown config variable #{key}"
|
33
|
-
end
|
34
|
-
end
|
35
|
-
end
|
36
|
-
|
37
|
-
private
|
38
|
-
|
39
|
-
def set(name)
|
40
|
-
key = "RACECAR_#{name.upcase}"
|
41
|
-
|
42
|
-
if @env.key?(key)
|
43
|
-
value = yield @env.fetch(key)
|
44
|
-
@config.set(name, value)
|
45
|
-
@loaded_keys << key
|
46
|
-
end
|
47
|
-
end
|
48
|
-
end
|
49
|
-
end
|