synapse 0.9.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +103 -8
- data/bin/synapse +1 -1
- data/lib/synapse/haproxy.rb +37 -13
- data/lib/synapse/service_watcher.rb +6 -4
- data/lib/synapse/service_watcher/base.rb +14 -0
- data/lib/synapse/service_watcher/dns.rb +12 -6
- data/lib/synapse/service_watcher/docker.rb +2 -2
- data/lib/synapse/service_watcher/zookeeper.rb +2 -2
- data/lib/synapse/service_watcher/zookeeper_dns.rb +232 -0
- data/lib/synapse/version.rb +1 -1
- data/spec/lib/synapse/service_watcher_base_spec.rb +10 -1
- data/spec/spec_helper.rb +2 -2
- data/spec/support/{config.rb → configuration.rb} +1 -1
- metadata +7 -6
data/README.md
CHANGED
@@ -1,4 +1,5 @@
|
|
1
1
|
[](https://travis-ci.org/airbnb/synapse)
|
2
|
+
[](http://inch-pages.github.io/github/airbnb/synapse)
|
2
3
|
|
3
4
|
# Synapse #
|
4
5
|
|
@@ -20,7 +21,7 @@ One solution to this problem is a discovery service, like [Apache Zookeeper](htt
|
|
20
21
|
However, Zookeeper and similar services have their own problems:
|
21
22
|
|
22
23
|
* Service discovery is embedded in all of your apps; often, integration is not simple
|
23
|
-
* The discovery layer itself
|
24
|
+
* The discovery layer itself is subject to failure
|
24
25
|
* Requires additional servers/instances
|
25
26
|
|
26
27
|
Synapse solves these difficulties in a simple and fault-tolerant way.
|
@@ -39,7 +40,7 @@ It is easy to write your own watchers for your use case, and we encourage submit
|
|
39
40
|
|
40
41
|
## Example Migration ##
|
41
42
|
|
42
|
-
|
43
|
+
Let's suppose your rails application depends on a Postgres database instance.
|
43
44
|
The database.yaml file has the DB host and port hardcoded:
|
44
45
|
|
45
46
|
```yaml
|
@@ -126,8 +127,9 @@ The name is just a human-readable string; it will be used in logs and notificati
|
|
126
127
|
Each value in the services hash is also a hash, and should contain the following keys:
|
127
128
|
|
128
129
|
* `discovery`: how synapse will discover hosts providing this service (see below)
|
129
|
-
* `default_servers`: the list of default servers providing this service; synapse uses these if
|
130
|
+
* `default_servers`: the list of default servers providing this service; synapse uses these if no others can be discovered
|
130
131
|
* `haproxy`: how will the haproxy section for this service be configured
|
132
|
+
* `shared_frontend`: optional: haproxy configuration directives for a shared http frontend (see below)
|
131
133
|
|
132
134
|
#### Service Discovery ####
|
133
135
|
|
@@ -136,7 +138,7 @@ Put these into the `discovery` section of the service hash, with these options:
|
|
136
138
|
|
137
139
|
##### Stub #####
|
138
140
|
|
139
|
-
The stub watcher
|
141
|
+
The stub watcher is useful in situations where you only want to use the servers in the `default_servers` list.
|
140
142
|
It has only one option:
|
141
143
|
|
142
144
|
* `method`: stub
|
@@ -156,7 +158,7 @@ We assume that the data contains a hostname and a port for service servers.
|
|
156
158
|
|
157
159
|
##### Docker #####
|
158
160
|
|
159
|
-
This watcher retrieves a list of [docker](http://www.docker.io/) containers via docker's [HTTP API](http://docs.docker.io/en/latest/api/docker_remote_api/).
|
161
|
+
This watcher retrieves a list of [docker](http://www.docker.io/) containers via docker's [HTTP API](http://docs.docker.io/en/latest/reference/api/docker_remote_api/).
|
160
162
|
It takes the following options:
|
161
163
|
|
162
164
|
* `method`: docker
|
@@ -176,11 +178,13 @@ Each hash in that section has the following options:
|
|
176
178
|
|
177
179
|
The `default_servers` list is used only when service discovery returns no servers.
|
178
180
|
In that case, the service proxy will be created with the servers listed here.
|
179
|
-
If you do not list any default servers, no proxy will be created.
|
181
|
+
If you do not list any default servers, no proxy will be created. The
|
182
|
+
`default_servers` will also be used in addition to discovered servers if the
|
183
|
+
`keep_default_servers` option is set.
|
180
184
|
|
181
185
|
#### The `haproxy` Section ####
|
182
186
|
|
183
|
-
This section is
|
187
|
+
This section is its own hash, which should contain the following keys:
|
184
188
|
|
185
189
|
* `port`: the port (on localhost) where HAProxy will listen for connections to the service.
|
186
190
|
* `server_port_override`: the port that discovered servers listen on; you should specify this if your discovery mechanism only discovers names or addresses (like the DNS watcher). If the discovery method discovers a port along with hostnames (like the zookeeper watcher) this option may be left out, but will be used in preference if given.
|
@@ -199,10 +203,101 @@ The `haproxy` section of the config file has the following options:
|
|
199
203
|
* `do_reloads`: whether or not Synapse will reload HAProxy (default to `true`)
|
200
204
|
* `global`: options listed here will be written into the `global` section of the HAProxy config
|
201
205
|
* `defaults`: options listed here will be written into the `defaults` section of the HAProxy config
|
202
|
-
* `bind_address`: force HAProxy to listen
|
206
|
+
* `bind_address`: force HAProxy to listen on this address (default is localhost)
|
207
|
+
* `shared_fronted`: (OPTIONAL) additional lines passed to the HAProxy config used to configure a shared HTTP frontend (see below)
|
203
208
|
|
204
209
|
Note that a non-default `bind_address` can be dangerous: it is up to you to ensure that HAProxy will not attempt to bind an address:port combination that is not already in use by one of your services.
|
205
210
|
|
211
|
+
### HAProxy shared HTTP Frontend ###
|
212
|
+
|
213
|
+
For HTTP-only services, it is not always necessary or desirable to dedicate a TCP port per service, since HAProxy can route traffic based on host headers.
|
214
|
+
To support this, the optional `shared_fronted` section can be added to both the `haproxy` section and each indvidual service definition: synapse will concatenate them all into a single frontend section in the generated haproxy.cfg file.
|
215
|
+
Note that synapse does not assemble the routing ACLs for you: you have to do that yourself based on your needs.
|
216
|
+
This is probably most useful in combination with the `service_conf_dir` directive in a case where the individual service config files are being distributed by a configuration manager such as puppet or chef, or bundled into service packages.
|
217
|
+
For example:
|
218
|
+
|
219
|
+
```yaml
|
220
|
+
{
|
221
|
+
"haproxy": {
|
222
|
+
"shared_frontend": [
|
223
|
+
"bind 127.0.0.1:8081"
|
224
|
+
],
|
225
|
+
"reload_command": "service haproxy reload",
|
226
|
+
"config_file_path": "/etc/haproxy/haproxy.cfg",
|
227
|
+
"socket_file_path": "/var/run/haproxy.sock",
|
228
|
+
"global": [
|
229
|
+
"daemon",
|
230
|
+
"user haproxy",
|
231
|
+
"group haproxy",
|
232
|
+
"maxconn 4096",
|
233
|
+
"log 127.0.0.1 local2 notice",
|
234
|
+
"stats socket /var/run/haproxy.sock"
|
235
|
+
],
|
236
|
+
"defaults": [
|
237
|
+
"log global",
|
238
|
+
"balance roundrobin"
|
239
|
+
]
|
240
|
+
},
|
241
|
+
"services": {
|
242
|
+
"service1": {
|
243
|
+
"discovery": {
|
244
|
+
"method": "zookeeper",
|
245
|
+
"path": "/nerve/services/service1",
|
246
|
+
"hosts": [ "0.zookeeper.example.com:2181" ]
|
247
|
+
},
|
248
|
+
"haproxy": {
|
249
|
+
"server_options": "check inter 2s rise 3 fall 2",
|
250
|
+
"shared_frontend": [
|
251
|
+
"acl is_service1 hdr_dom(host) -i service1.lb.example.com",
|
252
|
+
"use_backend service1 if is_service1"
|
253
|
+
],
|
254
|
+
"backend": [
|
255
|
+
"mode http"
|
256
|
+
]
|
257
|
+
}
|
258
|
+
},
|
259
|
+
"service2": {
|
260
|
+
"discovery": {
|
261
|
+
"method": "zookeeper",
|
262
|
+
"path": "/nerve/services/service2",
|
263
|
+
"hosts": [ "0.zookeeper.example.com:2181" ]
|
264
|
+
},
|
265
|
+
"haproxy": {
|
266
|
+
"server_options": "check inter 2s rise 3 fall 2",
|
267
|
+
"shared_frontend": [
|
268
|
+
"acl is_service1 hdr_dom(host) -i service2.lb.example.com",
|
269
|
+
"use_backend service2 if is_service2"
|
270
|
+
],
|
271
|
+
"backend": [
|
272
|
+
"mode http"
|
273
|
+
]
|
274
|
+
}
|
275
|
+
}
|
276
|
+
}
|
277
|
+
}
|
278
|
+
```
|
279
|
+
|
280
|
+
This would produce an haproxy.cfg much like the following:
|
281
|
+
|
282
|
+
```
|
283
|
+
backend service1
|
284
|
+
mode http
|
285
|
+
server server1.example.net:80 server1.example.net:80 check inter 2s rise 3 fall 2
|
286
|
+
|
287
|
+
backend service2
|
288
|
+
mode http
|
289
|
+
server server2.example.net:80 server2.example.net:80 check inter 2s rise 3 fall 2
|
290
|
+
|
291
|
+
frontend shared-frontend
|
292
|
+
bind 127.0.0.1:8081
|
293
|
+
acl is_service1 hdr_dom(host) -i service1.lb
|
294
|
+
use_backend service1 if is_service1
|
295
|
+
acl is_service2 hdr_dom(host) -i service2.lb
|
296
|
+
use_backend service2 if is_service2
|
297
|
+
```
|
298
|
+
|
299
|
+
Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need their own dedicated ports.
|
300
|
+
|
206
301
|
## Contributing
|
207
302
|
|
208
303
|
1. Fork it
|
data/bin/synapse
CHANGED
@@ -37,7 +37,7 @@ def parseconfig(filename)
|
|
37
37
|
raise ArgumentError, "config file does not exist:\n#{e.inspect}"
|
38
38
|
rescue Errno::EACCES => e
|
39
39
|
raise ArgumentError, "could not open config file:\n#{e.inspect}"
|
40
|
-
rescue YAML::
|
40
|
+
rescue YAML::SyntaxError => e
|
41
41
|
raise "config file #{filename} is not yaml:\n#{e.inspect}"
|
42
42
|
end
|
43
43
|
return c.to_ruby
|
data/lib/synapse/haproxy.rb
CHANGED
@@ -543,18 +543,35 @@ module Synapse
|
|
543
543
|
# generates a new config based on the state of the watchers
|
544
544
|
def generate_config(watchers)
|
545
545
|
new_config = generate_base_config
|
546
|
+
shared_frontend_lines = generate_shared_frontend
|
546
547
|
|
547
548
|
watchers.each do |watcher|
|
548
549
|
@watcher_configs[watcher.name] ||= parse_watcher_config(watcher)
|
549
|
-
|
550
550
|
new_config << generate_frontend_stanza(watcher, @watcher_configs[watcher.name]['frontend'])
|
551
551
|
new_config << generate_backend_stanza(watcher, @watcher_configs[watcher.name]['backend'])
|
552
|
+
if watcher.haproxy.include?('shared_frontend')
|
553
|
+
if @opts['shared_frontend'] == nil
|
554
|
+
log.warn "synapse: service #{watcher.name} contains a shared frontend section but the base config does not! skipping."
|
555
|
+
else
|
556
|
+
shared_frontend_lines << validate_haproxy_stanza(watcher.haproxy['shared_frontend'].map{|l| "\t#{l}"}, "frontend", "shared frontend section for #{watcher.name}")
|
557
|
+
end
|
558
|
+
end
|
552
559
|
end
|
560
|
+
new_config << shared_frontend_lines.flatten if shared_frontend_lines
|
553
561
|
|
554
562
|
log.debug "synapse: new haproxy config: #{new_config}"
|
555
563
|
return new_config.flatten.join("\n")
|
556
564
|
end
|
557
565
|
|
566
|
+
# pull out the shared frontend section if any
|
567
|
+
def generate_shared_frontend
|
568
|
+
return nil unless @opts.include?('shared_frontend')
|
569
|
+
log.debug "synapse: found a shared frontend section"
|
570
|
+
shared_frontend_lines = ["\nfrontend shared-frontend"]
|
571
|
+
shared_frontend_lines << validate_haproxy_stanza(@opts['shared_frontend'].map{|l| "\t#{l}"}, "frontend", "shared frontend")
|
572
|
+
return shared_frontend_lines
|
573
|
+
end
|
574
|
+
|
558
575
|
# generates the global and defaults sections of the config file
|
559
576
|
def generate_base_config
|
560
577
|
base_config = ["# auto-generated by synapse at #{Time.now}\n"]
|
@@ -593,20 +610,24 @@ module Synapse
|
|
593
610
|
})
|
594
611
|
|
595
612
|
# pick only those fields that are valid and warn about the invalid ones
|
596
|
-
config[section].
|
597
|
-
parsed_setting = setting.strip.gsub(/\s+/, ' ').downcase
|
598
|
-
if @@section_fields[section].any? {|field| parsed_setting.start_with?(field)}
|
599
|
-
true
|
600
|
-
else
|
601
|
-
log.warn "synapse: service #{watcher.name} contains invalid #{section} setting: '#{setting}'"
|
602
|
-
false
|
603
|
-
end
|
604
|
-
}
|
613
|
+
config[section] = validate_haproxy_stanza(config[section], section, watcher.name)
|
605
614
|
end
|
606
615
|
|
607
616
|
return config
|
608
617
|
end
|
609
618
|
|
619
|
+
def validate_haproxy_stanza(stanza, stanza_type, service_name)
|
620
|
+
return stanza.select {|setting|
|
621
|
+
parsed_setting = setting.strip.gsub(/\s+/, ' ').downcase
|
622
|
+
if @@section_fields[stanza_type].any? {|field| parsed_setting.start_with?(field)}
|
623
|
+
true
|
624
|
+
else
|
625
|
+
log.warn "synapse: service #{service_name} contains invalid #{stanza_type} setting: '#{setting}', discarding"
|
626
|
+
false
|
627
|
+
end
|
628
|
+
}
|
629
|
+
end
|
630
|
+
|
610
631
|
# generates an individual stanza for a particular watcher
|
611
632
|
def generate_frontend_stanza(watcher, config)
|
612
633
|
unless watcher.haproxy.has_key?("port")
|
@@ -642,7 +663,7 @@ module Synapse
|
|
642
663
|
# first, get a list of existing servers for various backends
|
643
664
|
begin
|
644
665
|
s = UNIXSocket.new(@opts['socket_file_path'])
|
645
|
-
s.write(
|
666
|
+
s.write("show stat\n")
|
646
667
|
info = s.read()
|
647
668
|
rescue StandardError => e
|
648
669
|
log.warn "synapse: unhandled error reading stats socket: #{e.inspect}"
|
@@ -690,9 +711,9 @@ module Synapse
|
|
690
711
|
cur_backends.each do |section, backends|
|
691
712
|
backends.each do |backend|
|
692
713
|
if enabled_backends[section].include? backend
|
693
|
-
command = "enable server #{section}/#{backend}
|
714
|
+
command = "enable server #{section}/#{backend}\n"
|
694
715
|
else
|
695
|
-
command = "disable server #{section}/#{backend}
|
716
|
+
command = "disable server #{section}/#{backend}\n"
|
696
717
|
end
|
697
718
|
|
698
719
|
# actually write the command to the socket
|
@@ -713,6 +734,8 @@ module Synapse
|
|
713
734
|
end
|
714
735
|
end
|
715
736
|
end
|
737
|
+
|
738
|
+
log.info "synapse: reconfigured haproxy"
|
716
739
|
end
|
717
740
|
|
718
741
|
# writes the config
|
@@ -741,6 +764,7 @@ module Synapse
|
|
741
764
|
# do the actual restart
|
742
765
|
res = `#{opts['reload_command']}`.chomp
|
743
766
|
raise "failed to reload haproxy via #{opts['reload_command']}: #{res}" unless $?.success?
|
767
|
+
log.info "synapse: restarted haproxy"
|
744
768
|
|
745
769
|
@last_restart = Time.now()
|
746
770
|
@restart_required = false
|
@@ -3,16 +3,18 @@ require "synapse/service_watcher/zookeeper"
|
|
3
3
|
require "synapse/service_watcher/ec2tag"
|
4
4
|
require "synapse/service_watcher/dns"
|
5
5
|
require "synapse/service_watcher/docker"
|
6
|
+
require "synapse/service_watcher/zookeeper_dns"
|
6
7
|
|
7
8
|
module Synapse
|
8
9
|
class ServiceWatcher
|
9
10
|
|
10
11
|
@watchers = {
|
11
|
-
'base'=>BaseWatcher,
|
12
|
-
'zookeeper'=>ZookeeperWatcher,
|
13
|
-
'ec2tag'=>EC2Watcher,
|
12
|
+
'base' => BaseWatcher,
|
13
|
+
'zookeeper' => ZookeeperWatcher,
|
14
|
+
'ec2tag' => EC2Watcher,
|
14
15
|
'dns' => DnsWatcher,
|
15
|
-
'docker' => DockerWatcher
|
16
|
+
'docker' => DockerWatcher,
|
17
|
+
'zookeeper_dns' => ZookeeperDnsWatcher,
|
16
18
|
}
|
17
19
|
|
18
20
|
# the method which actually dispatches watcher creation requests
|
@@ -40,6 +40,8 @@ module Synapse
|
|
40
40
|
@default_servers = opts['default_servers'] || []
|
41
41
|
@backends = @default_servers
|
42
42
|
|
43
|
+
@keep_default_servers = opts['keep_default_servers'] || false
|
44
|
+
|
43
45
|
# set a flag used to tell the watchers to exit
|
44
46
|
# this is not used in every watcher
|
45
47
|
@should_exit = false
|
@@ -91,5 +93,17 @@ module Synapse
|
|
91
93
|
|
92
94
|
log.warn "synapse: warning: a stub watcher with no default servers is pretty useless" if @default_servers.empty?
|
93
95
|
end
|
96
|
+
|
97
|
+
def set_backends(new_backends)
|
98
|
+
if @keep_default_servers
|
99
|
+
@backends = @default_servers + new_backends
|
100
|
+
else
|
101
|
+
@backends = new_backends
|
102
|
+
end
|
103
|
+
end
|
104
|
+
|
105
|
+
def reconfigure!
|
106
|
+
@synapse.reconfigure!
|
107
|
+
end
|
94
108
|
end
|
95
109
|
end
|
@@ -15,7 +15,11 @@ module Synapse
|
|
15
15
|
end
|
16
16
|
|
17
17
|
def ping?
|
18
|
-
!(resolver.getaddresses('airbnb.com').empty?)
|
18
|
+
@watcher.alive? && !(resolver.getaddresses('airbnb.com').empty?)
|
19
|
+
end
|
20
|
+
|
21
|
+
def discovery_servers
|
22
|
+
@discovery['servers']
|
19
23
|
end
|
20
24
|
|
21
25
|
private
|
@@ -23,7 +27,7 @@ module Synapse
|
|
23
27
|
raise ArgumentError, "invalid discovery method #{@discovery['method']}" \
|
24
28
|
unless @discovery['method'] == 'dns'
|
25
29
|
raise ArgumentError, "a non-empty list of servers is required" \
|
26
|
-
if
|
30
|
+
if discovery_servers.empty?
|
27
31
|
end
|
28
32
|
|
29
33
|
def watch
|
@@ -57,7 +61,7 @@ module Synapse
|
|
57
61
|
|
58
62
|
def resolve_servers
|
59
63
|
resolver.tap do |dns|
|
60
|
-
resolution =
|
64
|
+
resolution = discovery_servers.map do |server|
|
61
65
|
addresses = dns.getaddresses(server['host']).map(&:to_s)
|
62
66
|
[server, addresses.sort]
|
63
67
|
end
|
@@ -79,7 +83,8 @@ module Synapse
|
|
79
83
|
addresses.map do |address|
|
80
84
|
{
|
81
85
|
'host' => address,
|
82
|
-
'port' => server['port']
|
86
|
+
'port' => server['port'],
|
87
|
+
'name' => server['name'],
|
83
88
|
}
|
84
89
|
end
|
85
90
|
end
|
@@ -95,9 +100,10 @@ module Synapse
|
|
95
100
|
end
|
96
101
|
else
|
97
102
|
log.info "synapse: discovered #{new_backends.length} backends for service #{@name}"
|
98
|
-
|
103
|
+
set_backends(new_backends)
|
99
104
|
end
|
100
|
-
|
105
|
+
|
106
|
+
reconfigure!
|
101
107
|
end
|
102
108
|
end
|
103
109
|
end
|
@@ -111,9 +111,9 @@ module Synapse
|
|
111
111
|
end
|
112
112
|
else
|
113
113
|
log.info "synapse: discovered #{new_backends.length} backends for service #{@name}"
|
114
|
-
|
114
|
+
set_backends(new_backends)
|
115
115
|
end
|
116
|
-
|
116
|
+
reconfigure!
|
117
117
|
end
|
118
118
|
|
119
119
|
end
|
@@ -88,7 +88,7 @@ module Synapse
|
|
88
88
|
end
|
89
89
|
else
|
90
90
|
log.info "synapse: discovered #{new_backends.length} backends for service #{@name}"
|
91
|
-
|
91
|
+
set_backends(new_backends)
|
92
92
|
end
|
93
93
|
end
|
94
94
|
|
@@ -108,7 +108,7 @@ module Synapse
|
|
108
108
|
# Rediscover
|
109
109
|
discover
|
110
110
|
# send a message to calling class to reconfigure
|
111
|
-
|
111
|
+
reconfigure!
|
112
112
|
end
|
113
113
|
end
|
114
114
|
|
@@ -0,0 +1,232 @@
|
|
1
|
+
require 'synapse/service_watcher/base'
|
2
|
+
require 'synapse/service_watcher/dns'
|
3
|
+
require 'synapse/service_watcher/zookeeper'
|
4
|
+
|
5
|
+
require 'thread'
|
6
|
+
|
7
|
+
# Watcher for watching Zookeeper for entries containing DNS names that are
|
8
|
+
# continuously resolved to IP Addresses. The use case for this watcher is to
|
9
|
+
# allow services that are addressed by DNS to be reconfigured via Zookeeper
|
10
|
+
# instead of an update of the synapse config.
|
11
|
+
#
|
12
|
+
# The implementation builds on top of the existing DNS and Zookeeper watchers.
|
13
|
+
# This watcher creates a thread to manage the lifecycle of the DNS and
|
14
|
+
# Zookeeper watchers. This thread also publishes messages on a queue to
|
15
|
+
# indicate that DNS should be re-resolved (after the check interval) or that
|
16
|
+
# the DNS watcher should be shut down. The Zookeeper watcher waits for changes
|
17
|
+
# in backends from zookeeper and publishes those changes on an internal queue
|
18
|
+
# consumed by the DNS watcher. The DNS watcher blocks on this queue waiting
|
19
|
+
# for messages indicating that new servers are available, the check interval
|
20
|
+
# has passed (triggering a re-resolve), or that the watcher should shut down.
|
21
|
+
# The DNS watcher is responsible for the actual reconfiguring of backends.
|
22
|
+
module Synapse
|
23
|
+
class ZookeeperDnsWatcher < BaseWatcher
|
24
|
+
|
25
|
+
# Valid messages that can be passed through the internal message queue
|
26
|
+
module Messages
|
27
|
+
class InvalidMessageError < RuntimeError; end
|
28
|
+
|
29
|
+
# Indicates new servers identified by DNS names to be resolved. This is
|
30
|
+
# sent from Zookeeper on events that modify the ZK node. The payload is
|
31
|
+
# an array of hashes containing {'host', 'port', 'name'}
|
32
|
+
class NewServers < Struct.new(:servers); end
|
33
|
+
|
34
|
+
# Indicates that DNS should be re-resolved. This is sent by the
|
35
|
+
# ZookeeperDnsWatcher thread every check_interval seconds to cause a
|
36
|
+
# refresh of the IP addresses.
|
37
|
+
class CheckInterval; end
|
38
|
+
|
39
|
+
# Indicates that the DNS watcher should shut down. This is sent when
|
40
|
+
# stop is called.
|
41
|
+
class StopWatcher; end
|
42
|
+
|
43
|
+
# Saved instances of message types with contents that cannot vary. This
|
44
|
+
# reduces object allocation.
|
45
|
+
STOP_WATCHER_MESSAGE = StopWatcher.new
|
46
|
+
CHECK_INTERVAL_MESSAGE = CheckInterval.new
|
47
|
+
end
|
48
|
+
|
49
|
+
class Dns < Synapse::DnsWatcher
|
50
|
+
|
51
|
+
# Overrides the discovery_servers method on the parent class
|
52
|
+
attr_accessor :discovery_servers
|
53
|
+
|
54
|
+
def initialize(opts={}, synapse, message_queue)
|
55
|
+
@message_queue = message_queue
|
56
|
+
|
57
|
+
super(opts, synapse)
|
58
|
+
end
|
59
|
+
|
60
|
+
def stop
|
61
|
+
@message_queue.push(Messages::STOP_WATCHER_MESSAGE)
|
62
|
+
end
|
63
|
+
|
64
|
+
def watch
|
65
|
+
last_resolution = nil
|
66
|
+
while true
|
67
|
+
# Blocks on message queue, the message will be a signal to stop
|
68
|
+
# watching, to check a new set of servers from ZK, or to re-resolve
|
69
|
+
# the DNS (triggered every check_interval seconds)
|
70
|
+
message = @message_queue.pop
|
71
|
+
|
72
|
+
log.debug "synapse: received message #{message.inspect}"
|
73
|
+
|
74
|
+
case message
|
75
|
+
when Messages::StopWatcher
|
76
|
+
break
|
77
|
+
when Messages::NewServers
|
78
|
+
self.discovery_servers = message.servers
|
79
|
+
when Messages::CheckInterval
|
80
|
+
# Proceed to re-resolve the DNS
|
81
|
+
else
|
82
|
+
raise Messages::InvalidMessageError,
|
83
|
+
"Received unrecognized message: #{message.inspect}"
|
84
|
+
end
|
85
|
+
|
86
|
+
# Empty servers means we haven't heard back from ZK yet or ZK is
|
87
|
+
# empty. This should only occur if we don't get results from ZK
|
88
|
+
# within check_interval seconds or if ZK is empty.
|
89
|
+
if self.discovery_servers.nil? || self.discovery_servers.empty?
|
90
|
+
log.warn "synapse: no backends for service #{@name}"
|
91
|
+
else
|
92
|
+
# Resolve DNS names with the nameserver
|
93
|
+
current_resolution = resolve_servers
|
94
|
+
unless last_resolution == current_resolution
|
95
|
+
last_resolution = current_resolution
|
96
|
+
configure_backends(last_resolution)
|
97
|
+
end
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
102
|
+
private
|
103
|
+
|
104
|
+
# Validation is skipped as it has already occurred in the parent watcher
|
105
|
+
def validate_discovery_opts
|
106
|
+
end
|
107
|
+
end
|
108
|
+
|
109
|
+
class Zookeeper < Synapse::ZookeeperWatcher
|
110
|
+
def initialize(opts={}, synapse, message_queue)
|
111
|
+
super(opts, synapse)
|
112
|
+
|
113
|
+
@message_queue = message_queue
|
114
|
+
end
|
115
|
+
|
116
|
+
# Overrides reconfigure! to cause the new list of servers to be messaged
|
117
|
+
# to the DNS watcher rather than invoking a synapse reconfigure directly
|
118
|
+
def reconfigure!
|
119
|
+
# push the new backends onto the queue
|
120
|
+
@message_queue.push(Messages::NewServers.new(@backends))
|
121
|
+
end
|
122
|
+
|
123
|
+
private
|
124
|
+
|
125
|
+
# Validation is skipped as it has already occurred in the parent watcher
|
126
|
+
def validate_discovery_opts
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
def start
|
131
|
+
dns_discovery_opts = @discovery.select do |k,_|
|
132
|
+
k == 'nameserver'
|
133
|
+
end
|
134
|
+
|
135
|
+
zookeeper_discovery_opts = @discovery.select do |k,_|
|
136
|
+
k == 'hosts' || k == 'path'
|
137
|
+
end
|
138
|
+
|
139
|
+
@check_interval = @discovery['check_interval'] || 30.0
|
140
|
+
|
141
|
+
@message_queue = Queue.new
|
142
|
+
|
143
|
+
@dns = Dns.new(
|
144
|
+
mk_child_watcher_opts(dns_discovery_opts),
|
145
|
+
@synapse,
|
146
|
+
@message_queue
|
147
|
+
)
|
148
|
+
|
149
|
+
@zk = Zookeeper.new(
|
150
|
+
mk_child_watcher_opts(zookeeper_discovery_opts),
|
151
|
+
@synapse,
|
152
|
+
@message_queue
|
153
|
+
)
|
154
|
+
|
155
|
+
@zk.start
|
156
|
+
@dns.start
|
157
|
+
|
158
|
+
@watcher = Thread.new do
|
159
|
+
until @should_exit
|
160
|
+
# Trigger a DNS resolve every @check_interval seconds
|
161
|
+
sleep @check_interval
|
162
|
+
|
163
|
+
# Only trigger the resolve if the queue is empty, every other message
|
164
|
+
# on the queue would either cause a resolve or stop the watcher
|
165
|
+
if @message_queue.empty?
|
166
|
+
@message_queue.push(Messages::CHECK_INTERVAL_MESSAGE)
|
167
|
+
end
|
168
|
+
|
169
|
+
end
|
170
|
+
log.info "synapse: zookeeper_dns watcher exited successfully"
|
171
|
+
end
|
172
|
+
end
|
173
|
+
|
174
|
+
def ping?
|
175
|
+
@watcher.alive? && @dns.ping? && @zk.ping?
|
176
|
+
end
|
177
|
+
|
178
|
+
def stop
|
179
|
+
super
|
180
|
+
|
181
|
+
@dns.stop
|
182
|
+
@zk.stop
|
183
|
+
end
|
184
|
+
|
185
|
+
def backends
|
186
|
+
@dns.backends
|
187
|
+
end
|
188
|
+
|
189
|
+
private
|
190
|
+
|
191
|
+
def validate_discovery_opts
|
192
|
+
unless @discovery['method'] == 'zookeeper_dns'
|
193
|
+
raise ArgumentError, "invalid discovery method #{@discovery['method']}"
|
194
|
+
end
|
195
|
+
|
196
|
+
unless @discovery['hosts']
|
197
|
+
raise ArgumentError, "missing or invalid zookeeper host for service #{@name}"
|
198
|
+
end
|
199
|
+
|
200
|
+
unless @discovery['path']
|
201
|
+
raise ArgumentError, "invalid zookeeper path for service #{@name}"
|
202
|
+
end
|
203
|
+
end
|
204
|
+
|
205
|
+
# Method to generate a full config for the children (Dns and Zookeeper)
|
206
|
+
# watchers
|
207
|
+
#
|
208
|
+
# Notes on passing in the default_servers:
|
209
|
+
#
|
210
|
+
# Setting the default_servers here allows the Zookeeper watcher to return
|
211
|
+
# a list of backends based on the default servers when it fails to find
|
212
|
+
# any matching servers. These are passed on as the discovered backends
|
213
|
+
# to the DNS watcher, which will then watch them as normal for DNS
|
214
|
+
# changes. The default servers can also come into play if none of the
|
215
|
+
# hostnames from Zookeeper resolve to addresses in the DNS watcher. This
|
216
|
+
# should generally result in the expected behavior, but caution should be
|
217
|
+
# taken when deciding that this is the desired behavior.
|
218
|
+
def mk_child_watcher_opts(discovery_opts)
|
219
|
+
{
|
220
|
+
'name' => @name,
|
221
|
+
'haproxy' => @haproxy,
|
222
|
+
'discovery' => discovery_opts,
|
223
|
+
'default_servers' => @default_servers,
|
224
|
+
}
|
225
|
+
end
|
226
|
+
|
227
|
+
# Override reconfigure! as this class should not explicitly reconfigure
|
228
|
+
# synapse
|
229
|
+
def reconfigure!
|
230
|
+
end
|
231
|
+
end
|
232
|
+
end
|
data/lib/synapse/version.rb
CHANGED
@@ -41,6 +41,15 @@ describe Synapse::BaseWatcher do
|
|
41
41
|
default_servers = ['server1', 'server2']
|
42
42
|
let(:args) { testargs.merge({'default_servers' => default_servers}) }
|
43
43
|
it('sets default backends to default_servers') { expect(subject.backends).to equal(default_servers) }
|
44
|
+
|
45
|
+
context "with keep_default_servers set" do
|
46
|
+
let(:args) { testargs.merge({'default_servers' => default_servers, 'keep_default_servers' => true}) }
|
47
|
+
let(:new_backends) { ['discovered1', 'discovered2'] }
|
48
|
+
|
49
|
+
it('keeps default_servers when setting backends') do
|
50
|
+
subject.send(:set_backends, new_backends)
|
51
|
+
expect(subject.backends).to eq(default_servers + new_backends)
|
52
|
+
end
|
53
|
+
end
|
44
54
|
end
|
45
55
|
end
|
46
|
-
|
data/spec/spec_helper.rb
CHANGED
@@ -6,13 +6,13 @@
|
|
6
6
|
# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
|
7
7
|
require "#{File.dirname(__FILE__)}/../lib/synapse"
|
8
8
|
require 'pry'
|
9
|
-
require 'support/
|
9
|
+
require 'support/configuration'
|
10
10
|
|
11
11
|
RSpec.configure do |config|
|
12
12
|
config.treat_symbols_as_metadata_keys_with_true_values = true
|
13
13
|
config.run_all_when_everything_filtered = true
|
14
14
|
config.filter_run :focus
|
15
|
-
config.include
|
15
|
+
config.include Configuration
|
16
16
|
|
17
17
|
# Run specs in random order to surface order dependencies. If you find an
|
18
18
|
# order dependency and want to debug it, you can fix the order by providing
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: synapse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.10.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2014-
|
12
|
+
date: 2014-05-07 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: zk
|
@@ -141,12 +141,13 @@ files:
|
|
141
141
|
- lib/synapse/service_watcher/docker.rb
|
142
142
|
- lib/synapse/service_watcher/ec2tag.rb
|
143
143
|
- lib/synapse/service_watcher/zookeeper.rb
|
144
|
+
- lib/synapse/service_watcher/zookeeper_dns.rb
|
144
145
|
- lib/synapse/version.rb
|
145
146
|
- spec/lib/synapse/haproxy_spec.rb
|
146
147
|
- spec/lib/synapse/service_watcher_base_spec.rb
|
147
148
|
- spec/lib/synapse/service_watcher_docker_spec.rb
|
148
149
|
- spec/spec_helper.rb
|
149
|
-
- spec/support/
|
150
|
+
- spec/support/configuration.rb
|
150
151
|
- spec/support/minimum.conf.yaml
|
151
152
|
- synapse.gemspec
|
152
153
|
homepage: ''
|
@@ -163,7 +164,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
163
164
|
version: '0'
|
164
165
|
segments:
|
165
166
|
- 0
|
166
|
-
hash:
|
167
|
+
hash: 2945433216079037469
|
167
168
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
168
169
|
none: false
|
169
170
|
requirements:
|
@@ -172,7 +173,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
172
173
|
version: '0'
|
173
174
|
segments:
|
174
175
|
- 0
|
175
|
-
hash:
|
176
|
+
hash: 2945433216079037469
|
176
177
|
requirements: []
|
177
178
|
rubyforge_project:
|
178
179
|
rubygems_version: 1.8.23
|
@@ -184,5 +185,5 @@ test_files:
|
|
184
185
|
- spec/lib/synapse/service_watcher_base_spec.rb
|
185
186
|
- spec/lib/synapse/service_watcher_docker_spec.rb
|
186
187
|
- spec/spec_helper.rb
|
187
|
-
- spec/support/
|
188
|
+
- spec/support/configuration.rb
|
188
189
|
- spec/support/minimum.conf.yaml
|