synapse 0.2.1 → 0.8.0

data/.gitignore

@@ -18,3 +18,6 @@ tmp
 *~
 .vagrant
 .*sw?
+vendor/
+
+synapse.jar

data/.mailmap

@@ -0,0 +1,3 @@
+<igor.serebryany@airbedandbreakfast.com> <igor47@moomers.org>
+<martin.rhoads@airbnb.com> <ermal14@gmail.com>
+Pierre Carrier <pierre@gcarrier.fr>

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Martin Rhoads
+Copyright (c) 2013 Airbnb, Inc.
 
 MIT License
 
@@ -19,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Makefile

@@ -0,0 +1,6 @@
+build: synapse.jar
+
+synapse.jar:
+	jruby -S warble jar
+
+.PHONY: build push

data/README.md

@@ -54,10 +54,7 @@ Add a hash under `services` that looks like this:
 
 ```json
 {"services":
-    {
-      "name": "proddb",
-      "local_port": 3219,
-      "server_options": "check inter 2000 rise 3 fall 2",
+    "proddb": {
       "default_servers": [
         {
           "name": "default-db",
@@ -70,8 +67,16 @@ Add a hash under `services` that looks like this:
         "tag": "proddb",
         "value": "true"
       },
-      "listen": [
-      ]
+      "haproxy": {
+        "port": 3219,
+        "server_options": "check inter 2000 rise 3 fall 2",
+        "frontend": [
+          "mode tcp",
+        ],
+        "backend": [
+          "mode tcp",
+        ],
+      },
     },
 ...
 ```
@@ -114,15 +119,13 @@ The second is the `haproxy` section, which specifies how to configure and intera
 
 ### Configuring a Service ###
 
-Each service hash has the following options:
+The services are a hash, where the keys are the `name` of the service to be configured.
+The name is just a human-readable string; it will be used in logs and notifications.
+Each value in the services hash is also a hash, and should contain the following keys:
 
-* `name`: a human-readable name for the service, this is used in logs and notifications
-* `local_port`: the port (on localhost) where HAProxy will listen for connections to the serivce
 * `discovery`: how synapse will discover hosts providing this service (see below)
 * `default_servers`: the list of default servers providing this service; synapse uses these if none others can be discovered
-* `server_port_override`: the port that discovered servers listen on; if the discovery method discovers a port along with hostnames (like the zookeeper watcher) this option may be left out, but will be used in preference if given
-* `server_options`: the haproxy options for each `server` line of the service in HAProxy config; may be left out
-* `listen`: additional lines passed to the HAProxy config in the `listen` stanza of this service
+* `haproxy`: how will the haproxy section for this service be configured
 
 #### Service Discovery ####
 
@@ -149,6 +152,17 @@ The watcher assumes that each node under `path` represents a service server.
 Synapse attempts to decode the data in each of these nodes using JSON and also using Thrift under the standard Twitter service encoding.
 We assume that the data contains a hostname and a port for service servers.
 
+##### Docker #####
+
+This watcher retrieves a list of [docker](http://www.docker.io/) containers via docker's [HTTP API](http://docs.docker.io/en/latest/api/docker_remote_api/).
+It takes the following options:
+
+* `method`: docker
+* `servers`: a list of servers running docker as a daemon. Format is `{"name":"...", "host": "..."[, port: 4243]}`
+* `image_name`: find containers running this image
+* `container_port`: find containers forwarding this port
+* `check_interval`: how often to poll the docker API on each server. Default is 15s.
+
 #### Listing Default Servers ####
 
 You may list a number of default servers providing a service.
@@ -162,6 +176,17 @@ The `default_servers` list is used only when service discovery returns no server
 In that case, the service proxy will be created with the servers listed here.
 If you do not list any default servers, no proxy will be created.
 
+#### The `haproxy` Section ####
+
+This section is it's own hash, which should contain the following keys:
+
+* `port`: the port (on localhost) where HAProxy will listen for connections to the service.
+* `server_port_override`: the port that discovered servers listen on; you should specify this if your discovery mechanism only discovers names or addresses (like the DNS watcher). If the discovery method discovers a port along with hostnames (like the zookeeper watcher) this option may be left out, but will be used in preference if given.
+* `server_options`: the haproxy options for each `server` line of the service in HAProxy config; it may be left out.
+* `frontend`: additional lines passed to the HAProxy config in the `frontend` stanza of this service
+* `backend`: additional lines passed to the HAProxy config in the `backend` stanza of this service
+* `listen`: these lines will be parsed and placed in the correct `frontend`/`backend` section as applicable; you can put lines which are the same for the frontend and backend here.
+
 ### Configuring HAProxy ###
 
 The `haproxy` section of the config file has the following options:
@@ -172,6 +197,9 @@ The `haproxy` section of the config file has the following options:
 * `do_reloads`: whether or not Synapse will reload HAProxy (default to `true`)
 * `global`: options listed here will be written into the `global` section of the HAProxy config
 * `defaults`: options listed here will be written into the `defaults` section of the HAProxy config
+* `bind_address`: force HAProxy to listen  on this address (default is localhost)
+
+Note that a non-default `bind_address` can be dangerous: it is up to you to ensure that HAProxy will not attempt to bind an address:port combination that is not already in use by one of your services.
 
 ## Contributing
 
@@ -188,7 +216,8 @@ If you'd like to create a new service watcher:
 1. Create a file for your watcher in `service_watcher` dir
 2. Use the following template:
 ```ruby
-require_relative "./base"
+require 'synapse/service_watcher/base'
+
 module Synapse
   class NewWatcher < BaseWatcher
     def start

data/bin/synapse

@@ -3,18 +3,18 @@
 require 'json'
 require 'optparse'
 
-require_relative "../lib/synapse"
+require 'synapse'
 
 options={}
 
 # set command line options
 optparse = OptionParser.new do |opts|
-  opts.banner =<<EOB
+  opts.banner =<<-EOB
 Welcome to synapse
 
 Usage: synapse --config /path/to/synapse/config
-EOB
-  
+  EOB
+
   options[:config] = ENV['SYNAPSE_CONFIG']
   opts.on('-c config','--config config', String, 'path to synapse config') do |key,value|
     options[:config] = key
@@ -24,31 +24,39 @@ EOB
     puts opts
     exit
   end
-
 end
 
-
 # parse command line arguments
 optparse.parse!
 
-
-# parse synapse config file
-begin
-  config = JSON::parse(File.read(options[:config]))
-rescue Errno::ENOENT => e
-  raise ArgumentError, "config file does not exist:\n#{e.inspect}"
-rescue Errno::EACCES => e
-  raise ArgumentError, "could not open config file:\n#{e.inspect}"
-rescue JSON::ParserError => e
-  raise "config file #{options[:config]} is not json:\n#{e.inspect}" 
+def parseconfig(filename)
+  # parse synapse config file
+  begin
+    c = JSON::parse(File.read(filename))
+  rescue Errno::ENOENT => e
+    raise ArgumentError, "config file does not exist:\n#{e.inspect}"
+  rescue Errno::EACCES => e
+    raise ArgumentError, "could not open config file:\n#{e.inspect}"
+  rescue JSON::ParserError => e
+    raise "config file #{filename} is not json:\n#{e.inspect}"
+  end
+  return c
 end
 
+config = parseconfig(options[:config])
+config['services'] ||= {}
 
-# create synapse object
-s = Synapse::Synapse.new config
+if config.has_key?('service_conf_dir')
+  cdir = File.expand_path(config['service_conf_dir'])
+  if ! Dir.exists?(cdir)
+    raise "service conf dir does not exist:#{cdir}"
+  end
+  cfiles = Dir.glob(File.join(cdir, '*.json'))
+  cfiles.each { |x| config['services'][File.basename(x[/(.*)\.json$/, 1])] = parseconfig(x) }
+end
 
-# start synapse
+# run synapse
+s = Synapse::Synapse.new(config)
 s.run
 
-
-puts "exiting synapse"
+puts "synapse has exited"

data/config/hostheader_test.json

@@ -0,0 +1,71 @@
+{
+  "services": {
+    "service1": {
+      "default_servers": [
+        { "name": "default1", "host": "localhost", "port": 8080 }
+      ],
+      "discovery": {
+        "method": "dns",
+        "nameserver": "127.0.0.1",
+        "servers": [
+          "0.www.example.com",
+          "1.www.example.com"
+        ]
+      },
+      "haproxy": {
+        "server_options": "check inter 2s rise 3 fall 2",
+        "listen": [
+          "mode http",
+          "option httplog"
+        ],
+        "backend": [
+          "mode http",
+          "option httpchk GET /health HTTP/1.0"
+        ]
+      }
+    }
+  },
+  "haproxy": {
+    "reload_command": "sudo service haproxy reload",
+    "config_file_path": "/etc/haproxy/haproxy.cfg",
+    "socket_file_path": "/var/haproxy/stats.sock",
+    "do_writes": true,
+    "do_reloads": true,
+    "do_socket": true,
+    "global": [
+      "daemon",
+      "user haproxy",
+      "group haproxy",
+      "maxconn 4096",
+      "log     127.0.0.1 local0",
+      "log     127.0.0.1 local1 notice",
+      "stats   socket /var/haproxy/stats.sock mode 666 level admin"
+    ],
+    "defaults": [
+      "log      global",
+      "option   dontlognull",
+      "maxconn  2000",
+      "retries  3",
+      "timeout  connect 5s",
+      "timeout  client  1m",
+      "timeout  server  1m",
+      "option   redispatch",
+      "balance  roundrobin"
+    ],
+    "extra_sections": {
+      "listen stats :3212": [
+        "mode http",
+        "stats enable",
+        "stats uri /",
+        "stats refresh 5s"
+      ],
+      "frontend http-generic-in": [
+        "bind 127.0.0.1:80",
+        "acl is_service1 hdr_dom(host) -i service1.lb",
+        "acl is_cache hdr_dom(host) -i cache.lb",
+        "use_backend service1 if is_service1",
+        "use_backend cache if is_cache"
+      ]
+    }
+  }
+}

data/config/svcdir_test.json

@@ -0,0 +1,46 @@
+{
+  "service_conf_dir": "config/synapse_services",
+  "haproxy": {
+    "reload_command": "sudo service haproxy reload",
+    "config_file_path": "/etc/haproxy/haproxy.cfg",
+    "socket_file_path": "/var/haproxy/stats.sock",
+    "do_writes": true,
+    "do_reloads": true,
+    "do_socket": true,
+    "global": [
+      "daemon",
+      "user haproxy",
+      "group haproxy",
+      "maxconn 4096",
+      "log     127.0.0.1 local0",
+      "log     127.0.0.1 local1 notice",
+      "stats   socket /var/haproxy/stats.sock mode 666 level admin"
+    ],
+    "defaults": [
+      "log      global",
+      "option   dontlognull",
+      "maxconn  2000",
+      "retries  3",
+      "timeout  connect 5s",
+      "timeout  client  1m",
+      "timeout  server  1m",
+      "option   redispatch",
+      "balance  roundrobin"
+    ],
+    "extra_sections": {
+      "listen stats :3212": [
+        "mode http",
+        "stats enable",
+        "stats uri /",
+        "stats refresh 5s"
+      ],
+      "frontend http-generic-in": [
+        "bind 127.0.0.1:80",
+        "acl is_service1 hdr_dom(host) -i service1.lb",
+        "acl is_cache hdr_dom(host) -i cache.lb",
+        "use_backend service1 if is_service1",
+        "use_backend cache if is_cache"
+      ]
+    }
+  }
+}

data/config/synapse.conf.json

@@ -1,9 +1,6 @@
 {
-  "services": [
-    {
-      "name": "service1",
-      "local_port": 3213,
-      "server_options": "check inter 2s rise 3 fall 2",
+  "services": {
+    "service1": {
       "default_servers": [
         {
           "name": "default1",
@@ -13,22 +10,23 @@
       ],
       "discovery": {
         "method": "zookeeper",
-        "path": "/airbnb/service/service1",
+        "path": "/services/service1",
         "hosts": [
-          "zk0.airbnb.com:2181",
-          "zk1.airbnb.com:2181"
+          "localhost:2181"
         ]
       },
-      "listen": [
-        "mode http",
-        "option httpchk /health",
-        "http-check expect string OK"
-      ]
+      "haproxy": {
+        "port": 3213,
+        "server_options": "check inter 2s rise 3 fall 2",
+        "listen": [
+          "mode http",
+          "option httpchk /health",
+          "http-check expect string OK"
+        ]
+      }
+      
     },
-    {
-      "name": "service2",
-      "local_port": 3214,
-      "server_options": "check inter 2s rise 3 fall 2",
+    "service2": {
       "default_servers": [
         {
           "name": "default1",
@@ -38,18 +36,21 @@
       ],
       "discovery": {
         "method": "zookeeper",
-        "path": "/airbnb/service/service2",
+        "path": "/services/service2",
         "hosts": [
-          "zk0.airbnb.com:2181",
-          "zk1.airbnb.com:2181"
+          "localhost:2181"
         ]
       },
-      "listen": [
-        "mode http",
-        "option httpchk /health",
-      ]
+      "haproxy": {
+        "port": 3214,
+        "server_options": "check inter 2s rise 3 fall 2",
+        "listen": [
+          "mode http",
+          "option httpchk /health"
+        ]
+      }
     }
-  ],
+  },
   "haproxy": {
     "reload_command": "sudo service haproxy reload",
     "config_file_path": "/etc/haproxy/haproxy.cfg",
@@ -85,12 +86,5 @@
         "stats refresh 5s"
       ]
     }
-  },
-  "synapse": {
-    "self_check_port": 3210,
-    "self_check_proxy_port": 3211,
-    "zk_servers": [
-      "localhost:2181"
-    ]
   }
 }

data/config/synapse_services/service1.json

@@ -0,0 +1,24 @@
+{
+  "default_servers": [
+    { "name": "default1", "host": "localhost", "port": 8080 }
+  ],
+  "discovery": {
+    "method": "dns",
+    "nameserver": "10.10.1.13",
+    "servers": [
+      "0.www.example.com",
+      "1.www.example.com"
+    ]
+  },
+  "haproxy": {
+    "server_options": "check inter 2s rise 3 fall 2",
+    "listen": [
+      "mode http",
+      "option httplog"
+    ],
+    "backend": [
+      "mode http",
+      "option httpchk GET /health HTTP/1.0"
+    ]
+  }
+}