synapse 0.12.1 → 0.12.2

data/.travis.yml

@@ -1,5 +1,6 @@
 language: ruby
 cache: bundler
+sudo: false
 rvm:
   - 1.9.3
   - 2.0.0

data/README.md

@@ -15,7 +15,7 @@ In an environment like Amazon's EC2, all of the available workarounds are subopt
 
 * Round-robin DNS: Slow to converge, and doesn't work when applications cache DNS lookups (which is frequent)
 * Elastic IPs: slow to converge, limited in number, public-facing-only, which makes them less useful for internal services
-* ELB: Again, public-facing only, and only useful for HTTP
+* ELB: ultimately uses DNS (see above), can't tune load balancing, have to launch a new one for every service * region, autoscaling doesn't happen fast enough
 
 One solution to this problem is a discovery service, like [Apache Zookeeper](http://zookeeper.apache.org/).
 However, Zookeeper and similar services have their own problems:
@@ -92,38 +92,50 @@ HAProxy will be transparently reloaded, and your application will keep running w
 
 ## Installation
 
-Add this line to your application's Gemfile:
+To download and run the synapse binary, first install a version of ruby. Then,
+install synapse with:
 
-    gem 'synapse'
-
-And then execute:
+```bash
+$ mkdir -p /opt/smartstack/synapse
+# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc
+$ gem install synapse --install-dir /opt/smartstack/synapse --no-ri --no-rdoc
+```
 
-    $ bundle
+This will download synapse and its dependencies into /opt/smartstack/synapse. You
+might wish to omit the `--install-dir` flag to use your system's default gem
+path, however this will require you to run `gem install synapse` with root
+permissions.
 
-Or install it yourself as:
+You can now run the synapse binary like:
 
-    $ gem install synapse
-    
+```bash
+export GEM_PATH=/opt/smartstack/synapse
+/opt/smartstack/synapse/bin/synapse --help
+```
 
-Don't forget to install HAProxy prior to installing Synapse.
+Don't forget to install HAProxy too.
 
 ## Configuration ##
 
 Synapse depends on a single config file in JSON format; it's usually called `synapse.conf.json`.
-The file has two main sections.
-The first is the `services` section, which lists the services you'd like to connect.
-The second is the `haproxy` section, which specifies how to configure and interact with HAProxy.
+The file has three main sections.
+
+1. [`services`](#services): lists the services you'd like to connect.
+2. [`haproxy`](#haproxy): specifies how to configure and interact with HAProxy.
+3. [`file_output`](#file) (optional): specifies where to write service state to on the filesystem.
 
+<a name="services"/>
 ### Configuring a Service ###
 
-The services are a hash, where the keys are the `name` of the service to be configured.
+The `services` section is a hash, where the keys are the `name` of the service to be configured.
 The name is just a human-readable string; it will be used in logs and notifications.
 Each value in the services hash is also a hash, and should contain the following keys:
 
-* `discovery`: how synapse will discover hosts providing this service (see below)
+* [`discovery`](#discovery): how synapse will discover hosts providing this service (see below)
 * `default_servers`: the list of default servers providing this service; synapse uses these if no others can be discovered
-* `haproxy`: how will the haproxy section for this service be configured
+* [`haproxy`](#haproxysvc): how will the haproxy section for this service be configured
 
+<a name="discovery"/>
 #### Service Discovery ####
 
 We've included a number of `watchers` which provide service discovery.
@@ -183,6 +195,18 @@ be used in preference to the `AWS_` environment variables.
 * `aws_secret_access_key`: AWS secret key or set `AWS_SECRET_ACCESS_KEY` in the environment.
 * `aws_region`: AWS region (i.e. `us-east-1`) or set `AWS_REGION` in the environment.
 
+##### Marathon #####
+
+This watcher polls the Marathon API and retrieves a list of instances for a
+given application.
+
+It takes the following options:
+
+* `marathon_api_url`: Address of the marathon API (e.g. `http://marathon-master:8080`)
+* `application_name`: Name of the application in Marathon
+* `check_interval`: How often to request the list of tasks from Marathon (default: 10 seconds)
+* `port_index`: Index of the backend port in the task's "ports" array. (default: 0)
+
 #### Listing Default Servers ####
 
 You may list a number of default servers providing a service.
@@ -202,15 +226,7 @@ If you do not list any `default_servers`, and all backends for a service
 disappear then the previous known backends will be used.  Disable this behavior
 by unsetting `use_previous_backends`.
 
-#### The `file_output` Section ####
-
-This section controls whether or not synapse will write out service state
-to the filesystem in json format. This can be used for services that want to
-use discovery information but not go through HAProxy.
-
-* `output_directory`: the path to a directory on disk that service registrations
-should be written to.
-
+<a name="haproxysvc"/>
 #### The `haproxy` Section ####
 
 This section is its own hash, which should contain the following keys:
@@ -220,12 +236,15 @@ This section is its own hash, which should contain the following keys:
 * `server_options`: the haproxy options for each `server` line of the service in HAProxy config; it may be left out.
 * `frontend`: additional lines passed to the HAProxy config in the `frontend` stanza of this service
 * `backend`: additional lines passed to the HAProxy config in the `backend` stanza of this service
+* `backend_name`: The name of the generated HAProxy backend for this service
+  (defaults to the service's key in the `services` section)
 * `listen`: these lines will be parsed and placed in the correct `frontend`/`backend` section as applicable; you can put lines which are the same for the frontend and backend here.
 * `shared_frontend`: optional: haproxy configuration directives for a shared http frontend (see below)
 
+<a name="haproxy"/>
 ### Configuring HAProxy ###
 
-The `haproxy` section of the config file has the following options:
+The top level `haproxy` section of the config file has the following options:
 
 * `reload_command`: the command Synapse will run to reload HAProxy
 * `config_file_path`: where Synapse will write the HAProxy config file
@@ -249,6 +268,17 @@ The `haproxy` section of the config file has the following options:
 Note that a non-default `bind_address` can be dangerous.
 If you configure an `address:port` combination that is already in use on the system, haproxy will fail to start.
 
+<a name="file"/>
+### Configuring `file_output` ###
+
+This section controls whether or not synapse will write out service state
+to the filesystem in json format. This can be used for services that want to
+use discovery information but not go through HAProxy.
+
+* `output_directory`: the path to a directory on disk that service registrations
+should be written to.
+
+
 ### HAProxy shared HTTP Frontend ###
 
 For HTTP-only services, it is not always necessary or desirable to dedicate a TCP port per service, since HAProxy can route traffic based on host headers.
@@ -260,7 +290,8 @@ For example:
 
 ```yaml
  haproxy:
-  shared_frontend: "bind 127.0.0.1:8081"
+  shared_frontend:
+   - "bind 127.0.0.1:8081"
   reload_command: "service haproxy reload"
   config_file_path: "/etc/haproxy/haproxy.cfg"
   socket_file_path: "/var/run/haproxy.sock"
@@ -279,7 +310,8 @@ For example:
    discovery: 
     method: "zookeeper"
     path:  "/nerve/services/service1"
-    hosts: "0.zookeeper.example.com:2181"
+    hosts:
+     - "0.zookeeper.example.com:2181"
    haproxy:
     server_options: "check inter 2s rise 3 fall 2"
     shared_frontend:
@@ -298,7 +330,8 @@ For example:
     shared_frontend:
      - "acl is_service1 hdr_dom(host) -i service2.lb.example.com"
      - "use_backend service2 if is_service2
-    backend: "mode http"
+    backend:
+     - "mode http"
 
 ```
 
@@ -333,29 +366,5 @@ Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need thei
 
 ### Creating a Service Watcher ###
 
-If you'd like to create a new service watcher:
-
-1. Create a file for your watcher in `service_watcher` dir
-2. Use the following template:
-```ruby
-require 'synapse/service_watcher/base'
-
-module Synapse
-  class NewWatcher < BaseWatcher
-    def start
-      # write code which begins running service discovery
-    end
-
-    private
-    def validate_discovery_opts
-      # here, validate any required options in @discovery
-    end
-  end
-end
-```
-
-3. Implement the `start` and `validate_discovery_opts` methods
-4. Implement whatever additional methods your discovery requires
-
-When your watcher detects a list of new backends, you should call `set_backends` to
-store the new backends and update the HAProxy config.
+See the Service Watcher [README](lib/synapse/service_watcher/README.md) for
+how to add new Service Watchers.

data/lib/synapse.rb

@@ -1,18 +1,18 @@
+require 'logger'
+require 'json'
+
 require "synapse/version"
-require "synapse/service_watcher/base"
+require "synapse/log"
 require "synapse/haproxy"
 require "synapse/file_output"
 require "synapse/service_watcher"
-require "synapse/log"
 
-require 'logger'
-require 'json'
-
-include Synapse
 
 module Synapse
   class Synapse
+
     include Logging
+
     def initialize(opts={})
       # create the service watchers for all our services
       raise "specify a list of services to connect in the config" unless opts.has_key?('services')

data/lib/synapse/file_output.rb

@@ -1,4 +1,3 @@
-require 'synapse/log'
 require 'fileutils'
 require 'tempfile'
 
@@ -29,6 +28,7 @@ module Synapse
       watchers.each do |watcher|
         write_backends_to_file(watcher.name, watcher.backends)
       end
+      clean_old_watchers(watchers)
     end
 
     def write_backends_to_file(service_name, new_backends)
@@ -53,5 +53,16 @@ module Synapse
         return true
       end
     end
+
+    def clean_old_watchers(current_watchers)
+      # Cleanup old services that Synapse no longer manages
+      FileUtils.cd(@opts['output_directory']) do
+        present_files = Dir.glob('*.json')
+        managed_files = current_watchers.collect {|watcher| "#{watcher.name}.json"}
+        files_to_purge = present_files.select {|svc| not managed_files.include?(svc)}
+        log.info "synapse: purging unknown service files #{files_to_purge}" if files_to_purge.length > 0
+        FileUtils.rm(files_to_purge)
+      end
+    end
   end
 end

data/lib/synapse/haproxy.rb

@@ -1,6 +1,5 @@
 require 'fileutils'
 require 'json'
-require 'synapse/log'
 require 'socket'
 
 module Synapse
@@ -688,7 +687,7 @@ module Synapse
         "\nfrontend #{watcher.name}",
         config.map {|c| "\t#{c}"},
         "\tbind #{@opts['bind_address'] || 'localhost'}:#{watcher.haproxy['port']}",
-        "\tdefault_backend #{watcher.name}"
+        "\tdefault_backend #{watcher.haproxy.fetch('backend_name', watcher.name)}"
       ]
     end
 
@@ -705,6 +704,16 @@ module Synapse
       # setting the enabled state.
       watcher.backends.each do |backend|
         backend_name = construct_name(backend)
+        # If we have information in the state file that allows us to detect
+        # server option changes, use that to potentially force a restart
+        if backends.has_key?(backend_name)
+          old_backend = backends[backend_name]
+          if (old_backend.fetch('haproxy_server_options', "") !=
+              backend.fetch('haproxy_server_options', ""))
+            log.info "synapse: restart required because haproxy_server_options changed for #{backend_name}"
+            @restart_required = true
+          end
+        end
         backends[backend_name] = backend.merge('enabled' => true)
       end
 
@@ -713,13 +722,14 @@ module Synapse
       end
 
       stanza = [
-        "\nbackend #{watcher.name}",
+        "\nbackend #{watcher.haproxy.fetch('backend_name', watcher.name)}",
         config.map {|c| "\t#{c}"},
         backends.keys.shuffle.map {|backend_name|
           backend = backends[backend_name]
           b = "\tserver #{backend_name} #{backend['host']}:#{backend['port']}"
           b = "#{b} cookie #{backend_name}" unless config.include?('mode tcp')
-          b = "#{b} #{watcher.haproxy['server_options']}"
+          b = "#{b} #{watcher.haproxy['server_options']}" if watcher.haproxy['server_options']
+          b = "#{b} #{backend['haproxy_server_options']}" if backend['haproxy_server_options']
           b = "#{b} disabled" unless backend['enabled']
           b }
       ]

data/lib/synapse/service_watcher.rb

@@ -1,22 +1,8 @@
+require "synapse/log"
 require "synapse/service_watcher/base"
-require "synapse/service_watcher/zookeeper"
-require "synapse/service_watcher/ec2tag"
-require "synapse/service_watcher/dns"
-require "synapse/service_watcher/docker"
-require "synapse/service_watcher/zookeeper_dns"
 
 module Synapse
   class ServiceWatcher
-
-    @watchers = {
-      'base' => BaseWatcher,
-      'zookeeper' => ZookeeperWatcher,
-      'ec2tag' => EC2Watcher,
-      'dns' => DnsWatcher,
-      'docker' => DockerWatcher,
-      'zookeeper_dns' => ZookeeperDnsWatcher,
-    }
-
     # the method which actually dispatches watcher creation requests
     def self.create(name, opts, synapse)
       opts['name'] = name
@@ -25,10 +11,16 @@ module Synapse
         unless opts.has_key?('discovery') && opts['discovery'].has_key?('method')
 
       discovery_method = opts['discovery']['method']
-      raise ArgumentError, "Invalid discovery method #{discovery_method}" \
-        unless @watchers.has_key?(discovery_method)
-
-      return @watchers[discovery_method].new(opts, synapse)
+      watcher = begin
+        method = discovery_method.downcase
+        require "synapse/service_watcher/#{method}"
+        # zookeeper_dns => ZookeeperDnsWatcher, ec2tag => Ec2tagWatcher, etc ...
+        method_class  = method.split('_').map{|x| x.capitalize}.join.concat('Watcher')
+        self.const_get("#{method_class}")
+      rescue Exception => e
+        raise ArgumentError, "Specified a discovery method of #{discovery_method}, which could not be found: #{e}"
+      end
+      return watcher.new(opts, synapse)
     end
   end
 end

data/lib/synapse/service_watcher/README.md

@@ -0,0 +1,84 @@
+## Watcher Classes
+
+Watchers are the piece of Synapse that watch an external service registry
+and reflect those changes in the local HAProxy state. Watchers should conform
+to the interface specified by `BaseWatcher` and when your watcher has received
+an update from the service registry you should call
+`set_backends(new_backends)` to trigger a sync of your watcher state with local
+HAProxy state. See the [`Backend Interface`](#backend_interface) section for
+what service registrations Synapse understands.
+
+```ruby
+require "synapse/service_watcher/base"
+
+class Synapse::ServiceWatcher
+  class MyWatcher < BaseWatcher
+    def start
+      # write code which begins running service discovery
+    end
+
+    def stop
+      # write code which tears down the service discovery
+    end
+
+    def ping?
+      # write code to check in on the health of the watcher
+    end
+
+    private
+    def validate_discovery_opts
+      # here, validate any required options in @discovery
+    end
+
+    ... setup watches, poll, etc ... and call set_backends when you have new
+    ... backends to set
+
+  end
+end
+```
+
+### Watcher Plugin Inteface
+Synapse deduces both the class path and class name from the `method` key within
+the watcher configuration.  Every watcher is passed configuration with the
+`method` key, e.g. `zookeeper` or `ec2tag`.
+
+#### Class Location
+Synapse expects to find your class at `synapse/service_watcher/#{method}`. You
+must make your watcher available at that path, and Synapse can "just work" and
+find it.
+
+#### Class Name
+These method strings are then transformed into class names via the following
+function:
+
+```
+method_class  = method.split('_').map{|x| x.capitalize}.join.concat('Watcher')
+```
+
+This has the effect of taking the method, splitting on '_', capitalizing each
+part and recombining with an added 'Watcher' on the end. So `zookeeper_dns`
+becomes `ZookeeperDnsWatcher`, and `zookeeper` becomes `Zookeeper`. Make sure
+your class name is correct.
+
+<a name="backend_interface"/>
+### Backend interface
+Synapse understands the following fields in service backends (which are pulled
+from the service registries):
+
+`host` (string): The hostname of the service instance
+
+`port` (integer): The port running the service on `host`
+
+`name` (string, optional): The human readable name to refer to this service instance by
+
+`weight` (float, optional): The weight that this backend should get when load
+balancing to this service instance. Full support for updating HAProxy based on
+this is still a WIP.
+
+`haproxy_server_options` (string, optional): Any haproxy server options
+specific to this particular server. They will be applied to the generated
+`server` line in the HAProxy configuration. If you want Synapse to react to
+changes in these lines you will need to enable the `state_file_path` option
+in the main synapse configuration. In general the HAProxy backend level
+`haproxy.server_options` setting is preferred to setting this per server
+in your backends.

data/lib/synapse/service_watcher/base.rb

@@ -1,9 +1,9 @@
-require 'set'
 require 'synapse/log'
+require 'set'
 
-module Synapse
+class Synapse::ServiceWatcher
   class BaseWatcher
-    include Logging
+    include Synapse::Logging
 
     LEADER_WARN_INTERVAL = 30