synapse 0.14.0 → 0.14.1

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    YTEwNDE5ZTdlZDM4Njk2YmJkMDRjYWQzYTNiM2FkZTM0MzgyZWE4YQ==
-  data.tar.gz: !binary |-
-    ZjBmNmVmOTY2YWRiYTY1NWZmYTg3YjhiMThhY2NmNjIxNmE3ODkwYQ==
+SHA1:
+  metadata.gz: 1f4b4d3c79352195ba849dcbe72ae849a65f355b
+  data.tar.gz: 9270bec56704c96e37fc37862ee3bb3d4b2501a2
 SHA512:
-  metadata.gz: !binary |-
-    NjhlNDA1YjY1ODkxZWU4YzMyMzI2ODE1YWNjMzlmZWM4MTk1OTVjZTVkMjc4
-    ODFjMTVlNDEyYjVmMDk0YTAxMjM0MWMyNGRmOGFiYjdhNDM4OWFiNDFlMjE1
-    ZDMzNzZiYjg2MjgzY2VmYjQ2YjAxOGEwOTE0Y2MwMTkwNGFjMmU=
-  data.tar.gz: !binary |-
-    MGI0ZTBjNGQ5NjA0NmU1OWI0NzJhYmYyNDI2NWEyODI0MjA1NDM5NmYxNzEw
-    YWYxMzI3MDc2ZTEyMDFkM2U5MzkyY2U5NzBkNjQ4YjA4YWNhNTI0YmRlMzlj
-    N2JhM2MxYjQ5YWVhMzBlZWFhZTM3Y2RkMzVhOTNlMzc0ZmVkY2Y=
+  metadata.gz: 7df9f4fec05f203f66d1b5113df94e35e08d3f3ce5864ae1ba543829640a752d1938e4f9b04530782f7897427b0bbd6e6e07bc88f1ec6f7b003783ba154ed565
+  data.tar.gz: 2bde1c280db160799a9f52cfdedfdf28ce9060d78462a90538849984a9cffd60dad45d304562553bcc44453d440c81ada8c711e7edab97e9493bcf3594e87851

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    synapse (0.13.8)
+    synapse (0.14.1)
       aws-sdk (~> 1.39)
       docker-api (~> 1.7)
       logging (~> 1.8)

data/README.md

@@ -28,15 +28,29 @@ Synapse solves these difficulties in a simple and fault-tolerant way.
 
 ## How Synapse Works ##
 
-Synapse runs on your application servers; here at Airbnb, we just run it on every box we deploy.
-The heart of synapse is actually [HAProxy](http://haproxy.1wt.eu/), a stable and proven routing component.
+Synapse typically runs on your application servers, often every machine. At the heart of Synapse
+are proven routing components like [HAProxy](http://haproxy.1wt.eu/) or [NGINX](http://nginx.org/).
+
 For every external service that your application talks to, we assign a synapse local port on localhost.
 Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.
 
-Synapse comes with a number of `watchers`, which are responsible for service discovery.
-The synapse watchers take care of re-configuring the proxy so that it always points at available servers.
+Under the hood, Synapse sports `service_watcher`s for service discovery and
+`config_generators` for configuring local state (e.g. load balancer configs)
+based on that service discovery state.
+
+Synapse supports service discovery with with pluggable `service_watcher`s which
+take care of signaling to the `config_generators` so that they can react and
+reconfigure to point at available servers on the fly.
+
 We've included a number of default watchers, including ones that query zookeeper and ones using the AWS API.
-It is easy to write your own watchers for your use case, and we encourage submitting them back to the project.
+It is easy to write your own watchers for your use case, and install them as gems that
+extend Synapse's functionality. Check out the [docs](#createsw) on creating
+a watcher if you're interested, and if you think that the service watcher
+would be generally useful feel free to pull request with a link to your watcher.
+
+Synapse also has pluggable `config_generator`s, which are responsible for reacting to service discovery
+changes and writing out appropriate config. Right now HAProxy, and local files are built in, but you
+can plug your own in [easily](#createconfig).
 
 ## Example Migration ##
 
@@ -97,13 +111,22 @@ install synapse with:
 
 ```bash
 $ mkdir -p /opt/smartstack/synapse
+# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc
 
 # If you want to install specific versions of dependencies such as an older
 # version of the aws-sdk, the docker-api, etc, gem install that here *before*
-# gem installing synapse
+# gem installing synapse.
+
+# Example:
+# $ gem install aws-sdk -v XXX
 
-# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc
 $ gem install synapse --install-dir /opt/smartstack/synapse --no-ri --no-rdoc
+
+# If you want to install specific plugins such as watchers or config generators
+# gem install them *after* you install synapse.
+
+# Example:
+# $ gem install synapse-nginx --install-dir /opt/smartstack/synapse --no-ri --no-rdoc
 ```
 
 This will download synapse and its dependencies into /opt/smartstack/synapse. You
@@ -118,16 +141,26 @@ export GEM_PATH=/opt/smartstack/synapse
 /opt/smartstack/synapse/bin/synapse --help
 ```
 
-Don't forget to install HAProxy too.
+Don't forget to install HAProxy or NGINX or whatever proxy your `config_generator`
+is configuring.
 
 ## Configuration ##
 
 Synapse depends on a single config file in JSON format; it's usually called `synapse.conf.json`.
-The file has three main sections.
+The file has a `services` section that describes how services are discovered
+and configured, and then top level sections for every supported proxy or
+configuration section. For example, the default Synapse supports three sections:
 
-1. [`services`](#services): lists the services you'd like to connect.
-2. [`haproxy`](#haproxy): specifies how to configure and interact with HAProxy.
-3. [`file_output`](#file) (optional): specifies where to write service state to on the filesystem.
+* [`services`](#services): lists the services you'd like to connect.
+* [`haproxy`](#haproxy): specifies how to configure and interact with HAProxy.
+* [`file_output`](#file) (optional): specifies where to write service state to on the filesystem.
+* [`<your config generator here>`] (optional): configuration for your custom
+   configuration generators (e.g. nginx, vulcand, envoy, etc ..., w.e. you want).
+
+If you have synapse `config_generator` plugins installed, you'll want a top
+level as well, e.g.:
+* [`nginx`](https://github.com/jolynch/synapse-nginx#top-level-config) (optional):
+  configuration for how to configure and interact with NGINX.
 
 <a name="services"/>
 ### Configuring a Service ###
@@ -138,13 +171,21 @@ Each value in the services hash is also a hash, and must contain the following k
 
 * [`discovery`](#discovery): how synapse will discover hosts providing this service (see [below](#discovery))
 
-The services hash *should* contain a section on how to configure the routing
-component you wish to use for this particular service. The only choice currently
-is `haproxy`:
+The services hash *should* contain a section on how to configure each routing
+component you wish to use for this particular service. The current choices are
+`haproxy` but you can access others e.g. [`nginx`](https://github.com/jolynch/synapse-nginx)
+through [plugins](createconfig). Note that if you give a routing component at the top level
+but not at the service level the default is typically to make that service
+available via that routing component, sans listening ports. If you wish to only
+configure a single component explicitly pass the ``disabled`` option to the
+relevant routing component. For example if you want to only configure HAProxy and
+not NGINX for a particular service, you would pass ``disabled`` to the `nginx` section
+of that service's watcher config.
 
 * [`haproxy`](#haproxysvc): how will the haproxy section for this service be configured
+* [`nginx`](https://github.com/jolynch/synapse-nginx#service-watcher-config): how will the nginx section for this service be configured. **NOTE** to use this you must have the synapse-nginx [plugin](#plugins) installed.
 
-The services hash may contain the following keys:
+The services hash may contain the following additional keys:
 
 * `default_servers` (default: `[]`): the list of default servers providing this service; synapse uses these if no others can be discovered. See [Listing Default Servers](#defaultservers).
 * `keep_default_servers` (default: false): whether default servers should be added to discovered services
@@ -278,8 +319,17 @@ This section is its own hash, which should contain the following keys:
 * `disabled`: A boolean value indicating if haproxy configuration management
 for just this service instance ought be disabled. For example, if you want
 file output for a particular service but no HAProxy config. (default is ``False``)
-* `port`: the port (on localhost) where HAProxy will listen for connections to the service. If this is omitted, only a backend stanza (and no frontend stanza) will be generated for this service; you'll need to get traffic to your service yourself via the `shared_frontend` or manual frontends in `extra_sections`
-* `bind_address`: force HAProxy to listen on this address ( default is localhost ). Setting `bind_address` on a per service basis overrides the global `bind_address` in the top level `haproxy`. Having HAProxy listen for connections on different addresses ( example: service1 listen on 127.0.0.2:443 and service2 listen on 127.0.0.3:443) allows /etc/hosts entries to point to services.
+* `port`: the port (on localhost) where HAProxy will listen for connections to
+the service. If this is null, just the bind_address will be used (e.g. for
+unix sockets) and if omitted, only a backend stanza (and no frontend stanza)
+will be generated for this service. In the case of a bare backend, you'll need
+to get traffic to your service yourself via the `shared_frontend` or
+manual frontends in `extra_sections`
+* `bind_address`: force HAProxy to listen on this address (default is localhost).
+Setting `bind_address` on a per service basis overrides the global `bind_address`
+in the top level `haproxy`. Having HAProxy listen for connections on
+different addresses (example: service1 listen on 127.0.0.2:443 and service2
+listen on 127.0.0.3:443) allows /etc/hosts entries to point to services.
 * `bind_options`: optional: default value is an empty string, specify additional bind parameters, such as ssl accept-proxy, crt, ciphers etc.
 * `server_port_override`: **DEPRECATED**. Renamed [`backend_port_override`](#backend_port_override) and moved to the top level hash. This will be removed in future versions.
 * `server_options`: the haproxy options for each `server` line of the service in HAProxy config; it may be left out.
@@ -335,7 +385,6 @@ use discovery information but not go through HAProxy.
 * `output_directory`: the path to a directory on disk that service registrations
 should be written to.
 
-
 ### HAProxy shared HTTP Frontend ###
 
 For HTTP-only services, it is not always necessary or desirable to dedicate a TCP port per service, since HAProxy can route traffic based on host headers.
@@ -416,6 +465,9 @@ frontend shared-frontend
 Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need their own dedicated ports.
 
 ## Contributing
+Note that now that we have a fully dynamic include system for service watchers
+and configuration generators, you don't *have* to PR into the main tree, but
+please do contribute a [link](#plugins).
 
 1. Fork it
 2. Create your feature branch (`git checkout -b my-new-feature`)
@@ -423,12 +475,20 @@ Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need thei
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
 
+<a name="createsw"/>
 ### Creating a Service Watcher ###
 
 See the Service Watcher [README](lib/synapse/service_watcher/README.md) for
 how to add new Service Watchers.
 
+<a name="createconfig"/>
 ### Creating a Config Generator ###
 
 See the Config Generator [README](lib/synapse/config_generator/README.md) for
 how to add new Config Generators
+
+<a name="plugins"/>
+## Links to Synapse Plugins ##
+* [`synapse-nginx`](https://github.com/jolynch/synapse-nginx) Is a `config_generator`
+  which allows Synapse to automatically configure and administer a local NGINX
+  proxy.

data/lib/synapse/config_generator/haproxy.rb

@@ -997,9 +997,13 @@ class Synapse::ConfigGenerator
       )
       backend_name = watcher_config.fetch('backend_name', watcher.name)
 
+      # Explicit null value passed indicating no port needed
+      # For example if the bind_address is a unix port
+      bind_port = port.nil? ? '' : ":#{port}"
+
       bind_line = [
         "\tbind",
-        "#{bind_address}:#{port}",
+        "#{bind_address}#{bind_port}",
         watcher_config['bind_options']
       ].compact.join(' ')
 

data/lib/synapse/version.rb

@@ -1,3 +1,3 @@
 module Synapse
-  VERSION = "0.14.0"
+  VERSION = "0.14.1"
 end

data/spec/lib/synapse/haproxy_spec.rb

@@ -69,6 +69,15 @@ describe Synapse::ConfigGenerator::Haproxy do
     mockWatcher
   end
 
+  let(:mockwatcher_frontend_with_nil_port) do
+    mockWatcher = double(Synapse::ServiceWatcher)
+    allow(mockWatcher).to receive(:name).and_return('example_service6')
+    allow(mockWatcher).to receive(:config_for_generator).and_return({
+      'haproxy' => {'port' => nil, 'bind_address' => "unix@/foo/bar.sock"}
+    })
+    mockWatcher
+  end
+
   let(:mockwatcher_disabled) do
     mockWatcher = double(Synapse::ServiceWatcher)
     allow(mockWatcher).to receive(:name).and_return('disabled_watcher')
@@ -440,6 +449,11 @@ describe Synapse::ConfigGenerator::Haproxy do
     expect(subject.generate_frontend_stanza(mockwatcher_frontend_with_bind_options, mockConfig)).to eql(["\nfrontend example_service4", [], "\tbind localhost:2200 ssl no-sslv3 crt /path/to/cert/example.pem ciphers ECDHE-ECDSA-CHACHA20-POLY1305", "\tdefault_backend example_service4"])
   end
 
+  it 'generates frontend stanza with nil port' do
+    mockConfig= []
+    expect(subject.generate_frontend_stanza(mockwatcher_frontend_with_nil_port, mockConfig)).to eql(["\nfrontend example_service6", [], "\tbind unix@/foo/bar.sock", "\tdefault_backend example_service6"])
+  end
+
   it 'respects frontend bind_address ' do
     mockConfig = []
     expect(subject.generate_frontend_stanza(mockwatcher_frontend_with_bind_address, mockConfig)).to eql(["\nfrontend example_service5", [], "\tbind 127.0.0.3:2200", "\tdefault_backend example_service5"])

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: synapse
 version: !ruby/object:Gem::Version
-  version: 0.14.0
+  version: 0.14.1
 platform: ruby
 authors:
 - Martin Rhoads
@@ -10,160 +10,160 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-03-09 00:00:00.000000000 Z
+date: 2017-03-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.39'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.39'
 - !ruby/object:Gem::Dependency
   name: docker-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.7'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.7'
 - !ruby/object:Gem::Dependency
   name: zk
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.9.4
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.9.4
 - !ruby/object:Gem::Dependency
   name: logging
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.8'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.8'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 3.1.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 3.1.0
 - !ruby/object:Gem::Dependency
   name: factory_girl
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: pry-nav
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: webmock
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: timecop
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: Synapse is a daemon used to dynamically configure and manage local instances
@@ -179,10 +179,10 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .mailmap
-- .rspec
-- .travis.yml
+- ".gitignore"
+- ".mailmap"
+- ".rspec"
+- ".travis.yml"
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
@@ -235,17 +235,17 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.2.2
 signing_key: 
 specification_version: 4
 summary: Dynamic HAProxy configuration daemon