synapse 0.0.1 → 0.2.1

data/.gitignore

@@ -15,3 +15,6 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+*~
+.vagrant
+.*sw?

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/Gemfile.lock

@@ -0,0 +1,45 @@
+PATH
+  remote: .
+  specs:
+    synapse (0.1.4)
+      thrift (~> 0.9.0)
+      zk (~> 1.7.4)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    coderay (1.0.8)
+    diff-lcs (1.1.3)
+    little-plugger (1.1.3)
+    logging (1.7.2)
+      little-plugger (>= 1.1.3)
+    method_source (0.8.1)
+    pry (0.9.10)
+      coderay (~> 1.0.5)
+      method_source (~> 0.8)
+      slop (~> 3.3.1)
+    pry-nav (0.2.2)
+      pry (~> 0.9.10)
+    rspec (2.12.0)
+      rspec-core (~> 2.12.0)
+      rspec-expectations (~> 2.12.0)
+      rspec-mocks (~> 2.12.0)
+    rspec-core (2.12.0)
+    rspec-expectations (2.12.0)
+      diff-lcs (~> 1.1.3)
+    rspec-mocks (2.12.0)
+    slop (3.3.3)
+    thrift (0.9.0)
+    zk (1.7.5)
+      logging (~> 1.7.2)
+      zookeeper (~> 1.4.0)
+    zookeeper (1.4.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  pry
+  pry-nav
+  rspec
+  synapse!

data/README.md

@@ -1,6 +1,95 @@
-# Synapse
+# Synapse #
 
-Write a gem description
+Synapse is Airbnb's new system for service discovery.
+Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible.
+The end result is the ability to connect internal services together in a scalable, fault-tolerant way.
+
+## Motivation ##
+
+Synapse emerged from the need to maintain high-availability applications in the cloud.
+Traditional high-availability techniques, which involve using a CRM like [pacemaker](http://linux-ha.org/wiki/Pacemaker), do not work in environments where the end-user has no control over the networking.
+In an environment like Amazon's EC2, all of the available workarounds are suboptimal:
+
+* Round-robin DNS: Slow to converge, and doesn't work when applications cache DNS lookups (which is frequent)
+* Elastic IPs: slow to converge, limited in number, public-facing-only, which makes them less useful for internal services
+* ELB: Again, public-facing only, and only useful for HTTP
+
+One solution to this problem is a discovery service, like [Apache Zookeeper](http://zookeeper.apache.org/).
+However, Zookeeper and similar services have their own problems:
+
+* Service discovery is embedded in all of your apps; often, integration is not simple
+* The discovery layer itself it subject to failure
+* Requires additional servers/instances
+
+Synapse solves these difficulties in a simple and fault-tolerant way.
+
+## How Synapse Works ##
+
+Synapse runs on your application servers; here at Airbnb, we just run it on every box we deploy.
+The heart of synapse is actually [HAProxy](http://haproxy.1wt.eu/), a stable and proven routing component.
+For every external service that your application talks to, we assign a synapse local port on localhost.
+Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.
+
+Synapse comes with a number of `watchers`, which are responsible for service discovery.
+The synapse watchers take care of re-configuring the proxy so that it always points at available servers.
+We've included a number of default watchers, including ones that query zookeeper and ones using the AWS API.
+It is easy to write your own watchers for your use case, and we encourage submitting them back to the project.
+
+## Example Migration ##
+
+Lets suppose your rails application depends on a Postgre database instance.
+The database.yaml file has the DB host and port hardcoded:
+
+```yaml
+production:
+  database: mydb
+  host: mydb.example.com
+  port: 5432
+```
+
+You would like to be able to fail over to a different database in case the original dies.
+Let's suppose your instance is running in AWS and you're using the tag 'proddb' set to 'true' to indicate the prod DB.
+You set up synapse to proxy the DB connection on `localhost:3219` in the `synapse.conf.json` file.
+Add a hash under `services` that looks like this:
+
+```json
+{"services":
+    {
+      "name": "proddb",
+      "local_port": 3219,
+      "server_options": "check inter 2000 rise 3 fall 2",
+      "default_servers": [
+        {
+          "name": "default-db",
+          "host": "mydb.example.com",
+          "port": 5432
+        }
+      ],
+      "discovery": {
+        "method": "awstag",
+        "tag": "proddb",
+        "value": "true"
+      },
+      "listen": [
+      ]
+    },
+...
+```
+
+And then change your database.yaml file to look like this:
+
+```yaml
+production:
+  database: mydb
+  host: localhost
+  port: 3219
+```
+
+Start up synapse.
+It will configure HAProxy with a proxy from `localhost:3219` to your DB.
+It will attempt to find the DB using the AWS API; if that does not work, it will default to the DB given in `default_servers`.
+In the worst case, if AWS API is down and you need to change which DB your application talks to, simply edit the `synapse.conf.json` file, update the `default_servers` and restart synapse.
+HAProxy will be transparently reloaded, and your application will keep running without a hiccup.
 
 ## Installation
 
@@ -16,9 +105,73 @@ Or install it yourself as:
 
     $ gem install synapse
 
-## Usage
+## Configuration ##
+
+Synapse depends on a single config file in JSON format; it's usually called `synapse.conf.json`.
+The file has two main sections.
+The first is the `services` section, which lists the services you'd like to connect.
+The second is the `haproxy` section, which specifies how to configure and interact with HAProxy.
+
+### Configuring a Service ###
+
+Each service hash has the following options:
+
+* `name`: a human-readable name for the service, this is used in logs and notifications
+* `local_port`: the port (on localhost) where HAProxy will listen for connections to the serivce
+* `discovery`: how synapse will discover hosts providing this service (see below)
+* `default_servers`: the list of default servers providing this service; synapse uses these if none others can be discovered
+* `server_port_override`: the port that discovered servers listen on; if the discovery method discovers a port along with hostnames (like the zookeeper watcher) this option may be left out, but will be used in preference if given
+* `server_options`: the haproxy options for each `server` line of the service in HAProxy config; may be left out
+* `listen`: additional lines passed to the HAProxy config in the `listen` stanza of this service
 
-Write usage instructions here
+#### Service Discovery ####
+
+We've included a number of `watchers` which provide service discovery.
+Put these into the `discovery` section of the service hash, with these options:
+
+##### Stub #####
+
+The stub watcher, this is useful in situations where you only want to use the servers in the `default_servers` list.
+It has only one option:
+
+* `method`: stub
+
+##### Zookeeper #####
+
+This watcher retrieves a list of servers from zookeeper.
+It takes the following options:
+
+* `method`: zookeeper
+* `path`: the zookeeper path where ephemeral nodes will be created for each available service server
+* `hosts`: the list of zookeeper servers to query
+
+The watcher assumes that each node under `path` represents a service server.
+Synapse attempts to decode the data in each of these nodes using JSON and also using Thrift under the standard Twitter service encoding.
+We assume that the data contains a hostname and a port for service servers.
+
+#### Listing Default Servers ####
+
+You may list a number of default servers providing a service.
+Each hash in that section has the following options:
+
+* `name`: a human-readable name for the default server; must be unique
+* `host`: the host or IP address of the server
+* `port`: the port where the service runs on the `host`
+
+The `default_servers` list is used only when service discovery returns no servers.
+In that case, the service proxy will be created with the servers listed here.
+If you do not list any default servers, no proxy will be created.
+
+### Configuring HAProxy ###
+
+The `haproxy` section of the config file has the following options:
+
+* `reload_command`: the command Synapse will run to reload HAProxy
+* `config_file_path`: where Synapse will write the HAProxy config file
+* `do_writes`: whether or not the config file will be written (default to `true`)
+* `do_reloads`: whether or not Synapse will reload HAProxy (default to `true`)
+* `global`: options listed here will be written into the `global` section of the HAProxy config
+* `defaults`: options listed here will be written into the `defaults` section of the HAProxy config
 
 ## Contributing
 
@@ -27,3 +180,31 @@ Write usage instructions here
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+### Creating a Service Watcher ###
+
+If you'd like to create a new service watcher:
+
+1. Create a file for your watcher in `service_watcher` dir
+2. Use the following template:
+```ruby
+require_relative "./base"
+module Synapse
+  class NewWatcher < BaseWatcher
+    def start
+      # write code which begins running service discovery
+    end
+
+    private
+    def validate_discovery_opts
+      # here, validate any required options in @discovery
+    end
+  end
+end
+```
+
+3. Implement the `start` and `validate_discovery_opts` methods
+4. Implement whatever additional methods your discovery requires
+
+When your watcher detects a list of new backends, they should be written to `@backends`.
+You should then call `@synapse.configure` to force synapse to update the HAProxy config.

data/Vagrantfile

@@ -0,0 +1,112 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+this_dir = File.dirname(File.expand_path __FILE__)
+
+Vagrant::Config.run do |config|
+  # All Vagrant configuration is done here. The most common configuration
+  # options are documented and commented below. For a complete reference,
+  # please see the online documentation at vagrantup.com.
+
+  # Every Vagrant virtual environment requires a box to build off of.
+  config.vm.box = "precise64"
+
+  # The url from where the 'config.vm.box' box will be fetched if it
+  # doesn't already exist on the user's system.
+  # config.vm.box_url = "http://domain.com/path/to/above.box"
+
+  config.vm.box_url = 'http://files.vagrantup.com/precise64.box'
+
+  # Boot with a GUI so you can see the screen. (Default is headless)
+  # config.vm.boot_mode = :gui
+
+  # Assign this VM to a host-only network IP, allowing you to access it
+  # via the IP. Host-only networks can talk to the host machine as well as
+  # any other machines on the same network, but cannot be accessed (through this
+  # network interface) by any external networks.
+  # config.vm.network :hostonly, "192.168.33.10"
+
+  # Assign this VM to a bridged network, allowing you to connect directly to a
+  # network using the host's network device. This makes the VM appear as another
+  # physical device on your network.
+  # config.vm.network :bridged
+
+  # Forward a port from the guest to the host, which allows for outside
+  # computers to access the VM, whereas host only networking does not.
+  # config.vm.forward_port 80, 8080
+
+  # Share an additional folder to the guest VM. The first argument is
+  # an identifier, the second is the path on the guest to mount the
+  # folder, and the third is the path on the host to the actual folder.
+  # config.vm.share_folder "v-data", "/vagrant_data", "../data"
+
+  config.vm.share_folder 'this_dir', '/this_dir', this_dir
+
+  # Enable provisioning with Puppet stand alone.  Puppet manifests
+  # are contained in a directory path relative to this Vagrantfile.
+  # You will need to create the manifests directory and a manifest in
+  # the file base.pp in the manifests_path directory.
+  #
+  # An example Puppet manifest to provision the message of the day:
+  #
+  # # group { "puppet":
+  # #   ensure => "present",
+  # # }
+  # #
+  # # File { owner => 0, group => 0, mode => 0644 }
+  # #
+  # # file { '/etc/motd':
+  # #   content => "Welcome to your Vagrant-built virtual machine!
+  # #               Managed by Puppet.\n"
+  # # }
+  #
+  # config.vm.provision :puppet do |puppet|
+  #   puppet.manifests_path = "manifests"
+  #   puppet.manifest_file  = "base.pp"
+  # end
+
+  # Enable provisioning with chef solo, specifying a cookbooks path, roles
+  # path, and data_bags path (all relative to this Vagrantfile), and adding 
+  # some recipes and/or roles.
+  #
+  config.vm.provision :chef_solo do |chef|
+    # chef.cookbooks_path = "../my-recipes/cookbooks"
+    # chef.roles_path = "../my-recipes/roles"
+    # chef.data_bags_path = "../my-recipes/data_bags"
+    # chef.add_recipe "mysql"
+    # chef.add_role "web"
+  
+    # You may also specify custom JSON attributes:
+    # chef.json = { :mysql_password => "foo" }
+
+    chef.cookbooks_path = File.join(this_dir, "chef/cookbooks")
+    chef.add_recipe "synapse"
+  end
+
+  # Enable provisioning with chef server, specifying the chef server URL,
+  # and the path to the validation key (relative to this Vagrantfile).
+  #
+  # The Opscode Platform uses HTTPS. Substitute your organization for
+  # ORGNAME in the URL and validation key.
+  #
+  # If you have your own Chef Server, use the appropriate URL, which may be
+  # HTTP instead of HTTPS depending on your configuration. Also change the
+  # validation key to validation.pem.
+  #
+  # config.vm.provision :chef_client do |chef|
+  #   chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
+  #   chef.validation_key_path = "ORGNAME-validator.pem"
+  # end
+  #
+  # If you're using the Opscode platform, your validator client is
+  # ORGNAME-validator, replacing ORGNAME with your organization name.
+  #
+  # IF you have your own Chef Server, the default validation client name is
+  # chef-validator, unless you changed the configuration.
+  #
+  #   chef.validation_client_name = "ORGNAME-validator"
+
+  Vagrant::Config.run do |config|
+    config.vm.provision :shell, :path => "test.sh"
+  end
+end

data/bin/synapse

@@ -0,0 +1,54 @@
+#!/usr/bin/env ruby
+
+require 'json'
+require 'optparse'
+
+require_relative "../lib/synapse"
+
+options={}
+
+# set command line options
+optparse = OptionParser.new do |opts|
+  opts.banner =<<EOB
+Welcome to synapse
+
+Usage: synapse --config /path/to/synapse/config
+EOB
+  
+  options[:config] = ENV['SYNAPSE_CONFIG']
+  opts.on('-c config','--config config', String, 'path to synapse config') do |key,value|
+    options[:config] = key
+  end
+
+  opts.on( '-h', '--help', 'Display this screen' ) do
+    puts opts
+    exit
+  end
+
+end
+
+
+# parse command line arguments
+optparse.parse!
+
+
+# parse synapse config file
+begin
+  config = JSON::parse(File.read(options[:config]))
+rescue Errno::ENOENT => e
+  raise ArgumentError, "config file does not exist:\n#{e.inspect}"
+rescue Errno::EACCES => e
+  raise ArgumentError, "could not open config file:\n#{e.inspect}"
+rescue JSON::ParserError => e
+  raise "config file #{options[:config]} is not json:\n#{e.inspect}" 
+end
+
+
+# create synapse object
+s = Synapse::Synapse.new config
+
+# start synapse
+s.run
+
+
+puts "exiting synapse"

data/chef/converge

@@ -0,0 +1,4 @@
+#!/bin/bash -e
+
+cd `dirname $0`
+sudo /opt/vagrant_ruby/bin/chef-solo -c run.rb -j run.json

data/chef/cookbooks/lxc/recipes/default.rb

@@ -0,0 +1,2 @@
+package 'lxc'
+

data/chef/cookbooks/synapse/attributes/default.rb

@@ -0,0 +1 @@
+default.synapse.dir = '/opt/synapse'

data/chef/cookbooks/synapse/recipes/default.rb

@@ -0,0 +1,6 @@
+package 'haproxy'
+gem_package 'bundler'
+
+directory node.synapse.dir do 
+  recursive true
+end

data/chef/run.json

@@ -0,0 +1,8 @@
+{
+  "run_list": [
+    "recipe[synapse]"
+  ],
+  "synapse": {
+  }
+}
+

data/chef/run.rb

@@ -0,0 +1,2 @@
+this_dir = File.dirname(File.expand_path __FILE__)
+cookbook_path File.join(this_dir,'cookbooks')