nerve 0.0.1 → 0.2.1

data/.gitignore

@@ -15,3 +15,8 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+*.sw?
+\#*\#
+.\#*
+vendor
+.vagrant

data/.nerve.rc

@@ -0,0 +1,2 @@
+export DATA_BAG_DIR=/Users/martin/Dropbox/airbnb/src/chef/data_bags
+export COOKBOOK_DIR=/Users/martin/Dropbox/airbnb/src/chef/cookbooks

data/Gemfile

@@ -1,4 +1,2 @@
 source 'https://rubygems.org'
-
-# Specify your gem's dependencies in nerve.gemspec
 gemspec

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Martin Rhoads
+Copyright (c) 2013 Airbnb, Inc.
 
 MIT License
 
@@ -19,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,7 +1,21 @@
 # Nerve
 
+Nerve is a utility for tracking the status of machines and services.
+It runs locally on the boxes which make up a distributed system, and reports state information to a distributed key-value store.
+At Airbnb, we use Zookeeper as our key-value store.
+The combination of Nerve and [Synapse](https://github.com/airbnb/synapse) make service discovery in the cloud easy!
 
-## Installation
+## Motivation ##
+
+We already use [Synapse](https://github.com/airbnb/synapse) to discover remote services.
+However, those services needed boilerplate code to register themselves in [Zookeeper](zookeeper.apache.org/).
+Nerve simplifies underlying services, enables code reuse, and allows us to create a more composable system.
+It does so by factoring out the boilerplate into it's own application, which independenly handles monitoring and reporting.
+
+Beyond those benefits, nerve also acts as a general watchdog on systems.
+The information it reports can be used to take action from a certralized automation center: action like scaling distributed systems up or down or alerting ops or engineering about downtime.
+
+## Installation ##
 
 Add this line to your application's Gemfile:
 
@@ -15,8 +29,42 @@ Or install it yourself as:
 
     $ gem install nerve
 
-## Usage
+## Configuration ##
+
+Nerve depends on a single configuration file, in json format.
+It is usually called `nerve.conf.json`.
+An example config file is available in `example/nerve.conf.json`.
+The config file is composed of two main sections:
+
+* `instance_id`: the name under which your services will be registered in zookeeper
+* `services`: the hash (from service name to config) of the services nerve will be monitoring
+
+### Services Config ###
+
+Each service that nerve will be monitoring is specified in the `services` hash.
+The key is the name of the service, and the value is a configuration hash telling nerve how to monitor the service.
+The configuration contains the following options:
+
+* `port`: the default port for service checks; nerve will submit this the address `instance_id:port` to Zookeeper
+* `host`: the default host on which to make service checks; you should make this your *public* ip if you want to make sure your service is publically accessible
+* `zk_hosts`: a list of the zookeeper hosts comprising the [ensemble](https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup) that nerve will submit registration to
+* `zk_path`: the path (or [znode](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_zkDataModel_znodes)) where the registration will be created; nerve will create the [ephemeral node](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Ephemeral+Nodes) that is the registration as a child of this path
+* `check_interval`: the frequency with which service checks will be initiated; defaults to `500ms`
+* `checks`: a list of checks that nerve will perform; if all of the pass, the service will be registered; otherwise, it will be un-registered
+
+### Checks ###
+
+The core of nerve is a set of service checks.
+Each service can define a number of checks, and all of them must pass for the service to be registered.
+Although the exact parameters passed to each check are different, all take a number of common arguments:
 
+* `type`: (required) the kind of check; you can see available check types in the `lib/nerve/service_watcher` dir of this repo
+* `name`: (optional) a descriptive, human-readable name for the check; it will be auto-generated based on the other parameters if not specified
+* `host`: (optional) the host on which the check will be performed; defaults to the `host` of the service to which the check belongs
+* `port`: (optional) the port on which the check will be performed; like `host`, it defaults to the `port` of the service
+* `timeout`: (optional) maximum time the check can take; defaults to `100ms`
+* `rise`: (optional) how many consecutive checks must pass before the check is considered passing; defaults to 1
+* `fall`: (optional) how many consecutive checks must fail before the check is considered failing; defaults to 1
 
 ## Contributing
 

data/Vagrantfile

@@ -0,0 +1,121 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+unless ENV['COOKBOOK_DIR'] and ENV['DATA_BAG_DIR']
+  STDERR.puts "you need to set COOKBOOK_DIR and DATA_BAG_DIR as environment variables"
+  Kernel.exit 1
+end
+
+this_dir = File.dirname(File.expand_path __FILE__)
+
+STDERR.puts "mounting #{this_dir} as /root/this_dir"
+
+Vagrant::Config.run do |config|
+  # All Vagrant configuration is done here. The most common configuration
+  # options are documented and commented below. For a complete reference,
+  # please see the online documentation at vagrantup.com.
+
+  # Every Vagrant virtual environment requires a box to build off of.
+  config.vm.box = "precise64"
+
+  # The url from where the 'config.vm.box' box will be fetched if it
+  # doesn't already exist on the user's system.
+  # config.vm.box_url = "http://domain.com/path/to/above.box"
+
+  config.vm.box_url = 'http://files.vagrantup.com/precise64.box'
+
+  # Boot with a GUI so you can see the screen. (Default is headless)
+  # config.vm.boot_mode = :gui
+
+  # Assign this VM to a host-only network IP, allowing you to access it
+  # via the IP. Host-only networks can talk to the host machine as well as
+  # any other machines on the same network, but cannot be accessed (through this
+  # network interface) by any external networks.
+  # config.vm.network :hostonly, "192.168.33.10"
+
+  # Assign this VM to a bridged network, allowing you to connect directly to a
+  # network using the host's network device. This makes the VM appear as another
+  # physical device on your network.
+  # config.vm.network :bridged
+
+  # Forward a port from the guest to the host, which allows for outside
+  # computers to access the VM, whereas host only networking does not.
+  # config.vm.forward_port 80, 8080
+
+  # Share an additional folder to the guest VM. The first argument is
+  # an identifier, the second is the path on the guest to mount the
+  # folder, and the third is the path on the host to the actual folder.
+  # config.vm.share_folder "v-data", "/vagrant_data", "../data"
+
+  config.vm.share_folder 'this_dir', '/this_dir', this_dir
+
+  # Enable provisioning with Puppet stand alone.  Puppet manifests
+  # are contained in a directory path relative to this Vagrantfile.
+  # You will need to create the manifests directory and a manifest in
+  # the file base.pp in the manifests_path directory.
+  #
+  # An example Puppet manifest to provision the message of the day:
+  #
+  # # group { "puppet":
+  # #   ensure => "present",
+  # # }
+  # #
+  # # File { owner => 0, group => 0, mode => 0644 }
+  # #
+  # # file { '/etc/motd':
+  # #   content => "Welcome to your Vagrant-built virtual machine!
+  # #               Managed by Puppet.\n"
+  # # }
+  #
+  # config.vm.provision :puppet do |puppet|
+  #   puppet.manifests_path = "manifests"
+  #   puppet.manifest_file  = "base.pp"
+  # end
+
+  # Enable provisioning with chef solo, specifying a cookbooks path, roles
+  # path, and data_bags path (all relative to this Vagrantfile), and adding 
+  # some recipes and/or roles.
+  #
+  config.vm.provision :chef_solo do |chef|
+    # chef.cookbooks_path = "../my-recipes/cookbooks"
+    # chef.roles_path = "../my-recipes/roles"
+#    chef.data_bags_path = '/tmp/foo'
+    # chef.add_recipe "mysql"
+    # chef.add_role "web"
+  
+    # You may also specify custom JSON attributes:
+    # chef.json = { :mysql_password => "foo" }
+
+    chef.data_bags_path = ENV['DATA_BAG_DIR']
+    chef.cookbooks_path = ENV['COOKBOOK_DIR']
+    chef.add_recipe 'vagrant'
+    chef.add_recipe 'nerve'
+  end
+
+  # Enable provisioning with chef server, specifying the chef server URL,
+  # and the path to the validation key (relative to this Vagrantfile).
+  #
+  # The Opscode Platform uses HTTPS. Substitute your organization for
+  # ORGNAME in the URL and validation key.
+  #
+  # If you have your own Chef Server, use the appropriate URL, which may be
+  # HTTP instead of HTTPS depending on your configuration. Also change the
+  # validation key to validation.pem.
+  #
+  # config.vm.provision :chef_client do |chef|
+  #   chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
+  #   chef.validation_key_path = "ORGNAME-validator.pem"
+  # end
+  #
+  # If you're using the Opscode platform, your validator client is
+  # ORGNAME-validator, replacing ORGNAME with your organization name.
+  #
+  # IF you have your own Chef Server, the default validation client name is
+  # chef-validator, unless you changed the configuration.
+  #
+  #   chef.validation_client_name = "ORGNAME-validator"
+
+  # Vagrant::Config.run do |config|
+  #   config.vm.provision :shell, :path => "vagrant/init.sh"
+  # end
+end

data/bin/nerve

@@ -0,0 +1,56 @@
+#!/usr/bin/env ruby
+
+require 'json'
+require 'optparse'
+
+require 'nerve'
+
+options={}
+
+# set command line options
+optparse = OptionParser.new do |opts|
+  opts.banner =<<EOB
+Welcome to nerve
+
+Usage: nerve --config /path/to/nerve/config
+EOB
+
+  options[:config] = ENV['NERVE_CONFIG']
+  opts.on('-c config','--config config', String, 'path to nerve config') do |key,value|
+    options[:config] = key
+  end
+
+  opts.on( '-h', '--help', 'Display this screen' ) do
+    puts opts
+    exit
+  end
+
+end
+
+
+# parse command line arguments
+optparse.parse!
+
+
+# parse nerve config file
+begin
+  config = JSON::parse(File.read(options[:config]))
+rescue TypeError => e
+  raise ArgumentError, "you must pass in a '--config' option"
+rescue Errno::ENOENT => e
+  raise ArgumentError, "config file does not exist:\n#{e.inspect}"
+rescue Errno::EACCES => e
+  raise ArgumentError, "could not open config file:\n#{e.inspect}"
+rescue JSON::ParserError => e
+  raise "config file #{options[:config]} is not json:\n#{e.inspect}"
+end
+
+
+# create nerve object
+s = Nerve::Nerve.new config
+
+# start nerve
+s.run
+
+
+puts "exiting nerve"

data/example/nerve.conf.json

@@ -0,0 +1,50 @@
+{
+  "instance_id": "mymachine",
+  "services": {
+    "your_http_service": {
+      "port": 3000,
+      "host": "127.0.0.1",
+      "zk_hosts": ["localhost:2181"],
+      "zk_path": "/nerve/services/your_http_service/services",
+      "check_interval": 2,
+      "checks": [
+        {
+          "type": "http",
+          "uri": "/health",
+          "timeout": 0.2,
+          "rise": 3,
+          "fall": 2
+        }
+      ]
+    },
+    "your_tcp_service": {
+      "port": 6379,
+      "host": "127.0.0.1",
+      "zk_hosts": ["localhost:2181"],
+      "zk_path": "/nerve/services/your_tcp_service/services",
+      "check_interval": 2,
+      "checks": [
+        {
+          "type": "tcp",
+          "timeout": 0.2,
+          "rise": 3,
+          "fall": 2
+        }
+      ]
+    },
+    "rabbitmq_service": {
+      "port": 5672,
+      "host": "127.0.0.1",
+      "zk_hosts": ["localhost:2181"],
+      "zk_path": "/nerve/services/your_rabbitmq_service/services",
+      "check_interval": 2,
+      "checks": [
+        {
+          "type": "rabbitmq",
+          "username": "guest",
+          "password": "guest"
+        }
+      ]
+    }
+  }
+}

data/lib/nerve/log.rb

@@ -0,0 +1,24 @@
+module Nerve
+  module Logging
+
+    def log
+      @logger ||= Logging.logger_for(self.class.name)
+    end
+
+    # Use a hash class-ivar to cache a unique Logger per class:
+    @loggers = {}
+
+    class << self
+      def logger_for(classname)
+        @loggers[classname] ||= configure_logger_for(classname)
+      end
+
+      def configure_logger_for(classname)
+        logger = Logger.new(STDERR)
+        logger.level = Logger::INFO unless ENV['DEBUG']
+        logger.progname = classname
+        return logger
+      end
+    end
+  end
+end

data/lib/nerve/reporter.rb

@@ -0,0 +1,64 @@
+require 'zk'
+
+module Nerve
+  class Reporter
+    include Utils
+    include Logging
+
+    def initialize(opts)
+      %w{hosts path key}.each do |required|
+        raise ArgumentError, "you need to specify required argument #{required}" unless opts[required]
+      end
+
+      @path = opts['hosts'].shuffle.join(',') + opts['path']
+      @data = parse_data(opts['data'] || '')
+      @key = opts['key']
+      @key.insert(0,'/') unless @key[0] == '/'
+    end
+
+    def start()
+      log.info "nerve: waiting to connect to zookeeper at #{@path}"
+      @zk = ZK.new(@path)
+
+      log.info "nerve: successfully created zk connection to #{@path}"
+    end
+
+    def report_up()
+      zk_save
+    end
+
+    def report_down
+      zk_delete
+    end
+
+    def update_data(new_data='')
+      @data = parse_data(new_data)
+      zk_save
+    end
+
+    def ping?
+      return @zk.ping?
+    end
+
+    private
+
+    def zk_delete
+      @zk.delete(@key, :ignore => :no_node)
+    end
+
+    def zk_save
+      log.debug "nerve: writing data #{@data.class} to zk at #{@key} with #{@data.inspect}"
+      begin
+        @zk.set(@key,@data)
+      rescue ZK::Exceptions::NoNode => e
+        @zk.create(@key,:data => @data, :mode => :ephemeral)
+      end
+    end
+
+    def parse_data(data)
+      return data if data.class == String
+      return data.to_json
+    end
+
+  end
+end

data/lib/nerve/ring_buffer.rb

@@ -0,0 +1,30 @@
+module Nerve
+  class RingBuffer < Array
+    alias_method :array_push, :push
+    alias_method :array_element, :[]
+    
+    def initialize( size )
+      @ring_size = size.to_i
+      super( @ring_size )
+    end
+    
+    def average
+      self.inject(0.0) { |sum, el| sum + el } / self.size
+    end
+    
+    def push( element )
+      if length == @ring_size
+        shift # loose element
+      end
+      array_push element
+    end
+    
+    # Access elements in the RingBuffer
+    #
+    # offset will be typically negative!
+    #
+    def []( offset = 0 )
+      return self.array_element( - 1 + offset )
+    end
+  end
+end

data/lib/nerve/service_watcher/base.rb

@@ -0,0 +1,51 @@
+module Nerve
+  module ServiceCheck
+    class BaseServiceCheck
+      include Utils
+      include Logging
+
+      def initialize(opts={})
+        @timeout = opts['timeout'] ? opts['timeout'].to_f : 0.1
+        @rise    = opts['rise']    ? opts['rise'].to_i    : 1
+        @fall    = opts['fall']    ? opts['fall'].to_i    : 1
+        @name    = opts['name']    ? opts['name']         : "undefined"
+
+        @check_buffer = RingBuffer.new([@rise, @fall].max)
+        @last_result = nil
+      end
+
+      def up?
+        # do the check
+        check_result = !!ignore_errors do
+          check
+        end
+
+        # this is the first check -- initialize buffer
+        if @last_result == nil
+          @last_result = check_result
+          @check_buffer.size.times {@check_buffer.push check_result}
+          log.info "nerve: service check #{@name} initial check returned #{check_result}"
+        end
+
+        log.debug "nerve: service check #{@name} returned #{check_result}"
+        @check_buffer.push(check_result)
+
+        # we've failed if the last @fall times are false
+        unless @check_buffer.last(@fall).reduce(:|)
+          log.info "nerve: service check #{@name} transitions to down after #{@fall} failures" if @last_result
+          @last_result = false
+        end
+
+        # we've succeeded if the last @rise times is true
+        if @check_buffer.last(@rise).reduce(:&)
+          log.info "nerve: service check #{@name} transitions to up after #{@rise} successes" unless @last_result
+          @last_result = true
+        end
+
+        # otherwise return the last result
+        return @last_result
+      end
+    end
+  end
+end
+

data/lib/nerve/service_watcher/http.rb

@@ -0,0 +1,62 @@
+require 'nerve/service_watcher/base'
+
+module Nerve
+  module ServiceCheck
+    require 'net/http'
+
+    class HttpServiceCheck < BaseServiceCheck
+      def initialize(opts={})
+        super
+
+        %w{port uri}.each do |required|
+          raise ArgumentError, "missing required argument #{required} in http check" unless
+            opts[required]
+          instance_variable_set("@#{required}",opts[required])
+        end
+
+        @host        = opts['host'] || '127.0.0.1'
+        @ssl         = opts['ssl']  || false
+
+        @read_timeout = opts['read_timeout'] || @timeout
+        @open_timeout = opts['open_timeout'] || 0.2
+        @ssl_timeout  = opts['ssl_timeout']  || 0.2
+
+        @name        = "http-#{@host}:#{@port}#{@uri}"
+      end
+
+      def check
+        log.debug "running health check #{@name}"
+
+        connection = get_connection
+        response = connection.get(@uri)
+        code = response.code.to_i
+
+        log.debug "nerve: check #{@name} got response code #{code}"
+        if code >= 200 and code < 300
+          return true
+        else
+          return false
+        end
+      end
+
+      private
+      def get_connection
+        con = Net::HTTP.new(@host, @port)
+        con.read_timeout = @read_timeout
+        con.open_timeout = @open_timeout
+
+        if @ssl
+          con.use_ssl = true
+          con.ssl_timeout = @ssl_timeout
+          con.verify_mode = OpenSSL::SSL::VERIFY_NONE
+        end
+
+        return con
+      end
+
+    end
+
+    CHECKS ||= {}
+    CHECKS['http'] = HttpServiceCheck
+  end
+end

data/lib/nerve/service_watcher/rabbitmq.rb

@@ -0,0 +1,68 @@
+require 'nerve/service_watcher/base'
+require 'bunny'
+
+module Nerve
+  module ServiceCheck
+    class RabbitMQServiceCheck < BaseServiceCheck
+      require 'socket'
+      include Socket::Constants
+
+      def initialize(opts={})
+        super
+
+        raise ArgumentError, "missing required argument 'port' in rabbitmq check" unless opts['port']
+
+        @port = opts['port']
+        @host = opts['host']     || '127.0.0.1'
+        @user = opts['username'] || 'guest'
+        @pass = opts['password'] || 'guest'
+      end
+
+      def check
+        # the idea for this check was taken from the one in rabbitmq management
+        #  -- the aliveness_test:
+        # https://github.com/rabbitmq/rabbitmq-management/blob/9a8e3d1ab5144e3f6a1cb9a4639eb738713b926d/src/rabbit_mgmt_wm_aliveness_test.erl
+        log.debug "nerve: running rabbitmq health check #{@name}"
+
+        conn = Bunny.new(
+          :host => @host,
+          :port => @port,
+          :user => @user,
+          :pass => @pass,
+          :log_file => STDERR,
+          :continuation_timeout => @timeout,
+          :automatically_recover => false,
+          :heartbeat => false,
+          :threaded => false
+        )
+
+        begin
+          conn.start
+          ch = conn.create_channel
+
+          # create a queue, publish to it
+          log.debug "nerve: publishing to rabbitmq"
+          ch.queue('nerve')
+          ch.basic_publish('nerve test message', '', 'nerve', :mandatory => true, :expiration => 2 * 1000)
+
+          # read and ack the message
+          log.debug "nerve: consuming from rabbitmq"
+          delivery_info, properties, payload = ch.basic_get('nerve', :ack => true)
+
+          if payload
+            ch.acknowledge(delivery_info.delivery_tag)
+            return true
+          else
+            log.debug "nerve: rabbitmq consumption returned no payload"
+            return false
+          end
+        ensure
+          conn.close
+        end
+      end
+    end
+
+    CHECKS ||= {}
+    CHECKS['rabbitmq'] = RabbitMQServiceCheck
+  end
+end

data/lib/nerve/service_watcher/tcp.rb

@@ -0,0 +1,56 @@
+require 'nerve/service_watcher/base'
+
+module Nerve
+  module ServiceCheck
+    class TcpServiceCheck < BaseServiceCheck
+      require 'socket'
+      include Socket::Constants
+
+      def initialize(opts={})
+        super
+
+        raise ArgumentError, "missing required argument 'port' in tcp check" unless opts['port']
+
+        @port = opts['port']
+        @host = opts['host'] || '127.0.0.1'
+
+        @address = Socket.sockaddr_in(@port, @host)
+      end
+
+      def check
+        log.debug "nerve: running TCP health check #{@name}"
+
+        # create a TCP socket
+        socket = Socket.new(AF_INET, SOCK_STREAM, 0)
+
+        begin
+          # open a non-blocking connection
+          socket.connect_nonblock(@address)
+        rescue Errno::EINPROGRESS
+          # opening a non-blocking socket will usually raise
+          # this exception. it's just connect returning immediately,
+          # so it's not really an exception, but ruby makes it into
+          # one. if we got here, we are now free to wait until the timeout
+          # expires for the socket to be writeable
+          IO.select(nil, [socket], nil, @timeout)
+
+          # we should be connected now; allow any other exception through
+          begin
+            socket.connect_nonblock(@address)
+          rescue Errno::EISCONN
+            return true
+          end
+        else
+          # we managed to connect REALLY REALLY FAST
+          log.debug "nerve: connected to non-blocking socket without an exception"
+          return true
+        ensure
+          socket.close
+        end
+      end
+    end
+
+    CHECKS ||= {}
+    CHECKS['tcp'] = TcpServiceCheck
+  end
+end

data/lib/nerve/service_watcher.rb

@@ -0,0 +1,96 @@
+require 'nerve/service_watcher/tcp'
+require 'nerve/service_watcher/http'
+require 'nerve/service_watcher/rabbitmq'
+
+module Nerve
+  class ServiceWatcher
+    include Utils
+    include Logging
+
+    def initialize(service={})
+      log.debug "nerve: creating service watcher object"
+
+      # check that we have all of the required arguments
+      %w{name instance_id host port zk_hosts zk_path}.each do |required|
+        raise ArgumentError, "missing required argument #{required} for new service watcher" unless service[required]
+      end
+
+      @name = service['name']
+
+      # configure the reporter, which we use for talking to zookeeper
+      @reporter = Reporter.new({
+          'hosts' => service['zk_hosts'],
+          'path' => service['zk_path'],
+          'key' => "#{service['instance_id']}_#{@name}",
+          'data' => {'host' => service['host'], 'port' => service['port']},
+        })
+
+      # instantiate the checks for this service
+      @service_checks = []
+      service['checks'] ||= []
+      service['checks'].each do |check|
+        check['type'] ||= "undefined"
+        begin
+          service_check_class = ServiceCheck::CHECKS[check['type']]
+        rescue
+          raise ArgumentError,
+            "invalid service check type #{check['type']}; valid types: #{ServiceCheck::CHECKS.keys.join(',')}"
+        end
+
+        check['host'] ||= service['host']
+        check['port'] ||= service['port']
+        check['name'] ||= "#{@name} #{check['type']}-#{check['host']}:#{check['port']}"
+        @service_checks << service_check_class.new(check)
+      end
+
+      # how often do we initiate service checks?
+      @check_interval = service['check_interval'] || 0.5
+
+      log.debug "nerve: created service watcher for #{@name} with #{@service_checks.size} checks"
+    end
+
+    def run()
+      log.info "nerve: starting service watch #{@name}"
+
+      # begin by reporting down
+      @reporter.start()
+      @reporter.report_down
+      was_up = false
+
+      until $EXIT
+        @reporter.ping?
+
+        # what is the status of the service?
+        is_up = check?
+        log.debug "nerve: current service status for #{@name} is #{is_up.inspect}"
+
+        if is_up != was_up
+          if is_up
+            @reporter.report_up
+            log.info "nerve: service #{@name} is now up"
+          else
+            @reporter.report_down
+            log.warn "nerve: service #{@name} is now down"
+          end
+          was_up = is_up
+        end
+
+        # wait to run more checks
+        sleep @check_interval
+      end
+    rescue StandardError => e
+      log.error "nerve: error in service watcher #{@name}: #{e}"
+      raise e
+    ensure
+      log.info "nerve: ending service watch #{@name}"
+      $EXIT = true
+    end
+
+    def check?
+      @service_checks.each do |check|
+        return false unless check.up?
+      end
+      return true
+    end
+  end
+end

data/lib/nerve/utils.rb

@@ -0,0 +1,17 @@
+module Nerve
+  module Utils
+    def safe_run(command)
+      res = `#{command}`.chomp
+      raise "command '#{command}' failed to run:\n#{res}" unless $?.success?
+    end
+
+    def ignore_errors(&block)
+      begin
+        return yield
+      rescue Object => error
+        log.debug "ignoring error #{error.inspect}"
+        return false
+      end
+    end
+  end
+end

data/lib/nerve/version.rb

@@ -1,3 +1,3 @@
 module Nerve
-  VERSION = "0.0.1"
+  VERSION = "0.2.1"
 end

data/lib/nerve.rb

@@ -1,5 +1,69 @@
-require "nerve/version"
+require 'logger'
+require 'json'
+require 'timeout'
+
+require 'nerve/version'
+require 'nerve/utils'
+require 'nerve/log'
+require 'nerve/ring_buffer'
+require 'nerve/reporter'
+require 'nerve/service_watcher'
 
 module Nerve
-  # Your code goes here...
+  class Nerve
+
+    include Logging
+
+    def initialize(opts={})
+      # set global variable for exit signal
+      $EXIT = false
+
+      # trap int signal and set exit to true
+      %w{INT TERM}.each do |signal|
+        trap(signal) do
+          $EXIT = true
+        end
+      end
+
+      log.info 'nerve: starting up!'
+
+      # required options
+      log.debug 'nerve: checking for required inputs'
+      %w{instance_id services}.each do |required|
+        raise ArgumentError, "you need to specify required argument #{required}" unless opts[required]
+      end
+
+      @instance_id = opts['instance_id']
+
+      # create service watcher objects
+      log.debug 'nerve: creating service watchers'
+      @service_watchers=[]
+      opts['services'].each do |name, config|
+        @service_watchers << ServiceWatcher.new(config.merge({'instance_id' => @instance_id, 'name' => name}))
+      end
+
+      log.debug 'nerve: completed init'
+    end
+
+    def run
+      log.info 'nerve: starting run'
+      begin
+        children = []
+
+        log.debug 'nerve: launching service check threads'
+        @service_watchers.each do |watcher|
+          children << Thread.new{watcher.run}
+        end
+
+        log.debug 'nerve: main thread done, waiting for children'
+        children.each do |child|
+          child.join
+        end
+      ensure
+        $EXIT = true
+      end
+      log.info 'nerve: exiting'
+    end
+
+  end
 end

data/nerve.gemspec

@@ -6,8 +6,8 @@ require 'nerve/version'
 Gem::Specification.new do |gem|
   gem.name          = "nerve"
   gem.version       = Nerve::VERSION
-  gem.authors       = ["Martin Rhoads"]
-  gem.email         = ["martin.rhoads@airbnb.com"]
+  gem.authors       = ["Martin Rhoads", "Igor Serebryany", "Pierre Carrier"]
+  gem.email         = ["martin.rhoads@airbnb.com", "igor.serebryany@airbnb.com"]
   gem.description   = %q{description}
   gem.summary       = %q{summary}
   gem.homepage      = ""
@@ -16,4 +16,7 @@ Gem::Specification.new do |gem|
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
+
+  gem.add_runtime_dependency "zk", "~> 1.9.2"
+  gem.add_runtime_dependency "bunny", "= 1.0.0.rc2"
 end

metadata

@@ -1,29 +1,78 @@
 --- !ruby/object:Gem::Specification
 name: nerve
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.2.1
   prerelease: 
 platform: ruby
 authors:
 - Martin Rhoads
+- Igor Serebryany
+- Pierre Carrier
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-12-22 00:00:00.000000000 Z
-dependencies: []
+date: 2013-10-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: zk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.9.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.9.2
+- !ruby/object:Gem::Dependency
+  name: bunny
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.0.0.rc2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.0.0.rc2
 description: description
 email:
 - martin.rhoads@airbnb.com
-executables: []
+- igor.serebryany@airbnb.com
+executables:
+- nerve
 extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- .nerve.rc
 - Gemfile
 - LICENSE.txt
 - README.md
 - Rakefile
+- Vagrantfile
+- bin/nerve
+- example/nerve.conf.json
 - lib/nerve.rb
+- lib/nerve/log.rb
+- lib/nerve/reporter.rb
+- lib/nerve/ring_buffer.rb
+- lib/nerve/service_watcher.rb
+- lib/nerve/service_watcher/base.rb
+- lib/nerve/service_watcher/http.rb
+- lib/nerve/service_watcher/rabbitmq.rb
+- lib/nerve/service_watcher/tcp.rb
+- lib/nerve/utils.rb
 - lib/nerve/version.rb
 - nerve.gemspec
 homepage: ''