nerve 0.3.0 → 0.5.0

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+cache: bundler
+rvm:
+  - 1.9.3
+

data/CONTRIBUTING.md

@@ -0,0 +1,14 @@
+# Contributing Guidelines #
+
+Thanks for contributing to SmartStack!
+If you're opening a new PR, please ask for a merge into our `pull_requests` branch -- *not* `master`.
+This will allow us avoid a back-and-forth by quickly accepting your PR and then making minor changes or doing testing before merging into `master`.
+
+## Writing Checks ##
+
+We welcome additional service checks into the core of nerve.
+However, your checks must follow a few guidelines or they will not be accepted:
+
+* be sure to respect timeouts; checks that do not time-out will not be accepted
+* do NOT shell out; this becomes very expensive when done frequently
+* use well-tested, stable, core libraries whenever possible

data/Gemfile.lock

@@ -1,23 +1,33 @@
 PATH
   remote: .
   specs:
-    nerve (0.3.0)
-      bunny (= 1.0.0.rc2)
+    nerve (0.5.0)
+      bunny (= 1.1.0)
       zk (~> 1.9.2)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    amq-protocol (1.8.0)
-    bunny (1.0.0.rc2)
-      amq-protocol (>= 1.8.0)
+    amq-protocol (1.9.2)
+    bunny (1.1.0)
+      amq-protocol (>= 1.9.2)
+    diff-lcs (1.2.5)
     little-plugger (1.1.3)
     logging (1.7.2)
       little-plugger (>= 1.1.3)
-    zk (1.9.2)
+    rake (10.1.1)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.4)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.4)
+    zk (1.9.3)
       logging (~> 1.7.2)
       zookeeper (~> 1.4.0)
-    zookeeper (1.4.7)
+    zookeeper (1.4.8)
 
 PLATFORMS
   java
@@ -25,3 +35,5 @@ PLATFORMS
 
 DEPENDENCIES
   nerve!
+  rake
+  rspec

data/README.md

@@ -1,3 +1,5 @@
+[![Build Status](https://travis-ci.org/airbnb/nerve.png?branch=master)](https://travis-ci.org/airbnb/nerve)
+
 # Nerve
 
 Nerve is a utility for tracking the status of machines and services.
@@ -8,7 +10,7 @@ The combination of Nerve and [Synapse](https://github.com/airbnb/synapse) make s
 ## Motivation ##
 
 We already use [Synapse](https://github.com/airbnb/synapse) to discover remote services.
-However, those services needed boilerplate code to register themselves in [Zookeeper](zookeeper.apache.org/).
+However, those services needed boilerplate code to register themselves in [Zookeeper](http://zookeeper.apache.org/).
 Nerve simplifies underlying services, enables code reuse, and allows us to create a more composable system.
 It does so by factoring out the boilerplate into it's own application, which independenly handles monitoring and reporting.
 
@@ -36,7 +38,7 @@ It is usually called `nerve.conf.json`.
 An example config file is available in `example/nerve.conf.json`.
 The config file is composed of two main sections:
 
-* `instance_id`: the name under which your services will be registered in zookeeper
+* `instance_id`: the name nerve will submit when registering services; makes debugging easier
 * `services`: the hash (from service name to config) of the services nerve will be monitoring
 * `service_conf_dir`: path to a directory in which each json file will be interpreted as a service with the basename of the file minus the .json extension
 
@@ -46,13 +48,19 @@ Each service that nerve will be monitoring is specified in the `services` hash.
 The key is the name of the service, and the value is a configuration hash telling nerve how to monitor the service.
 The configuration contains the following options:
 
-* `port`: the default port for service checks; nerve will submit this the address `instance_id:port` to Zookeeper
-* `host`: the default host on which to make service checks; you should make this your *public* ip if you want to make sure your service is publically accessible
-* `zk_hosts`: a list of the zookeeper hosts comprising the [ensemble](https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup) that nerve will submit registration to
-* `zk_path`: the path (or [znode](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_zkDataModel_znodes)) where the registration will be created; nerve will create the [ephemeral node](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Ephemeral+Nodes) that is the registration as a child of this path
+* `host`: the default host on which to make service checks; you should make this your *public* ip to ensure your service is publically accessible
+* `port`: the default port for service checks; nerve will report the `host`:`port` combo via your chosen reporter
+* `reporter_type`: the mechanism used to report up/down information; depending on the reporter you choose, additional parameters may be required. Defaults to `zookeeper`
 * `check_interval`: the frequency with which service checks will be initiated; defaults to `500ms`
 * `checks`: a list of checks that nerve will perform; if all of the pass, the service will be registered; otherwise, it will be un-registered
 
+#### Zookeeper Reporter ####
+
+If you set your `reporter_type` to `"zookeeper"` you should also set these parameters:
+
+* `zk_hosts`: a list of the zookeeper hosts comprising the [ensemble](https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup) that nerve will submit registration to
+* `zk_path`: the path (or [znode](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_zkDataModel_znodes)) where the registration will be created; nerve will create the [ephemeral node](https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#Ephemeral+Nodes) that is the registration as a child of this path
+
 ### Checks ###
 
 The core of nerve is a set of service checks.

data/Rakefile

@@ -1 +1,7 @@
 require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :test => :spec
+task :default => :spec

data/bin/nerve

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 
-require 'json'
+require 'yaml'
 require 'optparse'
 
 require 'nerve'
@@ -20,7 +20,13 @@ EOB
     options[:config] = key
   end
 
-  opts.on( '-h', '--help', 'Display this screen' ) do
+  options[:instance_id] = ENV['NERVE_INSTANCE_ID']
+  opts.on('-i instance_id','--instance_id instance_id', String,
+    'reported as `name` to ZK; overrides instance id from config file') do |key,value|
+    options[:instance_id] = key
+  end
+
+  opts.on('-h', '--help', 'Display this screen') do
     puts opts
     exit
   end
@@ -34,15 +40,15 @@ optparse.parse!
 def parseconfig(filename)
   # parse synapse config file
   begin
-    c = JSON::parse(File.read(filename))
+    c = YAML::parse(File.read(filename))
   rescue Errno::ENOENT => e
     raise ArgumentError, "config file does not exist:\n#{e.inspect}"
   rescue Errno::EACCES => e
     raise ArgumentError, "could not open config file:\n#{e.inspect}"
-  rescue JSON::ParserError => e
-    raise "config file #{filename} is not json:\n#{e.inspect}"
+  rescue YAML::ParseError => e
+    raise "config file #{filename} is not yaml:\n#{e.inspect}"
   end
-  return c
+  return c.to_ruby
 end
 
 config = parseconfig(options[:config])
@@ -53,8 +59,12 @@ if config.has_key?('service_conf_dir')
   if ! Dir.exists?(cdir) then
     raise "service conf dir does not exist:#{cdir}"
   end
-  cfiles = Dir.glob(File.join(cdir, '*.json'))
-  cfiles.each { |x| config['services'][File.basename(x[/(.*)\.json$/, 1])] = parseconfig(x) }
+  cfiles = Dir.glob(File.join(cdir, '*.{yaml,json}'))
+  cfiles.each { |x| config['services'][File.basename(x[/(.*)\.(yaml|json)$/, 1])] = parseconfig(x) }
+end
+
+if options[:instance_id] && !options[:instance_id].empty?
+  config['instance_id'] = options[:instance_id]
 end
 
 # create nerve object

data/example/nerve.conf.json

@@ -3,8 +3,9 @@
   "service_conf_dir": "example/nerve_services",
   "services": {
     "your_http_service": {
+      "host": "1.2.3.4",
       "port": 3000,
-      "host": "127.0.0.1",
+      "reporter_type": "zookeeper",
       "zk_hosts": ["localhost:2181"],
       "zk_path": "/nerve/services/your_http_service/services",
       "check_interval": 2,
@@ -19,8 +20,9 @@
       ]
     },
     "your_tcp_service": {
+      "host": "1.2.3.4",
       "port": 6379,
-      "host": "127.0.0.1",
+      "reporter_type": "zookeeper",
       "zk_hosts": ["localhost:2181"],
       "zk_path": "/nerve/services/your_tcp_service/services",
       "check_interval": 2,
@@ -34,8 +36,9 @@
       ]
     },
     "rabbitmq_service": {
+      "host": "1.2.3.4",
       "port": 5672,
-      "host": "127.0.0.1",
+      "reporter_type": "zookeeper",
       "zk_hosts": ["localhost:2181"],
       "zk_path": "/nerve/services/your_rabbitmq_service/services",
       "check_interval": 2,

data/example/nerve_services/service1.json

@@ -1,6 +1,7 @@
 {
+  "host": "1.2.3.4",
   "port": 3000,
-  "host": "127.0.0.1",
+  "reporter_type": "zookeeper",
   "zk_hosts": ["localhost:2181"],
   "zk_path": "/nerve/services/your_http_service/services",
   "check_interval": 2,

data/lib/nerve/reporter/base.rb

@@ -0,0 +1,26 @@
+
+class Nerve::Reporter
+  class Base
+    include Nerve::Utils
+    include Nerve::Logging
+
+    def initialize(opts)
+    end
+
+    def start
+    end
+
+    def report_up
+    end
+
+    def report_down
+    end
+
+    def update_data(new_data='')
+    end
+
+    def ping?
+    end
+  end
+end
+

data/lib/nerve/reporter/zookeeper.rb

@@ -0,0 +1,70 @@
+require 'nerve/reporter/base'
+require 'zk'
+
+class Nerve::Reporter
+  class Zookeeper < Base
+    def initalize(service)
+      %w{zk_hosts zk_path instance_id host port}.each do |required|
+        raise ArgumentError, "missing required argument #{required} for new service watcher" unless service[required]
+      end
+      @path = service['zk_hosts'].shuffle.join(',') + opts['zk_path']
+      @data = parse_data({'host' => service['host'], 'port' => service['port'], 'name' => service['instance_id']})
+
+      @key = "/#{service['instance_id']}_"
+      @full_key = nil
+    end
+
+    def start()
+      log.info "nerve: waiting to connect to zookeeper at #{@path}"
+      @zk = ZK.new(@path)
+
+      log.info "nerve: successfully created zk connection to #{@path}"
+    end
+
+    def report_up()
+      zk_save
+    end
+
+    def report_down
+      zk_delete
+    end
+
+    def update_data(new_data='')
+      @data = parse_data(new_data)
+      zk_save
+    end
+
+    def ping?
+      return @zk.ping?
+    end
+
+    private
+
+    def zk_delete
+      if @full_key
+        @zk.delete(@full_key, :ignore => :no_node)
+        @full_key = nil
+      end
+    end
+
+    def zk_create
+      @full_key = @zk.create(@key, :data => @data, :mode => :ephemeral_sequential)
+    end
+
+    def zk_save
+      return zk_create unless @full_key
+
+      begin
+        @zk.set(@full_key, @data)
+      rescue ZK::Exceptions::NoNode
+        zk_create
+      end
+    end
+
+    def parse_data(data)
+      return data if data.class == String
+      return data.to_json
+    end
+  end
+end
+

data/lib/nerve/reporter.rb

@@ -1,64 +1,15 @@
-require 'zk'
 
 module Nerve
   class Reporter
-    include Utils
-    include Logging
-
-    def initialize(opts)
-      %w{hosts path key}.each do |required|
-        raise ArgumentError, "you need to specify required argument #{required}" unless opts[required]
-      end
-
-      @path = opts['hosts'].shuffle.join(',') + opts['path']
-      @data = parse_data(opts['data'] || '')
-      @key = opts['key']
-      @key.insert(0,'/') unless @key[0] == '/'
-    end
-
-    def start()
-      log.info "nerve: waiting to connect to zookeeper at #{@path}"
-      @zk = ZK.new(@path)
-
-      log.info "nerve: successfully created zk connection to #{@path}"
-    end
-
-    def report_up()
-      zk_save
-    end
-
-    def report_down
-      zk_delete
-    end
-
-    def update_data(new_data='')
-      @data = parse_data(new_data)
-      zk_save
-    end
-
-    def ping?
-      return @zk.ping?
-    end
-
-    private
-
-    def zk_delete
-      @zk.delete(@key, :ignore => :no_node)
-    end
-
-    def zk_save
-      log.debug "nerve: writing data #{@data.class} to zk at #{@key} with #{@data.inspect}"
-      begin
-        @zk.set(@key,@data)
-      rescue ZK::Exceptions::NoNode => e
-        @zk.create(@key,:data => @data, :mode => :ephemeral)
+    def self.new_from_service(service)
+      type = service['reporter_type'] || 'zookeeper'
+      reporter = begin
+        require "nerve/reporter/#{type.downcase}"
+        self.const_get(type.downcase.capitalize)
+      rescue Exception => e
+        raise ArgumentError, "specified a reporter_type of #{type}, which could not be found: #{e}"
       end
+      reporter.new(service)
     end
-
-    def parse_data(data)
-      return data if data.class == String
-      return data.to_json
-    end
-
   end
 end

data/lib/nerve/service_watcher.rb

@@ -11,19 +11,14 @@ module Nerve
       log.debug "nerve: creating service watcher object"
 
       # check that we have all of the required arguments
-      %w{name instance_id host port zk_hosts zk_path}.each do |required|
+      %w{name instance_id host port}.each do |required|
         raise ArgumentError, "missing required argument #{required} for new service watcher" unless service[required]
       end
 
       @name = service['name']
 
       # configure the reporter, which we use for talking to zookeeper
-      @reporter = Reporter.new({
-          'hosts' => service['zk_hosts'],
-          'path' => service['zk_path'],
-          'key' => "#{service['instance_id']}_#{@name}",
-          'data' => {'host' => service['host'], 'port' => service['port']},
-        })
+      @reporter = Reporter.new_from_service(service)
 
       # instantiate the checks for this service
       @service_checks = []
@@ -52,9 +47,7 @@ module Nerve
     def run()
       log.info "nerve: starting service watch #{@name}"
 
-      # begin by reporting down
       @reporter.start()
-      @reporter.report_down
       was_up = false
 
       until $EXIT

data/lib/nerve/version.rb

@@ -1,3 +1,3 @@
 module Nerve
-  VERSION = "0.3.0"
+  VERSION = "0.5.0"
 end

data/lib/nerve.rb

@@ -42,9 +42,12 @@ module Nerve
 
       begin
         sleep
-      rescue
-        log.debug 'nerve: sleep interrupted; exiting'
+      rescue StandardError => e
+        log.error 'nerve: encountered unexpected exception #{e.inspect} in main thread'
+        raise e
+      ensure
         $EXIT = true
+        log.warn 'nerve: reaping all watchers'
         @watchers.each do |name, watcher_thread|
           reap_watcher(name)
         end

data/nerve.gemspec

@@ -18,5 +18,8 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
 
   gem.add_runtime_dependency "zk", "~> 1.9.2"
-  gem.add_runtime_dependency "bunny", "= 1.0.0.rc2"
+  gem.add_runtime_dependency "bunny", "= 1.1.0"
+
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "rspec"
 end

data/spec/lib/nerve/reporter_spec.rb

@@ -0,0 +1,27 @@
+require 'spec_helper'
+
+describe Nerve::Reporter do
+  let(:subject) { {
+      'zk_hosts' => ['zkhost1', 'zkhost2'],
+      'zk_path' => 'zk_path',
+      'instance_id' => 'instance_id',
+      'host' => 'host',
+      'port' => 'port'
+    }
+  }
+  it 'can new_from_service' do
+    expect(Nerve::Reporter::Zookeeper).to receive(:new).with(subject).and_return('kerplunk')
+    expect(Nerve::Reporter.new_from_service(subject)).to eq('kerplunk')
+  end
+  it 'actually constructs an instance of a specific backend' do
+    expect(Nerve::Reporter.new_from_service(subject).is_a?(Nerve::Reporter::Zookeeper)).to eql(true)
+  end
+  it 'the reporter backend inherits from the base class' do
+    expect(Nerve::Reporter.new_from_service(subject).is_a?(Nerve::Reporter::Base)).to eql(true)
+  end
+  it 'throws ArgumentError if you ask for a reporter type which does not exist' do
+    subject['reporter_type'] = 'does_not_exist'
+    expect { Nerve::Reporter.new_from_service(subject) }.to raise_error(ArgumentError)
+  end
+end
+

data/spec/lib/nerve/reporter_zookeeper_spec.rb

@@ -0,0 +1,17 @@
+require 'spec_helper'
+require 'nerve/reporter/zookeeper'
+
+describe Nerve::Reporter::Zookeeper do
+  let(:subject) { {
+      'zk_hosts' => ['zkhost1', 'zkhost2'],
+      'zk_path' => 'zk_path',
+      'instance_id' => 'instance_id',
+      'host' => 'host',
+      'port' => 'port'
+    }
+  }
+  it 'actually constructs an instance' do
+    expect(Nerve::Reporter::Zookeeper.new(subject).is_a?(Nerve::Reporter::Zookeeper)).to eql(true)
+  end
+end
+

data/spec/spec_helper.rb

@@ -0,0 +1,20 @@
+# This file was generated by the `rspec --init` command. Conventionally, all
+# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
+# Require this file using `require "spec_helper"` to ensure that it is only
+# loaded once.
+#
+# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
+require "#{File.dirname(__FILE__)}/../lib/nerve"
+
+RSpec.configure do |config|
+  config.treat_symbols_as_metadata_keys_with_true_values = true
+  config.run_all_when_everything_filtered = true
+  config.filter_run :focus
+  config.include Config
+
+  # Run specs in random order to surface order dependencies. If you find an
+  # order dependency and want to debug it, you can fix the order by providing
+  # the seed, which is printed after each run.
+  #     --seed 1234
+  config.order = 'random'
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: nerve
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
   prerelease: 
 platform: ruby
 authors:
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-27 00:00:00.000000000 Z
+date: 2014-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: zk
@@ -36,7 +36,7 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 1.0.0.rc2
+        version: 1.1.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -44,7 +44,39 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 1.0.0.rc2
+        version: 1.1.0
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: description
 email:
 - martin.rhoads@airbnb.com
@@ -57,6 +89,8 @@ files:
 - .gitignore
 - .mailmap
 - .nerve.rc
+- .travis.yml
+- CONTRIBUTING.md
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
@@ -69,6 +103,8 @@ files:
 - lib/nerve.rb
 - lib/nerve/log.rb
 - lib/nerve/reporter.rb
+- lib/nerve/reporter/base.rb
+- lib/nerve/reporter/zookeeper.rb
 - lib/nerve/ring_buffer.rb
 - lib/nerve/service_watcher.rb
 - lib/nerve/service_watcher/base.rb
@@ -78,6 +114,10 @@ files:
 - lib/nerve/utils.rb
 - lib/nerve/version.rb
 - nerve.gemspec
+- spec/.gitkeep
+- spec/lib/nerve/reporter_spec.rb
+- spec/lib/nerve/reporter_zookeeper_spec.rb
+- spec/spec_helper.rb
 homepage: ''
 licenses: []
 post_install_message: 
@@ -102,4 +142,8 @@ rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: summary
-test_files: []
+test_files:
+- spec/.gitkeep
+- spec/lib/nerve/reporter_spec.rb
+- spec/lib/nerve/reporter_zookeeper_spec.rb
+- spec/spec_helper.rb