auto-consul 0.0.1 → 0.0.2

data/README.md

@@ -14,32 +14,29 @@ Export your AWS keys into the environment in each:
     export AWS_SECRET_ACCESS_KEY=...
     ```
 
-This will allow the AWS SDK to pick them up.
+This will allow the AWS SDK to pick them up.  (Note that on an EC2
+instance, the AWS SDK should seamlessly pick up any IAM roles associated
+with the instance, so these environment variables should not be
+necessary.)
 
-The server, screen A:
+Then, run the agent via auto-consul:
 
     auto-consul -r s3://my-bucket/consul/test-cluster \
                 -a 192.168.50.100 \
-                -n server1 \
+                -n agent1 \
+                -t 60 \
                 run
 
-Then, server screen B:
+This will launch the consul agent, and emit heartbeats roughly every
+minute.  The `-r` indicates the bucket to use as the heartbeat registry.
+The `-t` specifies the interval in seconds between heartbeats.  The
+`-a` and `-n` options map to consul's native `-bind` and `-node` options,
+respectively.
 
-    while true; do
-        auto-consul -r s3://my-bucket/consul/test-cluster \
-                    -a 192.168.50.100 \
-                    -n server1 \
-                    heartbeat
-        sleep 60
-    done
-
-The first launches the agent, the latter checks its run status and
-issues a heartbeat to the specified S3 bucket.
-
-Because this is the first server, there will be no heartbeats in the
-bucket (assuming a fresh bucket/key combination).  Therefore, the agent
-will be launched in server mode, along with the bootstrap option to
-initialize the raft cluster for state management.
+Because this is the first server, there will be no extant heartbeats in the
+bucket (assuming a fresh bucket/key combination) at startup.  Therefore,
+the agent will be launched in server mode, along with the bootstrap option
+to initialize the raft cluster for state management.
 
 Look in the S3 bucket above, under "servers", and you should see
 a timestamped entry like "20140516092731-server1".  This is produced
@@ -47,11 +44,12 @@ by the "heartbeat" command and allows new agents to discover active
 members of the cluster for joining.
 
 Having seen the server heartbeat, go to the agent vagrant box, and
-do something similar.  Screen A:
+do something similar.
 
     auto-consul -r s3://my-bucket/consul/test-cluster \
                 -a 192.168.50.101 \
                 -n agent1 \
+                -t 60 \
                 run
 
 In this case, the agent will discover the server via its heartbeat.  It
@@ -59,19 +57,38 @@ will know that we have enough servers (it defaults to only wanting one;
 that's fine for dev/testing but not good for availability) and thus
 simply join as a normal agent.
 
-Screen B:
+While the server sends heartbeats both to "servers" and "agents" in the
+bucket, the normal agent sends heartbeats only to "agents".
+
+## Alternative - separate heartbeat
+
+The auto-consul runner does not have to issue heartbeats itself; those
+can be left out entirely (but please understand this means automatic
+mode determination won't work), or run in a separate process with timing
+under more precise control.
+
+For example, in our vagrant both, the following commands are equivalent
+to the original server run.
+
+Server screen A:
+
+    auto-consul -r s3://my-bucket/consul/test-cluster \
+                -a 192.168.50.100 \
+                -n server1 \
+                run
+
+Then, server screen B:
 
     while true; do
         auto-consul -r s3://my-bucket/consul/test-cluster \
-                    -a 192.168.50.101 \
-                    -n agent1 \
+                    -a 192.168.50.100 \
+                    -n server1 \
                     heartbeat
         sleep 60
     done
 
-This generates heartbeats like the server did, but while the server
-sends heartbeats both to "servers" and "agents" in the bucket, the
-normal agent sends heartbeats only to "agents".
+The first launches the agent, the latter checks its run status and
+issues a heartbeat to the specified S3 bucket.
 
 # Mode determination
 
@@ -103,17 +120,12 @@ Each heartbeat tells us:
 - The timestamp of the heartbeat (the freshness)
 - The IP at which the node can be reached for cluster join operations.
 
-For now, it is necessary to run the heartbeat utility in parallel to the
-run utility.  In subsequent work we may want to have these things coordinated
-by one daemon, but given the experimental nature of this project it's not
-worth caring about just yet.
-
 The heartbeat asks consul for its status and from that determines if it
 is running as a server or regular agent (or if it is running at all).  If
 consul is not running at all, no heartbeat will be emitted.
 
 The default expiry is 120 seconds.  It is recommended that heartbeats fire
-at half that duration (60 seconds).
+at half that duration (60 seconds) or less.
 
 # Cluster join
 

data/bin/auto-consul

@@ -12,7 +12,7 @@ end
 
 class Command < OpenStruct
   def local
-    @local ||= AutoConsul::Local.bind_to_path(data)
+    @local ||= AutoConsul::Local.bind_to_path(data_dir)
   end
 
   def cluster
@@ -25,6 +25,8 @@ class Command < OpenStruct
 
   def do_set_mode
     cluster.set_mode! local, expiry, servers
+    # Healthy exit
+    0
   end
 
   def do_run
@@ -36,16 +38,37 @@ class Command < OpenStruct
 
   def do_direct_run
     runner = :no_op
-    runner = :run_agent! if local.agent?
-    runner = :run_server! if local.server?
+    runner = :agent_runner if local.agent?
+    runner = :server_runner if local.server?
     runner = AutoConsul::Runner.method(runner)
-    runner.call node, addr, expiry, local, cluster
+    monitor = runner.call(node, addr, expiry, local, cluster)
+    if with_heartbeat and ticks > 0
+      monitor.while_up do |ap|
+        # There's data exchange with the agent and with the registries
+        # per heartbeat, which with Ruby's memory management could cause
+        # process bloat over a long time.  So fork per heartbeat so we
+        # can avoid the bloat in the long-running process.
+        while true
+          sleep ticks
+          kid = fork { do_heartbeat }
+          Process.waitpid kid
+        end
+      end
+    end
+    monitor.run!
+    # Returns the exit status of the "consul agent" run.
+    monitor.wait
   end
 
   def do_heartbeat
     if state.running?
       cluster.servers.heartbeat! node, addr, expiry if state.server?
       cluster.agents.heartbeat! node, addr, expiry if state.agent?
+      # Healthy exit.
+      0
+    else
+      # Not running; can't heartbeat.
+      1
     end
   end
 
@@ -59,10 +82,12 @@ class Command < OpenStruct
   end
 end
 
-runner = Command.new(:data    => '/tmp/consul/state',
+runner = Command.new(:data_dir => '/tmp/consul/state',
                      :dc      => 'dc1',
+                     :with_heartbeat => false,
                      :expiry  => 120,
                      :servers => 1,
+                     :ticks   => 60,
                      :node    => Socket.gethostname.split('.', 2)[0])
 
 parser = OptionParser.new do |opts|
@@ -88,6 +113,11 @@ parser = OptionParser.new do |opts|
     runner.expiry = e.to_i
   end
 
+  opts.on("-t", "--ticks SECONDS", Integer, "The time between heartbeats (in seconds) for registry heartbeats; use of this activates a concurrent heartbeat thread for the 'run' command.") do |t|
+    runner.ticks = t.to_i
+    runner.with_heartbeat = true
+  end
+
   opts.on("-s", "--servers NUMBER", Integer, "The desired number of consul servers.") do |s|
     runner.servers = s.to_i
   end
@@ -101,7 +131,8 @@ end
 parser.parse!
 
 begin
-  runner.execute(ARGV.shift)
+  status = runner.execute(ARGV.shift)
+  exit status
 rescue UnknownCommandException => e
   puts e.message
   puts parser

data/lib/auto-consul/runner.rb

@@ -1,19 +1,143 @@
+require 'thread'
+
 module AutoConsul
   module Runner
+    INITIAL_VERIFY_SLEEP = 0.1
     SLEEP_INTERVAL = 2
     RETRIES = 5
 
-    def self.launch_and_join(agent_args, remote_ip=nil)
-      pid = spawn(*(['consul', 'agent'] + agent_args))
+    class AgentProcess
+      attr_reader :args
+      attr_reader :exit_code
+      attr_reader :pid
+      attr_reader :status
+      attr_reader :thread
+      attr_reader :stop_queue
+      attr_reader :stop_thread
 
-      # We really need to check that is running, but later.
-      return nil unless verify_running(pid)
+      def initialize args
+        @args = args.dup.freeze
+        @callbacks = {}
+      end
 
-      if not remote_ip.nil?
-        join remote_ip
+      def on_up &action
+        register_callback :up, &action
+      end
+
+      def on_down &action
+        register_callback :down, &action
+      end
+
+      def launch!
+        set_status :starting
+        @thread = Thread.new do
+          Thread.current.abort_on_exception = true
+          run_agent
+        end
+      end
+
+      def run_agent
+        handle_signals!
+        @pid = spawn(*(['consul', 'agent'] + args), :pgroup => true)
+        result = Process.waitpid2(@pid)
+        @exit_code = result[1].exitstatus
+        set_status :down
       end
 
-      pid
+      def handle_signals!
+        if @stop_queue.nil?
+          @stop_queue = Queue.new
+          @stop_thread = Thread.new do
+            while true
+              @stop_queue.pop
+              stop!
+            end
+          end
+          ['INT', 'TERM'].each do |sig|
+            Signal.trap(sig) do
+              @stop_queue << sig
+            end
+          end
+        end
+        nil
+      end
+
+      VALID_VERIFY_STATUSES = [nil, :starting]
+
+      def verify_up!
+        sleep INITIAL_VERIFY_SLEEP
+        tries = 0
+        while VALID_VERIFY_STATUSES.include?(status) and tries < RETRIES
+          sleep SLEEP_INTERVAL ** tries if tries > 0
+          if system('consul', 'info')
+            set_status :up
+          else
+            tries += 1
+          end
+        end
+      end
+
+      def on_stopping &action
+        register_callback :stopping, &action
+      end
+
+      VALID_STOP_STATUSES = [nil, :starting, :up, :stopping]
+      STOP_SIGNAL = "SIGINT"
+
+      def stop!
+        raise "The consul agent is not running (no pid)" if pid.nil?
+        raise "The consul agent is not running (status #{status.to_s})." unless VALID_STOP_STATUSES.include? status
+        set_status :stopping
+        Process.kill STOP_SIGNAL, pid
+      end
+
+      def register_callback on_status, &action
+        (@callbacks[on_status] ||= []) << action
+      end
+
+      def run!
+        launch!
+        verify_up!
+        status
+      end
+
+      def wait
+        if (t = thread).nil?
+          raise "The consul agent has not started within this runner."
+        end
+        t.join
+        exit_code
+      end
+
+      def while_up &action
+        on_up do |obj|
+          thread = Thread.new { action.call obj }
+          obj.on_stopping {|x| thread.kill }
+          obj.on_down {|x| thread.kill }
+        end
+      end
+
+      def run_callbacks on_status
+        if callbacks = @callbacks[on_status]
+          callbacks.each do |callback|
+            callback.call self
+          end
+        end
+      end
+
+      def set_status new_status
+        @status = new_status
+        run_callbacks new_status
+        new_status
+      end
+    end
+
+    def self.joining_runner(agent_args, remote_ip=nil)
+      runner = AgentProcess.new(agent_args)
+      if not remote_ip.nil?
+        runner.on_up {|a| join remote_ip}
+      end
+      runner
     end
 
     def self.verify_running pid
@@ -33,24 +157,21 @@ module AutoConsul
       hosts[0].data
     end
 
-    def self.run_agent! identity, bind_ip, expiry, local_state, registry
+    def self.agent_runner identity, bind_ip, expiry, local_state, registry
       remote_ip = pick_joining_host(registry.agents.members(expiry))
-      pid = launch_and_join(['-bind', bind_ip,
-                             '-data-dir', local_state.data_path,
-                             '-node', identity], remote_ip)
-      Process.wait pid
+      joining_runner(['-bind', bind_ip,
+                      '-data-dir', local_state.data_path,
+                      '-node', identity], remote_ip)
     end
 
-    def self.run_server! identity, bind_ip, expiry, local_state, registry
+    def self.server_runner identity, bind_ip, expiry, local_state, registry
       members = registry.servers.members(expiry)
       remote_ip = members.size > 0 ? pick_joining_host(members) : nil
 
       args = ['-bind', bind_ip, '-data-dir', local_state.data_path, '-node', identity, '-server']
       args << '-bootstrap' if members.size < 1
 
-      pid = launch_and_join(args, remote_ip)
-
-      Process.wait pid unless pid.nil?
+      joining_runner(args, remote_ip)
     end
   end
 end

data/spec/runner_spec.rb

@@ -1,6 +1,63 @@
 require 'spec-helper'
 
-shared_examples_for 'a consul agent run' do |method_name, registry_name, join_flag, args|
+shared_examples_for 'an unstoppable' do
+  before do
+    subject.set_status initial_status
+  end
+
+  before do
+    Process.should_not_receive(:kill)
+    subject.should_not_receive(:set_status)
+  end
+
+  it 'throws an exception' do
+    expect { subject.stop! }.to raise_error(/consul agent is not running/)
+  end
+end
+
+shared_examples_for 'stop signaler' do
+  before do
+    subject.set_status initial_status
+  end
+
+  before do
+    Process.should_receive(:kill).with("SIGINT", pid)
+  end
+
+  describe 'with no stopping callbacks' do
+    before do
+      subject.should_not_receive(:stopping_a!)
+      subject.should_not_receive(:stopping_b!)
+    end
+
+    it 'signals the agent process to stop' do
+      subject.stop!
+    end
+  end
+
+  describe 'with stopping callbacks' do
+    before do
+      subject.on_stopping do |o|
+        subject.stopping_a! o
+        expect(subject.status).to eq(:stopping)
+      end
+
+      subject.on_stopping do |o|
+        subject.stopping_b! o
+        expect(subject.status).to eq(:stopping)
+      end
+
+      subject.should_receive(:stopping_a!).with(subject)
+      subject.should_receive(:stopping_b!).with(subject)
+    end
+
+    it 'invokes the callbacks and signals the agent process to stop' do
+      subject.stop!
+    end
+  end
+end
+
+shared_examples_for 'a consul agent process runner' do |method_name, registry_name, join_flag, args|
   it 'properly launches consul agent' do
     members = []
     members << member if join_flag
@@ -8,7 +65,7 @@ shared_examples_for 'a consul agent run' do |method_name, registry_name, join_fl
     registry.should_receive(registry_name).with.and_return(reg = double)
     reg.should_receive(:members).with(expiry).and_return(members)
 
-    expected_args = (['consul', 'agent', '-bind', ip, '-data-dir', data_dir, '-node', identity] + args).collect do |e|
+    expected_args = (['-bind', ip, '-data-dir', data_dir, '-node', identity] + args).collect do |e|
                       if e.instance_of? Symbol
                         send e
                       else
@@ -16,23 +73,350 @@ shared_examples_for 'a consul agent run' do |method_name, registry_name, join_fl
                       end
                     end
 
-    AutoConsul::Runner.should_receive(:spawn).with(*expected_args).and_return(agent_pid = double)
+    runner = double("AgentProcess")
+    expect(AutoConsul::Runner::AgentProcess).to receive(:new).with(expected_args).and_return { runner }
 
-    # consul info retries to verify that it's running.
-    AutoConsul::Runner.should_receive(:system).with('consul', 'info').and_return(false, false, true)
+    if join_flag
+      expect(AutoConsul::Runner).to receive(:system).with('consul', 'join', remote_ip).and_return(true)
+      expect(runner).to receive(:on_up) do |&action|
+        # The callback mechanism is how we join the cluster.
+        action.call
+      end
+    else
+      expect(AutoConsul::Runner).to_not receive(:system)
+      expect(runner).to_not receive(:on_up)
+    end
 
-    AutoConsul::Runner.should_receive(:sleep).with(2)
-    AutoConsul::Runner.should_receive(:sleep).with(4)
-    AutoConsul::Runner.should_receive(:sleep).with(6)
+    callable = AutoConsul::Runner.method(method_name)
+    expect(callable.call(identity, ip, expiry, local_state, registry)).to be(runner)
+  end
+end
 
-    if join_flag
-      AutoConsul::Runner.should_receive(:system).with('consul', 'join', remote_ip).and_return(true)
+describe AutoConsul::Runner::AgentProcess do
+  let(:args) do
+    (1..3).collect do |i|
+      double("MockParam#{i.to_s}").to_s
     end
+  end
 
-    Process.should_receive(:wait).with(agent_pid)
+  subject { AutoConsul::Runner::AgentProcess.new args }
 
-    callable = AutoConsul::Runner.method(method_name)
-    callable.call(identity, ip, expiry, local_state, registry)
+  describe "#handle_signals!" do
+    let(:stop_thread) { double('StopThread', :block => Proc.new { raise 'Loop breakout bogus' }) }
+    let(:stop_queue) { [] }
+
+    before do
+      count = 0
+      # This is necessary to capture the block given to the thread,
+      # and to ensure it only happens once.
+      expect(Thread).to receive(:new) do |&block|
+        expect(count).to eq(0)
+        count += 1
+        stop_thread.stub(:block).and_return block
+      end
+      expect(Queue).to receive(:new).with.once.and_return(stop_queue)
+    end
+
+    it 'should register a queuing signal handler for SIGINT, SIGTERM' do
+      seen = {}
+      expect(Signal).to receive(:trap).exactly(2).times do |sig, &block|
+        seen[sig] = true
+        len = stop_queue.size
+        block.call
+        expect(stop_queue.size).to eq(len + 1)
+        expect(stop_queue[-1]).to eq(sig)
+      end
+      subject.handle_signals!
+      expect(seen).to eq({"INT" => true, "TERM" => true})
+    end
+
+    it 'should issue stop per queue member' do
+      Signal.stub(:trap)
+      subject.handle_signals!
+      sigs = ['INT', 'TERM', 'INT']
+      expect(stop_queue).to receive(:pop).exactly(sigs.size + 1).times do
+        raise 'Loop breakout' unless sigs.size > 0
+        sigs.pop
+      end
+      expect(subject).to receive(:stop!).with.exactly(3).times
+      expect { stop_thread.block.call }.to raise_exception(/Loop breakout/) 
+    end
+
+    it 'should only configure signals once' do
+      expect(Signal).to receive(:trap).exactly(2).times
+      subject.handle_signals!
+      subject.handle_signals!
+    end
+  end
+
+  describe "launch! method" do
+    let(:thread) { double('AgentThread') }
+    let(:pid) { double('AgentPid') }
+    let(:exit_code) { 2.to_i }
+
+    before do
+      # We'll have the AgentProcess take over signal handling, since it's
+      # bound up in the life cycle of the process.
+      subject.should_receive(:handle_signals!).with.once
+
+      # This sucks, but what are you gonna do?  It needs to be in a separate
+      # thread so the waitpid2 call doesn't block the main process.
+      subject.should_receive(:spawn).with(*(['consul', 'agent'] + args), :pgroup => true).and_return(pid)
+      process_status = double('ProcessStatus', :pid => pid,
+                                               :exitstatus => exit_code)
+      Process.should_receive(:waitpid2).with(pid).and_return([pid, process_status])
+      Thread.should_receive(:new) do |&block|
+        # The status should be :starting before invoking the thread.
+        expect(subject.status).to eq(:starting)
+        # Make sure we set abort_on_exception on the thread.
+        Thread.should_receive(:current).with.and_return(thread)
+        thread.should_receive(:abort_on_exception=).with(true)
+        block.call
+        # The block is what moves it to a status of :down.
+        # And sets the pid and exit code.
+        expect(subject.status).to eq(:down)
+        expect(subject.pid).to eq(pid)
+        expect(subject.exit_code).to eq(exit_code)
+        thread
+      end
+    end
+
+    it 'should invoke "agent consul" with given args and wait on the result' do
+      expect(subject.thread).to be_nil
+      expect(subject.pid).to be_nil
+      expect(subject.status).to be_nil
+      expect(subject.exit_code).to be_nil
+      subject.launch!
+      expect(subject.thread).to be(thread)
+    end
+
+    it 'should invoke "agent consul" and run callbacks after going down.' do
+      subject.on_down do |x|
+        x.down_a!
+      end
+
+      subject.on_down do |x|
+        x.down_b!
+      end
+
+      subject.should_receive(:down_a!).with
+      subject.should_receive(:down_b!).with
+
+      subject.launch!
+    end
+
+    # it should blow up if called with status other than nil, :down.
+  end
+
+  describe "verify_up! method" do
+    describe 'when check succeeds' do
+      before do
+        subject.should_receive(:sleep).with(0.1)
+        subject.should_receive(:system).with('consul', 'info').and_return(false, false, false, true)
+        subject.should_receive(:sleep).with(2)
+        subject.should_receive(:sleep).with(4)
+        subject.should_receive(:sleep).with(8)
+      end
+
+      it 'sets status to :up' do
+        subject.verify_up!
+        expect(subject.status).to eq(:up)
+      end
+
+      describe 'with callbacks' do
+        before do
+          subject.on_up do |obj|
+            subject.callback_a! obj
+            expect(subject.status).to eq(:up)
+          end
+
+          subject.on_up do |obj|
+            subject.callback_b! obj
+            expect(subject.status).to eq(:up)
+          end
+
+          subject.should_receive(:callback_a!).with(subject)
+          subject.should_receive(:callback_b!).with(subject)
+        end
+
+        it 'invokes up callbacks with itself as parameter' do
+          subject.verify_up!
+        end
+      end
+    end
+
+    describe 'when check fails' do
+      before do
+        subject.should_receive(:sleep).with(0.1)
+        subject.should_receive(:system).with('consul', 'info').and_return(false, false, false, false, false)
+        subject.should_receive(:sleep).with(2)
+        subject.should_receive(:sleep).with(4)
+        subject.should_receive(:sleep).with(8)
+        subject.should_receive(:sleep).with(16)
+      end
+      
+      it 'leaves the status alone' do
+        subject.should_not_receive(:set_status)
+        subject.verify_up!
+        expect(subject.status).to be_nil
+      end
+
+      describe 'with callbacks' do
+        before do
+          subject.on_up do |obj|
+            subject.callback_a! obj
+          end
+
+          subject.on_up do |obj|
+            subject.callback_b! obj
+          end
+
+          subject.should_not_receive(:callback_a!)
+          subject.should_not_receive(:callback_b!)
+        end
+
+        it 'does not invoke callbacks at all' do
+          subject.verify_up!
+        end
+      end
+
+      # it should blow up if called with status other than :starting, :up
+    end
+  end
+
+  describe 'stop! method' do
+    describe 'with a pid' do
+      let(:pid) { double("AgentPid") }
+      before { subject.stub(:pid).and_return(pid) }
+
+      describe 'in nil status' do
+        # Need an expression so compiler doesn't ignore the block
+        let(:initial_status) { nil && true }
+        it_behaves_like 'stop signaler'
+      end
+
+      describe 'in :starting status' do
+        let(:initial_status) { :starting.to_sym }
+        it_behaves_like 'stop signaler'
+      end
+
+      describe 'in :up status' do
+        let(:initial_status) { :up.to_sym }
+        it_behaves_like 'stop signaler'
+      end
+
+      describe 'in :stopping status' do
+        let(:initial_status) { :stopping.to_sym }
+        it_behaves_like 'stop signaler'
+      end
+
+      describe 'in :down status' do
+        let(:initial_status) { :down.to_sym }
+        it_behaves_like 'an unstoppable'
+      end
+    end
+
+    describe 'with no pid' do
+      before do
+        subject.stub(:pid).and_return(nil)
+      end
+
+      describe 'in nil status' do
+        # Need an expression so compiler doesn't ignore the block
+        let(:initial_status) { nil && true }
+        it_behaves_like 'an unstoppable'
+      end
+
+      describe 'in :starting status' do
+        let(:initial_status) { :starting.to_sym }
+        it_behaves_like 'an unstoppable'
+      end
+
+      describe 'in :up status' do
+        let(:initial_status) { :up.to_sym }
+        it_behaves_like 'an unstoppable'
+      end
+
+      describe 'in :stopping status' do
+        let(:initial_status) { :stopping.to_sym }
+        it_behaves_like 'an unstoppable'
+      end
+
+      describe 'in :down status' do
+        let(:initial_status) { :down.to_sym }
+        it_behaves_like 'an unstoppable'
+      end
+    end
+  end
+
+  describe ':run! method' do
+    it 'launches, then verifies up, and returns status' do
+      status = double('Status')
+      expect(subject).to receive(:launch!).with.ordered
+      expect(subject).to receive(:verify_up!).with.ordered
+      expect(subject).to receive(:status).with.ordered { status }
+      expect(subject.run!).to be(status)
+    end
+  end
+
+  describe ':wait method' do
+    it 'waits on the agent runner thread and returns the exit code' do
+      thread = double('Thread')
+      exit_code = double('ExitCode')
+      expect(subject).to receive(:thread).with.and_return { thread }
+      expect(thread).to receive(:join).with.and_return { thread }
+      expect(subject).to receive(:exit_code).with.and_return { exit_code }
+      expect(subject.wait).to be(exit_code)
+    end
+
+    it 'blows up if no thread is present' do
+      expect(subject).to receive(:thread).with.and_return { nil }
+      expect { subject.wait }.to raise_exception(/consul agent has not started/)
+    end
+  end
+
+  describe '#while_up method' do
+    let(:thread) { double('Thread') }
+    # Use this as the while_up block; it will verify that the block is invoked with
+    # the AgentProcess instance as sole parameter.
+    let(:action) { Proc.new {|o| expect(o).to be(subject)} }
+
+    describe 'when brought up' do
+      before do
+        # This "brings it up."
+        expect(subject).to receive(:on_up).and_yield(subject)
+
+        # And this happens in the on_up callback
+        expect(Thread).to receive(:new) do |&blk|
+          blk.call
+          thread
+        end
+      end
+
+      it 'registers an on_stopping that kills the thread for the given block' do
+        expect(thread).to receive(:kill).with
+        expect(subject).to receive(:on_stopping).and_yield(subject)
+        subject.while_up &action
+      end
+
+      it 'registers an on_down that kills the thread for the given block' do
+        expect(thread).to receive(:kill).with
+        expect(subject).to receive(:on_down).and_yield(subject)
+        subject.while_up &action
+      end
+    end
+
+    describe 'when never brought up' do
+      it 'never registers on_stopping or on_down handlers' do
+        expect(Thread).to_not receive(:new)
+        expect(thread).to_not receive(:kill)
+        expect(subject).to_not receive(:on_stopping)
+        expect(subject).to_not receive(:on_down)
+        # We receive it but we're not running anything.
+        expect(subject).to receive(:on_up)
+        subject.while_up &action
+      end
+    end
   end
 end
 
@@ -55,21 +439,21 @@ describe AutoConsul::Runner do
     registry.servers.stub(:servers).with(expiry).and_return(servers_list)
   end
 
-  describe :run_agent! do
-    it_behaves_like 'a consul agent run', :run_agent!, :agents, true, []
+  describe :agent_runner do
+    it_behaves_like 'a consul agent process runner', :agent_runner, :agents, true, []
   end
 
-  describe :run_server! do
+  describe :server_runner do
     describe 'with empty server registry' do
       # consul agent -bind 192.168.50.100 -data-dir /opt/consul/server/data -node vagrant-server -server -bootstrap
-      it_behaves_like 'a consul agent run', :run_server!, :servers, false, ['-server', '-bootstrap']
+      it_behaves_like 'a consul agent process runner', :server_runner, :servers, false, ['-server', '-bootstrap']
     end
 
     describe 'with other servers in registry' do
       # consul agent -bind 192.168.50.100 -data-dir /opt/consul/server/data -node vagrant-server -server
       # consul join some_ip
 
-      it_behaves_like 'a consul agent run', :run_server!, :servers, true, ['-server']
+      it_behaves_like 'a consul agent process runner', :server_runner, :servers, true, ['-server']
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: auto-consul
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors: