bosh-monitor 1.5.0.pre.1113

data/README

@@ -0,0 +1,80 @@
+h4. Synopsis
+
+BOSH Health Monitor (BHM) is a component that monitors health of one or multiple BOSH deployments. It processes heartbeats and alerts from BOSH agents and notifies interested parties if something goes wrong.
+
+h4. Heartbeats
+
+Agent sends periodic heartbeats to HM. Heartbeats are sent via message bus and have the following format:
+
+| *Subject* | hm.agent.heartbeat.<agent_id> |
+| *Payload* | none |
+
+h6. Heartbeat processing
+
+# If the agent is known to HM the last heartbeat timestamp gets updated. No analysis is attempted at this point, analyze agents routine is asynchronous to heartbeat processing.
+# If the agent is unknown it gets registered with HM with a warning flag set (we call them rogue agents). Next director poll will possibly include this agent to a list of managed agents and clear the flag. We might generate the alert if the flag hasn't been cleared for some (configurable) time.
+
+h4. Agents discovery
+
+HM polls director periodically to get the list of managed VMs:
+
+| *Endpoint* | GET /deployments/<deployment_name>/vms |
+| *Response* | JSON including agent ids, job names and indices for all managed VMs |
+
+When new agent is discovered it gets registered and added to a managed deployment. No active operations are performed to reach the agent and query it, we only rely on heartbeats and agent alerts.
+
+h4. Agents analysis
+
+This is a periodic operation that goes through all known agents. First it tries to go through all managed deployments, then analyzes rogue agents as well. The following procedure is used:
+
+# If agent missed more than N heartbeats the "Agent Missing" alert is generated.
+
+h4. Alerts
+
+Alert is a concept used by HM to flag and deliver information about important events. It includes the following data:
+
+# Id
+# Severity
+# Source (usually deployment/job/index tuple)
+# Timestamp
+# Description
+# Long description (optional)
+# Tags (optional)
+
+h6. Alert Processor
+
+Alert Processor is a module that registers incoming alerts and routes them to interested parties via appropriate delivery agent. It should conform to the following interface:
+
+| *Method*         | *Arguments* | *Description* |
+| *register_alert* | alert (object responding to :id, :severity, :timestamp, :description, :long_description, :source and :tags)  | Registers an alert and invokes a delivery agent. Delivery agent might or might not deliver alert immediately depending on the implementation, so Alert Processor shouldn't make any assumptions about delivery (i.e. agent might queue up several alerts and send them asynchronously. |
+| *add_delivery_agent* | delivery_agent, options | Adds a delivery agent to a processor |
+
+Alert id can be an arbitrary string however Alert Processor might use it to keep track of registered alerts and don't process the same alert twice. This way other HM modules can just blindly register any incoming alerts and leave the dedup step to the alert processor).
+
+Alerts are only persisted in HM memory (at least in the initial version) so losing HM leads to losing any undelivered alerts that might have been queued by a delivery agent or alert processor).
+
+If alert processor has more than one delivery agents associated with it then it notifies all of them in order (i.e. we want to notify both Zabbix and Pager Duty).
+
+h6. Delivery Agent
+
+Delivery Agent is a module that takes care of an alert delivery mechanism (such as an email, Pager Duty alert, writing to a journal or even silently discarding the alert). It should conform to the following interface:
+
+| *Method* | *Arguments* | *Description* |
+| *deliver* | alert | Delivers alert or queues it for delivery. |
+
+The initial implementation will have email and Pager Duty delivery agents.
+
+Alert Processor is not pluggable, it's just one of HM classes. Delivery agents are pluggable but generally not changed in a runtime but initialized using an HM configuration file on HM startup.
+
+h4. Alerts from agent
+
+HM subscribes to agent alerts on a message bus:
+
+| *Subject* | hm.agent.alert.<agent_id> |
+| *Payload* | JSON containing the following keys: id, service, event, action, description, timestamp, tags |
+
+BOSH Agent is responsible for mapping any underlying supervisor alert format to the expected JSON payload and send it to HM.
+
+HM is responsible for interpreting JSON payload and mapping it to a sequence of HM actions and possibly creating an HM alert compatible with Alert Processor module. HM never dedups incoming alerts outside of Alert Processor (this adds some overhead to an incoming alert parser but shouldn't be too bad). Malformed payloads are ignored.
+
+Job name and index are not featured in agent incoming alert, those are looked up in director. If heartbeat came from a rogue agent and we have no job name and/or index then we note that fact in alert description but don't try to be too worried about that (service name and agent id should be enough). We might consider including agent IP address as a part of heartbeat so we can track down rogue agents.

data/bin/bosh-monitor

@@ -0,0 +1,30 @@
+#!/usr/bin/env ruby
+
+require "bosh/monitor"
+require "optparse"
+
+config_file = nil
+
+opts = OptionParser.new do |opts|
+  opts.on("-c", "--config FILE", "configuration file") do |opt|
+    config_file = opt
+  end
+end
+
+opts.parse!(ARGV.dup)
+
+if config_file.nil?
+  puts opts
+  exit 1
+end
+
+runner = Bosh::Monitor::Runner.new(config_file)
+
+Signal.trap("INT") do
+  runner.stop
+  exit(1)
+end
+
+runner.run
+
+

data/bin/bosh-monitor-console

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+
+require 'bosh/monitor'
+require 'irb'
+require 'irb/completion'
+
+module Bosh
+  module Monitor
+
+    class Console
+      include YamlHelper
+
+      def self.start(context)
+        new.start(context)
+      end
+
+      def start(context)
+        config_file = nil
+
+        opts = OptionParser.new do |opt|
+          opt.on("-c", "--config [ARG]", "configuration file") { |c| config_file = c }
+        end
+
+        opts.parse!(ARGV)
+
+        if config_file.nil?
+          puts opts
+          exit 1
+        end
+
+        puts "=> Loading #{config_file}"
+        Bhm.config = load_yaml_file(config_file)
+
+        begin
+          require 'ruby-debug'
+          puts "=> Debugger enabled"
+        rescue LoadError
+          puts "=> ruby-debug not found, debugger disabled"
+        end
+
+        puts "=> Welcome to BOSH Health Monitor console"
+
+        IRB.start
+      end
+
+    end
+  end
+end
+
+Bhm::Console.start(self)
+

data/bin/listener

@@ -0,0 +1,58 @@
+#!/usr/bin/env ruby
+
+require "eventmachine"
+require "nats/client"
+
+class Listener
+
+  def self.start
+    new.start
+  end
+
+  def start
+    filter = nil
+    nats_uri = nil
+    nats_subject = nil
+
+    opts = OptionParser.new do |opt|
+      opt.on("-f", "--filter ARG") { |f| filter = f }
+      opt.on("-n", "--nats URI") { |n| nats_uri = n }
+      opt.on("-s", "--subject ARG") { |s| nats_subject = s }
+    end
+
+    opts.parse!(ARGV)
+
+    if nats_uri.nil?
+      puts "Usage: listener [options] <nats_uri>"
+    end
+
+    nats_client_options = {
+      :uri => nats_uri,
+      :autostart => false
+    }
+
+    @nats = NATS.connect(nats_client_options)
+
+    if nats_subject
+      puts "> NATS subject is set to `#{nats_subject}'"
+    else
+      nats_subject = "bosh.hm.events"
+    end
+
+    if filter
+      puts "> Filter is set to `#{filter}'"
+    end
+
+    puts "> Subscribing to events"
+    @nats.subscribe(nats_subject) do |msg|
+      if filter.nil? || msg =~ Regexp.new(Regexp.quote(filter))
+        puts "#{Time.now.strftime("%Y-%m-%d %H:%M:%S")} >> " + msg
+      end
+    end
+  end
+end
+
+EM.run do
+  Listener.start
+end
+

data/lib/bosh/monitor.rb

@@ -0,0 +1,72 @@
+module Bosh
+  module Monitor
+  end
+end
+
+Bhm = Bosh::Monitor
+
+begin
+  require 'fiber'
+rescue LoadError
+  unless defined? Fiber
+    $stderr.puts 'FATAL: HealthMonitor requires Ruby implementation that supports fibers'
+    exit 1
+  end
+end
+
+require 'ostruct'
+require 'set'
+
+require 'em-http-request'
+require 'eventmachine'
+require 'logging'
+require 'nats/client'
+require 'sinatra'
+require 'thin'
+require 'securerandom'
+require 'yajl'
+
+# Helpers
+require 'bosh/monitor/yaml_helper'
+
+# Basic blocks
+require 'bosh/monitor/agent'
+require 'bosh/monitor/config'
+require 'bosh/monitor/core_ext'
+require 'bosh/monitor/director'
+require 'bosh/monitor/director_monitor'
+require 'bosh/monitor/errors'
+require 'bosh/monitor/metric'
+require 'bosh/monitor/runner'
+require 'bosh/monitor/version'
+
+# Processing
+require 'bosh/monitor/agent_manager'
+require 'bosh/monitor/event_processor'
+
+# HTTP endpoints
+require 'bosh/monitor/api_controller'
+
+# Protocols
+require 'bosh/monitor/protocols/tsdb'
+
+# Events
+require 'bosh/monitor/events/base'
+require 'bosh/monitor/events/alert'
+require 'bosh/monitor/events/heartbeat'
+
+# Plugins
+require 'bosh/monitor/plugins/base'
+require 'bosh/monitor/plugins/dummy'
+require 'bosh/monitor/plugins/http_request_helper'
+require 'bosh/monitor/plugins/resurrector_helper'
+require 'bosh/monitor/plugins/cloud_watch'
+require 'bosh/monitor/plugins/datadog'
+require 'bosh/monitor/plugins/paging_datadog_client'
+require 'bosh/monitor/plugins/email'
+require 'bosh/monitor/plugins/logger'
+require 'bosh/monitor/plugins/nats'
+require 'bosh/monitor/plugins/pagerduty'
+require 'bosh/monitor/plugins/resurrector'
+require 'bosh/monitor/plugins/tsdb'
+require 'bosh/monitor/plugins/varz'

data/lib/bosh/monitor/agent.rb

@@ -0,0 +1,51 @@
+module Bosh::Monitor
+  class Agent
+
+    attr_reader   :id
+    attr_reader   :discovered_at
+    attr_accessor :updated_at
+
+    ATTRIBUTES = [ :deployment, :job, :index, :cid ]
+
+    ATTRIBUTES.each do |attribute|
+      attr_accessor attribute
+    end
+
+    def initialize(id, opts={})
+      raise ArgumentError, "Agent must have an id" if id.nil?
+
+      @id            = id
+      @discovered_at = Time.now
+      @updated_at    = Time.now
+      @logger        = Bhm.logger
+      @intervals     = Bhm.intervals
+
+      @deployment = opts[:deployment]
+      @job = opts[:job]
+      @index = opts[:index]
+      @cid = opts[:cid]
+    end
+
+    def name
+      if @deployment && @job && @index
+        "#{@deployment}: #{@job}(#{@index}) [id=#{@id}, cid=#{@cid}]"
+      else
+        state = ATTRIBUTES.inject([]) do |acc, attribute|
+          value = send(attribute)
+          acc << "#{attribute}=#{value}" if value
+          acc
+        end
+
+        "agent #{@id} [#{state.join(", ")}]"
+      end
+    end
+
+    def timed_out?
+      (Time.now - @updated_at) > @intervals.agent_timeout
+    end
+
+    def rogue?
+      (Time.now - @discovered_at) > @intervals.rogue_agent_alert && @deployment.nil?
+    end
+  end
+end

data/lib/bosh/monitor/agent_manager.rb

@@ -0,0 +1,295 @@
+module Bosh::Monitor
+
+  class AgentManager
+    attr_reader :heartbeats_received
+    attr_reader :alerts_received
+    attr_reader :alerts_processed
+
+    attr_accessor :processor
+
+    def initialize(event_processor)
+      # hash of agent id to agent structure (see add_agent())
+      @agents = { }
+
+      # hash of deployment name to set of agent ids
+      @deployments = { }
+
+      @logger = Bhm.logger
+      @heartbeats_received = 0
+      @alerts_received = 0
+      @alerts_processed = 0
+
+      @processor = event_processor
+    end
+
+    # Get a hash of agent id -> agent object for all agents associated with the deployment
+    def get_agents_for_deployment(deployment_name)
+      agent_ids = @deployments[deployment_name]
+      @agents.select { |key, value| agent_ids.include?(key) }
+    end
+
+    def lookup_plugin(name, options = {})
+      plugin_class = nil
+      begin
+        class_name = name.to_s.split("_").map(&:capitalize).join
+        plugin_class = Bosh::Monitor::Plugins.const_get(class_name)
+      rescue NameError => e
+        raise PluginError, "Cannot find `#{name}' plugin"
+      end
+
+      plugin_class.new(options)
+    end
+
+    def setup_events
+      Bhm.set_varz("heartbeats_received", 0)
+
+      @processor.enable_pruning(Bhm.intervals.prune_events)
+      Bhm.plugins.each do |plugin|
+        @processor.add_plugin(lookup_plugin(plugin["name"], plugin["options"]), plugin["events"])
+      end
+
+      Bhm.nats.subscribe("hm.agent.heartbeat.*") do |message, reply, subject|
+        process_event(:heartbeat, subject, message)
+      end
+
+      Bhm.nats.subscribe("hm.agent.alert.*") do |message, reply, subject|
+        process_event(:alert, subject, message)
+      end
+
+      Bhm.nats.subscribe("hm.agent.shutdown.*") do |message, reply, subject|
+        process_event(:shutdown, subject, message)
+      end
+    end
+
+    def agents_count
+      @agents.size
+    end
+
+    def deployments_count
+      @deployments.size
+    end
+
+    # Syncs deployments list received from director
+    # with HM deployments.
+    # @param deployments Array list of deployments returned by director
+    def sync_deployments(deployments)
+      managed = Set.new(deployments.map { |d| d["name"] })
+      all     = Set.new(@deployments.keys)
+
+      (all - managed).each do |stale_deployment|
+        @logger.warn("Found stale deployment #{stale_deployment}, removing...")
+        remove_deployment(stale_deployment)
+      end
+    end
+
+    def sync_agents(deployment, vms)
+      managed_agent_ids = @deployments[deployment] || Set.new
+      active_agent_ids  = Set.new
+
+      vms.each do |vm|
+        if add_agent(deployment, vm)
+          active_agent_ids << vm["agent_id"]
+        end
+      end
+
+      (managed_agent_ids - active_agent_ids).each do |agent_id|
+        remove_agent(agent_id)
+      end
+    end
+
+    def remove_deployment(name)
+      agent_ids = @deployments[name]
+
+      agent_ids.to_a.each do |agent_id|
+        @agents.delete(agent_id)
+      end
+
+      @deployments.delete(name)
+    end
+
+    def remove_agent(agent_id)
+      @agents.delete(agent_id)
+      @deployments.each_pair do |deployment, agents|
+        agents.delete(agent_id)
+      end
+    end
+
+    # Processes VM data from BOSH Director,
+    # extracts relevant agent data, wraps it into Agent object
+    # and adds it to a list of managed agents.
+    def add_agent(deployment_name, vm_data)
+      unless vm_data.kind_of?(Hash)
+        @logger.error("Invalid format for VM data: expected Hash, got #{vm_data.class}: #{vm_data}")
+        return false
+      end
+
+      agent_id = vm_data["agent_id"]
+      agent_cid = vm_data["cid"]
+
+      if agent_id.nil?
+        @logger.warn("No agent id for VM: #{vm_data}")
+        return false
+      end
+
+      # Idle VMs, we don't care about them, but we still want to track them
+      if vm_data["job"].nil?
+        @logger.debug("VM with no job found: #{agent_id}")
+      end
+
+      agent = @agents[agent_id]
+
+      if agent.nil?
+        @logger.debug("Discovered agent #{agent_id}")
+        agent = Agent.new(agent_id)
+        @agents[agent_id] = agent
+      end
+
+      agent.deployment = deployment_name
+      agent.job = vm_data["job"]
+      agent.index = vm_data["index"]
+      agent.cid = vm_data["cid"]
+
+      @deployments[deployment_name] ||= Set.new
+      @deployments[deployment_name] << agent_id
+      true
+    end
+
+    def analyze_agents
+      @logger.info "Analyzing agents..."
+      started = Time.now
+
+      processed = Set.new
+      count = 0
+
+      # Agents from managed deployments
+      @deployments.each_pair do |deployment_name, agent_ids|
+        agent_ids.each do |agent_id|
+          analyze_agent(agent_id)
+          processed << agent_id
+          count += 1
+        end
+      end
+
+      # Rogue agents (hey there Solid Snake)
+      (@agents.keys.to_set - processed).each do |agent_id|
+        @logger.warn("Agent #{agent_id} is not a part of any deployment")
+        analyze_agent(agent_id)
+        count += 1
+      end
+
+      @logger.info("Analyzed %s, took %s seconds" % [ pluralize(count, "agent"), Time.now - started ])
+      count
+    end
+
+    def analyze_agent(agent_id)
+      agent = @agents[agent_id]
+      ts = Time.now.to_i
+
+      if agent.nil?
+        @logger.error("Can't analyze agent #{agent_id} as it is missing from agents index, skipping...")
+        return false
+      end
+
+      if agent.timed_out? && agent.rogue?
+        # Agent has timed out but it was never
+        # actually a proper member of the deployment,
+        # so we don't really care about it
+        remove_agent(agent.id)
+        return
+      end
+
+      if agent.timed_out?
+        @processor.process(:alert,
+          severity: 2,
+          source: agent.name,
+          title: "#{agent.id} has timed out",
+          created_at: ts,
+          deployment: agent.deployment,
+          job: agent.job,
+          index: agent.index)
+      end
+
+      if agent.rogue?
+        @processor.process(:alert,
+          :severity => 2,
+          :source => agent.name,
+          :title => "#{agent.id} is not a part of any deployment",
+          :created_at => ts)
+      end
+
+      true
+    end
+
+    def process_event(kind, subject, payload = {})
+      kind = kind.to_s
+      agent_id = subject.split('.', 4).last
+      agent = @agents[agent_id]
+
+      if agent.nil?
+        # There might be more than a single shutdown event,
+        # we are only interested in processing it if agent
+        # is still managed
+        return if kind == "shutdown"
+
+        @logger.warn("Received #{kind} from unmanaged agent: #{agent_id}")
+        agent = Agent.new(agent_id)
+        @agents[agent_id] = agent
+      else
+        @logger.debug("Received #{kind} from #{agent_id}: #{payload}")
+      end
+
+      case payload
+      when String
+        message = Yajl::Parser.parse(payload)
+      when Hash
+        message = payload
+      end
+
+      case kind.to_s
+      when "alert"
+        on_alert(agent, message)
+      when "heartbeat"
+        on_heartbeat(agent, message)
+      when "shutdown"
+        on_shutdown(agent, message)
+      else
+        @logger.warn("No handler found for `#{kind}' event")
+      end
+
+    rescue Yajl::ParseError => e
+      @logger.error("Cannot parse incoming event: #{e}")
+    rescue Bhm::InvalidEvent => e
+      @logger.error("Invalid event: #{e}")
+    end
+
+    def on_alert(agent, message)
+      if message.is_a?(Hash) && !message.has_key?("source")
+        message["source"] = agent.name
+      end
+
+      @processor.process(:alert, message)
+      @alerts_processed += 1
+      Bhm.set_varz("alerts_processed", @alerts_processed)
+    end
+
+    def on_heartbeat(agent, message)
+      agent.updated_at = Time.now
+
+      if message.is_a?(Hash)
+        message["timestamp"] = Time.now.to_i if message["timestamp"].nil?
+        message["agent_id"] = agent.id
+        message["deployment"] = agent.deployment
+      end
+
+      @processor.process(:heartbeat, message)
+      @heartbeats_received += 1
+      Bhm.set_varz("heartbeats_received", @heartbeats_received)
+    end
+
+    def on_shutdown(agent, message)
+      @logger.info("Agent `#{agent.id}' shutting down...")
+      remove_agent(agent.id)
+    end
+
+  end
+end