bosh-monitor 1.2624.0 → 1.2640.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 235888ed3668f9879fc1115fdee790c8be428d97
+  data.tar.gz: aaa803354916d1e53d6a8131886d5d5bfb330a27
+SHA512:
+  metadata.gz: 8725456e1287862c434326a0e5ac7e0c1b06378bac2f972bfe8878a2372fa4220e06ce9e98f74ef204a77a0d0bcba486a530dc6df1c8dc3e4e2eaf822a5e6af9
+  data.tar.gz: a37746a0e2906592874766567857032e5b5cc8d1a373d5e05d26f11cf89a8e81f74371e5ea21f918dd55e4deac4c11d2ae2dc3926eef63b6c00cce437770f643

data/README.md

@@ -0,0 +1,105 @@
+## Synopsis
+
+BOSH Monitor is a component that listens to and responds to events (Heartbeats & Alerts) on the message bus (NATS). 
+
+The Monitor also includes a few primary components:
+- The Agent Monitor maintains a record of known agents (by heartbeat event subscription)
+- The Director Monitor maintains a record of known agents (by director HTTP polling).
+- The Agent Analyzer that analyzes agent state periodically and generates Alerts.
+- The HTTP Server that responds to the /varz endpoint
+
+The Monitor also supports generic event processing plugins that respond to Heartbeats & Alerts.
+
+## Heartbeat Events
+
+The Agent on each VM sends periodic heartbeats to the BOSH Monitor via the message bus (NATS).
+
+The message syntax is as follows:
+
+| *Subject* | *Payload* |
+|-----------|-----------|
+| hm.agent.heartbeat.\<agent_id\> | none |
+
+## Alert Events
+
+A BOSH Alert is a specific type of event sent by BOSH components via the message bus. 
+
+Alerts includes the following data:
+
+- Id
+- Severity
+- Source (usually deployment/job/index tuple)
+- Timestamp
+- Description
+- Long description (optional)
+- Tags (optional)
+
+## Event Handling Plugins
+
+Alerts are processed by a number of plugins that register to receive incoming alerts.
+
+Among the included plugins are:
+- Event Logger - Logs all events
+- Resurrector - Restarts VMs that have stopped heartbeating
+- PagerDuty - Sends various events to PagerDuty.com using their API
+- DataDog - Sends various events to DataDog.com using their API
+- AWS CloudWatch - Sends various events to Amazon's CloudWatch using their API
+- Emailer - Sends configurable Emails on events reciept
+
+Plugins should conform to the following interface:
+
+| *Method* | *Arguments* | *Description* |
+|----------|-------------|---------------|
+| *validate_options* | | Validates the plugin configuration options |
+| *run* | | Initializes the plugin process |
+| *process* | event | Processes an event (Bosh::Monitor::Events::Heartbeat or Bosh::Monitor::Events::Alert) |
+
+The event processor handles deduping duplicate events.
+
+Plugins are notified in the order that they were registered (based on configuration order).
+
+## Agent Monitor - Heartbeat Event Processing
+
+The Agent Monitor listens for heartbeat events on the message bus and handles them in the following way:
+
+- If the Agent is known to the Monitor then the last heartbeat timestamp gets updated. 
+- If the Agent is unknown to the Monitor then it is recorded with a flag that marks it as a "rogue agent". 
+
+No analysis is performed when a heatbeat is received. The Agent Analyzer process and Director Monitor polling are asynchronous to heartbeat event processing by the Agent Monitor.
+
+## Director Monitor - Agent Discovery
+
+The Director Monitor polls the Director periodically via HTTP to get the list of managed VMs.
+
+The message syntax is as follows:
+
+| *Method* | *Endpoint* | *Response* |
+|----------|------------|------------|
+| /deployments/\<deployment_name\>/vms | GET | JSON including agent ids, job names and indices for all managed VMs |
+
+- If a new agent is discovered via polling then it is recorded by the Monitor as part of the managed deployment. 
+- If a "rogue agent" is discovered via polling then its "rogue agent" flag is cleared.
+
+The Director Monitor does not actively poll the agents themselves, just the Director. The Director Monitor simply remembers the state of the world as reported by polling and event processing so that the difference can be analyzed.
+
+## Agent Analyzer
+
+The Agent Analyzer is a periodic process that generates "Agent Missing" alerts. 
+
+If an agent's heartbeat timestamp is not updated within the configured time period, the Agent Analyzer process will generate an "Agent Missing" alert.
+
+Both known VM agents and rogue agents may send "Agent Missing" alerts, but they have different configurable time periods.
+
+## Alerts from BOSH Agent
+
+The Monitor subscribes to Agent alerts of the following format:
+
+| *Subject* | *Payload* |
+|-----------|-----------|
+| hm.agent.alert.\<agent_id\> | JSON containing the following keys: id, service, event, action, description, timestamp, tags |
+
+BOSH Agent is responsible for mapping any underlying supervisor alert format to the expected JSON payload and sending it to BOSH Monitor.
+
+The Monitor is responsible for interpreting the JSON payload and mapping it to a sequence of Monitor & Plugin actions, possibly generating new alerts that bypass the message bus. Malformed payloads are ignored.
+
+Job name and index are not part of alerts from the Agent, those are looked up in the Director. If heartbeat came from a rogue agent and we have no job name and/or index then we note that fact in the alert description but don't try to be too worried about that (service name and agent id should be enough). We might consider including agent IP address as a part of heartbeat so we can track down rogue agents.

data/lib/bosh/monitor/version.rb

@@ -1,5 +1,5 @@
 module Bosh
   module Monitor
-    VERSION = '1.2624.0'
+    VERSION = '1.2640.0'
   end
 end

metadata

@@ -1,68 +1,60 @@
 --- !ruby/object:Gem::Specification
 name: bosh-monitor
 version: !ruby/object:Gem::Version
-  version: 1.2624.0
-  prerelease: 
+  version: 1.2640.0
 platform: ruby
 authors:
 - VMware
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-07-01 00:00:00.000000000 Z
+date: 2014-07-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: eventmachine
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.0.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.0.0
 - !ruby/object:Gem::Dependency
   name: logging
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.5.0
 - !ruby/object:Gem::Dependency
   name: em-http-request
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.3.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.3.0
 - !ruby/object:Gem::Dependency
   name: nats
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
@@ -70,7 +62,6 @@ dependencies:
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
@@ -78,102 +69,76 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: yajl-ruby
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.1.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.1.0
 - !ruby/object:Gem::Dependency
   name: thin
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.5.0
 - !ruby/object:Gem::Dependency
   name: sinatra
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.4.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.4.2
 - !ruby/object:Gem::Dependency
   name: aws-sdk
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 1.32.0
+        version: 1.44.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 1.32.0
+        version: 1.44.0
 - !ruby/object:Gem::Dependency
   name: dogapi
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.6.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.6.0
-- !ruby/object:Gem::Dependency
-  name: uuidtools
-  requirement: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '2.1'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: '2.1'
-description: ! 'BOSH Health Monitor
-
-  03f604'
+description: |-
+  BOSH Health Monitor
+  986896
 email: support@cloudfoundry.com
 executables:
 - bosh-monitor-console
@@ -182,6 +147,10 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- README.md
+- bin/bosh-monitor
+- bin/bosh-monitor-console
+- bin/listener
 - lib/bosh/monitor.rb
 - lib/bosh/monitor/agent.rb
 - lib/bosh/monitor/agent_manager.rb
@@ -214,36 +183,28 @@ files:
 - lib/bosh/monitor/runner.rb
 - lib/bosh/monitor/version.rb
 - lib/bosh/monitor/yaml_helper.rb
-- README
-- bin/bosh-monitor-console
-- bin/bosh-monitor
-- bin/listener
 homepage: https://github.com/cloudfoundry/bosh
 licenses:
 - Apache 2.0
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: 1.9.3
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -3444887689356418361
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.23.2
+rubygems_version: 2.2.2
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: BOSH Health Monitor
 test_files: []

data/README

@@ -1,80 +0,0 @@
-h4. Synopsis
-
-BOSH Health Monitor (BHM) is a component that monitors health of one or multiple BOSH deployments. It processes heartbeats and alerts from BOSH agents and notifies interested parties if something goes wrong.
-
-h4. Heartbeats
-
-Agent sends periodic heartbeats to HM. Heartbeats are sent via message bus and have the following format:
-
-| *Subject* | hm.agent.heartbeat.<agent_id> |
-| *Payload* | none |
-
-h6. Heartbeat processing
-
-# If the agent is known to HM the last heartbeat timestamp gets updated. No analysis is attempted at this point, analyze agents routine is asynchronous to heartbeat processing.
-# If the agent is unknown it gets registered with HM with a warning flag set (we call them rogue agents). Next director poll will possibly include this agent to a list of managed agents and clear the flag. We might generate the alert if the flag hasn't been cleared for some (configurable) time.
-
-h4. Agents discovery
-
-HM polls director periodically to get the list of managed VMs:
-
-| *Endpoint* | GET /deployments/<deployment_name>/vms |
-| *Response* | JSON including agent ids, job names and indices for all managed VMs |
-
-When new agent is discovered it gets registered and added to a managed deployment. No active operations are performed to reach the agent and query it, we only rely on heartbeats and agent alerts.
-
-h4. Agents analysis
-
-This is a periodic operation that goes through all known agents. First it tries to go through all managed deployments, then analyzes rogue agents as well. The following procedure is used:
-
-# If agent missed more than N heartbeats the "Agent Missing" alert is generated.
-
-h4. Alerts
-
-Alert is a concept used by HM to flag and deliver information about important events. It includes the following data:
-
-# Id
-# Severity
-# Source (usually deployment/job/index tuple)
-# Timestamp
-# Description
-# Long description (optional)
-# Tags (optional)
-
-h6. Alert Processor
-
-Alert Processor is a module that registers incoming alerts and routes them to interested parties via appropriate delivery agent. It should conform to the following interface:
-
-| *Method*         | *Arguments* | *Description* |
-| *register_alert* | alert (object responding to :id, :severity, :timestamp, :description, :long_description, :source and :tags)  | Registers an alert and invokes a delivery agent. Delivery agent might or might not deliver alert immediately depending on the implementation, so Alert Processor shouldn't make any assumptions about delivery (i.e. agent might queue up several alerts and send them asynchronously. |
-| *add_delivery_agent* | delivery_agent, options | Adds a delivery agent to a processor |
-
-Alert id can be an arbitrary string however Alert Processor might use it to keep track of registered alerts and don't process the same alert twice. This way other HM modules can just blindly register any incoming alerts and leave the dedup step to the alert processor).
-
-Alerts are only persisted in HM memory (at least in the initial version) so losing HM leads to losing any undelivered alerts that might have been queued by a delivery agent or alert processor).
-
-If alert processor has more than one delivery agents associated with it then it notifies all of them in order (i.e. we want to notify both Zabbix and Pager Duty).
-
-h6. Delivery Agent
-
-Delivery Agent is a module that takes care of an alert delivery mechanism (such as an email, Pager Duty alert, writing to a journal or even silently discarding the alert). It should conform to the following interface:
-
-| *Method* | *Arguments* | *Description* |
-| *deliver* | alert | Delivers alert or queues it for delivery. |
-
-The initial implementation will have email and Pager Duty delivery agents.
-
-Alert Processor is not pluggable, it's just one of HM classes. Delivery agents are pluggable but generally not changed in a runtime but initialized using an HM configuration file on HM startup.
-
-h4. Alerts from agent
-
-HM subscribes to agent alerts on a message bus:
-
-| *Subject* | hm.agent.alert.<agent_id> |
-| *Payload* | JSON containing the following keys: id, service, event, action, description, timestamp, tags |
-
-BOSH Agent is responsible for mapping any underlying supervisor alert format to the expected JSON payload and send it to HM.
-
-HM is responsible for interpreting JSON payload and mapping it to a sequence of HM actions and possibly creating an HM alert compatible with Alert Processor module. HM never dedups incoming alerts outside of Alert Processor (this adds some overhead to an incoming alert parser but shouldn't be too bad). Malformed payloads are ignored.
-
-Job name and index are not featured in agent incoming alert, those are looked up in director. If heartbeat came from a rogue agent and we have no job name and/or index then we note that fact in alert description but don't try to be too worried about that (service name and agent id should be enough). We might consider including agent IP address as a part of heartbeat so we can track down rogue agents.