scout_agent 3.0.7 → 3.1.0

data/CHANGELOG

@@ -1,3 +1,16 @@
+== 3.1.0
+
+* Fixed a bug where the monitor might not properly signal all subprocesses
+* Made the master agent forward shutdown requests to running missions
+* Enhanced the stop command to ensure all processes exit and to skip the waits
+  on KILL signals since they cannot be responded to
+* Improved the error message for forced stops to give next steps
+* Fixed a bug that would prevent the agent from daemonizing itself at start-up
+  if it had to clean out a stale PID file first
+* Added status database and log file maintenance to the snapshot and queue
+  commands
+* Completed full code documentation and clean-up pass
+
 == 3.0.7
 
 * Added a log() method (with a logger() alias) plugins can access to add

data/README

@@ -1,3 +1,45 @@
-= ReadMe
+= The Scout Agent
 
-Scout, Version 3
+This is the agent software installed on servers to work with {the Scout monitoring application}[http://scoutapp.com/].  See the sections below for details on how to install and use the agent, how to build your own plugins for it to track the data you care about, and how to add features to the agent itself.
+
+== How do I use the Scout agent?
+
+Installing the agent is just a simple gem install:
+
+  $ sudo gem install scout_agent
+
+Note that the gem requires Ruby 1.8.6 or higher and Rubygems 1.3.1 or higher.  It also doesn't run on Windows due to Ruby not supporting fork() there.
+
+Once the gem is installed, you need to identify yourself with the agent key (which looks like a7349498-bec3-4ddf-963c-149a666433a4) that you get from the Web application.  Just issue this command and have your key ready when it asks for it:
+
+  $ sudo scout_agent id
+
+At this point, you should be all set to run the agent.  You start it up with this command:
+
+  $ sudo scout_agent start
+
+The agent is a daemon, so it should return your prompt after it moves into the land of background processes.  It will be running though.  You can issue the following command if you want to check up on it:
+
+  $ scout_agent status
+
+With the agent running, you should be able to log into your account on the {Web application}[http://scoutapp.com/] to setup your list of plugins and see the agent delivering data.
+
+== How do I build my own plugins?
+
+Scout makes it very easy to build your own plugins for anything you need to monitor.  In a matter of minutes you could be tracking the user sign-ups in your application or anything else that's important to you.  Once you pipe some data into Scout you can take advantages of all the graphing and trend analysis we use for more traditional monitoring, like Rails applications.
+
+We have {a tutorial}[http://scoutapp.com/plugin_urls/static/creating_a_plugin] on the Web site that walks you through building a Scout plugin.
+
+== How do I hack on the agent?
+
+We try to keep the agent code fairly clean and documented, so it's hopefully not too tough to poke around in.  However, it is a big code base.  Let me give you the dime tour of where to look for things.  All paths below are relative to <tt>lib/scount_agent</tt>.
+
+<tt>dispatcher.rb</tt>, <tt>assignment.rb</tt>, and <tt>assignment/*</tt>:: This is the code the agent uses to interpret commands users give on the command-line.  You'll find configuration file loading (<tt>plan.rb</tt> is a configuration), switch parsing, command selection, and invocation in here.
+<tt>api.rb</tt>:: The ScoutAgent::API is the external interface for the queue and snapshot commands.  This can be used to push data into Scout, without even building a Plugin, or just to request an updated snapshot of the environment.
+<tt>lifeline.rb</tt> and <tt>agent.rb</tt>:: The ScoutAgent::Lifeline object monitors a ScoutAgent::Agent class, which is a major function of Scout, namely the plugin runner and the XMPP communication module.  This is a pretty typical multi-process heartbeat setup where the Agent is fork()ed into a separate process and then monitored for regular check-ins written to a shared pipe.
+<tt>agent/master_agent.rb</tt> and <tt>mission.rb</tt>:: Together these two pieces make up the heart of the agent.  The ScoutAgent::Agent::MasterAgent is the main event loop and ScoutAgent::Mission (aliased Plugin) are the pieces of code that get run in that loop.
+<tt>agent/communication_agent.rb</tt>, <tt>order.rb</tt>, and <tt>order/*</tt>:: This code is used to listen for supported commands over an XMPP connection.  The ScoutAgent::Agent::CommunicationAgent manages all the XMPP talking and ScoutAgent::Order and subclasses are the commands.
+<tt>database.rb</tt> and <tt>database/*</tt>:: This is a thin wrapper over Amalgalite[http://copiousfreetime.rubyforge.org/amalgalite/] (and SQLite databases by extension).  These are the memory of the agent and, with locking, the primary IPC used used by the agent.
+<tt>core_extensions.rb</tt>:: This file holds a handful of extensions that make sense in the context of the agent.  This is not an ActiveSupport size library, but just some simple niceties.  These extensions can be used in your own Plugins.
+
+We welcome additions to the agent and will incorporate patches if we feel they add to the platform as a whole.  Obviously, the easier we can understand what you did the easier it is to judge that, so tests and documentation are plusses to us.

data/Rakefile

@@ -118,7 +118,7 @@ task :upload_docs => :rdoc do
   )
   host       = "#{config['username']}@rubyforge.org"
   remote_dir = "/var/www/gforge-projects/#{SA_SPEC.rubyforge_project}"
-  local_dir  = 'doc/html'
+  local_dir  = "doc"
 
   sh "rsync -av --delete #{local_dir}/ #{host}:#{remote_dir}"
 end

data/TODO

@@ -1,3 +1,9 @@
 = To Do List
 
-Coming soon...
+* Build a `scout_agent help` command that provides general and command specific
+  help, perhaps by parsing the comments on assignments
+* Add SSL certificate verification show we ensure we are always talking with the
+  official Scout server
+* Add something like a SHA signature of the code to reports to help developers
+  see which version they are looking at data from
+* Improve test coverage all over the agent

data/lib/scout_agent.rb

@@ -64,9 +64,34 @@ module ScoutAgent
     wire_tap.tap      = $stdout unless skip_stdout or Plan.run_as_daemon?
     wire_tap
   end
+  
+  #
+  # A maintenance method used to remove log files written more than seven days
+  # ago.  This prevents the hard drive from slowly filling with log files and
+  # thus is called as part of the main event loop as well as after snapshot and
+  # queue commands.  A recent +log+ must be provided to notify of rotation
+  # errors.
+  # 
+  def self.remove_old_log_files(log)
+    Plan.log_dir.each_entry do |log_file|
+      if log_file.to_s =~ /\.(\d{4})(\d{2})(\d{2})\z/
+        log_day = Time.local(*$~.captures.map { |n| n.to_i })
+        if Time.now - log_day > 60 * 60 * 24 * 7
+          begin
+            (Plan.log_dir + log_file).unlink
+          rescue Exception => error  # file cannot be unlinked
+            log.error( "Failed to unlink old log file '#{log_file}':  " +
+                       "#{error.message} (#{error.class})." )
+            next
+          end
+          log.debug("Successfully unlinked old log file '#{log_file}'.")
+        end
+      end
+    end
+  end
 
   # The version of this agent.
-  VERSION = "3.0.7".freeze
+  VERSION = "3.1.0".freeze
   # A Pathname reference to the agent code directory, used in dynamic loading.
   LIB_DIR = Pathname.new(File.dirname(__FILE__)) + agent_name
 end

data/lib/scout_agent/agent/communication_agent.rb

@@ -6,9 +6,20 @@ require "scout_agent/order"
 
 module ScoutAgent
   class Agent
+    #
+    # This agent manages the XMPP connection with the server.  It mainly just
+    # listens for messages from the server and passes them on to matching Order
+    # instances.
+    # 
     class CommunicationAgent < Agent
+      # The number of seconds to wait before attempting another connection.
       RECONNECT_WAIT = 60
       
+      #
+      # Prepares a log() and passses it down to the Orders, which are also
+      # loaded here.  A list of trusted XMPP users is also prepared as part of
+      # this start-up.
+      # 
       def initialize
         super  # setup our log and status
 
@@ -26,6 +37,11 @@ module ScoutAgent
         }
       end
       
+      # 
+      # This method encupsulates the process of the XMPP listener, which is
+      # pretty much:  login, setup, listen for commands until told to stop, and 
+      # exit.
+      # 
       def run
         login
         update_status("Online since #{Time.now.utc.to_db_s}")
@@ -35,7 +51,8 @@ module ScoutAgent
         listen
         close_connection
       end
-      
+
+      # Triggers the shutdown process.
       def finish
         log.info("Shutting down.")
         if @shutdown_thread
@@ -45,8 +62,18 @@ module ScoutAgent
         end
       end
       
+      #######
       private
+      #######
       
+      #######################
+      ### XMPP Operations ###
+      #######################
+      
+      # 
+      # Prepares a Jabber::Client with identification and Exception handling, 
+      # then hands off to try_connection().
+      # 
       def login
         Thread.abort_on_exception = true  # make XMPP4R fail fast
         @agent_jid = Jabber::JID.new("#{agent_key}@#{jabber_server}/agent")
@@ -58,6 +85,10 @@ module ScoutAgent
         try_connection
       end
       
+      # 
+      # Loops over connection attempts until successfully reaching the server.
+      # There's a pause of <tt>RECONNECT_WAIT</tt> seconds between each attempt.
+      # 
       def try_connection
         @connecting = true
         until connect_and_authenticate?
@@ -68,13 +99,18 @@ module ScoutAgent
         @connecting = false
       end
       
+      #
+      # Attempts an XMPP connection and login.  The process will be cancelled if
+      # either part fails.  Returns +true+ if the entire process completes as
+      # expected, or +false+ otherwise.
+      # 
       def connect_and_authenticate?
         status("Connecting")
         close_connection
         begin
           no_warnings { @jabber.connect }
         rescue Exception => error  # connection failure
-          log.error("Failed to connect (#{error.class}:  #{error.message}).")
+          log.error("Failed to connect to XMPP server.")
           return false
         end
         begin
@@ -87,6 +123,10 @@ module ScoutAgent
         true
       end
       
+      #
+      # Builds an XMPP status with +message+ and arranges for it to be sent to
+      # the server in a separate Thread.
+      # 
       def update_status(message, status = nil)
         status("Queuing status change")
         presence        = Jabber::Presence.new
@@ -104,11 +144,19 @@ module ScoutAgent
         end
       end
       
+      # 
+      # Grabs a _roster_ that can be used to manage the subscription requests of
+      # this agent's XMPP user.
+      # 
       def fetch_roster
         status("Preparing connection")
         @roster = Jabber::Roster::Helper.new(@jabber)
       end
       
+      #
+      # Installs a callback that will accept all subscription requests from
+      # trusted XMPP identities.
+      # 
       def install_subscriptions_callback
         @roster.add_subscription_request_callback do |_, presence|
           log.info("Subscription request:  #{presence.from}")
@@ -121,6 +169,15 @@ module ScoutAgent
         end
       end
       
+      # 
+      # Installs a callback that reads XMPP messages looking for supported
+      # commands.  The identified commands are handed off to an Order subclass
+      # for execution.
+      # 
+      # Before processing a command, this process will strip a leading message
+      # ID, if provided.  A response is sent to the server aknowledging the
+      # receit of the message in such cases.
+      # 
       def install_messages_callback
         @jabber.add_message_callback do |message|
           log.info("Received message from #{message.from}:  #{message.body}")
@@ -144,6 +201,10 @@ module ScoutAgent
         end
       end
       
+      # 
+      # Sends a chat message with +body+ to the XMPP identity named by +who+.
+      # Messages are sent in a separate Thread.
+      # 
       def send_chat_message(who, body)
         status("Queuing message")
         message      = Jabber::Message.new(who, body)
@@ -160,6 +221,7 @@ module ScoutAgent
         end
       end
       
+      # Stops this main Thread to allow the listening Thread to take over.
       def listen
         log.info("Listening for commands.")
         status("Listening for commands")
@@ -167,6 +229,7 @@ module ScoutAgent
         Thread.stop
       end
       
+      # Closes our XMPP connection, if it's still open.
       def close_connection
         @jabber.close! if @jabber.is_connected?
       rescue Exception  # connection already closed
@@ -174,17 +237,27 @@ module ScoutAgent
         log.warn("Failed to close connection.")
       end
       
+      ###############
+      ### Helpers ###
+      ###############
+      
+      # Returns an appropriate key for the environment.
       def agent_key
         @agent_key ||= Plan.test_mode?                        ?
                        "a7349498-bec3-4ddf-963c-149a666433a4" :
                        Plan.agent_key
       end
       
+      # 
+      # Returns an appropriate server for the environment.  In non-test mode
+      # this will be the host() parsed out of the check-in URL.
+      # 
       def jabber_server
         @jabber_server ||= Plan.test_mode? ? "jabber.org" :
                                              URI.parse(Plan.server_url).host
       end
       
+      # Returns +true+ if +user+ matches any of our trusted XMPP identities.
       def trusted?(user)
         id = user.to_s
         @trusted.any? { |trusted| id =~ trusted }

data/lib/scout_agent/agent/master_agent.rb

@@ -6,16 +6,31 @@ require "scout_agent/mission"
 
 module ScoutAgent
   class Agent
+    #
+    # This agent is the main event loop for the platform.  It's primary function
+    # is to run each Mission downloaded from the server at the correct time.  As
+    # part of that, it regularly updates the Mission list from the server and
+    # also prepares and sends check-ins to the server after a set of mission
+    # runs.  Snapshots are also periodically run here and added to check-ins.
+    # 
+    # This loop also manages regular maintenance like <tt>VACUUM</tt>ing SQLite
+    # databases and cleaning out old log files.
+    # 
     class MasterAgent < Agent
+      #
+      # Prepares the primary event loop for execution.  The main function at
+      # this point is to ensure that we can load all of the needed databases.
+      # 
       def initialize
         super  # setup our log and status
         
-        @running   = true
-        @main_loop = nil
-        @server    = Server.new(log)
-        @db        = Database.load(:mission_log, log)
-        @queue     = Database.load(:queue,       log)
-        @snapshots = Database.load(:snapshots,   log)
+        @running     = true
+        @main_loop   = nil
+        @mission_pid = nil
+        @server      = Server.new(log)
+        @db          = Database.load(:mission_log, log)
+        @queue       = Database.load(:queue,       log)
+        @snapshots   = Database.load(:snapshots,   log)
         
         if [@db, @queue, @snapshots].any? { |db| db.nil? }
           log.fatal("Could not load all required databases.")
@@ -23,6 +38,11 @@ module ScoutAgent
         end
       end
       
+      #
+      # This method outlines the steps of the event loop:  update our plan from
+      # the server, run Missions and snapshots, check-in, handle maintenance, 
+      # and rest.
+      # 
       def run
         log.info("Running.")
         @main_loop = Thread.new do
@@ -42,31 +62,64 @@ module ScoutAgent
         @main_loop.join
       end
       
+      # 
+      # This method is called automatically when this Agent process receives an
+      # +ALRM+ signal and all it does is to wake up the event loop Thread, if it
+      # is not already running.  This allows the Agent to notice external
+      # changes quicker.
+      # 
       def notice_changes
         @main_loop.run if @main_loop
       rescue ThreadError  # Thread was already killed
         # do nothing:  we're shutting down and can't notice new things
       end
       
+      #
+      # This method is called automatically when this Agent process receives a
+      # stop request, like a +TERM+ signal.  It prepares the Agent to shutdown,
+      # but doesn't trigger an immediate stop.  Instead, the Agent will check to
+      # see if this has been called after each phase of the main event loop.
+      # This allows it to avoid repeating work when it relaunches.
+      # 
+      # This request is also forwarded to a currently running Mission, if there
+      # is one.
+      # 
       def finish
         if @running
           log.info("Shutting down.")
+          if @mission_pid
+            log.info("Forwarding shutdown request to the running mission.")
+            begin
+              Process.kill("TERM", @mission_pid)
+            rescue Exception  # unable to signal mission
+              log.warn("Mission could not be signaled.")
+              # do nothing:  mission already ended
+            end
+          end
         else
           log.warn("Received multiple shutdown signals.")
         end
         @running = false
         notice_changes
       end
-      
+
+      #######
       private
+      #######
       
       #############
       ### Agent ###
       #############
       
+      #
+      # Updates our plan from the server, if it hasn't changed.  This includes
+      # both the list of plugins to run as well as our list of commands involved
+      # in a snapshot.
+      # 
       def fetch_plan
         log.info("Fetching plan from server.")
         status("Fetching plan from server")
+        # read the plan
         headers = {}
         if not Plan.test_mode? and (old_plan = @db.current_plan)
           log.debug( "Adding If-Modified-Since for plan fetch:  " +
@@ -74,6 +127,7 @@ module ScoutAgent
           headers[:if_modified_since] = old_plan[:last_modified]
         end
         json_plan = @server.get_plan(headers)
+        # skip mission or empty plans
         if json_plan.nil?  # failed to retrieve plan
           log.warn("Could not retrieve plan from server.")
           return
@@ -83,24 +137,38 @@ module ScoutAgent
         else
           log.info("Received plan (#{json_plan.to_s.size} bytes).")
         end
+        # parse the plan
         begin
           ruby_plan = JSON.parse(json_plan.to_s)
         rescue JSON::ParserError  # bad JSON
           log.error("Plan from server was malformed JSON.")
           return  # skip plan update
         end
+        # update the local databases with the changes
         @db.update_plan( json_plan.headers[:last_modified],
                          Array(ruby_plan["plugins"]) )
         @snapshots.update_commands(Array(ruby_plan["commands"]))
       end
       
+      #
+      # This loop runs all Missions that have exceeded their run time wait, one
+      # at a time.  Missions are run in a child process to make it easy to time
+      # them out and clean up after them, as well as to ensure the agent isn't
+      # affected by their code.
+      # 
+      # The execution process outlined for a Mission here is:  create a child 
+      # process, compile the Mission code in that process, run the code in that
+      # process, and have that process record the results in the database so
+      # this process can access them.
+      # 
       def execute_missions
         status("Running missions")
         ran_a_mission = false
-        while mission = @db.current_mission
+        while mission = @db.current_mission  # loop over pending Missions
           log.info("Running #{mission[:name]} mission.")
           ran_a_mission = true
-          pid           = fork do
+          # run a Mission
+          @mission_pid  = fork do
             reset_environment
             compile_mission(mission)
             run_mission(mission)
@@ -108,10 +176,12 @@ module ScoutAgent
           end
           
           begin
+            # wait for the Mission to complete, or the timeout to expire
             Timeout.timeout(mission[:timeout]) do
-              Process.wait(pid)
+              Process.wait(@mission_pid)
             end
-            unless $?.success?
+            @mission_pid = nil
+            unless $?.success?  # record that the Mission exited with an error
               log.warn( "#{mission[:name]} exited with an error:  " +
                         "#{$?.exitstatus}." )
               @db.write_report(
@@ -121,8 +191,9 @@ module ScoutAgent
                 :body    => "Exit status:  #{$?.exitstatus}"
               )
             end
-          rescue Timeout::Error  # mission exceeded allowed execution
-            status = Process.term_or_kill(pid)
+          rescue Timeout::Error  # Mission exceeded allowed execution
+            status       = Process.term_or_kill(@mission_pid)
+            @mission_pid = nil
             log.error( "#{mission[:name]} took too long to run:  " +
                        "#{status && status.exitstatus}." )
             @db.write_report(
@@ -138,9 +209,16 @@ module ScoutAgent
             break
           end
         end
+        # we shouldn't wake up with nothing to do, so check for that
         log.warn("No missions to run.") unless ran_a_mission
       end
       
+      #
+      # Request a snapshot via the API.  This is a normal (non-forced) request,
+      # so only commands that have passed their interval will be run and it's
+      # likely that nothing at all will be done (because the last trip through
+      # the event loop ran a full snapshot, for example).
+      # 
       def prepare_snapshot
         if Plan.periodic_snapshots?
           status("Preparing a system snapshot")
@@ -150,10 +228,18 @@ module ScoutAgent
         end
       end
       
+      #
+      # Sends all generated data up to the Scout server.  This is not mission
+      # critical data so it is removed from the databases as it is sent.  This
+      # prevents something like a slow send that times out on our end but does
+      # eventually complete from causing us to later send duplicated data.
+      # 
       def checkin
+        # get the data from the databases
         reports   = @db.current_reports
         queued    = @queue.queued_reports
         snapshots = @snapshots.current_runs
+        # ensure we have something to send
         if reports.empty? and queued.empty? and snapshots.empty?
           log.warn("No data to report to the server.")
           return
@@ -161,16 +247,18 @@ module ScoutAgent
         
         log.info("Checking in with server.")
         status("Checking in with server")
+        # prepare the data for transport to the server
         checkin = { :reports => Array.new,
                     :hints   => Array.new,
                     :alerts  => Array.new,
                     :errors  => Array.new }
         (reports + queued).each do |report|
-          type                 =  report.delete_at(:type)
+          type                       =  report.delete_at(:type)
           checkin["#{type}s".to_sym] << report.to_hash
         end
         checkin[:snapshots] = snapshots.map { |run| run.to_hash }
         
+        # log some details about what we are sending
         report_dates = String.new
         if reports.first or queued.first
           dates                 = [ [reports.first, queued.first],
@@ -191,6 +279,7 @@ module ScoutAgent
                   "#{checkin[:alerts].size} alerts, "                    +
                   "and #{checkin[:errors].size} errors)#{report_dates} " +
                   "and #{snapshots.size} snapshot runs#{snapshot_dates}." )
+        # transmit the data and record the results
         if @server.post_checkin(checkin)
           log.info("Server received data.")
         else
@@ -198,6 +287,11 @@ module ScoutAgent
         end
       end
       
+      #
+      # Performs the regular maintenance needed to keep the agent from slowly
+      # filling the hard drive with data.  It <tt>VACUUM</tt>s databases to
+      # reclaim space and removes old log files.
+      # 
       def perform_maintenance
         log.info("Running maintenance tasks.")
         status("Running maintenance tasks")
@@ -213,23 +307,10 @@ module ScoutAgent
         end
         
         # clean out old logs
-        Plan.log_dir.each_entry do |log_file|
-          if log_file.to_s =~ /\.(\d{4})(\d{2})(\d{2})\z/
-            log_day = Time.local(*$~.captures.map { |n| n.to_i })
-            if Time.now - log_day > 60 * 60 * 24 * 7
-              begin
-                (Plan.log_dir + log_file).unlink
-              rescue Exception => error  # file cannot be unlinked
-                log.error( "Failed to unlink old log file '#{log_file}':  " +
-                           "#{error.message} (#{error.class})." )
-                next
-              end
-              log.debug("Successfully unlinked old log file '#{log_file}'.")
-            end
-          end
-        end
+        ScoutAgent.remove_old_log_files(log)
       end
       
+      # Rest for however much time we have before more work is needed.
       def wait_for_orders
         pause = @db.seconds_to_next_mission
         log.info("Waiting #{pause} seconds for next mission run.")
@@ -237,6 +318,10 @@ module ScoutAgent
         sleep pause
       end
       
+      # 
+      # Finish the shutdown process, if it was started while we were doing the
+      # last step of the main event loop.
+      # 
       def check_running_status
         exit unless @running
       end
@@ -245,9 +330,18 @@ module ScoutAgent
       ### Mission ###
       ###############
       
+      # 
+      # Reset our parent's signal handlers, authorize the Mission identity,
+      # prepare a log, and reset our status.
+      # 
       def reset_environment
         # swap out our parent's signal handlers
-        install_shutdown_handler { exit }
+        install_shutdown_handler do
+          Thread.new do
+            log.info("Shutting down.")
+            exit
+          end
+        end
 
         # clear the parent's identity and assume mine
         IDCard.me = nil
@@ -263,12 +357,13 @@ module ScoutAgent
         end
       end
       
+      # Build the +mission+ code or exit() with an error if it cannot be built.
       def compile_mission(mission)
         log.info("Compiling #{mission[:name]} mission.")
         status("Compiling")
         begin
           eval(mission[:code], TOPLEVEL_BINDING, mission[:name])
-        rescue Exception => error  # any compile error
+        rescue Exception => error       # any compile error
           raise if $!.is_a? SystemExit  # don't catch exit() calls
           log.error( "#{mission[:name]} could not be compiled:  " +
                      "#{error.message} (#{error.class})." )
@@ -282,6 +377,11 @@ module ScoutAgent
         end
       end
       
+      # 
+      # Create a Mission object from the code previously prepared, passing in
+      # details like <tt>:memory</tt> and <tt>:options</tt> from +mission+.
+      # Once this Mission is created, it is run().
+      # 
       def run_mission(mission)
         log.info("Preparing #{mission[:name]} mission.")
         if prepared = Mission.prepared
@@ -304,6 +404,7 @@ module ScoutAgent
         end
       end
       
+      # Report that +mission+ is now complete.
       def complete_mission(mission)
         log.info("#{mission[:name]} mission complete.")
       end