solanum 0.2.0 → 1.0.0

data/README.md

@@ -1,51 +1,57 @@
 Solanum
 =======
 
-This gem provides a domain-specific language (DSL) for collecting metrics
-data in Ruby. The `solanum` script takes a number of monitoring configuration
-scripts as arguments and periodically collects the metrics defined. The results
-can be printed to the console or sent to a [Riemann](http://riemann.io/) server.
-This requires the `riemann-client` gem to work.
-
-## Structure
-
-Solanum scripts define _sources_, which provide some string input when they are
-read. This input is processed by a set of _matchers_ for each source, which can
-generate named measurements from that data. A simple example would be a file
-source, which is read and matched line-by-line against a set of regular
-expressions.
-
-The emitted measurements can undergo a bit more processing before being
-reported. For example, some metrics are monotonically-increasing counters, and
-what we actually want is the _difference_ between each reading. For others, we
-may want to apply threshold-based states to the events. These are set by
-_service prototypes_, which are also defined in the scripts.
-
-## Examples
-
-Here's an example of reading some information about the current system memory:
-
-```ruby
-# Read memory usage.
-read "/proc/meminfo" do
-  match /^MemTotal:\s+(\d+) kB$/, cast: :to_i, scale: 1024, record: 'memory total bytes'
-  match /^MemFree:\s+(\d+) kB$/,  cast: :to_i, scale: 1024, record: 'memory free bytes'
-end
-
-# Calculate percentages from total space.
-compute do |metrics|
-  total = metrics['memory total bytes']
-  free = metrics['memory free bytes']
-  if total && free
-    metrics['memory free pct'] = free.to_f/total
-  end
-end
-
-# Define a service prototype with a threshold-based state.
-service 'memory free pct', state: thresholds(0.00, :critical, 0.10, :warning, 0.25, :ok)
-```
-
-See the files in the `examples` directory for more monitor configuration samples.
+This gem provides a monitoring daemon which can be configured to collect data
+from a variety of pluggable sources. The results can be printed to the console
+or sent to a [Riemann](http://riemann.io/) server. This requires the
+`riemann-client` gem to work.
+
+
+## Installation
+
+**TODO**
+
+
+## Metric Events
+
+Solanum represents each measurement datapoint as an _event_. Each event must
+have at minimum a `service` and `metric` with the measurement name and value,
+respectively. Events may also contain other attributes such as a `state`, `ttl`,
+`tags`, and so on - see the [Riemann concepts](http://riemann.io/concepts.html)
+page for more details.
+
+
+## Configuration
+
+Solanum is configured using one or more YAML files. These specify common event
+attributes, sources, and outputs.
+
+See the [example config](config.yml) in this repo for possible config options.
+
+### Defaults
+
+The `defaults` section of the config provides common attributes to apply to
+every event. This can be used to provide a common TTL, tags, and more.
+
+### Sources
+
+A _source_ is a class which extends `Solanum::Source` and implements the
+`collect!` method to return metric events. Solanum comes with several metric
+sources built in, including basic host-level monitoring of CPU usage, load,
+memory, diskstats, network, and more.
+
+Additional custom sources can be provided, as long as they are in Ruby's lib
+path for the daemon.
+
+### Outputs
+
+An _output_ is a destination to report the collected events to. The simplest
+one is the `print` output, which writes each event to STDOUT. This is useful for
+debugging, but you probably won't leave it on for deployed daemons.
+
+The other included choice is the `riemann` output, which sends each event to a
+Riemann monitoring server.
+
 
 ## License
 

data/bin/solanum

@@ -6,16 +6,13 @@ require 'optparse'
 require 'solanum'
 
 $options = {
-  riemann_host: nil,
-  riemann_port: 5555,
-  interval: 5,
-  verbose: false,
+  period: 10,
 }
 
 $defaults = {
-  host: %x{hostname}.chomp,
+  host: %x{hostname --fqdn}.chomp,
   tags: [],
-  ttl: 10,
+  ttl: 60,
 }
 
 def fail(msg, code=1)
@@ -23,25 +20,18 @@ def fail(msg, code=1)
   exit code
 end
 
-def log(msg)
-  puts "%s %s" % [Time.now.strftime("%H:%M:%S"), msg] if $options[:verbose]
-end
-
 # Parse command-line options.
 options = OptionParser.new do |opts|
-  opts.banner = "Usage: #{File.basename($0)} [options] <monitor config> [monitor config] ..."
+  opts.banner = "Usage: #{File.basename($0)} [options] <config.yml> [config2.yml ...]"
   opts.separator ""
   opts.separator "Event Attributes:"
   opts.on(      '--host HOST', "Event hostname (default: #{$defaults[:host]})") {|v| $defaults[:host] = v }
-  opts.on('-a', '--attribute KEY=VAL', "Attribute to add to the event (may be given multiple times)") {|attr| k,v = attr.split(/=/); if k and v then $defaults[k.intern] = v end }
+  opts.on('-a', '--attribute KEY=VAL', "Attribute to add to every event (may be given multiple times)") {|attr| k,v = attr.split(/=/); if k and v then $defaults[k.intern] = v end }
   opts.on('-t', '--tag TAG', "Tag to add to events (may be given multiple times)") {|v| $defaults[:tags] << v }
-  opts.on(      '--ttl SECONDS', "Default TTL for events (default: #{$options[:ttl]})") {|v| $defaults[:ttl] = v.to_i }
+  opts.on(      '--ttl SECONDS', "Default TTL for events (default: #{$defaults[:ttl]})") {|v| $defaults[:ttl] = v.to_i }
   opts.separator ""
   opts.separator "General Options:"
-  opts.on(      '--riemann-host HOST', "Riemann host to report events to") {|v| $options[:riemann_host] = v }
-  opts.on(      '--riemann-port PORT', "Riemann port (default: #{$options[:riemann_port]})") {|v| $options[:riemann_port] = v.to_i }
-  opts.on('-i', '--interval SECONDS', "Seconds between updates (default: #{$options[:interval]})") {|v| $options[:interval] = v.to_i }
-  opts.on('-v', '--verbose', "Print additional information to stdout") { $options[:verbose] = true }
+  opts.on('-p', '--period SECONDS', "Seconds between updates (default: #{$options[:period]})") {|v| $options[:period] = v.to_i }
   opts.on('-h', '--help', "Displays usage information") { print opts; exit }
 end
 options.parse!
@@ -49,46 +39,14 @@ options.parse!
 # Check usage.
 fail options if ARGV.empty?
 
-
-
-##### MONITORING CONFIGS #####
-
+# Construct monitoring system.
 $solanum = Solanum.new(ARGV)
 fail "No sources loaded!" if $solanum.sources.empty?
 
-if $options[:riemann_host]
-  begin
-    require 'riemann/client'
-  rescue LoadError
-    fail "ERROR: could not load Riemann client library! `gem install riemann-client` to enable reporting"
-  end
-
-  $riemann = Riemann::Client.new(host: $options[:riemann_host], port: $options[:riemann_port])
-end
-
-
-
-##### REPORT LOOP #####
-
+# Handle ^C interrupts gracefully.
 trap "SIGINT" do
   exit
 end
 
-loop do
-  $solanum.collect!
-  events = $solanum.build_events($defaults)
-
-  events.each do |event|
-    if $options[:verbose] || $riemann.nil?
-      puts "%-40s %5s (%s) %s" % [
-        event[:service], event[:metric],
-        event[:state].nil? ? "--" : event[:state],
-        event.inspect
-      ]
-    end
-
-    $riemann << event if $riemann
-  end
-
-  sleep $options[:interval]
-end
+# Scheduling loop.
+$solanum.run!

data/lib/solanum.rb

@@ -1,81 +1,136 @@
+require 'solanum/config'
+require 'solanum/schedule'
+require 'thread'
+
+
 # Class which wraps up an active Solanum monitoring system into an object.
-#
-# Author:: Greg Look
 class Solanum
-  attr_reader :sources, :services, :metrics
+  attr_reader :defaults, :sources, :outputs
 
-  require 'solanum/config'
-  require 'solanum/source'
+  # Merge two event attribute maps together, concatenating tags.
+  def self.merge_attrs(a, b)
+    stringify = lambda do |x|
+      o = {}
+      x.keys.each do |k|
+        o[k.to_s] = x[k]
+      end
+      o
+    end
+
+    if a.nil?
+      stringify[b]
+    elsif b.nil?
+      stringify[a]
+    else
+      a = stringify[a]
+      b = stringify[b]
+      tags = a['tags'] ? a['tags'].dup : []
+      tags.concat(b['tags']) if b['tags']
+      tags.uniq!
+      x = a.dup.merge(b)
+      x['tags'] = tags unless tags.empty?
+      x
+    end
+  end
 
 
-  # Loads the given monitoring scripts and initializes the sources and service
-  # definitions.
-  def initialize(scripts)
+  # Loads the given configuration file(s) and initializes the system.
+  def initialize(config_paths)
+    @defaults = {tags: []}
     @sources = []
-    @services = []
-    @metrics = {}
+    @outputs = []
 
-    scripts.each do |path|
-      begin
-        config = Solanum::Config.new(path)
-        @sources.concat(config.sources)
-        @services.concat(config.services)
-      rescue => e
-        STDERR.puts "Error loading monitor script #{path}: #{e}"
-      end
+    # Load and merge files.
+    config_paths.each do |path|
+      conf = Config.load_file(path)
+
+      # merge defaults, update tags
+      @defaults = Solanum.merge_attrs(@defaults, conf[:defaults])
+
+      # sources and outputs are additive
+      @sources.concat(conf[:sources])
+      @outputs.concat(conf[:outputs])
+    end
+
+    # Add default print output.
+    if @outputs.empty?
+      require 'solanum/output/print'
+      @outputs << Solanum::Output::Print.new()
     end
 
+    @defaults.freeze
+    @outputs.freeze
     @sources.freeze
-    @services.freeze
+
+    @schedule = Solanum::Schedule.new
+    @sources.each_with_index do |source, i|
+      @schedule.insert!(source.next_run, i)
+    end
   end
 
 
-  # Collects metrics from the given sources, in order. Updates the internal
-  # merged map of metric data.
-  def collect!
-    @old_metrics = @metrics
-    @metrics = @sources.reduce({}) do |metrics, source|
-      begin
-        new_metrics = source.collect(metrics) || {}
-        metrics.merge(new_metrics)
-      rescue => e
-        STDERR.puts "Error collecting metrics from #{source}: #{e}"
-        metrics
+  # Reschedule the given source for later running.
+  def reschedule!(source)
+    idx = nil
+    @sources.each_with_index do |s, i|
+      if s == source
+        idx = i
+        break
       end
     end
+    raise "Source #{source.inspect} is not present in source list!" unless idx
+    @schedule.insert!(source.next_run, idx)
+    @scheduler.wakeup
   end
 
 
-  # Builds full events from a set of service prototypes, old metrics, and new
-  # metrics.
-  def build_events(defaults={})
-    @metrics.keys.sort.map do |service|
-      value = @metrics[service]
-      prototype = @services.select{|m| m[0] === service }.map{|m| m[1] }.reduce({}, &:merge)
+  # Report a batch of events to all reporters.
+  def record!(events)
+    # TODO: does this need locking?
+    @outputs.each do |output|
+      output.write_events events
+    end
+  end
 
-      state = prototype[:state] ? prototype[:state].call(value) : :ok
-      tags = ((prototype[:tags] || []) + (defaults[:tags] || [])).uniq
-      ttl = prototype[:ttl] || defaults[:ttl]
 
-      if prototype[:diff]
-        last = @old_metrics[service]
-        if last && last <= value
-          value = value - last
-        else
-          value = nil
+  # Run collection from the given source in a new thread.
+  def collect_events!(source)
+    Thread.new do
+      begin
+        events = source.collect!
+        attrs = Solanum.merge_attrs(@defaults, source.attributes)
+        events = events.map do |event|
+          Solanum.merge_attrs(attrs, event)
         end
+        record! events
+      rescue => e
+        STDERR.puts "Error collecting events from source #{source.type}: #{e}"
       end
+      reschedule! source
+    end
+  end
+
 
-      if value
-        defaults.merge({
-          service: service,
-          metric: value,
-          state: state.to_s,
-          tags: tags,
-          ttl: ttl
-        })
+  # Runs the collection loop.
+  def run!
+    @scheduler = Thread.current
+    loop do
+      # Determine when next scheduled source should run, and sleep if needed.
+      duration = @schedule.next_wait || 1
+      if 0 < duration
+        sleep duration
+        next
       end
-    end.compact
+
+      # Get the next ready source.
+      idx = @schedule.pop_ready!
+      source = @sources[idx] if idx
+      next unless source
+      #puts "Source #{source.type} is ready to run!" # DEBUG
+
+      # Start thread to collect and report events.
+      collect_events! source
+    end
   end
 
 end

data/lib/solanum/config.rb

@@ -1,97 +1,90 @@
-require 'solanum/source'
+require 'yaml'
 
-class Solanum::Config
-  attr_reader :sources, :services
+class Solanum
+module Config
 
-  def initialize(path)
-    @sources = []
-    @services = []
-
-    instance_eval ::File.readlines(path).join, path, 1
-
-    raise "No sources loaded from monitor script: #{path}" if @sources.empty?
+  # Helper method to clear the type cache.
+  def self.clear_type_cache!
+    @@type_classes = {}
   end
 
 
-  private
-
-  # Registers a new source object. If a block is given, it is used to configure
-  # the source with instance_exec.
-  def register_source(source, config=nil)
-    source.instance_exec &config if config
-    @sources << source
-    source
-  end
+  # Resolve a type based on a library path.
+  def self.resolve_type(namespace, type, lib_path=nil, class_name=nil)
+    @@type_classes ||= {}
 
+    type_key = "#{namespace}:#{type}"
+    return @@type_classes[type_key] if @@type_classes.include?(type_key)
 
-  # Registers a source which runs a command and matches against output lines.
-  def run(command, &config)
-    register_source Solanum::Source::Command.new(command), config
-  end
+    lib_path ||= type.include?('/') ? type : "solanum/#{namespace}/#{type}"
+    if class_name
+      cls_path = class_name.split('::')
+    else
+      cls_path = lib_path.split('/').map {|w| w.capitalize }
+    end
 
+    begin
+      require lib_path
+      cls = cls_path.inject(Object) do |mod, class_name|
+        mod.const_get(class_name) if mod
+      end
+      STDERR.puts "Unable to resolve class #{cls_path.join('::')}" unless cls
+      @@type_classes[type_key] = cls
+    rescue LoadError => e
+      STDERR.puts "Unable to load code for #{type_key} type: #{e}"
+      @@type_classes[type_key] = nil
+    end
 
-  # Registers a source which matches against the lines in a file.
-  def read(path, &config)
-    register_source Solanum::Source::File.new(path), config
+    @@type_classes[type_key]
   end
 
 
-  # Registers a source which computes metrics directly.
-  def compute(&block)
-    register_source Solanum::Source::Compute.new(block)
+  # Resolves a type config string and constructs a new instance of it. Memoizes
+  # the results of loading the class in the `@@type_classes` field.
+  def self.construct_type(namespace, type, args)
+    cls = resolve_type(namespace, type, args['lib_path'], args['class'])
+    if cls.nil?
+      STDERR.puts "Skipping construction of failed #{namespace} type #{type}"
+      nil
+    else
+      begin
+        #puts "#{cls}.new(#{args.inspect})" # DEBUG
+        cls.new(args)
+      rescue => e
+        STDERR.puts "Error constructing #{namespace} type #{type}: #{args.inspect} #{e}"
+        nil
+      end
+    end
   end
 
 
-  # Registers a pair of [matcher, prototype] where matcher is generally a string
-  # or regex to match a service name, and prototype is a map of :ttl, :state,
-  # :tags, etc.
-  def service(service, prototype={})
-    @services << [service, prototype]
-  end
+  # Load the given configuration file. Returns a map with initialized :sources
+  # and :outputs.
+  def self.load_file(path)
+    config = File.open(path) {|f| YAML.load(f) }
 
+    defaults = config['defaults'] || {}
 
-  ##### HELPER METHODS #####
-
-  # Creates a state function based on thresholds. If the first argument is a
-  # symbol, it is taken as the default service state. Otherwise, arguments should
-  # be alternating numeric thresholds and state values to assign if the metric
-  # value exceeds the threshold.
-  #
-  # For example, for an 'availability' metric you often want to warn on low
-  # values. To assign a 'critical' state to values between 0% and 10%,
-  # 'warning' between 10% and 25%, and 'ok' above, use the following:
-  #
-  #     thresholds(0.00, :critical, 0.10, :warning, 0.25, :ok)
-  #
-  # For 'usage' metrics it's the inverse, giving low values ok states and
-  # warning about high values:
-  #
-  #     thresholds(:ok, 55, :warning, 65, :critical)
-  #
-  def thresholds(*args)
-    default_state = nil
-    default_state = args.shift unless args.first.kind_of? Numeric
-
-    # Check arguments.
-    raise "Thresholds must be paired with state values" unless args.count.even?
-    args.each_slice(2) do |threshold|
-      limit, state = *threshold
-      raise "Limits must be numeric: #{limit}" unless limit.kind_of? Numeric
-      raise "State values must be strings or symbols: #{state}" unless state.instance_of?(String) || state.instance_of?(Symbol)
+    # Construct sources from config.
+    source_configs = config['sources'] || []
+    sources = source_configs.map do |conf|
+      self.construct_type('source', conf['type'], conf)
     end
+    sources.reject!(&:nil?)
 
-    # State block.
-    lambda do |v|
-      state = default_state
-      args.each_slice(2) do |threshold|
-        if threshold[0] < v
-          state = threshold[1]
-        else
-          break
-        end
-      end
-      state
+    # Construct outputs from config.
+    output_configs = config['outputs'] || []
+    outputs = output_configs.map do |conf|
+      self.construct_type('output', conf['type'], conf)
     end
+    outputs.reject!(&:nil?)
+
+    {
+      defaults: defaults,
+      sources: sources,
+      outputs: outputs,
+    }
   end
 
 end
+end