pencil 0.2.0 → 0.2.2

data/bin/pencil

@@ -3,4 +3,4 @@ $: << File.dirname($0) + "/../lib"
 
 require "dash"
 
-Rack::Handler::WEBrick.run(Dash::App, :Port => Dash::App.port)
+Rack::Handler::Mongrel.run(Dash::App, :Port => Dash::App.port)

data/docs/pencil_options.md

@@ -0,0 +1,176 @@
+# Pencil Options
+Pencil configuration files are written in YAML. When pencil starts up, it
+searches for these files and loads them. See the main README.md for how
+these files should look.
+
+## General Configuration
+
+These are options that go under the :config key in pencil configuration files.
+
+* :graphite_url [String, required, no default]
+
+  The url of your graphite instance.
+
+* :url_opts [Hash, required, no default]
+
+  A map of default graph options.
+
+  In addition to <a href="#gopts">graph-level options</a>, an important default option
+  you should set under :url_opts is
+
+    :start: TIMESPEC
+
+  TIMESPEC should be a
+  [chronic](http://chronic.rubyforge.org/)-parsable time, and to be useful
+  should be relative to the current time (e.g. "8 hours ago")
+
+* :refresh_rate [Fixnum, optional, default 60]
+
+  How often to refresh a changing view, in seconds.
+
+  This doesn't apply to timeslices that aren't varying (i.e not current, see
+  <a href="#threshold">:now_threshold</a>).
+
+  Set this to false to disable automatic refreshing.
+
+* :host_sort [["builtin", "numeric", "sensible"], optional, default "sensible"]
+
+  Set to "builtin" to sort using ruby's builtin String sort.
+
+  Set to "numeric" to sort hosts numerically (i.e. match secondarily on the
+  first \d+).
+
+  Set to "sensible" if you want to sort like this:
+
+  http://www.bofh.org.uk/2007/12/16/comprehensible-sorting-in-ruby
+
+* :quantum [Fixnum, optional, no default value]
+
+  Map requests to NUM second intervals. Pencil floors request times to the
+  minute, and does some modular arithmetic to do this mapping. This is
+  especially useful for implementing a caching layer, so that many requests
+  coming in near-simultaneously won't require graphite to generate different
+  images for each request.
+
+  Adding &noq=1 to a pencil url will disable this in case you need
+  super-granularity for some reason, but you didn't hear it from me.
+
+* :date_format [String, optional, default "%X %x"]
+
+  strftime format for displaying dates.
+
+* :metric_format [String, optional, default "%m.%c.%h"]
+
+  The format your graphite metrics are stored in. For pencil to work your
+  metrics need to be composed of three distinct pieces, concatenated in some
+  regular fashion. The format strings are
+
+  * %m metric
+  * %c cluster
+  * %h host
+
+  If you want a literal %[mch] in your metric format string you likely have
+  bigger problems than not being able to do so.
+
+* <a name="threshold"/> :now\_threshold: [Fixnum, optional, default 300]
+
+  How many seconds before Time.now an end time is considered to still be 'now',
+  for the purposes of adding meta-refresh and displaying time intervals.
+
+## <a name="gopts"/> Graph-level Options
+This is a list of the supported graph-level options for pencil, which
+correspond to request(image)-level options for graphite. These options are
+key-value pairs, and are passed directly to graphite. Here is the list, with
+minor annotations:
+
+* vtitle: String (y-axis label)
+* yMin: Fixnum
+* yMax: Fixnum
+* lineWidth: Fixnum (line thickness in pixels)
+* areaMode: \[first, all, stacked\] (see graphite documentation)
+* template: \[noc, alphas\] (alphas inverts colors)
+* lineMode: staircase
+* bgcolor: String
+* graphOnly: bool (hide legend, axes, grid)
+* hideAxes: bool
+* hideGrid: bool
+* hideLegend: bool
+* fgcolor: String
+* fontSize: Fixnum
+* fontName: String (see your graphite instance for available fonts)
+* fontItalic: bool
+* fontBold: bool
+
+## Target-level Options
+This is a list of the supported target-level options for pencil. These are
+mosly a list of transformations graphite supports, including summation and
+scaling of metrics. You can apply them to individual metrics, or lists of
+metrics. See the example configs for how this works. Also see the graphite
+composer for the effects of these options, many of which are untested.
+
+### Combinations
+These functions take an arbitrary number of targets (usually simple metrics)
+for arguments.
+
+* sumSeries
+* averageSeries
+* minSeries
+* maxSeries
+* group
+
+### Transformations
+Some of these options take a single argument.
+
+* scale
+* offset
+* derivative
+* integral
+* nonNegativeDerivative
+* log BASE
+* timeShift
+* summarize
+* hitcount
+
+### Calculations
+These functions take an arbitrary number of targets (usually simple metrics)
+for arguments.
+
+* movingAverage
+* stdev
+* asPercent
+* diffSeries
+* ratio
+
+### Filters
+Most of these options take a single argument.
+
+* highestCurrent
+* lowestCurrent
+* nPercentile
+* currentAbove
+* currentBelow
+* highestAverage
+* lowestAverage
+* averageAbove
+* averageBelow
+* maximumAbove
+* maximumBelow
+* sortByMaxima
+* minimalist
+* limit
+* exclude
+
+### Special Operations
+* alias
+* key (alias for alias)
+* cumulative
+* drawAsInfinite
+* lineWidth
+* dashed
+* keepLastValue
+* substr
+* threshold
+* color
+
+Note: key and color are interpreted differently from the other options, which
+are more simply translated.

data/lib/config.rb

@@ -0,0 +1,103 @@
+require "rubygems"
+require "models"
+
+module Dash
+  class Config
+    include Dash::Models
+
+    attr_reader :dashboards
+    attr_reader :graphs
+    attr_reader :hosts
+    attr_reader :clusters
+    attr_reader :global_config
+
+    def initialize
+      port = 9292
+      @rawconfig = {}
+      @confdir = "."
+
+      optparse = OptionParser.new do |o|
+        o.on("-d", "--config-dir DIR",
+          "location of the config directory (default .)") do |arg|
+          @confdir = arg
+        end
+        o.on("-p", "--port PORT", "port to bind to (default 9292)") do |arg|
+          port = arg.to_i
+        end
+      end
+
+      optparse.parse!
+      reload!
+      @global_config[:port] = port
+    end
+
+    def reload!
+      configs = Dir.glob("#{@confdir}/*.y{a,}ml")
+      configs.each { |c| @rawconfig.merge!(YAML.load(File.read(c))) }
+
+      [:graphs, :dashboards, :config].each do |c|
+        if not @rawconfig[c]
+          raise "Missing config name '#{c.to_s}'"
+        end
+      end
+
+      @global_config = @rawconfig[:config]
+      # do some sanity checking of other configuration parameters
+      [:graphite_url, :url_opts].each do |c|
+        if not @global_config[c]
+          raise "Missing config name '#{c.to_s}'"
+        end
+      end
+
+      # possibly check more url_opts here as well
+      if @global_config[:url_opts][:start]
+        if !ChronicDuration.parse(@global_config[:url_opts][:start])
+          raise "bad default timespec in :url_opts"
+        end
+      end
+
+      @global_config[:default_colors] ||=
+        ["blue", "green", "yellow", "red", "purple", "brown", "aqua", "gold"]
+
+      if @global_config[:refresh_rate]
+        duration = ChronicDuration.parse(@global_config[:refresh_rate].to_s)
+        if !duration
+          raise "couldn't parse key :refresh_rate"
+        end
+        @global_config[:refresh_rate] = duration
+      end
+
+      @global_config[:metric_format] ||= "%m.%c.%h"
+      if @global_config[:metric_format] !~ /%m/
+        raise "missing metric (%m) in :metric_format"
+      elsif @global_config[:metric_format] !~ /%c/
+        raise "missing cluster (%c) in :metric_format"
+      elsif @global_config[:metric_format] !~ /%h/
+        raise "missing host (%h) in :metric_format"
+      end
+
+      graphs_new = []
+      @rawconfig[:graphs].each do |name, config|
+        graphs_new << Graph.new(name, config.merge(@global_config))
+      end
+
+      dashboards_new = []
+      @rawconfig[:dashboards].each do |name, config|
+        dashboards_new << Dashboard.new(name, config.merge(@global_config))
+      end
+
+      hosts_new = Set.new
+      clusters_new = Set.new
+
+      # generate host and cluster information at init time
+      graphs_new.each do |g|
+        hosts, clusters = g.hosts_clusters
+        hosts.each { |h| hosts_new << h }
+        clusters.each { |h| clusters_new << h }
+      end
+
+      @dashboards, @graphs = dashboards_new, graphs_new
+      @hosts, @clusters = hosts_new, clusters_new
+    end
+  end # Dash::Config
+end # Dash

data/lib/config.ru

@@ -0,0 +1,2 @@
+require 'dash'
+run Dash::App

data/lib/dash.rb

@@ -0,0 +1,153 @@
+#!/usr/bin/env ruby
+
+require "rubygems"
+require "namespace"
+
+require "config"
+require "erb"
+require "helpers"
+require "models"
+require "rack"
+require "sinatra/base"
+require "json"
+require "open-uri"
+require "yaml"
+require "chronic"
+require "chronic_duration"
+require "optparse"
+
+# fixme style.css isn't actually cached, you need to set something up with
+# rack to cache static files
+
+$:.unshift(File.dirname(__FILE__))
+
+module Dash
+  class App < Sinatra::Base
+    include Dash::Models
+    helpers Dash::Helpers
+    config = Dash::Config.new
+    set :config, config
+    set :port, config.global_config[:port]
+    set :run, true
+    use Rack::Session::Cookie, :expire_after => 126227700 # 4 years
+    set :root, File.dirname(__FILE__)
+    set :static, true
+
+    def initialize(settings={})
+      super
+    end
+
+    before do
+      session[:not] #fixme kludge is back
+      @request_time = Time.now
+      @dashboards = Dashboard.all
+      @no_graphs = false
+      # time stuff
+      start = param_lookup("start")
+      duration = param_lookup("duration")
+      @stime = Chronic.parse(start)
+      if @stime
+        @stime -= @stime.sec unless @params["noq"]
+      end
+      if duration
+        @duration = ChronicDuration.parse(duration)
+      else
+        @duration = @request_time.to_i - @stime.to_i
+      end
+
+      unless @params["noq"]
+        @duration -= (@duration % settings.config.global_config[:quantum]||1)
+      end
+
+      if @stime
+        @etime = Time.at(@stime + @duration)
+        @etime = @request_time if @etime > @request_time
+      else
+        @etime = @request_time
+      end
+
+      params[:stime] = @stime.to_i.to_s
+      params[:etime] = @etime.to_i.to_s
+      # fixme reload hosts after some expiry
+    end
+
+    get %r[^/(dash/?)?$] do
+      @no_graphs = true
+      redirect '/dash/global'
+    end
+
+    get '/dash/:cluster/:dashboard/:zoom/?' do
+      @cluster = params[:cluster]
+      @dash = Dashboard.find(params[:dashboard])
+      raise "Unknown dashboard: #{params[:dashboard].inspect}" unless @dash
+
+      @zoom = nil
+      @dash.graphs.each do |graph|
+        @zoom = graph if graph.name == params[:zoom]
+      end
+      raise "Unknown zoom parameter: #{params[:zoom]}" unless @zoom
+
+      @title = "dashboard :: #{@cluster} :: #{@dash['title']} :: #{params[:zoom]}"
+
+      if @cluster == "global"
+        erb :'dash-global-zoom'
+      else
+        erb :'dash-cluster-zoom'
+      end
+    end
+
+    get '/dash/:cluster/:dashboard/?' do
+      @cluster = params[:cluster]
+      @dash = Dashboard.find(params[:dashboard])
+      raise "Unknown dashboard: #{params[:dashboard].inspect}" unless @dash
+
+      @title = "dashboard :: #{@cluster} :: #{@dash['title']}"
+
+      if @cluster == "global"
+        erb :'dash-global'
+      else
+        erb :'dash-cluster'
+      end
+    end
+
+    get '/dash/:cluster/?' do
+      @no_graphs = true
+      @cluster = params[:cluster]
+      if @cluster == "global"
+        @title = "Overview"
+        erb :global
+      else
+        @title = "cluster :: #{params[:cluster]}"
+        erb :cluster
+      end
+    end
+
+    get '/host/:cluster/:host/?' do
+      @host = Host.new(params[:host], { "cluster" => params[:cluster] })
+      @cluster = params[:cluster]
+      raise "Unknown host: #{params[:host]} in #{params[:cluster]}" unless @host
+
+      @title = "#{@host.cluster} :: host :: #{@host.name}"
+
+      erb :host
+    end
+
+    # fixme make sure not to save shitty values for :start, :duration
+    # remove stime, etime
+    # there is definitely something wrong here
+    # disallow saving from, until
+    get '/saveprefs' do
+      puts 'saving prefs'
+      params.each do |k,v|
+        session[k] = v unless v.empty?
+      end
+      redirect URI.parse(request.referrer).path
+    end
+
+    get '/clear' do
+      puts "clearing prefs"
+      session.clear
+      redirect URI.parse(request.referrer).path
+    end
+  end # Dash::App
+end # Dash

data/lib/helpers.rb

@@ -0,0 +1,197 @@
+module Dash::Helpers
+  include Dash::Models
+
+  @@prefs = [["Start", "start"],
+             ["Duration", "duration"],
+             ["Width", "width"],
+             ["Height", "height"]]
+
+  # convert keys to symbols before lookup
+  def param_lookup(name)
+    sym_hash = {}
+    session.each { |k,v| sym_hash[k.to_sym] = v unless v.empty? }
+    params.each { |k,v| sym_hash[k.to_sym] = v unless v.empty? }
+    settings.config.global_config[:url_opts].merge(sym_hash)[name.to_sym]
+  end
+
+  def cluster_graph(g, cluster, title="wtf")
+    image_url = \
+    @dash.render_cluster_graph(g, cluster,
+                               :title => title,
+                               :dynamic_url_opts => merge_opts)
+    zoom_url = cluster_graph_link(@dash, g, cluster)
+    return image_url, zoom_url
+  end
+
+  def cluster_graph_link(dash, g, cluster)
+    link = dash.graph_opts[g]["click"] ||
+      "/dash/#{cluster}/#{dash.name}/#{g.name}"
+    return append_query_string(link)
+  end
+
+  def cluster_zoom_graph(g, cluster, host, title)
+    image_url = g.render_url([host.name], [cluster], :title => title,
+                             :dynamic_url_opts => merge_opts)
+    zoom_url = cluster_zoom_link(cluster, host)
+    return image_url, zoom_url
+  end
+
+  def cluster_zoom_link(cluster, host)
+    return append_query_string("/host/#{cluster}/#{host}")
+  end
+
+  def suggest_cluster_links(clusters, g)
+    links = []
+    clusters.each do |c|
+      href = append_query_string("/dash/#{c}/#{params[:dashboard]}/#{g.name}")
+      links << "<a href=\"#{href}\">#{c}</a>"
+    end
+    return "zoom (" + links.join(", ") + ")"
+  end
+
+  def suggest_dashboards_links(host, graph)
+    suggested = suggest_dashboards(host, graph)
+    return "" if suggested.length == 0
+
+    links = []
+    suggested.each do |d|
+      links << "<a href=\"/dash/#{host.cluster}/#{append_query_string(d)}\">" +
+        "#{d}</a>"
+    end
+    return "(" + links.join(", ") + ")"
+  end
+
+  # it's mildly annoying that when this set is empty there're no uplinks
+  # consider adding a link up to the cluster (which is best we can do)
+  def suggest_dashboards(host, graph)
+    ret = Set.new
+
+    host.graphs.each do |g|
+      Dashboard.find_by_graph(g).each do |d|
+        valid, _ = d.get_valid_hosts(g, host['cluster'])
+        ret << d.name if valid.member?(host)
+      end
+    end
+
+    return ret
+  end
+
+  # generate the input box fields, filled in to current parameters if specified
+  def input_boxes
+    @prefs = @@prefs
+    erb :'partials/input_boxes', :layout => false
+  end
+
+  def cookies_form
+    @prefs = @@prefs
+    erb :'partials/cookies_form', :layout => false
+  end
+
+  def refresh_button
+    @prefs = @@prefs
+    erb :'partials/refresh_button', :layout => false
+  end
+
+  def dash_link(dash, cluster)
+    return append_query_string("/dash/#{cluster}/#{dash.name}")
+  end
+
+  def cluster_link(cluster)
+    return append_query_string("/dash/#{cluster}")
+  end
+
+  def css_url
+    style = File.join(settings.root, "public/style.css")
+    mtime = File.mtime(style).to_i.to_s
+    return \
+    %Q[<link href="/style.css?#{mtime}" rel="stylesheet" type="text/css">]
+  end
+
+  def refresh
+    if settings.config.global_config[:refresh_rate] != false && nowish
+      rate = settings.config.global_config[:refresh_rate] || 60
+      return %Q[<meta http-equiv="refresh" content="#{rate}">]
+    end
+  end
+
+  def hosts_selector(hosts)
+    @hosts = hosts
+    erb :'partials/hosts_selector', :layout => false
+  end
+
+  def append_query_string(str)
+    v = str.dup
+    query = request.query_string.chomp("&permalink=1")
+    (v << "?#{query}") unless request.query_string.empty?
+    return v
+  end
+
+  def merge_opts
+    static_opts = ["cluster", "dashboard", "zoom", "host", "session_id"]
+    opts = params.dup
+    session.merge(opts).delete_if { |k,v| static_opts.member?(k) || v.empty? }
+  end
+
+  def cluster_switcher
+    erb :'partials/cluster_switcher', :layout => false
+  end
+
+  def dash_switcher
+    erb :'partials/dash_switcher', :layout => false
+  end
+
+  def graph_switcher
+    erb :'partials/graph_switcher', :layout => false
+  end
+
+  def cluster_selector
+    @clusters = settings.config.clusters.sort + ["global"]
+    erb :'partials/cluster_selector', :layout => false
+  end
+
+  def host_uplink
+    link = "/dash/#{append_query_string(@host.cluster)}"
+    "zoom out: <a href=\"#{link}\">#{@host.cluster}</a>"
+  end
+
+  def graph_uplink
+    link = append_query_string(request.path.split("/")[0..-2].join("/"))
+    "zoom out: <a href=\"#{link}\">#{@dash}</a>"
+  end
+
+  def dash_uplink
+    link = append_query_string(request.path.split("/")[0..-2].join("/"))
+    "zoom out: <a href=\"#{link}\">#{@params[:cluster]}</a>"
+  end
+
+  def nowish
+    if settings.config.global_config[:now_threshold] == false
+      return false
+    end
+    threshold = settings.config.global_config[:now_threshold] || 300
+    return @request_time.to_i - @etime.to_i < threshold
+  end
+
+  def range_string
+    format = settings.config.global_config[:date_format] || "%X %x"
+    if @stime && @etime
+      if nowish
+        "timeslice: from #{@stime.strftime(format)}"
+      else
+        "timeslice: #{@stime.strftime(format)} - #{@etime.strftime(format)}"
+      end
+    else
+      "invalid time range"
+    end
+
+  end
+
+  def permalink
+    return "" unless @stime && @duration
+    format = "%x %X" # chronic understands this
+    url = request.path + "?"
+    url << "&start=#{@stime.strftime(format)}"
+    url << "&duration=#{ChronicDuration.output(@duration)}"
+    "<a href=\"#{url}\">permalink</a>"
+  end
+end

data/lib/models/base.rb

@@ -0,0 +1,73 @@
+module Dash::Models
+  class Base
+    @@objects = Hash.new { |h, k| h[k] = Hash.new }
+    attr_reader :name
+
+    def initialize(name, params={})
+      @name = name
+      @match_name = name
+      @params = params
+      @@objects[self.class.to_s][name] = self
+    end
+
+    def self.find(name)
+      return @@objects[self.name][name] rescue []
+    end
+
+    def self.each(&block)
+      h = @@objects[self.name] rescue {}
+      h.each { |k, v| yield(k, v) }
+    end
+
+    def self.all
+      return @@objects[self.name].values
+    end
+
+    def [](key)
+      return @params[key] rescue []
+    end
+
+    def match(glob)
+      return true if glob == '*'
+      # convert glob to a regular expression
+      glob_parts = glob.split('*').collect { |s| Regexp.escape(s) }
+      if glob[0].chr == '*'
+        glob_re = /^.*#{glob_parts.join('.*')}$/
+      elsif glob[-1].chr == '*'
+        glob_re = /^#{glob_parts.join('.*')}.*$/
+      else
+        glob_re = /^#{glob_parts.join('.*')}$/
+      end
+      return @match_name.match(glob_re)
+    end
+
+    def multi_match(globs)
+      ret = false
+
+      globs.each do |glob|
+        ret = match(glob)
+        break if ret
+      end
+
+      return ret
+    end
+
+    def to_s
+      return @name
+    end
+
+    def <=>(other)
+      return to_s <=> other.to_s
+    end
+
+    def update_params(hash)
+      @params.merge!(hash)
+    end
+
+    # compose a metric using a :metric_format
+    # format string with %c for metric, %c for cluster, and %h for host
+    def compose_metric (m, c, h)
+      @params[:metric_format].dup.gsub("%m", m).gsub("%c", c).gsub("%h", h)
+    end
+  end # Dash::Models::Base
+end # Dash::Models