ustate-client 0.0.1

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+
+Copyright (c) 2011 Kyle Kingsbury
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,111 @@
+Overview
+======
+
+UState ("United States", "microstate", etc.) is a state aggregation daemon. It
+accepts a stream of state transitions and maintains an index of service states,
+which can be queried or forwarded to various handlers. A state is simply:
+
+    state {
+      host: A hostname, e.g. "api1", "foo.com",
+      service: e.g. "API port 8000 reqs/sec",
+      state: Any string less than 255 bytes, e.g. "ok", "warning", "critical",
+      time: The time that the service entered this state, in unix time,
+      description: Freeform text,
+      metric_f: A floating-point number associated with this state, e.g. the number of reqs/sec
+    }
+
+At http://showyou.com, we use UState to monitor the health and performance of hundreds of services across our infrastructure, including CPU, queries/second, latency bounds, disk usage, queues, and others.
+
+UState also includes a simple dashboard Sinatra app.
+
+Installing
+==========
+
+    git clone git://github.com/aphyr/ustate.git
+
+or
+
+    gem install ustate-client
+
+For the client:
+
+    gem install beefcake trollop
+
+For the server:
+
+    gem install treetop eventmachine sequel sqlite3 trollop beefcake
+
+For the dashboard:
+
+     gem install sinatra thin erubis sass
+
+Getting started
+===============
+
+To try it out, install all the gems above, and clone the repository. Start the server with
+
+    bin/server [--host host] [--port port]
+
+UState listens on TCP socket host:port, and accepts connections from clients. Start a basic testing client with
+
+   bin/test
+
+The tester spews randomly generated statistics at a server on the default local host and port. To see it in action, run the dashboard:
+
+  cd lib/ustate/dash
+  ../../../bin/dash
+
+
+The Dashboard
+=============
+
+The dashboard runs a file in the local directory: config.rb. That file can
+override any configuration options on the Dash class (hence all Sinatra
+configuration) as well as the Ustate client, etc.
+
+    set :port, 6000 # HTTP server on port 6000
+    config[:client][:host] = 'my.ustate.server'
+
+It also loads views from the local directory. Sinatra makes it awkward to
+compose multiple view directories, so you'll probably want to create your own
+view/ and config.rb. I've provided an example stylesheet, layout, and dashboard
+in lib/ustate/dash/views--as well as an extensive set of functions for laying
+out states corresponding to any query: see lib/ustate/dash/helper/renderer.rb.
+The way I figure, you're almost certainly going to want to write your own, so
+I'm going to give you the tools you need, and get out of your way.
+
+Protocol
+========
+
+A connection to UState is a stream of messages. Each message is a 4 byte
+network-endian integer *length*, followed by a Procol Buffers Message of
+*length* bytes. See lib/ustate/message.rb for the protobuf particulars.
+
+The server will accept a repeated list of States, and respond with a
+confirmation message with either an acknowledgement or an error. Check the
+success boolean in the Message.
+
+You can also query states using a very basic expression language. The grammar is specified as a Parsable Expression Grammar in query_string.treetop. Examples include:
+
+    state = "ok"
+    (service =~ "disk%") or (state == "critical" and host =~ "%.trioptimum.com")
+
+Search queries will return a message with repeated States matching that expression. An empty expression matches all states.
+
+Performance
+===========
+
+It's Ruby. It ain't gonna be fast. However, on my 4-year-old core 2 duo, I see >600 inserts/sec or queries/sec. The client is fully threadsafe, and performs well concurrently. I will continue to tune UState for latency and throughput, and welcome patches.
+
+For large installations, I plan to implement a selective forwarder. Local ustate servers can accept high volumes of states from a small set of nodes, and forward updates at a larger granularity to supervisors, and so forth, in a tree. The query language should be able to support proxying requests to the most recent source of a state, so very large sets of services can be maintained at high granularity.
+
+Goals
+=====
+
+Immediately, I'll be porting our internal email alerter to UState. Users register for interest in certain types of states or transitions, and receive emails when those events occur.
+
+In the medium term, I'll be connecting UState to Graphite (or perhaps another
+graphing tool) for metrics archival and soft-realtime graphs. I have an
+internal gnuplot system which is clunky and deserves retirement.
+
+When the protocol and architecture are finalized, I plan to reimplement the server in a faster language.

data/lib/ustate/client/query.rb

@@ -0,0 +1,17 @@
+module UState
+  class Client::Query
+    # Little query builder
+    def initialize
+      @predicate = nil
+    end
+
+    def method_missing(field)
+      field = field.to_sym
+      beefcake_field = UState::Query.fields.find { |f|
+        f.name == field
+      }
+      raise ArgumentError, "no such field #{field.inspect}" unless beefcake_field
+      beefcake_field.type::Proxy
+    end
+  end
+end

data/lib/ustate/client.rb

@@ -0,0 +1,108 @@
+class UState::Client
+  class Error < RuntimeError; end
+  class InvalidResponse < Error; end
+  class ServerError < Error; end
+  
+  require 'thread'
+  require 'socket'
+  require 'time'
+
+  HOST = '127.0.0.1'
+  PORT = 55956
+
+  TYPE_STATE = 1
+
+  attr_accessor :host, :port, :socket
+
+  def initialize(opts = {})
+    @host = opts[:host] || HOST
+    @port = opts[:port] || PORT
+    @locket = Mutex.new
+  end
+
+  # Send a state
+  def <<(state_opts)
+    # Create state
+    state = UState::State.new(state_opts)
+    state.time ||= Time.now.utc.to_i
+    state.host ||= Socket.gethostname
+
+    message = UState::Message.new :states => [state]
+
+    # Transmit
+    with_connection do |s|
+      s << message.encode_with_length
+      read_message s
+    end
+  end
+
+  def connect
+    @socket = TCPSocket.new(@host, @port)
+  end
+
+  def close
+    @locket.synchronize do
+      @socket.close
+    end
+  end
+
+  def connected?
+    not @socket.closed?
+  end
+
+  # Ask for states
+  def query(string = nil)
+    message = UState::Message.new query: UState::Query.new(string: string)
+    with_connection do |s|
+      s << message.encode_with_length
+      read_message s
+    end
+  end
+
+  # Read a message from a stream
+  def read_message(s)
+    if buffer = s.read(4) and buffer.size == 4
+      length = buffer.unpack('N').first
+      begin
+        str = s.read length
+        message = UState::Message.decode str
+      rescue => e
+        puts "Message was #{str.inspect}"
+        raise
+      end
+      
+      unless message.ok
+        puts "Failed"
+        raise ServerError, message.error
+      end
+      
+      message
+    else
+      raise InvalidResponse, "unexpected EOF"
+    end
+  end
+
+  # Yields a connection in the block.
+  def with_connection
+    tries = 0
+    
+    @locket.synchronize do
+      begin
+        tries += 1
+          yield (@socket or connect)
+      rescue IOError => e
+        raise if tries > 3
+        connect and retry
+      rescue Errno::EPIPE => e
+        raise if tries > 3
+        connect and retry
+      rescue Errno::ECONNREFUSED => e
+        raise if tries > 3
+        connect and retry
+      rescue Errno::ECONNRESET => e
+        raise if tries > 3
+        connect and retry
+      end
+    end
+  end
+end

data/lib/ustate/dash/controller/css.rb

@@ -0,0 +1,5 @@
+class UState::Dash
+  get '/css' do
+    scss :css, :layout => false
+  end
+end

data/lib/ustate/dash/controller/index.rb

@@ -0,0 +1,5 @@
+class UState::Dash
+  get '/' do
+    erubis :index
+  end
+end

data/lib/ustate/dash/helper/renderer.rb

@@ -0,0 +1,209 @@
+module UState
+  class Dash
+    helpers do
+      include Rack::Utils
+
+      alias_method :h, :escape_html
+
+      # Returns a scalar factor from 0.2 to 1, where 0.2 is "on the order of
+      # age_scale ago", and 1 is "very recent"
+      def age_fraction(time)
+        return 1 if time.nil?
+
+        x = 1 - ((Time.now.to_f - time) / Dash.config[:age_scale])
+        if x < 0.2
+          0.2
+        elsif x > 1
+          1
+        else
+          x
+        end
+      end
+
+      # Finds the longest common prefix of a list of strings.
+      # i.e. 'abc, 'ab', 'abdf' => 'ab'
+      def longest_common_prefix(strings, prefix = '')
+        return strings.first if strings.size <= 1
+
+        first = strings[0][0,1] or return prefix
+        tails = strings[1..-1].inject([strings[0][1..-1]]) do |tails, string|
+          if string[0,1] != first
+            return prefix
+          else
+            tails << string[1..-1]
+          end
+        end
+
+        longest_common_prefix(tails, prefix + first)
+      end
+
+      # An overview of states
+      def state_list(states)
+        ul(states.map { |s| state_short s })
+      end
+
+      def state_grid(states = Dash.client.query)
+        h2('States by Host') +
+        table(
+          *State.partition(states, :host).map do |host, states|
+            tr(
+              th(host, class: 'host'),
+              *State.sort(states, :service).map do |state|
+                state_short state
+              end
+            )
+          end
+        )
+      end
+
+      # Renders a state as the given HTML tag with a % width corresponding to
+      # metric / max.
+      def state_bar(s, opts = {})
+        opts = {tag: 'div', max: 1}.merge opts
+
+        return '' unless s
+        x = s.metric
+
+        # Text
+        text = case x
+        when Float
+          '%.2f' % x
+        when Integer
+          x.to_s
+        else
+          '?'
+        end
+
+        # Size
+        size = begin
+          (x || 0) * 100 / opts[:max]
+        rescue ZeroDivisionError
+          0
+        end
+
+        tag opts[:tag], h(text), 
+          :class => "state #{s.state}", 
+          style: "opacity: #{age_fraction s.time}; width: #{size}%", 
+          title: s.description
+      end
+
+      # Renders a set of states in a chart. Each row is a given host, each
+      # service is a column. Each state is shown as a bar with an inferred
+      # maximum for the entire service, so you can readily compare multiple
+      # hosts.
+      #
+      # Takes a a set of states and options:
+      #   title: the title of the chart. Inferred to be the longest common
+      #          prefix of all services.
+      #   maxima: maps each service to the maximum value used to display its
+      #           bar.
+      #   service_names: maps each service to a friendly name. Default service
+      #                  names have common prefixes removed.
+      #   hosts: an array of hosts for rows. Default is every host present in
+      #          states, sorted.
+      def state_chart(states, opts = {})
+        o = {
+          :maxima => {},
+          :service_names => {}
+        }.merge opts
+
+        # Get all services
+        services = states.map { |s| s.service }.compact.uniq.sort
+
+        # Figure out what name to use for each service.
+        prefix = longest_common_prefix services
+        service_names = services.inject({}) do |names, service|
+          names[service] = service[prefix.length..-1]
+          names
+        end.merge o[:service_names]
+
+        # Compute maximum for each service
+        maxima = states.inject(Hash.new(0)) do |m, s|
+          m[s.service] = [s.metric, m[s.service]].max
+          m
+        end.merge o[:maxima]
+
+        # Compute union of all hosts for these states, if no
+        # list of hosts explicitly given.
+        hosts = o[:hosts] || states.map do |state|
+          state.host
+        end.compact
+        hosts = hosts.uniq.sort
+
+        # Construct index
+        by = states.inject({}) do |index, s|
+          index[[s.host, s.service]] = s
+          index
+        end
+
+        # Title
+        title = o[:title] || prefix.capitalize rescue 'Unknown'
+
+        h2(title) +
+        table(
+          tr(
+            th,
+            *services.map do |service|
+              th service_names[service]
+            end
+          ),
+          *hosts.map do |host|
+            tr(
+              th(host),
+              *services.map do |service|
+                s = by[[host, service]]
+                td(
+                  s ? state_bar(s, max: maxima[service]) : nil
+                )
+              end
+            )
+          end,
+          :class => 'chart'
+        )
+      end
+
+      # Renders a state as a short tag.
+      def state_short(s, opts={tag: 'li'})
+        if s
+          "<#{opts[:tag]} class=\"state #{s.state}\" style=\"opacity: #{age_fraction s.time}\" title=\"#{h s.description}\">#{h s.host} #{h s.service}</#{opts[:tag]}>"
+        else
+          "<#{opts[:tag]} class=\"service\"></#{opts[:tag]}>"
+        end
+      end
+      
+      # Renders a time to an HTML tag.
+      def time(unix)
+        t = Time.at(unix)
+        "<time datetime=\"#{t.iso8601}\">#{t.strftime(Dash.config[:strftime])}</time>"
+      end
+
+      # Renders an HTML tag
+      def tag(tag, *a)
+        if Hash === a.last
+          opts = a.pop
+        else
+          opts = {}
+        end
+
+        attrs = opts.map do |k,v|
+          "#{k}=\"#{h v}\""
+        end.join ' '
+
+        content = if block_given?
+          a << yield
+        else
+          a
+        end.flatten.join("\n")
+
+        s = "<#{tag} #{attrs}>#{content}</#{tag}>"
+      end
+      
+      # Specific tag aliases
+      %w(div span h1 h2 h3 h4 h5 h6 ul ol li table th tr td u i b).each do |tag|
+        class_eval "def #{tag}(*a, &block)
+          tag #{tag.inspect}, *a, &block
+        end"
+      end
+    end 
+  end
+end