ustate-client 0.0.2 → 0.0.3

data/README.markdown

@@ -11,10 +11,27 @@ which can be queried or forwarded to various handlers. A state is simply:
       state: Any string less than 255 bytes, e.g. "ok", "warning", "critical",
       time: The time that the service entered this state, in unix time,
       description: Freeform text,
-      metric_f: A floating-point number associated with this state, e.g. the number of reqs/sec
+      metric_f: A floating-point number associated with this state, e.g. the number of reqs/sec,
+      once: A boolean, described below.
     }
 
-At http://showyou.com, we use UState to monitor the health and performance of hundreds of services across our infrastructure, including CPU, queries/second, latency bounds, disk usage, queues, and others.
+Normally, every state received by the server fires Index#on_state. When
+state.state changes, Index#on_state_change is called. You can, for example,
+register to send a single email whenever a state changes to :warning.
+
+:once states are transient. They fire Index#on_state and #on_state_once, but do
+*not* update the index. They can be used for events which are instantaneous;
+instead of sending {state: error} and {state: ok}, send {state: error,
+once:true}. 
+
+For example, recoverable errors may not hang your application, but
+should be processed by the email notifier. Sending a :once state with
+the error description means you can receive an email for each error,
+instead of two for entering and exiting the error state.
+
+At http://showyou.com, we use UState to monitor the health and performance of
+hundreds of services across our infrastructure, including CPU, queries/second,
+latency bounds, disk usage, queues, and others.
 
 UState also includes a simple dashboard Sinatra app.
 
@@ -39,22 +56,89 @@ For the dashboard:
 
      gem install sinatra thin erubis sass
 
-Getting started
-===============
+Demo
+====
 
 To try it out, install all the gems above, and clone the repository. Start the server with
 
-    bin/server [--host host] [--port port]
+    bin/server
 
 UState listens on TCP socket host:port, and accepts connections from clients. Start a basic testing client with
 
-   bin/test
+    bin/test
 
 The tester spews randomly generated statistics at a server on the default local host and port. To see it in action, run the dashboard:
 
-  cd lib/ustate/dash
-  ../../../bin/dash
+    cd lib/ustate/dash
+    ../../../bin/dash
+
+Server
+======
+
+The server loads a file in the working directory named config.rb. Override with
+--config-file.  Its contents are instance-evaled in the context of the current
+server. You can use this to extend ustate with additional behavior.
+
+Email
+-----
+
+config.rb:
+    # Email comes from this address (required):
+    emailer.from = 'ustate@your.net'
+
+    # Use this SMTP relay (default 127.0.0.1)
+    emailer.host = '123.4.56.7'
+    
+    # Receive mail when a state transition matches any of ...
+    emailer.tell 'you@gmail.com', 'state = "error" or state = "critical"'
+    emailer.tell 'you@gmail.com', 'service =~ "mysql%"'
+
+Custom hooks
+------------
+
+config.rb:
+    # Log all states received to console.
+    index.on_state do |s|
+      p s
+    end
+    
+    # Forward state transitions to another server.
+    require 'ustate/client'
+    client = UState::Client.new :host => '123.45.67.8'
+    index.on_state_change do |old, new|
+      client << new
+    end
+    index.on_state_once do |state|
+      client << state
+    end
+
+Client
+======
 
+You can use the git repo, or the gem.
+
+    gem install ustate-client
+
+Then:
+
+    require 'ustate'
+    require 'ustate/client'
+
+    # Create a client
+    c = UState::Client.new(
+      host: "my.host",    # Default localhost
+      port: 1234          # Default 55956
+    )
+    
+    # Insert a state
+    c << {
+      state: "ok",
+      service: "My service"
+    }
+
+    # Query for states
+    c.query.states # => [UState::State(state: 'ok', service: 'My service')]
+    c.query('state != "ok"').states # => []
 
 The Dashboard
 =============
@@ -90,20 +174,18 @@ You can also query states using a very basic expression language. The grammar is
     state = "ok"
     (service =~ "disk%") or (state == "critical" and host =~ "%.trioptimum.com")
 
-Search queries will return a message with repeated States matching that expression. An empty expression matches all states.
+Search queries will return a message with repeated States matching that expression. An null expression will return no states.
 
 Performance
 ===========
 
-It's Ruby. It ain't gonna be fast. However, on my 4-year-old core 2 duo, I see >600 inserts/sec or queries/sec. The client is fully threadsafe, and performs well concurrently. I will continue to tune UState for latency and throughput, and welcome patches.
+On a macbook pro 8,3, I see >1300 queries/sec or >1200 inserts/sec. The client is fully threadsafe, and performs well concurrently. I will continue to tune UState for latency and throughput, and welcome patches.
 
 For large installations, I plan to implement a selective forwarder. Local ustate servers can accept high volumes of states from a small set of nodes, and forward updates at a larger granularity to supervisors, and so forth, in a tree. The query language should be able to support proxying requests to the most recent source of a state, so very large sets of services can be maintained at high granularity.
 
 Goals
 =====
 
-Immediately, I'll be porting our internal email alerter to UState. Users register for interest in certain types of states or transitions, and receive emails when those events occur.
-
 In the medium term, I'll be connecting UState to Graphite (or perhaps another
 graphing tool) for metrics archival and soft-realtime graphs. I have an
 internal gnuplot system which is clunky and deserves retirement.

data/lib/ustate/emailer.rb

@@ -0,0 +1,92 @@
+module UState
+  class Emailer
+    require 'net/smtp'
+
+    attr_accessor :from
+    attr_accessor :host
+    attr_accessor :name
+
+    # Registers self with index.
+    # Options:
+    #   :host: The SMTP host to connect to. Default 'localhost'
+    #   :name: The From name used. Default "ustate".
+    #   :from: The From address used: e.g. "ustate@your_domain.com"
+    def initialize(index, opts = {})
+      opts = {
+        :name => 'ustate',
+        :host => 'localhost'
+      }.merge opts
+
+      @from = opts[:from]
+      @name = opts[:name]
+      @host = opts[:host]
+
+      @tell = {}
+
+      index.on_state_change &method(:receive)
+      index.on_state_once &method(:receive)
+    end
+
+    # Send an email to address about state.
+    def email(address, s)
+      raise ArgumentError, "no from address" unless @from
+      
+      # Subject
+      subject = "#{s.host} #{s.service} #{s.state}"
+      if s.once
+        subject << " transient "
+      else
+        subject << " is "
+      end
+      subject << s.state
+
+      # Body
+      body = "#{subject}: #{s.description}"
+
+      # SMTP message
+      message = <<EOF
+From: #{@name} <#{@from}>
+To: <#{address}>
+Subject: #{subject.gsub("\n", ' ')}
+
+#{body}
+EOF
+
+      Net::SMTP.start(@host) do |smtp|
+        smtp.send_message message, @from, address
+      end
+    end
+
+    # Dispatch emails to each address which is interested in this state
+    def receive(*states)
+      state = states.last
+      Thread.new do
+        @tell.each do |address, q|
+          if q === state
+            email address, state
+          end
+        end
+      end
+    end
+
+    # Notify email when a state matching query_string is
+    # received. Multiple calls are ORed together:
+    #
+    # emailer.tell 'aphyr@aphyr.com', 'state = "error"'
+    # emailer.tell 'aphyr@aphyr.com', 'host =~ "frontend%"'
+    def tell(email, query_string)
+      parser = QueryStringParser.new
+      q = parser.parse(query_string)
+      unless q
+        raise ArgumentError, "error parsing #{query_string.inspect} at line #{parser.failure_line}:#{parser.failure_column}: #{parser.failure_reason}"
+      end
+      q = q.query
+
+      @tell[email] = if existing = @tell[email]
+        Query::Or.new existing, q
+      else
+        q
+      end
+    end
+  end
+end

data/lib/ustate/query/and.rb

@@ -0,0 +1,12 @@
+class UState::Query
+  class And
+    def initialize(a,b)
+      @a = a
+      @b = b
+    end
+
+    def ===(state)
+      @a === state and @b === state
+    end
+  end
+end

data/lib/ustate/query/approximately.rb

@@ -0,0 +1,24 @@
+class UState::Query
+  class Approximately
+    def initialize(field, value)
+      @field = field
+      @value = case value
+        when String
+          r = value.chars.inject('') do |r, c|
+            if c == '%'
+              r << '.*'
+            else
+              r << Regexp.escape(c)
+            end
+          end
+          /^#{r}$/
+        else
+          value
+        end
+    end
+
+    def ===(state)
+      @value === state.send(@field) 
+    end
+  end
+end

data/lib/ustate/query/ast.rb

@@ -0,0 +1,6 @@
+require 'ustate/query/and'
+require 'ustate/query/approximately'
+require 'ustate/query/equals'
+require 'ustate/query/not_equals'
+require 'ustate/query/not'
+require 'ustate/query/or'

data/lib/ustate/query/equals.rb

@@ -0,0 +1,12 @@
+class UState::Query
+  class Equals
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+    
+    def ===(state)
+      state.send(@field) == @value
+    end
+  end
+end

data/lib/ustate/query/not.rb

@@ -0,0 +1,11 @@
+class UState::Query
+  class Not
+    def initialize(a)
+      @a = a
+    end
+
+    def ===(state)
+      not @a === state
+    end
+  end
+end

data/lib/ustate/query/not_equals.rb

@@ -0,0 +1,12 @@
+class UState::Query
+  class NotEquals
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+
+    def ===(state)
+      state.send(@field) != @value
+    end
+  end
+end

data/lib/ustate/query/or.rb

@@ -0,0 +1,12 @@
+class UState::Query
+  class Or
+    def initialize(a,b)
+      @a = a
+      @b = b
+    end
+
+    def ===(state)
+      @a === state or @b === state
+    end
+  end
+end

data/lib/ustate/query_string.rb

@@ -34,6 +34,12 @@ module UState
     end
 
     module Or2
+      def query
+        rest.elements.map { |x| x.and }.inject(first.query) do |a, sub|
+          Query::Or.new a, sub.query
+        end
+      end
+
       def sql
         rest.elements.map { |x| x.and }.
           inject(first.sql) do |a, sub|
@@ -135,6 +141,12 @@ module UState
     end
 
     module And2
+      def query
+        rest.elements.map { |x| x.primary }.inject(first.query) do |a, sub|
+          Query::And.new a, sub.query
+        end
+      end
+
       def sql
         rest.elements.map { |x| x.primary }.
           inject(first.sql) do |a, sub|
@@ -219,6 +231,10 @@ module UState
     end
 
     module Primary1
+      def query
+        x.query
+      end
+
       def sql
         x.sql
       end
@@ -348,6 +364,10 @@ module UState
     end
 
     module Approximately1
+      def query
+        Query::Approximately.new field.sql, string.sql
+      end
+
       def sql
         Sequel::SQL::StringExpression.like field.sql, string.sql
       end
@@ -424,6 +444,10 @@ module UState
     end
 
     module NotEquals1
+      def query
+        Query::NotEquals.new field.sql, value.sql
+      end
+      
       def sql
         Sequel::SQL::BooleanExpression.from_value_pairs({field.sql => value.sql}, :AND, true)
       end
@@ -500,6 +524,10 @@ module UState
     end
 
     module Equals1
+      def query
+        Query::Equals.new field.sql, value.sql
+      end
+
       def sql
         Sequel::SQL::BooleanExpression.from_value_pairs field.sql => value.sql
       end
@@ -529,21 +557,21 @@ module UState
         s0 << r2
         if r2
           i4 = index
-          if has_terminal?('=', false, index)
-            r5 = instantiate_node(SyntaxNode,input, index...(index + 1))
-            @index += 1
+          if has_terminal?('==', false, index)
+            r5 = instantiate_node(SyntaxNode,input, index...(index + 2))
+            @index += 2
           else
-            terminal_parse_failure('=')
+            terminal_parse_failure('==')
             r5 = nil
           end
           if r5
             r4 = r5
           else
-            if has_terminal?('==', false, index)
-              r6 = instantiate_node(SyntaxNode,input, index...(index + 2))
-              @index += 2
+            if has_terminal?('=', false, index)
+              r6 = instantiate_node(SyntaxNode,input, index...(index + 1))
+              @index += 1
             else
-              terminal_parse_failure('==')
+              terminal_parse_failure('=')
               r6 = nil
             end
             if r6

data/lib/ustate/query_string.treetop

@@ -5,6 +5,12 @@ module UState
     
     rule or
       first:and rest:(space 'or' space and)* {
+        def query
+          rest.elements.map { |x| x.and }.inject(first.query) do |a, sub|
+            Query::Or.new a, sub.query
+          end
+        end
+
         def sql
           rest.elements.map { |x| x.and }.
             inject(first.sql) do |a, sub|
@@ -16,6 +22,12 @@ module UState
 
     rule and
       first:primary rest:(space 'and' space primary)* {
+        def query
+          rest.elements.map { |x| x.primary }.inject(first.query) do |a, sub|
+            Query::And.new a, sub.query
+          end
+        end
+
         def sql
           rest.elements.map { |x| x.primary }.
             inject(first.sql) do |a, sub|
@@ -27,6 +39,10 @@ module UState
 
     rule primary
       '(' space? x:or space? ')' {
+        def query
+          x.query
+        end
+
         def sql
           x.sql
         end
@@ -41,6 +57,10 @@ module UState
 
     rule approximately
       field space? '=~' space? string {
+        def query
+          Query::Approximately.new field.sql, string.sql
+        end
+
         def sql
           Sequel::SQL::StringExpression.like field.sql, string.sql
         end
@@ -49,6 +69,10 @@ module UState
 
     rule not_equals
       field space? '!=' space? value {
+        def query
+          Query::NotEquals.new field.sql, value.sql
+        end
+        
         def sql
           Sequel::SQL::BooleanExpression.from_value_pairs({field.sql => value.sql}, :AND, true)
         end
@@ -56,7 +80,11 @@ module UState
     end
  
     rule equals
-      field space? ('=' / '==') space? value {
+      field space? ('==' / '=') space? value {
+        def query
+          Query::Equals.new field.sql, value.sql
+        end
+
         def sql
           Sequel::SQL::BooleanExpression.from_value_pairs field.sql => value.sql
         end

data/lib/ustate/server.rb

@@ -12,7 +12,8 @@ module UState
     require 'ustate/server/index'
     require 'ustate/server/backends'
     require 'treetop'
-    require 'ustate/query_string.rb'
+    require 'ustate/query_string'
+    require 'ustate/query/ast'
 
     attr_accessor :backends
     attr_accessor :index
@@ -29,6 +30,11 @@ module UState
       setup_signals
     end
 
+    def emailer(opts = {})
+      require 'ustate/emailer'
+      @emailer ||= UState::Emailer.new(@index, opts)
+    end
+
     def start
       @index.start
      

data/lib/ustate/server/index.rb

@@ -19,6 +19,10 @@ module UState
       @threads = opts[:threads] || THREADS
       @pool = []
 
+      @on_state_change = []
+      @on_state_once = []
+      @on_state = []
+
       setup_db
     end
 
@@ -39,13 +43,42 @@ module UState
       end
     end
 
-    def on_state_change(old, new)
+    def on_state_change(old = nil, new = nil, &block)
+      if block_given?
+        @on_state_change |= [block]
+      else
+        @on_state_change.each do |callback|
+          callback.call old, new
+        end
+      end
+    end
+
+    def on_state_once(state = nil, &block)
+      if block_given?
+        @on_state_once |= [block]
+      else
+        @on_state_once.each do |callback|
+          callback.call state
+        end
+      end
     end
 
-    def on_state(state)
+    def on_state(state = nil, &block)
+      if block_given?
+        @on_state |= [block]
+      else
+        @on_state.each do |callback|
+          callback.call state
+        end
+      end
     end
 
     def process(s)
+      if s.once
+        on_state_once s
+        return on_state s
+      end
+
       if current = @db[:states][host: s.host, service: s.service]
         # Update
         if current[:time] <= s.time

data/lib/ustate/state.rb

@@ -6,6 +6,7 @@ class UState::State
   optional :service, :string, 3
   optional :host, :string, 4
   optional :description, :string, 5
+  optional :once, :bool, 6
   optional :metric_f, :float, 15
 
   def metric

data/lib/ustate/version.rb

@@ -1,3 +1,3 @@
 module UState
-  VERSION = '0.0.2'
+  VERSION = '0.0.3'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ustate-client
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,12 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2011-09-05 00:00:00.000000000Z
+date: 2011-09-06 00:00:00.000000000 -07:00
+default_executable: 
 dependencies:
 - !ruby/object:Gem::Dependency
   name: beefcake
-  requirement: &77275260 !ruby/object:Gem::Requirement
+  requirement: &15107700 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +22,10 @@ dependencies:
         version: 0.3.5
   type: :runtime
   prerelease: false
-  version_requirements: *77275260
+  version_requirements: *15107700
 - !ruby/object:Gem::Dependency
   name: trollop
-  requirement: &77274800 !ruby/object:Gem::Requirement
+  requirement: &15107220 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,7 +33,7 @@ dependencies:
         version: 1.16.2
   type: :runtime
   prerelease: false
-  version_requirements: *77274800
+  version_requirements: *15107220
 description: 
 email: aphyr@aphyr.com
 executables: []
@@ -40,32 +41,41 @@ extensions: []
 extra_rdoc_files: []
 files:
 - lib/ustate.rb
+- lib/ustate/query/not_equals.rb
+- lib/ustate/query/equals.rb
+- lib/ustate/query/or.rb
+- lib/ustate/query/approximately.rb
+- lib/ustate/query/and.rb
+- lib/ustate/query/ast.rb
+- lib/ustate/query/not.rb
 - lib/ustate/query_string.treetop
-- lib/ustate/message.rb
+- lib/ustate/server.rb
+- lib/ustate/emailer.rb
+- lib/ustate/query.rb
 - lib/ustate/version.rb
-- lib/ustate/server/graphite.rb
+- lib/ustate/state.rb
+- lib/ustate/client/query.rb
+- lib/ustate/query_string.rb
+- lib/ustate/server/index.rb
 - lib/ustate/server/backends.rb
-- lib/ustate/server/backends/base.rb
 - lib/ustate/server/backends/tcp.rb
-- lib/ustate/server/index.rb
+- lib/ustate/server/backends/base.rb
+- lib/ustate/server/graphite.rb
 - lib/ustate/server/connection.rb
-- lib/ustate/server.rb
-- lib/ustate/dash.rb
-- lib/ustate/state.rb
-- lib/ustate/query.rb
-- lib/ustate/client/query.rb
 - lib/ustate/client.rb
+- lib/ustate/message.rb
+- lib/ustate/dash/helper/renderer.rb
+- lib/ustate/dash/views/css.scss
 - lib/ustate/dash/views/index.erubis
 - lib/ustate/dash/views/layout.erubis
-- lib/ustate/dash/views/css.scss
 - lib/ustate/dash/state.rb
 - lib/ustate/dash/controller/index.rb
 - lib/ustate/dash/controller/css.rb
 - lib/ustate/dash/config.rb
-- lib/ustate/dash/helper/renderer.rb
-- lib/ustate/query_string.rb
+- lib/ustate/dash.rb
 - LICENSE
 - README.markdown
+has_rdoc: true
 homepage: https://github.com/aphyr/ustate
 licenses: []
 post_install_message: 
@@ -86,7 +96,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: ustate-client
-rubygems_version: 1.8.10
+rubygems_version: 1.6.2
 signing_key: 
 specification_version: 3
 summary: Client for the distributed state server ustate.