ustate-client 0.0.6 → 0.0.7

data/README.markdown

@@ -56,6 +56,8 @@ For the dashboard:
 
      gem install sinatra thin erubis sass
 
+We run UState from Upstart. An example job is in docs/upstart.conf.
+
 Demo
 ====
 
@@ -79,6 +81,29 @@ The server loads a file in the working directory named config.rb. Override with
 --config-file.  Its contents are instance-evaled in the context of the current
 server. You can use this to extend ustate with additional behavior.
 
+Expiring States
+---------------
+
+The reaper periodically kills states matching queries which are too old. It
+will ensure that any state matching a query will be present for *at least* that
+many seconds. For instance:
+
+    # States expire after 10 seconds
+    reaper.default = 10
+
+    # Except for daily stats, which last 2 days
+    reaper.reap 'service =~ "%daily%"', 2 * 24 * 3600
+
+    # We need to know RIGHT AWAY if the fridge fails to check in.
+    reaper.reap 'host = "fridge"', 1
+
+In this configuration, daily updates from host fridge will stay around for 2
+days, everything not on the fridge expires after 10 seconds, and other fridge
+updates are kept for only one second.
+
+Note that the reaper does some query recomposition which can lead to
+inefficient patterns. Writing an optimizer is on my list.
+
 Email
 -----
 
@@ -138,7 +163,7 @@ config.rb:
 
     # And also include the disk use on all nodes
     graphite.graph 'service = "disk"'
-    
+ 
 Custom hooks
 ------------
 
@@ -215,6 +240,9 @@ out states corresponding to any query: see lib/ustate/dash/helper/renderer.rb.
 The way I figure, you're almost certainly going to want to write your own, so
 I'm going to give you the tools you need, and get out of your way.
 
+An example config.rb, additional controllers, views, and public directory are
+all in doc/dash. Should give you ideas for extending the dashboard for your own needs.
+
 Protocol
 ========
 
@@ -229,8 +257,17 @@ success boolean in the Message.
 You can also query states using a very basic expression language. The grammar is specified as a Parsable Expression Grammar in query_string.treetop. Examples include:
 
     state = "ok"
+    
     (service =~ "disk%") or (state == "critical" and host =~ "%.trioptimum.com")
 
+    metric_f > 2.0 and not host = "tau ceti 5"
+    
+    # All states
+    true
+
+    # No states
+    false
+
 Search queries will return a message with repeated States matching that expression. An null expression will return no states.
 
 Performance
@@ -240,11 +277,13 @@ On a macbook pro 8,3, I see >1300 queries/sec or >1200 inserts/sec. The client i
 
 For large installations, I plan to implement a selective forwarder. Local ustate servers can accept high volumes of states from a small set of nodes, and forward updates at a larger granularity to supervisors, and so forth, in a tree. The query language should be able to support proxying requests to the most recent source of a state, so very large sets of services can be maintained at high granularity.
 
-Goals
+Future directions
 =====
 
-In the medium term, I'll be connecting UState to Graphite (or perhaps another
-graphing tool) for metrics archival and soft-realtime graphs. I have an
-internal gnuplot system which is clunky and deserves retirement.
+Several people have mentioned wanting to query historical states; to replay the events in ustate over time. There are some difficulties here; notably that compressing hundreds of millions of states can make it a little tricky to query states over the entire dataset. If we restrict ourselves to specific time ranges, storing sequential states as protocol buffers compressed with snappy could work, especially if only *state* changes are written. Storing only state deltas might work as well.
+
+It'd be interesting to subscribe to states matching a query and receive states pushed to you as soon as they change.
+
+Should be easy to add a UDP acceptor for states as well. Have to figure out eventmachine with multiple backends.
 
 When the protocol and architecture are finalized, I plan to reimplement the server in a faster language.

data/lib/ustate/query/and.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class And
+  class And < Node
     def initialize(a,b)
       @a = a
       @b = b
@@ -8,5 +8,9 @@ class UState::Query
     def ===(state)
       @a === state and @b === state
     end
+
+    def inspect
+      inspect_helper @a, @b
+    end
   end
 end

data/lib/ustate/query/approximately.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class Approximately
+  class Approximately < Node
     def initialize(field, value)
       @field = field
       @value = case value
@@ -20,5 +20,9 @@ class UState::Query
     def ===(state)
       @value === state.send(@field) 
     end
+
+    def inspect
+      inspect_helper @field, @value
+    end
   end
 end

data/lib/ustate/query/ast.rb

@@ -1,6 +1,13 @@
+require 'ustate/query/node'
 require 'ustate/query/and'
 require 'ustate/query/approximately'
 require 'ustate/query/equals'
-require 'ustate/query/not_equals'
+require 'ustate/query/false'
+require 'ustate/query/greater'
+require 'ustate/query/greater_equal'
+require 'ustate/query/less'
+require 'ustate/query/less_equal'
 require 'ustate/query/not'
+require 'ustate/query/not_equals'
 require 'ustate/query/or'
+require 'ustate/query/true'

data/lib/ustate/query/equals.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class Equals
+  class Equals < Node
     def initialize(field, value)
       @field = field
       @value = value
@@ -8,5 +8,9 @@ class UState::Query
     def ===(state)
       state.send(@field) == @value
     end
+
+    def inspect
+      inspect_helper @field, @value
+    end
   end
 end

data/lib/ustate/query/false.rb

@@ -0,0 +1,10 @@
+class UState::Query
+  class False < Node
+    def initialize
+    end
+
+    def ===(state)
+      false
+    end
+  end
+end

data/lib/ustate/query/greater.rb

@@ -0,0 +1,16 @@
+class UState::Query
+  class Greater < Node
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+    
+    def ===(state)
+      x = state.send(@field) and x > @value
+    end
+
+    def inspect
+      inspect_helper @field, @value
+    end
+  end
+end

data/lib/ustate/query/greater_equal.rb

@@ -0,0 +1,16 @@
+class UState::Query
+  class GreaterEqual < Node
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+    
+    def ===(state)
+      x = state.send(@field) and x >= @value
+    end
+
+    def inspect
+      inspect_helper @field, @value
+    end
+  end
+end

data/lib/ustate/query/less.rb

@@ -0,0 +1,16 @@
+class UState::Query
+  class Less < Node
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+    
+    def ===(state)
+      x = state.send(@field) and x < @value
+    end
+
+    def inspect
+      inspect_helper @field, @value
+    end
+  end
+end

data/lib/ustate/query/less_equal.rb

@@ -0,0 +1,16 @@
+class UState::Query
+  class LessEqual < Node
+    def initialize(field, value)
+      @field = field
+      @value = value
+    end
+    
+    def ===(state)
+      x = state.send(@field) and x <= @value
+    end
+
+    def inspect
+      inspect_helper @field, @value
+    end
+  end
+end

data/lib/ustate/query/node.rb

@@ -0,0 +1,15 @@
+class UState::Query
+  class Node
+    def indent(s, d = 1)
+      ("  " * d) + s.gsub("\n", "\n" + ("  " * d))
+    end
+
+    def inspect_helper(*kids)
+      "#{self.class}\n#{indent kids.map { |k| k.inspect }.join("\n")}"
+    end
+
+    def inspect
+      inspect_helper
+    end
+  end
+end

data/lib/ustate/query/not.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class Not
+  class Not < Node
     def initialize(a)
       @a = a
     end
@@ -7,5 +7,9 @@ class UState::Query
     def ===(state)
       not @a === state
     end
+
+    def inspect
+      inspect_helper @a
+    end
   end
 end

data/lib/ustate/query/not_equals.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class NotEquals
+  class NotEquals < Node
     def initialize(field, value)
       @field = field
       @value = value
@@ -8,5 +8,9 @@ class UState::Query
     def ===(state)
       state.send(@field) != @value
     end
+
+    def inspect
+      inspect_helper @field, @value
+    end
   end
 end

data/lib/ustate/query/or.rb

@@ -1,5 +1,5 @@
 class UState::Query
-  class Or
+  class Or < Node
     def initialize(a,b)
       @a = a
       @b = b
@@ -8,5 +8,9 @@ class UState::Query
     def ===(state)
       @a === state or @b === state
     end
+
+    def inspect
+      inspect_helper @a, @b
+    end
   end
 end

data/lib/ustate/query/true.rb

@@ -0,0 +1,10 @@
+class UState::Query
+  class True < Node
+    def initialize
+    end
+
+    def ===(state)
+      true
+    end
+  end
+end

data/lib/ustate/query_string.rb

@@ -125,7 +125,7 @@ module UState
         elements[2]
       end
 
-      def primary
+      def p
         elements[3]
       end
     end
@@ -142,13 +142,13 @@ module UState
 
     module And2
       def query
-        rest.elements.map { |x| x.primary }.inject(first.query) do |a, sub|
+        rest.elements.map { |x| x.p }.inject(first.query) do |a, sub|
           Query::And.new a, sub.query
         end
       end
 
       def sql
-        rest.elements.map { |x| x.primary }.
+        rest.elements.map { |x| x.p }.
           inject(first.sql) do |a, sub|
             a & sub.sql
           end
@@ -167,47 +167,71 @@ module UState
       end
 
       i0, s0 = index, []
-      r1 = _nt_primary
+      i1 = index
+      r2 = _nt_not
+      if r2
+        r1 = r2
+      else
+        r3 = _nt_primary
+        if r3
+          r1 = r3
+        else
+          @index = i1
+          r1 = nil
+        end
+      end
       s0 << r1
       if r1
-        s2, i2 = [], index
+        s4, i4 = [], index
         loop do
-          i3, s3 = index, []
-          r4 = _nt_space
-          s3 << r4
-          if r4
+          i5, s5 = index, []
+          r6 = _nt_space
+          s5 << r6
+          if r6
             if has_terminal?('and', false, index)
-              r5 = instantiate_node(SyntaxNode,input, index...(index + 3))
+              r7 = instantiate_node(SyntaxNode,input, index...(index + 3))
               @index += 3
             else
               terminal_parse_failure('and')
-              r5 = nil
+              r7 = nil
             end
-            s3 << r5
-            if r5
-              r6 = _nt_space
-              s3 << r6
-              if r6
-                r7 = _nt_primary
-                s3 << r7
+            s5 << r7
+            if r7
+              r8 = _nt_space
+              s5 << r8
+              if r8
+                i9 = index
+                r10 = _nt_not
+                if r10
+                  r9 = r10
+                else
+                  r11 = _nt_primary
+                  if r11
+                    r9 = r11
+                  else
+                    @index = i9
+                    r9 = nil
+                  end
+                end
+                s5 << r9
               end
             end
           end
-          if s3.last
-            r3 = instantiate_node(SyntaxNode,input, i3...index, s3)
-            r3.extend(And0)
+          if s5.last
+            r5 = instantiate_node(SyntaxNode,input, i5...index, s5)
+            r5.extend(And0)
           else
-            @index = i3
-            r3 = nil
+            @index = i5
+            r5 = nil
           end
-          if r3
-            s2 << r3
+          if r5
+            s4 << r5
           else
             break
           end
         end
-        r2 = instantiate_node(SyntaxNode,input, i2...index, s2)
-        s0 << r2
+        r4 = instantiate_node(SyntaxNode,input, i4...index, s4)
+        s0 << r4
       end
       if s0.last
         r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
@@ -223,6 +247,80 @@ module UState
       r0
     end
 
+    module Not0
+      def space
+        elements[1]
+      end
+
+      def p
+        elements[2]
+      end
+    end
+
+    module Not1
+      def query
+        Query::Not.new p.query
+      end
+
+      def sql 
+        ~ p.sql
+      end
+    end
+
+    def _nt_not
+      start_index = index
+      if node_cache[:not].has_key?(index)
+        cached = node_cache[:not][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      i0, s0 = index, []
+      if has_terminal?('not', false, index)
+        r1 = instantiate_node(SyntaxNode,input, index...(index + 3))
+        @index += 3
+      else
+        terminal_parse_failure('not')
+        r1 = nil
+      end
+      s0 << r1
+      if r1
+        r2 = _nt_space
+        s0 << r2
+        if r2
+          i3 = index
+          r4 = _nt_not
+          if r4
+            r3 = r4
+          else
+            r5 = _nt_primary
+            if r5
+              r3 = r5
+            else
+              @index = i3
+              r3 = nil
+            end
+          end
+          s0 << r3
+        end
+      end
+      if s0.last
+        r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
+        r0.extend(Not0)
+        r0.extend(Not1)
+      else
+        @index = i0
+        r0 = nil
+      end
+
+      node_cache[:not][start_index] = r0
+
+      r0
+    end
+
     module Primary0
       def x
         elements[2]
@@ -330,20 +428,50 @@ module UState
       end
 
       i0 = index
-      r1 = _nt_equals
+      r1 = _nt_true
       if r1
         r0 = r1
       else
-        r2 = _nt_not_equals
+        r2 = _nt_false
         if r2
           r0 = r2
         else
-          r3 = _nt_approximately
+          r3 = _nt_less_equal
           if r3
             r0 = r3
           else
-            @index = i0
-            r0 = nil
+            r4 = _nt_less
+            if r4
+              r0 = r4
+            else
+              r5 = _nt_greater_equal
+              if r5
+                r0 = r5
+              else
+                r6 = _nt_greater
+                if r6
+                  r0 = r6
+                else
+                  r7 = _nt_equals
+                  if r7
+                    r0 = r7
+                  else
+                    r8 = _nt_not_equals
+                    if r8
+                      r0 = r8
+                    else
+                      r9 = _nt_approximately
+                      if r9
+                        r0 = r9
+                      else
+                        @index = i0
+                        r0 = nil
+                      end
+                    end
+                  end
+                end
+              end
+            end
           end
         end
       end
@@ -433,6 +561,326 @@ module UState
       r0
     end
 
+    module LessEqual0
+      def field
+        elements[0]
+      end
+
+      def value
+        elements[4]
+      end
+    end
+
+    module LessEqual1
+      def query
+        Query::LessEqual.new field.sql, value.sql
+      end
+
+      def sql
+        Sequel::SQL::BooleanExpression.new(:<=, field.sql, value.sql)
+      end
+    end
+
+    def _nt_less_equal
+      start_index = index
+      if node_cache[:less_equal].has_key?(index)
+        cached = node_cache[:less_equal][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      i0, s0 = index, []
+      r1 = _nt_field
+      s0 << r1
+      if r1
+        r3 = _nt_space
+        if r3
+          r2 = r3
+        else
+          r2 = instantiate_node(SyntaxNode,input, index...index)
+        end
+        s0 << r2
+        if r2
+          if has_terminal?('<=', false, index)
+            r4 = instantiate_node(SyntaxNode,input, index...(index + 2))
+            @index += 2
+          else
+            terminal_parse_failure('<=')
+            r4 = nil
+          end
+          s0 << r4
+          if r4
+            r6 = _nt_space
+            if r6
+              r5 = r6
+            else
+              r5 = instantiate_node(SyntaxNode,input, index...index)
+            end
+            s0 << r5
+            if r5
+              r7 = _nt_value
+              s0 << r7
+            end
+          end
+        end
+      end
+      if s0.last
+        r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
+        r0.extend(LessEqual0)
+        r0.extend(LessEqual1)
+      else
+        @index = i0
+        r0 = nil
+      end
+
+      node_cache[:less_equal][start_index] = r0
+
+      r0
+    end
+
+    module Less0
+      def field
+        elements[0]
+      end
+
+      def value
+        elements[4]
+      end
+    end
+
+    module Less1
+      def query
+        Query::Less.new field.sql, value.sql
+      end
+
+      def sql
+        Sequel::SQL::BooleanExpression.new(:<, field.sql, value.sql)
+      end
+    end
+
+    def _nt_less
+      start_index = index
+      if node_cache[:less].has_key?(index)
+        cached = node_cache[:less][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      i0, s0 = index, []
+      r1 = _nt_field
+      s0 << r1
+      if r1
+        r3 = _nt_space
+        if r3
+          r2 = r3
+        else
+          r2 = instantiate_node(SyntaxNode,input, index...index)
+        end
+        s0 << r2
+        if r2
+          if has_terminal?('<', false, index)
+            r4 = instantiate_node(SyntaxNode,input, index...(index + 1))
+            @index += 1
+          else
+            terminal_parse_failure('<')
+            r4 = nil
+          end
+          s0 << r4
+          if r4
+            r6 = _nt_space
+            if r6
+              r5 = r6
+            else
+              r5 = instantiate_node(SyntaxNode,input, index...index)
+            end
+            s0 << r5
+            if r5
+              r7 = _nt_value
+              s0 << r7
+            end
+          end
+        end
+      end
+      if s0.last
+        r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
+        r0.extend(Less0)
+        r0.extend(Less1)
+      else
+        @index = i0
+        r0 = nil
+      end
+
+      node_cache[:less][start_index] = r0
+
+      r0
+    end
+
+    module GreaterEqual0
+      def field
+        elements[0]
+      end
+
+      def value
+        elements[4]
+      end
+    end
+
+    module GreaterEqual1
+      def query
+        Query::GreaterEqual.new field.sql, value.sql
+      end
+
+      def sql
+        Sequel::SQL::BooleanExpression.new(:>=, field.sql, value.sql)
+      end
+    end
+
+    def _nt_greater_equal
+      start_index = index
+      if node_cache[:greater_equal].has_key?(index)
+        cached = node_cache[:greater_equal][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      i0, s0 = index, []
+      r1 = _nt_field
+      s0 << r1
+      if r1
+        r3 = _nt_space
+        if r3
+          r2 = r3
+        else
+          r2 = instantiate_node(SyntaxNode,input, index...index)
+        end
+        s0 << r2
+        if r2
+          if has_terminal?('>=', false, index)
+            r4 = instantiate_node(SyntaxNode,input, index...(index + 2))
+            @index += 2
+          else
+            terminal_parse_failure('>=')
+            r4 = nil
+          end
+          s0 << r4
+          if r4
+            r6 = _nt_space
+            if r6
+              r5 = r6
+            else
+              r5 = instantiate_node(SyntaxNode,input, index...index)
+            end
+            s0 << r5
+            if r5
+              r7 = _nt_value
+              s0 << r7
+            end
+          end
+        end
+      end
+      if s0.last
+        r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
+        r0.extend(GreaterEqual0)
+        r0.extend(GreaterEqual1)
+      else
+        @index = i0
+        r0 = nil
+      end
+
+      node_cache[:greater_equal][start_index] = r0
+
+      r0
+    end
+
+    module Greater0
+      def field
+        elements[0]
+      end
+
+      def value
+        elements[4]
+      end
+    end
+
+    module Greater1
+      def query
+        Query::Greater.new field.sql, value.sql
+      end
+
+      def sql
+        Sequel::SQL::BooleanExpression.new(:>, field.sql, value.sql)
+      end
+    end
+
+    def _nt_greater
+      start_index = index
+      if node_cache[:greater].has_key?(index)
+        cached = node_cache[:greater][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      i0, s0 = index, []
+      r1 = _nt_field
+      s0 << r1
+      if r1
+        r3 = _nt_space
+        if r3
+          r2 = r3
+        else
+          r2 = instantiate_node(SyntaxNode,input, index...index)
+        end
+        s0 << r2
+        if r2
+          if has_terminal?('>', false, index)
+            r4 = instantiate_node(SyntaxNode,input, index...(index + 1))
+            @index += 1
+          else
+            terminal_parse_failure('>')
+            r4 = nil
+          end
+          s0 << r4
+          if r4
+            r6 = _nt_space
+            if r6
+              r5 = r6
+            else
+              r5 = instantiate_node(SyntaxNode,input, index...index)
+            end
+            s0 << r5
+            if r5
+              r7 = _nt_value
+              s0 << r7
+            end
+          end
+        end
+      end
+      if s0.last
+        r0 = instantiate_node(SyntaxNode,input, i0...index, s0)
+        r0.extend(Greater0)
+        r0.extend(Greater1)
+      else
+        @index = i0
+        r0 = nil
+      end
+
+      node_cache[:greater][start_index] = r0
+
+      r0
+    end
+
     module NotEquals0
       def field
         elements[0]
@@ -623,24 +1071,34 @@ module UState
       end
 
       i0 = index
-      r1 = _nt_double_quoted_string
+      r1 = _nt_true
       if r1
         r0 = r1
       else
-        r2 = _nt_float
+        r2 = _nt_false
         if r2
           r0 = r2
         else
-          r3 = _nt_integer
+          r3 = _nt_double_quoted_string
           if r3
             r0 = r3
           else
-            r4 = _nt_null
+            r4 = _nt_float
             if r4
               r0 = r4
             else
-              @index = i0
-              r0 = nil
+              r5 = _nt_integer
+              if r5
+                r0 = r5
+              else
+                r6 = _nt_null
+                if r6
+                  r0 = r6
+                else
+                  @index = i0
+                  r0 = nil
+                end
+              end
             end
           end
         end
@@ -651,6 +1109,84 @@ module UState
       r0
     end
 
+    module True0
+      def query
+        Query::True.new
+      end
+        
+      def ruby_value
+        true
+      end
+
+      def sql
+        Sequel::TRUE
+      end
+    end
+
+    def _nt_true
+      start_index = index
+      if node_cache[:true].has_key?(index)
+        cached = node_cache[:true][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      if has_terminal?('true', false, index)
+        r0 = instantiate_node(SyntaxNode,input, index...(index + 4))
+        r0.extend(True0)
+        @index += 4
+      else
+        terminal_parse_failure('true')
+        r0 = nil
+      end
+
+      node_cache[:true][start_index] = r0
+
+      r0
+    end
+
+    module False0
+      def query
+        Query::False.new
+      end
+        
+      def ruby_value
+        false
+      end
+      
+      def sql
+        Sequel::FALSE
+      end
+    end
+
+    def _nt_false
+      start_index = index
+      if node_cache[:false].has_key?(index)
+        cached = node_cache[:false][index]
+        if cached
+          cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+          @index = cached.interval.end
+        end
+        return cached
+      end
+
+      if has_terminal?('false', false, index)
+        r0 = instantiate_node(SyntaxNode,input, index...(index + 5))
+        r0.extend(False0)
+        @index += 5
+      else
+        terminal_parse_failure('false')
+        r0 = nil
+      end
+
+      node_cache[:false][start_index] = r0
+
+      r0
+    end
+
     module Integer0
     end
 
@@ -902,8 +1438,20 @@ module UState
                 r0 = r5
                 r0.extend(Field0)
               else
-                @index = i0
-                r0 = nil
+                if has_terminal?("time", false, index)
+                  r6 = instantiate_node(SyntaxNode,input, index...(index + 4))
+                  @index += 4
+                else
+                  terminal_parse_failure("time")
+                  r6 = nil
+                end
+                if r6
+                  r0 = r6
+                  r0.extend(Field0)
+                else
+                  @index = i0
+                  r0 = nil
+                end
               end
             end
           end