bud 0.9.2 → 0.9.3

data/History.txt

@@ -1,4 +1,18 @@
-== 0.9.2 / ???
+== 0.9.3 / 2012-08-20
+
+* Change behavior of accum() aggregate to return a Set, rather than an Array in
+  an unspecified order
+* Fix several serious bugs in caching/invalidation of materialized operator
+  state (#276, #278, #279)
+* Avoid possible spurious infinite loop with dbm-backed collections
+* Optimize aggregation/grouping performance
+* Fix bugs and improve performance for materialization of sort operator
+* Fix REBL regression with push-based runtime (#274)
+* Minor performance optimizations for simple projection rules
+* Remove dependency on gchart
+* Built-in support for code coverage with MRI 1.9 and SimpleCov
+
+== 0.9.2 / 2012-05-19
 
 * Add new aggregate functions: bool_and() and bool_or()
 * Fix bugs in notin() stratification and implementation (#271)
@@ -16,7 +30,7 @@
   * Previous behavior was to ignore additional fields, but this was found to be
     error-prone
 * Remove builtin support for BUST (web services API); this avoids the need to
-  depend on the json, nestful and i18n gems.
+  depend on the json, nestful and i18n gems
 
 == 0.9.0 / 2012-03-21
 
@@ -39,5 +53,4 @@
   * Support for Bloom-based signal handling has been removed
   * Support for the "with" syntax has been removed
   * The Bloom-based "deployment" framework has been removed
-  * Support for Tokyo Cabinet-based collections has been removed
-
+  * Support for Tokyo Cabinet-backed collections has been removed

data/README.md

@@ -37,6 +37,11 @@ To run the unit tests:
     % gem install minitest      # unless already installed
     % cd test; ruby ts_bud.rb
 
+To run the unit tests and produce a code coverage report:
+
+    % gem install simplecov    # unless already installed
+    % cd test; COVERAGE=1 ruby ts_bud.rb
+
 ## Optional Dependencies
 
 The bud gem has a handful of mandatory dependencies. It also has one optional

data/docs/cheat.md

@@ -239,7 +239,7 @@ Finally, we output every tuple of `bc` that does *not* appear in `t`.
 * `bc.group([:col1, :col2], min(:col3))`.  *akin to min(col3) GROUP BY col1,col2*
   * exemplary aggs: `min`, `max`, `bool_and`, `bool_or`, `choose`
   * summary aggs: `sum`, `avg`, `count`
-  * structural aggs: `accum`
+  * structural aggs: `accum` *accumulates inputs into a Set*
 * `bc.argmax([:attr1], :attr2)`      *returns the bc items per attr1 that have highest attr2*
 * `bc.argmin([:attr1], :attr2)`
 * `bc.argagg(:exemplary_agg_name, [:attr1], :attr2))`.  *generalizes argmin/max: returns the bc items per attr1 that are chosen by the exemplary

data/docs/getstarted.md

@@ -14,7 +14,7 @@ This installs four things:
 
 * The `Bud` module, to embed Bloom code in Ruby.
 * The `rebl` executable: an interactive shell for trying out Bloom.
-* The `budplot` and `budvis` executables: graphical tools for visualizing and debugging Bloom programs.
+* The `budplot`, `budvis`, and `budtimelines` executables: graphical tools for visualizing and debugging Bloom programs.
 
 ## First Blooms ##
 

data/lib/bud.rb

@@ -315,12 +315,12 @@ module Bud
 
     # Check scan and merge_targets to see if any builtin_tables need to be added as well.
     @scanners.each do |scs|
-      @app_tables += scs.values.map {|s| s.collection}
+      @app_tables.merge(scs.values.map {|s| s.collection})
     end
     @merge_targets.each do |mts| #mts == merge_targets at stratum
-      @app_tables += mts
+      @app_tables.merge(mts)
     end
-    @app_tables = @app_tables.nil? ? [] : @app_tables.to_a
+    @app_tables = @app_tables.to_a
 
     # for each stratum create a sorted list of push elements in topological order
     @push_sorted_elems = []
@@ -352,18 +352,18 @@ module Bud
       end
     end
 
-    # sanity check
+    # Sanity check
     @push_sorted_elems.each do |stratum_elems|
-      stratum_elems.each do |se|
-        se.check_wiring
-      end
+      stratum_elems.each {|se| se.check_wiring}
     end
 
-    # create sets of elements and collections to invalidate or rescan at the beginning of each tick.
+    # Create sets of elements and collections to invalidate or rescan at the
+    # beginning of each tick
     prepare_invalidation_scheme
 
-    # For all tables that are accessed (scanned) in a stratum higher than the one they are updated in, set
-    # a flag to track deltas accumulated in that tick (see: collection.tick_delta)
+    # For all tables that are accessed (scanned) in a stratum higher than the
+    # one they are updated in, set a flag to track deltas accumulated in that
+    # tick (see: collection.tick_delta)
     stratum_accessed = {}
     (@num_strata-1).downto(0) do |stratum|
       @scanners[stratum].each_value do |s|
@@ -384,24 +384,38 @@ module Bud
     end
   end
 
-  # All collections (elements included) are semantically required to erase any cached information at the start of a tick
-  # and start from a clean slate. prepare_invalidation_scheme prepares a just-in-time invalidation scheme that
-  # permits us to preserve data from one tick to the next, and to keep things in incremental mode unless there's a
-  # negation.
+  # All collections (elements included) are semantically required to erase any
+  # cached information at the start of a tick and start from a clean
+  # slate. prepare_invalidation_scheme prepares a just-in-time invalidation
+  # scheme that permits us to preserve data from one tick to the next, and to
+  # keep things in incremental mode unless there's a negation.
+  #
   # This scheme solves the following constraints.
-  # 1. A full scan of an element's contents results in downstream elements getting full scans themselves (i.e no \
-  #    deltas). This effect is transitive.
-  # 2. Invalidation of an element's cache results in rebuilding of the cache and a consequent fullscan. See next.
-  # 3. Invalidation of an element requires upstream elements to rescan their contents, or to transitively pass the
-  #    request on further upstream. Any element that has a cache can rescan without passing on the request to higher
-  #    levels.
   #
-  # This set of constraints is solved once during wiring, resulting in four data structures
-  # @default_invalidate = set of elements and tables to always invalidate at every tick. Organized by stratum
-  # @default_rescan = set of elements and tables to always scan fully in the first iteration of every tick.
-  # scanner[stratum].invalidate = Set of elements to additionally invalidate if the scanner's table is invalidated at
-  #  run-time
-  # scanner[stratum].rescan = Similar to above.
+  # 1. A full scan of an element's contents results in downstream elements
+  #    getting full scans themselves (i.e., no deltas). This effect is
+  #    transitive.
+  # 2. Invalidation of an element's cache results in rebuilding of the cache and
+  #    a consequent fullscan. See next.
+  # 3. Invalidation of an element requires upstream elements to rescan their
+  #    contents, or to transitively pass the request on further upstream. Any
+  #    element that has a cache can rescan without passing on the request to
+  #    higher levels.
+  #
+  # This set of constraints is solved once during wiring, resulting in four data
+  # structures:
+  #
+  # @default_invalidate = Set of elements and tables to always invalidate at
+  #                       every tick.
+  #
+  # @default_rescan = Set of elements and tables to always scan fully in the
+  #                   first iteration of every tick.
+  #
+  # scanner[stratum].invalidate_set = Set of elements to additionally invalidate
+  #                                   if the scanner's table is invalidated at
+  #                                   run-time.
+  #
+  # scanner[stratum].rescan_set = Similar to above.
   def prepare_invalidation_scheme
     num_strata = @push_sorted_elems.size
     if $BUD_SAFE
@@ -443,22 +457,26 @@ module Bud
 
     num_strata.times do |stratum|
       @push_sorted_elems[stratum].each do |elem|
-        if elem.rescan_at_tick
-          rescan << elem
-        end
+        rescan << elem if elem.rescan_at_tick
 
         if elem.outputs.any?{|tab| not(tab.class <= PushElement) and nm_targets.member? tab.qualified_tabname.to_sym }
-          rescan += elem.wired_by
+          rescan.merge(elem.wired_by)
         end
       end
       rescan_invalidate_tc(stratum, rescan, invalidate)
     end
 
+    puts "(PRE) Default rescan: #{rescan.inspect}" if $BUD_DEBUG
+    puts "(PRE) Default inval: #{invalidate.inspect}" if $BUD_DEBUG
+
     prune_rescan_invalidate(rescan, invalidate)
     # transitive closure
     @default_rescan = rescan.to_a
     @default_invalidate = invalidate.to_a
 
+    puts "(POST) Default rescan: #{rescan.inspect}" if $BUD_DEBUG
+    puts "(POST) Default inval: #{invalidate.inspect}" if $BUD_DEBUG
+
     # Now compute for each table that is to be scanned, the set of dependent
     # tables and elements that will be invalidated if that table were to be
     # invalidated at run time.
@@ -475,7 +493,8 @@ module Bud
         invalidate = dflt_invalidate.clone
         rescan_invalidate_tc(stratum, rescan, invalidate)
         prune_rescan_invalidate(rescan, invalidate)
-        to_reset += rescan + invalidate
+        to_reset.merge(rescan)
+        to_reset.merge(invalidate)
         # Give the diffs (from default) to scanner; these are elements that are
         # dependent on this scanner
         diffscan = (rescan - dflt_rescan).find_all {|elem| elem.class <= PushElement}
@@ -953,7 +972,7 @@ module Bud
   # One timestep of Bloom execution. This MUST be invoked from the EventMachine
   # thread; it is not intended to be called directly by client code.
   def tick_internal
-    puts "#{object_id}/#{port} : =============================================" if $BUD_DEBUG
+    puts "#{object_id}/#{port} : ============================================= (#{@budtime})" if $BUD_DEBUG
     begin
       starttime = Time.now if options[:metrics]
       if options[:metrics] and not @endtime.nil?
@@ -979,7 +998,7 @@ module Bud
         num_strata = @push_sorted_elems.size
         # The following loop invalidates additional (non-default) elements and
         # tables that depend on the run-time invalidation state of a table.
-        # Loop once to set the flags
+        # Loop once to set the flags.
         num_strata.times do |stratum|
           @scanners[stratum].each_value do |scanner|
             if scanner.rescan
@@ -1034,7 +1053,6 @@ module Bud
       invoke_callbacks
       @budtime += 1
       @inbound.clear
-
       @reset_list.each {|e| e.invalidated = false; e.rescan = false}
 
     ensure

data/lib/bud/aggs.rb

@@ -1,3 +1,5 @@
+require 'set'
+
 module Bud
   ######## Agg definitions
   class Agg #:nodoc: all
@@ -12,7 +14,7 @@ module Bud
     #     b. :keep tells the caller to save this input
     #     c. :replace tells the caller to keep this input alone
     #     d. :delete, [t1, t2, ...] tells the caller to delete the remaining tuples
-    #  For things that do not descend from ArgExemplary, the 2nd part can simply be nil.
+    #  For aggs that do not inherit from ArgExemplary, the 2nd part can simply be nil.
     def trans(the_state, val)
       return the_state, :ignore
     end
@@ -22,16 +24,12 @@ module Bud
     end
   end
 
-  class Exemplary < Agg #:nodoc: all
-  end
-
   # ArgExemplary aggs are used by argagg. Canonical examples are min/min (argmin/max)
   # They must have a trivial final method and be monotonic, i.e. once a value v
-  # is discarded in favor of another, v can never be the final result
-
+  # is discarded in favor of another, v can never be the final result.
   class ArgExemplary < Agg #:nodoc: all
     def tie(the_state, val)
-      (the_state == val)
+      the_state == val
     end
     def final(the_state)
       the_state
@@ -184,7 +182,7 @@ module Bud
       return retval, nil
     end
     def final(the_state)
-      the_state[0]*1.0 / the_state[1]
+      the_state[0].to_f / the_state[1]
     end
   end
 
@@ -196,7 +194,7 @@ module Bud
 
   class Accum < Agg #:nodoc: all
     def init(x)
-      [x]
+      [x].to_set
     end
     def trans(the_state, val)
       the_state << val
@@ -205,8 +203,7 @@ module Bud
   end
 
   # aggregate method to be used in Bud::BudCollection.group.
-  # accumulates all x inputs into an array. note that the order of the elements
-  # in the resulting array is undefined.
+  # accumulates all x inputs into a set.
   def accum(x)
     [Accum.new, x]
   end

data/lib/bud/bud_meta.rb

@@ -23,10 +23,12 @@ class BudMeta #:nodoc: all
       # slot each rule into the stratum corresponding to its lhs pred (from stratum_map)
       stratified_rules = Array.new(top_stratum + 2) { [] }  # stratum -> [ rules ]
       @bud_instance.t_rules.each do |rule|
-        if rule.op.to_s == '<='
+        if rule.op == '<='
           # Deductive rules are assigned to strata based on the basic Datalog
           # stratification algorithm
           belongs_in = stratum_map[rule.lhs]
+          # If the rule body doesn't reference any collections, it won't be
+          # assigned a stratum, so just place it in stratum zero
           belongs_in ||= 0
           stratified_rules[belongs_in] << rule
         else
@@ -112,18 +114,16 @@ class BudMeta #:nodoc: all
     #    a.b.c == s(:call, s1,  :c, (:args))
     # where s1 == s(:call, s2,  :b, (:args))
     # where s2 == s(:call, nil, :a, (:args))
-
     tag, recv, name, args = pt
-    return nil unless tag == :call or args.length == 1
+    return nil unless tag == :call and args.length == 1
 
     if recv
       qn = get_qual_name(recv)
       return nil if qn.nil? or qn.size == 0
-      qn = "#{qn}.#{name}"
+      return "#{qn}.#{name}"
     else
-      qn = name.to_s
+      return name.to_s
     end
-    qn
   end
 
   # Perform some basic sanity checks on the AST of a rule block. We expect a
@@ -156,9 +156,9 @@ class BudMeta #:nodoc: all
 
       # Check that LHS references a named collection
       lhs_name = get_qual_name(lhs)
-      return [n, "Unexpected lhs format: #{lhs}"] if lhs.nil?
+      return [n, "unexpected lhs format: #{lhs}"] if lhs_name.nil?
       unless @bud_instance.tables.has_key? lhs_name.to_sym
-        return [n, "Collection does not exist: '#{lhs_name}'"]
+        return [n, "collection does not exist: '#{lhs_name}'"]
       end
 
       return [n, "illegal operator: '#{op}'"] unless [:<, :<=].include? op
@@ -194,21 +194,21 @@ class BudMeta #:nodoc: all
     nodes = {}
     bud.t_depends.each do |d|
       #t_depends [:bud_instance, :rule_id, :lhs, :op, :body] => [:nm]
-      lhs = (nodes[d.lhs.to_s] ||= Node.new(d.lhs.to_s, :init, 0, [], true, false, false, false))
+      lhs = (nodes[d.lhs] ||= Node.new(d.lhs, :init, 0, [], true, false, false, false))
       lhs.in_lhs = true
-      body = (nodes[d.body.to_s] ||= Node.new(d.body.to_s, :init, 0, [], false, true, false, false))
+      body = (nodes[d.body] ||= Node.new(d.body, :init, 0, [], false, true, false, false))
       temporal = d.op != "<="
       lhs.edges << Edge.new(body, d.op, d.nm, temporal)
       body.in_body = true
     end
 
-    nodes.values.each {|n| calc_stratum(n, false, false, [n.name])}
+    nodes.each_value {|n| calc_stratum(n, false, false, [n.name])}
     # Normalize stratum numbers because they may not be 0-based or consecutive
     remap = {}
     # if the nodes stratum numbers are [2, 3, 2, 4], remap = {2 => 0, 3 => 1, 4 => 2}
-    nodes.values.map {|n| n.stratum}.uniq.sort.each_with_index{|num, i|
+    nodes.values.map {|n| n.stratum}.uniq.sort.each_with_index do |num, i|
       remap[num] = i
-    }
+    end
     stratum_map = {}
     top_stratum = -1
     nodes.each_pair do |name, n|
@@ -232,9 +232,9 @@ class BudMeta #:nodoc: all
       node.status = :in_process
       node.edges.each do |edge|
         node.is_neg_head = edge.neg
-        next if edge.op != "<="
+        next unless edge.op == "<="
         body_stratum = calc_stratum(edge.to, (neg or edge.neg), (edge.temporal or temporal), path + [edge.to.name])
-        node.is_neg_head = false #reset for next edge
+        node.is_neg_head = false # reset for next edge
         node.stratum = max(node.stratum, body_stratum + (edge.neg ? 1 : 0))
       end
       node.status = :done
@@ -253,15 +253,15 @@ class BudMeta #:nodoc: all
     bud.t_provides.each do |p|
       pred, input = p.interface, p.input
       if input
-        unless preds_in_body.include? pred.to_s
+        unless preds_in_body.include? pred
           # input interface is underspecified if not used in any rule body
           bud.t_underspecified << [pred, true] # true indicates input mode
           out.puts "Warning: input interface #{pred} not used"
         end
       else
-        unless preds_in_lhs.include? pred.to_s
+        unless preds_in_lhs.include? pred
           # output interface underspecified if not in any rule's lhs
-          bud.t_underspecified << [pred, false]  #false indicates output mode.
+          bud.t_underspecified << [pred, false]  # false indicates output mode
           out.puts "Warning: output interface #{pred} not used"
         end
       end

data/lib/bud/collections.rb

@@ -1,7 +1,6 @@
 require 'msgpack'
 
 $struct_classes = {}
-$EMPTY_HASH = {}
 module Bud
   ########
   #--
@@ -74,6 +73,10 @@ module Bud
       @qualified_tabname ||= @bud_instance.toplevel?  ? tabname : "#{@bud_instance.qualified_name}.#{tabname}".to_sym
     end
 
+    def inspect
+      "#{self.class}:#{self.object_id.to_s(16)} [#{qualified_tabname}]"
+    end
+
     # The user-specified schema might come in two forms: a hash of Array =>
     # Array (key_cols => remaining columns), or simply an Array of columns (if
     # no key_cols were specified). Return a pair: [list of (all) columns, list
@@ -102,10 +105,6 @@ module Bud
       return [cols, key_cols]
     end
 
-    def inspect
-      "#{self.class}:#{self.object_id.to_s(16)} [#{qualified_tabname}]"
-    end
-
     # produces the schema in a format that is useful as the schema specification for another table
     public
     def schema
@@ -189,6 +188,8 @@ module Bud
     def pro(the_name=tabname, the_schema=schema, &blk)
       if @bud_instance.wiring?
         pusher = to_push_elem(the_name, the_schema)
+        # If there is no code block evaluate, use the scanner directly
+        return pusher if blk.nil?
         pusher_pro = pusher.pro(&blk)
         pusher_pro.elem_name = the_name
         pusher_pro.tabname = the_name
@@ -260,7 +261,7 @@ module Bud
 
     public
     def non_temporal_predecessors
-      @wired_by.map {|elem| elem if elem.outputs.include? self}
+      @wired_by.select {|elem| elem.outputs.include? self}
     end
 
     public
@@ -393,7 +394,7 @@ module Bud
 
     private
     def get_key_vals(t)
-      @key_colnums.map {|i| t[i]}
+      t.values_at(*@key_colnums)
     end
 
     public
@@ -567,7 +568,7 @@ module Bud
       unless @delta.empty?
         puts "#{qualified_tabname}.tick_delta delta --> storage (#{@delta.size} elems)" if $BUD_DEBUG
         @storage.merge!(@delta)
-        @tick_delta += @delta.values if accumulate_tick_deltas
+        @tick_delta.concat(@delta.values) if accumulate_tick_deltas
         @delta.clear
       end
 
@@ -607,7 +608,7 @@ module Bud
       end
       unless @delta.empty?
         @storage.merge!(@delta)
-        @tick_delta += @delta.values if accumulate_tick_deltas
+        @tick_delta.concat(@delta.values) if accumulate_tick_deltas
         @delta.clear
       end
       unless @new_delta.empty?
@@ -704,14 +705,6 @@ module Bud
       return to_push_elem.reduce(initial, &blk)
     end
 
-    public
-    def pretty_print_instance_variables
-      # list of attributes (in order) to print when pretty_print is called.
-      important = ["@tabname", "@storage", "@delta", "@new_delta", "@pending"]
-      # everything except bud_instance
-      important + (self.instance_variables - important - ["@bud_instance"])
-    end
-
     public
     def uniquify_tabname # :nodoc: all
       # just append current number of microseconds
@@ -747,7 +740,7 @@ module Bud
       srcs = non_temporal_predecessors
       if srcs.any? {|e| rescan.member? e}
         invalidate << self
-        rescan += srcs
+        rescan.merge(srcs)
       end
     end
 
@@ -1145,7 +1138,7 @@ module Bud
       # No cache to invalidate. Also, tables do not invalidate dependents,
       # because their own state is not considered invalidated; that happens only
       # if there were pending deletes at the beginning of a tick (see tick())
-      puts "******** invalidate_cache called on BudTable"
+      puts "******** invalidate_cache called on BudTable" if $BUD_DEBUG
     end
 
     public
@@ -1190,7 +1183,7 @@ module Bud
 
     public
     def each(&block)
-      @expr.call.each {|i| yield i}
+      @expr.call.each(&block)
     end
 
     public
@@ -1221,7 +1214,7 @@ module Bud
 
     public
     def each(&blk)
-      each_raw {|l| blk.call(l)}
+      each_raw(&blk)
     end
   end
 end