bud 0.9.6 → 0.9.7

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    MTY4MjlkMTY1NjZlYTZiOTVjNmYyMmIzYTliNDVlZDIwZjA0YjY4Yw==
-  data.tar.gz: !binary |-
-    MTE0ZGY3MDg5NzBmMTJkYTExNDNjYTcyNTFhOThmMzk0ZTM4YzgxOA==
-!binary "U0hBNTEy":
-  metadata.gz: !binary |-
-    OGJmYmMwZjU4Y2RlZWUyNDVkMGFkYTczZDhlNjk5MGQ5NmVlZDIyYzQyMjAz
-    NzEyZWI0YTI1MjVjM2VhZGRjMWE4ZDM4MGVlYzA4YzliYmJjNGUwZjEyY2I2
-    YjZhOTVjNzk5MGVjNWJkMTA0YWRmODk3MDAyYmVlOTM4MmVmYzA=
-  data.tar.gz: !binary |-
-    YjZkOTlkYzQwODJkZTI4YjU0YmFhMmY4MDY5Mjg3Y2ZjMzUyN2ZmZTAyOGI5
-    M2M3NWQ5OTgyM2U2ZjA3NGIzMjkwZjhjMTBhNWFmYzc3OTNiYmZjOWMzNmU3
-    MTkzZmI1MzJlODNkMmE2Y2E1NzU5ODYzNDlhM2Q3N2Q3MDk2Yzg=
+SHA1:
+  metadata.gz: 3d9ec3dd7f92dded5c77e541617853e0954fccaf
+  data.tar.gz: 37c31e760cbfe5305ebb73d3ff6e2417e499022d
+SHA512:
+  metadata.gz: a9fe2678a4aa143ebfa6ae677426bc60593550bda6209edc056855a6bca1b4f7ee5b4fc99e773d74679ca57fc615da7128e6db4622c855d45fda60e138bb78de
+  data.tar.gz: 240d532afbe76c4a7740f65276e7d0bb405b52661e8dde63fc42e54a5d40deb6d23c0bea1ef7d4a9e0e0e060c837533a0e3202c3ab41f33779861df6db673509

data/History.txt

@@ -1,3 +1,44 @@
+== 0.9.7 / 2013-04-22
+
+* Avoid raising an exception when Bud is used with Ruby 2.0. There isn't
+  anything special that Bud itself needs to do to support Ruby 2.0, but
+  RubyParser doesn't properly support 2.0 yet. So using 2.0-only syntax in Bud
+  rules is unlikely to work (until RubyParser is updated), but no other problems
+  have been observed.
+* Reject <= on collections from outside Bloom rules (#289, Aaron Davidson). The
+  previous behavior was error-prone: inserting into a scratch collection via <=
+  was not rejected but the inserted data would immediately be discarded at the
+  beginning of the next tick. Hence, it is simpler to require <+ or <~ for all
+  insertions from outside Bloom rules.
+* Fix bug in join operators whose qualifiers reference collections defined
+  inside imported modules (#301)
+* Fix bug in join operators when the same table and column name appears on the
+  LHS of multiple join predicates (#313)
+* Fix bug in outer joins with multiple predicates (#315)
+* Fix several bugs in rescan/invalidation logic for tables in the presence of
+  upstream deletions (#303)
+* Fix regression in file_reader collections (#304)
+* Improve chaining of notin operators
+* Optimize join evaluation with multiple predicates
+* Optimize notin self joins
+* Improve error reporting for syntax errors on temp collections (#310)
+* Add Travis CI integration hooks (#300, Josh Rosen)
+
+Lattices:
+
+* Rename lmap#apply_monotone to lmap#apply. Also, allow #apply to take either
+  monotone functions or morphisms, and improve error reporting.
+* Add lmap#filter. This takes an lmap whose values are elements of the lbool
+  lattice and returns an lmap containing only those k/v pairs where the value is
+  true.
+* More intuitive lattice equi-join syntax (lset#eqjoin). Rather than specifying
+  the join predicate(s) using array indexes, instead allow them to be specified
+  using a hash of field names, as in traditional Bloom joins. This only works
+  when the lset contains Structs.
+* Remove lset#project, and allow lset#eqjoin to take zero predicates; in the
+  latter case, eqjoin computes the Cartesian product.
+* Support deletion rules (<-) with lattice values on the RHS
+
 == 0.9.6 / 2013-02-25
 
 * Support syntax sugar for initializing lattices (#294). For example, rather

data/README.md

@@ -11,11 +11,12 @@ documentation.
 Main deficiencies at this point are:
 
 - No Ruby constraints: Within Bloom programs the full power of Ruby is also
-  available, including mutable state.  This allows programmers to get outside
-  the Bloom framework and lose cleanliness.
+  available, including mutable state. This allows programmers to get outside the
+  Bloom framework and lose cleanliness.
 
-- Compatibility: Bud only works with Ruby (MRI) 1.8.7 and 1.9. JRuby and other
-  Ruby implementations are currently not supported.
+- Compatibility: Bud only works with Ruby (MRI) 1.8.7 and 1.9. Bud also has
+  experimental support for Ruby 2.0. JRuby and other Ruby implementations are
+  currently not supported.
 
 ## Installation
 

data/docs/cheat.md

@@ -21,15 +21,23 @@ As in Ruby, backslash is used to escape a newline.<br>
     end
     
 ## State Declarations ##
-A `state` block contains Bud collection definitions. A Bud collection is a *set*
-of *facts*; each fact is an array of Ruby values. Note that collections do not
-contain duplicates (inserting a duplicate fact into a collection is ignored).
+
+A `state` block contains definitions of two kinds of program state:
+*collections* and *lattices*. A Bud collection is a *set* of *facts*; each fact
+is an array of Ruby values. Note that collections do not contain duplicates
+(inserting a duplicate fact into a collection is ignored).
 
 Like a table in a relational database, a subset of the columns in a collection
 makeup the collection's _key_. Attempting to insert two facts into a collection
 that agree on the key columns (but are not duplicates) results in a runtime
 exception.
 
+A lattice represents a value that *grows* over time, where the definition of
+"growth" depends on the kind of lattice in question. For example, an `lset`
+lattice contains a set of facts that grows over time (similar to a traditional
+Bud collection), whereas an `lmax` lattice holds an increasing integer
+value. For more information on lattices, see the section below.
+
 ### Default Declaration Syntax ###
 *BudCollection :name, [keys] => [values]*
 
@@ -245,12 +253,6 @@ Finally, we output every tuple of `bc` that does *not* appear in `t`.
 * `bc.argagg(:exemplary_agg_name, [:attr1], :attr2))`.  *generalizes argmin/max: returns the bc items per attr1 that are chosen by the exemplary
 aggregate named*
 
-### Built-in Aggregates: ###
-
-* Exemplary aggs: `min`, `max`, `choose`
-* Summary aggs: `count`, `sum`, `avg`
-* Structural aggs: `accum`
-
 Note that custom aggregation can be written using `reduce`.
 
 ## Collection Combination (Join) ###
@@ -313,6 +315,103 @@ The schema of a temp collection in inherited from the rhs; if the rhs has no
 schema, a simple one is manufactured to suit the data found in the rhs at
 runtime: `[c0, c1, ...]`.
 
+## Lattices ##
+
+In addition to traditional Bud collections and relational-style statements that
+operate over collections, Bud also supports lattices and rules that operate over
+lattices. Lattices provide a way to represent values that *grow over time*,
+where the definition of "growth" depends on the kind of lattice. The following
+built-in lattice types are currently supported:
+
+<table>
+  <tr>
+    <td><b>Name</b></td>
+    <td><b>Description</b></td>
+    <td><b>Initial Value</b></td>
+    <td><b>Monotone Functions</b></td>
+  </tr>
+
+  <tr>
+    <td><code>lbool</code></td>
+    <td>Threshold test (<code>false</code> => <code>true</code> conditional)</td>
+    <td>false</td>
+    <td>when_true</td>
+  </tr>
+
+  <tr>
+    <td><code>lmax</code></td>
+    <td>Increasing numeric value</td>
+    <td>-&infin;</td>
+    <td>gt(n), gt_eq(n), +(n), -(n)</td>
+  </tr>
+
+  <tr>
+    <td><code>lmin</code></td>
+    <td>Decreasing numeric value</td>
+    <td>+&infin;</td>
+    <td>lt(n), lt_eq(n), +(n), -(n)</td>
+  </tr>
+
+  <tr>
+    <td><code>lset</code></td>
+    <td>Growing set of values</td>
+    <td>empty set</td>
+    <td>contains?, eqjoin, filter, intersect, product, project, size</td>
+  </tr>
+
+  <tr>
+    <td><code>lpset</code></td>
+    <td>Growing set of non-negative numeric values</td>
+    <td>empty set</td>
+    <td>contains?, eqjoin, filter, intersect, product, project, size, sum</td>
+  </tr>
+
+  <tr>
+    <td><code>lbag</code></td>
+    <td>Growing multiset of values</td>
+    <td>empty multiset</td>
+    <td>contains?, multiplicity, intersect, product, project, size</td>
+  </tr>
+
+  <tr>
+    <td><code>lmap</code></td>
+    <td>Map from keys to lattice values</td>
+    <td>empty map</td>
+    <td>at, intersect, key?, key_set, project, size</td>
+  </tr>
+</table>
+
+Lattices can be declared in `state` blocks in a similar manner to traditional
+Bud collections. Similarly, Bloom rules can invoke functions on lattice
+values. A simple Bloom program that uses lattices to compute a quorum vote is as
+follows:
+
+```ruby
+QUORUM_SIZE = 5
+RESULT_ADDR = "example.org"
+
+class QuorumVote
+  include Bud
+
+  state do
+    channel :vote_chn, [:@addr, :voter_id]
+    channel :result_chn, [:@addr]
+    lset    :votes
+    lmax    :vote_cnt
+    lbool   :vote_done
+  end
+
+  bloom do
+    votes      <= vote_chn {|v| v.voter_id}
+    vote_cnt   <= votes.size
+    got_quorum <= vote_cnt.gt_eq(QUORUM_SIZE)
+    result_chn <~ got_quorum.when_true { [RESULT_ADDR] }
+  end
+end
+```
+
+For more information on lattice support in Bloom, see this [recent paper](http://db.cs.berkeley.edu/papers/socc12-blooml.pdf).
+
 ## Bud Modules ##
 A Bud module combines state (collections) and logic (Bloom rules). Using modules allows your program to be decomposed into a collection of smaller units.
 

data/lib/bud/aggs.rb

@@ -36,7 +36,7 @@ module Bud
 
   class Min < ArgExemplary #:nodoc: all
     def trans(the_state, val)
-      if the_state < val
+      if (the_state <=> val) < 0
         return the_state, :ignore
       elsif the_state == val
         return the_state, :keep
@@ -53,8 +53,10 @@ module Bud
 
   class Max < ArgExemplary #:nodoc: all
     def trans(the_state, val)
-      if the_state > val
+      if (the_state <=> val) > 0
         return the_state, :ignore
+      elsif the_state == val
+        return the_state, :keep
       else
         return val, :replace
       end

data/lib/bud/bud_meta.rb

@@ -17,7 +17,7 @@ class BudMeta #:nodoc: all
       # stratum_map = {fully qualified pred => stratum}. Copy stratum_map data
       # into t_stratum format.
       raise unless @bud_instance.t_stratum.to_a.empty?
-      @bud_instance.t_stratum <= stratum_map.to_a
+      @bud_instance.t_stratum.merge(stratum_map.to_a)
 
       # slot each rule into the stratum corresponding to its lhs pred (from stratum_map)
       stratified_rules = Array.new(top_stratum + 2) { [] }  # stratum -> [ rules ]
@@ -265,9 +265,6 @@ class BudMeta #:nodoc: all
       da = ::DepAnalysis.new
       da.providing <+ @bud_instance.tables[:t_provides].to_a
       da.depends <+ @bud_instance.t_depends.map{|d| [d.lhs, d.op, d.body, d.nm]}
-
-      #@bud_instance.tables[:t_provides].each {|t| da.providing <+ t}
-      #@bud_instance.tables[:t_depends].each {|t| da.depends_tc <+ t}
       da.tick_internal
       @dependency_analysis = da
     end

data/lib/bud/collections.rb

@@ -14,11 +14,11 @@ module Bud
   class BudCollection
     include Enumerable
 
-    attr_accessor :bud_instance, :tabname  # :nodoc: all
-    attr_reader :cols, :key_cols # :nodoc: all
+    attr_accessor :bud_instance  # :nodoc: all
+    attr_reader :tabname, :cols, :key_cols # :nodoc: all
     attr_reader :struct
     attr_reader :storage, :delta, :new_delta, :pending, :tick_delta # :nodoc: all
-    attr_reader :wired_by, :to_delete
+    attr_reader :wired_by, :scanner_cnt
     attr_accessor :invalidated, :rescan
     attr_accessor :is_source
     attr_accessor :accumulate_tick_deltas # updated in bud.do_wiring
@@ -28,6 +28,7 @@ module Bud
       @bud_instance = bud_instance
       @invalidated = true
       @is_source = true # unless it shows up on the lhs of some rule
+      @scanner_cnt = 0
       @wired_by = []
       @accumulate_tick_deltas = false
       init_schema(given_schema) unless given_schema.nil? and defer_schema
@@ -62,7 +63,7 @@ module Bud
       if @cols.empty?
         @cols = nil
       else
-        @struct = ($struct_classes[@cols] ||= Struct.new(*@cols))
+        @struct = ($struct_classes[@cols] ||= Bud::TupleStruct.new(*@cols))
         @structlen = @struct.members.length
       end
       setup_accessors
@@ -131,11 +132,19 @@ module Bud
         end
       end
 
-      # set up schema accessors, which are class methods
+      # Setup schema accessors, which are class methods. Note that the same
+      # table/column name might appear multiple times on the LHS of a single
+      # join (e.g., (foo * bar).combos(foo.x => bar.y, foo.x => bar.z)). Because
+      # the join predicates are represented as a hash, we need the two instances
+      # of foo.x to be distinct values (otherwise the resulting hash will only
+      # have a single key). Hence, we add a unique ID to the value returned by
+      # schema accessors.
       @cols_access = Module.new do
         sc.each_with_index do |c, i|
           define_method c do
-            [@tabname, i, c]
+            @counter ||= 0
+            @counter += 1
+            [qualified_tabname, i, c, @counter]
           end
         end
       end
@@ -188,11 +197,8 @@ module Bud
       if @bud_instance.wiring?
         pusher = to_push_elem(the_name, the_schema)
         # If there is no code block evaluate, use the scanner directly
-        return pusher if blk.nil?
-        pusher_pro = pusher.pro(&blk)
-        pusher_pro.elem_name = the_name
-        pusher_pro.tabname = the_name
-        pusher_pro
+        pusher = pusher.pro(&blk) unless blk.nil?
+        pusher
       else
         rv = []
         self.each do |t|
@@ -209,7 +215,7 @@ module Bud
     def each_with_index(the_name=tabname, the_schema=schema, &blk)
       if @bud_instance.wiring?
         pusher = to_push_elem(the_name, the_schema)
-        pusher.each_with_index(the_name, the_schema, &blk)
+        pusher.each_with_index(&blk)
       else
         super(&blk)
       end
@@ -251,7 +257,7 @@ module Bud
     end
 
     def rename(the_name, the_schema=nil, &blk)
-      raise unless @bud_instance.wiring?
+      raise Bud::Error unless @bud_instance.wiring?
       # a scratch with this name should have been defined during rewriting
       unless @bud_instance.respond_to? the_name
         raise Bud::Error, "rename failed to define a scratch named #{the_name}"
@@ -397,7 +403,7 @@ module Bud
     private
     def raise_pk_error(new, old)
       key = get_key_vals(old)
-      raise Bud::KeyConstraintError, "key conflict inserting #{new.inspect} into \"#{tabname}\": existing tuple #{old.inspect}, key = #{key.inspect}"
+      raise Bud::KeyConstraintError, "key conflict inserting #{new.inspect} into \"#{qualified_tabname}\": existing tuple #{old.inspect}, key = #{key.inspect}"
     end
 
     private
@@ -408,7 +414,7 @@ module Bud
     private
     def prep_tuple(o)
       return o if o.class == @struct
-      if o.class == Array
+      if o.kind_of? Array
         if @struct.nil?
           sch = (1 .. o.length).map{|i| "c#{i}".to_sym}
           init_schema(sch)
@@ -446,7 +452,7 @@ module Bud
                       when @delta.object_id; "delta"
                       when @new_delta.object_id; "new_delta"
                     end
-        puts "#{qualified_tabname}.#{storetype} ==> #{t}"
+        puts "#{qualified_tabname}.#{storetype} ==> #{t.inspect}"
       end
       return if t.nil? # silently ignore nils resulting from map predicates failing
       t = prep_tuple(t)
@@ -600,6 +606,10 @@ module Bud
     public
     # instantaneously merge items from collection +o+ into +buf+
     def <=(collection)
+      unless bud_instance.toplevel.inside_tick
+        raise Bud::CompileError, "illegal use of <= outside of bloom block, use <+ instead"
+      end
+
       merge(collection)
     end
 
@@ -712,6 +722,7 @@ module Bud
                                           self, the_schema)
         toplevel.scanners[this_stratum][[oid, the_name]] = scanner
         toplevel.push_sources[this_stratum][[oid, the_name]] = scanner
+        @scanner_cnt += 1
       end
       return toplevel.scanners[this_stratum][[oid, the_name]]
     end
@@ -839,6 +850,11 @@ module Bud
     public
     def add_rescan_invalidate(rescan, invalidate)
       srcs = non_temporal_predecessors
+
+      # XXX: this seems wrong. We might rescan a node for many reasons (e.g.,
+      # because another one of the node's outputs needs to be refilled). We only
+      # need to invalidate + rescan this scratch if one of the inputs to this
+      # collection is *invalidated*.
       if srcs.any? {|e| rescan.member? e}
         invalidate << self
         rescan.merge(srcs)
@@ -1012,7 +1028,7 @@ module Bud
       if @payload_struct.nil?
         payload_cols = cols.dup
         payload_cols.delete_at(@locspec_idx)
-        @payload_struct = Struct.new(*payload_cols)
+        @payload_struct = Bud::TupleStruct.new(*payload_cols)
         @payload_colnums = payload_cols.map {|k| cols.index(k)}
       end
 
@@ -1078,11 +1094,9 @@ module Bud
     # add the terminal file descriptor to the EM event loop.
     private
     def read_line
-      toplevel = @bud_instance.toplevel
-      if @prompt
-        get_out_io.print("#{tabname} > ")
-      end
+      get_out_io.print("#{tabname} > ") if @prompt
 
+      toplevel = @bud_instance.toplevel
       in_io = toplevel.options[:stdin]
       input_str = in_io.gets
       return false if input_str.nil? # Hit EOF
@@ -1113,7 +1127,7 @@ module Bud
     public
     def tick #:nodoc: all
       unless @pending.empty?
-        @delta = @pending #  pending used for input tuples in this case.
+        @delta = @pending       # pending used for input tuples in this case
         @tick_delta = @pending.values
         @pending.clear
       else
@@ -1121,7 +1135,7 @@ module Bud
         @delta.clear
         @tick_delta.clear
       end
-      @invalidated = true # channels and terminals are always invalidated.
+      @invalidated = true       # channels and terminals are always invalidated
     end
 
     public
@@ -1213,8 +1227,8 @@ module Bud
     public
     def tick #:nodoc: all
       if $BUD_DEBUG
-        puts "#{tabname}. storage -= pending deletes" unless @to_delete.empty? and @to_delete_by_key.empty?
-        puts "#{tabname}. delta += pending" unless @pending.empty?
+        puts "#{tabname}.storage -= pending deletes" unless @to_delete.empty? and @to_delete_by_key.empty?
+        puts "#{tabname}.delta += pending" unless @pending.empty?
       end
       @tick_delta.clear
       deleted = nil
@@ -1242,7 +1256,9 @@ module Bud
     end
 
     def invalidated=(val)
-      raise Bud::Error, "internal error: must not set invalidate on tables"
+      # Might be reset to false at end-of-tick, but shouldn't be set to true
+      raise Bud::Error, "cannot not set invalidate on table '#{@tabname}'" if val
+      super
     end
 
     def pending_delete(o)
@@ -1256,6 +1272,12 @@ module Bud
         add_merge_target
         tbl = register_coll_expr(o)
         tbl.pro.wire_to(self, :delete)
+      elsif o.class <= Bud::LatticePushElement
+        add_merge_target
+        o.wire_to(self, :delete)
+      elsif o.class <= Bud::LatticeWrapper
+        add_merge_target
+        o.to_push_elem.wire_to(self, :delete)
       else
         unless o.nil?
           o = o.uniq.compact if o.respond_to?(:uniq)
@@ -1293,7 +1315,7 @@ module Bud
       # No cache to invalidate. Also, tables do not invalidate dependents,
       # because their own state is not considered invalidated; that happens only
       # if there were pending deletes at the beginning of a tick (see tick())
-      puts "******** invalidate_cache called on BudTable" if $BUD_DEBUG
+      puts "******** invalidate_cache called on table '#{@tabname}'" if $BUD_DEBUG
     end
 
     public
@@ -1339,6 +1361,7 @@ module Bud
     public
     def each(&block)
       v = @expr.call
+      return if v.nil? or v == [nil]
 
       # XXX: Gross hack. We want to support RHS expressions that do not
       # necessarily return BudCollections (they might instead return lattice
@@ -1367,6 +1390,11 @@ module Bud
       # NEEDS A TRY/RESCUE BLOCK
       @fd = File.open(@filename, "r")
       @linenum = 0
+      @invalidated = true
+    end
+    
+    def tick
+      @invalidated = true
     end
 
     public

data/lib/bud/executor/README.rescan

@@ -3,7 +3,8 @@ Notes on Invalidate and Rescan in Bud
 
 (I'll use 'downstream' to mean rhs to lhs (like in budplot). In every stratum,
 data originates at scanned sources at the "top", winds its way through various
-PushElements and ends up in a collection at the "bottom". I'll also the term
+PushElements and ends up in a collection at the "bottom". That is, data flows
+from "upstream" producers to "downstream" consumers. I'll also the term
 "elements" to mean both dataflow nodes (PushElements) and collections).
 
 Invalidation strategy works through two flags/signals, rescan and