bud 0.9.4 → 0.9.5

data/History.txt

@@ -1,3 +1,26 @@
+== 0.9.5 / 2012-11-24
+
+* Lattice branch (Bloom^L) merged
+* Compatibility with recent versions of ruby_parser (3.0.2+) and ruby2ruby
+  (2.0.1+) -- #. Older versions of these two gems are no longer supported
+* Add support for aggregate functions that take multiple input columns
+* Add built-in aggregate function accum_pair(x, y), which produces a Set of
+  pairs (two-element arrays [x,y])
+* Support user-specified code blocks in payloads(), argagg(), argmin() and
+  argmax()
+* Change behavior of BudChannel#payloads for channels with two
+  columns. Previously we returned a single *column* (scalar) value in this case;
+  now we always return a tuple with k-1 columns
+* More consistent behavior for BudCollection#sort when used outside Bloom
+  programs
+* Restore support for each_with_index() over Bud collections
+* Restore functionality of Zookeeper-backed Bud collections and fix
+  incompatibility with recent (> 0.4.4) versions of the Zookeeper gem
+* Optimize parsing of Bloom statements, particularly for large Bloom programs
+* Fix bug in argagg state materialization
+* Fix bug in chaining argmin() or argmax() expressions
+* Fix bug in chaining notin() expressions
+
 == 0.9.4 / 2012-09-06
 
 * Optimize grouping performance

data/bin/budlabel

@@ -0,0 +1,63 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'bud'
+require 'getopt/std'
+require 'bud/labeling/labeling'
+require 'bud/labeling/bloomgraph'
+require 'bud/labeling/budplot_style'
+
+$LOAD_PATH.unshift(".")
+
+@opts = Getopt::Std.getopts("r:i:p:O:CP")
+
+unless @opts["r"] and @opts["i"]
+  puts "USAGE:"
+  puts "-r REQUIRE"
+  puts "-i INCLUDE"
+  puts "[-p INCLUDE PATH]"
+  puts "[-O <FMT> Output a graphviz representation of the module in FMT format (pdf if not specified)."
+  puts "-C Concise output -- Associate a single label with each output interface"
+  puts "-P Path-based output -- For each output interface, attribute a label to paths from each input interface"
+  exit
+end
+
+hreadable = {
+  "D" => "Diffluent: Nondeterministic output contents.",
+  "A" => "Asynchronous.  Nondeterministic output orders.",
+  "N" => "Nonmonotonic.  Output contents are sensitive to input orders.",
+  "Bot" => "Monotonic.  Order-insensitive and retraction-free."
+}
+
+if @opts["p"]
+  $LOAD_PATH.unshift @opts["p"]
+end
+
+require @opts["r"]
+c = Label.new(@opts["i"])
+
+puts "--- Report for module #{@opts["i"]} ---"
+
+if @opts["C"]
+  puts "---------------"
+  puts "Output\t\tLabel"
+  puts "---------------"
+  c.output_report.each_pair do |k, v|
+    puts [k, hreadable[v]].join("\t")
+  end
+end
+
+if @opts["P"]
+  c.path_report.each_pair do |output, inpaths|
+    puts ""
+    puts "--------------------"
+    puts "Output\tInput\tLabel"
+    puts "--------------------"
+    puts output
+    inpaths.each_pair do |inp, lbl|
+      puts "\t#{inp}\t#{hreadable[lbl]}"
+    end
+  end
+end
+
+c.write_graph(@opts["O"]) if @opts["O"]

data/bin/budtimelines

@@ -10,7 +10,7 @@ include VizUtil
 
 module Depends
   state do
-    table :depends, [:rid, :lhs, :op, :rhs, :nm]
+    table :depends, [:rid, :lhs, :op, :rhs, :nm, :in_body]
   end
 end
 

data/docs/cheat.md

@@ -239,7 +239,7 @@ Finally, we output every tuple of `bc` that does *not* appear in `t`.
 * `bc.group([:col1, :col2], min(:col3))`.  *akin to min(col3) GROUP BY col1,col2*
   * exemplary aggs: `min`, `max`, `bool_and`, `bool_or`, `choose`
   * summary aggs: `sum`, `avg`, `count`
-  * structural aggs: `accum` *accumulates inputs into a Set*
+  * structural aggs: `accum`, `accum_pair` *accumulates inputs into a Set; accum_pair takes two inputs and accumulates a Set of pairs (two element arrays)*
 * `bc.argmax([:attr1], :attr2)`      *returns the bc items per attr1 that have highest attr2*
 * `bc.argmin([:attr1], :attr2)`
 * `bc.argagg(:exemplary_agg_name, [:attr1], :attr2))`.  *generalizes argmin/max: returns the bc items per attr1 that are chosen by the exemplary

data/docs/getstarted.md

@@ -128,7 +128,7 @@ Now that we've seen a bit of Bloom, we're ready to write our first interesting s
 
 Even though we're getting ahead of ourselves, let's have a peek at the Bloom statements that implement the server in `examples/chat/chat_server.rb`:
 
-    nodelist <= signup.payloads
+    nodelist <= connect { |c| [c.client, c.nick] }
     mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
 
 That's it!  There is one statement for each of the two sentences describing the behavior of the "basic idea" above.  We'll go through these two statements in more detail shortly.  But it's nice to see right away how concisely and naturally a Bloom program can fit our intuitive description of a distributed service.
@@ -139,11 +139,11 @@ Now that we've satisfied our need to peek, let's take this a bit more methodical
 
     module ChatProtocol
       state do
+        channel :connect, [:@addr, :client] => [:nick]
         channel :mcast
-        channel :connect
       end
       
-    DEFAULT_ADDR = "localhost:12345"
+      DEFAULT_ADDR = "localhost:12345"
     end
 
 This defines a [Ruby mixin module](http://www.ruby-doc.org/docs/ProgrammingRuby/html/tut_modules.html) called `ChatProtocol` that has a couple special Bloom features:
@@ -167,7 +167,7 @@ Given this protocol (and the Ruby constant at the bottom), we're now ready to ex
       state { table :nodelist }
 
       bloom do
-        nodelist <= connect.payloads
+        nodelist <= connect { |c| [c.client, c.nick] }
         mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
       end
     end
@@ -190,11 +190,11 @@ With those preliminaries aside, we have our first `bloom` block, which is how Bl
 
 The first is pretty simple: 
 
-     nodelist <= connect.payloads
+     nodelist <= connect { |c| [c.client, c.nick] }
 
-This says that whenever messages arrive on the channel named "connect", their payloads (i.e. their non-address field) should be instantaneously merged into the table nodelist, which will store them persistently.  Note that nodelist has a \[key/val\] pair structure, so we expect the payloads will have that structure as well.
+This says that whenever messages arrive on the channel named "connect", the client address and user-provided nickname should be instantaneously merged into the table "nodelist", which will store them persistently.  Note that nodelist has a \[key/val\] pair structure, so it is suitable for storing pairs of (IP address, nickname).
 
-The next Bloom statement is more complex.  Remember the description in the "basic idea" at the beginning of this section: the server needs to accept inbound chat messages from clients, and forward them to other clients.  
+The next Bloom statement is more complex.  Remember the description in the "basic idea" at the beginning of this section: the server needs to accept inbound chat messages from clients and forward them to other clients.  
 
     mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
     
@@ -233,7 +233,7 @@ And here's the code:
       end
 
       bootstrap do
-        connect <~ [[@server, [ip_port, @nick]]]
+        connect <~ [[@server, ip_port, @nick]]
       end
 
       bloom do

data/examples/chat/README.md

@@ -7,3 +7,5 @@ To run the chat example, do each of the following in a different terminal:
     # ruby chat.rb bob
 
     # ruby chat.rb harvey
+
+Note that the "backports" gem should be installed.

data/examples/chat/chat.rb

@@ -1,6 +1,7 @@
 require 'rubygems'
+require 'backports'
 require 'bud'
-require 'chat_protocol'
+require_relative 'chat_protocol'
 
 class ChatClient
   include Bud
@@ -13,7 +14,7 @@ class ChatClient
   end
 
   bootstrap do
-    connect <~ [[@server, [ip_port, @nick]]]
+    connect <~ [[@server, ip_port, @nick]]
   end
 
   bloom do

data/examples/chat/chat_protocol.rb

@@ -1,7 +1,7 @@
 module ChatProtocol
   state do
+    channel :connect, [:@addr, :client] => [:nick]
     channel :mcast
-    channel :connect
   end
 
   DEFAULT_ADDR = "localhost:12345"

data/examples/chat/chat_server.rb

@@ -1,6 +1,7 @@
 require 'rubygems'
+require 'backports'
 require 'bud'
-require 'chat_protocol'
+require_relative 'chat_protocol'
 
 class ChatServer
   include Bud
@@ -9,7 +10,7 @@ class ChatServer
   state { table :nodelist }
 
   bloom do
-    nodelist <= connect.payloads
+    nodelist <= connect { |c| [c.client, c.nick] }
     mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
   end
 end

data/lib/bud/aggs.rb

@@ -1,5 +1,3 @@
-require 'set'
-
 module Bud
   ######## Agg definitions
   class Agg #:nodoc: all
@@ -207,4 +205,20 @@ module Bud
   def accum(x)
     [Accum.new, x]
   end
+
+  class AccumPair < Agg #:nodoc: all
+    def init(fst, snd)
+      [[fst, snd]].to_set
+    end
+    def trans(the_state, fst, snd)
+      the_state << [fst, snd]
+      return the_state, nil
+    end
+  end
+
+  # aggregate method to be used in Bud::BudCollection.group.
+  # accumulates x, y inputs into a set of pairs (two element arrays).
+  def accum_pair(x, y)
+    [AccumPair.new, x, y]
+  end
 end

data/lib/bud/bud_meta.rb

@@ -1,6 +1,5 @@
 require 'bud/rewrite'
 
-
 class BudMeta #:nodoc: all
   def initialize(bud_instance, declarations)
     @bud_instance = bud_instance
@@ -111,11 +110,11 @@ class BudMeta #:nodoc: all
 
   def get_qual_name(pt)
     # expect to see a parse tree corresponding to a dotted name
-    #    a.b.c == s(:call, s1,  :c, (:args))
-    # where s1 == s(:call, s2,  :b, (:args))
-    # where s2 == s(:call, nil, :a, (:args))
-    tag, recv, name, args = pt
-    return nil unless tag == :call and args.length == 1
+    #    a.b.c == s(:call, s1,  :c)
+    # where s1 == s(:call, s2,  :b)
+    # where s2 == s(:call, nil, :a)
+    tag, recv, name, *args = pt
+    return nil unless tag == :call and args.empty?
 
     if recv
       qn = get_qual_name(recv)
@@ -127,23 +126,16 @@ class BudMeta #:nodoc: all
   end
 
   # Perform some basic sanity checks on the AST of a rule block. We expect a
-  # rule block to consist of a :defn, a nested :scope, and then a sequence of
+  # rule block to consist of a :defn whose body consists of a sequence of
   # statements. Each statement is a :call node. Returns nil (no error found), a
   # Sexp (containing an error), or a pair of [Sexp, error message].
   def check_rule_ast(pt)
-    # :defn format: node tag, block name, args, nested scope
-    return pt if pt.sexp_type != :defn
-    scope = pt[3]
-    return pt if scope.sexp_type != :scope
-    block = scope[1]
-
-    block.each_with_index do |n,i|
-      if i == 0
-        return pt if n != :block
-        next
-      end
+    # :defn format: node tag, block name, args, body_0, ..., body_n
+    tag, name, args, *body = pt
+    return pt if tag != :defn
 
-      next if i == 1 and n.sexp_type == :nil # a block got rewritten to an empty block
+    body.each_with_index do |n,i|
+      next if i == 0 and n == s(:nil) # a block got rewritten to an empty block
 
       # Check for a common case
       if n.sexp_type == :lasgn
@@ -157,8 +149,9 @@ class BudMeta #:nodoc: all
       # Check that LHS references a named collection
       lhs_name = get_qual_name(lhs)
       return [n, "unexpected lhs format: #{lhs}"] if lhs_name.nil?
-      unless @bud_instance.tables.has_key? lhs_name.to_sym
-        return [n, "collection does not exist: '#{lhs_name}'"]
+      unless @bud_instance.tables.has_key? lhs_name.to_sym or
+             @bud_instance.lattices.has_key? lhs_name.to_sym
+        return [n, "Collection does not exist: '#{lhs_name}'"]
       end
 
       return [n, "illegal operator: '#{op}'"] unless [:<, :<=].include? op
@@ -171,13 +164,11 @@ class BudMeta #:nodoc: all
       # XXX: We don't check for illegal superators (e.g., "<--"). That would be
       # tricky, because they are encoded as a nested unary op in the rule body.
       if op == :<
-        return n unless rhs.sexp_type == :arglist
-        body = rhs[1]
-        return n unless body.sexp_type == :call
-        op_tail = body[2]
+        return n unless rhs.sexp_type == :call
+        op_tail = rhs[2]
         return n unless [:~, :-@, :+@].include? op_tail
-        rhs_args = body[3]
-        return n if rhs_args.sexp_type != :arglist or rhs_args.length != 1
+        rhs_args = rhs[3..-1]
+        return n unless rhs_args.empty?
       end
     end
 
@@ -193,7 +184,7 @@ class BudMeta #:nodoc: all
     bud = @bud_instance.toplevel
     nodes = {}
     bud.t_depends.each do |d|
-      #t_depends [:bud_instance, :rule_id, :lhs, :op, :body] => [:nm]
+      #t_depends [:bud_instance, :rule_id, :lhs, :op, :body] => [:nm, :in_body]
       lhs = (nodes[d.lhs] ||= Node.new(d.lhs, :init, 0, [], true, false, false, false))
       lhs.in_lhs = true
       body = (nodes[d.body] ||= Node.new(d.body, :init, 0, [], false, true, false, false))

data/lib/bud/collections.rb

@@ -1,5 +1,3 @@
-require 'msgpack'
-
 $struct_classes = {}
 module Bud
   ########
@@ -59,6 +57,7 @@ module Bud
       end
 
       @key_colnums = @key_cols.map {|k| @cols.index(k)}
+      @val_colnums = val_cols.map {|k| @cols.index(k)}
 
       if @cols.empty?
         @cols = nil
@@ -195,7 +194,24 @@ module Bud
         pusher_pro.tabname = the_name
         pusher_pro
       else
-        @storage.map(&blk)
+        rv = []
+        self.each do |t|
+          t = blk.call(t)
+          rv << t unless t.nil?
+        end
+        rv
+      end
+    end
+
+    # XXX: Although we support each_with_index over Bud collections, using it is
+    # probably not a great idea: the index assigned to a given collection member
+    # is not defined by the language semantics.
+    def each_with_index(the_name=tabname, the_schema=schema, &blk)
+      if @bud_instance.wiring?
+        pusher = to_push_elem(the_name, the_schema)
+        pusher.each_with_index(the_name, the_schema, &blk)
+      else
+        super(&blk)
       end
     end
 
@@ -218,7 +234,7 @@ module Bud
         end
         elem.set_block(&f)
         toplevel.push_elems[[self.object_id, :flatten]] = elem
-        return elem
+        elem
       else
         @storage.flat_map(&blk)
       end
@@ -230,14 +246,16 @@ module Bud
         pusher = self.pro
         pusher.sort("sort#{object_id}", @bud_instance, @cols, &blk)
       else
-        @storage.sort
+        @storage.values.sort(&blk)
       end
     end
 
     def rename(the_name, the_schema=nil, &blk)
       raise unless @bud_instance.wiring?
       # a scratch with this name should have been defined during rewriting
-      raise Bud::Error, "rename failed to define a scratch named #{the_name}" unless @bud_instance.respond_to? the_name
+      unless @bud_instance.respond_to? the_name
+        raise Bud::Error, "rename failed to define a scratch named #{the_name}"
+      end
       pro(the_name, the_schema, &blk)
     end
 
@@ -377,6 +395,11 @@ module Bud
       raise Bud::KeyConstraintError, "key conflict inserting #{new.inspect} into \"#{tabname}\": existing tuple #{old.inspect}, key = #{key.inspect}"
     end
 
+    private
+    def is_lattice_val(v)
+      v.kind_of? Bud::Lattice
+    end
+
     private
     def prep_tuple(o)
       return o if o.class == @struct
@@ -391,9 +414,16 @@ module Bud
         raise Bud::TypeError, "array or struct type expected in \"#{qualified_tabname}\": #{o.inspect}"
       end
 
+      @key_colnums.each do |i|
+        next if i >= o.length
+        if is_lattice_val(o[i])
+          raise Bud::TypeError, "lattice value cannot be a key for #{qualified_tabname}: #{o[i].inspect}"
+        end
+      end
       if o.length > @structlen
         raise Bud::TypeError, "too many columns for \"#{qualified_tabname}\": #{o.inspect}"
       end
+
       return @struct.new(*o)
     end
 
@@ -420,13 +450,44 @@ module Bud
     end
 
     # Merge "tup" with key values "key" into "buf". "old" is an existing tuple
-    # with the same key columns as "tup" (if any such tuple exists).
+    # with the same key columns as "tup" (if any such tuple exists). If "old"
+    # exists and "tup" is not a duplicate, check whether the two tuples disagree
+    # on a non-key, non-lattice value; if so, raise a PK error. Otherwise,
+    # construct and return a merged tuple by using lattice merge functions.
     private
     def merge_to_buf(buf, key, tup, old)
-      if old.nil?       # no matching tuple found
+      if old.nil?               # no matching tuple found
         buf[key] = tup
-      elsif old != tup  # ignore duplicates
-        raise_pk_error(tup, old)
+        return
+      end
+      return if tup == old      # ignore duplicates
+
+      # Check for PK violation
+      @val_colnums.each do |i|
+        old_v = old[i]
+        new_v = tup[i]
+
+        unless old_v == new_v || (is_lattice_val(old_v) && is_lattice_val(new_v))
+          raise_pk_error(tup, old)
+        end
+      end
+
+      # Construct new tuple version. We discard the newly-constructed tuple if
+      # merging every lattice field doesn't yield a new value.
+      new_t = null_tuple
+      saw_change = false
+      @val_colnums.each do |i|
+        if old[i] == tup[i]
+          new_t[i] = old[i]
+        else
+          new_t[i] = old[i].merge(tup[i])
+          saw_change = true if new_t[i].reveal != old[i].reveal
+        end
+      end
+
+      if saw_change
+        @key_colnums.each {|k| new_t[k] = old[k]}
+        buf[key] = new_t
       end
     end
 
@@ -508,6 +569,12 @@ module Bud
         add_merge_target
         tbl = register_coll_expr(o)
         tbl.pro.wire_to self
+      elsif o.class <= Bud::LatticePushElement
+        add_merge_target
+        o.wire_to self
+      elsif o.class <= Bud::LatticeWrapper
+        add_merge_target
+        o.to_push_elem.wire_to self
       else
         unless o.nil?
           o = o.uniq.compact if o.respond_to?(:uniq)
@@ -557,6 +624,12 @@ module Bud
         add_merge_target
         tbl = register_coll_expr(o)
         tbl.pro.wire_to(self, :pending)
+      elsif o.class <= Bud::LatticePushElement
+        add_merge_target
+        o.wire_to(self, :pending)
+      elsif o.class <= Bud::LatticeWrapper
+        add_merge_target
+        o.to_push_elem.wire_to(self, :pending)
       else
         pending_merge(o)
       end
@@ -656,10 +729,10 @@ module Bud
     # for each distinct value of the grouping key columns, return the items in that group
     # that have the value of the exemplary aggregate +aggname+
     public
-    def argagg(aggname, gbkey_cols, collection)
+    def argagg(aggname, gbkey_cols, collection, &blk)
       elem = to_push_elem
       gbkey_cols = gbkey_cols.map{|k| canonicalize_col(k)} unless gbkey_cols.nil?
-      retval = elem.argagg(aggname, gbkey_cols, canonicalize_col(collection))
+      retval = elem.argagg(aggname, gbkey_cols, canonicalize_col(collection), &blk)
       # PushElement inherits the schema accessors from this Collection
       retval.extend @cols_access
       retval
@@ -669,37 +742,46 @@ module Bud
     # that group that have the minimum value of the attribute +col+. Note that
     # multiple tuples might be returned.
     public
-    def argmin(gbkey_cols, col)
-      argagg(:min, gbkey_cols, col)
+    def argmin(gbkey_cols, col, &blk)
+      argagg(:min, gbkey_cols, col, &blk)
     end
 
     # for each distinct value of the grouping key columns, return the items in
     # that group that have the maximum value of the attribute +col+. Note that
     # multiple tuples might be returned.
     public
-    def argmax(gbkey_cols, col)
-      argagg(:max, gbkey_cols, col)
+    def argmax(gbkey_cols, col, &blk)
+      argagg(:max, gbkey_cols, col, &blk)
     end
 
     # form a collection containing all pairs of items in +self+ and items in
     # +collection+
     public
     def *(collection)
-      elem1 = to_push_elem
-      return elem1.join(collection)
+      return to_push_elem.join(collection)
+    end
+
+    def prep_aggpairs(aggpairs)
+      aggpairs.map do |ap|
+        agg, *rest = ap
+        if rest.empty?
+          [agg]
+        else
+          [agg] + rest.map {|c| canonicalize_col(c)}
+        end
+      end
     end
 
     def group(key_cols, *aggpairs, &blk)
-      elem = to_push_elem
       key_cols = key_cols.map{|k| canonicalize_col(k)} unless key_cols.nil?
-      aggpairs = aggpairs.map{|ap| [ap[0], canonicalize_col(ap[1])].compact} unless aggpairs.nil?
-      return elem.group(key_cols, *aggpairs, &blk)
+      aggpairs = prep_aggpairs(aggpairs)
+      return to_push_elem.group(key_cols, *aggpairs, &blk)
     end
 
     def notin(collection, *preds, &blk)
       elem1 = to_push_elem
       elem2 = collection.to_push_elem
-      return elem1.notin(elem2, preds, &blk)
+      return elem1.notin(elem2, *preds, &blk)
     end
 
     def canonicalize_col(col)
@@ -725,6 +807,7 @@ module Bud
       false
     end
 
+    # tick_delta for scratches is @storage, so iterate over that instead
     public
     def each_tick_delta(&block)
       @storage.each_value(&block)
@@ -870,28 +953,45 @@ module Bud
           raise Bud::Error, "'#{t[@locspec_idx]}', channel '#{@tabname}'" if the_locspec[0].nil? or the_locspec[1].nil? or the_locspec[0] == '' or the_locspec[1] == ''
         end
         puts "channel #{qualified_tabname}.send: #{t}" if $BUD_DEBUG
-        toplevel.dsock.send_datagram([qualified_tabname.to_s, t].to_msgpack,
-                                     the_locspec[0], the_locspec[1])
+
+        # Convert the tuple into a suitable wire format. Because MsgPack cannot
+        # marshal arbitrary Ruby objects that we need to send via channels (in
+        # particular, lattice values and Class instances), we first encode such
+        # values using Marshal, and then encode the entire tuple with
+        # MsgPack. Obviously, this is gross. The wire format also includes an
+        # array of indices, indicating which fields hold Marshall'd objects.
+        marshall_indexes = []
+        wire_tuple = Array.new(t.length)
+        t.each_with_index do |f, i|
+          if [Bud::Lattice, Class].any?{|t| f.class <= t}
+            marshall_indexes << i
+            wire_tuple[i] = Marshal.dump(f)
+          else
+            wire_tuple[i] = f
+          end
+        end
+        wire_str = [qualified_tabname.to_s, wire_tuple, marshall_indexes].to_msgpack
+        toplevel.dsock.send_datagram(wire_str, the_locspec[0], the_locspec[1])
       end
       @pending.clear
     end
 
     public
     # project to the non-address fields
-    def payloads
-      return self.pro if @is_loopback
-
-      if cols.size > 2
-        # bundle up each tuple's non-locspec fields into an array
-        retval = case @locspec_idx
-          when 0 then self.pro{|t| t.values_at(1..(t.size-1))}
-          when (schema.size - 1) then self.pro{|t| t.values_at(0..(t.size-2))}
-          else self.pro{|t| t.values_at(0..(@locspec_idx-1), @locspec_idx+1..(t.size-1))}
-        end
-      else
-        # just return each tuple's non-locspec field value
-        retval = self.pro{|t| t[(@locspec_idx == 0) ? 1 : 0]}
+    def payloads(&blk)
+      return self.pro(&blk) if @is_loopback
+
+      if @payload_struct.nil?
+        payload_cols = cols.dup
+        payload_cols.delete_at(@locspec_idx)
+        @payload_struct = Struct.new(*payload_cols)
+        @payload_colnums = payload_cols.map {|k| cols.index(k)}
+      end
+
+      retval = self.pro do |t|
+        @payload_struct.new(*t.values_at(*@payload_colnums))
       end
+      retval = retval.pro(&blk) unless blk.nil?
       return retval
     end
 
@@ -903,6 +1003,12 @@ module Bud
       elsif o.class <= Proc
         tbl = register_coll_expr(o)
         tbl.pro.wire_to(self, :pending)
+      elsif o.class <= Bud::LatticePushElement
+        add_merge_target
+        o.wire_to(self, :pending)
+      elsif o.class <= Bud::LatticeWrapper
+        add_merge_target
+        o.to_push_elem.wire_to(self, :pending)
       else
         pending_merge(o)
       end
@@ -946,7 +1052,7 @@ module Bud
             port = toplevel.port
             EventMachine::schedule do
               socket = EventMachine::open_datagram_socket("127.0.0.1", 0)
-              socket.send_datagram([tabname, tup].to_msgpack, ip, port)
+              socket.send_datagram([tabname, tup, []].to_msgpack, ip, port)
             end
           end
         rescue Exception
@@ -1195,7 +1301,19 @@ module Bud
 
     public
     def each(&block)
-      @expr.call.each(&block)
+      v = @expr.call
+
+      # XXX: Gross hack. We want to support RHS expressions that do not
+      # necessarily return BudCollections (they might instead return lattice
+      # values or hashes). Since it isn't easy to distinguish between these two
+      # cases statically, instead we just always use CollExpr; at runtime, if
+      # the value doesn't look like a traditional Bloom collection, we don't try
+      # to break it up into tuples.
+      if v.class <= Array || v.class <= BudCollection
+        v.each(&block)
+      else
+        yield v
+      end
     end
 
     public