bud 0.9.4 → 0.9.9

data/docs/getstarted.md

@@ -66,9 +66,11 @@ Second, note that our Bud program's one statement merges the values on its right
 ### Tables and Scratches ###
 Before we dive into writing server code, let's try a slightly more involved single-timestep example.  Start up rebl again, and paste in the following:
 
-    table :clouds
-    clouds <= [[1, "Cirrus"], [2, "Cumulus"]]
-    stdio <~ clouds.inspected
+``` ruby
+table :clouds
+clouds <= [[1, "Cirrus"], [2, "Cumulus"]]
+stdio <~ clouds.inspected
+```
     
 Now tick your rebl, but don't quit yet.  
 
@@ -128,8 +130,10 @@ Now that we've seen a bit of Bloom, we're ready to write our first interesting s
 
 Even though we're getting ahead of ourselves, let's have a peek at the Bloom statements that implement the server in `examples/chat/chat_server.rb`:
 
-    nodelist <= signup.payloads
-    mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
+``` ruby
+nodelist <= connect { |c| [c.client, c.nick] }
+mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
+```
 
 That's it!  There is one statement for each of the two sentences describing the behavior of the "basic idea" above.  We'll go through these two statements in more detail shortly.  But it's nice to see right away how concisely and naturally a Bloom program can fit our intuitive description of a distributed service.
 
@@ -137,14 +141,16 @@ That's it!  There is one statement for each of the two sentences describing the
 
 Now that we've satisfied our need to peek, let's take this a bit more methodically.  First we need declarations for the various Bloom collections we'll be using.  We put the declarations that are common to both client and server into file `examples/chat/chat_protocol.rb`:
 
-    module ChatProtocol
-      state do
-        channel :mcast
-        channel :connect
-      end
-      
-    DEFAULT_ADDR = "localhost:12345"
-    end
+``` ruby
+module ChatProtocol
+  state do
+    channel :connect, [:@addr, :client] => [:nick]
+    channel :mcast
+  end
+  
+  DEFAULT_ADDR = "localhost:12345"
+end
+```
 
 This defines a [Ruby mixin module](http://www.ruby-doc.org/docs/ProgrammingRuby/html/tut_modules.html) called `ChatProtocol` that has a couple special Bloom features:
 
@@ -156,33 +162,34 @@ This defines a [Ruby mixin module](http://www.ruby-doc.org/docs/ProgrammingRuby/
 
 Given this protocol (and the Ruby constant at the bottom), we're now ready to examine `examples/chat/chat_server.rb` in more detail:
 
-    require 'rubygems'
-    require 'bud'
-    require 'chat_protocol'
+``` ruby
+require 'rubygems'
+require 'bud'
+require_relative 'chat_protocol'
 
-    class ChatServer
-      include Bud
-      include ChatProtocol
+class ChatServer
+  include Bud
+  include ChatProtocol
 
-      state { table :nodelist }
-
-      bloom do
-        nodelist <= connect.payloads
-        mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
-      end
-    end
-
-    if ARGV.first
-      addr = ARGV.first
-    else
-      addr = ChatProtocol::DEFAULT_ADDR
-    end
-
-    ip, port = addr.split(":")
-    puts "Server address: #{ip}:#{port}"
-    program = ChatServer.new(:ip => ip, :port => port.to_i)
-    program.run
+  state { table :nodelist }
 
+  bloom do
+    nodelist <= connect { |c| [c.client, c.nick] }
+    mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
+  end
+end
+
+if ARGV.first
+  addr = ARGV.first
+else
+  addr = ChatProtocol::DEFAULT_ADDR
+end
+
+ip, port = addr.split(":")
+puts "Server address: #{ip}:#{port}"
+program = ChatServer.new(:ip => ip, :port => port.to_i)
+program.run_fg
+```
     
 The first few lines get the appropriate Ruby classes and modules loaded via `require`.  We then define the ChatServer class which mixes in the `Bud` module and the ChatProtocol module we looked at above.  Then we have another `state` block that declares one additional collection, the `nodelist` table.  
 
@@ -190,14 +197,18 @@ With those preliminaries aside, we have our first `bloom` block, which is how Bl
 
 The first is pretty simple: 
 
-     nodelist <= connect.payloads
+``` ruby
+nodelist <= connect { |c| [c.client, c.nick] }
+```
 
-This says that whenever messages arrive on the channel named "connect", their payloads (i.e. their non-address field) should be instantaneously merged into the table nodelist, which will store them persistently.  Note that nodelist has a \[key/val\] pair structure, so we expect the payloads will have that structure as well.
+This says that whenever messages arrive on the channel named "connect", the client address and user-provided nickname should be instantaneously merged into the table "nodelist", which will store them persistently.  Note that nodelist has a \[key/val\] pair structure, so it is suitable for storing pairs of (IP address, nickname).
 
-The next Bloom statement is more complex.  Remember the description in the "basic idea" at the beginning of this section: the server needs to accept inbound chat messages from clients, and forward them to other clients.  
+The next Bloom statement is more complex.  Remember the description in the "basic idea" at the beginning of this section: the server needs to accept inbound chat messages from clients and forward them to other clients.  
+
+``` ruby
+mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
+```
 
-    mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
-    
 The first thing to note is the lhs and operator in this statement.  We are merging items (asynchronously, of course!) into the mcast channel, where they will be sent to their eventual destination.  
 
 The rhs is our first introduction to the `*` operator of Bloom collections, and the `pairs` method after it.  You can think of the `*` operator as "all-pairs": it produces a Bloom collection containing all pairs of mcast and nodelist items.  The `pairs` method iterates through these pairs, passing them through a code block via the block arguments `m` and `n`. Finally, for each such pair the block produces an item containing the `key` attribute of the nodelist item, and the `val` attribute of the mcast item.  This is structured as a proper \[address, val\] entry to be merged back into the mcast channel.  Putting this together, this statement *multicasts inbound payloads on the mcast channel to all nodes in the chat*.
@@ -205,7 +216,7 @@ The rhs is our first introduction to the `*` operator of Bloom collections, and
 The remaining lines of plain Ruby simply instantiate and run the ChatServer class (which includes the `Bud` module) using an ip and port given on the command line (or the default from ChatProtocol.rb).
 
 #### `*`'s and Clouds ####
-You can think of out use of the `*` operator in the rhs of the second statement in a few different ways:
+You can think of our use of the `*` operator on the rhs of the second statement in a few different ways:
 
 * If you're familiar with event-loop programming, this implements an *event handler* for messages on the mcast channel: whenever an mcast message arrives, this handler performs lookups in the nodelist table to form new messages.  (It is easy to add "filters" to these handlers as arguments to `pairs`.)  The resulting messages are dispatched via the mcast channel accordingly.  This is a very common pattern in Bloom programs: handling channel messages via lookups in a table.
 
@@ -218,49 +229,51 @@ Given our understanding of the server, the client should be pretty simple.  It n
 
 And here's the code:
 
-    require 'rubygems'
-    require 'bud'
-    require 'chat_protocol'
-
-    class ChatClient
-      include Bud
-      include ChatProtocol
-
-      def initialize(nick, server, opts={})
-        @nick = nick
-        @server = server
-        super opts
-      end
-
-      bootstrap do
-        connect <~ [[@server, [ip_port, @nick]]]
-      end
-
-      bloom do
-        mcast <~ stdio do |s|
-          [@server, [ip_port, @nick, Time.new.strftime("%I:%M.%S"), s.line]]
-        end
-
-        stdio <~ mcast { |m| [pretty_print(m.val)] }
-      end
-
-      # format chat messages with timestamp on the right of the screen
-      def pretty_print(val)
-        str = val[1].to_s + ": " + (val[3].to_s || '')
-        pad = "(" + val[2].to_s + ")"
-        return str + " "*[66 - str.length,2].max + pad
-      end
-    end
+``` ruby
+require 'rubygems'
+require 'bud'
+require_relative 'chat_protocol'
+
+class ChatClient
+  include Bud
+  include ChatProtocol
+
+  def initialize(nick, server, opts={})
+    @nick = nick
+    @server = server
+    super opts
+  end
 
-    if ARGV.length == 2
-      server = ARGV[1]
-    else
-      server = ChatProtocol::DEFAULT_ADDR
+  bootstrap do
+    connect <~ [[@server, ip_port, @nick]]
+  end
+
+  bloom do
+    mcast <~ stdio do |s|
+      [@server, [ip_port, @nick, Time.new.strftime("%I:%M.%S"), s.line]]
     end
 
-    puts "Server address: #{server}"
-    program = ChatClient.new(ARGV[0], server, :read_stdin => true)
-    program.run_fg
+    stdio <~ mcast { |m| [pretty_print(m.val)] }
+  end
+
+  # format chat messages with timestamp on the right of the screen
+  def pretty_print(val)
+    str = val[1].to_s + ": " + (val[3].to_s || '')
+    pad = "(" + val[2].to_s + ")"
+    return str + " "*[66 - str.length,2].max + pad
+  end
+end
+
+if ARGV.length == 2
+  server = ARGV[1]
+else
+  server = ChatProtocol::DEFAULT_ADDR
+end
+
+puts "Server address: #{server}"
+program = ChatClient.new(ARGV[0], server, :stdin => $stdin)
+program.run_fg
+```
 
 The ChatClient class has a typical Ruby `initialize` method that sets up two local instance variables: one for this client's nickname, and another for the 'IP:port' address string for the server.  It then calls the initializer of the Bud superclass passing along a hash of options.
 
@@ -293,4 +306,4 @@ In this section we saw a number of features that we missed in our earlier single
 * **the * operator and pairs method**: the way to combine items from multiple collections.
 
 # The Big Picture and the Details #
-Now that you've seen some working Bloom code, hopefully you're ready to delve deeper.  The [README](README.md) provides links to places you can go for more information.  Have fun and [stay in touch](http://groups.google.com/group/bloom-lang)!
+Now that you've seen some working Bloom code, hopefully you're ready to delve deeper.  The [README](README.md) provides links to places you can go for more information.  Have fun and [stay in touch](http://groups.google.com/group/bloom-lang)!

data/docs/operational.md

@@ -58,7 +58,7 @@ Have a look at the following classic "transitive closure" example, which compute
 
     state do
       table :link, [:from, :to, :cost]
-      table :path, [:from, :to, :cost]
+      table :path, [:from, :to,  :cost]
     end
 
     bloom :make_paths do
@@ -67,7 +67,7 @@ Have a look at the following classic "transitive closure" example, which compute
 
       # recurse: path of length n+1 made by a link to a path of length n
       path <= (link*path).pairs(:to => :from) do |l,p|
-        [l.from, p.to, l.cost+p.cost]
+        [l.from, p.to, l.cost + p.cost]
       end
     end
     
@@ -97,4 +97,4 @@ Note that it is possible to write a program in Bloom that is *unstratifiable*: t
 
     glass <= one_item {|t| ['full'] if glass.empty? }
 
-Consider the case where we start out with glass being empty.  Then we know the fact `glass.empty?`, and the bloom statement says that `(glass.empty? => not glass.empty?)` which is equivalent to `(glass.empty? and not glass.empty?)` which is a contradiction.  The Bud runtime detects cycles through non-monotonicity for you automatically when you instantiate your class.
+Consider the case where we start out with glass being empty.  Then we know the fact `glass.empty?`, and the bloom statement says that `(glass.empty? => not glass.empty?)` which is equivalent to `(glass.empty? and not glass.empty?)` which is a contradiction.  The Bud runtime detects cycles through non-monotonicity for you automatically when you instantiate your class.

data/examples/basics/paths.rb

@@ -36,7 +36,7 @@ program = ShortestPaths.new
 
 # populate our little example.  we put two links between 'a' and 'b'
 # to see whether our shortest-paths code does the right thing.
-program.link <= [['a', 'b', 1],
+program.link <+ [['a', 'b', 1],
                  ['a', 'b', 4],
                  ['b', 'c', 1],
                  ['c', 'd', 1],
@@ -48,6 +48,6 @@ program.shortest.to_a.sort.each {|t| puts t.inspect}
 puts "----"
 
 # now lets add an extra link and recompute
-program.link << ['e', 'f', 1]
+program.link <+ [['e', 'f', 1]]
 program.tick
 program.shortest.to_a.sort.each {|t| puts t.inspect}

data/examples/chat/README.md

@@ -7,3 +7,5 @@ To run the chat example, do each of the following in a different terminal:
     # ruby chat.rb bob
 
     # ruby chat.rb harvey
+
+Note that the "backports" gem should be installed.

data/examples/chat/chat.rb

@@ -1,6 +1,7 @@
 require 'rubygems'
+require 'backports'
 require 'bud'
-require 'chat_protocol'
+require_relative 'chat_protocol'
 
 class ChatClient
   include Bud
@@ -13,7 +14,7 @@ class ChatClient
   end
 
   bootstrap do
-    connect <~ [[@server, [ip_port, @nick]]]
+    connect <~ [[@server, ip_port, @nick]]
   end
 
   bloom do

data/examples/chat/chat_protocol.rb

@@ -1,8 +1,8 @@
 module ChatProtocol
   state do
+    channel :connect, [:@addr, :client] => [:nick]
     channel :mcast
-    channel :connect
   end
 
-  DEFAULT_ADDR = "localhost:12345"
+  DEFAULT_ADDR = "127.0.0.1:12345"
 end

data/examples/chat/chat_server.rb

@@ -1,6 +1,7 @@
 require 'rubygems'
+require 'backports'
 require 'bud'
-require 'chat_protocol'
+require_relative 'chat_protocol'
 
 class ChatServer
   include Bud
@@ -9,7 +10,7 @@ class ChatServer
   state { table :nodelist }
 
   bloom do
-    nodelist <= connect.payloads
+    nodelist <= connect { |c| [c.client, c.nick] }
     mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }
   end
 end

data/lib/bud.rb

@@ -1,16 +1,27 @@
 require 'rubygems'
+gem 'ruby2ruby', '>= 2.0.1'
+gem 'ruby_parser', '>= 3.0.2'
+
 require 'eventmachine'
 require 'msgpack'
+require 'ruby2ruby'
+require 'ruby_parser'
+require 'set'
 require 'socket'
 require 'superators19'
 require 'thread'
-require 'bud/errors'
 
+require 'bud/errors'
 require 'bud/monkeypatch'
 
 require 'bud/aggs'
 require 'bud/bud_meta'
 require 'bud/collections'
+require 'bud/executor/elements.rb'
+require 'bud/executor/group.rb'
+require 'bud/executor/join.rb'
+require 'bud/lattice-core'
+require 'bud/lattice-lib'
 require 'bud/metrics'
 require 'bud/rtrace'
 require 'bud/server'
@@ -19,10 +30,6 @@ require 'bud/storage/dbm'
 require 'bud/storage/zookeeper'
 require 'bud/viz'
 
-require 'bud/executor/elements.rb'
-require 'bud/executor/group.rb'
-require 'bud/executor/join.rb'
-
 ILLEGAL_INSTANCE_ID = -1
 SIGNAL_CHECK_PERIOD = 0.2
 
@@ -61,12 +68,13 @@ $bud_instances = {}        # Map from instance id => Bud instance
 # :main: Bud
 module Bud
   attr_reader :budtime, :inbound, :options, :meta_parser, :viz, :rtracer, :dsock
-  attr_reader :tables, :builtin_tables, :channels, :zk_tables, :dbm_tables, :app_tables
+  attr_reader :tables, :builtin_tables, :channels, :zk_tables, :dbm_tables, :app_tables, :lattices
   attr_reader :push_sources, :push_elems, :push_joins, :scanners, :merge_targets
-  attr_reader :this_stratum, :this_rule, :rule_orig_src, :done_bootstrap
+  attr_reader :this_stratum, :this_rule_context, :done_bootstrap
+  attr_reader :inside_tick
   attr_accessor :stratified_rules
   attr_accessor :metrics, :periodics
-  attr_accessor :this_rule_context, :qualified_name
+  attr_accessor :qualified_name
   attr_reader :running_async
 
   # options to the Bud runtime are passed in a hash, with the following keys
@@ -101,14 +109,12 @@ module Bud
   #   * <tt>:dbm_dir</tt> filesystem directory to hold DBM-backed collections
   #   * <tt>:dbm_truncate</tt> if true, DBM-backed collections are opened with +OTRUNC+
   def initialize(options={})
-    # capture the binding for a subsequent 'eval'. This ensures that local
-    # variable names introduced later in this method don't interfere with
-    # table names used in the eval block.
     options[:dump_rewrite] ||= ENV["BUD_DUMP_REWRITE"].to_i > 0
     options[:dump_ast]     ||= ENV["BUD_DUMP_AST"].to_i > 0
     options[:print_wiring] ||= ENV["BUD_PRINT_WIRING"].to_i > 0
     @qualified_name = ""
     @tables = {}
+    @lattices = {}
     @channels = {}
     @dbm_tables = {}
     @zk_tables = {}
@@ -130,7 +136,8 @@ module Bud
     @instance_id = ILLEGAL_INSTANCE_ID # Assigned when we start running
     @metrics = {}
     @endtime = nil
-    @this_stratum = 0
+    @this_stratum = -1
+    @this_rule_id = -1
     @push_sorted_elems = nil
     @running_async = false
     @bud_started = false
@@ -144,23 +151,22 @@ module Bud
     # NB: If using an ephemeral port (specified by port = 0), the actual port
     # number won't be known until we start EM
 
+    load_lattice_defs
     builtin_state
     resolve_imports
     call_state_methods
 
-    @declarations = self.class.instance_methods.select {|m| m =~ /^__bloom__.+$/}.map {|m| m.to_s}
-
     @viz = VizOnline.new(self) if @options[:trace]
     @rtracer = RTrace.new(self) if @options[:rtrace]
 
     do_rewrite
     if toplevel == self
       # initialize per-stratum state
-      num_strata = @stratified_rules.length
-      @scanners = num_strata.times.map{{}}
-      @push_sources = num_strata.times.map{{}}
-      @push_joins = num_strata.times.map{[]}
-      @merge_targets = num_strata.times.map{Set.new}
+      @num_strata = @stratified_rules.length
+      @scanners = @num_strata.times.map{{}}
+      @push_sources = @num_strata.times.map{{}}
+      @push_joins = @num_strata.times.map{[]}
+      @merge_targets = @num_strata.times.map{Set.new}
     end
   end
 
@@ -248,16 +254,21 @@ module Bud
           tables[qname.to_sym] = t
         end
       end
+      mod_inst.lattices.each_pair do |name, t|
+        qname = "#{local_name}.#{name}".to_sym
+        raise Bud::Error if lattices.has_key? qname
+        lattices[qname] = t
+      end
       mod_inst.t_rules.each do |imp_rule|
         qname = "#{local_name}.#{imp_rule.lhs}"
         self.t_rules << [imp_rule.bud_obj, imp_rule.rule_id, qname, imp_rule.op,
-                         imp_rule.src, imp_rule.orig_src, imp_rule.nm_funcs_called]
+                         imp_rule.src, imp_rule.orig_src, imp_rule.unsafe_funcs_called]
       end
       mod_inst.t_depends.each do |imp_dep|
         qlname = "#{local_name}.#{imp_dep.lhs}"
         qrname = "#{local_name}.#{imp_dep.body}"
         self.t_depends << [imp_dep.bud_obj, imp_dep.rule_id, qlname,
-                           imp_dep.op, qrname, imp_dep.nm]
+                           imp_dep.op, qrname, imp_dep.nm, imp_dep.in_body]
       end
       mod_inst.t_provides.each do |imp_pro|
         qintname = "#{local_name}.#{imp_pro.interface}"
@@ -304,14 +315,13 @@ module Bud
   end
 
   def do_wiring
-    @num_strata = @stratified_rules.length
-
     @stratified_rules.each_with_index { |rules, stratum| eval_rules(rules, stratum) }
 
     # Prepare list of tables that will be actively used at run time. First, all
-    # the user-defined ones.  We start @app_tables off as a set, then convert to
-    # an array later.
+    # the user-defined tables and lattices.  We start @app_tables off as a set,
+    # then convert to an array later.
     @app_tables = (@tables.keys - @builtin_tables.keys).map {|t| @tables[t]}.to_set
+    @app_tables.merge(@lattices.values)
 
     # Check scan and merge_targets to see if any builtin_tables need to be added as well.
     @scanners.each do |scs|
@@ -331,11 +341,11 @@ module Bud
       seen = Set.new(working)
       sorted_elems = [] # sorted elements in this stratum
       while not working.empty?
-        sorted_elems += working
+        sorted_elems.concat(working)
         wired_to = []
         working.each do |e|
           e.wirings.each do |out|
-            if (out.class <= PushElement and not seen.member?(out))
+            if ((out.class <= PushElement || out.class <= LatticePushElement) and not seen.member?(out))
               seen << out
               wired_to << out
             end
@@ -352,6 +362,24 @@ module Bud
       end
     end
 
+    # We create "orphan" scanners for collections that don't appear on the RHS
+    # of any rules, but do appear on the LHS of at least one rule. These
+    # scanners aren't needed to compute the fixpoint, but they are used as part
+    # of rescan/invalidation (e.g., if an orphaned collection receives a manual
+    # deletion operation, we need to arrange for the collection to be
+    # re-filled).
+    @orphan_scanners = []       # Pairs of [scanner, stratum]
+    @app_tables.each do |t|
+      next unless t.class <= Bud::BudCollection         # skip lattice wrappers
+      next if t.scanner_cnt > 0
+
+      stratum = collection_stratum(t.qualified_tabname.to_s)
+      # if the collection also doesn't appear on any LHSs, skip it
+      next if stratum.nil?
+      @orphan_scanners << [Bud::ScannerElement.new(t.tabname, self, t, t.schema),
+                           stratum]
+    end
+
     # Sanity check
     @push_sorted_elems.each do |stratum_elems|
       stratum_elems.each {|se| se.check_wiring}
@@ -417,19 +445,17 @@ module Bud
   #
   # scanner[stratum].rescan_set = Similar to above.
   def prepare_invalidation_scheme
-    num_strata = @push_sorted_elems.size
     if $BUD_SAFE
-      @app_tables = @tables.values # No tables excluded
+      @app_tables = @tables.values + @lattices.values # No collections excluded
 
       rescan = Set.new
       invalidate = @app_tables.select {|t| t.class <= BudScratch}.to_set
-      num_strata.times do |stratum|
+      @num_strata.times do |stratum|
         @push_sorted_elems[stratum].each do |elem|
           invalidate << elem
           rescan << elem
         end
       end
-      #prune_rescan_invalidate(rescan, invalidate)
       @default_rescan = rescan.to_a
       @default_invalidate = invalidate.to_a
       @reset_list = [] # Nothing to reset at end of tick. It'll be overwritten anyway
@@ -438,15 +464,19 @@ module Bud
 
     # By default, all tables are considered sources unless they appear on the
     # lhs.  We only consider non-temporal rules because invalidation is only
-    # about this tick.  Also, we track (in nm_targets) those tables that are the
-    # targets of user-defined code blocks that call non-monotonic functions
-    # (such as budtime). Elements that feed these tables are forced to rescan
-    # their contents, and thus forced to re-execute these code blocks.
-    nm_targets = Set.new
+    # about this tick.  Also, we track (in unsafe_targets) those tables that are
+    # the targets of user-defined code blocks that call "unsafe" functions that
+    # produce a different value in every tick (e.g., budtime). Elements that
+    # feed these tables are forced to rescan their contents, and thus forced to
+    # re-execute these code blocks.
+    unsafe_targets = Set.new
     t_rules.each do |rule|
       lhs = rule.lhs.to_sym
-      @tables[lhs].is_source = false if rule.op == "<="
-      nm_targets << lhs if rule.nm_funcs_called
+      if rule.op == "<="
+        # Note that lattices cannot be sources
+        @tables[lhs].is_source = false if @tables.has_key? lhs
+      end
+      unsafe_targets << lhs if rule.unsafe_funcs_called
     end
 
     # Compute a set of tables and elements that should be explicitly told to
@@ -455,54 +485,102 @@ module Bud
     invalidate = @app_tables.select {|t| t.invalidate_at_tick}.to_set
     rescan = Set.new
 
-    num_strata.times do |stratum|
+    @num_strata.times do |stratum|
       @push_sorted_elems[stratum].each do |elem|
         rescan << elem if elem.rescan_at_tick
 
-        if elem.outputs.any?{|tab| not(tab.class <= PushElement) and nm_targets.member? tab.qualified_tabname.to_sym }
+        if elem.outputs.any?{|tab| not(tab.class <= PushElement) and not(tab.class <= LatticePushElement) and unsafe_targets.member? tab.qualified_tabname.to_sym }
           rescan.merge(elem.wired_by)
         end
       end
       rescan_invalidate_tc(stratum, rescan, invalidate)
     end
 
-    prune_rescan_invalidate(rescan, invalidate)
-    # transitive closure
     @default_rescan = rescan.to_a
     @default_invalidate = invalidate.to_a
 
-    puts "Default rescan: #{rescan.inspect}" if $BUD_DEBUG
-    puts "Default inval: #{invalidate.inspect}" if $BUD_DEBUG
+    if $BUD_DEBUG
+      puts "Default rescan: #{rescan.inspect}"
+      puts "Default inval: #{invalidate.inspect}"
+      puts "Unsafe targets: #{unsafe_targets.inspect}"
+    end
 
-    # Now compute for each table that is to be scanned, the set of dependent
-    # tables and elements that will be invalidated if that table were to be
-    # invalidated at run time.
+    # For each collection that is to be scanned, compute the set of dependent
+    # tables and elements that will need invalidation and/or rescan if that
+    # table were to be invalidated at runtime.
     dflt_rescan = rescan
     dflt_invalidate = invalidate
     to_reset = rescan + invalidate
-    num_strata.times do |stratum|
-      @scanners[stratum].each_value do |scanner|
-        # If it is going to be always invalidated, it doesn't need further
-        # examination
-        next if dflt_rescan.member? scanner
-
-        rescan = dflt_rescan + [scanner]  # add scanner to scan set
-        invalidate = dflt_invalidate.clone
-        rescan_invalidate_tc(stratum, rescan, invalidate)
-        prune_rescan_invalidate(rescan, invalidate)
-        to_reset.merge(rescan)
-        to_reset.merge(invalidate)
-        # Give the diffs (from default) to scanner; these are elements that are
-        # dependent on this scanner
-        diffscan = (rescan - dflt_rescan).find_all {|elem| elem.class <= PushElement}
-        scanner.invalidate_at_tick(diffscan, (invalidate - dflt_invalidate).to_a)
-      end
+    each_scanner do |scanner, stratum|
+      # If it is going to be always invalidated, it doesn't need further
+      # examination. Lattice scanners also don't get invalidated.
+      next if dflt_rescan.member? scanner
+      next if scanner.class <= LatticeScanner
+
+      rescan = dflt_rescan.clone
+      invalidate = dflt_invalidate + [scanner.collection]
+      rescan_invalidate_tc(stratum, rescan, invalidate)
+      prune_rescan_set(rescan)
+
+      # Make sure we reset the rescan/invalidate flag for this scanner at
+      # end-of-tick, but we can remove the scanner from its own
+      # rescan_set/inval_set.
+      to_reset.merge(rescan)
+      to_reset.merge(invalidate)
+      rescan.delete(scanner)
+      invalidate.delete(scanner.collection)
+
+      # Give the diffs (from default) to scanner; these are elements that are
+      # dependent on this scanner
+      diffscan = (rescan - dflt_rescan).find_all {|elem| elem.class <= PushElement}
+      scanner.invalidate_at_tick(diffscan, (invalidate - dflt_invalidate).to_a)
     end
     @reset_list = to_reset.to_a
+
+    # For each lattice, find the collections that should be rescanned when there
+    # is a new delta for the lattice. That is, if we have a rule like:
+    # "t2 <= t1 {|t| [t.key, lat_foo]}", whenever there is a delta on lat_foo we
+    # should rescan t1 (to produce tuples with the updated lat_foo value).
+    #
+    # TODO:
+    # (1) if t1 is fed by rules r1 and r2 but only r1 references lattice x,
+    #     don't trigger rescan of r2 on deltas for x (hard)
+    t_depends.each do |dep|
+      src, target_name = dep.body.to_sym, dep.lhs.to_sym
+      if @lattices.has_key? src and dep.in_body
+        src_lat = @lattices[src]
+        if @tables.has_key? target_name
+          target = @tables[target_name]
+        else
+          target = @lattices[target_name]
+        end
+
+        # Conservatively, we rescan all the elements that feed the lhs (target)
+        # collection via positive (non-deletion) rules; we then also need to
+        # potentially rescan ancestors of those elements as well (e.g., setting
+        # a stateless PushElement to rescan does nothing; we want to tell its
+        # ancestor ScannerElement to rescan).
+        #
+        # XXX: do we need to consider all transitively reachable nodes for
+        # rescan?
+        lat_rescan = target.positive_predecessors.to_set
+        lat_inval = Set.new
+        target.positive_predecessors.each do |e|
+          e.add_rescan_invalidate(lat_rescan, lat_inval)
+        end
+        src_lat.rescan_on_delta.merge(lat_rescan)
+      end
+    end
   end
 
-  # given rescan, invalidate sets, compute transitive closure
+  # Given rescan, invalidate sets, compute transitive closure
   def rescan_invalidate_tc(stratum, rescan, invalidate)
+    # XXX: hack. If there's nothing in the given stratum, don't do
+    # anything. This can arise if we have an orphan scanner whose input is a
+    # non-monotonic operator; the stratum(LHS) = stratum(RHS) + 1, but there's
+    # nothing else in stratum(LHS).
+    return if @push_sorted_elems[stratum].nil?
+
     rescan_len = rescan.size
     invalidate_len = invalidate.size
     while true
@@ -515,12 +593,24 @@ module Bud
     end
   end
 
-  def prune_rescan_invalidate(rescan, invalidate)
+  def prune_rescan_set(rescan)
     rescan.delete_if {|e| e.rescan_at_tick}
   end
 
+  def each_scanner
+    @num_strata.times do |stratum|
+      @scanners[stratum].each_value do |scanner|
+        yield scanner, stratum
+      end
+    end
+
+    @orphan_scanners.each do |scanner,stratum|
+      yield scanner, stratum
+    end
+  end
+
   def do_rewrite
-    @meta_parser = BudMeta.new(self, @declarations)
+    @meta_parser = BudMeta.new(self)
     @stratified_rules = @meta_parser.meta_rewrite
   end
 
@@ -641,16 +731,30 @@ module Bud
   # method blocks until Bud has been shutdown. If +stop_em+ is true, the
   # EventMachine event loop is also shutdown; this will interfere with the
   # execution of any other Bud instances in the same process (as well as
-  # anything else that happens to use EventMachine).
+  # anything else that happens to use EventMachine). We always shutdown the EM
+  # loop if there are no more running Bud instances (this does interfere with
+  # other EM-using apps, but it is necessary).
   def stop(stop_em=false, do_shutdown_cb=true)
     schedule_and_wait do
       do_shutdown(do_shutdown_cb)
     end
 
+    # If we're shutting down the last active Bud instance, shutdown the EM event
+    # loop as well. This is probably good practice in general, but it also
+    # prevents weird EM behavior -- it seems as though EM::ConnectionNotBound
+    # exceptions can be raised if the EM event loop is left running and
+    # subsequent events arrive.
+    $signal_lock.synchronize {
+      stop_em = true if $bud_instances.empty? and EventMachine::reactor_running?
+    }
+
     if stop_em
       Bud.stop_em_loop
-      EventMachine::reactor_thread.join
+      unless Thread.current == EventMachine::reactor_thread
+        EventMachine::reactor_thread.join
+      end
     end
+
     report_metrics if options[:metrics]
   end
   alias :stop_bg :stop
@@ -962,10 +1066,42 @@ module Bud
     end
     bootstrap
 
-    @tables.each_value {|t| t.bootstrap} if toplevel == self
+    if toplevel == self
+      @tables.each_value {|t| t.bootstrap}
+      @lattices.each_value {|l| l.bootstrap}
+    end
     @done_bootstrap = true
   end
 
+  def do_invalidate_rescan
+    @default_rescan.each {|elem| elem.rescan = true}
+    @default_invalidate.each {|elem|
+      elem.invalidated = true
+      # Call tick on tables here itself. The rest below
+      elem.invalidate_cache unless elem.class <= PushElement
+    }
+
+    # The following loop invalidates additional (non-default) elements and
+    # tables that depend on the run-time invalidation state of a table.  Loop
+    # once to set the flags.
+    each_scanner do |scanner, stratum|
+      if scanner.rescan
+        scanner.rescan_set.each {|e| e.rescan = true}
+        scanner.invalidate_set.each {|e|
+          e.invalidated = true
+          e.invalidate_cache unless e.class <= PushElement
+        }
+      end
+    end
+
+    # Loop a second time to actually call invalidate_cache.  We can't merge this
+    # with the loops above because some versions of invalidate_cache (e.g.,
+    # join) depend on the rescan state of other elements.
+    @num_strata.times do |stratum|
+      @push_sorted_elems[stratum].each {|e| e.invalidate_cache if e.invalidated}
+    end
+  end
+
   # One timestep of Bloom execution. This MUST be invoked from the EventMachine
   # thread; it is not intended to be called directly by client code.
   def tick_internal
@@ -985,32 +1121,7 @@ module Bud
       else
         # inform tables and elements about beginning of tick.
         @app_tables.each {|t| t.tick}
-        @default_rescan.each {|elem| elem.rescan = true}
-        @default_invalidate.each {|elem|
-          elem.invalidated = true
-          # Call tick on tables here itself. The rest below
-          elem.invalidate_cache unless elem.class <= PushElement
-        }
-
-        num_strata = @push_sorted_elems.size
-        # The following loop invalidates additional (non-default) elements and
-        # tables that depend on the run-time invalidation state of a table.
-        # Loop once to set the flags.
-        num_strata.times do |stratum|
-          @scanners[stratum].each_value do |scanner|
-            if scanner.rescan
-              scanner.rescan_set.each {|e| e.rescan = true}
-              scanner.invalidate_set.each {|e|
-                e.invalidated = true
-                e.invalidate_cache unless e.class <= PushElement
-              }
-            end
-          end
-        end
-        # Loop a second time to actually call invalidate_cache
-        num_strata.times do |stratum|
-          @push_sorted_elems[stratum].each {|e| e.invalidate_cache if e.invalidated}
-        end
+        do_invalidate_rescan
       end
 
       receive_inbound
@@ -1074,14 +1185,14 @@ module Bud
   end
 
   # Return the stratum number of the given collection.
-  # NB: if a collection is not referenced by any rules, it is not currently
-  # assigned to a strata.
+  # NB: if a collection does not appear on the lhs or rhs of any rules, it is
+  # not currently assigned to a strata.
   def collection_stratum(collection)
     t_stratum.each do |t|
       return t.stratum if t.predicate == collection
     end
 
-    raise Bud::Error, "no such collection: #{collection}"
+    return nil
   end
 
   private
@@ -1099,25 +1210,26 @@ module Bud
     @periodics = table :periodics_tbl, [:pername] => [:period]
 
     # for BUD reflection
-    table :t_rules, [:bud_obj, :rule_id] => [:lhs, :op, :src, :orig_src, :nm_funcs_called]
-    table :t_depends, [:bud_obj, :rule_id, :lhs, :op, :body] => [:nm]
+    table :t_cycle, [:predicate, :via, :neg, :temporal]
+    table :t_depends, [:bud_obj, :rule_id, :lhs, :op, :body] => [:nm, :in_body]
     table :t_provides, [:interface] => [:input]
-    table :t_underspecified, t_provides.schema
+    table :t_rule_stratum, [:bud_obj, :rule_id] => [:stratum]
+    table :t_rules, [:bud_obj, :rule_id] => [:lhs, :op, :src, :orig_src, :unsafe_funcs_called]
     table :t_stratum, [:predicate] => [:stratum]
-    table :t_cycle, [:predicate, :via, :neg, :temporal]
     table :t_table_info, [:tab_name, :tab_type]
     table :t_table_schema, [:tab_name, :col_name, :ord, :loc]
+    table :t_underspecified, t_provides.schema
 
     # Identify builtin tables as such
     @builtin_tables = @tables.clone if toplevel
   end
 
-  # Handle any inbound tuples off the wire. Received messages are placed
-  # directly into the storage of the appropriate local channel. The inbound
-  # queue is cleared at the end of the tick.
+  # Handle external inputs: channels, terminals, and periodics. Received
+  # messages are placed directly into the storage of the appropriate local
+  # collection. The inbound queue is cleared at the end of the tick.
   def receive_inbound
     @inbound.each do |tbl_name, msg_buf|
-      puts "channel #{tbl_name} rcv:  #{msg_buf}" if $BUD_DEBUG
+      puts "recv via #{tbl_name}: #{msg_buf}" if $BUD_DEBUG
       msg_buf.each do |b|
         tables[tbl_name] << b
       end
@@ -1142,17 +1254,20 @@ module Bud
     # of PushElements
     @this_stratum = strat_num
     rules.each_with_index do |rule, i|
-      @this_rule_context = rule.bud_obj # user-supplied code blocks will be evaluated in this context at run-time
+      # user-supplied code blocks will be evaluated in this context at run-time
+      @this_rule_context = rule.bud_obj
       begin
         eval_rule(rule.bud_obj, rule.src)
       rescue Exception => e
-        err_msg = "** Exception while wiring rule: #{rule.src}\n ****** #{e}"
+        err_msg = "** Exception while wiring rule: #{rule.orig_src}\n ****** #{e}"
         # Create a new exception for accomodating err_msg, but reuse original backtrace
         new_e = (e.class <= Bud::Error) ? e.class.new(err_msg) : Bud::Error.new(err_msg)
         new_e.set_backtrace(e.backtrace)
         raise new_e
       end
     end
+    @this_rule_context = nil
+    @this_stratum = -1
   end
 
   ######## ids and timers
@@ -1183,10 +1298,10 @@ module Bud
         EventMachine::release_machine
         EventMachine::instance_variable_set('@reactor_running', false)
       end
+
       # Shutdown all the Bud instances inherited from the parent process, but
       # don't invoke their shutdown callbacks
       Bud.shutdown_all_instances(false)
-
       $got_shutdown_signal = false
       $signal_handler_setup = false
 
@@ -1206,16 +1321,16 @@ module Bud
   end
 
   # Signal handling. If multiple Bud instances are running inside a single
-  # process, we want a SIGINT or SIGTERM signal to cleanly shutdown all of them.
+  # process, we want a SIGINT or SIGTERM signal to cleanly shutdown all of
+  # them. Note that we don't try to do any significant work in the signal
+  # handlers themselves: we just set a flag that is checked by a periodic timer.
   def self.init_signal_handlers(b)
     $signal_lock.synchronize {
-      # If we setup signal handlers and then fork a new process, we want to
-      # reinitialize the signal handler in the child process.
+      # Initialize or re-initialize signal handlers if necessary.
       unless b.options[:signal_handling] == :none || $signal_handler_setup
         EventMachine::PeriodicTimer.new(SIGNAL_CHECK_PERIOD) do
           if $got_shutdown_signal
             Bud.shutdown_all_instances
-            Bud.stop_em_loop
             $got_shutdown_signal = false
           end
         end