tupelo 0.20 → 0.21

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8f1def09de170ac1f78bdc5bfe99374b02566dab
-  data.tar.gz: fe906b6b7f0724a056b93de2144b5b198b77c577
+  metadata.gz: 3ab41264ead7c8233c4e48f9f568ae68fb04e843
+  data.tar.gz: 01baffe87a234f70b3f2f84de84f38f62924d152
 SHA512:
-  metadata.gz: b033a673de3a993f6cec2bdb76beca10797a37c579e6bba09c19073a1eea875e683d24cdd28ca2450378602f3ce74a8870ee14ede361ae35e5cf7e4a95fa460b
-  data.tar.gz: 43e3999fe505e7eca3c483839e6dcbde880f9cdc7791bb2b88541ce43de9e506b47cc16354d595f051456146b45ea4a78422d9419b25e1e028eaa93a51cb716c
+  metadata.gz: 6808153d5b48baf8e376a72d4179ca20851da83833f938dfa00120da1ec20fe9f4fe504b4006bd0c88f243d9a445f6856ecc70af6013d27c1f13140b5946b67a
+  data.tar.gz: e57be7395a7d81c6cde4e3ec8c21f93761e76f1dc8c7d2f70a6331f59e563c5b7e0bd903119f2f86c2e161231f5cdfc4210144bdf223327c5f158d139caf5c2a

data/README.md

@@ -113,17 +113,27 @@ Ssh is used to set up the remote processes. Additionally, with the `--tunnel` co
 Limitations
 ===========
 
-The main limitation of tupelo is that **all network communication passes through a single process**, the message sequencer. This process has minimal state and minimal computation. The state is just a counter and the network connections (no storage of tuples or other application data). The computation is just counter increment and message dispatch (no transaction execution or searches). A transaction requires just one message (possibly with many recipients) to pass through the sequencer. The message sequencer can be light and fast.
+Bottleneck
+----------
+
+The main limitation of tupelo is that, except for read-only operations, **all tuple operations pass through a single process**, the message sequencer.
+
+The sequencer has minimal state and minimal computation. The state is just a counter and the network connections (no storage of tuples or other application data). The computation is just counter increment and message dispatch (no transaction execution or searches). A transaction requires just one message (possibly with many recipients) to pass through the sequencer. The message sequencer can be light and fast.
 
 Nevertheless, this process is a bottleneck. Each message traverses two hops, to and from the sequencer. Each tupelo client must be connected to the sequencer to transact on tuples (aside from local reads).
 
 **Tupelo will always have this limitation.** It is essential to the design of the system. By accepting this cost, we get some benefits, discussed in the next section.
 
+Clients may communicate other data over side channels that do not go through the sequencer. For [example](example/socket-broker.rb), they can use the tuplespace to coordinate task assignments, data locations (perhaps external to the tuplespace), TCP hosts and ports, and other metadata, and then use direct connections for the data. The archiver, which is a special client that brings newly connected clients up to date, is another example of direct client-to-client connections.
+
+Other limitations
+-----------------
+
 The message sequencer is also a SPoF (single point of failure), but this is not inherent in the design. A future version of tupelo will have options for failover or clustering of the sequencer, perhaps based on [raft](http://raftconsensus.github.io), with a cost of increased latency and complexity. (However, redundancy and failover of *application* data and computation *is* supported by the current implementation; app data and computations are distributed among the client processes.)
 
 There are some limitations that may result from naive application of tupelo: high client memory use, high bandwidth use, high client cpu use. These resource issues can often be controlled with [subspaces](doc/subspace.md) and specialized data structures and data stores. There are several examples addressing these problems. Another approach is to use the tuplespace for low volume references to high volume data.
 
-Also, see the discussion in [transactions](doc/transactions.md) on limitations of transactions across subspaces.
+Also, see the discussion in [transactions](doc/transactions.md) on limitations of transactions across subspaces. It's likely that these limitations will soon be lifted, at the cost of increased latency (only for cross-subspace transactions).
 
 This implementation is also limited in efficiency because of its use of Ruby.
 
@@ -149,7 +159,9 @@ As noted above, the sequencer assigns an incrementing sequence number, or *tick*
 
 * relatively easy data replication: all subscribers to a subspace replicate that subspace, possibly with different storage implementations;
 
-* the current state of the tuplespace can be computed from a earlier state by replaying the transactions in sequence;
+* even though storage is distributed, the client programming model is that all tuples are in the same place at the same time; there is no need to reason about multiple clocks or clock skew;
+
+* the current state of the tuplespace can be computed from an earlier state by replaying the transactions in sequence;
 
 * the evolution of system state over time is observable, and tupelo provides the tools to do so: the `--trace` switch, the `#trace` api, and the `tspy` program.
 

data/bench/{tuplespace.rb → tuplestore.rb}

@@ -1,17 +1,17 @@
-# Benchmark the default tuplespace (which is not intended to be fast).
+# Benchmark the default tuplestore (which is not intended to be fast).
 
 module Tupelo
   class Client; end
 end
 
-require 'tupelo/client/tuplespace'
+require 'tupelo/client/tuplestore'
 require 'benchmark'
 
 N_TUPLES = 100_000
 N_DELETES = 10_000
 
 Benchmark.bm(20) do |b|
-  ts = Tupelo::Client::SimpleTuplespace.new
+  ts = Tupelo::Client::SimpleTupleStore.new
 
   b.report('insert') do
     N_TUPLES.times do |i|

data/bin/tspy

@@ -28,12 +28,12 @@ if ARGV.delete("-h") or ARGV.delete("--help")
 end
 
 require 'tupelo/app'
-require 'tupelo/archiver/tuplespace'
+require 'tupelo/archiver/tuplestore'
 
 Tupelo.application do
   # Use hash-and-count-based storage, for efficiency (this client never
   # does take or read).
-  local tuplespace: [Tupelo::Archiver::Tuplespace, zero_tolerance: 1000] do
+  local tuplestore: [Tupelo::Archiver::TupleStore, zero_tolerance: 1000] do
     trap :INT do
       exit!
     end

data/bin/tup

@@ -61,9 +61,14 @@ if ARGV.delete("-h") or ARGV.delete("--help")
       --yaml
       --json
       --msgpack   <-- default
+
+      --symbol-keys
+      --string-keys
+                  for json and msgpack, represent hash keys as symbols
+                  or strings (see doc/faq for more details)
       
       --persist-dir DIR
-                  load and save tuplespace to DIR
+                  load and save tuplestore to DIR
                     (only needs to be set on first tup invocation)
 
       --subscribe TAG,TAG,...
@@ -78,7 +83,10 @@ require 'tupelo/app'
 
 argv, tupelo_opts = Tupelo.parse_args(ARGV)
 
-pubsub = argv.delete("--pubsub") # not a standard tupelo opt
+# non-standard tupelo opts:
+pubsub = argv.delete("--pubsub")
+symbol_keys = argv.delete("--symbol-keys")
+string_keys = argv.delete("--string-keys")
 
 if i=argv.index("--subscribe") # default is to subscribe to all
   argv.delete("--subscribe")
@@ -142,9 +150,18 @@ Tupelo.application(
   end
 
   client_opts = {}
+
+  if string_keys
+    client_opts[:symbolize_keys] = false
+  end
+
+  if symbol_keys
+    client_opts[:symbolize_keys] = true
+  end
+
   if pubsub
     client_opts[:arc] = nil
-    client_opts[:tuplespace] = TupClient::NullTuplespace
+    client_opts[:tuplestore] = TupClient::NullTupleStore
   end
   
   if subscribed_tags

data/example/multi-tier/{kvspace.rb → kvstore.rb}

@@ -10,8 +10,8 @@
 # that can be represented as pairs. (See memo2.rb.)
 #
 # This store also manages meta tuples, which it keeps in an array, just like
-# the default Tuplespace class does.
-class KVSpace
+# the default TupleStore class does.
+class KVStore
   include Enumerable
 
   attr_reader :tag, :hash, :metas

data/example/multi-tier/memo2.rb

@@ -1,7 +1,7 @@
-# Better, but more complex, implementation of memo.rb. Uses a custom tuplespace
+# Better, but more complex, implementation of memo.rb. Uses a custom tuplestore
 # that is optimized for storing key-value data, rather than general tuples.
 # Also, subscribes to just the relevant subspace. Consequently, this example
-# should scale up to large memo spaces much better than memo.rb, which uses
+# should scale up to large memo stores much better than memo.rb, which uses
 # linear search.
 #
 # Depends on the sinatra, json, and http gems.
@@ -10,7 +10,7 @@ require 'json'
 
 fork do
   require 'tupelo/app'
-  require_relative 'kvspace.rb'
+  require_relative 'kvstore'
 
   Tupelo.application do
     local do
@@ -21,7 +21,7 @@ fork do
       ])
     end
 
-    child tuplespace: [KVSpace, "memo"], subscribe: ["memo"] do |client|
+    child tuplestore: [KVStore, "memo"], subscribe: ["memo"] do |client|
       require 'sinatra/base'
 
       Class.new(Sinatra::Base).class_eval do

data/example/riemann/v2/event-sql.rb

@@ -0,0 +1,56 @@
+require 'sequel'
+
+    @db = Sequel.sqlite
+    @db.create_table :events do
+      primary_key   :id # id is not significant to our app
+      text          :host, null: false ## need this ?
+      text          :service, null: false
+      text          :state
+      number        :time
+      text          :description
+      number        :metric
+      number        :ttl
+
+      index         [:host, :service]
+    end
+
+    @db.create_table :tags do
+      foreign_key   :event_id, :events
+      text          :tag
+      index         :tag
+      primary_key   [:event_id, :tag]
+    end
+
+    @db.create_table :customs do
+      foreign_key   :event_id, :events
+      text          :key
+      text          :value
+      index         :key
+      primary_key   [:event_id, :key]
+    end
+
+require 'pp'
+#@db.tables.each do |table|
+#  pp @db.schema(table)
+#end
+
+events = @db[:events]
+tags = @db[:tags]
+customs = @db[:customs]
+
+events << {host: "foo.bar", service: "httpd"}
+events << {host: "foo.bar", service: "sshd"}
+tags << {event_id: 1, tag: "red"}
+customs << {event_id: 1, key: "a", value: "1"}
+customs << {event_id: 1, key: "b", value: "2"}
+customs << {event_id: 2, key: "b", value: "3"}
+
+pp events.all
+pp tags.all
+pp customs.all
+
+full_events = events.join(:tags, :event_id => :id).join(:customs, :event_id => :events__id)
+
+p full_events
+pp full_events.all
+

data/example/riemann/v2/expirer.rb

@@ -5,7 +5,7 @@ class Tupelo::Client
   #
   # A little more complex than v1, but more efficient.
   #
-  # This version uses the rbtree to manage the space itself (in the worker
+  # This version uses the rbtree to manage the tuplestore itself (in the worker
   # thread), instead of using an rbtree to manage the scheduler (in the client
   # thread). It also uses a custom template class to perform range-based queries
   # of the rbtree key, which is not explicitly stored in the tuples. The rbree
@@ -15,6 +15,9 @@ class Tupelo::Client
   def run_expirer_v2
     loop do
       event = read subspace("event") # event with lowest time+ttl
+        # This just happens to be true, but alternately we could define
+        # OrderedEventTemplate.frist() to pass in a template that explicitly
+        # finds the lowest key.
       dt_next_expiration = event["time"] + event["ttl"] - Time.now.to_f
       begin
         transaction timeout: dt_next_expiration do

data/example/riemann/v2/hash-store.rb

@@ -1,9 +1,9 @@
-require 'tupelo/archiver/tuplespace'
+require 'tupelo/archiver/tuplestore'
 
 # Store for any kind of tuple, and faster for lookups of literal tuples,
 # rather than matching with templates or other queries.
-# See also example/multitier/kvspace.rb.
-class HashStore < Tupelo::Archiver::Tuplespace
+# See also example/multitier/kvstore.rb.
+class HashStore < Tupelo::Archiver::TupleStore
   def initialize zero_tolerance: 1000
     super
   end
@@ -14,11 +14,11 @@ class HashStore < Tupelo::Archiver::Tuplespace
       super
 
     else
-      # We added this case to Archiver::Tuplespace just so the read(..)
-      # will work correctly on tuples that are already in the space
+      # We added this case to Archiver::TupleStore just so the read(..)
+      # will work correctly on tuples that are already in the store
       # when this process starts up. After that point, incoming tuples
       # are matched directly against the CRITICAL_EVENT template without
-      # searching the space.
+      # searching the store.
 
       # We're not going to use Client#take in this client, so there's no need
       # to handle the distinct_from keyword argument.

data/example/riemann/v2/ordered-event-store.rb

@@ -37,7 +37,7 @@ end
 # other spaces/stores without modification.
 #
 # This store also manages meta tuples, which it keeps in an array,
-# just like the default Tuplespace class does. Actually, any tuple for which
+# just like the default TupleStore class does. Actually, any tuple for which
 # `tuple["time"] + tuple["ttl"]` raises an exception will go in the metas,
 # but in this example the process only subscribes to events and metas.
 #

data/example/riemann/v2/riemann.rb

@@ -27,6 +27,8 @@ require 'tupelo/app'
 require_relative '../event-subspace'
 require_relative '../producer'
 require_relative 'expirer'
+require_relative 'hash-store'
+require_relative 'sqlite-event-store'
 
 N_PRODUCERS = 3
 N_CONSUMERS = 2
@@ -34,6 +36,7 @@ N_CONSUMERS = 2
 Tupelo.application do
   local do
     define_event_subspace
+    EVENT_SPACE = client.subspace("event")
   end
 
   if USE_HTTP
@@ -53,7 +56,8 @@ Tupelo.application do
 
   N_CONSUMERS.times do |i|
     # stores events indexed by host, service
-    child subscribe: "event", passive: true do ### tuplespace: sqlite
+    child tuplestore: [SqliteEventStore, EVENT_SPACE],
+          subscribe: "event", passive: true do
       log.progname = "consumer #{i}"
       read subspace("event") do |event|
         log.info event ### need filtering, actions, etc.
@@ -62,8 +66,7 @@ Tupelo.application do
   end
   
   # critical event alerter
-  require_relative 'hash-store'
-  child tuplespace: HashStore, subscribe: "event", passive: true do
+  child tuplestore: HashStore, subscribe: "event", passive: true do
     log.progname = "alerter"
     read Tupelo::Client::CRITICAL_EVENT do |event|
       log.error event
@@ -80,7 +83,7 @@ Tupelo.application do
   end
 
   # expirer: stores current events and looks for events that can be expired.
-  child tuplespace: OrderedEventStore, subscribe: "event", passive: true do
+  child tuplestore: OrderedEventStore, subscribe: "event", passive: true do
     log.progname = "expirer"
     run_expirer_v2
   end

data/example/riemann/v2/sqlite-event-store.rb

@@ -0,0 +1,174 @@
+require 'sequel'
+
+# Hard-coded to work with tuples belonging to the "event" subspace 
+# and with the SqliteEventStore table defined below. This template is designed
+# for range queries on host, or on host and service, using the composite index.
+class EventTemplate
+  attr_reader :host, :service, :space
+
+  # host and service can be intervals or single values or nil to match any value
+  def initialize host: nil, service: nil, space: nil
+    @host = host
+    @service = service
+    @space = space
+  end
+
+  # we only need to define this method if we plan to wait for event tuples
+  # locally using this template, i.e. read(template) or take(template).
+  # Non-waiting queries (such as read_all) just use #find_in.
+  def === tuple
+    @space === tuple and
+    !@host || @host === tuple["host"] and
+    !@service || @service === tuple["service"]
+  end
+
+  # Optimized search function to find a template match that exists already in
+  # the table. For operations that wait for a match, #=== is used instead.
+  def find_in events, distinct_from: []
+    where_terms = {}
+    where_terms[:host] = @host if @host
+    where_terms[:service] = @service if @service
+
+    matches = events.
+      ### select all but id
+      where(where_terms).
+      limit(distinct_from.size + 1).all
+
+    distinct_from.each do |tuple|
+      if i=matches.index(tuple)
+        matches.delete_at i
+      end
+    end
+
+    matches.first ## get the tags and customs
+    ### convert sym to string?
+  end
+end
+
+# A tuple store that is optimized for event data. These tuples are stored in an
+# in-memory sqlite database table. Tuples that do not fit this pattern (such
+# as metatuples) are stored in an array, as in the default TupleStore class.
+class SqliteEventStore
+  include Enumerable
+
+  attr_reader :events, :metas, :space
+
+  # space should be client.subspace("event"), but really we only need
+  # `space.pot` the portable object template for deciding membership.
+  def initialize space
+    @space = space
+    clear
+  end
+  
+  def clear
+    @db = Sequel.sqlite
+    @db.create_table :events do
+      primary_key   :id # id is not significant to our app
+      text          :host, null: false ## need this ?
+      text          :service, null: false
+      text          :state
+      number        :time
+      text          :description
+      number        :metric
+      number        :ttl
+
+      index         [:host, :service]
+    end
+
+    @db.create_table :tags do
+      foreign_key   :event_id, :events
+      text          :tag
+      index         :tag
+      primary_key   [:event_id, :tag]
+    end
+
+    @db.create_table :customs do
+      foreign_key   :event_id, :events
+      text          :key
+      text          :value
+      index         :key
+      primary_key   [:event_id, :key]
+    end
+
+    @events = @db[:events]
+    @metas = []
+  end
+
+  def each
+    events.each do |row|
+      ## extra queries for tags and custom 
+      # yuck, gotta convert symbol keys to string keys:
+      tuple = row.inject({}) {|h, (k,v)| h[k.to_s] = v; h}
+      yield tuple
+    end
+    metas.each do |tuple|
+      yield tuple
+    end
+  end
+
+  def insert tuple
+    case tuple
+    when space
+      events << tuple ## minus tags and customs
+    else
+      metas << tuple
+    end
+  end
+
+  def delete_once tuple
+    case tuple
+    when space
+      # yuck, gotta convert string keys to symbol keys:
+      row = tuple.inject({}) {|h, (k,v)| h[k.to_sym] = v; h}
+      id = events.select(:id).where(row).limit(1)
+      count = events.where(id: id).delete
+
+      if count == 0
+        false
+      elsif count == 1
+        true
+      else
+        raise "internal error: primary key, id, was not unique"
+      end
+
+    else
+      if i=metas.index(tuple)
+        metas.delete_at i
+      end
+    end
+  end
+
+  def transaction inserts: [], deletes: [], tick: nil
+    deletes.each do |tuple|
+      delete_once tuple or raise "bug"
+    end
+
+    inserts.each do |tuple|
+      insert tuple.freeze ## should be deep_freeze
+    end
+  end
+
+  def find_distinct_matches_for templates
+    templates.inject([]) do |tuples, template|
+      tuples << find_match_for(template, distinct_from: tuples)
+    end
+  end
+
+  def find_match_for template, distinct_from: []
+    case template
+    when EventTemplate
+      template.find_in events, distinct_from: distinct_from
+    else
+      ## if template can match subspace
+      # Fall back to linear search, same as default tuplestore.
+      find do |tuple|
+        template === tuple and not distinct_from.any? {|t| t.equal? tuple}
+      end
+      ## else
+      ##   metas.find do |tuple|
+      ##     template === tuple and not distinct_from.any? {|t| t.equal? tuple}
+      ##   end
+      ## end
+    end
+  end
+end

data/example/sqlite/poi-client.rb

@@ -0,0 +1,11 @@
+require 'tupelo/client'
+require_relative 'poi-store'
+
+class PoiClient < Tupelo::Client
+  def initialize *args, poispace: nil, subscribe: [], **opts
+    super *args, **opts,
+      tuplestore: [PoiStore, poispace.spec],
+      subscribe: [poispace.tag] + [subscribe].flatten,
+      symbolize_keys: true # for ease of use with sequel DB interface
+  end
+end

data/example/sqlite/poi-store.rb

@@ -1,52 +1,16 @@
 require 'sequel'
-
-# Hard-coded to work with tuples belonging to the "poi" subspace 
-# and with the PoiStore table defined below.
-class PoiTemplate
-  attr_reader :lat, :lng, :poispace
-
-  # lat and lng can be intervals or single values or nil to match any value
-  def initialize lat: nil, lng: nil, poispace: nil
-    @lat = lat
-    @lng = lng
-    @poispace = poispace
-  end
-
-  # we only need to define this method if we plan to wait for poi tuples
-  # locally using this template, i.e. read(template) or take(template)
-  def === tuple
-    @poispace === tuple and
-    !@lat || @lat === tuple["lat"] and
-    !@lng || @lng === tuple["lng"]
-  end
-
-  def find_in table, distinct_from: []
-    where_terms = {}
-    where_terms[:lat] = @lat if @lat
-    where_terms[:lng] = @lng if @lng
-
-    matches = table.
-      select(:lat, :lng, :desc).
-      where(where_terms).
-      limit(distinct_from.size + 1).all
-
-    distinct_from.each do |tuple|
-      if i=matches.index(tuple)
-        matches.delete_at i
-      end
-    end
-
-    matches.first
-  end
-end
+require_relative 'poi-template'
 
 # A tuple store that is optimized for POI data. These tuples are stored in an
 # in-memory sqlite database table. Tuples that do not fit this pattern (such
-# as metatuples) are stored in an array, as in the default Tuplespace class.
+# as metatuples) are stored in an array, as in the default TupleStore class.
 class PoiStore
   include Enumerable
 
-  attr_reader :table, :metas, :poispace
+  attr_reader :table, :metas
+  
+  # Template for matching all POI tuples.
+  attr_reader :poi_template
 
   def self.define_poispace client
     client.define_subspace("poi",
@@ -54,14 +18,24 @@ class PoiStore
       lng:  Numeric,
       desc: String
     )
-    client.subspace("poi") ## is this awkward?
+    client.subspace("poi")
+      # this waits for ack of write of subspace metatuple, and then
+      # it returns the Subspace object, which contains a tag and a template
+      # spec, from which we can later construct a correct template in
+      # initialize.
   end
 
-  # poispace should be client.subspace("poi"), but really we only need
-  # poispace.pot
-  def initialize poispace
-    @poispace = poispace
+  def initialize spec, client: nil
+    @poi_template = client.worker.pot_for(spec)
+      # calling #pot_for in client means that resulting template
+      # will have keys converted as needed (in the case of this client,
+      # to symbols).
+
     clear
+
+    # To be more general, we could inspect the spec to determine which keys to
+    # use when creating and querying the table, and in the PoiTemplate class
+    # above. This example just assumes the key names are always lat, lng, desc.
   end
   
   def clear
@@ -82,8 +56,8 @@ class PoiStore
   end
 
   def each
-    table.each do |row|
-      yield "lat" => row[:lat], "lng" => row[:lng], "desc" => row[:desc]
+    table.select(:lat, :lng, :desc).each do |row|
+      yield row
     end
     metas.each do |tuple|
       yield tuple
@@ -92,7 +66,7 @@ class PoiStore
 
   def insert tuple
     case tuple
-    when poispace
+    when poi_template
       table << tuple
     else
       metas << tuple
@@ -101,12 +75,10 @@ class PoiStore
 
   def delete_once tuple
     case tuple
-    when poispace
-      id = table.select(:id).where(
-        lat:  tuple["lat"],
-        lng:  tuple["lng"],
-        desc: tuple["desc"]
-      ).limit(1)
+    when poi_template
+      id = table.select(:id).
+        where(tuple).
+        limit(1)
       count = table.where(id: id).delete
 
       if count == 0
@@ -157,4 +129,14 @@ class PoiStore
       ## end
     end
   end
+
+  def find_all_matches_for template, &bl
+    case template
+    when PoiTemplate
+      template.find_all_in table, &bl
+    else
+      # Fall back to linear search using #each and #===.
+      grep template, &bl
+    end
+  end
 end