tupelo 0.19 → 0.20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a8a67711763af58aa46979622519450ba198e603
-  data.tar.gz: fc399bd00b0b69ec5938a0d3ffae68155c186ae7
+  metadata.gz: 8f1def09de170ac1f78bdc5bfe99374b02566dab
+  data.tar.gz: fe906b6b7f0724a056b93de2144b5b198b77c577
 SHA512:
-  metadata.gz: 0f3570788a47744267fc361e25bd50b56bf327177764fb428317cb51ed1870c7e86acc8df53ef28d60278f9c902057b2a5275f86eba29189ea80f28259ddfc9f
-  data.tar.gz: 85a3be5a56f49d9518c2ed1409d992654e3f36f80847b441d7f4d84bf29afba541c11164c4052f97e208916e9a2a4cbf57aeef04736207160d4c6b4cd2d69e25
+  metadata.gz: b033a673de3a993f6cec2bdb76beca10797a37c579e6bba09c19073a1eea875e683d24cdd28ca2450378602f3ce74a8870ee14ede361ae35e5cf7e4a95fa460b
+  data.tar.gz: 43e3999fe505e7eca3c483839e6dcbde880f9cdc7791bb2b88541ce43de9e506b47cc16354d595f051456146b45ea4a78422d9419b25e1e028eaa93a51cb716c

data/README.md

@@ -1,35 +1,49 @@
 Tupelo
 ==
 
-A tuplespace that is fast, scalable, and language agnostic. Tupelo is designed for distribution of both computation and storage (disk and memory), in a unified language that has both transactional and tuple-operation (read/write/take) semantics.
+Tupelo is a language-agnostic tuplespace for coordination of distributed programs. It is designed for distribution of both computation and storage, on disk and in memory, with pluggable storage adapters. Its programming model is small and semantically transparent: there are tuples (built from arrays, hashes, and scalars), a few operations on tuples (read, write, take), and transactions composed of these operations. This data-centric model, unlike RPC and most forms of messaging, decouples application endpoints from each other, not only in space and time, but also in referential structure: processes refer to data rather than to other processes.
+
+Tupelo is inspired by Masatoshi Seki's Rinda in the Ruby standard library, which in turn is based on David Gelernter's Linda. The programming models of Tupelo and Rinda are similar, except for the lack of transactions in Rinda. However, the implementations of the two are nearly opposite in architectural approach.
+
+This repository contains the reference implementation in Ruby, with documentation, tests, benchmarks, and examples. Implementations in other languages must communicate with this one.
 
-This is the reference implementation in ruby. It should be able to communicate with implementations in other languages.
 
 Documentation
 ============
 
+Introductory
+------------
 * [Tutorial](doc/tutorial.md)
+* [Examples](example)
 * [FAQ](doc/faq.md)
-* [Comparisons](doc/compare.md)
+
+In Depth
+--------
 * [Transactions](doc/transactions.md)
 * [Replication](doc/replication.md)
 * [Subspaces](doc/subspace.md)
 * [Causality](doc/causality.md)
 * [Concurrency](doc/concurrency.md)
-* [Examples](example/)
+
+Big Picture
+-----------
+* [Comparisons](doc/compare.md)
+* [Planned future work](doc/future.md)
 
 Internals
 ---------
-* [Architecture and protocol](doc/arch.md)
+* [Architecture](doc/arch.md)
+* [Protocols](doc/protocol.md)
 
 Talk
 ----
-* [Abstract](sfdc.md) and [slides](doc/sfdc.pdf) for San Francisco Distributed Computing meetup
+* [Abstract](sfdc.md) and [slides](doc/sfdc.pdf) for San Francisco Distributed Computing meetup, December 2013.
+
 
 Getting started
 ==========
 
-1. Install ruby 2.0 or 2.1 (not 1.9) from http://ruby-lang.org. Examples and tests will not work on windows (they use fork and unix sockets) or jruby, though probably the underying libs will (using tcp sockets).
+1. Install ruby 2.0 or 2.1 (not 1.9) from http://ruby-lang.org. Examples and tests will not work on Windows (they use fork and unix sockets) or JRuby, though probably the underying libs will (using tcp sockets on Windows).
 
 2. Install the gem and its dependencies (you may need to `sudo` this):
 
@@ -44,68 +58,114 @@ Getting started
         >> t [nil, nil]
         => ["hello", "world"]
 
-4. Take a look at the [FAQ](doc/faq.md), [tutorial](doc/tutorial.md), and the many [examples](example/).
+4. Take a look at the [FAQ](doc/faq.md), the [tutorial](doc/tutorial.md), and the many [examples](example).
 
 
 Applications
 =======
 
-Tupelo is a flexible base layer for various distributed programming paradigms: job queues, dataflow, map-reduce, etc. Using subspaces, it's also a transactional, replicated datastore with pluggable storage providers.
+Tupelo is a flexible base layer for various distributed programming patterns and techniques, which are explored in the examples: job queues, shared configuration and state, load balancing, service discovery, in-memory data grids, message queues, publish/subscribe, dataflow, map-reduce, and both optimistic and pessimistic (lock/lease) concurrency control.
 
+Tupelo can be used to impose a unified transactional structure and distributed access model on a mixture of programs and languages (polyglot computation) and a mixture of data stores (polyglot persistence), with consistent replication.
 
-Advantages
-==========
 
-Tupelo can be used to impose a unified transactional structure and distributed access model on a mixture of programs and stores. ("Polyglot persistence".) Need examples....
+Example
+-------
 
-Speed (latency, throughput):
+This program counts prime numbers in an interval by distributing the problem to a set of hosts:
 
-* minimal system-wide bottlenecks
+    require 'tupelo/app/remote'
 
-* non-blocking socket reads
+    hosts = %w{itchy scratchy lisa bart} # ssh hosts with key-based auth
 
-* read -- local and hence very fast
+    Tupelo.tcp_application do
+      hosts.each do |host|
+        remote host: host, passive: true, eval: %{
+          require 'prime' # ruby stdlib for prime factorization
+          loop do
+            _, input = take(["input", Integer])
+            write ["output", input, input.prime_division]
+          end
+        }
+      end
 
-* write -- fast, pipelined (waiting for acknowledgement is optional);  
+      local do
+        inputs = 1_000_000_000_000 .. 1_000_000_000_200
 
-* transactions -- combine several takes and writes, reducing latency and avoiding locking
+        inputs.each do |input|
+          write ["input", input]
+        end
 
-Can use optimal data structure for each subspace of tuplespace.
+        count = 0
+        inputs.size.times do |i|
+          _, input, factors = take ["output", Integer, nil]
+          count += 1 if factors.size == 1 and factors[0][1] == 1
+          print "\rChecked #{i}"
+        end
 
-Decouples storage from query. (E.g. archiver for storage, optimized for just insert, delete, dump. And in-memory data structure, such as red-black tree, optimized for sorted query.)
+        puts "\nThere are #{count} primes in #{inputs}"
+      end
+    end
 
-Each client can have its own matching agorithms and api -- matching is not part of the comm protocol, which is defined purely in terms of tuples.
+Ssh is used to set up the remote processes. Additionally, with the `--tunnel` command line argument, all tuple communication is tunneled over ssh. More examples like this are in [example/map-reduce](example/map-reduce).
 
-Data replication is easy--hard to avoid in fact.
 
 Limitations
 ===========
 
-Better for small messages, because they tend to propagate widely.
+The main limitation of tupelo is that **all network communication passes through a single process**, the message sequencer. This process has minimal state and minimal computation. The state is just a counter and the network connections (no storage of tuples or other application data). The computation is just counter increment and message dispatch (no transaction execution or searches). A transaction requires just one message (possibly with many recipients) to pass through the sequencer. The message sequencer can be light and fast.
+
+Nevertheless, this process is a bottleneck. Each message traverses two hops, to and from the sequencer. Each tupelo client must be connected to the sequencer to transact on tuples (aside from local reads).
 
-May stress network and local memory (but subspaces can help).
+**Tupelo will always have this limitation.** It is essential to the design of the system. By accepting this cost, we get some benefits, discussed in the next section.
 
-Worker thread has cpu cost (but subspaces can help).
+The message sequencer is also a SPoF (single point of failure), but this is not inherent in the design. A future version of tupelo will have options for failover or clustering of the sequencer, perhaps based on [raft](http://raftconsensus.github.io), with a cost of increased latency and complexity. (However, redundancy and failover of *application* data and computation *is* supported by the current implementation; app data and computations are distributed among the client processes.)
 
-What other potential problems and how does tupelo solve them?
+There are some limitations that may result from naive application of tupelo: high client memory use, high bandwidth use, high client cpu use. These resource issues can often be controlled with [subspaces](doc/subspace.md) and specialized data structures and data stores. There are several examples addressing these problems. Another approach is to use the tuplespace for low volume references to high volume data.
 
+Also, see the discussion in [transactions](doc/transactions.md) on limitations of transactions across subspaces.
 
-Future
-======
+This implementation is also limited in efficiency because of its use of Ruby.
 
-- Subspaces. Redundancy, for read-heavy data stores (redundant array of in-memory sqlite, for example). Clients managing different subspaces may benefit by using different stores and algorithms.
+Finally, it must be understood that work on tupelo is still in early, experimental stages. **The tupelo software should not yet be relied on for applications where failure resistance and recovery are important.** The current version is suited for things like batch processing (especially complex dataflow topologies), which can be restarted after failure, or other distributed systems that have short lifespans or are disposable.
+
+
+Benefits
+========
 
-- More persistence options.
+As noted above, the sequencer assigns an incrementing sequence number, or *tick*, to each transaction and dispatches it to the clients, who take on all the burden of tuple computation and storage. This design choice leads to:
 
-- Fail-over. Robustness.
+* strong consistency: all clients have the same view of the tuplespace at a given tick of the global clock;
 
-- Investigate nio4r for faster networking, especially with many clients.
+* deterministic transaction execution across processes: transactions complete in two network hops, and transactions reference concrete tuples, not templates or queries that require further searching;
 
-- Interoperable client and server implementations in C, Python, Go, Elixir?
+* high concurrency: no interprocess locking or coordination is needed to prepare or execute transactions;
 
-- UDP multicast to further reduce the bottleneck in the message sequencer.
+* efficient distribution of transaction workload off of the critical path: transaction preparation (finding matching tuples) is performed by just the client initiating the transaction, and transaction execution is performed only by clients that subscribe to subspaces relevant to the transaction;
 
-- Tupelo as a service; specialized and replicated subspace managers as services.
+* client-side logic within transactions: any client state can be accessed while preparing a transaction, and each client is free to use any template and search mechanism (deterministic or not), possibly taking advantage of the client's specialized tuple storage;
+
+* zero-latency reads: clients store subscribed tuples locally, so searching and waiting for matching tuples are local operations;
+
+* relatively easy data replication: all subscribers to a subspace replicate that subspace, possibly with different storage implementations;
+
+* the current state of the tuplespace can be computed from a earlier state by replaying the transactions in sequence;
+
+* the evolution of system state over time is observable, and tupelo provides the tools to do so: the `--trace` switch, the `#trace` api, and the `tspy` program.
+
+Additional benefits (not related to message sequencing) include:
+
+* the `tup` program for interactively starting and connecting to tupelo instances;
+
+* a framework for starting and controlling child and remote processes connected to the tuplespace;
+
+* options to tunnel connections over ssh and through firewalls, for running in public clouds and other insecure environments;
+
+* choice of object serialization method (msgpack, json, marshal, yaml);
+
+* choice of UNIX or TCP sockets.
+
+Process control and tunneling are available independently of tupelo using the easy-serve gem.
 
 
 Development
@@ -138,9 +198,11 @@ Other gems:
 
 * yajl-ruby (only used to support --json option)
 
+* nio4r (optional dependency of funl)
+
 Optional gems for some of the examples:
 
-* sinatra, http, sequel, sqlite, rbtree, leveldb-native
+* sinatra, json, http, sequel, sqlite, rbtree, leveldb-native, lmdb
 
 Contact
 =======

data/bin/tup

@@ -66,11 +66,9 @@ if ARGV.delete("-h") or ARGV.delete("--help")
                   load and save tuplespace to DIR
                     (only needs to be set on first tup invocation)
 
-      --use-subspaces
-                  enable subspaces for this tupelo service
-                    (only needs to be set on first tup invocation)
       --subscribe TAG,TAG,...
                   subscribe to specified subspaces; use "" for none
+                  by default, tup client subscribes to everything
 
   END
   exit
@@ -82,8 +80,6 @@ argv, tupelo_opts = Tupelo.parse_args(ARGV)
 
 pubsub = argv.delete("--pubsub") # not a standard tupelo opt
 
-use_subspaces = argv.delete("--use-subspaces") # not a standard tupelo opt
-
 if i=argv.index("--subscribe") # default is to subscribe to all
   argv.delete("--subscribe")
   subscribed_tags = argv.delete_at(i).split(",")
@@ -156,8 +152,6 @@ Tupelo.application(
   end
 
   local TupClient, **client_opts do
-    use_subspaces! if use_subspaces
-
     log.info {"cpu time: %.2fs" % Process.times.inject {|s,x|s+x}}
     log.info {"starting shell."}
 

data/bugs/take-write.rb

@@ -15,5 +15,13 @@ Tupelo.application do
     status, tick, cid, op = note.wait
     p op # should "read [1]", not "write [1]; take [1]"
     # this is just an optimization, not really a bug
+
+    # however, need to be careful about this optimization, since
+    #   transaction {take [1]; take [1]; write [1]}
+    # is not the same as
+    #   transaction {take [1]; read [1]}
+    # but rather more like
+    #   transaction {take [1]; read_distinct [1]}
+    # except #read_distinct is not defined.
   end
 end

data/example/bingo/bingo-v2.rb

@@ -0,0 +1,20 @@
+
+__END__
+  # dealer
+  child passive: true do
+    loop do
+      # transaction -- add to v2
+      _, player_id = take ["buy", nil]
+      
+    end
+  end
+  
+  # player -- in v3 use subspace per player
+end
+
+## how to buy 4 without increasing contention risk
+
+## reduce contention for cards by randomizing or cons. hashing?
+
+ ## swapping these lines -> more contention?
+ 

data/example/broker-queue.rb

@@ -0,0 +1,35 @@
+# more like how you would do it in redis, except that the queue is not stored in
+# the central server, so operations on it are not a bottleneck, FWIW
+
+require 'tupelo/app'
+
+N_PLAYERS = 10
+
+Tupelo.application do
+  N_PLAYERS.times do
+    # sleep rand / 10 # reduce contention -- could also randomize inserts
+    child do
+      me = client_id
+      write name: me
+      
+      you = transaction do
+        game = read_nowait(
+          player1: nil,
+          player2: me)
+        break game["player1"] if game
+      
+        unless take_nowait name: me
+          raise Tupelo::Client::TransactionFailure
+        end
+
+        you = take(name: nil)["name"]
+        write(
+          player1: me,
+          player2: you)
+        you
+      end
+
+      log "now playing with #{you}"
+    end
+  end
+end

data/example/child-of-child.rb

@@ -0,0 +1,34 @@
+require 'tupelo/app'
+
+### need a programmatic way to start up clients
+
+Tupelo.application do |app|
+
+  app.child do ## local still hangs
+    3.times do |i|
+      app.child do
+        write [i]
+        log "wrote #{i}"
+      end
+    end
+
+    3.times do
+      log take [nil]
+    end
+  end
+end
+
+__END__
+
+this hangs sometimes but not always:
+
+  tick    cid status operation
+A: client 3: wrote 0
+A: client 4: wrote 1
+     1      3        batch write [0]
+     2      4        batch write [1]
+A: client 2: [0]
+     3      2        atomic take [0]
+     4      2        atomic take [1]
+A: client 2: [1]
+A: client 5: wrote 2

data/example/consistent-hash.rb

@@ -9,8 +9,6 @@ N_ITER = 1000
 
 Tupelo.application do
   local do
-    use_subspaces!
-
     N_BINS.times do |id|
       define_subspace id, [id, Numeric, Numeric]
     end

data/example/counters/lock.rb

@@ -0,0 +1,24 @@
+require 'tupelo/app'
+
+N_PROCS = 3
+N_ITER = 10
+
+Tupelo.application do
+  pids = N_PROCS.times.map do
+    child do
+      N_ITER.times do
+        c = take count: nil
+          # unlike optimistic.rb, a system/network failure here
+          # could cause this tuple to be lost. To safeguard: example/lease.rb
+        write count: c["count"] + 1
+        sleep 0.1
+      end
+    end
+  end
+  
+  local do
+    write count: 0
+    pids.each {|pid| Process.waitpid pid}
+    log read count: nil
+  end
+end

data/example/counters/merge.rb

@@ -0,0 +1,35 @@
+require 'tupelo/app'
+
+N_PROCS = 3
+N_ITER = 10
+
+Tupelo.application do
+  pids = N_PROCS.times.map do
+    child do
+      N_ITER.times do
+        write count: 1
+        sleep 0.1
+      end
+    end
+  end
+  
+  local do
+    # note: no need to init counter
+    pids.each {|pid| Process.waitpid pid}
+
+    # but we cannot read the counter(s) without first merging them
+    # see also example/dedup.rb
+    while transaction do
+      c1 = take_nowait count: nil
+      c2 = take_nowait count: nil
+      if c1 and c2
+        write count: c1["count"] + c2["count"]
+        true
+      elsif c1
+        log c1
+      else
+        log count: 0
+      end
+    end
+  end
+end

data/example/counters/optimistic.rb

@@ -0,0 +1,29 @@
+require 'tupelo/app'
+
+N_PROCS = 3
+N_ITER = 10
+
+Tupelo.application do
+  pids = N_PROCS.times.map do
+    child do
+      N_ITER.times do
+        transaction do
+          c = take count: nil
+          write count: c["count"] + 1
+        end
+        sleep 0.1
+      end
+    end
+  end
+  
+  local do
+    write count: 0
+      # we have to make sure we write this initial counter only once,
+      # which is easy in this case, but not always. See merge.rb for
+      # an approach that allows multiple initializations.
+    
+    pids.each {|pid| Process.waitpid pid}
+      # could also use tuples to do this
+    log read count: nil
+  end
+end

data/example/dataflow.rb

@@ -0,0 +1,21 @@
+# http://slashdot.org/topic/bi/how-is-reactive-different-from-procedural-programming
+# http://developers.slashdot.org/story/14/01/13/2119202/how-reactive-programming-differs-from-procedural-programming
+# bud and bloom
+
+# TODO: DSL for expressing data dependency relations
+
+require 'tupelo/app'
+
+N_WORKERS = 2
+
+Tupelo.application do
+  N_WORKERS.times do
+    child passive: true do
+      
+    end
+  end
+
+  local do
+    #write 
+  end
+end

data/example/dedup.rb

@@ -0,0 +1,45 @@
+# How to deduplicate a tuple.
+#
+# Problem: we have some clients, each of which writes the same tuple.
+# How can we remove duplicates?
+#
+# Run with --trace to see how this works.
+
+require 'tupelo/app'
+
+N_CLIENTS = 5
+
+T = [1] # whatever
+
+Tupelo.application do
+  N_CLIENTS.times do
+    child do
+      unless read_nowait T  # try not to write dups, but...
+        write_wait T        # T is possibly not unique
+      end
+
+      # After writing T, each client tries to reduce the T population to one
+      # tuple.
+      catch do |done|
+        loop do
+          transaction do
+            if take_nowait T and take_nowait T
+              write T
+            else
+              throw done # don't take or write anything
+            end
+          end
+        end
+      end
+
+      # At the tick on which the last transaction above from this client
+      # completes, there is a unique T, but of course that may change in the
+      # future, for example, if another client's `write_wait T` was delayed in
+      # flight over the network). But in that case, the other client will also
+      # perform the same de-dup code.
+
+      count = read_all(T).size
+      log "count = #{count}"
+    end
+  end
+end

data/example/map-reduce/ex.rb

@@ -0,0 +1,32 @@
+require 'tupelo/app/remote'
+
+hosts = %w{od1 od2 ut} * 4 # list of ssh hosts; must not require password
+
+Tupelo.tcp_application do
+  hosts.each do |host|
+    remote host: host, passive: true, eval: %{
+      require 'prime' # ruby stdlib for prime factorization
+      loop do
+        _, input = take(["input", Integer])
+        write ["output", input, input.prime_division]
+      end
+    }
+  end
+
+  local do
+    inputs = 1_000_000_000_000 .. 1_000_000_000_200
+
+    inputs.each do |input|
+      write ["input", input]
+    end
+
+    count = 0
+    inputs.size.times do |i|
+      _, input, factors = take ["output", Integer, nil]
+      count += 1 if factors.size == 1 and factors[0][1] == 1
+      print "\rChecked #{i}"
+    end
+    
+    puts "\nThere are #{count} primes in #{inputs}"
+  end
+end

data/example/multi-tier/memo2.rb

@@ -14,8 +14,6 @@ fork do
 
   Tupelo.application do
     local do
-      use_subspaces!
-
       define_subspace("memo", [
         "memo",   # tag is encoded in each tuple, for recognizing
         String,   # key in the cache, must be string

data/example/pregel/dist-opt.rb

@@ -0,0 +1,15 @@
+#
+# Minor optimization:
+
+class KeyMatcher
+  def initialize i, n
+    @i = i
+    @n = n
+  end
+
+  def === id
+    id % @n == @i
+  end
+end
+
+vertex = take id: v_id_matcher, step: step, rank: nil, active: true

data/example/riemann/event-subspace.rb

@@ -46,6 +46,8 @@ class Tupelo::Client
     custom:       nil
   }.freeze
 
+  # Also could be a subspace of the event subspace, but for now, we can just
+  # use it as a template to select expired events out of the event subspace.
   EXPIRED_EVENT = {
     host:         nil,
     service:      nil,

data/example/riemann/expiration-dbg.rb

@@ -0,0 +1,15 @@
+class Tupelo::Client
+  def run_expiration_debugger
+    read Tupelo::Client::EXPIRED_EVENT do |event|
+      event_exp = event["time"] + event["ttl"]
+      delta = Time.now.to_f - event_exp
+      if delta > 0.1
+        log.warn "expired late by %6.4f seconds: #{event}" % delta
+      elsif delta < 0
+        log.warn "expired too soon: #{event}"
+      else
+        log.info "expired on time: #{event}"
+      end
+    end
+  end
+end