tupelo 0.9 → 0.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 241bdc610aa513222cd452c8624bcf69cd9b2686
-  data.tar.gz: 674756c0a1156745007199a4e476df72df7183ef
+  metadata.gz: 892628e201d400fa9592a01b1fd62c4bb8e09ab5
+  data.tar.gz: 2ca45f4f124ad6db32e2880557004f5c0553cf9f
 SHA512:
-  metadata.gz: c6b2e338a9d11abfe109ed608b8b51807bc69994a75824978c37f0b2ad00327efc080b8e8ee1eb85303b9fee3961045fbae1b2b9bc6fa5c6776a743e7f818f67
-  data.tar.gz: 026cf80f84292493865f5f644f54ce883defa9351d99c6268c02791d500edd88cbeaa0cc4c25c556d0a96f3cee424f33dea1a4d97d1f421bfd1532803a323d21
+  metadata.gz: 0b8bfcdbb7143f543372cd0bdad95feb49305a47e57e0368146e3c224f33e1fddb261f9f019ec4c4a503a4250d91ecc653a93e5792c27e4f47cfd7ca33727a76
+  data.tar.gz: 931b57069c7e06452429eb71e083c568be953a1e37de54cd1b9541107c6a79884e7d64645ed8765777c9ede5f18589e3c35d5057e946e82b8daa70c18180b49d

data/README.md

@@ -21,6 +21,7 @@ Documentation
 ============
 
 * [FAQ](doc/faq.md)
+* [Subspaces](doc/subspace.md)
 
 Getting started
 ==========
@@ -110,7 +111,7 @@ Getting started
           ...
         end
 
-  Note that a local match is still not a guarantee of `x_final == x_optimistic`. Another process may take `x_optimistic` first, and the take will be re-executed. (Think of #take_nowait as a way of saying "take a match, but don't bother trying if there is no match known at this time.")
+  Note that a local match is still not a guarantee of `x_final == x_optimistic`. Another process may take `x_optimistic` first, and the take will be re-executed. (Think of #take_nowait as a way of saying "take a match, but don't bother trying if there is no match known at this time.") Similarly, #take_nowait returning nil is not a guarantee that a match does not exist: another process could have written a match later than the time of the local search.
 
   Perform a general transaction:
 
@@ -124,7 +125,17 @@ Getting started
           end
 
   Note that the block may execute more than once, if there is competition for the tuples that you are trying to #take or #read. When the block exits, however, the transaction is final and universally accepted by all clients.
+
+  Tuples written or taken during a transaction affect subsequent operations in the transaction without modifying the tuplespace or affecting other concurrent transactions (until the transaction completes):
   
+        transaction do |t|
+          t.write [3]
+          p t.read [3] # => 3
+          p read_all   # => [] # note read_all called on client, not trans.
+          t.take [3]
+          p t.read_nowait [3] # => nil
+        end
+
   You can timeout a transaction:
   
         transaction timeout: 1 do
@@ -180,6 +191,8 @@ A tuplespace is a service for coordination, configuration, and control of concur
 
 See https://en.wikipedia.org/wiki/Tuple_space for general information and history. This project is strongly influenced by Masatoshi Seki's Rinda implementation, part of the Ruby standard library. See http://pragprog.com/book/sidruby/the-druby-book for a good introduction to rinda and druby.
 
+See http://dbmsmusings.blogspot.com/2010/08/problems-with-acid-and-how-to-fix-them.html for an explanation of the imporance of determinism in distributed transaction systems.
+
 What is a tuple?
 ----------------
 
@@ -210,6 +223,8 @@ In other words, a tuple is a fairly general object, though this depends on the s
 
 It's kind of like a "JSON object", except that, when using the json serializer, the hash keys can only be strings. In the msgpack case, keys have no special limitations. In the case of the marshal and yaml modes, tuples can contain many other kinds of objects.
 
+The empty tuples `[]` and `{}` are allowed, but bare values such as `3.14` or `false` are not tuples by themselves.
+
 One other thing to keep in mind: in the array case, the order of the elements is significant. In the hash case, the order is not significant. So these are both true:
 
     [1,2] != [2,1]
@@ -292,6 +307,17 @@ Transactions have a significant disadvantage compared to using take/write to loc
 
 Transactions do have an advantage over using take/write to lock/unlock tuples: there is no possibility of deadlock. See [example/deadlock.rb](example/deadlock.rb) and [example/parallel.rb](example/parallel.rb).
 
+Another advantage of tranactions is that it is possible to guarantee continuous existence of a time-series of tuples. For example, suppose that tuples matching `{step: Numeric}` indicate the progress of some activity. With transactions, you can guarantee that there is exactly one matching tuple at any time. So any client which reads this template will find a match without blocking.
+
+Another use of transactions: forcing a retry when something changes:
+
+    transaction do
+      step = read(step: nil)["step"]
+      take value: nil, step: step
+    end
+
+This code waits on the existence of a value, but retries if the step changes while waiting. See example/pregel/distributed.rb for a use of this techinique.
+
 Tupelo transactions are ACID in the following sense. They are Atomic and Isolated -- this is enforced by the transaction processing in each client. Consistency is enforced by the underlying message sequencer: each client's copy of the space is the deterministic result of the same sequence of operations. Durability is optional, but can be provided by the persistent archiver or other clients.
 
 On the CAP spectrum, tupelo tends towards consistency: for all clients, write and take operations are applied in the same order, so the state of the entire system up through a given tick of discrete time is universally agreed upon. Of course, because of the difficulties of distributed systems, one client may not yet have seen the same range of ticks as another.
@@ -404,6 +430,8 @@ To compare
 
 * doozer, etcd
 
+* serf -- tupelo has lower latency and is transactional, but at a cost compared to serf; tupelo semantics is closer to databases
+
 * arakoon
 
 * hazelcast
@@ -471,6 +499,8 @@ Each inner serialization method ("blobber") has its own advantages and drawbacks
 
 For most purposes, msgpack is a good choice, so it is the default.
 
+The sending client's tupelo library must make sure that there is no aliasing within the list of tuples (this is only an issue for Marshal and YAML, since msgpack and json do not support references).
+
 
 Development
 ===========

data/bugs/read-take.rb

@@ -0,0 +1,19 @@
+require 'tupelo/app'
+
+Tupelo.application do
+  local do
+    write_wait [1]
+
+    note = notifier
+
+    transaction do 
+      read [1]
+      take [1]
+    end
+    
+    note.wait
+    status, tick, cid, op = note.wait
+    p op # should "take [1]", not "take [1]; read [1]"
+    # this is just an optimization, not really a bug
+  end
+end

data/bugs/take-write.rb

@@ -1,14 +1,14 @@
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.local do |client|
-    client.write_wait [1]
+Tupelo.application do
+  local do
+    write_wait [1]
 
-    note = client.notifier
+    note = notifier
 
-    client.transaction do |t|
-      x = t.take [1]
-      t.write x
+    transaction do
+      x = take [1]
+      write x
     end
     
     note.wait

data/example/app-and-tup.rb

@@ -11,19 +11,22 @@
 #
 # w [2]
 # w [3]
-# ra [nil] # => [[5]]
+# ra # => [[5]]
 # w [7.4]
-# ra [nil] # => [[12.4]]
+# ra # => [[12.4]]
 
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.child do |client|
+filename = "servers-#$$.yaml"
+puts "run this in another shell: tup #{filename}"
+
+Tupelo.application servers_file: filename do
+  child do
     loop do
-      client.transaction do |t|
-        x, = t.take [Numeric]
-        y, = t.take [Numeric]
-        t.write [x + y]
+      transaction do
+        x, = take [Numeric]
+        y, = take [Numeric]
+        write [x + y]
       end
     end
   end

data/example/async-transaction.rb

@@ -2,15 +2,15 @@ require 'tupelo/app'
 
 # see also cancel.rb
 
-Tupelo.application do |app|
-  app.child do |client|
-    t = client.transaction.async do |t|
-      t.write ["pong"]
-      t.take ["ping"]
+Tupelo.application do
+  child do
+    t = transaction.async do
+      write ["pong"]
+      take ["ping"]
     end
     
-    client.write ["ping"]
-    puts client.take ["pong"]
+    write ["ping"]
+    puts take ["pong"]
     puts t.value
   end
 end

data/example/balance-xfer-locking.rb

@@ -1,28 +1,28 @@
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.child do |client|
-    client.write(
+Tupelo.application do
+  child do
+    write(
       {name: "alice", balance: 1000},
       {name: "bob", balance: 200}
     )
     10.times do |i|
-      alice = client.take(name: "alice", balance: Numeric)
-      client.log alice
+      alice = take(name: "alice", balance: Numeric)
+      log alice
       alice = alice.dup
       alice["balance"] -= 10
-      client.write_wait alice
+      write_wait alice
       sleep 0.1
     end
     
-    client.log client.read_all(name: /^(?:alice|bob)$/, balance: nil)
+    log read_all(name: /^(?:alice|bob)$/, balance: nil)
   end
   
-  app.child do |client|
+  child do
     sleep 0.3
 
-    src = client.take(name: "alice", balance: Numeric)
-    dst = client.take(name: "bob", balance: Numeric)
+    src = take(name: "alice", balance: Numeric)
+    dst = take(name: "bob", balance: Numeric)
 
     if src["balance"] < 500
       abort "insufficient funds -- not attempting transfer"
@@ -41,10 +41,10 @@ Tupelo.application do |app|
     src["balance"] -= 500
     dst["balance"] += 500
 
-    w = client.write src, dst
-    client.log "attempting to set #{[src, dst]}"
+    w = write src, dst
+    log "attempting to set #{[src, dst]}"
 
     w.wait
-    client.log client.read_all(name: /^(?:alice|bob)$/, balance: nil)
+    log read_all(name: /^(?:alice|bob)$/, balance: nil)
   end
 end

data/example/balance-xfer-retry.rb

@@ -1,27 +1,27 @@
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.child do |client|
-    client.write(
+Tupelo.application do
+  child do
+    write(
       {name: "alice", balance: 1000},
       {name: "bob", balance: 200}
     )
     10.times do |i|
-      alice = client.take(name: "alice", balance: Numeric)
-      client.log alice
+      alice = take(name: "alice", balance: Numeric)
+      log alice
       alice = alice.dup
       alice["balance"] -= 10
-      client.write_wait alice
+      write_wait alice
       sleep 0.1
     end
     
-    client.log client.read_all(name: /^(?:alice|bob)$/, balance: nil)
+    log read_all(name: /^(?:alice|bob)$/, balance: nil)
   end
   
-  app.child do |client|
-    client.transaction do |t|
-      src = t.take(name: "alice", balance: Numeric)
-      dst = t.take(name: "bob", balance: Numeric)
+  child do
+    transaction do
+      src = take(name: "alice", balance: Numeric)
+      dst = take(name: "bob", balance: Numeric)
 
       if src["balance"] < 500
         abort "insufficient funds -- not attempting transfer"
@@ -37,19 +37,19 @@ Tupelo.application do |app|
         # force fail -- the tuples this client is trying to take
         # will be gone when it wakes up
       
-      client.log "attempting to set #{[src, dst]}"
-      t.write src, dst
+      log "attempting to set #{[src, dst]}"
+      write src, dst
 
       if false # enable this to see how failures are retried
         begin
-          t.commit.wait
+          commit.wait
         rescue => ex
-          client.log "retrying after #{ex}"
+          log "retrying after #{ex}"
           raise
         end
       end
     end
     
-    client.log client.read_all(name: /^(?:alice|bob)$/, balance: nil)
+    log read_all(name: /^(?:alice|bob)$/, balance: nil)
   end
 end

data/example/balance-xfer.rb

@@ -1,17 +1,17 @@
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.child do |client|
-    client.write(
+Tupelo.application do
+  child do
+    write(
       {name: "alice", balance: 1000},
       {name: "bob", balance: 200}
     )
   end
   
-  app.child do |client|
-    client.transaction do |t|
-      src = t.take(name: "alice", balance: Numeric)
-      dst = t.take(name: "bob", balance: Numeric)
+  child do
+    transaction do
+      src = take(name: "alice", balance: Numeric)
+      dst = take(name: "bob", balance: Numeric)
 
       if src["balance"] < 500
         abort "insufficient funds -- not attempting transfer"
@@ -23,11 +23,11 @@ Tupelo.application do |app|
       src["balance"] -= 500
       dst["balance"] += 500
 
-      t.write src, dst
+      write src, dst
     end
     # transaction will block if balances have changed since the read.
     # see balance-xfer-retry.rb
     
-    client.log client.read_all(name: /^(?:alice|bob)$/, balance: nil)
+    log read_all(name: /^(?:alice|bob)$/, balance: nil)
   end
 end

data/example/boolean-match.rb

@@ -1,15 +1,15 @@
 require 'tupelo/app'
 require 'tupelo/util/boolean'
 
-Tupelo.application do |app|
-  app.local do |client|
-    tm = client.match_any [0..2, String], [3..5, Hash]
+Tupelo.application do
+  local do
+    tm = match_any [0..2, String], [3..5, Hash]
     
-    client.write(
+    write(
       [0, "a"], [1, {b: 0}], [2, "c"],
       [3, "a"], [4, {b: 0}], [5, "c"]
     ).wait
 
-    client.log client.read_all tm
+    log read_all tm
   end
 end

data/example/bounded-retry.rb

@@ -3,32 +3,32 @@ require 'tupelo/app'
 N = 2
 K = 1
 
-Tupelo.application do |app|
+Tupelo.application do
   (N+K).times do
-    app.child do |client|
+    child do
       catch :gave_up do
         tries = 0
 
-        r = client.take [Integer] do |val|
+        r = take [Integer] do |val|
           tries += 1
           if tries >= N
-            client.log "giving up on #{val}"
+            log "giving up on #{val}"
             throw :gave_up
           end
-          client.log "trying to take #{val}"
+          log "trying to take #{val}"
         end
 
-        client.log "took #{r.inspect}"
+        log "took #{r.inspect}"
       end
     end
   end
   
   sleep 0.01
 
-  app.child do |client|
+  child do
     N.times do |i|
-      client.write [i]
-      client.log "wrote #{[i]}"
+      write [i]
+      log "wrote #{[i]}"
       sleep 0.1
     end
   end

data/example/broker-locking.rb

@@ -7,37 +7,37 @@ N_PLAYERS = 10
 
 token = ["token"] # only the holder of the token can arrange games
 
-Tupelo.application do |app|
-  app.local do |client|
-    client.write token
+Tupelo.application do
+  local do
+    write token
   end
   
   N_PLAYERS.times do
-    app.child do |client|
-      me = client.client_id
+    child do
+      me = client_id
 
-      client.take token # bottleneck and fragile until 'client.write token'
-      other_player = client.read_nowait(name: nil)
+      take token # bottleneck and fragile until 'write token'
+      other_player = read_nowait(name: nil)
         # sleep 1 # program takes ~N_PLAYERS sec to finish
 
       if other_player
-        client.take other_player
-        client.write(
+        take other_player
+        write(
           player1: me,
           player2: other_player["name"])
-        client.write token
+        write token
         you = other_player["name"]
 
       else
-        client.write(name: me)
-        client.write token
-        game = client.read(
+        write(name: me)
+        write token
+        game = read(
           player1: nil,
           player2: me)
         you = game["player1"]
       end
 
-      client.log "now playing with #{you}"
+      log "now playing with #{you}"
     end
   end
 end

data/example/broker-optimistic.rb

@@ -9,15 +9,15 @@ require 'tupelo/app'
 
 N_PLAYERS = 10
 
-Tupelo.application do |app|
+Tupelo.application do
   N_PLAYERS.times do
     # sleep rand / 10 # reduce contention -- could also randomize inserts
-    app.child do |client|
-      me = client.client_id
-      client.write name: me
+    child do
+      me = client_id
+      write name: me
       
       begin
-        t = client.transaction
+        t = transaction
         if t.take_nowait name: me
           you = t.take(name: nil)["name"]
           t.write(
@@ -28,14 +28,14 @@ Tupelo.application do |app|
           t.fail!
         end
       rescue Tupelo::Client::TransactionFailure => ex
-        game = client.read_nowait(
+        game = read_nowait(
           player1: nil,
           player2: me)
         retry unless game
         you = game["player1"]
       end
 
-      client.log "now playing with #{you}"
+      log "now playing with #{you}"
     end
   end
 end

data/example/cancel.rb

@@ -1,17 +1,17 @@
 require 'tupelo/app'
 
-Tupelo.application do |app|
-  app.child do |client|
+Tupelo.application do
+  child do
     ats = (0..4).map do |i|
-      client.transaction.async do |t|
-        t.take ["start"]
-        t.write [i]
+      transaction.async do
+        take ["start"]
+        write [i]
       end
     end
     
     [0,1,2,4].each {|i| ats[i].cancel}
 
-    client.write ["start"]
-    p client.take [Integer]
+    write ["start"]
+    p take [Integer]
   end
 end

data/example/concurrent-transactions.rb

@@ -2,37 +2,37 @@ require 'tupelo/app'
 
 N = 100
 
-Tupelo.application do |app|
-  app.child do |client|
-    client.write [0, 0]
+Tupelo.application do
+  child do
+    write [0, 0]
 
     t1 = Thread.new do
       N.times do
-        client.transaction do |t|
-          t.take ["reader ready"]
-          x, y = t.take [nil, nil]
-          t.write [x+1, y]
-          t.write ["data ready"]
+        transaction do
+          take ["reader ready"]
+          x, y = take [nil, nil]
+          write [x+1, y]
+          write ["data ready"]
         end
       end
     end
     
     t2 = Thread.new do
       N.times do
-        client.transaction do |t|
-          t.take ["reader ready"]
-          x, y = t.take [nil, nil]
-          t.write [x, y+1]
-          t.write ["data ready"]
+        transaction do
+          take ["reader ready"]
+          x, y = take [nil, nil]
+          write [x, y+1]
+          write ["data ready"]
         end
       end
     end
     
     loop do
-      client.write ["reader ready"]
-      client.take ["data ready"]
-      x, y = client.read [nil, nil]
-      client.log "%3d %3d" % [x, y]
+      write ["reader ready"]
+      take ["data ready"]
+      x, y = read [nil, nil]
+      log "%3d %3d" % [x, y]
       break if x == N and y == N
     end
   end

data/example/custom-class.rb

@@ -10,20 +10,20 @@ end
 require 'tupelo/app'
 
 # Must use marshal or yaml -- msgpack and json do not support custom classes.
-Tupelo.application blob_type: 'marshal' do |app|
-  app.child do |client|
+Tupelo.application blob_type: 'marshal' do
+  child do
     f = Foo.new; f.x = 3
     p f
 
-    client.write [f]
+    write [f]
 
-    p client.read [nil]
-    p client.read [Foo]
-    p client.read [f]
+    p read [nil]
+    p read [Foo]
+    p read [f]
 
-    p client.take [Foo]
+    p take [Foo]
 
-    client.write [f]
-    p client.take [f]
+    write [f]
+    p take [f]
   end  
 end

data/example/custom-search.rb

@@ -14,14 +14,14 @@ class MyClient < Tupelo::Client
   end
 end
 
-Tupelo.application do |app|
-  app.local MyClient do |client|
-    client.write [41, 42, 43]
-    client.write [42, 42, 42]
-    client.write [42, 42]
-    client.write_wait [42] # make sure all writes up to this one have completed
+Tupelo.application do
+  local MyClient do
+    write [41, 42, 43]
+    write [42, 42, 42]
+    write [42, 42]
+    write_wait [42] # make sure all writes up to this one have completed
 
-    client.log client.read_all [nil, nil, nil]
-    client.log client.read_all_diagonal 42
+    log read_all
+    log read_all_diagonal 42
   end
 end