tupelo 0.4 → 0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 526ed0a8bc267debbd03d9306433256bc225ff41
-  data.tar.gz: 1be5ebae3fff130e0d0dde261490604f4936d78e
+  metadata.gz: 1f296ad08b928805d512dc88cbdc0f5b1196fde6
+  data.tar.gz: f25b3ac8ef8fac55f5a9affc1b33cf42409aebc6
 SHA512:
-  metadata.gz: a98abb4bf4465e3e166c5295065984120f62a06021e369aa53db12f8ad83308253cb170a2b63fca017c8b83a0abfc1d15c29268af1b22ac5bca89b0d3119ffb4
-  data.tar.gz: 24a8d32264737f932a124abccaf9c9512febd28473930b762baaada6c159d8a5a7afb805b8ee55ac251764c9c19162b9185e29efd2508d06cbc1186c4a5ae066
+  metadata.gz: 2a88ccb3f784fc9106fb4d92c522fb26e61ae7ec0ad24fecbb805a94c4f01329ad717e035ed2f9fc8b795076aaec9c840d2601759452afb8502f61b7999ef86f
+  data.tar.gz: 4d3662603023f504618ea0f3f8e84ef076dce52f86150634ad4d8e540a610279f3604bb5071b6bd5e16c074bf330dcd7914f70397a52397e1966dae9d55ba8f3

data/README.md

@@ -21,7 +21,7 @@ Tupelo differs from other spaces in several ways:
 Getting started
 ==========
 
-1. Install ruby 2 (not 1.9) from http://ruby-lang.org. Examples and tests will not work on windows (they use fork and unix sockets), though probably the underying libs will (using tcp sockets).
+1. Install ruby 2 (not 1.9) from http://ruby-lang.org. Examples and tests will not work on windows (they use fork and unix sockets) or jruby, though probably the underying libs will (using tcp sockets).
 
 2. Install the gem and its dependencies (you may need to `sudo` this):
 
@@ -132,9 +132,17 @@ Getting started
 
         ls -d /usr/local/lib/ruby/gems/*/gems/tupelo*
 
-  Note that all bin and example programs accept blob type (e.g., --msgpaack, --json) on command line (it only needs to be specified for server -- the clients discover it). Also, all these programs accept log level on command line. The default is --warn. The --info level is a good way to get an idea of what is happening, without the verbosity of --debug.
+  Note that all bin and example programs accept blob type (e.g., --msgpack, --json) on command line (it only needs to be specified for server -- the clients discover it). Also, all these programs accept log level on command line. The default is --warn. The --info level is a good way to get an idea of what is happening, without the verbosity of --debug.
 
-6. Deugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful, and see the debugger client in example/lock-mgr.rb.
+6. Debugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful. There is also the similar --trace switch that is available to all bin and example programs, for example:
+
+      tick    cid status operation
+         1      2        batch write ["x", 1]
+         2      2        batch write ["y", 2]
+         3      3        atomic take ["x", 1], ["y", 2]
+
+
+  The `Tupelo.application` command, provided by `tupelo/app`, is the source of all these options and is available to your programs.
 
 
 What is a tuplespace?
@@ -147,7 +155,7 @@ See https://en.wikipedia.org/wiki/Tuple_space for general information and histor
 What is a tuple?
 ----------------
 
-A tuple is the unit of information in a tuplespace. It is immutable in the context of the tuplespace -- you can write a tuple into the space and you can read or take one from the space, but you cannot update a tuple within a space.
+A tuple is the unit of information in a tuplespace. It is immutable in the context of the tuplespace -- you can write a tuple into the space and you can read or take one from the space, but you cannot update a tuple within a space. A tuples does not have an identity other than the data it is made from.
 
 A tuple is either an array:
 
@@ -174,6 +182,8 @@ In other words, a tuple is a fairly general object, though this depends on the s
 
 It's kind of like a "JSON object", except that in the json blob case, the hash keys can only be strings. In the msgpack case, keys have no special limitations. In the case of the marshal and yaml modes, tuples can contain many other kinds of objects.
 
+A tuplespace can contain multiple copies of the same tuple.
+
 What is a template?
 -------------------
 
@@ -193,6 +203,12 @@ but not these tuples:
 
 The nil wildcard matches anything. The Range, Regexp, and Class entries function as wildcards because of the way they define the #=== (match) method. See ruby docs for general information on "threequals" matching.
 
+Every tuple can also be used as a template. The template:
+
+    [4, 7, "foobar", "xyz"]
+
+matches itself.
+
 Here's a template for matching some hash tuples:
 
     {name: String, location: "home"}
@@ -206,6 +222,10 @@ A template doesn't have to be a tuple pattern with wildcards, though. It can be
     read_all Array
     read_all Object
 
+An optional library, `tupelo/util/boolean`, provides a #match_any method to construct the boolean or of other templates:
+
+    read_all match_any( [1,2,3], {foo: "bar"} )
+
 Unlike in some tuplespace implementations, templates are a client-side concept (except for subspace-defining templates), which is a source of efficiency and scalability. Matching operations (which can be computationally heavy) are performed on the client, rather than on the server, which would bottleneck the whole system.
 
 What are the operations on tuples?
@@ -232,17 +252,25 @@ Transactions are not just about batching up operations into a more efficient pac
 
 Transactions give you a means of optimistic locking: the transaction proceeds in a way that depends on preconditions. See example/increment.rb for a very simple example. Not only can you make a transaction depend on the existence of a tuple, you can make the effect of the transaction a function of existing tuples (see example/transaction-logic.rb and example/broker-optimistic.rb).
 
-If you prefer classical tuplespace locking, you can simply use certain tuples as locks, using take/write to lock/unlock them. See the examples. If you have a lot of contention and want to avoid the thundering herd, see example/lock-mgr-with-queue.rb.
+If you prefer classical tuplespace locking, you can simply use certain tuples as locks, using take/write to lock/unlock them. See the examples, such as example/broker-locking.rb. If you have a lot of contention and want to avoid the thundering herd, see example/lock-mgr-with-queue.rb.
 
 If an optimistic transaction fails (for example, it is trying to take a tuple, but the tuple has just been taken by another transaction), then the transaction block is re-executed, possibly waiting for new matches to the templates. Application code must be aware of the possible re-execution of the block. This is better explained in the examples...
 
-Tupelo transactions are ACID in the following sense. They are Atomic and Isolated -- this is enforced by the transaction processing in each client. Consistency is enforced by the underlying message sequencer (each client's copy of the space is the deterministic result of the same sequence of operations). Durability is optional, but can be provided by the archiver (to be implemented) or other clients.
+Transactions have a significant disadvantage compared to lock tuples: a transaction can protect only resources that are represented in the tuplespace, whereas a lock can protect anything: a file, a device, a service, etc. This is because a transaction begins and ends within a single instant of logical (tuplespace) time, whereas a lock tuple can be taken out for an arbitrary duration of real time. Furthermore, the instant of logical time in which a transaction takes effect may occur at different wall-clock times on different processes, even on the same host.
+
+Tupelo transactions are ACID in the following sense. They are Atomic and Isolated -- this is enforced by the transaction processing in each client. Consistency is enforced by the underlying message sequencer: each client's copy of the space is the deterministic result of the same sequence of operations. Durability is optional, but can be provided by the archiver (to be implemented) or other clients.
 
 On the CAP spectrum, tupelo tends towards consistency.
 
 Tupelo transactions do not require two-phase commit, because they are less powerful than general transactions. Each client has enough information to decide (in the same way as all other clients) whether the transaction succeeds or fails. This has performance advantages, but imposes some limitations on transactions over subspaces that are known to one client but not another.
 
 
+Syntax
+======
+
+You can use tupelo with a simplified syntax, like a "domain-specific language". Each construct with a block can be used in either of two forms, with an explicit block param or without. Compare example/add-dsl.rb and example/add.rb.
+
+
 Advantages
 ==========
 
@@ -252,6 +280,8 @@ Speed (latency, throughput):
 
 * minimal system-wide bottlenecks
 
+* non-blocking socket reads
+
 * read -- local and hence very fast
 
 * write -- fast, pipelined (waiting for acknowledgement is optional);  
@@ -322,6 +352,8 @@ Tupelo also supports custom classes in tuples, but only with marshal / yaml; mus
 
 Both: tuples can be arrays or hashes.
 
+Spaces have an advantage over distributed hash tables: different clients may acccess tuples in terms of different dimensions. For example, a producer generates [producer_id, value]; a consumer looks for [nil, SomeParticularValues]. Separation of concerns, decoupling in the data space.
+
 
 To compare
 ----------
@@ -342,6 +374,7 @@ To compare
 
 * datomic -- similar distribution of "facts", but not tuplespace
 
+* job queues: sidekiq, resque, delayedjob, http://queues.io
 
 Architecture
 ============
@@ -354,7 +387,7 @@ Two central processes:
 
 Specialized clients:
 
-* archiver -- dumps tuplespace state to clients joining the system later than t=0
+* archiver -- dumps tuplespace state to clients joining the system later than t=0; at least one archiver is required, unless all clients start at t=0.
 
 * tup -- command line shell for accessing (and creating) tuplespaces
 
@@ -368,14 +401,14 @@ General application clients:
 
 * worker thread manages local tuplespace state and requests to modify or access it
 
-* client threads construct transactions and wait for results (communicating with the worker thread over queues)
+* client threads construct transactions and wait for results (communicating with the worker thread over queues); they may also use asynchronous transactions
 
 Protocol
 --------
 
 Nothing in the protocol specifies local searching or storage, or matching, or notification, or templating. That's all up to each client. The protocol only contains tuples and operations on them (take, write, pulse, read), combined into transactions.
 
-The protocol has two layers. The outer (message) layer is 6 fields, managed by the funl gem, using msgpack for serialization.
+The protocol has two layers. The outer (message) layer is 6 fields, managed by the funl gem, using msgpack for serialization. All socket reads are non-blocking, so a slow sender will not block other activity in the system.
 
 The inner (blob) layer manages one of those 6 field using msgpack (by default), marshal, json, or yaml. This layer contains the transaction operations. The blob is not unpacked by the server, only by clients.
 

data/Rakefile

@@ -29,14 +29,14 @@ namespace :test do
   desc "Run system tests"
   task :system do |t|
     FileList["test/system/*.rb"].each do |f|
-      ruby f
+      ruby "-Itest/lib", f
     end
   end
 
   desc "Run stress tests"
   task :stress do |t|
     FileList["test/stress/*.rb"].each do |f|
-      ruby f
+      ruby "-Itest/lib", f
     end
   end
 end
@@ -67,10 +67,16 @@ end
 namespace :release do
   desc "Diff to latest release"
   task :diff do
-    latest = `git describe --abbrev=0 --tags --match '#{PRJ}-*'`
+    latest = `git describe --abbrev=0 --tags --match '#{PRJ}-*'`.chomp
     sh "git diff #{latest}"
   end
 
+  desc "Log to latest release"
+  task :log do
+    latest = `git describe --abbrev=0 --tags --match '#{PRJ}-*'`.chomp
+    sh "git log #{latest}.."
+  end
+
   task :is_new_version do
     abort "#{tag} exists; update version!" unless `git tag -l #{tag}`.empty?
   end

data/bin/tspy

@@ -31,10 +31,15 @@ require 'tupelo/app'
 
 Tupelo.application do |app|
   app.local do |client|
+    trap :INT do
+      exit!
+    end
+
     note = client.notifier
-    client.log "%10s %10s %10s %s" % %w{ status tick client operation }
+    client.log "%10s %10s %10s %s" % %w{ tick client status operation }
     loop do
-      client.log "%10s %10d %10d %p" % note.wait
+      status, tick, cid, op = note.wait
+      client.log "%10d %10d %10s %p" % [tick, cid, status, op]
     end
   end
 end

data/bin/tup

@@ -155,7 +155,7 @@ EasyServe.start ez_opts do |ez|
       "starting shell. Commands: #{TupClient::CMD_ALIASES.join(", ")}"
     }
 
-    require 'tupelo/irb-shell'
+    require 'tupelo/app/irb-shell'
     IRB.start_session(client)
 
     client.stop

data/example/add-dsl.rb

@@ -0,0 +1,27 @@
+# A little ruby magic to sweeten the syntax. Compare add.rb
+#
+# The old syntax is still accepted. Switching between syntaxes is caused by the
+# presence of the '|...|' form. When present, it's important to keep in mind
+# that 'self' and instance vars inside the block are not the same as outside the
+# block. In other respects, such as local vars, these closures behave normally.
+# This is ruby's famous instance_eval gotcha.
+
+require 'tupelo/app'
+
+Tupelo.application do
+  child do
+    write ['x', 1]
+    write ['y', 2]
+  end
+  
+  child do
+    sum =
+      transaction do
+        _, x = take ['x', Numeric]
+        _, y = take ['y', Numeric]
+        x + y
+      end
+    
+    log "sum = #{sum}"
+  end
+end

data/example/boolean-match.rb

@@ -1,26 +1,9 @@
 require 'tupelo/app'
-
-class Tupelo::Client
-  class Or
-    attr_reader :templates
-
-    def initialize worker, templates
-      @templates = templates.map {|template| worker.make_template(template)}
-    end
-
-    def === obj
-      templates.any? {|template| template === obj}
-    end
-  end
-  
-  def or *templates
-    Or.new(worker, templates)
-  end
-end
+require 'tupelo/util/boolean'
 
 Tupelo.application do |app|
   app.local do |client|
-    tm = client.or [0..2, String], [3..5, Hash]
+    tm = client.match_any [0..2, String], [3..5, Hash]
     
     client.write(
       [0, "a"], [1, {b: 0}], [2, "c"],

data/example/broker-optimistic-v2.rb

@@ -0,0 +1,33 @@
+# Modified in two ways:
+# - use simplified block syntax
+# - make it fit in a transaction do..end block
+
+require 'tupelo/app'
+
+N_PLAYERS = 10
+
+Tupelo.application do
+  N_PLAYERS.times do
+    # sleep rand / 10 # reduce contention -- could also randomize inserts
+    child do
+      me = client_id
+      write name: me
+      
+      you = transaction do
+        game = read_nowait(
+          player1: nil,
+          player2: me)
+        break game["player1"] if game
+      
+        take_nowait(name: me) or fail!
+        you = take(name: nil)["name"]
+        write(
+          player1: me,
+          player2: you)
+        you
+      end
+
+      log "now playing with #{you}"
+    end
+  end
+end

data/example/broker-optimistic.rb

@@ -2,6 +2,8 @@
 # broker-locking.rb, but it has far fewer bottlenecks (try inserting some sleep
 # 1 calls), and it is not possible for a token to be lost (leaving the lock in a
 # locked state) if a process dies.
+# However, this version is vulnerable to contention. Try this: N_PLAYERS=40
+# and comment out the sleep line.
 
 require 'tupelo/app'
 
@@ -9,29 +11,27 @@ N_PLAYERS = 10
 
 Tupelo.application do |app|
   N_PLAYERS.times do
+    # sleep rand / 10 # reduce contention -- could also randomize inserts
     app.child do |client|
       me = client.client_id
       client.write name: me
-      you = nil
       
-      1.times do
-        begin
-          t = client.transaction
-          if t.take_nowait name: me
-            you = t.take(name: nil)["name"]
-            t.write(
-              player1: me,
-              player2: you)
-            t.commit.wait
-            break
-          end
-        rescue Tupelo::Client::TransactionFailure => ex
+      begin
+        t = client.transaction
+        if t.take_nowait name: me
+          you = t.take(name: nil)["name"]
+          t.write(
+            player1: me,
+            player2: you)
+          t.commit.wait
+        else
+          t.fail!
         end
-          
+      rescue Tupelo::Client::TransactionFailure => ex
         game = client.read_nowait(
           player1: nil,
           player2: me)
-        redo unless game
+        retry unless game
         you = game["player1"]
       end
 

data/example/dphil-optimistic-v2.rb

@@ -0,0 +1,37 @@
+# Dining philosopher example. Like dphil-optimistic.rb, but replaces
+# each take-write pair with a read. Same effect, inside a transaction,
+# and it is a little faster.
+
+require 'tupelo/app'
+
+N_PHIL = 5
+N_ITER = 10
+
+Tupelo.application do
+  N_PHIL.times do |i|
+    child do
+      log.progname << ": phil #{i}"
+      write ["eat", i, 0] # amount eaten
+
+      N_ITER.times do
+        transaction do
+          # lock the resource (in transaction, so optimistically):
+          read ["chopstick", i]
+          read ["chopstick", (i+1)%N_PHIL]
+
+          # use the resource:
+          _,_,count = take ["eat", i, nil]
+          write ["eat", i, count+1]
+        end
+      end
+      
+      log "ate #{read(["eat", i, nil])[2]}"
+    end
+  end
+  
+  local do
+    N_PHIL.times do |i|
+      write ["chopstick", i]
+    end
+  end
+end

data/example/dphil-optimistic.rb

@@ -0,0 +1,44 @@
+# Dining philosopher example. Like dphil.rb, but uses transactions and can work
+# correctly without the "room tickets", which are really global locks. We
+# optimistically let all philosophers dine at the same time. This version is
+# several times faster than the locking version (try increasing N_PHIL). Note,
+# however, that in the transaction-based version only operations within the
+# transaction are protected from concurrent users. That may not be suitable if
+# the intent of the lock is to protect some resource external to the tuplespace.
+
+require 'tupelo/app'
+
+N_PHIL = 5
+N_ITER = 10
+
+Tupelo.application do
+  N_PHIL.times do |i|
+    child do
+      log.progname << ": phil #{i}"
+      write ["eat", i, 0] # amount eaten
+
+      N_ITER.times do
+        transaction do
+          # lock the resource (in transaction, so optimistically):
+          c0 = take ["chopstick", i]
+          c1 = take ["chopstick", (i+1)%N_PHIL]
+
+          # use the resource:
+          _,_,count = take ["eat", i, nil]
+          write ["eat", i, count+1]
+
+          # release the resource:
+          write c0, c1
+        end
+      end
+      
+      log "ate #{read(["eat", i, nil])[2]}"
+    end
+  end
+  
+  local do
+    N_PHIL.times do |i|
+      write ["chopstick", i]
+    end
+  end
+end

data/example/dphil.rb

@@ -0,0 +1,42 @@
+# Dining philosopher example, based on:
+#
+# http://www.lindaspaces.com/book/chap9.htm
+
+require 'tupelo/app'
+
+N_PHIL = 5
+N_ITER = 10
+
+Tupelo.application do
+  N_PHIL.times do |i|
+    child do
+      log.progname << ": phil #{i}"
+      write ["eat", i, 0] # amount eaten
+
+      N_ITER.times do
+        # lock the resource:
+        ticket = take ["room ticket"]
+        c0 = take ["chopstick", i]
+        c1 = take ["chopstick", (i+1)%N_PHIL]
+        
+        # use the resource:
+        _,_,count = take ["eat", i, nil]
+        write ["eat", i, count+1]
+
+        # release the resource:
+        write c0, c1, ticket
+      end
+      
+      log "ate #{read(["eat", i, nil])[2]}"
+    end
+  end
+  
+  local do
+    N_PHIL.times do |i|
+      write ["chopstick", i]
+    end
+    (N_PHIL-1).times do
+      write ["room ticket"]
+    end
+  end
+end

data/example/lock-mgr-with-queue.rb

@@ -2,25 +2,15 @@
 # problem. You can observe this by seeing "FAILED" in the output of the former
 # but not in the latter. This means that competing offers are resolved by the
 # queue, rather than by propagating them to all clients.
+#
+# Run with --trace
 
 require 'tupelo/app'
 
 N = 3
 
 Tupelo.application do |app|
-  app.child do |client| # a debugger client, to see what's happening
-    note = client.notifier
-    puts "%4s %4s %10s %s" % %w{ tick cid status operation }
-    loop do
-      status, tick, cid, op = note.wait
-      unless status == :attempt
-        s = status == :failure ? "FAILED" : ""
-        puts "%4d %4d %10s %p" % [tick, cid, s, op]
-      end
-    end
-  end
-  
-  app.child do |client| # the lock manager
+  app.child passive: true do |client| # the lock manager
     client.log.progname << " (lock mgr)"
     waiters = Queue.new
 
@@ -50,7 +40,7 @@ Tupelo.application do |app|
     app.child do |client|
       client.write ["request", "resource", client.client_id, 0.5]
       
-      10.times do |j|
+      2.times do |j|
         # Now we are ouside of transaction, but still no other client may use
         # "resource" until lock expires (or is otherwise removed), as long
         # as all clients follow the read protocol below.
@@ -64,7 +54,7 @@ Tupelo.application do |app|
     end
   end
   
-  app.child do |client|
+  app.child passive: true do |client|
     # This client never even tries to lock the resource, so it cannot write.
     client.transaction do |t|
       t.read ["resource", client.client_id, nil]

data/example/lock-mgr.rb

@@ -1,21 +1,12 @@
+# Run with --trace
+
 require 'tupelo/app'
 
 N = 3
 
 Tupelo.application do |app|
-  app.child do |client| # a debugger client, to see what's happening
-    note = client.notifier
-    puts "%4s %4s %10s %s" % %w{ tick cid status operation }
-    loop do
-      status, tick, cid, op = note.wait
-      unless status == :attempt
-        s = status == :failure ? "FAILED" : ""
-        puts "%4d %4d %10s %p" % [tick, cid, s, op]
-      end
-    end
-  end
-  
-  app.child do |client| # the lock manager
+  app.child passive: true do |client| # the lock manager
+    client.log.progname << " (lock mgr)"
     loop do
       client.write ["resource", "none"]
       lock_id, client_id, duration = client.read ["resource", nil, nil]
@@ -35,9 +26,9 @@ Tupelo.application do |app|
         t.write ["resource", client.client_id, 0.5]
       end
       # Thundering herd -- all N clients can respond at the same time.
-      # A better example would have a queue -- see lock-mgr-with-queue.rb.
+      # Can be avoided with a queue -- see lock-mgr-with-queue.rb.
       
-      10.times do |j|
+      2.times do |j|
         # Now we are ouside of transaction, but still no other client may use
         # "resource" until lock expires (or is otherwise removed), as long
         # as all clients follow the read protocol below.
@@ -51,7 +42,7 @@ Tupelo.application do |app|
     end
   end
   
-  app.child do |client|
+  app.child passive: true do |client|
     # This client never even tries to lock the resource, so it cannot write.
     client.transaction do |t|
       t.read ["resource", client.client_id, nil]

data/example/map-reduce-v2.rb

@@ -1,41 +1,9 @@
 require 'tupelo/app'
+require 'tupelo/util/boolean'
 
 N = 2 # how many cpus do you want to use for mappers?
-VERBOSE = ARGV.delete "-v"
-
-class Tupelo::Client
-  class Or
-    attr_reader :templates
-
-    def initialize worker, templates
-      @templates = templates.map {|template| worker.make_template(template)}
-    end
-
-    def === obj
-      templates.any? {|template| template === obj}
-    end
-  end
-  
-  def or *templates
-    Or.new(worker, templates)
-  end
-end
 
 Tupelo.application do |app|
-  if VERBOSE
-    app.child do |client| # a debugger client, to see what's happening
-      note = client.notifier
-      puts "%4s %4s %10s %s" % %w{ tick cid status operation }
-      loop do
-        status, tick, cid, op = note.wait
-        unless status == :attempt
-          s = status == :failure ? "FAILED" : ""
-          puts "%4d %4d %10s %p" % [tick, cid, s, op]
-        end
-      end
-    end
-  end
-
   app.child do |client|
     document = "I will not map reduce in class\n" * 10
     lineno = 0
@@ -51,7 +19,7 @@ Tupelo.application do |app|
     results = Hash.new(0)
     lines_remaining = lineno
     results_remaining = 0
-    result_template = client.or(
+    result_template = client.match_any(
       {word: String, count: Integer},
       {lineno: Integer, result_count: Integer}
     )
@@ -70,12 +38,11 @@ Tupelo.application do |app|
       end
     end
 
-    client.log "DONE. results = #{results}"
-    client.log "Press ^C to exit"
+    client.log "results = #{results}"
   end
   
   N.times do |i|
-    app.child do |client|
+    app.child passive: true do |client|
       client.log.progname = "mapper #{i}"
       
       loop do