tupelo 0.5 → 0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1f296ad08b928805d512dc88cbdc0f5b1196fde6
-  data.tar.gz: f25b3ac8ef8fac55f5a9affc1b33cf42409aebc6
+  metadata.gz: 1bd2a467d93d42b9237dc027c9ee33160c7f44a3
+  data.tar.gz: 16c3d47bc0acd033f1f8be8bc4578310edb3f653
 SHA512:
-  metadata.gz: 2a88ccb3f784fc9106fb4d92c522fb26e61ae7ec0ad24fecbb805a94c4f01329ad717e035ed2f9fc8b795076aaec9c840d2601759452afb8502f61b7999ef86f
-  data.tar.gz: 4d3662603023f504618ea0f3f8e84ef076dce52f86150634ad4d8e540a610279f3604bb5071b6bd5e16c074bf330dcd7914f70397a52397e1966dae9d55ba8f3
+  metadata.gz: 1f20d07a716db5de35606c9500562863c64a5f3e74129336bfb9d52221b2751385d3d2c8a6ca873bd9841c403edafcc5911df776a143bf83a743d0d257af7ce6
+  data.tar.gz: 40b77909cf54701fabfaf35d189dd15f9ebc8beb4bbfa8a6a55b2cc6ee1eb512eab682c5c26df375e94a939d1a018a1f463e1400ae9131412819ebd09f4c093c

data/README.md

@@ -97,7 +97,7 @@ Getting started
 
   There is no guarantee that `x_final == x_optimistic`. The block may execute more than once.
 
-  Take a tuple matching a template, but only if a local match exist (otherwise return nil):
+  Take a tuple matching a template, but only if a local match exists (otherwise return nil):
 
         take_nowait <template>
 
@@ -105,12 +105,11 @@ Getting started
           ...
         end
 
-  Note that a local match is still not a guarantee of returning that tuple immediately -- another process may take it first, blocking this process.
-
+  Note that a local match is still not a guarantee of `x_final == x_optimistic`. Another process may take `x_optimistic` first, and the take will be re-executed. 
   Perform a general transaction:
 
         result =
-          tr do |t|           # tr is alias for transaction
+          transaction do |t|
             rval = t.read ... # optimistic value
             t.write ...
             t.pulse ...
@@ -136,11 +135,12 @@ Getting started
 
 6. Debugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful. There is also the similar --trace switch that is available to all bin and example programs, for example:
 
-      tick    cid status operation
-         1      2        batch write ["x", 1]
-         2      2        batch write ["y", 2]
-         3      3        atomic take ["x", 1], ["y", 2]
-
+  ```
+    tick    cid status operation
+       1      2        batch write ["x", 1]
+       2      2        batch write ["y", 2]
+       3      3        atomic take ["x", 1], ["y", 2]
+  ```
 
   The `Tupelo.application` command, provided by `tupelo/app`, is the source of all these options and is available to your programs.
 
@@ -155,7 +155,7 @@ See https://en.wikipedia.org/wiki/Tuple_space for general information and histor
 What is a tuple?
 ----------------
 
-A tuple is the unit of information in a tuplespace. It is immutable in the context of the tuplespace -- you can write a tuple into the space and you can read or take one from the space, but you cannot update a tuple within a space. A tuples does not have an identity other than the data it is made from.
+A tuple is the unit of information in a tuplespace. It is immutable in the context of the tuplespace -- you can write a tuple into the space and you can read or take one from the space, but you cannot update a tuple within a space. A tuple does not have an identity other than the data it contains. A tuplespace can contain multiple copies of the same tuple. (In the ruby client, two tuples are considered the same when they are #==.)
 
 A tuple is either an array:
 
@@ -182,8 +182,6 @@ In other words, a tuple is a fairly general object, though this depends on the s
 
 It's kind of like a "JSON object", except that in the json blob case, the hash keys can only be strings. In the msgpack case, keys have no special limitations. In the case of the marshal and yaml modes, tuples can contain many other kinds of objects.
 
-A tuplespace can contain multiple copies of the same tuple.
-
 What is a template?
 -------------------
 
@@ -222,7 +220,7 @@ A template doesn't have to be a tuple pattern with wildcards, though. It can be
     read_all Array
     read_all Object
 
-An optional library, `tupelo/util/boolean`, provides a #match_any method to construct the boolean or of other templates:
+An optional library, `tupelo/util/boolean`, provides a #match_any method to construct the boolean `or` of other templates:
 
     read_all match_any( [1,2,3], {foo: "bar"} )
 
@@ -237,7 +235,7 @@ What are the operations on tuples?
 
 * take - search the space for matching tuples, waiting if none found, removing the tuple if found
 
-* pulse - write and take the tuple; readers see it, but it cannot be taken (nto a classical tuplespace operation)
+* pulse - write and take the tuple; readers see it, but it cannot be taken by other client, and it cannot be read later (this is not a classical tuplespace operation, but is useful for publish-subscribe communication patterns)
 
 These operations have a few variations (wait vs nowait) and options (timeouts).
 
@@ -250,9 +248,9 @@ However, it may take some time to prepare the transaction. This is true in terms
 
 Transactions are not just about batching up operations into a more efficient package (though you can do that with the #batch api). A transaction makes the combined operations execute atomically: the transaction finishes only when all of its operations can be successfully performed. Writes and pulses can always succeed, but takes and reads only succeed if the tuples exist.
 
-Transactions give you a means of optimistic locking: the transaction proceeds in a way that depends on preconditions. See example/increment.rb for a very simple example. Not only can you make a transaction depend on the existence of a tuple, you can make the effect of the transaction a function of existing tuples (see example/transaction-logic.rb and example/broker-optimistic.rb).
+Transactions give you a means of optimistic locking: the transaction proceeds in a way that depends on preconditions. See [example/increment.rb](example/increment.rb) for a very simple example. Not only can you make a transaction depend on the existence of a tuple, you can make the effect of the transaction a function of existing tuples (see [example/transaction-logic.rb](example/transaction-logic.rb) and [example/broker-optimistic.rb](example/broker-optimistic.rb)).
 
-If you prefer classical tuplespace locking, you can simply use certain tuples as locks, using take/write to lock/unlock them. See the examples, such as example/broker-locking.rb. If you have a lot of contention and want to avoid the thundering herd, see example/lock-mgr-with-queue.rb.
+If you prefer classical tuplespace locking, you can simply use certain tuples as locks, using take/write to lock/unlock them. See the examples, such as [example/broker-locking.rb](example/broker-locking.rb). If you have a lot of contention and want to avoid the thundering herd, see [example/lock-mgr-with-queue.rb](example/lock-mgr-with-queue.rb).
 
 If an optimistic transaction fails (for example, it is trying to take a tuple, but the tuple has just been taken by another transaction), then the transaction block is re-executed, possibly waiting for new matches to the templates. Application code must be aware of the possible re-execution of the block. This is better explained in the examples...
 
@@ -260,7 +258,7 @@ Transactions have a significant disadvantage compared to lock tuples: a transact
 
 Tupelo transactions are ACID in the following sense. They are Atomic and Isolated -- this is enforced by the transaction processing in each client. Consistency is enforced by the underlying message sequencer: each client's copy of the space is the deterministic result of the same sequence of operations. Durability is optional, but can be provided by the archiver (to be implemented) or other clients.
 
-On the CAP spectrum, tupelo tends towards consistency.
+On the CAP spectrum, tupelo tends towards consistency: for all clients, write and take operations are applied in the same order, so the state of the entire system up through a given tick of discrete time is universally agreed upon. Of course, because of the difficulties of distributed systems, one client may not yet have seen the same range of ticks as another.
 
 Tupelo transactions do not require two-phase commit, because they are less powerful than general transactions. Each client has enough information to decide (in the same way as all other clients) whether the transaction succeeds or fails. This has performance advantages, but imposes some limitations on transactions over subspaces that are known to one client but not another.
 
@@ -268,7 +266,7 @@ Tupelo transactions do not require two-phase commit, because they are less power
 Syntax
 ======
 
-You can use tupelo with a simplified syntax, like a "domain-specific language". Each construct with a block can be used in either of two forms, with an explicit block param or without. Compare example/add-dsl.rb and example/add.rb.
+You can use tupelo with a simplified syntax, like a "domain-specific language". Each construct with a block can be used in either of two forms, with an explicit block param or without. Compare [example/add-dsl.rb](example/add-dsl.rb) and [example/add.rb](example/add.rb).
 
 
 Advantages
@@ -290,6 +288,8 @@ Speed (latency, throughput):
 
 Can use optimal data structure for each subspace of tuplespace.
 
+Decouples storage from query. (E.g. archiver for storage, optimized for just insert, delete, dump. And in-memory data structure, such as red-black tree, optimized for sorted query.)
+
 Each client can have its own matching agorithms and api -- matching is not part of the comm protocol, which is defined purely in terms of tuples.
 
 Data replication is easy--hard to avoid in fact.
@@ -348,7 +348,7 @@ Rinda has a severe bottleneck, though: all matching, waiting, etc. are performed
 
 Rinda is rpc-based, which is slower and also more vulnerable due to the extra client-server state; tupelo is imlemented on a message layer, rather than rpc. This also helps with pipelined writes.
 
-Tupelo also supports custom classes in tuples, but only with marshal / yaml; must define #==; see example/custom-class.rb
+Tupelo also supports custom classes in tuples, but only with marshal / yaml; must define #==; see [example/custom-class.rb](example/custom-class.rb)
 
 Both: tuples can be arrays or hashes.
 
@@ -372,10 +372,12 @@ To compare
 
 * lmax -- minimal spof
 
-* datomic -- similar distribution of "facts", but not tuplespace
+* datomic -- similar distribution of "facts", but not tuplespace; similar use of pluggable storage managers
 
 * job queues: sidekiq, resque, delayedjob, http://queues.io
 
+* pubsubs: kafka
+
 Architecture
 ============
 

data/example/broker-optimistic-v2.rb

@@ -11,20 +11,24 @@ Tupelo.application do
     # sleep rand / 10 # reduce contention -- could also randomize inserts
     child do
       me = client_id
+      you = nil
       write name: me
       
-      you = transaction do
+      transaction do
         game = read_nowait(
           player1: nil,
           player2: me)
-        break game["player1"] if game
+
+        if game
+          you = game["player1"]
+          break
+        end
       
         take_nowait(name: me) or fail!
         you = take(name: nil)["name"]
         write(
           player1: me,
           player2: you)
-        you
       end
 
       log "now playing with #{you}"

data/example/broker-queue.rb

@@ -0,0 +1,35 @@
+# more like how you would do it in redis, except that the queue is not stored in
+# the central server, so operations on it are not a bottleneck, FWIW
+
+require 'tupelo/app'
+
+N_PLAYERS = 10
+
+Tupelo.application do
+  N_PLAYERS.times do
+    # sleep rand / 10 # reduce contention -- could also randomize inserts
+    child do
+      me = client_id
+      write name: me
+      
+      you = transaction do
+        game = read_nowait(
+          player1: nil,
+          player2: me)
+        break game["player1"] if game
+      
+        unless take_nowait name: me
+          raise Tupelo::Client::TransactionFailure
+        end
+
+        you = take(name: nil)["name"]
+        write(
+          player1: me,
+          player2: you)
+        you
+      end
+
+      log "now playing with #{you}"
+    end
+  end
+end

data/example/load-balancer.rb

@@ -0,0 +1,51 @@
+require 'tupelo/app'
+
+N_WORKERS = 10
+N_CLIENTS = 2
+
+Tupelo.application do
+  
+  N_WORKERS.times do |i|
+    child passive: true do
+      log.progname = "worker #{i}"
+      worker_delay = rand 1.0..3.0 # how long it takes this worker to do a task
+
+      loop do
+        _, req_id, req_dat = take ["request", Integer, nil]
+          # we could modify this client's tuplespace storage so that
+          # the requst are taken in req_id order
+
+        log.info "handling request #{req_id} for #{req_dat.inspect}"
+        sleep worker_delay
+        write ["response", req_id]
+        log.info "handled request #{req_id} for #{req_dat.inspect}"
+      end
+    end
+  end
+  
+  N_CLIENTS.times do |i|
+    child do
+      log.progname = "client #{i}"
+
+      req_data = 1..20 # task spec here
+      
+      req_data.each do |req_dat|
+        req_id = nil
+        transaction do
+          # grouping the following ops in a transaction is not necessary for
+          # correctness, but it does reduce latency
+          _, req_id = take ["next_req_id", Integer]
+          write ["next_req_id", req_id + 1]
+          write ["request", req_id, req_dat]
+        end
+
+        take ["response", req_id]
+        log "got response for request #{req_id}"
+      end
+    end
+  end
+  
+  local do
+    write ["next_req_id", 0]
+  end
+end

data/example/message-bus.rb

@@ -0,0 +1,59 @@
+# Asynchronously deliver messages via a shared "data bus" aka "data hub".
+# A message is written to a channel. It remains there until a new value is
+# written to the channel. A reader can pick it up at any time. Compare
+# pubsub.rb.
+#
+# Run with the --show-final-state switch to verify that the last tuple
+# written to a channel does stay there.
+
+require 'tupelo/app'
+
+show_final_state = ARGV.delete "--show-final-state"
+
+N_PUBS = 6
+N_SUBS = 6
+N_CHAN = 3
+
+Tupelo.application do
+  channels = (0...N_CHAN).map {|i| "channel #{i}"}
+  
+  N_PUBS.times do |pi|
+    child do
+      read ['start']
+      delay = pi
+      sleep delay
+      ch = channels[ pi % channels.size ]
+      transaction do
+        take_nowait [ch, nil]
+        write [ch, "pub #{pi} slept for #{delay} sec"]
+      end
+    end
+  end
+
+  N_SUBS.times do |si|
+    child passive: true do # passive means process will exit when pubs exit
+      log.progname = "sub #{si}"
+      ch = channels[ si % channels.size ]
+      loop do
+        sleep 1
+        t = read_nowait [ch, nil]
+        log t if t
+      end
+    end
+  end
+  
+  if show_final_state
+    child passive: true do
+      log.progname = "final"
+      def self.stop
+        log read_all
+        super
+      end
+      sleep
+    end
+  end
+
+  local do
+    write ['start']
+  end
+end

data/example/parallel.rb

@@ -0,0 +1 @@
+# like gnu parallel

data/example/pubsub.rb

@@ -0,0 +1,53 @@
+# Synchronously deliver published messages to subscribers. Subscriber gets the
+# message only if waiting while the message is sent.
+# Compare message-bus.rb.
+#
+# Run with the --show-final-state switch to verify that published tuples
+# don't stay in the space.
+
+require 'tupelo/app'
+
+show_final_state = ARGV.delete "--show-final-state"
+
+N_PUBS = 6
+N_SUBS = 6
+N_CHAN = 3
+
+Tupelo.application do
+  channels = (0...N_CHAN).map {|i| "channel #{i}"}
+  
+  N_PUBS.times do |pi|
+    child do
+      read ['start']
+      delay = pi/10.0
+      sleep delay
+      ch = channels[ pi % channels.size ]
+      pulse [ch, "pub #{pi} slept for #{delay} sec"]
+    end
+  end
+
+  N_SUBS.times do |si|
+    child passive: true do # passive means process will exit when pubs exit
+      log.progname = "sub #{si}"
+      ch = channels[ si % channels.size ]
+      loop do
+        log read [ch, nil]
+      end
+    end
+  end
+  
+  if show_final_state
+    child passive: true do
+      log.progname = "final"
+      def self.stop
+        log read_all
+        super
+      end
+      sleep
+    end
+  end
+
+  local do
+    write ['start']
+  end
+end

data/example/pulse.rb

@@ -1,3 +1,5 @@
+# See pubsub.rb for a more interesting use of pulse.
+
 require 'tupelo/app'
 
 Tupelo.application do |app|

data/example/remote.rb

@@ -0,0 +1,28 @@
+# simple use of remote to start process on remote host and connect it to
+# the tupelo app
+
+require 'tupelo/app/remote'
+
+host = ARGV.shift or abort "usage: #$0 <ssh-hostname>"
+
+Tupelo.tcp_application do
+  remote host: host do
+    write host: `hostname`.chomp, mode: "drb", client: client_id
+      # this actually returns local hostname, because the block executes
+      # locally -- only the tupelo ops are remote. So this mode is really
+      # only for examples and tests, not production.
+  end
+  
+  remote host: host, eval: %{
+    write host: `hostname`.chomp, mode: "eval", client: client_id
+  }
+  # rather than embed large chunks of code in the string, it's better to
+  # load or require a file and pass self (which is a Client instance) to
+  # a method in that file.
+  
+  local do
+    2.times do
+      log take host: nil, mode: nil, client: nil
+    end
+  end
+end

data/example/take-many.rb

@@ -0,0 +1,14 @@
+# This is a simple way to take multiple tuples at once.
+
+require 'tupelo/app'
+
+Tupelo.application do
+  child do
+    result = transaction { (1..3).map {|i| take [i]} }
+    log result
+  end
+  
+  child do
+    write [2], [1], [3]
+  end
+end

data/example/take-nowait.rb

@@ -0,0 +1,20 @@
+# Run this with --trace to see that, even in the FAIL case, take_nowait never
+# hangs waiting for a match. The "ready" tuple is just to keep the take
+# requests fairly close in time, increasing the chance of transaction failure.
+
+require 'tupelo/app'
+
+Tupelo.application do
+  20.times do
+    child do
+      read ["ready"]
+      r = take_nowait [1]
+      log "winner! result = #{r.inspect}" if r
+    end
+  end
+
+  local do
+    write [1]
+    write ["ready"]
+  end
+end

data/lib/tupelo/app/remote.rb

@@ -0,0 +1,68 @@
+require 'tupelo/app'
+
+module Tupelo
+  class AppBuilder
+    # Perform client operations on another host.
+    #
+    # There are two modes: drb and eval. In the drb mode, a block executes
+    # locally, performing tuplespace operations by proxy to the remote
+    # tupelo client instance. Results of the ops are available in the local
+    # process. This is very useful for testing and examples.
+    #
+    # In the eval mode, a string is evaluated on the remote host in the context
+    # of a client instance.
+    #
+    # There are some minor limitations compared to #child:
+    #
+    # In eval mode, the code string is always treated as in the arity 0 case
+    # of #child, in other words "DSL mode", in other words the self is the
+    # client.
+    #
+    # Unlike #child, there is no mode that returns a Client instance.
+    #
+    def remote client_class = Client,
+        client_lib: 'tupelo/client', host: nil, **opts
+      require 'easy-serve/remote'
+      raise if opts[:passive] ## not supported yet
+      ## detach option so that remote process doesn't keep ssh connection
+      snames = :seqd, :cseqd, :arcd
+
+      if opts[:eval]
+        ez.remote *snames, host: host, **opts, eval: %{
+          require #{client_lib.inspect}
+          
+          seqd, cseqd, arcd = *conns
+          client_class = Object.const_get(#{client_class.name.inspect})
+
+          begin
+            log.progname = "client <starting in \#{log.progname}>"
+            client = client_class.new(
+              seq: seqd, cseq: cseqd, arc: arcd, log: log)
+            client.start do
+              log.progname = "client \#{client.client_id}"
+            end
+            client.instance_eval #{opts[:eval].inspect}
+          ensure
+            client.stop if client
+          end
+        }
+
+      elsif block_given?
+        block = Proc.new
+        ez.remote *snames, host: host, **opts do |seqd, cseqd, arcd|
+          run_client client_class,
+              seq: seqd, cseq: cseqd, arc: arcd, log: log do |client|
+            if block.arity == 0
+              client.instance_eval &block
+            else
+              yield client
+            end
+          end
+        end
+
+      else
+        raise ArgumentError, "cannot select remote mode based on arguments"
+      end
+    end
+  end
+end

data/lib/tupelo/app.rb

@@ -75,6 +75,16 @@ module Tupelo
     end
   end
 
+  # same as application, but with tcp sockets the default
+  def self.tcp_application argv: ARGV,
+        servers_file: nil, blob_type: nil,
+        seqd_addr:  [:tcp, nil, 0],
+        cseqd_addr: [:tcp, nil, 0],
+        arcd_addr:  [:tcp, nil, 0], &block
+    application argv: ARGV, servers_file: servers_file, blob_type: blob_type,
+      seqd_addr: seqd_addr, cseqd_addr: cseqd_addr, arcd_addr: arcd_addr, &block
+  end
+
   #blob_type: 'msgpack' # the default
   #blob_type: 'marshal' # if you need to pass general ruby objects
   #blob_type: 'yaml' # less general ruby objects, but cross-language

data/lib/tupelo/version.rb

@@ -1,3 +1,3 @@
 module Tupelo
-  VERSION = "0.5"
+  VERSION = "0.6"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tupelo
 version: !ruby/object:Gem::Version
-  version: '0.5'
+  version: '0.6'
 platform: ruby
 authors:
 - Joel VanderWerf
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-07-31 00:00:00.000000000 Z
+date: 2013-08-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: atdo
@@ -79,6 +79,7 @@ files:
 - README.md
 - COPYING
 - Rakefile
+- lib/tupelo/app/remote.rb
 - lib/tupelo/app/irb-shell.rb
 - lib/tupelo/app/trace.rb
 - lib/tupelo/client.rb
@@ -96,9 +97,11 @@ files:
 - bench/pipeline.rb
 - bugs/write-read.rb
 - bugs/take-write.rb
+- example/pubsub.rb
 - example/timeout-trans.rb
 - example/tiny-client.rb
 - example/add.rb
+- example/parallel.rb
 - example/app-and-tup.rb
 - example/small.rb
 - example/bounded-retry.rb
@@ -124,7 +127,10 @@ files:
 - example/fail-and-retry.rb
 - example/dphil-optimistic-v2.rb
 - example/broker-optimistic-v2.rb
+- example/remote.rb
+- example/take-nowait.rb
 - example/optimist.rb
+- example/message-bus.rb
 - example/balance-xfer-locking.rb
 - example/increment.rb
 - example/custom-class.rb
@@ -133,8 +139,11 @@ files:
 - example/broker-optimistic.rb
 - example/notify.rb
 - example/small-simplified.rb
+- example/broker-queue.rb
 - example/async-transaction.rb
 - example/boolean-match.rb
+- example/load-balancer.rb
+- example/take-many.rb
 - example/dphil.rb
 - test/lib/testable-worker.rb
 - test/lib/mock-seq.rb