tupelo 0.14 → 0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f4eb65df5eb526b3a885d5f0ce41736bb3ccc216
-  data.tar.gz: 96f7e2b11e86b299af62b6598081564e98ceb159
+  metadata.gz: 79bc191b109fc7e11c3200a9e993725299558ead
+  data.tar.gz: 736b92e95728520e46c0516e857a26a2fd74e766
 SHA512:
-  metadata.gz: bd91077b980c38858cb3b4549cec5df43ccb91a58bc209c026fe9db23c9ec6a90486c1cecd8f92c564d420fa26ef00cda93fa6b92e6e9af171d9abd23af78717
-  data.tar.gz: b5de5df9f3e7baf6a02a4440b05e0cbb9a3fa04d37d5e46db3e0950ff5d59e29c23ac49809a9810f94fdd1a54916c47c98ce05a651167e72c9cfe8d60bd227ea
+  metadata.gz: 33db9c43e57b0405190165d965a09099b842dcc7bab9767ee7ec9f3ac11536de37fd8cc56f97709daa1c4c40c69a4f9055634ef318c90211148856c38a38adb9
+  data.tar.gz: a0b6f65625f735ec0b24bb06a02f07102661bf45b483a3b013ac7cafb4b3fceb27ac10711a7311638d2e305b2a522ea7eea1951739255ce0b31de10ee9bbe338

data/README.md

@@ -1,5 +1,3 @@
-**NEWS**: Come hear a talk on Tupelo on December 11 in San Francisco at the [SF Distributed Computing meetup](http://www.meetup.com/San-Francisco-Distributed-Computing/events/153886592/). Abstract: [doc/sfdc.txt](doc/sfdc.md).
-
 tupelo
 ==
 
@@ -25,6 +23,7 @@ Documentation
 
 * [FAQ](doc/faq.md)
 * [Subspaces](doc/subspace.md)
+* [Abstract](sfdc.md) and [slides](doc/sfdc.pdf) for San Francisco Distributed Systems meetup
 
 Getting started
 ==========
@@ -78,7 +77,12 @@ Getting started
 
         read_nowait <template>
 
-  Read all tuples matching a template, no waiting:
+  Note that neither #read nor #read_nowait wait for any previously issued writes to complete. The difference is that #read waits for a match to exist and #read_nowait does not. Compare:
+  
+        write [1]; read_nowait [1]        # ==> nil, probably
+        write [2]; read [2]               # ==> [2]
+
+  Read all tuples matching a template, no waiting (like #read_nowait):
 
         ra <template>
         read_all <template>
@@ -144,6 +148,17 @@ Getting started
           p t.read_nowait [3] # => nil
         end
 
+  Be careful about context within the do...end. If you omit the `|t|` block argument, then all operations are automatically scoped to the transaction, rather than the client. The following is equivalent to the previous example:
+  
+        client = self # local var that we can use inside the block
+        transaction do
+          write [3]
+          p read [3]
+          p client.read_all
+          take [3]
+          p read_nowait [3]
+        end
+
   You can timeout a transaction:
   
         transaction timeout: 1 do
@@ -162,11 +177,11 @@ Getting started
 
 4. Run tup with a server file so that two sessions can interact. Do this in two terminals in the same dir:
 
-        $ tup svr
+        $ tup sv
 
-  (The 'svr' argument names a file that the first instance of tup uses to store information like socket addresses and the second instance uses to connect. The first instance starts the servers as child processes. However, both instances appear in the terminal as interactive shells.)
+  (The 'sv' argument names a file that the first instance of tup uses to store information like socket addresses and the second instance uses to connect. The first instance starts the servers as child processes. However, both instances appear in the terminal as interactive shells.)
   
-  To do this on two hosts, copy the svr file and edit its hostname params as needed.
+  To do this on two hosts, copy the sv file and, if necessary, edit its connect_host field.
 
 5. Look at the examples. You may need to dig a bit to find the gem installation. For example:
 
@@ -176,17 +191,17 @@ Getting started
 
 6. Debugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful; it shows all tuplespace events in sequence that they occur. For example, run
 
-        $ tspy svr
+        $ tspy sv
 
-  in another terminal after running `tup svr`. The output shows the clock tick, sending client, operation, and operation status (success or failure).
+  in another terminal after running `tup sv`. The output shows the clock tick, sending client, operation, and operation status (success or failure).
 
   There is also the similar --trace switch that is available to all bin and example programs. This turns on diagnostic output for each transaction. For example:
 
   ```
     tick    cid status operation
-       1      2        batch write ["x", 1]
-       2      2        batch write ["y", 2]
-       3      3        atomic take ["x", 1], ["y", 2]
+       1      2        write ["x", 1]
+       2      2        write ["y", 2]
+       3      3        take ["x", 1], ["y", 2]
   ```
 
   The `Tupelo.application` command, provided by `tupelo/app`, is the source of all these options and is available to your programs. It's a kind of lightweight process deployment and control framework; however `Tupelo.application` is not necessary to use tupelo.
@@ -303,7 +318,7 @@ Transactions combine operations into a group that take effect at the same instan
 
 However, it may take some time to prepare the transaction. This is true in terms of both real time (clock and process) and logical time (global sequence of operations). Preparing a transaction means finding tuples that match the criteria of the read and take operations. Finding tuples may require searching (locally) for tuples, or waiting for new tuples to be written by others. Also, the transaction may fail even after matching tuples are found (when another process takes tuples of interest). Then the transaction needs to be prepared again. Once prepared, transaction is sent to all clients, where it may either succeed (in all clients) or fail (for the same reason as before--someone else grabbed one of our tuples). If it fails, then the preparation begins again. A transaction guarantees that, when it completes, all the operations were performed on the tuples at the same logical time. It does not guarantee that the world stands still while one process is inside the `transaction {...}` block.
 
-Transactions are not just about batching up operations into a more efficient package (though you can do that with the #batch api). A transaction makes the combined operations execute atomically: the transaction finishes only when all of its operations can be successfully performed. Writes and pulses can always succeed, but takes and reads only succeed if the tuples exist.
+Transactions are not just about batching up operations into a more efficient package. A transaction makes the combined operations execute atomically: the transaction finishes only when all of its operations can be successfully performed. Writes and pulses can always succeed, but takes and reads only succeed if the tuples exist.
 
 Transactions give you a means of optimistic locking: the transaction proceeds in a way that depends on preconditions. See [example/increment.rb](example/increment.rb) for a very simple example. Not only can you make a transaction depend on the existence of a tuple, you can make the effect of the transaction a function of existing tuples (see [example/transaction-logic.rb](example/transaction-logic.rb) and [example/broker-optimistic.rb](example/broker-optimistic.rb)).
 

data/bin/tspy

@@ -3,18 +3,18 @@
 if ARGV.delete("-h") or ARGV.delete("--help")
   puts <<-END
     Usage:
-      #$0 servers_file
+      #$0 services_file
 
-      Connect to the tuplespace specified by the servers_file and
+      Connect to the tuplespace specified by the services_file and
       use the notification api to print out all events.
       
       For example, you can start a tup with
       
-          tup svr
+          tup sv
       
       and then in another terminal
       
-          tspy svr
+          tspy sv
 
     Options:
     

data/bin/tup

@@ -5,21 +5,21 @@ if ARGV.delete("-h") or ARGV.delete("--help")
     Usage:
       #$0
 
-      #$0 servers_file
-      #$0 servers_file <script...
+      #$0 services_file
+      #$0 services_file <script...
 
-      #$0 servers_file unix [path]
-      #$0 servers_file tcp [host [port]]]
+      #$0 services_file unix [path]
+      #$0 services_file tcp [host [port]]]
     
-    The first form starts a tuplespace server as a child process. Then it
+    The first form starts a tuplespace service as a child process. Then it
     enters an interactive session in which the current object is the proxy to
     that tuplespace. This form is useful for an isolated tuplespace for
     simple experiments.
     
-    The second form tries to open the servers_file. If it cannot, then as in the
-    first form, it starts a tuplespace server child process and writes its
-    address to servers_file. If it can open the servers_file, then it simply
-    connects to the referenced tuplespace server. In either case, as in the
+    The second form tries to open the services_file. If it cannot, then as in
+    the first form, it starts a tuplespace service child process and writes its
+    address to services_file. If it can open the services_file, then it simply
+    connects to the referenced tuplespace service. In either case, as in the
     first form, an interactive session starts. This form (in its two variants)
     is useful for starting two sessions operating on the same tuplespace.
 
@@ -29,13 +29,13 @@ if ARGV.delete("-h") or ARGV.delete("--help")
     transcripts.
 
     The fourth and fifth forms are like the previous, but can be used to expose
-    the server on more or less public sockets with specified addresses. In the
-    tcp case, the servers_file can be copied to other hosts and used with tup
-    to connect to the servers (adjust the host references as needed). The
+    the service on more or less public sockets with specified addresses. In the
+    tcp case, the services_file can be copied to other hosts and used with tup
+    to connect to the services (adjust the host references as needed). The
     default for unix is, as in the first three forms, a path in a tmpdir. The
     default for tcp is localhost with port 0, and hence a dynamically chosen
-    port. These forms are only for starting a new server; connecting to an
-    existing server uses the simpler form "#$0 servers_file".
+    port. These forms are only for starting a new service; connecting to an
+    existing service uses the simpler form "#$0 services_file".
 
     Options:
     
@@ -62,9 +62,10 @@ if ARGV.delete("-h") or ARGV.delete("--help")
       
       --persist-dir DIR
                   load and save tuplespace to DIR
+                    (only needs to be set on first tup invocation)
 
       --use-subspaces
-                  enable subspaces for this tupelo server
+                  enable subspaces for this tupelo service
                     (only needs to be set on first tup invocation)
       --subscribe TAG,TAG,...
                   subscribe to specified subspaces; use "" for none
@@ -86,13 +87,22 @@ if i=argv.index("--subscribe") # default is to subscribe to all
   subscribed_tags = argv.delete_at(i).split(",")
 end
 
-servers_file = argv.shift
-addr = argv.shift(3)
+services_file = argv.shift
+proto = (argv.shift || :unix).to_sym
+addr = {proto: proto}
+case proto
+when :unix
+  addr[:path] = argv.shift
+when :tcp
+  addr[:bind_host] = argv.shift
+  addr[:port] = argv.shift
+  addr[:port] = Integer(addr[:port]) if addr[:port]
+end
 
 Tupelo.application(
   argv: argv,
   **tupelo_opts,
-  servers_file: servers_file,
+  services_file: services_file,
   seqd_addr: addr,
   cseqd_addr: addr, # using same addr causes autoincrement of port/filename
   arcd_addr: addr) do

data/example/app-and-tup.rb

@@ -1,7 +1,7 @@
 # It's very easy to connect tup to an existing app.
 # Just run this file, and then do (in another terminal):
 #
-# ../bin/tup servers-nnnn.yaml
+# ../bin/tup services-nnnn.yaml
 #
 # where nnnn is determined by looking in this dir. You can also
 # set the filename explicitly (first ARGV), rather than let it be generated
@@ -17,10 +17,10 @@
 
 require 'tupelo/app'
 
-filename = "servers-#$$.yaml"
+filename = "services-#$$.yaml"
 puts "run this in another shell: tup #{filename}"
 
-Tupelo.application servers_file: filename do
+Tupelo.application services_file: filename do
   child do
     loop do
       transaction do

data/example/broker-queue.rb

@@ -0,0 +1,35 @@
+# more like how you would do it in redis, except that the queue is not stored in
+# the central server, so operations on it are not a bottleneck, FWIW
+
+require 'tupelo/app'
+
+N_PLAYERS = 10
+
+Tupelo.application do
+  N_PLAYERS.times do
+    # sleep rand / 10 # reduce contention -- could also randomize inserts
+    child do
+      me = client_id
+      write name: me
+      
+      you = transaction do
+        game = read_nowait(
+          player1: nil,
+          player2: me)
+        break game["player1"] if game
+      
+        unless take_nowait name: me
+          raise Tupelo::Client::TransactionFailure
+        end
+
+        you = take(name: nil)["name"]
+        write(
+          player1: me,
+          player2: you)
+        you
+      end
+
+      log "now playing with #{you}"
+    end
+  end
+end

data/example/chat/chat-nohistory.rb

@@ -2,7 +2,7 @@
 
 require 'tupelo/app'
 
-svr = "chat-nohistory.yaml"
+sv = "chat-nohistory.yaml"
 
 Thread.abort_on_exception = true
 
@@ -13,7 +13,7 @@ def display_message msg
   puts "#{from}@#{time_str}> #{line}"
 end
 
-Tupelo.tcp_application servers_file: svr do
+Tupelo.tcp_application services_file: sv do
   me = argv.shift
 
   local do

data/example/chat/chat.rb

@@ -15,7 +15,7 @@
 
 require 'tupelo/app'
 
-svr = "chat.yaml"
+sv = "chat.yaml"
 history_period = 60 # seconds -- discard _my_ messages older than this
 
 Thread.abort_on_exception = true
@@ -27,7 +27,7 @@ def display_message msg
   puts "#{from}@#{time_str}> #{line}"
 end
 
-Tupelo.tcp_application servers_file: svr do
+Tupelo.tcp_application services_file: sv do
   me = argv.shift
 
   local do

data/example/child-of-child.rb

@@ -0,0 +1,34 @@
+require 'tupelo/app'
+
+### need a programmatic way to start up clients
+
+Tupelo.application do |app|
+
+  app.child do ## local still hangs
+    3.times do |i|
+      app.child do
+        write [i]
+        log "wrote #{i}"
+      end
+    end
+
+    3.times do
+      log take [nil]
+    end
+  end
+end
+
+__END__
+
+this hangs sometimes but not always:
+
+  tick    cid status operation
+A: client 3: wrote 0
+A: client 4: wrote 1
+     1      3        batch write [0]
+     2      4        batch write [1]
+A: client 2: [0]
+     3      2        atomic take [0]
+     4      2        atomic take [1]
+A: client 2: [1]
+A: client 5: wrote 2

data/example/fish01.rb

@@ -0,0 +1,48 @@
+# This works, but requires a fix-up step.
+
+require 'tupelo/app'
+
+Tupelo.application do
+  2.times do
+    child passive: true do
+      loop do
+        fish = nil
+
+        transaction do
+          fish, _ = take([String])
+          n, _ = take_nowait([Integer, fish])
+          if n
+            write [n + 1, fish]
+          else
+            write [1, fish] # another process might also write this, so ...
+          end
+        end
+        ### what if both processes die here?
+        transaction do # ... fix up the two tuples.
+          n1, _ = take_nowait [Integer, fish]; abort unless n1
+          n2, _ = take_nowait [Integer, fish]; abort unless n2
+          #log "fixing: #{[n1 + n2, fish]}"
+          write [n1 + n2, fish]
+        end
+      end
+    end
+  end
+
+  local do
+    seed = 3
+    srand seed
+    log "seed = #{seed}"
+    
+    fishes = %w{ trout marlin char salmon }
+
+    a = fishes * 10
+    a.shuffle!
+    a.each do |fish|
+      write [fish]
+    end
+
+    fishes.each do |fish|
+      log take [10, fish]
+    end
+  end
+end

data/example/map-reduce/remote-map-reduce.rb

@@ -2,12 +2,14 @@
 
 require 'tupelo/app/remote'
 
+tunnel = !!ARGV.delete("--tunnel")
+
 hosts = ARGV.shift or abort "usage: #$0 <ssh-hostname>,<ssh-hostname>,..."
 hosts = hosts.split(",")
 
 Tupelo.tcp_application do
   hosts.each do |host|
-    remote host: host, passive: true, eval: %{
+    remote host: host, passive: true, tunnel: tunnel, eval: %{
       loop do
         len = take([String])[0].size
         write [len]

data/example/pregel/dist-opt.rb

@@ -0,0 +1,15 @@
+#
+# Minor optimization:
+
+class KeyMatcher
+  def initialize i, n
+    @i = i
+    @n = n
+  end
+
+  def === id
+    id % @n == @i
+  end
+end
+
+vertex = take id: v_id_matcher, step: step, rank: nil, active: true

data/example/small.rb

@@ -12,18 +12,18 @@ log_level = case
   else Logger::WARN
 end
 
-EasyServe.start(servers_file: "small-servers.yaml") do |ez|
+EasyServe.start(services_file: "small-services.yaml") do |ez|
   log = ez.log
   log.level = log_level
   log.progname = "parent"
 
-  ez.start_servers do
+  ez.start_services do
     arc_to_seq_sock, seq_to_arc_sock = UNIXSocket.pair
     arc_to_cseq_sock, cseq_to_arc_sock = UNIXSocket.pair
     
-    ez.server :seqd do |svr|
+    ez.service :seqd do |sv|
       require 'funl/message-sequencer'
-      seq = Funl::MessageSequencer.new svr, seq_to_arc_sock, log: log,
+      seq = Funl::MessageSequencer.new sv, seq_to_arc_sock, log: log,
         blob_type: 'msgpack' # the default
         #blob_type: 'marshal' # if you need to pass general ruby objects
         #blob_type: 'yaml' # less general ruby objects, but cross-language
@@ -31,15 +31,15 @@ EasyServe.start(servers_file: "small-servers.yaml") do |ez|
       seq.start
     end
     
-    ez.server :cseqd do |svr|
+    ez.service :cseqd do |sv|
       require 'funl/client-sequencer'
-      cseq = Funl::ClientSequencer.new svr, cseq_to_arc_sock, log: log
+      cseq = Funl::ClientSequencer.new sv, cseq_to_arc_sock, log: log
       cseq.start
     end
 
-    ez.server :arcd do |svr|
+    ez.service :arcd do |sv|
       require 'tupelo/archiver'
-      arc = Tupelo::Archiver.new svr,
+      arc = Tupelo::Archiver.new sv,
         seq: arc_to_seq_sock, cseq: arc_to_cseq_sock, log: log
       arc.start
     end

data/example/subspaces/addr-book-v1.rb

@@ -0,0 +1,106 @@
+## TODO
+##
+## scaling params
+
+require 'tupelo/app'
+
+ab_tag = "my address book"
+ab_sort_field = 1
+ab_val_field = 2
+cmd_tag = "#{ab_tag} commands"
+resp_tag = "#{ab_tag} responses"
+
+Tupelo.application do
+  local do
+    use_subspaces!
+
+    # Subspace for tuples belonging to the addr book.
+    define_subspace(
+      tag:          ab_tag,
+      template:     [
+        {value: ab_tag},
+        {type:  "string"},  # name <-- ab_sort_field references this field
+        nil                 # address; can be any object <-- ab_val_field
+      ]
+    )
+    
+    # Subspace for commands for fetch and delete.
+    # We can't use #read and #take because then the requesting client
+    # would have to subscribe to the ab_tag subspace.
+    define_subspace(
+      tag:          cmd_tag,
+      template:     [
+        {value: cmd_tag},
+        {type:  "string"},  # cmd name
+        {type:  "list"}     # arguments
+      ]
+    )
+
+    # Subspace for responses to commands. Identify the command this is in
+    # response to by copying it (alternately, could use ids).
+    define_subspace(
+      tag:          resp_tag,
+      template:     [
+        {value: resp_tag},
+        {type:  "string"},  # cmd name
+        {type:  "list"},    # arguments
+        nil                 # result of query -- type depends on command
+      ]
+    )
+  end
+
+  ## Could set N_SORTED_SET_SPACE > 1, but lookups are so fast it would
+  ## just lead to contention and redundant computation. Redundancy is useful
+  ## though.
+
+  # Inserts are just writes, which are handled by Worker and SortedSetSpace,
+  # so this child's app loop only needs to handle special commands: fetch and
+  # delete, which are delegated to the SortedSetSpace.
+  child tuplespace: [SortedSetSpace, ab_tag, ab_sort_field, ab_val_field],
+        subscribe: [ab_tag, cmd_tag], passive: true do
+    loop do
+      transaction do
+        _, cmd, args = take(subspace cmd_tag)
+
+        case cmd
+        when "delete"
+          args.each do |name|
+            take [ab_tag, name, nil]
+          end
+
+        when "fetch"
+          name = args[0]
+          _, _, addr = read [ab_tag, name, nil]
+          write [resp_tag, name, args, addr]
+
+        when "next", "prev"
+          name = args[0]
+          _, name2, addr = read SortedSetTemplate[ab_tag, cmd, name]
+          write [resp_tag, name, args, name2, addr]
+
+        when "first", "last"
+          _, name, addr = read SortedSetTemplate[ab_tag, cmd]
+          write [resp_tag, name, args, name, addr]
+
+        else # maybe write an error message in a tuple
+          log.error "bad command: #{cmd}"
+        end
+      end
+    end
+  end
+  
+  child subscribe: resp_tag do
+    # write some ab entries
+    write [ab_tag, "McFirst, Firsty", "123 W. Crescent Terrace"]
+    write [ab_tag, "Secondismus, Deuce", "456 S. West Way"]
+
+    # make some queries
+    write [cmd_tag, "first", []]
+    *, name, addr = take [resp_tag, "first", [], nil, nil]
+    log "first entry: #{name} => #{addr}"
+    
+    write [cmd_tag, "next", [name]]
+    *, name, addr = take [resp_tag, "next", [name], nil, nil]
+    log "next entry: #{name} => #{addr}"
+  end
+end