tupelo 0.6 → 0.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1bd2a467d93d42b9237dc027c9ee33160c7f44a3
-  data.tar.gz: 16c3d47bc0acd033f1f8be8bc4578310edb3f653
+  metadata.gz: 74456cde00385b5340494724ee0affc7cf6e2534
+  data.tar.gz: 331f3970d1fd41a702ada650f5701616de24b184
 SHA512:
-  metadata.gz: 1f20d07a716db5de35606c9500562863c64a5f3e74129336bfb9d52221b2751385d3d2c8a6ca873bd9841c403edafcc5911df776a143bf83a743d0d257af7ce6
-  data.tar.gz: 40b77909cf54701fabfaf35d189dd15f9ebc8beb4bbfa8a6a55b2cc6ee1eb512eab682c5c26df375e94a939d1a018a1f463e1400ae9131412819ebd09f4c093c
+  metadata.gz: bcfe41b674e798ba15761f6740bc74a11963a793538a725ec393fefd4328eb9053e019d95c0f4701cb2ca680dc52c2d2e8dce48921be7cd68532c639b4c61d90
+  data.tar.gz: 82b13b6aa0675069946428033e5d032ca4418409c5901da00cac76cf139d7cc34afa428c4e666e42370b69241c797cabb55b375cfd34e5d8bd5bc83f2e6bf9a9

data/README.md

@@ -105,7 +105,8 @@ Getting started
           ...
         end
 
-  Note that a local match is still not a guarantee of `x_final == x_optimistic`. Another process may take `x_optimistic` first, and the take will be re-executed. 
+  Note that a local match is still not a guarantee of `x_final == x_optimistic`. Another process may take `x_optimistic` first, and the take will be re-executed. (Think of #take_nowait as a way of saying "take a match, but don't bother trying if there is no match known at this time.")
+
   Perform a general transaction:
 
         result =
@@ -133,7 +134,7 @@ Getting started
 
   Note that all bin and example programs accept blob type (e.g., --msgpack, --json) on command line (it only needs to be specified for server -- the clients discover it). Also, all these programs accept log level on command line. The default is --warn. The --info level is a good way to get an idea of what is happening, without the verbosity of --debug.
 
-6. Debugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful. There is also the similar --trace switch that is available to all bin and example programs, for example:
+6. Debugging: in addition to the --info switch on all bin and example programs, bin/tspy is also really useful. There is also the similar --trace switch that is available to all bin and example programs. This switch diagnostic output for each transaction. For example:
 
   ```
     tick    cid status operation
@@ -142,7 +143,7 @@ Getting started
        3      3        atomic take ["x", 1], ["y", 2]
   ```
 
-  The `Tupelo.application` command, provided by `tupelo/app`, is the source of all these options and is available to your programs.
+  The `Tupelo.application` command, provided by `tupelo/app`, is the source of all these options and is available to your programs. It's a kind of lightweight process deployment and control framework; however it is not necessary to use tupelo.
 
 
 What is a tuplespace?
@@ -180,7 +181,7 @@ In other words, a tuple is a fairly general object, though this depends on the s
 
 * hashes
 
-It's kind of like a "JSON object", except that in the json blob case, the hash keys can only be strings. In the msgpack case, keys have no special limitations. In the case of the marshal and yaml modes, tuples can contain many other kinds of objects.
+It's kind of like a "JSON object", except that, when using the json serializer, the hash keys can only be strings. In the msgpack case, keys have no special limitations. In the case of the marshal and yaml modes, tuples can contain many other kinds of objects.
 
 What is a template?
 -------------------
@@ -244,7 +245,7 @@ Transactions and optimistic concurrency
 
 Transactions combine operations into a group that take effect at the same instant in (logical) time, isolated from other transactions.
 
-However, it may take some time to prepare the transaction. This is true in terms of both real time (clock and process) and logical tiem (global sequence of operations). Preparing a transaction means finding tuples that match the criteria of the read and take operations. Finding tuples may require searching (locally) for tuples, or waiting for new tuples to be written by others. Also, the transaction may fail even after matching tuples are found (when another process takes tuples of interest). Then the transaction needs to be prepared again. Once prepared, transaction is sent to all clients, where it may either succeed (in all clients) or fail (for the same reason as before--someone else grabbed one of our tuples). If it fails, then the preparation can begin again. A transaction guarantees that, when it completes, all the operations were performed on the tuples at the same logical time. It does not guarantee that the world stands still while one process is inside the `transaction {...}` block.
+However, it may take some time to prepare the transaction. This is true in terms of both real time (clock and process) and logical time (global sequence of operations). Preparing a transaction means finding tuples that match the criteria of the read and take operations. Finding tuples may require searching (locally) for tuples, or waiting for new tuples to be written by others. Also, the transaction may fail even after matching tuples are found (when another process takes tuples of interest). Then the transaction needs to be prepared again. Once prepared, transaction is sent to all clients, where it may either succeed (in all clients) or fail (for the same reason as before--someone else grabbed one of our tuples). If it fails, then the preparation begins again. A transaction guarantees that, when it completes, all the operations were performed on the tuples at the same logical time. It does not guarantee that the world stands still while one process is inside the `transaction {...}` block.
 
 Transactions are not just about batching up operations into a more efficient package (though you can do that with the #batch api). A transaction makes the combined operations execute atomically: the transaction finishes only when all of its operations can be successfully performed. Writes and pulses can always succeed, but takes and reads only succeed if the tuples exist.
 
@@ -378,6 +379,8 @@ To compare
 
 * pubsubs: kafka
 
+* spark, storm
+
 Architecture
 ============
 
@@ -410,9 +413,9 @@ Protocol
 
 Nothing in the protocol specifies local searching or storage, or matching, or notification, or templating. That's all up to each client. The protocol only contains tuples and operations on them (take, write, pulse, read), combined into transactions.
 
-The protocol has two layers. The outer (message) layer is 6 fields, managed by the funl gem, using msgpack for serialization. All socket reads are non-blocking, so a slow sender will not block other activity in the system.
+The protocol has two layers. The outer (message) layer is 6 fields, managed by the funl gem, using msgpack for serialization. All socket reads are non-blocking (using msgpack's stream mode), so a slow sender will not block other activity in the system.
 
-The inner (blob) layer manages one of those 6 field using msgpack (by default), marshal, json, or yaml. This layer contains the transaction operations. The blob is not unpacked by the server, only by clients.
+One of those 6 fields is a data blob, containing the actual transaction and tuple information. The inner (blob) layer manages that field using msgpack (by default), marshal, json, or yaml. This layer contains the transaction operations. The blob is not unpacked by the server, only by clients.
 
 Each inner serialization method ("blobber") has its own advantages and drawbacks:
 
@@ -422,8 +425,6 @@ Each inner serialization method ("blobber") has its own advantages and drawbacks
 
 * msgpack and json (yajl) are both relatively efficient (in terms of packet size, as well as parse/emit time)
 
-* msgpack and json both support non-blocking (buffered) reads, which can avoid bottlenecks due to slow senders or bad networks.
-
 * msgpack and json support the least diversity of objects (just "JSON objects"), but msgpack also supports hash keys that are objects rather than just strings.
 
 For most purposes, msgpack is a good choice, so it is the default.

data/example/remote-map-reduce.rb

@@ -0,0 +1,27 @@
+require 'tupelo/app/remote'
+
+hosts = ARGV.shift or abort "usage: #$0 <ssh-hostname>,<ssh-hostname>,..."
+hosts = hosts.split(",")
+
+Tupelo.tcp_application do
+  hosts.each do |host|
+    remote host: host, passive: true, eval: %{
+      loop do
+        len = take([String])[0].size
+        write [len]
+      end
+    }
+  end
+
+  local do
+    input = [ ["We are going to"], ["calcula"], ["te the len"],
+              ["gth of this "], ["sentence."] ]
+    write *input
+    sum = 0
+    input.size.times do
+      sum += take([Numeric])[0]
+    end
+    log "sum = #{sum}, correct sum = #{input.flatten.join.size}"
+    sleep 2
+  end
+end

data/example/remote.rb

@@ -13,15 +13,20 @@ Tupelo.tcp_application do
       # only for examples and tests, not production.
   end
   
-  remote host: host, eval: %{
+  remote host: host, log: true, eval: %{
     write host: `hostname`.chomp, mode: "eval", client: client_id
   }
   # rather than embed large chunks of code in the string, it's better to
   # load or require a file and pass self (which is a Client instance) to
   # a method in that file.
   
+  remote host: host, log: true, passive: true, eval: %{
+    write host: `hostname`.chomp, mode: "eval", client: client_id
+    sleep # since passive, app can stop this process
+  }
+  
   local do
-    2.times do
+    3.times do
       log take host: nil, mode: nil, client: nil
     end
   end

data/example/small.rb

@@ -58,7 +58,7 @@ EasyServe.start(servers_file: "small-servers.yaml") do |ez|
     client.stop if client # gracefully exit the tuplespace management thread
   end
 
-  ez.client :seqd, :cseqd, :arcd do |seqd, cseqd, arcd|
+  ez.child :seqd, :cseqd, :arcd do |seqd, cseqd, arcd|
     run_client seq: seqd, cseq: cseqd, arc: arcd, log: log do |client|
       client.write [2, 3, "frogs"]
       _, s = client.take ["animals", nil]
@@ -66,7 +66,7 @@ EasyServe.start(servers_file: "small-servers.yaml") do |ez|
     end
   end
 
-  ez.client :seqd, :cseqd, :arcd do |seqd, cseqd, arcd|
+  ez.child :seqd, :cseqd, :arcd do |seqd, cseqd, arcd|
     run_client seq: seqd, cseq: cseqd, arc: arcd, log: log do |client|
       x, y, s = client.take [Numeric, Numeric, String]
       s2 = ([s] * (x + y)).join(" ")

data/lib/tupelo/app/remote.rb

@@ -23,14 +23,13 @@ module Tupelo
     def remote client_class = Client,
         client_lib: 'tupelo/client', host: nil, **opts
       require 'easy-serve/remote'
-      raise if opts[:passive] ## not supported yet
       ## detach option so that remote process doesn't keep ssh connection
       snames = :seqd, :cseqd, :arcd
 
       if opts[:eval]
         ez.remote *snames, host: host, **opts, eval: %{
           require #{client_lib.inspect}
-          
+
           seqd, cseqd, arcd = *conns
           client_class = Object.const_get(#{client_class.name.inspect})
 

data/lib/tupelo/app.rb

@@ -46,7 +46,7 @@ module Tupelo
     # way. Then you do not have to manually interrupt the whole application when
     # the active processes are done. See examples.
     def child client_class = Client, passive: false, &block
-      ez.client :seqd, :cseqd, :arcd, passive: passive do |seqd, cseqd, arcd|
+      ez.child :seqd, :cseqd, :arcd, passive: passive do |seqd, cseqd, arcd|
         run_client client_class,
                    seq: seqd, cseq: cseqd, arc: arcd, log: log do |client|
           if block

data/lib/tupelo/version.rb

@@ -1,3 +1,3 @@
 module Tupelo
-  VERSION = "0.6"
+  VERSION = "0.7"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tupelo
 version: !ruby/object:Gem::Version
-  version: '0.6'
+  version: '0.7'
 platform: ruby
 authors:
 - Joel VanderWerf
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-27 00:00:00.000000000 Z
+date: 2013-08-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: atdo
@@ -118,6 +118,7 @@ files:
 - example/hash-tuples.rb
 - example/pulse.rb
 - example/transaction-logic.rb
+- example/remote-map-reduce.rb
 - example/map-reduce.rb
 - example/balance-xfer.rb
 - example/add-dsl.rb