tupelo 0.21 → 0.22

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3ab41264ead7c8233c4e48f9f568ae68fb04e843
-  data.tar.gz: 01baffe87a234f70b3f2f84de84f38f62924d152
+  metadata.gz: 79341ae98464d4246f83672dc84801f0ec5c9979
+  data.tar.gz: 128e129fa77dfc86ddd3097a89d1563e9f20b8b6
 SHA512:
-  metadata.gz: 6808153d5b48baf8e376a72d4179ca20851da83833f938dfa00120da1ec20fe9f4fe504b4006bd0c88f243d9a445f6856ecc70af6013d27c1f13140b5946b67a
-  data.tar.gz: e57be7395a7d81c6cde4e3ec8c21f93761e76f1dc8c7d2f70a6331f59e563c5b7e0bd903119f2f86c2e161231f5cdfc4210144bdf223327c5f158d139caf5c2a
+  metadata.gz: d88280189aed380cc55f30182fd58a3e66daa58094bc80ca19c444df4719706b0a0ef427910b6b67fe276c5d73eb677159b02c61fab1a6f99af5f10561aa28e8
+  data.tar.gz: 1bb0816c7e482a3b6bd15ff42d6542c5a8fad88ce22b51cb36098a0b954501292f8fd8fdce7e59dd955b5d6f705169e67dbe8f607fd12b95bcf6ffb7afd777db

data/README.md

@@ -1,12 +1,42 @@
 Tupelo
 ==
 
-Tupelo is a language-agnostic tuplespace for coordination of distributed programs. It is designed for distribution of both computation and storage, on disk and in memory, with pluggable storage adapters. Its programming model is small and semantically transparent: there are tuples (built from arrays, hashes, and scalars), a few operations on tuples (read, write, take), and transactions composed of these operations. This data-centric model, unlike RPC and most forms of messaging, decouples application endpoints from each other, not only in space and time, but also in referential structure: processes refer to data rather than to other processes.
+Tupelo is a language-agnostic tuplespace for coordination of distributed programs. It is designed for distribution of both computation and storage, on disk and in memory, with pluggable storage adapters. Its programming model is small and semantically transparent: there are tuples (built from arrays, hashes, and scalars), a few operations on tuples (read, write, take), and transactions composed of these operations. This data-centric model, unlike RPC and most forms of messaging, decouples application endpoints from each other, not only in space and time, but also in referential structure: processes refer to data rather than to other processes or to channels.
 
 Tupelo is inspired by Masatoshi Seki's Rinda in the Ruby standard library, which in turn is based on David Gelernter's Linda. The programming models of Tupelo and Rinda are similar, except for the lack of transactions in Rinda. However, the implementations of the two are nearly opposite in architectural approach.
 
 This repository contains the reference implementation in Ruby, with documentation, tests, benchmarks, and examples. Implementations in other languages must communicate with this one.
 
+News
+====
+
+* 2014-June-26: Slides and videos from my talk on Calvin, which has some ideas in common with tupelo (plus a lot more):
+  * http://www.meetup.com/papers-we-love-too/events/171291972
+  * https://speakerdeck.com/paperswelove/pwlsf-number-4-equals-joel-vanderwerf-on-calvin
+  * https://plus.google.com/events/cprsf9edesa0ghm3b8c416ppa7o
+
+* 2014-April-7: Want to use these fancy stores inside a client that you run from the command line? See [here](example/sqlite/README.md) and [here](example/riemann/README.md).
+
+* 2014-April-1: [Riemann example](example/riemann) uses more stores in various roles:
+
+  * sqlite store for the consumer replicas, with indexes for
+    * service/host/time
+    * tags
+    * custom data
+  * rbtree store for the expirer
+  * hash store for the alerter
+
+* 2014-March-19: release 0.21
+    
+    * correct use of `store` and `space` terms everywhere
+    * sqlite example improvements
+    * API additions
+      * tuplestores can define #find_all_matches_for
+        * for efficient search in #read_all
+      * msgpack and json options to deserialize keys as symbols
+        * invoke via Client option and tup switch
+
+See also https://github.com/vjoel/tupelo/releases.
 
 Documentation
 ============
@@ -22,6 +52,7 @@ In Depth
 * [Transactions](doc/transactions.md)
 * [Replication](doc/replication.md)
 * [Subspaces](doc/subspace.md)
+* [Tuple stores](doc/tuplestores.md)
 * [Causality](doc/causality.md)
 * [Concurrency](doc/concurrency.md)
 
@@ -68,46 +99,7 @@ Tupelo is a flexible base layer for various distributed programming patterns and
 
 Tupelo can be used to impose a unified transactional structure and distributed access model on a mixture of programs and languages (polyglot computation) and a mixture of data stores (polyglot persistence), with consistent replication.
 
-
-Example
--------
-
-This program counts prime numbers in an interval by distributing the problem to a set of hosts:
-
-    require 'tupelo/app/remote'
-
-    hosts = %w{itchy scratchy lisa bart} # ssh hosts with key-based auth
-
-    Tupelo.tcp_application do
-      hosts.each do |host|
-        remote host: host, passive: true, eval: %{
-          require 'prime' # ruby stdlib for prime factorization
-          loop do
-            _, input = take(["input", Integer])
-            write ["output", input, input.prime_division]
-          end
-        }
-      end
-
-      local do
-        inputs = 1_000_000_000_000 .. 1_000_000_000_200
-
-        inputs.each do |input|
-          write ["input", input]
-        end
-
-        count = 0
-        inputs.size.times do |i|
-          _, input, factors = take ["output", Integer, nil]
-          count += 1 if factors.size == 1 and factors[0][1] == 1
-          print "\rChecked #{i}"
-        end
-
-        puts "\nThere are #{count} primes in #{inputs}"
-      end
-    end
-
-Ssh is used to set up the remote processes. Additionally, with the `--tunnel` command line argument, all tuple communication is tunneled over ssh. More examples like this are in [example/map-reduce](example/map-reduce).
+See the [example section](#examples) below and the [examples](example) directory.
 
 
 Limitations
@@ -180,14 +172,148 @@ Additional benefits (not related to message sequencing) include:
 Process control and tunneling are available independently of tupelo using the easy-serve gem.
 
 
+Examples
+========
+
+Distributed processing
+----------------------
+
+This program counts prime numbers in an interval by distributing the problem to a set of hosts:
+
+    require 'tupelo/app/remote'
+
+    hosts = %w{itchy scratchy lisa bart} # ssh hosts with key-based auth
+
+    Tupelo.tcp_application do
+      hosts.each do |host|
+        remote host: host, passive: true, eval: %{
+          require 'prime' # ruby stdlib for prime factorization
+          loop do
+            _, input = take(["input", Integer])
+            write ["output", input, input.prime_division]
+          end
+        }
+      end
+
+      local do
+        inputs = 1_000_000_000_000 .. 1_000_000_000_200
+
+        inputs.each do |input|
+          write ["input", input]
+        end
+
+        count = 0
+        inputs.size.times do |i|
+          _, input, factors = take ["output", Integer, nil]
+          count += 1 if factors.size == 1 and factors[0][1] == 1
+          print "\rChecked #{i}"
+        end
+
+        puts "\nThere are #{count} primes in #{inputs}"
+      end
+    end
+
+Ssh is used to set up the remote processes. Additionally, with the `--tunnel` command line argument, all tuple communication is tunneled over ssh. More examples like this are in [example/map-reduce](example/map-reduce), [example/pregel](example/pregel), and [example/parallel.rb](example/parallel.rb).
+
+Distributed storage
+-------------------
+
+Here's an example that creates an in-memory sqlite in one client with a table for Points of Interest (POI). A second client populates that table by writing POI tuples and then executes a SQL delete by writing a tuple with the deletion parameters.
+
+    require 'tupelo/app'
+    require_relative 'poi-client' # run this in example/sqlite
+
+    Tupelo.application do
+      local do
+        POISPACE = PoiStore.define_poispace(self)
+        define_subspace("cmd", {id: nil, cmd: String, arg: nil})
+        define_subspace("rsp", {id: nil, result: nil})
+      end
+
+      child PoiClient, poispace: POISPACE, subscribe: "cmd", passive: true do
+        loop do
+          req = take subspace("cmd")
+          case req[:cmd]
+          when "delete box"
+            lat = req[:arg][:lat]; lng = req[:arg][:lng]
+            template = PoiTemplate.new(poi_template: subspace("poi"),
+              lat: lat[0]..lat[1], lng: lng[0]..lng[1])
+            deleted = []
+            transaction do
+              while poi = take_nowait(template)
+                deleted << poi
+              end
+            end
+            write id: req[:id], result: deleted
+          end
+        end
+      end
+
+      child subscribe: "rsp" do
+        write lat: 1.2, lng: 3.4, desc: "foo"
+        write lat: 5.6, lng: 7.8, desc: "bar"
+        write lat: 1.3, lng: 3.5, desc: "baz"
+
+        write id: 1, cmd: "delete box", arg: {lat: [1.0, 1.4], lng: [3.0, 4.0]}
+        rsp = take id: 1, result: nil
+        log "deleted: #{rsp["result"]}"
+      end
+    end
+
+The output should be something like this:
+
+    A: client 3: deleted: [{"lat"=>1.2, "lng"=>3.4, "desc"=>"foo"}, {"lat"=>1.3, "lng"=>3.5, "desc"=>"baz"}]
+
+See [example/sqlite](example/sqlite) for the complete example. More advanced versions of this example have remote, replicated sqlites for redundancy and load distribution.
+
+Web app coordination
+--------------------
+
+This example runs several sinatra web apps and uses tupelo to set up a chat network between their users.
+
+    require 'tupelo/app'
+    require 'sinatra/base'
+
+    Tupelo.application do
+      [9001, 9002, 9003].each do |port|
+        child do |client|
+          Class.new(Sinatra::Base).class_eval do
+            post '/send' do
+              client.write ["message", params["dest"], params["text"]]
+            end
+
+            get '/recv' do
+              "%s for %s: %s\n" %
+                (client.take ["message", params["dest"], String])
+            end
+
+            set :port, port
+            run!
+          end
+        end
+      end
+    end
+
+You can use curl to chat:
+
+    $ curl 'localhost:9001/send?text=hello&dest=fred' -d ''
+
+and
+
+    $ curl 'localhost:9003/recv?dest=fred'
+    message for fred: hello
+
+Note that the `recv` call waits for a message if none is available.
+
+See also [example/multi-tier](example/multi-tier) and the chat server in [example/chat](example/chat).
+
+
 Development
 ===========
 
 Patches and bug reports are most welcome.
 
-This project is hosted at
-
-https://github.com/vjoel/tupelo
+This project is hosted at https://github.com/vjoel/tupelo
 
 Dependencies
 ------------
@@ -219,7 +345,7 @@ Optional gems for some of the examples:
 Contact
 =======
 
-Joel VanderWerf, vjoel@users.sourceforge.net, @JoelVanderWerf.
+Joel VanderWerf, vjoel@users.sourceforge.net, [@JoelVanderWerf](https://twitter.com/JoelVanderWerf).
 
 License and Copyright
 ========

data/bin/tup

@@ -75,6 +75,17 @@ if ARGV.delete("-h") or ARGV.delete("--help")
                   subscribe to specified subspaces; use "" for none
                   by default, tup client subscribes to everything
 
+      --store class
+      --store class,subspace
+                  Use a store of the specified class for the client's
+                  local tuplestore. If subspace is given, pass the subspace's
+                  spec as the first argument to <class>.new.
+                  (The -I and -r options are useful.)
+
+      -I path     append dir to $LOAD_PATH (the space char is necessary)
+      
+      -r file     require file (the space char is necessary)
+
   END
   exit
 end
@@ -93,6 +104,24 @@ if i=argv.index("--subscribe") # default is to subscribe to all
   subscribed_tags = argv.delete_at(i).split(",")
 end
 
+__store = nil
+if i=argv.index("--store")
+  argv.delete("--store")
+  __store = argv.delete_at(i).split(",")
+end
+
+_r_files = []
+if i=argv.index("-r")
+  argv.delete("-r")
+  _r_files << argv.delete_at(i)
+end
+
+_I_dirs = []
+if i=argv.index("-I")
+  argv.delete("-I")
+  _I_dirs << argv.delete_at(i)
+end
+
 services_file = argv.shift
 proto = (argv.shift || :unix).to_sym
 addr = {proto: proto}
@@ -113,6 +142,24 @@ Tupelo.application(
   cseqd_addr: addr, # using same addr causes autoincrement of port/filename
   arcd_addr: addr) do
 
+  $LOAD_PATH.unshift(*_I_dirs)
+  _r_files.each do |f|
+    require f
+  end
+
+  if __store
+    store_args = [Object.const_get(__store[0])]
+    store_subspace = __store[1]
+    if store_subspace
+      local subscribe: nil do
+        log.debug "getting spec for subspace #{store_subspace.inspect}"
+        store_args << subspace(store_subspace, wait: true).spec
+      end
+    end
+  else
+    store_args = nil
+  end
+
   class TupClient < Tupelo::Client
     alias w write_wait
     alias pl pulse_wait
@@ -168,6 +215,10 @@ Tupelo.application(
     client_opts[:subscribe] = subscribed_tags
   end
 
+  if store_args
+    client_opts[:tuplestore] = store_args
+  end
+
   local TupClient, **client_opts do
     log.info {"cpu time: %.2fs" % Process.times.inject {|s,x|s+x}}
     log.info {"starting shell."}

data/example/counters/merge.rb

@@ -17,18 +17,38 @@ Tupelo.application do
     # note: no need to init counter
     pids.each {|pid| Process.waitpid pid}
 
-    # but we cannot read the counter(s) without first merging them
-    # see also example/dedup.rb
-    while transaction do
+    # The next block of code reads the "current value" of the counter.
+    #
+    # "Current" means globally correct as of the last global
+    # tick received at the local client.
+    #
+    # The "value" of the counter is defined to be the sum of all n
+    # occurring in some tuple matching {count: nil}.
+    #
+    # The difficulty is that we cannot read the counter without
+    # first merging all the matching count tuples. In a sense, this
+    # is like CRDTs in an eventually consistent DB.
+    #
+    # See also example/dedup.rb.
+    log "merging..." while transaction do
       c1 = take_nowait count: nil
       c2 = take_nowait count: nil
       if c1 and c2
         write count: c1["count"] + c2["count"]
         true
+          # We had to merge, so try again.
       elsif c1
         log c1
+          # At the tick of the second take_nowait, exactly one count
+          # tuple exists, so we can safely report that value as the
+          # global count, with the understanding that the count
+          # may have changed at a later tick (even by the time the
+          # transaction commits).
+        false
       else
         log count: 0
+          # At the time of this txn, no count tuple exists.
+        false
       end
     end
   end

data/example/multi-tier/multi-sinatras.rb

@@ -3,6 +3,10 @@
 # http.rb example and shows how to use tupelo to coordinate multiple sinatra
 # instances.
 #
+# This could easily be adapted to run the servers on remote hosts (using
+# #remote instead of #child) and to restrict the network traffic to just
+# the subspaces each client needs (using subscribe).
+#
 # Depends on the sinatra and http gems.
 
 PORTS = [9001, 9002, 9003]
@@ -29,6 +33,7 @@ fork do
           end
           
           get '/recv' do
+            ## actually this should nt be a 'get' but a 'post' or 'delete'
             dest = params["dest"]
             _, _, text = client.take ["message", dest, String]
             text

data/example/riemann/event-subspace.rb

@@ -25,10 +25,7 @@ class Tupelo::Client
                               # event is considered valid for. Expired states
                               # may be removed from the index.
 
-      custom:       nil       # Any data
-                              # (not quite same as riemann's custom event attrs,
-                              # which are just arbitrary key-value pairs;
-                              # tupelo does not permit wildcards in keys)
+      custom:       Hash      # Arbitrary key-value pairs (not just strings).
     })
   end
 

data/example/riemann/expiration-dbg.rb

@@ -1,4 +1,6 @@
 class Tupelo::Client
+  # For each newly expired event, check whether it was expired on time,
+  # and log the event and the result of the check.
   def run_expiration_debugger
     read Tupelo::Client::EXPIRED_EVENT do |event|
       event_exp = event["time"] + event["ttl"]

data/example/riemann/producer.rb

@@ -9,7 +9,7 @@ class Tupelo::Client
       tags:         [],
       metric:       0,
       ttl:          0,
-      custom:       nil
+      custom:       {}
     }.freeze
   end
 
@@ -36,11 +36,12 @@ class Tupelo::Client
     30.times do |ei|
       e_cpu = base_event.merge(
         service:  "service #{i}",
-        state:    ei==10 ? "critical" : "ok",
+        state:    (ei%10==0) ? "critical" : "ok",
         time:     Time.now.to_f,
         ttl:      0.2,
         tags:     ["cpu", "cumulative"],
-        metric:   Process.times.utime + Process.times.stime
+        metric:   Process.times.utime + Process.times.stime,
+        custom:   {foo: ["bar", 42*i + ei]}
       )
       write_event e_cpu
       sleep 0.2