zk 0.9.1 → 1.0.0.rc.1

data/.gitignore

@@ -1,10 +1,10 @@
 *.gem
 .bundle
-Gemfile.lock
 pkg/*
-Gemfile.lock
 *.log
 html
 .yardoc
 .rspec
 .rvmrc
+Gemfile.*
+wiki

data/Gemfile

@@ -1,14 +1,13 @@
-# this is here for doing internal builds in our environment
-source ENV['MBOX_BUNDLER_SOURCE'] if ENV['MBOX_BUNDLER_SOURCE']
-source "http://rubygems.org"
+source :rubygems
 
 # gem 'slyphon-zookeeper', :path => '~/zookeeper'
 
 group :development do
-  gem 'pry'
   gem 'rake'
 end
 
+gem 'pry', :group => [:development, :test]
+
 group :docs do
   gem 'yard', '~> 0.7.5'
 

data/README.markdown

@@ -54,9 +54,11 @@ In addition to all of that, I would like to think that the public API the ZK::Cl
 
 ## Caveats
 
-ZK strives to be a complete, correct, and convenient way of interacting with ZooKeeper. There are a few weak points in the implementation:
+ZK strives to be a complete, correct, and convenient way of interacting with ZooKeeper. There are a few things to be aware of:
 
-* Starting with 0.9 there is only *one* event dispatch thread (in 0.8 there are 5). It is *very important* that you don't block the event delivery thread.
+* In versions &lte; 0.9 there is only *one* event dispatch thread. It is *very important* that you don't block the event delivery thread. In 1.0, there is one delivery thread by default, but you can adjust the level of concurrency, allowing more control and convenience for building your event-driven app.
+
+* ZK uses threads. You will have to use synchronization primitives if you want to avoid getting hurt. There are use cases that do not require you to think about this, but as soon as you want to register for events, you're using multiple threads. 
 
 * If you're not familiar with developing solutions with zookeeper, you should read about [sessions][] and [watches][] in the Programmer's Guide. Even if you *are* familiar, you should probably go read it again. 
 
@@ -66,7 +68,7 @@ ZK strives to be a complete, correct, and convenient way of interacting with Zoo
 
 * ZK::Client supports asynchronous calls of all basic methods (get, set, delete, etc.) however these versions are kind of inconvenient to use. For a fully evented stack, try [zk-eventmachine][], which is designed to be compatible and convenient to use in event-driven code.
 
-* ZooKeeper "chroot" [connection syntax][chroot] is currently being developed and should work for most cases. Right now we require that the root path exist before the chrooted client is used, but that may change [in the near future](https://github.com/slyphon/zk/issues/7).
+* ZooKeeper "chroot" [connection syntax][chroot] should work for most cases. Right now we require that the root path exist before the chrooted client is used, but that may change [in the near future](https://github.com/slyphon/zk/issues/7).
 
 [twitter/zookeeper]: https://github.com/twitter/zookeeper
 [async-branch]: https://github.com/slyphon/zk/tree/dev%2Fasync-conveniences
@@ -75,14 +77,21 @@ ZK strives to be a complete, correct, and convenient way of interacting with Zoo
 [sessions]: http://zookeeper.apache.org/doc/current/zookeeperProgrammers.html#ch_zkSessions 
 [watches]: http://zookeeper.apache.org/doc/r3.3.5/zookeeperProgrammers.html#ch_zkWatches
 
+## Users
+
+* [papertrail](http://papertrailapp.com/): Hosted log management service
+* [redis\_failover](https://github.com/ryanlecompte/redis_failover): Redis client/server failover managment system
+* [DCell](https://github.com/celluloid/dcell): Distributed ruby objects, built on top of the super cool [Celluloid](https://github.com/celluloid/celluloid) framework.
+
+
 ## Dependencies
 
-* The [slyphon-zookeeper gem][szk-gem] ([repo][szk-repo], branch with Gemfile [here][szk-repo-bundler]), which adds JRuby compatibility and a full suite of tests to the excellent [twitter/zookeeper][] project. _(I'm hoping to get this merged upstream, but it's a large change and, you know, people have day jobs)_. 
+* The [slyphon-zookeeper gem][szk-gem] ([repo][szk-repo]), which adds JRuby compatibility and a full suite of tests to the excellent [twitter/zookeeper][] project. 
 
 * For JRuby, the [slyphon-zookeeper\_jar gem][szk-jar-gem] ([repo][szk-jar-repo]), which just wraps the upstream zookeeper driver jar in a gem for easy installation
 
 [szk-gem]: https://rubygems.org/gems/slyphon-zookeeper
-[szk-repo]: https://github.com/slyphon/zookeeper/tree/dev/xplatform
+[szk-repo]: https://github.com/slyphon/zookeeper
 [szk-repo-bundler]: https://github.com/slyphon/zookeeper/tree/dev/gemfile/
 [szk-jar-gem]: https://rubygems.org/gems/slyphon-zookeeper_jar
 [szk-jar-repo]: https://github.com/slyphon/zookeeper_jar

data/RELEASES.markdown

@@ -1,5 +1,73 @@
 This file notes feature differences and bugfixes contained between releases. 
 
+### v1.0.0 ###
+
+* Support for 1.8.7 WILL BE *DROPPED* in v1.1. You've been warned.
+
+* Threaded client (the default one) will now automatically reconnect (i.e. `reopen()`) if a `SESSION_EXPIRED` or `AUTH_FAILED` event is received. Thanks to @eric for pointing out the _nose-on-your-face obviousness_ and importance of this. If users want to handle these events themselves, and not automatically reopen, you can pass `:reconnect => false` to the constructor.
+
+* allow for both :sequence and :sequential arguments to create, because I always forget which one is the "right one"
+
+* add zk.register(:all) to recevie node updates for all nodes (i.e. not filtered on path)
+
+* add 'interest' feature to zk.register, now you can indicate what kind of events should be delivered to the given block (previously you had to do that filtering inside the block). The default behavior is still the same, if no 'interest' is given, then all event types for the given path will be delivered to that block. 
+  
+    zk.register('/path', :created) do |event|
+      # event.node_created? will always be true
+    end
+
+    # or multiple kinds of events
+
+    zk.register('/path', [:created, :changed]) do |event|
+      # (event.node_created? or event.node_changed?) will always be true
+    end
+
+* create now allows you to pass a path and options, instead of requiring the blank string
+
+    zk.create('/path', '', :sequential => true)
+
+    # now also
+
+    zk.create('/path', :sequential => true)
+
+* fix for shutdown: close! called from threadpool will do the right thing
+
+* Chroot users rejoice! By default, ZK.new will create a chrooted path for you. 
+    
+    ZK.new('localhost:2181/path', :chroot => :create) # the default, create the path before returning connection
+
+    ZK.new('localhost:2181/path', :chroot => :check)  # make sure the chroot exists, raise if not
+
+    ZK.new('localhost:2181/path', :chroot => :do_nothing) # old default behavior
+
+    # and, just for kicks
+    
+    ZK.new('localhost:2181', :chroot => '/path') # equivalent to 'localhost:2181/path', :chroot => :create
+
+* Most of the event functionality used is now in a ZK::Event module. This is still mixed into the underlying slyphon-zookeeper class, but now all of the important and relevant methods are documented, and Event appears as a first-class citizen.
+
+### v0.9.1 ###
+
+The "Don't forget to update the RELEASES file before pushing a new release" release
+
+* Fix a fairly bad bug in event de-duplication (diff: http://is.gd/a1iKNc)
+	
+	This is fairly edge-case-y but could bite someone. If you'd set a watch
+	when doing a get that failed because the node didn't exist, any subsequent
+	attempts to set a watch would fail silently, because the client thought that the
+	watch had already been set.
+	
+	We now wrap the operation in the setup_watcher! method, which rolls back the
+	record-keeping of what watches have already been set for what nodes if an
+	exception is raised.
+	
+	This change has the side-effect that certain operations (get,stat,exists?,children)
+	will block event delivery until completion, because they need to have a consistent
+	idea about what events are pending, and which have been delivered. This also means
+	that calling these methods represent a synchronization point between user threads
+	(these operations can only occur serially, not simultaneously).
+
+
 ### v0.9.0 ###
 
 * Default threadpool size has been changed from 5 to 1. This should only affect people who are using the Election code.
@@ -11,4 +79,4 @@ This file notes feature differences and bugfixes contained between releases.
 * Began work on an experimental Multiplexed client, that would allow multithreaded clients to more effectively share a single connection by making all requests asynchronous behind the scenes, and using a queue to provide a synchronous (blocking) API. 
 
 
-# vim:ft=markdown
+# vim:ft=markdown:sts=2:sw=2:et

data/Rakefile

@@ -1,36 +1,68 @@
-# require 'rubygems'
-# gem 'rdoc', '~> 2.5'
-# require 'rdoc/task'
+require 'benchmark'
+gemset_name = 'zk'
 
-# RDoc::Task.new do |rd|
-#   rd.title = 'ZK Documentation'
-#   rd.rdoc_files.include("lib/**/*.rb")
-# end
+# this nonsense with the Gemfile symlinks is a bundler optimization
 
-gemset_name = 'zk'
+GEMSPEC_NAME = 'zk.gemspec'
+
+%w[1.8.7 1.9.2 jruby rbx 1.9.3].each do |ns_name|
+  rvm_ruby = (ns_name == 'rbx') ? "rbx-2.0.testing" : ns_name
+
+  ruby_with_gemset        = "#{rvm_ruby}@#{gemset_name}"
+  create_gemset_task_name = "mb:#{ns_name}:create_gemset"
+  bundle_task_name        = "mb:#{ns_name}:bundle_install"
+  rspec_task_name         = "mb:#{ns_name}:run_rspec"
+
+  phony_gemfile_link_name = "Gemfile.#{ns_name}"
+  phony_gemfile_lock_name = "#{phony_gemfile_link_name}.lock"
 
-%w[1.8.7 1.9.2 1.9.3 jruby].each do |rvm_ruby|
-  ruby_with_gemset = "#{rvm_ruby}@#{gemset_name}"
-  create_gemset_task_name = "mb:#{rvm_ruby}:create_gemset"
-  bundle_task_name        = "mb:#{rvm_ruby}:bundle_install"
-  rspec_task_name         = "mb:#{rvm_ruby}:run_rspec"
+  file phony_gemfile_link_name do
+    # apparently, rake doesn't deal with symlinks intelligently :P
+    ln_s('Gemfile', phony_gemfile_link_name) unless File.symlink?(phony_gemfile_link_name)
+  end
+
+  task :clean do
+    rm_rf [phony_gemfile_lock_name, phony_gemfile_lock_name]
+  end
 
   task create_gemset_task_name do
     sh "rvm #{rvm_ruby} do rvm gemset create #{gemset_name}"
   end
 
-  task bundle_task_name => create_gemset_task_name do
-    rm_f 'Gemfile.lock'
-    sh "rvm #{ruby_with_gemset} do bundle install"
+  task bundle_task_name => [phony_gemfile_link_name, create_gemset_task_name] do
+    sh "rvm #{ruby_with_gemset} do bundle install --gemfile #{phony_gemfile_link_name}"
   end
 
   task rspec_task_name => bundle_task_name do
-    sh "rvm #{ruby_with_gemset} do bundle exec rspec spec --fail-fast"
+    sh "rvm #{ruby_with_gemset} do env BUNDLE_GEMFILE=#{phony_gemfile_link_name} bundle exec rspec spec --fail-fast"
+  end
+
+  task "mb:#{ns_name}" => rspec_task_name
+
+  task "mb:test_all_rubies" => rspec_task_name
+end
+
+task 'mb:test_all' do
+  tm = Benchmark.realtime do
+    Rake::Task['mb:test_all_rubies'].invoke
+  end
+
+  $stderr.puts "Test run took: #{tm}"
+end
+
+
+namespace :yard do
+  task :clean do
+    rm_rf '.yardoc'
   end
 
-  task "mb:test_all" => rspec_task_name
+  task :server => :clean do
+    sh "yard server --reload"
+  end
 end
 
+task :clean => 'yard:clean'
+
 namespace :spec do
   task :define do
     require 'rubygems'

data/docs/examples/block_until_node_deleted_ex.rb

@@ -0,0 +1,63 @@
+# docs/examples/block_until_node_deleted_ex.rb
+
+require 'zk'
+
+class BlockUntilNodeDeleted
+  attr_reader :zk
+
+  def initialize
+    @zk = ZK.new
+    @path = @zk.create('/zk-examples', sequence: true, ephemeral: true)
+  end
+
+  def block_until_node_deleted(abs_node_path)
+    queue = Queue.new
+
+    ev_sub = zk.register(abs_node_path) do |event|
+      if event.node_deleted?
+        queue.enq(:deleted) 
+      else
+        if zk.exists?(abs_node_path, :watch => true)
+          # node still exists, wait for next event (better luck next time)
+        else
+          # ooh! surprise! it's gone!
+          queue.enq(:deleted) 
+        end
+      end
+    end
+   
+    # set up the callback, but bail if we don't need to wait
+    return true unless zk.exists?(abs_node_path, :watch => true)  
+
+    queue.pop # block waiting for node deletion
+    true
+  ensure
+    ev_sub.unsubscribe
+  end
+
+  def run
+    waiter = Thread.new do
+      $stderr.puts "waiter thread, about to block"
+      block_until_node_deleted(@path)
+      $stderr.puts "waiter unblocked"
+    end
+
+    # This is not a good way to wait on another thread in general (it's
+    # busy-waiting) but simple for this example.
+    #
+    Thread.pass until waiter.status == 'sleep'
+
+    # we now know the other thread is waiting for deletion
+    # so give 'em a thrill
+    @zk.delete(@path)
+
+    waiter.join
+
+    $stderr.puts "hooray! success!"
+  ensure
+    @zk.close!
+  end
+end
+
+BlockUntilNodeDeleted.new.run
+

data/docs/examples/events_01.rb

@@ -0,0 +1,36 @@
+require 'thread'
+require 'zk'
+
+class Events
+  def initialize
+    @zk = ZK.new
+    @queue = Queue.new
+    @path = '/zk-example-events01'
+  end
+
+  def do_something_with(data)
+    puts "I was told to say #{data.inspect}"
+  end
+
+  def run
+    @zk.delete(@path) rescue ZK::Exceptions::NoNode
+
+    @zk.register(@path) do |event|
+      if event.node_changed? or event.node_created?
+        # fetch the latest data
+        data = @zk.get(@path).first
+        do_something_with(data)
+        @queue.push(:got_event)
+      end
+    end
+
+    @zk.stat(@path, watch: true)
+    @zk.create(@path, 'Hello, events!')
+
+    @queue.pop
+  ensure
+    @zk.close!
+  end
+end
+
+Events.new.run

data/docs/examples/events_02.rb

@@ -0,0 +1,41 @@
+# docs/examples/events_02.rb
+
+require 'thread'
+require 'zk'
+
+class Events
+  def initialize
+    @zk = ZK.new
+    @queue = Queue.new
+    @path = '/zk-example-events01'
+  end
+
+  def do_something_with(data)
+    puts "I was told to say #{data.inspect}"
+    @queue.push(:got_event)
+  end
+
+  def run
+    @zk.register(@path) do |event|
+      if event.node_changed? or event.node_created?
+        data = @zk.get(@path, watch: true).first    # fetch the latest data and re-set watch
+        do_something_with(data)
+      end
+    end
+
+    @zk.delete(@path) rescue ZK::Exceptions::NoNode
+    @zk.stat(@path, watch: true)
+    @zk.create(@path, 'Hello, events!')
+
+    @queue.pop
+
+    @zk.set(@path, "ooh, an update!")
+
+    @queue.pop
+  ensure
+    @zk.close!
+  end
+end
+
+Events.new.run
+

data/lib/{z_k → zk}/client/base.rb

@@ -70,6 +70,7 @@ module ZK
       #    
       #   ZK::Client.new("zk01:2181,zk02:2181/chroot/path")
       #
+      # @abstract Overridden in subclasses
       def initialize(host, opts={})
         # no-op
       end
@@ -88,7 +89,16 @@ module ZK
       public
 
       # reopen the underlying connection
-      # returns state of connection after operation
+      #
+      # The `timeout` param is here mainly for legacy support.
+      #
+      # @param [Numeric] timeout how long should we wait for
+      #   the connection to reach a connected state before returning. Note that
+      #   the method will not raise and will return whether the connection
+      #   reaches the 'connected' state or not. The default is actually to use
+      #   the same value that was passed to the constructor for 'timeout'
+      #
+      # @return [Symbol] state of connection after operation
       def reopen(timeout=nil)
         timeout ||= @session_timeout # XXX: @session_timeout ?
         cnx.reopen(timeout)
@@ -98,14 +108,6 @@ module ZK
 
       # close the underlying connection and clear all pending events.
       #
-      # @note it is VERY IMPORTANT that when using the threaded client that you
-      #   __NOT CALL THIS ON THE EVENT DISPATCH THREAD__. If you do call it when
-      #   `event_dispatch_thread?` is true, you will get a big fat Kernel.warn
-      #   and nothing will happen. There is currently no safe way to have the event
-      #   thread cause a shutdown in the main thread (with good reason). There is also
-      #   no way to get an exception from the event thread (they're currently
-      #   just logged and swallowed), so this is the least horrible remedy available.
-      #   
       def close!
         event_handler.clear!
         wrap_state_closed_error { cnx.close unless cnx.closed? }
@@ -129,18 +131,16 @@ module ZK
       # creating sequential node with the same path argument, the call will never
       # throw a NodeExists exception.
       # 
-      # @todo clean up the verbiage around watchers
-      #
-      # This operation, if successful, will trigger all the watches left on the
-      # node of the given path by exists and get API calls, and the watches left
-      # on the parent node by children API calls.
-      # 
       # If a node is created successfully, the ZooKeeper server will trigger the
       # watches on the path left by exists calls, and the watches on the parent
       # of the node by children calls.
       #
-      # @param [String] path absolute path of the znode
-      # @param [String] data the data to create the znode with
+      # @overload create(path, opts={})
+      #   creates a znode at the absolute `path` with blank data and given
+      #   options
+      #
+      # @overload create(path, data, opts={})
+      #   creates a znode at the absolute `path` with given data and options
       # 
       # @option opts [Integer] :acl defaults to <tt>ZookeeperACLs::ZOO_OPEN_ACL_UNSAFE</tt>, 
       #   otherwise the ACL for the node. Should be a `ZOO_*` constant defined under the 
@@ -150,6 +150,9 @@ module ZK
       #
       # @option opts [bool] :sequence (false) if true, the created node will be sequential
       #
+      # @option opts [bool] :sequential (false) alias for :sequence option. if both are given
+      #   an ArgumentError is raised
+      #
       # @option opts [ZookeeperCallbacks::StringCallback] :callback (nil) provide a callback object
       #   that will be called when the znode has been created
       # 
@@ -193,18 +196,18 @@ module ZK
       #
       #   # or you can also do:
       #
-      #   zk.create("/path", '', :mode => :persistent_sequence)
+      #   zk.create("/path", '', :mode => :persistent_sequential)
       #   # => "/path0"
       #
       #
       # @example create ephemeral and sequential node
       #
-      #   zk.create("/path", '', :sequential => true, :ephemeral => true)
+      #   zk.create("/path", '', :sequence => true, :ephemeral => true)
       #   # => "/path0"
       #
       #   # or you can also do:
       #
-      #   zk.create("/path", "foo", :mode => :ephemeral_sequence)
+      #   zk.create("/path", "foo", :mode => :ephemeral_sequential)
       #   # => "/path0"
       #
       # @example create a child path
@@ -214,12 +217,12 @@ module ZK
       #
       # @example create a sequential child path
       #
-      #   zk.create("/path/child", "bar", :sequential => true, :ephemeral => true)
+      #   zk.create("/path/child", "bar", :sequence => true, :ephemeral => true)
       #   # => "/path/child0"
       #
       #   # or you can also do:
       #
-      #   zk.create("/path/child", "bar", :mode => :ephemeral_sequence)
+      #   zk.create("/path/child", "bar", :mode => :ephemeral_sequential)
       #   # => "/path/child0"
       #
       # @hidden_example create asynchronously with callback object
@@ -245,7 +248,25 @@ module ZK
       #
       #   zk.create("/path", "foo", :callback => callback, :context => context)
       #
-      def create(path, data='', opts={})
+      def create(path, *args)
+        opts = args.extract_options!
+
+        # be somewhat strict about how many arguments we accept.
+        if args.length > 1
+          raise ArgumentError, "create takes path, an optional data argument, and options, you passed: (#{path}, *#{args})"
+        end
+
+        # argh, terrible documentation bug, allow for :sequential, analagous to :sequence
+        if opts.has_key?(:sequential)
+          if opts.has_key?(:sequence)
+            raise ArgumentError, "Only one of :sequential or :sequence options can be given, opts: #{opts}"
+          end
+
+          opts[:sequence] = opts.delete(:sequential)
+        end
+
+        data = args.first || ''
+
         h = { :path => path, :data => data, :ephemeral => false, :sequence => false }.merge(opts)
 
         if mode = h.delete(:mode)
@@ -695,12 +716,12 @@ module ZK
         end
       end
 
-      # returns the session_id of the underlying connection
+      # @return [Fixnum] the session_id of the underlying connection
       def session_id
         cnx.session_id
       end
 
-      # returns the session_passwd of the underlying connection
+      # @return [String] the session_passwd of the underlying connection
       def session_passwd
         cnx.session_passwd
       end
@@ -716,6 +737,10 @@ module ZK
       # This method will return an {EventHandlerSubscription} instance that can be used
       # to remove the block from further updates by calling its `.unsubscribe` method.
       #
+      # You can specify a list of event types after the path that you wish to
+      # receive in your block. This allows you to register different blocks for
+      # different types of events.
+      #
       # @note All node watchers are one-shot handlers. After an event is delivered to
       #   your handler, you *must* re-watch the node to receive more events. This
       #   leads to a pattern you will find throughout ZK code that avoids races,
@@ -743,8 +768,30 @@ module ZK
       #     do_something_when_node_deleted  # call the callback
       #   end
       #
+      # @example only creation events
+      #
+      #   sub = zk.register('/path/to/znode', :created) do |event|
+      #     # do something when the node is created
+      #   end
+      #
+      # @example only changed or children events
+      #
+      #   sub = zk.register('/path/to/znode', [:changed, :child]) do |event|
+      #     if event.node_changed?
+      #       # do something on change
+      #     else
+      #       # we know it's a child event
+      #     end
+      #   end
+      #
+      # @param [String,:all] path the znode path you want to listen to, or the
+      #   special value :all, that will cause the block to be delivered events
+      #   for all znode paths
       #
-      # @param [String] path the path you want to listen to
+      # @param [Array,Symbol,nil] interests a symbol or array-of-symbols indicating
+      #   which events you would like the block to be called for. Valid events
+      #   are :created, :deleted, :changed, and :child. If nil, the block will
+      #   receive all events
       #
       # @param [Block] block the block to execute when a watch event happpens
       #
@@ -756,9 +803,10 @@ module ZK
       #
       # @see ZooKeeper::WatcherEvent
       # @see ZK::EventHandlerSubscription
+      # @see https://github.com/slyphon/zk/wiki/Events the wiki page on using events effectively
       #
-      def register(path, &block)
-        event_handler.register(path, &block)
+      def register(path, interests=nil, &block)
+        event_handler.register(path, interests, &block)
       end
 
       # returns true if the caller is calling from the event dispatch thread
@@ -772,6 +820,15 @@ module ZK
         raise Exceptions::EventDispatchThreadException, msg if event_dispatch_thread?
       end
 
+      # called directly from the zookeeper event thread with every event, before they
+      # get dispatched to the user callbacks. used by client implementations for
+      # critical events like session_expired, so that we don't compete for
+      # threads in the threadpool.
+      #
+      # @private
+      def raw_event_handler(event)
+      end
+
       protected
         # @private
         def check_rc(hash, inputs=nil)
@@ -798,6 +855,6 @@ module ZK
           nil
         end
     end # Base
-  end   # Client
-end     # ZK
+  end # Client
+end # ZK
 

data/lib/{z_k → zk}/client/conveniences.rb

@@ -18,7 +18,6 @@ module ZK
       # @yield [] the block that should be run in the threadpool, if `callable`
       #   isn't given
       #
-      # @private
       def defer(callable=nil, &block)
         @threadpool.defer(callable, &block)
       end