etcd-rb 1.0.0.pre1 → 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f5096dd7b703a7f42348e4eaae037cfa642e054f
+  data.tar.gz: 8f93a5c5affe7ffaaa0557a7df893a2fa6d9f00c
+SHA512:
+  metadata.gz: 64631fd825ea6326288c8730f60539afd63e62f12015a657232e663153564ce628025f78b5c7026844901c790a76cbcddd70bb36963af529e7c3f8871b29a7d4
+  data.tar.gz: a2170e3f1d2ea0d9dfac6353d221539f593cc534083782bf2a53704907429adf9cbd2da3ec844a2d6d956571ce9ee08b50fe7416160127edb449dc2d8d5bf8d4

data/README.md

@@ -5,50 +5,197 @@
 
 # Requirements
 
-A modern Ruby, compatible with 1.9.3 or later. Continously tested with MRI 1.9.3, 2.0.0 and JRuby 1.7.x.
-
-An etcd cluster. _Currently incompatible with the most recent versions of etcd because they return the wrong URI for the leader._
+  - A modern Ruby, compatible with 1.9.3 or later. Continously tested with MRI 1.9.3, 2.0.0 and JRuby 1.7.x.
+  - Linux/OSX OS
+  - to have a local Etcd binary, run
+    - `$ sh/install-etcd.sh`
 
 # Installation
 
-    gem install etcd-rb --prerelease
+    gem install etcd-rb
 
 # Quick start
 
 ```ruby
 require 'etcd'
 
-client = Etcd::Client.new
+client = Etcd::Client.connect(uris: 'http://localhost:4001')
+client.connect
 client.set('/foo', 'bar')
 client.get('/foo')
 ```
 
-See the full [API documentation](http://rubydoc.info/gems/etcd-rb/frames) for more. All core features are supported, including test-and-set, TTL, watches -- as well as a few convenience features like continuous watching.
+See the full [API documentation](http://rubydoc.info/github/iconara/etcd-rb/master/frames) for more. All core features are supported, including test-and-set, TTL, watches -- as well as a few convenience features like continuous watching.
+
+
+# Development
+    $ git clone https://github.com/iconara/etcd-rb.git
+    $ cd etcd-rb
+    # will compile the etcd binary in the tmp folder to use for testing
+    $ sh/install-etcd.sh
+    $ bundle install
+    # make your changes
+    $ sh/test
+
+
+# Development helpers
+    # console for quick REPL testing
+    $ sh/c
+
+    # start/stop Etcd cluster
+    $ sh/cluster <start/stop/reset>
+
+    # install Etcd binary to tmp folder
+    $ sh/install-etcd.sh
+
+    # run tests
+    $ sh/test
+
+
+# Playing in shell
+    # load console with etcd-rb code
+    $ sh/c
+    > ClusterController.start_cluster
+    > seed_uris = ["http://127.0.0.1:4001", "http://127.0.0.1:4002", "http://127.0.0.1:4003"]
+    > client = Etcd::Client.connect(:uris => seed_uris)
+
 
 # Features
 
-## Continuous watches: observers
+### Continuous watches: observers - [Example](#example-observers)
 
 Most of the time when you use watches with etcd you want to immediately re-watch the key when you get a change notification. The `Client#observe` method handles this for you, including re-watching with the last seen index, so that you don't miss any updates.
 
-## Automatic leader detection
+### Automatic leader detection - [Example](#example-automatic-leader-detection)
+
+All writes go to the leader-node. When the leader is re-elected, next request triggers a redirect and re-evaluation for
+the cluster status on the client side. This happens transparently to you.
+
+### Automatic failover & retry - [Example](#example-automatic-failover)
+
+If a request fails, client will try to get cluster configuration from all given seed URIs until first valid response. Then the original request will be retried. This also happens transparently to you.
+
+Watches are a special case, since they use long polling, they will break when the leader goes down. After a failover observers reestablish their watches with the new leader. Again - this happens transparently to you :)
+
+
+### Heartbeating - [Example](#example-heartbeating)
+
+To ensure that you have the most up-to-date cluster status and your observers are registered against the current leader node, initiate the client with `:heartbeat_freq  (in seconds) parameter. This will start a background thread, that will periodially check the leader status, which in case of leader re-election will trigger the failover.
+
+### Example: Automatic Leader Detection
+
+```ruby
+$ sh/c
+# ensure we have a cluster with 3 nodes
+ClusterController.start_cluster
+client = Etcd::Client.test_client
+# => <Etcd::Client ["http://127.0.0.1:4001", "http://127.0.0.1:4002", "http://127.0.0.1:4003"]>
+client.leader
+# => <Etcd::Node - node2 (leader) - http://127.0.0.1:4002>
+ClusterController.kill_node(client.cluster.leader.name)
+client.get("foo")
+client.leader # leader has changed!
+#=> <Etcd::Node - node3 (leader) - http://127.0.0.1:4003>
+```
+
+### Example: Automatic Failover
 
-You can point the client to any node in the etcd cluster, it will ask that node for the current leader and direct all subsequent requests directly to the leader to avoid unnecessary redirects. When the leader changes, detected by a redirect, the new leader will be registered and used instead of the previous.
+```ruby
+# start with
+# $ sh/c to have ClusterController available :)
+seed_uris = ["http://127.0.0.1:4001", "http://127.0.0.1:4002", "http://127.0.0.1:4003"]
+client = Etcd::Client.connect(:uris => seed_uris)
+
+
+## set some values
+client.set("foo", "bar")
+client.get("foo") # => bar
+client.get("does-not-exist") # => nil
+
+## kill leader node
+ClusterController.kill_node(client.cluster.leader.name)
+
+## client still trucking on
+client.get("foo") # => bar
+
+## we have visibility into cluster status
+puts client.cluster.nodes.map(&:status) # => [:running, :down, :running]
 
-## Automacic failover & retry
+# will leave only one process running by killing the next leader node
+ClusterController.kill_node(client.cluster.leader.name)
 
-When connecting for the first time, and when the leader changes, the list of nodes in the cluster is cached. Should the node that the client is talking to become unreachable, the client will attempt to connect to the next known node, until it finds one that responds. The first node to respond will be asked for the current leader, which will then be used for subsequent request.
+# but since we have no leader with one process, all requests will fail
+client.get("foo") # raises AllNodesDownError error
+
+puts client.cluster.nodes.map(&:status) # => [:running, :down, :down]
+client.cluster.leader # => nil
+
+## now start up the cluster in another terminal by executing
+ClusterController.start_cluster
+
+## client works again
+client.get("foo") # => bar
+
+```
+
+
+### Example: Observers
+
+```ruby
+$ sh/c
+# ensure we have a cluster with 3 nodes
+ClusterController.start_cluster
+# test_client method is only sugar for local development
+client = Etcd::Client.test_client
+
+# your block can get value, key and info of the change, that you are observing
+client.observe('/foo') do |v,k,info|
+  puts "v #{v}, k: #{k}, info: #{info}"
+end
+
+# this will trigger the observer
+client.set("foo", "bar")
+# let's kill the leader of the cluster to demonstrate the re-watching feature
+ClusterController.kill_node(client.cluster.leader.name)
+# still triggering the observer!
+client.set("foo", "bar")
+```
+
+
+### Example: Heartbeating
+
+```ruby
+$ sh/c
+# ensure we have a cluster with 3 nodes
+ClusterController.start_cluster
+client = Etcd::Client.test_client(:heartbeat_freq => 5)
+
+# your block can get value, key and info of the change, that you are observing
+client.observe('/foo') do |v,k,info|
+  puts "v #{v}, k: #{k}, info: #{info}"
+end
+
+### START A NEW console with $ `sh/c` helper
+client = Etcd::Client.test_client
+# this will trigger the observer in the first console
+client.set("foo", "bar")
+# let's kill the leader of the cluster to demonstrate re-watching && heartbeating for all active clients
+ClusterController.kill_node(client.cluster.leader.name)
+# still triggering the observer in the first console
+# you might loose some changes in the 5-seconds hearbeating window... You should be aware of that.
+client.set("foo", "bar")
+```
 
-This is handled completely transparently to you.
 
-Watches are a special case, since they use long polling, they will break when the leader goes down. Observers will attempt to reestablish their watches with the new leader.
 
 # Changelog & versioning
 
 Check out the [releases on GitHub](https://github.com/iconara/etcd-rb/releases). Version numbering follows the [semantic versioning](http://semver.org/) scheme.
 
+
 # How to contribute
 
+
 Fork the repository, make your changes in a topic branch that branches off from the right place in the history (HEAD isn't necessarily always right), make your changes and finally submit a pull request.
 
 Follow the style of the existing code, make sure that existing tests pass, and that everything new has good test coverage. Put some effort into writing clear and concise commit messages, and write a good pull request description.

data/lib/etcd/client/failover.rb

@@ -0,0 +1,109 @@
+module Etcd
+  class Client
+
+
+    # @param options [Hash]
+    # @option options [Array] :uris (['http://127.0.0.1:4001']) seed uris with etcd cluster nodes
+    # @option options [Float] :heartbeat_freq (0.0) check-frequency for leader status (in seconds)
+    #  Heartbeating will start only for non-zero values
+    def initialize(options={})
+      @observers      = {}
+      @seed_uris      = options[:uris] || ['http://127.0.0.1:4001']
+      @heartbeat_freq = options[:heartbeat_freq].to_f
+      http_client.redirect_uri_callback = method(:handle_redirected)
+    end
+
+    # Create a new client and connect it to the etcd cluster.
+    #
+    # This method is the preferred way to create a new client, and is the
+    # equivalent of `Client.new(options).connect`. See {#initialize} and
+    # {#connect} for options and details.
+    #
+    # @see #initialize
+    # @see #connect
+    def self.connect(options={})
+      self.new(options).connect
+    end
+
+    # Connects to the etcd cluster
+    #
+    # @see #update_cluster
+    def connect
+      update_cluster
+      start_heartbeat_if_needed
+      self
+    end
+
+
+    # Creates a Cluster-instance from `@seed_uris`
+    # and stores the cluster leader information
+    def update_cluster
+      logger.debug("update_cluster: enter")
+      begin
+        @cluster = Etcd::Cluster.init_from_uris(*seed_uris)
+        @leader  = @cluster.leader
+        @status  = :up
+        logger.debug("update_cluster: after success")
+        refresh_observers
+        @cluster
+      rescue AllNodesDownError => e
+        logger.debug("update_cluster: failed")
+        raise e
+      end
+    end
+
+    # kinda magic accessor-method:
+    # - will reinitialize leader && cluster if needed
+    def leader
+      @leader ||= cluster && cluster.leader || update_cluster && self.leader
+    end
+
+    def leader_uri
+      leader && leader.etcd
+    end
+
+
+    def start_heartbeat_if_needed
+      logger.debug("client - starting heartbeat")
+      @heartbeat = Etcd::Heartbeat.new(self, @heartbeat_freq)
+      @heartbeat.start_heartbeat_if_needed
+    end
+
+    # Only happens on attempted write to a follower node in cluster. Means:
+    # - leader changed since last update
+    # Solution: just get fresh cluster status
+    def handle_redirected(uri, response)
+      update_cluster
+      http_client.default_redirect_uri_callback(uri, response)
+    end
+
+
+private
+    # :uri and :request_data are the only methods calling :leader method
+    # so they both need to handle the case for missing leader in cluster
+    def uri(key, action=S_KEYS)
+      raise AllNodesDownError unless leader
+      key = "/#{key}" unless key.start_with?(S_SLASH)
+      "#{leader_uri}/v1/#{action}#{key}"
+    end
+
+
+    def request_data(method, uri, args={})
+      logger.debug("request_data:  #{method} - #{uri} #{args.inspect}")
+      begin
+        super
+      rescue Errno::ECONNREFUSED, HTTPClient::TimeoutError => e
+        logger.debug("request_data:  re-election handling")
+        old_leader_uri = @leader.etcd
+        update_cluster
+        if @leader
+          uri = uri.gsub(old_leader_uri, @leader.etcd)
+          retry
+        else
+          raise AllNodesDownError
+        end
+      end
+    end
+
+  end
+end

data/lib/etcd/client/observing.rb

@@ -0,0 +1,60 @@
+module Etcd
+  class Client
+
+    # Sets up a continuous watch of a key or prefix.
+    #
+    # This method works like {#watch} (which is used behind the scenes), but
+    # will re-watch the key or prefix after receiving a change notificiation.
+    #
+    # When re-watching the index of the previous change notification is used,
+    # so no subsequent changes will be lost while a change is being processed.
+    #
+    # Unlike {#watch} this method as asynchronous. The watch handler runs in a
+    # separate thread (currently a new thread is created for each invocation,
+    # keep this in mind if you need to watch many different keys), and can be
+    # cancelled by calling `#cancel` on the returned object.
+    #
+    # Because of implementation details the watch handler thread will not be
+    # stopped directly when you call `#cancel`. The thread will be blocked until
+    # the next change notification (which will be ignored). This will have very
+    # little effect on performance since the thread will not be runnable. Unless
+    # you're creating lots of observers it should not matter. If you want to
+    # make sure you wait for the thread to stop you can call `#join` on the
+    # returned object.
+    #
+    # @example Creating and cancelling an observer
+    #   observer = client.observe('/foo') do |value|
+    #     # do something on changes
+    #   end
+    #   # ...
+    #   observer.cancel
+    #
+    # @return [#cancel, #join] an observer object which you can call cancel and
+    #   join on
+    def observe(prefix, &handler)
+      ob = Observer.new(self, prefix, handler).tap(&:run)
+      @observers[prefix] = ob
+      ob
+    end
+
+    def observers_overview
+      observers.map do |_, observer|
+        observer.pp_status
+      end
+    end
+
+    def refresh_observers_if_needed
+      refresh_observers if observers.values.any?{|x| not x.status}
+    end
+
+
+    # Re-initiates watches after leader election
+    def refresh_observers
+      logger.debug("refresh_observers: enter")
+      observers.each do |_, observer|
+        observer.rerun unless observer.status
+      end
+    end
+
+  end
+end

data/lib/etcd/client/protocol.rb

@@ -0,0 +1,186 @@
+module Etcd
+  class Client
+
+    # Sets the value of a key.
+    #
+    # Accepts an optional `:ttl` which is the number of seconds that the key
+    # should live before being automatically deleted.
+    #
+    # @param key [String] the key to set
+    # @param value [String] the value to set
+    # @param options [Hash]
+    # @option options [Fixnum] :ttl (nil) an optional time to live (in seconds)
+    #   for the key
+    # @return [String] The previous value (if any)
+    def set(key, value, options={})
+      body       = {:value => value}
+      body[:ttl] = options[:ttl] if options[:ttl]
+      data       = request_data(:post, key_uri(key), body: body)
+      data[S_PREV_VALUE]
+    end
+
+    # Gets the value or values for a key.
+    #
+    # If the key represents a directory with direct decendants (e.g. "/foo" for
+    # "/foo/bar") a hash of keys and values will be returned.
+    #
+    # @param key [String] the key or prefix to retrieve
+    # @return [String, Hash] the value for the key, or a hash of keys and values
+    #   when the key is a prefix.
+    def get(key)
+      data = request_data(:get, key_uri(key))
+      return nil unless data
+      if data.is_a?(Array)
+        data.each_with_object({}) do |e, acc|
+          acc[e[S_KEY]] = e[S_VALUE]
+        end
+      else
+        data[S_VALUE]
+      end
+    end
+
+    # Atomically sets the value for a key if the current value for the key
+    # matches the specified expected value.
+    #
+    # Returns `true` when the operation succeeds, i.e. when the specified
+    # expected value matches the current value. Returns `false` otherwise.
+    #
+    # Accepts an optional `:ttl` which is the number of seconds that the key
+    # should live before being automatically deleted.
+    #
+    # @param key [String] the key to set
+    # @param value [String] the value to set
+    # @param expected_value [String] the value to compare to the current value
+    # @param options [Hash]
+    # @option options [Fixnum] :ttl (nil) an optional time to live (in seconds)
+    #   for the key
+    # @return [true, false] whether or not the operation succeeded
+    def update(key, value, expected_value, options={})
+      body       = {:value => value, :prevValue => expected_value}
+      body[:ttl] = options[:ttl] if options[:ttl]
+      data       = request_data(:post, key_uri(key), body: body)
+      !! data
+    end
+
+    # Remove a key and its value.
+    #
+    # The previous value is returned, or `nil` if the key did not exist.
+    #
+    # @param key [String] the key to remove
+    # @return [String] the previous value, if any
+    def delete(key)
+      data = request_data(:delete, key_uri(key))
+      return nil unless data
+      data[S_PREV_VALUE]
+    end
+
+    # Returns true if the specified key exists.
+    #
+    # This is a convenience method and equivalent to calling {#get} and checking
+    # if the value is `nil`.
+    #
+    # @return [true, false] whether or not the specified key exists
+    def exists?(key)
+      !!get(key)
+    end
+
+    # Returns info about a key, such as TTL, expiration and index.
+    #
+    # For keys with values the returned hash will include `:key`, `:value` and
+    # `:index`. Additionally for keys with a TTL set there will be a `:ttl` and
+    # `:expiration` (as a UTC `Time`).
+    #
+    # For keys that represent directories with no direct decendants (e.g. "/foo"
+    # for "/foo/bar/baz") the `:dir` key will have the value `true`.
+    #
+    # For keys that represent directories with direct decendants (e.g. "/foo"
+    # for "/foo/bar") a hash of keys and info will be returned.
+    #
+    # @param key [String] the key or prefix to retrieve
+    # @return [Hash] a with info about the key, the exact contents depend on
+    #   what kind of key it is.
+    def info(key)
+      data = request_data(:get, uri(key))
+      return nil unless data
+      if data.is_a?(Array)
+        data.each_with_object({}) do |d, acc|
+          info = extract_info(d)
+          info.delete(:action)
+          acc[info[:key]] = info
+        end
+      else
+        info = extract_info(data)
+        info.delete(:action)
+        info
+      end
+    end
+
+    # Watches a key or prefix and calls the given block when with any changes.
+    #
+    # This method will block until the server replies. There is no way to cancel
+    # the call.
+    #
+    # The parameters to the block are the value, the key and a hash of
+    # additional info. The info will contain the `:action` that caused the
+    # change (`:set`, `:delete` etc.), the `:key`, the `:value`, the `:index`,
+    # `:new_key` with the value `true` when a new key was created below the
+    # watched prefix, `:previous_value`, if any, `:ttl` and `:expiration` if
+    # applicable.
+    #
+    # The reason why the block parameters are in the order`value`, `key` instead
+    # of `key`, `value` is because you almost always want to get the new value
+    # when you watch, but not always the key, and most often not the info. With
+    # this order you can leave out the parameters you don't need.
+    #
+    # @param prefix [String] the key or prefix to watch
+    # @param options [Hash]
+    # @option options [Fixnum] :index (nil) the index to start watching from
+    # @yieldparam [String] value the value of the key that changed
+    # @yieldparam [String] key the key that changed
+    # @yieldparam [Hash] info the info for the key that changed
+    # @return [Object] the result of the given block
+    def watch(prefix, options={})
+      if options[:index]
+        parameters = {:index => options[:index]}
+        data       = request_data(:post, watch_uri(prefix), query: parameters)
+      else
+        data       = request_data(:get, watch_uri(prefix), query: {})
+      end
+
+      info         = extract_info(data)
+      yield info[:value], info[:key], info
+    end
+
+
+    def key_uri(key)
+      uri(key, S_KEYS)
+    end
+
+    def watch_uri(key)
+      uri(key, S_WATCH)
+    end
+
+private
+
+    def extract_info(data)
+      info = {
+        :key   => data[S_KEY],
+        :value => data[S_VALUE],
+        :index => data[S_INDEX],
+      }
+      expiration_s          = data[S_EXPIRATION]
+      ttl                   = data[S_TTL]
+      previous_value        = data[S_PREV_VALUE]
+      action_s              = data[S_ACTION]
+      info[:expiration]     = Time.iso8601(expiration_s) if expiration_s
+      info[:ttl]            = ttl if ttl
+      info[:new_key]        = data[S_NEW_KEY] if data.include?(S_NEW_KEY)
+      info[:dir]            = data[S_DIR] if data.include?(S_DIR)
+      info[:previous_value] = previous_value if previous_value
+      info[:action]         = action_s.downcase.to_sym if action_s
+      info
+    end
+
+
+  end
+end