haproxy-cluster 0.0.1

data/README.md

@@ -0,0 +1,56 @@
+haproxy-cluster
+===============
+
+> "Can we survive a rolling restart?"
+>
+> "How many concurrent connections right now across all load balancers?"
+
+While there are already a handfull of [HA Proxy](http://haproxy.1wt.edu) abstraction layers on RubyGems, I wanted to be able to answer questions like those above and more, quickly, accurately, and easily. So here's one more for the pile.
+
+`HAProxyCluster::Member` provides an ORM for HA Proxy's status page.
+
+`HAProxyCluster` provides a simple map/reduce-inspired framework on top of `HAProxyCluster::Member`.
+
+`haproxy_cluster` provides a shell scripting interface for `HAProxyCluster`. Exit codes are meaningful and intended to be useful from Nagios.
+
+Do you deploy new code using a sequential restart of application servers? Using this common pattern carelessly can result in too many servers being down at the same time, and cutomers seeing errors. `haproxy_cluster` can prevent this by ensuring that every load balancer agrees that the application is up at each stage in the deployment. In the example below, we will deploy a new WAR to three Tomcat instances which are fronted by two HA Proxy instances. HA Proxy has been configured with `option httpchk /check`, a path which only returns an affirmative status code when the application is ready to serve requests.
+
+```bash
+#!bin/bash
+set -o errexit
+servers="server1.example.com server2.example.com server3.example.com"
+load_balancers="https://lb1.example.com:8888 http://lb2.example.com:8888"
+
+for server in $servers ; do
+    haproxy_cluster --timeout=300 --eval "wait_until(true){ myapp.rolling_restartable? }" $load_balancers
+    scp myapp.war $server:/opt/tomcat/webapps
+done
+```
+
+The code block passed to `--eval` will not return until every load balancer reports that at least 80% of the backend servers defined for "myapp" are ready to serve requests. If this takes more than 5 minutes (300 seconds), the whole deployment is halted.
+
+Maybe you'd like to know how many transactions per second your whole cluster is processing.
+
+    $ haproxy_cluster --eval 'poll{ puts members.map{|m|m.myapp.rate}.inject(:+) }' $load_balancers
+
+Installation
+------------
+
+`gem install haproxy-cluster`
+
+Requires Ruby 1.9.2 and depends on RestClient.
+
+Non-Features
+------------
+
+* Doesn't try to modify configuration files. Use [haproxy-tools](https://github.com/subakva/haproxy-tools), [rhaproxy](https://github.com/jjuliano/rhaproxy), [haproxy_join](https://github.com/joewilliams/haproxy_join), or better yet, [Chef](http://www.opscode.com/chef) for that.
+* Doesn't talk to sockets, yet. Use [haproxy-ruby](https://github.com/inkel/haproxy-ruby) for now if you need this. I intend to add support for this using `Net::SSH` and `socat(1)` but for now HTTP is enough for my needs.
+
+ProTip
+------
+
+HA Proxy's awesome creator Willy Tarrreau loves [big text files](http://haproxy.1wt.eu/download/1.5/doc/configuration.txt) and [big, flat web pages](http://haproxy.1wt.eu/). If smaller, hyperlinked documents are more your style, you should know about the two alternative documentation sources:
+
+* http://code.google.com/p/haproxy-docs/
+* http://cbonte.github.com/haproxy-dconv/configuration-1.5.html
+

data/bin/haproxy_cluster

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+require "haproxy_cluster/cli"

data/lib/haproxy_cluster.rb

@@ -0,0 +1,95 @@
+require 'haproxy_cluster/member'
+require 'thread'
+
+class HAProxyCluster
+  def initialize(members = [])
+    @members = []
+    threads = []
+    members.each do |url|
+      threads << Thread.new do
+        @members << HAProxyCluster::Member.new(url)
+      end
+    end
+    threads.each{|t|t.join}
+  end
+
+  attr_accessor :members
+
+  # Poll the cluster, executing the given block with fresh data at the
+  # prescribed interval.
+  def poll(interval = 1.0)
+    first = true
+    loop do
+      start = Time.now
+      map { poll! } unless first
+      first = false
+      yield
+      sleep interval - (Time.now - start)
+    end
+  end
+
+  # Poll the entire cluster using exponential backoff until the the given
+  # block's return value always matches the condition (expressed as boolean or
+  # range).
+  #
+  # A common form of this is:
+  #
+  #   wait_for(true) do
+  #     api.servers.map{|s|s.ok?}
+  #   end
+  #
+  # This block would not return until every member of the cluster is available
+  # to serve requests.
+  # 
+  # wait_for(1!=1){false} #=> true
+  # wait_for(1==1){true}  #=> true
+  # wait_for(1..3){2}     #=> true
+  # wait_for(true){sleep} #=> Timeout
+  def wait_for (condition, &code)
+    results = map(&code)
+    delay = 1.5
+    loop do
+      if reduce(condition, results.values.flatten)
+        return true
+      end
+      if delay > 60
+        puts "Too many timeouts, giving up"
+        return false
+      end
+      delay *= 2
+      sleep delay
+      map { poll! }
+      results = map(&code)
+    end
+  end
+
+  # Run the specified code against every memeber of the cluster. Results are
+  # returned as a Hash, with member.to_s being the key.
+  def map (&code)
+    threads = []
+    results = {}
+    @members.each do |member|
+      threads << Thread.new do
+        results[member.to_s] = member.instance_exec &code
+      end
+    end
+    threads.each{|t|t.join}
+    return results
+  end
+
+  # Return true or false depending on the relationship between `condition` and `values`.
+  # `condition` may be specified as true, false, or a Range object.
+  # `values` is an Array of whatever type is appropriate for the condition.
+  def reduce (condition, values)
+    case condition.class.to_s
+    when "Range"
+      values.each{ |v| return false unless condition.cover? v }
+    when "TrueClass", "FalseClass" 
+      values.each{ |v| return false unless v == condition }
+    else
+      raise ArgumentError.new("Got #{condition.class.to_s} but TrueClass, FalseClass, or Range expected")
+    end
+    return true
+  end
+
+end

data/lib/haproxy_cluster/backend.rb

@@ -0,0 +1,32 @@
+require 'haproxy_cluster/stats_container'
+require 'haproxy_cluster/server_collection'
+
+class HAProxyCluster
+
+  class Backend < StatsContainer
+
+    def initialize
+      @servers = ServerCollection.new
+      super
+    end
+
+    attr_accessor :servers
+
+    def name
+      self.pxname
+    end
+
+    def rolling_restartable? (enough = 80)
+      up_servers = @servers.map{ |s| s.ok? }.count
+      if up_servers == 0
+        return true    # All servers are down; can't hurt!
+      elsif Rational(up_servers,@servers.count) >= Rational(enough,100)
+        return true    # Minumum % is satisfied
+      else
+        return false   # Not enough servers are up to handle restarting #{number_to_restart} at a time.
+      end
+    end
+
+  end
+
+end

data/lib/haproxy_cluster/cli.rb

@@ -0,0 +1,61 @@
+require 'rubygems'
+require 'optparse'
+require 'ostruct'
+require 'pp'
+require 'thread'
+require 'timeout'
+require 'haproxy_cluster/version'
+require 'haproxy_cluster'
+
+options = OpenStruct.new
+OptionParser.new do |opts|
+  opts.banner = "Usage: #{File.basename $0} ARGS URL [URL] [...]"
+  opts.on("-e", "--eval=CODE", "Ruby code block to be evaluated") do |o|
+    options.code_string = o
+  end
+  opts.on("-v", "--verbose", "Verbose logging") do
+    # TODO Need better coverage here
+    RestClient.log = STDERR
+  end
+  opts.on("--csv", "Assume result will be an Array of Arrays and emit as CSV") do
+    options.csv = true
+  end
+  opts.on("-t", "--timeout=SECONDS", "Give up after TIMEOUT seconds") do |o|
+    options.timeout = o.to_f
+  end
+  opts.on_tail("--version", "Show version") do
+    puts HAProxyCluster::Version
+    exit
+  end
+  opts.separator "URL should be the root of an HA Proxy status page, either http:// or https://"
+end.parse!
+options.urls = ARGV
+
+
+if options.code_string
+
+  if options.timeout
+    result = Timeout::timeout(options.timeout) do 
+      Kernel.eval( options.code_string, HAProxyCluster.new(options.urls).instance_eval("binding") )
+    end
+  else
+    result = Kernel.eval( options.code_string, HAProxyCluster.new(options.urls).instance_eval("binding") )
+  end
+
+  case result.class.to_s
+  when "TrueClass","FalseClass"
+    exit result == true ? 0 : 1
+  when "Hash"
+    pp result
+  when "Array"
+    if options.csv
+      result.each{|row| puts row.to_csv}
+    else
+      pp result
+    end
+  else
+    puts result
+  end
+
+end
+

data/lib/haproxy_cluster/member.rb

@@ -0,0 +1,54 @@
+require 'csv'
+require 'rest-client'
+require 'haproxy_cluster/backend'
+require 'haproxy_cluster/server'
+
+class HAProxyCluster
+  class Member
+    BACKEND = 1
+    SERVER  = 2
+
+    def initialize(source)
+      @source = source
+      @backends = Hash.new { |h,k| h[k] = Backend.new }
+      if source =~ /https?:/
+        @type = :url
+      else
+        @type = :file
+      end
+      poll!
+    end
+
+    def poll!
+      case @type
+      when :url
+        csv = RestClient.get(@source + ';csv').gsub(/^# /,'').gsub(/,$/,'')
+      when :file
+        File.read(@source)
+      end
+      CSV.parse(csv, { :headers => :first_row, :converters => :all, :header_converters => [:downcase,:symbol] } ) do |row|
+        case row[:type]
+        when BACKEND
+          @backends[ row[:pxname].to_sym ].stats.merge! row.to_hash 
+        when SERVER
+          @backends[ row[:pxname].to_sym ].servers << Server.new(row.to_hash, self)
+        end
+      end
+    end
+
+    attr_accessor :backends, :source, :type
+
+    def get_binding; binding; end
+    def to_s; @source; end
+
+    # Allow Backends to be accessed by dot-notation
+    def method_missing(m, *args, &block)
+      if @backends.has_key? m
+        @backends[m]
+      else
+        super
+      end
+    end
+
+  end
+end

data/lib/haproxy_cluster/server.rb

@@ -0,0 +1,60 @@
+require 'haproxy_cluster/stats_container'
+
+class HAProxyCluster
+
+  class Server < StatsContainer
+
+    def initialize (stats,member)
+      @member = member
+      super stats
+    end
+
+    def name
+      self.svname 
+    end
+
+    def backup?
+      self.bck == 1
+    end
+
+    def ok?
+      self.status == 'UP'
+    end
+
+    def enable!
+      modify! :enable
+    end
+
+    def disable!
+      modify! :disable
+    end
+
+    def wait_until_ok
+      return true if self.ok?
+      start = Time.now
+      until self.ok?
+        raise Timeout if Time.now > start + 10
+        sleep 1
+        @member.poll!
+      end
+      return true
+    end
+
+    private
+
+    def modify! (how)
+      case @member.type
+      when :url
+        RestClient.post @member.source, { :s => self.name, :action => how, :b => self.pxname }
+        @member.poll!
+      else
+        raise "Not implemented: #{how} on #{@member.type}"
+      end
+      return self.status
+    end
+
+    class Timeout < RuntimeError ; end
+
+  end
+
+end

data/lib/haproxy_cluster/server_collection.rb

@@ -0,0 +1,9 @@
+class HAProxyCluster
+  class ServerCollection < Array
+    def find(string)
+      self.select do |s|
+        s.name == string
+      end.first
+    end
+  end
+end

data/lib/haproxy_cluster/stats_container.rb

@@ -0,0 +1,23 @@
+require 'haproxy_cluster'
+
+# Backends present summary statistics for the servers they contain, and
+# individual servers also present their own specific data.
+class HAProxyCluster
+  class StatsContainer
+
+    def initialize(stats = {})
+      @stats = stats
+    end
+
+    attr_accessor :stats
+
+    def method_missing(m, *args, &block)
+      if @stats.has_key? m
+        @stats[m]
+      else
+        super
+      end
+    end
+
+  end
+end

data/lib/haproxy_cluster/version.rb

@@ -0,0 +1,3 @@
+class HAProxyCluster
+  VERSION = "0.0.1"
+end

metadata

@@ -0,0 +1,104 @@
+--- !ruby/object:Gem::Specification
+name: haproxy-cluster
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Jacob Elder
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-07-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rest-client
+  requirement: &70251272059180 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *70251272059180
+description: ! "haproxy-cluster\n===============\n\n> \"Can we survive a rolling restart?\"\n>\n>
+  \"How many concurrent connections right now across all load balancers?\"\n\nWhile
+  there are already a handfull of [HA Proxy](http://haproxy.1wt.edu) abstraction layers
+  on RubyGems, I wanted to be able to answer questions like those above and more,
+  quickly, accurately, and easily. So here's one more for the pile.\n\n`HAProxyCluster::Member`
+  provides an ORM for HA Proxy's status page.\n\n`HAProxyCluster` provides a simple
+  map/reduce-inspired framework on top of `HAProxyCluster::Member`.\n\n`haproxy_cluster`
+  provides a shell scripting interface for `HAProxyCluster`. Exit codes are meaningful
+  and intended to be useful from Nagios.\n\nDo you deploy new code using a sequential
+  restart of application servers? Using this common pattern carelessly can result
+  in too many servers being down at the same time, and cutomers seeing errors. `haproxy_cluster`
+  can prevent this by ensuring that every load balancer agrees that the application
+  is up at each stage in the deployment. In the example below, we will deploy a new
+  WAR to three Tomcat instances which are fronted by two HA Proxy instances. HA Proxy
+  has been configured with `option httpchk /check`, a path which only returns an affirmative
+  status code when the application is ready to serve requests.\n\n```bash\n#!bin/bash\nset
+  -o errexit\nservers=\"server1.example.com server2.example.com server3.example.com\"\nload_balancers=\"https://lb1.example.com:8888
+  http://lb2.example.com:8888\"\n\nfor server in $servers ; do\n    haproxy_cluster
+  --timeout=300 --eval \"wait_until(true){ myapp.rolling_restartable? }\" $load_balancers\n
+  \   scp myapp.war $server:/opt/tomcat/webapps\ndone\n```\n\nThe code block passed
+  to `--eval` will not return until every load balancer reports that at least 80%
+  of the backend servers defined for \"myapp\" are ready to serve requests. If this
+  takes more than 5 minutes (300 seconds), the whole deployment is halted.\n\nMaybe
+  you'd like to know how many transactions per second your whole cluster is processing.\n\n
+  \   $ haproxy_cluster --eval 'poll{ puts members.map{|m|m.myapp.rate}.inject(:+)
+  }' $load_balancers\n\nInstallation\n------------\n\n`gem install haproxy-cluster`\n\nRequires
+  Ruby 1.9.2 and depends on RestClient.\n\nNon-Features\n------------\n\n* Doesn't
+  try to modify configuration files. Use [haproxy-tools](https://github.com/subakva/haproxy-tools),
+  [rhaproxy](https://github.com/jjuliano/rhaproxy), [haproxy_join](https://github.com/joewilliams/haproxy_join),
+  or better yet, [Chef](http://www.opscode.com/chef) for that.\n* Doesn't talk to
+  sockets, yet. Use [haproxy-ruby](https://github.com/inkel/haproxy-ruby) for now
+  if you need this. I intend to add support for this using `Net::SSH` and `socat(1)`
+  but for now HTTP is enough for my needs.\n\nProTip\n------\n\nHA Proxy's awesome
+  creator Willy Tarrreau loves [big text files](http://haproxy.1wt.eu/download/1.5/doc/configuration.txt)
+  and [big, flat web pages](http://haproxy.1wt.eu/). If smaller, hyperlinked documents
+  are more your style, you should know about the two alternative documentation sources:\n\n*
+  http://code.google.com/p/haproxy-docs/\n* http://cbonte.github.com/haproxy-dconv/configuration-1.5.html\n\n"
+email:
+- jacob.elder@gmail.com
+executables:
+- haproxy_cluster
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- bin/haproxy_cluster
+- lib/haproxy_cluster/backend.rb
+- lib/haproxy_cluster/cli.rb
+- lib/haproxy_cluster/member.rb
+- lib/haproxy_cluster/server.rb
+- lib/haproxy_cluster/server_collection.rb
+- lib/haproxy_cluster/stats_container.rb
+- lib/haproxy_cluster/version.rb
+- lib/haproxy_cluster.rb
+homepage: https://github.com/jelder/haproxy_cluster
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ~>
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.11
+signing_key: 
+specification_version: 3
+summary: Inspect and manipulate collections of HA Proxy instances
+test_files: []
+has_rdoc: