broham 0.0.1 → 0.0.2

data/README.textile

@@ -1,17 +1,105 @@
-= broham
+h2. Broham: A simple, global, highly-available, none-too-bright service registry.
 
-Description goes here.
+**Broham always knows where his bros are, bro!** Using broham, a newly-created cloud machine can annouce its availability for a certain role ("nfs_server" or "db_slave-2"), allowing any other interested nodes to discover its public_ip, private_ip, etc.
+
+It uses Amazon's SimpleDB service, so the registry is global, highly-available, secure, accessible from any programming language (or even @curl@ if you're clever), and simple. As for Broham himself, he fills out a suit OK but isn't too bright: there's no monitoring or logging, for instance.
+
+Broham plays nicely with "Chef":http://github.com/opscode/chef and indeed was written to coordinate node assignment for a "hadoop cluster chef setup.":http://github.com/mrflip/hadoop_cluster_chef
+
+Examples:
+
+<pre><code>
+  require 'broham'
+  Settings.access_key_id     = 'XXXXXXXXXX'
+  Settings.secret_access_key = 'XXXXXXXXXX'
+
+  # create a context for the 'chad' cluster
+  class Chad < Broham::Cluster ; end
+  Chad.establish_connection
+  Chad.create_domain
+
+  # On NFS server start, register an nfs share
+  Chad.register_nfs_share '/home'
+  #=> #<Chad @attributes={"timestamp"=>["20100405072638"], "client_path"=>["/home"], "server_path"=>["/home"], "role"=>["nfs_server"], "public_ip"=>["250.249.248.247"], "private_ip"=>["192.168.69.22"], "default_ip"=>["192.168.69.22"]}
+  # On the nfs clients, get the local IP and server-side path of the share, ready to insert into /etc/fstab
+  Chad.nfs_device_path 
+  #=> "192.168.69.22:/home"
+
+  # Register as one of many nodes with a given role
+  Chad.register_as_next 'dj'
+  #=> #<Chad @attributes={"timestamp"=>["20100405072931"], "role"=>["dj-1"], "idx"=>["1"], "default_ip"=>["192.168.69.10"], "public_ip"=>["250.249.248.247"], "private_ip"=>["192.168.69.10"]}
+  
+  # Find the highest-yet-registered node in the 'dj' role and immediately
+  # register as the next one. Even if a thundering herd of hosts try to register
+  # in this role, Broham will ensure that exactly one host claims each index.
+  Chad.host('dj') ; Chad.register_as_next 'dj'
+  #=> #<Chad @attributes={"timestamp"=>["20100405073626"], "role"=>["dj"], "idx"=>["2"], "default_ip"=>["192.168.69.14"], "public_ip"=>["250.249.248.247"], "private_ip"=>["192.168.69.14"]}]
+  #=> #<Chad @attributes={"timestamp"=>["20100405073626"], "role"=>["dj-4"], "idx"=>["4"], "default_ip"=>["192.168.69.22"], "public_ip"=>["250.249.248.247"], "private_ip"=>["192.168.69.22"]}]
+</code></pre>
+
+Alternate interface:
+
+<pre><code>
+  Chad.yo! 'beer-stand'
+  #<Chad @attributes={"timestamp"=>["20100405071446"], "role"=>["beer-stand"], "public_ip"=>["250.249.248.247"], "private_ip"=>["10.0.69.69"]}
+
+  Chad.sup? 'beer-stand'
+  #<Chad @attributes={"timestamp"=>["20100405071446"], "role"=>["beer-stand"], "public_ip"=>["250.249.248.247"], "private_ip"=>["10.0.69.69"]}
+</code></pre>
+
+h4. Commandline Interface
+
+<pre><code>
+  $ broham-register chad nfs_server  --set-path=/home
+  Registering as nfs_server in chad cluster, with {:path=>"/home"}
+  {"timestamp":["20100405092638"],"role":["nfs_server"],"path":["/home"],"default_ip":["192.168.69.22"]}
+
+  $ broham-host chad nfs_server
+  {"timestamp":["20100405092638"],"role":["nfs_server"],"path":["/home"],"default_ip":["192.168.69.22"]}
+
+  # Alternative interface works too.
+  $ broham-sup chad nfs_server
+  {"timestamp":["20100405092638"],"role":["nfs_server"],"path":["/home"],"default_ip":["192.168.69.22"]}
+
+  $ broham-unregister-all chad nfs_server
+  Removing nfs_server from chad cluster
+  {"timestamp":["20100405092638"],"role":["nfs_server"],"path":["/home"],"default_ip":["192.168.69.22"]}
+</code></pre>
+
+h4. Setup
+
+For setup, we recommend configliere
+    
+<pre><code>
+  require 'configliere'
+  Configliere.use :config_file
+  Settings.read 'broham.yaml'; Settings.resolve!
+</code></pre>
+
+h3. 
+
+bq. Like Broseph Stalin, you are leading the way to the dictatorship of the broletariate. It is truly revbrolutionary. Like the Bro v. Wade of our generation. You brobliterate the enemy from the very peak of Mt. Brolympus. That's some shit. That's brolific. But that's the kind of bro you are. -- "Zach Caldwell":http://j.mp/amongbros
+
+
+h3. Warnings!
+
+Make sure you are using a recent (>= 1.11,0) version of right_aws, and set the SDB_API_VERSION environment variable to '2009-04-15':
+
+<pre><code>
+  export SDB_API_VERSION='2009-04-15'
+</code></pre>
+
+For @register_as_next@, you'll need a version of right_aws that supports conditional puts: "http://github.com/mrflip/right_aws":http://github.com/mrflip/right_aws
+  
+
+h3. Note on Patches/Pull Requests
 
-== Note on Patches/Pull Requests
- 
 * Fork the project.
 * Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a
-  future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Add tests for it. This is important so I don't break it in a future version unintentionally.
+* Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
-== Copyright
+h3. Copyright
 
 Copyright (c) 2010 Philip (flip) Kromer. See LICENSE for details.

data/VERSION

@@ -1 +1 @@
-0.0.1
+0.0.2

data/bin/broham-host

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'broham'
+require 'broham/script'
+require 'json'
+
+cluster = Broham.get_cluster_settings
+cluster_name, role_name = [Settings[:cluster_name], Settings[:role_name]]
+
+$stderr.puts %Q{Retrieving '#{role_name}' for cluster '#{cluster_name}'}
+resp = cluster.host role_name
+$stderr.puts resp.to_pretty_json

data/bin/broham-register

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'broham'
+require 'broham/script'
+Settings.define :set, :description => %Q{Any arg prefixed with "--set" will become an extra arg to register: 'broham-register foo --set-path=/path/to/foo' sets :path => '/path/to/foo' as an additional attribute}, :type => Hash
+
+cluster = Broham.get_cluster_settings
+cluster_name, role_name, broham_args = [Settings[:cluster_name], Settings[:role_name], Settings[:set]||{}]
+
+$stderr.puts %Q{Registering as #{role_name} in #{cluster_name} cluster, with #{broham_args.inspect}}
+resp = cluster.register role_name, broham_args
+node = cluster.host role_name
+$stderr.puts resp.to_pretty_json
+

data/bin/broham-unregister-all

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'broham'
+require 'broham/script'
+
+cluster = Broham.get_cluster_settings
+cluster_name, role_name = [Settings[:cluster_name], Settings[:role_name] ]
+
+$stderr.puts %Q{Removing all #{role_name} from #{cluster_name} cluster}
+resps = cluster.unregister_all role_name
+$stderr.puts resps.map{|resp| resp.to_pretty_json }
+

data/broham.gemspec

@@ -5,28 +5,31 @@
 
 Gem::Specification.new do |s|
   s.name = %q{broham}
-  s.version = "0.0.1"
+  s.version = "0.0.2"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Philip (flip) Kromer"]
   s.date = %q{2010-04-05}
   s.description = %q{Bro! Broham always knows where his bros are, bro. Using broham, a newly-created cloud machine can annouce its availability for a certain role ("nfs_server" or "db_slave-2"), allowing any other interested nodes to discover its public_ip, private_ip, etc. See also: http://j.mp/amongbros}
   s.email = %q{flip@infochimps.org}
+  s.executables = ["broham-host", "broham-register", "broham-unregister-all"]
   s.extra_rdoc_files = [
     "LICENSE",
-     "README.rdoc",
      "README.textile"
   ]
   s.files = [
     ".document",
      ".gitignore",
      "LICENSE",
-     "README.rdoc",
      "README.textile",
      "Rakefile",
      "VERSION",
+     "bin/broham-host",
+     "bin/broham-register",
+     "bin/broham-unregister-all",
      "broham.gemspec",
      "lib/broham.rb",
+     "lib/broham/script.rb",
      "spec/broham_spec.rb",
      "spec/spec.opts",
      "spec/spec_helper.rb"

data/lib/broham.rb

@@ -1,10 +1,15 @@
-require './settings.rb'
+# SimpleDB interface
 require 'right_aws'
 RightAws::SdbInterface::API_VERSION = '2009-04-15'
 require 'sdb/active_sdb'
+# Machine information
 require 'ohai'
 OHAI_INFO = Ohai::System.new unless defined?(OHAI_INFO)
 OHAI_INFO.all_plugins
+# Settings from Configliere
+require 'configliere'; Configliere.use :define
+Settings.define :access_key,        :required => true, :description => "Amazon AWS access key ID     -- found in your AWS console (http://bit.ly/awsconsole)"
+Settings.define :secret_access_key, :required => true, :description => "Amazon AWS secret access key -- found in your AWS console (http://bit.ly/awsconsole)"
 
 #
 # Make sure you are using a recent (>= 1.11,0) version of right_aws, and set the
@@ -13,105 +18,131 @@ OHAI_INFO.all_plugins
 #
 
 #
+# Broham expects a hash constant +Settings+ with values for +:secret_access_key+
+# and +:access_key+. The configliere gem (http://github.com/mrflip/configliere)
+# can help with that.
 #
-#
-class Broham < RightAws::ActiveSdb::Base
-  # Returns the last-registered host in the given role
-  def self.host role
-    select_by_role(role, :order => 'timestamp DESC')
-  end
+module Broham
+  class Cluster < RightAws::ActiveSdb::Base
+    # Returns the last-registered host in the given role
+    def self.host role
+      select_by_role(role, :order => 'timestamp DESC')
+    end
 
-  # Returns all hosts in the given role
-  def self.hosts role
-    select_all_by_role(role, :order => 'timestamp DESC')
-  end
+    # Returns all hosts in the given role
+    def self.hosts role
+      select_all_by_role(role, :order => 'timestamp DESC')
+    end
 
-  def self.register role, attrs={}
-    ahost = host(role) || new
-    ahost.attributes = ({:role => role, :timestamp => timestamp,
-        :private_ip => my_private_ip, :public_ip => my_public_ip, :default_ip => my_default_ip,
-        :fqdn => my_fqdn}.merge(attrs))
-    success = ahost.save
-    success ? self.new(success) : false
-  end
+    def self.host_attrs(role)
+      { :role => role, :timestamp => timestamp,
+        :private_ip => my_private_ip, :public_ip => my_public_ip, :default_ip => my_default_ip, :fqdn => my_fqdn  }
+    end
 
-  def yo!(*args)      register *args ; end
-  def sup?(*args)      host *args    ; end
-  def sup_bros?(*args) hosts *args   ; end
+    def self.register role, attrs={}
+      ahost = host(role) || new
+      ahost.attributes = (host_attrs(role).merge(attrs))
+      success = ahost.save
+      success ? self.new(success) : false
+    end
 
+    #
+    # Enlists as the next among many machines filling the given role.
+    #
+    # This is just a simple counter: it doesn't check whether the machine is
+    # already enlisted under a different index, or whether there are missing
+    # indices.
+    #
+    # It uses conditional save to be sure that the count is consistent
+    #
+    def self.register_as_next role, attrs={}
+      my_idx = 0
+      100.times do
+        ahost = host(role) || new
+        current_max_idx = ahost[:idx] && ahost[:idx].first
+        my_idx          = (current_max_idx.to_i + 1)
+        ahost.attributes = host_attrs(role).merge({ :idx => my_idx.to_s }.merge(attrs))
+        expected = current_max_idx ? {:idx => (current_max_idx.to_i + rand(5)).to_s} : {}
+        success = ahost.save_if(expected)
+        break if success
+      end
+      register role+'-'+my_idx.to_s, { :idx => my_idx }.merge(attrs)
+    end
 
-  #
-  # Enlists as the next among many machines filling the given role.
-  #
-  # This is just a simple counter: it doesn't check whether the machine is
-  # already enlisted under a different index, or whether there are missing
-  # indices.
-  #
-  # It uses conditional save to be sure that the count is consistent
-  #
-  def self.register_as_next role, attrs={}
-    my_idx = 0
-    100.times do
-      ahost = host(role) || new
-      current_max_idx = ahost[:idx] && ahost[:idx].first
-      my_idx          = (current_max_idx.to_i + 1)
-      ahost.attributes = ({:role => role, :timestamp => timestamp, :idx => my_idx.to_s }.merge(attrs))
-      expected = current_max_idx ? {:idx => (current_max_idx.to_i + rand(5)).to_s} : {}
-      success = ahost.save_if(expected)
-      break if success
+    # alternative syntax for #register
+    def self.yo!(*args)       register *args ; end
+    # alternative syntax for #register_as_next
+    def self.yo_yo_yo!(*args) register_as_next *args ; end
+    # alternative syntax for #host
+    def self.sup?(*args)      host *args    ; end
+    # alternative syntax for #hosts
+    def self.sup_bros?(*args) hosts *args   ; end
+
+    #
+    # Removes all registrations for the given role
+    #
+    def self.unregister_all role
+      select_all_by_role(role).each(&:delete)
     end
-    register role+'-'+my_idx.to_s, { :idx => my_idx }.merge(attrs)
-  end
 
-  #
-  # Removes all registrations for the given role
-  #
-  def self.unregister_all role
-    select_all_by_role(role).each(&:delete)
-  end
+    #
+    # Registration attributes
+    #
+
+    def self.my_private_ip()        OHAI_INFO[:cloud][:private_ips].first rescue nil ; end
+    def self.my_public_ip()         OHAI_INFO[:cloud][:public_ips].first  rescue nil ; end
+    def self.my_default_ip()        OHAI_INFO[:ipaddress]                            ; end
+    def self.my_fqdn()              OHAI_INFO[:fqdn]                                 ; end
+    def self.my_availability_zone() OHAI_INFO[:ec2][:availability_zone]              ; end
+    def self.timestamp()            Time.now.utc.strftime("%Y%m%d%H%M%S")            ; end
+
+    def private_ip()        self['private_ip'       ] || default_ip ; end
+    def public_ip()         self['public_ip'        ] || default_ip ; end
+    def default_ip()        self['default_ip'       ] ; end
+    def fqdn()              self['fqdn'             ] ; end
+    def availability_zone() self['availability_zone'] ; end
+    def idx()
+      self['idx']
+    end
 
-  #
-  # Registration attributes
-  #
+    def self.establish_connection
+      @connection ||= RightAws::ActiveSdb.establish_connection(Settings[:access_key], Settings[:secret_access_key])
+    end
 
-  def self.my_private_ip()        OHAI_INFO[:cloud][:private_ips].first rescue nil ; end
-  def self.my_public_ip()         OHAI_INFO[:cloud][:public_ips].first  rescue nil ; end
-  def self.my_default_ip()        OHAI_INFO[:ipaddress]                            ; end
-  def self.my_fqdn()              OHAI_INFO[:fqdn]                                 ; end
-  def self.my_availability_zone() OHAI_INFO[:ec2][:availability_zone]              ; end
-  def self.timestamp()            Time.now.utc.strftime("%Y%m%d%H%M%S")            ; end
-
-  def private_ip()      self['private_ip'       ] ; end
-  def public_ip()       self['public_ip'        ] ; end
-  def default_ip()      self['default_ip'       ] ; end
-  def fqdn()            self['fqdn'             ] ; end
-  def availability_zone() self['availability_zone'] ; end
-  def idx() self['idx']
-  end
+    # Register an nfs server share
+    def self.register_nfs_share server_path, client_path=nil, role='nfs_server'
+      client_path ||= server_path
+      register(role, :server_path => server_path, :client_path => client_path)
+    end
 
-  def self.establish_connection
-    @connection ||= RightAws::ActiveSdb.establish_connection(Settings[:access_key], Settings[:secret_access_key])
-  end
+    # NFS: device path, for stuffing into /etc/fstab
+    def self.nfs_device_path role='nfs_server'
+      nfs_server = host(role) or return
+      [nfs_server.private_ip, nfs_server[:server_path]].join(':')
+    end
 
-  #
-  #
-  #
+    # Hadoop: master jobtracker node
+    def self.hadoop_jobtracker(role='hadoop_jobtracker') ; host(role) ; end
+    # Hadoop: master namenode
+    def self.hadoop_namenode(  role='hadoop_namenode')   ; host(role) ; end
+    # Hadoop: cloudera desktop node
+    def self.cloudera_desktop( role='cloudera_desktop')  ; host(role) ; end
+
+    def to_hash() attributes ; end
+    def to_pretty_json
+      to_hash.reject{|k,v| k.to_s == 'id'}.to_json
+    end
 
-  # NFS: device path, for stuffing into /etc/fstab
-  def self.nfs_device_path role='nfs_server'
-    nfs_server = host(role) or return
-    [nfs_server[:private_ip], nfs_server[:remote_path]].join(':')
   end
 
-  # Hadoop: master jobtracker node
-  def self.hadoop_jobtracker(role='hadoop_jobtracker') ; host(role) ; end
-  # Hadoop: master namenode
-  def self.hadoop_namenode(  role='hadoop_namenode')   ; host(role) ; end
-  # Hadoop: cloudera desktop node
-  def self.cloudera_desktop( role='cloudera_desktop')  ; host(role) ; end
 
-  def to_hash() attributes ; end
+  #
+  # Metaprogramming
+  #
+  def self.new cluster
+    cluster_klass = '::'+Extlib::Inflection.classify(cluster.to_s)
+    module_eval(%Q{ class #{cluster_klass} < Broham::Cluster ; end })
+    cluster_klass.constantize rescue nil
+  end
 end
 
-Broham.establish_connection
-Broham.create_domain

data/lib/broham/script.rb

@@ -0,0 +1,23 @@
+module Broham
+
+  def self.get_cluster_settings
+    Configliere.use :commandline, :config_file
+    Settings.read('broham.yaml')
+    Settings.resolve!
+    Settings[:cluster_name] = Settings.rest.shift
+    Settings[:role_name]    = Settings.rest.shift
+    check_args!
+    cluster = Broham.new(Settings[:cluster_name])
+    cluster.establish_connection
+    cluster.create_domain
+    cluster
+  end
+
+  def self.check_args!
+    if (Settings[:cluster_name].blank? || Settings[:role_name].blank?)
+      warn "Please supply a cluster name and a role as the first two arguments"
+      exit(-1)
+    end
+  end
+
+end

metadata

@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
   segments: 
   - 0
   - 0
-  - 1
-  version: 0.0.1
+  - 2
+  version: 0.0.2
 platform: ruby
 authors: 
 - Philip (flip) Kromer
@@ -45,24 +45,28 @@ dependencies:
   version_requirements: *id002
 description: "Bro! Broham always knows where his bros are, bro. Using broham, a newly-created cloud machine can annouce its availability for a certain role (\"nfs_server\" or \"db_slave-2\"), allowing any other interested nodes to discover its public_ip, private_ip, etc. See also: http://j.mp/amongbros"
 email: flip@infochimps.org
-executables: []
-
+executables: 
+- broham-host
+- broham-register
+- broham-unregister-all
 extensions: []
 
 extra_rdoc_files: 
 - LICENSE
-- README.rdoc
 - README.textile
 files: 
 - .document
 - .gitignore
 - LICENSE
-- README.rdoc
 - README.textile
 - Rakefile
 - VERSION
+- bin/broham-host
+- bin/broham-register
+- bin/broham-unregister-all
 - broham.gemspec
 - lib/broham.rb
+- lib/broham/script.rb
 - spec/broham_spec.rb
 - spec/spec.opts
 - spec/spec_helper.rb

data/README.rdoc

@@ -1,38 +0,0 @@
-h2. Broham: A simple, global, highly-available, none-too-bright service registry. Broham always knows where his bros are, bro.
-
-Bro! Broham always knows where his bros are, bro. Using broham, a newly-created
-cloud machine can annouce its availability for a certain role ("nfs_server" or
-"db_slave-2"), allowing any other interested nodes to discover its public_ip,
-private_ip, etc.
-
-Examples:
-
-
-
-Alternate interface:
-
-
-
----------------------
-
-h3. 
-
-bq. Like Broseph Stalin, you are leading the way to the dictatorship of the
-broletariate. It is truly revbrolutionary. Like the Bro v. Wade of our
-generation. You brobliterate the enemy from the very peak of
-Mt. Brolympus. That's some shit. That's brolific. But that's the kind of bro you
-are. -- "Zach Caldwell":http://j.mp/amongbros
-
-h3. Note on Patches/Pull Requests
- 
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a
-  future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
-
-h3. Copyright
-
-Copyright (c) 2010 Philip (flip) Kromer. See LICENSE for details.