cmdb 0.1.0 → 2.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9212599111d82e3f236f06fb05aaf87ec0932155
-  data.tar.gz: 87200c6133a80e8f2bd02f02697aa0d42c694622
+  metadata.gz: df72629dd0ba9fcb24a33607ca6216e9a70ff188
+  data.tar.gz: 2daf8320e35c3cd58234d86b452b8574cc76e944
 SHA512:
-  metadata.gz: 86b209d6f11b166e981cd7a0eeac50f10a78a8efa5ef4db7c973642c4eb26aa519741b7ffbdbb3e1e266f794e9429c19fb646bdddb749ae18cdd0429f4fc3fa6
-  data.tar.gz: 33d7b4855c49ece993167e1dbfd7da504a337fae128a60096d037fcaa51f8b0815658b2ac2ffb32d1875563464b16e1447a85316429dc0d08a0e0c760fbf48e0
+  metadata.gz: 97c03113ce4aa1c50718edadd6c9649096a22916bf14a16721df59987f56745c8150131a5bbb63d6d6f2c3c9e4db59fff2568f2480e0e3488468400b4cc92253
+  data.tar.gz: 38cc4d3c2701dfc66b2d3f27470851bd14128e5ec4b3548e85fa292daced4d609cb43f4d63746a73e449cd410e78f109ea27f8f822b99e73a0d4a45f4fa6fd7c

data/.travis.yml

@@ -8,6 +8,5 @@ before_install:
 before_script:
 - bundle install
 script: 
-- bundle exec rake ci:spec
-- bundle exec rake ci:cucumber
-
+- bundle exec rake spec
+- bundle exec rake cucumber

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cmdb (0.1.0)
+    cmdb (2.6.0)
       diplomat (~> 0.15)
       listen (~> 3.0)
       trollop (~> 2.0)

data/README.md

@@ -1,47 +1,50 @@
 # CMDB
 
-[![TravisCI][travis_ci_img]](https://magnum.travis-ci.com/rightscale/cmdb)
-[travis_ci_img]: https://magnum.travis-ci.com/rightscale/cmdb.svg?token=ETyVMY7RGypDf6U8z6jc&branch=master
+[![TravisCI][travis_ci_img]](https://travis-ci.org/rightscale/cmdb)
+[travis_ci_img]: https://travis-ci.org/rightscale/cmdb.svg?branch=master
+
+*NOTE:* this gem is under heavy development and it is likely that v3 will contain several interface-breaking
+changes and simplifications. We encourage you to play with our toys and give us feedback on how you would
+like to see the project evolve, but if you use this gem for production-grade software, please make sure to
+pin to version `~> 2.6` in your Gemfile to avoid breakage!
 
 CMDB is a Ruby interface for consuming data from one or more configuration management databases
-(CMDBs). It is intended to support multiple CM technologies, including:
+(CMDBs) and making that information available to Web applications.
+
+It is intended to support multiple CM technologies, including:
   - JSON/YAML files on a local disk
   - consul
   - (someday) etcd
   - (someday) ZooKeeper
 
-The CMDB's command-line tool can also facilitate debugging by watching your application's files
-and sending SIGHUP (or the signal of your choice) to the app server if anything changes.
-
 Maintained by
  - [RightScale Inc.](https://www.rightscale.com)
 
 ## Why should I use this gem?
 
-By using a standardized API for querying inputs, your application is decoupled from the underlying
-configuration management database that our Operations team is using. Today we happen to use JSON
-files that live under /etc/rightscale, but tomorrow we might use a key/value store such as etcd or
-consul (possibly in combination with flat files). This gem will deal with all of that complexity
-and provide you with a simple interface for reading your app's configuration.
+CMDB supports two primary use cases:
+
+  1. Decouple your modern (12-factor) application from the CM mechanism that is used to deploy it,
+     transforming CMDB keys and values into the enviroment variables that your app expects.
+  2. Help you deploy your "legacy" application that expects its configuration to be written to
+     disk files, rewriting those files with data taken from the CMDB.
 
 The gem has two primary interfaces:
-- The `cmdb shim` command populates the environment with config data and/or rewrites hardcoded
+- The `cmdb shim` command populates the environment with values and/or rewrites hardcoded
   config files, then spawns your application. It can also be told to watch the filesystem for changes and
   send a signal e.g. `SIGHUP` to your application, bringing reload-on-edit functionality to any app.
-- The `CMDB::Interface` object provides a programmatic API for querying configuration. Its `#to_h`
+- The `CMDB::Interface` object provides a programmatic API for querying CMDBs. Its `#to_h`
   method transforms the whole configuration into an environment-friendly hash if you prefer to seed the
-  environment yourself.
-
-The CMDB gem is friendly to 12-factor and "Rails-style" apps and can be used with or without the shim,
-depending on your application's needs. 
+  environment yourself, without using the shim.
 
 # Getting Started
 
 ## Create CMDB Data Files
- 
+
 The shim looks in two locations to find data files. In order of precedence:
-1. `/var/lib/cmdb` -- typically used at deployment time
-2. `~/.cmdb` -- useful for developers when testing the app
+
+  1. `/var/lib/cmdb` -- typically at deployment time
+  2. `~/.cmdb` -- useful for developers when testing the app
 
 The base name (minus extension) of each file is important; it determines the top-level name of
 the keys in that file and it *must be unique* across all of the directories. For instance,
@@ -59,7 +62,7 @@ the following contents:
 
 ## Invoke the CMDB Shim
 
-For non-Ruby applications, or for situations where CMDB values are required 
+For non-Ruby applications, or for situations where CMDB values are required
 outside of the context of interpreted code, use `cmdb shim` to run
 your application. The shim can do several things for you:
 
@@ -83,8 +86,8 @@ for CMDB values that are serialized to the environment (as a JSON document, in t
 
 ### Rewriting configuration files with CMDB values
 
-If the `--dir` option is provided, the shim recursively scans your working 
-directory (`Dir.pwd`) for data files that contain replacement tokens; when a token is 
+If the `--dir` option is provided, the shim recursively scans your working
+directory (`Dir.pwd`) for data files that contain replacement tokens; when a token is
 found, it substitutes the corresponding CMDB key's value.
 
 Replacement tokens look like this: `<<name.of.my.key>>` and can appear anywhere in a file as a YAML
@@ -93,7 +96,7 @@ or JSON _value_ (but never a key).
 Replacement tokens should appear inside string literals in your configuration files so they don't
 invalidate syntax or render the files unparsable by other tools.
 
-The shim performs replacement in-memory and saves all of the edits at once, making the rewrite 
+The shim performs replacement in-memory and saves all of the edits at once, making the rewrite
 operation nearly atomic. If any keys are missing, then no files are changed on disk and the shim
 exits with a helpful error message.
 
@@ -115,7 +118,7 @@ I can run the following command in my application's root directory:
 
     bundle exec cmdb shim --dir=config rackup
 
-This will rewrite the files under config, replacing my configuration files as 
+This will rewrite the files under config, replacing my configuration files as
 follows:
 
     # config/database.yml
@@ -172,6 +175,27 @@ value of a CMDB key can be a string, boolean, number, nil, or a list of any of t
 
 CMDB keys *cannot* contain maps/hashes, nor can lists contain differently-typed data.
 
+When a CMDB key is accessed through the Ruby API or referenced with a file-rewrite <<token>>, its
+name always begins with the file or path name of its *source* (JSON file, consul path, etc).
+
+When a CMDB key is written into the process environment or accessed via `Source#to_h`, its name
+is "bare" and the source name is irrelevant.
+
+If we use a `--consul-prefix` of `/kv/rightscale/intregration/shard403/common`
+then a key names would look like `common.debug.enabled` and environment names
+would look like `DEBUG_ENABLED`. The same is true if we load a `common.json`
+file source from `/var/lib/cmdb`.
+
+A future version of cmdb will harmonize the treatment of names; the prefix
+will be insignificant to the key name and keys will look like environment
+variables.
+
+## Network Data Sources
+
+To read from a consul server, pass `--consul-url` with a consul server address
+and `--consul-prefix` one or more times with a top-level path to treat as a
+named source.
+
 ## Disk-Based Data Sources
 
 When the CMDB interface is initialized, it searches two directories for YAML files:
@@ -212,10 +236,10 @@ on the value of RACK_ENV or RAILS_ENV:
   - unset, development or test: CMDB chooses the highest-precedence file and ignores the others
     after printing a warning. Files in `/etc` win over files in `$HOME`, which win over
     files in the working directory.
-     
+
   - any other environment: CMDB fails with an error message that describes the problem and
     the locations of the overlapping files.
-   
+
 ### Ambiguous Key Names
 
 Consider a file that defines the following variables:
@@ -244,7 +268,3 @@ of the keys could change if the structure of the YML file changes.
 For this reason, any YAML file that defines an "ambiguous" key name will cause an error at
 initialization time. To avoid ambiguous key names, think of your YAML file as a tree and remember
 that _leaf nodes must define data_ and _internal nodes must define structure_.
-
-## Network Data Sources
-
-TODO: add support for etcd or similar

data/cmdb.gemspec

@@ -1,6 +1,11 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'cmdb/version'
+
 Gem::Specification.new do |spec|
   spec.name          = 'cmdb'
-  spec.version       = '0.1.0'
+  spec.version       = CMDB::VERSION
   spec.authors       = ['RightScale']
   spec.email         = ['rubygems@rightscale.com']
 
@@ -10,6 +15,8 @@ Gem::Specification.new do |spec|
   spec.license       = 'MIT'
 
   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
   spec.required_ruby_version = Gem::Requirement.new("~> 2.0")

data/exe/cmdb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'trollop'
+require 'cmdb'
+
+if gemspec = Gem.loaded_specs['cmdb']
+  gemspec_version = gemspec.version
+else
+  require_relative '../lib/cmdb/version'
+  gemspec_version = CMDB::VERSION
+end
+
+commands = {}
+CMDB::Commands.constants.each do |konst|
+  name = konst.to_s.downcase
+  commands[name] = CMDB::Commands.const_get(konst.to_sym)
+end
+
+# Use a Trollop parser for help/banner display, but do not actually parse
+# anything just yet.
+command_list = commands.keys - ['help']
+command_info = command_list.map { |c| "       * #{c}" }.join("\n")
+p = Trollop::Parser.new do
+  version "cmdb #{gemspec_version} (c) 2013-2014 RightScale, Inc."
+  banner <<-EOS
+A command-line interface for configuration management.
+
+Usage:
+       cmdb <command> [options]
+
+Where <command> is one of:
+#{command_info}
+
+To get help on a command:
+       cmdb help command
+  EOS
+
+  stop_on commands.keys
+end
+
+opts = Trollop::with_standard_exception_handling p do
+  raise Trollop::HelpNeeded if ARGV.empty?
+  p.parse ARGV
+  cmd = ARGV.shift
+  klass = commands[cmd]
+
+  if klass
+    klass.create.run
+  else
+    raise ArgumentError, "Unrecognized command #{cmd}"
+  end
+end

data/lib/cmdb/commands/help.rb

@@ -0,0 +1,75 @@
+require 'logger'
+require 'listen'
+
+module CMDB::Commands
+  class Help
+    def self.create
+      options = Trollop.options do
+        banner <<-EOS
+The 'shim' command adapts your applications for use with CMDB without coupling them to
+the CMDB RubyGem (or forcing you to write your applications in Ruby). It works by
+manipulating the environment or filesystem to make CMDB inputs visible, then invoking
+your app.
+
+To use the shim with apps that read configuration from the filesystem, use the --dir
+option to tell the shim where to rewrite configuration files. It will look for tokens
+in JSON or YML that look like <<cmdb.key.name>> and replace them with the value of
+the specified key.
+
+To use the shim with 12-factor apps, use the --env option to tell the shim to load
+every CMDB key into the environment. When using --env, the prefix of each key is
+omitted from the environment variable name, e.g. "common.database.host" is
+represented as DATABASE_HOST.
+
+To support "development mode" and reload your app whenever its files change on disk,
+use the --reload option and specify the name of a CMDB key that will enable this
+behavior.
+
+Usage:
+cmdb shim [options] -- <command_to_exec> [options_for_command]
+
+Where [options] are selected from:
+        EOS
+        opt :dir,
+            "Directory to scan for key-replacement tokens in data files",
+            :type => :string
+        opt :consul_url,
+            "The URL for talking to consul",
+            :type => :string
+        opt :consul_prefix,
+            "The prefix to use when getting keys from consul, can be specified more than once",
+            :type => :string,
+            :multi => true
+        opt :keys,
+            "Override search path(s) for CMDB key files",
+            :type => :strings
+        opt :pretend,
+            "Check for errors, but do not actually launch the app or rewrite files",
+            :default => false
+        opt :quiet,
+            "Don't print any output",
+            :default => false
+        opt :reload,
+            "CMDB key that enables reload-on-edit",
+            :type => :string
+        opt :reload_signal,
+            "Signal to send to app server when code is edited",
+            :type => :string,
+            :default => "HUP"
+        opt :env,
+            "Add CMDB keys to the app server's process environment",
+            :default => false
+        opt :user,
+            "Switch to named user before executing app",
+            :type => :string
+        opt :root,
+            "Promote named subkey to the root when it is present in a namespace",
+            :type => :string
+      end
+
+      self.new(ARGV, options)
+    end
+
+
+  end
+end

data/lib/cmdb/commands/shim.rb

@@ -0,0 +1,298 @@
+require 'logger'
+require 'listen'
+
+module CMDB::Commands
+  class Shim
+    def self.create
+      options = Trollop.options do
+        banner <<-EOS
+The 'shim' command adapts your applications for use with CMDB without coupling them to
+the CMDB RubyGem (or forcing you to write your applications in Ruby). It works by
+manipulating the environment or filesystem to make CMDB inputs visible, then invoking
+your app.
+
+To use the shim with apps that read configuration from the filesystem, use the --dir
+option to tell the shim where to rewrite configuration files. It will look for tokens
+in JSON or YML that look like <<cmdb.key.name>> and replace them with the value of
+the specified key.
+
+To use the shim with 12-factor apps, use the --env option to tell the shim to load
+every CMDB key into the environment. When using --env, the prefix of each key is
+omitted from the environment variable name, e.g. "common.database.host" is
+represented as DATABASE_HOST.
+
+To support "development mode" and reload your app whenever its files change on disk,
+use the --reload option and specify the name of a CMDB key that will enable this
+behavior.
+
+Usage:
+cmdb shim [options] -- <command_to_exec> [options_for_command]
+
+Where [options] are selected from:
+        EOS
+        opt :dir,
+            "Directory to scan for key-replacement tokens in data files",
+            :type => :string
+        opt :consul_url,
+             "The URL for talking to consul",
+             :type => :string
+        opt :consul_prefix,
+             "The prefix to use when getting keys from consul, can be specified more than once",
+             :type => :string,
+             :multi => true
+        opt :keys,
+            "Override search path(s) for CMDB key files",
+            :type => :strings
+        opt :pretend,
+            "Check for errors, but do not actually launch the app or rewrite files",
+            :default => false
+        opt :quiet,
+            "Don't print any output",
+            :default => false
+        opt :reload,
+            "CMDB key that enables reload-on-edit",
+            :type => :string
+        opt :reload_signal,
+            "Signal to send to app server when code is edited",
+            :type => :string,
+            :default => "HUP"
+        opt :env,
+            "Add CMDB keys to the app server's process environment",
+            :default => false
+        opt :user,
+            "Switch to named user before executing app",
+          :type => :string
+        opt :root,
+            "Promote named subkey to the root when it is present in a namespace",
+          :type => :string
+      end
+
+      self.new(ARGV, options)
+    end
+
+    # Irrevocably change the current user for this Unix process by calling the
+    # setresuid system call. This sets both the uid and gid (to the user's primary
+    # group).
+    #
+    # @param [String] login name of user to switch to
+    # @return [true]
+    # @raise [ArgumentError] if the named user does not exist
+    def self.drop_privileges(login)
+      pwent = Etc.getpwnam(login)
+      Process::Sys.setresgid(pwent.gid, pwent.gid, pwent.gid)
+      Process::Sys.setresuid(pwent.uid, pwent.uid, pwent.uid)
+      true
+    end
+
+    # @return [CMDB::Interface]
+    attr_reader :cmdb
+
+    # Create a Shim.
+    # @param [Array] command collection of string to pass to Kernel#exec; 0th element is the command name
+    # @options [String] :condif_dir
+    def initialize(command, options={})
+      @command         = command
+      @dir             = options[:dir]
+      @consul_url      = options[:consul_url]
+      @consul_prefixes = options[:consul_prefix]
+      @keys            = options[:keys] || []
+      @pretend         = options[:pretend]
+      @reload          = options[:reload]
+      @signal          = options[:reload_signal]
+      @env             = options[:env]
+      @user            = options[:user]
+      @root            = options[:root]
+
+      unless @keys.empty?
+        CMDB::FileSource.base_directories = @keys
+      end
+      unless @consul_url.nil?
+        CMDB::ConsulSource.url = @consul_url
+      end
+      if !@consul_prefixes.nil? && !@consul_prefixes.empty?
+        CMDB::ConsulSource.prefixes = @consul_prefixes
+      end
+
+      if options[:quiet]
+        CMDB.log.level = Logger::FATAL
+      end
+    end
+
+    # Run the shim.
+    #
+    # @raise [SystemExit] if something goes wrong
+    def run
+      @cmdb = CMDB::Interface.new(:root=>@root)
+
+      rewrote   = rewrite_files
+      populated = populate_environment
+
+      if (!rewrote && !populated && !@pretend && @command.empty?)
+        CMDB.log.warn "CMDB: nothing to do; please specify --dir, --env, or a command to run"
+        exit 7
+      end
+
+      launch_app
+    rescue CMDB::BadKey => e
+      CMDB.log.fatal "CMDB: Bad Key: malformed CMDB key '#{e.key}'"
+      exit 1
+    rescue CMDB::BadValue => e
+      CMDB.log.fatal "CMDB: Bad Value: illegal value for CMDB key '#{e.key}' in source #{e.url}"
+      exit 2
+    rescue CMDB::BadData => e
+      CMDB.log.fatal "CMDB: Bad Data: malformed CMDB data in source #{e.url}"
+      exit 3
+    rescue CMDB::ValueConflict => e
+      CMDB.log.fatal "CMDB: Value Conflict: #{e.message}"
+      e.sources.each do |s|
+        CMDB.log.fatal " - #{s.url}"
+      end
+      exit 4
+    rescue CMDB::NameConflict => e
+      CMDB.log.fatal "CMDB: Name Conflict: #{e.message}"
+      e.keys.each do |k|
+        CMDB.log.fatal " - #{k}"
+      end
+      exit 4
+    rescue CMDB::EnvironmentConflict => e
+      CMDB.log.fatal "CMDB: Environment Conflict: #{e.message}"
+      exit 5
+    rescue Errno::ENOENT => e
+      CMDB.log.fatal "CMDB: missing file or directory #{e.message}"
+      exit 6
+    end
+
+    private
+
+    # @return [Boolean]
+    def rewrite_files
+      return false unless @dir
+
+      CMDB.log.info 'Starting rewrite of configuration...'
+
+      rewriter  = CMDB::Rewriter.new(@dir)
+
+      total = rewriter.rewrite(@cmdb)
+
+      if rewriter.missing_keys.any?
+        missing = rewriter.missing_keys.map { |k| "  #{k}" }.join("\n")
+        CMDB.log.error "Cannot rewrite configuration; #{rewriter.missing_keys.size} missing keys:\n#{missing}"
+
+        exit -rewriter.missing_keys.size
+      end
+
+      report_rewrite(total)
+
+      if @pretend
+        false
+      else
+        rewriter.save
+      end
+    end
+
+    # @return [Boolean]
+    def populate_environment
+      return false unless @env
+
+      env = @cmdb.to_h
+
+      env.keys.each do |k|
+        raise CMDB::EnvironmentConflict.new(k) if ENV.key?(k)
+      end
+
+      if @pretend
+        false
+      else
+        env.each_pair do |k, v|
+          ENV[k] = v
+        end
+        true
+      end
+    end
+
+    def launch_app
+      if @command.any?
+        # log this now, so that it gets logged even if @pretend
+        CMDB.log.info "App will run as user #{@user}" if @user
+
+        if @reload && @cmdb.get(@reload)
+          CMDB.log.info "SIG#{@signal}-on-edit is enabled; fork app"
+          fork_and_watch_app unless @pretend
+        else
+          CMDB.log.info "SIG#{@signal}-on-edit is disabled; exec app"
+          exec_app unless @pretend
+        end
+      end
+    end
+
+    def exec_app
+      self.class.drop_privileges(@user) if @user
+      exec(*@command)
+    end
+
+    def fork_and_watch_app
+      # let the child share our stdio handles on purpose
+      pid = fork do
+        exec_app
+      end
+
+      CMDB.log.info("App (pid %d) has been forked; watching %s" % [pid, ::Dir.pwd])
+
+      t0 = Time.at(0)
+
+      listener = Listen.to(::Dir.pwd) do |modified, added, removed|
+        modified = modified.select { |fn| interesting?(fn) }
+        added = added.select { |fn| interesting?(fn) }
+        removed = removed.select { |fn| interesting?(fn) }
+        next if modified.empty? && added.empty? && removed.empty?
+
+        begin
+          dt = Time.now - t0
+          if dt > 15
+            Process.kill(@signal, pid)
+            CMDB.log.info "Sent SIG%s to app (pid %d) because (modified,created,deleted)=(%d,%d,%d)" %
+                          [@signal, pid, modified.size, added.size, removed.size]
+            t0 = Time.now
+          else
+            CMDB.log.error "Skipped SIG%s to app (pid %d) due to timeout (%d)" %
+                           [@signal, pid, dt]
+          end
+        rescue
+          CMDB.log.error "Skipped SIG%s to app (pid %d) due to %s" %
+                        [@signal, pid, $!.to_s]
+        end
+      end
+
+      listener.start
+
+      wpid, wstatus = nil, nil
+
+      loop do
+        begin
+          wpid, wstatus = Process.wait2(-1, Process::WNOHANG)
+          if wpid == pid && wstatus.exited?
+            break
+          elsif wpid
+            CMDB.log.info("Descendant (pid %d) has waited with %s" % [wpid, wstatus.inspect])
+          end
+        rescue
+          CMDB.log.error "Skipped wait2 to app (pid %d) due to %s" %
+                         [pid, $!.to_s]
+        end
+        sleep(1)
+      end
+
+      CMDB.log.info("App (pid %d) has exited with %s" % [wpid, wstatus.inspect])
+      listener.stop
+      exit(wstatus.exitstatus || 43)
+    end
+
+    def report_rewrite(total)
+      CMDB.log.info "Replaced #{total} variables in #{@dir}"
+    end
+
+    def interesting?(fn)
+      !File.basename(fn).start_with?('.')
+    end
+  end
+end

data/lib/cmdb/commands.rb

@@ -0,0 +1,7 @@
+module CMDB
+  module Commands
+  end
+end
+
+require 'cmdb/commands/help'
+require 'cmdb/commands/shim'

data/lib/cmdb/consul_source.rb

@@ -0,0 +1,77 @@
+require 'diplomat'
+
+module CMDB
+  class ConsulSource
+    # Regular expression to match array values
+    ARRAY_VALUE = /^\[(.*)\]$/
+
+    # The url to communicate with consul
+    @@url = nil
+    # The prefixes to use when getting keys from consul
+    @@prefixes = nil
+
+    def self.url=(url)
+      @@url = url
+    end
+
+    def self.url
+      @@url
+    end
+
+    def self.prefixes=(prefixes)
+      @@prefixes = prefixes
+    end
+
+    def self.prefixes
+      @@prefixes
+    end
+
+    # Initialize the configuration for consul source
+    def initialize(prefix)
+      Diplomat.configure do |config|
+        config.url = @@url
+      end
+      @prefix = prefix
+    end
+
+    # Get a single key from consul
+    def get(key)
+      value = Diplomat::Kv.get(dot_to_slash(key))
+      process_value(value)
+    rescue Diplomat::KeyNotFound
+      nil
+    end
+
+    # Not implemented for consul source
+    def each_pair(&block)
+      prefix = @prefix || ''
+      all = Diplomat::Kv.get(prefix, recurse: true)
+      all.each do |item|
+        dotted_prefix = prefix.split('/').join('.')
+        dotted_key = item[:key].split('/').join('.')
+        key = dotted_prefix == '' ? dotted_key : dotted_key.split("#{dotted_prefix}.").last
+        value = process_value(item[:value])
+        puts "Key: #{key}, Value: #{value}"
+        block.call(dotted_key.split("#{dotted_prefix}.").last, process_value(item[:value]))
+      end
+    rescue Diplomat::KeyNotFound
+    end
+
+    private
+
+    def process_value(val)
+      return JSON.load(val)
+    rescue Exception
+      return val
+    end
+
+    # Converts the dotted notation to a slashed notation. If a @@prefix is set, it applies the prefix.
+    # @example
+    #   "common.proxy.endpoints" => common/proxy/endpoints (or) shard403/common/proxy/endpoints
+    def dot_to_slash(key)
+      key = "#{@prefix}.#{key}" unless @prefix.nil?
+      key.split('.').join('/')
+    end
+  end
+end
+

data/lib/cmdb/file_source.rb

@@ -0,0 +1,102 @@
+require 'uri'
+
+module CMDB
+  # Data source that is backed by a YAML file that lives in the filesystem. The name of the YAML
+  # file becomes the top-level key under which all values in the YAML are exposed, preserving
+  # their exact structure as parsed by YAML.
+  #
+  # @example Use my.yml as a CMDB source
+  #    source = FileSource.new('/tmp/my.yml') # contains a top-level stanza named "database"
+  #    source['my']['database']['host'] # => 'db1-1.example.com'
+  class FileSource
+    # @return [URI] a file:// URL describing where this source's data comes from
+    attr_reader :prefix, :url
+
+    @@base_directories = ['/var/lib/cmdb', File.expand_path('~/.cmdb')]
+
+    # List of directories that will be searched (in order) for YML files at load time.
+    # @return [Array] collection of String
+    def self.base_directories
+      @@base_directories
+    end
+
+    # @param [Array] bd collection of String absolute paths to search for YML files
+    def self.base_directories=(bd)
+      @@base_directories = bd
+    end
+
+    # Construct a new FileSource from an input YML file.
+    # @param [String,Pathname] filename path to a YAML file
+    # @raise [BadData] if the file's content is malformed
+    def initialize(filename, root=nil)
+      filename = File.expand_path(filename)
+      @url = URI.parse("file://#{filename}")
+      @prefix = File.basename(filename, ".*")
+      @extension = File.extname(filename)
+      @data = {}
+
+      raw_bytes = File.read(filename)
+      raw_data  = nil
+
+      begin
+        case @extension
+        when /jso?n?$/i
+          raw_data = JSON.load(raw_bytes)
+        when /ya?ml$/i
+          raw_data = YAML.load(raw_bytes)
+        else
+          raise BadData.new(self.url, 'file with unknown extension; expected js(on) or y(a)ml')
+        end
+      rescue Exception
+        raise BadData.new(self.url, 'CMDB data file')
+      end
+
+      raw_data = raw_data[root] if !root.nil? && raw_data.key?(root)
+      flatten(raw_data, @prefix, @data)
+    end
+
+    # Get the value of key.
+    #
+    # @return [nil,String,Numeric,TrueClass,FalseClass,Array] the key's value, or nil if not found
+    def get(key)
+      @data[key]
+    end
+
+    # Enumerate the keys in this source, and their values.
+    #
+    # @yield every key/value in the source
+    # @yieldparam [String] key
+    # @yieldparam [Object] value
+    def each_pair(&block)
+      # Strip the prefix in the key and call the block
+      @data.each_pair { |k, v| block.call(k.split("#{@prefix}.").last, v) }
+    end
+
+    private
+
+    def flatten(data, prefix, output)
+      data.each_pair do |key, value|
+        key = "#{prefix}.#{key}"
+        case value
+        when Hash
+          flatten(value, key, output)
+        when Array
+          if value.all? { |e| e.is_a?(String) } ||
+               value.all? { |e| e.is_a?(Numeric) } ||
+               value.all? { |e| e == true } ||
+               value.all? { |e| e == false }
+            output[key] = value
+          else
+            # mismatched arrays: not allowed
+            raise BadValue.new(self.url, key, value)
+          end
+        when String, Numeric, TrueClass, FalseClass
+          output[key] = value
+        else
+          # nil and anything else: not allowed
+          raise BadValue.new(self.url, key, value)
+        end
+      end
+    end
+  end
+end

data/lib/cmdb/interface.rb

@@ -0,0 +1,165 @@
+require 'json'
+
+module CMDB
+  class Interface
+    # Create a new instance of the CMDB interface.
+    # @option settings [String] root name of subkey to consider as root
+    def initialize(settings={})
+      @root = settings[:root] if settings
+
+      namespaces = {}
+
+      load_file_sources(namespaces)
+      check_overlap(namespaces)
+
+      @sources = []
+      # Load from consul source first if one is available.
+      if !ConsulSource.url.nil?
+        if ConsulSource.prefixes.nil? || ConsulSource.prefixes.empty?
+          @sources << ConsulSource.new('')
+        else
+          ConsulSource.prefixes.each do |prefix|
+            @sources << ConsulSource.new(prefix)
+          end
+        end
+      end
+      # Register valid sources with CMDB
+      namespaces.each do |_, v|
+        @sources << v.first
+      end
+    end
+
+    # Retrieve the value of a CMDB key, searching all sources in the order they were initialized.
+    #
+    # @return [Object,nil] the value of the key, or nil if key not found
+    # @param [String] key
+    # @raise [BadKey] if the key name is malformed
+    def get(key)
+      raise BadKey.new(key) unless key =~ VALID_KEY
+      value = nil
+
+      @sources.each do |s|
+        value = s.get(key)
+        break unless value.nil?
+      end
+
+      value
+    end
+
+    # Retrieve the value of a CMDB key; raise an exception if the key is not found.
+    #
+    # @return [Object,nil] the value of the key
+    # @param [String] key
+    # @raise [MissingKey] if the key is absent from the CMDB
+    # @raise [BadKey] if the key name is malformed
+    def get!(key)
+      get(key) || raise(MissingKey.new(key))
+    end
+
+    # Enumerate all of the keys in the CMDB.
+    #
+    # @yield every key/value in the CMDB
+    # @yieldparam [String] key
+    # @yieldparam [Object] value
+    # @return [Interface] always returns self
+    def each_pair(&block)
+      @sources.each do |s|
+        s.each_pair(&block)
+      end
+
+      self
+    end
+
+    # Transform the entire CMDB into a flat Hash that can be merged into ENV.
+    # Key names are transformed into underscore-separated, uppercase strings;
+    # all runs of non-alphanumeric, non-underscore characters are tranformed
+    # into a single underscore.
+    #
+    # The transformation rules make it possible for key names to conflict,
+    # e.g. "apple.orange.pear" and "apple.orange_pear" cannot exist in
+    # the same flat hash. This method checks for such conflicts and raises
+    # rather than returning bad data.
+    #
+    # @raise [NameConflict] if two or more key names transform to the same
+    def to_h
+      values = {}
+      sources = {}
+
+      each_pair do |key, value|
+        env_key = key_to_env(key)
+        value = JSON.dump(value) unless value.is_a?(String)
+
+        if sources.key?(env_key)
+          raise NameConflict.new(env_key, [sources[env_key], key])
+        else
+          sources[env_key] = key
+          values[env_key] = value_to_env(value)
+        end
+      end
+
+      values
+    end
+
+    private
+
+    # Scan for CMDB data files and index them by namespace
+    def load_file_sources(namespaces)
+      # Consult standard base directories for data files
+      directories = FileSource.base_directories
+
+      # Also consult working dir in development environments
+      if CMDB.development?
+        local_dir   = File.join(Dir.pwd, '.cmdb')
+        directories += [local_dir]
+      end
+
+      directories.each do |dir|
+        (Dir.glob(File.join(dir, '*.js')) + Dir.glob(File.join(dir, '*.json'))).each do |filename|
+          source = FileSource.new(filename, @root)
+          namespaces[source.prefix] ||= []
+          namespaces[source.prefix] << source
+        end
+
+        (Dir.glob(File.join(dir, '*.yml')) + Dir.glob(File.join(dir, '*.yaml'))).each do |filename|
+          source = FileSource.new(filename, @root)
+          namespaces[source.prefix] ||= []
+          namespaces[source.prefix] << source
+        end
+      end
+    end
+
+    # Check for overlapping namespaces and react appropriately. This can happen when a file
+    # of a given name is located in more than one of the key-search directories. We tolerate
+    # this in development mode, but raise an exception otherwise.
+    def check_overlap(namespaces)
+      overlapping = namespaces.select { |_, sources| sources.size > 1 }
+      overlapping.each do |ns, sources|
+        exc = ValueConflict.new(ns, sources)
+
+        if CMDB.development?
+          CMDB.log.warn exc.message
+        else
+          raise exc
+        end
+      end
+    end
+
+    # Make an environment variable out of a key name
+    def key_to_env(key)
+      env_name = key
+      env_name.gsub!(/[^A-Za-z0-9_]+/,'_')
+      env_name.upcase!
+      env_name
+    end
+
+    # Make a CMDB value storable in the process environment (ENV hash)
+    def value_to_env(value)
+      case value
+      when String
+        value
+      else
+        JSON.dump(value)
+      end
+    end
+  end
+end

data/lib/cmdb/rewriter.rb

@@ -0,0 +1,141 @@
+require 'yaml'
+require 'json'
+
+module CMDB
+  # Tool that visits every file in a hierarchy and rewrites its references to CMDB inputs.
+  #
+  # References look like <<name.of.key>> and may be replaced with a scalar value
+  # or an array depending on the type of the input value.
+  class Rewriter
+    attr_reader :missing_keys
+
+    # Create a new shim to rewrite config files in a chosen dir and all subdirs.
+    # config_dir is transformed into an absolute path (if it isn't already)
+    # before any rewrite operations occur.
+    #
+    # @param [String,Pathname] config_dir
+    def initialize(config_dir)
+      @dir = File.expand_path(config_dir)
+      @rewriters = []
+    end
+
+    # Substitute CMDB input values into config files whenever a replacement token is encountered.
+    #
+    # @param [CMDB::Interface] cmdb
+    # @return [Integer] the number of variables replaced
+    def rewrite(cmdb)
+      raise Errno::ENOENT.new(@dir) unless File.directory?(@dir)
+
+      visit(@dir)
+      total = 0
+      @rewriters.each { |rw| total += rw.rewrite(cmdb) }
+
+      @missing_keys = @rewriters.map { |rw| rw.missing_keys}.flatten.uniq.sort
+
+      total
+    end
+
+    def save
+      @rewriters.each { |rw| rw.save }
+      true
+    end
+
+    private
+
+    # Recursively scan location for files to rewrite.
+    def visit(location)
+      if File.file?(location)
+        scan(location)
+      elsif File.directory?(location)
+        entries = Dir.glob("#{location}/*")
+        subdirs = entries.select { |e| File.directory?(e) }
+        files   = entries.select { |e| File.file?(e) }
+
+        subdirs.each do |entry|
+          visit(entry)
+        end
+
+        files.each do |entry|
+          visit(entry)
+        end
+      end
+    end
+
+    # Load a data file and attach a rewriter to it.
+    def scan(file)
+      case File.extname(file)
+      when '.yml', '.yaml'
+        @rewriters << FileRewriter.new(file, YAML)
+      when '.js', '.json'
+        @rewriters << FileRewriter.new(file, JSON)
+      end
+    end
+  end
+
+  # Tool that rewrites the contents of a single YAML, JSON or similar data file.
+  # The rewriting is done in-memory and isn't saved back to disk until someone
+  # calls #save, allowing the caller to check #missing_keys before making a
+  # decision whether to proceed.
+  class FileRewriter
+    # Regexp that matches a well-formed replacement token in YML or JSON
+    REPLACEMENT_TOKEN = /^<<(.*)>>$/
+
+    attr_reader :missing_keys
+
+    # Load YAML, JSON or similar into memory as a Ruby object graph.
+    def initialize(file, encoder)
+      @file = file
+      @encoder = encoder
+      @data = @encoder.load(File.read(file))
+    end
+
+    # Peform CMDB input replacement on in-memory objects. Validate that the result can be saved.
+    #
+    # @return [Integer] number of variables replaced
+    def rewrite(cmdb)
+      @total = 0
+      @missing_keys = []
+      @data = visit(cmdb, @data)
+
+      # Very important; DO NOT REMOVE. This is how we validate that #save will work.
+      @encoder.dump(@data)
+      raise Errno::EACCES.new(@file) unless File.writable?(@file)
+
+      @total
+    end
+
+    def save
+      File.open(@file, 'w') { |f| f.write @encoder.dump(@data) }
+    end
+
+    private
+
+    # Recurse through an object graph, finding replacement tokens and substituting the
+    # corresponding CMDB values.
+    def visit(cmdb, node)
+      if node.is_a?(Hash)
+        result = {}
+        node.each_pair do |k, v|
+          result[k] = visit(cmdb, v)
+        end
+      elsif node.is_a?(Array)
+        result = []
+        node.each do |v|
+          result << visit(cmdb, v)
+        end
+      elsif node.is_a?(String) && (m = REPLACEMENT_TOKEN.match(node))
+        value = cmdb.get(m[1])
+        if value.nil?
+          @missing_keys << m[1]
+        else
+          result = value
+          @total += 1
+        end
+      else
+        result = node
+      end
+
+      result
+    end
+  end
+end

data/lib/cmdb/version.rb

@@ -0,0 +1,3 @@
+module CMDB
+  VERSION = '2.6.0'
+end

data/lib/cmdb.rb

@@ -1,2 +1,140 @@
+require 'set'
+require 'singleton'
+
 module CMDB
+  # Values of RACK_ENV/RAILS_ENV that are considered to be "development," which relaxes
+  # certain runtime sanity checks.
+  DEVELOPMENT_ENVIRONMENTS = [nil, 'development', 'test'].freeze
+
+  # Regexp that matches valid key names. Key names consist of one or more dot-separated words;
+  # each word must begin with a lowercase alpha character and may contain alphanumerics or
+  # underscores.
+  VALID_KEY = /^[a-z][a-z0-9_]*(?:\.[a-z][a-z0-9_]*)*$/
+
+  class Error < StandardError; end
+
+  # Client asserted the existence of a key that does not exist in the CMDB.
+  class MissingKey < Error
+    # @return [String] the name of the offending key
+    attr_reader :key
+
+    # @param [String] name
+    def initialize(key)
+      @key = key
+      super("Key '#{key}' not found in CMDB")
+    end
+  end
+
+  # Client asked for an invalid or malformed key name.
+  class BadKey < Error
+    # @return [String] the name of the offending key
+    attr_reader :key
+
+    # @param [String] name
+    def initialize(key)
+      @key = key
+      super("Malformed key '#{key}'")
+    end
+  end
+
+  # A value of an unsupported type was encountered in the CMDB.
+  class BadValue < Error
+    # @return [URI] filesystem or network location of the bad value
+    attr_reader :url
+
+    # @return [String] the name of the key that contained the bad value
+    attr_reader :key
+
+    # @param [URI] url filesystem or network location of the bad value
+    # @param [String] key CMDB key name under which the bad value was found
+    # @param [Object] value the bad value itself
+    def initialize(url, key, value)
+      @url = url
+      @key = key
+      super("Values of type #{value.class.name} are unsupported")
+    end
+  end
+
+  # Malformed data was encountered in the CMDB or in an app's filesystem.
+  class BadData < Error
+    # @return [URI] filesystem or network location of the bad data
+    attr_reader :url
+
+    # @param [URI] url filesystem or network location of the bad value
+    # @param [String] context brief description of where data was found e.g. 'CMDB data file' or 'input config file'
+    def initialize(url, context=nil)
+      @url = url
+      super("Malformed data encountered #{(' in ' + context) if context}")
+    end
+  end
+
+  # Two or more sources contain keys for the same namespace; this is only allowed in development
+  # environments.
+  class ValueConflict < Error
+    attr_reader :sources
+
+    def initialize(ns, sources)
+      @sources = sources
+      super("Keys for namespace #{ns} are defined in #{sources.size} overlapping sources")
+    end
+  end
+
+  # Deprecated name for ValueConflict
+  Conflict = ValueConflict
+
+  # Two or more keys in different namespaces have an identical name. This isn't an error
+  # when CMDB is used to refer to keys by their full, qualified name, but it can become
+  # an issue when loading keys into the environment for 12-factor apps to process.
+  class NameConflict < Error
+    attr_reader :env
+    attr_reader :keys
+
+    def initialize(env, keys)
+      @env = env
+      @keys = keys
+      super("#{env} corresponds to #{keys.size} different keys")
+    end
+  end
+
+  # The CMDB is being exported to ENV, but one of its keys would overwrite a value that
+  # is already present in ENV. This should never happen, because the CMDB is designed to
+  # _augment_ the environment by providing a place to store boring (static, non-secret)
+  # inputs, and a given input should be set either in the CMDB or the environment, never
+  # both.
+  #
+  class EnvironmentConflict < Error
+    attr_reader :key
+
+    def initialize(key)
+      @key = key
+      super("#{key} is already present in the environment; cannot override with CMDB values")
+    end
+  end
+
+  module_function
+
+  def log
+    unless @log
+      @log = Logger.new(STDOUT)
+      @log.level = Logger::WARN
+    end
+
+    @log
+  end
+
+  def log=(log)
+    @log = log
+  end
+
+  # Determine whether CMDB is running in a development environment.
+  # @return [Boolean]
+  def development?
+    DEVELOPMENT_ENVIRONMENTS.include?(ENV['RACK_ENV'] || ENV['RAILS_ENV'])
+  end
 end
+
+require 'cmdb/consul_source'
+require 'cmdb/file_source'
+require 'cmdb/interface'
+require 'cmdb/rewriter'
+require 'cmdb/commands'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cmdb
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 2.6.0
 platform: ruby
 authors:
 - RightScale
 autorequire: 
-bindir: bin
+bindir: exe
 cert_chain: []
-date: 2016-02-27 00:00:00.000000000 Z
+date: 2016-02-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: listen
@@ -69,7 +69,8 @@ dependencies:
 description: Reads CMDB variables from files, Consul, and elsewhere.
 email:
 - rubygems@rightscale.com
-executables: []
+executables:
+- cmdb
 extensions: []
 extra_rdoc_files: []
 files:
@@ -85,11 +86,20 @@ files:
 - bin/console
 - bin/setup
 - cmdb.gemspec
+- exe/cmdb
 - fixtures/Gemfile
 - fixtures/Gemfile.lock
 - fixtures/app/widgets.rb
 - fixtures/config.ru
 - lib/cmdb.rb
+- lib/cmdb/commands.rb
+- lib/cmdb/commands/help.rb
+- lib/cmdb/commands/shim.rb
+- lib/cmdb/consul_source.rb
+- lib/cmdb/file_source.rb
+- lib/cmdb/interface.rb
+- lib/cmdb/rewriter.rb
+- lib/cmdb/version.rb
 homepage: https://github.com/rightscale/cmdb
 licenses:
 - MIT