bjn_inventory 1.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 92ee73b61dcc5d4e4d701b5e5007a99ad20b36a8
+  data.tar.gz: d2be8a63d506b5e4497e3db88a564a9a71c55c57
+SHA512:
+  metadata.gz: e42bd1e6b24bf66b89cc20a5c50c26877110c4d6f1edfaafe224a945f4917d4f78a119ac184d07538a06079f821b57ceabe4b2a896014ffaee94a1299bde145e
+  data.tar.gz: e676b4aeabdf0078d3b0d5098e87c8c7260bda80f0c3bfc568802711d4ba541727b8d510e13a47869e614dc75bbfc05edab286aeeb93ea817585c0e52e2bd1f2

data/.gitignore

@@ -0,0 +1,15 @@
+*.swp
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/d42/
+/aws/
+/inv/
+tasks/package/artifacts
+/tasks/test/reports/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.13.1

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in bjn_inventory.gemspec
+gemspec

data/README.md

@@ -0,0 +1,227 @@
+# BjnInventory
+
+This gem is designed to help materialize a standardized inventory of devices according to:
+
+* A model you specify
+* Multiple inventory sources you configure
+* A mapping between each source type and your standard model
+
+The materialized inventory can easily be converted to different formats:
+* Raw JSON or YAML
+* Ansible dynamic inventory
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'bjn_inventory'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install bjn_inventory
+
+## Usage
+
+An inventory is a list of devices, created from a specification, that specifies:
+* A device model
+* One or more sources, each of which has:
+  - A source-specific filename, URL or other location information
+  - A source-specific origin, which maps source data to the standard model
+* Context data
+  - These are just JSON files that you can refer to in your map and model
+
+Example usage (for an Ansible dynamic inventory script--but see `ansible-from`):
+```ruby
+require 'bjn_inventory'
+
+manifest = JSON.parse(File.read('inventory.json'))
+inventory = BjnInventory.new(manifest)
+ansible = {
+  group_by: [
+    environment,
+    region,
+    roles
+  ],
+  groups: {
+    webservers: ['www', 'stage-www']
+  }
+}
+# Adds a method to Array to convert to Ansible dynamic inventory output
+puts inventory.to_ansible(ansible)
+```
+
+inventory.json
+```ruby
+{ "model": "/etc/inventory/device.json",
+  "context": "/etc/inventory/data/context",
+  "sources": [
+    { "file": "/etc/inventory/data/device42.json",
+      "rules": "/etc/inventory/maps/device42.rb" },
+    { "file": "/etc/inventory/data/aws.json",
+      "rules": "/etc/inventory/maps/aws.rb" }
+  ]
+}
+```
+
+## Model
+
+When **bjn_inventory** produces an inventory, each device conforms
+to the device model. They have only the fields specified in the
+model, and defaults are filled in according to the model.
+
+Your inventory sources may not conform to the model. That's where
+there are [Rules#rules] to map the source entries into proper devices,
+according to the model you provide.
+
+The model generally takes the form of a JSON file, but can be embedded
+directly in the inventory specification.
+
+### Merge Rules
+
+When two devices need to be merged (for example, you are invoking
+**BjnInventory::Inventory#by(key)** and they have the same value
+for the *key* field), a new Device object is created with field
+values taken from the second, merged with the first (similar to
+a Hash merge). This merge is done according to the device model
+and the following rules:
+
+* Non-**nil** values take precedence over **nil**
+* Hashes are merged shallowly according to a standard
+  Ruby hash merge
+* Arrays are concatenated, except duplicate values from
+  the second device are not added
+* The second device's other values take precedence
+
+The resulting merged device's **#origin** method returns an
+array of the different origins used to merge it together.
+
+## Sources
+
+Each source you specify is read, in order. When you invoke
+**BjnInventory::Inventory#by(key)**, all sources are used. If two
+entries have the same *key*, they are merged together using the
+[merge rules#merge-rules] (basically, it merges intelligently based on
+the default values in the model). Order is strictly preserved here, so
+sources listed later in the list have data precedence over those
+listed earlier (like Ruby merge logic).
+
+Right now, only two kinds of sources are supported: inline, with the
+`entries` key, where the inventory entries are specified directly in
+the source; or with the `file` key, where the file must contain a JSON
+array of objects.
+
+In other words, it's assumed that you're separately downloading your
+source of inventory into JSON files for **bjn_inventory** to operate on.
+In the future, a download command or plugin may be allowed here.
+
+This package also provides a downloader command for AWS EC2. Use
+`aws-ec2-source` to download and minimally process an EC2 instance
+list to provide an inventory source.
+
+### Rules
+
+The mapping rules allow you to specify field values in the model and
+what calculation to perform from a source to derive them in the
+following DSL (domain-specific language). The rules consist of either
+text or a filename.
+
+The `origin` command takes a string, which specifies (for your
+convenience) the origin type of the resulting devices. For example,
+your AWS mapping rules might use `origin 'aws'` to identify resulting
+devices as having come from the AWS source. You can reuse rules
+files for different sources, or not, so it's up to you whether this
+origin represents a particular source of data or a particular kind
+of device.
+
+The `map` command takes a hash with a field name as a key and a mapping
+rule type (such as `ruby` or `jsonpath`). For example, the following
+rule specifies that the `name` field in your device model comes from
+either the `fqdn` field from the source entry, or the `name` field, if
+`fqdn` is **nil**:
+
+```ruby
+map name: ruby { |data| data['fqdn'] || data['name'] }
+```
+
+As an alternative to this syntax, you can mention the field name
+directly and omit the `ruby` keyword:
+
+```ruby
+name { |data| data['fqdn'] || data['name'] }
+```
+
+These are examples of **ruby** rules, which take the form of a Ruby
+block. This block accepts up to two arguments: the first is the data
+from the source entry (the "raw" data) in the form of a Hash, with
+the default values from the device model added; the second is the
+current **BjnInventory::Device** object. For example, if your model
+contains the `name` and `domain` fields, the following rule specifies
+a `fqdn` field that uses them:
+
+```ruby
+fqdn { |_data, device| device.name + '.' + device.domain }
+```
+
+There are other rule types available:
+
+A `jsonpath` rule uses a JSONPath expression in a string to set
+the device field. For example, the following rule sets the
+`name` field using the value of the `Name` tag (assuming AWS
+tagging, e.g. when using `aws-ec2-source`):
+
+```ruby
+name jsonpath '$.tags[?(@["key"]=="Name")].value'
+```
+
+A `synonym` rule simply makes the field a synonym for another
+device model field. For example, the following rule makes the
+field `management_ip` exactly synonymous with the `ip_address`
+field:
+
+```ruby
+management_ip synonym :ip_address
+```
+
+An `always` rule simply uses a constant value for that field.
+For example, the following rule sets the value of `system_type`
+to `ec2_instance` unconditionally:
+
+```ruby
+system_type always 'ec2_instance'
+```
+
+## Design Decisions
+
+* No calculated fields in the model.
+* No filtering: you can either filter the sources, or you can filter
+  the inventory after producing it
+* No validation of model fields (exactly, though merge is sensitive to
+  the model)
+* For now, no fancy file finding (filenames in manifests must be
+  absolute, or correct with respect to wherever you're running the
+  software). Possibly this might change with the addition of the
+  ability to pass a filename to **BjnInventory::Inventory.new()**.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install
+dependencies. Then, run `rake spec` to run the tests. You can also run
+`bin/console` for an interactive prompt that will allow you to
+experiment.
+
+To install this gem onto your local machine, run `bundle exec rake
+install`. To release a new version, update the version number in
+`version.rb`, and then run `bundle exec rake release`, which will
+create a git tag for the version, push git commits and tags, and push
+the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports can be sent to Ops Tools <tools+bjn_inventory@bluejeans.com>.
+

data/Rakefile

@@ -0,0 +1,17 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+
+task :report do
+    rm_r 'tasks/test/reports' if Dir.exist? 'tasks/test/reports'
+    mkdir_p 'tasks/test/reports'
+    # This command always succeeds; failed tests will be caught by the '--format p' output
+    sh 'rspec spec --format p --force-color --format RspecJunitFormatter --out tasks/test/reports/results.xml || :'
+end
+
+task :test do
+    sh 'rspec spec --format p --force-color'
+end

data/bin/ansible-from

@@ -0,0 +1,48 @@
+#!/usr/bin/env ruby
+
+# Drive an Ansible inventory from a file manifest
+# In order to use dynamically, still need to wrap
+# with a shell script (Ansible won't pass arguments)
+
+require 'trollop'
+require 'json'
+require 'logger'
+require 'bjn_inventory'
+require 'bjn_inventory/ansible'
+
+parser = Trollop::Parser.new do
+    version BjnInventory::VERSION
+    banner <<-USAGE.gsub(/^\s{8}/,'')
+        Usage:
+          ansible-from [options]
+    USAGE
+
+    opt :ansible, 'Specify ansible groupings file', required: true, type: :string
+    opt :manifest, 'Specify the manifest that defines this inventory', required: true, type: :string
+    opt :debug, 'Enable debug output', :short => '-D'
+    opt :syslog, 'Log to Syslog', :short => '-S'
+    stop_on_unknown
+end
+
+opt = Trollop::with_standard_exception_handling parser do
+    parser.parse(ARGV)
+end
+
+if opt[:syslog]
+    require 'syslog/logger'
+    logger = Syslog::Logger.new 'ansible-from'
+else
+    logger = Logger.new STDERR
+end
+
+if opt[:debug]
+    logger.level = Logger::DEBUG
+else
+    logger.level = Logger::WARN
+end
+
+manifest = JSON.parse(File.read(opt[:manifest]))
+ansible_spec = JSON.parse(File.read(opt[:ansible]))
+
+inventory = BjnInventory::Inventory.new(manifest.merge({logger: logger}))
+puts JSON.pretty_generate(inventory.by('name').to_ansible(ansible_spec))

data/bin/aws-ec2-source

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+
+require 'trollop'
+require 'logger'
+
+require 'bjn_inventory/version'
+require 'bjn_inventory/source_command/aws_ec2'
+
+parser = Trollop::Parser.new do
+    version BjnInventory::VERSION
+    banner <<-USAGE.gsub(/^\s{8}/,'')
+        Usage:
+          aws-ec2-source [options]
+    USAGE
+
+    opt :region, "Specify region (default is all regions)", type: :string, multi: true
+    opt :profile, "Specify AWS client profile", type: :string
+    opt :filters, 'Specify AWS EC2 filters in JSON (e.g. `[{"name":"tag:ENVIRONMENT","values":["Production"]}]`)', type: :string
+    opt :output, "Send output to file instead of stdout", type: :string
+    opt :verbose, "Produce verbose output", short: '-v'
+    opt :debug, 'Enable debug output', short: '-D'
+    opt :syslog, 'Log to Syslog', short: '-S'
+    stop_on_unknown
+end
+
+opt = Trollop::with_standard_exception_handling parser do
+    parser.parse(ARGV)
+end
+
+if opt[:syslog]
+    require 'syslog/logger'
+    logger = Syslog::Logger.new 'aws-ec2-source'
+else
+    logger = Logger.new STDERR
+end
+
+if opt[:debug]
+    logger.level = Logger::DEBUG
+elsif opt[:verbose]
+    logger.level = Logger::INFO
+else
+    logger.level = Logger::WARN
+end
+
+command = BjnInventory::SourceCommand::AwsEc2.new(opt.merge({ logger: logger }))
+command.run

data/bin/aws-rds-source

@@ -0,0 +1,47 @@
+#!/usr/bin/env ruby
+
+require 'trollop'
+require 'logger'
+
+require 'bjn_inventory/version'
+require 'bjn_inventory/source_command/aws_rds'
+
+parser = Trollop::Parser.new do
+    version BjnInventory::VERSION
+    banner <<-USAGE.gsub(/^\s{8}/,'')
+        Usage:
+          aws-rds-source [options]
+    USAGE
+
+    opt :region, "Specify region (default is all regions)", type: :string, multi: true
+    opt :profile, "Specify AWS client profile", type: :string
+    opt :filters, 'Specify AWS RDS filters in JSON (e.g. `[{"name":"db-cluster-id","values":["arn..."]}]`)', type: :string
+    opt :tag_filters, 'Specify tags and their required values (e.g. `[{"key":"ENVIRONMENT","values":["Production"]}]`)', type: :string
+    opt :output, "Send output to file instead of stdout", type: :string
+    opt :verbose, "Produce verbose output", short: '-v'
+    opt :debug, 'Enable debug output', short: '-D'
+    opt :syslog, 'Log to Syslog', short: '-S'
+    stop_on_unknown
+end
+
+opt = Trollop::with_standard_exception_handling parser do
+    parser.parse(ARGV)
+end
+
+if opt[:syslog]
+    require 'syslog/logger'
+    logger = Syslog::Logger.new 'aws-rds-source'
+else
+    logger = Logger.new STDERR
+end
+
+if opt[:debug]
+    logger.level = Logger::DEBUG
+elsif opt[:verbose]
+    logger.level = Logger::INFO
+else
+    logger.level = Logger::WARN
+end
+
+command = BjnInventory::SourceCommand::AwsRds.new(opt.merge({ logger: logger }))
+command.run

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "bjn_inventory"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start