barking_iguana-compound 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e3c49e4fa77451a0f78931175dfeedfa2736d3a8
+  data.tar.gz: dc148ad917a0182a1a8e94db603a5d12f53c5a85
+SHA512:
+  metadata.gz: f4a53fe0a6346889d69ccf9fa1933c4743ae4932afbde85799237ac837be6c10bb538cb58b14d202a2a53a12b9a4b2e1c79e2dcbe0a0b380312b93badc712eb3
+  data.tar.gz: 4890023cfae6d8c76e4fa97fa9cceb14f39930a37861b73f7a35eea57b0f96eaaf0d114a6ff684ff283b0083d9c5d11d9e572e933e8c3b12d31fd1c907c47f36

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in barking_iguana-compound.gemspec
+gemspec

data/README.md

@@ -0,0 +1,179 @@
+# BarkingIguana::Compound
+
+Compound testing for Ansible atoms.
+
+When you need to test several roles and/or playbooks together over their lifetime e.g. testing database failover and recovery.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'barking_iguana-compound'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install barking_iguana-compound
+
+## Usage
+
+Install it, as per above.
+
+Normally you'll use Compond as part of a Rake task, so include it in your
+`Rakefile`. Tell it to define Rake tasks for your compound tests - we assume
+these live in `test/compound` - and where your playbooks live, your control
+repository:
+
+```ruby
+require 'barking_iguana/compound'
+ansible_control_repo = File.dirname __FILE__
+BarkingIguana::Compound.new('test/compound', ansible_control_repo).define_rake_tasks
+```
+
+Now let's define a test managing hosts files. It's a trivial test, but it
+suffices to demonstrate the capacilities of Compound.
+
+### Writing a compound test
+
+Create a directory for the test:
+
+    $ mkdir -p test/compound/hosts_file_management
+
+Each test will have several _stages_, which represent parts of the expected
+lifecycle of your servers. For example, you set up your servers using Ansible,
+to you probably have a setup stage. You may be setting up a HA cluster, so
+perhaps you have a stage where the master in the cluster fails.
+
+Each stage involves several actions. We'll cover them in detail below, but in
+summary:
+
+The `setup` action runs first. For each stage this action decides which virtual
+machines should be powered on or off during the rest of the stage.
+
+Next, the `converge` action runs the playbook for that stage, if one is provided
+by you, to allow things to happen. The playbook may apply roles or kill
+processes, whatever you need to simulate what's happening to your servers at
+that part of the lifecycle.
+
+Finally, the `verify` stage runs some serverspec tests which you will define
+against the results of the `converge` action. This allows you to check what
+happens after the tasks in the playbook have been run e.g. you may verify that a
+standby server has become the master.
+
+Each stage lives in a directory inside the test. They're executed in
+alphabetical order, and you are encouraged to prefix each stage with numbers to
+make it very clear which one should execute in which order.
+
+We'll cover each of those actions in greater detail now, for the setup stage of
+our hosts file example.
+
+#### Test Stages
+
+Start by creating the directory for that stage:
+
+    $ mkdir -p test/compound/hosts_file_management/000-setup
+
+##### The Setup Action
+
+This action controls which virtual machines are available for you to work with,
+by looking at an Ansible inventory file in the stage directory and starting all
+the hosts it finds.
+
+Let's create a simple inventory file with 2 hosts and 1 group.
+
+The inventory file for each stage lives in the stage directory in a file called
+`inventory`, so in our example so for that's `test/compound/hosts_file_management/000-setup/inventory`.
+
+They're just normal Ansible inventory files, with only one restriction: the
+`ansible_host` _must_ start with `10.8.`.
+
+```
+[linux]
+host001 ansible_host=10.8.100.11
+host002 ansible_host=10.8.100.12
+```
+
+When the setup action for a stage with this inventory is run, Compound will
+launch two virtual machines for you to test with, `host001` and `host002` with
+the respective IP addresses.
+
+##### The Converge Action
+
+Now that virtual machines are available to use, the converge action can run
+the playbook for your stage against them.
+
+Playbooks live in the stage directory, in a file called `playbook.yml`. I'm
+inventive like that. For our example stage that's `test/compound/hosts_file_management/000-setup/playbook.yml`.
+
+For our setup action we'll apply the `hosts` role to all hosts.
+
+```
+---
+- hosts: all
+  roles:
+    - hosts
+```
+
+After a converge, Compound will attempt to verify whatever you ask it to.
+Onwards, to the verify action.
+
+##### The Verify Action
+
+Now we run automated tests to assert that our virtual machines behave like we
+think they should, after the stage playbook has been applied in the converge
+action.
+
+The verify action is based around serverspec tests, arranged around the name of
+the host which they test.
+
+For example, to test `host001` we'd create tests under a `test/compound/hosts_file_management/host001/`
+directory. Each test file must be suffixed with `_spec.rb`. A simple example
+would be checking that the other host of the pair is resolvable now that the
+hosts role has been applied to the hosts:
+
+```ruby
+# test/compound/hosts_file_management/host001/resolving_spec.rb
+describe host('host002') do
+  it "is resolvable" do
+    expect(subject).to be_resolvable
+  end
+end
+```
+
+```ruby
+# test/compound/hosts_file_management/host002/resolving_spec.rb
+describe host('host001') do
+  it "is resolvable" do
+    expect(subject).to be_resolvable
+  end
+end
+```
+
+Compound will take the appropriate actions to make sure your tests are run on
+the correct hosts, you don't need to worry about SSH keys, passwords or ports.
+
+### Running the tests
+
+Since we've asked Compound to define rake tasks above, we can run those. The tasks generated are based on the directory names we use in the tests. The above test can be run like this:
+
+    $ bundle exec rake compound:hosts_file_management
+
+You can see a list of all tests by asking Rake to list them:
+
+    $ bundle exec rake -T
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/barking_iguana-compound.
+

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/barking_iguana-compound.gemspec

@@ -0,0 +1,44 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'barking_iguana/compound/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "barking_iguana-compound"
+  spec.version       = BarkingIguana::Compound::VERSION
+  spec.authors       = ["Craig R Webster"]
+  spec.email         = ["craig@barkingiguana.com"]
+
+  spec.summary       = %q{Compound testing of Ansible playbooks}
+  spec.description   = spec.summary
+  spec.homepage      = "https://github.com/barkingiguana/compound"
+
+  # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
+  # to allow pushing to a single host or delete this section to allow pushing to any host.
+  if spec.respond_to?(:metadata)
+    spec.metadata['allowed_push_host'] = "https://rubygems.org"
+  else
+    raise "RubyGems 2.0 or newer is required to protect against " \
+      "public gem pushes."
+  end
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.13"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+
+  spec.add_dependency 'barking_iguana-logging'
+  spec.add_dependency 'barking_iguana-benchmark'
+  spec.add_dependency 'mixlib-shellout'
+  spec.add_dependency 'rspec-wait'
+  spec.add_dependency 'ansible_spec'
+  spec.add_dependency 'colorize'
+  spec.add_dependency 'hostlist_expression'
+  spec.add_dependency 'oj'
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "barking_iguana/compound"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/barking_iguana/compound.rb

@@ -0,0 +1,27 @@
+require 'barking_iguana/logging'
+require 'barking_iguana/benchmark'
+
+require 'erb'
+require 'mixlib/shellout'
+require 'hostlist_expression'
+require 'oj'
+require 'yaml'
+
+require 'barking_iguana/compound/version'
+require 'barking_iguana/compound/ansible'
+require 'barking_iguana/compound/ansible/inventory'
+require 'barking_iguana/compound/ansible/inventory_parser'
+require 'barking_iguana/compound/ansible/playbook'
+require 'barking_iguana/compound/test_stage'
+require 'barking_iguana/compound/test'
+require 'barking_iguana/compound/test_suite'
+require 'barking_iguana/compound/host_manager'
+require 'barking_iguana/compound/host'
+require 'barking_iguana/compound/vagrant'
+
+module BarkingIguana
+  module Compound
+  end
+end
+
+BarkingIguana::Logging.default_level = Logger.const_get(ENV['LOG_LEVEL'].upcase) if ENV['LOG_LEVEL']

data/lib/barking_iguana/compound/ansible.rb

@@ -0,0 +1,13 @@
+module BarkingIguana
+  module Compound
+    module Ansible
+      def self.inventory *args
+        Inventory.new *args
+      end
+
+      def self.playbook *args
+        Playbook.new *args
+      end
+    end
+  end
+end

data/lib/barking_iguana/compound/ansible/inventory.rb

@@ -0,0 +1,28 @@
+module BarkingIguana
+  module Compound
+    module Ansible
+      class Inventory
+        attr_accessor :path
+        private :path=
+
+        def initialize path
+          self.path = path
+        end
+
+        def hostgroups
+          InventoryParser.load_targets path
+        end
+
+        def hosts
+          h = hostgroups
+          h.reject! { |hg| hg[0].empty? }
+          hosts = h.values.flatten.uniq { |h| h['uri'] }
+          hosts.map do |data|
+            name = data['name'].gsub(/ .*/, '')
+            Host.new name: name, uri: data['uri']
+          end
+        end
+      end
+    end
+  end
+end

data/lib/barking_iguana/compound/ansible/inventory_parser.rb

@@ -0,0 +1,165 @@
+module BarkingIguana
+  module Compound
+    module Ansible
+      class InventoryParser
+        def self.get_variables(host, group_idx, hosts=nil)
+          vars = {}
+          p = self.get_properties
+
+          # roles default
+          p[group_idx]['roles'].each do |role|
+            vars = load_vars_file(vars ,"roles/#{role}/defaults/main.yml")
+          end
+
+          # all group
+          vars = load_vars_file(vars ,'group_vars/all', true)
+
+          # each group vars
+          if p[group_idx].has_key?('group')
+            vars = load_vars_file(vars ,"group_vars/#{p[group_idx]['group']}", true)
+          end
+
+          # each host vars
+          vars = load_vars_file(vars ,"host_vars/#{host}", true)
+
+          # site vars
+          if p[group_idx].has_key?('vars')
+            vars = merge_variables(vars, p[group_idx]['vars'])
+          end
+
+          # roles vars
+          p[group_idx]['roles'].each do |role|
+            vars = load_vars_file(vars ,"roles/#{role}/vars/main.yml")
+          end
+
+          # multiple host and children dependencies group vars
+          unless hosts.nil? || p[group_idx]["hosts_childrens"].nil?
+            hosts_childrens = p[group_idx]["hosts_childrens"]
+            next_find_target = hosts
+            while(!next_find_target.nil? && hosts_childrens.size > 0)
+              vars = load_vars_file(vars ,"group_vars/#{next_find_target}", true)
+              group_vars_file = find_group_vars_file(hosts_childrens,next_find_target)
+              next_find_target = group_vars_file
+              hosts_childrens.delete(group_vars_file)
+            end
+          end
+
+          return vars
+
+        end
+
+        # param  hash   {"server"=>["192.168.0.103"], "databases"=>["192.168.0.104"], "pg:children"=>["server", "databases"]}
+        # param  search ":children"
+        # param  k      "pg:children"
+        # return {"server"=>["192.168.0.103"], "databases"=>["192.168.0.104"], "pg"=>["192.168.0.103", "192.168.0.104"]}
+        def self.get_parent(hash,search,k)
+          k_parent = k.gsub(search,'')
+          arry = Array.new
+          hash["#{k}"].each{|group|
+            next if hash["#{group}"].nil?
+            arry = arry + hash["#{group}"]
+          }
+          h = Hash.new
+          h["#{k_parent}"] = arry
+          return h
+        end
+
+        def self.load_targets(file)
+          f = File.open(file).read
+          groups = Hash.new
+          group = ''
+          hosts = Hash.new
+          hosts.default = Hash.new
+          f.each_line{|line|
+            line = line.chomp
+            # skip
+            next if line.start_with?('#') #comment
+            next if line.empty? == true   #null
+
+            # get group
+            if line.start_with?('[') && line.end_with?(']')
+              group = line.gsub('[','').gsub(']','')
+              groups["#{group}"] = Array.new
+              next
+            end
+
+            # get host
+            host_name = line.split[0]
+            if group.empty? == false
+              if groups.has_key?(line)
+                groups["#{group}"] << line
+                next
+              elsif host_name.include?("[") && host_name.include?("]")
+                # www[01:50].example.com
+                # db-[a:f].example.com
+                hostlist_expression(line,":").each{|h|
+                  host = hosts[h.split[0]]
+                  groups["#{group}"] << get_inventory_param(h).merge(host)
+                }
+                next
+              else
+                # 1つのみ、かつ:を含まない場合
+                # 192.168.0.1
+                # 192.168.0.1 ansible_ssh_host=127.0.0.1 ...
+                host = hosts[host_name]
+                groups["#{group}"] << get_inventory_param(line).merge(host)
+                next
+              end
+            else
+              if host_name.include?("[") && host_name.include?("]")
+                hostlist_expression(line, ":").each{|h|
+                  hosts[h.split[0]] = get_inventory_param(h)
+                }
+              else
+                hosts[host_name] = get_inventory_param(line)
+              end
+            end
+          }
+
+          # parse children [group:children]
+          search = Regexp.new(":children".to_s)
+          groups.keys.each{|k|
+            unless (k =~ search).nil?
+              # get group parent & merge parent
+              groups.merge!(get_parent(groups,search,k))
+              # delete group children
+              if groups.has_key?("#{k}") && groups.has_key?("#{k.gsub(search,'')}")
+                groups.delete("#{k}")
+              end
+            end
+          }
+          return groups
+        end
+
+        # param ansible_ssh_port=22
+        # return: hash
+        def self.get_inventory_param(line)
+          host = Hash.new
+          # 初期値
+          host['name'] = line
+          host['port'] = 22
+          if line.include?(":") # 192.168.0.1:22
+            host['uri']  = line.split(":")[0]
+            host['port'] = line.split(":")[1].to_i
+            return host
+          end
+          # 192.168.0.1 ansible_ssh_port=22
+          line.split.each{|v|
+            unless v.include?("=")
+              host['uri'] = v
+            else
+              key,value = v.split("=")
+              host['port'] = value.to_i if key == "ansible_ssh_port" or key == "ansible_port"
+              host['private_key'] = value if key == "ansible_ssh_private_key_file"
+              host['user'] = value if key == "ansible_ssh_user" or key == "ansible_user"
+              host['uri'] = value if key == "ansible_ssh_host" or key == "ansible_host"
+              host['pass'] = value if key == "ansible_ssh_pass"
+              host['connection'] = value if key == "ansible_connection"
+            end
+          }
+          return host
+        end
+      end
+    end
+  end
+end