dopi 0.17.0

data/README.md

@@ -0,0 +1,309 @@
+# Dopi
+
+DOPi is the "inner" part of the Deployment Orchestrater for Puppet (DOP).
+It is the part that connects into your nodes and runs commands in a defined
+order.
+
+The main purpose of DOPi is to get your nodes into a state where they can
+run Puppet or any other config management. It will also allows you to
+orchestrate this Puppet runs so you can setup your nodes in the desired order.
+
+DOPi orchestrates puppet runs, mco calls and custom commands over different nodes
+
+DOPi uses a DOP plan file to find out what it has to run in what order on
+which nodes. To learn more about the syntax of this DOP plan file make sure
+you checkout the Documentation in [dop_common](https://github.com/swisscom/dop_common).
+
+If you are new to DOPi make sure you check out the [getting started guide](doc/getting_started.md).
+
+## Change Log
+
+Dopi is currently under heavy development and should not be considered stable. If you are
+upgrading make sure you carefully ready the [Change Log](CHANGELOG.md)
+
+## DOPi as a library
+
+### Install 
+
+Add this line to your application's Gemfile:
+
+    gem 'dopi'
+
+And then execute:
+
+    $ bundle
+
+### Usage Example
+
+    require 'dopi'
+
+    Dopi.configure do |config|
+      config.role_variable = 'my_role'
+      config.role_default  = 'base'
+    end
+
+    plan_parser = DopCommon::Plan.new(YAML.load_file(plan_file))
+    plan = Dopi::Plan.new(plan_parser)
+    plan.run
+
+    puts "Plan status: #{plan.state.to_s}"
+    plan.steps.each do |step|
+      puts "[#{step.state.to_s}] #{step.name}"
+      step.commands.each do |command|
+        puts "  [#{command.state.to_s}] #{command.node.fqdn}"
+      end
+    end
+
+#### With DOP plan cache
+
+This will persist the plan in the DOP plan cache.
+
+    require 'dopi'
+
+    Dopi.configure do |config|
+      config.role_variable = 'my_role'
+      config.role_default  = 'base'
+    end
+
+    plan = Dopi.add_plan(plan_file)
+    Dopi.run_plan(plan)
+
+    puts "Plan status: #{plan.state.to_s}"
+    plan.steps.each do |step|
+      puts "[#{step.state.to_s}] #{step.name}"
+      step.commands.each do |command|
+        puts "  [#{command.state.to_s}] #{command.node.fqdn}"
+      end
+    end
+    
+### DOPi as a CLI
+
+### Install
+
+Install the gem
+
+    $ gem install dopi
+
+Help on all available options
+
+    $ dopi help
+
+### Usage Example
+
+First you have to add a plan to the plan cache:
+
+    $ dopi add spec/data/plan/example_deploment_plan_test.yaml 
+    example_deploment_plan_test 
+
+This will return the plan name which can be used to run other
+commands on that plan. You can get a list of all the plans in the
+cache by running:
+
+    $ dopi list
+    example_deploment_plan_test
+
+You can get information about the state of a plan with the show command
+and the name of the plan:
+
+    $ dopi show example_deploment_plan_test
+    [ready] test_run
+      [ready] mysql01.example.com
+      [ready] web01.example.com
+      [ready] web02.example.com
+      [ready] haproxy01.example.com
+      [ready] haproxy02.example.com
+    [ready] Make sure we can login to all nodes
+      [ready] mysql01.example.com
+      [ready] web01.example.com
+      [ready] web02.example.com
+      [ready] haproxy01.example.com
+      [ready] haproxy02.example.com
+    [ready] ssh_test_run
+      [ready] mysql01.example.com
+    [ready] run_puppet
+      [ready] mysql01.example.com
+      [ready] web01.example.com
+      [ready] web02.example.com
+      [ready] haproxy01.example.com
+      [ready] haproxy02.example.com
+    [ready] run_puppet2
+      [ready] mysql01.example.com
+      [ready] web01.example.com
+      [ready] web02.example.com
+      [ready] haproxy01.example.com
+      [ready] haproxy02.example.com
+
+You can run the plan with the run command and the name:
+
+    $ dopi run example_deploment_plan_test
+
+
+## Plan File Format
+
+For a general description of the DOP plan file format, please see the
+[dop_common](https://github.com/swisscom/dop_common/blob/master/README.md)
+documentation. The documentation in this gem will focus on the command hashes for all
+the basic plugins which are shipped with DOPi and on how to create your own custom plugins.
+
+### How to use Plugins
+
+DOPi uses plugins to run commands on the nodes. Each step in the plan has one
+command and as many verify_commands as needed. DOPi will run all the verify_commands
+before the command and will run the command only if one of them fails.
+
+In general a plugin is specified like this:
+
+```YAML
+    - name "My new Step"
+      nodes: 'all'
+      command:
+        plugin: 'my_plugin_name'
+        parameter1: 'foo'
+        parameter2: 'bar'
+```
+
+Some of the Plugins don't actually need parameters, so they can be called with the short form:
+
+```YAML
+    - name "My new Step"
+      nodes: 'all'
+      command: 'my_simple_plugin'
+```
+
+### Verify Commands
+
+It is usually a good idea to check if a step is required to run. This way you can make your
+plans idempotent. You can define any number of verify commands. If they all are successful
+DOPi will skip the run. There are a hand full of plugins who are written exactly for this
+purpose.
+
+```YAML
+    - name "Create file if it does not exist"
+      command:
+        verify_commands:
+          - plugin; 'ssh/file_exists'
+            file: '/tmp/somefile'
+        plugin: 'ssh/custom'
+        exec: 'echo'
+        arguments: "'Hello World' > /tmp/somefile"
+```
+
+### Generic Plugin Parameters
+
+There are some generic parameters every plugin supports:
+
+#### plugin_timeout (optional)
+
+`default: 300`
+
+The time in seconds after which DOPi will kill the thread and mark the step as failed.
+
+#### verify_after_run (optional)
+
+`default: false`
+
+The verify commands will be executed again after the command run and the step will
+only succeed if the verify commands all successful.
+
+### Command Execution Plugins
+
+This are the plugins generally used in steps as commands
+
+[custom](doc/plugins/custom.md)
+
+[ssh/custom](doc/plugins/ssh/custom.md)
+
+[ssh/wait_for_login](doc/plugins/ssh/wait_for_login.md)
+
+[ssh/reboot](doc/plugins/ssh/reboot.md)
+
+[ssh/puppet_agent_run](doc/plugins/ssh/puppet_agent_run.md)
+
+[ssh/file_replace](doc/plugins/ssh/file_replace.md)
+
+[ssh/file_deploy](doc/plugins/ssh/file_deploy.md)
+
+[mco/rpc](doc/plugins/mco/rpc.md)
+
+[winrm/cmd](doc/plugins/winrm/cmd.md)
+
+[winrm/powershell](doc/plugins/winrm/powershell.md)
+
+[winrm/wait_for_login](doc/plugins/winrm/wait_for_login.md)
+
+[winrm/reboot](doc/plugins/winrm/reboot.md)
+
+[winrm/puppet_agent_run](doc/plugins/winrm/puppet_agent_run.md)
+
+### Verification Plugins
+
+This are some helper plugins that check stuff on the nodes. They are
+usefull for verify_commands. However, every normal plugin can be used
+as a verify_command and vice versa.
+
+[ssh/file_contains](doc/plugins/ssh/file_contains.md)
+
+[ssh/file_exists](doc/plugins/ssh/file_exists.md)
+
+[winrm/file_contains](doc/plugins/winrm/file_contains.md)
+
+[winrm/file_exists](doc/plugins/winrm/file_exists.md)
+
+## Example Plans
+
+There are some examples for DOPi in the sources which are also used for tests
+
+[DOPi test environment setup](spec/integration/dopi/build_dop_test_environment.yaml)
+
+More can be found in the plans directory.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+### Run the test suit
+
+Most of the tests depend on Vagrant to create an actual test environment where the DOPi
+plugins can be tested under real conditions.
+
+To setup the test suit you need a working vagrant (https://www.vagrantup.com/) installation
+with the dop_common gem added as a plugin:
+
+    cd /tmp
+    git clone https://github.com/swisscom/dop_common.git
+    cd dop_common
+    gem build dop_common.gemspec
+    vagrant plugin install ./dop_common-*.gem
+
+After you install the plugin you have to setup the test machines with the rake task:
+
+    cd /path/to/dopi/
+    bundle install --path .bundle
+    bundle exec rake spec:prep
+
+You should always rerun 'spec:prep' to make sure your test environment is started
+and setup correctly.
+
+The tests will connect to the machines and for now you require some hosts file entries
+to work correctly. Add the following lines to your /etc/hosts:
+
+    # Host entries for DOPi test environment
+    192.168.56.101 puppetmaster.example.com
+    192.168.56.102 broker.example.com
+    192.168.56.103 linux01.example.com
+    192.168.56.104 linux02.example.com
+    192.168.56.105 linux03.example.com
+    192.168.56.106 windows01.example.com
+
+Now you are ready to run the test suit:
+
+    bundle exec rake
+
+If you reboot your machine or stop the test machines you can make sure the test
+environment is ready and built by simply running the setup again:
+
+    bundle exec rake testenv:setup

data/Rakefile

@@ -0,0 +1,44 @@
+require 'rspec/core/rake_task'
+
+namespace :spec do
+  desc 'setup the test environment (this builds multiple virtual machines with vagrant and virtualbox)'
+  task :prep do
+    Bundler.with_clean_env do
+      sh('vagrant up')
+      hiera = 'spec/fixtures/puppet/hiera.yaml'
+      plan  = 'spec/fixtures/testenv_plan.yaml'
+      sh('bundle package --all')
+      sh("bundle exec bin/dopi --verbosity debug --trace --hiera_yaml #{hiera} oneshot #{plan}")
+    end
+  end
+
+  desc 'destory the test environment'
+  task :clean do
+    Bundler.with_clean_env do
+      sh('vagrant destroy')
+    end
+  end
+
+  RSpec::Core::RakeTask.new(:unit) do |t|
+    t.pattern = 'spec/unit/**/*_spec.rb'
+  end
+
+  RSpec::Core::RakeTask.new(:integration) do |t|
+    t.pattern = 'spec/integration/**/*_spec.rb'
+  end
+
+  desc 'Run single plan file from spec/integration/dopi/plans/<name>.yaml, <name> is taken from env DOPI_TEST_PLAN'
+  RSpec::Core::RakeTask.new(:plan) do |t|
+    t.pattern = 'spec/integration/dopi/plan.rb'
+  end
+
+  desc 'Run single plan file from spec/integration/dopi/fail_check_plans/<name>.yaml and expect it to fail, <name> is taken from env DOPI_TEST_PLAN'
+  RSpec::Core::RakeTask.new(:failplan) do |t|
+    t.pattern = 'spec/integration/dopi/failplan.rb'
+  end
+end
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+

data/Vagrantfile

@@ -0,0 +1,64 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# This will create the dop test machines with from a dop plan
+# Please make sure you have installed dop_common as a Vagrant plugin
+#
+# $ vagrant plugin install path/to/dop_common-x.x.x.gem
+#
+require 'dop_common'
+
+DOP_PLAN = 'spec/fixtures/testenv_plan.yaml'
+
+hash = YAML.load(DopCommon::PreProcessor.load_plan(DOP_PLAN))
+plan = DopCommon::Plan.new(hash)
+
+Vagrant.configure(2) do |config|
+
+  # Create vagrant boxes from the plan file
+  plan.nodes.each do |node|
+
+    config.vm.define node.name do |machine|
+      machine.vm.box = node.image
+
+      interface = node.interfaces.first
+      machine.vm.network "private_network", ip: interface.ip
+
+      # windows/linux specific settings
+      if node.name[/^windows/, 0]
+        machine.vm.guest = :windows
+        machine.vm.hostname = node.name.split('.', 2)[0]
+      else
+        machine.vm.hostname = node.name
+      end
+
+      # disable the default folder sync on the nodes
+      # and rsync all the stuff to the puppetmaster
+      if node.name == 'puppetmaster.example.com'
+        config.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ".bundle/"
+      else
+        config.vm.synced_folder ".", "/vagrant", disabled: true
+      end
+    end
+  end
+
+
+  # Test run nodes
+  config.vm.define 'rhel6.example.com' do |machine|
+    machine.vm.box = 'puppetlabs/centos-6.6-64-nocm'
+    machine.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ".bundle/"
+    machine.vm.provision 'shell', inline: 'yum install -y gcc git epel-release centos-release-scl'
+    machine.vm.provision 'shell', inline: 'yum install -y ncurses-devel sshpass'
+    machine.vm.provision 'shell', inline: 'yum install -y rh-ruby22 rh-ruby22-ruby-devel rh-ruby22-rubygem-bundler'
+  end
+
+  config.vm.define 'rhel7.example.com' do |machine|
+    machine.vm.box = 'puppetlabs/centos-7.0-64-nocm'
+    machine.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ".bundle/"
+    machine.vm.provision 'shell', inline: 'yum install -y gcc git centos-release-scl'
+    machine.vm.provision 'shell', inline: 'yum install -y ncurses-devel sshpass'
+    machine.vm.provision 'shell', inline: 'yum install -y rh-ruby22 rh-ruby22-ruby-devel rh-ruby22-rubygem-bundler'
+  end
+
+end
+

data/bin/dopi

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require 'dopi/cli'
+exit Dopi::Cli.run(ARGV)

data/doc/getting_started.md

@@ -0,0 +1,247 @@
+# Getting started with DOPi
+
+This guide is a good starting point if you are new to DOPi and want to
+learn how to use it to automate your node provisioning.
+
+To start make sure you installed DOPi according to the instructions in
+the [README](README.md).
+
+You will find all the examples written in this tutorial in the example
+directory next to this file.
+
+## Hello World
+
+Let's start with a typical hello world plan file and analyze what
+parts are required to actually use DOPi. Use your favored editor to
+create a file called 'hello_world.yaml' with the following content:
+
+    name: 'hello_world'
+
+    infrastructures:
+      'test':
+        type: 'baremetal'
+
+    nodes:
+      'testlinux.example.com':
+        infrastructure: 'test'
+
+    steps:
+      - name: 'write hello world'
+        nodes: 'all'
+        command:
+          plugin: 'custom'
+          exec: 'echo'
+          arguments: '"hello world"'
+
+Now let's try to use DOPi to run that plan. There are usually a
+few steps required to run a plan. Here is what you have to do:
+
+    $ dopi add hello_world.yaml
+    New plan hello_world was added
+    hello_world
+
+We just added the 'hello_world' plan to our plan cache. DOPi tells
+us that this was successful and returns the handle 'hello_world'.
+We will require this handle to do other stuff with the plan we
+just added. The handle is in fact the same as the 'name' attribute
+at the very top of the plan file. It is always good to chose a name
+which describes best what the content is. You can use letters,
+numbers, '-' and '_' but no spaces or any other kind of character in
+the name.
+
+The plan cache is just a directory which holds all your added plans
+and some files for the state of the plans. This cache will also be
+used by the dop-hiera plugin if you have any configuration data in
+your plans (more on this later).
+
+The standard directory for the plan cache is ~/.dop/cache if you run
+DOPi as user or /var/lib/dop/plans if you run it as root. You can also
+overwrite this with the global '--plan_cache_dir' option.
+
+To list what plans are already added in your plan cache you can simply
+list them with DOPi:
+
+    $ dopi list
+    hello_world
+
+We can see that it contains our 'hello_world' plan file. To see what
+the state of the plan is and what steps are defined we can show
+the content of the plan with DOPi:
+
+    $ dopi show hello_world
+    [ready] hello_world
+      [ready] default
+        [ready] write hello world
+          [ready] testlinux.example.com
+
+Now we see a whole hierarchy with every part in the state 'ready'.
+The first line represents the state of the whole plan. You can only
+run a plan which is in the state ready. If part of the plan is in
+the state 'failed' it will reflect that in the plan state.
+
+The second line represents the step set. You can separate your steps
+in your plan into different sets, so they can be executed independendly
+of eachother (more on that later). We did not specify a step set in our
+hello_world.yaml, so DOPi will create a default step set for us.
+
+The third line represents the step and it is named after the name
+attribute we specified in the first step in the hello_world.yaml file.
+A step is executed on a number of nodes and they are listed under each
+step with their state, which represents the last line in the output.
+
+So now that we know that our plan is ready we can execute it for the
+first time. But maybe we want to check first what it actually does.
+To accomplish this we run the plan in 'noop' mode first to see
+exactly what commands get executed:
+
+    $ dopi run --noop hello_world
+    Starting signal handling
+    Starting to run step 'write hello world'
+      [Command] testlinux.example.com : Running command custom
+      [Command] testlinux.example.com : (NOOP) Executing 'echo "hello world"' for command custom
+      [Command] testlinux.example.com : (NOOP) Environment: {"DOP_NODE_FQDN"=>"testlinux.example.com"}
+    [ready] hello_world
+      [ready] default
+        [ready] write hello world
+          [ready] testlinux.example.com
+
+Now you can see that DOPi will run the command 'echo "hello world"' as
+we have specified in the plan file. Now let's remove the 'noop' option
+and actually run it:
+
+    $ dopi run hello_world
+    Starting signal handling
+    Starting to run step 'write hello world'
+      [Command] testlinux.example.com : Running command custom
+      [Command] testlinux.example.com : custom [OK]
+    Step 'write hello world' successfully finished.
+    [done] hello_world
+      [done] default
+        [done] write hello world
+          [done] testlinux.example.com
+
+DOPi successfully executed the plan and the state changed from 'ready'
+to 'done'.
+
+## Connecting over SSH
+
+Until now we only executed commands on the local machine. But the main
+purpose of DOPi is to connect to a machine and execute some commands
+to make it ready or execute a configuration management tool like Puppet.
+
+To make it easy to demonstrate this we will use the ssh plugin like you
+normaly would to connect to a remote machine, but we will configure DOPi
+so it actually connect to localhost.
+
+First we add a network to the infrastructures hash:
+
+    infrastructures:
+      'test':
+        type: 'baremetal'
+        networks:
+          'localhost':
+            ip_pool:
+              from: '127.0.0.2'
+              to:   '127.0.0.250'
+            ip_netmask: '255.255.255.0'
+            ip_defgw: '127.0.0.1'
+
+We defined the network 'localhost' and a range of IPs. Now we have to
+add our existing machine into this network by adding a network interface:
+
+    nodes:
+      'testlinux.example.com':
+        infrastructure: 'test'
+        interfaces:
+          'eth0':
+            network: 'localhost'
+            ip: '127.0.0.2'
+
+Then we need to make sure we can access the local machine with SSH. We
+have to provide some credentials to login to the machine. This is done
+in the credentials hash:
+
+    credentials:
+      'test-credentials':
+        type: 'ssh_key'
+        username: 'myuser'
+        private_key: '/home/myuser/.ssh/id_rsa'
+
+In this case we will use an SSH private key to connect to local host.
+Make sure your public key is in your '~/.ssh/authorized_keys' file.
+You can also define a username/password pair if you don't want to use
+a private key, make sure you check the syntax in the documentation.
+
+Now we change the plugin in the step and use the 'ssh/custom' plugin
+instead of 'custom'. DOPi will now connect to the node and execute
+the command via SSH. We also need to tell the plugin what credentials
+it should use. Make sure the step looks like this now:
+
+    steps:
+      - name: 'write hello world'
+        nodes: 'all'
+        command:
+          plugin: 'ssh/custom'
+          credentials: 'test-credentials'
+          exec: 'echo'
+          arguments: '"hello world"'
+
+This tells DOPi to use the previously defined credentials to login to
+the machine. Let's update the plan:
+
+    $ bundle exec dopi update --plan hello_world.yaml hello_world
+    Updating plan hello_world
+    Plan hello_world was removed
+    New plan hello_world was added
+    hello_world
+
+Updating the plan will always reset the state of the the steps since
+we may completely change them and there is no way for DOPi to know
+what states should be preserves (This may change in the future).
+
+Let's run in noop mode:
+
+    $ bundle exec dopi run --noop hello_world
+    Starting signal handling
+    Starting to run step 'write hello world'
+      [Command] testlinux.example.com : Running command ssh/custom
+      [Command] testlinux.example.com : (NOOP) Executing 'ssh  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  -q  -o ChallengeResponseAuthentication=no  -o PasswordAuthentication=no  -i /home/myuser/.ssh/id_rsa myuser@127.0.0.2 "DOP_NODE_FQDN=testlinux.example.com echo \"hello world\""' for command ssh/custom
+      [Command] testlinux.example.com : (NOOP) Environment: {"DOP_NODE_FQDN"=>"testlinux.example.com"}
+    [ready] hello_world
+      [ready] default
+        [ready] write hello world
+          [ready] testlinux.example.com
+
+We see that DOPi will actually use ssh to connect to the node and
+execute the step.
+
+    $ bundle exec dopi run hello_world 
+    Starting signal handling
+    Starting to run step 'write hello world'
+      [Command] testlinux.example.com : Running command ssh/custom
+      [Command] testlinux.example.com : ssh/custom [OK]
+    Step 'write hello world' successfully finished.
+    [done] hello_world
+      [done] default
+        [done] write hello world
+          [done] testlinux.example.com
+
+We can successfully run the step.
+
+## Idempotency
+
+As we have seen before if we update a plan we lose the state and
+have to rerun all the steps. Sometimes this may not be what we want.
+In any case, it is always good practice to write your steps in a way
+so you can rerun them as many times as you want.
+
+In DOPi this is made easy with verification commands which run prior
+to the actual command and check if we can skip the step or not.
+
+
+
+DOPi oneshot adds the plan to the plan cache, runs it and at the
+end removes it from the cache again. We will learn more about the
+different methods to run a plan later, for now we will use the
+oneshot command because it is useful to test or if you don't care
+about the state of a plan.

data/doc/getting_started_examples/001_hello_world.yaml

@@ -0,0 +1,17 @@
+name: 'hello_world'
+
+infrastructures:
+  'test':
+    type: 'baremetal'
+
+nodes:
+  'testlinux.example.com':
+    infrastructure: 'test'
+
+steps:
+  - name: 'write hello world'
+    nodes: 'all'
+    command:
+      plugin: 'custom'
+      exec: 'echo'
+      arguments: '"hello world"'