dopv 0.11.0

data/README.md

@@ -0,0 +1,214 @@
+# Dopv
+
+Dopv orchestrates deployments of nodes. A node can be a virtual machine or a bare-metal compute node.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'dopv'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install dopv
+
+## Usage
+
+### Library
+Deploy a plan
+```ruby
+require 'dopv'
+plan = ::Dopv::load_plan(plan_file)
+data_volumes_db = ::Dopv::load_data_volumes_db(db_file)
+::Dopv::run_plan(plan, data_volumes_db)
+```
+Undeploy a plan while keeping data volumes
+```ruby
+require 'dopv'
+plan = ::Dopv::load_plan(plan_file)
+data_volumes_db = ::Dopv::load_data_volumes_db(db_file)
+::Dopv::run_plan(plan, data_volumes_db, :undeploy)
+```
+Undeploy a plan and remove data volumes from infrastructure provider (stack) as
+well as from persistent data volumes DB:
+```ruby
+require 'dopv'
+plan = ::Dopv::load_plan(plan_file)
+data_volumes_db = ::Dopv::load_data_volumes_db(db_file)
+::Dopv::run_plan(plan, data_volumes_db, :undeploy, true)
+```
+
+### CLI
+A command line interface utility `dopv` is provided.
+
+#### Getting help
+A help can be obtained by calling `dopv --help`:
+```
+$ dopv --help
+NAME
+    dopv - DOPv command line tool
+
+SYNOPSIS
+    dopv [global options] command [command options] [arguments...]
+
+VERSION
+    0.7.0
+
+GLOBAL OPTIONS
+    --help                         - Show this message
+    --logfile, -l path_to_log_file - Log file (default: none)
+    --[no-]trace, -t               - Show stacktrace on crash
+    --verbosity, -v level          - Verbosity of the command line tool (default: info)
+    --version                      - Display the program version
+
+COMMANDS
+    add      - Add a new plan file to the plan cache
+    deploy   - Deploy a plan
+    export   - Export the internal disk state into a local diskdb file
+    help     - Shows a list of commands or help for one command
+    import   - Import a diskdb file into the internal state store
+    list     - Show the list of plans in the plan store
+    remove   - Remove an existing plan from the plan cache
+    undeploy - Undeploy a plan
+    update   - Update the plan and/or the plan state for a given plan yaml or plan name.
+    validate - Validate plan file
+```
+
+#### Validating a plan
+The deployment plan `/tmp/foo_plan.yml` could be validated by issuing:
+```
+$ dopv validate -p /tmp/foo_plan.yml
+Plan valid.
+```
+
+Please note that the plan is always validated before any action takes place.
+
+#### Adding a plan
+In order to add a plan to plan cache, use `dopv` `add` command:
+```
+$ dopv add /tmp/bar_plan.yml
+```
+
+Please note that this feature is considered experimental and should be used with extreme care.
+
+#### Updating a plan
+In order to update a plan in the cache after some local changes have been introduced, use `dopv`
+`update` command:
+```
+$ dopv update /tmp/bar_plan.yml
+```
+
+One may use `-c` or `-i` in order to remove the existing disk information and start with a clean
+state or to ignore the update and to merely set it to the latest version.
+
+Please note that this feature is considered experimental and should be used with extreme care.
+
+#### Listing plans in cache
+One may use the `list` command to check which plans are stored in the plan cache.
+
+#### Deploying a plan
+To get a help on deploy CLI options launch `dopv help deploy` argument:
+```
+$ dopv help deploy
+NAME
+    deploy - Deploy a plan
+
+SYNOPSIS
+    dopv [global options] deploy [command options]
+
+COMMAND OPTIONS
+    --diskdb, -d path_to_db_file                                             - Use a local diskdb file and import/export it automatically (default: none)
+    --exclude_nodes=node01.example.com,node02.example.com,/example\.com$/    - Exclude this nodes from the run (default: )
+    --exclude_nodes_by_config='{"var1": ["val1", "/val2/"], "var2": "val2"}' - Exclude nodes with this config from the run (You have to specify a JSON hash here) (default: {})
+    --exclude_roles=role01,role01,/^rolepattern/                             - Exclude this roles from the run (default: )
+    --nodes=node01.example.com,node02.example.com,/example\.com$/            - Run plans for this nodes only (default: )
+    --nodes_by_config='{"var1": ["val1", "/val2/"], "var2": "val2"}'         - Run plans for this nodes with this config only (You have to specify a JSON hash here) (default: {})
+    --plan, -p path_to_plan_file                                             - plan name from the store or plan file to deploy. If a plan file is given DOPv will run in oneshot mode and add/remove
+                                                                               the plan automatically to the plan store (required, default: none)
+    --roles=role01,role01,/^rolepattern/                                     - Run plans for this roles only (default: )
+```
+To deploy a plan located at `/tmp/plan.yaml` and store and/or load persistent disks database located at `/tmp/pdisks.yaml` one can launch `dopv` with following options:
+```
+$ dopv deploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml
+```
+
+To deploy only nodes matching a regular expression `^foo-[1-9]+\.bar\.baz$` that are defined in a plan called `/tmp/plan.yaml` and store and/or load persistent disks database located at `/tmp/pdisks.yaml` one can launch `dopv` with following options:
+```
+$ dopv deploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml --nodes=/^foo-[1-9]+\.bar\.baz$/
+```
+
+Please note that disk database file is created if it does not exist.
+
+#### Undeploying a plan
+To get a help on undeploy CLI options launch `dopv` `help undeploy` argument:
+```
+$ dopv help undeploy
+NAME
+    undeploy - Undeploy a plan
+
+SYNOPSIS
+    dopv [global options] undeploy [command options]
+
+COMMAND OPTIONS
+    --diskdb, -d path_to_db_file - (default: none)
+    --plan, -p path_to_plan_file - (required, default: none)
+ ME
+    undeploy - Undeploy a plan
+
+SYNOPSIS
+    dopv [global options] undeploy [command options]
+
+COMMAND OPTIONS
+    --diskdb, -d path_to_db_file                                             - Use a local diskdb file and import/export it automatically (default: none)
+    --exclude_nodes=node01.example.com,node02.example.com,/example\.com$/    - Exclude this nodes from the run (default: )
+    --exclude_nodes_by_config='{"var1": ["val1", "/val2/"], "var2": "val2"}' - Exclude nodes with this config from the run (You have to specify a JSON hash here) (default: {})
+    --exclude_roles=role01,role01,/^rolepattern/                             - Exclude this roles from the run (default: )
+    --nodes=node01.example.com,node02.example.com,/example\.com$/            - Run plans for this nodes only (default: )
+    --nodes_by_config='{"var1": ["val1", "/val2/"], "var2": "val2"}'         - Run plans for this nodes with this config only (You have to specify a JSON hash here) (default: {})
+    --plan, -p path_to_plan_file                                             - plan name from the store or plan file to undeploy. If a plan file is given DOPv will run in oneshot mode and
+                                                                               add/remove the plan automatically to the plan store (required, default: none)
+    --[no-]rmdisk, -r                                                        - Remove the disks
+    --roles=role01,role01,/^rolepattern/                                     - Run plans for this roles only (default: )   --[no-]rmdisk, -r            -
+```
+
+In order to destroy a deployment specified by a plan located at `/tmp/plan.yaml` and persistent disks database located at `/tmp/pdisks.yaml` one can launch `dopv` with following options:
+```
+$ dopv undeploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml
+```
+If you also want to remove the data volumes of each node and remove their records from persistent data volumes DB, please specify `-r` or `--rmdisk` option as shown bellow:
+```
+$ dopv undeploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml -r
+```
+If you'd like to selectively remove a node called `foo.bar.baz` as well as nodes that match regular expression `^foo[1-5]\.bar\.baz$` and their data disks, you would use `--nodes` filter:
+```
+$ dopv undeploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml -r --nodes=foo.bar.baz,/^foo[1-5]\.bar\.baz$/
+```
+If you'd like to remove all nodes but those matching `foo.bar.baz` as well as nodes that match regular expression `^foo[1-5]\.bar\.baz$` and their data disks, you would use `--exclude_nodes` filter:
+```
+$ dopv undeploy -p  /tmp/pdisks.yaml -d /tmp/pdisks.yaml -r --exclude_nodes=foo.bar.baz,/^foo[1-5]\.bar\.baz$/
+```
+
+#### Logging
+By default `dopv` logs messages with `INFO` level and higher to standard output. In order to log to a file `-l` can be specified. The `-v` option is used to set a different log threshold. Following is an example of logging everything (`DEBUG` and above) into `/tmp/dopv.log` during plan deployment:
+```
+$ dopv -l /tmp/dopv.log -v debug deploy -p /tmp/plan.yaml -d /tmp/disks.yaml
+```
+
+## Node Filtering Notes
+Please note that `dopv` filtering by configuration and roles is not yet supported.
+
+## Plan
+A plan format and description can be found [here](https://github.com/swisscom/dop_common/blob/master/doc/plan_format_v0.0.1.md). A plan example can be found [here](https://github.com/swisscom/dop_common/blob/master/doc/plan_format_v0.0.1.md)
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/dopv

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require 'dopv/cli'
+exit Dopv::Cli.run(ARGV)

data/dopv.gemspec

@@ -0,0 +1,52 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'dopv/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "dopv"
+  spec.version       = Dopv::VERSION
+  spec.authors       = ["Pavol Dilung", "Andreas Zuber"]
+  spec.email         = ["pavol.dilung@swisscom.com", "azuber@puzzle.ch"]
+  spec.description   = %q{Deployment orchestrator for VMs}
+  spec.summary       = %q{Deployment orchestrator for VMs}
+  spec.homepage      = "https://github.com/swisscom/dopv"
+  spec.license       = "Apache-2.0"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec"
+  spec.add_development_dependency "rspec-mocks"
+  spec.add_development_dependency "rspec-command"
+  spec.add_development_dependency "simplecov"
+  spec.add_development_dependency "pry"
+  if RUBY_VERSION < "2"
+    spec.add_development_dependency "pry-debugger"
+  else
+    spec.add_development_dependency "pry-byebug"
+  end
+  # Newer guard listen versions do not work properly with RB <= 2.2.2
+  spec.add_development_dependency "listen", "~> 3.0.8"
+  spec.add_development_dependency "guard-ctags-bundler"
+
+  spec.add_dependency "json", "~> 1.8"
+  spec.add_dependency 'logger-colors', '~> 1'
+  #spec.add_dependency "rest-client", "~> 1.7"
+  spec.add_dependency "rbovirt", "~> 0.1", '>= 0.1.3'
+  spec.add_dependency "rbvmomi", "~> 1.8.2"
+  spec.add_dependency "net-ssh", "< 3.0" # fog dependcy but net-ssh >= 3.x require ruby 2.x
+  spec.add_dependency "fog-google", "< 0.1.1" # fog dependcy but net-ssh >= 3.x require ruby 2.x
+  spec.add_dependency "fog-profitbricks", "~> 0.0.5" # fog dependency but fog-profitbricks > 0.5 requires ruby 2.x
+  spec.add_dependency "fog", "~> 1.36.0"
+  spec.add_dependency "gli", "~> 2.13.1"
+  spec.add_dependency "dop_common", "~> 0.13", '>= 0.13.0'
+  spec.add_dependency 'parallel', '~> 1'
+  if RUBY_VERSION < "2"
+    spec.add_dependency "rest-client", "< 2.0"
+  end
+end

data/lib/dopv.rb

@@ -0,0 +1,166 @@
+require 'dopv/version'
+require 'dopv/log'
+require 'dopv/persistent_disk'
+require 'dopv/plan'
+require 'dopv/infrastructure'
+require 'dopv/state_store'
+require 'dop_common'
+require 'etc'
+require 'parallel'
+
+module Dopv
+  extend DopCommon::NodeFilter
+
+  DEFAULT_MAX_IN_FLIGHT = 5
+
+  def self.valid?(plan_file)
+    hash, _ = plan_store.read_plan_file(plan_file)
+    plan = DopCommon::Plan.new(hash)
+    plan.valid?
+  end
+
+  def self.add(plan_file)
+    raise StandardError, 'Plan not valid; did not add' unless valid?(plan_file)
+    plan_name = plan_store.add(plan_file)
+    Dopv.update_state(plan_name)
+    plan_name
+  end
+
+  def self.update_plan(plan_file, options = {})
+    raise StandardError, 'Plan not valid; did not add' unless valid?(plan_file)
+    plan_name = plan_store.update(plan_file)
+    update_state(plan_name, options)
+    plan_name
+  end
+
+  def self.update_state(plan_name, options = {})
+    plan_store.run_lock(plan_name) do
+      state_store = Dopv::StateStore.new(plan_name, plan_store)
+      state_store.update(options)
+    end
+  end
+
+  def self.remove(plan_name, remove_dopi_state = true, remove_dopv_state = false)
+    plan_store.remove(plan_name, remove_dopi_state, remove_dopv_state)
+  end
+
+  def self.list
+    plan_store.list
+  end
+
+  def self.deploy(plan_name, options = {})
+    run(:deploy, plan_name, options)
+  end
+
+  def self.undeploy(plan_name, options = {})
+    run(:undeploy, plan_name, options)
+  end
+
+  def self.refresh(plan_name, options = {})
+    run(:refresh, plan_name, options)
+  end
+
+  def self.export_state(plan_name)
+    ensure_plan_exists(plan_name)
+    state_store = Dopv::StateStore.new(plan_name, plan_store)
+    state_store.export
+  end
+
+  def self.export_state_file(plan_name, state_file)
+    ensure_plan_exists(plan_name)
+    File.open(state_file, 'w+') do |diskdb|
+      diskdb << YAML.dump(Dopv.export_state(plan_name))
+    end
+  end
+
+  def self.import_state(plan_name, data_volumes_db)
+    ensure_plan_exists(plan_name)
+    plan_store.run_lock(plan_name) do
+      state_store = Dopv::StateStore.new(plan_name, plan_store)
+      state_store.import(data_volumes_db)
+    end
+  end
+
+  def self.import_state_file(plan_name, state_file)
+    ensure_plan_exists(plan_name)
+    Dopv.import_state(plan_name, YAML.load_file(state_file))
+  end
+
+  private
+
+  def self.plan_store
+    @plan_store ||= DopCommon::PlanStore.new(DopCommon.config.plan_store_dir)
+  end
+
+  def self.ensure_plan_exists(plan_name)
+    unless plan_store.list.include?(plan_name)
+      raise StandardError, "The plan #{plan_name} does not exist in the plan store"
+    end
+  end
+
+  def self.get_plan(plan_name)
+    raise StandardError, 'Please update the plan state, there are pending updates' if pending_updates?(plan_name)
+    plan_parser = plan_store.get_plan(plan_name)
+    Dopv::Plan.new(plan_parser)
+  end
+
+  def self.pending_updates?(plan_name)
+    state_store = Dopv::StateStore.new(plan_name, plan_store)
+    state_store.pending_updates?
+  end
+
+  def self.run(operation, plan_name, options)
+    ensure_plan_exists(plan_name)
+    update_state(plan_name)
+    plan = get_plan(plan_name)
+    run_options = merge_default_options(options)
+    nodes = filter_nodes(plan.nodes, run_options[:run_for_nodes])
+
+    context_log_path = File.join(DopCommon.config.log_dir, "#{run_options[:run_id]}-#{plan_name}")
+    node_names = nodes.map{|n| n.name}
+    context_logger = DopCommon::ThreadContextLogger.new(context_log_path, node_names)
+
+    plan_store.run_lock(plan_name) do
+      state_store = Dopv::StateStore.new(plan_name, plan_store)
+      in_parallel(plan, nodes) do |node|
+        context_logger.log_context = node.name
+        case operation
+        when :deploy   then Dopv::Infrastructure::bootstrap_node(node, state_store)
+        when :undeploy then Dopv::Infrastructure::destroy_node(node, state_store, run_options[:rmdisk])
+        when :refresh  then Dopv::Infrastructure::refresh_node(node, state_store)
+        end
+      end
+    end
+  end
+
+  def self.merge_default_options(options)
+    {
+      :run_for_nodes => :all,
+      :rmdisk        => false,
+      :run_id        => Time.now.strftime('%Y%m%d-%H%M%S'),
+    }.merge(options)
+  end
+
+  def self.in_parallel(plan, nodes)
+    errors = false
+    infras = nodes.group_by {|node| node.infrastructure}
+    Parallel.each(infras.keys, :in_threads => infras.keys.length) do |infra|
+      Dopv.log.debug("Spawning control thread for infra #{infra.name}")
+      max_in_flight = infra.max_in_flight || plan.max_in_flight || DEFAULT_MAX_IN_FLIGHT
+      Dopv.log.debug("Threads for infra #{infra.name}: #{max_in_flight}")
+      Parallel.each(infras[infra], :in_threads => max_in_flight) do |node|
+        Dopv.log.debug("Spawning thread for node #{node.name}.")
+        begin
+          Dopv.log.debug("Yielding node #{node.name}.")
+          yield(node)
+        rescue => e
+          errors = true
+          Dopv.log.error("There was an error while processing node #{node.name}: #{e}")
+          raise Parallel::Break
+        end
+      end
+    end
+    raise "Errors detected during plan run" if errors
+  end
+
+end

data/lib/dopv/cli.rb

@@ -0,0 +1,54 @@
+#
+# DOPv command line main module
+#
+
+require 'gli'
+require 'dop_common/cli/node_selection'
+require 'dop_common/cli/log'
+require 'dop_common/cli/global_options'
+require 'dopv'
+require 'dopv/cli/command_validate'
+require 'dopv/cli/command_add'
+require 'dopv/cli/command_remove'
+require 'dopv/cli/command_list'
+require 'dopv/cli/command_update'
+require 'dopv/cli/command_import'
+require 'dopv/cli/command_export'
+require 'dopv/cli/command_run'
+require 'logger/colors'
+
+module Dopv
+  module Cli
+    include GLI::App
+    extend self
+
+    trace = false
+
+    program_desc 'DOPv command line tool'
+    version Dopv::VERSION
+
+    subcommand_option_handling :normal
+    arguments :strict
+
+    DopCommon::Cli.global_options(self)
+
+    pre do |global,command,options,args|
+      DopCommon.configure = global
+      ENV['GLI_DEBUG'] = 'true' if global[:trace] == true
+      DopCommon::Cli.initialize_logger('dopv.log', global[:log_level], global[:verbosity], global[:trace])
+      true
+    end
+
+    command_validate(self)
+    command_add(self)
+    command_remove(self)
+    command_list(self)
+    command_update(self)
+    command_import(self)
+    command_export(self)
+    command_run(self, :deploy)
+    command_run(self, :undeploy)
+    command_run(self, :refresh)
+
+  end
+end