vcloud-edge_gateway 0.5.0 → 1.0.0

data/.gitignore

@@ -11,6 +11,7 @@ replace_variables.sh
 /pkg/
 vendor/
 coverage/
+spec/integration/vcloud_tools_testing_config.yaml
 results.html
 fog_integration_test.config
 .idea

data/.travis.yml

@@ -2,8 +2,5 @@
 language: ruby
 rvm:
   - 1.9.3
-branches:
-  except:
-    - master
 notifications:
   email: false

data/CHANGELOG.md

@@ -1,3 +1,29 @@
+## 1.0.0 (2014-06-04)
+
+Features:
+
+  - `vcloud-edge-configure` now prints diff output. Colour is optional.
+    Defaults to off when STDOUT is redirected.
+  - `vcloud-edge-configure --dry-run` new argument to print the diff without
+    modifying the remote edge gateway.
+  - `vcloud-edge-configure --version` now only returns the version string
+    and no usage information.
+  - A side effect of changes to the executable means that exceptions from
+    Vcloud::EdgeGateway and Vcloud::Core will now result in a stacktrace
+    being returned by the CLI, which we'll retain for now until we refine
+    the error messages.
+
+API changes:
+
+  - Vcloud::EdgeGateway::Configure returns a hash, keyed by service name, of
+    HashDiff#diff arrays. It will be empty if there are no differences.
+  - Vcloud::EdgeGateway::Configure#update takes a `dry_run` argument which
+    defaults to false. When set to true it won't update the remote Edge GW.
+
+Bugfixes:
+
+  - The `vcloud-configure-edge` command has been renamed to `vcloud-edge-configure`.
+
 ## 0.5.0 (2014-05-15)
 
 Bugfixes:
@@ -33,6 +59,10 @@ Features:
   - Now uses the config loader and validator in vcloud-core rather than its own duplicate.
   - Require fog v1.21 to allow use of FOG_VCLOUD_TOKEN via ENV as an alternative to a .fog file
 
+Documentation credits:
+
+  - Thanks to @Azulinho who added some example configuration.
+
 ## 0.2.2 (2014-03-05)
 
 Bugfixes:

data/CONTRIBUTING.md

@@ -0,0 +1,66 @@
+# Contributing to vCloud Edge Gateway
+
+We really welcome contributions.
+
+## A quick guide on how to contribute
+
+1. Clone the repo:
+
+        git clone git@github.com:alphagov/vcloud-edge_gateway.git
+
+2. Run `bundle` to get the required dependecies
+
+3. Run the tests. Pull requests that add features must include unit tests,
+   so it is good to ensure you've got them passing to begin with.
+
+        bundle exec rake
+
+   If you have access to a live environment for testing, it would be great
+   if you could run the integration tests too - for more details on the
+   set-up for that, please see the [integration tests README]
+   (https://github.com/alphagov/vcloud-edge_gateway/blob/master/spec/integration/README.md)
+
+4. Add your functionality or bug fix and a test for your change. Only refactoring and
+   documentation changes do not require tests. If the functionality is at all complicated
+   then it is likely that more than one test will be required. If you would like help
+   with writing tests please do ask us.
+
+5. Make sure all the tests pass, including the integration tests if possible.
+
+6. Update the [CHANGELOG](https://github.com/alphagov/vcloud-edge_gateway/blob/master/CHANGELOG.md)
+   with a short description of what the change is. This may be a feature, a bugfix, or an
+   API change. If your change is documenation or refactoring, you do not need to add a line
+   to the CHANGELOG.
+
+7. Fork the repo, push to your fork, and submit a pull request.
+
+## How soon will we respond?
+
+We will comment on your pull request within two working days. However, we might not be able to review it immediately.
+
+We may come back to you with comments and suggestions, and if we would like you to make changes, we will close the pull request as well as adding details of the changes we'd like you to make.
+
+If you feel your pull request has been outstanding too long, please feel free to bump it by making a comment on it.
+
+## Guidelines for making a pull request
+
+    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
+    "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this document are to be
+    interpreted as described in RFC 2119.
+
+## In order for a pull request to be accepted, it MUST
+
+- Include at least one test (unless it is documentation or refactoring). If you have any questions about how to write tests, please ask us, we will be happy to help
+- Follow our [Git style guide](https://github.com/alphagov/styleguides/blob/master/git.md)
+- Include a clear summary in the pull request comments as to what the change is and why
+  you are making it
+- Be readable - we might ask you to change unclear variable names or obscure syntactic sugar
+- Have [good commit messages](http://robots.thoughtbot.com/5-useful-tips-for-a-better-commit-message)
+  that explain the change being made in that commit. Don't be afraid to write a lot in the
+  detail.
+
+## In order for a pull request to be accepted, it SHOULD
+
+- Include a line in the CHANGELOG unless it is a refactoring or documentation change
+- If it is code, follow our [Ruby style guide](https://github.com/alphagov/styleguides/blob/master/ruby.md)
+- If it is documentation, follow the [GDS content style guide](https://www.gov.uk/design-principles/style-guide/style-points)

data/README.md

@@ -23,12 +23,12 @@ Or install it yourself as:
 
 To configure an Edge Gateway:
 
-    $ vcloud-configure-edge input.yaml
+    $ vcloud-edge-configure input.yaml
 
 To use [mustache](http://mustache.github.io) templates so that rulesets can
 be re-used between environments:
 
-    $ vcloud-configure-edge --template-vars vars.yaml input.yaml.mustache
+    $ vcloud-edge-configure --template-vars vars.yaml input.yaml.mustache
 
 ## Credentials
 
@@ -60,7 +60,7 @@ credentials that allow it to talk to a vCloud Director environment.
 
         FOG_CREDENTIAL=test_credentials \
             FOG_VCLOUD_TOKEN=AAAABBBBBCCCCCCDDDDDDEEEEEEFFFFF= \
-            vcloud-configure-edge input.yaml
+            vcloud-edge-configure input.yaml
 
   You may find it easier to export one or both of the values as environment variables.
 
@@ -71,7 +71,7 @@ credentials that allow it to talk to a vCloud Director environment.
 ### Configure edge gateway services
 
 You can configure the following services on an existing edgegateway using
-`vcloud-configure-edge`.
+`vcloud-edge-configure`.
 
 - firewall_service
 - nat_service
@@ -80,7 +80,7 @@ You can configure the following services on an existing edgegateway using
 NB: DHCP and VPN Services are not yet supported by the Fog platform underneath.
 Support for these is being considered.
 
-The `vcloud-configure-edge` tool takes an input YAML file describing one
+The `vcloud-edge-configure` tool takes an input YAML file describing one
 or more of these services and updates the edge gateway configuration to match,
 obeying the following rules:
 
@@ -237,7 +237,7 @@ The vCloud Director load balancer service is quite basic, but supports the follo
 * Ability to persist sessions to the same backend member node, via a variety of
   means (eg HTTP cookie value, SSL session ID, source IP hash).
 
-`vcloud-configure-edge` supports all of the above features.
+`vcloud-edge-configure` supports all of the above features.
 
 It is also worth noting that the vCloud Director load balancer *does not support*:
 
@@ -423,9 +423,14 @@ cat edges.out | jq '
       '
 ```
 
-### Full configuration examples
+### Full configuration examples and schemas
 
-You can find full configuration examples in the `examples` folder.
+Full configuration examples are in the [`examples/`][examples] folder.
+
+Configuration schemas are in [`lib/vcloud/edge_gateway/schema/`][schema].
+
+[examples]: /examples
+[schema]: /lib/vcloud/edge_gateway/schema
 
 ## The vCloud API
 
@@ -453,6 +458,9 @@ Run the integration tests (slower and requires a real environment):
 
     bundle exec rake integration
 
+You need access to a suitable vCloud Director organization to run the integration tests. See the [integration tests
+README](/spec/integration/README.md) for further details.
+
 ### References
 
 * [vCloud Director Edge Gateway documentation](http://pubs.vmware.com/vcd-51/topic/com.vmware.vcloud.admin.doc_51/GUID-ADE1DCAB-874F-45A9-9337-1E971DAC0F7D.html)

data/Rakefile

@@ -23,6 +23,6 @@ task :publish_gem do
 end
 
 require 'rubocop/rake_task'
-Rubocop::RakeTask.new(:rubocop) do |task|
+RuboCop::RakeTask.new(:rubocop) do |task|
   task.options = ['--lint']
 end

data/bin/vcloud-configure-edge

@@ -1,31 +1,4 @@
 #!/usr/bin/env ruby
 
-require 'methadone'
-require 'vcloud/edge_gateway'
-
-class App
-  include Methadone::Main
-  include Methadone::CLILogging
-
-  main do |config_file|
-    if options['template-vars']
-      vse = Vcloud::EdgeGateway::Configure.new(config_file, options['template-vars'])
-    else
-      vse = Vcloud::EdgeGateway::Configure.new(config_file)
-    end
-    vse.update
-  end
-
-  arg :config_file
-  on('--template-vars FILE', 'Enable templating with variables from this file')
-
-  description '
-  vcloud-edge_gateway allows you to configure an EdgeGateway with an input
-  file which may optionally be a Mustache template.
-
-  See https://github.com/alphagov/vcloud-edge_gateway for more info'
-
-  version Vcloud::EdgeGateway::VERSION
-
-  go!
-end
+$stderr.puts "This command has been renamed. Please use: vcloud-edge-configure"
+exit 1

data/bin/vcloud-edge-configure

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'vcloud/edge_gateway'
+
+Vcloud::EdgeGateway::Cli.new(ARGV).run

data/jenkins.sh

@@ -1,8 +1,15 @@
 #!/bin/bash -x
 set -e
+
+git clean -ffdx
 bundle install --path "${HOME}/bundles/${JOB_NAME}"
 bundle exec rake
 
+# Obtain the integration test parameters
+git clone git@github.gds:gds/vcloud-tools-testing-config.git
+mv vcloud-tools-testing-config/vcloud_tools_testing_config.yaml spec/integration/
+rm -rf vcloud-tools-testing-config
+
 RUBYOPT="-r ./tools/fog_credentials" bundle exec rake integration
 
 bundle exec rake publish_gem

data/jenkins_integration_tests.sh

@@ -1,5 +1,12 @@
 #!/bin/bash -x
 set -e
+
+git clean -ffdx
 bundle install --path "${HOME}/bundles/${JOB_NAME}"
 
+# Obtain the integration test parameters
+git clone git@github.gds:gds/vcloud-tools-testing-config.git
+mv vcloud-tools-testing-config/vcloud_tools_testing_config.yaml spec/integration/
+rm -rf vcloud-tools-testing-config
+
 RUBYOPT="-r ./tools/fog_credentials" bundle exec rake integration

data/lib/vcloud/edge_gateway/cli.rb

@@ -0,0 +1,137 @@
+require 'optparse'
+
+module Vcloud
+  module EdgeGateway
+    class Cli
+      ANSI_RESET = "\033[0m"
+      ANSI_RED   = "\033[31m"
+      ANSI_GREEN = "\033[32m"
+
+      def initialize(argv_array)
+        @usage_text = nil
+        @config_file = nil
+        @options = {
+          :template_vars => nil,
+          :colour => STDOUT.tty?,
+          :dry_run => nil,
+        }
+
+        parse(argv_array)
+      end
+
+      def run
+        config_args = [@config_file]
+        if @options[:template_vars]
+          config_args << @options[:template_vars]
+        end
+
+        update_args = []
+        if @options[:dry_run]
+          update_args << @options[:dry_run]
+        end
+
+        conf = Vcloud::EdgeGateway::Configure.new(*config_args)
+        diff = conf.update(*update_args)
+        puts render_diff(diff)
+      end
+
+      private
+
+      def parse(args)
+        opt_parser = OptionParser.new do |opts|
+          opts.banner = <<-EOS
+Usage: #{$0} [options] config_file
+
+vcloud-edge_gateway allows you to configure an EdgeGateway with an input
+file which may optionally be a Mustache template.
+
+It will always output a diff of the changes between the remote config and
+your local config.
+
+See https://github.com/alphagov/vcloud-edge_gateway for more info
+          EOS
+
+          opts.separator ""
+          opts.separator "Options:"
+
+          opts.on("--dry-run", "Don't apply configuration changes") do
+            @options[:dry_run] = true
+          end
+
+          opts.on("--template-vars FILE", "Enable templating with variables from this file") do |f|
+            @options[:template_vars] = f
+          end
+
+          opts.on("--[no-]colour", "Disable/enable colour output. Enabled by default unless output is redirected") do |c|
+            @options[:colour] = c
+          end
+
+          opts.on("-h", "--help", "Print usage and exit") do
+            $stderr.puts opts
+            exit
+          end
+
+          opts.on("--version", "Display version and exit") do
+            puts Vcloud::EdgeGateway::VERSION
+            exit
+          end
+        end
+
+        @usage_text = opt_parser.to_s
+        begin
+          opt_parser.parse!(args)
+        rescue OptionParser::InvalidOption => e
+          exit_error_usage(e)
+        end
+
+        exit_error_usage("must supply config_file") unless args.size == 1
+        @config_file = args.first
+      end
+
+      def exit_error_usage(error)
+        $stderr.puts "#{$0}: #{error}"
+        $stderr.puts @usage_text
+        exit 2
+      end
+
+      def render_diff(diff)
+        lines = diff.collect { |service_name, service_diff|
+          service_diff.collect { |diff_tuple|
+            key = "#{service_name}.#{diff_tuple[1]}"
+            case diff_tuple[0]
+            when "-"
+              diff_line_rem(key, diff_tuple[2])
+            when "+"
+              diff_line_add(key, diff_tuple[2])
+            when "~"
+              [
+                diff_line_rem(key, diff_tuple[2]),
+                diff_line_add(key, diff_tuple[3]),
+              ]
+            end
+          }
+        }
+
+        lines.join("\n")
+      end
+
+      def diff_line_add(key, value)
+        line = "+ #{key}: #{value}"
+        if @options.fetch(:colour)
+          line = "#{ANSI_GREEN}#{line}#{ANSI_RESET}"
+        end
+
+        line
+      end
+
+      def diff_line_rem(key, value)
+        line = "- #{key}: #{value}"
+        if @options.fetch(:colour)
+          line = "#{ANSI_RED}#{line}#{ANSI_RESET}"
+        end
+
+        line
+      end
+    end
+  end
+end

data/lib/vcloud/edge_gateway/configuration_differ.rb

@@ -1,3 +1,5 @@
+require 'hashdiff'
+
 module Vcloud
   module EdgeGateway
     class ConfigurationDiffer
@@ -25,5 +27,4 @@ module Vcloud
 
     end
   end
-
 end

data/lib/vcloud/edge_gateway/configure.rb

@@ -1,15 +1,13 @@
-require 'hashdiff'
-
 module Vcloud
   module EdgeGateway
     class Configure
 
-      def initialize(config_file = nil, vars_file = nil)
+      def initialize(config_file=nil, vars_file=nil)
         config_loader = Vcloud::Core::ConfigLoader.new
         @local_config = config_loader.load_config(config_file, Vcloud::EdgeGateway::Schema::EDGE_GATEWAY_SERVICES, vars_file)
       end
 
-      def update
+      def update(dry_run=false)
         edge_gateway = Vcloud::Core::EdgeGateway.get_by_name @local_config[:gateway]
         remote_config = edge_gateway.vcloud_attributes[:Configuration][:EdgeGatewayServiceConfiguration]
         edge_gateway_interface_list = edge_gateway.interfaces
@@ -21,10 +19,16 @@ module Vcloud
         )
 
         if proposed_config.update_required?
-          edge_gateway.update_configuration proposed_config.config
+          if dry_run
+            Vcloud::Core.logger.info("EdgeGateway::Configure.update: Dry run. Skipping.")
+          else
+            edge_gateway.update_configuration proposed_config.config
+          end
         else
           Vcloud::Core.logger.info("EdgeGateway::Configure.update: Configuration is already up to date. Skipping.")
         end
+
+        proposed_config.diff
       end
 
     end

data/lib/vcloud/edge_gateway/edge_gateway_configuration.rb

@@ -2,63 +2,69 @@ module Vcloud
   module EdgeGateway
     class EdgeGatewayConfiguration
 
+      attr_reader :config, :diff
+
       def initialize(local_config, remote_config, edge_gateway_interfaces)
-        @local_config = local_config
-        @remote_config = remote_config
-        @edge_gateway_interfaces = edge_gateway_interfaces
-        @config = { }
-        @update_required = nil
+        @config, @diff = generate_new_config(local_config, remote_config, edge_gateway_interfaces)
       end
 
       def update_required?
-        @update_required = false
+        @config.any?
+      end
+
+      private
+      def generate_new_config(local_config, remote_config, edge_gateway_interfaces)
+        new_config = { }
+        diff = { }
+
+        firewall_service_config =
+          EdgeGateway::ConfigurationGenerator::FirewallService.new.
+            generate_fog_config(local_config[:firewall_service])
 
-        firewall_service_config = EdgeGateway::ConfigurationGenerator::FirewallService.new.generate_fog_config(@local_config[:firewall_service])
         unless firewall_service_config.nil?
-          differ = EdgeGateway::FirewallConfigurationDiffer.new(firewall_service_config, @remote_config[:FirewallService])
+          differ = EdgeGateway::FirewallConfigurationDiffer.new(
+            remote_config[:FirewallService],
+            firewall_service_config
+          )
           unless differ.diff.empty?
-            @config[:FirewallService] = firewall_service_config
-            @update_required = true
+            diff[:FirewallService] = differ.diff
+            new_config[:FirewallService] = firewall_service_config
           end
         end
 
         nat_service_config = EdgeGateway::ConfigurationGenerator::NatService.new(
-          @local_config[:nat_service],
-          @edge_gateway_interfaces
+          local_config[:nat_service],
+          edge_gateway_interfaces
         ).generate_fog_config
 
         unless nat_service_config.nil?
-          differ = EdgeGateway::NatConfigurationDiffer.new(nat_service_config, @remote_config[:NatService])
+          differ = EdgeGateway::NatConfigurationDiffer.new(
+            remote_config[:NatService],
+            nat_service_config
+          )
           unless differ.diff.empty?
-            @config[:NatService] = nat_service_config
-            @update_required = true
+            diff[:NatService] = differ.diff
+            new_config[:NatService] = nat_service_config
           end
         end
 
         load_balancer_service_config =
           EdgeGateway::ConfigurationGenerator::LoadBalancerService.new(
-            @edge_gateway_interfaces
-          ).generate_fog_config(@local_config[:load_balancer_service])
+            edge_gateway_interfaces
+          ).generate_fog_config(local_config[:load_balancer_service])
 
         unless load_balancer_service_config.nil?
           differ = EdgeGateway::LoadBalancerConfigurationDiffer.new(
-                     load_balancer_service_config,
-                     @remote_config[:LoadBalancerService]
-                   )
+            remote_config[:LoadBalancerService],
+            load_balancer_service_config
+          )
           unless differ.diff.empty?
-            @config[:LoadBalancerService] = load_balancer_service_config
-            @update_required = true
+            diff[:LoadBalancerService] = differ.diff
+            new_config[:LoadBalancerService] = load_balancer_service_config
           end
         end
 
-        @update_required
-      end
-
-      def config
-        if @update_required.nil?
-          update_required?
-        end
-        @config
+        return new_config, diff
       end
 
     end

data/lib/vcloud/edge_gateway/version.rb

@@ -1,6 +1,6 @@
 module Vcloud
   module EdgeGateway
-    VERSION = '0.5.0'
+    VERSION = '1.0.0'
   end
 end
 

data/lib/vcloud/edge_gateway.rb

@@ -8,6 +8,7 @@ require 'vcloud/edge_gateway/schema/firewall_service'
 require 'vcloud/edge_gateway/schema/load_balancer_service'
 require 'vcloud/edge_gateway/schema/edge_gateway'
 
+require 'vcloud/edge_gateway/cli'
 require 'vcloud/edge_gateway/configure'
 require 'vcloud/edge_gateway/configuration_generator/id_ranges'
 require 'vcloud/edge_gateway/configuration_generator/firewall_service'

data/spec/integration/README.md

@@ -0,0 +1,38 @@
+# Running vCloud Edge Gateway Integration Tests
+
+## Prerequisites
+
+- Access to a suitable vCloud Director organisation.
+
+  **NB** It is not safe to run them against an environment that is in use
+  (e.g. production, preview) as many of the tests clear down all config at
+  the beginning and/or end to ensure the environment is as the tests expect.
+
+- A config file with the settings configured.
+
+  There is a [template file](spec/integration/vcloud_tools_testing_config.yaml.template) to
+  help with this. Copy the template file to `spec/integration/vcloud_tools_testing_config.yaml`
+  and update with parameters suitable for your environment.
+
+- You need to include the set-up for your testing environment in your
+  [fog file](https://github.com/alphagov/vcloud-core#credentials).
+
+- The tests use the [vCloud Tools Tester](http://rubygems.org/gems/vcloud-tools-tester) gem.
+  You do not need to install this, `bundler` will do this for you.
+
+## Parameters
+
+````
+default:                # This is the fog credential that refers to your testing environment, e.g. `test_credential`
+  network_1:            # Primary network name
+  network_1_id:         # Primary network ID
+  network_1_ip:         # Primary network IP
+  edge_gateway:         # Edge gateway name
+  provider_network:     # Provider (external-facing) network name
+  provider_network_id:  # Provider network ID
+  provider_network_ip:  # Provider network IP
+````
+
+## To run the tests
+
+  `FOG_CREDENTIAL=test_credential bundle exec integration`