beaker-google 1.0.0 → 1.2.0

data/CHANGELOG.md

@@ -1,12 +1,52 @@
 # Changelog
 
-## [1.0.0](https://github.com/voxpupuli/beaker-google/tree/1.0.0) (2022-10-31)
+## [1.2.0](https://github.com/voxpupuli/beaker-google/tree/1.2.0) (2024-06-07)
+
+[Full Changelog](https://github.com/voxpupuli/beaker-google/compare/1.1.0...1.2.0)
+
+**Implemented enhancements:**
+
+- CI: Run on Ruby 3.2 and 3.3 as well [\#52](https://github.com/voxpupuli/beaker-google/pull/52) ([bastelfreak](https://github.com/bastelfreak))
+
+**Fixed bugs:**
+
+- Fix call to Google::Apis::ComputeV1::Instance.new\(\) to work on Ruby 3 [\#51](https://github.com/voxpupuli/beaker-google/pull/51) ([silug](https://github.com/silug))
+
+**Merged pull requests:**
+
+- CI: build gem with --strict & --verbose [\#53](https://github.com/voxpupuli/beaker-google/pull/53) ([bastelfreak](https://github.com/bastelfreak))
+- Cleanup for rubocop [\#50](https://github.com/voxpupuli/beaker-google/pull/50) ([silug](https://github.com/silug))
+
+## [1.1.0](https://github.com/voxpupuli/beaker-google/tree/1.1.0) (2024-05-22)
+
+[Full Changelog](https://github.com/voxpupuli/beaker-google/compare/1.0.0...1.1.0)
+
+**Implemented enhancements:**
+
+- Move timestamp in firewall name to a different value [\#45](https://github.com/voxpupuli/beaker-google/issues/45)
+- Add ability to configure the firewall [\#40](https://github.com/voxpupuli/beaker-google/pull/40) ([jaevans](https://github.com/jaevans))
+- Add ability to set custom hostname [\#39](https://github.com/voxpupuli/beaker-google/pull/39) ([Andy-Adrian](https://github.com/Andy-Adrian))
+
+**Closed issues:**
+
+- Add ability to set custom hostname [\#38](https://github.com/voxpupuli/beaker-google/issues/38)
+- network ports should be configurable [\#34](https://github.com/voxpupuli/beaker-google/issues/34)
+
+**Merged pull requests:**
+
+- Fix name conflicts [\#48](https://github.com/voxpupuli/beaker-google/pull/48) ([jaevans](https://github.com/jaevans))
+- Bump actions/checkout from 2 to 4 [\#43](https://github.com/voxpupuli/beaker-google/pull/43) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump github/codeql-action from 1 to 2 [\#42](https://github.com/voxpupuli/beaker-google/pull/42) ([dependabot[bot]](https://github.com/apps/dependabot))
+- dependabot: check for github actions and bundler [\#41](https://github.com/voxpupuli/beaker-google/pull/41) ([bastelfreak](https://github.com/bastelfreak))
+- Update fakefs requirement from ~\> 1.8 to ~\> 2.4 [\#37](https://github.com/voxpupuli/beaker-google/pull/37) ([dependabot[bot]](https://github.com/apps/dependabot))
+
+## [1.0.0](https://github.com/voxpupuli/beaker-google/tree/1.0.0) (2022-11-02)
 
 [Full Changelog](https://github.com/voxpupuli/beaker-google/compare/0.5.0...1.0.0)
 
 **Fixed bugs:**
 
-- exec sysprep commands in child processes [\#31](https://github.com/voxpupuli/beaker-google/pull/31) ([AndyAdrian-OP](https://github.com/AndyAdrian-OP))
+- exec sysprep commands in child processes [\#31](https://github.com/voxpupuli/beaker-google/pull/31) ([Andy-Adrian](https://github.com/Andy-Adrian))
 
 ## [0.5.0](https://github.com/voxpupuli/beaker-google/tree/0.5.0) (2022-10-04)
 

data/Gemfile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 source ENV.fetch('GEM_SOURCE', 'https://rubygems.org')
 
 gemspec
@@ -18,6 +20,7 @@ group :acceptance_testing do
   gem 'beaker', *location_for(ENV.fetch('BEAKER_VERSION', '~> 4.0'))
 end
 
-group :release do
-  gem 'github_changelog_generator', require: false
+group :release, optional: true do
+  gem 'faraday-retry', '~> 2.1', require: false
+  gem 'github_changelog_generator', '~> 1.16.4', require: false
 end

data/README.md

@@ -10,7 +10,7 @@
 
 Beaker library to use the Google hypervisor
 
-# How to use this wizardry
+# How to use this module
 
 This is a gem that allows you to use hosts with [Google Compute](https://cloud.google.com/compute) hypervisor with [Beaker](https://github.com/voxpupuli/beaker).
 
@@ -46,6 +46,7 @@ The behavior of this library can be configured using either the beaker host conf
 | gce_zone             | true             |                                  | The zone to place compute instances in. The region is calculated from the zone name.                                                                                                                                                                                                          |
 | gce_network          | false            | Default                          | The name of the network to attach to instances. If the project uses the default network, this and `gce_subnetwork` can be left empty.                                                                                                                                                         |
 | gce_subnetwork       | false            | Default                          | The name of the subnetwork to attach to the instances network interface. If the Default network is not used, this must be supplied.                                                                                                                                                           |
+| gce_ports            | false            | `[ ]`                            | A comma separated list of ports to add to the external firewall. Each port is specified in the format `number/protocol` where protocol is one of `tcp`, `udp`, `icmp`, `esp`, `ah`, `ipip`, or `sctp`. **NOTE:** Port `22/tcp` is required for beaker to function and is automatically added to the firewall. |
 | gce_ssh_private_key  | false            | $HOME/.ssh/google_compute_engine | The file path of the private key to use to connect to instances. If using the key created by the gcloud tool, this can be left blank.                                                                                                                                                         |
 | gce_ssh_public_key   | false            | <gce_ssh_private_key>.pub        | The file path of the public key to upload to the instance. If left blank, attempt to use the file at `gce_ssh_private_key` with a `.pub` extension.                                                                                                                                           |
 | gce_machine_type     | false            | e2-standard-4                    | The machine type to use for the instance. If the `BEAKER_gce_machine_type` environment variable is set, it will be used for all hosts.                                                                                                                                                        |
@@ -55,14 +56,24 @@ The behavior of this library can be configured using either the beaker host conf
 
 All the variables in the list can be set in the Beaker host configuration file, or the ones starting with `gce_` can be overridden by environment variables in the form `BEAKER_gce_...`. i.e. To override the `gce_machine_type` setting in the environment, set `BEAKER_gce_machine_type`.
 
+## Networking
+
+Each run of beaker creates a pair of firewalls to protect the hosts, and internal host-to-host firewall, and an external firewall between the hosts and the internet.
+
+The internal firewall allows all communication between hosts in the node set, while keeping them isolated from any other Beaker jobs that may be running in the same environment. This firewall is attached to all hosts in the test set, and allows all `tcp`, `udp`, and `icmp` traffic.
+
+The external firewall allows outside communication from the internet into the hosts in the test. Due to constrains of the Beaker system, the firewall accepts any source IP (`0.0.0.0/0`) and the SSH port (`22/tcp`) is always allowed. Other ports may be added by using the `gce_ports` configuration option or the `BEAKER_gce_ports` environment variable as a comma separated list of `port/proto` values where `port` is the port number or range, and proto is one of `tcp`, `udp`, `icmp`, `esp`, `ah`, `ipip`, or `sctp`. The port number is only used for `tcp`, `udp`, and `sctp` protocols. Any value, such as `-1` can be used for the other protocols.
+
+VM Instances created during this process will by default use the automatically-generated instance name as the hostname in the VM OS.  Set the `BEAKER_set_gce_hostname` environment variable to `1` to override this behavior and configure the VM OS with the name defined in the nodeset as the hostname.
+
 # Cleanup
 
 In cases where the beaker process is killed before finishing, it may leave resources in GCP. These resources will need to be manually deleted.
 
 | Resource Type | Name Pattern        | Count                                       |
 | ------------- | ------------------- | ------------------------------------------- |
-| Firewall      | `beaker-<number>-*` | 1                                           |
-| Instance      | `beaker-*`          | One or more depending on test configuration |
+| Firewall      | `beaker-<number>-*` | 2                                           |
+| Instance      | `beaker-<number>-*` | One or more depending on test configuration |
 
 # Contributing
 

data/Rakefile

@@ -1,5 +1,13 @@
+# frozen_string_literal: true
+
 require 'rspec/core/rake_task'
 
+begin
+  require 'voxpupuli/rubocop/rake'
+rescue LoadError
+  # the voxpupuli-rubocop gem is optional
+end
+
 ###########################################################
 #
 #   Documentation Tasks
@@ -13,12 +21,12 @@ def running?(cmdline)
   found = ps.lines.grep(/#{Regexp.quote(cmdline)}/)
   raise StandardError, "Found multiple YARD Servers. Don't know what to do." if found.length > 1
 
-  yes = found.empty? ? false : true
+  yes = !found.empty?
   [yes, found.first]
 end
 
 def pid_from(output)
-  output.squeeze(' ').strip.split(' ')[1]
+  output.squeeze(' ').strip.split[1]
 end
 
 desc 'Start the documentation server in the foreground'
@@ -44,7 +52,7 @@ namespace :docs do
     Dir.chdir(__dir__)
     output = `bundle exec yard doc`
     puts output
-    raise 'Errors/Warnings during yard documentation generation' if output =~ /\[warn\]|\[error\]/
+    raise 'Errors/Warnings during yard documentation generation' if /\[warn\]|\[error\]/.match?(output)
 
     Dir.chdir(original_dir)
   end
@@ -63,6 +71,7 @@ namespace :docs do
     end
   end
 
+  desc 'Alias for `background`'
   task(:bg) { Rake::Task['docs:background'].invoke }
 
   desc 'Check the status of the documentation server'
@@ -84,10 +93,10 @@ namespace :docs do
       puts "Found a YARD Server running with pid #{pid}"
       `kill #{pid}`
       puts 'Stopping...'
-      yes, output = running?(DOCS_DAEMON)
+      yes, = running?(DOCS_DAEMON)
       if yes
         `kill -9 #{pid}`
-        yes, output = running?(DOCS_DAEMON)
+        yes, = running?(DOCS_DAEMON)
         if yes
           puts 'Could not Stop Server!'
         else
@@ -106,9 +115,10 @@ begin
   require 'rubygems'
   require 'github_changelog_generator/task'
 rescue LoadError
+  # the github_changelog_generator gem is optional
 else
   GitHubChangelogGenerator::RakeTask.new :changelog do |config|
-    config.exclude_labels = %w{duplicate question invalid wontfix wont-fix skip-changelog}
+    config.exclude_labels = %w[duplicate question invalid wontfix wont-fix skip-changelog]
     config.user = 'voxpupuli'
     config.project = 'beaker-google'
     gem_version = Gem::Specification.load("#{config.project}.gemspec").version

data/beaker-google.gemspec

@@ -1,35 +1,36 @@
-# -*- encoding: utf-8 -*-
-$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift File.expand_path('lib', __dir__)
 require 'beaker-google/version'
 
 Gem::Specification.new do |s|
-  s.name        = "beaker-google"
+  s.name        = 'beaker-google'
   s.version     = BeakerGoogle::VERSION
-  s.authors     = ["Puppet", "Voxpupuli"]
-  s.email       = ["voxpupuli@groups.io"]
-  s.homepage    = "https://github.com/voxpupuli/beaker-google"
-  s.summary     = %q{Beaker DSL Extension Helpers!}
-  s.description = %q{Google Compute Engine support for the Beaker acceptance testing tool.}
-  s.license     = 'Apache2'
+  s.authors     = %w[Puppet Voxpupuli]
+  s.email       = ['voxpupuli@groups.io']
+  s.homepage    = 'https://github.com/voxpupuli/beaker-google'
+  s.summary     = 'Beaker DSL Extension Helpers!'
+  s.description = 'Google Compute Engine support for the Beaker acceptance testing tool.'
+  s.license     = 'Apache-2.0'
 
   s.files         = `git ls-files`.split("\n")
-  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
-  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
-  s.require_paths = ["lib"]
+  s.executables   = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
+  s.require_paths = ['lib']
 
-  s.required_ruby_version = Gem::Requirement.new('>= 2.4')
+  s.required_ruby_version = Gem::Requirement.new('>= 2.7')
 
   # Testing dependencies
-  s.add_development_dependency 'rspec', '~> 3.0'
-  s.add_development_dependency 'rspec-its'
-  s.add_development_dependency 'fakefs', '~> 1.8'
-  s.add_development_dependency 'rake', '~> 13.0'
-  s.add_development_dependency 'simplecov'
+  s.add_development_dependency 'fakefs', '~> 2.4'
   s.add_development_dependency 'pry', '~> 0.10'
+  s.add_development_dependency 'rake', '~> 13.0'
+  s.add_development_dependency 'rspec', '~> 3.0'
+  s.add_development_dependency 'rspec-its', '~> 1.3'
+  s.add_development_dependency 'simplecov', '~> 0.22.0'
+  s.add_development_dependency 'voxpupuli-rubocop', '~> 2.7.0'
 
   # Documentation dependencies
-  s.add_development_dependency 'yard'
-  s.add_development_dependency 'thin'
+  s.add_development_dependency 'thin', '~> 1.8', '>= 1.8.2'
+  s.add_development_dependency 'yard', '~> 0.9.36'
 
   # Run time dependencies
   s.add_runtime_dependency 'stringify-hash', '~> 0.0.0'

data/bin/beaker-google

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'rubygems' unless defined?(Gem)
 require 'beaker-google'

data/lib/beaker/hypervisor/google.rb

@@ -1,4 +1,8 @@
+# frozen_string_literal: true
+
 require 'beaker/hypervisor/google_compute'
 
-class Beaker::Google < Beaker::GoogleCompute
+module Beaker
+  class Google < Beaker::GoogleCompute
+  end
 end

data/lib/beaker/hypervisor/google_compute.rb

@@ -1,12 +1,13 @@
-require 'time'
+# frozen_string_literal: true
+
+require 'securerandom'
 
 module Beaker
   # Beaker support for the Google Compute Engine.
   class GoogleCompute < Beaker::Hypervisor
     SLEEPWAIT = 5
 
-    # Hours before an instance is considered a zombie
-    ZOMBIE = 3
+    WINDOWS_IMAGE_PROJECT = %w[windows-cloud windows-sql-cloud].freeze
 
     # Do some reasonable sleuthing on the SSH public key for GCE
 
@@ -18,9 +19,12 @@ module Beaker
     # @raise [Error] if the private key can not be found
     def find_google_ssh_private_key
       private_keyfile = ENV.fetch('BEAKER_gce_ssh_public_key',
-        File.join(ENV.fetch('HOME', nil), '.ssh', 'google_compute_engine'))
-      private_keyfile = @options[:gce_ssh_private_key] if @options[:gce_ssh_private_key] && !File.exist?(private_keyfile)
+                                  File.join(Dir.home, '.ssh', 'google_compute_engine'))
+      if @options[:gce_ssh_private_key] && !File.exist?(private_keyfile)
+        private_keyfile = @options[:gce_ssh_private_key]
+      end
       raise("Could not find GCE Private SSH key at '#{keyfile}'") unless File.exist?(private_keyfile)
+
       @options[:gce_ssh_private_key] = private_keyfile
       private_keyfile
     end
@@ -36,6 +40,7 @@ module Beaker
       public_keyfile = private_keyfile << '.pub'
       public_keyfile = @options[:gce_ssh_public_key] if @options[:gce_ssh_public_key] && !File.exist?(public_keyfile)
       raise("Could not find GCE Public SSH key at '#{keyfile}'") unless File.exist?(public_keyfile)
+
       @options[:gce_ssh_public_key] = public_keyfile
       public_keyfile
     end
@@ -79,32 +84,53 @@ module Beaker
       @options = options
       @logger = options[:logger]
       @hosts = google_hosts
-      @firewall = ''
+      @external_firewall_name = ''
+      @internal_firewall_name = ''
       @gce_helper = GoogleComputeHelper.new(options)
     end
 
     # Create and configure virtual machines in the Google Compute Engine,
     # including their associated disks and firewall rules
     def provision
-      start = Time.now
-      test_group_identifier = "beaker-#{start.to_i}-"
+      test_group_identifier = "beaker-#{SecureRandom.hex(4)}"
 
       # set firewall to open pe ports
       network = @gce_helper.get_network
 
-      @firewall = test_group_identifier + generate_host_name
+      @external_firewall_name = "#{test_group_identifier}-external"
 
-      @gce_helper.create_firewall(@firewall, network)
+      # Always allow ssh from anywhere as it's needed for Beaker to run
+      @gce_helper.create_firewall(
+        @external_firewall_name,
+        network,
+        allow: @options[:gce_ports] + ['22/tcp'],
+        source_ranges: ['0.0.0.0/0'],
+        target_tags: [test_group_identifier],
+      )
 
-      @logger.debug("Created Google Compute firewall #{@firewall}")
+      @logger.debug("Created External Google Compute firewall #{@external_firewall_name}")
 
-      @hosts.each do |host|
+      # Create a firewall that opens everything between all the hosts in this test group
+      @internal_firewall_name = "#{test_group_identifier}-internal"
+      internal_ports = ['1-65535/tcp', '1-65535/udp', '-1/icmp']
+      @gce_helper.create_firewall(
+        @internal_firewall_name,
+        network,
+        allow: internal_ports,
+        source_tags: [test_group_identifier],
+        target_tags: [test_group_identifier],
+      )
+      @logger.debug("Created test group Google Compute firewall #{@internal_firewall_name}")
 
+      @hosts.each do |host|
         machine_type_name = ENV.fetch('BEAKER_gce_machine_type', host['gce_machine_type'])
         raise "Must provide a machine type name in 'gce_machine_type'." if machine_type_name.nil?
+
         # Get the GCE machine type object for this host
         machine_type = @gce_helper.get_machine_type(machine_type_name)
-        raise "Unable to find machine type named #{machine_type_name} in region #{@compute.default_zone}" if machine_type.nil?
+        if machine_type.nil?
+          raise "Unable to find machine type named #{machine_type_name} in region #{@compute.default_zone}"
+        end
 
         # Find the image to use to create the new VM.
         # Either `image` or `family` must be set in the configuration. Accepted formats
@@ -119,7 +145,7 @@ module Beaker
         if host[:image]
           image_selector = host[:image]
           # Do we have a project name?
-          if %r{/}.match?(image_selector)
+          if image_selector.include?('/')
             image_project, image_name = image_selector.split('/')[0..1]
           else
             image_project = @gce_helper.options[:gce_project]
@@ -130,7 +156,7 @@ module Beaker
         elsif host[:family]
           image_selector = host[:family]
           # Do we have a project name?
-          if %r{/}.match?(image_selector)
+          if image_selector.include?('/')
             image_project, family_name = image_selector.split('/')
           else
             image_project = @gce_helper.options[:gce_project]
@@ -142,7 +168,7 @@ module Beaker
           raise('You must specify either :image or :family')
         end
 
-        unique_host_id = test_group_identifier + generate_host_name
+        unique_host_id = "#{test_group_identifier}-#{generate_host_name}"
 
         boot_size = host['volume_size'] || img.disk_size_gb
 
@@ -155,19 +181,29 @@ module Beaker
         host['vmhostname'] = unique_host_id
 
         # add a new instance of the image
-        operation = @gce_helper.create_instance(host['vmhostname'], img, machine_type, boot_size)
+        operation = @gce_helper.create_instance(host['vmhostname'], img, machine_type, boot_size, host.name)
         unless operation.error.nil?
-          raise "Unable to create Google Compute Instance #{host.name}: [#{operation.error.errors[0].code}] #{operation.error.errors[0].message}"
+          raise "Unable to create Google Compute Instance #{
+            host.name
+          }: [#{
+            operation.error.errors[0].code
+          }] #{
+            operation.error.errors[0].message
+          }"
         end
+
         @logger.debug("Created Google Compute instance for #{host.name}: #{host['vmhostname']}")
         instance = @gce_helper.get_instance(host['vmhostname'])
 
+        @gce_helper.add_instance_tag(host['vmhostname'], test_group_identifier)
+        @logger.debug("Added network tag #{test_group_identifier} to instance")
+
         # Make sure we have a non root/Adminsitor user to log in as
-        if host['user'] == "root" || host['user'] == "Administrator" || host['user'].empty?
-          initial_user = 'google_compute'
-        else
-          initial_user = host['user']
-        end
+        initial_user = if host['user'] == 'root' || host['user'] == 'Administrator' || host['user'].empty?
+                         'google_compute'
+                       else
+                         host['user']
+                       end
 
         # add metadata to instance, if there is any to set
         # mdata = format_metadata
@@ -175,18 +211,18 @@ module Beaker
         mdata = [
           {
             key: 'ssh-keys',
-            value: "#{initial_user}:#{File.read(find_google_ssh_public_key).strip}"
+            value: "#{initial_user}:#{File.read(find_google_ssh_public_key).strip}",
           },
           # For now oslogin needs to be disabled as there's no way to log in as root and it would
           # take too much work on beaker to add sudo support to everything
           {
             key: 'enable-oslogin',
-            value: 'FALSE'
+            value: 'FALSE',
           },
         ]
-        
+
         # Check for google's default windows images and turn on ssh if found
-        if image_project == "windows-cloud" || image_project == "windows-sql-cloud"
+        if WINDOWS_IMAGE_PROJECT.include?(image_project)
           # Turn on SSH on GCP's default windows images
           mdata << {
             key: 'enable-windows-ssh',
@@ -194,12 +230,12 @@ module Beaker
           }
           mdata << {
             key: 'sysprep-specialize-script-cmd',
-            value: 'start /wait googet -noconfirm=true update && start /wait googet -noconfirm=true install google-compute-engine-ssh',
+            value: 'start /wait googet -noconfirm=true update && start /wait googet -noconfirm=true install google-compute-engine-ssh', # rubocop:disable Layout/LineLength
           }
           # Some versions of windows don't seem to add the OpenSSH directory to the path which prevents scp from working
           mdata << {
             key: 'sysprep-specialize-script-ps1',
-            value: '[Environment]::SetEnvironmentVariable( "PATH", "$ENV:PATH;C:\Program Files\OpenSSH", [EnvironmentVariableTarget]::Machine )',
+            value: '[Environment]::SetEnvironmentVariable( "PATH", "$ENV:PATH;C:\Program Files\OpenSSH", [EnvironmentVariableTarget]::Machine )', # rubocop:disable Layout/LineLength
           }
         end
         unless mdata.empty?
@@ -210,9 +246,6 @@ module Beaker
 
         host['ip'] = instance.network_interfaces[0].access_configs[0].nat_ip
 
-        # Add the new host to the firewall
-        @gce_helper.add_firewall_tag(@firewall, host['vmhostname'])
-
         if host['disable_root_ssh'] == true
           @logger.info('Not enabling root ssh as disable_root_ssh is true')
         else
@@ -235,7 +268,8 @@ module Beaker
     # Shutdown and destroy virtual machines in the Google Compute Engine,
     # including their associated disks and firewall rules
     def cleanup
-      @gce_helper.delete_firewall(@firewall)
+      @gce_helper.delete_firewall(@external_firewall_name)
+      @gce_helper.delete_firewall(@internal_firewall_name)
 
       @hosts.each do |host|
         # TODO: Delete any other disks attached during the instance creation