beaker-google 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,15 +1,7 @@
1
1
  ---
2
- !binary "U0hBMQ==":
3
- metadata.gz: !binary |-
4
- ZmUwMDUwOWJjOWQ0ZDA4NjA4OTExYTdiNzM0ODRjN2M5NDQ2OWQ3OA==
5
- data.tar.gz: !binary |-
6
- NDkzYzcyNDQ3OWVhZGY2NzdjOGNmZmRkYTY4NzE2NmQxZmY5YjM0Ng==
2
+ SHA256:
3
+ metadata.gz: 38ff789a0865a3b6dd349f7543b6097b1f79e7b957744ba51d855e3460da4530
4
+ data.tar.gz: f812dc39fdb14f8a22ed9ffb735a2c2df12f58772e9287a7ec0b48d9e6be3fa7
7
5
  SHA512:
8
- metadata.gz: !binary |-
9
- YmQ5MDFiZGJhMWY0OWIyZjgyNTc3OTM0NDliOGIzYmJiMjVhYzhmNjJlYzc4
10
- YzQ4ZmFiZWI4NzQ0ZTFkYjEyZjlhY2ZlZDk4ZWZhMjRhN2FlOTY0NGFkODE2
11
- NDE0ZDc2MGMwMDcyNDM0ZGIwNmVlZWUxMjNhMGY3MzM2MTAwNTY=
12
- data.tar.gz: !binary |-
13
- MzMwNzc3MGY2NzNmZjQyMmY5N2ViZTk3NWUzZTAxN2JiMzBlM2MxNjAwZmE5
14
- NWVlNGJiODY0ZWM3YzdkYTJjZWNiMGUwMjMyM2RjN2YxZjU2NzcyZjZlODMz
15
- NDczYWY5OGE1YWQ3NmNlYTI4MjcyZWFkNWE1OGNjOWVkOWQyNDY=
6
+ metadata.gz: 81f7d94daad049188c62d4f4897a688e6b0987f514f10c8075a1f7cd55e552a792976ceb06ce3bf0143f760114c49860468db578dcab268e7331cac343754ad1
7
+ data.tar.gz: 9fadfb95152db012aaa528c628229d8ff5378f9cd17a0f3e41773a941eddbdd7dacca2675ee8382eac9db0b3148fef1027adc87c563d50f32d8fef07929ccf07
@@ -0,0 +1,8 @@
1
+ version: 2
2
+ updates:
3
+ - package-ecosystem: bundler
4
+ directory: "/"
5
+ schedule:
6
+ interval: daily
7
+ time: "13:00"
8
+ open-pull-requests-limit: 10
@@ -0,0 +1,23 @@
1
+ name: Snyk Scan
2
+ on:
3
+ workflow_dispatch:
4
+ push:
5
+ branches:
6
+ - master
7
+ jobs:
8
+ security:
9
+ runs-on: ubuntu-latest
10
+ steps:
11
+ - uses: actions/checkout@master
12
+ - name: setup ruby
13
+ uses: ruby/setup-ruby@v1
14
+ with:
15
+ ruby-version: 2.5.9
16
+ - name: create lock
17
+ run: bundle lock
18
+ - name: Run Snyk to check for vulnerabilities
19
+ uses: snyk/actions/ruby@master
20
+ env:
21
+ SNYK_TOKEN: ${{ secrets.SNYK_DIO_KEY }}
22
+ with:
23
+ command: monitor
data/CODEOWNERS ADDED
@@ -0,0 +1,2 @@
1
+ * @puppetlabs/release-engineering
2
+
data/Gemfile CHANGED
@@ -18,7 +18,7 @@ end
18
18
  # We don't put beaker in as a test dependency because we
19
19
  # don't want to create a transitive dependency
20
20
  group :acceptance_testing do
21
- gem "beaker", *location_for(ENV['BEAKER_VERSION'] || '~> 3.0')
21
+ gem "beaker", *location_for(ENV['BEAKER_VERSION'] || '~> 4.0')
22
22
  end
23
23
 
24
24
 
data/README.md CHANGED
@@ -1,21 +1,39 @@
1
1
  # beaker-google
2
2
 
3
- Beaker library to use google hypervisor
3
+ Beaker library to use the Google hypervisor
4
4
 
5
5
  # How to use this wizardry
6
6
 
7
- This is a gem that allows you to use hosts with [google compute](google_compute_engine.md) hypervisor with [beaker](https://github.com/puppetlabs/beaker).
7
+ This is a gem that allows you to use hosts with [google compute](google_compute_engine.md) hypervisor with [Beaker](https://github.com/puppetlabs/beaker).
8
8
 
9
- ### Right Now? (beaker 3.x)
9
+ See the [documentation](docs/manual.md) for the full manual.
10
10
 
11
- This gem is already included as [beaker dependency](https://github.com/puppetlabs/beaker/blob/master/beaker.gemspec) for you, so you don't need to do anything special to use this gem's functionality with beaker.
11
+ Beaker will automatically load the appropriate hypervisors for any given hosts file, so as long as your project dependencies are satisfied there's nothing else to do. No need to `require` this library in your tests.
12
12
 
13
- ### In beaker's Next Major Version? (beaker 4.x)
13
+ ## With Beaker 3.x
14
+ This gem is already included as [beaker dependency](https://github.com/puppetlabs/beaker/blob/master/beaker.gemspec)
15
+ for you, so you don't need to do anything special to use this gem's
16
+ functionality with Beaker.
14
17
 
15
- In beaker's next major version, the requirement for beaker-google will be pulled
16
- from that repo. When that happens, then the usage pattern will change. In order
17
- to use this then, you'll need to include beaker-google as a dependency right
18
- next to beaker itself.
18
+ This library is included as a dependency of Beaker 3.x versions, so there's nothing to do.
19
+
20
+ ## With Beaker 4.x
21
+
22
+ As of Beaker 4.0, all hypervisor and DSL extension libraries have been removed and are no longer dependencies. In order to use a specific hypervisor or DSL extension library in your project, you will need to include them alongside Beaker in your Gemfile or project.gemspec. E.g.
23
+
24
+ ~~~ruby
25
+ # Gemfile
26
+ gem 'beaker', '~>4.0'
27
+ gem 'beaker-aws'
28
+ # project.gemspec
29
+ s.add_runtime_dependency 'beaker', '~>4.0'
30
+ s.add_runtime_dependency 'beaker-aws'
31
+ ~~~
32
+
33
+ In Beaker's next major version, the requirement for `beaker-google` will be
34
+ pulled from that repo. When that happens, then the usage pattern will change.
35
+ In order to use this then, you'll need to include `beaker-google` as a dependency
36
+ right next to beaker itself.
19
37
 
20
38
  # Contributing
21
39
 
@@ -20,19 +20,23 @@ Gem::Specification.new do |s|
20
20
  # Testing dependencies
21
21
  s.add_development_dependency 'rspec', '~> 3.0'
22
22
  s.add_development_dependency 'rspec-its'
23
- s.add_development_dependency 'fakefs', '~> 0.6'
23
+ # pin fakefs for Ruby < 2.3
24
+ if RUBY_VERSION < "2.3"
25
+ s.add_development_dependency 'fakefs', '~> 0.6', '< 0.14'
26
+ else
27
+ s.add_development_dependency 'fakefs', '~> 0.6'
28
+ end
24
29
  s.add_development_dependency 'rake', '~> 10.1'
25
30
  s.add_development_dependency 'simplecov'
26
31
  s.add_development_dependency 'pry', '~> 0.10'
27
32
 
28
33
  # Documentation dependencies
29
34
  s.add_development_dependency 'yard'
30
- s.add_development_dependency 'markdown'
31
35
  s.add_development_dependency 'thin'
32
36
 
33
37
  # Run time dependencies
34
38
  s.add_runtime_dependency 'stringify-hash', '~> 0.0.0'
35
- s.add_runtime_dependency 'google-api-client', '~> 0.9'
39
+ s.add_runtime_dependency 'google-api-client', '~> 0.8'
36
40
 
37
41
  end
38
42
 
data/docs/manual.md ADDED
@@ -0,0 +1,122 @@
1
+ # Using the Google Compute Engine Provisioner
2
+
3
+ This [Beaker](https://github.com/puppetlabs/beaker) provisioner manages instances from pre-existing [Google Compute Engine](https://cloud.google.com/compute) images.
4
+
5
+ Currently supported [GCE images](https://cloud.google.com/compute/docs/images#os-compute-support) include:
6
+ * Debian
7
+ * CentOS
8
+ * Red Hat Enterprise Linux
9
+
10
+ ## Prerequisites
11
+
12
+ * A [Google Compute Engine project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
13
+ * An active [service account](https://cloud.google.com/compute/docs/access/service-accounts)
14
+ to your Google Compute Engine project, along with the following
15
+ information:
16
+ * The service account **private key file** (named xxx-privatekey.p12).
17
+ * The service account **email address** (named xxx@developer.gserviceaccount.com).
18
+ * The service account **password**.
19
+ * A **ssh keypair**, [set up for GCE](https://developers.google.com/compute/docs/console#sshkeys)
20
+ * Load the private key into your `ssh-agent` prior to running Beaker (or
21
+ leave off a password, but this is not recommended)
22
+ * Name the pair `google_compute_engine`
23
+ * You can tell Beaker to use a different public key by setting `BEAKER_gce_ssh_public_key` to the path of your public key file
24
+ * Place the [public key](https://git-scm.com/book/gr/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key) in your Google Compute Engine [project metadata](https://cloud.google.com/compute/docs/storing-retrieving-metadata)
25
+ * `key`: `sshKeys`
26
+ * `value` is the contents of your `google_compute.pub` with `google_compute:` _prepended_, eg: `google_compute:ssh-rsaAAA(...) user@machine.local`
27
+
28
+ ## Setting Up Beaker
29
+
30
+ You are now ready to set up the Beaker [nodeset](https://github.com/puppetlabs/beaker-rspec#typical-workflow) for your GCE instances
31
+
32
+ ### Example GCE Hosts File
33
+
34
+ ```yaml
35
+ HOSTS:
36
+ debian-7-master:
37
+ roles:
38
+ - server
39
+ - master
40
+ - agent
41
+ platform: debian-7-x86_64
42
+ hypervisor: google
43
+ # The GCE image name to use
44
+ image: debian-7-XXXX
45
+
46
+ centos-6-agent:
47
+ roles:
48
+ - agent
49
+ platform: el-6-x86_64
50
+ hypervisor: google
51
+ # The GCE image name to use
52
+ image: centos-6-XXXX
53
+ # The GCE Machine type to use
54
+ gce_machine_type: n1-highmem-2
55
+
56
+ redhat-7-agent:
57
+ roles:
58
+ - agent
59
+ platform: el-7-x86_64
60
+ hypervisor: google
61
+ # The GCE image name to use
62
+ image: redhat-7-XXXX
63
+
64
+ CONFIG:
65
+ type: aio
66
+
67
+ # Optional: The number of seconds to attempt execution and then quit
68
+ timeout: 10
69
+
70
+ # You don't have to specify this if you set the environment variable
71
+ # BEAKER_gce_project
72
+ gce_project: google-compute-project-name
73
+
74
+ # Service account key
75
+ #
76
+ # If not defined, this key will try to be found at
77
+ # $HOME/.beaker/gce/project-name.p12
78
+ #
79
+ # You can also specify the key using the environment variable
80
+ # BEAKER_gce_keyfile
81
+ gce_keyfile: /path/to/*****-privatekey.p12
82
+
83
+ # This is almost always the same!
84
+ #
85
+ # You don't have to specify this unless you have a custom 'gce_password'
86
+ #
87
+ # You can also specify this by setting the environment variable
88
+ # BEAKER_gce_password
89
+ gce_password: notasecret
90
+
91
+ # Service account email
92
+ #
93
+ # You don't have to specify this if you set the environment variable
94
+ # BEAKER_gce_email
95
+ gce_email: *********@developer.gserviceaccount.com
96
+ ```
97
+
98
+ Google Compute cloud instances and disks are deleted after test runs, but it is
99
+ up to the owner of the Google Compute Engine project to ensure that any zombie
100
+ instances/disks are properly removed should Beaker fail during cleanup.
101
+
102
+ **NOTE:** If zombie resources are left in GCE the code helps you identify runs
103
+ that have not been cleaned up by prefixing all of the assets with the **same**
104
+ identifier that follows the pattern of `beaker-TIMESTAMP`.
105
+
106
+ ### Tips and Tricks
107
+
108
+ To most easily use this plugin with GCE projects, you may want to do the following:
109
+
110
+ 1. `mkdir -p $HOME/.beaker/gce`
111
+ 2. `cd $HOME/.beaker/gce`
112
+ 3. Add the following to a file named `config` with your values in place
113
+
114
+ ```bash
115
+ export BEAKER_gce_ssh_public_key="$HOME/.beaker/gce/rsa.pub"
116
+ export BEAKER_gce_project='my-beaker-tests'
117
+ export BEAKER_gce_keyfile="$HOME/.beaker/Beaker_Testing.p12"
118
+ export BEAKER_gce_email='myuser@my-beaker-tests.iam.gserviceaccount.com'
119
+ ```
120
+
121
+ 4. Copy your testing account `p12` file into `$HOME/.beaker/gce/Beaker_Testing.p12`
122
+ 5. Run your test after running `source $HOME/.beaker/gce/config`
@@ -1,4 +1,4 @@
1
1
  require 'beaker/hypervisor/google_compute'
2
2
 
3
- class Google < GoogleCompute
3
+ class Beaker::Google < Beaker::GoogleCompute
4
4
  end
@@ -1,33 +1,65 @@
1
1
  require 'time'
2
2
 
3
3
  module Beaker
4
- #Beaker support for the Google Compute Engine.
4
+
5
+ # Beaker support for the Google Compute Engine.
5
6
  class GoogleCompute < Beaker::Hypervisor
6
7
 
7
8
  SLEEPWAIT = 5
8
- #number of hours before an instance is considered a zombie
9
+
10
+ # Hours before an instance is considered a zombie
9
11
  ZOMBIE = 3
10
12
 
11
- #Create the array of metaData, each member being a hash with a :key and a :value. Sets
12
- #:department, :project and :jenkins_build_url.
13
+ # Do some reasonable sleuthing on the SSH public key for GCE
14
+ def find_google_ssh_public_key
15
+ keyfile = ENV.fetch('BEAKER_gce_ssh_public_key', File.join(ENV['HOME'], '.ssh', 'google_compute_engine.pub'))
16
+
17
+ if @options[:gce_ssh_public_key] && !File.exist?(keyfile)
18
+ keyfile = @options[:gce_ssh_public_key]
19
+ end
20
+
21
+ raise("Could not find GCE Public SSH Key at '#{keyfile}'") unless File.exist?(keyfile)
22
+
23
+ return keyfile
24
+ end
25
+
26
+ # Create the array of metaData, each member being a hash with a :key and a
27
+ # :value. Sets :department, :project and :jenkins_build_url.
13
28
  def format_metadata
14
29
  [ {:key => :department, :value => @options[:department]},
15
30
  {:key => :project, :value => @options[:project]},
16
- {:key => :jenkins_build_url, :value => @options[:jenkins_build_url]} ].delete_if { |member| member[:value].nil? or member[:value].empty?}
31
+ {:key => :jenkins_build_url, :value => @options[:jenkins_build_url]},
32
+ {:key => :sshKeys, :value => "google_compute:#{File.read(find_google_ssh_public_key).strip}" }
33
+ ].delete_if { |member| member[:value].nil? or member[:value].empty?}
17
34
  end
18
35
 
19
- #Create a new instance of the Google Compute Engine hypervisor object
20
- #@param [<Host>] google_hosts The array of google hosts to provision, may ONLY be of platforms /centos-6-.*/ and
21
- # /debian-7-.*/. We currently only support the Google Compute provided templates.
22
- #@param [Hash{Symbol=>String}] options The options hash containing configuration values
23
- #@option options [String] :gce_project The Google Compute Project name to connect to
24
- #@option options [String] :gce_keyfile The location of the Google Compute service account keyfile
25
- #@option options [String] :gce_password The password for the Google Compute service account key
26
- #@option options [String] :gce_email The email address for the Google Compute service account
27
- #@option options [String] :gce_machine_type A Google Compute machine type used to create instances, defaults to n1-highmem-2
28
- #@option options [Integer] :timeout The amount of time to attempt execution before quiting and exiting with failure
36
+ # Create a new instance of the Google Compute Engine hypervisor object
37
+ #
38
+ # @param [<Host>] google_hosts The Array of google hosts to provision, may
39
+ # ONLY be of platforms /centos-*/, /debian-*/, /rhel-*/, /suse-*/. Only
40
+ # supports the Google Compute provided templates.
41
+ #
42
+ # @param [Hash{Symbol=>String}] options The options hash containing
43
+ # configuration values @option options [String] :gce_project The Google
44
+ # Compute Project name to connect to
45
+ #
46
+ # @option options [String] :gce_keyfile The location of the Google Compute
47
+ # service account keyfile
48
+ #
49
+ # @option options [String] :gce_password The password for the Google Compute
50
+ # service account key
51
+ #
52
+ # @option options [String] :gce_email The email address for the Google
53
+ # Compute service account
54
+ #
55
+ # @option options [String] :gce_machine_type A Google Compute machine type
56
+ # used to create instances, defaults to n1-highmem-2
57
+ #
58
+ # @option options [Integer] :timeout The amount of time to attempt execution
59
+ # before quiting and exiting with failure
29
60
  def initialize(google_hosts, options)
30
61
  require 'beaker/hypervisor/google_compute_helper'
62
+
31
63
  @options = options
32
64
  @logger = options[:logger]
33
65
  @hosts = google_hosts
@@ -35,66 +67,77 @@ module Beaker
35
67
  @gce_helper = GoogleComputeHelper.new(options)
36
68
  end
37
69
 
38
- #Create and configure virtual machines in the Google Compute Engine, including their associated disks and firewall rules
39
- #Currently ONLY supports Google Compute provided templates of CENTOS-6 and DEBIAN-7
70
+ # Create and configure virtual machines in the Google Compute Engine,
71
+ # including their associated disks and firewall rules
40
72
  def provision
41
- try = 1
42
73
  attempts = @options[:timeout].to_i / SLEEPWAIT
43
74
  start = Time.now
44
75
 
45
- #get machineType resource, used by all instances
76
+ test_group_identifier = "beaker-#{start.to_i}-"
77
+
78
+ # get machineType resource, used by all instances
46
79
  machineType = @gce_helper.get_machineType(start, attempts)
47
80
 
48
- #set firewall to open pe ports
81
+ # set firewall to open pe ports
49
82
  network = @gce_helper.get_network(start, attempts)
50
- @firewall = generate_host_name
83
+ @firewall = test_group_identifier + generate_host_name
51
84
  @gce_helper.create_firewall(@firewall, network, start, attempts)
52
85
 
53
86
  @logger.debug("Created Google Compute firewall #{@firewall}")
54
87
 
55
88
 
56
89
  @hosts.each do |host|
57
- gplatform = Platform.new(host[:image] || host[:platform])
90
+ if host[:image]
91
+ gplatform = host[:image]
92
+ elsif host[:platform]
93
+ gplatform = Platform.new(host[:platform])
94
+ else
95
+ raise('You must specify either :image or :platform, or both as necessary')
96
+ end
97
+
58
98
  img = @gce_helper.get_latest_image(gplatform, start, attempts)
59
- host['diskname'] = generate_host_name
99
+
100
+ unique_host_id = test_group_identifier + generate_host_name
101
+
102
+ host['diskname'] = unique_host_id
60
103
  disk = @gce_helper.create_disk(host['diskname'], img, start, attempts)
61
104
  @logger.debug("Created Google Compute disk for #{host.name}: #{host['diskname']}")
62
105
 
63
- #create new host name
64
- host['vmhostname'] = generate_host_name
106
+ # create new host name
107
+ host['vmhostname'] = unique_host_id
65
108
  #add a new instance of the image
66
109
  instance = @gce_helper.create_instance(host['vmhostname'], img, machineType, disk, start, attempts)
67
110
  @logger.debug("Created Google Compute instance for #{host.name}: #{host['vmhostname']}")
68
111
 
69
- #add metadata to instance, if there is any to set
112
+ # add metadata to instance, if there is any to set
70
113
  mdata = format_metadata
71
- if not mdata.empty?
114
+ unless mdata.empty?
72
115
  @gce_helper.setMetadata_on_instance(host['vmhostname'], instance['metadata']['fingerprint'],
73
116
  mdata,
74
117
  start, attempts)
75
118
  @logger.debug("Added tags to Google Compute instance #{host.name}: #{host['vmhostname']}")
76
119
  end
77
120
 
78
- #get ip for this host
121
+ # get ip for this host
79
122
  host['ip'] = instance['networkInterfaces'][0]['accessConfigs'][0]['natIP']
80
123
 
81
- #configure ssh
124
+ # configure ssh
82
125
  default_user = host['user']
83
126
  host['user'] = 'google_compute'
84
127
 
85
- disable_se_linux(host, @options)
86
128
  copy_ssh_to_root(host, @options)
87
129
  enable_root_login(host, @options)
88
130
  host['user'] = default_user
89
131
 
90
- #shut down connection, will reconnect on next exec
132
+ # shut down connection, will reconnect on next exec
91
133
  host.close
92
134
 
93
135
  @logger.debug("Instance ready: #{host['vmhostname']} for #{host.name}}")
94
136
  end
95
137
  end
96
138
 
97
- #Shutdown and destroy virtual machines in the Google Compute Engine, including their associated disks and firewall rules
139
+ # Shutdown and destroy virtual machines in the Google Compute Engine,
140
+ # including their associated disks and firewall rules
98
141
  def cleanup()
99
142
  attempts = @options[:timeout].to_i / SLEEPWAIT
100
143
  start = Time.now
@@ -110,18 +153,18 @@ module Beaker
110
153
 
111
154
  end
112
155
 
113
- #Shutdown and destroy Google Compute instances (including their associated disks and firewall rules)
114
- #that have been alive longer than ZOMBIE hours.
156
+ # Shutdown and destroy Google Compute instances (including their associated
157
+ # disks and firewall rules) that have been alive longer than ZOMBIE hours.
115
158
  def kill_zombies(max_age = ZOMBIE)
116
159
  now = start = Time.now
117
160
  attempts = @options[:timeout].to_i / SLEEPWAIT
118
161
 
119
- #get rid of old instances
162
+ # get rid of old instances
120
163
  instances = @gce_helper.list_instances(start, attempts)
121
164
  if instances
122
165
  instances.each do |instance|
123
166
  created = Time.parse(instance['creationTimestamp'])
124
- alive = (now - created ) /60 /60
167
+ alive = (now - created )/60/60
125
168
  if alive >= max_age
126
169
  #kill it with fire!
127
170
  @logger.debug("Deleting zombie instance #{instance['name']}")
@@ -131,14 +174,16 @@ module Beaker
131
174
  else
132
175
  @logger.debug("No zombie instances found")
133
176
  end
134
- #get rid of old disks
177
+
178
+ # get rid of old disks
135
179
  disks = @gce_helper.list_disks(start, attempts)
136
180
  if disks
137
181
  disks.each do |disk|
138
182
  created = Time.parse(disk['creationTimestamp'])
139
- alive = (now - created ) /60 /60
183
+ alive = (now - created )/60/60
140
184
  if alive >= max_age
141
- #kill it with fire!
185
+
186
+ # kill it with fire!
142
187
  @logger.debug("Deleting zombie disk #{disk['name']}")
143
188
  @gce_helper.delete_disk( disk['name'], start, attempts )
144
189
  end
@@ -146,10 +191,11 @@ module Beaker
146
191
  else
147
192
  @logger.debug("No zombie disks found")
148
193
  end
149
- #get rid of non-default firewalls
194
+
195
+ # get rid of non-default firewalls
150
196
  firewalls = @gce_helper.list_firewalls( start, attempts)
151
197
 
152
- if firewalls and not firewalls.empty?
198
+ if firewalls && !firewalls.empty?
153
199
  firewalls.each do |firewall|
154
200
  @logger.debug("Deleting non-default firewall #{firewall['name']}")
155
201
  @gce_helper.delete_firewall( firewall['name'], start, attempts )
@@ -157,8 +203,6 @@ module Beaker
157
203
  else
158
204
  @logger.debug("No zombie firewalls found")
159
205
  end
160
-
161
206
  end
162
-
163
207
  end
164
208
  end