beaker-google 0.1.0 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,15 +1,7 @@
1
1
  ---
2
- !binary "U0hBMQ==":
3
- metadata.gz: !binary |-
4
- ZmUwMDUwOWJjOWQ0ZDA4NjA4OTExYTdiNzM0ODRjN2M5NDQ2OWQ3OA==
5
- data.tar.gz: !binary |-
6
- NDkzYzcyNDQ3OWVhZGY2NzdjOGNmZmRkYTY4NzE2NmQxZmY5YjM0Ng==
2
+ SHA256:
3
+ metadata.gz: 38ff789a0865a3b6dd349f7543b6097b1f79e7b957744ba51d855e3460da4530
4
+ data.tar.gz: f812dc39fdb14f8a22ed9ffb735a2c2df12f58772e9287a7ec0b48d9e6be3fa7
7
5
  SHA512:
8
- metadata.gz: !binary |-
9
- YmQ5MDFiZGJhMWY0OWIyZjgyNTc3OTM0NDliOGIzYmJiMjVhYzhmNjJlYzc4
10
- YzQ4ZmFiZWI4NzQ0ZTFkYjEyZjlhY2ZlZDk4ZWZhMjRhN2FlOTY0NGFkODE2
11
- NDE0ZDc2MGMwMDcyNDM0ZGIwNmVlZWUxMjNhMGY3MzM2MTAwNTY=
12
- data.tar.gz: !binary |-
13
- MzMwNzc3MGY2NzNmZjQyMmY5N2ViZTk3NWUzZTAxN2JiMzBlM2MxNjAwZmE5
14
- NWVlNGJiODY0ZWM3YzdkYTJjZWNiMGUwMjMyM2RjN2YxZjU2NzcyZjZlODMz
15
- NDczYWY5OGE1YWQ3NmNlYTI4MjcyZWFkNWE1OGNjOWVkOWQyNDY=
6
+ metadata.gz: 81f7d94daad049188c62d4f4897a688e6b0987f514f10c8075a1f7cd55e552a792976ceb06ce3bf0143f760114c49860468db578dcab268e7331cac343754ad1
7
+ data.tar.gz: 9fadfb95152db012aaa528c628229d8ff5378f9cd17a0f3e41773a941eddbdd7dacca2675ee8382eac9db0b3148fef1027adc87c563d50f32d8fef07929ccf07
@@ -0,0 +1,8 @@
1
+ version: 2
2
+ updates:
3
+ - package-ecosystem: bundler
4
+ directory: "/"
5
+ schedule:
6
+ interval: daily
7
+ time: "13:00"
8
+ open-pull-requests-limit: 10
@@ -0,0 +1,23 @@
1
+ name: Snyk Scan
2
+ on:
3
+ workflow_dispatch:
4
+ push:
5
+ branches:
6
+ - master
7
+ jobs:
8
+ security:
9
+ runs-on: ubuntu-latest
10
+ steps:
11
+ - uses: actions/checkout@master
12
+ - name: setup ruby
13
+ uses: ruby/setup-ruby@v1
14
+ with:
15
+ ruby-version: 2.5.9
16
+ - name: create lock
17
+ run: bundle lock
18
+ - name: Run Snyk to check for vulnerabilities
19
+ uses: snyk/actions/ruby@master
20
+ env:
21
+ SNYK_TOKEN: ${{ secrets.SNYK_DIO_KEY }}
22
+ with:
23
+ command: monitor
data/CODEOWNERS ADDED
@@ -0,0 +1,2 @@
1
+ * @puppetlabs/release-engineering
2
+
data/Gemfile CHANGED
@@ -18,7 +18,7 @@ end
18
18
  # We don't put beaker in as a test dependency because we
19
19
  # don't want to create a transitive dependency
20
20
  group :acceptance_testing do
21
- gem "beaker", *location_for(ENV['BEAKER_VERSION'] || '~> 3.0')
21
+ gem "beaker", *location_for(ENV['BEAKER_VERSION'] || '~> 4.0')
22
22
  end
23
23
 
24
24
 
data/README.md CHANGED
@@ -1,21 +1,39 @@
1
1
  # beaker-google
2
2
 
3
- Beaker library to use google hypervisor
3
+ Beaker library to use the Google hypervisor
4
4
 
5
5
  # How to use this wizardry
6
6
 
7
- This is a gem that allows you to use hosts with [google compute](google_compute_engine.md) hypervisor with [beaker](https://github.com/puppetlabs/beaker).
7
+ This is a gem that allows you to use hosts with [google compute](google_compute_engine.md) hypervisor with [Beaker](https://github.com/puppetlabs/beaker).
8
8
 
9
- ### Right Now? (beaker 3.x)
9
+ See the [documentation](docs/manual.md) for the full manual.
10
10
 
11
- This gem is already included as [beaker dependency](https://github.com/puppetlabs/beaker/blob/master/beaker.gemspec) for you, so you don't need to do anything special to use this gem's functionality with beaker.
11
+ Beaker will automatically load the appropriate hypervisors for any given hosts file, so as long as your project dependencies are satisfied there's nothing else to do. No need to `require` this library in your tests.
12
12
 
13
- ### In beaker's Next Major Version? (beaker 4.x)
13
+ ## With Beaker 3.x
14
+ This gem is already included as [beaker dependency](https://github.com/puppetlabs/beaker/blob/master/beaker.gemspec)
15
+ for you, so you don't need to do anything special to use this gem's
16
+ functionality with Beaker.
14
17
 
15
- In beaker's next major version, the requirement for beaker-google will be pulled
16
- from that repo. When that happens, then the usage pattern will change. In order
17
- to use this then, you'll need to include beaker-google as a dependency right
18
- next to beaker itself.
18
+ This library is included as a dependency of Beaker 3.x versions, so there's nothing to do.
19
+
20
+ ## With Beaker 4.x
21
+
22
+ As of Beaker 4.0, all hypervisor and DSL extension libraries have been removed and are no longer dependencies. In order to use a specific hypervisor or DSL extension library in your project, you will need to include them alongside Beaker in your Gemfile or project.gemspec. E.g.
23
+
24
+ ~~~ruby
25
+ # Gemfile
26
+ gem 'beaker', '~>4.0'
27
+ gem 'beaker-aws'
28
+ # project.gemspec
29
+ s.add_runtime_dependency 'beaker', '~>4.0'
30
+ s.add_runtime_dependency 'beaker-aws'
31
+ ~~~
32
+
33
+ In Beaker's next major version, the requirement for `beaker-google` will be
34
+ pulled from that repo. When that happens, then the usage pattern will change.
35
+ In order to use this then, you'll need to include `beaker-google` as a dependency
36
+ right next to beaker itself.
19
37
 
20
38
  # Contributing
21
39
 
@@ -20,19 +20,23 @@ Gem::Specification.new do |s|
20
20
  # Testing dependencies
21
21
  s.add_development_dependency 'rspec', '~> 3.0'
22
22
  s.add_development_dependency 'rspec-its'
23
- s.add_development_dependency 'fakefs', '~> 0.6'
23
+ # pin fakefs for Ruby < 2.3
24
+ if RUBY_VERSION < "2.3"
25
+ s.add_development_dependency 'fakefs', '~> 0.6', '< 0.14'
26
+ else
27
+ s.add_development_dependency 'fakefs', '~> 0.6'
28
+ end
24
29
  s.add_development_dependency 'rake', '~> 10.1'
25
30
  s.add_development_dependency 'simplecov'
26
31
  s.add_development_dependency 'pry', '~> 0.10'
27
32
 
28
33
  # Documentation dependencies
29
34
  s.add_development_dependency 'yard'
30
- s.add_development_dependency 'markdown'
31
35
  s.add_development_dependency 'thin'
32
36
 
33
37
  # Run time dependencies
34
38
  s.add_runtime_dependency 'stringify-hash', '~> 0.0.0'
35
- s.add_runtime_dependency 'google-api-client', '~> 0.9'
39
+ s.add_runtime_dependency 'google-api-client', '~> 0.8'
36
40
 
37
41
  end
38
42
 
data/docs/manual.md ADDED
@@ -0,0 +1,122 @@
1
+ # Using the Google Compute Engine Provisioner
2
+
3
+ This [Beaker](https://github.com/puppetlabs/beaker) provisioner manages instances from pre-existing [Google Compute Engine](https://cloud.google.com/compute) images.
4
+
5
+ Currently supported [GCE images](https://cloud.google.com/compute/docs/images#os-compute-support) include:
6
+ * Debian
7
+ * CentOS
8
+ * Red Hat Enterprise Linux
9
+
10
+ ## Prerequisites
11
+
12
+ * A [Google Compute Engine project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
13
+ * An active [service account](https://cloud.google.com/compute/docs/access/service-accounts)
14
+ to your Google Compute Engine project, along with the following
15
+ information:
16
+ * The service account **private key file** (named xxx-privatekey.p12).
17
+ * The service account **email address** (named xxx@developer.gserviceaccount.com).
18
+ * The service account **password**.
19
+ * A **ssh keypair**, [set up for GCE](https://developers.google.com/compute/docs/console#sshkeys)
20
+ * Load the private key into your `ssh-agent` prior to running Beaker (or
21
+ leave off a password, but this is not recommended)
22
+ * Name the pair `google_compute_engine`
23
+ * You can tell Beaker to use a different public key by setting `BEAKER_gce_ssh_public_key` to the path of your public key file
24
+ * Place the [public key](https://git-scm.com/book/gr/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key) in your Google Compute Engine [project metadata](https://cloud.google.com/compute/docs/storing-retrieving-metadata)
25
+ * `key`: `sshKeys`
26
+ * `value` is the contents of your `google_compute.pub` with `google_compute:` _prepended_, eg: `google_compute:ssh-rsaAAA(...) user@machine.local`
27
+
28
+ ## Setting Up Beaker
29
+
30
+ You are now ready to set up the Beaker [nodeset](https://github.com/puppetlabs/beaker-rspec#typical-workflow) for your GCE instances
31
+
32
+ ### Example GCE Hosts File
33
+
34
+ ```yaml
35
+ HOSTS:
36
+ debian-7-master:
37
+ roles:
38
+ - server
39
+ - master
40
+ - agent
41
+ platform: debian-7-x86_64
42
+ hypervisor: google
43
+ # The GCE image name to use
44
+ image: debian-7-XXXX
45
+
46
+ centos-6-agent:
47
+ roles:
48
+ - agent
49
+ platform: el-6-x86_64
50
+ hypervisor: google
51
+ # The GCE image name to use
52
+ image: centos-6-XXXX
53
+ # The GCE Machine type to use
54
+ gce_machine_type: n1-highmem-2
55
+
56
+ redhat-7-agent:
57
+ roles:
58
+ - agent
59
+ platform: el-7-x86_64
60
+ hypervisor: google
61
+ # The GCE image name to use
62
+ image: redhat-7-XXXX
63
+
64
+ CONFIG:
65
+ type: aio
66
+
67
+ # Optional: The number of seconds to attempt execution and then quit
68
+ timeout: 10
69
+
70
+ # You don't have to specify this if you set the environment variable
71
+ # BEAKER_gce_project
72
+ gce_project: google-compute-project-name
73
+
74
+ # Service account key
75
+ #
76
+ # If not defined, this key will try to be found at
77
+ # $HOME/.beaker/gce/project-name.p12
78
+ #
79
+ # You can also specify the key using the environment variable
80
+ # BEAKER_gce_keyfile
81
+ gce_keyfile: /path/to/*****-privatekey.p12
82
+
83
+ # This is almost always the same!
84
+ #
85
+ # You don't have to specify this unless you have a custom 'gce_password'
86
+ #
87
+ # You can also specify this by setting the environment variable
88
+ # BEAKER_gce_password
89
+ gce_password: notasecret
90
+
91
+ # Service account email
92
+ #
93
+ # You don't have to specify this if you set the environment variable
94
+ # BEAKER_gce_email
95
+ gce_email: *********@developer.gserviceaccount.com
96
+ ```
97
+
98
+ Google Compute cloud instances and disks are deleted after test runs, but it is
99
+ up to the owner of the Google Compute Engine project to ensure that any zombie
100
+ instances/disks are properly removed should Beaker fail during cleanup.
101
+
102
+ **NOTE:** If zombie resources are left in GCE the code helps you identify runs
103
+ that have not been cleaned up by prefixing all of the assets with the **same**
104
+ identifier that follows the pattern of `beaker-TIMESTAMP`.
105
+
106
+ ### Tips and Tricks
107
+
108
+ To most easily use this plugin with GCE projects, you may want to do the following:
109
+
110
+ 1. `mkdir -p $HOME/.beaker/gce`
111
+ 2. `cd $HOME/.beaker/gce`
112
+ 3. Add the following to a file named `config` with your values in place
113
+
114
+ ```bash
115
+ export BEAKER_gce_ssh_public_key="$HOME/.beaker/gce/rsa.pub"
116
+ export BEAKER_gce_project='my-beaker-tests'
117
+ export BEAKER_gce_keyfile="$HOME/.beaker/Beaker_Testing.p12"
118
+ export BEAKER_gce_email='myuser@my-beaker-tests.iam.gserviceaccount.com'
119
+ ```
120
+
121
+ 4. Copy your testing account `p12` file into `$HOME/.beaker/gce/Beaker_Testing.p12`
122
+ 5. Run your test after running `source $HOME/.beaker/gce/config`
@@ -1,4 +1,4 @@
1
1
  require 'beaker/hypervisor/google_compute'
2
2
 
3
- class Google < GoogleCompute
3
+ class Beaker::Google < Beaker::GoogleCompute
4
4
  end
@@ -1,33 +1,65 @@
1
1
  require 'time'
2
2
 
3
3
  module Beaker
4
- #Beaker support for the Google Compute Engine.
4
+
5
+ # Beaker support for the Google Compute Engine.
5
6
  class GoogleCompute < Beaker::Hypervisor
6
7
 
7
8
  SLEEPWAIT = 5
8
- #number of hours before an instance is considered a zombie
9
+
10
+ # Hours before an instance is considered a zombie
9
11
  ZOMBIE = 3
10
12
 
11
- #Create the array of metaData, each member being a hash with a :key and a :value. Sets
12
- #:department, :project and :jenkins_build_url.
13
+ # Do some reasonable sleuthing on the SSH public key for GCE
14
+ def find_google_ssh_public_key
15
+ keyfile = ENV.fetch('BEAKER_gce_ssh_public_key', File.join(ENV['HOME'], '.ssh', 'google_compute_engine.pub'))
16
+
17
+ if @options[:gce_ssh_public_key] && !File.exist?(keyfile)
18
+ keyfile = @options[:gce_ssh_public_key]
19
+ end
20
+
21
+ raise("Could not find GCE Public SSH Key at '#{keyfile}'") unless File.exist?(keyfile)
22
+
23
+ return keyfile
24
+ end
25
+
26
+ # Create the array of metaData, each member being a hash with a :key and a
27
+ # :value. Sets :department, :project and :jenkins_build_url.
13
28
  def format_metadata
14
29
  [ {:key => :department, :value => @options[:department]},
15
30
  {:key => :project, :value => @options[:project]},
16
- {:key => :jenkins_build_url, :value => @options[:jenkins_build_url]} ].delete_if { |member| member[:value].nil? or member[:value].empty?}
31
+ {:key => :jenkins_build_url, :value => @options[:jenkins_build_url]},
32
+ {:key => :sshKeys, :value => "google_compute:#{File.read(find_google_ssh_public_key).strip}" }
33
+ ].delete_if { |member| member[:value].nil? or member[:value].empty?}
17
34
  end
18
35
 
19
- #Create a new instance of the Google Compute Engine hypervisor object
20
- #@param [<Host>] google_hosts The array of google hosts to provision, may ONLY be of platforms /centos-6-.*/ and
21
- # /debian-7-.*/. We currently only support the Google Compute provided templates.
22
- #@param [Hash{Symbol=>String}] options The options hash containing configuration values
23
- #@option options [String] :gce_project The Google Compute Project name to connect to
24
- #@option options [String] :gce_keyfile The location of the Google Compute service account keyfile
25
- #@option options [String] :gce_password The password for the Google Compute service account key
26
- #@option options [String] :gce_email The email address for the Google Compute service account
27
- #@option options [String] :gce_machine_type A Google Compute machine type used to create instances, defaults to n1-highmem-2
28
- #@option options [Integer] :timeout The amount of time to attempt execution before quiting and exiting with failure
36
+ # Create a new instance of the Google Compute Engine hypervisor object
37
+ #
38
+ # @param [<Host>] google_hosts The Array of google hosts to provision, may
39
+ # ONLY be of platforms /centos-*/, /debian-*/, /rhel-*/, /suse-*/. Only
40
+ # supports the Google Compute provided templates.
41
+ #
42
+ # @param [Hash{Symbol=>String}] options The options hash containing
43
+ # configuration values @option options [String] :gce_project The Google
44
+ # Compute Project name to connect to
45
+ #
46
+ # @option options [String] :gce_keyfile The location of the Google Compute
47
+ # service account keyfile
48
+ #
49
+ # @option options [String] :gce_password The password for the Google Compute
50
+ # service account key
51
+ #
52
+ # @option options [String] :gce_email The email address for the Google
53
+ # Compute service account
54
+ #
55
+ # @option options [String] :gce_machine_type A Google Compute machine type
56
+ # used to create instances, defaults to n1-highmem-2
57
+ #
58
+ # @option options [Integer] :timeout The amount of time to attempt execution
59
+ # before quiting and exiting with failure
29
60
  def initialize(google_hosts, options)
30
61
  require 'beaker/hypervisor/google_compute_helper'
62
+
31
63
  @options = options
32
64
  @logger = options[:logger]
33
65
  @hosts = google_hosts
@@ -35,66 +67,77 @@ module Beaker
35
67
  @gce_helper = GoogleComputeHelper.new(options)
36
68
  end
37
69
 
38
- #Create and configure virtual machines in the Google Compute Engine, including their associated disks and firewall rules
39
- #Currently ONLY supports Google Compute provided templates of CENTOS-6 and DEBIAN-7
70
+ # Create and configure virtual machines in the Google Compute Engine,
71
+ # including their associated disks and firewall rules
40
72
  def provision
41
- try = 1
42
73
  attempts = @options[:timeout].to_i / SLEEPWAIT
43
74
  start = Time.now
44
75
 
45
- #get machineType resource, used by all instances
76
+ test_group_identifier = "beaker-#{start.to_i}-"
77
+
78
+ # get machineType resource, used by all instances
46
79
  machineType = @gce_helper.get_machineType(start, attempts)
47
80
 
48
- #set firewall to open pe ports
81
+ # set firewall to open pe ports
49
82
  network = @gce_helper.get_network(start, attempts)
50
- @firewall = generate_host_name
83
+ @firewall = test_group_identifier + generate_host_name
51
84
  @gce_helper.create_firewall(@firewall, network, start, attempts)
52
85
 
53
86
  @logger.debug("Created Google Compute firewall #{@firewall}")
54
87
 
55
88
 
56
89
  @hosts.each do |host|
57
- gplatform = Platform.new(host[:image] || host[:platform])
90
+ if host[:image]
91
+ gplatform = host[:image]
92
+ elsif host[:platform]
93
+ gplatform = Platform.new(host[:platform])
94
+ else
95
+ raise('You must specify either :image or :platform, or both as necessary')
96
+ end
97
+
58
98
  img = @gce_helper.get_latest_image(gplatform, start, attempts)
59
- host['diskname'] = generate_host_name
99
+
100
+ unique_host_id = test_group_identifier + generate_host_name
101
+
102
+ host['diskname'] = unique_host_id
60
103
  disk = @gce_helper.create_disk(host['diskname'], img, start, attempts)
61
104
  @logger.debug("Created Google Compute disk for #{host.name}: #{host['diskname']}")
62
105
 
63
- #create new host name
64
- host['vmhostname'] = generate_host_name
106
+ # create new host name
107
+ host['vmhostname'] = unique_host_id
65
108
  #add a new instance of the image
66
109
  instance = @gce_helper.create_instance(host['vmhostname'], img, machineType, disk, start, attempts)
67
110
  @logger.debug("Created Google Compute instance for #{host.name}: #{host['vmhostname']}")
68
111
 
69
- #add metadata to instance, if there is any to set
112
+ # add metadata to instance, if there is any to set
70
113
  mdata = format_metadata
71
- if not mdata.empty?
114
+ unless mdata.empty?
72
115
  @gce_helper.setMetadata_on_instance(host['vmhostname'], instance['metadata']['fingerprint'],
73
116
  mdata,
74
117
  start, attempts)
75
118
  @logger.debug("Added tags to Google Compute instance #{host.name}: #{host['vmhostname']}")
76
119
  end
77
120
 
78
- #get ip for this host
121
+ # get ip for this host
79
122
  host['ip'] = instance['networkInterfaces'][0]['accessConfigs'][0]['natIP']
80
123
 
81
- #configure ssh
124
+ # configure ssh
82
125
  default_user = host['user']
83
126
  host['user'] = 'google_compute'
84
127
 
85
- disable_se_linux(host, @options)
86
128
  copy_ssh_to_root(host, @options)
87
129
  enable_root_login(host, @options)
88
130
  host['user'] = default_user
89
131
 
90
- #shut down connection, will reconnect on next exec
132
+ # shut down connection, will reconnect on next exec
91
133
  host.close
92
134
 
93
135
  @logger.debug("Instance ready: #{host['vmhostname']} for #{host.name}}")
94
136
  end
95
137
  end
96
138
 
97
- #Shutdown and destroy virtual machines in the Google Compute Engine, including their associated disks and firewall rules
139
+ # Shutdown and destroy virtual machines in the Google Compute Engine,
140
+ # including their associated disks and firewall rules
98
141
  def cleanup()
99
142
  attempts = @options[:timeout].to_i / SLEEPWAIT
100
143
  start = Time.now
@@ -110,18 +153,18 @@ module Beaker
110
153
 
111
154
  end
112
155
 
113
- #Shutdown and destroy Google Compute instances (including their associated disks and firewall rules)
114
- #that have been alive longer than ZOMBIE hours.
156
+ # Shutdown and destroy Google Compute instances (including their associated
157
+ # disks and firewall rules) that have been alive longer than ZOMBIE hours.
115
158
  def kill_zombies(max_age = ZOMBIE)
116
159
  now = start = Time.now
117
160
  attempts = @options[:timeout].to_i / SLEEPWAIT
118
161
 
119
- #get rid of old instances
162
+ # get rid of old instances
120
163
  instances = @gce_helper.list_instances(start, attempts)
121
164
  if instances
122
165
  instances.each do |instance|
123
166
  created = Time.parse(instance['creationTimestamp'])
124
- alive = (now - created ) /60 /60
167
+ alive = (now - created )/60/60
125
168
  if alive >= max_age
126
169
  #kill it with fire!
127
170
  @logger.debug("Deleting zombie instance #{instance['name']}")
@@ -131,14 +174,16 @@ module Beaker
131
174
  else
132
175
  @logger.debug("No zombie instances found")
133
176
  end
134
- #get rid of old disks
177
+
178
+ # get rid of old disks
135
179
  disks = @gce_helper.list_disks(start, attempts)
136
180
  if disks
137
181
  disks.each do |disk|
138
182
  created = Time.parse(disk['creationTimestamp'])
139
- alive = (now - created ) /60 /60
183
+ alive = (now - created )/60/60
140
184
  if alive >= max_age
141
- #kill it with fire!
185
+
186
+ # kill it with fire!
142
187
  @logger.debug("Deleting zombie disk #{disk['name']}")
143
188
  @gce_helper.delete_disk( disk['name'], start, attempts )
144
189
  end
@@ -146,10 +191,11 @@ module Beaker
146
191
  else
147
192
  @logger.debug("No zombie disks found")
148
193
  end
149
- #get rid of non-default firewalls
194
+
195
+ # get rid of non-default firewalls
150
196
  firewalls = @gce_helper.list_firewalls( start, attempts)
151
197
 
152
- if firewalls and not firewalls.empty?
198
+ if firewalls && !firewalls.empty?
153
199
  firewalls.each do |firewall|
154
200
  @logger.debug("Deleting non-default firewall #{firewall['name']}")
155
201
  @gce_helper.delete_firewall( firewall['name'], start, attempts )
@@ -157,8 +203,6 @@ module Beaker
157
203
  else
158
204
  @logger.debug("No zombie firewalls found")
159
205
  end
160
-
161
206
  end
162
-
163
207
  end
164
208
  end