cem_acpt 0.10.9 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/settings.local.json +7 -0
- data/CLAUDE.md +87 -0
- data/Gemfile.lock +3 -1
- data/cem_acpt.gemspec +1 -0
- data/exe/cem_acpt +1 -0
- data/exe/cem_acpt_image +1 -0
- data/lib/cem_acpt/cli.rb +25 -6
- data/lib/cem_acpt/config/base.rb +105 -30
- data/lib/cem_acpt/config/cem_acpt.rb +1 -1
- data/lib/cem_acpt/config/cem_acpt_image.rb +8 -0
- data/lib/cem_acpt/image_builder/provision_commands.rb +4 -6
- data/lib/cem_acpt/image_builder.rb +30 -9
- data/lib/cem_acpt/platform/gcp.rb +43 -4
- data/lib/cem_acpt/platform.rb +1 -0
- data/lib/cem_acpt/provision/terraform/linux.rb +1 -1
- data/lib/cem_acpt/provision/terraform/os_data.rb +1 -1
- data/lib/cem_acpt/provision/terraform/windows.rb +1 -1
- data/lib/cem_acpt/provision/terraform.rb +19 -4
- data/lib/cem_acpt/test_runner.rb +9 -1
- data/lib/cem_acpt/version.rb +1 -1
- data/lib/terraform/gcp/linux/main.tf +145 -145
- data/lib/terraform/gcp/windows/main.tf +64 -64
- data/lib/terraform/image/gcp/linux/main.tf +84 -76
- metadata +24 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1e3e1c15beb777d80ea6a65241d1433901f6d627eabd4f11be9547f79e941c03
|
|
4
|
+
data.tar.gz: bef63cc44f4e536da21599b01d00dc51524f60240e42ad22f23df6e5fadb36b7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 28f8b379445b92a4ff5e7df0fcd1a2f7385d7a6ee86c47bfdd825be23beb79c60752580e8c62c29f39d5fa5c3f623cc4834c4237679f74b19b847448ed16518f
|
|
7
|
+
data.tar.gz: 681f5f556c8ddcdc523db369c207e19c96e7378a82299862d10002fa9cc2d02ea8e6a3a416d84fd68594d194218e415f1a137b511bd0786386e0330a714518c5
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
## What This Project Is
|
|
4
|
+
|
|
5
|
+
`cem_acpt` is a Ruby gem providing an acceptance testing CLI for Puppet SCE (Security Compliance Enforcement), formerly known as Puppet CEM (Compliance Enforcement Modules). It provisions cloud nodes via Terraform, applies a Puppet manifest to the node, runs infrastructure tests via Goss, and tears everything down. It is also capable of running Puppet Bolt tests, which validate Bolt tasks and plans. A second binary, `cem_acpt_image`, builds the base VM images used by the test runner.
|
|
6
|
+
|
|
7
|
+
## Commands
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
bundle install # Install dependencies
|
|
11
|
+
|
|
12
|
+
bundle exec rake spec # Run all the spec tests
|
|
13
|
+
bundle exec rake spec SPEC=spec/path/to/file_spec.rb # Run a single spec file
|
|
14
|
+
|
|
15
|
+
rubocop # Lint
|
|
16
|
+
rubocop -a # Auto-fix lint issues
|
|
17
|
+
|
|
18
|
+
bundle exec rake build # Build the gem (saves to pkg/cem_acpt-<semver>.gem)
|
|
19
|
+
bundle exec gem install pkg/cem_acpt-X.X.X.gem # Install the gem with the given semver (X.X.X should be replaced with a version)
|
|
20
|
+
|
|
21
|
+
bundle exec exe/cem_acpt -h # CLI help
|
|
22
|
+
bundle exec exe/cem_acpt -Y # Print merged config (useful for debugging)
|
|
23
|
+
bundle exec exe/cem_acpt -X # Explain config merge sources
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
RuboCop target is Ruby 3.2. Line length limit is 200. Many complexity metrics (AbcSize, MethodLength, ClassLength) are disabled.
|
|
27
|
+
|
|
28
|
+
## Architecture
|
|
29
|
+
|
|
30
|
+
### Entry Points
|
|
31
|
+
|
|
32
|
+
- `exe/cem_acpt` → `lib/cem_acpt.rb` → dispatches to `TestRunner::Runner`
|
|
33
|
+
- `exe/cem_acpt_image` → `lib/cem_acpt.rb` → dispatches to `ImageBuilder::TerraformBuilder`
|
|
34
|
+
- CLI parsing lives in `lib/cem_acpt/cli.rb` (Ruby `OptionParser`)
|
|
35
|
+
|
|
36
|
+
### Configuration System (`lib/cem_acpt/config/`)
|
|
37
|
+
|
|
38
|
+
Config is merged from four sources, lowest to highest priority:
|
|
39
|
+
1. Environment variables (`CEM_ACPT_` prefix; nested keys use `__`)
|
|
40
|
+
2. User config at `~/.cem_acpt/config.yaml`
|
|
41
|
+
3. `--config FILE` option
|
|
42
|
+
4. Other CLI flags
|
|
43
|
+
|
|
44
|
+
`Config::Base` handles the merge logic via `deep_merge`. Subclasses `Config::CemAcpt` and `Config::CemAcptImage` define the schema for each command.
|
|
45
|
+
|
|
46
|
+
### Test Runner Lifecycle (`lib/cem_acpt/test_runner.rb`)
|
|
47
|
+
|
|
48
|
+
1. **Pre-provision** – Build the Puppet module tarball (`Utils::Puppet`)
|
|
49
|
+
2. **Provision** – Spin up GCP nodes via Terraform (`Provision::Terraform`), generate ephemeral SSH keys, install the module
|
|
50
|
+
3. **Execute** – Run action groups: `:goss` (async, HTTP) and `:bolt` (sync, Puppet tasks)
|
|
51
|
+
4. **Cleanup** – `terraform destroy` unless `--no-destroy-nodes`
|
|
52
|
+
|
|
53
|
+
### Platform Abstraction (`lib/cem_acpt/platform/`)
|
|
54
|
+
|
|
55
|
+
`Platform::Base` defines the interface; `Platform::Gcp` implements GCP-specific behaviour. Platforms are loaded dynamically by name from the config.
|
|
56
|
+
|
|
57
|
+
### Actions (`lib/cem_acpt/actions.rb`)
|
|
58
|
+
|
|
59
|
+
The `Actions` class orchestrates test execution. Action groups can be filtered with `--only-actions` / `--except-actions`. Goss actions run async via `async-http`; Bolt actions run synchronously.
|
|
60
|
+
|
|
61
|
+
### Terraform Templates (`lib/terraform/`)
|
|
62
|
+
|
|
63
|
+
Terraform HCL lives inside the gem under `lib/terraform/`. Paths:
|
|
64
|
+
- `lib/terraform/gcp/linux/` and `lib/terraform/gcp/windows/` – test node provisioning
|
|
65
|
+
- `lib/terraform/image/gcp/linux/` – image-building node provisioning
|
|
66
|
+
|
|
67
|
+
### Logging (`lib/cem_acpt/logging/`)
|
|
68
|
+
|
|
69
|
+
Supports simultaneous STDOUT + file logging. Pass `-I` / `--CI` for GitHub Actions–formatted output. Pass `--trace` to enable Ruby `TracePoint` debugging.
|
|
70
|
+
|
|
71
|
+
## Test Structure
|
|
72
|
+
|
|
73
|
+
Acceptance test cases live under `spec/acceptance/` (in the consuming module, not this gem):
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
spec/acceptance/<framework>_<os>-<version>_<firewall>_<profile>_<level>/
|
|
77
|
+
manifest.pp # Puppet manifest to apply
|
|
78
|
+
goss.yaml # Goss assertions
|
|
79
|
+
bolt.yaml # (optional) Bolt task assertions
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Unit tests mirror `lib/` under `spec/`. Fixtures are in `spec/fixtures/`.
|
|
83
|
+
|
|
84
|
+
## Conventions
|
|
85
|
+
|
|
86
|
+
- Document all Ruby code using Yard comments
|
|
87
|
+
- All changes should have tests
|
data/Gemfile.lock
CHANGED
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
PATH
|
|
2
2
|
remote: .
|
|
3
3
|
specs:
|
|
4
|
-
cem_acpt (0.
|
|
4
|
+
cem_acpt (0.11.0)
|
|
5
5
|
async-http (>= 0.60, < 0.70)
|
|
6
6
|
bcrypt_pbkdf (>= 1.0, < 2.0)
|
|
7
7
|
deep_merge (>= 1.2, < 2.0)
|
|
8
|
+
dotenv (>= 3.2, < 4.0)
|
|
8
9
|
ed25519 (>= 1.0, < 2.0)
|
|
9
10
|
puppet-modulebuilder (>= 0.0.1)
|
|
10
11
|
winrm (>= 2.3, < 3.0)
|
|
@@ -44,6 +45,7 @@ GEM
|
|
|
44
45
|
deep_merge (1.2.2)
|
|
45
46
|
diff-lcs (1.6.1)
|
|
46
47
|
docile (1.4.1)
|
|
48
|
+
dotenv (3.2.0)
|
|
47
49
|
ed25519 (1.3.0)
|
|
48
50
|
erubi (1.13.1)
|
|
49
51
|
ffi (1.17.1)
|
data/cem_acpt.gemspec
CHANGED
|
@@ -30,6 +30,7 @@ Gem::Specification.new do |spec|
|
|
|
30
30
|
spec.add_runtime_dependency 'async-http', '>= 0.60', '< 0.70'
|
|
31
31
|
spec.add_runtime_dependency 'bcrypt_pbkdf', '>= 1.0', '< 2.0'
|
|
32
32
|
spec.add_runtime_dependency 'deep_merge', '>= 1.2', '< 2.0'
|
|
33
|
+
spec.add_runtime_dependency 'dotenv', '>= 3.2', '< 4.0'
|
|
33
34
|
spec.add_runtime_dependency 'ed25519', '>= 1.0', '< 2.0'
|
|
34
35
|
spec.add_runtime_dependency 'puppet-modulebuilder', '>= 0.0.1'
|
|
35
36
|
spec.add_runtime_dependency 'winrm', '>= 2.3', '< 3.0'
|
data/exe/cem_acpt
CHANGED
data/exe/cem_acpt_image
CHANGED
data/lib/cem_acpt/cli.rb
CHANGED
|
@@ -4,6 +4,14 @@ require 'optparse'
|
|
|
4
4
|
|
|
5
5
|
module CemAcpt
|
|
6
6
|
module Cli
|
|
7
|
+
OPTIONS_DESC = [
|
|
8
|
+
'Set options. Example: -O "param1=value1,param2=value2"',
|
|
9
|
+
'For nested options, use dot notation to specify the nesting. For example,',
|
|
10
|
+
'-O "nested.param1=value1,nested.param2=value2"',
|
|
11
|
+
'Use this to set any config option or other runtime option. For example, to set the max_run_duration for a test',
|
|
12
|
+
'node, you would use: -O "node_data.max_run_duration=<duration in seconds>"',
|
|
13
|
+
].freeze
|
|
14
|
+
|
|
7
15
|
def self.parse_opts_for(command)
|
|
8
16
|
cmd = command
|
|
9
17
|
options = {}
|
|
@@ -56,8 +64,13 @@ module CemAcpt
|
|
|
56
64
|
options[:dry_run] = true
|
|
57
65
|
end
|
|
58
66
|
|
|
59
|
-
opts.on('--no-build-images', 'Do not build images
|
|
67
|
+
opts.on('--no-build-images', 'Do not build images from the provisioned nodes.') do
|
|
68
|
+
options[:no_build_images] = true
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
opts.on('--provision-only', 'Provision the nodes only. Equivalent to --no-build-images and --no-destroy-nodes.') do
|
|
60
72
|
options[:no_build_images] = true
|
|
73
|
+
options[:no_destroy_nodes] = true
|
|
61
74
|
end
|
|
62
75
|
|
|
63
76
|
opts.on('--no-linux', 'Do not build Linux images') do
|
|
@@ -83,10 +96,16 @@ module CemAcpt
|
|
|
83
96
|
options[:log_file] = file
|
|
84
97
|
end
|
|
85
98
|
|
|
86
|
-
opts.on('-O', '--options OPTS', '
|
|
87
|
-
params = o.split(',').map { |s| s.split('=') }.to_h
|
|
88
|
-
params.
|
|
89
|
-
|
|
99
|
+
opts.on('-O', '--options OPTS', OPTIONS_DESC.join(' ')) do |o|
|
|
100
|
+
params = o.split(',').map { |s| s.split('=', 2) }.to_h
|
|
101
|
+
params.each do |k, v|
|
|
102
|
+
keys = k.split('.') # Split the option key into parts for nested hashes
|
|
103
|
+
last_key = keys.pop # The last part is the actual key to set the value for
|
|
104
|
+
nested_hash = keys.reduce(options) do |h, key|
|
|
105
|
+
h[key.to_sym] ||= {} # Create nested hash if it doesn't exist
|
|
106
|
+
h[key.to_sym] # Return the nested hash for the next iteration
|
|
107
|
+
end
|
|
108
|
+
nested_hash[last_key.to_sym] = v # Set the value for the last key
|
|
90
109
|
end
|
|
91
110
|
end
|
|
92
111
|
|
|
@@ -115,7 +134,7 @@ module CemAcpt
|
|
|
115
134
|
options[:verbose] = true
|
|
116
135
|
end
|
|
117
136
|
|
|
118
|
-
opts.on('-S', '--no-
|
|
137
|
+
opts.on('-S', '--no-ephemeral-ssh-key', 'Do not generate an ephemeral SSH key for test suites') do
|
|
119
138
|
options[:no_ephemeral_ssh_key] = true
|
|
120
139
|
end
|
|
121
140
|
|
data/lib/cem_acpt/config/base.rb
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require 'digest'
|
|
4
4
|
require 'deep_merge'
|
|
5
|
+
require 'dotenv/load'
|
|
5
6
|
require 'fileutils'
|
|
6
7
|
require 'json'
|
|
7
8
|
require 'yaml'
|
|
@@ -13,6 +14,24 @@ module CemAcpt
|
|
|
13
14
|
module Config
|
|
14
15
|
using CemAcpt::CoreExt::ExtendedHash
|
|
15
16
|
|
|
17
|
+
# Provides a wrapper for secret values. This is used to prevent secrets from being printed in logs or error messages.
|
|
18
|
+
class Secret
|
|
19
|
+
attr_reader :key, :value
|
|
20
|
+
|
|
21
|
+
def initialize(key, value)
|
|
22
|
+
@value = value
|
|
23
|
+
@key = key
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def to_s
|
|
27
|
+
"Secret(#{key}=****)"
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def inspect
|
|
31
|
+
"#<#{self.class}:#{object_id.to_s(16)} key=#{key} value=**** >"
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
16
35
|
# Base class for other config classes
|
|
17
36
|
# Child classes should provide the following constant:
|
|
18
37
|
# - VALID_KEYS - provide an array of valid top-level keys for the config as symbols
|
|
@@ -36,6 +55,7 @@ module CemAcpt
|
|
|
36
55
|
provisioner
|
|
37
56
|
puppet
|
|
38
57
|
quiet
|
|
58
|
+
secrets
|
|
39
59
|
terraform
|
|
40
60
|
user_config
|
|
41
61
|
verbose
|
|
@@ -46,10 +66,25 @@ module CemAcpt
|
|
|
46
66
|
merge_nil_values: true,
|
|
47
67
|
}.freeze
|
|
48
68
|
|
|
69
|
+
def self.inherited(subclass)
|
|
70
|
+
subclass.instance_variable_set(:@load_hook, nil)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Add a block to be called after the config is loaded but before it is validated and frozen. The block will be
|
|
74
|
+
# passed the config hash and can be used to modify the config before it is finalized.
|
|
75
|
+
def self.load_hook(&block)
|
|
76
|
+
if block_given?
|
|
77
|
+
@load_hook = block
|
|
78
|
+
else
|
|
79
|
+
@load_hook
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
|
|
49
83
|
attr_reader :config, :env_vars
|
|
50
84
|
|
|
51
85
|
def initialize(opts: {}, config_file: nil, load_user_config: true)
|
|
52
86
|
@load_user_config = load_user_config
|
|
87
|
+
@explanation = {}
|
|
53
88
|
load(opts: opts, config_file: config_file)
|
|
54
89
|
end
|
|
55
90
|
|
|
@@ -78,11 +113,12 @@ module CemAcpt
|
|
|
78
113
|
|
|
79
114
|
# Load the configuration from the environment variables, config file, and opts
|
|
80
115
|
# The order of precedence is:
|
|
81
|
-
# 1.
|
|
82
|
-
# 2.
|
|
83
|
-
# 3.
|
|
84
|
-
# 4.
|
|
85
|
-
# 5.
|
|
116
|
+
# 1. static options (set in this class)
|
|
117
|
+
# 2. Runtime options
|
|
118
|
+
# 3. Runtime config file
|
|
119
|
+
# 4. User config file
|
|
120
|
+
# 5. Environment variables
|
|
121
|
+
# 6. Defaults
|
|
86
122
|
# @param opts [Hash] The options to load
|
|
87
123
|
# @param config_file [String] The config file to load
|
|
88
124
|
# @return [self] This object with the config loaded
|
|
@@ -92,9 +128,24 @@ module CemAcpt
|
|
|
92
128
|
add_env_vars!(@config)
|
|
93
129
|
@config.deep_merge!(user_config, **DEEP_MERGE_OPTS) if user_config && @load_user_config
|
|
94
130
|
@config.deep_merge!(config_from_file, **DEEP_MERGE_OPTS) if config_from_file
|
|
95
|
-
|
|
131
|
+
if @options
|
|
132
|
+
@config.deep_merge!(@options, **DEEP_MERGE_OPTS)
|
|
133
|
+
@options.each do |key, _value|
|
|
134
|
+
add_config_explanation(key, "runtime option")
|
|
135
|
+
end
|
|
136
|
+
end
|
|
96
137
|
add_static_options!(@config)
|
|
138
|
+
# Run the load hook if it is defined. This allows child classes to modify the config after it has been loaded
|
|
139
|
+
# but before it is validated and frozen.
|
|
140
|
+
block = self.class.load_hook
|
|
141
|
+
instance_eval(&block) if block
|
|
97
142
|
@config.format! # Symbolize keys of all hashes
|
|
143
|
+
# Remove any keys that are not in the valid keys list for this config. This prevents invalid config options
|
|
144
|
+
# from being set.
|
|
145
|
+
@config.select! { |key, _| valid_keys.include?(key) }
|
|
146
|
+
# Wrap secrets in the config with the Secret class to prevent them from being accidentally printed in logs
|
|
147
|
+
# or error messages. WARNING: Secrets can leak from Terraform logging.
|
|
148
|
+
wrap_secrets!
|
|
98
149
|
validate_config!
|
|
99
150
|
@dot_key_cache = {}
|
|
100
151
|
# Freeze the config so it can't be modified
|
|
@@ -106,21 +157,9 @@ module CemAcpt
|
|
|
106
157
|
|
|
107
158
|
# Returns a string representation of how the config was loaded
|
|
108
159
|
def explain
|
|
109
|
-
explanation
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
next if source_vals.nil? || source_vals.empty?
|
|
113
|
-
|
|
114
|
-
# The loop below will overwrite the value of explanation[key] if the same key is found in multiple sources
|
|
115
|
-
# This is intentional, as the last source to set the value is the one that should be used
|
|
116
|
-
source_vals.each do |key, value|
|
|
117
|
-
explanation[key] = source if @config.dget(key.to_s) == value
|
|
118
|
-
end
|
|
119
|
-
end
|
|
120
|
-
explained = explanation.each_with_object([]) do |(key, value), ary|
|
|
121
|
-
ary << "Key '#{key}' from source '#{value}'"
|
|
122
|
-
end
|
|
123
|
-
explained.join("\n")
|
|
160
|
+
@explanation.each_with_object([]) { |(key, values), ary|
|
|
161
|
+
ary << "#{key}:\n -->#{values.join("\n -->")}"
|
|
162
|
+
}.join("\n")
|
|
124
163
|
end
|
|
125
164
|
|
|
126
165
|
def [](key)
|
|
@@ -166,17 +205,24 @@ module CemAcpt
|
|
|
166
205
|
end
|
|
167
206
|
alias quiet? quiet_mode?
|
|
168
207
|
|
|
169
|
-
def to_yaml
|
|
170
|
-
@config.to_yaml
|
|
208
|
+
def to_yaml(expose_secrets: false)
|
|
209
|
+
return @config.to_yaml unless @config.key?(:secrets)
|
|
210
|
+
|
|
211
|
+
serializable_secrets = @config[:secrets].transform_values { |v| v.is_a?(Secret) ? (expose_secrets ? v.value : v.to_s) : v }
|
|
212
|
+
@config.merge(secrets: serializable_secrets).to_yaml
|
|
171
213
|
end
|
|
172
214
|
|
|
173
|
-
def to_json(*args)
|
|
174
|
-
@config.to_json(*args)
|
|
215
|
+
def to_json(*args, expose_secrets: false)
|
|
216
|
+
return @config.to_json(*args) unless @config.key?(:secrets)
|
|
217
|
+
|
|
218
|
+
serializable_secrets = @config[:secrets].transform_values { |v| v.is_a?(Secret) ? (expose_secrets ? v.value : v.to_s) : v }
|
|
219
|
+
@config.merge(secrets: serializable_secrets).to_json(*args)
|
|
175
220
|
end
|
|
176
221
|
|
|
177
222
|
private
|
|
178
223
|
|
|
179
224
|
attr_reader :options
|
|
225
|
+
attr_writer :config
|
|
180
226
|
|
|
181
227
|
def user_config_dir
|
|
182
228
|
@user_config_dir ||= File.join(Dir.home, '.cem_acpt')
|
|
@@ -206,11 +252,20 @@ module CemAcpt
|
|
|
206
252
|
env_var.sub('CEM_ACPT_', '').gsub(%r{__}, '.').downcase
|
|
207
253
|
end
|
|
208
254
|
|
|
255
|
+
def add_config_explanation(key, source)
|
|
256
|
+
@explanation[key] ||= []
|
|
257
|
+
@explanation[key] << source unless @explanation[key].include?(source)
|
|
258
|
+
end
|
|
259
|
+
|
|
209
260
|
def add_static_options!(config)
|
|
210
261
|
config.dset('user_config.dir', user_config_dir)
|
|
262
|
+
add_config_explanation('user_config.dir', 'static value')
|
|
211
263
|
config.dset('user_config.file', user_config_file)
|
|
264
|
+
add_config_explanation('user_config.file', 'static value')
|
|
212
265
|
config.dset('provisioner', 'terraform')
|
|
266
|
+
add_config_explanation('provisioner', 'static value')
|
|
213
267
|
config.dset('terraform.dir', terraform_dir)
|
|
268
|
+
add_config_explanation('terraform.dir', 'static value')
|
|
214
269
|
set_third_party_env_vars!(config)
|
|
215
270
|
end
|
|
216
271
|
|
|
@@ -221,10 +276,13 @@ module CemAcpt
|
|
|
221
276
|
def set_third_party_env_vars!(config)
|
|
222
277
|
if ENV['RUNNER_DEBUG'] == '1'
|
|
223
278
|
config.dset('log_level', 'debug')
|
|
279
|
+
add_config_explanation('log_level', "environment variable 'RUNNER_DEBUG' (static value)")
|
|
224
280
|
config.dset('verbose', true)
|
|
281
|
+
add_config_explanation('verbose', "environment variable 'RUNNER_DEBUG' (static value)")
|
|
225
282
|
end
|
|
226
283
|
return unless ENV['GITHUB_ACTIONS'] == 'true' || ENV['CI'] == 'true'
|
|
227
284
|
config.dset('ci_mode', true)
|
|
285
|
+
add_config_explanation('ci_mode', "environment variable 'GITHUB_ACTIONS' or 'CI' (static value)")
|
|
228
286
|
end
|
|
229
287
|
|
|
230
288
|
# Used to source the config during loading of config files.
|
|
@@ -253,6 +311,9 @@ module CemAcpt
|
|
|
253
311
|
# Set the parameterized defaults
|
|
254
312
|
config_file = ENV["#{env_var_prefix}_CONFIG_FILE"] if config_file.nil?
|
|
255
313
|
@config.dset('config_file', config_file) if config_file
|
|
314
|
+
@config.each do |key, _value|
|
|
315
|
+
add_config_explanation(key, "default value")
|
|
316
|
+
end
|
|
256
317
|
@options = opts || {}
|
|
257
318
|
end
|
|
258
319
|
|
|
@@ -267,6 +328,7 @@ module CemAcpt
|
|
|
267
328
|
next unless valid_keys.include?(key.split('.').first.to_sym) # Skip if the key is not a known config key
|
|
268
329
|
@env_vars[key] = value
|
|
269
330
|
config.dset(key, value)
|
|
331
|
+
add_config_explanation(key, "environment variable '#{env_var}'")
|
|
270
332
|
end
|
|
271
333
|
end
|
|
272
334
|
|
|
@@ -282,15 +344,14 @@ module CemAcpt
|
|
|
282
344
|
@user_config
|
|
283
345
|
end
|
|
284
346
|
|
|
285
|
-
# def config_from_file
|
|
286
|
-
# {}
|
|
287
|
-
# end
|
|
288
|
-
|
|
289
347
|
def load_config_file(config_file)
|
|
290
348
|
return {} if config_file.nil? || config_file.empty? || !File.exist?(File.expand_path(config_file))
|
|
291
349
|
|
|
292
350
|
loaded = load_yaml(config_file)
|
|
293
351
|
loaded.format!
|
|
352
|
+
loaded.each do |key, _value|
|
|
353
|
+
add_config_explanation(key, "config file '#{File.expand_path(config_file)}'")
|
|
354
|
+
end
|
|
294
355
|
loaded
|
|
295
356
|
end
|
|
296
357
|
|
|
@@ -300,7 +361,7 @@ module CemAcpt
|
|
|
300
361
|
conf_file = find_option('config_file')
|
|
301
362
|
return {} if conf_file.nil? || conf_file.empty?
|
|
302
363
|
|
|
303
|
-
unless conf_file
|
|
364
|
+
unless conf_file.is_a?(String)
|
|
304
365
|
warn "Invalid config_file type '#{conf_file.class}'. Must be a String."
|
|
305
366
|
return {}
|
|
306
367
|
end
|
|
@@ -329,6 +390,12 @@ module CemAcpt
|
|
|
329
390
|
end
|
|
330
391
|
end
|
|
331
392
|
|
|
393
|
+
def wrap_secrets!
|
|
394
|
+
return unless @config.key?(:secrets)
|
|
395
|
+
|
|
396
|
+
@config[:secrets].merge!(@config[:secrets]) { |key, value, _| Secret.new(key, value) }
|
|
397
|
+
end
|
|
398
|
+
|
|
332
399
|
def validate_config!
|
|
333
400
|
return unless @config && !valid_keys.empty?
|
|
334
401
|
|
|
@@ -345,10 +412,18 @@ module CemAcpt
|
|
|
345
412
|
def create_terraform_dir!
|
|
346
413
|
raise 'Cannot create terraform dir without a user config dir' unless Dir.exist? user_config_dir
|
|
347
414
|
|
|
415
|
+
checksum = nil
|
|
348
416
|
if File.exist?(module_terraform_checksum_file)
|
|
349
417
|
checksum = File.read(module_terraform_checksum_file).strip
|
|
350
418
|
return if checksum == module_terraform_checksum
|
|
351
419
|
end
|
|
420
|
+
if checksum
|
|
421
|
+
warn 'Updating local terraform files with latest from gem...'
|
|
422
|
+
warn "Old checksum: #{checksum}"
|
|
423
|
+
warn "New checksum: #{module_terraform_checksum}"
|
|
424
|
+
else
|
|
425
|
+
warn 'Copying terraform files to local config...'
|
|
426
|
+
end
|
|
352
427
|
module_terraform_dir = File.expand_path(File.join(__dir__, '..', '..', 'terraform'))
|
|
353
428
|
FileUtils.rm_rf(File.join(user_config_dir, 'terraform'))
|
|
354
429
|
FileUtils.cp_r(module_terraform_dir, user_config_dir)
|
|
@@ -44,7 +44,7 @@ module CemAcpt
|
|
|
44
44
|
quiet: false,
|
|
45
45
|
test_data: {
|
|
46
46
|
for_each: {
|
|
47
|
-
collection: %w[
|
|
47
|
+
collection: %w[puppet8],
|
|
48
48
|
},
|
|
49
49
|
vars: {},
|
|
50
50
|
name_pattern_vars: %r{^(?<framework>[a-z]+)_(?<image_fam>[a-z0-9-]+)_(?<firewall>[a-z]+)_(?<framework_vars>[-_a-z0-9]+)$},
|
|
@@ -12,8 +12,15 @@ module CemAcpt
|
|
|
12
12
|
images
|
|
13
13
|
image_name_filter
|
|
14
14
|
no_build_images
|
|
15
|
+
node_data
|
|
15
16
|
].freeze
|
|
16
17
|
|
|
18
|
+
# Uses the load hook to filter the images based on the image_name_filter config option.
|
|
19
|
+
load_hook do
|
|
20
|
+
self.config[:images].delete_if { |k, _| !k.match(self.config[:image_name_filter]) }
|
|
21
|
+
self.add_config_explanation('images', "filtered by 'image_name_filter' config option")
|
|
22
|
+
end
|
|
23
|
+
|
|
17
24
|
def env_var_prefix
|
|
18
25
|
'CEM_ACPT_IMAGE'
|
|
19
26
|
end
|
|
@@ -35,6 +42,7 @@ module CemAcpt
|
|
|
35
42
|
log_format: 'text',
|
|
36
43
|
no_destroy_nodes: false,
|
|
37
44
|
no_ephemeral_ssh_key: false,
|
|
45
|
+
node_data: {},
|
|
38
46
|
platform: {
|
|
39
47
|
name: 'gcp',
|
|
40
48
|
},
|
|
@@ -50,14 +50,12 @@ module CemAcpt
|
|
|
50
50
|
commands = []
|
|
51
51
|
# Replace the commented out username and password lines with the actual username and password
|
|
52
52
|
# This is necessary to access the private Puppet platform repository
|
|
53
|
-
#
|
|
53
|
+
#
|
|
54
54
|
# Updating the username first
|
|
55
55
|
commands.push("sudo sed -i 's/^.*#username=.*$/username=forge-key/' /etc/yum.repos.d/puppet#{@puppet_version}-release.repo")
|
|
56
56
|
|
|
57
57
|
# Updating the password next
|
|
58
|
-
|
|
59
|
-
# "sudo sed -i 's/#password=/password=#{ENV['PUPPET_AUTH_TOKEN']}/' /etc/yum.repos.d/puppet#{@puppet_version}-release.repo"
|
|
60
|
-
commands.push("sudo sed -i 's/^.*#password=.*$/password=$PUPPET_AUTH_TOKEN/' /etc/yum.repos.d/puppet#{@puppet_version}-release.repo")
|
|
58
|
+
commands.push("sudo sed -i \"s/^.*#password=.*$/password=$PUPPET_AUTH_TOKEN/\" /etc/yum.repos.d/puppet#{@puppet_version}-release.repo")
|
|
61
59
|
commands
|
|
62
60
|
end
|
|
63
61
|
|
|
@@ -122,9 +120,9 @@ module CemAcpt
|
|
|
122
120
|
|
|
123
121
|
def set_repo_username_and_password
|
|
124
122
|
commands = []
|
|
125
|
-
# This is necessary to access the private Puppet platform repository
|
|
123
|
+
# This is necessary to access the private Puppet platform repository
|
|
126
124
|
# Modifying the conf file at /et/apt/auth.conf.d/puppetcore.conf by adding the username and password
|
|
127
|
-
|
|
125
|
+
|
|
128
126
|
# Adding the server URL
|
|
129
127
|
commands.push("sudo echo machine apt-puppetcore.puppet.com | sudo tee /etc/apt/auth.conf.d/apt-puppetcore-puppet.conf > /dev/null")
|
|
130
128
|
# Adding the username
|
|
@@ -61,6 +61,9 @@ module CemAcpt
|
|
|
61
61
|
image_types = []
|
|
62
62
|
image_types << [@linux_tfvars, 'linux'] unless no_linux?
|
|
63
63
|
image_types << [@windows_tfvars, 'windows'] unless no_windows?
|
|
64
|
+
if image_types.empty?
|
|
65
|
+
raise 'No images to build. Ensure the images config is populated and that --filter (if used) matches at least one image.'
|
|
66
|
+
end
|
|
64
67
|
return dry_run(image_types) if @config.get('dry_run')
|
|
65
68
|
|
|
66
69
|
@working_dir = new_working_dir
|
|
@@ -75,8 +78,10 @@ module CemAcpt
|
|
|
75
78
|
terraform_apply(os_str, DEFAULT_PLAN_NAME)
|
|
76
79
|
output = JSON.parse(terraform_output(os_str, 'node-data', json: true))
|
|
77
80
|
output.each do |instance_name, data|
|
|
78
|
-
|
|
79
|
-
|
|
81
|
+
unless @config.get('no_destroy_nodes')
|
|
82
|
+
logger.info('CemAcpt::ImageBuilder') { "Stopping instance #{instance_name}..." }
|
|
83
|
+
@exec.run('compute', 'instances', 'stop', instance_name)
|
|
84
|
+
end
|
|
80
85
|
unless @config.get('no_build_images')
|
|
81
86
|
deprecate_old_images_in_family(data['image_family'])
|
|
82
87
|
create_image_from_disk(data['disk_link'], image_name_from_image_family(data['image_family']), data['image_family'])
|
|
@@ -210,8 +215,6 @@ module CemAcpt
|
|
|
210
215
|
end
|
|
211
216
|
|
|
212
217
|
def terraform_init
|
|
213
|
-
raise 'Cannot initialize Terraform, both no_linux and no_windows are true' if no_linux? && no_windows?
|
|
214
|
-
|
|
215
218
|
logger.debug('CemAcpt::ImageBuilder') { 'Initializing Terraform' }
|
|
216
219
|
in_os_dir('linux') do
|
|
217
220
|
terraform.init({ input: false, no_color: true })
|
|
@@ -225,7 +228,7 @@ module CemAcpt
|
|
|
225
228
|
in_os_dir(os_dir) do
|
|
226
229
|
logger.debug('CemAcpt::ImageBuilder') { "Creating Terraform plan #{plan_name} for #{os_dir}" }
|
|
227
230
|
logger.verbose('CemAcpt::ImageBuilder') { "Using vars:\n#{JSON.pretty_generate(tfvars)}" }
|
|
228
|
-
terraform.plan({ input: false, no_color: true, plan: plan_name, vars: tfvars }, { environment: environment })
|
|
231
|
+
terraform.plan({ input: false, no_color: true, plan: plan_name, vars: terraform_vars(tfvars) }, { environment: environment })
|
|
229
232
|
end
|
|
230
233
|
end
|
|
231
234
|
|
|
@@ -247,7 +250,7 @@ module CemAcpt
|
|
|
247
250
|
in_os_dir(os_dir) do
|
|
248
251
|
logger.debug('CemAcpt::ImageBuilder') { "Destroying Terraform resources for #{os_dir}" }
|
|
249
252
|
logger.verbose('CemAcpt::ImageBuilder') { "Using vars:\n#{JSON.pretty_generate(tfvars)}" }
|
|
250
|
-
terraform.destroy({ auto_approve: true, input: false, no_color: true, vars: tfvars }, { environment: environment })
|
|
253
|
+
terraform.destroy({ auto_approve: true, input: false, no_color: true, vars: terraform_vars(tfvars) }, { environment: environment })
|
|
251
254
|
end
|
|
252
255
|
end
|
|
253
256
|
|
|
@@ -258,17 +261,35 @@ module CemAcpt
|
|
|
258
261
|
end
|
|
259
262
|
end
|
|
260
263
|
|
|
264
|
+
# Keys in node_data that are passed to Terraform. Other keys are used only for Ruby-side logic.
|
|
265
|
+
TERRAFORM_NODE_DATA_KEYS = %i[image_family machine_type max_run_duration base_image disk_size provision_commands].freeze
|
|
266
|
+
|
|
267
|
+
# Returns a copy of tfvars safe to pass directly to TerraformCmd:
|
|
268
|
+
# - Secret values are unwrapped to their plain string values
|
|
269
|
+
# - node_data entries are filtered to only the keys defined in the Terraform variable type
|
|
270
|
+
def terraform_vars(tfvars)
|
|
271
|
+
result = tfvars.transform_values do |v|
|
|
272
|
+
v.is_a?(CemAcpt::Config::Secret) ? v.value : v
|
|
273
|
+
end
|
|
274
|
+
result[:node_data] = result[:node_data].transform_values do |node|
|
|
275
|
+
node.slice(*TERRAFORM_NODE_DATA_KEYS)
|
|
276
|
+
end
|
|
277
|
+
result
|
|
278
|
+
end
|
|
279
|
+
|
|
261
280
|
# @return [Hash] A hash of the Terraform variables to use
|
|
262
281
|
def new_tfvars(config)
|
|
282
|
+
if !config.has?('secrets.puppet_auth_token')
|
|
283
|
+
raise ArgumentError, 'Missing required config value: secrets.puppet_auth_token'
|
|
284
|
+
end
|
|
263
285
|
tfvars = { node_data: {} }
|
|
264
286
|
private_key, public_key, _ = new_ephemeral_ssh_keys
|
|
265
287
|
tfvars[:private_key] = private_key if private_key
|
|
266
288
|
tfvars[:public_key] = public_key if public_key
|
|
289
|
+
tfvars[:puppet_auth_token] = config.get('secrets.puppet_auth_token')
|
|
267
290
|
config.get('images').each do |image_name, image|
|
|
268
|
-
next if config.get('image_name_filter')&.respond_to?(:match?) && !config.get('image_name_filter').match?(image_name)
|
|
269
|
-
|
|
270
291
|
platform = new_platform(config, **tfvars)
|
|
271
|
-
tfvars.merge
|
|
292
|
+
tfvars = platform.platform_data.merge(tfvars)
|
|
272
293
|
provision_commands = CemAcpt::ImageBuilder::ProvisionCommands.provision_commands(
|
|
273
294
|
config,
|
|
274
295
|
image_name: image_name,
|