vanagon 0.13.1 → 0.14.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: c1fbe8fc537feff6f6fe49b077602c6f2fa1dff5
4
- data.tar.gz: 577e9e051c48bc9d2c9da9f7b1b08dabdd62e87c
3
+ metadata.gz: 072c1ccfd65eb990500787ebd5cfa03d25a4ae09
4
+ data.tar.gz: 7e3a2687a3e8a8b367fe195d1d853ce782d46005
5
5
  SHA512:
6
- metadata.gz: cd644943daee7fcaa360bd40a1285c7c341f6736b8750a7157373662de3329b8c200ac713bff758a75c9aea74f54d1a3f2eedb7aa3d7034a529917ed819c08c1
7
- data.tar.gz: c98eb69b2ac57756d663ff61d83d2c1c3dbf35d925fd9f4c7cc4219585c4f01208ee895a950871c293077ea080f754b1389f269a0d25d6b32e03f292c14a3034
6
+ metadata.gz: d015eede00d49017508043fd1f235b46b2ef61b4e608abd75cb8fec863fafe4d5b0f63a4dbe1a26c6a32cae47424f8ec63ae33603cb943a960b8588b9126982f
7
+ data.tar.gz: 86c8a5b9ea8dce5c3d0b3fb83f0c1280e8dc94ce37f1256f5c0707bc229e570e4570f6a83fae8b0138655e270a3711c80a91ff27f33f07f34a3979aeb3dda3a6
data/README.md CHANGED
@@ -11,7 +11,6 @@
11
11
  - [Configuration and Usage](#configuration-and-usage)
12
12
  - [`build` usage](#build-usage)
13
13
  - [`inspect` usage](#inspect-usage)
14
- - [`devkit` usage](#devkit-usage)
15
14
  - [Engines](#engines)
16
15
  - [Amazon Ec2](#amazon-ec2)
17
16
  - [Contributing](#contributing)
@@ -33,13 +32,6 @@ Vanagon builds up a Makefile and packaging files (specfile for RPM,
33
32
  control/rules/etc for DEB) and copies them to a remote host, where make can be
34
33
  invoked to build all of the components and make a package of the contents.
35
34
 
36
- Vanagon also provides a devkit command that will prepare a machine as a
37
- development environment for the entire project, or restricted to individual
38
- components of the project. The devkit command installs all required build tools,
39
- creates a master makefile for the project, and configures, builds, and installs
40
- all components. The result is an environment where you can work on individual
41
- components, then rebuild the project and test the installed artifacts.
42
-
43
35
  ### How is it pronounced?
44
36
 
45
37
  Vanagon (/ˈvænəgɪn/) sounds like "van again." It does not sound like "van wagon" or "van and gone."
@@ -255,43 +247,6 @@ build-time configuration, environment variables, and expected artifacts.
255
247
 
256
248
  ---
257
249
 
258
- ### `devkit` usage
259
-
260
- The devkit command has positional arguments and position independent flagged
261
- arguments.
262
-
263
- #### Arguments (position dependent)
264
-
265
- ##### project name
266
- As in `build` arguments.
267
-
268
- ##### platform name
269
- As in `build` arguments.
270
-
271
- ##### component names <optional>
272
- Specifies specific components that should be built. If components are not
273
- specified, then all components in the project will be built. If components
274
- are specified as arguments, then any in the project that aren't specified
275
- as arguments will be retrieved from packages rather than built from source.
276
-
277
- #### Flagged arguments
278
-
279
- **Note:** command flags can be used anywhere in the command.
280
-
281
- Supports all flagged arguments from the `build` command.
282
-
283
- ##### -t HOST, --target HOST
284
- As in the `build` target host optional argument.
285
-
286
- #### Flags
287
-
288
- **Note:** command flags can be used anywhere in the command.
289
-
290
- ##### -h, --help
291
- Display command-line help.
292
-
293
- ---
294
-
295
250
  Engines
296
251
  ---
297
252
 
data/bin/ship CHANGED
@@ -12,4 +12,17 @@ else
12
12
  require 'packaging'
13
13
  Pkg::Util::RakeUtils.load_packaging_tasks
14
14
  Pkg::Util::RakeUtils.invoke_task('pl:jenkins:ship', 'artifacts', 'output')
15
+ begin
16
+ Pkg::Util::RakeUtils.invoke_task('pl:jenkins:ship_to_artifactory', 'output')
17
+ rescue
18
+ warn <<-DOC
19
+ Unable to ship packages to artifactory. Please make sure you are pointing to a
20
+ recent version of packaging in your Gemfile. Please also make sure you include
21
+ the artifactory gem in your Gemfile.
22
+
23
+ i.e.,
24
+ gem 'packaging', :github => 'puppetlabs/packaging', branch: '1.0.x'
25
+ gem 'artifactory'
26
+ DOC
27
+ end
15
28
  end
@@ -92,6 +92,15 @@ class Vanagon
92
92
  # preinstall_actions is a two-dimensional Array, describing scripts that
93
93
  # should be executed before a given component is installed.
94
94
  attr_accessor :preinstall_actions
95
+ # install_triggers is a one-dimensional Array of OpenStructs, describing scripts that
96
+ # should be executed when a package is installed or upgraded
97
+ attr_accessor :install_triggers
98
+ # interest_triggers is a one-dimensional Array of OpenStructs, describing scripts that
99
+ # should be executed when a package identifies an interest trigger
100
+ attr_accessor :interest_triggers
101
+ # activate_triggers is a one-dimentional Array of Strings, describing scripts that
102
+ # should be executed when a package identifies an activate trigger
103
+ attr_accessor :activate_triggers
95
104
  # postinstall_actions is a two-dimensional Array, describing scripts that
96
105
  # should be executed after a given component is installed.
97
106
  attr_accessor :postinstall_actions
@@ -106,6 +115,12 @@ class Vanagon
106
115
  # usually a String, but not required to be.
107
116
  attr_accessor :cleanup_source
108
117
 
118
+ # When dealing with compiled artifacts generated by setting
119
+ # `project.generate_archives true`, you only need to install the component.
120
+ # By setting install_only to true the component 'build' will exclude the
121
+ # unpack, patch, configure, build, and check steps.
122
+ attr_accessor :install_only
123
+
109
124
  # Loads a given component from the configdir
110
125
  #
111
126
  # @param name [String] the name of the component
@@ -152,9 +167,13 @@ class Vanagon
152
167
  @environment = Vanagon::Environment.new
153
168
  @sources = []
154
169
  @preinstall_actions = []
170
+ @install_triggers = []
171
+ @interest_triggers = []
172
+ @activate_triggers = []
155
173
  @postinstall_actions = []
156
174
  @preremove_actions = []
157
175
  @postremove_actions = []
176
+ @install_only = false
158
177
  end
159
178
 
160
179
  # Adds the given file to the list of files and returns @files.
@@ -193,7 +212,9 @@ class Vanagon
193
212
 
194
213
  # @return [Set] a list of unique mirror URIs that should be used to
195
214
  # retrieve the upstream source before attempting to retrieve from
196
- # whatever URI was defined for #uri.
215
+ # whatever URI was defined for #uri. If no mirrors are set and the
216
+ # deprecated rewrite system has been configured, this will return
217
+ # rewritten URIs
197
218
  def mirrors
198
219
  @mirrors ||= Set.new [deprecated_rewrite_url].compact
199
220
  end
@@ -243,10 +264,12 @@ class Vanagon
243
264
  end
244
265
 
245
266
  # This method is deprecated and private. It will return
246
- # a rewritten URL if there's any value for #url.
267
+ # a rewritten URL if there's any value for #url and a
268
+ # rewrite rule for that URL has been set
247
269
  def deprecated_rewrite_url
248
270
  return nil unless url
249
- Vanagon::Component::Source::Rewrite.parse_and_rewrite(url)
271
+ rewritten_url = Vanagon::Component::Source::Rewrite.parse_and_rewrite(url)
272
+ url == rewritten_url ? nil : rewritten_url
250
273
  end
251
274
  private :deprecated_rewrite_url
252
275
 
@@ -416,21 +416,61 @@ class Vanagon
416
416
  eos
417
417
  end
418
418
 
419
+ # Checks that the array of pkg_state is valid (install AND/OR upgrade).
420
+ # Returns vanagon error if invalid
421
+ #
422
+ # @param pkg_state [Array] array of pkg_state input to test
423
+ def check_pkg_state_array(pkg_state)
424
+ if pkg_state.empty? || (pkg_state - ["install", "upgrade"]).any?
425
+ raise Vanagon::Error, "#{pkg_state} should be an array containing one or more of ['install', 'upgrade']"
426
+ end
427
+ end
428
+
419
429
  # Adds action to run during the preinstall phase of packaging
420
430
  #
421
- # @param pkg_state [Array] the state in which the scripts should execute. Can be
422
- # one or multiple of 'install' and 'upgrade'.
431
+ # @param pkg_state [String, Array] the package state during which the scripts should execute.
432
+ # Accepts either a single string ("install" or "upgrade"), or an Array of Strings (["install", "upgrade"]).
423
433
  # @param scripts [Array] the Bourne shell compatible scriptlet(s) to execute
424
434
  def add_preinstall_action(pkg_state, scripts)
425
435
  pkg_state = Array(pkg_state)
426
436
  scripts = Array(scripts)
427
-
428
- if pkg_state.empty? || !(pkg_state - ["install", "upgrade"]).empty?
429
- raise Vanagon::Error, "#{pkg_state} should be an array containing one or more of ['install', 'upgrade']"
430
- end
437
+ check_pkg_state_array(pkg_state)
431
438
  @component.preinstall_actions << OpenStruct.new(:pkg_state => pkg_state, :scripts => scripts)
432
439
  end
433
440
 
441
+ # Adds trigger for scripts to be run on specified pkg_state.
442
+ #
443
+ # @param pkg_state [String, Array] the package state during which the scripts should execute.
444
+ # Accepts either a single string ("install" or "upgrade"), or an Array of Strings (["install", "upgrade"]).
445
+ # @param scripts [Array] the rpm pkg scriptlet(s) to execute
446
+ # @param pkg [String] the package the trigger will be set in
447
+ def add_rpm_install_triggers(pkg_state, scripts, pkg)
448
+ pkg_state = Array(pkg_state)
449
+ scripts = Array(scripts)
450
+ check_pkg_state_array(pkg_state)
451
+ @component.install_triggers << OpenStruct.new(:pkg_state => pkg_state, :scripts => scripts, :pkg => pkg)
452
+ end
453
+
454
+ # Adds interest trigger based on the specified packaging state and interest name.
455
+ #
456
+ # @param pkg_state [String, Array] the package state during which the scripts should execute.
457
+ # Accepts either a single string ("install" or "upgrade"), or an Array of Strings (["install", "upgrade"]).
458
+ # @param scripts [Array] the scripts to run for the interest trigger
459
+ # @param interest_name [String] the name of the interest trigger
460
+ def add_debian_interest_triggers(pkg_state, scripts, interest_name)
461
+ pkg_state = Array(pkg_state)
462
+ scripts = Array(scripts)
463
+ check_pkg_state_array(pkg_state)
464
+ @component.interest_triggers << OpenStruct.new(:pkg_state => pkg_state, :scripts => scripts, :interest_name => interest_name)
465
+ end
466
+
467
+ # Adds activate trigger name to be watched
468
+ #
469
+ # @param activate_name [String] the activate trigger name
470
+ def add_debian_activate_triggers(activate_name)
471
+ @component.activate_triggers << OpenStruct.new(:activate_name => activate_name)
472
+ end
473
+
434
474
  # Adds action to run during the postinstall phase of packaging
435
475
  #
436
476
  # @param pkg_state [Array] the state in which the scripts should execute. Can be
@@ -439,10 +479,7 @@ class Vanagon
439
479
  def add_postinstall_action(pkg_state, scripts)
440
480
  pkg_state = Array(pkg_state)
441
481
  scripts = Array(scripts)
442
-
443
- if pkg_state.empty? || !(pkg_state - ["install", "upgrade"]).empty?
444
- raise Vanagon::Error, "#{pkg_state} should be an array containing one or more of ['install', 'upgrade']"
445
- end
482
+ check_pkg_state_array(pkg_state)
446
483
  @component.postinstall_actions << OpenStruct.new(:pkg_state => pkg_state, :scripts => scripts)
447
484
  end
448
485
 
@@ -479,6 +516,10 @@ class Vanagon
479
516
  def license(license)
480
517
  @component.license = license
481
518
  end
519
+
520
+ def install_only(install_only)
521
+ @component.install_only = install_only
522
+ end
482
523
  end
483
524
  end
484
525
  end
@@ -47,8 +47,8 @@ class Vanagon
47
47
  # in the returned rules.
48
48
  #
49
49
  # @return [Array<Makefile::Rule>]
50
- def rules
51
- list = [
50
+ def rules # rubocop:disable Metrics/AbcSize
51
+ list_of_rules = [
52
52
  component_rule,
53
53
  unpack_rule,
54
54
  patch_rule,
@@ -59,11 +59,20 @@ class Vanagon
59
59
  clean_rule,
60
60
  clobber_rule,
61
61
  ]
62
+
63
+ if component.install_only
64
+ list_of_rules = [
65
+ component_rule,
66
+ install_rule,
67
+ clean_rule,
68
+ clobber_rule,
69
+ ]
70
+ end
62
71
  if project.cleanup
63
- list << cleanup_rule
72
+ list_of_rules << cleanup_rule
64
73
  end
65
74
 
66
- list
75
+ list_of_rules
67
76
  end
68
77
 
69
78
  # Generate a top level rule to build this component.
@@ -148,13 +157,20 @@ class Vanagon
148
157
 
149
158
  # Install this component.
150
159
  rule("install") do |r|
151
- r.dependencies = ["#{component.name}-check"]
160
+ r.dependencies = ["#{component.name}-check"] unless component.install_only
152
161
  unless component.install.empty?
153
- r.recipe << andand_multiline(
154
- component.environment_variables,
155
- "cd #{component.get_build_dir}",
156
- component.install
157
- )
162
+ if component.install_only
163
+ r.recipe << andand_multiline(
164
+ component.environment_variables,
165
+ component.install
166
+ )
167
+ else
168
+ r.recipe << andand_multiline(
169
+ component.environment_variables,
170
+ "cd #{component.get_build_dir}",
171
+ component.install
172
+ )
173
+ end
158
174
  end
159
175
 
160
176
  after_install_patches = component.patches.select { |patch| patch.after == "install" }
@@ -43,11 +43,12 @@ class Vanagon
43
43
  # Constructor for the Http source type
44
44
  #
45
45
  # @param url [String] url of the http source to fetch
46
- # @param sum [String] sum to verify the download against
46
+ # @param sum [String] sum to verify the download against or URL to fetch
47
+ # sum from
47
48
  # @param workdir [String] working directory to download into
48
49
  # @param sum_type [String] type of sum we are verifying
49
50
  # @raise [RuntimeError] an exception is raised is sum is nil
50
- def initialize(url, sum:, workdir:, sum_type:, **options)
51
+ def initialize(url, sum:, workdir:, sum_type:, **options) # rubocop:disable Metrics/AbcSize
51
52
  unless sum
52
53
  fail "sum is required to validate the http source"
53
54
  end
@@ -62,6 +63,20 @@ class Vanagon
62
63
  @sum = sum
63
64
  @workdir = workdir
64
65
  @sum_type = sum_type
66
+
67
+ if Vanagon::Component::Source::Http.valid_url?(@sum)
68
+ sum_file = download(@sum)
69
+ File.open(File.join(@workdir, sum_file)) do |file|
70
+ # the sha1 files generated during archive creation are formatted
71
+ # "<sha1sum> <filename>". This will also work for sources that
72
+ # only contain the checksum.
73
+ remote_sum = file.read.split.first
74
+ unless remote_sum
75
+ fail "Downloaded checksum file seems to be empty, make sure you have the correct URL"
76
+ end
77
+ @sum = remote_sum
78
+ end
79
+ end
65
80
  end
66
81
 
67
82
  # Download the source from the url specified. Sets the full path to the
@@ -77,7 +77,7 @@ class Vanagon
77
77
  end
78
78
 
79
79
  def cleanup_workdir
80
- FileUtils.rm_rf(@workdir)
80
+ FileUtils.rm_rf(workdir)
81
81
  end
82
82
 
83
83
  def self.configdir
@@ -115,25 +115,29 @@ class Vanagon
115
115
  raise Vanagon::Error, "Project requires a version set, all is lost."
116
116
  end
117
117
 
118
- @engine.startup(@workdir)
118
+ @engine.startup(workdir)
119
119
 
120
120
  $stderr.puts "Target is #{@engine.target}"
121
121
  retry_task { install_build_dependencies }
122
- retry_task { @project.fetch_sources(@workdir) }
122
+ retry_task { @project.fetch_sources(workdir) }
123
123
 
124
- @project.make_makefile(@workdir)
125
- @project.make_bill_of_materials(@workdir)
126
- @project.generate_packaging_artifacts(@workdir)
124
+ @project.make_makefile(workdir)
125
+ @project.make_bill_of_materials(workdir)
126
+ @project.generate_packaging_artifacts(workdir)
127
127
  @project.save_manifest_json
128
- @engine.ship_workdir(@workdir)
128
+ @engine.ship_workdir(workdir)
129
129
  @engine.dispatch("(cd #{@engine.remote_workdir}; #{@platform.make})")
130
130
  @engine.retrieve_built_artifact
131
131
 
132
- unless @preserve
132
+ if %i[never on-failure].include? @preserve
133
133
  @engine.teardown
134
134
  cleanup_workdir
135
135
  end
136
136
  rescue => e
137
+ if [:never].include? @preserve
138
+ @engine.teardown
139
+ cleanup_workdir
140
+ end
137
141
  $stderr.puts e
138
142
  $stderr.puts e.backtrace.join("\n")
139
143
  raise e
@@ -143,36 +147,17 @@ class Vanagon
143
147
  end
144
148
  end
145
149
 
146
- def render
150
+ def render # rubocop:disable Metrics/AbcSize
147
151
  # Simple sanity check for the project
148
152
  if @project.version.nil? or @project.version.empty?
149
153
  raise Vanagon::Error, "Project requires a version set, all is lost."
150
154
  end
151
155
 
152
156
  $stderr.puts "rendering Makefile"
153
- retry_task { @project.fetch_sources(@workdir) }
154
- @project.make_bill_of_materials(@workdir)
155
- @project.generate_packaging_artifacts(@workdir)
156
- @project.make_makefile(@workdir)
157
- end
158
-
159
- def prepare(workdir = nil) # rubocop:disable Metrics/AbcSize
160
- @workdir = workdir ? FileUtils.mkdir_p(workdir).first : Dir.mktmpdir
161
- @engine.startup(@workdir)
162
-
163
- $stderr.puts "Devkit on #{@engine.target}"
164
-
165
- install_build_dependencies
166
- @project.fetch_sources(@workdir)
167
- @project.make_makefile(@workdir)
168
- @project.make_bill_of_materials(@workdir)
169
- # Builds only the project, skipping packaging into an artifact.
170
- @engine.ship_workdir(@workdir)
171
- @engine.dispatch("(cd #{@engine.remote_workdir}; #{@platform.make} #{@project.name}-project)")
172
- rescue => e
173
- $stderr.puts e
174
- $stderr.puts e.backtrace.join("\n")
175
- raise e
157
+ retry_task { @project.fetch_sources(workdir) }
158
+ @project.make_bill_of_materials(workdir)
159
+ @project.generate_packaging_artifacts(workdir)
160
+ @project.make_makefile(workdir)
176
161
  end
177
162
 
178
163
  # Retry the provided block, use the retry count and timeout
@@ -10,7 +10,7 @@ class Vanagon
10
10
  def initialize(platform, target = nil, **opts)
11
11
  super
12
12
 
13
- @pooler = "http://vmpooler.delivery.puppetlabs.net"
13
+ @available_poolers = ["http://vmpooler.delivery.puppetlabs.net", "https://nspooler-service-prod-1.delivery.puppetlabs.net"]
14
14
  @token = load_token
15
15
  @required_attributes << "vmpooler_template"
16
16
  end
@@ -76,36 +76,58 @@ class Vanagon
76
76
  private :read_vmfloaty_token
77
77
 
78
78
  # This method is used to obtain a vm to build upon using the Puppet Labs'
79
- # vmpooler (https://github.com/puppetlabs/vmpooler)
79
+ # vmpooler (https://github.com/puppetlabs/vmpooler) or other pooler technologies
80
+ # leveraging the same API
80
81
  # @raise [Vanagon::Error] if a target cannot be obtained
81
- def select_target # rubocop:disable Metrics/AbcSize
82
+ def select_target
83
+ @available_poolers.each do |current_pooler|
84
+ @pooler = select_target_from(current_pooler)
85
+ break unless @pooler.empty?
86
+ end
87
+ raise Vanagon::Error, "Something went wrong getting a target vm to build on, maybe the pool for #{build_host_name} is empty?" if @pooler.empty?
88
+ end
89
+
90
+ # Attempt to provision a host from a specific pooler.
91
+ #
92
+ def select_target_from(pooler) # rubocop:disable Metrics/AbcSize
82
93
  response = Vanagon::Utilities.http_request(
83
- "#{@pooler}/vm",
94
+ "#{pooler}/vm",
84
95
  'POST',
85
96
  '{"' + build_host_name + '":"1"}',
86
97
  { 'X-AUTH-TOKEN' => @token }
87
98
  )
88
99
  if response["ok"]
89
- @target = response[build_host_name]['hostname'] + '.' + response['domain']
100
+ @target = response[build_host_name]['hostname']
101
+ # The nspooler does not use 'domain' in the response: 'hostname' just returns the fqdn.
102
+ # in the future we should make the two APIs the same in this sense, but for now, just check
103
+ # if 'domain' is a thing and use it if so.
104
+ if response['domain']
105
+ @target += '.' + response['domain']
106
+ end
90
107
  Vanagon::Driver.logger.info "Reserving #{@target} (#{build_host_name}) [#{@token ? 'token used' : 'no token used'}]"
91
-
92
- tags = {
93
- 'tags' => {
94
- 'jenkins_build_url' => ENV['BUILD_URL'],
95
- 'project' => ENV['JOB_NAME'] || 'vanagon',
96
- 'created_by' => ENV['USER'] || ENV['USERNAME'] || 'unknown'
97
- }
98
- }
99
-
100
- Vanagon::Utilities.http_request(
101
- "#{@pooler}/vm/#{response[build_host_name]['hostname']}",
102
- 'PUT',
103
- tags.to_json,
104
- { 'X-AUTH-TOKEN' => @token }
105
- )
108
+ add_tags_to_target(pooler, response[build_host_name]['hostname'])
106
109
  else
107
- raise Vanagon::Error, "Something went wrong getting a target vm to build on, maybe the pool for #{build_host_name} is empty?"
110
+ pooler = ''
108
111
  end
112
+ pooler
113
+ end
114
+
115
+ # Add tags to a provisioned target using the pooler API
116
+ #
117
+ def add_tags_to_target(pooler, hostname)
118
+ tags = {
119
+ 'tags' => {
120
+ 'jenkins_build_url' => ENV['BUILD_URL'],
121
+ 'project' => ENV['JOB_NAME'] || 'vanagon',
122
+ 'created_by' => ENV['USER'] || ENV['USERNAME'] || 'unknown'
123
+ }
124
+ }
125
+ Vanagon::Utilities.http_request(
126
+ "#{pooler}/vm/#{hostname}",
127
+ 'PUT',
128
+ tags.to_json,
129
+ { 'X-AUTH-TOKEN' => @token }
130
+ )
109
131
  end
110
132
 
111
133
  # This method is used to tell the vmpooler to delete the instance of the