vara 0.17.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 08cc7a49aec754f09f066fc8649926fda5f74ddd
+  data.tar.gz: 0f14ee0ed5fc429451dc1cc868ea45ea0e5d54d5
+SHA512:
+  metadata.gz: 551fee9ff87c07a9ae255d482bb50d4bfe79b816cf6fa6377c5e9f53cb09aa9d0bfff3403b66c16ff647751856ebfa7a719a4118bb6fde770f716d2b997ad69e
+  data.tar.gz: 314fa6195a5f9e20cc5afd2451928f328b39a2adb68ede1b3b17772ebbf7163efa3532b18a786a70e297e2a6296863756c21b1065843c9ee6f4feb602089e76d

data/LEGAL.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2014-2015 Pivotal Software, Inc. All rights reserved.
+
+Unauthorized use, copying or distribution of this source code via any
+medium is strictly prohibited without the express written consent of
+Pivotal Software, Inc.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,297 @@
+# Vara
+
+A gem for building products to be consumed by Ops Manager.
+
+## Complementary tools
+
+While Vara focuses on building a .pivotal from a complete product repository, you may find some of the following tools complement your workflow with Vara:
+
+* [*Vöxtur*](https://github.com/pivotal-cf-experimental/voxtur) to automatically update metadata_parts/binaries.yml from a built release tarball.
+
+## Usage
+
+### Build .pivotal file
+
+In most circumstances, you will only need to run `vara build-pivotal PRODUCT_DIR`, which calls other lower-level vara commands.
+
+    vara build-pivotal ~/workspace/p-runtime
+
+will build a .pivotal file using the stemcell, release tarball, and compiled packages as specified by the product metadata.
+If the stemcell, release tarball, or compiled packages do not exist on disk, they will be downloaded.
+
+The product file will be named `#{product_name}-#{product_version}.pivotal`, where both variables come from the product metadata YAML.
+
+The product file's contents will look like:
+
+```
+cf-1.1.0.0.pivotal
+├── compiled_packages
+│   └── cf-158.1-dev-bosh-vsphere-esxi-ubuntu-1471.2.tgz
+├── content_migrations
+│   └── migrations.yml
+├── metadata
+│   └── cf.yml
+├── releases
+│   └── cf-158.1-dev.tgz
+└── stemcells
+    └── bosh-stemcell-1471_2-vsphere-esxi-ubuntu.tgz
+```
+
+The `content_migrations/migrations.yml` is copied from the product folder's `content_migrations/migrations.yml`.
+The `metadata/cf.yml` file is copied from the product folder's `metadata/cf.yml` if it exists.
+Otherwise the metadata YAML is assembled from `metadata_parts/{handcraft,binaries}.yml`.
+The release file is copied from the product folder's `releases` folder.
+The stemcell file is copied from the product folder's `stemcells` folder.
+
+### The PRODUCT_DIR
+
+Vara expects a product directory that resembles the layout of the .pivotal zip contents.
+
+```
+p-runtime/
+├── .gitignore
+├── compiled_packages
+│   └── .gitkeep
+├── content_migrations
+│   └── .gitkeep
+├── content_migrations_parts
+│   ├── base.yml
+│   └── migrations
+│       ├── from_1.1.0.0.yml
+│       └── from_1.2.0.0.yml
+├── metadata
+│   └── .gitkeep
+├── metadata_parts
+│   ├── binaries.yml
+│   └── handcraft.yml
+├── releases
+│   └── .gitkeep
+└── stemcells
+    └── .gitkeep
+```
+
+The contents of `compiled_packages`, `releases`, and `stemcells` are large tarballs.
+Large binary files are discouraged from being checked into git and other source control systems.
+To work around this limitation, vara examines the metadata YAML and downloads these files from a URL if necessary.
+
+The `compiled_packages`, `releases`, and `stemcells` directories are populated during the `download-artifacts` step of building a .pivotal.
+The `metadata` directory is populated during the `build-metadata` step.
+The `content_migrations` directory is populated during the `build-migrations` step.
+
+The .gitignore file should include these entries to prevent large binary files from being committed by mistake.
+These entries also prevent vara from failing various operations due to a dirty git working tree.
+
+```
+releases/
+stemcells/
+compiled_packages/
+*.pivotal
+*.pivotal.yml
+*.pivotal.md5
+```
+
+### Re-generate Metadata .yml
+
+#### Using metadata_parts
+
+To only build the product metadata file from the two separate metadata_parts files:
+
+	vara build-metadata ~/workspace/p-runtime
+
+This will create a `metadata/${PRODUCT_NAME}.yml` file composed of the two files located in `${product_dir}/metadata_parts`:
+
+* binaries.yml (includes stemcells, release, and optional compiled_packages information)
+* handcraft.yml (includes everything else, including product name (e.g. *cf*), product_version (e.g. *1.2.0.0*)
+
+(where PRODUCT_NAME is derived from the product name entry in handcraft.yml)
+
+*[These files were split in order to maintain the comments and the formatting of contained in the handcraft.yml file while allowing the information contained in binaries.yml to be updated by utilities such as `vara-update-metadata`]
+
+Include the following entry in .gitignore if vara dynamically generates the metadata .yml.
+
+```
+metadata/*.yml
+```
+
+#### Automatically increasing a product's prerelease version
+
+You can set the `product_version` to something like `1.2.3.4$PRERELEASE_VERSION$` and during metadata generation, vara will expand that into something like `1.2.3.4.alpha.88.dedbeef` where:
+
+* *alpha* is the "cycle" as specified by the --cycle flag to `vara build-metadata`, which defaults to alpha
+* *88* is the number of commits in the current branch
+* *dedbeef* is the short commit hash of the current revision on the current branch
+
+#### Specifying a product version
+
+You can set the `product_version` to something specific like `1.2.3-build.1` or `1.2.3` by specifying the `--version` flag, e.g.:
+
+```
+vara build-metadata --version=1.2.3-build.1 .
+vara build-migrations --version=1.2.3 .
+vara build-pivotal --version=1.2.3-rc.3 .
+```
+
+#### Specifying a final version
+
+You can remove the `$PRERELEASE_VERSION$` from the product version by using the `--final` flag. So for a product version that is in
+`handcraft.yml` as `1.2.3.4$PRERELEASE_VERSION$` running
+
+```
+vara build-pivotal . --final
+vara build-metadata . --final
+vara build-migrations . --final
+```
+
+would result in a product version of `1.2.3.4`.
+
+#### Specifying an external release
+
+You can update multiple releases when building the pivotal by passing one or more external release files using the `--external-releases` flag.
+
+```
+vara build-pivotal . --external-releases=some-release.yml
+vara build-pivotal . --external-release=some-release.yml,another-release.yml
+```
+
+An external release file contains an array of manifest snippets that replace entries in `metadata_parts/binaries.yml`. For example a sample external release file could be the following. This would update the `cf` entry in binaries.yml when running the pivotal.
+
+```
+- file: cf-214.tgz
+  name: cf
+  version: '214'
+  md5: 20cd54b93c23a7fb232d5c25b2082666
+  url: http://bosh.io/d/github.com/cloudfoundry/cf-release?v=214
+```
+
+The flag will use the specified files to replace the entries that matches the `name` property of an already existing release inside of the `binaries.yml` file.
+
+### Lint product metadata
+
+#### Syntax
+
+`vara lint <Product-Metadata>`
+
+#### Description
+
+Running `vara lint ~/workspace/p-runtime/cf.yml` will inspect the product template for malformed data.
+`vara build-pivotal` executes `vara lint` before running `vara zip-pivotal`, therefore if malformed data exists,
+the resulting .pivotal file will not be created.
+
+### Re-generate content migrations
+
+To only compose the content migrations file from the files in the `content_migrations_parts` directory:
+
+    vara build-migrations ~/workspace/p-runtime
+
+This will create a `content_migrations/migrations.yml` file composed of the files in `${product_dir}/content_migrations_parts`:
+
+* base.yml
+  * Typically includes the `product`, `installation_version`, and `to_version` keys
+  * Optionally includes the `migrations` key, which is allowed to have any number of entries underneath it
+* migrations/from_1.2.3.4.yml
+  * Can be named anything you want with a .yml extension, but by convention is `from_${version}.yml`
+  * Content is the hash of a single entry to be inserted under the migrations key (e.g. has `from_version` and `rules` as top-level keys)
+
+A migration file in the `migrations` folder may have a `base.yml` file like:
+
+```
+product: example-product
+installation_version: "1.1"
+to_version: "1.2.0.0$PRERELEASE_VERSION$"
+```
+
+and a rule entry like:
+
+```
+from_version: 1.1.0.0
+rules:
+- type: update
+  selector: "product_version"
+  to: "1.2.0.0$PRERELEASE_VERSION$"
+```
+
+and then `$PRERELEASE_VERSION$` will be replaced in both places using the same rules as above in the metadata file.
+
+Include the following entry in .gitignore if vara dynamically generates the metadata .yml.
+
+```
+content_migrations/*.yml
+```
+
+#### A Warning on Auto-Generated Prerelease versions
+
+Ops Manager does not allow wildcards in the `from_version` field of migrations.
+Therefore, if you have deployed a prerelease version, you will not be able to upgrade to another prerelease version unless you explicitly write a migration between the two.
+We expect that the normal workflow will not include upgrading between prereleases.
+
+Any prerelease products published externally will require an additional stanza in future migrations.yml for upgradability.
+
+### Download missing artifacts
+
+Running `vara download-artifacts ~/workspace/p-runtime/cf.yml` will download any missing compiled_packages, releases, and stemcells, inserting them into the respective directories.
+For stemcells, vara assumes that the stemcell is a public bosh stemcell and infers the URL based on the `file` attribute of the stemcell hash.
+For releases and compiled packages, vara uses the `url` attribute of the respective hashes in the product metadata.
+
+For example:
+
+```
+releases:
+- file: cf-172.tgz
+  name: cf
+  version: '172'
+  md5: 777fe352515612841a3d96af12054947
+  url: https://example.s3.amazonaws.com/cf-172.tgz
+```
+
+#### Checksum Validation
+
+Artifacts can use either `MD5` or `SHA1` checksums for release and stemcell validation. For example:
+
+```
+releases:
+- file: cf-172.tgz
+  name: cf
+  version: '172'
+  md5: 777fe352515612841a3d96af12054947
+  sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
+  url: https://example.s3.amazonaws.com/cf-172.tgz
+```
+
+You must specify at least one checksum.
+
+#### AWS S3 support
+
+Release artifacts can be downloaded from an S3 bucket. For example:
+
+```
+releases:
+- file: cf-172.tgz
+  name: cf
+  version: '172'
+  md5: 777fe352515612841a3d96af12054947
+  sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
+  aws:
+    access_key_id: <some-access-key>
+    secret_access_key: <some-secret-key>
+    region: us-west-1
+    bucket_name: <some-bucket>
+    filename: path/to/s3/object.tgz
+```
+
+You should only specify `url` or `aws`, not both.
+
+### Zipping the .pivotal
+
+If your metadata file has been generated and your artifacts have been downloaded, running `vara zip-pivotal ~/workspace/p-runtime` will put all of those contents into a new .pivotal in your working directory.
+
+
+## LEGAL
+
+Copyright (c) 2014-2015 Pivotal Software, Inc.
+All rights reserved.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bin/vara

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+# Copyright (c) 2014-2015 Pivotal Software, Inc.
+# All rights reserved.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+# USE OR OTHER DEALINGS IN THE SOFTWARE.
+require 'vara/cli'
+require 'vara/cli_helper'
+
+Vara::Cli.start(ARGV)

data/lib/vara.rb

@@ -0,0 +1,14 @@
+require 'vara/version'
+
+module Vara
+  # Your code goes here...
+end
+
+# Copyright (c) 2014-2015 Pivotal Software, Inc.
+# All rights reserved.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+# USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/vara/cli.rb

@@ -0,0 +1,96 @@
+require 'thor'
+
+module Vara
+  class Cli < Thor
+    def self.exit_on_failure?
+      true
+    end
+
+    desc 'build-metadata PRODUCT_DIR', 'Compose metadata from metadata_parts'
+    option 'cycle', default: 'alpha'
+    option 'final', type: :boolean, desc: 'When true, $PRERELEASE_VERSION$ is removed instead of substituted'
+    option 'version', type: :string, desc: 'When given, use the version as specified'
+    option 'external_releases', desc: 'Comma delimited paths to release metadata files to use when generating the .pivotal'
+    def build_metadata(product_dir)
+      require 'vara/prerelease_versioner'
+      require 'vara/static_versioner'
+      require 'vara/product_metadata_processor'
+      versioner = Vara::PrereleaseVersioner.new(File.expand_path(product_dir), options.fetch('cycle'))
+      versioner.override_prerelease_version!('') if options['final']
+      if options['version'] && !options['version'].empty?
+        versioner = Vara::StaticVersioner.new(File.expand_path(product_dir), options.fetch('version'))
+      end
+      metadata_path = Vara::ProductMetadataProcessor.new(
+        File.expand_path(product_dir),
+        versioner,
+        external_release_paths: options.fetch('external_releases', '').split(',')
+      ).process
+      say("#{metadata_path} has been created")
+      metadata_path
+    end
+
+    desc 'build-migrations PRODUCT_DIR', 'Compose content_migrations from content_migrations_parts'
+    option 'cycle', default: 'alpha'
+    option 'final', type: :boolean, desc: 'When true, $PRERELEASE_VERSION$ is removed instead of substituted'
+    option 'version', type: :string, desc: 'When given, use the version as specified'
+    def build_migrations(product_dir)
+      require 'vara/prerelease_versioner'
+      require 'vara/static_versioner'
+      require 'vara/content_migrations_processor'
+      versioner = Vara::PrereleaseVersioner.new(File.expand_path(product_dir), options.fetch('cycle'))
+      versioner.override_prerelease_version!('') if options['final']
+      if options['version'] && !options['version'].empty?
+        versioner = Vara::StaticVersioner.new(File.expand_path(product_dir), options.fetch('version'))
+      end
+      content_migrations_path = Vara::ContentMigrationsProcessor.new(File.expand_path(product_dir), versioner).process
+      say("#{content_migrations_path} has been created")
+      content_migrations_path
+    end
+
+    desc 'lint PRODUCT_METADATA', 'Lints the product metadata file'
+    def lint(product_metadata)
+      require 'vara/linter'
+      linter = Vara::Linter.build(File.expand_path(product_metadata))
+      linter.lint!
+    end
+
+    desc 'download-artifacts PRODUCT_METADATA', 'Download the artifacts from the product metadata file'
+    def download_artifacts(product_metadata)
+      require 'vara/product_resource_downloader'
+      product_resource_downloader = Vara::ProductResourceDownloader.build(File.expand_path(product_metadata))
+      product_resource_downloader.download
+    end
+
+    desc 'zip-pivotal PRODUCT_DIR', 'Zips up the .pivotal product given that all artifacts are synced locally'
+    def zip_pivotal(product_dir)
+      require 'vara/product'
+      product = Vara::Product.new(File.expand_path(product_dir))
+      product.build
+      say("file #{product.path} has been created")
+      say("file #{product.bom_path} has been created")
+      say("file #{product.md5_path} has been created")
+    end
+
+    desc 'build-pivotal PRODUCT_DIR', 'Generate a .pivotal from a vara-flavored product repository'
+    option 'final', type: :boolean, desc: 'When true, $PRERELEASE_VERSION$ is removed instead of substituted'
+    option 'version', type: :string, desc: 'When given, use the version as specified'
+    option 'external_releases', desc: 'Comma delimited paths to release metadata files to use when generating the .pivotal'
+    option 'cycle', default: 'alpha'
+    def build_pivotal(product_dir)
+      metadata_path = build_metadata(product_dir)
+      build_migrations(product_dir)
+      lint(metadata_path)
+      download_artifacts(metadata_path)
+      zip_pivotal(product_dir)
+    end
+  end
+end
+
+# Copyright (c) 2014-2015 Pivotal Software, Inc.
+# All rights reserved.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+# USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/vara/cli_helper.rb

@@ -0,0 +1,12 @@
+require 'vara/log'
+
+Vara::Log.stdout_mode!
+
+# Copyright (c) 2014-2015 Pivotal Software, Inc.
+# All rights reserved.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+# USE OR OTHER DEALINGS IN THE SOFTWARE.