concourse-deployer 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e35d5f547ba50ab86ab55fa4e7767cf17ac17e051e7fc2908f192e5633528da2
+  data.tar.gz: bfd4742437ff9911aca9b9d7302a3df8ebb3c2e8cf83651120ad779da8ac2c7d
+SHA512:
+  metadata.gz: ff1ce32001402f065b54fe410a0f690b729a6d1a4e5ca1a73065c7daadbde7e913c040b462556b1481407781d10aeb3986d342276e623327dce915cf645713db
+  data.tar.gz: 24a6f65a6bd1d0facd44948f89a88c5df47b3969ab6f36e68e897883334b5e18dc48739acf8591e73a3b5711e85348c1b871b5fd811e83c5a1982a0d197548c0

data/.gitignore

@@ -0,0 +1,12 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+
+# rspec failure tracking
+.rspec_status

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog for `concourse-deployer`
+
+## v0.1.0 / 2019-01-04
+
+First release.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at mike.dalessio@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in concourse-deployer.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Mike Dalessio
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,306 @@
+# Concourse::Deployer
+
+Provides easy installation and maintenance of an opinionated [Concourse](https://concourse-ci.org) deployment.
+
+- external Postgres database
+- Github auth integration
+- LetsEncrypt integration for SSL cert management
+- Windows™ workers
+
+Today this only supports deployment to GCP.
+
+
+## TL;DR
+
+These five commands will give you a full Concourse deployment, with user-friendly prompting for configuration to external resources like a Postgres database and Github auth.
+
+``` sh
+rake bbl:gcp:init[GCP_PROJECT_ID]
+rake bbl:gcp:up
+rake bosh:init
+rake bosh:update
+rake bosh:deploy
+```
+
+You can create and deploy a LetsEncrypt SSL cert:
+
+``` sh
+rake letsencrypt:create letsencrypt:backup letsencrypt:import
+rake bosh:deploy
+```
+
+## Requirements
+
+This gem requires:
+
+* `bbl` ~> 6.9.0 (https://github.com/cloudfoundry/bosh-bootloader/releases)
+* `bosh` ~> 5.2.0 (https://github.com/cloudfoundry/bosh-cli/releases)
+* `terraform` (https://www.terraform.io/downloads.html)
+* `gcloud` (https://cloud.google.com/sdk/downloads)
+* `direnv` (https://direnv.net/)
+* `git-crypt` (https://www.agwa.name/projects/git-crypt/)
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'concourse-deployer'
+```
+
+And then run `bundle` or else install it directly with `gem install concourse-deployer`.
+
+
+## Usage
+
+In your Rakefile:
+
+``` ruby
+require "concourse/deployer"
+
+Concourse::Deployer.new.create_tasks!
+```
+
+Available tasks:
+
+``` sh
+rake bbl:gcp:init[gcp_project_id]  # initialize bosh-bootloader for GCP
+rake bbl:gcp:up                    # terraform your environment and deploy the bosh director
+rake bosh:deploy                   # deploy concourse
+rake bosh:init                     # prepare the concourse bosh deployment
+rake bosh:update                   # upload stemcells and releases to the director
+rake bosh:update:ubuntu_stemcell   # upload ubuntu stemcell to the director
+rake letsencrypt:backup            # backup web:/etc/letsencrypt to local disk
+rake letsencrypt:create            # create a cert
+rake letsencrypt:import            # import letsencrypt keys into `secrets.yml` from backup
+rake letsencrypt:renew             # renew the certificate
+rake letsencrypt:restore           # restore web:/etc/letsencrypt from backup
+```
+
+See full instructions below.
+
+
+## A Note on Security
+
+It's incredibly important that you don't risk leaking your credentials by committing them in the clear to a public git repository. This gem will use `git-crypt` to ensure files are encrypted which contain sensitive credentials.
+
+Files which contain sensitive data:
+
+* `service-account.key.json`
+* `bbl-state.json`
+* `secrets.yml`
+* `cluster-creds.yml`
+* the `vars` subdirectory
+* `letsencrypt.tar.gz` (if you're using the letsencrypt SSL cert functionality)
+
+You will see these files listed in `.gitattributes` invoking git-crypt for them.
+
+
+## Deploying to GCP
+
+### Step 0: create a GCP project, and create and config a Postgres database
+
+Spin up a postgres database. Note the following information as you do so:
+
+* password
+* IP address
+
+To set up connectivity to it, we'll first create client SSL certs, then only allow access via SSL, and finally allow inbound connections from any source IP (so long as it's via SSL).
+
+1. Under "SSL", create a client SSL cert, and download `client-key.pem`, `client-cert.pem`, and `server-ca.pem` for later use.
+2. Click "Allow only secured connections"
+3. Under "Authorization", add "0.0.0.0/0" as an allowed network.
+4. Under "Databases", create a database named `atc`.
+
+Using an external db is a little annoying to do, but in the opinion of the author, it's worth it to have state persisted outside of the bosh-administered cluster, so that it can be torn down and rebuilt easily when necessary.
+
+
+### Step 1: Initialize bosh-bootloader and the project directory
+
+``` sh
+$ rake bbl:gcp:init[gcp_project_id]
+```
+
+This will:
+
+* check that required dependencies are installed,
+* create an `.envrc` file with environment variables for bbl to work with GCP,
+* create `.gitattributes` entries to prevent sensitive files from being committed in the clear,
+* create a GCP service account, associate it with your project, and give it the necessary permissions,
+* and save that GCP service account information in `service-account.key.json`
+
+__NOTE:__ At this point, if you want to use a region/zone besides us-central1, you can edit your `.envrc`.
+
+__NOTE:__ `service-account.key.json` contains sensitive data.
+
+
+### Step 2: `bbl up`
+
+``` sh
+$ rake bbl:gcp:up
+```
+
+Go get a coffee. In about 5 minutes, this will:
+
+* terraform a GCP environment,
+* spin up VMs running a bosh director and a jumpbox (a.k.a. "bastion host"),
+* create a load balancer with an external IP,
+* and save your state and credentials into `bbl-state.json` and the `vars` subdirectory.
+
+__NOTE:__ This task is idempotent. If you want to upgrade your bosh director (or stemcell) using a future version of bbl, you can re-run this (but read the bbl upgrade notes first).
+
+__NOTE:__ `bbl-state.json` and `vars` contain sensitive data.
+
+
+### Step 3: Prepare the Bosh deployment for Concourse
+
+``` sh
+$ rake bosh:init
+```
+
+This will:
+
+* create a git submodule with a clone of [`concourse-bosh-deployment`](https://github.com/concourse/concourse-bosh-deployment)
+* create a `secrets.yml` file with credentials and external configuration you'll use to access concourse
+
+You may be prompted for several things at this step, including:
+
+* database IP and password (noted in Step 0 above)
+* database server CA, client cert, and client key filenames (downloaded in Step 0 above)
+* Github OAuth2 credentials
+
+
+__NOTE:__ `secrets.yml` contains sensitive data.
+
+__NOTE:__ This task is idempotent! You can re-run this whenever you like.
+
+__NOTE:__ If you'd like a github user, team, or org to be members of Concourse's admin team, edit `secrets.yml` and add them to the `/main_team` section.
+
+
+### Step 4: Upload stemcell to the director
+
+``` sh
+$ rake bosh:update
+```
+
+This will:
+
+* upload to the director the latest GCP stemcell
+* upload all necessary Bosh releases
+
+__NOTE:__ This task is idempotent! If you want to upgrade your releases or stemcell in the future, you should re-run this.
+
+
+### Step 5: deploy!
+
+``` sh
+$ rake bosh:deploy
+```
+
+This will:
+
+* create or update a `cluster-creds.yml` file with automatically-generated cluster credentials,
+* bosh-deploy Concourse
+
+__NOTE:__ `cluster-creds.yml` and `secrets.yml` contain sensitive data.
+
+__NOTE:__ This task is idempotent! Yay Bosh.
+
+
+## Other Fun Things This Gem Does
+
+### Scale your Concourse deployment
+
+Your first deployment will spin up one (1) web VM, and two (2) Linux worker VMs. But you can scale these numbers up as needed by editing the file `scale-vars.yml`, whose default contents looks like:
+
+```yaml
+---
+web_instances: 1
+worker_instances: 2
+```
+
+Edit this file as appropriate for your needs, and re-run `rake bosh:deploy`.
+
+
+### Manage your letsencrypt SSL cert
+
+``` sh
+$ rake letsencrypt:backup
+$ rake letsencrypt:create
+$ rake letsencrypt:restore
+$ rake letsencrypt:import
+$ rake letsencrypt:renew
+```
+
+__NOTE:__ These tasks will create and use `letsencrypt.tar.gz` which contains sensitive data.
+
+
+### Custom bosh ops files
+
+If you want to perform any custom operations on the manifest, put them in a file named `operations.yml` and they'll be pulled in as the __final__ ops file during deployment.
+
+
+## Upgrading `bbl`
+
+When a new version of bosh-bootloader comes out, just [download it](https://github.com/cloudfoundry/bosh-bootloader/releases) and make sure it's in your path as `bbl` (check by running `bbl -v`) and then:
+
+``` sh
+$ rake bbl:gcp:up
+```
+
+... which will generate a new plan and then update the jumpbox, director, and cloud config. (See https://github.com/cloudfoundry/bosh-bootloader/blob/master/docs/upgrade.md for details.)
+
+Make sure to commit into source control all the changes in your project directory (`bbl-state.json`, `vars/`, `bosh-deployment/`, etc.).
+
+
+## Upgrading `concourse-bosh-deployment`
+
+If a new version of concourse comes out, and you'd like to upgrade, first read the [release notes for Concourse](https://concourse-ci.org/download.html) to check for any relevant breaking changes.
+
+Then:
+
+``` sh
+$ rake bosh:update:concourse_deployment
+$ rake bosh:deploy
+```
+
+Make sure you commit to source control the updated git submodule.
+
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/flavorjones/concourse-deployer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+
+## TODO
+
+- [ ] update windows stemcell
+- [ ] include windows worker in manifest
+- [ ] deploy windows ruby tools release to the windows vms
+- [x] +      x_frame_options: "SAMEORIGIN"
+- [x] +      container_placement_strategy: random
+- [ ] enable encryption https://concourse.ci/encryption.html
+- [ ] allow scaling up/down by locally setting number of VMs (currently hardcoded in gem)
+- [ ] start using https://github.com/dpb587/caddy-bosh-release instead of the letsencrypt rake tasks
+
+
+Things to follow up on:
+
+- [x] upgrading! ZOMG
+- [ ] consider swapping secrets-wizarding and rake task for deploy for a shell script that's user-modifiable
+- [ ] bbl feature for suspending/unsuspending the director VM?
+- [ ] stack driver add-on?
+- [ ] metrics? https://concourse-ci.org/metrics.html
+- [ ] credhub for credential management? https://concourse-ci.org/creds.html
+
+
+Things I'm not immediately planning to do but that might be nice:
+
+- [ ] ops file to make the cloud-config come in under default GCP quota
+- [ ] ops files for a few variations on size/cost tradeoffs

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "concourse/deployer"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/concourse-deployer.gemspec

@@ -0,0 +1,30 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'concourse/deployer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "concourse-deployer"
+  spec.version       = Concourse::Deployer::VERSION
+  spec.authors       = ["Mike Dalessio"]
+  spec.email         = ["mike.dalessio@gmail.com"]
+
+  spec.summary       = %q{Rake tasks to help BOSH-deploy a Concourse CI environment.}
+  spec.description   = %q{concourse-deployer provides an ease-of-use layer on top of bosh-bootloader and bosh, to ease the initial install process and maintenance.}
+  spec.homepage      = "https://github.com/flavorjones/concourse-deployer"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "term-ansicolor"
+  spec.add_dependency "nokogiri"
+
+  spec.add_development_dependency "bundler", "~> 1.14"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+end

data/lib/concourse/deployer.rb

@@ -0,0 +1,363 @@
+require "concourse/deployer/version"
+require "concourse/deployer/utils"
+require "erb"
+require "open-uri"
+require "nokogiri"
+require "yaml"
+require "rake"
+
+module Concourse
+  class Deployer
+    include Rake::DSL
+    include Concourse::Deployer::Utils
+
+    GCP_SERVICE_ACCOUNT_FILE = "service-account.key.json"
+    ENVRC_FILE               = ".envrc"
+
+    BBL_STATE_FILE           = "bbl-state.json"
+    BBL_VARS_DIR             = "vars"
+
+    BOSH_DEPLOYMENT          = "concourse"
+    BOSH_SECRETS             = "secrets.yml"
+    BOSH_VARS_STORE          = "cluster-creds.yml"
+    BOSH_OPERATIONS          = "operations.yml"
+
+    CONCOURSE_SCALE_VARS     = "scale-vars.yml"
+
+    LETSENCRYPT_BACKUP_FILE  = "letsencrypt.tar.gz"
+
+    def bbl_init
+      unless_which "bbl", "https://github.com/cloudfoundry/bosh-bootloader/releases"
+      unless_which "bosh", "https://github.com/cloudfoundry/bosh-cli/releases"
+      unless_which "terraform", "https://www.terraform.io/downloads.html"
+    end
+
+    def bbl_gcp_prompt_for_service_account
+      return true unless File.exist?(GCP_SERVICE_ACCOUNT_FILE)
+
+      overwrite = prompt "A #{GCP_SERVICE_ACCOUNT_FILE} file already exists. Do you want to overwrite it? (y/n)", "n"
+      return !! (overwrite =~ /^y/i)
+    end
+
+    def bbl_gcp_init project_id
+      bbl_init
+      unless_which "gcloud", "https://cloud.google.com/sdk/downloads"
+      ensure_in_gitcrypt GCP_SERVICE_ACCOUNT_FILE
+      ensure_in_envrc "BBL_GCP_PROJECT_ID", project_id
+      ensure_in_envrc "BBL_IAAS", "gcp"
+      ensure_in_envrc "BBL_GCP_REGION", "us-central1"
+      ensure_in_envrc "BBL_GCP_SERVICE_ACCOUNT_KEY", GCP_SERVICE_ACCOUNT_FILE
+
+      if bbl_gcp_prompt_for_service_account
+        service_account_name = "concourse-bbl-service-account"
+
+        sh %Q{gcloud --project=#{project_id} iam service-accounts create '#{service_account_name}'}
+        sh %Q{gcloud --project=#{project_id} iam service-accounts keys create '#{GCP_SERVICE_ACCOUNT_FILE}' --iam-account '#{service_account_name}@#{project_id}.iam.gserviceaccount.com'}
+        sh %Q{gcloud projects add-iam-policy-binding '#{project_id}' --member 'serviceAccount:#{service_account_name}@#{project_id}.iam.gserviceaccount.com' --role 'roles/editor'}
+      end
+    end
+
+    def bbl_gcp_up
+      unless ENV['BBL_GCP_PROJECT_ID']
+        error "Environment variable BBL_GCP_PROJECT_ID is not set. Did you run `rake bbl:gcp:init` and `direnv allow`?"
+      end
+
+      ensure_in_gitcrypt BBL_STATE_FILE
+      ensure_in_gitcrypt "#{BBL_VARS_DIR}/*"
+      ensure_in_envrc 'eval "$(bbl print-env)"'
+
+      note ""
+      note "running `bbl up` on GCP ... go get a coffee."
+      note "(If you get an error about 'Access Not Configured', follow the URL in the error message and enable API access for your project!)"
+      note ""
+      sh "bbl plan --lb-type concourse"
+      sh "bbl up --lb-type concourse"
+    end
+
+    def bosh_init
+      ensure_git_submodule "https://github.com/concourse/concourse-bosh-deployment", "master"
+      ensure_in_gitcrypt BOSH_SECRETS
+      ensure_in_envrc "BOSH_DEPLOYMENT", BOSH_DEPLOYMENT
+
+      bosh_secrets do |v|
+        v["local_user"] = (v["local_user"] || {}).tap do |local_user|
+          local_user["username"] = "concourse"
+          local_user["password"] ||= if which "apg"
+                                       `apg -n1`.strip
+                                     else
+                                       prompt "Please enter a password"
+                                     end
+        end
+
+        v["external_dns_name"] ||= prompt("Please enter a DNS name if you have one", bbl_external_ip)
+
+        v["postgres_host"] ||= prompt("External postgres host IP")
+        v["postgres_port"] ||= prompt("External postgres port", 5432)
+        v["postgres_role"] ||= prompt("External postgres role", "postgres")
+        v["postgres_password"] ||= prompt("External postgres password")
+
+        v["postgres_client_cert"] = (v["postgres_client_cert"] || {}).tap do |cert|
+          cert["certificate"] ||= prompt_for_file_contents "Path to client-cert.pem"
+          cert["private_key"] ||= prompt_for_file_contents "Path to client-key.pem" 
+        end
+        v["postgres_ca_cert"] = (v["postgres_ca_cert"] || {}).tap do |cert|
+          cert["certificate"] ||= prompt_for_file_contents "Path to server-ca.pem"
+        end
+
+        if v["github_client"].nil?
+          if prompt("Would you like to configure a github oauth2 application", "n") =~ /^y/i
+            v["github_client"] = {}.tap do |gc|
+              gc["username"] = prompt "Github Client ID"
+              gc["password"] = prompt "Github Client Secret"
+            end
+            v["main_team"] ||= {}.tap do |mt|
+              mt["github_users"] ||= []
+              mt["github_orgs"] ||= []
+              mt["github_teams"] ||= []
+            end
+          end
+        end
+      end
+    end
+
+    def bosh_update_concourse_deployment
+      update_git_submodule "https://github.com/concourse/concourse-bosh-deployment", "master"
+    end
+
+    def bosh_update_ubuntu_stemcell
+      bosh_update_stemcell "bosh-google-kvm-ubuntu-xenial-go_agent"
+    end
+
+    # def bosh_update_windows_stemcell
+    #   bosh_update_stemcell "bosh-google-kvm-windows2012R2-go_agent"
+    # end
+
+    # def bosh_update_concourse_windows_release
+    #   # bosh_update_from_git_repo "https://github.com/pivotal-cf-experimental/concourse-windows-release"
+    #   bosh_update_release "pivotal-cf-experimental/concourse-windows-worker-release"
+    # end
+
+    # def bosh_update_windows_ruby_dev_tools
+    #   # bosh_update_from_git_repo "https://github.com/flavorjones/windows-ruby-dev-tools-release"
+    #   bosh_update_release "flavorjones/windows-ruby-dev-tools-release"
+    # end
+
+    # def bosh_update_windows_utilities_release
+    #   bosh_update_release "cloudfoundry-incubator/windows-utilities-release"
+    # end
+
+    def bosh_deploy
+      unless File.exists?(BOSH_SECRETS)
+        error "File #{BOSH_SECRETS} does not exist. Please run `rake bosh:init` first."
+      end
+
+      ensure_in_gitcrypt BOSH_SECRETS
+      ensure_in_gitcrypt BOSH_VARS_STORE
+
+      ensure_file CONCOURSE_SCALE_VARS do |f|
+        f.write({"web_instances" => 1, "worker_instances" => 2}.to_yaml)
+      end
+
+      external_dns_name = bosh_secrets['external_dns_name']
+      external_url = "https://#{external_dns_name}"
+
+      # command will be run in the bosh deployment submodule's cluster directory
+      command = [].tap do |c|
+        c << "bosh deploy concourse.yml"
+        # c << "--no-redact" # DEBUG
+        c << "-l ../versions.yml"
+        c << "-l ../../#{BOSH_SECRETS}"
+        c << "--vars-store ../../#{BOSH_VARS_STORE}"
+        c << "-o operations/basic-auth.yml"
+        c << "-o operations/privileged-http.yml"
+        c << "-o operations/privileged-https.yml"
+        c << "-o operations/tls.yml"
+        c << "-o operations/tls-vars.yml"
+        c << "-o operations/web-network-extension.yml"
+        c << "-o operations/external-postgres.yml"
+        c << "-o operations/external-postgres-tls.yml"
+        c << "-o operations/external-postgres-client-cert.yml"
+        c << "-o operations/worker-ephemeral-disk.yml"
+        c << "-o operations/x-frame-options-sameorigin.yml"
+        c << "-o operations/container-placement-strategy-random.yml"
+        c << "-o operations/scale.yml"
+        c << "-o ../../#{BOSH_OPERATIONS}" if File.exists?(BOSH_OPERATIONS)
+        c << "-o operations/github-auth.yml" if bosh_secrets["github_client"]
+        c << "--var network_name=default"
+        c << "--var external_host='#{external_dns_name}'"
+        c << "--var external_url='#{external_url}'"
+        c << "--var web_vm_type=default"
+        c << "--var worker_vm_type=default"
+        c << "--var worker_ephemeral_disk=50GB_ephemeral_disk"
+        c << "--var deployment_name=#{BOSH_DEPLOYMENT}"
+        c << "--var web_network_name=private"
+        c << "--var web_network_vm_extension=lb"
+        c << "-l ../../#{CONCOURSE_SCALE_VARS}"
+      end.join(" ")
+
+      Dir.chdir("concourse-bosh-deployment/cluster") do
+        sh command
+      end
+    end
+
+    def letsencrypt_create
+      external_dns_name = bosh_secrets['external_dns_name']
+      if external_dns_name == bbl_external_ip
+        error "Please set your external DNS name in #{BOSH_SECRETS}"
+      end
+
+      sh "bosh ssh web -c 'sudo chmod 777 /tmp'"
+      sh "bosh ssh web -c 'sudo add-apt-repository -y ppa:certbot/certbot'"
+      sh "bosh ssh web -c 'sudo apt-get update'"
+      sh "bosh ssh web -c 'sudo apt-get install -y certbot'"
+      begin
+        sh "bosh stop web"
+        note "logging you into the web server. run this command: sudo certbot certonly --standalone -d \"#{external_dns_name}\""
+        sh "bosh ssh web"
+      ensure
+        sh "bosh start web"
+      end
+    end
+
+    def letsencrypt_backup
+      ensure_in_gitcrypt LETSENCRYPT_BACKUP_FILE
+      sh %Q{bosh ssh web -c 'sudo tar -zcvf /var/tmp/#{LETSENCRYPT_BACKUP_FILE} -C /etc letsencrypt'}
+      sh %Q{bosh scp web:/var/tmp/#{LETSENCRYPT_BACKUP_FILE} .}
+    end
+
+    def letsencrypt_import
+      ensure_in_gitcrypt LETSENCRYPT_BACKUP_FILE
+      external_dns_name = bosh_secrets['external_dns_name']
+
+      begin
+        sh "tar -zxf #{LETSENCRYPT_BACKUP_FILE}"
+        note "importing certificate and private key for #{external_dns_name} ..."
+        bosh_secrets do |v|
+          v["atc_tls"] ||= {}
+          v["atc_tls"]["certificate"] = File.read "letsencrypt/live/#{external_dns_name}/fullchain.pem"
+          v["atc_tls"]["private_key"] = File.read "letsencrypt/live/#{external_dns_name}/privkey.pem"
+        end
+      ensure
+        sh "rm -rf letsencrypt"
+      end
+    end
+
+    def letsencrypt_restore
+      ensure_in_gitcrypt LETSENCRYPT_BACKUP_FILE
+      sh "bosh ssh web -c 'sudo rm -rf /etc/letsencrypt /var/tmp/#{LETSENCRYPT_BACKUP_FILE}'"
+      sh "bosh scp #{LETSENCRYPT_BACKUP_FILE} web:/var/tmp"
+      sh "bosh ssh web -c 'sudo tar -zxvf /var/tmp/#{LETSENCRYPT_BACKUP_FILE} -C /etc'"
+      sh "bosh ssh web -c 'sudo chown -R root:root /etc/letsencrypt'"
+    end
+
+    def letsencrypt_renew
+      sh "bosh ssh web -c 'sudo chmod 1777 /tmp'" # see https://github.com/cloudfoundry/bosh-linux-stemcell-builder/issues/39
+      sh "bosh ssh web -c 'sudo add-apt-repository -y ppa:certbot/certbot'"
+      sh "bosh ssh web -c 'sudo apt-get update'"
+      sh "bosh ssh web -c 'sudo apt-get install -y certbot'"
+      begin
+        sh "bosh stop web"
+        sh "bosh ssh web -c 'sudo certbot renew'"
+      ensure
+        sh "bosh start web"
+      end
+    end
+
+    def create_tasks!
+      namespace "bbl" do
+        namespace "gcp" do
+          desc "initialize bosh-bootloader for GCP"
+          task "init", ["gcp_project_id"] do |t, args|
+            gcp_project_id = args["gcp_project_id"]
+            unless gcp_project_id
+              error "You must specify an existing GCP project id, like `rake #{t.name}[unique-project-name]`"
+            end
+            bbl_gcp_init gcp_project_id
+          end
+
+          desc "terraform your environment and deploy the bosh director"
+          task "up" do
+            bbl_gcp_up
+          end
+        end
+      end
+
+      namespace "bosh" do
+        desc "prepare the concourse bosh deployment"
+        task "init" do
+          bosh_init
+        end
+
+        desc "macro task for all `update` subtasks"
+        task "update" => [
+               "bosh:update:concourse_deployment",
+               "bosh:update:ubuntu_stemcell",
+             ]
+
+        namespace "update" do
+          desc "update the git submodule for concourse-bosh-deployment"
+          task "concourse_deployment" do
+            bosh_update_concourse_deployment
+          end
+
+          desc "upload ubuntu stemcell to the director"
+          task "ubuntu_stemcell" do
+            bosh_update_ubuntu_stemcell
+          end
+
+#       desc "upload windows stemcell to the director"
+#       task "windows_stemcell" do
+#         bosh_update_windows_stemcell
+#       end
+
+#       desc "upload concourse windows release to the director"
+#       task "concourse_windows_release" do
+#         bosh_update_concourse_windows_release
+#       end
+
+#       desc "upload windows-ruby-dev-tools release to the director"
+#       task "windows_ruby_dev_tools" do
+#         bosh_update_windows_ruby_dev_tools
+#       end
+
+#       desc "upload windows-utilities release to the director"
+#       task "windows_utilities_release" do
+#         bosh_update_windows_utilities_release
+#       end
+        end
+
+        desc "deploy concourse"
+        task "deploy" do
+          bosh_deploy
+        end
+      end
+
+      namespace "letsencrypt" do
+        desc "create a cert"
+        task "create" do
+          letsencrypt_create
+        end
+
+        desc "backup web:/etc/letsencrypt to local disk"
+        task "backup" do
+          letsencrypt_backup
+        end
+
+        desc "import letsencrypt keys into `#{BOSH_SECRETS}` from backup"
+        task "import" do
+          letsencrypt_import
+        end
+
+        desc "restore web:/etc/letsencrypt from backup"
+        task "restore" do
+          letsencrypt_restore
+        end
+
+        desc "renew the certificate"
+        task "renew" do
+          letsencrypt_renew
+        end
+      end
+    end
+  end
+end

data/lib/concourse/deployer/utils.rb

@@ -0,0 +1,214 @@
+require 'term/ansicolor'
+
+module Concourse
+  class Deployer
+    module Utils
+      include Term::ANSIColor
+
+      GITIGNORE_FILE           = ".gitignore"
+      GITATTRIBUTES_FILE       = ".gitattributes"
+
+      def sh command
+        running "(in #{Dir.pwd}) #{command}"
+        super command, verbose: false
+      end
+
+      def running message
+        print bold, red, "RUNNING: ", reset, message, "\n"
+      end
+
+      def note message
+        print bold, green, "NOTE: ", reset, message, "\n"
+      end
+
+      def important message
+        print bold, "NOTE: ", message, reset, "\n"
+      end
+
+      def error message, continue=false
+        print red, bold, "ERROR: #{message}", reset, "\n"
+        exit 1 unless continue
+      end
+
+      def ensure_file filename, &block
+        return if File.exist?(filename)
+        File.open(filename, "w") do |f|
+          block.call f
+        end
+      end
+
+      def ensure_in_gitignore file_glob
+        if File.exist?(GITIGNORE_FILE)
+          if File.read(GITIGNORE_FILE).split("\n").include?(file_glob)
+            note "found '#{file_glob}' already present in #{GITIGNORE_FILE}"
+            return
+          end
+        end
+        note "adding '#{file_glob}' to #{GITIGNORE_FILE}"
+        File.open(GITIGNORE_FILE, "a") { |f| f.puts file_glob }
+      end
+
+      def ensure_in_gitcrypt file_glob
+        crypt_entry = "#{file_glob} filter=git-crypt diff=git-crypt"
+        if File.exist?(GITATTRIBUTES_FILE)
+          if File.read(GITATTRIBUTES_FILE).split("\n").include?(crypt_entry)
+            note "found '#{file_glob}' already git-crypted in #{GITATTRIBUTES_FILE}"
+            return
+          end
+        end
+        note "adding '#{file_glob}' as git-crypted to #{GITATTRIBUTES_FILE}"
+        File.open(GITATTRIBUTES_FILE, "a") { |f| f.puts crypt_entry }
+      end
+
+      def ensure_in_envrc entry_key, entry_value=nil
+        entries = if File.exist?(ENVRC_FILE)
+                    File.read(ENVRC_FILE).split("\n")
+                  else
+                    Array.new
+                  end
+
+        if entry_value
+          #
+          #  set an env var
+          #
+          entry_match = /^export #{entry_key}=/
+          entry_contents = "export #{entry_key}=#{entry_value}"
+
+          found_entry = entries.grep(entry_match).first
+
+          if found_entry.nil?
+            note "adding '#{entry_key}=#{entry_value}' to #{ENVRC_FILE}"
+            File.open(ENVRC_FILE, "a") { |f| f.puts entry_contents }
+          else
+            if found_entry == entry_contents
+              note "found '#{entry_key}=#{entry_value}' already present in #{ENVRC_FILE}"
+              return
+            else
+              note "overwriting '#{entry_key}' entry with '#{entry_value}' in #{ENVRC_FILE}"
+              entries.map! do |jentry|
+                jentry =~ entry_match ? entry_contents : jentry
+              end
+              File.open(ENVRC_FILE, "w") { |f| f.puts entries.join("\n") }
+            end
+          end
+        else
+          #
+          #  add a line of bash
+          #
+          entry_contents = entry_key
+          found_entry = entries.find { |line| line == entry_contents }
+
+          if found_entry.nil?
+            note "adding '#{entry_contents}' to #{ENVRC_FILE}"
+            File.open(ENVRC_FILE, "a") { |f| f.puts entry_contents }
+          else
+            note "found '#{entry_contents}' already present in #{ENVRC_FILE}"
+            return
+          end
+        end
+      end
+
+      def ensure_git_submodule repo_url, commitish
+        repo_name = File.basename repo_url
+        sh "git submodule add '#{repo_url}'" unless Dir.exists?(repo_name)
+        Dir.chdir(repo_name) do
+          sh "git checkout '#{commitish}'"
+        end
+      end
+
+      def update_git_submodule repo_url, commitish
+        ensure_git_submodule repo_url, commitish
+
+        repo_name = File.basename repo_url
+        Dir.chdir(repo_name) do
+          sh "git remote update"
+          sh "git pull --rebase"
+        end
+      end
+
+      def which command
+        found = `which #{command}`
+        return $?.success? ? found : nil
+      end
+
+      def unless_which command, whereto
+        if which command
+          note "found command '#{command}'"
+          return
+        end
+        error "please install '#{command}' by visiting #{whereto}"
+      end
+
+      def prompt query, default=nil
+        loop do
+          message = query
+          message += " [#{default}]" if default
+          message += ": "
+          print bold, message, reset
+          answer = STDIN.gets.chomp.strip
+          if answer.empty?
+            return default if default
+            error "Please provide an answer.", true
+          else
+            return answer
+          end
+        end
+      end
+
+      def prompt_for_file_contents query
+        loop do
+          path = prompt query
+          return File.read(path) if File.exists?(path)
+          error("File '#{path}' does not exist.", true)
+        end
+      end
+
+      def bbl_external_ip
+        `bbl lbs`.split(":").last.strip
+      end
+
+      def bosh_secrets &block
+        vars = File.exists?(BOSH_SECRETS) ? YAML.load_file(BOSH_SECRETS) : {}
+        return vars unless block_given?
+
+        yield vars
+        File.open(BOSH_SECRETS, "w") { |f| f.write vars.to_yaml }
+        vars
+      end
+
+      def bosh_update_stemcell name
+        doc = Nokogiri::XML(open("https://bosh.io/stemcells/#{name}"))
+        url = doc.at_xpath("//a[contains(text(), 'Light Stemcell')]/@href")
+        if url.nil?
+          error "Could not find the latest stemcell `#{name}`"
+        end
+        sh "bosh upload-stemcell #{url}"
+      end
+
+      def bosh_update_release repo
+        doc = Nokogiri::XML(open("https://bosh.io/releases/github.com/#{repo}?all=1"))
+        url = doc.at_xpath("//a[contains(text(), 'Release Tarball')]/@href")
+        if url.nil?
+          error "Could not find the latest release `#{repo}`"
+        end
+        if url.value =~ %r{\A/}
+          url = "https://bosh.io#{url}"
+        end
+        sh "bosh upload-release #{url}"
+      end
+
+      def bosh_update_from_git_repo git
+        dirname = File.basename(git)
+        Dir.mktmpdir do |dir|
+          Dir.chdir dir do
+            sh "git clone '#{git}'"
+            Dir.chdir dirname do
+              sh "bosh create-release"
+              sh "bosh upload-release"
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/concourse/deployer/version.rb

@@ -0,0 +1,5 @@
+module Concourse
+  class Deployer
+    VERSION = "0.1.0"
+  end
+end

metadata

@@ -0,0 +1,128 @@
+--- !ruby/object:Gem::Specification
+name: concourse-deployer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Mike Dalessio
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2019-01-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: term-ansicolor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.14'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: concourse-deployer provides an ease-of-use layer on top of bosh-bootloader
+  and bosh, to ease the initial install process and maintenance.
+email:
+- mike.dalessio@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- concourse-deployer.gemspec
+- lib/concourse/deployer.rb
+- lib/concourse/deployer/utils.rb
+- lib/concourse/deployer/version.rb
+homepage: https://github.com/flavorjones/concourse-deployer
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.1
+signing_key: 
+specification_version: 4
+summary: Rake tasks to help BOSH-deploy a Concourse CI environment.
+test_files: []