shiplane 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f140db779603d5107c232dd5dca0665c251b39df409c3590498c4485b6df9f1d
+  data.tar.gz: aac47c05deecae0cfbbfbbb1888c664d305ec62434165ee89a3fb68bc6a85881
+SHA512:
+  metadata.gz: 37359831f595b65b12f55756668023e46574626267fb16a71b459404772424f67605a4e614c570a304746d703b9f94b628d64fde64015ff6e3353a3f5469067d
+  data.tar.gz: eae6e737e636a325cad67d06da705cab114dbb7e4468fb7b107e3f03186c91c80748bdcf6831eac39b54b8c04e8033b0602376e698f1b255621ca5a919e3f41a

data/README.md

@@ -0,0 +1,73 @@
+# Shiplane
+Convert your development docker-compose yaml files into production-ready docker deployments
+
+## The Mission
+### Empower Developers
+Shiplane is about empowering developers to get more done, in less time, with less effort. It is intended to amplify a developer's skills and efforts for little cost. It shouldn't take hours and days to setup a working Shiplane environment. It should take minutes to get a working, sane, and relatively secure solution out of the box. Once working, everything should be tweakable as needed.
+
+### Platform Agnostic
+Shiplane doesn't care what language you use, what orchestration tool you use, or even what kind of OS you are deploying to or from. Shiplane is merely a well-traveled, sane, and easy to use path through the Docker ecosystem that gets you from having nothing to having a working system.
+
+## What does Shiplane do for me?
+### Environment Consistency
+Your development compose environment already defines all your application's dependencies, so why should you need to recreate your entire environment just to deploy to production? It not only duplicates work, but increases the likelihood of differences between environments that developers are always attempting to mitigate. Shiplane helps you minimize any differences so that the transition from your local, development Docker environment is just another push of code to your CI.
+
+### Taming the Wild West
+The sheer number of Docker tools out there is staggering. While they all solve some problem, it can be difficult to decide WHICH tools you want to chain together to make your toolset as a developer new to the Docker ecosystem. Maybe you're not as big as Google and you don't need Kubernetes right now or maybe you just picked up Docker last week, but see it's potential and you're seeking the quickest path to a pipeline right now despite not understanding everything just yet. Shiplane is designed to help you move from having nothing to having a complete Docker ecosystem in a matter of minutes - not hours, days, or weeks.
+
+### Appropriate division of responsibilities
+Docker places a lot of power in the hands of developers to define and manage their development environments and how their code is deployed. This is both good and bad. The good side of things includes developers no longer being subject to the work of other groups in your company to provision and deploy. However, the downsides include things like developers having control over platform security, the keys to the kingdom, and infrastructure. While these aren't necessarily evil, these are typically the domain of an infrastructure team to whom these responsibilities are specifically designated. How can these responsibilities be delegated appropriately without taking away the power that the Docker ecosystem provides developers? There are some existing ways to solve these problems, but they require complex systems to be setup to do so. Shiplane is DESIGNED to make this happen out of the box from day 1. It is designed specifically to allow infrastructure teams to be inserted into the process before developers do any work (infrastructure/security can provide base docker images) and during the CI pipeline (so that infrastructure/security teams can manage keys). This allows you to deploy a docker ecosystem quickly and easily with Shiplane and remain confident that you start off pretty secure and are able to continue using it as your processes evolve.
+
+### Makes it easy to adapt and scale
+Shiplane is actually designed to be easy to use early on in the process, scale with you, be easy to switch to new tools (say you want to switch from a pure Docker deployment to using Kubernetes), and yet be easy to get rid of should you decide it is no longer for you.
+
+### Ok, surely there has to be a catch?
+#### Not really, no. Ok, one minor catch (that you can help me with by contributing!)
+While Shiplane is language and platform agnostic and you will be able to deploy that Python project or your Java project, it is written in Ruby at this time and therefore requires your machine to have Ruby installed. The instructions for doing so are included below. That's it. That's the only caveat. And, if you are so inclined, I would LOVE to accept PRs to help me create a Shiplane binary or otherwise remove this dependency entirely. There is an issue open [here](https://github.com/kirillian/shiplane/issues/11) to discuss this problem and potential fixes for it.
+
+## Installation
+### Installing Ruby
+#### TODO
+
+### Installing Shiplane
+#### Installing locally to a project
+1. Install Bundler if you don't already have it:
+```sh
+gem install bundler
+```
+
+2. You can install Shiplane by adding (if you already have one, skip this first step) a Gemfile with the following contents:
+```Gemfile
+source 'https://rubygems.org'
+
+ruby '2.6.2'
+```
+
+3. Add the following lines to your Gemfile:
+```
+gem 'shiplane' require: false
+gem 'shiplane_bootstrappers_chef', require: false
+gem 'shiplane_deployers_capistrano_docker', require: false
+```
+
+4. Run bundler to install everything:
+```sh
+bundle install
+```
+
+#### Installing globally
+```sh
+gem install shiplane
+gem install shiplane_bootstrappers_chef
+gem install shiplane_deployers_capistrano_docker
+```
+
+### Adding the shiplane.yml to your project folder
+You will need to have a shiplane.yml file in your project's folder in order to configure shiplane.
+You can generate a default version of this file via the following command:
+```sh
+rake shiplane:install[<application_name>]
+```
+
+### Using Shiplane
+#### TODO

data/lib/capistrano/shiplane/version.rb

@@ -0,0 +1,5 @@
+module Capistrano
+  module Shiplane
+    VERSION = Shiplane::Version
+  end
+end

data/lib/capistrano/shiplane.rb

@@ -0,0 +1,6 @@
+require 'shiplane/host'
+require 'shiplane/deploy/container_configuration'
+require 'shiplane/deploy/network_configuration'
+
+load File.expand_path('../tasks/capistrano_stubs.rake', __FILE__)
+load File.expand_path('../tasks/shiplane.rake', __FILE__)

data/lib/capistrano/tasks/capistrano_stubs.rake

@@ -0,0 +1,122 @@
+namespace :deploy do
+  task :finishing do
+  end
+
+  task :new_release_path do
+  end
+
+  task :check do
+  end
+
+  task :set_current_revision  do
+  end
+end
+
+
+# deploy
+#   deploy:check
+#   deploy:check:directories
+#   deploy:check:linked_dirs
+#   deploy:check:linked_files
+#   deploy:check:make_linked_dirs
+#   deploy:cleanup
+#   deploy:cleanup_rollback
+#   deploy:finished
+#   deploy:finishing
+#   deploy:finishing_rollback
+#   deploy:log_revision
+#   deploy:published
+#   deploy:publishing
+#   deploy:revert_release
+#   deploy:reverted
+#   deploy:reverting
+#   deploy:rollback
+#   deploy:set_current_revision
+#   deploy:started
+#   deploy:starting
+#   deploy:symlink:linked_dirs
+#   deploy:symlink:linked_files
+#   deploy:symlink:release
+#   deploy:symlink:shared
+#   deploy:updated
+#   deploy:updating
+#   install
+
+
+  # task :print_config_variables do
+  # end
+
+  # task updating: :new_release_path do
+  # end
+
+  # task :reverting do
+  # end
+
+  # task :publishing do
+  # end
+
+  # task :finishing do
+  # end
+
+  # task :finishing_rollback do
+  # end
+
+  # task :finished do
+  # end
+
+
+
+  # namespace :check do
+  #   task :directories do
+  #   end
+
+  #   task :linked_dirs do
+  #   end
+
+  #   task :make_linked_dirs do
+  #   end
+
+  #   task :linked_files do
+  #   end
+  # end
+
+  # namespace :symlink do
+  #   task :release do
+  #   end
+
+  #   task :shared do
+  #   end
+
+  #   task :linked_dirs do
+  #   end
+
+  #   task :linked_files do
+  #   end
+  # end
+
+  # task :cleanup do
+  # end
+
+  # task :cleanup_rollback do
+  # end
+
+  # task :log_revision do
+  # end
+
+  # task revert_release: :rollback_release_path do
+  # end
+
+  # task :new_release_path do
+  # end
+
+  # task :rollback_release_path do
+  # end
+
+  task :set_current_revision  do
+  end
+
+  # task :set_previous_revision do
+  # end
+
+  # task :restart
+  # task :failed

data/lib/capistrano/tasks/shiplane.rake

@@ -0,0 +1,16 @@
+require 'rake'
+require 'shiplane'
+
+desc "Convert Docker Container Artifacts From Development to Production and Upload to Docker Hub"
+task :shiplane, [:appname, :sha] => ['shiplane:default']
+
+namespace :shiplane do
+  task :default do |t, args|
+    sha = `git rev-parse HEAD`.chomp
+    invoke "shiplane:build", sha
+  end
+
+  task :build, :sha do |t, args|
+    Shiplane::Build.build_latest!(args['sha'], fetch(:stage))
+  end
+end

data/lib/generators/shiplane/install/templates/Capfile.erb

@@ -0,0 +1,13 @@
+# Load DSL and set up stages
+require 'capistrano/setup'
+require 'capistrano/framework'
+
+require 'capistrano/ssh_doctor'
+require "airbrussh/capistrano"
+
+<% gems.each do |gem| -%>
+require 'capistrano/<%= gem %>'
+<% end -%>
+
+# Load custom tasks from `lib/capistrano/tasks` if you have any defined
+Dir.glob('lib/capistrano/tasks/**/*.rake').each { |r| import r }

data/lib/generators/shiplane/install/templates/deploy.rb.erb

@@ -0,0 +1,132 @@
+Airbrussh.configure do |config|
+  config.command_output = false
+end
+
+# config valid only for current version of Capistrano
+lock '3.11.0'
+
+set :application, 'podcaster'
+set :repo_url, 'git@github.com:kirillian/podcaster.git'
+
+set :deploy_to, '/var/www/battlecryforfreedom'
+
+# Default value for :log_level is :debug
+# set :log_level, :debug
+
+# Default value for :pty is false
+# set :pty, true
+
+# Default value for :linked_files is []
+set :linked_files, %w( config/database.yml config/secrets.yml .env.production CHANGELOG.md config/sidekiq_scheduler.yml )
+
+# Default value for linked_dirs is []
+set :linked_dirs, %w( log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system )
+
+# Default value for default_env is {}
+# set :default_env, { path: "/opt/ruby/bin:$PATH" }
+
+# Default value for keep_releases is 5
+# set :keep_releases, 5
+
+set :rbenv_type, :system
+set :rbenv_ruby, File.read('.ruby-version').strip
+
+set :rbenv_prefix, "RBENV_ROOT=#{fetch(:rbenv_path)} RBENV_VERSION=#{fetch(:rbenv_ruby)} #{fetch(:rbenv_path)}/bin/rbenv exec"
+set :rbenv_map_bins, %w{rake gem bundle ruby rails sidekiq sidekiqctl}
+set :rbenv_roles, :app
+
+SSHKit.config.umask = "022"
+
+set :ssh_options, {
+  forward_agent: true
+}
+
+set :passenger_restart_with_touch, true
+
+set :branch, proc { `git rev-list --max-count=1 --abbrev-commit #{ENV["REVISION"] || ENV["BRANCH"] || "master"}`.chomp }.call
+
+set :rollbar_token, '07b2c99baf9b43b087ce0dc96957b52f'
+set :rollbar_env, Proc.new { fetch :stage }
+set :rollbar_role, Proc.new { :app }
+
+set :sha, `git rev-parse HEAD`.chomp
+
+set :shiplane_docker_registry_username, ENV['DOCKERHUB_USERNAME']
+set :shiplane_docker_registry_password, ENV['DOCKERHUB_PASSWORD']
+
+set :shiplane_networks, {
+  battlecryforfreedom: {
+    connections: [
+      "nginx_reverse_proxy",
+      "nginx-proxy-letsencrypt",
+    ],
+  }
+}
+
+set :shiplane_containers, {
+  app: {
+    alias: 'battlecryforfreedom-app',
+    volumes: [],
+    environment: [],
+    expose: 3000,
+    capistrano_role: "docker",
+    repo: "kirillian2/podcaster",
+    command: 'bin/start',
+    virtual_host: "battlecryforfreedom.com",
+    letsencrypt_email: "john.epperson@battlecryforfreedom.com",
+    networks: [
+      "battlecryforfreedom"
+    ],
+  },
+  sidekiq: {
+    alias: 'battlecryforfreedom-sidekiq',
+    volumes: [],
+    environment: [],
+    capistrano_role: "docker",
+    repo: "kirillian2/podcaster",
+    command: 'bin/start_sidekiq_workers',
+    networks: [
+      "battlecryforfreedom"
+    ],
+  },
+  redis: {
+    alias: 'redis',
+    volumes: [
+      "/var/lib/redis:/var/lib/redis/data",
+    ],
+    environment: [],
+    expose: 6379,
+    publish: 6379,
+    capistrano_role: "docker",
+    repo: "redis",
+    tag: "4.0.9-alpine",
+    networks: [
+      "battlecryforfreedom"
+    ],
+  },
+  postgres: {
+    alias: 'postgres',
+    volumes: [
+      "/var/lib/postgres/data:/var/lib/postgresql/data",
+    ],
+    expose: 5432,
+    publish: 5432,
+    capistrano_role: "docker",
+    repo: "postgres",
+    tag: "9.6",
+    networks: [
+      "battlecryforfreedom"
+    ],
+  },
+}
+
+# namespace :deploy do
+#   before :started, :capture_revision
+#   before :started, :upload_changelog
+
+#   # config/deploy.rb
+#   # after 'deploy:symlink:release', 'letsencrypt:register_client'
+#   # after 'letsencrypt:register_client', 'letsencrypt:authorize_domain'
+#   # after 'letsencrypt:authorize_domain', 'letsencrypt:obtain_certificate'
+#   # after 'letsencrypt:obtain_certificate', 'nginx:reload'
+# end

data/lib/generators/shiplane/install/templates/production.rb.erb

@@ -0,0 +1,61 @@
+# server-based syntax
+# ======================
+# Defines a single server with a list of roles and multiple properties.
+# You can define all roles on a single server, or split them:
+
+# server 'example.com', user: 'deploy', roles: %w{app db web}, my_property: :my_value
+# server 'example.com', user: 'deploy', roles: %w{app web}, other_property: :other_value
+# server 'db.example.com', user: 'deploy', roles: %w{db}
+
+set :rails_env, 'production'
+
+# role-based syntax
+# ==================
+
+# Defines a role with one or multiple servers. The primary server in each
+# group is considered to be the first unless any  hosts have the primary
+# property set. Specify the username and a domain or IP for the server.
+# Don't use `:all`, it's a meta role.
+
+# role :app, %w{deploy@example.com}, my_property: :my_value
+# role :web, %w{user1@primary.com user2@additional.com}, other_property: :other_value
+# role :db,  %w{deploy@example.com}
+
+
+
+# Configuration
+# =============
+# You can set any configuration variable like in config/deploy.rb
+# These variables are then only loaded and set in this stage.
+# For available Capistrano configuration variables see the documentation page.
+# http://capistranorb.com/documentation/getting-started/configuration/
+# Feel free to add new variables to customise your setup.
+
+
+
+# Custom SSH Options
+# ==================
+# You may pass any option but keep in mind that net/ssh understands a
+# limited set of options, consult the Net::SSH documentation.
+# http://net-ssh.github.io/net-ssh/classes/Net/SSH.html#method-c-start
+#
+# Global options
+# --------------
+#  set :ssh_options, {
+#    keys: %w(/home/rlisowski/.ssh/id_rsa),
+#    forward_agent: false,
+#    auth_methods: %w(password)
+#  }
+#
+# The server-based syntax can be used to override options:
+# ------------------------------------
+# server 'example.com',
+#   user: 'user_name',
+#   roles: %w{web app},
+#   ssh_options: {
+#     user: 'user_name', # overrides user setting above
+#     keys: %w(/home/user_name/.ssh/id_rsa),
+#     forward_agent: false,
+#     auth_methods: %w(publickey password)
+#     # password: 'please use keys'
+#   }

data/lib/generators/shiplane/install/templates/production_dockerfile_stages.erb

@@ -0,0 +1,44 @@
+# Build Production Intermediate Image
+FROM base as production-intermediate
+
+COPY --from=intermediate $APP_PATH $APP_PATH
+COPY . $APP_PATH
+
+WORKDIR $APP_PATH
+
+ARG GITHUB_TOKEN
+ENV GITHUB_TOKEN $GITHUB_TOKEN
+ENV SHIPLANE building
+ENV RAILS_ENV production
+
+RUN mkdir .git && \
+  touch .git/config
+
+RUN git config --global url."https://$GITHUB_TOKEN:@github.com/".insteadOf "https://github.com/"
+RUN git config --global --add url."https://$GITHUB_TOKEN:@github.com/".insteadOf "ssh://git@github.com/"
+
+RUN bundle install --deployment --jobs=4 --without development test
+
+RUN yarn install --production
+
+RUN rm -rf node_modules/n2-styles/test/
+
+RUN bundle exec rake assets:precompile
+
+RUN sed -i "s/${GITHUB_TOKEN}//" Gemfile.lock
+
+
+# Build Production Image
+FROM base as production
+
+COPY --from=production-intermediate $APP_PATH $APP_PATH
+
+WORKDIR $APP_PATH
+
+RUN bundle config --local path vendor/bundle
+RUN bundle config --local without development:test:assets
+
+ENV RAILS_ENV production
+ENV SHIPLANE running
+ENV RAILS_LOG_TO_STDOUT true
+ENV RAILS_SERVE_STATIC_FILES true

data/lib/generators/shiplane/install/templates/shiplane.yml.erb

@@ -0,0 +1,44 @@
+project:
+  appname: <%= app_name %>
+  version_control: git
+  # The origin url on Github
+  # origin: username/repo_title
+bootstrap:
+  env_file: .env.production
+  chef-bootstrapper:
+    package_name: chefdk_3.6.57-1_amd64.deb
+    package_url: https://packages.chef.io/files/stable/chefdk/3.6.57/ubuntu/18.04/chefdk_3.6.57-1_amd64.deb
+build:
+  settings_folder: .shiplane
+  environment_file: .env.production
+  compose_filepath: docker-compose.yml
+  dockerfile_filepath: Dockerfile
+# Define the artifacts that you want built here
+  artifacts:
+    container-name:
+      tag: <%= app_name %>-base:latest
+      repo: kirillian2/<%= app_name %>
+      ports:
+        - "3000:3000"
+  compose:
+    whitelist:
+      - services.container-name
+    blacklist:
+      - services.container-name.tty
+      - services.container-name.stdin_open
+      - services.container-name.volumes
+      - services.container-name.ports
+      - services.container-name.depends_on
+deploy:
+  servers:
+    # put the server domain or ip address here
+    server-url:
+      # Only set this flag if you need docker to run as the sudo user (default Ubuntu AMI on AWS requires this)
+      requires_sudo: false
+      # Put SSH options here
+      # ssh_options:
+      #   user: a_user
+      #   keys: "path/to/ssh/keys"
+      #   forward_agent: true
+      #   auth_methods:
+      #     - publickey

data/lib/shiplane/build.rb

@@ -0,0 +1,101 @@
+require 'fileutils'
+require 'dotenv'
+
+require_relative 'checkout_artifact'
+require_relative 'convert_compose_file'
+require_relative 'convert_dockerfile'
+require_relative 'configuration'
+
+module Shiplane
+  class Build
+    extend Forwardable
+    attr_accessor :sha, :postfix, :tag_latest
+
+    delegate %i(build_config project_config) => :shiplane_config
+
+    def initialize(sha, postfix:, tag_latest: false)
+      @sha = sha
+      @tag_latest = tag_latest
+      @postfix = postfix
+
+      Dotenv.overload File.join(Dir.pwd, build_config.fetch('environment_file', '.env'))
+    end
+
+    def appname
+      @appname ||= project_config['appname']
+    end
+
+    def project_folder_name
+      @project_folder_name ||= "#{appname}-#{sha}"
+    end
+
+    def project_folder
+      @project_folder ||= File.join(Dir.pwd, 'docker_builds', appname, project_folder_name)
+    end
+
+    def shiplane_config
+      @shiplane_config ||= Shiplane::Configuration.new
+    end
+
+    def docker_config
+      @docker_config ||= YAML.load(File.new(File.join(project_folder, 'docker-compose.yml')))
+    end
+
+    def buildable_artifacts
+      build_config.fetch('artifacts', {})
+    end
+
+    def build!
+      unless File.exist?(File.join(project_folder, Shiplane::SHIPLANE_CONFIG_FILENAME))
+        Shiplane::CheckoutArtifact.checkout!(sha)
+        Shiplane::ConvertComposeFile.convert_output!(project_folder, sha)
+      end
+
+      buildable_artifacts.each do |(artifact_name, attributes)|
+        compose_context = docker_config.fetch('services', {}).fetch(artifact_name.to_s, {})
+        Shiplane::ConvertDockerfile.convert_output!(project_folder, attributes, compose_context)
+
+        FileUtils.cd project_folder do
+          steps(artifact_name, attributes).select{|step| step.fetch(:condition, true) }.each do |step|
+            puts step[:notify_before] if step.has_key? :notify_before
+            success = system(step[:command])
+            raise StepFailureException.new(step[:command], artifact_name) unless success
+            puts step[:notify_after] if step.has_key? :notify_after
+          end
+        end
+      end
+    rescue StepFailureException => e
+      puts e.message
+      raise if ENV['RAISE_EXCEPTIONS_ON_FAILED_BUILD']
+    end
+
+    def steps(artifact_name, attributes)
+      [
+        { command: "docker-compose build --no-cache #{artifact_name}", notify_before: "Building Artifact: #{artifact_name}...", notify_after: "Docker Compose Built", stop_on_failure: true },
+        { command: "docker tag #{attributes['repo']}:#{sha} #{attributes['repo']}:#{postfix}-#{sha}", notify_before: "Tagging Build...", stop_on_failure: true, condition: !!postfix },
+        { command: "docker tag #{attributes['repo']}:#{sha} #{attributes['repo']}:#{postfix}-latest", notify_before: "Tagging Build...", stop_on_failure: true, condition: !!postfix && tag_latest },
+        { command: "docker tag #{attributes['repo']}:#{sha} #{attributes['repo']}:latest", notify_before: "Tagging Latest Build...", stop_on_failure: true, condition: tag_latest },
+        { command: "echo '#{ENV['DOCKERHUB_PASSWORD']}' | docker login --username #{ENV['DOCKERHUB_USERNAME']} --password-stdin", notify_before: "Logging into DockerHub...", stop_on_failure: true },
+        { command: "docker push '#{attributes['repo']}:#{sha}'", notify_before: "Pushing Image", notify_after: "Completed Artifact: #{artifact_name}...", stop_on_failure: true },
+        { command: "docker push '#{attributes['repo']}:#{postfix}-#{sha}'", notify_before: "Pushing #{postfix} Image", notify_after: "Completed Artifact: #{artifact_name}...", stop_on_failure: true, condition: !!postfix },
+        { command: "docker push '#{attributes['repo']}:#{postfix}-latest'", notify_before: "Pushing Latest #{postfix} Image", notify_after: "Completed Latest Artifact: #{artifact_name}...", stop_on_failure: true, condition: !!postfix && tag_latest },
+        { command: "docker push '#{attributes['repo']}:latest'", notify_before: "Pushing Latest Image", notify_after: "Completed Latest Artifact: #{artifact_name}...", stop_on_failure: true, condition: tag_latest },
+      ]
+    end
+
+    def self.build!(sha, postfix = nil)
+      new(sha, postfix: postfix).build!
+    end
+
+    def self.build_latest!(sha, postfix = nil)
+      new(sha, postfix: postfix, tag_latest: true).build!
+    end
+  end
+end
+
+class StepFailureException < RuntimeError
+  def initialize(command, artifact_name)
+    message = "Command [#{command}] failed for artifact: #{artifact_name}" if artifact_name
+    super(message)
+  end
+end