conjur-debify 3.0.0.pre.1118 → 3.0.0

data/README.md

@@ -0,0 +1,303 @@
+# Debify
+
+Debify is a tool used for building and testing DAP appliance packages.
+It is mainly used to package and publish debian packages that are consumed into the
+appliance image in its build stage. However, it also packages and publishes an
+RPM package whenever it does so for a debian.
+
+## Installation
+
+There are two different ways of installing debify: as a gem, or as a Docker image.
+
+### Installing the gem
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'conjur-debify'
+```
+
+And then execute:
+
+```sh-session
+$ bundle
+```
+
+Or install it yourself as a ruby gem:
+
+```sh-session
+$ gem install conjur-debify
+```
+
+### Installing the Docker image
+Pull the Docker image:
+
+```sh-session
+$ VERSION=3.0.0
+$ docker pull registry.tld/conjurinc/debify:$VERSION
+```
+
+Images are tagged with the version specified in [VERSION](./VERSION)
+
+Use the `config` subcommand to get a copy of the wrapper script and the secret definitions for publishing:
+
+```sh-session
+$ docker run --rm debify:$VERSION config script > docker-debify
+$ chmod +x docker-debify
+# Optionally, if publishing a deb
+$ docker run --rm debify:$VERSION config secrets > publishing-secrets.yml
+```
+
+Running `docker-debify` will then start a container configured to run debify:
+
+```sh-session
+$ ./docker-debify help
+NAME
+    debify - Utility commands for building and testing Conjur appliance Debian packages
+
+SYNOPSIS
+    debify [global options] command [command options] [arguments...]
+
+VERSION
+    3.0.0
+
+
+GLOBAL OPTIONS
+    --env=arg           - Set an environment variable (e.g. TERM=xterm) when starting a container (may be used more than once, default:
+                          none)
+    --help              - Show this message
+    --[no-]local-bundle - Mount local bundle to reuse gems from previous installation
+    --version           - Display the program version
+
+COMMANDS
+    clean          - Clean current working directory of non-Git-managed files
+    config         - Show the given configuration
+    detect-version - Auto-detect and print the repository version
+    help           - Shows a list of commands or help for one command
+    initconfig     - Initialize the config file using current global options
+    package        - Build a debian package for a project
+    publish        - Publish a debian package to apt repository
+    sandbox        - Setup a development sandbox for a Conjur debian package in a Conjur appliance container
+    test           - Test a Conjur debian package in a Conjur appliance container
+```
+
+
+Note that debify itself creates images and starts containers, so it
+needs access to the host's `docker.sock`. Additionally, it requires
+that it be started in root directory of the project being packaged.
+
+## Build a package
+
+Builds a Conjur Debian package from a Ruby gem.
+
+```
+$ debify help package
+NAME
+    package - Build a debian package for a project
+    
+SYNOPSIS
+    debify [global options] package [command options] project_name -- <fpm-arguments>
+b
+DESCRIPTION
+    The package is built using fpm (https://github.com/jordansissel/fpm).
+
+    The project directory is required to contain:
+
+    * A Gemfile and Gemfile.lock * A shell script called debify.sh
+
+    debify.sh is invoked by the package build process to create any custom files, other than the project source tree. For example, config files can be
+    created in /opt/conjur/etc.
+
+    The distrib folder in the project source tree is intended to create scripts for package pre-install, post-install etc. The distrib folder is not
+    included in the deb package, so its contents should be copied to the file system or packaged using fpm arguments.
+
+    All arguments to this command which follow the double-dash are propagated to the fpm command. 
+
+COMMAND OPTIONS
+    -d, --dir=arg     - Set the current working directory (default: none)
+    -v, --version=arg - Specify the deb version; by default, it's read from the VERSION file (default: none)
+```
+
+### Example usage
+
+```sh-session
+$ package_name=$(debify package -d example -v 0.0.1 example -- --post-install /distrib/postinstall.sh)
+$ echo $package_name
+conjur-example_0.0.1_amd64.deb
+```
+
+## Test a package
+
+```
+$ debify help test
+NAME
+    test - Test a Conjur debian package in a Conjur appliance container
+
+SYNOPSIS
+    debify [global options] test [command options] project-name test-script
+
+DESCRIPTION
+    First, a Conjur appliance container is created and started. By default, the container image is registry.tld/conjur-appliance-cuke-master. An image tag
+    MUST be supplied. This image is configured with all the CONJUR_ environment variables setup for the local environment (appliance URL, cert path, admin
+    username and password, etc). The project source tree is also mounted into the container, at /src/<project-name>.
+
+    This command then waits for Conjur to initialize and be healthy. It proceeds by installing the conjur-<project-name>_latest_amd64.deb from the project
+    working directory.
+
+    Then the evoke "test-install" command is used to install the test code in the /src/<project-name>. Basically, the development bundle is installed and
+    the database configuration (if any) is setup.
+
+    Next, an optional "configure-script" from the project source tree is run, with the container id as the program argument. This command waits for Conjur
+    to be healthy again.
+
+    Finally, a test script from the project source tree is run, again with the container id as the program argument.
+
+    Then the Conjur container is deleted (use --keep to leave it running). 
+
+COMMAND OPTIONS
+    -c, --configure-script=arg - Shell script to configure the appliance before testing (default: none)
+    -d, --dir=arg              - Set the current working directory (default: none)
+    -i, --image=arg            - Image name (default: registry.tld/conjur-appliance-cuke-master)
+    -k, --[no-]keep            - Keep the Conjur appliance container after the command finishes
+    --[no-]pull                - Pull the image, even if it's in the Docker engine already (default: enabled)
+    -t, --image-tag=arg        - Image tag, e.g. 4.5-stable, 4.6-stable (default: none)
+```
+
+### Example usage
+
+```sh-session
+$ debify test -i conjur-appliance-cuke-master --image-tag 4.6-dev --no-pull -d example example test.sh
+```
+
+## Publish a package
+
+```
+$ debify help publish
+NAME
+    publish - Publish a debian package to apt repository
+
+SYNOPSIS
+    debify [global options] publish [command options] distribution project-name
+
+DESCRIPTION
+    Publishes a deb created with `debify package` to our private apt repository.
+
+    "distribution" should match the major/minor version of the Conjur appliance you want to install to.
+
+    The package name is a required option. The package version can be specified as a CLI option, or it will be auto-detected from Git.
+
+    --component should be 'stable' if run after package tests pass or 'testing' if the package is not yet ready for release. If you don't specify the component, it will be set to
+    'testing' unless the current git branch is 'master' or 'origin/master'. The git branch is first detected from the env var GIT_BRANCH, and then by checking `git rev-parse
+    --abbrev-ref HEAD` (which won't give you the answer you want when detached).
+
+COMMAND OPTIONS
+    -c, --component=arg - Maturity stage of the package, 'testing' or 'stable' (default: none)
+    -d, --dir=arg       - Set the current working directory (default: none)
+    -v, --version=arg   - Specify the deb package version; by default, it's computed automatically (default: none)
+```
+
+### Example usage
+
+You will need read permission for the `ci/artifactory/users/jenkins/username` and `ci/artifactory/users/jenkins/password` variables in order to run this command from your local machine.
+
+```sh-session
+$ debify publish -c stable 0.0.1 example
+[Thread 0] Uploading artifact: https://conjurinc.artifactoryonline.com/conjurinc/debian-local/conjur-example_0.1.1-c9fd618_amd64.deb;deb.distribution=0.1.1;deb.component=possum;deb.architecture=amd64
+[Thread 0] Artifactory response: 201 Created
+Uploaded 1 artifacts to Artifactory.
+```
+
+## Create a development session in a Conjur appliance container
+
+
+```
+$ debify help sandbox
+NAME
+    sandbox - Setup a development sandbox for a Conjur debian package in a Conjur appliance container
+
+SYNOPSIS
+    debify [global options] sandbox [command options] 
+
+DESCRIPTION
+    First, a Conjur appliance container is created and started. By default, the container image is 
+    registry.tld/conjur-appliance-cuke-master. An image tag MUST be supplied. This image
+    is configured with all the CONJUR_ environment variables setup for the local environment (appliance URL, 
+    cert path, admin username and password, etc). The project source tree is
+    also mounted into the container, at /src/<project-name>, where <project-name> is taken from the name of the 
+    current working directory.
+
+    Once in the container, use "/opt/conjur/evoke/bin/dev-install" to install the development bundle of your project. 
+
+COMMAND OPTIONS
+    --bind=arg          - Bind another source directory into the container. Use <src>:<dest>, where both are full paths. (default: none)
+    -d, --dir=arg       - Set the current working directory (default: none)
+    -i, --image=arg     - Image name (default: registry.tld/conjur-appliance-cuke-master)
+    --[no-]pull         - 'docker pull' the Conjur container image
+    -t, --image-tag=arg - Image tag, e.g. 4.5-stable, 4.6-stable (default: none)
+```
+
+### Example usage
+
+```sh-session
+authz $ debify sandbox -t $(cat VERSION_APPLIANCE)-stable
+... much logging
+authz $ docker exec -it authz-sandbox bash
+root@7d4217655332:/src/authz# /opt/conjur/evoke/bin/dev-install authz
+...
+root@7d4217655332:/src/authz# export RAILS_ENV=test
+root@7d4217655332:/src/authz# bundle exec rake db:migrate
+```
+
+## Usage with docker-compose
+
+As of v1.10.0, both the `test` and `sandbox` subcommands support the `--net` switch. This allows you to specify a network to which the Conjur appliance container should be attached. 
+
+There are a variety of ways to make use of this feature. One
+possiblity is creating a network using `docker network create`, then
+attaching both the docker-compose services, as well as the Conjur
+appliance container created by debify, to it. 
+
+As a (somewhat contrived) example, create a new docker network:
+
+```sh-session
+$ docker network create testnet
+```
+
+Use a docker-compose file like [example/docker-compose.yml](example/docker-compose.yml)
+
+```yaml
+version: "2"
+networks:
+  svcnet:
+    external:
+      name: testnet
+services:
+  db:
+    image: postgres
+    container_name: mydb
+    networks:
+      - svcnet
+```
+
+Bring up the db service:
+
+```sh-session
+debify $ cd example
+example $ docker-compose up -d
+```
+
+Start a sandbox, see that it can resolve the hostname `mydb`:
+
+```sh-session
+
+example $ debify sandbox -t 5.0-stable --net testnet
+example $ docker exec -it example-sandbox /bin/bash
+root@7d4217655332:/src/example# getent hosts mydb
+172.19.0.2      mydb
+```
+
+
+## Contributing
+
+For instructions on how to get started and 
+descriptions of our development workflows, please see our
+[contributing guide](CONTRIBUTING.md).

data/Rakefile

@@ -0,0 +1,75 @@
+require 'rake/clean'
+require 'rubygems'
+require 'rubygems/package_task'
+require 'rdoc/task'
+
+def cucumber?
+  require 'cucumber'
+  require 'cucumber/rake/task'
+rescue LoadError
+  false
+end
+
+def rspec?
+  require 'rspec/core/rake_task'
+  require 'ci/reporter/rake/rspec'
+end
+  
+Rake::RDocTask.new do |rd|
+  rd.main = "README.rdoc"
+  rd.rdoc_files.include("README.rdoc","lib/**/*.rb","bin/**/*")
+  rd.title = 'Your application title'
+end
+
+spec = eval(File.read('debify.gemspec'))
+
+Gem::PackageTask.new(spec) do |pkg|
+end
+
+if cucumber?
+  CUKE_RESULTS = 'features/reports'
+
+  desc 'Run features'
+  Cucumber::Rake::Task.new(:features) do |t|
+    opts = [
+      "features",
+      "--format",
+      "junit",
+      "-o",
+      CUKE_RESULTS,
+      "--format",
+      "pretty",
+      "-x"]
+    opts += ["--tags", ENV['TAGS']] if ENV['TAGS']
+    opts += ["--tags", "not @skip"]
+    t.cucumber_opts = opts
+    t.fork = false
+  end
+
+  desc 'Run features tagged as work-in-progress (@wip)'
+  Cucumber::Rake::Task.new('features:wip') do |t|
+    tag_opts = %w[--tags @wip]
+    opts = [
+      "features",
+      "--format",
+      "junit",
+      "-o",
+      CUKE_RESULTS,
+      "--format",
+      "pretty",
+      "-x",
+      "-s"]
+    t.cucumber_opts = opts + tag_opts
+    t.fork = false
+  end
+
+  task :cucumber => :features
+  task 'cucumber:wip' => 'features:wip'
+  task :wip => 'features:wip'
+end
+
+if rspec?
+  desc 'Run specs'
+  RSpec::Core::RakeTask.new(:spec)
+  task :spec => 'ci:setup:rspec'
+end

data/VERSION

@@ -1 +1 @@
-3.0.0-1118
+3.0.0

data/bin/debify

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require 'gli'
+require 'conjur/debify'
+
+exit run(ARGV)

data/build.sh

@@ -0,0 +1,8 @@
+#!/bin/bash -ex
+
+if [ ! -f "VERSION" ]; then
+  echo -n "0.0.1.dev" > VERSION
+fi
+
+VERSION=$(< VERSION)
+docker build --build-arg VERSION=$VERSION -t debify:$VERSION .

data/ci/test.sh

@@ -0,0 +1,8 @@
+#!/bin/bash -ex
+
+bundle
+
+for target in spec cucumber; do
+  bundle exec rake $target
+done
+

data/debify.gemspec

@@ -0,0 +1,36 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'conjur/debify/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "conjur-debify"
+  spec.version       = Conjur::Debify::VERSION
+  spec.authors       = ["CyberArk Software, Inc."]
+  spec.email         = ["conj_maintainers@cyberark.com"]
+  spec.summary       = %q{Utility commands to build and package Conjur services as Debian packages}
+  spec.homepage      = "https://github.com/conjurinc/debify"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").append("VERSION")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+  
+  spec.add_dependency "gli"
+  spec.add_dependency "docker-api", "~> 2.0"
+  spec.add_dependency "conjur-cli" , "~> 6"
+  spec.add_dependency "conjur-api", "~> 5.3"
+  spec.add_development_dependency "bundler", ">= 2.2.30"
+  spec.add_development_dependency "fakefs", "~> 0"
+  spec.add_development_dependency "rake", "~> 13.0"
+  
+  # Pin to cucumbe v2. cucumber v3 changes (breaks) the behavior of
+  # unmatched capture groups with \(d+). In v3, the value of such a
+  # group is 0 instead of nil, which breaks aruba's "I successfully
+  # run...." steps.
+  spec.add_development_dependency "cucumber", '~> 7.1'
+  spec.add_development_dependency "aruba", "~> 2.0"
+  spec.add_development_dependency 'rspec', '~> 3.10'
+  spec.add_development_dependency 'ci_reporter_rspec', '~> 1.0'
+end

data/distrib/conjur_creds.rb

@@ -0,0 +1,7 @@
+require 'netrc'
+Netrc.configure do |c|
+  c[:allow_permissive_netrc_file] = true
+end
+
+creds = Netrc.read('/root/.netrc')[ENV['CONJUR_APPLIANCE_URL'] + '/authn']
+print "#{creds.login} #{creds.password}" if creds

data/distrib/docker-debify

@@ -0,0 +1,50 @@
+#!/bin/bash -e
+
+# If we're running in jenkins, there will be a conjur.identity file
+# with Conjur creds in it. Otherwise, assume the user's netrc has
+# them.
+if [[ -f /etc/conjur.identity ]]; then
+  netrc=/etc/conjur.identity
+else
+  netrc=$HOME/.netrc
+fi
+
+: ${CONJURRC=/etc/conjur.conf}
+
+conjur_config() {
+  local name=$1; shift
+  grep $name $CONJURRC | awk '{print $2}' | tr -d '"'
+}
+
+export CONJUR_APPLIANCE_URL=$(conjur_config appliance_url)
+export CONJUR_SSL_CERTIFICATE="$(< $(conjur_config cert_file))"
+
+[[ -f "$HOME/.debifyrc" ]] && rc_arg="-v $HOME/.debifyrc:/root/.debifyrc:ro"
+
+: ${DEBIFY_ENVFILE=debify.env}
+[[ -f $DEBIFY_ENVFILE ]] &&  envfile_arg="--env-file $DEBIFY_ENVFILE"
+
+# Mounting docker socket is required because subcommands launch
+# containers.
+#
+# The environment variables can't go into an env-file, because docker
+# doesn't handle env-file variables the same way it handles
+# command-line variables. In particular, when a variable in an
+# env-file is unset in the calling enviroment, it gets set in the
+# container without a value. When such a variable is mentioned on the
+# command line, it doesn't get set in the container.
+tty=$(tty -s && echo "-t" || true)
+docker run -i $tty --rm \
+  -e GLI_DEBUG -e DEBUG \
+  -e CONJUR_APPLIANCE_URL -e CONJUR_SSL_CERTIFICATE \
+  -e GIT_BRANCH -e BRANCH_NAME \
+  -e ARTIFACTORY_USER -e ARTIFACTORY_PASSWORD \
+  -e HOME \
+  ${envfile_arg} \
+  -v "$PWD:$PWD" -w "$PWD" \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v "${HOME}:${HOME}" \
+  -v "${netrc}:${HOME}/.netrc:ro" \
+  ${rc_arg} \
+  ${DEBIFY_ENTRYPOINT+--entrypoint $DEBIFY_ENTRYPOINT} \
+  ${DEBIFY_IMAGE-registry.tld/conjurinc/debify:@@DEBIFY_VERSION@@} "$@"

data/distrib/entrypoint.sh

@@ -0,0 +1,19 @@
+#!/bin/bash -e
+
+# Make sure we don't echo commands as executed, otherwise the user's
+# Conjur API key will show up in the logs.
+set +x
+
+creds=( $(ruby /debify/distrib/conjur_creds.rb) )
+
+# If there are creds, use them to log in to the registry.
+#
+# If there are no creds, any commands that do
+# Docker stuff will fail, but the non-Docker commands (e.g. the config
+# subcommands) will work fine.
+if [[ ${#creds[*]} > 0 ]]; then
+  echo -n "${creds[1]}" | docker login registry.tld -u ${creds[0]} --password-stdin >/dev/null 2>&1
+fi
+
+exec debify "$@"
+

data/distrib/script

@@ -0,0 +1 @@
+docker-debify

data/distrib/secrets

@@ -0,0 +1 @@
+secrets.yml

data/distrib/secrets.yml

@@ -0,0 +1,2 @@
+ARTIFACTORY_USER: !var ci/artifactory/users/jenkins/username
+ARTIFACTORY_PASSWORD: !var ci/artifactory/users/jenkins/password

data/example/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+group :development do
+  gem 'pry'
+end
+
+group :test do
+  gem 'rspec'
+end

data/example/Gemfile.lock

@@ -0,0 +1,32 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    coderay (1.1.2)
+    diff-lcs (1.3)
+    method_source (0.9.0)
+    pry (0.11.3)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    rspec (3.7.0)
+      rspec-core (~> 3.7.0)
+      rspec-expectations (~> 3.7.0)
+      rspec-mocks (~> 3.7.0)
+    rspec-core (3.7.1)
+      rspec-support (~> 3.7.0)
+    rspec-expectations (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-mocks (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-support (3.7.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  pry
+  rspec
+
+BUNDLED WITH
+   2.1.4

data/example/debify.sh

@@ -0,0 +1,3 @@
+#!/bin/bash -e
+
+echo "running debify.sh"

data/example/distrib/postinstall.sh

@@ -0,0 +1,8 @@
+#!/bin/sh -e
+# Example postinstall script.
+
+case $1 in
+  configure)
+	echo 'Executing post-install script'
+    ;;
+esac

data/example/docker-compose.yml

@@ -0,0 +1,11 @@
+version: "2"
+networks:
+  svcnet:
+    external:
+      name: testnet
+services:
+  db:
+    image: postgres
+    container_name: mydb
+    networks:
+      - svcnet

data/example/net-test.sh

@@ -0,0 +1,7 @@
+#!/bin/bash -ex
+
+cid=$1
+
+docker exec $cid curl -s http://other_host > /dev/null
+
+echo Test succeeded

data/example/test.sh

@@ -0,0 +1,4 @@
+#!/bin/bash -ex
+
+ruby -rrspec -e 'puts RSpec::Version::STRING'
+echo Test succeeded

data/features/detect_version.feature

@@ -0,0 +1,12 @@
+@announce-output
+Feature: Automatic version string
+
+  Scenario: 'example' project gets a default version
+    When I run `env DEBUG=true GLI_DEBUG=true debify detect-version -d ../../example`
+    Then the exit status should be 0
+    And the output should match /\d+.\d+.\d+-\d+-.*/
+
+  @skip
+  Scenario: Test @skip tag, failed by default
+    When I run `env DEBUG=true GLI_DEBUG=true debify detect-version -d ../../example`
+    Then the exit status should be 1

data/features/package.feature

@@ -0,0 +1,23 @@
+@announce-output
+Feature: Packaging
+
+  Background:
+    # We use version 0.0.1-suffix to verify that RPM converts dashes to underscores
+    # in the version as we expect
+    Given I successfully run `env DEBUG=true GLI_DEBUG=true debify package -d ../../example -v 0.0.1-suffix example -- --post-install /distrib/postinstall.sh`
+    And I successfully run `env DEBUG=true GLI_DEBUG=true debify package -d ../../example --output rpm  -v 0.0.1-suffix example  -- --post-install /distrib/postinstall.sh`
+
+  Scenario: 'example' project can be packaged successfully
+    Then the stdout should contain "conjur-example_0.0.1-suffix_amd64.deb"
+    And the stdout should contain "conjur-example-dev_0.0.1-suffix_amd64.deb"
+    And the stdout should contain "conjur-example-0.0.1_suffix-1.x86_64.rpm"
+    And the stdout should contain "conjur-example-dev-0.0.1_suffix-1.x86_64.rpm"
+
+  Scenario: 'clean' command will delete non-Git-managed files
+    When I successfully run `env DEBUG=true GLI_DEBUG=true debify clean -d ../../example --force`
+    And I successfully run `find ../../example`
+    Then the stdout from "find ../../example" should not contain "conjur-example_0.0.1-suffix_amd64.deb"
+    And the stdout from "find ../../example" should not contain "conjur-example-0.0.1_suffix-1.x86_64.rpm"
+
+  Scenario: 'example' project can be published
+    When I successfully run `env DEBUG=true GLI_DEBUG=true debify publish -v 0.0.1-suffix -d ../../example 5.0 example`