mantra 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 993ccf8b86579e3bcf76e407de934c053bafd6d2
-  data.tar.gz: d8904bfab735ac2ff3eafadbbe6ecaf457847c3d
+  metadata.gz: f071f30737a66c7b81b0b0100831abfe981a5032
+  data.tar.gz: 1c0015a38b4ecd69be8d38688bb6c8284d5ff6dc
 SHA512:
-  metadata.gz: 7d32e81a7b46e059f15136db0665a0670b7b30fe31b39628b34a9a7ae9643a0c4617925bcb11543039a1297faf9e85b05b8e40cf4f8f56f4403c8b3eb0cc7770
-  data.tar.gz: 06f8c0903e36c59a67f3dd71500ca5834d7890b44de7c0331bea02fc4ccb24644f447ccb3bae92003d551fa83e889769ad149c0c2822d1eabbb432be465ff515
+  metadata.gz: 5a85bda3e4389db1b75a9bfde752f5aa8185ad03fec4caadf8239c4bd4a266891167d57388fb1699b11fe2e5c154763642c97dfef51e712e6c66daada7e9af2b
+  data.tar.gz: fec5710aaed34e7961fe4a63ff405390bb416a57370bce2b44f9d5e4303e1fb3b492bbf0ed6bda5d56de342a7e4a4178497bdc6b048c3949240ae48dfd50e2d5

data/.gitignore

@@ -7,3 +7,5 @@
 /pkg/
 /spec/reports/
 /tmp/
+*.gem
+.ruby-*

data/.rspec

@@ -1,2 +1,3 @@
 --format documentation
 --color
+--require spec_helper

data/.travis.yml

@@ -1,4 +1,10 @@
 language: ruby
 rvm:
-  - 2.3.0
-before_install: gem install bundler -v 1.11.2
+  - 2.1.10
+  - 2.2.2
+  - 2.3.1
+before_install:
+  - gem install bundler -v 1.11.2
+  - bundle install
+script:
+  - "bundle exec rake"

data/README.md

@@ -1,34 +1,144 @@
-# Mantra
+# Mantra - मंत्र
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/mantra`. To experiment with that code, run `bin/console` for an interactive prompt.
+[![Build Status](https://travis-ci.org/allomov/mantra.svg?branch=master)](https://travis-ci.org/allomov/mantra)
 
-TODO: Delete this and the text above, and describe your gem
+### Description
 
-## Installation
+Mantra (/ˈmæntrə/, abr. **Man**ifest **Tra**nsformation) is a tool to ease work with manifests.
+
+It allows to do following:
+
+* find manifest parts 
+* extract cloud config from you BOSH v1 manifest
+* convert your manifest into spiff templates with
+  * extracted passwords
+  * extracted certificates to different files
+  * templatized network configuration
+  * extracted or changed properties
+* and much more
+
+Mantra is tend to be an easily extendable tool, so you can add new commands and transforms.
+
+### Concepts
+
+While some actions performed on manifest can be automated, anther require special configuration and extra knowledges about source manifes. Here is a small glossery that I use:
+
+`Transformation Manifest` - config file that describes how your manifest should be udpated
+`Source Manifest` - your source BOSH manifest, if you run templatizers on source manifest they can change it.
+`Target Manifest` - manifest that will contain extracted template properties, to restore source manifest you'll need to run merge tool with resulting source manifest.
+`Transform` - a ruby class that declares how specific transformation should be done.
+
+### Examples
+
+Find and output in json some manifest parts:
+
+```
+mantra find jobs[name=consul].properties -m manifest.yml -f json
+# alias: mantra f
+```
 
-Add this line to your application's Gemfile:
+Transform BOSH v1 manifest to extract cloud config:
 
-```ruby
-gem 'mantra'
+```
+mantra transform -c transformation-manifest.yml -m manifest.yml
+# mantra t
 ```
 
-And then execute:
+Here is an example of `transformation-manifest.yml`:
+
+```yaml
+global:
+  source: manifest.yml
+  target: cloud-config.yml
+transforms:
+- filter:
+    sections: ["networks", "compilation", "update", "resource_pools", "disk_pools"]
+- rename:
+    section: "resource_pools"
+    to: "vm_types"
+- rename:
+    section: "disk_pools"
+    to: "disk_types"
+- type: add
+    section:
+      az: z1
+    to: "networks[].subnets[]"
+- add:
+    section:
+      az: z1
+    to: "update"
+- add:
+    section:
+      az: z1
+    to: "compilation"
+- add:
+    section:
+      azs:
+      - name: z1
+    to: ""
+```
 
-    $ bundle
 
-Or install it yourself as:
+```yaml
+global_vars:
+  source: manifest.yml
+transforms:
+- extract:
+    section: jobs
+    to:   workspace/jobs.yml  
+- extract-certs:
+    from: workspace/properties.yml
+    to:   workspace/secrets.yml
+- extract-certs-to-files:
+    from: workspace/secrets.yml
+    to:   workspace/secrets
+- extract:
+    section: jobs
+    to:
+      file: manifest.yml
+      with_scope: meta.jobs
+- extract:
+    source: manifest.yml
+    section: resource_pools
+- extract:
+    section: resource_pools
+    to: resource_pools.yml
+- extract:
+    section: networks
+    to: networks.yml
+- templatize-ip-address:
+    from:  networks.yml
+    to: workspace/stub.yml
+    quads:
+      - range: "1-2"
+        scope: meta.networks.cf.prefix
+        with_value: "192.168"
+      - number: 3
+        scope: meta.networks.cf.quad1
+        with_value: 2
+      - number: 3
+        scope: meta.networks.cf.quad2
+        with_value: 3
+- templatize-value:
+    source: manifest.yml
+    to: workspace/stub.yml
+    value: cfdomain.com
+    in: "*.cfdomain.com"
+    scope: meta.domain
+```
 
-    $ gem install mantra
 
-## Usage
+## Installation
 
-TODO: Write usage instructions here
+Install it as a gem:
 
-## Development
+```
+    $ gem install mantra
+```
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+## Dependencies
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+`openssl`!
 
 ## Contributing
 

data/bin/mantra

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift File.expand_path('../lib', __FILE__)
+require "mantra"
+
+if ARGV.empty?
+  puts Mantra::Command.usage
+  exit 0
+end
+
+command_type = ARGV.shift
+options = {type: command_type.to_sym, args: ARGV}
+
+command = Mantra::Command.create(options)
+
+command.run
+

data/ci/README.md

@@ -0,0 +1,6 @@
+### How to use
+
+```
+fly set-pipeline -t tutorial -c pipeline.yml --load-vars-from stub.yml -p merrill-manifest-generator
+fly unpause-pipeline -p merrill-manifest-generator
+```

data/ci/docker/Dockerfile

@@ -0,0 +1,20 @@
+FROM ruby:2.3.0
+MAINTAINER Alex Lomov <lomov.as@gmail.com>
+# should go to allomov/ruby#manifest-generator
+
+RUN apt-get -y update --fix-missing 
+RUN apt-get -y install git git-core curl gawk g++ gcc make libc6-dev libreadline6-dev \  
+    zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 autoconf \  
+    libgdbm-dev libncurses5-dev automake libtool bison pkg-config libffi-dev
+
+RUN apt-get install -y postgresql postgresql-contrib postgresql-client libpq5 libpq-dev
+
+RUN apt-get install -y nodejs npm cmake
+RUN ln -s  /usr/bin/nodejs  /usr/bin/node 
+
+RUN gem install bundler --no-ri --no-rdoc
+
+COPY manifest-generator /manifest-generator
+
+WORKDIR /manifest-generator
+RUN bundle install

data/ci/docker/build-docker-image

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+cp Dockerfile ../../../Dockerfile
+
+pushd ../../..
+  docker build -t allomov/ruby:manifest-generator .
+  docker push allomov/ruby:manifest-generator
+popd
+
+rm ../../../Dockerfile

data/ci/pipeline.yml

@@ -0,0 +1,42 @@
+---
+resources:
+- name: merrill-manifest-generator-dev
+  type: git
+  source:
+    uri: {{github-repo}}
+    branch: manifest-generator-dev
+    private_key: {{git-repo-private-key}}
+- name: slack-notification
+  type: slack-notification
+  source:
+    url: {{slack-notification-url}}
+
+jobs:
+- name: dev-version
+  plan:
+  - get: merrill-manifest-generator-dev
+    trigger: true
+  - task: run-tests
+    file: merrill-manifest-generator-dev/manifest-generator/ci/tasks/run-tests-dev.yml
+    on_failure:
+      do:
+      - put: slack-notification
+        params:
+          channel: {{slack-channel}}
+          username: {{slack-username}}
+          icon_emoji: ":robot_face:"
+          text: "@here dev version of manifest-generator seems to be broken"
+- name: stable-version
+  plan:
+  - get: merrill-manifest-generator-stable
+    trigger: true
+  - task: run-tests
+    file: merrill-manifest-generator-stable/manifest-generator/ci/tasks/run-tests-stable.yml
+    on_failure:
+      do:
+      - put: slack-notification
+        params:
+          channel: {{slack-channel}}
+          username: {{slack-username}}
+          icon_emoji: ":robot_face:"
+          text: "@here stable version of manifest-generator seems to be broken"

data/ci/scripts/run-tests

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+bundle exec rake

data/ci/stub.example.yml

@@ -0,0 +1,6 @@
+slack-notification-url: "https://https://hooks.slack.com/services/XXXXXX/XXXXXX/ID"
+slack-channel: "#general"
+slack-username: "Merrill-Manifest-Generator-Bot"
+github-repo: git@github.com:pivotalservices/merrill.git
+git-repo-private-key: |
+  Your private key here

data/ci/stub.yml

@@ -0,0 +1,33 @@
+---
+slack-notification-url: "https://hooks.slack.com/services/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"
+slack-channel: "#general"
+slack-username: "Merrill-Manifest-Generator-Bot"
+github-repo: git@github.com:pivotalservices/merrill.git
+git-repo-private-key: |
+  -----BEGIN RSA PRIVATE KEY-----
+  MIIEogIBAAKCAQEAz/SJ469kiOr3CysK7Tqp1ZQPYFviObH6N4ucmAiWb78KxqM+
+  BBMEsmdo/2Pe0CKfiMGW14LA4JlpQIguuNixUiSffBo7zlfxigm8MpgdF3i7kqOv
+  Ffv+zOpLfq1i0rRz+cV9RsowRxbRkWekiy0YqDqVW4pyzfXIMd81dhjQh25w6Zs+
+  PFyAlg9bU7QUl0F9vljqpHii0n7hCL2hcKN7Jq0XF4AHPyQg24UgagU955wvlWvx
+  iJZ4fRcG1bSNTCJvParS7Pl/wnCC73WJ0d8MMOVHbk3G0lYPtbaCwUctZ47MjB4f
+  j6+40sfkfoJZQ6eb43AHnCZVypS652hcKGiLpQIDAQABAoIBAGdKZ1MYQy9exeZe
+  MscyZDkZmama+fr+U/JS/izl1xMaKRgQ7TTI3YRfqN9xk7J+AA/FWE8jOvpUONs8
+  yTg3KZxLZIwX26zaMbcGrtvSgmCuLzze4JBi4mSV23nIVVWjHBfr6BXJR8GHrAA1
+  Ip2egKjOsHEPi4aRliZvX/aQzML3SxcpttyxaRT5ytokMSjNSNRk2fTdcCliLbnD
+  +UOsGwWiZpC1BDZXHeKsnyGLgK5cgG4Pygnr0gDZY73wflnGHWFBQhIaLfdZdBb6
+  zXyYD6XGM3p+SUOL0JxWLFCMLWntpzn0LRx3RBmicXiW82dLrjVo1nQw9+TtMOv3
+  kWrO9OECgYEA6FpU0AngbKDnWAjG0peYrIKFgF/9Gk60JCQsXkHyNJ/mVt4sjFtG
+  y5Ajh6QxKReF1Be+W7di2CH9/Sb0xYdeiFAWtqNeBEOMOZYAV22zoFAb97SIlSrT
+  oZSquzY6jXldZU1lH0Pknklmz8Y4jcRSHvUKDXlr2MHvXMP7RIHKvTcCgYEA5R6O
+  m/LgMVn8ahB8jTSVPO41Ely1oH6dUghaQEqFf0MJmKBKESq+SHId8wHD9Sbuirg/
+  /KtYBe0ouvtIddX4AIoO5NVOudUGzLZEBntHUnpAln/P3COJkd0LdbN/5okguMkb
+  uMHJdqL/OWAUGOVzpM+DxK53c70r1VpIr3IETAMCgYAVP1VsJMsSA/DD2yGbVmOO
+  uuzMXXq+sh3FBE3OXMdySoD8xppaR2qE6Oxm1fvs3wrM51Lv/igazZj1Ul+lxIg/
+  fNtm3i+VwjuLdI0ZcZ2wdkLodujagy0RJJAsSHw4PbXtspMhtcqtMtck/uJjDGXa
+  ENxXeRgEnU9vMzjB/gB1JwKBgDIttAvhcEo+q/JKtAx01K5oeQURetGwiSHbJHDq
+  ITyS4PolOLNzygKxw29iFpmNMjb9TT44BGof24i/2cm817i+g66ILNq0Fe4liNZm
+  j2M31Qdg7JQI0Gs6nvTjIALMS3dMzB51inrKDd1x9lTRg7c3n2Mm1pBpgL0ULdEG
+  xnwjAoGAD+S5jsF2m0amYbR3pUwWFHmThXhB+ud6Sx8WVt4lL4zPA0doh8yTrt0h
+  srNaVWjGC6DHW2u4f1ArkGB3Csi/RtYFGV06V2j9Z5vgEf/1BCW0peiLqsiUQfRo
+  zsZ7pCigQzaSJDYjkvUXibgXHK7cuZ+D6CdQVy1QGuxWsP4lzc0=
+  -----END RSA PRIVATE KEY-----

data/ci/tasks/run-tests-dev.yml

@@ -0,0 +1,10 @@
+---
+platform: linux
+image_resource:
+  type: docker-image
+  source: {repository: "allomov/ruby", tag: "manifest-generator"}
+inputs:
+  - name: merrill-manifest-generator-dev
+run:
+  path: ci/scripts/run-tests
+  dir: merrill-manifest-generator-dev/manifest-generator

data/ci/tasks/run-tests-stable.yml

@@ -0,0 +1,13 @@
+---
+platform: linux
+image_resource:
+  type: docker-image
+  source: {repository: "allomov/ruby", tag: "manifest-generator"}
+inputs:
+  - name: merrill-manifest-generator-stable
+run:
+  # path: ci/scripts/run-tests
+  # dir: merrill-manifest-generator-stable/manifest-generator
+  path: bash
+  args:
+    - "ci/scripts/run-tests"

data/lib/mantra/certificate.rb

@@ -0,0 +1,35 @@
+module Mantra
+  module Certificate
+    class DigitalCertificate
+      TEMPLATE_MESSAGE = "Difital Certificate from file:"
+      attr_accessor :path, :type, :value
+      def initialize(options)
+        self.path  = options[:path]
+        self.type  = options[:type].strip
+        self.value = options[:value].strip
+      end
+
+      def private_key?
+        self.type.include? "PRIVATE KEY"
+      end
+
+      def public_key?
+        self.type == "PUBLIC KEY"
+      end
+
+      def certificate?
+        self.type == "CERTIFICATE"
+      end
+
+      def scope
+        self.path.to_scope
+      end
+
+    end
+  end
+end
+
+# require "mantra/certificates/string_ext"
+require "mantra/certificates/cortege"
+require "mantra/certificates/group"
+# require "mantra/certificates"

data/lib/mantra/certificates/cortege.rb

@@ -0,0 +1,64 @@
+require "tempfile"
+
+module Mantra
+  class Certificates
+    class Cortege
+
+      attr_accessor :public_key, :private_key, :certificate
+      def initialize(options)
+        self.public_key  = options[:public_key]
+        self.private_key = options[:private_key]
+        self.certificate = options[:certificate]
+      end
+
+      def matches?
+        if self.has_private_key? && self.has_certificate?
+          return private_key_matches_certificate?
+        elsif self.has_private_key? && self.has_public_key?
+          return private_key_matches_public_key?
+        else
+          puts "Warning: don't know what to do with following certificate cortege #{self.inspect}"
+          return false
+        end
+      end
+
+      def has_certificate?
+        !!self.certificate
+      end
+
+      def has_public_key?
+        !!self.public_key
+      end
+
+      def has_private_key?
+        !!self.private_key
+      end
+
+      private
+
+      # https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs
+      # https://www.sslshopper.com/article-most-common-openssl-commands.html
+      # TODO: find out if there are ruby wrappers for this calls
+      def private_key_matches_certificate?
+        private_key_file_path = temp_file_wrapper(self.private_key.value, "domain.key")
+        certificate_file_path = temp_file_wrapper(self.certificate.value, "domain.crt")
+        private_key_md5 = `openssl rsa -noout -modulus -in #{private_key_file_path} 2> /dev/null | openssl md5`
+        certificate_md5 = `openssl x509 -noout -modulus -in #{certificate_file_path} 2> /dev/null | openssl md5`
+        private_key_md5 == certificate_md5
+      end
+
+      def private_key_matches_public_key?
+        private_key_file_path  = temp_file_wrapper(self.private_key.value, "domain.key")
+        private_key_public_key = `openssl rsa -in #{private_key_file_path} -pubout 2> /dev/null`
+        private_key_public_key.strip == self.public_key.value
+      end
+
+      def temp_file_wrapper(string, filename)
+        file = Tempfile.new(filename)
+        file.write(string)
+        file.close
+        file.path
+      end
+    end
+  end
+end

data/lib/mantra/certificates/group.rb

@@ -0,0 +1,26 @@
+require "forwardable"
+
+# this class acts as it was simple DigitalCertificate
+module Mantra
+  class Certificates
+    class Group
+      extend Forwardable
+
+      attr_accessor :certificates
+      alias_method :collection, :certificates
+
+      def_delegators :@certificates, :sample
+      def_delegators :sample, :certificate?, :public_key?, :private_key?, :value
+
+
+      def initialize(certificate_list)
+        self.certificates = certificate_list
+      end
+
+      def private_keys
+        self.certificates.select { |c| c.private_key? }
+      end
+
+    end
+  end
+end

data/lib/mantra/command.rb

@@ -0,0 +1,51 @@
+require "optparse"
+
+module Mantra
+  class Command
+    include Helpers::ObjectWithType
+
+    def self.option_descriptors
+      @options ||= []
+    end
+
+    def self.usage
+      "\n\tUsage:  \tmantra <action> [options]\n" +
+      "\tactions:\t#{self.subclasses.map { |s| s.type }.join(", ")}\n\n"
+    end
+
+    def self.option(name, long_option, short_option, description)
+      self.option_descriptors << [name, long_option, short_option, description]
+      self.send(:define_method, name) do
+        @options[name.to_s]
+      end
+    end
+
+    def run
+      parse_options
+      perform
+    end
+
+    def initialize(options)
+      @args = options[:args]
+    end
+
+    attr_accessor :options
+    def parse_options
+      @options = {}
+      OptionParser.new do |options_parser|
+        options_parser.banner = "Usage: mantra #{} [options]"
+        self.class.option_descriptors.each do |option|
+          option_name = option.shift.to_s
+          options_parser.on(*option) do |value|
+            @options[option_name] = value
+          end
+        end
+      end.parse!
+      @options
+    end
+
+  end
+end
+
+require "mantra/commands/find"
+require "mantra/commands/transform"

data/lib/mantra/commands/find.rb

@@ -0,0 +1,38 @@
+require "yaml"
+require "json"
+
+module Mantra
+  class Commands
+    class Find < Command
+      type :find
+      alias_type :f
+      option :scope,       "--scope scope-path",       "-s scope",         "path scope you want to find"
+      option :manifest,    "--manifest manifest-file", "-m manifest-file", "manifest"
+      option :format,      "--format format",          "-f format",        "format (yaml or json)"
+      option :with_parent, "--with-parent",            "-p",               "format (yaml or json)"
+
+      def perform
+        m = Manifest.new(manifest)
+        results = m.select(scope)
+
+        results = results.map do |element|
+          element.parent
+        end if with_parent
+
+        results = results.map do |element|
+          element.to_ruby_object
+        end
+
+        case format
+        when "yaml", "yml", "y", nil
+          puts results.to_yaml
+        when "json", "j"
+          puts JSON.pretty_generate(results)
+        else
+          raise "unknown format"
+        end
+      end
+
+    end
+  end
+end

data/lib/mantra/commands/transform.rb

@@ -0,0 +1,38 @@
+require "yaml"
+
+module Mantra
+  class Commands
+    class Transform < Command
+      type :transform
+      alias_type :t
+      attr_accessor :manifest
+
+      option :transform_config_path, "--config CONFIG",     "-c", "Transformation config path"
+      option :manifest_path,         "--manifest MANIFEST", "-m", "Manifest path"
+
+      def transform_config
+        @transform_config ||= YAML.load_file(transform_config_path)
+      end
+
+      def transforms
+        previous_transform = self
+        transform_config["transforms"].map do |t|
+          t.merge!(previous_transform: previous_transform)
+          previous_transform = Mantra::Transform.create(t)
+        end
+      end
+
+      def manifest
+        @manifest ||= Manifest.new(self.manifest_path)
+      end
+
+      def perform
+        transforms.each do |t|
+          t.run
+        end
+        puts manifest.to_ruby_object.to_yaml
+      end
+
+    end
+  end
+end