cps-property-generator 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b480ea73120071abacc074c6279e1da3b9f62d53
+  data.tar.gz: e2092c1a6218e51058d1ff049cca4e4bc8f19ddd
+SHA512:
+  metadata.gz: f8ceb169740e2ebfb3c0d26379f690507954b166a674081b60080556ecc7994ae9ad4aa0f8856cc4748e81ed89c8730a27e185da864a7679532377d77a642e3b
+  data.tar.gz: b9d8808e6608a4dc7dd0f5625401fba02083d8141f3aca8e97e9472d0246c7ed494f559c5450e3d26dc6b318eaa258a7881fb53719c6e7315e65cdea41a7f682

data/README.md

@@ -0,0 +1,115 @@
+## **Getting Started**
+
+#### Creating your properties project:
+##### Step 1: Creating your config.yml
+1. Create your top level directory for your property project.
+
+    `mkdir example-properties`
+
+2. Create a config directory and inside a config.yml file.
+
+```sh
+cd example-properties/
+mkdir config
+cd config
+touch config.yml
+```
+
+The config.yml will define essential configurations that the generator needs to create the property json files.
+In the config.yml file we need three keys set to explain our project to the generator.
+
+1. The `environments` key. This key will define a array of the environments that we are setting properties for. Environments must be unique and have a one to one mapping with amazon aws regions.
+2. The `accounts` key. This key will define a array of amazon accounts properties will be uploaded to and served in. 
+3. The `environment_configs` key. This key will define three things. 
+    * The one to one mapping or aws regions to environments.
+    * The account a environment lives in.
+    * Interpolations for a given environment. Interpolations will be explained in a separate section. 
+    
+Here is a example config.yml
+```yaml
+environments:
+  - 'my-test-env1'
+  - 'my-test-env2'
+
+accounts:
+  - 123456789012
+  - 987654321098
+
+environment_configs:
+  my-test-env1:
+    region: 'us-east-1'
+    account: 123456789012
+    interpolations:
+      region: 'us-east-1'
+      cloud: 'test-cloud-1'
+      domain: 'my1.com'
+  my-test-env2:
+    region: 'eu-central-1'
+    account: 987654321098
+    interpolations:
+      region: 'eu-central-1'
+      cloud: 'test-cloud-2'
+      domain: 'my2.com'
+```
+    
+##### Step 2: Creating your globals
+Globals are properties that get mixed in with service definitions. The hierarchy of the global definition sets the ruling for what that property can supersede. 
+The globals supersedence order is as follows. Any Environmental globals will overwrite account globals. The resultant merge of environmental and account globals overwrite definitions in the top level globals.yml. In short
+Superscedence from least to greatest is globals.yml, account globals, environment globals. To define globals follow the steps below.
+###### Top level globals
+1. Create a globals folder
+```sh
+cd example-properties/
+mkdir globals
+```
+2. If you would like top level globals then create a `globals.yml` in your globals folder.
+```sh
+cd globals/
+touch globals.yml
+```
+3. Define your yaml values in the globals.yml
+###### Account globals
+1. Create a folder called the account id of your aws account.
+2. In the folder created above create an YAML file named after the account id. 
+3. Define your account level globals in the YAML file you created above.
+###### Environment Globals
+1. Create a folder called `environments` inside your folder named after your account.
+2. Inside the `environments` folder create a yaml file named after the environment you would like to define globals for. Only environments defined in your config are supported. The environment must also be mapped in the config to the account the sharing the same name as the folder the environment global yaml file lives in. 
+3. In the newly created environment's yaml file you may define your globals.
+
+##### Step 3: Creating your service definitions
+Service definitions have the highest level of supersedence and will overwrite matching global definitions. 
+To create you service definitions you need to create the services folder and then the services yaml files.
+```sh
+cd example-properties/
+mkdir services
+cd services
+touch my-service-name.yml
+```
+Service definitions consist of three parts `default`, `environments`, and `encrypted`. Encrypted definitions overwrite environment definitions which will overwrite default definitions. Here is a example of a service file. The name of your service file MUST be the same as your service. 
+```yaml
+default:
+  database.host: 'my.database.{domain}'
+  database.port: 3306
+
+environments:
+  my-test-env-1:
+    thread.pool.size: 12
+  my-test-env-2:
+    thread.pool.size: 8
+
+encrypted:
+  my-test-env-1:
+   my.encrypted.property:
+    $ssm:
+      region: us-east-1
+      encrypted: PRETEND_ENCRYPTED_PROPERTY_CIPHERTEXT
+```
+###### Adding interpolations
+An interpolation is a  value that will be dynamically substituted during generation with the correct value for the environment being generated. Interpolations are declared in the config for a given environment. Once declared an interpolation can be used in a property definition by referencing it in braces. If we were to reference the domain interpolation from the example config above we would use `{domain}`.
+
+##### Step 4: Generating Your Properties (Using the CLI)
+The bin directory contains the generator.rb cli. An example of running the cli is below. The `project_path` argument specifies the path to the properties repo we are generating a uploading properties from. You must be able to create a session with s3 to upload.
+```sh
+./generator.rb generate --project_path "~/projects/project-properties" --upload true --upload_account "123456789012" --upload_region "us-east-1" --upload_bucket "propertiesbucket.my-cloud.com"
+```

data/bin/cps-property-generator

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+#
+require 'thor'
+require 'yaml'
+require_relative '../lib/generator/generator.rb'
+require_relative '../lib/linter/linter.rb'
+class GeneratorCLI < ::Thor
+
+  desc 'generate', 'Generate properties'
+  option 'project_path', banner: 'PROJECT_PATH', type: :string, desc: 'Path to the property project to generate properties for'
+  option 'output', banner: 'OUTPUT', type: :string, desc: 'Output path for locally dumping generated outputs', :default => '/tmp/'
+  option 'upload', banner: 'UPLOAD', type: :boolean, desc: 'Whether to upload or not', :default => false
+  option 'upload_account', banner: 'UPLOAD_ACCOUNT', type: :string, desc: 'The account you are uploading properties to'
+  option 'upload_region', banner: 'UPLOAD_REGION', type: :string, desc: 'The region your property bucket is in'
+  option 'upload_bucket', banner: 'UPLOAD_BUCKET', type: :string, desc: 'The bucket you are uploading properties to.'
+
+  def generate
+
+    generator = PropertyGenerator::Generator.new(options)
+    out = generator.generate
+    if options['upload']
+      if options['upload_account'].nil? || options['upload_region'].nil? && options['upload_bucket'].nil?
+        puts 'Please specify upload configs'
+      else
+        generator.upload(out, options)
+      end
+
+    end
+  end
+
+  desc 'lint', 'Lint YAML files for properties repo'
+  option 'project_path', banner: 'PROJECT_PATH', type: :string, desc: 'Path to the property project to generate properties for'
+  def lint
+    linter = PropertyGenerator::Linter.new(options['project_path'])
+    linter.run_tests
+    linter.display_report
+    if linter.fail_check
+      return 1
+    else
+      return 0
+    end
+  end
+end
+
+GeneratorCLI.start(ARGV)

data/lib/generator/config.rb

@@ -0,0 +1,32 @@
+module PropertyGenerator
+  require 'yaml'
+  class Config
+
+    attr_accessor :configs
+
+    def initialize(project_path)
+      @project_path = project_path
+    end
+
+    def configs
+      @configs ||= read_configs
+    end
+
+    def environments
+      configs['environments']
+    end
+
+    def accounts
+      configs['accounts']
+    end
+
+    def environment_configs
+      configs['environment_configs']
+    end
+
+    def read_configs
+      file = "#{@project_path}/config/config.yml"
+      YAML.load_file(file)
+    end
+  end
+end

data/lib/generator/generator.rb

@@ -0,0 +1,50 @@
+require_relative 'config'
+require_relative 'globals'
+require_relative 'service'
+require_relative 'helpers'
+
+module PropertyGenerator
+  class Generator
+    require 'fileutils'
+    require 'securerandom'
+    include PropertyGenerator
+    # purpose: initialize globals and configs
+    # serve as a broker between tasks
+    def initialize(options)
+      project_path =  File.expand_path(options['project_path'])
+      @configs = PropertyGenerator::Config.new(project_path)
+      @globals = PropertyGenerator::Globals.new(project_path, @configs)
+      @globals = @globals.globals
+      @output_path =  "#{File.expand_path(options['output'])}/properties/#{SecureRandom.hex}"
+      puts "Properties will be output here #{@output_path}"
+      @service_list = PropertyGenerator.read_services(project_path)
+
+    end
+
+    def generate
+      output = []
+      @service_list.each do | service, path|
+        service_instance = PropertyGenerator::Service.new(YAML.load_file(path), @configs, @globals)
+        service_instance.service
+        service_instance.interpolate
+
+        out = PropertyGenerator.writer(service, service_instance.service, @configs, @output_path)
+        (output << out).flatten!
+      end
+      output
+    end
+
+    def upload(out, config)
+      upload_account = config['upload_account']
+      upload_region = config['upload_region']
+      upload_bucket = config['upload_bucket']
+      out.each do |file|
+        next unless file.include?("#{upload_account}") && file.include?("#{upload_region}")
+         file_region = file.split("/")[-2]
+         PropertyGenerator.sync(upload_region, upload_account, upload_bucket, file, file_region)
+      end
+    end
+
+  end
+end
+

data/lib/generator/globals.rb

@@ -0,0 +1,85 @@
+module PropertyGenerator
+  class Globals
+    require 'yaml'
+    attr_accessor :globals
+
+    def initialize(project_path, config)
+      @project_path = project_path
+      @environments = config.environments
+      @environment_configs = config.environment_configs
+      @accounts = config.accounts
+    end
+
+    def globals
+      @globals ||= condense_globals
+    end
+
+
+    def get_main_global
+      top_level = {}
+      if File.exists?("#{@project_path}/globals/globals.yml")
+        top_level = YAML.load_file("#{@project_path}/globals/globals.yml")
+      end
+      top_level
+    end
+
+    def get_account_globals
+      data = {}
+      @accounts.each do |account|
+        next unless Dir.exists?("#{@project_path}/globals/accounts/#{account}")
+        account_default_file = "#{@project_path}/globals/accounts/#{account}/#{account}.yml"
+        data[account] = YAML.load_file(account_default_file) if File.exists?(account_default_file)
+      end
+      data
+    end
+
+    def get_environment_globals
+      data = {}
+      @accounts.each do |account|
+        next unless Dir.exists?("#{@project_path}/globals/accounts/#{account}/environments")
+        data[account] = {}
+        @environments.each do |env|
+          next unless File.exists?("#{@project_path}/globals/accounts/#{account}/environments/#{env}.yml")
+          data[account][env] = YAML.load_file("#{@project_path}/globals/accounts/#{account}/environments/#{env}.yml")
+        end
+      end
+      data
+    end
+
+
+    #merge environment globals with account globals.
+    def condense_globals
+      condensed = {}
+      # get account and the environmental hash's for said account
+      environment_globals = get_environment_globals
+      account_globals = get_account_globals
+      main_global = get_main_global
+      # nothing to do here if everything is empty
+      return condensed if environment_globals.empty? && account_globals.empty? && main_global.empty?
+
+      environment_globals.each do |account, env_global |
+        # get the env and the values
+        env_global.each do |env, hash|
+          account_globals[account] ||= {}
+          # set the environment globals to be the account global merged with the env globals
+          env_global[env] = account_globals[account].merge(hash) unless hash.empty?
+          condensed[env] = env_global[env]
+        end
+      end
+
+      unless main_global.empty?
+        # All environments need the main global definitions
+        @environments.each do |env|
+          # If a key/value pair for a environment has not been defined set one so we can merge
+          condensed[env] ||= {}
+          # We need to merge into the globals so any env configs overwrite main global configs.
+          # Dup so we dont modify the original object
+          main_global_dup = main_global.dup
+          condensed[env] = main_global_dup.merge(condensed[env])
+        end
+      end
+      condensed
+    end
+
+  end
+end

data/lib/generator/service.rb

@@ -0,0 +1,89 @@
+module PropertyGenerator
+  class Service
+    attr_accessor :service
+    def initialize(service_data, config, globals)
+      @service_data = service_data
+      @environments = config.environments
+      @globals = globals
+      @environment_configs = config.environment_configs
+      set_service
+    end
+
+    def set_service
+      service_data = merge_env_default(@service_data, @environments)
+      @service = merge_service_with_globals(@globals, service_data, @environments)
+    end
+
+    def service
+      @service
+    end
+
+    def interpolate
+      environments = @environments
+      #read in config
+      #interate through environment and substitute config for values for that environment
+      environments.each do | env|
+        #get the map of config for a env
+        interpolations = @environment_configs[env]['interpolations']
+
+        #interate through the properties for an environment and gsub the config
+        @service[env].each do | property_key, property_value|
+          property_value_dup = property_value.dup
+          interpolations.each do |matcher_key, matcher_value|
+            if property_value.class == String && property_value_dup.include?("{#{matcher_key}}")
+              @service[env][property_key] = property_value_dup.gsub!("{#{matcher_key}}", matcher_value)
+            end
+          end
+        end
+      end
+      service
+    end
+
+    def merge_env_default(data, environments)
+      #creates a hash of the enviornments merged with the defaults
+      # {service => {env1 =>  {properties},
+      #             env2 => {properties}
+      #           }
+      # }
+      output = {}
+      default = data['default']
+
+      environments.each do |env|
+        default_clone = default.dup
+        #if nil, use set to environments as nothing to merge env with
+        data['environments'] ||= {}
+        data['environments'][env] ||= {}
+        environment_data = data['environments'][env].dup
+        if data['encrypted']
+          encrypted = data['encrypted'][env].dup unless data['encrypted'][env].nil?
+          environment_data = data['environments'][env].merge(encrypted) unless encrypted.nil?
+        end
+        if default_clone.nil?
+          merged = environment_data
+        else
+          merged = default_clone.merge(environment_data)
+        end
+        output[env] = merged
+      end
+      output
+    end
+
+    def merge_service_with_globals(globals_data, service_data, environments)
+      #service will now overwrite globals, merging will be done for each environment
+      output = {}
+      envs = environments
+      envs.each do |env|
+        globals_clone = globals_data.dup
+        if globals_clone[env].nil? || globals_clone[env] == false
+          merged = service_data[env]
+        else
+          merged = globals_clone[env].merge(service_data[env])
+        end
+        output[env] = merged
+      end
+      output
+    end
+
+
+  end
+end

data/lib/helpers/helpers.rb

@@ -0,0 +1,81 @@
+module PropertyGenerator
+  require 'json'
+  require 'fileutils'
+  require 'aws-sdk-s3'
+  class << self
+
+    def test_runner(object, test_list)
+      results = {}
+      test_list.each do |test|
+        results[test] = object.send(test)
+      end
+      results
+    end
+
+    def get_list_of_files(path, ignore_list)
+      #Returns a list of files in given path
+      #Ignores files in a given ignore list
+      Dir.glob(path + "/**/*").select{ |e| File.file? e unless ignore_list.include?(e.split('/')[(e.split('/')).length - 1])}
+    end
+
+    def valid_paths(path)
+      valid_paths = []
+      list_of_file_paths = get_list_of_files(path, [])
+      list_of_file_paths.each do |file_path|
+        begin
+          YAML.load_file(file_path)
+          valid_paths << file_path
+        rescue
+          next
+        end
+      end
+      valid_paths
+    end
+
+    def invalid_paths(path, ignore_list)
+      invalid_paths = []
+      list_of_file_paths = get_list_of_files(path, ignore_list)
+      list_of_file_paths.each do |file_path|
+        begin
+          YAML.load_file(file_path)
+        rescue
+          invalid_paths << file_path
+        end
+      end
+      invalid_paths
+    end
+
+    def read_services(path)
+      Dir.glob("#{path}/services/*.{yaml,yml}").each_with_object({}) do |file, acc|
+        name = File.basename(file)[/(?<service>.*)\.ya?ml$/, :service]
+        path = File.absolute_path(file)
+        acc[name] = path
+      end
+    end
+
+    def writer(service_name, finalized, configs, output_path)
+      output = []
+      envs = configs.environments
+      environmental_configs =  configs.environment_configs
+      envs.each do | env|
+        account = environmental_configs[env]["account"]
+        region = environmental_configs[env]["region"]
+        json = JSON.pretty_generate({"properties" => finalized[env]})
+        FileUtils.mkdir_p("#{output_path}/#{account}/#{region}/") unless Dir.exist?("#{output_path}/#{account}/#{region}/")
+        File.write("#{output_path}/#{account}/#{region}/#{service_name}.json", json)
+        output << "#{output_path}/#{account}/#{region}/#{service_name}.json"
+      end
+      output
+    end
+
+    def sync(region, account, bucket, file, file_region)
+      s3 = Aws::S3::Resource.new(region: region)
+      filename = file.split("/").last
+      puts "Destination: #{account}/#{file_region}/#{filename}"
+      puts "Uploading: #{file}"
+      obj = s3.bucket(bucket).object("#{account}/#{file_region}/#{filename}")
+      obj.upload_file(file)
+    end
+
+  end
+end