cpl 2.0.1 → 2.1.0

data/docs/tips.md

@@ -3,7 +3,7 @@
 1. [GVCs vs. Orgs](#gvcs-vs-orgs)
 2. [RAM](#ram)
 3. [Remote IP](#remote-ip)
-4. [ENV Values](#env-values)
+4. [Secrets and ENV Values](/docs/secrets-and-env-values.md)
 5. [CI](#ci)
 6. [Memcached](#memcached)
 7. [Sidekiq](#sidekiq)
@@ -70,45 +70,6 @@ pick those up and automatically populate `request.remote_ip`.
 
 So `REMOTE_ADDR` should not be used directly, only `request.remote_ip`.
 
-## ENV Values
-
-You can store ENV values used by a container (within a workload) within Control Plane at the following levels:
-
-1. Workload Container
-2. GVC
-
-For your "review apps," it is convenient to have simple ENVs stored in plain text in your source code. You will want to
-keep some ENVs, like the Rails' `SECRET_KEY_BASE`, out of your source code. For staging and production apps, you will
-set these values directly at the GVC or workload levels, so none of these ENV values are committed to the source code.
-
-For storing ENVs in the source code, we can use a level of indirection so that you can store an ENV value in your source
-code like `cpln://secret/my-app-review-env-secrets.SECRET_KEY_BASE` and then have the secret value stored at the org
-level, which applies to your GVCs mapped to that org.
-
-You can do this during the initial app setup, like this:
-
-1. Add the templates for `app` and `secrets` to `.controlplane/templates`
-2. Ensure that the `app` template includes the `identity`
-3. Ensure that the `app` template is listed in `setup_app_templates` for the app in `.controlplane/controlplane.yml`
-4. Run `cpl apply-template secrets -a $APP_NAME` (one-time setup)
-5. Run `cpl setup-app -a $APP_NAME`
-6. The secrets, secrets policy and identity will be automatically created, along with the proper binding
-7. In the Control Plane console, upper left "Manage Org" menu, click on "Secrets"
-8. Find the created secret (it will be in the `$APP_PREFIX-secrets` format) and add the secret env vars there
-9. Use `cpln://secret/...` in the app to access the secret env vars (e.g., `cpln://secret/$APP_PREFIX-secrets.SOME_VAR`)
-
-Here are the manual steps for reference. We recommend that you follow the steps above:
-
-1. In the upper left of the Control Plane console, "Manage Org" menu, click on "Secrets"
-2. Create a secret with `Secret Type: Dictionary` (e.g., `my-secrets`) and add the secret env vars there
-3. In the upper left "Manage GVC" menu, click on "Identities"
-4. Create an identity (e.g., `my-identity`)
-5. Navigate to the workload that you want to associate with the identity created
-6. Click "Identity" on the left menu and select the identity created
-7. In the lower left "Access Control" menu, click on "Policies"
-8. Create a policy with `Target Kind: Secret` and add a binding with the `reveal` permission for the identity created
-9. Use `cpln://secret/...` in the app to access the secret env vars (e.g., `cpln://secret/my-secrets.SOME_VAR`)
-
 ## CI
 
 **Note:** Docker builds much slower on Apple Silicon, so try configuring CI to build the images when using Apple

data/examples/controlplane.yml

@@ -31,9 +31,6 @@ aliases:
     # 2. Each file can contain many objects, such as in the case of templates that create a resource, like `postgres`.
     # 3. While the naming often corresponds to a workload or other object name, the naming is arbitrary. 
     #    Naming does not need to match anything other than the file name without the `.yml` extension.
-    #
-    # If you're going to use secrets, you need to apply the `secrets.yml` template separately (one-time setup):
-    # `cpl apply-template secrets -a my-app`
     setup_app_templates:
       - app
       - redis
@@ -42,6 +39,9 @@ aliases:
       - rails
       - sidekiq
 
+    # Uncomment next line to skips secrets setup when running `cpl setup-app`.
+    # skip_secrets_setup: true
+
     # Only needed if using a custom secrets name.
     # The default is '{APP_PREFIX}-secrets'. For example:
     # - for an app 'my-app-staging' with `match_if_app_name_starts_with` set to `false`,
@@ -108,6 +108,15 @@ apps:
     # e.g., "my-app-review-pr123", "my-app-review-anything-goes", etc.
     match_if_app_name_starts_with: true
 
+    # Hooks can be either a script path that exists in the app image or a command.
+    # They're run in the context of `cpl run` with the latest image.
+    hooks:
+      # Used by the command `cpl setup-app` to run a hook after creating the app.
+      post_creation: bundle exec rake db:prepare
+
+      # Used by the command `cpl delete` to run a hook before deleting the app.
+      pre_deletion: bundle exec rake db:drop
+
   my-app-production:
     <<: *common
 

data/lib/command/apply_template.rb

@@ -8,7 +8,8 @@ module Command
     OPTIONS = [
       app_option(required: true),
       location_option,
-      skip_confirm_option
+      skip_confirm_option,
+      add_app_identity_option
     ].freeze
     DESCRIPTION = "Applies application-specific configs from templates"
     LONG_DESCRIPTION = <<~DESC
@@ -39,45 +40,27 @@ module Command
       cpl apply-template app postgres redis rails -a $APP_NAME
       ```
     EX
+    VALIDATIONS = %w[config templates].freeze
+
+    def call # rubocop:disable Metrics/MethodLength
+      @template_parser = TemplateParser.new(config)
+      @names_to_filenames = config.args.to_h do |name|
+        [name, @template_parser.template_filename(name)]
+      end
 
-    def call # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
       ensure_templates!
 
       @created_items = []
       @failed_templates = []
       @skipped_templates = []
 
-      @asked_for_confirmation = false
-
-      pending_templates = templates.select do |template|
-        if template == "app"
-          confirm_app(template)
-        else
-          confirm_workload(template)
-        end
-      end
-
-      progress.puts if @asked_for_confirmation
-
-      @deprecated_variables = []
-
-      pending_templates.each do |template, filename|
-        step("Applying template '#{template}'", abort_on_error: false) do
-          items = apply_template(filename)
-          unless items
-            report_failure(template)
-            next false
-          end
-
-          items.each do |item|
-            report_success(item)
-          end
-          true
-        end
+      templates = @template_parser.parse(@names_to_filenames.values)
+      pending_templates = confirm_templates(templates)
+      add_app_identity_template(pending_templates) if config.options[:add_app_identity]
+      pending_templates.each do |template|
+        apply_template(template)
       end
 
-      warn_deprecated_variables
-
       print_created_items
       print_failed_templates
       print_skipped_templates
@@ -87,18 +70,21 @@ module Command
 
     private
 
-    def templates
-      @templates ||= config.args.to_h do |template|
-        [template, "#{config.app_cpln_dir}/templates/#{template}.yml"]
+    def template_kind(template)
+      case template["kind"]
+      when "gvc"
+        "app"
+      else
+        template["kind"]
       end
     end
 
     def ensure_templates!
-      missing_templates = templates.reject { |_template, filename| File.exist?(filename) }.to_h
+      missing_templates = @names_to_filenames.reject { |_, filename| File.exist?(filename) }
       return if missing_templates.empty?
 
-      missing_templates_str = missing_templates.map do |template, filename|
-        "  - #{template} (#{filename})"
+      missing_templates_str = missing_templates.map do |name, filename|
+        "  - #{name} (#{filename})"
       end.join("\n")
       progress.puts("#{Shell.color('Missing templates:', :red)}\n#{missing_templates_str}\n\n")
 
@@ -113,10 +99,10 @@ module Command
     end
 
     def confirm_app(template)
-      app = cp.fetch_gvc
+      app = cp.fetch_gvc(template["name"])
       return true unless app
 
-      confirmed = confirm_apply("App '#{config.app}' already exists, do you want to re-create it?")
+      confirmed = confirm_apply("App '#{template['name']}' already exists, do you want to re-create it?")
       return true if confirmed
 
       report_skipped(template)
@@ -124,63 +110,67 @@ module Command
     end
 
     def confirm_workload(template)
-      workload = cp.fetch_workload(template)
+      workload = cp.fetch_workload(template["name"])
       return true unless workload
 
-      confirmed = confirm_apply("Workload '#{template}' already exists, do you want to re-create it?")
+      confirmed = confirm_apply("Workload '#{template['name']}' already exists, do you want to re-create it?")
       return true if confirmed
 
       report_skipped(template)
       false
     end
 
-    def apply_template(filename) # rubocop:disable Metrics/MethodLength
-      data = File.read(filename)
-                 .gsub("{{APP_ORG}}", config.org)
-                 .gsub("{{APP_NAME}}", config.app)
-                 .gsub("{{APP_LOCATION}}", config.location)
-                 .gsub("{{APP_LOCATION_LINK}}", app_location_link)
-                 .gsub("{{APP_IMAGE}}", latest_image)
-                 .gsub("{{APP_IMAGE_LINK}}", app_image_link)
-                 .gsub("{{APP_IDENTITY}}", app_identity)
-                 .gsub("{{APP_IDENTITY_LINK}}", app_identity_link)
-                 .gsub("{{APP_SECRETS}}", app_secrets)
-                 .gsub("{{APP_SECRETS_POLICY}}", app_secrets_policy)
+    def confirm_templates(templates) # rubocop:disable Metrics/MethodLength
+      @asked_for_confirmation = false
+
+      pending_templates = templates.select do |template|
+        case template["kind"]
+        when "gvc"
+          confirm_app(template)
+        when "workload"
+          confirm_workload(template)
+        else
+          true
+        end
+      end
 
-      find_deprecated_variables(data)
+      progress.puts if @asked_for_confirmation
+
+      pending_templates
+    end
 
-      # Kept for backwards compatibility
-      data = data
-             .gsub("APP_ORG", config.org)
-             .gsub("APP_GVC", config.app)
-             .gsub("APP_LOCATION", config.location)
-             .gsub("APP_IMAGE", latest_image)
+    def add_app_identity_template(templates)
+      app_template_index = templates.index { |template| template["name"] == config.app }
+      app_identity_template_index = templates.index { |template| template["name"] == config.identity }
 
-      # Don't read in YAML.safe_load as that doesn't handle multiple documents
-      cp.apply_template(data)
+      return unless app_template_index && app_identity_template_index.nil?
+
+      # Adding the identity template right after the app template is important since:
+      # a) we can't create the identity at the beginning because the app doesn't exist yet
+      # b) we also can't create it at the end because any workload templates associated with it will fail to apply
+      templates.insert(app_template_index + 1, build_app_identity_hash)
     end
 
-    def new_variables
+    def build_app_identity_hash
       {
-        "APP_ORG" => "{{APP_ORG}}",
-        "APP_GVC" => "{{APP_NAME}}",
-        "APP_LOCATION" => "{{APP_LOCATION}}",
-        "APP_IMAGE" => "{{APP_IMAGE}}"
+        "kind" => "identity",
+        "name" => config.identity
       }
     end
 
-    def find_deprecated_variables(data)
-      @deprecated_variables.push(*new_variables.keys.select { |old_key| data.include?(old_key) })
-      @deprecated_variables = @deprecated_variables.uniq.sort
-    end
-
-    def warn_deprecated_variables
-      return unless @deprecated_variables.any?
+    def apply_template(template) # rubocop:disable Metrics/MethodLength
+      step("Applying template for #{template_kind(template)} '#{template['name']}'", abort_on_error: false) do
+        items = cp.apply_hash(template)
+        unless items
+          report_failure(template)
+          next false
+        end
 
-      message = "Please replace these variables in the templates, " \
-                "as support for them will be removed in a future major version bump:"
-      deprecated = @deprecated_variables.map { |old_key| "  - #{old_key} -> #{new_variables[old_key]}" }.join("\n")
-      progress.puts("\n#{Shell.color("DEPRECATED: #{message}", :yellow)}\n#{deprecated}")
+        items.each do |item|
+          report_success(item)
+        end
+        true
+      end
     end
 
     def report_success(item)
@@ -205,14 +195,14 @@ module Command
     def print_failed_templates
       return unless @failed_templates.any?
 
-      failed = @failed_templates.map { |template| "  - #{template}" }.join("\n")
+      failed = @failed_templates.map { |template| "  - [#{template_kind(template)}] #{template['name']}" }.join("\n")
       progress.puts("\n#{Shell.color('Failed to apply templates:', :red)}\n#{failed}")
     end
 
     def print_skipped_templates
       return unless @skipped_templates.any?
 
-      skipped = @skipped_templates.map { |template| "  - #{template}" }.join("\n")
+      skipped = @skipped_templates.map { |template| "  - [#{template_kind(template)}] #{template['name']}" }.join("\n")
       progress.puts("\n#{Shell.color('Skipped templates (already exist):', :blue)}\n#{skipped}")
     end
   end

data/lib/command/base.rb

@@ -8,6 +8,10 @@ module Command
 
     include Helpers
 
+    VALIDATIONS_WITHOUT_ADDITIONAL_OPTIONS = %w[config].freeze
+    VALIDATIONS_WITH_ADDITIONAL_OPTIONS = %w[templates].freeze
+    ALL_VALIDATIONS = VALIDATIONS_WITHOUT_ADDITIONAL_OPTIONS + VALIDATIONS_WITH_ADDITIONAL_OPTIONS
+
     # Used to call the command (`cpl NAME`)
     # NAME = ""
     # Displayed when running `cpl help` or `cpl help NAME` (defaults to `NAME`)
@@ -32,8 +36,8 @@ module Command
     HIDE = false
     # Whether or not to show key information like ORG and APP name in commands
     WITH_INFO_HEADER = true
-
-    NO_IMAGE_AVAILABLE = "NO_IMAGE_AVAILABLE"
+    # Which validations to run before the command
+    VALIDATIONS = %w[config].freeze
 
     def initialize(config)
       @config = config
@@ -269,6 +273,7 @@ module Command
     def self.skip_secret_access_binding_option(required: false)
       {
         name: :skip_secret_access_binding,
+        new_name: :skip_secrets_setup,
         params: {
           desc: "Skips secret access binding",
           type: :boolean,
@@ -277,6 +282,17 @@ module Command
       }
     end
 
+    def self.skip_secrets_setup_option(required: false)
+      {
+        name: :skip_secrets_setup,
+        params: {
+          desc: "Skips secrets setup",
+          type: :boolean,
+          required: required
+        }
+      }
+    end
+
     def self.run_release_phase_option(required: false)
       {
         name: :run_release_phase,
@@ -378,57 +394,65 @@ module Command
         }
       }
     end
-    # rubocop:enable Metrics/MethodLength
 
-    def self.all_options
-      methods.grep(/_option$/).map { |method| send(method.to_s) }
+    def self.validations_option(required: false)
+      {
+        name: :validations,
+        params: {
+          banner: "VALIDATION_1,VALIDATION_2,...",
+          desc: "Which validations to run " \
+                "(must be separated by a comma)",
+          type: :string,
+          required: required,
+          default: VALIDATIONS_WITHOUT_ADDITIONAL_OPTIONS.join(","),
+          valid_regex: /^(#{ALL_VALIDATIONS.join("|")})(,(#{ALL_VALIDATIONS.join("|")}))*$/
+        }
+      }
     end
 
-    def self.all_options_by_key_name
-      all_options.each_with_object({}) do |option, result|
-        option[:params][:aliases]&.each { |current_alias| result[current_alias.to_s] = option }
-        result["--#{option[:name]}"] = option
-      end
+    def self.skip_post_creation_hook_option(required: false)
+      {
+        name: :skip_post_creation_hook,
+        params: {
+          desc: "Skips post-creation hook",
+          type: :boolean,
+          required: required
+        }
+      }
     end
 
-    def latest_image_from(items, app_name: config.app, name_only: true)
-      matching_items = items.select { |item| item["name"].start_with?("#{app_name}:") }
-
-      # Or special string to indicate no image available
-      if matching_items.empty?
-        name_only ? "#{app_name}:#{NO_IMAGE_AVAILABLE}" : nil
-      else
-        latest_item = matching_items.max_by { |item| extract_image_number(item["name"]) }
-        name_only ? latest_item["name"] : latest_item
-      end
+    def self.skip_pre_deletion_hook_option(required: false)
+      {
+        name: :skip_pre_deletion_hook,
+        params: {
+          desc: "Skips pre-deletion hook",
+          type: :boolean,
+          required: required
+        }
+      }
     end
 
-    def latest_image(app = config.app, org = config.org, refresh: false)
-      @latest_image ||= {}
-      @latest_image[app] = nil if refresh
-      @latest_image[app] ||=
-        begin
-          items = cp.query_images(app, org)["items"]
-          latest_image_from(items, app_name: app)
-        end
+    def self.add_app_identity_option(required: false)
+      {
+        name: :add_app_identity,
+        params: {
+          desc: "Adds app identity template if it does not exist",
+          type: :boolean,
+          required: required
+        }
+      }
     end
+    # rubocop:enable Metrics/MethodLength
 
-    def latest_image_next(app = config.app, org = config.org, commit: nil)
-      # debugger
-      commit ||= config.options[:commit]
-
-      @latest_image_next ||= {}
-      @latest_image_next[app] ||= begin
-        latest_image_name = latest_image(app, org)
-        image = latest_image_name.split(":").first
-        image += ":#{extract_image_number(latest_image_name) + 1}"
-        image += "_#{commit}" if commit
-        image
-      end
+    def self.all_options
+      methods.grep(/_option$/).map { |method| send(method.to_s) }
     end
 
-    def extract_image_commit(image_name)
-      image_name.match(/_(\h+)$/)&.captures&.first
+    def self.all_options_by_key_name
+      all_options.each_with_object({}) do |option, result|
+        option[:params][:aliases]&.each { |current_alias| result[current_alias.to_s] = option }
+        result["--#{option[:name]}"] = option
+      end
     end
 
     # NOTE: use simplified variant atm, as shelljoin do different escaping
@@ -486,30 +510,6 @@ module Command
       @cp ||= Controlplane.new(config)
     end
 
-    def app_location_link
-      "/org/#{config.org}/location/#{config.location}"
-    end
-
-    def app_image_link
-      "/org/#{config.org}/image/#{latest_image}"
-    end
-
-    def app_identity
-      "#{config.app}-identity"
-    end
-
-    def app_identity_link
-      "/org/#{config.org}/gvc/#{config.app}/identity/#{app_identity}"
-    end
-
-    def app_secrets
-      config.current[:secrets_name] || "#{config.app_prefix}-secrets"
-    end
-
-    def app_secrets_policy
-      config.current[:secrets_policy_name] || "#{app_secrets}-policy"
-    end
-
     def ensure_docker_running!
       result = Shell.cmd("docker", "version", capture_stderr: true)
       return if result[:success]
@@ -517,13 +517,24 @@ module Command
       raise "Can't run Docker. Please make sure that it's installed and started, then try again."
     end
 
-    private
+    def run_command_in_latest_image(command, title:)
+      # Need to prefix the command with '.controlplane/'
+      # if it's a file in the '.controlplane' directory,
+      # for backwards compatibility
+      path = Pathname.new("#{config.app_cpln_dir}/#{command}").expand_path
+      command = ".controlplane/#{command}" if File.exist?(path)
+
+      progress.puts("Running #{title}...\n\n")
 
-    # returns 0 if no prior image
-    def extract_image_number(image_name)
-      return 0 if image_name.end_with?(NO_IMAGE_AVAILABLE)
+      begin
+        Cpl::Cli.start(["run", "-a", config.app, "--image", "latest", "--", command])
+      rescue SystemExit => e
+        progress.puts
 
-      image_name.match(/:(\d+)/)&.captures&.first.to_i
+        raise "Failed to run #{title}." if e.status.nonzero?
+
+        progress.puts("Finished running #{title}.\n\n")
+      end
     end
   end
 end

data/lib/command/build_image.rb

@@ -27,7 +27,7 @@ module Command
 
       progress.puts("Building image from Dockerfile '#{dockerfile}'...\n\n")
 
-      image_name = latest_image_next
+      image_name = cp.latest_image_next
       image_url = "#{config.org}.registry.cpln.io/#{image_name}"
 
       commit = config.options[:commit]
@@ -41,7 +41,7 @@ module Command
       progress.puts("\nPushed image to '/org/#{config.org}/image/#{image_name}'.\n\n")
 
       step("Waiting for image to be available", retry_on_failure: true) do
-        image_name == latest_image(refresh: true)
+        image_name == cp.latest_image(refresh: true)
       end
     end
   end

data/lib/command/cleanup_images.rb

@@ -56,7 +56,7 @@ module Command
       return images unless cp.fetch_gvc(app)
 
       # If app exists, remove latest image, because we don't want to delete the image that is currently deployed
-      latest_image_name = latest_image_from(images, app_name: app)
+      latest_image_name = cp.latest_image_from(images, app_name: app)
       images.reject { |image| image["name"] == latest_image_name }
     end
 

data/lib/command/cleanup_stale_apps.rb

@@ -49,7 +49,7 @@ module Command
             app_name = gvc["name"]
 
             images = cp.query_images(app_name)["items"].select { |item| item["name"].start_with?("#{app_name}:") }
-            image = latest_image_from(images, app_name: app_name, name_only: false)
+            image = cp.latest_image_from(images, app_name: app_name, name_only: false)
             next unless image
 
             created_date = DateTime.parse(image["created"])

data/lib/command/copy_image_from_upstream.rb

@@ -66,8 +66,8 @@ module Command
       step("Fetching upstream image URL") do
         cp.profile_switch(@upstream_profile)
         upstream_image = config.options[:image]
-        upstream_image = latest_image(@upstream, @upstream_org) if !upstream_image || upstream_image == "latest"
-        @commit = extract_image_commit(upstream_image)
+        upstream_image = cp.latest_image(@upstream, @upstream_org) if !upstream_image || upstream_image == "latest"
+        @commit = cp.extract_image_commit(upstream_image)
         @upstream_image_url = "#{@upstream_org}.registry.cpln.io/#{upstream_image}"
       end
     end
@@ -75,7 +75,7 @@ module Command
     def fetch_app_image_url
       step("Fetching app image URL") do
         cp.profile_switch("default")
-        app_image = latest_image_next(config.app, config.org, commit: @commit)
+        app_image = cp.latest_image_next(config.app, config.org, commit: @commit)
         @app_image_url = "#{config.org}.registry.cpln.io/#{app_image}"
       end
     end

data/lib/command/delete.rb

@@ -6,13 +6,17 @@ module Command
     OPTIONS = [
       app_option(required: true),
       workload_option,
-      skip_confirm_option
+      skip_confirm_option,
+      skip_pre_deletion_hook_option
     ].freeze
     DESCRIPTION = "Deletes the whole app (GVC with all workloads, all volumesets and all images) or a specific workload"
     LONG_DESCRIPTION = <<~DESC
       - Deletes the whole app (GVC with all workloads, all volumesets and all images) or a specific workload
       - Also unbinds the app from the secrets policy, as long as both the identity and the policy exist (and are bound)
       - Will ask for explicit user confirmation
+      - Runs a pre-deletion hook before the app is deleted if `hooks.pre_deletion` is specified in the `.controlplane/controlplane.yml` file
+      - If the hook exits with a non-zero code, the command will stop executing and also exit with a non-zero code
+      - Use `--skip-pre-deletion-hook` to skip the hook if specified in `controlplane.yml`
     DESC
     EXAMPLES = <<~EX
       ```sh
@@ -51,6 +55,7 @@ module Command
       check_images
       return unless confirm_delete(config.app)
 
+      run_pre_deletion_hook unless config.options[:skip_pre_deletion_hook]
       unbind_identity_from_policy
       delete_volumesets
       delete_gvc
@@ -119,19 +124,26 @@ module Command
     end
 
     def unbind_identity_from_policy
-      return if cp.fetch_identity(app_identity).nil?
+      return if cp.fetch_identity(config.identity).nil?
 
-      policy = cp.fetch_policy(app_secrets_policy)
+      policy = cp.fetch_policy(config.secrets_policy)
       return if policy.nil?
 
       is_bound = policy["bindings"].any? do |binding|
-        binding["principalLinks"].any? { |link| link == app_identity_link }
+        binding["principalLinks"].any? { |link| link == config.identity_link }
       end
       return unless is_bound
 
       step("Unbinding identity from policy for app '#{config.app}'") do
-        cp.unbind_identity_from_policy(app_identity_link, app_secrets_policy)
+        cp.unbind_identity_from_policy(config.identity_link, config.secrets_policy)
       end
     end
+
+    def run_pre_deletion_hook
+      pre_deletion_hook = config.current.dig(:hooks, :pre_deletion)
+      return unless pre_deletion_hook
+
+      run_command_in_latest_image(pre_deletion_hook, title: "pre-deletion hook")
+    end
   end
 end