cpl 0.7.0 → 1.0.1

data/docs/commands.md

@@ -1,6 +1,6 @@
 <!-- NOTE: This file is automatically generated by running `script/generate_commands_docs`. Do NOT edit it manually. -->
 
-### Common Options
+## Common Options
 
 ```
 -a XXX, --app XXX         app ref on Control Plane (GVC)
@@ -9,7 +9,7 @@
 This `-a` option is used in most of the commands and will pick all other app configurations from the project-specific
 `.controlplane/controlplane.yml` file.
 
-### Commands
+## Commands
 
 ### `apply-template`
 

data/docs/tips.md

@@ -0,0 +1,177 @@
+# Tips
+
+1. [GVCs vs. Orgs](#gvcs-vs-orgs)
+2. [RAM](#ram)
+3. [Remote IP](#remote-ip)
+4. [ENV Values](#env-values)
+5. [CI](#ci)
+6. [Memcached](#memcached)
+7. [Sidekiq](#sidekiq)
+   - [Quieting Non-Critical Workers During Deployments](#quieting-non-critical-workers-during-deployments)
+   - [Setting Up a Pre Stop Hook](#setting-up-a-pre-stop-hook)
+   - [Setting Up a Liveness Probe](#setting-up-a-liveness-probe)
+8. [Useful Links](#useful-links)
+
+## GVCs vs. Orgs
+
+- A "GVC" roughly corresponds to a Heroku "app."
+- Images are available at the org level.
+- Multiple GVCs within an org can use the same image.
+- You can have different images within a GVC and even within a workload. This flexibility is one of the key differences
+  compared to Heroku apps.
+
+## RAM
+
+Any workload replica that reaches the max memory is terminated and restarted. You can configure alerts for workload
+restarts and the percentage of memory used in the Control Plane UX.
+
+Here are the steps for configuring an alert for the percentage of memory used:
+
+1. Navigate to the workload that you want to configure the alert for
+2. Click "Metrics" on the left menu to go to Grafana
+3. On Grafana, go to the alerting page by clicking on the alert icon in the sidebar
+4. Click on "New alert rule"
+5. In the "Set a query and alert condition" section, select "Grafana managed alert"
+6. There should be a default query named `A`
+7. Change the data source of the query to `metrics`
+8. Click "Code" on the top right of the query and enter `mem_used{workload="workload_name"} / mem_reserved{workload="workload_name"} * 100`
+   (replace `workload_name` with the name of the workload)
+9. There should be a default expression named `B`, with the type `Reduce`, the function `Last`, and the input `A` (this
+   ensures that we're getting the last data point of the query)
+10. There should be a default expression named `C`, with the type `Threshold`, and the input `B` (this is where you
+    configure the condition for firing the alert, e.g., `IS ABOVE 95`)
+11. You can then preview the alert and check if it's firing or not based on the example time range of the query
+12. In the "Alert evaluation behavior" section, you can configure how often the alert should be evaluated and for how
+    long the condition should be true before firing (for example, you might want the alert only to be fired if the
+    percentage has been above `95` for more than 20 seconds)
+13. In the "Add details for your alert" section, fill out the name, folder, group, and summary for your alert
+14. In the "Notifications" section, you can configure a label for the alert if you're using a custom notification policy,
+    but there should be a default root route for all alerts
+15. Once you're done, save and exit in the top right of the page
+16. Click "Contact points" on the top menu
+17. Edit the `grafana-default-email` contact point and add the email where you want to receive notifications
+18. You should now receive notifications for the alert in your email
+
+![](assets/grafana-alert.png)
+
+The steps for configuring an alert for workload restarts are almost identical, but the code for the query would be
+`container_restarts`.
+
+For more information on Grafana alerts, see: https://grafana.com/docs/grafana/latest/alerting/
+
+## Remote IP
+
+The actual remote IP of the workload container is in the 127.0.0.x network, so that will be the value of the
+`REMOTE_ADDR` env var.
+
+However, Control Plane additionally sets the `x-forwarded-for` and `x-envoy-external-address` headers (and others - see:
+https://docs.controlplane.com/concepts/security#headers). On Rails, the `ActionDispatch::RemoteIp` middleware should
+pick those up and automatically populate `request.remote_ip`.
+
+So `REMOTE_ADDR` should not be used directly, only `request.remote_ip`.
+
+## ENV Values
+
+You can store ENV values used by a container (within a workload) within Control Plane at the following levels:
+
+1. Workload Container
+2. GVC
+
+For your "review apps," it is convenient to have simple ENVs stored in plain text in your source code. You will want to
+keep some ENVs, like the Rails' `SECRET_KEY_BASE`, out of your source code. For staging and production apps, you will
+set these values directly at the GVC or workload levels, so none of these ENV values are committed to the source code.
+
+For storing ENVs in the source code, we can use a level of indirection so that you can store an ENV value in your source
+code like `cpln://secret/my-app-review-env-secrets.SECRET_KEY_BASE` and then have the secret value stored at the org
+level, which applies to your GVCs mapped to that org.
+
+Here is how you do this:
+
+1. In the upper left "Manage Org" menu, click on "Secrets"
+2. Create a secret with `Secret Type: Dictionary` (e.g., `my-secrets`) and add the secret env vars there
+3. In the upper left "Manage GVC" menu, click on "Identities"
+4. Create an identity (e.g., `my-identity`)
+5. Navigate to the workload that you want to associate with the identity created
+6. Click "Identity" on the left menu and select the identity created
+7. In the lower left "Access Control" menu, click on "Policies"
+8. Create a policy with `Target Kind: Secret` and add a binding with the `reveal` permission for the identity created
+9. Use `cpln://secret/...` in the app to access the secret env vars (e.g., `cpln://secret/my-secrets.SOME_VAR`)
+
+## CI
+
+**Note:** Docker builds much slower on Apple Silicon, so try configuring CI to build the images when using Apple
+hardware.
+
+Make sure to create a profile on CI before running any `cpln` or `cpl` commands.
+
+```sh
+CPLN_TOKEN=...
+cpln profile create default --token ${CPLN_TOKEN}
+```
+
+Also, log in to the Control Plane Docker repository if building and pushing an image.
+
+```sh
+cpln image docker-login
+```
+
+## Memcached
+
+On the workload container for Memcached (using the `memcached:alpine` image), configure the command with the args
+`-l 0.0.0.0`.
+
+To do this:
+
+1. Navigate to the workload container for Memcached
+2. Click "Command" on the top menu
+3. Add the args and save
+
+![](assets/memcached.png)
+
+## Sidekiq
+
+### Quieting Non-Critical Workers During Deployments
+
+To avoid locks in migrations, we can quiet non-critical workers during deployments. Doing this early enough in the CI
+allows all workers to finish jobs gracefully before deploying the new image.
+
+There's no need to unquiet the workers, as that will happen automatically after deploying the new image.
+
+```sh
+cpl run 'rails runner "Sidekiq::ProcessSet.new.each { |w| w.quiet! unless w[%q(hostname)].start_with?(%q(criticalworker.)) }"' -a my-app
+```
+
+### Setting Up a Pre Stop Hook
+
+By setting up a pre stop hook in the lifecycle of the workload container for Sidekiq, which sends "QUIET" to the workers,
+we can ensure that all workers will finish jobs gracefully before Control Plane stops the replica. That also works
+nicely for multiple replicas.
+
+A couple of notes:
+
+- We can't use the process name as regex because it's Ruby, not Sidekiq.
+- We need to add a space after `sidekiq`; otherwise, it sends `TSTP` to the `sidekiqswarm` process as well, and for some
+  reason, that doesn't work.
+
+So with `^` and `\s`, we guarantee it's sent only to worker processes.
+
+```sh
+pkill -TSTP -f ^sidekiq\s
+```
+
+To do this:
+
+1. Navigate to the workload container for Sidekiq
+2. Click "Lifecycle" on the top menu
+3. Add the command and args below "Pre Stop Hook" and save
+
+![](assets/sidekiq-pre-stop-hook.png)
+
+### Setting Up a Liveness Probe
+
+To set up a liveness probe on port 7433, see: https://github.com/arturictus/sidekiq_alive
+
+## Useful Links
+
+- For best practices for the app's Dockerfile, see: https://lipanski.com/posts/dockerfile-ruby-best-practices
+- For migrating from Heroku Postgres to RDS, see: https://pelle.io/posts/hetzner-rds-postgres

data/examples/circleci.yml

@@ -8,8 +8,8 @@ build-staging:
     - image: cimg/ruby:3.1-node
   resource_class: large
   environment:
-    CPLN_ORG: myorg
-    APP_NAME: myapp-staging
+    CPLN_ORG: my-org
+    APP_NAME: my-app-staging
   steps:
     - checkout
     - setup_remote_docker:
@@ -21,14 +21,13 @@ build-staging:
           cpln profile create default --token ${CPLN_TOKEN} --org ${CPLN_ORG} --gvc ${APP_NAME}
           cpln image docker-login
 
-          git clone https://github.com/shakacode/heroku-to-control-plane ~/heroku-to-control-plane
-          sudo ln -s ~/heroku-to-control-plane/cpl /usr/local/bin/cpl
+          gem install cpl
     - run:
         name: Containerize and push image
         command: cpl build-image -a ${APP_NAME}
     - run:
         name: Database tasks
-        command: cpl run:detached rails db:migrate -a ${APP_NAME} --image latest
+        command: cpl run:detached -a ${APP_NAME} --image latest -- rails db:migrate
     - run:
         name: Deploy image
         command: cpl deploy-image -a ${APP_NAME}
@@ -43,7 +42,7 @@ build-review-app:
     - image: cimg/ruby:3.1-node
   resource_class: large
   environment:
-    CPLN_ORG: my-org-name
+    CPLN_ORG: my-org
   steps:
     - checkout
     - setup_remote_docker:
@@ -65,7 +64,7 @@ build-review-app:
     - run:
         name: Provision review app if needed
         command: |
-          if ! cpl exist -a ${APP_NAME}; then
+          if ! cpl exists -a ${APP_NAME}; then
             cpl setup-app -a ${APP_NAME}
             echo "export NEW_APP=true" >> $BASH_ENV
           fi
@@ -77,9 +76,9 @@ build-review-app:
         name: Database tasks
         command: |
           if [ -n "${NEW_APP}" ]; then
-            cpl run:detached 'LOG_LEVEL=warn rails db:reset' -a ${APP_NAME} --image latest
+            cpl run:detached -a ${APP_NAME} --image latest -- LOG_LEVEL=warn rails db:reset
           else
-            cpl run:detached 'LOG_LEVEL=warn rails db:migrate' -a ${APP_NAME} --image latest
+            cpl run:detached -a ${APP_NAME} --image latest -- LOG_LEVEL=warn rails db:migrate
           fi
     - run:
         name: Deploy image

data/examples/controlplane.yml

@@ -1,47 +1,49 @@
+# Keys beginning with "cpln_" correspond to your settings in Control Plane.
+
 aliases:
   common: &common
-    # Org name for staging. (customize to your needs)
-    # Production apps will use a different Control Plane org, specified below, for security.
-    # keys beginning with CPLN correspond to your settings in Control Plane
+    # Organization name for staging (customize to your needs).
+    # Production apps will use a different organization, specified below, for security.
     cpln_org: my-org-staging
 
-    # Example apps use only location. CPLN offers the ability to use multiple locations.
-    # TODO -- allow specfication of multiple locations
+    # Example apps use only one location. Control Plane offers the ability to use multiple locations.
+    # TODO: Allow specification of multiple locations.
     default_location: aws-us-east-2
 
     # Configure the workload name used as a template for one-off scripts, like a Heroku one-off dyno.
     one_off_workload: rails
 
-    # Workloads that are application itself and are using application docker image
+    # Workloads that are for the application itself and are using application Docker images.
     app_workloads:
       - rails
       - sidekiq
 
-    # Additional "service type" workloads, using non-application docker images
+    # Additional "service type" workloads, using non-application Docker images.
     additional_workloads:
       - redis
       - postgres
       - memcached
 
-    # Configure the workload name used when maintenance mode is on (defaults to 'maintenance')
+    # Configure the workload name used when maintenance mode is on (defaults to "maintenance")
     maintenance_workload: maintenance
 
 apps:
   my-app-staging:
-    # Use the values from the common section above
+    # Use the values from the common section above.
     <<: *common
   my-app-review:
     <<: *common
-    # if match_if_app_name_starts_with == true, then use this config for app names beginning,
-    # like my-app-review-pr123 or my-app-review-anything-goes
+    # If `match_if_app_name_starts_with` is `true`, then use this config for app names starting with this name,
+    # e.g., "my-app-review-pr123", "my-app-review-anything-goes", etc.
     match_if_app_name_starts_with: true
   my-app-production:
     <<: *common
-    # Use a different org for production
+    # Use a different organization for production.
     cpln_org: my-org-production
-    # Allows running command 'cpl pipeline-promote my-app-staging' to promote the staging app to production
+    # Allows running the command `cpl promote-app-from-upstream -a my-app-production`
+    # to promote the staging app to production.
     upstream: my-app-staging
   my-app-other:
     <<: *common
-    # you can specify different dockerfile relative to .controlplane folder, default is just 'Dockerfile'
+    # You can specify a different `Dockerfile` relative to the `.controlplane/` directory (defaults to "Dockerfile").
     dockerfile: ../some_other/Dockerfile

data/lib/command/base.rb

@@ -187,28 +187,22 @@ module Command
       end
     end
 
-    def wait_for(title)
-      progress.print "- Waiting for #{title}"
-      until yield
-        progress.print(".")
-        sleep(1)
-      end
-      progress.puts
-    end
-
     def wait_for_workload(workload)
-      wait_for("workload to start") { cp.fetch_workload(workload) }
+      step("Waiting for workload", retry_on_failure: true) do
+        cp.fetch_workload(workload)
+      end
     end
 
     def wait_for_replica(workload, location)
-      wait_for("replica") do
+      step("Waiting for replica", retry_on_failure: true) do
         cp.workload_get_replicas_safely(workload, location: location)&.dig("items", 0)
       end
     end
 
     def ensure_workload_deleted(workload)
-      progress.puts "- Ensure workload is deleted"
-      cp.delete_workload(workload)
+      step("Deleting workload") do
+        cp.delete_workload(workload)
+      end
     end
 
     def latest_image_from(items, app_name: config.app, name_only: true)
@@ -216,7 +210,7 @@ module Command
 
       # Or special string to indicate no image available
       if matching_items.empty?
-        "#{app_name}:#{NO_IMAGE_AVAILABLE}"
+        name_only ? "#{app_name}:#{NO_IMAGE_AVAILABLE}" : nil
       else
         latest_item = matching_items.max_by { |item| extract_image_number(item["name"]) }
         name_only ? latest_item["name"] : latest_item

data/lib/command/cleanup_stale_apps.rb

@@ -51,6 +51,7 @@ module Command
 
             images = cp.image_query(app_name)["items"].filter { |item| item["name"].start_with?("#{app_name}:") }
             image = latest_image_from(images, app_name: app_name, name_only: false)
+            next unless image
 
             created_date = DateTime.parse(image["created"])
             diff_in_days = (now - created_date).to_i

data/lib/command/run.rb

@@ -56,24 +56,26 @@ module Command
 
     attr_reader :location, :workload, :one_off, :container
 
-    def call
+    def call # rubocop:disable Metrics/MethodLength
       @location = config[:default_location]
       @workload = config.options["workload"] || config[:one_off_workload]
       @one_off = "#{workload}-run-#{rand(1000..9999)}"
 
-      clone_workload
+      step("Cloning workload '#{workload}' on app '#{config.options[:app]}' to '#{one_off}'") do
+        clone_workload
+      end
+
       wait_for_workload(one_off)
       wait_for_replica(one_off, location)
       run_in_replica
     ensure
+      progress.puts
       ensure_workload_deleted(one_off)
     end
 
     private
 
     def clone_workload # rubocop:disable Metrics/MethodLength
-      progress.puts "- Cloning workload '#{workload}' on '#{config.options[:app]}' to '#{one_off}'"
-
       # Create a base copy of workload props
       spec = cp.fetch_workload!(workload).fetch("spec")
       container_spec = spec["containers"].detect { _1["name"] == workload } || spec["containers"].first
@@ -127,7 +129,7 @@ module Command
         if config.options[:terminal_size]
           rows, cols = config.options[:terminal_size].split(",")
         else
-          rows, cols = `stty -a`.match(/(\d+)\s*rows;\s*(\d+)\s*columns/).captures
+          rows, cols = `stty size`.split(/\s+/)
         end
         script += "stty rows #{rows}\nstty cols #{cols}\n" if rows && cols
       end
@@ -137,7 +139,7 @@ module Command
     end
 
     def run_in_replica
-      progress.puts "- Connecting"
+      progress.puts("Connecting...\n\n")
       command = %(bash -c 'eval "$CONTROLPLANE_RUNNER"')
       cp.workload_exec(one_off, location: location, container: container, command: command)
     end

data/lib/command/run_detached.rb

@@ -47,24 +47,28 @@ module Command
 
     attr_reader :location, :workload, :one_off, :container
 
-    def call
+    def call # rubocop:disable Metrics/MethodLength
       @location = config[:default_location]
       @workload = config.options["workload"] || config[:one_off_workload]
       @one_off = "#{workload}-runner-#{rand(1000..9999)}"
 
-      clone_workload
+      step("Cloning workload '#{workload}' on app '#{config.options[:app]}' to '#{one_off}'") do
+        clone_workload
+      end
+
       wait_for_workload(one_off)
       show_logs_waiting
     ensure
-      ensure_workload_deleted(one_off)
+      if cp.fetch_workload(one_off)
+        progress.puts
+        ensure_workload_deleted(one_off)
+      end
       exit(1) if @crashed
     end
 
     private
 
     def clone_workload # rubocop:disable Metrics/MethodLength
-      progress.puts "- Cloning workload '#{workload}' on '#{config.options[:app]}' to '#{one_off}'"
-
       # Get base specs of workload
       spec = cp.fetch_workload!(workload).fetch("spec")
       container_spec = spec["containers"].detect { _1["name"] == workload } || spec["containers"].first
@@ -121,17 +125,17 @@ module Command
     end
 
     def show_logs_waiting # rubocop:disable Metrics/MethodLength
-      progress.puts "- Scheduled, fetching logs (it is cron job, so it may take up to a minute to start)"
+      progress.puts("Scheduled, fetching logs (it's a cron job, so it may take up to a minute to start)...\n\n")
       begin
         while cp.fetch_workload(one_off)
           sleep(WORKLOAD_SLEEP_CHECK)
           print_uniq_logs
         end
       rescue RuntimeError => e
-        progress.puts "ERROR: #{e}"
+        progress.puts(Shell.color("ERROR: #{e}", :red))
         retry
       end
-      progress.puts "- Finished workload and logger"
+      progress.puts("\nFinished workload and logger.")
     end
 
     def print_uniq_logs

data/lib/core/scripts.rb

@@ -10,7 +10,7 @@ module Scripts
         -H "Authorization: ${CONTROLPLANE_TOKEN}" -s | grep -o '"replicas":[0-9]*' | grep -o '[0-9]*')
 
       if [ "$REPLICAS_QTY" -gt 0 ]; then
-        echo "-- MULTIPLE REPLICAS ATTEMPT !!!! replicas: $REPLICAS_QTY"
+        echo "-- MULTIPLE REPLICAS ATTEMPT: $REPLICAS_QTY --"
         exit -1
       fi
     SHELL
@@ -24,7 +24,7 @@ module Scripts
 
   # NOTE: please escape all '/' as '//' (as it is ruby interpolation here as well)
   def http_dummy_server_ruby
-    'require "socket";s=TCPServer.new(ENV["PORT"]);' \
+    'require "socket";s=TCPServer.new(ENV["PORT"] || 80);' \
       'loop do c=s.accept;c.puts("HTTP/1.1 200 OK\\nContent-Length: 2\\n\\nOk");c.close end'
   end
 

data/lib/cpl/version.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
 module Cpl
-  VERSION = "0.7.0"
+  VERSION = "1.0.1"
+  MIN_CPLN_VERSION = "0.0.71"
 end

data/lib/cpl.rb

@@ -40,16 +40,57 @@ end
 module Cpl
   class Error < StandardError; end
 
-  class Cli < Thor
+  class Cli < Thor # rubocop:disable Metrics/ClassLength
     package_name "cpl"
     default_task :no_command
 
     def self.start(*args)
+      check_cpln_version
+      check_cpl_version
       fix_help_option
 
       super(*args)
     end
 
+    def self.check_cpln_version # rubocop:disable Metrics/MethodLength
+      return if @checked_cpln_version
+
+      @checked_cpln_version = true
+
+      result = `cpln --version 2>/dev/null`
+      if $CHILD_STATUS.success?
+        data = JSON.parse(result)
+
+        version = data["npm"]
+        min_version = Cpl::MIN_CPLN_VERSION
+        if Gem::Version.new(version) < Gem::Version.new(min_version)
+          ::Shell.abort("Current 'cpln' version: #{version}. Minimum supported version: #{min_version}. " \
+                        "Please update it with 'npm update -g @controlplane/cli'.")
+        end
+      else
+        ::Shell.abort("Can't find 'cpln' executable. Please install it with 'npm install -g @controlplane/cli'.")
+      end
+    end
+
+    def self.check_cpl_version # rubocop:disable Metrics/MethodLength
+      return if @checked_cpl_version
+
+      @checked_cpl_version = true
+
+      result = `gem search ^cpl$ --remote 2>/dev/null`
+      return unless $CHILD_STATUS.success?
+
+      matches = result.match(/cpl \((.+)\)/)
+      return unless matches
+
+      version = Cpl::VERSION
+      latest_version = matches[1]
+      return unless Gem::Version.new(version) < Gem::Version.new(latest_version)
+
+      ::Shell.warn("You are not using the latest 'cpl' version. Please update it with 'gem update cpl'.")
+      $stderr.puts
+    end
+
     # This is so that we're able to run `cpl COMMAND --help` to print the help
     # (it basically changes it to `cpl --help COMMAND`, which Thor recognizes)
     # Based on https://stackoverflow.com/questions/49042591/how-to-add-help-h-flag-to-thor-command

data/script/check_command_docs

@@ -0,0 +1,3 @@
+#!/bin/sh
+bundle exec rake update_command_docs
+git diff --exit-code || exit 1

data/script/{generate_commands_docs → update_command_docs}

@@ -46,7 +46,7 @@ file_data =
   <<~DATA
     <!-- NOTE: This file is automatically generated by running `script/generate_commands_docs`. Do NOT edit it manually. -->
 
-    ### Common Options
+    ## Common Options
 
     ```
     -a XXX, --app XXX         app ref on Control Plane (GVC)
@@ -55,7 +55,7 @@ file_data =
     This `-a` option is used in most of the commands and will pick all other app configurations from the project-specific
     `.controlplane/controlplane.yml` file.
 
-    ### Commands
+    ## Commands
 
     #{commands_str}
   DATA

data/templates/daily-task.yml

@@ -4,23 +4,24 @@ spec:
   # https://docs.controlplane.com/reference/workload#cron-configuration
   type: cron
   job:
-    # Run daily job at 2am, see cron docs
+    # Run daily job at 2am (see cron docs)
     schedule: 0  2  *  *  *
     # Never or OnFailure
     restartPolicy: Never
   containers:
     - name: daily-task
+      cpu: 50m
+      memory: 256Mi
       args:
         - bundle
         - exec
         - rails
         - db:prepare
-      cpu: 50m
       inheritEnv: true
-      image: '/org/APP_ORG/image/APP_IMAGE'
-      memory: 256Mi
+      image: "/org/APP_ORG/image/APP_IMAGE"
   defaultOptions:
     autoscaling:
+      minScale: 1
       maxScale: 1
     capacityAI: false
   firewallConfig:

data/templates/maintenance.yml

@@ -6,15 +6,16 @@ spec:
     - name: maintenance
       env:
         - name: PORT
-          value: '3000'
+          value: "3000"
         - name: PAGE_URL
-          value: ''
-      image: 'shakacode/maintenance-mode'
+          value: ""
+      image: "shakacode/maintenance-mode"
       ports:
         - number: 3000
           protocol: http
   defaultOptions:
     autoscaling:
+      minScale: 1
       maxScale: 1
     capacityAI: false
     timeoutSeconds: 60

data/templates/memcached.yml

@@ -7,15 +7,16 @@ spec:
       cpu: 3m
       memory: 10Mi
       args:
-        - '-l'
+        - "-l"
         - 0.0.0.0
-      image: 'memcached:alpine'
+      image: "memcached:alpine"
       ports:
         - number: 11211
           protocol: tcp
   defaultOptions:
     autoscaling:
       metric: latency
+      minScale: 1
       maxScale: 1
     capacityAI: false
   firewallConfig: