cpl 1.0.0 → 1.0.2

data/docs/migrating.md

@@ -0,0 +1,262 @@
+# Steps to Migrate from Heroku to Control Plane
+
+We recommend following along with
+[this example project](https://github.com/shakacode/react-webpack-rails-tutorial).
+
+1. [Clone the Staging Environment](#clone-the-staging-environment)
+   - [Review Special Gems](#review-special-gems)
+   - [Create a Minimum Bootable Config](#create-a-minimum-bootable-config)
+2. [Create the Review App Process](#create-the-review-app-process)
+   - [Database for Review Apps](#database-for-review-apps)
+   - [Redis and Memcached for Review Apps](#redis-and-memcached-for-review-apps)
+3. [Deploy to Production](#deploy-to-production)
+
+## Clone the Staging Environment
+
+By cloning the staging environment on Heroku, you can speed up the initial provisioning of the app on Control Plane
+without compromising your current environment.
+
+Consider migrating just the web dyno first, and get other types of dynos working afterward. You can also move the
+add-ons to Control Plane later once the app works as expected.
+
+First, create a new Heroku app with all the add-ons, copying the data from the current staging app.
+
+Then, copy project-specific configs to a `.controlplane/` directory at the top of your project. `cpl` will pick those up
+depending on which project folder tree it runs. Thus, this automates running several projects with different configs
+without explicitly switching configs.
+
+Edit the `.controlplane/controlplane.yml` file as needed. Note that the `my-app-staging` name used in the examples below
+is defined in this file. See
+[this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/controlplane.yml).
+
+Before the initial setup, add the templates for the app to the `.controlplane/controlplane.yml` file, using the `setup`
+key, e.g.:
+
+```yaml
+my-app-staging:
+  <<: *common
+  setup:
+    - gvc
+    - redis
+    - memcached
+    - rails
+    - sidekiq
+```
+
+Note how the templates correspond to files in the `.controlplane/templates/` directory. These files will be used by the
+`cpl setup-app` and `cpl apply-template` commands.
+
+Ensure that env vars point to the Heroku add-ons in the template for the app (`.controlplane/templates/gvc.yml`). See
+[this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/templates/gvc.yml).
+
+After that, create a Dockerfile in `.controlplane/Dockerfile` for your deployment. See
+[this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/Dockerfile).
+
+You should have a folder structure similar to the following:
+
+```sh
+app_main_folder/
+  .controlplane/
+    Dockerfile          # Your app's Dockerfile, with some Control Plane changes.
+    controlplane.yml
+    entrypoint.sh       # App-specific - edit as needed.
+    templates/
+      gvc.yml
+      memcached.yml
+      rails.yml
+      redis.yml
+      sidekiq.yml
+```
+
+The example
+[`.controlplane/` directory](https://github.com/shakacode/react-webpack-rails-tutorial/tree/master/.controlplane)
+already contains these files.
+
+Finally, check the app for any Heroku-specific code and update it, such as the `HEROKU_SLUG_COMMIT` env var and other
+env vars beginning with `HEROKU_`. You should add some logic to check for the Control Plane equivalents - it might be
+worth adding a `CONTROLPLANE` env var to act as a feature flag and help run different code for Heroku and Control Plane
+until the migration is complete.
+
+You might want to [review special gems](#review-special-gems) and
+[create a minimum bootable config](#create-a-minimum-bootable-config).
+
+At first, do the deployments from the command line. Then set up CI scripts to trigger the deployment upon merges to
+master/main.
+
+Use these commands for the initial setup and deployment:
+
+```sh
+# Provision infrastructure (one-time-only for new apps) using templates.
+cpl setup-app -a my-app-staging
+
+# Build and push image with auto-tagging, e.g., "my-app-staging:1_456".
+cpl build-image -a my-app-staging --commit 456
+
+# Prepare database.
+cpl run:detached -a my-app-staging --image latest -- rails db:prepare
+
+# Deploy latest image.
+cpl deploy-image -a my-app-staging
+
+# Open app in browser.
+cpl open -a my-app-staging
+```
+
+Then for promoting code upgrades:
+
+```sh
+# Build and push new image with sequential tagging, e.g., "my-app-staging:2".
+cpl build-image -a my-app-staging
+
+# Or build and push new image with sequential tagging and commit SHA, e.g., "my-app-staging:2_ABC".
+cpl build-image -a my-app-staging --commit ABC
+
+# Run database migrations (or other release tasks) with latest image, while app is still running on previous image.
+# This is analogous to the release phase.
+cpl run:detached -a my-app-staging --image latest -- rails db:migrate
+
+# Deploy latest image.
+cpl deploy-image -a my-app-staging
+```
+
+### Review Special Gems
+
+Make sure to review "special" gems which might be related to Heroku, e.g.:
+
+- `rails_autoscale_agent`. It's specific to Heroku, so it must be removed.
+- `puma_worker_killer`. In general, it's unnecessary on Control Plane, as Kubernetes containers will restart on their
+  own logic and may not restart at all if everything is ok.
+- `rack-timeout`. It could possibly be replaced with Control Plane's `timeout` option.
+
+You can use the `CONTROLPLANE` env var to separate the gems, e.g.:
+
+```ruby
+# Gemfile
+group :staging, :production do
+	gem "rack-timeout"
+
+  unless ENV.key?("CONTROLPLANE")
+	  gem "rails_autoscale_agent"
+    gem "puma_worker_killer"
+  end
+end
+```
+
+### Create a Minimum Bootable Config
+
+You can try to create a minimum bootable config to migrate parts of your app gradually. To do that, follow these steps:
+
+1. Rename the existing `application.yml` file to some other name (e.g., `application.old.yml`)
+2. Create a new **minimal** `application.yml` file, e.g.:
+
+```yaml
+SECRET_KEY_BASE: "123"
+# This should be enabled for `rails s`, not `rails assets:precompile`.
+# DATABASE_URL: postgres://localhost:5432/dbname
+# RAILS_SERVE_STATIC_FILES: "true"
+
+# You will add whatever env vars are required here later.
+```
+
+3. Try running `RAILS_ENV=production CONTROLPLANE=true rails assets:precompile`
+   (theoretically, this should work without any additional env vars)
+4. Fix whatever code needs to be fixed and add missing env vars
+   (the fewer env vars are needed, the cleaner the `Dockerfile` will be)
+5. Enable `DATABASE_URL` and `RAILS_SERVE_STATIC_FILES` env vars
+6. Try running `RAILS_ENV=production CONTROLPLANE=true rails s`
+7. Fix whatever code needs to be fixed and add required env vars to `application.yml`
+8. Try running your **production** entrypoint command, e.g.,
+   `RAILS_ENV=production RACK_ENV=production CONTROLPLANE=true puma -C config/puma.rb`
+9. Fix whatever code needs to be fixed and add required env vars to `application.yml`
+
+Now you should have a minimal bootable config.
+
+Then you can temporarily set the `LOG_LEVEL=debug` env var and disable unnecessary services to help with the process,
+e.g.:
+
+```yaml
+DISABLE_SPRING: "true"
+SCOUT_MONITOR: "false"
+RACK_TIMEOUT_SERVICE_TIMEOUT: "0"
+```
+
+## Create the Review App Process
+
+Add an entry for review apps to the `.controlplane/controlplane.yml` file. By adding a `match_if_app_name_starts_with`
+key with the value `true`, any app that starts with the entry's name will use this config. Doing this allows you to
+configure an entry for, e.g., `my-app-review`, and then create review apps starting with that name (e.g.,
+`my-app-review-1234`, `my-app-review-5678`, etc.). Here's an example:
+
+```yaml
+  my-app-review:
+    <<: *common
+    match_if_app_name_starts_with: true
+    setup:
+      - gvc
+      - redis
+      - memcached
+      - rails
+      - sidekiq
+```
+
+In your CI scripts, you can create a review app using some identifier (e.g., the number of the PR on GitHub).
+
+```yaml
+# On CircleCI, you can use `echo $CIRCLE_PULL_REQUEST | grep -Eo '[0-9]+$'` to extract the number of the PR.
+PR_NUM=$(... extract the number of the PR here ...)
+echo "export APP_NAME=my-app-review-$PR_NUM" >> $BASH_ENV
+
+# Only create the app if it doesn't exist yet, as we may have multiple triggers for the review app
+# (such as when a PR gets updated).
+if ! cpl exists -a ${APP_NAME}; then
+  cpl setup-app -a ${APP_NAME}
+  echo "export NEW_APP=true" >> $BASH_ENV
+fi
+
+# The `NEW_APP` env var that we exported above can be used to either reset or migrate the database before deploying.
+if [ -n "${NEW_APP}" ]; then
+  cpl run:detached 'LOG_LEVEL=warn rails db:reset' -a ${APP_NAME} --image latest
+else
+  cpl run:detached 'LOG_LEVEL=warn rails db:migrate_and_wait_replica' -a ${APP_NAME} --image latest
+fi
+```
+
+Then follow the same steps for the initial deployment or code upgrades.
+
+### Database for Review Apps
+
+For the review app resources, these should be handled as env vars in the template for the app
+(`.controlplane/templates/gvc.yml`), .e.g.:
+
+```yaml
+- name: DATABASE_URL
+  value: postgres://postgres:XXXXXXXX@cpln-XXXX-staging.XXXXXX.us-east-1.rds.amazonaws.com:5432/APP_GVC
+```
+
+Notice that `APP_GVC` is the app name, which is used as the database name on RDS, so that each review app gets its own
+database on the one RDS instance used for all review apps, which would be, e.g., `my-app-review-1234`.
+
+### Redis and Memcached for Review Apps
+
+So long as no persistence is needed for Redis and Memcached, we have templates for workloads that should be sufficient
+for review apps in the `templates/` directory of this repository. Using these templates results in considerable cost
+savings compared to paying for the resources on Heroku.
+
+```yaml
+- name: MEMCACHE_SERVERS
+  value: memcached.APP_GVC.cpln.local
+- name: REDIS_URL
+  value: redis://redis.APP_GVC.cpln.local:6379
+```
+
+## Deploy to Production
+
+Only try deploying to production once staging and review apps are working well.
+
+For simplicity, keep add-ons running on Heroku initially. You could move over the database to RDS first. However, it's a
+bit simpler to isolate any differences in cost and performance by first moving over your compute to Control Plane.
+
+Ensure that your Control Plane compute is in the AWS region `US-EAST-1`; otherwise, you'll have noticeable extra latency
+with your calls to resources. You might also have egress charges from Control Plane.
+
+Use the `cpl promote-app-from-upstream` command to promote the staging app to production.

data/docs/tips.md

@@ -0,0 +1,177 @@
+# Tips
+
+1. [GVCs vs. Orgs](#gvcs-vs-orgs)
+2. [RAM](#ram)
+3. [Remote IP](#remote-ip)
+4. [ENV Values](#env-values)
+5. [CI](#ci)
+6. [Memcached](#memcached)
+7. [Sidekiq](#sidekiq)
+   - [Quieting Non-Critical Workers During Deployments](#quieting-non-critical-workers-during-deployments)
+   - [Setting Up a Pre Stop Hook](#setting-up-a-pre-stop-hook)
+   - [Setting Up a Liveness Probe](#setting-up-a-liveness-probe)
+8. [Useful Links](#useful-links)
+
+## GVCs vs. Orgs
+
+- A "GVC" roughly corresponds to a Heroku "app."
+- Images are available at the org level.
+- Multiple GVCs within an org can use the same image.
+- You can have different images within a GVC and even within a workload. This flexibility is one of the key differences
+  compared to Heroku apps.
+
+## RAM
+
+Any workload replica that reaches the max memory is terminated and restarted. You can configure alerts for workload
+restarts and the percentage of memory used in the Control Plane UX.
+
+Here are the steps for configuring an alert for the percentage of memory used:
+
+1. Navigate to the workload that you want to configure the alert for
+2. Click "Metrics" on the left menu to go to Grafana
+3. On Grafana, go to the alerting page by clicking on the alert icon in the sidebar
+4. Click on "New alert rule"
+5. In the "Set a query and alert condition" section, select "Grafana managed alert"
+6. There should be a default query named `A`
+7. Change the data source of the query to `metrics`
+8. Click "Code" on the top right of the query and enter `mem_used{workload="workload_name"} / mem_reserved{workload="workload_name"} * 100`
+   (replace `workload_name` with the name of the workload)
+9. There should be a default expression named `B`, with the type `Reduce`, the function `Last`, and the input `A` (this
+   ensures that we're getting the last data point of the query)
+10. There should be a default expression named `C`, with the type `Threshold`, and the input `B` (this is where you
+    configure the condition for firing the alert, e.g., `IS ABOVE 95`)
+11. You can then preview the alert and check if it's firing or not based on the example time range of the query
+12. In the "Alert evaluation behavior" section, you can configure how often the alert should be evaluated and for how
+    long the condition should be true before firing (for example, you might want the alert only to be fired if the
+    percentage has been above `95` for more than 20 seconds)
+13. In the "Add details for your alert" section, fill out the name, folder, group, and summary for your alert
+14. In the "Notifications" section, you can configure a label for the alert if you're using a custom notification policy,
+    but there should be a default root route for all alerts
+15. Once you're done, save and exit in the top right of the page
+16. Click "Contact points" on the top menu
+17. Edit the `grafana-default-email` contact point and add the email where you want to receive notifications
+18. You should now receive notifications for the alert in your email
+
+![](assets/grafana-alert.png)
+
+The steps for configuring an alert for workload restarts are almost identical, but the code for the query would be
+`container_restarts`.
+
+For more information on Grafana alerts, see: https://grafana.com/docs/grafana/latest/alerting/
+
+## Remote IP
+
+The actual remote IP of the workload container is in the 127.0.0.x network, so that will be the value of the
+`REMOTE_ADDR` env var.
+
+However, Control Plane additionally sets the `x-forwarded-for` and `x-envoy-external-address` headers (and others - see:
+https://docs.controlplane.com/concepts/security#headers). On Rails, the `ActionDispatch::RemoteIp` middleware should
+pick those up and automatically populate `request.remote_ip`.
+
+So `REMOTE_ADDR` should not be used directly, only `request.remote_ip`.
+
+## ENV Values
+
+You can store ENV values used by a container (within a workload) within Control Plane at the following levels:
+
+1. Workload Container
+2. GVC
+
+For your "review apps," it is convenient to have simple ENVs stored in plain text in your source code. You will want to
+keep some ENVs, like the Rails' `SECRET_KEY_BASE`, out of your source code. For staging and production apps, you will
+set these values directly at the GVC or workload levels, so none of these ENV values are committed to the source code.
+
+For storing ENVs in the source code, we can use a level of indirection so that you can store an ENV value in your source
+code like `cpln://secret/my-app-review-env-secrets.SECRET_KEY_BASE` and then have the secret value stored at the org
+level, which applies to your GVCs mapped to that org.
+
+Here is how you do this:
+
+1. In the upper left "Manage Org" menu, click on "Secrets"
+2. Create a secret with `Secret Type: Dictionary` (e.g., `my-secrets`) and add the secret env vars there
+3. In the upper left "Manage GVC" menu, click on "Identities"
+4. Create an identity (e.g., `my-identity`)
+5. Navigate to the workload that you want to associate with the identity created
+6. Click "Identity" on the left menu and select the identity created
+7. In the lower left "Access Control" menu, click on "Policies"
+8. Create a policy with `Target Kind: Secret` and add a binding with the `reveal` permission for the identity created
+9. Use `cpln://secret/...` in the app to access the secret env vars (e.g., `cpln://secret/my-secrets.SOME_VAR`)
+
+## CI
+
+**Note:** Docker builds much slower on Apple Silicon, so try configuring CI to build the images when using Apple
+hardware.
+
+Make sure to create a profile on CI before running any `cpln` or `cpl` commands.
+
+```sh
+CPLN_TOKEN=...
+cpln profile create default --token ${CPLN_TOKEN}
+```
+
+Also, log in to the Control Plane Docker repository if building and pushing an image.
+
+```sh
+cpln image docker-login
+```
+
+## Memcached
+
+On the workload container for Memcached (using the `memcached:alpine` image), configure the command with the args
+`-l 0.0.0.0`.
+
+To do this:
+
+1. Navigate to the workload container for Memcached
+2. Click "Command" on the top menu
+3. Add the args and save
+
+![](assets/memcached.png)
+
+## Sidekiq
+
+### Quieting Non-Critical Workers During Deployments
+
+To avoid locks in migrations, we can quiet non-critical workers during deployments. Doing this early enough in the CI
+allows all workers to finish jobs gracefully before deploying the new image.
+
+There's no need to unquiet the workers, as that will happen automatically after deploying the new image.
+
+```sh
+cpl run 'rails runner "Sidekiq::ProcessSet.new.each { |w| w.quiet! unless w[%q(hostname)].start_with?(%q(criticalworker.)) }"' -a my-app
+```
+
+### Setting Up a Pre Stop Hook
+
+By setting up a pre stop hook in the lifecycle of the workload container for Sidekiq, which sends "QUIET" to the workers,
+we can ensure that all workers will finish jobs gracefully before Control Plane stops the replica. That also works
+nicely for multiple replicas.
+
+A couple of notes:
+
+- We can't use the process name as regex because it's Ruby, not Sidekiq.
+- We need to add a space after `sidekiq`; otherwise, it sends `TSTP` to the `sidekiqswarm` process as well, and for some
+  reason, that doesn't work.
+
+So with `^` and `\s`, we guarantee it's sent only to worker processes.
+
+```sh
+pkill -TSTP -f ^sidekiq\s
+```
+
+To do this:
+
+1. Navigate to the workload container for Sidekiq
+2. Click "Lifecycle" on the top menu
+3. Add the command and args below "Pre Stop Hook" and save
+
+![](assets/sidekiq-pre-stop-hook.png)
+
+### Setting Up a Liveness Probe
+
+To set up a liveness probe on port 7433, see: https://github.com/arturictus/sidekiq_alive
+
+## Useful Links
+
+- For best practices for the app's Dockerfile, see: https://lipanski.com/posts/dockerfile-ruby-best-practices
+- For migrating from Heroku Postgres to RDS, see: https://pelle.io/posts/hetzner-rds-postgres

data/examples/circleci.yml

@@ -8,8 +8,8 @@ build-staging:
     - image: cimg/ruby:3.1-node
   resource_class: large
   environment:
-    CPLN_ORG: myorg
-    APP_NAME: myapp-staging
+    CPLN_ORG: my-org
+    APP_NAME: my-app-staging
   steps:
     - checkout
     - setup_remote_docker:
@@ -21,14 +21,13 @@ build-staging:
           cpln profile create default --token ${CPLN_TOKEN} --org ${CPLN_ORG} --gvc ${APP_NAME}
           cpln image docker-login
 
-          git clone https://github.com/shakacode/heroku-to-control-plane ~/heroku-to-control-plane
-          sudo ln -s ~/heroku-to-control-plane/cpl /usr/local/bin/cpl
+          gem install cpl
     - run:
         name: Containerize and push image
         command: cpl build-image -a ${APP_NAME}
     - run:
         name: Database tasks
-        command: cpl run:detached rails db:migrate -a ${APP_NAME} --image latest
+        command: cpl run:detached -a ${APP_NAME} --image latest -- rails db:migrate
     - run:
         name: Deploy image
         command: cpl deploy-image -a ${APP_NAME}
@@ -43,7 +42,7 @@ build-review-app:
     - image: cimg/ruby:3.1-node
   resource_class: large
   environment:
-    CPLN_ORG: my-org-name
+    CPLN_ORG: my-org
   steps:
     - checkout
     - setup_remote_docker:
@@ -65,7 +64,7 @@ build-review-app:
     - run:
         name: Provision review app if needed
         command: |
-          if ! cpl exist -a ${APP_NAME}; then
+          if ! cpl exists -a ${APP_NAME}; then
             cpl setup-app -a ${APP_NAME}
             echo "export NEW_APP=true" >> $BASH_ENV
           fi
@@ -77,9 +76,9 @@ build-review-app:
         name: Database tasks
         command: |
           if [ -n "${NEW_APP}" ]; then
-            cpl run:detached 'LOG_LEVEL=warn rails db:reset' -a ${APP_NAME} --image latest
+            cpl run:detached -a ${APP_NAME} --image latest -- LOG_LEVEL=warn rails db:reset
           else
-            cpl run:detached 'LOG_LEVEL=warn rails db:migrate' -a ${APP_NAME} --image latest
+            cpl run:detached -a ${APP_NAME} --image latest -- LOG_LEVEL=warn rails db:migrate
           fi
     - run:
         name: Deploy image

data/examples/controlplane.yml

@@ -1,47 +1,49 @@
+# Keys beginning with "cpln_" correspond to your settings in Control Plane.
+
 aliases:
   common: &common
-    # Org name for staging. (customize to your needs)
-    # Production apps will use a different Control Plane org, specified below, for security.
-    # keys beginning with CPLN correspond to your settings in Control Plane
+    # Organization name for staging (customize to your needs).
+    # Production apps will use a different organization, specified below, for security.
     cpln_org: my-org-staging
 
-    # Example apps use only location. CPLN offers the ability to use multiple locations.
-    # TODO -- allow specfication of multiple locations
+    # Example apps use only one location. Control Plane offers the ability to use multiple locations.
+    # TODO: Allow specification of multiple locations.
     default_location: aws-us-east-2
 
     # Configure the workload name used as a template for one-off scripts, like a Heroku one-off dyno.
     one_off_workload: rails
 
-    # Workloads that are application itself and are using application docker image
+    # Workloads that are for the application itself and are using application Docker images.
     app_workloads:
       - rails
       - sidekiq
 
-    # Additional "service type" workloads, using non-application docker images
+    # Additional "service type" workloads, using non-application Docker images.
     additional_workloads:
       - redis
       - postgres
       - memcached
 
-    # Configure the workload name used when maintenance mode is on (defaults to 'maintenance')
+    # Configure the workload name used when maintenance mode is on (defaults to "maintenance")
     maintenance_workload: maintenance
 
 apps:
   my-app-staging:
-    # Use the values from the common section above
+    # Use the values from the common section above.
     <<: *common
   my-app-review:
     <<: *common
-    # if match_if_app_name_starts_with == true, then use this config for app names beginning,
-    # like my-app-review-pr123 or my-app-review-anything-goes
+    # If `match_if_app_name_starts_with` is `true`, then use this config for app names starting with this name,
+    # e.g., "my-app-review-pr123", "my-app-review-anything-goes", etc.
     match_if_app_name_starts_with: true
   my-app-production:
     <<: *common
-    # Use a different org for production
+    # Use a different organization for production.
     cpln_org: my-org-production
-    # Allows running command 'cpl pipeline-promote my-app-staging' to promote the staging app to production
+    # Allows running the command `cpl promote-app-from-upstream -a my-app-production`
+    # to promote the staging app to production.
     upstream: my-app-staging
   my-app-other:
     <<: *common
-    # you can specify different dockerfile relative to .controlplane folder, default is just 'Dockerfile'
+    # You can specify a different `Dockerfile` relative to the `.controlplane/` directory (defaults to "Dockerfile").
     dockerfile: ../some_other/Dockerfile

data/lib/command/base.rb

@@ -210,7 +210,7 @@ module Command
 
       # Or special string to indicate no image available
       if matching_items.empty?
-        "#{app_name}:#{NO_IMAGE_AVAILABLE}"
+        name_only ? "#{app_name}:#{NO_IMAGE_AVAILABLE}" : nil
       else
         latest_item = matching_items.max_by { |item| extract_image_number(item["name"]) }
         name_only ? latest_item["name"] : latest_item

data/lib/command/cleanup_stale_apps.rb

@@ -51,6 +51,7 @@ module Command
 
             images = cp.image_query(app_name)["items"].filter { |item| item["name"].start_with?("#{app_name}:") }
             image = latest_image_from(images, app_name: app_name, name_only: false)
+            next unless image
 
             created_date = DateTime.parse(image["created"])
             diff_in_days = (now - created_date).to_i

data/lib/command/ps_start.rb

@@ -42,9 +42,7 @@ module Command
 
       @workloads.reverse_each do |workload|
         step("Waiting for workload '#{workload}' to be ready", retry_on_failure: true) do
-          cp.fetch_workload_deployments(workload)&.dig("items")&.any? do |item|
-            item.dig("status", "ready")
-          end
+          cp.workload_deployments_ready?(workload, expected_status: true)
         end
       end
     end

data/lib/command/ps_stop.rb

@@ -6,7 +6,7 @@ module Command
     OPTIONS = [
       app_option(required: true),
       workload_option,
-      wait_option("workload to be not ready")
+      wait_option("workload to not be ready")
     ].freeze
     DESCRIPTION = "Stops workloads in app"
     LONG_DESCRIPTION = <<~DESC
@@ -41,10 +41,8 @@ module Command
       progress.puts
 
       @workloads.each do |workload|
-        step("Waiting for workload '#{workload}' to be not ready", retry_on_failure: true) do
-          cp.fetch_workload_deployments(workload)&.dig("items")&.all? do |item|
-            !item.dig("status", "ready")
-          end
+        step("Waiting for workload '#{workload}' to not be ready", retry_on_failure: true) do
+          cp.workload_deployments_ready?(workload, expected_status: false)
         end
       end
     end

data/lib/command/ps_wait.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module Command
+  class PsWait < Base
+    NAME = "ps:wait"
+    OPTIONS = [
+      app_option(required: true),
+      workload_option
+    ].freeze
+    DESCRIPTION = "Waits for workloads in app to be ready after re-deployment"
+    LONG_DESCRIPTION = <<~DESC
+      - Waits for workloads in app to be ready after re-deployment
+    DESC
+    EXAMPLES = <<~EX
+      ```sh
+      # Waits for all workloads in app.
+      cpl ps:wait -a $APP_NAME
+
+      # Waits for a specific workload in app.
+      cpl ps:swait -a $APP_NAME -w $WORKLOAD_NAME
+      ```
+    EX
+
+    def call
+      @workloads = [config.options[:workload]] if config.options[:workload]
+      @workloads ||= config[:app_workloads] + config[:additional_workloads]
+
+      @workloads.reverse_each do |workload|
+        step("Waiting for workload '#{workload}' to be ready", retry_on_failure: true) do
+          cp.workload_deployments_ready?(workload, expected_status: true)
+        end
+      end
+    end
+  end
+end

data/lib/core/controlplane.rb

@@ -149,6 +149,26 @@ class Controlplane # rubocop:disable Metrics/ClassLength
     api.workload_deployments(workload: workload, gvc: gvc, org: org)
   end
 
+  def workload_deployment_version_ready?(version, next_version, expected_status:)
+    return false unless version["workload"] == next_version
+
+    version["containers"]&.all? do |_, container|
+      ready = container.dig("resources", "replicas") == container.dig("resources", "replicasReady")
+      expected_status == true ? ready : !ready
+    end
+  end
+
+  def workload_deployments_ready?(workload, expected_status:)
+    deployments = fetch_workload_deployments(workload)["items"]
+    deployments.all? do |deployment|
+      next_version = deployment.dig("status", "expectedDeploymentVersion")
+
+      deployment.dig("status", "versions")&.all? do |version|
+        workload_deployment_version_ready?(version, next_version, expected_status: expected_status)
+      end
+    end
+  end
+
   def workload_set_image_ref(workload, container:, image:)
     cmd = "cpln workload update #{workload} #{gvc_org}"
     cmd += " --set spec.containers.#{container}.image=/org/#{config.org}/image/#{image}"