cpl 2.0.0.rc.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84efbe4c7082edcef9193b7a4116af57177b54f52373e4e4bca7e6b277f1b846
-  data.tar.gz: 9e047c8b354e6e0a01b28b84e4b3c74c0f5dd675c8d1f5113119803f1f5bcabb
+  metadata.gz: c304577e2c02b5778a463b82ec9a72792bd10b727687402d66a504d390673fae
+  data.tar.gz: dc28bb00740914362abba26bc61cee999f7ac2d955218dca0ea6dbeef58d465e
 SHA512:
-  metadata.gz: 6efc92fbcb3ea76e6ec44561178e719afc80af717790771b9c2c8358f564eb94b9c76b8aef8367c401e6e98f2faf9c0a8e628371a3193e038843339d12f7058e
-  data.tar.gz: e43668ec1adc98bd399396a35cb092d8d940df1d6bfea3ecb09b99e1be48f8ff7f0c625d8c4936eb61d01ceec4e3c8fb6a2eb7018f6ee91c1b0751dc7847f93f
+  metadata.gz: 4812e651d14114d8d7bccf893cf50107dc9ba24b35679d633fc36d37589dfbae9d93eb8e8dc172bc66e4feb6039d6ab7c2093c0f35b1a544acab01b55c722c68
+  data.tar.gz: 53013a61c2bcedc3ad10664f8d4a8874731013bb740abdb42fdac34dd82c3dc960b2643e9dea8227335040d2c2730eff6ed0cf47973d9344ddf5c5fc1bba7a99

data/CHANGELOG.md

@@ -14,13 +14,35 @@ Changes since the last non-beta release.
 
 _Please add entries here for your pull requests that are not yet released._
 
+## [2.0.0.rc.1] - 2024-05-11
+
+### BREAKING CHANGES
+
+- Commands that finished with a failure now exit with code `64` instead of `1`. [PR 132](https://github.com/shakacode/heroku-to-control-plane/pull/132) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Bumped minimum `cpln` version to `2.0.1` (`cpln workload cron get` is required). [PR 171](https://github.com/shakacode/heroku-to-control-plane/pull/171) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `run:cleanup` command has been removed. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `deploy-image` command now runs the release script in the context of the `run` command. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+
+### Fixed
+
+- Fixed race conditions when using latest image in `run` command. [PR 163](https://github.com/shakacode/heroku-to-control-plane/pull/163) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+
 ### Added
 
+- Added options to `run` command to override the workload container's `--cpu`, `--memory`, and `--entrypoint`. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Added `--workload` option to `delete` command to delete a specific workload. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Added `--replica` option to `logs` command to see logs from a specific replica. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Added `--replica` option to `ps:stop` command to stop a specific replica. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
 - Added option to set custom names for secrets and secrets policy, using `secrets_name` and `secrets_policy_name` in `controlplane.yml`. [PR 159](https://github.com/shakacode/heroku-to-control-plane/pull/159) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
 
 ### Changed
 
+- An error is now raised if the org does not exist. [PR 167](https://github.com/shakacode/heroku-to-control-plane/pull/167) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- Common options are now shown in help. [PR 169](https://github.com/shakacode/heroku-to-control-plane/pull/169) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `run` command now uses a single reusable cron workload and works for both interactive and non-interactive jobs. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `run:detached` command has been deprecated in favor of `run`. [PR 151](https://github.com/shakacode/heroku-to-control-plane/pull/151) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
 - `deploy-image` command now raises an error if image does not exist. [PR 153](https://github.com/shakacode/heroku-to-control-plane/pull/153) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
+- `delete` command now unbinds identity from policy (if bound) when deleting app. [PR 170](https://github.com/shakacode/heroku-to-control-plane/pull/170) by [Rafael Gomes](https://github.com/rafaelgomesxyz).
 
 ## [1.4.0] - 2024-03-20
 
@@ -164,7 +186,8 @@ _Please add entries here for your pull requests that are not yet released._
 
 - Initial release
 
-[Unreleased]: https://github.com/shakacode/heroku-to-control-plane/compare/v1.4.0...HEAD
+[Unreleased]: https://github.com/shakacode/heroku-to-control-plane/compare/v2.0.0.rc.1...HEAD
+[2.0.0.rc.1]: https://github.com/shakacode/heroku-to-control-plane/compare/v1.4.0...v2.0.0.rc.1
 [1.4.0]: https://github.com/shakacode/heroku-to-control-plane/compare/v1.3.0...v1.4.0
 [1.3.0]: https://github.com/shakacode/heroku-to-control-plane/compare/v1.2.0...v1.3.0
 [1.2.0]: https://github.com/shakacode/heroku-to-control-plane/compare/v1.1.2...v1.2.0

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cpl (2.0.0.rc.0)
+    cpl (2.0.0)
       debug (~> 1.7.1)
       dotenv (~> 2.8.1)
       jwt (~> 2.8.1)
@@ -52,7 +52,7 @@ GEM
     rdoc (6.6.3.1)
       psych (>= 4.0.0)
     regexp_parser (2.9.0)
-    reline (0.5.6)
+    reline (0.5.7)
       io-console (~> 0.5)
     rexml (3.2.6)
     rspec (3.12.0)

data/docs/commands.md

@@ -106,6 +106,7 @@ cpl copy-image-from-upstream -a $APP_NAME --upstream-token $UPSTREAM_TOKEN --ima
 ### `delete`
 
 - Deletes the whole app (GVC with all workloads, all volumesets and all images) or a specific workload
+- Also unbinds the app from the secrets policy, as long as both the identity and the policy exist (and are bound)
 - Will ask for explicit user confirmation
 
 ```sh

data/docs/migrating.md

@@ -47,7 +47,7 @@ Note how the templates correspond to files in the `.controlplane/templates/` dir
 `cpl setup-app` and `cpl apply-template` commands.
 
 Ensure that env vars point to the Heroku add-ons in the template for the app (`.controlplane/templates/app.yml`). See
-[this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/templates/app.yml).
+[this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/templates/gvc.yml).
 
 After that, create a Dockerfile in `.controlplane/Dockerfile` for your deployment. See
 [this example](https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/.controlplane/Dockerfile).

data/docs/redis.md

@@ -1,15 +1,31 @@
-# Migrating Redis database from Heroku infrastructure
+# Migrating Redis databases
 
-**General considerations:**
+There are two templates examples in this repo:
+- `redis` - basic non-persistent template. It is good for review-apps or staging or where no persistence is required
+- `redis2` - basic persistent template. Good for production where persistence is needed, but cluster is overkill.
+
+## Option 1: use SLAVEOF (easier way)
+
+1. create a redis workload that will accept data
+2. execute `SLAVEOF source_host source_port`, if needed use `masterauth` to provide auth details
+3. wait for replication to pick up all changes (usually quickly), use `INFO` or `DBSIZE` to check progress
+4. stop app completely and ensure nothing is writing to any of redises
+5. execute `SLAVEOF no one` to disconnect replication
+6. switch `REDIS_URL` in the app to point to new server
+7. start the app
+
+## Option 2: use Redis-RIOT (harder way, where option 1 is not possible)
+
+### General considerations:
 
 1. Heroku uses self-signed TLS certificates, which are not verifiable. It needs special handling by setting
-TLS verification to `none`, otherwise most apps are not able to connect.
+The tool that satisfies those criteria is [Redis-RIOT](https://developer.redis.com/riot/riot-redis/index.html)
 
 2. We are moving to private Redis that don't have a public URL, so have to do it from a Control Plane GVC container.
 
 The tool that satisfies those criteria is [Redis-RIOT](https://developer.redis.com/riot/riot-redis/index.html)
 
-**Heroku Redis:**
+### Heroku Redis:
 
 As Redis-RIOT says, master redis should have keyspace-notifications set to `KA` to be able to do live replication.
 To do that:
@@ -23,7 +39,7 @@ Connect to heroku Redis CLI:
 heroku redis:cli -a my-app
 ```
 
-**Control Plane Redis:**
+### Control Plane Redis:
 
 Connect to Control Plane Redis CLI:
 
@@ -36,10 +52,10 @@ apt-get update
 apt-get install redis -y
 
 # connect to local cloud Redis
-redis-cli -u MY_CONTROL_PLANE_REDIS_URL
+redis-cli -u MY_CONTROL_PLANE_REDIS_URL -p 6379
 ```
 
-**Useful Redis CLI commands:**
+### Useful Redis CLI commands:
 
 Quick-check keys qty:
 ```
@@ -49,7 +65,7 @@ info keyspace
 db0:keys=9496,expires=2941,avg_ttl=77670114535
 ```
 
-**Create a Control Plane sync workload**
+### Create a Control Plane sync workload
 
 ```
 name: riot-redis
@@ -76,7 +92,7 @@ command args:
   live
 ```
 
-**Sync process**
+### Sync process
 
 1. open 1st terminal window with heroku redis CLI, check keys qty
 2. open 2nd terminal window with controlplane redis CLI, check keys qty

data/lib/command/delete.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Command
-  class Delete < Base
+  class Delete < Base # rubocop:disable Metrics/ClassLength
     NAME = "delete"
     OPTIONS = [
       app_option(required: true),
@@ -11,6 +11,7 @@ module Command
     DESCRIPTION = "Deletes the whole app (GVC with all workloads, all volumesets and all images) or a specific workload"
     LONG_DESCRIPTION = <<~DESC
       - Deletes the whole app (GVC with all workloads, all volumesets and all images) or a specific workload
+      - Also unbinds the app from the secrets policy, as long as both the identity and the policy exist (and are bound)
       - Will ask for explicit user confirmation
     DESC
     EXAMPLES = <<~EX
@@ -48,6 +49,7 @@ module Command
       check_images
       return unless confirm_delete(config.app)
 
+      unbind_identity_from_policy
       delete_volumesets
       delete_gvc
       delete_images
@@ -113,5 +115,21 @@ module Command
         end
       end
     end
+
+    def unbind_identity_from_policy
+      return if cp.fetch_identity(app_identity).nil?
+
+      policy = cp.fetch_policy(app_secrets_policy)
+      return if policy.nil?
+
+      is_bound = policy["bindings"].any? do |binding|
+        binding["principalLinks"].any? { |link| link == app_identity_link }
+      end
+      return unless is_bound
+
+      step("Unbinding identity from policy") do
+        cp.unbind_identity_from_policy(app_identity_link, app_secrets_policy)
+      end
+    end
   end
 end

data/lib/command/run.rb

@@ -87,9 +87,9 @@ module Command
     MAGIC_END = "---cpl run command finished---"
 
     attr_reader :interactive, :detached, :location, :original_workload, :runner_workload,
-                :container, :image_link, :image_changed, :job, :replica, :command
+                :container, :expected_deployed_version, :job, :replica, :command
 
-    def call # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
+    def call # rubocop:disable Metrics/MethodLength
       @interactive = config.options[:interactive] || interactive_command?
       @detached = config.options[:detached]
       @log_method = config.options[:log_method]
@@ -117,11 +117,6 @@ module Command
       update_runner_workload
       wait_for_runner_workload_update
 
-      # NOTE: need to wait some time before starting the job,
-      # otherwise the image may not be updated yet
-      # TODO: need to figure out if there's a better way to do this
-      sleep 1 if image_changed
-
       start_job
       wait_for_replica_for_job
 
@@ -177,41 +172,54 @@ module Command
       end
     end
 
-    def update_runner_workload # rubocop:disable Metrics/MethodLength
-      step("Updating runner workload '#{runner_workload}' based on '#{original_workload}'") do
+    def update_runner_workload # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
+      step("Updating runner workload '#{runner_workload}' based on '#{original_workload}'") do # rubocop:disable Metrics/BlockLength
+        @expected_deployed_version = cp.cron_workload_deployed_version(runner_workload)
+        should_update = false
+
         _, original_container_spec = base_workload_specs(original_workload)
         spec, container_spec = base_workload_specs(runner_workload)
 
         # Override image if specified
         image = config.options[:image]
-        if image
-          image = latest_image if image == "latest"
-          @image_link = "/org/#{config.org}/image/#{image}"
-        else
-          @image_link = original_container_spec["image"]
+        image_link = if image
+                       image = latest_image if image == "latest"
+                       "/org/#{config.org}/image/#{image}"
+                     else
+                       original_container_spec["image"]
+                     end
+        if container_spec["image"] != image_link
+          container_spec["image"] = image_link
+          should_update = true
         end
-        @image_changed = container_spec["image"] != image_link
-        container_spec["image"] = image_link
 
         # Container overrides
-        container_spec["cpu"] = config.options[:cpu] if config.options[:cpu]
-        container_spec["memory"] = config.options[:memory] if config.options[:memory]
+        if config.options[:cpu] && container_spec["cpu"] != config.options[:cpu]
+          container_spec["cpu"] = config.options[:cpu]
+          should_update = true
+        end
+        if config.options[:memory] && container_spec["memory"] != config.options[:memory]
+          container_spec["memory"] = config.options[:memory]
+          should_update = true
+        end
+
+        next true unless should_update
 
         # Update runner workload
+        @expected_deployed_version += 1
         cp.apply_hash("kind" => "workload", "name" => runner_workload, "spec" => spec)
       end
     end
 
     def wait_for_runner_workload_create
       step("Waiting for runner workload '#{runner_workload}' to be created", retry_on_failure: true) do
-        cp.fetch_workload(runner_workload)
+        !cp.cron_workload_deployed_version(runner_workload).nil?
       end
     end
 
     def wait_for_runner_workload_update
       step("Waiting for runner workload '#{runner_workload}' to be updated", retry_on_failure: true) do
-        _, container_spec = base_workload_specs(runner_workload)
-        container_spec["image"] == image_link
+        cp.cron_workload_deployed_version(runner_workload) >= expected_deployed_version
       end
     end
 

data/lib/core/controlplane.rb

@@ -8,6 +8,8 @@ class Controlplane # rubocop:disable Metrics/ClassLength
     @api = ControlplaneApi.new
     @gvc = config.app
     @org = config.org
+
+    ensure_org_exists! if org
   end
 
   # profile
@@ -245,6 +247,16 @@ class Controlplane # rubocop:disable Metrics/ClassLength
     perform_yaml(cmd)
   end
 
+  def cron_workload_deployed_version(workload)
+    current_deployment = fetch_workload_deployments(workload)&.dig("items")&.first
+    return nil unless current_deployment
+
+    ready = current_deployment.dig("status", "ready")
+    last_processed_version = current_deployment.dig("status", "lastProcessedVersion")
+
+    ready ? last_processed_version : nil
+  end
+
   # volumeset
 
   def fetch_volumesets(a_gvc = gvc)
@@ -329,6 +341,11 @@ class Controlplane # rubocop:disable Metrics/ClassLength
     perform!(cmd)
   end
 
+  def unbind_identity_from_policy(identity_link, policy)
+    cmd = "cpln policy remove-binding #{policy} --org #{org} --identity #{identity_link} --permission reveal"
+    perform!(cmd)
+  end
+
   # apply
   def apply_template(data) # rubocop:disable Metrics/MethodLength
     Tempfile.create do |f|
@@ -394,6 +411,18 @@ class Controlplane # rubocop:disable Metrics/ClassLength
 
   private
 
+  def org_exists?
+    items = api.list_orgs["items"]
+    items.any? { |item| item["name"] == org }
+  end
+
+  def ensure_org_exists!
+    return if org_exists?
+
+    raise "Can't find org '#{org}', please create it in the Control Plane dashboard " \
+          "or ensure that the name is correct."
+  end
+
   # `output_mode` can be :all, :errors_only or :none.
   # If not provided, it will be determined based on the `HIDE_COMMAND_OUTPUT` env var
   # or the return value of `Shell.should_hide_output?`.

data/lib/core/controlplane_api.rb

@@ -1,6 +1,10 @@
 # frozen_string_literal: true
 
 class ControlplaneApi # rubocop:disable Metrics/ClassLength
+  def list_orgs
+    api_json("/org", method: :get)
+  end
+
   def gvc_list(org:)
     api_json("/org/#{org}/gvc", method: :get)
   end

data/lib/cpl/version.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
 module Cpl
-  VERSION = "2.0.0.rc.0"
-  MIN_CPLN_VERSION = "0.0.71"
+  VERSION = "2.0.0"
+  MIN_CPLN_VERSION = "2.0.1"
 end

data/lib/cpl.rb

@@ -146,9 +146,22 @@ module Cpl
       ::Command::Base.all_commands.merge(deprecated_commands)
     end
 
+    def self.process_option_params(params)
+      # Ensures that if no value is provided for a non-boolean option (e.g., `cpl command --option`),
+      # it defaults to an empty string instead of the option name (which is the default Thor behavior)
+      params[:lazy_default] ||= "" if params[:type] != :boolean
+
+      params
+    end
+
     @commands_with_required_options = []
     @commands_with_extra_options = []
 
+    ::Command::Base.common_options.each do |option|
+      params = process_option_params(option[:params])
+      class_option(option[:name], **params)
+    end
+
     all_base_commands.each do |command_key, command_class| # rubocop:disable Metrics/BlockLength
       deprecated = deprecated_commands[command_key]
 
@@ -157,7 +170,7 @@ module Cpl
       usage = command_class::USAGE.empty? ? name : command_class::USAGE
       requires_args = command_class::REQUIRES_ARGS
       default_args = command_class::DEFAULT_ARGS
-      command_options = command_class::OPTIONS + ::Command::Base.common_options
+      command_options = command_class::OPTIONS
       accepts_extra_options = command_class::ACCEPTS_EXTRA_OPTIONS
       description = command_class::DESCRIPTION
       long_description = command_class::LONG_DESCRIPTION
@@ -175,12 +188,7 @@ module Cpl
       long_desc(long_description)
 
       command_options.each do |option|
-        params = option[:params]
-
-        # Ensures that if no value is provided for a non-boolean option (e.g., `cpl command --option`),
-        # it defaults to an empty string instead of the option name (which is the default Thor behavior)
-        params[:lazy_default] ||= "" if params[:type] != :boolean
-
+        params = process_option_params(option[:params])
         method_option(option[:name], **params)
       end
 
@@ -208,7 +216,7 @@ module Cpl
         end
 
         begin
-          Cpl::Cli.validate_options!(options, command_options)
+          Cpl::Cli.validate_options!(options)
 
           config = Config.new(args, options, required_options)
 
@@ -227,11 +235,11 @@ module Cpl
     check_unknown_options!(except: @commands_with_extra_options)
     stop_on_unknown_option!
 
-    def self.validate_options!(options, command_options)
+    def self.validate_options!(options)
       options.each do |name, value|
         raise "No value provided for option '#{name}'." if value.to_s.strip.empty?
 
-        params = command_options.find { |option| option[:name].to_s == name }[:params]
+        params = ::Command::Base.all_options.find { |option| option[:name].to_s == name }[:params]
         next unless params[:valid_regex]
 
         raise "Invalid value provided for option '#{name}'." unless value.match?(params[:valid_regex])

data/templates/memcached.yml

@@ -4,8 +4,8 @@ spec:
   type: standard
   containers:
     - name: memcached
-      cpu: 3m
-      memory: 10Mi
+      cpu: 25m
+      memory: 32Mi
       args:
         - "-l"
         - 0.0.0.0
@@ -15,7 +15,7 @@ spec:
           protocol: tcp
   defaultOptions:
     autoscaling:
-      metric: latency
+      metric: disabled
       minScale: 1
       maxScale: 1
     capacityAI: false

data/templates/postgres.yml

@@ -23,7 +23,7 @@ spec:
           uri: "scratch://postgres-vol"
   defaultOptions:
     autoscaling:
-      metric: latency
+      metric: disabled
       minScale: 1
       maxScale: 1
     capacityAI: false

data/templates/redis.yml

@@ -4,15 +4,15 @@ spec:
   type: standard
   containers:
     - name: redis
-      cpu: 3m
-      memory: 20Mi
+      cpu: 25m
+      memory: 32Mi
       image: "redis:latest"
       ports:
         - number: 6379
           protocol: tcp
   defaultOptions:
     autoscaling:
-      metric: latency
+      metric: disabled
       minScale: 1
       maxScale: 1
     capacityAI: false

data/templates/redis2.yml

@@ -0,0 +1,37 @@
+kind: volumeset
+name: redis-data
+spec:
+  fileSystemType: ext4
+  initialCapacity: 10
+  performanceClass: general-purpose-ssd
+---
+kind: workload
+name: redis2
+spec:
+  type: stateful
+  containers:
+    - name: redis
+      args:
+        - '--appendonly'
+        - 'yes'
+        - '--maxmemory'
+        - 25mb
+      cpu: 25m
+      memory: 32Mi
+      image: "redis:latest"
+      ports:
+        - number: 6379
+          protocol: tcp
+      volumes:
+        - path: /data
+          recoveryPolicy: retain
+          uri: cpln://volumeset/redis-data
+  defaultOptions:
+    autoscaling:
+      metric: disabled
+      minScale: 1
+      maxScale: 1
+    capacityAI: false
+  firewallConfig:
+    internal:
+      inboundAllowType: same-gvc

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cpl
 version: !ruby/object:Gem::Version
-  version: 2.0.0.rc.0
+  version: 2.0.0
 platform: ruby
 authors:
 - Justin Gordon
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-05-10 00:00:00.000000000 Z
+date: 2024-05-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: debug
@@ -310,6 +310,7 @@ files:
 - templates/postgres.yml
 - templates/rails.yml
 - templates/redis.yml
+- templates/redis2.yml
 - templates/secrets.yml
 - templates/sidekiq.yml
 homepage: https://github.com/shakacode/heroku-to-control-plane
@@ -328,9 +329,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: 2.7.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubygems_version: 3.4.21
 signing_key: