cpflow 5.1.0 → 5.2.0

data/docs/tips.md

@@ -6,18 +6,28 @@
 4. [CPU](#cpu)
 5. [Remote IP](#remote-ip)
 6. [Secrets and ENV Values](/docs/secrets-and-env-values.md)
-7. [CI](#ci)
-8. [Logs](#logs)
-9. [Memcached](#memcached)
-10. [Sidekiq](#sidekiq)
+7. [Telemetry](#telemetry)
+8. [CI](#ci)
+9. [Logs](#logs)
+10. [Grafana and OpenTelemetry](#grafana-and-opentelemetry)
+11. [Memcached](#memcached)
+12. [Sidekiq](#sidekiq)
     - [Quieting Non-Critical Workers During Deployments](#quieting-non-critical-workers-during-deployments)
     - [Setting Up a Pre Stop Hook](#setting-up-a-pre-stop-hook)
     - [Setting Up a Liveness Probe](#setting-up-a-liveness-probe)
-11. [Minimizing Review App Costs](#minimizing-review-app-costs)
-    - [Scale the Web Workload to Zero](#scale-the-web-workload-to-zero)
+13. [Minimizing Non-Production App Costs](#minimizing-non-production-app-costs)
+    - [Share One Control Plane Postgres for Staging and Review Apps](#share-one-control-plane-postgres-for-staging-and-review-apps)
+    - [Enable Capacity AI for Demo and Starter Staging Apps](#enable-capacity-ai-for-demo-and-starter-staging-apps)
     - [Delete or Pause Abandoned Apps with `cleanup-stale-apps`](#delete-or-pause-abandoned-apps-with-cleanup-stale-apps)
     - [Pause and Resume with `ps:stop` / `ps:start`](#pause-and-resume-with-psstop--psstart)
-12. [Useful Links](#useful-links)
+14. [Right-Sizing Non-Production Workloads](#right-sizing-non-production-workloads)
+    - [Enable Capacity AI on Idle Workloads](#enable-capacity-ai-on-idle-workloads)
+    - [Don't Autoscale Idle Workloads on CPU](#dont-autoscale-idle-workloads-on-cpu)
+    - [Right-Size Reserved CPU and Memory](#right-size-reserved-cpu-and-memory)
+    - [Drop Workloads You Don't Use](#drop-workloads-you-dont-use)
+    - [Share One Postgres Across Non-Production Apps](#share-one-postgres-across-non-production-apps)
+    - [Keep Templates as the Source of Truth](#keep-templates-as-the-source-of-truth)
+15. [Useful Links](#useful-links)
 
 ## GVCs vs. Orgs
 
@@ -114,6 +124,14 @@ So `REMOTE_ADDR` should not be used directly, only `request.remote_ip`.
 > **Warning:** Do not use `REMOTE_ADDR` for authentication, rate limiting, auditing, or IP allowlists. Always use
 > framework-specific mechanisms that understand proxy headers (such as Rails' `request.remote_ip`).
 
+## Telemetry
+
+If your app emits OpenTelemetry, StatsD, or structured log signals, run an
+OpenTelemetry Collector as a Control Plane workload in the same GVC and point
+application env vars at the collector's internal service name. See the
+[telemetry guide](https://www.shakacode.com/control-plane-flow/docs/telemetry/) for the template shape, recommended ports,
+review-app guardrails, and troubleshooting commands.
+
 ## CI
 
 **Note:** Docker builds much slower on Apple Silicon, so try configuring CI to build the images when using Apple
@@ -202,6 +220,15 @@ To check for truncation, compare line count to the limit: `wc -l < incident.log`
 likely cut off. Prefer narrowing the time window (and concatenating the sub-ranges) over raising `--limit`, since the
 server-side cap may be lower than the flag value.
 
+## Grafana and OpenTelemetry
+
+Control Plane's built-in Grafana gives useful workload metrics such as CPU, memory, restarts, and request rate. For
+Rails applications that need app-level request latency, database spans, Redis spans, Sidekiq job metrics, or
+trace-to-log correlation, add OpenTelemetry and an internal collector workload that exposes generated Prometheus
+metrics.
+
+See [Grafana and OpenTelemetry on Control Plane](/docs/grafana-opentelemetry.md) for the full setup guide.
+
 ## Memcached
 
 On the workload container for Memcached (using the `memcached:alpine` image), configure the command with the args
@@ -266,49 +293,328 @@ To do this:
 
 To set up a liveness probe on port 7433, see: https://github.com/arturictus/sidekiq_alive
 
-## Minimizing Review App Costs
+## Minimizing Non-Production App Costs
 
 Long-tail review apps — PRs that linger for days or weeks with little traffic — can drive up Control Plane spend if every
 workload runs full-time. `cpflow` already provides several knobs to manage this without custom orchestration.
+The same cost-control pass applies to public demos, starter staging apps, and long-lived review apps: start with
+Capacity AI for app workloads, then reserve true scale-to-zero for apps where cold starts and planned migrations are
+acceptable.
 
 > **Note:** Scaling workloads to zero or stopping review apps does not reduce costs from external databases, managed
 > Redis instances, object storage, or other third-party services. Those continue to bill independently of Control Plane
 > workload state.
 
-### Scale the Web Workload to Zero
+### Share One Control Plane Postgres for Staging and Review Apps
+
+For non-production Rails apps, a per-GVC Postgres workload is often the largest avoidable review-app cost. Each app can
+end up with its own always-on Postgres replica and its own volume. If staging/review data can be reset, create one
+shared Postgres GVC in the staging org, then point staging and review app GVCs at separate logical databases inside that
+single Postgres instance.
+
+Use separate logical databases per app or review app. Do not point multiple Rails apps at the same database/schema unless
+they intentionally share migrations and data. For example:
 
-`templates/rails.yml` ships with `type: standard`, `minScale: 1`, `maxScale: 1`. That's a safe default for production,
-but for review apps where cold-start latency is acceptable you can switch the web workload to a serverless type that
-scales to zero replicas when idle. Apply the snippet below to your project's `.controlplane/templates/rails.yml`, or
-create a review-app-specific template (for example `rails-review.yml`) and list it under `setup_app_templates` for the
-review-app entry in `.controlplane/controlplane.yml`.
+```text
+Shared GVC (in the staging org):
+  staging-shared-postgres
+    postgres workload
+    shared-postgres-vs volume
+
+Client GVCs (each points to a separate logical database):
+  react-webpack-rails-tutorial-staging
+  react-webpack-rails-tutorial-review-pr-123
+  react-on-rails-starter-staging
+  react-on-rails-starter-review-pr-123
+```
+
+The shared Postgres workload must accept internal traffic from other GVCs. `same-gvc` is not enough when the database
+lives in a separate GVC; use `same-org`, or `workload-list` if you can keep an explicit allowlist current. `same-org` is
+convenient for trusted staging orgs, but every workload in the org can reach the database port, including production
+workloads if production GVCs share the same org. Use `workload-list` for a tighter blast radius if you can automate
+entries as review apps appear and disappear.
 
 ```yaml
-# Only `type` and `minScale` change from templates/rails.yml; `maxScale`, `capacityAI` and `timeoutSeconds`
-# are shown for context so the full `defaultOptions` block reaches the destination intact.
-# Update the relevant fields in your full templates/rails.yml (or a review-app-specific template); keep
-# containers, firewallConfig, identityLink, and everything else from that file intact.
+kind: volumeset
+name: shared-postgres-vs
+spec:
+  fileSystemType: ext4
+  initialCapacity: 10
+  performanceClass: general-purpose-ssd
+  snapshots:
+    createFinalSnapshot: true
+    retentionDuration: 7d
+    # Periodic snapshots need a schedule; without one, only a final snapshot is taken when the volumeset is deleted.
+    schedule: "0 2 * * *" # daily at 02:00 UTC; adjust to your retention needs
+
+---
+kind: workload
+name: postgres
+spec:
+  type: stateful
+  containers:
+    - name: postgres
+      image: postgres:17 # pin a specific patch (e.g. postgres:17.x) for reproducible stateful deploys
+      cpu: 250m
+      memory: 512Mi
+      env:
+        - name: PGDATA
+          value: /var/lib/postgresql/data/pg_data
+        - name: POSTGRES_DB
+          value: postgres
+        - name: POSTGRES_USER
+          value: postgres
+        - name: POSTGRES_PASSWORD
+          # Recommended after adding the workload identity/policy binding:
+          # value: cpln://secret/shared-postgres-password.password
+          # Plain-value fallback for disposable non-production experiments only.
+          # Do not commit this file with a real password in place.
+          value: REPLACE_WITH_NON_PRODUCTION_PASSWORD
+      ports:
+        - number: 5432
+          protocol: tcp
+      volumes:
+        - uri: cpln://volumeset/shared-postgres-vs
+          path: /var/lib/postgresql/data
+          recoveryPolicy: retain # keep the volume if the workload is deleted; clean up manually when no longer needed
+  defaultOptions:
+    autoscaling:
+      metric: disabled
+      minScale: 1
+      maxScale: 1
+    capacityAI: false
+  firewallConfig:
+    internal:
+      inboundAllowType: same-org
+```
+
+`POSTGRES_DB: postgres` initializes the administrative database for the server. Apps should use their own logical
+databases, not the administrative `postgres` database.
+
+Field note: `100m` CPU and `256Mi` memory were enough for tiny Rails migrations, but a real staging seed that inserted
+hundreds of thousands of rows caused Postgres to log `server process ... terminated by signal 9: Killed`. `250m` and
+`512Mi` handled the same seed while still replacing multiple always-on per-app Postgres workloads.
+
+For review apps, keep the logical database name unique per app. `cpflow`'s default app template uses `{{APP_NAME}}` in
+both the Postgres host and database name:
+
+```yaml
+- name: DATABASE_URL
+  value: postgres://the_user:the_password@postgres.{{APP_NAME}}.cpln.local:5432/{{APP_NAME}}
+```
+
+Create a review-only app template by copying `.controlplane/templates/app.yml` to
+`.controlplane/templates/app-review.yml`. In that copy, point `DATABASE_URL` at a per-review-app Control Plane secret.
+Trusted automation should create the secret with a full URL whose database name is still that review app's
+`{{APP_NAME}}`:
+
+```yaml
+spec:
+  env:
+    - name: DATABASE_URL
+      value: cpln://secret/{{APP_NAME}}-database.DATABASE_URL
+```
+
+Control Plane's `cpln://secret/...` syntax replaces the entire env value; it is not substring interpolation, so avoid
+forms such as `postgres://cpln://secret/...@...`. A full per-app URL secret also avoids depending on any runtime
+ordering between secret resolution and `$(VAR)` env-var expansion. Because `{{APP_NAME}}-database` is a separate secret
+from the app dictionary secret, trusted automation must create that secret and add it to the app identity's reveal policy
+before the workload starts; otherwise workloads cannot resolve the `cpln://secret/...` value. For trusted staging/review
+apps where a single shared database role is acceptable, you can still create one URL secret per app that reuses the same
+database user/password while keeping the database name unique.
+
+The `cpln://secret/NAME.FIELD` field syntax resolves only against **dictionary** secrets; an `opaque` or `tls` secret
+leaves the workload with an empty or literal `cpln://...` string rather than a clear error. Define the database secret as
+a dictionary, apply it with `cpln apply -f secret.yaml`, and confirm the app identity's policy grants `reveal` on it
+before the workload starts:
+
+```yaml
+kind: secret
+name: my-app-review-pr-123-database
+type: dictionary
+data:
+  DATABASE_URL: postgres://the_user:the_password@postgres.staging-shared-postgres.cpln.local:5432/my-app-review-pr-123
+```
+
+`cpflow` can automate this secret-and-policy wiring. Declare a `shared_secret_grants` entry on the review app and
+reference the generated `{{SHARED_SECRET_DATABASE}}` placeholder in your templates instead of hardcoding the secret name;
+`cpflow setup-app`, `deploy-image`, `delete`, and `cleanup-stale-apps` then keep the policy binding and cleanup automatic.
+See [Shared Secrets for Review Apps](secrets-and-env-values.md#shared-secrets-for-review-apps) for the full setup.
+
+`{{APP_NAME}}` keeps databases separate by convention, not by itself as a security boundary. If review apps can run
+untrusted PR code, do not give every review app the same database role with `CREATEDB` or ownership of every review
+database. Prefer one of these safer models:
+
+1. Create a database and role per review app, store that app's URL/credentials in its DB secret, and grant the role only to
+   its own database.
+2. Keep review app database roles low-privilege and run create/drop cleanup from trusted admin automation against the
+   shared Postgres workload.
+
+A single shared role/password is acceptable only for trusted staging apps or review apps where database separation is a
+cost-control convenience rather than a security boundary. The hook example below assumes the review app role is allowed
+to create its own logical database; if you choose admin-owned cleanup, run create/drop steps from trusted automation
+instead of from the review app workload.
+
+Then point the review-app entry at the review-only template and remove the per-PR Postgres workload:
+
+```yaml
+my-app-review:
+  match_if_app_name_starts_with: true
+  setup_app_templates:
+    # postgres removed, so no per-PR database workload is created
+    - app-review   # was: app
+    - redis
+    - rails
+  additional_workloads:
+    - redis        # postgres removed
+  hooks:
+    post_creation: bundle exec rails db:prepare
+```
+
+The `post_creation` hook creates only that review app's logical database because the database name is still
+`{{APP_NAME}}`. Do not rely on a generic `pre_deletion: rails db:drop` hook for shared databases: `cpflow delete` runs
+the pre-deletion hook before it removes or suspends the app workloads, so live Rails/worker processes can still hold
+connections and make PostgreSQL reject the drop. Stop the review app workloads first, or run trusted admin cleanup
+against the shared Postgres workload with `DROP DATABASE ... WITH (FORCE)`.
+
+Rails apps with multiple production databases need each connection isolated. Either set connection-specific URLs such as
+`CACHE_DATABASE_URL`, `QUEUE_DATABASE_URL`, and `CABLE_DATABASE_URL`, or make the database names in `config/database.yml`
+derive from an app-specific environment variable. If the database names are hard-coded, every review app for that repo
+will collide inside the shared Postgres instance.
+
+Suggested cutover order:
+
+1. Create the shared Postgres GVC, workload, and volume.
+2. Create the app roles and logical databases, or make sure the app role has `CREATEDB` and let `rails db:prepare`
+   create them. For admin-created databases, generate the password in trusted automation, store it in the matching app DB
+   secret, then run the setup from an interactive `psql` session so the password is not written into shell history or
+   process arguments:
+
+   ```sh
+   cpln workload exec postgres --org ORG --gvc staging-shared-postgres --stdin --tty -- psql -U postgres
+   ```
+
+   Then enter the SQL in `psql`:
+
+   ```sql
+   CREATE ROLE "my-app-staging" LOGIN;
+   \password "my-app-staging"
+   CREATE DATABASE "my-app-staging" OWNER "my-app-staging";
+   \connect "my-app-staging"
+   GRANT ALL ON SCHEMA public TO "my-app-staging";
+   ```
+
+   In CI, run equivalent SQL through a secret-aware step that does not echo the password, add it to process arguments, or
+   persist it in logs.
+
+3. Update staging/review GVC environment values to the shared host.
+4. Run `cpflow run -a APP -- bin/rails db:prepare` for each app.
+5. Force redeploy app workloads so live replicas pick up the new GVC env.
+6. Stop the old per-app Postgres workloads and smoke test the apps.
+7. Delete the old Postgres workloads and volumes only after smoke tests pass.
+8. When a review app is deleted, drop its logical database from the shared instance so orphaned review databases do not
+   accumulate. For the most reliable cleanup, stop the app workloads first, then run the drop from trusted admin
+   automation or directly against the shared Postgres workload. Use `WITH (FORCE)` on PostgreSQL 13+ to terminate
+   remaining sessions:
+
+   ```sh
+   cpln workload exec postgres --org ORG --gvc staging-shared-postgres -- \
+     psql -U postgres -c 'DROP DATABASE IF EXISTS "my-app-review-pr-123" WITH (FORCE);'
+   ```
+
+   `cpln workload exec` runs `psql` inside the container over its local Unix socket, which the official Postgres image
+   grants the `postgres` superuser `trust` auth — so no `PGPASSWORD` or `-W` flag is required here.
+
+When updating URL-like env values, prefer applying a full GVC YAML update with `cpln apply`, then re-read the GVC env to
+confirm the new reference took effect:
+
+```sh
+cpln apply -f my-app-review-pr-123-gvc.yaml
+cpln gvc get my-app-review-pr-123 -o yaml | grep DATABASE_URL
+```
+
+`cpln gvc update --set` also works, but treat it as a known-fragile shortcut. Quote the entire `path=value` expression,
+or the CLI can leave the old value in place while the command appears superficially successful. The `spec.env.NAME.value`
+path relies on env-array lookup by name, so verify against your installed CLI version before relying on it:
+
+```sh
+cpln gvc update my-app-review-pr-123 \
+  --set 'spec.env.DATABASE_URL.value=cpln://secret/my-app-review-pr-123-database.DATABASE_URL'
+```
+
+A few tradeoffs remain even after the cost savings:
+
+- **Noisy neighbor risk.** All staging/review apps share one server's CPU, RAM, disk, and connection pool. A runaway
+  query or connection leak in one app can affect the others; per-app connection caps or PgBouncer can help. Mind
+  Postgres's own `max_connections` (default 100): a staging seed running alongside several review apps, each at Rails'
+  default `pool: 5`, can exhaust it before any query runs. The official image ignores a `POSTGRES_MAX_CONNECTIONS` env
+  var; raise the server limit with a `-c max_connections=N` server argument or a custom `postgresql.conf`, and lower
+  `pool:` in `config/database.yml` for review apps as the simplest app-side lever.
+- **Operational ownership.** Backups, restores, password rotation, sizing, and access control move to the shared server.
+- **Other trusted services can use the same pattern.** Redis and Memcached can also be shared for trusted apps, but a
+  per-app key prefix or logical database index is only conventional separation when apps share credentials. If review
+  app code is not trusted, use enforced isolation such as per-app ACL users/credentials or separate instances.
+
+### Enable Capacity AI for Demo and Starter Staging Apps
+
+`templates/rails.yml` ships with CPU autoscaling pinned to one replica (`minScale: 1`, `maxScale: 1`) and
+`capacityAI: false`. That's a conservative production-safe default, but for public demos, starter staging apps, and
+long-lived review apps, Capacity AI can right-size CPU and memory allocation while keeping the same warm replica count.
+For these non-production apps, keep the Rails workload as `type: standard`, disable the explicit autoscaling metric,
+and enable Capacity AI. Apply the snippet below to your project's `.controlplane/templates/rails.yml`, or create an
+environment-specific template (for example `rails-review.yml` or `rails-demo-staging.yml`) and list it under
+`setup_app_templates` for the matching app entry in `.controlplane/controlplane.yml`.
+
+```yaml
+# Only `autoscaling.metric` and `capacityAI` change from templates/rails.yml.
+# `type: standard` is shown here to confirm this is not a serverless migration.
+# Keep containers, firewallConfig, identityLink, and everything else from the template intact.
 kind: workload
 name: rails
 spec:
-  type: serverless
+  type: standard
   defaultOptions:
     autoscaling:
-      minScale: 0
+      minScale: 1
       maxScale: 1
-    capacityAI: false    # keep your existing value
-    timeoutSeconds: 60   # keep your existing value
+      metric: disabled
+    capacityAI: true
 ```
 
-See [`templates/rails.yml`](/templates/rails.yml) for the full default — `containers`, `firewallConfig`,
+See [`templates/rails.yml`](https://github.com/shakacode/control-plane-flow/blob/main/templates/rails.yml) for the full default — `containers`, `firewallConfig`,
 `identityLink`, and the other required fields must be preserved when you copy the snippet above.
 
-Control Plane spins the workload back up on the next request. Only `type: serverless` workloads support `minScale: 0`;
-`type: standard` always keeps at least one replica running.
+This is not the same as scale-to-zero. Capacity AI can reduce over-allocation for mostly idle demos, but it will not
+make costs approach zero when a workload has steady RAM usage or background load. Expect it to settle over several
+hours, and treat memory sizing as a separate cost lever.
+
+Shared Postgres is the usual exception: keep shared databases manually sized rather than enabling Capacity AI
+indiscriminately. Apply this guidance to stateless app/service workloads first (Rails, renderers, workers, and similar
+staging-only services). Stateful workloads are not supported by Capacity AI, so keep stateful Redis, Elasticsearch,
+Mongo, and similar support services manually sized unless you intentionally deploy them as supported stateless
+workloads.
+
+If you intentionally need true idle scale-to-zero, use a separate `type: serverless` workload with `minScale: 0` and
+an HTTP wake-up autoscaling metric such as `rps` or `concurrency`:
+
+```yaml
+kind: workload
+name: rails
+spec:
+  type: serverless
+  defaultOptions:
+    autoscaling:
+      minScale: 0
+      maxScale: 1
+      metric: rps
+      target: 1
+```
+
+Existing `type: standard` workloads cannot change to `serverless` in place; that requires a planned delete/recreate
+migration and can interrupt traffic.
 
-Tradeoff: the first request after a quiet period pays the cold-start cost (typically 15–60 seconds for a Rails
-image, depending on app size and boot configuration). For review apps that's usually fine; for production it
-usually isn't.
+> **Warning:** Treat a `standard` to `serverless` conversion as an operational migration because deleting a running
+> workload can interrupt traffic.
 
 > **Note:** if you later suspend the app with `cpflow ps:stop`, Control Plane will not auto-wake it on the next
 > request. Run `cpflow ps:start` explicitly first. See
@@ -390,7 +696,130 @@ No re-deploy is needed; the workloads come back with the same images they had be
 > keep running while only the web tier sleeps. `cpflow ps:stop -a $APP_NAME` suspends every configured workload, web
 > included, and `cleanup-stale-apps --mode=stop` applies the same pause behavior to stale review apps.
 
+## Right-Sizing Non-Production Workloads
+
+[Minimizing Non-Production App Costs](#minimizing-non-production-app-costs) above focuses on review-app lifecycle
+controls: scale-to-zero, explicit pauses, and stale app cleanup. Long-lived staging and demo apps are the other common
+source of avoidable Control Plane spend: they tend to keep generously-sized workloads running full-time. The levers
+below apply to any non-production environment (staging, demos, and review apps alike).
+
+### Enable Capacity AI on Idle Workloads
+
+Control Plane bills the CPU and memory a running replica *reserves*. With `minScale: 1` and
+Capacity AI off, a workload reserves its full `cpu`/`memory` around the clock, even when the
+app is idle. **Capacity AI** lets Control Plane right-size that reservation toward actual
+usage, so an idle non-production workload costs a fraction of its ceiling.
+
+Set it in `defaultOptions`:
+
+```yaml
+kind: workload
+name: rails
+spec:
+  defaultOptions:
+    capacityAI: true
+```
+
+Also disable CPU-utilization autoscaling for idle non-production workloads; the
+next section shows the complete `capacityAI` and autoscaling shape together.
+
+Tradeoff: Control Plane reprovisions the replica when it adjusts the reservation. For
+stateless web/renderer workloads that's negligible. For stateful workloads, see the
+[guidance above](#enable-capacity-ai-for-demo-and-starter-staging-apps) — Postgres,
+Redis, Elasticsearch, Mongo, and similar services should remain manually sized.
+
+### Don't Autoscale Idle Workloads on CPU
+
+CPU-utilization autoscaling adds nothing for an idle non-production app and works against
+Capacity AI. Disable it and let Capacity AI handle right-sizing:
+
+```yaml
+kind: workload
+name: rails
+spec:
+  defaultOptions:
+    capacityAI: true
+    autoscaling:
+      metric: disabled
+      minScale: 1
+      maxScale: 1
+```
+
+(For the web tier you can go further and scale to zero — see
+[Enable Capacity AI for Demo and Starter Staging Apps](#enable-capacity-ai-for-demo-and-starter-staging-apps).)
+
+### Right-Size Reserved CPU and Memory
+
+The shipped templates use production-leaning defaults. Check each workload's reserved
+`cpu`/`memory` against its real usage — the workload's **Metrics** tab in Control Plane
+shows Grafana CPU/memory graphs — because non-production workloads are routinely
+over-provisioned.
+
+Postgres is the usual offender: a demo or staging database does **not** need a full core.
+Pinning `cpu: 1000m` keeps a whole reserved CPU running 24/7, while an idle Postgres
+typically sits at single-digit millicores. Something like `cpu: 250m` / `memory: 512Mi`
+is a field-tested non-production starting point; raise memory toward `1Gi` if the
+workload's Metrics tab shows pressure during seeds, imports, or larger staging datasets.
+
+```yaml
+kind: workload
+name: postgres
+spec:
+  containers:
+    - name: postgres
+      cpu: 250m
+      memory: 512Mi
+```
+
+### Drop Workloads You Don't Use
+
+Every workload listed under `app_workloads` / `additional_workloads` is another full-time
+container. Remove the ones a non-production app doesn't actually need.
+
+A common one is a separate background-job worker when the app has no jobs to run. On Rails
+8, [Solid Queue](https://github.com/rails/solid_queue) can run inside Puma instead of as its
+own workload — set `SOLID_QUEUE_IN_PUMA=true` when the app uses the Rails 8 default
+`config/puma.rb`, or add `plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]` manually for
+apps upgraded from Rails 7. Then drop the `worker` workload from `app_workloads` and
+`setup_app_templates` in `.controlplane/controlplane.yml`, and delete its template.
+Solid Queue is database-backed, so job processing needs no Redis; if your app uses Redis for caching or Action Cable,
+keep that workload.
+
+### Share One Postgres Across Non-Production Apps
+
+Running a dedicated Postgres workload — and its SSD volume — for every staging and review
+app multiplies standing cost. For non-production, several apps can share a single Postgres
+server, each using its own database:
+
+- Point each app's `DATABASE_URL` environment variable (in `.controlplane/templates/`) at the shared instance — for
+  example `postgres://user:pass@postgres.staging-shared-postgres.cpln.local:5432/my_app_staging` — and give each app a
+  distinct database name in the path.
+- Set `inboundAllowType` to the narrowest scope that covers your use case — `workload-list` gives the tightest blast
+  radius when you can keep an explicit per-workload allowlist current, while `same-org` is the practical default when
+  client apps are too dynamic to maintain manually. `same-gvc` only works when the shared Postgres and every client app
+  live in the same GVC, which is not the cross-GVC setup described in
+  [Share One Control Plane Postgres for Staging and Review Apps](#share-one-control-plane-postgres-for-staging-and-review-apps).
+  Overly broad allow-types expand the attack surface, especially when review apps can run untrusted PR code.
+- Store shared database credentials in Control Plane secrets for long-lived staging and demos; plaintext
+  `DATABASE_URL` values are only reasonable for disposable non-production experiments.
+- Prefer per-app database roles over a shared superuser or broad `CREATEDB` role, especially when review apps can run
+  untrusted PR code.
+
+A managed alternative is a single small RDS instance hosting many databases; see
+[Hetzner RDS Postgres](https://pelle.io/posts/hetzner-rds-postgres).
+
+### Keep Templates as the Source of Truth
+
+It's tempting to tune `cpu`, `capacityAI`, or autoscaling directly in the Control Plane UI.
+Don't: `cpflow apply-template` reconciles workloads from your `.controlplane/templates/`, so console edits are
+overwritten when it runs next; non-interactive CI runs with `--yes` do that silently, while interactive runs prompt
+before re-creating existing workloads. Make cost changes in the templates and deploy them.
+
+If you want drift caught automatically, manage long-lived environments with Terraform via
+[`cpflow terraform`](/docs/terraform/overview.md) — `terraform plan` reports any difference
+between the repo and live infrastructure before you apply.
+
 ## Useful Links
 
 - For best practices for the app's Dockerfile, see: https://lipanski.com/posts/dockerfile-ruby-best-practices
-- For migrating from Heroku Postgres to RDS, see: https://pelle.io/posts/hetzner-rds-postgres
+- For Hetzner RDS Postgres, see: https://pelle.io/posts/hetzner-rds-postgres

data/examples/controlplane.yml

@@ -135,6 +135,8 @@ apps:
       post_creation: bundle exec rake db:prepare
 
       # Used by the command `cpflow delete` to run a hook before deleting the app.
+      # For a shared database, prefer admin-side cleanup instead: `cpflow delete` runs this hook before removing the
+      # workloads, so live connections can block the drop. See docs/tips.md ("Share One Control Plane Postgres").
       pre_deletion: bundle exec rake db:drop
 
   my-app-production:

data/lib/command/ai_github_flow_prompt.rb

@@ -30,9 +30,9 @@ module Command
 
     def prompt
       <<~PROMPT
-        Set up Control Plane GitHub Flow for this repo. Start with `cpflow github-flow-readiness` and stop on any reported blockers. The repo must be deployable from a clean clone: published package versions, complete runtime scaffold, and a production Dockerfile that can build the app. If any package version is unpublished, inaccessible from CI, or requires credentials that are not already modeled in the repo or GitHub settings, stop and report the blocker instead of generating workflow files. If the repo is a legacy sample pinned to an obsolete Ruby or Bundler toolchain, if it does not even have a production Dockerfile yet, or if it is a monorepo without an already-decided single app boundary for this flow, stop and report that as a prerequisite instead of forcing the rollout.
+        Set up Control Plane GitHub Flow for this repo. First make sure the `cpflow` CLI is available: use the repo's existing `bundle exec cpflow` if present, otherwise install the published `cpflow` Ruby gem with `gem install cpflow`; if neither is possible, stop and report that blocker. Use the same `cpflow` invocation for the rest of the rollout. Start with `cpflow github-flow-readiness` and stop on any reported blockers. The repo must be deployable from a clean clone: published package versions, complete runtime scaffold, and a production Dockerfile that can build the app. If any package version is unpublished, inaccessible from CI, or requires credentials that are not already modeled in the repo or GitHub settings, stop and report the blocker instead of generating workflow files. If the repo is a legacy sample pinned to an obsolete Ruby or Bundler toolchain, if it does not even have a production Dockerfile yet, or if it is a monorepo without an already-decided single app boundary for this flow, stop and report that as a prerequisite instead of forcing the rollout.
 
-        If `.controlplane/` is missing, run `cpflow generate`. Treat the generated app names as the repo-name default (`#{inferred_app_prefix}`) and rename them only if the project needs a different prefix. Then run `cpflow generate-github-actions` (or `cpflow generate-github-actions --staging-branch BRANCH` when staging should deploy from a branch other than `main`/`master`), keep review apps opt-in via `+review-app-deploy`, make sure any `STAGING_APP_BRANCH` repository variable is also present in the generated staging workflow's `on.push.branches` filter, and list the GitHub secrets and variables that must be configured. Do not hand-edit duplicated upstream refs into the generated wrappers: the only downstream Control Plane Flow pin should be the reusable workflow `uses: ...@vX.Y.Z` value generated from the installed `cpflow` gem version, and upstream workflows load their matching shared actions automatically. When bumping the `cpflow` gem in a downstream repo, run `cpflow update-github-actions` (or `bundle exec cpflow update-github-actions`) and validate with `bin/test-cpflow-github-flow` in the same PR so the checked-in wrappers move to the matching release tag. Keep the standard path simple: review apps require only `CPLN_TOKEN_STAGING` when the generated review app config can be inferred. For shared review-app resources such as one staging database, use `shared_secret_grants` and `{{SHARED_SECRET_DATABASE}}` placeholders instead of hardcoding the base app secret name; this keeps review-app policy binding and cleanup automatic while avoiding per-PR database cost. Document the one-time Control Plane bootstrap command for persistent staging and production apps with `cpflow setup-app --skip-post-creation-hook`; for existing apps or later template updates, document `cpflow apply-template` and the need for the app identity to have `reveal` on the app secret policy. Do not imply the staging deploy or promotion workflows create those persistent GVCs. For production promotion, document a protected `production` GitHub Environment with required reviewers, prevent self-review, and `CPLN_TOKEN_PRODUCTION` stored as an environment secret, not as a repository or organization secret.
+        If `.controlplane/` is missing, run `cpflow generate`. Treat the generated app names as the repo-name default (`#{inferred_app_prefix}`) and rename them only if the project needs a different prefix. Then run `cpflow generate-github-actions` (or `cpflow generate-github-actions --staging-branch BRANCH` when staging should deploy from a branch other than `main`/`master`), keep review apps opt-in via `+review-app-deploy`, make sure any `STAGING_APP_BRANCH` repository variable is also present in the generated staging workflow's `on.push.branches` filter, and list the GitHub secrets and variables that must be configured. Do not hand-edit duplicated upstream refs into the generated wrappers: the only downstream Control Plane Flow pin should be the reusable workflow `uses: ...@vX.Y.Z` value generated from the installed `cpflow` gem version, and upstream workflows load their matching shared actions automatically. When bumping the `cpflow` gem in a downstream repo, run `cpflow update-github-actions` (or `bundle exec cpflow update-github-actions`) and validate with `bin/test-cpflow-github-flow` in the same PR so the checked-in wrappers move to the matching release tag. Keep the normal generated review-app setup simple: review apps require only `CPLN_TOKEN_STAGING` when the generated review app config can be inferred. For public demos, starter staging apps, and long-lived review apps, keep the app workload `type: standard` with one warm replica, set its autoscaling metric to `disabled`, and enable `capacityAI: true` so Control Plane can right-size CPU and memory allocation at that fixed replica count. Shared Postgres and other stateful workloads are the usual exceptions and should stay manually sized; Capacity AI is for supported stateless app/service workloads. If true idle scale-to-zero is explicitly required, create a separate `serverless` workload before first deploy or plan a delete/recreate migration because Control Plane will not change an existing `standard` workload to `serverless` in place. For shared review-app resources such as one staging database, use `shared_secret_grants` and `{{SHARED_SECRET_DATABASE}}` placeholders instead of hardcoding the base app secret name; this keeps review-app policy binding and cleanup automatic while avoiding per-PR database cost. Document the one-time Control Plane bootstrap command for persistent staging and production apps with `cpflow setup-app --skip-post-creation-hook`; for existing apps or later template updates, document `cpflow apply-template` and the need for the app identity to have `reveal` on the app secret policy. Do not imply the staging deploy or promotion workflows create those persistent GVCs. For production promotion, document a protected `production` GitHub Environment with required reviewers, prevent self-review, and `CPLN_TOKEN_PRODUCTION` stored as an environment secret, not as a repository or organization secret.
 
         Keep Node available in the final image if asset compilation or SSR depends on ExecJS, Yarn, `pnpm`, or npm after the main install layer. Make sure the generated Dockerfile uses a Ruby base image compatible with the app's declared Ruby requirement. Preserve repo-defined frontend build hooks: if `config/shakapacker.yml` defines a `precompile_hook`, or React on Rails enables `config.auto_load_bundle = true`, confirm the generated Dockerfile runs that codegen step before `rails assets:precompile`. If `config/database.yml` shows SQLite in production, confirm that the generated scaffold uses persistent `db` and `storage` volumes plus a release script that runs `rails db:prepare`; otherwise keep the default Postgres workload. If the public workload is not named `rails`, set `PRIMARY_WORKLOAD` or adjust the generated workflows. Inspect the Dockerfile and package sources for private GitHub dependencies or `RUN --mount=type=ssh`; if present, wire `DOCKER_BUILD_SSH_KEY`, optionally set `DOCKER_BUILD_SSH_KNOWN_HOSTS` for non-GitHub SSH hosts, and keep `DOCKER_BUILD_EXTRA_ARGS` to newline-delimited single tokens such as `--build-arg=FOO=bar`.
 

data/lib/command/base.rb

@@ -49,7 +49,7 @@ module Command
 
     def self.all_commands # rubocop:disable Metrics/MethodLength
       Dir["#{__dir__}/**/*.rb"].each_with_object({}) do |file, result|
-        content = File.read(file)
+        content = File.read(file, encoding: "UTF-8")
 
         classname = content.match(/^\s+class (?!Base\b)(\w+) < (?:.*(?!Command::)Base)(?:$| .*$)/)&.captures&.first
         next unless classname
@@ -96,7 +96,7 @@ module Command
       }
     end
 
-    def self.workload_option(required: false)
+    def self.workload_option(required: false, repeatable: false)
       {
         name: :workload,
         params: {
@@ -104,6 +104,7 @@ module Command
           banner: "WORKLOAD_NAME",
           desc: "Workload name",
           type: :string,
+          repeatable: repeatable,
           required: required
         }
       }
@@ -583,6 +584,20 @@ module Command
       success
     end
 
+    def wait_for_workloads_ready(workloads, reverse: false)
+      workloads_to_wait_for = reverse ? workloads.reverse : workloads
+
+      workloads_to_wait_for.each do |workload|
+        if cp.workload_suspended?(workload)
+          progress.puts("Workload '#{workload}' is suspended. Skipping...")
+        else
+          step("Waiting for workload '#{workload}' to be ready", retry_on_failure: true) do
+            cp.workload_deployments_ready?(workload, location: config.location, expected_status: true)
+          end
+        end
+      end
+    end
+
     def cp
       @cp ||= Controlplane.new(config)
     end