cpflow 5.1.1 → 5.2.0

data/docs/secrets-and-env-values.md

@@ -9,10 +9,33 @@ For your "review apps," it is convenient to have simple ENVs stored in plain tex
 keep some ENVs, like the Rails' `SECRET_KEY_BASE`, out of your source code. For staging and production apps, you will
 set these values directly at the GVC or workload levels, so none of these ENV values are committed to the source code.
 
+## Review app secrets
+
+Review apps run application code from pull requests. For repositories with external contributors, do not put sensitive
+production or long-lived staging secrets in review apps. A secret reference such as `cpln://secret/...` protects the
+value while it is stored in Control Plane configuration, but the value becomes readable by application code after it is
+mounted into the review-app workload. Use generated dummy values, disposable databases, review-only renderer credentials,
+and revocable service tokens for review apps.
+
 For storing ENVs in the source code, we can use a level of indirection so that you can store an ENV value in your source
 code like `cpln://secret/my-app-review-env-secrets.SECRET_KEY_BASE` and then have the secret value stored at the org
 level, which applies to your GVCs mapped to that org.
 
+Avoid pointing review-app templates at shared production secret dictionaries. If staging and review apps live in the same
+Control Plane org, keep review-app secret dictionaries separate from persistent staging secrets, and restrict the Control
+Plane identity bound to review workloads so it can reveal only the values those workloads need.
+
+Review-app identity and policy templates in `.controlplane/templates/`, plus any `shared_secret_grants` in
+`controlplane.yml`, are read from the PR branch at deploy time for same-repository PRs. `cpflow setup-app` applies
+templates at first review-app creation, and `cpflow deploy-image` can bind configured shared-secret policies on later
+redeploys. A PR can modify either path to bind the review workload's identity to any secret dictionary, including staging
+ones. Each generated trigger path has its own fork guard; see
+[Review app security for repositories with external contributors](./ci-automation.md#review-app-security-for-repositories-with-external-contributors)
+for details. Removing any guard leaves one or more trigger paths unprotected.
+
+For repositories with external contributors, treat the entire review-app identity and policy scope as untrusted and ensure
+the staging token cannot reach sensitive resources even if the PR changes the template.
+
 For setting up secrets, you'll need:
 
 - **Org-level Secret:** This is where the values will be stored.
@@ -75,8 +98,12 @@ updated, which helps existing review apps recover after the config is added. `cp
 remove those shared policy bindings when a review app is deleted.
 
 For shared databases, keep runtime data isolated by using a per-review-app database name, schema, or tenant key. A common
-pattern is to keep the host, user, and password in the shared secret, then have `hooks.post_creation` create the PR-specific
-database/schema and `hooks.pre_deletion` drop it.
+pattern is to keep the host, user, and password in the shared secret, then have `hooks.post_creation` create the
+PR-specific database/schema. Avoid a generic `hooks.pre_deletion` that drops the database: `cpflow delete` runs the
+pre-deletion hook before it removes the app workloads, so live connections can make PostgreSQL reject the drop. Stop the
+review app workloads first, or run cleanup from trusted admin automation against the shared Postgres workload. See
+[Share One Control Plane Postgres for Staging and Review Apps](tips.md#share-one-control-plane-postgres-for-staging-and-review-apps)
+for the full pattern.
 
 Here are the manual steps for reference. We recommend that you follow the steps above:
 

data/docs/sidebars.ts

@@ -0,0 +1,70 @@
+import type {SidebarsConfig} from '@docusaurus/plugin-content-docs';
+
+const sidebars: SidebarsConfig = {
+  docsSidebar: [
+    {type: 'doc', id: 'index', label: 'Documentation Guide'},
+    {type: 'doc', id: 'README', label: 'Overview'},
+    {
+      type: 'category',
+      label: 'Start Here',
+      collapsed: false,
+      items: [
+        {type: 'doc', id: 'migrating-heroku-to-control-plane', label: 'Migrate from Heroku'},
+        {type: 'doc', id: 'ci-automation', label: 'Automate GitHub Flow'},
+        {type: 'doc', id: 'ai-github-flow-prompt', label: 'AI Rollout Prompt'},
+        {type: 'doc', id: 'commands', label: 'Command Reference'},
+      ],
+    },
+    {
+      type: 'category',
+      label: 'Core Operations',
+      items: [
+        'secrets-and-env-values',
+        'dns',
+        'thruster',
+        'troubleshooting',
+        'tips',
+      ],
+    },
+    {
+      type: 'category',
+      label: 'Telemetry',
+      link: {type: 'doc', id: 'telemetry/index'},
+      items: [
+        'telemetry/collector',
+        'telemetry/application-instrumentation',
+        'telemetry/pipelines',
+        'telemetry/review-apps',
+        'grafana-opentelemetry',
+        'telemetry/troubleshooting',
+      ],
+    },
+    {
+      type: 'category',
+      label: 'Data Services',
+      items: [
+        'postgres',
+        'rds-private-networking',
+        'redis',
+      ],
+    },
+    {
+      type: 'category',
+      label: 'Terraform',
+      items: [
+        'terraform/overview',
+        'terraform/details',
+      ],
+    },
+    {
+      type: 'category',
+      label: 'Project',
+      items: [
+        {type: 'doc', id: 'releasing', label: 'Releasing the Gem'},
+        {type: 'doc', id: 'changelog', label: 'Changelog'},
+      ],
+    },
+  ],
+};
+
+export default sidebars;

data/docs/telemetry/application-instrumentation.md

@@ -0,0 +1,161 @@
+# Application Instrumentation
+
+Application instrumentation is the code and configuration that creates telemetry
+signals. `cpflow` deploys the infrastructure around that instrumentation, but it
+does not add tracing or metrics libraries to your application.
+
+## Standard Environment Variables
+
+Set these at the GVC level when every app workload should inherit them. Set them
+on one workload container when only that workload should emit telemetry.
+
+The snippets below are fragments, not complete templates. In a `cpflow`
+template, put the `env` list under `spec.env` for GVC-level values or under the
+target workload container's `spec.containers[].env` list for workload-only
+values. When setting values directly in the Control Plane console or with
+`cpln`, replace `{{APP_NAME}}` with the actual app name before applying it.
+
+```yaml
+env:
+  - name: OTEL_EXPORTER_OTLP_ENDPOINT
+    value: "http://open-telemetry-collector.{{APP_NAME}}.cpln.local:4318"
+  - name: OTEL_EXPORTER_OTLP_PROTOCOL
+    value: "http/protobuf"
+  - name: OTEL_RESOURCE_ATTRIBUTES
+    value: "deployment.environment=staging,service.namespace=example"
+# This GVC-level snippet intentionally omits OTEL_SERVICE_NAME.
+# Set OTEL_SERVICE_NAME per workload container, such as example-web or
+# example-worker, and set inheritEnv: true on each container that should
+# receive these shared values.
+```
+
+Change `deployment.environment=staging` to match the real environment. When this
+value is set at the GVC level, each target workload container must set
+`inheritEnv: true` to receive it. Use workload container env instead when only
+one workload should receive the telemetry settings. Keep `OTEL_SERVICE_NAME`
+workload-specific at the container level.
+
+The `http://` collector endpoint is intended for a collector in the same GVC with
+`same-gvc` inbound firewall isolation. Use `https://` only when a shared
+telemetry endpoint terminates TLS; otherwise use the internal collector hostname
+and protocol configured for that collector.
+
+When using HTTP transport (`http/protobuf` or `http/json`), modern stable
+OpenTelemetry SDKs treat `OTEL_EXPORTER_OTLP_ENDPOINT` as a base URL and append
+`/v1/traces`, `/v1/metrics`, or `/v1/logs` automatically. For OTLP over gRPC,
+most stable SDKs expect `http://host:port` for insecure connections and
+`https://host:port` for TLS. Some older or pre-stable SDKs accepted a bare
+`host:port`; check your SDK's documentation. For older or pre-stable SDKs, use
+signal-specific variables such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` with the
+full signal path when needed.
+
+`ENABLE_OPEN_TELEMETRY` is not a standard OpenTelemetry environment variable.
+Use it only if your application code explicitly reads that flag. Otherwise, use
+standard SDK configuration such as `OTEL_SERVICE_NAME` and
+`OTEL_EXPORTER_OTLP_ENDPOINT`, plus your framework's instrumentation setup.
+
+## Optional StatsD TCP Variables
+
+Use these only when your application client can send StatsD over TCP and your
+collector has a matching `statsd/tcp` receiver. These environment variable names
+are examples for application code that reads them; many StatsD clients require
+TCP transport to be configured explicitly in code.
+
+```yaml
+env:
+  - name: STATSD_HOST
+    value: "open-telemetry-collector.{{APP_NAME}}.cpln.local"
+  - name: STATSD_PORT
+    value: "9127"
+  - name: STATSD_PROTOCOL
+    value: "tcp"
+```
+
+`9127` is project-specific. The StatsD protocol default is `8125/UDP`; use the
+port your collector's `statsd/tcp` receiver is configured to bind.
+
+When applying with `cpln` directly, replace `{{APP_NAME}}` with the actual app
+name.
+
+## Generic Ruby Example
+
+This example uses deliberately generic metric names. Keep labels low-cardinality.
+Initialize your client with TCP transport before emitting metrics. The exact
+constructor name varies by library, but do not rely on the client's default
+transport when the collector listens on `statsd/tcp`.
+
+```ruby
+statsd = MyStatsDClient.new(
+  host: ENV.fetch("STATSD_HOST"),
+  port: Integer(ENV.fetch("STATSD_PORT", "9127")),
+  protocol: ENV.fetch("STATSD_PROTOCOL", "tcp")
+)
+
+statsd.increment(
+  "example.tasks.completed",
+  tags: ["task_type:background", "status:success"]
+)
+
+statsd.timing(
+  "example.jobs.duration_ms",
+  elapsed_ms,
+  tags: ["job_type:scheduled"]
+)
+```
+
+Avoid tags such as `user_id`, `request_id`, raw URLs, or free-form error
+messages. They create high-cardinality metrics and can leak sensitive data.
+
+## Generic Node.js Example
+
+```javascript
+const statsdHost = process.env.STATSD_HOST;
+if (!statsdHost) {
+  throw new Error("STATSD_HOST is not set");
+}
+
+const statsd = createStatsDClient({
+  host: statsdHost,
+  port: (() => {
+    const port = parseInt(process.env.STATSD_PORT || "9127", 10);
+    if (!Number.isFinite(port)) {
+      throw new Error(`Invalid STATSD_PORT: ${process.env.STATSD_PORT}`);
+    }
+    return port;
+  })(),
+  protocol: process.env.STATSD_PROTOCOL || "tcp",
+});
+
+statsd.increment("example.requests.completed", 1, {
+  route: "health_check",
+  status: "success",
+});
+
+statsd.histogram("example.worker.duration_ms", elapsedMs, {
+  worker: "default",
+});
+```
+
+## Service Names
+
+Use names that identify the workload, not a real organization, account, or user.
+
+Good:
+
+- `example-web`
+- `example-worker`
+- `example-scheduler`
+
+Avoid:
+
+- real organization names
+- account names
+- user identifiers
+- business-specific nouns copied from one product into reusable docs
+
+## Logs
+
+Keep logs structured, redacted, and useful without a collector. If your
+application sends OTLP logs, use the same collector endpoint as traces and
+metrics. If it logs to stdout/stderr, use `cpflow logs` for live tailing and the
+platform log backend for historical queries.

data/docs/telemetry/collector.md

@@ -0,0 +1,297 @@
+# Collector Workload
+
+The OpenTelemetry Collector is a telemetry router. It receives data from
+application workloads, optionally processes or samples it, and exports it to one
+or more backends.
+
+The two files that must agree are:
+
+1. The Control Plane workload template, which exposes ports.
+2. The collector `config.yaml`, which binds receivers and exporters to those
+   same ports.
+
+If these files disagree, telemetry can fail silently.
+
+## Control Plane Workload Template
+
+Add a template such as
+`.controlplane/templates/open-telemetry-collector.yml`.
+
+```yaml
+kind: workload
+name: open-telemetry-collector
+spec:
+  type: standard
+  containers:
+    - name: open-telemetry-collector
+      image: "registry.example.com/example/open-telemetry-collector:0.155.0"
+      args:
+        - "--config=/etc/otelcol-contrib/config.yaml"
+      # Uncomment when config.yaml references ${env:TELEMETRY_BACKEND_TOKEN}.
+      # env:
+      #   - name: TELEMETRY_BACKEND_TOKEN
+      #     value: cpln://secret/{{APP_NAME}}-telemetry-backend.TELEMETRY_BACKEND_TOKEN
+      cpu: 100m
+      memory: 256Mi
+      ports:
+        # OTLP over HTTP from application SDKs.
+        - number: 4318
+          protocol: http
+        # StatsD over TCP, not UDP. Delete this port unless your collector
+        # config enables statsd/tcp and app clients set TCP transport.
+        # 9127 is project-specific; StatsD defaults to 8125/UDP.
+        # When deleting it, also remove receivers.statsd/tcp and statsd/tcp
+        # from service.pipelines.metrics.receivers in config.yaml.
+        - number: 9127
+          protocol: tcp
+        # Prometheus-formatted metrics exposed by the collector.
+        - number: 9292
+          protocol: http
+  defaultOptions:
+    autoscaling:
+      metric: disabled
+      minScale: 1
+      maxScale: 1
+    capacityAI: false
+  firewallConfig:
+    internal:
+      inboundAllowType: same-gvc
+    external:
+      # Prefer a narrow allowlist for production. Use the hostnames or CIDRs
+      # required by your telemetry backend.
+      outboundAllowHostname:
+        - telemetry-backend.example.com
+  # Use a collector-specific identity when the collector reads telemetry
+  # backend secrets that app workloads should not reveal.
+  identityLink: "//identity/{{APP_NAME}}-otel-collector-identity"
+```
+
+Use `outboundAllowCIDR` instead of `outboundAllowHostname` when your backend
+requires IP ranges. Avoid leaving collector egress open to `0.0.0.0/0` in
+production unless your security model explicitly accepts that risk.
+
+Create the collector identity and a secret policy in a separate
+`open-telemetry-collector-secrets` template before applying the workload
+template if the collector reads backend tokens from Control Plane secrets:
+
+```yaml
+kind: identity
+name: "{{APP_NAME}}-otel-collector-identity"
+description: "{{APP_NAME}}-otel-collector-identity"
+
+---
+kind: policy
+name: "{{APP_NAME}}-otel-collector-secrets"
+description: "{{APP_NAME}}-otel-collector-secrets"
+bindings:
+  - permissions:
+      - reveal
+    principalLinks:
+      - "//gvc/{{APP_NAME}}/identity/{{APP_NAME}}-otel-collector-identity"
+targetKind: secret
+targetLinks:
+  - "//secret/{{APP_NAME}}-telemetry-backend"
+```
+
+Create the `{{APP_NAME}}-telemetry-backend` dictionary secret with a
+`TELEMETRY_BACKEND_TOKEN` key, or replace that name consistently before
+applying the templates. Keep this policy scoped to only the backend secret the
+collector needs.
+
+`cpflow apply-template` replaces `{{APP_NAME}}` with the actual app name. When
+applying the YAML directly with `cpln`, replace `{{APP_NAME}}` manually before
+running `cpln apply`.
+
+Keep the collector image pinned to a tested release and update it as part of
+normal dependency maintenance. Do not rely on a floating `latest` tag for
+production workloads.
+
+## Delivering Collector Config
+
+The official collector image does not include your application-specific
+`config.yaml`. A simple Control Plane pattern is to build a small collector image
+that starts from the pinned upstream image and copies the config into the path
+used by the workload command.
+
+```dockerfile
+FROM otel/opentelemetry-collector-contrib:0.155.0
+
+COPY config.yaml /etc/otelcol-contrib/config.yaml
+```
+
+Build this image, publish it to your registry, and set the workload template
+`image:` field to that published image. The upstream image is only the base image
+for this Dockerfile because it does not contain your `config.yaml`.
+
+Baking the config into the image keeps collector startup deterministic. If your
+team already manages runtime files through Control Plane, you can instead mount a
+secret-backed file with a `cpln://secret/...` URI and `path:` entry, following
+the same pattern used by other file-delivery templates.
+
+## Matching Collector Config
+
+Mount or bake the collector config at the path passed to `--config`. The contrib
+image convention is `/etc/otelcol-contrib/config.yaml`; use a different path only
+when your image or command is built for it.
+
+The collector config must bind every exposed port. This minimal config receives
+OTLP over HTTP, optionally receives StatsD over TCP, and exposes metrics for
+Prometheus scraping.
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4318
+
+  statsd/tcp:
+    endpoint: 0.0.0.0:9127
+    transport: tcp
+    aggregation_interval: 60s  # Use 5s during initial validation.
+
+processors:
+  memory_limiter:
+    check_interval: 1s
+    limit_percentage: 80
+    spike_limit_percentage: 20
+
+  batch: {}
+
+exporters:
+  prometheus:
+    endpoint: 0.0.0.0:9292
+
+  # Replace this placeholder with your real trace/log backend.
+  otlphttp/backend:
+    endpoint: "https://telemetry-backend.example.com"
+    # This is a base URL. The exporter appends /v1/traces, /v1/metrics,
+    # and /v1/logs automatically. Do not include the signal path here.
+    # headers:
+    #   Authorization: "Bearer ${env:TELEMETRY_BACKEND_TOKEN}"
+
+  # Optional validation-only exporter. Do not leave debug in production
+  # pipelines because it writes telemetry payloads to collector logs.
+  # debug:
+  #   verbosity: basic
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlphttp/backend]
+
+    metrics:
+      receivers: [otlp, statsd/tcp]
+      processors: [memory_limiter, batch]
+      exporters: [prometheus, otlphttp/backend]
+
+    logs:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlphttp/backend]
+```
+
+Before applying the config, replace `telemetry-backend.example.com` with your
+real backend endpoint and headers. Keep `memory_limiter` before `batch`, and
+keep the Prometheus exporter only when your platform or backend scrapes collector
+metrics from port `9292`. For push-only metrics backends, remove `prometheus`
+from the metrics pipeline and keep `otlphttp/backend`. If your application only
+sends OTLP metrics and does not use StatsD, also remove the `statsd/tcp`
+receiver block and `statsd/tcp` from the `metrics.receivers` list. See
+[StatsD and UDP](#statsd-and-udp) for the full three-piece removal.
+
+If you have no OTLP push backend at all, remove `otlphttp/backend` from the
+traces, metrics, and logs pipelines and delete the `otlphttp/backend` exporter
+block entirely.
+
+Store backend tokens such as `TELEMETRY_BACKEND_TOKEN` in Control Plane secrets
+and bind them only to the collector workload identity shown in the template
+above. Create the collector identity and a policy that grants `reveal` on only
+the telemetry backend secret before applying the collector workload. When the
+collector config references `${env:TELEMETRY_BACKEND_TOKEN}`, also add the
+matching `env` entry to the collector workload so Control Plane injects the
+secret value at startup. Use the app identity placeholder only when the collector
+does not need secrets that are isolated from app workloads. See
+[Secrets and ENV Values](../secrets-and-env-values.md) for the recommended
+pattern.
+
+The Prometheus exporter exposes an unauthenticated scrape endpoint. With
+`same-gvc` firewall isolation, any workload in the same GVC that can reach port
+`9292` can read exported metrics. Widen collector inbound access only when that
+is acceptable.
+
+Port `9292` is an example scrape port. Configure your scraper to match the port
+you expose in the workload template and bind in the collector config.
+
+## StatsD and UDP
+
+The upstream OpenTelemetry StatsD receiver defaults to UDP on port `8125`.
+Control Plane workload templates in this repository currently use `http` and
+`tcp` ports, not `udp`. Do not document or deploy UDP StatsD on Control Plane
+unless you have verified that the platform supports the required UDP forwarding
+path for your workload.
+
+Safer defaults:
+
+1. Prefer OTLP metrics over HTTP on `4318`.
+2. Use StatsD over TCP only when your app client and collector build support it.
+3. Treat UDP StatsD as an environment-specific advanced option.
+
+If you remove StatsD, remove all three pieces together: the workload port, the
+`statsd/tcp` receiver block, and `statsd/tcp` from the metrics pipeline
+receivers.
+
+## `controlplane.yml`
+
+Include the collector in setup and informational commands:
+
+```yaml
+aliases:
+  common: &common
+    setup_app_templates:
+      - app
+      - worker
+      - open-telemetry-collector-secrets
+      - open-telemetry-collector
+
+    app_workloads:
+      - app
+      - worker
+
+    additional_workloads:
+      - open-telemetry-collector
+
+apps:
+  example:
+    <<: *common
+```
+
+Use `open-telemetry-collector-secrets` for the identity and policy YAML shown
+above. List it before `open-telemetry-collector` so `cpflow setup-app` creates
+the collector identity and secret reveal policy before applying the workload
+that references `identityLink`.
+
+Keep `app_workloads` limited to real application workloads such as `app` and
+`worker`. Put the collector under `additional_workloads` so informational
+commands show it without treating it as an app process.
+
+For an existing app, apply the identity and policy template before applying the
+collector workload:
+
+```sh
+cpflow apply-template open-telemetry-collector-secrets -a $APP_NAME
+cpflow apply-template open-telemetry-collector -a $APP_NAME
+```
+
+For a brand-new app, `cpflow setup-app` applies both collector templates with
+the other entries listed in `setup_app_templates`, in the order shown above.
+
+## Port Agreement Checklist
+
+The Control Plane workload template and collector config must agree on every
+exposed port. See
+[Verify Port Agreement](troubleshooting.md#4-verify-port-agreement) for the
+canonical port checklist. Remove any workload port that is not enabled in the
+collector config.