cpflow 5.1.0 → 5.2.0

data/docs/telemetry/index.md

@@ -0,0 +1,152 @@
+# Telemetry
+
+This guide explains how to run application telemetry on Control Plane with
+`cpflow`. It is intentionally generic: the examples use placeholder service
+names, simple metric names, and backend-neutral collector configuration.
+
+`cpflow` does not instrument application code. It helps deploy the Control Plane
+workloads and environment values that already-instrumented applications use to
+send traces, metrics, and logs.
+
+## Quick Navigation
+
+| Page | Use it for |
+| --- | --- |
+| [Collector workload](collector.md) | Control Plane workload template, collector ports, and matching `config.yaml` |
+| [Application instrumentation](application-instrumentation.md) | Generic app env vars and simple Ruby/Node examples |
+| [Pipelines](pipelines.md) | Receivers, processors, exporters, and how signals flow |
+| [Review apps](review-apps.md) | Review-app isolation, sampling, secrets, and egress controls |
+| [Troubleshooting](troubleshooting.md) | Commands and checks for missing telemetry |
+
+## Recommended Shape
+
+```mermaid
+flowchart LR
+  app["Application workload(s)"]
+  collector["Collector workload"]
+  backend["Telemetry backend(s)"]
+  dashboard["Dashboards and alerts"]
+
+  app -->|"OTLP HTTP :4318"| collector
+  app -->|"Optional StatsD TCP :9127"| collector
+  collector -->|"Traces, metrics, logs"| backend
+  backend --> dashboard
+```
+
+Keep the collector inside the same GVC as the application unless your team has a
+separate, intentionally shared telemetry GVC. A same-GVC collector keeps review,
+staging, and production wiring easy to reason about and avoids accidental
+cross-environment data mixing.
+
+## What `cpflow` Provides
+
+`cpflow apply-template` already applies ordinary Control Plane workload
+templates from `.controlplane/templates`. Telemetry can therefore be added with
+project templates and app environment values; no custom `cpflow telemetry`
+command is required.
+
+Use `cpflow` for:
+
+1. Deploying an OpenTelemetry Collector workload template.
+2. Applying GVC or workload environment variables that point apps at the
+   collector.
+3. Reusing the same setup shape for review, staging, and production apps.
+4. Tailing application and collector logs with `cpflow logs`.
+
+Application code is still responsible for:
+
+1. Installing OpenTelemetry, StatsD, Prometheus, or framework-specific
+   instrumentation libraries.
+2. Setting service names and resource attributes.
+3. Creating custom spans or metrics.
+4. Filtering sensitive data before it leaves the process.
+
+## Signal Flow
+
+```mermaid
+flowchart TB
+  subgraph app["Application process"]
+    trace_sdk["Trace SDK"]
+    metric_sdk["Metric SDK"]
+    otlp_logs["OTLP logs"]
+    stdout_logs["stdout/stderr logs"]
+  end
+
+  subgraph collector["Collector workload"]
+    receivers["Receivers"]
+    processors["Processors"]
+    exporters["Exporters"]
+  end
+
+  traces["Trace backend"]
+  metrics["Metrics backend"]
+  logs["OTLP log backend"]
+  platform_logs["Platform log capture"]
+  platform_logs_backend["Platform log backend"]
+
+  trace_sdk -->|"OTLP traces"| receivers
+  metric_sdk -->|"OTLP metrics or StatsD TCP"| receivers
+  otlp_logs -->|"OTLP logs"| receivers
+  stdout_logs -->|"bypasses collector"| platform_logs
+  platform_logs -->|"cpflow logs and historical queries"| platform_logs_backend
+  receivers --> processors
+  processors --> exporters
+  exporters --> traces
+  exporters --> metrics
+  exporters --> logs
+```
+
+stdout/stderr logs bypass the collector and are captured directly by the
+platform. Use `cpflow logs` for live tailing.
+
+For new instrumentation, prefer OTLP over HTTP on port `4318`. It is the most
+portable path because it can carry traces, metrics, and logs through one
+well-known protocol. Use StatsD only when your application already has a simple
+StatsD client or when a metric library cannot emit OTLP yet.
+
+## Minimal Rollout Checklist
+
+1. Add `.controlplane/templates/open-telemetry-collector-secrets.yml` and
+   `.controlplane/templates/open-telemetry-collector.yml`.
+2. Package a collector `config.yaml` that binds every port exposed by the
+   workload template.
+3. Build the collector image and push it to your registry. Update the workload
+   template's `image:` field to that pushed image reference before applying it.
+4. Add both collector templates to `setup_app_templates` in
+   `.controlplane/controlplane.yml` (see
+   [Collector `controlplane.yml`](collector.md#controlplaneyml)):
+   `open-telemetry-collector-secrets` first, then
+   `open-telemetry-collector`.
+5. Add the collector to `additional_workloads` in
+   `.controlplane/controlplane.yml`.
+6. Set application env vars such as `OTEL_SERVICE_NAME`,
+   `OTEL_EXPORTER_OTLP_ENDPOINT`, and `OTEL_EXPORTER_OTLP_PROTOCOL`.
+7. For an existing app, apply both collector templates in the same order:
+   identity and policy first, then the collector workload.
+
+   ```sh
+   cpflow apply-template open-telemetry-collector-secrets -a $APP_NAME
+   cpflow apply-template open-telemetry-collector -a $APP_NAME
+   ```
+
+   For a brand-new app, `cpflow setup-app` applies the configured templates in
+   `setup_app_templates` order.
+8. Confirm collector logs, application exporter logs, and backend ingestion.
+
+The collector workload and its `config.yaml` must be kept in sync. If the
+workload exposes `4318`, `9127`, or `9292`, the collector config must bind those
+same ports. Exposing a port in Control Plane does not automatically enable a
+collector receiver or exporter.
+
+## Generic Naming
+
+Use generic service names, metric names, and resource attributes in reusable
+telemetry templates. See
+[Service Names](application-instrumentation.md#service-names) for examples and
+avoidance rules.
+
+## More Detail
+
+Start with [Collector workload](collector.md), then read
+[Application instrumentation](application-instrumentation.md). Use
+[Troubleshooting](troubleshooting.md) when signals do not appear in the backend.

data/docs/telemetry/pipelines.md

@@ -0,0 +1,98 @@
+# Telemetry Pipelines
+
+Collectors organize telemetry into signal-specific pipelines. Each pipeline has
+receivers, optional processors, and exporters.
+
+```mermaid
+flowchart LR
+  receiver["Receiver"]
+  processor["Processor"]
+  exporter["Exporter"]
+
+  receiver --> processor
+  processor --> exporter
+```
+
+## Pipeline Components
+
+| Component | Role | Examples |
+| --- | --- | --- |
+| Receiver | Accepts telemetry from apps | OTLP HTTP, StatsD TCP |
+| Processor | Modifies, batches, filters, or samples telemetry | `batch`, `memory_limiter`, `attributes` |
+| Exporter | Sends telemetry to a backend or exposes it for scrape | OTLP exporter, Prometheus exporter, debug exporter |
+
+## Recommended Initial Pipelines
+
+```mermaid
+flowchart TB
+  app["Application"]
+  otlp["OTLP receiver :4318"]
+  statsd["Optional StatsD TCP metrics receiver :9127"]
+  traces_processors["Trace processors"]
+  metrics_processors["Metrics processors"]
+  logs_processors["Log processors"]
+  traces["Trace exporter"]
+  metrics_push["Metrics push exporter"]
+  prometheus["Prometheus scrape exporter :9292"]
+  logs["Log exporter"]
+
+  app -->|"OTLP traces, metrics, logs"| otlp
+  app -->|"StatsD metrics only"| statsd
+  otlp -->|"traces"| traces_processors --> traces
+  otlp -->|"metrics"| metrics_processors
+  statsd -->|"metrics"| metrics_processors
+  metrics_processors --> metrics_push
+  metrics_processors --> prometheus
+  otlp -->|"logs"| logs_processors --> logs
+```
+
+Start with direct application telemetry:
+
+1. OTLP traces from the app SDK.
+2. OTLP metrics from the app SDK, or StatsD TCP if that is what the app already
+   supports.
+3. Structured logs to stdout/stderr, or OTLP logs if your logging library
+   supports them.
+
+## Direct Metrics First
+
+Prefer direct metrics from application code when you can change the code. Direct
+metrics are explicit, cheap, and easy to test.
+
+Good direct metrics:
+
+- `example.requests.completed`
+- `example.jobs.duration_ms`
+- `example.queue.depth`
+- `example.cache.hit`
+
+Use low-cardinality labels such as:
+
+- `status`
+- `route_name`
+- `worker`
+- `deployment.environment`
+
+## Derived Metrics
+
+Derived metrics can be useful, but they should not be the default for new work.
+
+| Source | Use when | Caution |
+| --- | --- | --- |
+| Spans | You already have spans and need latency/error histograms | Attribute choices can create high cardinality |
+| Logs | You cannot change legacy code yet | Regex processing is expensive and fragile |
+
+When a derived metric proves valuable, prefer moving it into direct application
+instrumentation later.
+
+## Sampling
+
+Sampling belongs in the collector when you need centralized control. A common
+policy shape is:
+
+1. Keep all error traces.
+2. Keep a small percentage of fast successful traces.
+3. Keep a larger percentage of slow successful traces.
+
+Record sampling decisions in the collector config so agents and humans know why
+some traces are absent.

data/docs/telemetry/review-apps.md

@@ -0,0 +1,55 @@
+# Review Apps
+
+Review-app telemetry should help debug a change without giving untrusted or
+short-lived environments access to production telemetry credentials.
+
+```mermaid
+flowchart TB
+  subgraph review["Review app GVC"]
+    app["Review app workloads"]
+    collector["Review collector"]
+  end
+
+  subgraph staging["Staging telemetry"]
+    backend["Non-production backend"]
+  end
+
+  app --> collector
+  collector --> backend
+```
+
+## Recommended Defaults
+
+1. Use a collector inside each review app GVC, or a staging-only shared collector
+   that is intentionally isolated from production.
+2. Keep collector inbound access internal with `same-gvc` when the collector is
+   inside the review app GVC. For a staging-only shared collector in another
+   GVC, configure explicit cross-GVC ingress instead of broad public access.
+3. Do not give review apps production telemetry tokens.
+4. Use lower sampling rates and shorter retention for review apps.
+5. Avoid sending request bodies, credentials, or personally identifiable
+   information in spans, labels, or logs.
+6. Restrict outbound egress to the non-production telemetry backend.
+
+## Environment Separation
+
+Use resource attributes to make review telemetry easy to filter:
+
+```yaml
+env:
+  - name: OTEL_RESOURCE_ATTRIBUTES
+    value: "deployment.environment=review,service.namespace=example"
+```
+
+If every review app has a unique name, keep that name in a bounded attribute only
+when your backend can handle the cardinality. Do not put pull request titles,
+branch names, user names, or request IDs into metric labels.
+
+## Secrets
+
+Store backend tokens in Control Plane secrets and bind only the collector
+identity that needs them. See [Secrets and ENV Values](../secrets-and-env-values.md)
+for the general Control Plane secret pattern.
+
+Review apps should use non-production tokens. If no non-production token exists,
+keep telemetry local to logs until one is available.

data/docs/telemetry/troubleshooting.md

@@ -0,0 +1,92 @@
+# Telemetry Troubleshooting
+
+Use this checklist when telemetry does not appear in the backend.
+
+Set `APP_NAME` before running the examples:
+
+```sh
+APP_NAME=your-app-name
+```
+
+## 1. Confirm Workloads
+
+```sh
+cpflow ps -a $APP_NAME -w open-telemetry-collector
+cpflow ps -a $APP_NAME -w app
+```
+
+Replace `app` with the workload you are checking.
+
+`cpflow ps` lists running replicas. If the collector is deployed but scaled to
+zero, restart or scale it before debugging collector config or application
+exporter settings.
+
+## 2. Check Collector Logs
+
+```sh
+cpflow logs -a $APP_NAME -w open-telemetry-collector
+```
+
+Look for:
+
+- config parse errors
+- receiver bind failures
+- exporter authentication failures
+- backend DNS or connection failures
+- dropped data due to memory or queue limits
+
+## 3. Check Application Logs
+
+```sh
+cpflow logs -a $APP_NAME -w app
+```
+
+Look for exporter errors from the application SDK. Common causes are:
+
+- wrong `OTEL_EXPORTER_OTLP_ENDPOINT`
+- app instrumentation not enabled
+- collector service name typo
+- unsupported protocol value
+- backend token missing from the collector
+
+## 4. Verify Port Agreement
+
+The Control Plane workload template and collector config must agree.
+
+| Symptom | Check |
+| --- | --- |
+| App cannot export OTLP | Workload exposes `4318`; collector binds `0.0.0.0:4318` |
+| StatsD metrics missing | Workload exposes `9127`; collector has `statsd/tcp` on `0.0.0.0:9127` |
+| Prometheus scrape empty | Workload exposes `9292`; collector has `prometheus` exporter on `0.0.0.0:9292` |
+
+## 5. Verify DNS
+
+From an app workload shell, check that the collector service name resolves:
+
+```sh
+getent hosts open-telemetry-collector.${APP_NAME}.cpln.local
+```
+
+Set `APP_NAME` to the actual app name. If your image does not include `getent`,
+use an equivalent DNS tool available in the image.
+
+## 6. Keep The First Test Simple
+
+Before adding sampling, filtering, derived metrics, or multiple exporters:
+
+1. Start with one app workload.
+2. Send OTLP traces over HTTP.
+3. Export to a debug exporter or one known backend. To enable the debug
+   exporter, uncomment the `debug:` block in your collector config, add `debug`
+   to the relevant pipeline exporters list, and rebuild or remount the config.
+   Remove the debug exporter before production use; it writes full telemetry
+   payloads to collector logs.
+4. Confirm data appears.
+5. Add metrics and logs after traces work.
+
+Use the debug exporter only for short validation windows. It writes full
+telemetry payloads to collector logs, so remove or disable it before sending
+production traffic or attributes that may contain sensitive data.
+
+Small steps make it much easier to tell whether the app, collector, network, or
+backend is the source of the problem.

data/docs/terraform/example/.controlplane/controlplane.yml

@@ -19,7 +19,6 @@ apps:
     <<: *common
     hooks:
       post_creation: bundle exec rake db:prepare
-      pre_deletion: bundle exec rake db:drop
 
   rails-app-production:
     <<: *common

data/docs/terraform/overview.md

@@ -15,6 +15,17 @@ You can continue working with CPLN configuration files in YAML format and start
 3. **Multi-Cloud Support**: With Terraform, you can manage resources across multiple cloud providers seamlessly, allowing for a more flexible architecture.
 4. **Plan and Apply**: Terraform provides a clear plan of what changes will be made before applying them, reducing the risk of unintended modifications.
 
+## Terraform and `cpflow` Are Complementary
+
+Generating Terraform does not replace `cpflow`. The two cover different parts of the workflow, and most teams use both:
+
+- **`cpflow` YAML + `cpflow deploy-image`** fits **ephemeral, fast-moving environments** — Heroku-style review apps in particular. Dynamic per-PR app names, [`cleanup-stale-apps`](/docs/tips.md#delete-or-pause-abandoned-apps-with-cleanup-stale-apps), [`ps:stop`/`ps:start`](/docs/tips.md#pause-and-resume-with-psstop--psstart), and the build → release → deploy flow are first-class, and there is no Terraform state to manage for apps that come and go.
+- **Generated Terraform + `terraform apply`** fits **long-lived, stable environments** (staging and production) where you want a tracked state file, drift detection, explicit dependency ordering, and integration with other Terraform-managed infrastructure.
+
+Because `cpflow terraform generate` produces the Terraform from the same `.controlplane/templates/`, adopting Terraform for your stable environments does not mean maintaining two separate configurations — the YAML templates remain the source of truth, and you regenerate when they change.
+
+A practical split: keep review apps on the `cpflow deploy-image` flow, and manage staging and production with the generated Terraform. `terraform plan` then surfaces drift for Terraform-managed resources before you apply — see [Tips → Keep Templates as the Source of Truth](/docs/tips.md#keep-templates-as-the-source-of-truth).
+
 ## Usage
 
 Let's take a look at how to deploy a [simple Rails application](https://github.com/shakacode/control-plane-flow/tree/main/docs/terraform/example/.controlplane/controlplane.yml) on CPLN using Terraform: