@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-release-engineering/references/deploy-cli-concrete-recipe.md

@@ -0,0 +1,312 @@
+# Deploy CLI — Concrete Recipe
+
+Working implementation of the deploy CLI described in `custom-control-plane-boundary.md`. Distilled from a production CLI that ships every release of a multi-language platform.
+
+## Surface — 5 commands
+
+```
+deploy-cli compile        Compile source into a build artifact (tar.gz)
+deploy-cli build          Build a Docker image from the artifact, push to registry
+deploy-cli deploy-config  Preview the rendered k8s YAML (optionally save to file)
+deploy-cli review         Create an approval task, optionally wait for the result
+deploy-cli deploy         Apply the deploy via control plane; stream watch events via WebSocket
+```
+
+Each command:
+- Takes a small set of named flags (parsed from struct tags via reflection — see CLI builder below).
+- Calls the control plane's HTTP API.
+- Returns 0 on success, non-zero on failure.
+- Prints structured progress to stdout.
+
+Importantly, the CLI **never** applies deploy/update state directly to k8s, service discovery, mesh control plane, or etcd. Deploy-config, review, deploy, rollback, and traffic-policy changes route through the control plane. Source compilation and image build may call pinned project build tools and the image registry behind typed CLI commands, but long-lived deploy/update behavior must stay declarative and auditable.
+
+## Pipeline-name parsing convention
+
+The CI/CD system passes a `pipeline_name` flag in the form `[lane_name] P.S.M XXXX` or `【lane_name】P.S.M XXXX` (Chinese brackets accepted). The CLI parses both `lane_name` and `PSM` from this single string:
+
+```go
+laneNameReg := regexp.MustCompile(`[\[【]([0-9a-zA-Z-]+)[]】]`)
+psmReg      := regexp.MustCompile(`[0-9a-zA-Z-_]+\.[0-9a-zA-Z-_]+\.[0-9a-zA-Z-_]+`)
+
+// Validate: exactly one lane name + exactly one PSM in the string.
+// Reject otherwise — "[a] [b] x.y.z" or "[a] x.y.z m.n.o" is ambiguous.
+```
+
+Why this convention: external CI/CD platforms (e.g. cloud-provider continuous pipeline) name their pipelines manually with a project-recognizable label. Embedding lane + PSM in the name means the CLI doesn't need separate flags for them, and a glance at the pipeline name tells operators what's deploying where.
+
+If your platform passes lane and PSM as explicit flags, skip this parser. The convention is one less round-trip; explicit flags are clearer.
+
+## Image naming convention
+
+```
+<registry-host>/<namespace>/<psm>:<lane-type>-<image-version>
+```
+
+Where:
+- `registry-host`: private image registry hostname.
+- `namespace`: separate namespace per environment tier (e.g. one for offline lanes, one for online lanes — two-tier organization for RBAC + cleanup policies).
+- `psm`: service PSM as-is.
+- `lane-type`: `prod` (online) or `dev` (offline) — derived from `lane.LaneType`.
+- `image-version`: caller-supplied, typically `<git-commit-sha>` or `<release-tag>`.
+
+This naming lets you:
+- Filter all online images via the namespace prefix.
+- Quickly identify the lane type from the tag.
+- Garbage-collect dev images on a shorter retention than prod.
+
+## Compile flow
+
+```
+deploy-cli compile --workspace <dir> --pipeline_name "[lane] x.y.z ..."
+  │
+  ├─ parse pipeline_name → psm, lane
+  ├─ control plane: GetLaneServiceDetail(lane, psm)
+  │   → buildScript (from cluster config, or service config, or default ./build.sh)
+  │   → runtime (from env RUNTIME, or cluster config, or default e.g. "go:1.25")
+  ├─ execute embedded compile.sh with env vars:
+  │     CP_WORKSPACE, PSM, LANE, BUILD_SCRIPT, RUNTIME
+  │   → runs user's build script (sh "$BUILD_SCRIPT") in workspace
+  │   → packs ./output/* into output.tar.gz in workspace
+  ├─ generate Dockerfile in workspace from embedded template:
+  │     FROM <registry>/<runtime>
+  │     USER <runtime-user>
+  │     WORKDIR /opt/<org>/app
+  │     ADD output.tar.gz .
+  │     ENTRYPOINT ["./bootstrap.sh"]
+  └─ done; tar.gz + Dockerfile ready in workspace
+```
+
+Embedded compile.sh:
+
+```sh
+#!/bin/bash
+set -e
+mkdir -p "$CP_WORKSPACE/.gomod"
+cd "$CP_WORKSPACE"
+rm -rf output/
+sh "$BUILD_SCRIPT"               # user's build script writes to ./output
+cd output
+shopt -s dotglob
+tar cvzf output.tar.gz *
+shopt -u dotglob
+mv output.tar.gz "$CP_WORKSPACE"
+```
+
+The user's `build.sh` (in the service repo) is responsible for writing the actual artifact to `./output`. Compile orchestrates: cache setup, output packing, Dockerfile generation.
+
+`bootstrap.sh` lives inside the user's output tarball — it is the service-specific entrypoint. Compile doesn't generate it.
+
+## Build flow
+
+```
+deploy-cli build --pipeline_name "[lane] x.y.z ..." --image_version <v>
+                 [--registry_user <u> --registry_pwd <p>]
+  │
+  ├─ parse pipeline_name → psm, lane
+  ├─ control plane: ListLane(lane) → lane type → image namespace + tag
+  ├─ registry credentials from flag OR env (REGISTRY_USERNAME / REGISTRY_PASSWORD)
+  ├─ registry host from env (REGISTRY_HOST) — required, no default
+  └─ execute embedded build.sh with env vars:
+        PSM, LANE, NAMESPACE, IMAGE, REGISTRY_USERNAME, REGISTRY_PASSWORD, REGISTRY_HOST
+      → docker login → docker build -t $IMAGE -f Dockerfile .
+                     → docker push $IMAGE → docker rmi $IMAGE
+```
+
+Embedded build.sh:
+
+```sh
+#!/bin/bash
+set -e
+# Use --password-stdin to keep password out of process args / CI command logs.
+echo "$REGISTRY_PASSWORD" | docker login -u "$REGISTRY_USERNAME" --password-stdin "$REGISTRY_HOST"
+docker build -t "$IMAGE" -f Dockerfile .
+docker push "$IMAGE"
+docker rmi "$IMAGE"
+```
+
+**Never pass passwords with `-p` / `--password` flags** — they appear in `ps`, `/proc/<pid>/cmdline`, container event logs, and CI build output. `--password-stdin` is the universal correct form. Pair with:
+- CI secret masking (most CI platforms scrub matching strings from logs).
+- Short-lived registry tokens (rotate per CI run if the registry supports it).
+- Image registry RBAC: build-time tokens have push-only scope to the target namespace, not org-wide.
+
+Why bash not Go: docker CLI is the path of least friction; Go reimplementations of docker daemon interaction are fragile. Embed the script, execute with `exec.CommandContext`.
+
+## Deploy-config flow (preview YAML)
+
+```
+deploy-cli deploy-config --pipeline_name "[lane] x.y.z ..." --image_version <v>
+                          [--use_lane <override>] [--save_path <file>]
+  │
+  ├─ parse pipeline_name → psm, lane
+  ├─ compute image URI
+  ├─ if special PSM (control plane itself): render embedded YAML template
+  │     with placeholders <psm> <appName> <laneName> <image>
+  ├─ else: control plane: GetLaneServiceDeployConfig(lane, psm, image)
+  │     → returns server-rendered YAML
+  ├─ print YAML to stdout
+  └─ if --save_path given: write to file
+```
+
+The preview lets operators eyeball the YAML before deploying — useful for "is this template change going to do what I think" review.
+
+The "special PSM" branch handles the chicken-egg case: when deploying the control plane itself, the control plane API may be unavailable, so the CLI ships an embedded template for those services.
+
+## Review flow
+
+```
+deploy-cli review --workspace_id <id> --pipeline_id <id> --pipeline_name "..."
+                  --pipeline_run_id <id> [--need_approve <n>=1] [--expire <dur>=24h]
+                  [--wait_result]
+  │
+  ├─ validate: pipeline_run_id and pipeline_name required
+  ├─ parse pipeline_name → psm, lane
+  ├─ control plane: CreateReviewTask(
+  │       lane, psm, workspace_id, pipeline_id, pipeline_name, run_id,
+  │       expire, need_approve_num)
+  ├─ control plane sends chat-platform card to reviewers (out-of-band)
+  └─ if --wait_result:
+        poll GetReviewTask(task_id) every 5s
+          tolerate up to 5 consecutive errors before failing
+          until: status != Pending OR elapsed > expire
+        on result: print approver list + decisions
+```
+
+The review task is bound to the **pipeline run id** — one approval per CI/CD pipeline run. The pipeline runs `review --wait_result`, which blocks until approved or expired; then runs `deploy`.
+
+If approval expires or is rejected, `deploy` should not be called. The CI/CD orchestrator enforces this — typically by checking the review command's exit code.
+
+## Deploy flow (the actual k8s apply)
+
+```
+deploy-cli deploy --pipeline_name "[lane] x.y.z ..." --image_version <v>
+                  [--use_lane <override>]
+  │
+  ├─ parse pipeline_name → psm, lane (or use --use_lane override)
+  ├─ compute image URI
+  ├─ open WebSocket to control plane: /v1/deploy_lane_service
+  │   query: ?psm=<psm>&name=<lane>&image=<image>&interactive=true
+  ├─ control plane:
+  │     start k8s Deployment apply
+  │     start k8s.Watch on the Deployment
+  │     for each watch event: format as JSON → write to WebSocket
+  ├─ CLI reads events from WebSocket with 5-min read deadline (reset per message):
+  │     event = { type, app_name, ready_replica, expect_replica,
+  │              condition_* (optional), is_result_event, is_success }
+  │   each event printed as a status line, e.g.:
+  │     [MODIFIED] <app>(2/3)
+  │     [MODIFIED-CONDITION] <app>(2/3), Available - True ...: Deployment available
+  ├─ stop reading on is_result_event=true
+  └─ exit 0 if is_success, else non-zero
+```
+
+This gives the user real-time feedback. A deploy that gets stuck (e.g. ImagePullBackOff) surfaces conditions immediately, not after a timeout.
+
+## CLI builder via reflection (generic)
+
+A reusable pattern for adding new commands without boilerplate:
+
+```go
+func MustBuildCommand[P any](name, usage string, action func(*P) error) *cli.Command
+```
+
+Define a params struct with `json` + `usage` + `default` tags:
+
+```go
+type DeployParams struct {
+    PipelineName string `json:"pipeline_name" usage:"Pipeline name in '[lane] psm ...' format"`
+    ImageVersion string `json:"image_version" usage:"Image version tag"`
+    UseLane      string `json:"use_lane" usage:"Override lane (optional)"`
+}
+
+var Deploy = MustBuildCommand[DeployParams]("deploy", "Deploy a service", deployHandler)
+```
+
+The builder uses reflection to:
+- Read each field's `json` tag → flag name.
+- Read `usage` tag → flag help text.
+- Read `default` tag → default value.
+- Detect field type (string/bool/int/int64/float64) → register the right flag type.
+
+Supported types: `string`, `bool`, `int`, `int64`, `float64`. Slices via comma-separated strings, parsed with helpers (`GetStringSlice`, `GetInt64Slice`, etc.).
+
+Adding a new command is one struct + one handler function + one `MustBuildCommand` line.
+
+## Control plane host override
+
+The CLI looks up the control plane at:
+
+```
+host = env.<CONTROL_PLANE_HOST> || <default-host>
+lane_for_api = env.<CONTROL_PLANE_LANE>   // optional; injected as request header
+```
+
+So a developer testing against a non-prod control plane (e.g. a dev-lane deployment of the control plane itself) sets one env var. The CLI's HTTP middleware adds the `<lane-header>` to outgoing requests, and the mesh routes them to the right lane of the control plane.
+
+This lets the platform team dogfood the control plane in non-prod lanes before promoting it to prod.
+
+**Security**: open host override is a real attack surface. A compromised CI environment that sets `<CONTROL_PLANE_HOST>` to an attacker-controlled URL would have the deploy CLI build + deploy via malicious rendered YAML. Production lanes MUST add safeguards:
+- **Pinned host allowlist**: a build-time-baked list of allowed control-plane hosts. Override permitted only to hosts in the list (typically: prod, pre-prod, staging variants).
+- **TLS identity validation**: pin the control-plane TLS cert fingerprint or require a specific CA; reject unexpected certs even on HTTPS.
+- **Audience-bound tokens**: CLI's SSO token specifies the target host as audience; refuses to send the token to a different host. Prevents token reuse against a redirected control plane.
+- **Break-glass approval for prod override**: setting `<CONTROL_PLANE_HOST>` for a prod lane requires an approval ticket; CLI refuses without it.
+- **Audit log on override**: every CLI invocation with non-default host emits an audit entry to the platform's central audit log (not just local).
+
+## Special-PSM bypass (chicken-and-egg)
+
+A handful of platform services (typically the control plane itself + a secrets/auth service) are marked as "special PSM". For those:
+
+- Image namespace logic uses lane-name prefix instead of querying the control plane (which may not be reachable when deploying itself).
+- Deploy-config uses an embedded YAML template instead of calling the rendering API.
+
+Limit the special-PSM set to the minimum necessary for self-bootstrap. Adding regular services here creates inconsistent paths and audit gaps.
+
+## What this confirms about `custom-control-plane-boundary.md`
+
+Mapping the recommended boundary to this concrete CLI:
+
+| Boundary rule | CLI evidence |
+|---|---|
+| Deploy/update calls API; never k8s/etcd directly | ✅ Deploy-config, review, deploy, rollback, and traffic-policy changes go through `cloudapi.Client` HTTP |
+| Audit every deploy event | ✅ control plane records each deploy; CLI just initiates |
+| WebSocket for live deploy progress | ✅ `/deploy_lane_service` over WS with 5-min read deadline |
+| Approval via chat platform | ✅ control plane sends card; CLI's `review --wait_result` polls task status |
+| Stateless beyond audit + tasks | ✅ CLI is fully stateless; control plane holds task state |
+| Single endpoint, single auth | ✅ `CONTROL_PLANE_HOST` env, lane-header optional |
+| No business logic | ✅ CLI is pure orchestration |
+| Break-glass kubectl open | ✅ Admin can bypass; CLI doesn't gatekeep cluster access |
+| Generic helpers | ✅ Generic `MustBuildCommand[P]` for new commands |
+
+The recommended boundary holds in practice. The CLI is small (~12 files, ~3.5kLOC), maintained by a small platform team, and ships every release.
+
+## Pipeline integration (external CI/CD)
+
+The CLI is designed to be called from a CI/CD pipeline runner (cloud-provider continuous-pipeline / GitLab CI / GitHub Actions / Tekton). A typical pipeline:
+
+```yaml
+stages:
+  - name: compile
+    script: deploy-cli compile --workspace . --pipeline_name "$PIPELINE_NAME"
+  - name: build
+    script: deploy-cli build --pipeline_name "$PIPELINE_NAME" --image_version "$VERSION"
+  - name: review
+    script: deploy-cli review --pipeline_run_id "$RUN_ID" --pipeline_name "$PIPELINE_NAME" --wait_result
+    timeout: 24h    # matches review expire
+  - name: deploy
+    script: deploy-cli deploy --pipeline_name "$PIPELINE_NAME" --image_version "$VERSION"
+```
+
+The pipeline runner provides:
+- `PIPELINE_NAME` (encodes lane + PSM)
+- `VERSION` (image version, typically `$COMMIT_SHA`)
+- `RUN_ID` (unique per pipeline run)
+- `WORKSPACE_ID`, `PIPELINE_ID` (for review task linking)
+
+The pipeline runner handles auth (image-registry credentials, control-plane SSO token) via environment variables. The CLI inherits them.
+
+## Verification
+
+- Run `compile + build + deploy-config + deploy` end-to-end against a dev lane; expect each stage idempotent (re-running same command produces same result).
+- Override `CONTROL_PLANE_HOST` to a stub server; CLI calls hit the stub, no production traffic.
+- Set `CONTROL_PLANE_LANE` to a dev lane; verify the mesh routes CLI's calls to the dev control plane.
+- Kill the control plane mid-deploy; CLI surfaces the disconnection cleanly (not hung indefinitely).
+- Add a new command via `MustBuildCommand[NewParams]("name", ...)`; expect compile + register + work without other code changes.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-release-engineering/references/deploy-pipeline.md

@@ -0,0 +1,165 @@
+# Deploy Pipeline
+
+From source commit to running pod, with audit and rollback.
+
+## Stages
+
+```
+commit
+   │
+   ▼
+build:       source @ commit + version → image digest
+             record: digest, commit, builder version, build time
+   │
+   ▼
+manifest:    image digest + lane policy → k8s YAML (Deployment, Service, etc.)
+             + mesh objects (VirtualService subset, DestinationRule)
+             + discovery lane marker (registry tag or k8s label)
+   │
+   ▼
+approve:     review task (for high-risk classes)
+             approval recorded with reviewer, change summary, rollback plan
+   │
+   ▼
+apply:       control plane writes desired state
+             k8s reconciles
+             mesh + discovery updates propagate
+   │
+   ▼
+verify:      readiness probes pass
+             smoke tests pass
+             SLIs measured during bake window
+   │
+   ▼
+promote OR abort:
+             promote: shift traffic forward (canary → stable)
+             abort:   rollback to previous known-good
+```
+
+Each stage is observable, audit-logged, and re-runnable (or in the case of `apply`, idempotent).
+
+## Three control-plane spines
+
+### A) Custom orchestration API
+
+A platform-team service exposes:
+
+```
+POST   /service                                  create or update service definition
+POST   /lane                                     create lane
+POST   /lane/<lane>/service/<svc>/deploy         deploy a service version into a lane
+POST   /lane/<lane>/service/<svc>/traffic        update traffic weight
+POST   /canary-check-task                         start a canary check
+GET    /canary-check-task/<id>                    status + SLI evidence
+POST   /review-task                                request approval
+POST   /review-task/<id>/approve                   approve
+POST   /service/<svc>/rollback                     rollback to previous
+POST   /service/<svc>/reload-mesh-config           push VirtualService update
+```
+
+A CLI client wraps these. A web UI provides the same surface for non-CLI users.
+
+**Strengths**: deep integration with internal RBAC, audit, lane-first model, custom approval workflow.
+
+**Weaknesses**: you maintain the API and its tools; off-the-shelf reuse is limited.
+
+### B) GitOps (Argo CD / Flux)
+
+Desired state is a tree of YAML in git. A controller in-cluster watches the repo and reconciles k8s state to match.
+
+- Deploys = git PR + merge. Approval = PR review.
+- Rollback = revert the commit.
+- Audit = git history.
+
+**Strengths**: declarative, reusable, no custom code to maintain; built-in review via PR.
+
+**Weaknesses**: PR-based approval is coarse; lane-explosion can clutter the repo; secret handling needs an extra tool (Sealed Secrets, External Secrets Operator).
+
+**OpenGitOps conformance gate**: when picking GitOps as the spine, test the implementation against the 4 OpenGitOps principles (CNCF Sandbox v1.0 documents released by the GitOps Working Group, per `opengitops.dev`): (1) **Declarative** — desired state expressed as data, not imperative scripts; if production state lives in `kubectl apply` shell scripts or "Helm render from git tag, runs on master", the spine is not GitOps even if it uses Argo CD as a viewer. (2) **Immutable & Versioned** — desired state stored with full version history; values stored in mutable ConfigMaps that get overwritten outside git, or environment-specific overrides applied at runtime by an operator, fail this. (3) **Pulled Automatically** (canonical OpenGitOps wording) — agents pull from the source of truth, not push from CI; a CI job that runs `argocd app sync --force` on merge defeats the pull model and turns the agent into a push pipeline. (4) **Continuous Reconciliation** — agents continuously observe and re-apply, not "deploy once and forget"; reconciliation interval, drift-detection alert routing, and self-healing toggle are first-class config decisions, not defaults left untouched. **Self-healing scope MUST be bounded explicitly**: legitimate out-of-band controllers (HPA / VPA setting replica counts and resource requests, cert-manager rotating cert Secrets, image-update operators bumping digests, security-patch automation) write fields that GitOps reconciliation would otherwise revert in a fight. Configure the GitOps agent to ignore controller-owned field paths (`spec.replicas` when HPA is in scope, Secrets owned by external-secrets-operator, image digests owned by an image-updater bot) via the agent's diff-ignore mechanism (Argo CD `ignoreDifferences`, Flux `ignore` annotation). Drift outside the ignore set should ALERT before auto-revert on production resources, not silently re-apply — the team needs the chance to intervene if the drift came from an emergency manual fix. Architecture review on a GitOps-spine platform must verify all 4; meeting only 2-3 produces "GitOps-shaped" deployments that lose half the model's benefit (typical failure: declarative + versioned + pulled, but reconciliation disabled because "drift fixes break things" — that platform is one manual `kubectl` away from silent divergence).
+
+**ArgoCD OCI source (2024-2025)**: per `argo-cd.readthedocs.io/en/latest/user-guide/oci/`, Argo CD now supports OCI artifacts as a first-class application source — desired-state manifests packaged as OCI artifacts (referenced from a registry) sit alongside the git source. This buys: bundle a versioned manifest set in the same registry as container images (one supply-chain, one access policy); cosign-sign the manifest bundle and verify at sync time; cache via the OCI distribution spec the platform already uses for images. **ApplicationSet OCI generator** is in proposal — until shipped, OCI-driven ApplicationSets need the Plugin Generator with a custom OCI puller. **Version cadence**: ArgoCD's documented support policy is patches for the two most recent minors, so when 3.x is the current line the older minors (e.g. 2.13 until 3.1, 2.14 until 3.2) have time-limited maintenance windows, NOT open-ended LTS — schedule the platform-wide upgrade off the older line before its window closes. Coordinate platform-wide ArgoCD upgrades with the v2.14→3.0 migration notes in `argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.14-3.0/` before flipping production clusters; watch the release-cadence page for the current minor before committing a multi-cluster upgrade plan. **Pre-upgrade check**: any ApplicationSet using the Plugin Generator as the OCI workaround needs validation that the proposed native OCI generator (when it ships) matches the plugin's input/output contract — migrating without a contract diff produces silent ApplicationSet behavior change. **Mutable tag footgun (P0 conflict with OpenGitOps "Immutable & Versioned" above)**: OCI artifacts referenced by tag CAN be overwritten in most registries (`docker push` against an existing tag silently replaces the underlying manifest), and an Argo CD sync against a mutable tag will silently pick up the new manifests with no PR / no audit trail / no rollback breadcrumb — defeating the immutability principle this skill demands of the GitOps spine. Required mitigations: (a) reference OCI app sources by digest (`@sha256:...`) or by an immutable-tag policy enforced at the registry (no overwrites on the artifact repo); (b) when tag references are unavoidable, configure Argo CD to require a digest pin via signature verification (cosign-sign + verify-at-sync); (c) treat a tag-only OCI reference on a production app as a platform finding equivalent to a `:latest` image pin — not GitOps-conformant.
+
+### C) Pipeline tool (Spinnaker / Tekton / Jenkins / GitHub Actions)
+
+A pipeline runner executes named stages with gates between.
+
+**Strengths**: long sequenced flows, complex multi-cluster orchestration, broad existing patterns.
+
+**Weaknesses**: pipeline state separate from cluster state; rollback is "run a different pipeline"; less declarative.
+
+## Picking a spine
+
+| Signal | Pick |
+|---|---|
+| Small team, want off-the-shelf | GitOps |
+| Heavy approval / audit / RBAC requirements | Custom API |
+| Long pre-deploy sequences (build, scan, test, multiple envs in order) | Pipeline tool |
+| Lane explosion (>50 named lanes) | Custom API or GitOps with auto-generation |
+| Hybrid budgets | Custom API that emits to GitOps repo; explicit boundary owner |
+
+Do not run two without a documented ownership boundary. The most common production trap is "we have Argo AND a custom deploy API AND a pipeline AND ad-hoc kubectl" — every team picks differently and rollback is undefined.
+
+## CLI shape
+
+A platform CLI is the developer's primary surface regardless of spine. Typical commands:
+
+```
+plat build --name=<svc> --version=<v> [--push]
+plat compile --name=<svc>                       # local build for testing
+plat deploy --lane=<lane> --service=<svc> --version=<v>
+plat deploy-config --lane=<lane> --service=<svc>  # update env-specific config
+plat review --service=<svc> --change=<summary>  # create review task
+plat rollback --service=<svc> [--to-version=<v>]
+plat status --lane=<lane>
+plat logs --lane=<lane> --service=<svc>          # shortcut to log search
+plat trace --log-id=<id>                          # shortcut to trace UI
+```
+
+CLI calls the control-plane API (or commits to git for GitOps). It must work from a developer laptop with no extra setup beyond auth.
+
+## Build pipeline specifics
+
+```
+source@commit + version
+   │
+   ▼
+builder image (pinned version, golden image)
+   │
+   ▼
+Dockerfile path: per language pattern (e.g. golang-app.dockerfile)
+   │
+   ▼
+Build args:
+  - GOPRIVATE / language-private-proxy
+  - GOINSECURE / language-insecure-flag (only if necessary)
+  - GIT_HOST mapping (only if necessary; consider private registry mirror instead)
+   │
+   ▼
+Image tag: <registry>/<svc>:<version>
+   │
+   ▼
+Push to private registry
+   │
+   ▼
+Record: image digest (SHA), commit, build time, builder version
+```
+
+Floating tags (`latest`, branch name) are forbidden in deploys. The control plane references digest, not tag.
+
+Builder image is itself versioned and updated on a schedule. Drift in the builder is invisible until reproduce-from-source fails.
+
+## Idempotency
+
+- `build` with same `(commit, version)` produces the same digest. If it doesn't, the build env is non-deterministic; fix that first.
+- `deploy` with the same `(lane, service, digest)` is a no-op.
+- `rollback` to a digest already running is a no-op.
+
+Idempotency means CI can retry without side effects.
+
+## Failure modes
+
+- Builder image pulled stale → reproducibility breaks. Mitigation: pin and version.
+- Registry credentials expire → CI breaks silently. Mitigation: rotation pipeline.
+- k8s apply fails partway → state mismatch. Mitigation: declarative reconciler that retries.
+- Approval workflow deadlock (everyone offline) → emergency override path documented.
+- Mesh reload non-atomic → some pods serve old VirtualService briefly. Mitigation: connection drain.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-release-engineering/references/env-and-lane-matrix.md

@@ -0,0 +1,126 @@
+# Environment and Lane Matrix
+
+## Why "lane" instead of "environment"
+
+"Environment" suggests a small fixed set: `dev`, `staging`, `prod`. Real platforms grow:
+
+- Per-developer lanes (one developer's in-progress version of a couple of services).
+- Per-integration-test lanes (ephemeral, deleted after test run).
+- Long-lived stages (`dev`, `pre`, `staging`, `prod`).
+- Canary subsets within prod (`prod-canary-1pct`).
+- Shadow traffic targets (`prod-shadow`).
+- Stress-test lanes.
+
+All share routing primitives (registry tag or k8s label plus mesh routing — see `platform-service-connectivity`). The release skill treats them uniformly: each lane is a deploy target.
+
+## Lane object shape
+
+```yaml
+lane:
+  name: <unique-name>
+  parent: <fallback-lane>          # if request lane has no instances, fall to parent
+  type: long-lived | ephemeral | canary | shadow | stress
+  expires_at: <timestamp>           # required for ephemeral; absent for long-lived
+  owner: <team-or-individual>
+  services: [<service-names>]       # services deployed in this lane
+  traffic_weight: <0-100>            # only for canary/shadow subsets within a parent
+  protected: true | false           # protected lanes cannot be deleted without explicit override
+```
+
+Long-lived lanes are protected. Ephemeral lanes auto-delete past expiry.
+
+## Standard long-lived lanes
+
+| Lane | Purpose | Traffic source |
+|---|---|---|
+| `prod` | live user traffic | external clients, ingress |
+| `pre` | pre-production validation | internal smoke + load tests |
+| `staging` | pre-pre, integration tests | CI |
+| `dev` | shared dev environment | engineers, internal tools |
+
+Some platforms add `gray` or `inner-prod` for internal employee dogfooding. Add only when there's a real signal it produces; don't proliferate.
+
+## Per-developer lanes
+
+A developer working on service A wants to test against the real `prod` versions of B, C, D, while running their own A. Lane mechanism:
+
+1. Developer deploys their A under lane `dev-<name>`.
+2. Their requests carry `<lane-header>: dev-<name>`.
+3. Mesh VirtualService routes:
+   - A: to `dev-<name>` subset (their version).
+   - B, C, D: no `dev-<name>` subset → fall back to `prod` (or `dev` shared, depending on policy).
+4. Developer's `dev-<name>` lane expires after N days (e.g. 7) unless renewed.
+
+This lets one developer iterate without cluster-wide effect. Multiplied by team size, it works only if lane creation is cheap (control-plane API, not manual k8s).
+
+## Ephemeral lanes for CI
+
+Integration tests that need real services (not mocks) create a lane per test run:
+
+```
+lane name: it-<test-id>-<commit>
+parent: pre
+type: ephemeral
+expires_at: now + 4h
+services: [services-under-test]
+```
+
+CI deploys the under-test versions to the lane, runs tests with `<lane-header>: it-...`, then deletes the lane. Lane lifetime is bounded by test duration + grace.
+
+## Canary lanes
+
+A canary is a subset of `prod` with its own subset name:
+
+```
+lane name: prod-canary-v2.5.0
+parent: prod
+type: canary
+traffic_weight: 1   # 1% to start
+services: [the-one-being-rolled-out]
+```
+
+VirtualService weight shifts from canary to stable over the rollout window. See `canary-and-rollout-strategy.md`.
+
+## Shadow lanes
+
+Shadow receives a mirror of prod traffic without affecting user response:
+
+```
+lane name: prod-shadow
+parent: prod
+type: shadow
+services: [services-being-shadowed]
+```
+
+Shadow target's responses are dropped. Metrics from shadow MUST be tagged so they don't contaminate prod SLIs.
+
+## Cluster / region mapping
+
+Lanes live in clusters. Common topologies:
+
+| Topology | When |
+|---|---|
+| Single cluster, all lanes | Small platform; everything fits |
+| Per-region cluster, lanes mirrored | Geographic latency or compliance |
+| Per-lane-type cluster | Heavy stress / shadow isolation |
+| Per-team cluster | Late stage; high cost; only when noise isolation is critical |
+
+Document the mapping. Each lane declares its cluster. Cross-cluster lane traffic requires the platform's chosen cross-cluster discovery or mesh federation path (`platform-service-connectivity`).
+
+## Lane policies
+
+Beyond pure routing, lanes carry policy:
+
+- **Resource quotas**: dev / canary lanes get smaller pod counts.
+- **Data isolation**: prod and pre have different DB instances; sharing is a leak.
+- **Network egress**: dev may have looser external access; prod may have stricter.
+- **Observability retention**: prod traces kept 30 days; ephemeral lanes 1 day.
+
+Policies are enforced by namespace labels + admission controllers + mesh AuthorizationPolicy.
+
+## Verification
+
+- `lane list` returns all lanes with type and owner. `lane get <name>` returns services + expiry + traffic weight.
+- A new ephemeral lane is reachable end-to-end within minutes of creation.
+- A lane deleted leaves no orphaned routing/discovery resources (Deployment, Service, VirtualService subset, registry tag or k8s label selector).
+- A request with an unknown lane header falls back to the documented parent lane and logs the fallback (so silent fallback can be detected).