@1aboveio/skills 0.0.0-stage → 1.0.0

package/skills/cicd/cloud-build/SKILL.md

@@ -0,0 +1,707 @@
+---
+name: cloud-build
+description: "Build or debug Cloud Build pipelines for Cloud Run: cloudbuild.yaml, triggers, Artifact Registry, builds, migrations, deploys, secrets, VPC, notifications. Use cloud-deploy for staged promotion. NOT for GitHub Actions PR checks (github-actions)."
+release-version: 0.36.0
+---
+**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
+# Cloud Build Direct Deploy to Cloud Run
+
+## Related Skills
+
+- [google-cloud](../google-cloud/SKILL.md) - Centralized GCP operations
+  - [IAM patterns](../google-cloud/references/iam.md) - Service accounts, roles, IAP, token minting
+  - [VPC configuration](../google-cloud/references/vpc.md) - Networks, subnets, Cloud Run with VPC
+  - [Secrets Manager](../google-cloud/references/secrets.md) - Secret setup, access, Cloud Run integration
+- [cloud-deploy](../cloud-deploy/SKILL.md) - Complex deployments with Cloud Deploy + Skaffold
+- [cloud-debug](../cloud-debug/SKILL.md) - Debugging Cloud Build/Deploy failures
+
+## When to Use
+
+- Single container deployments
+- Services handling own migrations (e.g., Airflow standalone)
+- No need for staged promotion with verification
+
+For complex deployments with migrations and verification, use `cloud-deploy` skill.
+
+## Architecture
+
+**GitFlow (default)** — a long-lived integration branch plus a production branch:
+
+```
+dev push ────→ Cloud Build ────→ Cloud Run (dev)
+PR to master ─→ Cloud Build ────→ Cloud Run (staging)
+v*-rc.* tag ──→ Cloud Build ────→ Cloud Run (prod, 0% traffic candidate)
+```
+
+**GitHub Flow** — one long-lived branch; see [GitHub Flow variant](#github-flow-variant) for the trigger set and what changes:
+
+```
+PR opened ─────→ Cloud Build ────→ build + test only, no deploy
+merge to main ─→ Cloud Build ────→ Cloud Run (staging, 100% traffic)
+v*-rc.* tag ───→ Cloud Build ────→ Cloud Run (prod, 0% traffic candidate)
+```
+
+**The prod fork is identical in both.** It keys off the rc tag, not off a branch — which is the practical payoff of triggering production on a tag: the release path stops caring what the branching model is.
+
+## Environment Configuration
+
+### Triggers
+
+- **Dev**: `dev` branch push — active development
+- **Staging**: PR to `master` — PR validation
+- **Prod**: `v*-rc.*` tag push — builds a candidate revision at 0% traffic; the release lane verifies it and shifts traffic
+
+### Database
+
+- **Dev**: Dedicated dev DB — migrations skipped for faster iteration
+- **Staging**: Dedicated staging DB — runs migrations + test seeds
+- **Prod**: Production DB — runs migrations + prod seeds
+
+Dev skips migrations/seeds during build for speed. Cloud Run Jobs are still configured so they can be triggered manually when needed.
+
+**For Cloud SQL connection patterns, see [google-cloud skill VPC reference](../google-cloud/references/vpc.md) — Cloud SQL Connection Standard.**
+
+### Build Steps
+
+All environments run: **setup env → build → update job → deploy → test**
+
+Staging + Prod additionally run: **migrate → seed**
+
+Step descriptions:
+- **setup env**: Initialize environment-specific variables (VPC, secrets)
+- **build**: Build and push Docker image to Artifact Registry
+- **update job**: Configure Cloud Run Jobs (for migrate/seed) — does not execute them
+- **migrate**: Execute database migration job
+- **deploy**: Deploy to Cloud Run service
+- **seed**: Execute database seed job
+- **test**: Run integration tests against deployed service
+
+### Migration Best Practice: Skip When Schema Inputs Are Unchanged
+
+If migrations run through Cloud Build, prefer a single combined **migrate-if-needed** step rather than separate "check", "update job", and "execute job" steps. The step should:
+
+1. Hash only the files that define database shape
+2. Compare that hash to the last applied hash stored outside the build workspace
+3. Skip migration image build, job update, and job execution when the hash is unchanged
+
+Use a persistent marker such as a GCS object:
+
+```text
+gs://$PROJECT_ID_cloudbuild/migrate-markers/{service}-${_ENV}.digest
+```
+
+Why:
+- Avoids slow no-op migration runs on every deploy
+- Works with Cloud Build's shallow clone and source mirror limitations
+- Keeps migration execution tied to actual schema changes rather than every app change
+
+Do not hash the whole repo. Hash only migration-relevant inputs for the ORM in use.
+
+Practical rules:
+- Do **not** gate migrations with `git diff HEAD^ HEAD`. Merge commits, shallow clones, and repository-event checkouts can compare against the wrong parent and produce false positives.
+- The **first** build after adopting digest-based gating should usually run migrations once, because there is no previous marker yet. That is expected marker seeding, not a failure of the gate.
+- After a successful migration job, write the new digest marker even if the job only reports "database schema is up to date". Otherwise every later build will look like a first run.
+- Prefer `gcr.io/google.com/cloudsdktool/cloud-sdk:slim` and keep the skip path to a single always-started SDK step. A split "decide" + "maybe-run" pattern is usually slower for the common unchanged-schema case.
+- If Cloud Build UI still shows a migration step, check whether it truly executed a Cloud Run Job or only started a step that exited early. Those are different.
+
+### Prisma Example
+
+For Prisma, prefer hashing:
+- `prisma/schema.prisma`
+- `prisma/migrations/**`
+
+If the digest is unchanged, skip all migration job work. A pragmatic pattern is a single slim SDK step that:
+- computes the digest
+- compares it with the stored marker
+- exits immediately when unchanged
+- otherwise deploys and executes the migration job
+- writes the new digest back to GCS after success
+
+Example:
+
+```yaml
+  - name: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
+    id: migrate-if-needed
+    entrypoint: bash
+    args:
+      - -ceu
+      - |
+        digest_file="/workspace/migration-digest.txt"
+        marker="gs://${PROJECT_ID}_cloudbuild/migrate-markers/{service}-${_ENV}.digest"
+        current_digest="$$( \
+          { \
+            printf '%s\0' prisma/schema.prisma; \
+            find prisma/migrations -type f -print0; \
+          } | sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1 \
+        )"
+        previous_digest="$$(gcloud storage cat "$${marker}" 2>/dev/null || true)"
+
+        if [[ -n "$${previous_digest}" && "$${previous_digest}" == "$${current_digest}" ]]; then
+          echo "Prisma schema unchanged; skipping migrate job."
+          exit 0
+        fi
+
+        printf '%s' "$${current_digest}" > "$${digest_file}"
+
+        migration_script="$$(cat <<'EOF'
+        status_output=$$(pnpm prisma migrate status 2>&1 || true)
+        echo "$${status_output}"
+        if grep -qi "Database schema is up to date" <<<"$${status_output}"; then
+          echo "Database already up to date"
+          exit 0
+        fi
+        if grep -qi "Following migration" <<<"$${status_output}" || grep -qi "have not yet been applied" <<<"$${status_output}"; then
+          pnpm prisma migrate deploy
+          exit 0
+        fi
+        echo "Unexpected prisma migrate status output"
+        exit 1
+        EOF
+        )"
+
+        gcloud run jobs deploy {service}-${_ENV}-migrate \
+          --image=$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}:$COMMIT_SHA \
+          --region=$LOCATION \
+          --command=bash \
+          --args=-ceu \
+          --args="$${migration_script}"
+
+        gcloud run jobs execute {service}-${_ENV}-migrate --region=$LOCATION --wait --format=none
+        gcloud storage cp "$${digest_file}" "$${marker}"
+```
+
+### Tortoise ORM + Aerich Example
+
+For Tortoise ORM, there is no single canonical schema file like Prisma. Hash:
+- `aerich.ini` or equivalent Aerich config
+- `migrations/**` or `aerich/**` migration files
+- the model packages that define Tortoise tables
+
+The same preference applies here: a combined digest-check + migrate-if-needed step is usually better than separate decision and execution steps unless the pipeline has a strong reason to split them.
+
+Example:
+
+```yaml
+  - name: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
+    id: check-migrations
+    entrypoint: bash
+    args:
+      - -ceu
+      - |
+        marker="gs://${PROJECT_ID}_cloudbuild/migrate-markers/{service}-${_ENV}.digest"
+        current_digest="$$( \
+          { \
+            printf '%s\0' aerich.ini; \
+            find migrations -type f -print0 2>/dev/null || true; \
+            find app/models -type f \\( -name '*.py' -o -name '*.sql' \\) -print0; \
+          } \
+          | sort -z \
+          | xargs -0 sha256sum \
+          | sha256sum \
+          | cut -d' ' -f1)"
+        previous_digest="$$(gcloud storage cat "$${marker}" 2>/dev/null || true)"
+
+        if [[ -n "$${previous_digest}" && "$${previous_digest}" == "$${current_digest}" ]]; then
+          echo "Aerich inputs unchanged; skipping migrate job."
+          exit 0
+        fi
+
+        touch /workspace/.migrate-needed
+        printf '%s' "$${current_digest}" > /workspace/.new-migration-digest
+```
+
+For Aerich, keep the migration command deterministic inside the Cloud Run Job:
+
+```bash
+aerich upgrade
+```
+
+If you generate migrations in CI, treat that as a separate workflow with tighter review controls. For normal deploys, prefer committing generated migration files and only applying them in Cloud Build.
+
+## cloudbuild.yaml
+
+```yaml
+substitutions:
+  _ENV: dev
+  _MIN_INSTANCES: '0'
+  _DATABASE_URL_SECRET: {service}-database-url-${_ENV}
+  _SECRET_KEY: {service}-secret-key
+  # VPC configuration per environment
+  _VPC_NETWORK_DEV: 'default'
+  _VPC_SUBNET_DEV: 'default'
+  _VPC_NETWORK_STAGING: 'default'
+  _VPC_SUBNET_STAGING: 'default'
+  _VPC_NETWORK_PROD: 'prod'
+  _VPC_SUBNET_PROD: 'prod'
+
+steps:
+  # Initialize environment-specific variables
+  - name: 'bash'
+    env:
+      - '_ENV=${_ENV}'
+      - '_VPC_NETWORK_DEV=${_VPC_NETWORK_DEV}'
+      - '_VPC_NETWORK_STAGING=${_VPC_NETWORK_STAGING}'
+      - '_VPC_NETWORK_PROD=${_VPC_NETWORK_PROD}'
+      - '_VPC_SUBNET_DEV=${_VPC_SUBNET_DEV}'
+      - '_VPC_SUBNET_STAGING=${_VPC_SUBNET_STAGING}'
+      - '_VPC_SUBNET_PROD=${_VPC_SUBNET_PROD}'
+    args:
+      - '-c'
+      - |
+        ENV_UPPER=$$(echo "$$_ENV" | tr '[:lower:]' '[:upper:]')
+        NETWORK_VAR="_VPC_NETWORK_$$ENV_UPPER"
+        SUBNET_VAR="_VPC_SUBNET_$$ENV_UPPER"
+        VPC_NETWORK="$${!NETWORK_VAR}"
+        VPC_SUBNET="$${!SUBNET_VAR}"
+        echo "VPC_NETWORK=$$VPC_NETWORK" > /workspace/env.sh
+        echo "VPC_SUBNET=$$VPC_SUBNET" >> /workspace/env.sh
+        echo "Initialized: VPC network=$$VPC_NETWORK, subnet=$$VPC_SUBNET"
+
+  # Build the Docker image
+  - name: 'gcr.io/cloud-builders/docker'
+    args:
+      - 'build'
+      - '-t'
+      - '$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}:$COMMIT_SHA'
+      - '-t'
+      - '$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}:latest'
+      - '.'
+
+  # Push to Artifact Registry
+  - name: 'gcr.io/cloud-builders/docker'
+    args:
+      - 'push'
+      - '--all-tags'
+      - '$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}'
+
+  # Deploy to Cloud Run
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    entrypoint: bash
+    args:
+      - '-c'
+      - |
+        source /workspace/env.sh
+        gcloud run deploy {service}-${_ENV} \
+          --image=$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}:$COMMIT_SHA \
+          --region=$LOCATION \
+          --platform=managed \
+          --min-instances=${_MIN_INSTANCES} \
+          --max-instances=1 \
+          --memory=2Gi \
+          --cpu=1 \
+          --port=8080 \
+          --cpu-boost \
+          --no-cpu-throttling \
+          --timeout=300 \
+          --set-secrets=DATABASE_URL=${_DATABASE_URL_SECRET}:latest,SECRET_KEY=${_SECRET_KEY}:latest \
+          --network=$${VPC_NETWORK} \
+          --subnet=$${VPC_SUBNET}
+
+options:
+  logging: CLOUD_LOGGING_ONLY
+  machineType: E2_HIGHCPU_8
+```
+
+## Infrastructure Configuration
+
+For IAM, VPC, Secrets, and IAP patterns, see the [google-cloud](../google-cloud/SKILL.md) skill:
+
+- **VPC**: [google-cloud/vpc.md](../google-cloud/references/vpc.md) - Network/subnet setup for Cloud Run
+- **Secrets**: [google-cloud/secrets.md](../google-cloud/references/secrets.md) - Secret Manager setup and Cloud Run integration
+- **IAM/IAP**: [google-cloud/iam.md](../google-cloud/references/iam.md) - Service accounts, roles, IAP access, token minting
+
+### Build Triggers
+
+```bash
+# Dev
+gcloud builds triggers create github --name="{service}-dev" \
+  --repo-name="REPO" --repo-owner="OWNER" --branch-pattern="^dev$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=dev,_MIN_INSTANCES=0"
+
+# Staging
+gcloud builds triggers create github --name="{service}-staging" \
+  --repo-name="REPO" --repo-owner="OWNER" --pull-request-pattern="^master$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=staging,_MIN_INSTANCES=0"
+
+# Prod — fires on release-candidate tags minted by the release lane, not on branch push
+gcloud builds triggers create github --name="{service}-prod" \
+  --repo-name="REPO" --repo-owner="OWNER" --tag-pattern="^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=prod,_MIN_INSTANCES=1"
+```
+
+**Footgun: the prod tag pattern must not match the final `v<version>` tag.** The release lane mints `v1.4.0-rc.1` to produce a candidate, and then — after the candidate verifies ALIVE and traffic is at 100% — mints `v1.4.0`. A loose pattern like `^v.*$` matches both, so the final tag fires a second production build that rebuilds and redeploys an already-live release at 0% traffic for no reason. Anchor the `-rc.<n>` suffix, as above, and check the pattern against both tag shapes before creating the trigger.
+
+Dev and staging triggers are unchanged: `dev` still builds on branch push, staging still builds on PRs to `master`.
+
+### GitHub Flow variant
+
+One long-lived branch (`main`), short-lived feature branches, no `dev`. Three triggers, the same `cloudbuild.yaml`:
+
+```bash
+# PR — build and test only. No deploy: a PR is not an environment.
+gcloud builds triggers create github --name="{service}-pr" \
+  --repo-name="REPO" --repo-owner="OWNER" --pull-request-pattern="^main$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=ci"
+
+# Staging — every merge to main deploys at 100% and records the SHA-keyed verdict
+gcloud builds triggers create github --name="{service}-staging" \
+  --repo-name="REPO" --repo-owner="OWNER" --branch-pattern="^main$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=staging,_MIN_INSTANCES=0"
+
+# Prod — byte-identical to the GitFlow prod trigger
+gcloud builds triggers create github --name="{service}-prod" \
+  --repo-name="REPO" --repo-owner="OWNER" --tag-pattern="^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$" \
+  --build-config="cloudbuild.yaml" --substitutions="_ENV=prod,_MIN_INSTANCES=1"
+```
+
+**`_ENV=staging` takes the dev fork's behavior, not the GitFlow staging behavior.** It deploys at 100% and runs the smoke as a **non-blocking recorder** publishing a `dev-smoke`-style commit status — because under GitHub Flow this environment *is* the integration environment. Reuse the [dev fork step](#integration-dev-deploy-100-record-the-verdict-never-fail-the-build) verbatim, matching on `staging` instead of `dev`. Everything else — build, migrations, seeds, the prod fork — is unchanged.
+
+**What the recorded verdict means here is weaker, and that is the one thing to understand.** In GitFlow, the promoted SHA and the validated SHA are the same commit: `dev` is validated, and the promotion fast-forwards it into `main`. Under GitHub Flow there is only one branch, so the release lane cuts `release/<version>` at a validated SHA, merges it back into `main`, and the **shipped SHA is that merge commit** — which also carries anything else that landed on `main` meanwhile. Those commits never earned a verdict.
+
+So under GitHub Flow the SHA-keyed staging verdict is a **pre-filter, not the release gate**. The gate is the prod candidate at 0% traffic, which validates the *exact shipped SHA* on real infrastructure before any user reaches it. That is sound precisely because of the 0%-candidate design — under a deploy-then-smoke pipeline the same topology would ship unvalidated commits straight to users.
+
+**Order is forced: merge first, then tag.** The release tag must name a SHA on the production branch, so the release branch cannot be rc-tagged before it lands on `main`. Merge the release PR, then mint the rc on the merge commit. `rush-release` owns this sequencing.
+
+## Build Notifications
+
+Build notifications can be routed via Cloud Build Pub/Sub topic to external services (Slack, Discord, webhooks). Configure the notification sink at the project level.
+
+**Critical:** Do NOT set `options.pubsubTopic` in `cloudbuild.yaml` — it overrides the built-in topic and can break your notification pipeline.
+
+## Post-deploy smoke gate (Cloud Run)
+
+A "deploy SUCCESS" only proves the revision is `Ready` — not that the real built image, with real secrets/env, actually works. The bug class this catches lives *only* in the deployed artifact (a secret with a trailing newline, an env substitution like `_ENV` stripped at deploy, a file not `COPY`ed into the image, a `redirect_uri` from the container bind origin, traffic pinned to a stale revision). No PR-lane or `next dev` test sees it. Run a curated **smoke gate** against the deployed revision — the canonical doctrine, the manifest format, and the `--target` semantics live in the **`smoke` skill** (`skills/engineering/smoke`); this section is only the Cloud Run/Cloud Build *placement*.
+
+**The pipeline owns artifacts. The release lane owns exposure.** That seam decides what each fork of the build is allowed to do. The build's job ends at producing something verifiable — an image, a revision, a verdict. Deciding that users should see it is the release lane's job, not a build step's.
+
+**Where it blocks depends on who the environment serves.** The blocking placement protects *users*, so it belongs on the **production** path. An integration environment (`dev`) has no users to protect — there, a blocking smoke only converts small bugs and infra flakes into failed builds and full rebuild cycles. So the same seam cuts differently per fork, and each fork's contract output is different:
+
+- **`dev` output: a SHA-keyed verdict.** Deploy at 100%, smoke as a non-blocking recorder, and publish the verdict as a commit status. When configured as required CI, it helps `rush-release` select the newest green trunk SHA; the production candidate smoke remains the definitive release gate.
+- **prod output: a 0%-traffic candidate revision.** Deploy with `--no-traffic`, surface the revision name and candidate URL, and **stop**. The build does not smoke it and does not shift traffic — `rush-release` does both.
+
+**Why the prod gate moved out of the build.** The gate did not disappear; it moved to where it can be operated:
+
+- **Validation becomes retryable without a rebuild.** A red smoke inside the build means re-triggering the whole build to retry a config or secret fix. Against an already-deployed candidate URL it is a re-run measured in seconds.
+- **The traffic shift sits behind a human authorization point.** A build step promoting traffic is an automatic decision made by whoever pushed a tag. In the release lane it is an explicit act with a person and a version behind it.
+- **The pipeline can change without the release doctrine changing.** `cloudbuild.yaml` is free to change builders, regions, or step order; the release contract is still "give me a candidate at 0%."
+
+### Production: build a candidate, stop before traffic
+
+The prod fork's entire contract is three facts:
+
+- **In:** a tag push matching `v*-rc.*` — release-candidate tags, minted by the release lane.
+- **Out:** a candidate revision deployed at **0% traffic**, with its revision name and candidate URL discoverable by the caller.
+- **Never:** it does not validate, and it does not shift traffic.
+
+```
+tag v*-rc.* → build → deploy --no-traffic --tag candidate → publish {revision, url} → END
+                                                                    │
+          rush-release: golden-path prod-preview (read-only) ──────┤
+                                        ALIVE → update-traffic 100%, tag v<version>
+                                        DEAD  → delete the 0% revision (traffic never moved)
+```
+
+Because traffic never moved, a DEAD verdict is not a rollback and has no user impact — the candidate revision is simply deleted.
+
+```yaml
+  # Prod fork: deploy the candidate at 0% traffic, then stop.
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    id: deploy-candidate
+    entrypoint: bash
+    args:
+      - '-c'
+      - |
+        [[ "${_ENV}" == "prod" ]] || exit 0
+        source /workspace/env.sh
+        gcloud run deploy {service}-${_ENV} \
+          --image=$LOCATION-docker.pkg.dev/$PROJECT_ID/{repo}/{service}:$COMMIT_SHA \
+          --region=$LOCATION \
+          --platform=managed \
+          --min-instances=${_MIN_INSTANCES} \
+          --max-instances=1 \
+          --memory=2Gi \
+          --cpu=1 \
+          --port=8080 \
+          --cpu-boost \
+          --no-cpu-throttling \
+          --timeout=300 \
+          --set-secrets=DATABASE_URL=${_DATABASE_URL_SECRET}:latest,SECRET_KEY=${_SECRET_KEY}:latest \
+          --network=$${VPC_NETWORK} \
+          --subnet=$${VPC_SUBNET} \
+          --no-traffic \
+          --tag=candidate
+
+  # Publish the candidate coordinates for the release lane. Build ends here.
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    id: publish-candidate
+    entrypoint: bash
+    args:
+      - '-c'
+      - |
+        [[ "${_ENV}" == "prod" ]] || exit 0
+        svc="$$(gcloud run services describe {service}-${_ENV} --region=$LOCATION --format=json)"
+
+        # Read BOTH coordinates off the same `candidate` traffic entry. Do not reach for
+        # .status.latestCreatedRevisionName: "latest" and "the one I just tagged" are the
+        # same thing only when nothing else is deploying, and the case where they diverge
+        # is exactly the case you need this to be right — a concurrent build would hand the
+        # release lane one revision's name beside another revision's URL.
+        revision="$$(jq -r '.status.traffic[] | select(.tag=="candidate") | .revisionName' <<<"$${svc}")"
+        url="$$(jq -r '.status.traffic[] | select(.tag=="candidate") | .url' <<<"$${svc}")"
+        [[ -n "$${url}" && "$${url}" != "null" ]] || { echo "no candidate tag URL"; exit 1; }
+        [[ -n "$${revision}" && "$${revision}" != "null" ]] || { echo "no candidate revision"; exit 1; }
+
+        # Prove the tagged revision runs THIS build's image before publishing it as the
+        # candidate. Without this, a failed deploy that left the previous candidate in place
+        # publishes stale coordinates the release lane would validate and promote.
+        running="$$(gcloud run revisions describe "$${revision}" --region=$LOCATION \
+          --format='value(spec.containers[0].image)')"
+        [[ "$${running}" == *":$COMMIT_SHA" ]] || {
+          echo "candidate $${revision} runs $${running}, not $COMMIT_SHA"; exit 1; }
+
+        jq -n --arg r "$${revision}" --arg u "$${url}" --arg s "$COMMIT_SHA" --arg t "$TAG_NAME" \
+          '{revision:$$r, candidate_url:$$u, commit_sha:$$s, rc_tag:$$t}' \
+          > /workspace/candidate.json
+        gcloud storage cp /workspace/candidate.json \
+          "gs://$${PROJECT_ID}_cloudbuild/candidates/{service}-$TAG_NAME.json"
+        cat /workspace/candidate.json
+```
+
+**The `candidate` tag moves; the revision name does not.** Each prod build re-points `--tag=candidate` at its own new revision, so the tag URL always means "the newest candidate," not "the candidate I built." That is why the published JSON carries the **revision name** and why the release lane deletes a failed candidate **by name**. `rush-release` therefore refuses a second release in flight: its moving URL would silently point the earlier release at the later build.
+
+**Surface the coordinates as a GCS object keyed by the RC tag, not as log output.** The release lane already knows the tag it pushed, so it can fetch `candidates/{service}-<rc-tag>.json` without knowing the build ID — no build lookup, no log scraping, and the object outlives log retention. The `cat` at the end is a convenience for humans reading the build, not the interface. Derive the URL from `gcloud run services describe` rather than from `gcloud run deploy` output: the tag URL lives in `status.traffic[]`, and describing the service after the deploy also proves the tag actually landed (the guard above fails the build if it did not).
+
+**Verification-time policy belongs to the release lane.** Against production, the release lane runs smoke under a **read-only production Validation Profile** (`prod-preview`/`prod-active`); the profile's semantics are the `golden-path` skill's to document, not this file's.
+
+**Notification.** A prod build that fails now means the *candidate was never produced* — it flows into the **existing Cloud Build → Pub/Sub notification** above, no new sink. The smoke verdict is no longer a build event at all; it belongs to the release lane.
+
+**Debug-loop tip.** Moving validation out of the build is exactly what makes retrying it cheap — this used to be an optional "split it into a second trigger" refinement, and it is now the default. Re-running smoke against the already-deployed candidate URL costs seconds and needs no rebuild. (For staged multi-environment promotion, prefer `cloud-deploy`'s verify phase, which gates promotion and is likewise retryable without a rebuild.)
+
+### Integration (`dev`): deploy 100%, record the verdict, never fail the build
+
+`dev` deploys straight to **100% traffic** on build success — no `--no-traffic` staging, no smoke step that can fail the build. Smoke still runs, as a **recorder**:
+
+```
+build → deploy (100% traffic) → smoke → publish SHA-keyed verdict (never fails the build)
+```
+
+```yaml
+  # Dev fork: deploy at 100% traffic (the standard deploy step above), then record a verdict.
+  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
+    id: record-smoke-verdict
+    entrypoint: bash
+    secretEnv: ['GITHUB_TOKEN']
+    args:
+      - '-c'
+      - |
+        [[ "${_ENV}" == "dev" ]] || exit 0
+        post_status() {   # $1=state $2=description
+          curl -sS -X POST \
+            -H "Authorization: Bearer $${GITHUB_TOKEN}" \
+            -H "Accept: application/vnd.github+json" \
+            "https://api.github.com/repos/OWNER/REPO/statuses/$COMMIT_SHA" \
+            -d "$$(jq -n --arg s "$$1" --arg d "$$2" \
+                  '{state:$$s, context:"dev-smoke", description:$$d}')"
+        }
+
+        svc="$$(gcloud run services describe {service}-${_ENV} --region=$LOCATION --format=json)"
+        serving="$$(jq -r '.status.traffic[] | select(.percent==100) | .revisionName' <<<"$${svc}")"
+        url="$$(jq -r '.status.url' <<<"$${svc}")"
+
+        # Identify the serving revision by the IMAGE it runs, not by its name: revision
+        # names are Cloud Run's own counter (`{service}-dev-00042-abc`) and carry no SHA.
+        # The image tag is the only thing that ties the running code to this commit.
+        running="$$(gcloud run revisions describe "$${serving}" --region=$LOCATION \
+          --format='value(spec.containers[0].image)')"
+        if [[ "$${running}" != *":$COMMIT_SHA" ]]; then
+          # Traffic is pinned to a stale revision. Smoking now would test OLD code and
+          # publish a green verdict against THIS SHA — a false ALIVE that the release lane
+          # would later read as authorization to promote. Fail the status, run nothing.
+          post_status failure "traffic on $${serving} running $${running##*:} — not this SHA; no verdict earned"
+          exit 0
+        fi
+
+        command -v node || { apt-get update -qq && apt-get install -yqq nodejs; }; gp="node scripts/golden-paths/golden-path.mjs"
+        $$gp run --profile dev-active --target "$${url}" --expect-sha "$COMMIT_SHA"
+        $$gp verdict --profile dev-active --target "$${url}" && state=success || state=failure
+        post_status "$${state}" "golden-path $${state}"
+
+        exit 0   # never fail the dev build on a DEAD verdict
+
+availableSecrets:
+  secretManager:
+    - versionName: projects/$PROJECT_ID/secrets/github-status-token/versions/latest
+      env: GITHUB_TOKEN
+```
+
+- After traffic shifts, **confirm the serving revision is running this commit's image**, then run smoke against the live service URL. Capture the exit code instead of letting it fail the step. Compare **images, not revision names** — Cloud Run names revisions from its own counter, so a name-based check silently never matches and degrades to no check at all. On a mismatch, publish `failure` and **run no smoke**: traffic is on older code, so any verdict it produced would be a green light keyed to a SHA that never served.
+- **Publish the verdict as a commit status on the SHA** (context `dev-smoke`, state `success`/`failure`, description naming the broken path on DEAD). When branch protection requires that status, it is part of the green CI set from which `rush-release` selects its trunk SHA. The image for a SHA is immutable, so the recorded verdict stays useful after `dev` moves on; production still gets its own `prod-preview` gate.
+- The step then **exits 0 regardless of verdict**: a DEAD verdict marks the SHA unpromotable; it must not block the next dev iteration.
+- **Non-blocking must not mean silent.** On DEAD, still notify (a Pub/Sub message from the step, or a webhook on the `dev-smoke` status change) so a broken dev is known in minutes, not discovered at release time.
+
+## Rules
+
+### Entrypoint Fail-Fast
+
+All `entrypoint.sh` scripts that use case statements MUST fail on unknown or missing commands. Never silently default to starting a server or executing arbitrary commands.
+
+```bash
+# WRONG - silently starts server on unknown command
+*)
+    exec python -m app.main
+    ;;
+
+# WRONG - executes arbitrary commands
+*)
+    exec "$@"
+    ;;
+
+# CORRECT - fail fast with clear error
+"")
+    echo "Error: No command specified."
+    echo "Usage: entrypoint.sh {migrate|seed|test|serve}"
+    exit 1
+    ;;
+*)
+    echo "Error: Unknown command '$1'"
+    echo "Usage: entrypoint.sh {migrate|seed|test|serve}"
+    exit 1
+    ;;
+```
+
+**Why:** Silent defaults cause Cloud Run Jobs to hang indefinitely when passed unrecognized commands (e.g., a `seed` command that doesn't exist). The job appears stuck because it started a server instead of failing immediately.
+
+### Debug Master Branch Issues by Comparing to Dev
+
+When debugging and fixing issues on the master branch, always ask: **"Why didn't this occur on the dev branch?"**
+
+This question helps identify:
+- **Environment differences**: VPC, secrets, min instances, database configuration
+- **Missing build steps**: Dev skips migrations/seeds - did this mask an issue?
+- **Trigger differences**: Dev triggers on branch push, staging on PR, prod on a `v*-rc.*` tag push
+- **Data differences**: Dev may have different seed data or database state
+
+If the issue only manifests on master/prod:
+1. Compare the cloudbuild.yaml substitutions between environments
+2. Check if migrations ran correctly (dev skips them)
+3. Verify secrets are correctly configured for the environment
+4. Confirm VPC/network settings match expectations
+
+**Why:** Issues that appear only in production often stem from environment-specific configuration that was never tested because dev has different settings. Systematically comparing environments catches configuration drift early.
+
+## Key Patterns
+
+- **Substitutions**: `${_ENV}` for Cloud Build substitutions
+- **Bash escaping**: `$$VAR` for bash variables (becomes `$VAR`)
+- **Env file**: `/workspace/env.sh` persists variables across steps
+- **Min instances**: 0 dev/staging, 1 prod
+- **Machine type**: `E2_HIGHCPU_8` for faster builds
+- **Startup flags**: `--cpu-boost --no-cpu-throttling --timeout=300` for slow-starting containers
+- **Entrypoint**: Always fail fast on unknown commands (see Rules)
+
+For VPC, Secrets, and IAM patterns, see [google-cloud](../google-cloud/SKILL.md) and its reference files.
+
+## Debug
+
+```bash
+# List recent builds
+gcloud builds list --region=$LOCATION --limit=5
+
+# Find builds for a specific branch
+gcloud builds list --filter="trigger.branchName=YOUR_BRANCH" --region=$LOCATION
+
+# Builds on main branch that failed
+gcloud builds list --filter="trigger.branchName=main AND status=FAILURE" --region=$LOCATION
+
+# Detailed output with branch and commit info
+gcloud builds list --filter="trigger.branchName=dev" --region=$LOCATION --limit=10 \
+  --format="table(id,startTime,status,trigger.branchName,substitutions.SHORT_SHA)"
+
+# View logs for a specific build
+gcloud builds log BUILD_ID --region=$LOCATION
+
+# Describe build status
+gcloud builds describe BUILD_ID --region=$LOCATION
+
+# Check Cloud Run service status
+gcloud run services describe {svc}-{env} --region=$LOCATION
+```
+
+## Best Practices
+
+### Enable IAP Protection on Cloud Run
+
+See [google-cloud/iam.md](../google-cloud/references/iam.md#cloud-run-native-iap) for:
+- Enabling IAP on Cloud Run with `--no-allow-unauthenticated` and `--iap`
+- Granting access to domain users, individual users, and service accounts
+- IAP in Cloud Build pipelines
+
+For programmatic access to IAP-protected resources, see [google-cloud/iam.md#service-account-authentication-jwt-pattern](../google-cloud/references/iam.md#service-account-authentication-jwt-pattern) for:
+- Service account JWT authentication pattern for E2E tests, CI/CD runners, and inter-service calls
+
+## Closed-Loop Execution (Monitor → Debug → Fix → Monitor)
+
+**Goal:** Never stop after `gcloud builds log --stream`. Always declare the next step and continue until DONE criteria are met or escalated.
+
+### DONE / Exit Criteria
+- **Cloud Build:** Build status is `SUCCESS` and all steps complete without errors.
+- **Cloud Run Deploy:** Service revision is updated and `Ready` with desired traffic.
+- **Verification:** Basic smoke check passes (health endpoint or minimal request) and no new errors in service logs for a short window.
+
+### Timeboxed Monitoring
+1. **Stream build logs** for 5-10 minutes:
+   ```bash
+   gcloud builds log BUILD_ID --region=$LOCATION --stream
+   ```
+2. **If still running**, describe build to pinpoint the current step:
+   ```bash
+   gcloud builds describe BUILD_ID --region=$LOCATION
+   ```
+3. **If failing**, identify the failing step (by step name/ID), then drill into its logs or related service/job logs.
+
+### FAILURE vs SUCCESS (Next Commands)
+
+**On FAILURE:**
+- Declare next step: "Next step: inspect failing step `<step-name>` logs and related Cloud Run/Job logs."
+- Run:
+  ```bash
+  gcloud builds describe BUILD_ID --region=$LOCATION
+  gcloud builds log BUILD_ID --region=$LOCATION
+  gcloud run services describe {svc}-{env} --region=$LOCATION
+  ```
+- Apply fix (config/code), then **re-run build**, and loop back to monitoring.
+
+**On SUCCESS:**
+- Declare next step: "Next step: verify deployment and check service logs."
+- Run:
+  ```bash
+  gcloud run services describe {svc}-{env} --region=$LOCATION
+  gcloud run services logs read {svc}-{env} --region=$LOCATION --limit=50
+  ```
+- Perform a minimal verification (health endpoint or basic request). If clean, declare DONE.
+
+### Next Step Declaration Rule
+After any log stream or describe command, **explicitly state the next step** before taking it. This prevents stalls after log streaming.
+
+### Escalation / Ask-Confirmation Boundary (Risky Ops)
+Ask for confirmation **before** any of the following:
+- Database migrations or seeds in prod
+- Rolling back to a prior revision
+- Changing IAM, VPC, or secrets
+- Deleting resources or data
+
+### Incident Checklist (Copy/Paste)
+
+```
+[ ] Identify build ID and region
+[ ] Stream build logs (5-10 min)
+[ ] Describe build to find current/failing step
+[ ] If failing: capture step name and error
+[ ] Inspect related Cloud Run service/job logs
+[ ] Declare next step explicitly
+[ ] Apply fix and re-run build
+[ ] Verify Cloud Run revision is Ready + traffic
+[ ] Smoke check endpoint
+[ ] Confirm no new errors in logs (short window)
+[ ] Declare DONE or escalate with reason
+```