@yottagraph-app/aether-instructions 1.1.48 → 1.1.49

package/commands/deploy_job.md

@@ -0,0 +1,227 @@
+# Deploy Compute Job
+
+Deploy a Cloud Run Job from the `jobs/` directory via the Broadchurch Portal.
+
+## Overview
+
+A "compute job" is a containerized Python (or any-language) entrypoint
+that runs on Google Cloud Run Jobs. Use it for:
+
+- **Cron jobs** (set `schedule:` in `job.yaml` — Cloud Scheduler is wired up automatically)
+- **Event-triggered batch work** (HTTP-triggered from your Vercel app or an Agent Engine tool)
+- **Heavy compute** (entity enrichment, scoring, ETL, exports, aggregations)
+- **Workflow steps** (called from a Cloud Workflow definition under `workflows/`)
+
+This command triggers a Cloud Build → Cloud Run Job deploy through
+the Portal. No local GCP credentials needed.
+
+The job must live in `jobs/<name>/` with at minimum:
+
+```
+jobs/<name>/
+├── main.py             # Entrypoint (or any executable; see Dockerfile)
+├── requirements.txt    # Python deps (only required if no custom Dockerfile)
+├── job.yaml            # Manifest: resources, schedule, env
+└── Dockerfile          # Optional — auto-generated if missing
+```
+
+**Prerequisite:** The project must have a valid `broadchurch.yaml`
+(created during provisioning).
+
+---
+
+## Step 1: Read Configuration
+
+Read `broadchurch.yaml` from the project root.
+
+```bash
+cat broadchurch.yaml
+```
+
+**If the file does not exist:**
+
+> This project hasn't been provisioned yet. Create it in the Broadchurch Portal first.
+
+Stop here.
+
+Extract these values:
+
+- `tenant.org_id` (tenant org ID)
+- `gateway.url` (Portal Gateway URL)
+
+---
+
+## Step 2: Discover Jobs
+
+List the directories under `jobs/`:
+
+```bash
+ls -d jobs/*/
+```
+
+**If no directories exist:**
+
+> No jobs found. Create one by making a directory under `jobs/` with the structure above.
+> See `docs/COMPUTE_JOBS.md` for guidance, or copy `jobs/example_job/` as a starting point.
+
+Stop here.
+
+**Skip `example_job`** — this is a template placeholder and should
+never be deployed. Filter it out before proceeding.
+
+**If multiple jobs remain:** Deploy all of them. If called interactively
+(not from `/build_my_app`), ask the user which one to deploy.
+
+**If only one job remains:** Proceed with it — no confirmation needed.
+
+**Important:** Job directory names should use underscores; the deploy
+workflow translates them to Cloud Run-friendly hyphens automatically.
+
+---
+
+## Step 3: Validate Job Structure
+
+For the selected job directory, verify the required files exist:
+
+```bash
+ls jobs/<name>/main.py jobs/<name>/job.yaml
+```
+
+If `Dockerfile` exists, the deploy uses it as-is. If not, the deploy
+auto-generates a Python 3.12 Dockerfile that runs `python main.py`.
+
+If using the auto-Dockerfile, `requirements.txt` must also exist:
+
+```bash
+ls jobs/<name>/requirements.txt 2>/dev/null
+```
+
+Validate the manifest is well-formed:
+
+```bash
+yq -e '.name // ""' jobs/<name>/job.yaml >/dev/null
+```
+
+---
+
+## Step 4: Ensure Code is Pushed
+
+The deployment workflow runs on the code in the GitHub repo, not the
+local working directory:
+
+```bash
+git status
+```
+
+**If there are uncommitted changes in `jobs/<name>/`:**
+
+> Your job code has local changes that aren't pushed yet. The
+> deployment will use the version on GitHub. Would you like me to
+> commit and push first?
+
+If yes, commit and push. If no, warn them and continue.
+
+---
+
+## Step 5: Trigger Deployment
+
+Call the Portal API to trigger the deploy workflow:
+
+```bash
+curl -sf -X POST "<GATEWAY_URL>/api/projects/<ORG_ID>/deploy" \
+  -H "Content-Type: application/json" \
+  -d '{"type": "job", "name": "<JOB_NAME>"}'
+```
+
+**If this fails with 404:** The job directory may not exist on GitHub
+yet. Push your code first.
+
+**If this succeeds:** The Portal has triggered the `deploy-job.yml`
+GitHub Actions workflow.
+
+---
+
+## Step 6: Monitor Progress
+
+> Deployment triggered! The compute job is being deployed via GitHub Actions.
+>
+> - **Job:** <name>
+> - **Workflow:** deploy-job.yml
+>
+> This typically takes 2-5 minutes (container build + Cloud Run Job create/update).
+> You can monitor progress:
+>
+> - In the Broadchurch Portal under your project's "Jobs" tab
+> - On GitHub: `https://github.com/<REPO>/actions`
+>
+> Once complete:
+>
+> - The job is callable via the Portal "Run now" button
+> - If `schedule:` is set in `job.yaml`, Cloud Scheduler will trigger it automatically
+> - Run history is visible in the Portal's "Jobs" tab
+
+---
+
+## Step 7: (Optional) Trigger a Test Run
+
+After deployment, trigger an ad-hoc run to verify the job works:
+
+```bash
+curl -sf -X POST "<GATEWAY_URL>/api/projects/<ORG_ID>/jobs/<JOB_NAME>/run" \
+  -H "Content-Type: application/json" \
+  -d '{}'
+```
+
+Then poll for results:
+
+```bash
+curl -sf "<GATEWAY_URL>/api/projects/<ORG_ID>/jobs/<JOB_NAME>/runs" | jq '.runs[0]'
+```
+
+Each run has a `status` field. Terminal statuses are: `Succeeded`,
+`Failed`, `Cancelled`.
+
+---
+
+## Troubleshooting
+
+### Build fails
+
+Check the GitHub Actions logs for the `Deploy Compute Job` workflow.
+Common issues:
+
+- **"requirements.txt" errors**: list every Python dep your `main.py` imports.
+- **Custom Dockerfile**: ensure the `CMD` actually runs your entrypoint.
+- **Memory/CPU mismatch**: Cloud Run Jobs require 1 vCPU per 4 GiB memory minimum (see Google Cloud docs).
+
+### Job times out
+
+Increase `task_timeout` in `job.yaml`. Cloud Run Jobs supports up to 24
+hours per task. For longer-running work, split into shards
+(`task_count: N` + `parallelism: N`) or escalate to GCP Batch.
+
+### Schedule doesn't fire
+
+Cloud Scheduler entries are named `job-<name>`. Check in the Cloud
+Console (Cloud Scheduler → us-central1) and verify:
+
+- The cron expression is valid
+- The OAuth service account email matches the tenant SA
+- The target URL points at the Cloud Run Job's `:run` endpoint
+
+### Need to update an existing job
+
+Just run `/deploy_job` again. It will rebuild the container, update
+the Cloud Run Job in place, and reconcile the Cloud Scheduler entry
+to match the current `job.yaml`.
+
+### Want to delete a job
+
+Delete via the Portal "Jobs" tab, or:
+
+```bash
+curl -sf -X DELETE "<GATEWAY_URL>/api/projects/<ORG_ID>/jobs/<JOB_NAME>"
+```
+
+This removes the Cloud Run Job, its Cloud Scheduler entry, and the
+Portal's job registration.

package/commands/deploy_workflow.md

@@ -0,0 +1,189 @@
+# Deploy Cloud Workflow
+
+Deploy a Cloud Workflow definition from the `workflows/` directory via
+the Broadchurch Portal.
+
+## Overview
+
+A Cloud Workflow orchestrates one or more compute jobs. It's the
+declarative DAG layer on top of the job runtime — useful for:
+
+- **Multi-step pipelines** ("for each entity: enrich, score, write")
+- **Fan-out / fan-in** patterns (parallel jobs that converge)
+- **Retry / error-branch logic** managed by the workflow runner, not the jobs
+- **Scheduled multi-step orchestration** (set `schedule:` in `manifest.yaml`)
+
+Cloud Workflows handles state, retries, fan-out, and observability.
+Your jobs stay stateless and idempotent — that's the design point.
+
+The workflow must live in `workflows/<name>/` with:
+
+```
+workflows/<name>/
+├── workflow.yaml       # Cloud Workflows DSL — the DAG itself
+└── manifest.yaml       # Name, schedule, log level, schedule input
+```
+
+**Prerequisite:** The project must have a valid `broadchurch.yaml`,
+and any jobs the workflow references must already be deployed via
+`/deploy_job`.
+
+---
+
+## Step 1: Read Configuration
+
+```bash
+cat broadchurch.yaml
+```
+
+**If the file does not exist:** Stop and tell the user the project
+isn't provisioned yet.
+
+Extract:
+
+- `tenant.org_id`
+- `gateway.url`
+
+---
+
+## Step 2: Discover Workflows
+
+```bash
+ls -d workflows/*/
+```
+
+**If no directories exist:** Suggest copying `workflows/example_workflow/`
+or creating a new one matching the structure above. Stop here.
+
+**Skip `example_workflow`** — it's a template placeholder. Filter it out.
+
+**If multiple remain:** Ask which to deploy.
+**If one remains:** Proceed without confirmation.
+
+---
+
+## Step 3: Validate Workflow Structure
+
+```bash
+ls workflows/<name>/workflow.yaml workflows/<name>/manifest.yaml
+```
+
+Validate `workflow.yaml` is syntactically valid:
+
+```bash
+yq -e '.main.steps' workflows/<name>/workflow.yaml >/dev/null
+```
+
+Validate `manifest.yaml` has a name:
+
+```bash
+yq -e '.name' workflows/<name>/manifest.yaml >/dev/null
+```
+
+---
+
+## Step 4: Ensure Referenced Jobs Are Deployed
+
+Cloud Workflows can't deploy referenced jobs for you. Check the
+workflow definition for any `googleapis.run.v1.namespaces.jobs.run`
+calls and verify those jobs exist:
+
+```bash
+grep -oP 'jobs/\K[a-z0-9-]+' workflows/<name>/workflow.yaml | sort -u
+```
+
+For each job name found, confirm it appears in the Portal:
+
+```bash
+curl -sf "<GATEWAY_URL>/api/projects/<ORG_ID>/jobs" | jq '.jobs[].job_name'
+```
+
+If a referenced job isn't deployed yet:
+
+> The workflow `<name>` references job `<job>` which isn't deployed.
+> Deploy it first with `/deploy_job <job>`.
+
+---
+
+## Step 5: Ensure Code is Pushed
+
+```bash
+git status
+```
+
+If there are uncommitted changes in `workflows/<name>/`, prompt to
+commit and push.
+
+---
+
+## Step 6: Trigger Deployment
+
+```bash
+curl -sf -X POST "<GATEWAY_URL>/api/projects/<ORG_ID>/deploy" \
+  -H "Content-Type: application/json" \
+  -d '{"type": "workflow", "name": "<WORKFLOW_NAME>"}'
+```
+
+**If 404:** The directory may not exist on GitHub yet. Push first.
+
+**If success:** The Portal has triggered the `deploy-workflow.yml`
+GitHub Actions workflow.
+
+---
+
+## Step 7: Monitor Progress
+
+> Workflow deployment triggered.
+>
+> - **Workflow:** <name>
+> - **GitHub Action:** deploy-workflow.yml
+>
+> Typically completes in under a minute (workflow definitions are tiny).
+>
+> Once deployed:
+>
+> - The workflow is executable via the Portal "Run now" button on the Workflows tab
+> - If `schedule:` is set, Cloud Scheduler invokes it automatically
+> - Execution history is in the Portal Workflows tab
+
+---
+
+## Step 8: (Optional) Trigger a Test Execution
+
+```bash
+curl -sf -X POST "<GATEWAY_URL>/api/projects/<ORG_ID>/workflows/<WORKFLOW_NAME>/run" \
+  -H "Content-Type: application/json" \
+  -d '{"input": {"limit": 10}}'
+```
+
+Then poll for the result:
+
+```bash
+curl -sf "<GATEWAY_URL>/api/projects/<ORG_ID>/workflows/<WORKFLOW_NAME>/executions" | jq '.executions[0]'
+```
+
+---
+
+## Troubleshooting
+
+### Workflow rejected with parsing errors
+
+Cloud Workflows YAML has strict syntax. Common gotchas:
+
+- Indentation must use spaces (no tabs)
+- `${...}` expressions must be quoted strings in YAML
+- `parallel for` requires `value` and `range` (or `in`) under `for:`
+- `args:` for `googleapis.run.v1.*` must include `name: namespaces/<project>/jobs/<job>`
+
+Reference: https://cloud.google.com/workflows/docs/reference/syntax
+
+### Workflow times out
+
+Cloud Workflows has a 1-year max duration but each step has its own
+limits. Long-running step calls (`googleapis.run.v1.namespaces.jobs.run`)
+should use `connector_params: {timeout: <seconds>}` to bound them.
+
+### Schedule doesn't fire
+
+Cloud Scheduler entries are named `workflow-<name>`. Verify in the
+Cloud Console.

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@yottagraph-app/aether-instructions",
-    "version": "1.1.48",
+    "version": "1.1.49",
     "description": "Cursor commands and skills for Aether development",
     "files": [
         "commands",