graph-agents-cli 0.3.1__py3-none-any.whl

graph_agents_cli/skills/data/graph-agents-cli-deploy/SKILL.md

@@ -0,0 +1,357 @@
+---
+name: graph-agents-cli-deploy
+description: >
+  This skill should be used when the user wants to "deploy an agent",
+  "deploy to Kubernetes", "deploy to our cluster", "set up CI/CD", "set up
+  Argo CD", "configure secrets", "rotate a key", "promote to production",
+  "check cluster prerequisites", "deploy to kind/k3s/minikube", "deploy
+  air-gapped", or "troubleshoot a deployment". Covers the deployment modes
+  (direct local-load, direct registry, helm-push, argocd), the dev/staging/
+  prod environments, the secrets procedure and rotation, the Argo CD PR
+  flow and the single production merge gate, the required GitHub
+  environment and branch-protection settings, `infra check`, local-load dev
+  clusters, and the disconnected profile. Part of the graph-agents-cli
+  skills suite. Do NOT use for agent code (graph-agents-cli-langgraph-code),
+  evaluation (graph-agents-cli-eval), scaffolding
+  (graph-agents-cli-scaffold), or tracing (graph-agents-cli-observability).
+metadata:
+  author: graph-agents-cli contributors
+  license: Apache-2.0
+  version: "0.3.1"
+  requires:
+    bins:
+      - graph-agents-cli
+    install: "uv tool install git+https://github.com/ss7172/graph-agents-cli@v0.3.1"
+---
+
+# Deployment guide
+
+> **Requires:** `graph-agents-cli`, plus `docker`, `helm`, `kubectl`, `git`, a kubeconfig, `gh` for
+> GitHub-hosted CD, and `argocd` when `cd: argocd`. The CLI shells out to these; it never installs cluster components. If the
+> project has no deployment target, `/graph-agents-cli-scaffold` adds one with `scaffold enhance`.
+
+> **Never deploy without explicit human approval.** In `argocd` mode you open a PR; a code owner
+> merges it. You never merge it.
+
+## Reference files
+
+| File | Contents |
+|---|---|
+| `references/kubernetes.md` | Helm chart values and render-time refusals, pod security, probes, published routes (`route.publicPaths`), traffic entry (Gateway API / Ingress / TLS), metrics and NetworkPolicy, Postgres and Redis toggles, HPA/PDB, local-load dev clusters, rollback, disconnected profile |
+| `references/gitops.md` | The argocd mode: `Application` manifests, the `deploy` PR flow, staging auto-merge, production promotion, rollback by revert, `--restart` under self-heal |
+| `references/secrets.md` | The Secret contract, allow-listed and required keys, env-file and context rules, `secrets apply/status` (server-side apply, key merge, `API_KEY` handling), ownership in CD modes, rotation |
+| `references/github-settings.md` | Required GitHub environment and branch-protection settings, `CODEOWNERS`, secrets and variables (`DEPLOY_KUBECONFIG`, `GH_PR_TOKEN`), the self-hosted runner, what `infra check` reports |
+
+---
+
+## Deployment modes (determined by the manifest's `cd` and the environment)
+
+| Mode | Who applies the release | What `deploy` does |
+|---|---|---|
+| *direct, local-load* (`cd: skip`, local cluster) | `deploy` | builds the image with the `docker` CLI, loads it into the kind/k3d/k3s/minikube node (nothing for Docker Desktop, Rancher Desktop, OrbStack), applies the Secret from the allow-listed keys of the env file, runs `helm upgrade --install --wait` |
+| *direct, registry* (`cd: skip`, any other cluster) | `deploy` | builds, pushes to the manifest's registry, applies the Secret, runs `helm upgrade --install --wait` |
+| *helm-push* (`cd: helm-push`) | GitHub Actions on a self-hosted runner | CI builds and pushes; the runner job runs `deploy --env <env> --image <ref> --context <ctx> --yes`, which only runs helm (after checking the live Secret's required keys). From a workstation `dev` is allowed; `staging`/`prod` are refused outside CI (`GITHUB_ACTIONS=true`) **even with `--image`**, unless `--force-direct` |
+| *argocd* (`cd: argocd`) | Argo CD inside the cluster | **`deploy` never runs helm, never contacts the cluster and never merges.** `deploy --env <env> --image <ref>` writes the desired state: it updates `image.tag` in `deployment/helm/<name>/values-<env>.yaml` (and nothing else), commits on a branch `deploy/<env>/<short sha>` built with git plumbing from `origin/main` (the developer's checkout is never switched), and opens or updates a pull request with `gh` (else GitHub REST with `GITHUB_TOKEN`; `GH_HOST=<host>` for a GitHub Enterprise Server). `--image` names the CI-pushed image; without it the tag is `--tag` or the short git sha with a warning. Argo reconciles from `main`, so nothing reaches the cluster until the PR merges, and a PR touching `values-prod.yaml` merges only after the production approval, whoever opened it. In argocd environments only `--status`, `--restart`, and `secrets` touch the cluster |
+
+Local-load is decided from the cluster itself (its nodes), confirmed with `kind get clusters`,
+`k3d cluster list` or `minikube profile list`; the context name decides only when the nodes
+cannot be read. `deploy --dry-run` prints the docker, helm, kubectl, and gh commands plus the
+rendered manifests, changes nothing and never prompts. It runs the same checks as the real run,
+including a read-only read of the live Secret, so it refuses (exit 1) a deploy the real run
+would refuse for a missing required key (and says so when the cluster cannot be read).
+`helm dependency build deployment/helm/<name>` is printed with the `[dry-run]` prefix and still
+executed when a declared subchart is missing from `charts/`, because it only writes into the
+chart directory and the `helm template` render needs the subcharts. Use `--dry-run` to show the
+user what will happen.
+
+Images are tagged with the **short commit SHA**: `${GITHUB_SHA::7}` in the `staging` workflow,
+`git rev-parse --short HEAD` for a workstation `deploy`, plus `-dirty-<YYYYmmddHHMMSS>` (with a
+warning) when the tree has uncommitted changes outside `deployment/`, `.github/`, `tests/` and
+`docs/`; a UTC timestamp outside git; `--tag TAG` overrides. The argocd branch is
+`deploy/<env>/<short sha>` and the `promote-to-prod` `image_tag` input is a short sha. A
+placeholder registry (`ghcr.io/CHANGE-ME`) or an invalid image reference is exit 3 before
+`docker` runs. The chart refuses an empty `image.tag` and an unquoted numeric one.
+
+Exit codes: `0` ok; `1` refused (policy or mode, a declined or missing context confirmation, a
+Secret missing a required key, `--status` on a rollout that is not complete within `--timeout`);
+`2` tool failure (helm/kubectl/docker non-zero or missing from `PATH`, a failed rollout, a
+`--restart` whose new pods do not become ready, another helm operation holding the release,
+`helm dependency build` failing because `registry-1.docker.io` is unreachable); `3`
+configuration error (no env file outside dev, an unknown context, a placeholder registry, a
+`CHANGE-ME` left in the chart `env` outside dev, `jwt` without a verification key, issuer or
+audience outside dev, a blank `gateway.parentRef.name`).
+
+## Environments
+
+`dev`, `staging`, `prod`. Each has a values file `deployment/helm/<name>/values-<env>.yaml`, a
+namespace `<name>-<env>` (recorded with its kube context under `environments:` in the manifest),
+a Secret `<name>-app`, and under argocd an `Application`. `dev` may be a local-load cluster.
+
+| Values file | Defaults |
+|---|---|
+| `values-dev.yaml` | `env.APP_ENV=dev`, `secretOptional: true`, `postgresql.enabled=true` (bundled subchart; Redis too under `langgraph-server`), `gateway.enabled=false` |
+| `values-staging.yaml` | `secretOptional: false` (pods need the Secret), `postgresql.enabled=false` (external DSN from the Secret), gateway on, hostnames blank for the operator to fill |
+| `values-prod.yaml` | as staging, plus `replicaCount: 2`, a PDB, topology spread and larger requests |
+
+## Rules `deploy` and `secrets apply` follow
+
+- **Env file:** `--env-file`, else `.env.<env>`. Only `dev` falls back to `.env`; any other
+  environment without one exits 3 (local development keys never reach staging or prod).
+- **Kube context:** `--context`, else `environments.<env>.context`, else the kubeconfig's current
+  context. Outside `dev` the current context needs a confirmation (a prompt at a terminal,
+  `--yes` otherwise; exit 1 without). An explicit context missing from the kubeconfig is exit 3.
+  The resolved context and API server are printed before anything happens. Prefer recording the
+  context in the manifest for staging and prod.
+- **Order (direct mode):** project checks (chart, values, image reference, env file, chart `env`
+  placeholders, `jwt` settings the chart lists); the context; a read-only check that the Secret
+  the deploy would produce holds every required key (exit 1 before anything is built) and that
+  `jwt` can verify tokens with it; a refusal (exit 2) while another helm operation holds the
+  release, with the command that clears a lock left by an interrupted helm; then build, load or
+  push, the namespace (created when missing), the Secret, and `helm upgrade --install --wait
+  --timeout <--timeout, default 5m>`.
+- **Chart env placeholders:** a `CHANGE-ME` left in the merged chart `env` (an API base URL from
+  `api add`, `OPENAI_BASE_URL` of an `openai-compatible` project) is refused outside dev in every
+  mode (exit 3, naming the key and the values file) and a warning in dev.
+- **`jwt` settings:** outside dev (or when `APP_ENV` is not `dev`) the chart env or the Secret
+  must provide a verification key (`AUTH_JWT_JWKS_URL` or `AUTH_JWT_PUBLIC_KEY`), the issuer and
+  the audience, or `deploy` exits 3 (the pods would refuse to start). A setting the chart env
+  lists, even empty, overrides the Secret. In dev a missing key source is a warning (the pods
+  start and answer 503).
+- **Failed rollout:** the pods' states, this release's warning events since the deploy started
+  and the logs are printed, then with `--atomic` (default) the release is rolled back to the
+  newest good revision, or a first install that never succeeded is uninstalled. Only the
+  revision this run created is ever undone. Once the release is back where it was, the app
+  Secret (and `<name>-metrics`) this deploy applied is restored to its previous values, or
+  deleted if this deploy created it, unless someone changed it since. `--no-atomic` leaves the
+  failed revision and the new Secret values in place and says so.
+- **Same image:** when the release already runs the image being deployed, `deploy` says so up
+  front; after the upgrade it reports when no pod was replaced and, if the Secret changed, that
+  `deploy --restart` is what makes the pods read it.
+- **Protected environments:** `deploy --env staging|prod` refuses while the manifest says
+  `auth_policy_implemented: false` (the `custom` stub).
+
+## Standard procedure
+
+1. **Gate:** `graph-agents-cli eval run` exits 0 and the user approved deploying.
+2. **Prerequisites:** `graph-agents-cli infra check --env <env>`. Read-only; reports the required
+   tools and the kube context, Gateway API CRDs and classes (only when `gateway.enabled`) /
+   Ingress classes, cert-manager (only when `tls.certManager.enabled`), Argo CD (only when `cd:
+   argocd`), metrics-server (only when `hpa.enabled`), the namespace, the image pull secret, the
+   app Secret and its required keys, the `jwt` verification settings, the ServiceMonitor's token
+   Secret, whether an external DSN requires TLS, every `CHANGE-ME` placeholder (registry, chart
+   image and env, CODEOWNERS, Argo CD `repoURL`), where pending approvals are kept when
+   `api-policy.yaml` gates calls (the `approvals` table of the app database; a warning under
+   `CHECKPOINTER=memory`, which loses paused runs on a restart), and, when `gh` is logged in,
+   the environment protection, branch protection and (helm-push) `DEPLOY_KUBECONFIG` settings.
+   Nothing is created; install hints are printed for the operator, and every printed command
+   runs as is.
+3. **Values:** fill `hostname`, `parentRef`, `tls`, `resources` in `values-<env>.yaml` and review
+   `route.publicPaths` (what the Gateway or Ingress publishes). For network isolation copy the
+   `networkPolicy` block of `deployment/helm/<name>/examples/networkpolicy.yaml` (DNS, the
+   database, the model endpoint, public HTTPS for hosted APIs) and set its addresses. For an
+   external database use a least-privileged role and `sslmode=verify-full` (see
+   `references/kubernetes.md`). These are config files: never
+   overwritten by upgrade, never containing secrets. `gateway.parentRef.name` is **required**
+   whenever `gateway.enabled` (the staging/prod default): `deploy` checks the merged values
+   before building or pushing anything and exits 3 naming the file when it is blank. Set
+   `appUrl` (or a hostname) so the A2A card advertises a reachable URL. Keep `image:` a nested
+   mapping (`image:` newline `tag: ...`), never an inline `{}` map, because `deploy` rewrites
+   `image.tag` textually. Record `environments.<env>.context` in the manifest.
+4. **Secrets:** `graph-agents-cli secrets apply --env <env>` (from `.env.<env>`). CD modes: the
+   manifest's `secrets.owner` runs it from a workstation with cluster access; CI never holds
+   application secrets; Argo never manages the Secret. `secrets status --env <env>` lists keys
+   without values and exits 1 when a required key is missing. `deploy` refuses to touch Secrets
+   in `helm-push` and `argocd` modes and prints the procedure.
+5. **Deploy:** `graph-agents-cli deploy --env <env>` (direct), or let CI run it (`helm-push`), or
+   open the PR (`argocd`).
+6. **Verify:** `graph-agents-cli deploy --status --env <env>` (a bounded rollout status, 60 s by
+   default, with the image, helm revision and each pod's readiness and restarts; diagnostics and
+   exit 1 when not ready; `argocd app get` in argocd mode), then
+   `graph-agents-cli run --url https://<host> "hello"` with the environment's credential in
+   `GRAPH_AGENTS_CLI_API_KEY` (the environment's `API_KEY` under `shared-bearer`, a user's token
+   under `jwt`; kept out of argv and shell history), or `--header` / `--cookie` under `custom`.
+   Readiness is `/ready` (probed inside the cluster; not published on the route).
+
+## Agents calling agents
+
+- **One project:** `peer add` writes the peer's URL into each `values-<env>.yaml` from a
+  template with `{env}`:
+  `graph-agents-cli peer add orders --cluster-url 'http://orders-agent.orders-agent-{env}.svc.cluster.local'`.
+- **Several projects:** `graph-agents-system.yaml` names them, the agents each calls, the
+  issuer and the environments. `graph-agents-cli system apply --dry-run`, then without it,
+  writes both sides of every edge per environment: the callers' peer URLs,
+  `TOKEN_EXCHANGE_URL` and `TOKEN_EXCHANGE_CLIENT_ID`; the called agents' `appUrl` (the URL
+  callers dial, which their card advertises: SC05), `AUTH_JWT_AUDIENCE` when empty and the
+  callers' actor ids in `AUTH_ALLOWED_ACTORS` (under `jwt` an agent not listed gets 403); and
+  NetworkPolicy `egressTo`/`ingressFrom` rules. It never writes gates, secrets or `.env`.
+  `graph-agents-cli system check --env <env>` (add `--live` after a deploy), then
+  `graph-agents-cli system deploy --env <env>` (the agents called first, `--parallel` at
+  once; outside dev every project records its kube context). Give the issuer's admin
+  `graph-agents-cli system delegations`: what each client may exchange for.
+- **Callers** keep `TOKEN_EXCHANGE_CLIENT_SECRET` and `PRINCIPAL_HASH_SALT` in `secrets.keys`;
+  outside dev `deploy` refuses a peer credential over plain http (except loopback,
+  single-label and `.svc` hosts) and an exchange API without the issuer settings.
+- **Keep A2A internal:** when only agents call an agent, drop `/a2a/<agent>` from
+  `route.publicPaths` (SC13) and turn on `networkPolicy`.
+- **Sizing:** replicas share A2A tasks only on Postgres (more than one replica with in-memory
+  tasks is SC06). Agents sharing one database server open replicas x (`DB_POOL_MAX_SIZE` + 1)
+  connections each: set `database.max_connections` in the file (SC10), and above about 120
+  connections give agents their own databases or a session-mode PgBouncer.
+
+## Secrets and rotation (summary)
+
+- Only variables in the manifest's `secrets.keys` are exported: the provider key
+  (`OPENAI_API_KEY` | `ANTHROPIC_API_KEY` | `GOOGLE_API_KEY` | `MODEL_API_KEY`), `JUDGE_API_KEY`,
+  `POSTGRES_DSN` (fastapi) or `DATABASE_URI` + `REDIS_URI` (langgraph-server), `API_KEY`,
+  `LANGSMITH_API_KEY`, the `token_env` of every `auth: bearer` API in `api-policy.yaml`, and
+  `AUTH_JWT_SECRET` automatically under `jwt` with HS*. Never the whole env file. Add other
+  secrets (`METRICS_TOKEN`, `PRINCIPAL_HASH_SALT`) to the list.
+- Server-side apply; allow-listed keys the env file leaves out are kept from the live Secret.
+- `API_KEY`: the live key wins; it is replaced only when the env file sets another **and**
+  `--rotate-api-key` is passed. Under `shared-bearer` (the only policy that reads it), when
+  neither has one, a key is generated, applied and written to the env file (0600), never printed;
+  `jwt` and `custom` never get one generated. Values must be single-line.
+- `METRICS_TOKEN` is also written alone into `<name>-metrics`, the Secret the ServiceMonitor
+  reads (Prometheus never needs the app Secret).
+- **Rotation:** put the new value in `.env.<env>`, `secrets apply` (with `--rotate-api-key` for
+  `API_KEY`), then `deploy --restart --env <env>` (`kubectl rollout restart`, then it waits for
+  the new pods and prints diagnostics if they do not become ready), because an externally
+  managed Secret does not change the pod template. In argocd environments the restart is drift
+  that self-heal may revert; the warning suggests an Argo resource action instead.
+- Details: `references/secrets.md`.
+
+## argocd PR flow and the single production gate (summary)
+
+- Staging: the `staging` workflow (on `main`) builds and pushes `<registry>/<name>:<short sha>`
+  (`${GITHUB_SHA::7}`), writes the tag into `values-staging.yaml` on `deploy/staging/<short sha>`
+  (built on the latest `main`, superseding older staging PRs) and opens a PR with auto-merge;
+  Argo syncs staging with self-heal. A PR opened with the workflow `GITHUB_TOKEN` never triggers
+  `pr_checks`, so auto-merge on that required check needs a fine-grained PAT or GitHub App token
+  (pull-request and contents write) in the `GH_PR_TOKEN` repository secret.
+- Production: desired state changes **only** through a PR touching `values-prod.yaml`, opened by
+  the `promote-to-prod` workflow (running in the GitHub `production` environment) or by a
+  workstation `deploy --env prod --image <ref>`. Neither path merges. The merge requires review
+  from the code owners of `values-prod.yaml`, cannot be self-approved, and requires `pr_checks`
+  to pass. That merge is the single gate. Prod `Application` has no automated sync; an operator
+  syncs in Argo after the merge.
+- Rollback is a git revert (a PR under the same gate). No inbound access from GitHub to the
+  cluster. Details: `references/gitops.md`.
+
+## Required GitHub settings (summary)
+
+Repository configuration a workflow cannot create for itself; the operator sets it, `infra
+check` reports it, the scaffolded README documents it:
+
+- Environment `production`: required reviewers (at least one), prevent self-review, deployment
+  branches restricted to `main`, optional wait timer. Environment `staging`: deployment branches
+  restricted to `main`.
+- `helm-push`: the `DEPLOY_KUBECONFIG` secret in **each environment** (never a repository
+  secret), and a self-hosted runner with `kubectl` and `curl`.
+- Branch protection on `main` (argocd and helm-push): pull requests required; code-owner review
+  required; dismiss stale approvals; prevent self-approval; `pr_checks` required; auto-merge
+  allowed.
+- `.github/CODEOWNERS` owns `deployment/` (except the dev and staging values), `.github/`,
+  `api-policy.yaml`, `tests/eval/`, the extensions and the manifest; replace the
+  `@CHANGE-ME/production-approvers` placeholder.
+- Repository secret `GH_PR_TOKEN` so the desired-state PRs opened by CI trigger `pr_checks`.
+- Optional: the provider key secret (or `MODEL_PROVIDER` / `MODEL_NAME` variables) so the
+  `pr_checks` eval gate runs on a real model.
+- GitHub Enterprise Server: set `GH_HOST=<host>` (plus `GH_ENTERPRISE_TOKEN` or `GITHUB_TOKEN`)
+  where `deploy` runs so argocd-mode PRs go to that host.
+- Details: `references/github-settings.md`.
+
+## Local-load dev clusters
+
+With `cd: skip` and a local cluster (kind, k3d, k3s on this machine, minikube, Docker Desktop,
+Rancher Desktop, OrbStack), `deploy --env dev` skips the registry: it builds with `docker build`,
+loads the image into the node (`kind load docker-image`, `k3d image import`, `docker save` then
+`k3s ctr images import`, `minikube image load`; nothing for the shared-daemon clusters), applies
+the Secret, runs `helm dependency build` when the subcharts are missing, and runs helm with
+`values-dev.yaml` (bundled Postgres, and Redis under `langgraph-server`, gateway off,
+`APP_ENV=dev` so `/playground` is served). The k3s import runs without `sudo` and needs root on
+most k3s hosts. Reach it with `kubectl -n <name>-dev port-forward svc/<name> 8000:80`. Only the
+`docker` CLI is supported for builds.
+
+## Disconnected profile
+
+The one configuration in which the whole lifecycle runs without internet access:
+
+| Component | Requirement |
+|---|---|
+| Agent model | `MODEL_PROVIDER=openai-compatible`, `OPENAI_BASE_URL` at an in-cluster or on-network server (vLLM, TGI, Ollama) with a tool-capable model |
+| Judge | same via `JUDGE_*`; may be the same endpoint |
+| Runtime | `fastapi` (the LangGraph Server image checks for a licence at startup; the local `langgraph dev` server needs none) |
+| Python deps | private index or mirror (`UV_INDEX_URL`), `install --locked` |
+| Images | base images mirrored into `--registry` (`PYTHON_IMAGE` / `UV_IMAGE` build args); the chart's Bitnami subcharts vendored under `deployment/helm/<name>/charts/` and their images mirrored, otherwise `deploy` runs `helm dependency build` against `registry-1.docker.io` (exit 2 offline) |
+| Tracing | `TRACING_ENABLED=true` with OTLP to an in-cluster collector, or off; no LangSmith |
+| Evals | local datasets, deterministic checks, judge run in the project's environment against the on-network endpoint; `eval submit` disabled |
+| CLI | installed from a mirror (`GRAPH_AGENTS_CLI_INSTALL_SPEC`); `GRAPH_AGENTS_CLI_NO_UPDATE_CHECK=1`; skills from the wheel bundle |
+| Scaffold | built-in templates only |
+| CD | only with an on-network GitHub Enterprise Server; otherwise `cd: skip` and direct-mode `deploy` |
+
+`infra check --profile disconnected` and `login --profile disconnected` verify these and fail on
+any hosted dependency. "Runs locally" (orchestration on the developer's machine) is not "runs
+disconnected" (this profile).
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `deploy` exit 3 "No env file for <env>" | create `.env.<env>` with the allow-listed keys, or pass `--env-file`; `.env` is used for `dev` only |
+| `deploy` exit 1 "Refusing to deploy to <env> on the kubeconfig's current context" | record `environments.<env>.context` in the manifest, pass `--context <name>`, or `--yes` after checking the printed context |
+| `deploy` exit 3 "Kube context 'x' ... is not in the kubeconfig" | fix the manifest's context or `--context`; the known contexts are listed |
+| `deploy` exit 1 "Secret ... is missing required key(s)" | add them to `.env.<env>` and re-run (direct), or run `secrets apply` (helm-push); or drop the key from `secrets.keys` if the environment does not need it |
+| `deploy` exit 2 "Another helm operation ... is in progress" | wait for it (`helm history`); if nothing else runs, the printed `helm rollback` / `helm uninstall` clears the lock an interrupted helm left |
+| `deploy` exit 2 "helm upgrade failed ... rolled back to revision N" | read the printed pod diagnostics (states, this rollout's events, logs); the Secret was restored too ("Secret ... was restored"); fix and deploy again |
+| "Secret ... keeps this deploy's values for ..." after a failed deploy | the release stayed on the failed revision (`--no-atomic`, another deploy): roll it back, then re-apply the previous values with `secrets apply --env-file <previous file>` |
+| `deploy` exit 3 "still hold(s) the placeholder CHANGE-ME" | set the real value (an API base URL, `OPENAI_BASE_URL`) in `values-<env>.yaml` or `values.yaml` |
+| `deploy` / `infra check` "The jwt auth policy cannot verify tokens" | set `AUTH_JWT_JWKS_URL` (or `AUTH_JWT_PUBLIC_KEY`), `AUTH_JWT_ISSUER` and `AUTH_JWT_AUDIENCE` under `env:` in `values-<env>.yaml` (an empty value there overrides the Secret) |
+| `deploy --status` exit 1 "did not complete within" | read the printed pods, events and logs; `--timeout` waits longer |
+| `deploy --restart` exit 2 "did not become ready" | the old pods keep serving; fix the cause (often a Secret value) and restart again, or `kubectl rollout undo` |
+| "No pods were replaced" / "The Secret changed ... run deploy --restart" | same image and chart values: the pods read a changed Secret only at start |
+| Warning "POSTGRES_DSN does not require TLS" | add `sslmode=verify-full` (and `sslrootcert`) to the DSN; see `references/kubernetes.md` |
+| `deploy` exit 1 "Refusing to deploy <env> from outside CI in helm-push mode" | run from the CI runner, or `--force-direct` for a deliberate workstation deploy (dev is always allowed) |
+| `deploy` exit 1 "--env-file is not accepted in argocd mode" (or helm-push) | Secrets are applied separately: `secrets apply --env <env>`; `deploy --env <env> --image <ref>` only opens the PR (argocd) or runs helm (helm-push) |
+| `deploy` exit 1 "the manifest records auth_policy_implemented: false" | implement `app/policies/custom.py`, flip the manifest flag |
+| `build` / `deploy` exit 3 "still the placeholder 'ghcr.io/CHANGE-ME'" | `graph-agents-cli scaffold enhance --registry <host>/<org>` (sets `create_params.registry` in the manifest, `image.repository` in the chart values and `IMAGE_REPOSITORY` in `.github/agent.env`); `build` also takes `--registry` |
+| `deploy` exit 3 "gateway.parentRef.name is blank" | set `gateway.parentRef.name` in `values-<env>.yaml` (or `gateway.enabled: false` / `ingress.enabled: true`) |
+| chart error "image.tag is empty" / "must be a quoted string" | deploy with a built image (`deploy` passes the tag); quote tags in values files (`tag: "0123456"`) |
+| `deploy` exit 3 "is a digest reference" | pass `<registry>/<repo>:<tag>`; the chart has no `image.digest` |
+| `deploy` exit 3 "not committed on origin/main" (argocd) | commit `values-<env>.yaml` on `main` first; the PR is built from the base branch's copy, never the working tree |
+| `secrets apply` exit 3 "must be single-line" | put the value on one line (for example base64) or create the Secret with kubectl directly |
+| "API_KEY in <file> differs from the live Secret; the live key is kept" | intended; pass `--rotate-api-key` to replace it, then `deploy --restart` |
+| A2A client dials `127.0.0.1:8000` after fetching the card | `APP_URL` is unset in the pod: set `appUrl` or a gateway/ingress hostname in the values file |
+| An agent's calls to another agent fail with 403 "Delegated caller ... is not allowed here (AUTH_ALLOWED_ACTORS)" | list the id the error names (the `act.sub` of the caller's exchanged tokens) in the called agent's `AUTH_ALLOWED_ACTORS`: `system apply` lists the caller's `actor_id` (default its `client_id`), so set `actor_id` when the issuer names it differently; then `deploy` it |
+| "<peer>'s agent card names <url> as its A2A endpoint, not the URL this agent calls" | set the called agent's `appUrl` to the URL the caller dials (`system apply` does); `graph-agents-cli system check --env <env>` (SC05) or `peer show <name> --check` finds it |
+| `deploy` exit 2 "missing in charts/ directory" or `helm dependency build` failed (429) | `registry-1.docker.io` is unreachable or rate-limited; retry, authenticate, or vendor the charts under `deployment/helm/<name>/charts/` |
+| `deploy` exit 2 naming a tool | helm, kubectl, docker, git or gh is not on `PATH` |
+| Staging PR opened by CI never runs `pr_checks` | PRs opened with `GITHUB_TOKEN` do not trigger workflows; store a PAT or App token as `GH_PR_TOKEN` |
+| helm-push job: "DEPLOY_KUBECONFIG is empty" | store the kubeconfig as the `DEPLOY_KUBECONFIG` secret of that GitHub environment |
+| Workflow step "Load project settings" fails on `.github/agent.env` | only the six known names are accepted; rename `CLI_VERSION_PIN` to `GRAPH_AGENTS_CLI_SPEC` |
+| `k3s ctr images import` permission denied | the import needs root on that host |
+| Pod `CreateContainerConfigError` | the Secret `<name>-app` is missing (required outside dev): `secrets apply`, then `secrets status --env <env>` |
+| Pod not Ready, `/ready` 503 | the database is unreachable from the pod: check the DSN in the Secret and network policies |
+| `ImagePullBackOff` | pull secret missing (`imagePullSecrets` in values; `infra check` reports it) or the image was not pushed / loaded |
+| No route / 404 at the hostname | `gateway.parentRef` or `ingress.className` unset, or the path is not in `route.publicPaths`; `infra check` lists classes |
+| TLS errors | `tls.existingSecret` name wrong, or cert-manager not installed while `tls.certManager.enabled` |
+| 401 from `run --url` | wrong credential for the policy; `secrets status`; pass `--header` |
+| 503 on every request under `custom` | the stub is still in place |
+| Argo shows `OutOfSync` after `--restart` | self-heal reverted the restart annotation; use an Argo resource action |
+| `ApiPolicyError` in tool results after deploy (outside dev, clients see `The tool call did not succeed. Reference: <error_id>`; the pod log has `tool call failed (error_id=...)` with its `error_type`, and `api call refused: ...` naming the rule) | the deployed `api-policy.yaml` differs from local (it is baked into the image; rebuild), or a base URL / token variable is missing in the pod (`api call not sent`); "limits.max_calls_per_run" or "rate_per_minute" in the refusal means a limit was reached (per run / per replica) |
+
+## Not covered by this skill
+
+- Writing the auth policy or tools: `/graph-agents-cli-langgraph-code`.
+- Adding the Kubernetes target or CD mode to a project: `/graph-agents-cli-scaffold`.
+- The eval gate that precedes deployment: `/graph-agents-cli-eval`.
+- Tracing configuration after deploy: `/graph-agents-cli-observability`.
+- Installing cluster components (Gateway controller, cert-manager, Argo CD, metrics-server),
+  provisioning clusters or registries, or a managed-cloud deployment target: none exist here.
+- External Secrets Operator, Sealed Secrets, Kustomize, Terraform, GitLab CI: not in this release.
+
+## Migration note
+
+Compared with google-agents-cli: Agent Runtime, Cloud Run, and GKE targets, Terraform
+provisioning (`infra single-project`, `infra cicd`), Cloud Build, Secret Manager, and Agent
+Gateway are gone. Any Kubernetes cluster via a Helm chart replaces them; secrets are Kubernetes
+Secrets applied from allow-listed `.env` keys; `infra check` is read-only.

graph_agents_cli/skills/data/graph-agents-cli-deploy/references/github-settings.md

@@ -0,0 +1,113 @@
+# Required GitHub settings
+
+These are repository settings a workflow cannot create for itself with the default token. The
+operator sets them once; `graph-agents-cli infra check` reports whether they exist when a
+`GITHUB_TOKEN` (or `gh auth`) is available; the scaffolded README documents them. The CLI never
+creates or changes them.
+
+## Environments
+
+| Environment | Settings |
+|---|---|
+| `production` | required reviewers (at least one); "prevent self-review" enabled; deployment branches restricted to `main`; optional wait timer; in `helm-push` mode the `DEPLOY_KUBECONFIG` secret |
+| `staging` | deployment branches restricted to `main`; no reviewers; in `helm-push` mode the `DEPLOY_KUBECONFIG` secret |
+
+The production jobs (`promote-to-prod` in both modes) declare `environment: production` and run
+only from `main`.
+
+## Branch protection on `main` (argocd and helm-push modes)
+
+- Pull requests required.
+- Required review from code owners.
+- "Dismiss stale approvals" enabled.
+- "Prevent self-approval" enabled.
+- `pr_checks` as a required status check.
+- No bypass for the Actions token, except the staging auto-merge mechanism (auto-merge allowed
+  on the repository; the staging PR merges once `pr_checks` passes).
+
+## `CODEOWNERS`
+
+Scaffolded at `.github/CODEOWNERS` for every project:
+
+```
+/deployment/ @CHANGE-ME/production-approvers          # kubernetes target only
+/deployment/helm/*/values-dev.yaml                    # unowned, so the staging PR can auto-merge
+/deployment/helm/*/values-staging.yaml
+/.github/ @CHANGE-ME/production-approvers             # workflows, agent.env, this file
+/api-policy.yaml @CHANGE-ME/production-approvers
+/tests/eval/ @CHANGE-ME/production-approvers
+/graph-agents-cli-extensions.yaml @CHANGE-ME/production-approvers
+/extensions/ @CHANGE-ME/production-approvers
+/.graph-agents-cli/ @CHANGE-ME/production-approvers
+/graph-agents-cli-manifest.yaml @CHANGE-ME/production-approvers
+```
+
+Replace the placeholder team (`infra check` reports it: a required item under `helm-push` or
+`argocd`, a warning under `skip`; GitHub ignores unknown owners, so the gate would require
+nobody). With code-owner review required, a PR touching `values-prod.yaml`, the policy, the eval
+gate inputs, the workflows or the extensions cannot merge without a production approver, whoever
+opened it.
+
+## Secrets and variables
+
+| Name | Kind | Mode | Purpose |
+|---|---|---|---|
+| (none for images on GHCR) | | argocd, helm-push | `GITHUB_TOKEN` pushes to `ghcr.io/<org>` |
+| `REGISTRY_USERNAME`, `REGISTRY_PASSWORD` | repository secrets | any CD mode with a non-GHCR registry | `docker login` in `staging.yaml` |
+| `DEPLOY_KUBECONFIG` | **environment** secret of `staging` and of `production` | helm-push | the self-hosted runner's cluster credentials, written under `RUNNER_TEMP` and removed after the job. Never a repository secret: any branch's workflow could read that one and it would bypass the production reviewers. The retired repository secret `KUBECONFIG` is no longer read; delete it |
+| `GH_PR_TOKEN` | repository secret | argocd, helm-push | a fine-grained PAT or GitHub App token with pull-request and contents write access, used to push the desired-state branches and open the PRs (`secrets.GH_PR_TOKEN \|\| secrets.GITHUB_TOKEN`). A PR opened with the workflow `GITHUB_TOKEN` does not trigger `pr_checks`, so auto-merge on a required check needs it |
+| provider key (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `MODEL_API_KEY`) | repository secret | any | the `pr_checks` eval gate runs on the project's real provider when its key exists; tests always run on the fake model |
+| `MODEL_PROVIDER`, `MODEL_NAME` | repository variables | any | override the eval gate's model (`MODEL_NAME` is required when `MODEL_PROVIDER` differs from the project's provider) |
+| `JUDGE_MODEL_PROVIDER`, `JUDGE_MODEL_NAME`, `JUDGE_BASE_URL`; `JUDGE_API_KEY` | variables; secret | any | a separate eval judge |
+
+Without a provider key the gate runs on the deterministic fake model and prints the warning
+"Eval gate is not a quality signal". **Never** application secrets (`API_KEY`, `POSTGRES_DSN`,
+provider keys for the deployed app) in GitHub: those are Kubernetes Secrets applied by the owner
+(`secrets.md`).
+
+## Self-hosted runner (helm-push)
+
+A runner with network access to the cluster (the jobs use `runs-on: self-hosted`), with `kubectl`
+and `curl` on `PATH` (the jobs resolve the kube context and verify the rollout with them; `uv` is
+installed by the job). Its kubeconfig (`DEPLOY_KUBECONFIG`) needs rights to deploy the release
+and to read the app Secret (deploy checks its required keys).
+
+## GitHub Enterprise Server
+
+Set `GH_HOST=<host>` (or `GITHUB_HOST`; `GITHUB_SERVER_URL` is honoured inside Actions) with
+`GH_ENTERPRISE_TOKEN` or `GITHUB_TOKEN` where `deploy` runs, so argocd-mode PRs go to that host
+(API base `https://<host>/api/v3`). `infra check --profile disconnected` and
+`login --profile disconnected` treat GitHub-hosted CI as outside the disconnected profile unless
+`GH_HOST` / `GITHUB_SERVER_URL` names an on-network GHES.
+
+## What `infra check` reports
+
+Through the `gh` CLI (`gh api`, so `gh auth login` or `GITHUB_TOKEN`), skipped when `cd: skip`:
+whether the `production` environment exists with required reviewers, prevent self-review and
+deployment branches restricted; whether the `staging` environment exists with restricted
+branches; whether `main` branch protection requires pull requests, code-owner review, dismisses
+stale approvals and lists `pr_checks` as a required status check. For `helm-push` it also
+reports whether `DEPLOY_KUBECONFIG` exists as a secret of each environment and warns about a
+repository-level (or shared organization) `KUBECONFIG` / `DEPLOY_KUBECONFIG` secret; these rows
+are informational when the account cannot read secrets. When the GitHub API does not answer, the
+GitHub rows collapse into one informational row. Independently of GitHub it reports every
+`CHANGE-ME` placeholder (registry, chart `image.repository` and `env` (required outside dev,
+a warning in dev, as `deploy` treats it), CODEOWNERS, Argo CD `repoURL`), and with `--env` the
+`jwt` policy's verification settings (a JWKS URL or public key; issuer and audience outside
+dev), the ServiceMonitor's token Secret and whether an external DSN requires TLS. Rows for
+components the environment does not use (a disabled Gateway, cert-manager, metrics-server,
+Argo CD) are `skip` with no install hint. It never changes anything.
+
+```bash
+GITHUB_TOKEN=... graph-agents-cli infra check --env prod --json
+```
+
+## Verifying by hand
+
+```bash
+gh api repos/<owner>/<repo>/environments
+gh api repos/<owner>/<repo>/environments/production
+gh api repos/<owner>/<repo>/environments/production/secrets
+gh api repos/<owner>/<repo>/branches/main/protection
+gh api repos/<owner>/<repo> --jq .allow_auto_merge
+```

graph_agents_cli/skills/data/graph-agents-cli-deploy/references/gitops.md

@@ -0,0 +1,137 @@
+# GitOps: the `argocd` CD mode
+
+With `cd: argocd`, Argo CD inside the cluster reconciles from `main`. `graph-agents-cli deploy`
+never runs helm, never contacts the cluster and never merges; it writes desired state and opens
+a pull request.
+
+## Scaffolded artefacts
+
+- `deployment/argocd/application-<env>.yaml`: one Argo `Application` per environment;
+  `targetRevision: main`, `path: deployment/helm/<name>`,
+  `valueFiles: [values.yaml, values-<env>.yaml]`, destination namespace `<name>-<env>`
+  (`CreateNamespace`). `dev` and `staging` have automated sync with self-heal; **prod has no
+  automated sync** (an operator syncs in Argo after the merge). They ignore the data of the
+  chart-managed `<name>-postgresql-auth` Secret (`RespectIgnoreDifferences`), so the dev database
+  password is not regenerated on every sync. Set `repoURL` (a `CHANGE-ME` placeholder that
+  `infra check` reports). An environment shows a comparison error until its values file has an
+  image tag: the chart refuses an empty one.
+- `.github/workflows/pr_checks.yaml`: ruff, unit and integration tests on the fake model, `lint`
+  (with the API-policy check) and the eval gate. The gate uses the project's real provider when
+  its key is a repository secret (or the `MODEL_PROVIDER` / `MODEL_NAME` repository variables);
+  on the fake model it warns that the gate only checks the plumbing.
+- `.github/workflows/staging.yaml` (on `main`, or by hand from `main`): builds and pushes
+  `<registry>/<name>:<short sha>` (`${GITHUB_SHA::7}`), then writes the tag (double-quoted) into
+  `values-staging.yaml` on branch `deploy/staging/<short sha>`, built on the latest `main`, and
+  opens a PR with auto-merge (`gh pr merge --auto --squash`); branch protection must permit
+  auto-merge. It closes older open `deploy/staging/*` PRs it supersedes (with a comment, deleting
+  their branches) and does nothing when `main` or an open PR already carries this build or a
+  newer one, so a re-run of an older build never moves staging back. PRs whose tag is not a
+  commit (a workstation `<sha>-dirty-<time>`) are left alone. Pushes that only change
+  `values-*.yaml` do not start the workflow, so a merged staging PR never triggers another build.
+- `.github/workflows/promote-to-prod.yaml`: runs in the GitHub `production` environment (which
+  controls who may *initiate* a promotion from CI), only from `main`; its `image_tag` input is the
+  short sha (empty = the tag in `values-staging.yaml`). It runs
+  `uvx --from "$GRAPH_AGENTS_CLI_SPEC" graph-agents-cli deploy --env prod --image <registry>/<name>:<short sha> --yes`,
+  which opens a PR updating `values-prod.yaml`. It does not merge.
+- Both workflows use `secrets.GH_PR_TOKEN || secrets.GITHUB_TOKEN`. A pull request opened with
+  the workflow `GITHUB_TOKEN` does not trigger `pr_checks` (GitHub never starts workflows from
+  `GITHUB_TOKEN` events), so auto-merge on a required check needs a fine-grained PAT or GitHub
+  App token stored as the `GH_PR_TOKEN` repository secret, with pull-request **and contents**
+  write access (the staging job pushes the branch with it).
+- Every job that runs graph-agents-cli sets `GRAPH_AGENTS_CLI_DISABLE_OVERRIDES=1`, so project or
+  user extensions cannot replace `lint`, `eval` or `deploy` in CI and CD.
+- `.github/agent.env` (rendered for every project, read only by the workflows): `IMAGE_REPOSITORY`,
+  `RELEASE_NAME`, `CHART_PATH`, `RUNTIME`, `CD` and `GRAPH_AGENTS_CLI_SPEC` (where CI installs the
+  CLI: the creating version's git tag). It is data, never sourced: one `NAME=VALUE` per line, `#`
+  comments and blank lines skipped, one pair of surrounding quotes and a trailing ` # comment`
+  dropped. Any other name (including the retired `CLI_VERSION_PIN`), a duplicate or a control
+  character fails the `Load project settings` step before anything is exported. To add a setting,
+  extend `known` in all three workflows.
+- `.github/CODEOWNERS`: `/.github/`, `/CODEOWNERS`, `/api-policy.yaml`, `/tests/eval/`, the
+  extensions files, the manifest, and `/deployment/` except `values-dev.yaml` and
+  `values-staging.yaml` (left unowned so the staging PR can auto-merge) ->
+  `@CHANGE-ME/production-approvers` (placeholder to fill; `infra check` reports it).
+
+The `Application` manifests and the values files are config: `upgrade` never overwrites them.
+
+## The `deploy` PR flow
+
+```bash
+graph-agents-cli deploy --env staging --image ghcr.io/acme/my-agent:abc1234
+graph-agents-cli deploy --env prod --image ghcr.io/acme/my-agent:abc1234
+```
+
+1. Reads `values-<env>.yaml` **as `origin/main` holds it** (`git cat-file blob origin/main:<path>`;
+   `HEAD` when `origin/main` is unavailable), changes `image.tag` and nothing else. Your working
+   tree is not modified and never used: a checkout behind `main` or with local edits cannot leak
+   into the PR, and "nothing to change" is judged on the base branch (a closed, unmerged PR is
+   re-opened by a re-run). A values file not committed on the base is exit 3. Paths are
+   repository-relative, so a project below the git root works.
+2. Builds branch `deploy/<env>/<short sha>` with git plumbing (`hash-object --stdin`,
+   `read-tree`, `update-index`, `write-tree`, `commit-tree`, `update-ref`) and pushes it with
+   `git push --force-with-lease=refs/heads/<branch>:<sha on origin>`; when `origin/<branch>`
+   already holds the change it prints "nothing to push", so a retry is idempotent. Your checkout
+   and index are never switched. A git identity must be configured (`commit-tree` needs one; the
+   scaffolded workflow sets it).
+3. Opens or updates a PR against `main` with `gh` when available, else GitHub REST with
+   `GITHUB_TOKEN`, `GH_TOKEN` or `GH_ENTERPRISE_TOKEN`. Only GitHub and GitHub Enterprise Server
+   hosts are supported; a GHES host is recognised from `GH_HOST` (or `GITHUB_HOST`,
+   `GITHUB_SERVER_URL`), API base `https://<host>/api/v3`. Without `gh` and without a token the
+   branch is still pushed and the command exits 3 with instructions.
+4. Prints the PR URL and stops. **You never merge it.** Argo reconciles after the merge.
+
+`deploy` prints "No cluster is contacted" in this mode: no kube context is used, so `--context`
+and `--yes` change nothing here. `--env-file` and `--rotate-api-key` are refused (exit 1):
+Secrets are applied with `secrets apply`. `--dry-run` prints the values rewrite, the git plumbing
+and the `gh pr list/create` commands. `--image` names the CI-pushed image (a tagged reference; a
+digest reference is refused with exit 3 because the chart has no `image.digest`; a repository
+other than the chart's `image.repository` gets a warning, since only `image.tag` is written);
+without it `deploy` uses `--tag` or the short git sha and warns that the image must already have
+been pushed by CI (the CLI does not build or push in this mode). A chart `image.repository` that
+still holds `CHANGE-ME` is exit 3.
+
+## The single production gate
+
+Production desired state changes only through a PR touching `values-prod.yaml`. Two paths open
+such a PR (`promote-to-prod` from CI, `deploy --env prod` from a workstation); neither merges.
+The merge requires:
+
+- review from the production approvers in `CODEOWNERS` for `values-prod.yaml`,
+- no self-approval (branch protection "prevent self-approval" and environment "prevent
+  self-review"),
+- `pr_checks` green.
+
+Argo watches `main`, so approval necessarily precedes the production change on both paths. No
+inbound access from GitHub to the cluster exists. No Argo Image Updater is used. After deploying,
+Argo's own health assessment (the readiness probe on `/ready`) shows whether the rollout landed;
+the workflows have no cluster access by design.
+
+## Operating in argocd environments
+
+| Need | Command |
+|---|---|
+| Status | `graph-agents-cli deploy --status --env <env>` (`argocd app get <name>-<env>`; without the `argocd` CLI a bounded `kubectl rollout status` plus the pods' readiness, exit 1 when not ready) |
+| Restart after secret rotation | `graph-agents-cli deploy --restart --env <env>` (`kubectl rollout restart`, then waits for the new pods); the CLI warns that self-heal may revert the annotation and recommends an Argo resource action (`argocd app actions run <name>-<env> restart --kind Deployment`) |
+| Secrets | `graph-agents-cli secrets apply/status --env <env>` from the owner's workstation; Argo never manages the app Secret |
+| Rollback | revert the values change on `main` through a PR (same gate); for emergencies `argocd app history` / `argocd app rollback` on the Argo side, then reconcile git |
+| Diff before merge | `argocd app diff <name>-<env> --revision <branch>` |
+
+`--status`, `--restart` and `secrets` follow the kube context rules (recorded context or
+`--context`; outside dev the current context needs a confirmation or `--yes`). Direct
+`helm upgrade` or `kubectl apply` against an argocd environment is drift: self-heal reverts it
+(dev, staging) or Argo reports `OutOfSync` (prod). Do not do it.
+
+## `helm-push` for comparison
+
+A self-hosted GitHub runner inside the network gets the cluster credentials from the
+`DEPLOY_KUBECONFIG` secret of the `staging` or `production` **environment** (written under
+`RUNNER_TEMP` and removed after the job), resolves one kube context (the manifest's
+`environments.<env>.context`, else the kubeconfig's current one), runs
+`graph-agents-cli deploy --env <env> --image <ref> --context "$KUBE_CONTEXT" --yes` (helm only),
+and verifies the rollout (`kubectl rollout status`, `/health` and `/ready` through a
+port-forward; the runner needs `kubectl` and `curl`). The `production` job declares
+`environment: production` and is gated by that environment's reviewers. Workstation deploys are
+allowed for `dev`; `staging`/`prod` are refused outside CI (`GITHUB_ACTIONS=true`) even with
+`--image`, unless `--force-direct`. Secrets are still provisioned by the owner from a
+workstation, never by CI; the runner's kubeconfig needs permission to read the Secret (deploy
+checks its required keys).