@leverege/build-tools 2.114.0-DEVOP-578.2 → 2.114.0

package/README.md

@@ -44,6 +44,55 @@ Provides: `color()`, `errorExit()`, `warnOnError()`, `exitOnError()`, `generateP
 
 ### Helm & Kubernetes Deployment
 
+#### The "field is immutable" error and `helmup --mutate`
+
+A routine `helmup` will occasionally fail with an error like:
+
+```
+Error: UPGRADE FAILED: cannot patch "my-service" with kind Deployment:
+Deployment.apps "my-service" is invalid: spec.selector: Invalid value: ...:
+field is immutable
+```
+
+**Why it happens.** Kubernetes enforces immutability on certain fields after a resource is first
+created. The most common is `spec.selector` on a Deployment — the label selector that determines
+which pods belong to the workload. Once set at creation time, it can never be patched in place;
+any attempt to change it is rejected by the API server outright, regardless of how the change
+arrives (kubectl apply, helm upgrade, etc.). Helm has no built-in way to recover from this on its
+own.
+([Kubernetes docs — label selector updates](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates))
+
+At Leverege, this most commonly surfaces when a service chart is upgraded from the older
+self-contained flat chart structure to the newer Leverege Base Chart library approach
+(`leverege-base-charts`). As the platform's chart conventions have evolved and improved over the
+years, migrating a chart with `init-my-chart` can alter the rendered selector labels, leaving the
+live Deployment's immutable `spec.selector` incompatible with what helm now wants to apply.
+
+**What `helmup --mutate` does.** Before invoking helm, helmup deletes the `Deployment` object
+for the service by name (`kubectl delete deployments.apps <service>`) and waits for the deletion
+to complete. Helm then runs its normal upgrade, which recreates the Deployment from scratch with
+the new selector. Because the object is gone before helm runs, there is no immutable-field
+conflict to hit.
+
+helmup will ask for confirmation before deleting and prints a warning that brief downtime is
+coming. The upgrade proceeds automatically once you confirm.
+
+**Side effects to be aware of:**
+
+- *Brief downtime* — pods are terminated when the Deployment is deleted and do not come back
+  until helm recreates the object. The duration is typically a few seconds.
+- *LoadBalancers and static IPs are unaffected* — only the Deployment object is deleted; Services
+  (and their external IPs) remain untouched throughout.
+- *StatefulSets are not handled* — `--mutate` targets `deployments.apps` only. If you hit an
+  immutable field on a StatefulSet (e.g. `spec.volumeClaimTemplates`), you must delete the
+  StatefulSet manually using `--cascade=orphan` to preserve PVCs, then re-run helmup normally.
+  ([Kubernetes docs — StatefulSet limitations](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations))
+
+In short: reach for `--mutate` whenever you see the immutable field error on a Deployment. It is
+targeted, explicit, and safe — the only cost is the brief downtime while the object is recreated.
+
+---
+
 **`helmup`** — Context-safe Helm deployment tool. Installs or upgrades one or more services.
 ```bash
 helmup <service> [service...]   # deploy one or more services
@@ -170,8 +219,98 @@ gcp-snapshots --project <id> [--age <days>] [--list|--delete]
 
 ### Kubernetes Utilities
 
-**`k8x`** — Opens an interactive shell (or specific CLI) on a running pod, with fuzzy
-finder support for pod selection. Requires `fzf`.
+**`k8s`** — Unified Kubernetes toolkit with cluster-context safety via `overwhelm`. All
+subcommands verify the active cluster context before acting to prevent cross-cluster
+contamination.
+
+```bash
+k8s fwd <target>                    # port-forward to a named cluster endpoint
+k8s exe [target]                    # exec into a pod — shortcut or fuzzy match
+k8s scale <up|dn> [deployments...]  # scale deployments up or down
+k8s roll [deployments...]           # rolling restart of one or more deployments
+k8s log [service]                   # stream pod logs with pino-pretty formatting
+k8s cryo init                       # generate k8s-cryo.yaml from live cluster state
+k8s cryo hibernate                  # scale all workloads to zero and drain node pools
+k8s cryo wake                       # restore node pools and workload replica counts
+```
+
+**`k8s fwd`** — Port-forwards a named cluster service to localhost. Handles `Ctrl+C` cleanly.
+```bash
+k8s fwd elastic    # localhost:9200 → elasticsearch8:9200
+k8s fwd redis      # localhost:6379 → redis-master:6379
+k8s fwd valkey     # localhost:6379 → valkey-master:6379
+k8s fwd prom       # localhost:9090 → prometheus-operated:9090
+k8s fwd alert      # localhost:9093 → alertmanager-operated:9093
+k8s fwd grafana    # localhost:3000 → prometheus-stack-grafana:3000
+k8s fwd <target> --local <port>     # override the local port
+```
+
+**`k8s exe`** — Execs into a running pod. Named shortcuts resolve to the right namespace
+and CLI automatically; any other string is used as a fuzzy filter against pods in the
+default namespace.
+```bash
+k8s exe               # interactive pod picker (all namespaces)
+k8s exe cnpg          # bash on a cnpg-operands pod
+k8s exe es8           # bash on an elasticsearch8 node
+k8s exe grafana       # bash on grafana (prometheus namespace)
+k8s exe grafold       # bash on legacy grafana (monitoring namespace)
+k8s exe prometheus    # bash on prometheus-operator pod
+k8s exe psql          # psql -U postgres on a CNPG node
+k8s exe redis         # redis-cli on redis master
+k8s exe valkey        # valkey-cli on valkey master
+k8s exe <pattern>     # fuzzy match against pod names in default namespace
+```
+
+**`k8s scale`** — Scales one or more deployments in a namespace. Notifies Slack on each
+result. Interactive picker if no deployment names are given.
+```bash
+k8s scale up api-server message-processor    # scale to 1 replica
+k8s scale dn api-server                      # scale to 0
+k8s scale up                                 # interactive: pick from listed deployments
+k8s scale dn -n monitoring grafana           # scale in a specific namespace
+```
+
+**`k8s roll`** — Rolling restart of one or more deployments (`kubectl rollout restart`).
+Notifies Slack on each result. Interactive picker if no names given.
+```bash
+k8s roll api-server authz-server
+k8s roll                          # interactive picker
+k8s roll -n monitoring grafana
+```
+
+**`k8s log`** — Streams pod logs with pino-pretty formatting. Follows the stream until
+`Ctrl+C`. Can tee raw output to a timestamped file.
+```bash
+k8s log                             # interactive pod picker (all namespaces)
+k8s log api-server                  # stream by app= label selector, default namespace
+k8s log api-server -n staging       # specify namespace
+k8s log api-server --tee            # also save raw logs to api-server-<timestamp>.log
+k8s log api-server --tee my.log     # save to a named file
+k8s log api-server --no-pretty      # raw output, skip pino-pretty
+k8s log api-server --tail 200       # start with last 200 lines (default 50)
+```
+
+**`k8s cryo`** — Cluster hibernation and wake tool. Uses `k8s-cryo.yaml` in the current
+directory to define shutdown order, dependency phases, node pool sizing, and CNPG operator
+awareness. Intended to be run from a cluster directory (e.g. `plat-k8s/tst-builder/`).
+```bash
+k8s cryo init                 # scan live cluster and generate k8s-cryo.yaml
+k8s cryo init --force         # overwrite existing k8s-cryo.yaml
+k8s cryo hibernate            # stop all workloads and drain node pools
+k8s cryo wake                 # restore nodes and scale workloads back up
+k8s cryo hibernate ./path/to/k8s-cryo.yaml   # explicit config path
+```
+`k8s cryo init` scans the live cluster for deployments, StatefulSets, CNPG clusters and
+poolers, node pools, and system namespaces (traefik, cert-manager, external-secrets, velero,
+estafette, prometheus, monitoring). It queries live replica counts for variable-size
+StatefulSets (timescale slave, valkey replicas, elasticsearch data) and emits a complete
+`k8s-cryo.yaml`. Review the generated file — especially the Project phase member list and
+any cluster-specific replica counts — before committing.
+
+---
+
+**`k8x`** *(deprecated — use `k8s exe`)* — Opens an interactive shell (or specific CLI)
+on a running pod, with fuzzy finder support for pod selection. Requires `fzf`.
 ```bash
 k8x               # open bash on any pod (fuzzy select)
 k8x cnpg          # bash on a cnpg cluster node
@@ -182,15 +321,16 @@ k8x redis         # redis-cli on redis master
 k8x valkey        # valkey-cli on valkey master
 ```
 
-**`k8scale`** — Scales Kubernetes deployments or performs rolling restarts.
+**`k8scale`** *(deprecated — use `k8s scale` and `k8s roll`)* — Scales Kubernetes
+deployments or performs rolling restarts.
 ```bash
 k8scale up [services]    # scale up to minReplicas
 k8scale dn [services]    # scale down to zero
 k8scale roll [services]  # rolling restart
 ```
 
-**`klog`** — Streams logs from a pod with pino-pretty formatting. Saves to a timestamped
-log file. Requires `fzf` for interactive selection.
+**`klog`** *(deprecated — use `k8s log`)* — Streams logs from a pod with pino-pretty
+formatting. Saves to a timestamped log file. Requires `fzf` for interactive selection.
 ```bash
 klog                     # fuzzy-select pod and stream logs
 klog <app-name>          # stream logs for named app
@@ -357,6 +497,9 @@ BUILD_TOOLS_DEBUG=1 docker-to-registry
 
 | Tool | Replacement |
 |------|-------------|
+| `k8x` | `k8s exe` |
+| `k8scale` | `k8s scale` / `k8s roll` |
+| `klog` | `k8s log` |
 | `pkgck` | `repo-panopticon` |
 | `docker-to-registry-py` | `docker-to-registry` |
 | `chart-to-museum` | `push-my-chart` |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@leverege/build-tools",
-  "version": "2.114.0-DEVOP-578.2",
+  "version": "2.114.0",
   "description": "A collection of build / support tools for Leverege developers",
   "main": "index.js",
   "repository": {
@@ -100,7 +100,7 @@
     "execa": "^10.0.1",
     "find-yarn-workspace-root": "^2.0.0",
     "glob": "^13.0.6",
-    "googleapis": "^173.0.0",
+    "googleapis": "^174.0.1",
     "handlebars": "^4.7.9",
     "ignore": "^7.0.6",
     "inquirer": "^14.0.2",

package/src/helm-charts/llm-prompt-server/helmup.bootstrap

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+function createLlmPromptServerJwtSecret() {
+  # Create a placeholder k8s secret so llm-prompt-server starts cleanly on first install.
+  # ESO will then be wired up to manage the actual JWT credentials from GCP Secret Manager.
+  kubectl get secret llm-prompt-server-jwt --namespace default &> $DEVNULL
+  if [ $? -ne 0 ];
+  then
+    printf "\nCreating placeholder `color g llm-prompt-server-jwt` secret in default namespace\n"
+    kubectl create secret generic llm-prompt-server-jwt \
+      --namespace default \
+      --from-literal=config='{"jwtSub":"llm-prompt-server","jwtSecret":"bootstrap","jwtIssuer":"imagineService"}'
+    warnOnError $? "Failed to create llm-prompt-server-jwt placeholder secret"
+  else
+    printf "\n`color g llm-prompt-server-jwt` secret already exists - skipping\n"
+  fi
+
+  # Wire up the ExternalSecret only if ESO is enabled for this cluster.
+  # Set ESO_ENABLED=true in overwhelm.yaml to activate (default: false).
+  if [ "${ESO_ENABLED}" = "true" ];
+  then
+    showInstalling "llm-prompt-server JWT ExternalSecret"
+    cat<<LLM_JWT_SECRET | kubectl apply -f -
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: llm-prompt-server-jwt
+  namespace: default
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: gcp-secret-manager
+    kind: ClusterSecretStore
+  target:
+    name: llm-prompt-server-jwt
+    creationPolicy: Merge
+    template:
+      engineVersion: v2
+      data:
+        config: '{"jwtSub":"llm-prompt-server","jwtSecret":"{{ .JWT_SERVICE_SECRET }}","jwtIssuer":"imagineService"}'
+  data:
+    - secretKey: JWT_SERVICE_SECRET
+      remoteRef:
+        key: JWT_SERVICE_SECRET
+LLM_JWT_SECRET
+  else
+    printf "\n`color y '***WARNING:'` ESO_ENABLED is not set - skipping llm-prompt-server JWT ExternalSecret\n"
+    printf "  Set ESO_ENABLED=true in overwhelm.yaml and re-run helmup llm-prompt-server to wire up the JWT secret.\n\n"
+  fi
+}
+
+createLlmPromptServerJwtSecret

package/src/helmup.sh

@@ -20,7 +20,7 @@ function giveHelp() {
       `color g "--k8s-match"` Only proceed if k8s context is correctly set
       `color g "--dry-run"`   Show what helm would do without applying changes
       `color g "--debug"`     Enable verbose debug output
-      `color g "--mutate"`    Forces a deployment mutation
+      `color g "--mutate"`    Delete the k8s deployment object before upgrading (causes downtime)
       `color g "--help"`      Show this help message
 
     Invocation Modes:
@@ -106,7 +106,6 @@ PLATFORM=(
   db-curator
   emailer
   imagine
-  llm-prompt-server
   message-processor
   messenger
   resource-server
@@ -121,6 +120,7 @@ AUXILIARY=(
   argocd
   fota-server
   geotile-server
+  llm-prompt-server
   overdose
   pgbouncer
   pubsub-pulse
@@ -128,6 +128,7 @@ AUXILIARY=(
   pusher
   reason
   transponder-dh
+  triton-inference-server
   vin-decoder-server
   vms-server
 )
@@ -655,23 +656,53 @@ DELETE_WAIT
 
   # plugins pass HELMUP as a single quoted string - split it into args without using eval
   [[ $# -eq 1 ]] && set -- $1
-  "$@" --set "helmup.culprit=$culprit" --set "helmup.deployT=$deployT"
-  
+
+  # Tee helm output to a temp file so we can inspect it on failure while still
+  # streaming it to the terminal in real-time. 2>&1 captures stderr (where helm
+  # writes UPGRADE FAILED messages) alongside stdout.
+  local HELM_LOG
+  HELM_LOG=$(mktemp)
+  "$@" --set "helmup.culprit=$culprit" --set "helmup.deployT=$deployT" 2>&1 | tee "$HELM_LOG"
+  local HELM_EXIT=${PIPESTATUS[0]}
+
   # usually the helm command will run without error but a common failure is caused
   # by the lack of use of our protonVPN profile which causes the i/o timeout from helm
-  if [[ $? -ne 0 ]];
+  if [[ $HELM_EXIT -ne 0 ]];
   then
-    cat<<HELMUP_FAIL
+    if grep -q "field is immutable" "$HELM_LOG";
+    then
+      cat<<IMMUTABLE_FIELD
+
+  $RED_ERROR Helm upgrade failed — an immutable field has changed.
+
+  `color y "This typically happens when a service chart has been updated to use the new Leverege
+  Base Chart structure and the Deployment's label selector has changed. Kubernetes does not
+  allow selector changes on an existing Deployment — the object must be deleted and recreated."`
+
+  `color g "Re-run with --mutate to let helmup delete and recreate the Deployment for you:"`
+
+    `color c "helmup $SERVICE --mutate"`
+
+  `color y "Note: --mutate causes brief downtime while the Deployment is recreated.
+  The service's LoadBalancer, static IP, and any PVCs are unaffected."`
+
+IMMUTABLE_FIELD
+    else
+      cat<<HELMUP_FAIL
 
   $RED_ERROR helmup failed for => `color y "$SERVICE"`
 
   `color y "Please investigate the issue, and if it isn't obvious then check your VPN connectivity!"`
 
 HELMUP_FAIL
+    fi
 
+    rm -f "$HELM_LOG"
     exit 1
   fi
 
+  rm -f "$HELM_LOG"
+
   case "$INVOKED_AS" in
     "helmup"|"helmdn")
       sendToSlack "$*"