toga-ai 1.0.673 → 1.0.674

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -5,6 +5,7 @@
 | [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php, worker2/composer.json |
 | [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/_shared/040-write-instance-id.sh, worker2/.platform/hooks/_shared/041-write-region.sh, worker2/.platform/hooks/_shared/042-write-eb-environment.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, saml/.platform/hooks/_shared/040-write-instance-id.sh, saml/.platform/hooks/_shared/041-write-region.sh, saml/.platform/hooks/_shared/042-write-eb-environment.sh, saml/.platform/hooks/prestart/040-write-instance-id.sh, saml/.platform/hooks/prestart/041-write-region.sh, saml/.platform/hooks/postdeploy/040-write-instance-id.sh, saml/.platform/hooks/postdeploy/041-write-region.sh, saml/.platform/hooks/postdeploy/042-write-eb-environment.sh, saml/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, saml/ebs/register_instance_to_shared_application_load_balancer.php, saml/composer.json, api2/.platform/hooks/prebuild/_shared/042-write-eb-environment.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
 | [All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)](features/all-client-email-queue-monitor.md) | `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0 outbound email queue (`Logs_<Client>.Email` — see [2.0 Email Send Pipeli | worker2/Worker/Monitor/Operations.php, _underscore/Database.php, _underscore/Query.php, dbchanges2/Logs_Client/2026-05-21 - Email.sql |
+| [API 2.0 Stability Monitor (Monitor/Operations/ApiStability)](features/api-stability-monitor.md) | `_Worker_Monitor_Operations::ApiStability()` is a cross-client/operational **end-to-end round-trip probe** of the production 2.0 API. | worker2/Worker/Monitor/Operations.php, dbchanges2/Core/2026-08-28a - API 2.0 Stability Monitor Cron.sql |
 | [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
 | [Callback Scheduling ("call me back") — worker2 AI-BDR](features/callback-scheduling.md) | The **AI-BDR callback path**: what happens between a prospect saying *"call me back later"* on a Vapi call and the dialer actually placing that second call. | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
 | [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |

package/knowledge/2.0/apps/worker2/features/api-stability-monitor.md

@@ -0,0 +1,140 @@
+---
+title: API 2.0 Stability Monitor (Monitor/Operations/ApiStability)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-28
+owners: ["jcardinal"]
+files:
+  - worker2/Worker/Monitor/Operations.php
+  - dbchanges2/Core/2026-08-28a - API 2.0 Stability Monitor Cron.sql
+related:
+  - ./oneuptime-worker2-monitoring.md
+  - ./netsuite-integrations-monitor.md
+  - ./all-client-email-queue-monitor.md
+  - ./elastic-beanstalk-health-monitor.md
+---
+
+## Summary
+
+`_Worker_Monitor_Operations::ApiStability()` is a cross-client/operational **end-to-end
+round-trip probe** of the production 2.0 API. Unlike a log-scraper (blind to a dead
+endpoint) or a cheap health-endpoint liveness probe, it performs a real authenticated
+call chain against `api.togahub.com/v2` and only reports OK when the API round-trips a
+representative read correctly. It follows the established "dumb reporter, smart monitor"
+OneUptime push pattern — same shape as the sibling
+[NetsuiteIntegrations](./netsuite-integrations-monitor.md) and
+[EmailQueue](./all-client-email-queue-monitor.md) monitors on
+`_Worker_Monitor_Operations`. Shared internal operational infrastructure, **not** a client
+feature.
+
+**Motivating incident.** A `Core.Records` model failed to deploy to production. Because the
+2.0 API requires a model for every `Core.Records` entry, any call touching the missing
+model returned HTTP 500 — while the Elastic Beanstalk instances stayed "healthy" and the
+`/v2/health` short-circuit route kept returning 200. A monitor that actually round-trips
+the API would have gone red immediately; a health-endpoint probe and an EB health check
+both stayed green. This probe closes that gap.
+
+## Key files / entry points
+
+- `worker2/Worker/Monitor/Operations.php` — `abstract class _Worker_Monitor_Operations`;
+  `public static function ApiStability(): string`, route
+  `Monitor/Operations/ApiStability`. All config is `ALL_CAPS` in-method locals (no
+  arguments): `$ONEUPTIME_URL` (push credential), `$API_BASE`, timeouts, expected statuses,
+  `$EXPECTED_CLIENT_AUTH_TYPE = 'SSO'`, `$DOMAINS_QUERY`, and `$ORIGIN`.
+- `dbchanges2/Core/2026-08-28a - API 2.0 Stability Monitor Cron.sql` — the `Core.CronJobs`
+  seed row (see below).
+
+## How it works
+
+Two sequential sub-checks against **production** `api.togahub.com/v2` (the `$API_BASE` is
+hardcoded — the monitor watches prod regardless of which environment runs it). HTTP goes
+through `_ApiRequest` with `ENCODE__JSON` (so `execute()` returns the decoded object) and a
+fresh `transactionId` per request via `_String::generateUuid()`:
+
+1. **Auth** — `POST /v2/auth/public?transactionId=<uuid>` with an `Origin` header (see the
+   gotcha below). Expect **HTTP 201**; extract the bearer token from `data.tokens.access`.
+2. **Representative read** — `GET /v2/domains?<fields/join/where>&transactionId=<uuid>` with
+   `Authorization: Bearer <token>` and the same `Origin`. The query is a representative
+   joined read across `Domains` + `Environments` + `ClientAuthentications` + `Apps`, scoped
+   to the `compass.togasupply.com` PRODUCTION domain. Expect **HTTP 200 AND**
+   `data.domains[0].ClientAuthentications.type === 'SSO'` (`$EXPECTED_CLIENT_AUTH_TYPE`).
+   This is the exact call that failed during the missing-model incident.
+
+Any failed sub-check sets `alarm = 'DEGRADED'` (else `OK`). The method pushes
+`{status:'reporting', alarm, authStatus, domainsStatus, failureCount, failures,
+checkedAtUtc}` to OneUptime, then returns a structured summary string recorded in
+`WorkerJobs.output`.
+
+Compass USA's production supply domain is used only as a **representative** live call — this
+is shared platform/liveness knowledge, not client-specific behavior.
+
+### Never-throw / non-fatal push
+
+Logging is off, `throwExceptionsOnFailure` is off, and the push is wrapped so a
+monitoring-side failure never fails the worker job — the standard reporter contract from
+[the OneUptime push-metric pattern](./oneuptime-worker2-monitoring.md).
+
+## Cron seed (dbchanges2)
+
+`dbchanges2/Core/2026-08-28a - API 2.0 Stability Monitor Cron.sql` — one `Core.CronJobs`
+row: action `Monitor/Operations/ApiStability`, schedule `*/5 * * * *` (every 5 min, all
+hours), `maxExecutionTime` 120, `parameters` NULL (no arguments), **`isActive = 0`**
+(ships inert per the deploy-inert convention — flip to 1 after the OneUptime monitor exists
+and its push URL is set). Core-database-only references (cluster-isolation satisfied); no
+`Core.Records`/`RecordFields` involved.
+
+## Gotchas / known issues
+
+- **`POST /v2/auth/public` returns HTTP 401 without an `Origin` request header.** The
+  public-auth endpoint carries **no** client credential; the API resolves which client/app
+  the public token is minted for from the request's `Origin` header. A server-side call must
+  set it explicitly (`Origin: https://compass.togasupply.com`) — the browser supplies Origin
+  automatically, which is why the front end's public-token flow
+  (`toga-blox-npm/src/api/auth.ts` POSTs to `/auth/public` with a null body + JSON
+  content-type) never sees this. Send the same `Origin` on the `/domains` call too, mirroring
+  the browser. This is the non-obvious fact for anyone scripting a server-side call to the
+  public-auth endpoint. (Verified 2026-08-28: with Origin, auth → 201 and domains → 200 +
+  `SSO`; without it, auth → 401.)
+- **The push URL is a credential** — it lives only as the in-method `$ONEUPTIME_URL` local,
+  never logged and never written into a doc or into dbchanges2.
+- **Watches PRODUCTION regardless of runner.** `$API_BASE` is hardcoded to prod, so a
+  non-prod worker still probes prod — intended, but do not "fix" it to the running env.
+
+## OneUptime import artifact
+
+Provisioned by importing an Incoming-Request monitor export (a local staging artifact,
+`C:\STAGE\monitor-export-api-2.0-stability-2026-08-28.json` — **not** a repo file) with three
+criteria: body Contains `"alarm":"DEGRADED"` → Degraded + incident (templated with
+`{{requestBody.failures}}` etc.); body Contains `"alarm":"OK"` → Operational;
+not-received-25-min → Offline + incident. Per the
+[activation-ordering trap](./oneuptime-worker2-monitoring.md#gotchas--known-issues), the
+absence criterion ships `isEnabled: false` — enable it after the first push lands.
+Project-specific status/severity ObjectIDs were cloned from an existing monitor export.
+
+## Change history
+
+- 2026-08-28 — Built `ApiStability()` on `_Worker_Monitor_Operations` — a two-step
+  end-to-end round-trip probe (auth/public → 201 → bearer, then a joined `/v2/domains` read
+  → 200 + `SSO`) against production `api.togahub.com/v2`, plus the `Core.CronJobs` seed
+  (`*/5 * * * *`, `isActive=0`) and an Incoming-Request OneUptime import. Motivated by a
+  missing `Core.Records` model that 500'd real calls while EB/health-endpoint stayed green.
+  Recorded the durable gotcha that `POST /v2/auth/public` **401s without an `Origin`
+  header** (the API resolves the client from Origin). Verified working end-to-end after the
+  Origin fix. (jcardinal)
+
+## Related docs
+
+- [OneUptime push-metric monitors for 2.0 workers](./oneuptime-worker2-monitoring.md) — the
+  push/token pattern, OneUptime criteria, import-JSON schema, deploy-inert and
+  activation-ordering conventions, and the liveness-probe-vs-log-scraper distinction.
+- [NetSuite Integrations Monitor](./netsuite-integrations-monitor.md) and
+  [All-Client Email Queue Monitor](./all-client-email-queue-monitor.md) — sibling
+  `_Worker_Monitor_Operations` monitors modeled on the same reporter contract.
+- [Elastic Beanstalk Health Monitor](./elastic-beanstalk-health-monitor.md) — EB-level
+  health this probe deliberately reaches past (EB stayed healthy during the incident).
+</content>
+</invoke>

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-20
+updated: 2026-08-28
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -195,6 +195,21 @@ touches it. Rules for the probe:
   duplicate credentials into a second repo and race the shared access token. Accept and record the
   coverage gap rather than forking the client.
 
+#### A round-trip probe is stronger than a health-endpoint probe
+
+A health endpoint that short-circuits before any real work (`/v2/health` → 200) proves the
+process is up, but **not** that the API can actually serve a request. A missing `Core.Records`
+model once 500'd every real call while `/v2/health` and EB health both stayed green. Where an
+integration matters, add an **end-to-end round-trip probe** that authenticates and reads a
+representative record, asserting on the returned data — not just the HTTP status. First
+instance: [API 2.0 Stability Monitor](./api-stability-monitor.md), which mints a public token
+then reads `/v2/domains` and asserts `ClientAuthentications.type === 'SSO'`.
+
+**Gotcha for a server-side public-auth call:** `POST /v2/auth/public` returns **HTTP 401
+without an `Origin` request header** — the endpoint carries no client credential and resolves
+the client/app from `Origin`. The browser supplies it automatically; a scripted call must set
+it explicitly. See the API-stability doc for details.
+
 ### Tunable thresholds live in `Core.CronJobs.parameters`, everything else stays a local
 
 The alarm threshold is the **one** setting that goes in the DB, as
@@ -356,6 +371,12 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-08-28 — Added the **round-trip probe** refinement (an authenticated end-to-end read
+  that asserts on returned data is stronger than a `/v2/health` short-circuit or EB health
+  check — a missing `Core.Records` model 500'd real calls while both stayed green) and the
+  durable gotcha that `POST /v2/auth/public` **401s without an `Origin` header** (the API
+  resolves the client from Origin). First instance:
+  [API 2.0 Stability Monitor](./api-stability-monitor.md). (jcardinal)
 - 2026-08-24 — Documented the **monitor import-JSON schema** the runbook previously described
   only conceptually: the `oneuptime-resource-export` envelope, `monitorType` must be the
   spaced display name `"Incoming Request"`, the `_type`-tagged
@@ -410,6 +431,7 @@ check for another client.
   5-min cadence → 10/15-min heartbeat timing. (jcardinal)
 
 ## Related docs
+- [API 2.0 Stability Monitor](./api-stability-monitor.md) — the first end-to-end round-trip probe instance (auth/public + /domains SSO assertion)
 - [Monitoring Framework](./monitoring-framework.md) — the parallel DB-driven, email-alert monitoring pattern
 - [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md) — `_Cloud::getS3Objects()` used to list the EDI bucket
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor

package/knowledge/INDEX.md

@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 70 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 57 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 58 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 13 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.673",
+  "version": "1.0.674",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",