toga-ai 1.0.495 → 1.0.496

package/knowledge/1.0/apps/worker/INDEX.md

@@ -10,5 +10,6 @@
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/netsuite/rest.php, library/app/systemmonitor/netsuiteintegration.php |
 | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
+| [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
 | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/workflows/diagnosing-frozen-cron-checkins.md

@@ -0,0 +1,103 @@
+---
+title: Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-03
+owners: ["dfranks"]
+files:
+  - worker/.ebextensions/cron.config
+  - library/app/worker.php
+related:
+  - ../architecture.md
+  - ../features/oneuptime-worker-uptime-monitoring.md
+  - ../../library/features/cron-execution-monitoring.md
+---
+
+## Summary
+
+When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`**
+check-ins, the intuitive diagnosis — a wedged `App_Framework::isProcessRunning()` lock — is
+usually **wrong**. The common cause is **Elastic Beanstalk deployment churn replacing the EC2
+instances underneath the crons**. This workflow is the ordered path that distinguishes the two
+and the interpretation traps along the way.
+
+## Where to look (name mapping)
+
+Sentry project **`worker1`** is the 1.0 worker tier running in the Elastic Beanstalk
+environment **`agilant-worker`** (application "Agilant", `us-west-2`, account
+`502614707982`, reachable with the `default` SSO profile). It is a multi-instance autoscaled
+fleet. **There is no EB environment literally named `worker1`** — the Sentry project name and
+the EB environment name do not match, and looking for the wrong one costs real time.
+
+## The failure mode
+
+An EB deploy that aborts — events read like `Failed to deploy application`,
+`Unsuccessful command execution on instance id(s) ...`,
+`Cannot complete command execution ... no longer running` — causes EB to **replace
+instances**. In-flight cron processes die with their host, and each Sentry monitor's
+`lastCheckIn` freezes at whatever its now-terminated instance last reported. Nothing is
+stuck; nothing needs clearing.
+
+Two consequences worth knowing:
+
+- **Check-ins resume on their own**, typically within ~1 minute of the deploy completing.
+  No manual intervention, no lock to clear, no instance to restart.
+- EB may warn that an aborted deploy left instances on **mixed application versions**. That
+  does need a re-deploy of the intended version — the recovery of the crons does not imply
+  the fleet is on the version you meant to ship.
+
+## Diagnostic path
+
+1. **List the project's monitors and find the stall boundary.**
+   `GET https://sentry.io/api/0/organizations/<org>/monitors/?project=worker1` — read each
+   monitor's `environments[].lastCheckIn`. The cluster of timestamps that all stop at roughly
+   the same moment is the boundary.
+2. **Pull per-monitor check-in history.**
+   `GET .../monitors/<slug>/checkins/` — a `missed` entry **with no `duration`** means the run
+   never started or never reported. A run that started and then hung looks different (it has a
+   start and no clean check-out). This is the first signal separating "process never launched"
+   from "process wedged".
+3. **KEY DISCRIMINATOR — are non-monitored crons on the same host still throwing ordinary
+   exceptions in the same Sentry project's issue stream?** If other crons are still erroring
+   *after* the stall boundary, then PHP runs and Sentry connectivity both work and the host is
+   alive. That rules out "host down / Sentry unreachable" and points at instance replacement
+   or a per-cron blockage.
+4. **Confirm against Elastic Beanstalk.**
+   ```
+   aws elasticbeanstalk describe-events --environment-name agilant-worker --start-time <ISO>
+   aws elasticbeanstalk describe-environment-health --environment-name agilant-worker --attribute-names All
+   ```
+   Correlate deploy / abort / instance-add / instance-remove timestamps against the stall
+   boundary. A deploy window that brackets the boundary is the answer.
+
+## Interpretation caveats
+
+- **Timezones.** Sentry returns **UTC**; the developer reporting the symptom is usually
+  quoting local **CDT (UTC-5)**. Convert before concluding the windows "don't line up" — they
+  usually do.
+- **Establish each monitor's baseline before calling a stale timestamp new.** The `worker1`
+  monitor list chronically contains monitors that have been silently dead for weeks or months
+  for reasons unrelated to any current incident. During an incident these are noise and must
+  not be read as "failed to recover." (Deliberately not listing which ones — that goes stale.
+  Check each monitor's own history.)
+- **Separate high-frequency from low-frequency monitors.** A 5-minute monitor should recover
+  within minutes; an hourly or daily one will not move until its next scheduled slot, and its
+  silence proves nothing.
+- **Alerting-hygiene gap.** Chronically dead cron monitors mean nobody is being alerted on
+  those jobs at all. Worth a separate follow-up whenever you notice one.
+
+## After recovery
+
+A multi-minute cron outage on this tier means missed windows for order processing, email
+sending, and inbound order imports. These are catch-up-on-next-run designs and generally
+self-drain — but **eyeball queue depth / backlog after recovery rather than assuming clean**.
+
+## Change history
+
+- 2026-08-03 — Documented from a production diagnostic session: frozen cron timestamps +
+  `missed` check-in flood on `worker1` traced to EB deployment churn on `agilant-worker`, not
+  a wedged `isProcessRunning()` lock. (dfranks)

package/knowledge/2.0/apps/_underscore/features/email-send-pipeline.md

@@ -7,7 +7,7 @@ client: shared
 type: feature
 status: active
 updated: 2026-07-29
-owners: ["bala", tcox]
+owners: ["bala", "tcox", "dfranks"]
 files:
   - _underscore/Email.php
   - worker2/Worker/Infrastructure/Email/Send.php
@@ -71,11 +71,47 @@ SMTP** using PHPMailer, then flips each row to `SENT` or `FAILED`. So a 2.0 emai
   `DevTeam@goagilant.com` is a known-good verified sender identity in worker2 config. See
   [ai-bdr web-funnel-app.md](../../ai-bdr/features/web-funnel-app.md).
 
+- **⚠ READ-AFTER-WRITE ON THE READER REPLICA LOSES `send()`'S OWN LOG ROW (deterministic).**
+  `_Email::send()` INSERTs the `PENDING` `Logs_[Client].Email` row, then reads `$emailLog->id`
+  back. On a `_Model`, `save()` sets the PK in memory from `getInsertId()` but does **not** clear
+  `_model_needsToBeInitialized`, so the next `->id` access triggers `__get` → `initialize()`,
+  which issues a fresh `SELECT ... WHERE id=<new id>`. `DB_CLIENT_LOGS` is registered with a
+  **separate `readHost`**, so that SELECT is routed to the **reader replica** — but the INSERT is
+  still inside an **uncommitted** lazy transaction, so it has produced no binlog event and the
+  replica **can never** see it during this request. The read returns 0 rows and `initialize()`
+  throws `"exactly 1 row expected, 0"`. This is **100% deterministic, not a lag race** — an
+  uncommitted write cannot replicate. A connection *can* see its own uncommitted writes
+  (read-your-own-writes), so forcing the read onto the writer is the fix.
+  - **Why only some emails hit it:** approval/order-placed emails send from inside the **api2
+    CRUD POST path** (and worker2), which call `_Database::setIsReadHostEnabled(false)`
+    (`V2.php` create loop; worker2 `Controller/Index.php`) — their identical read-back hits the
+    **writer** and succeeds. The scripted-API path (`/email-templates/sendEmail`) never suppresses
+    the read host, so it throws every time. This was the root cause of the Compass In-Transit
+    email outage — see
+    [`compass-partial-in-transit-delivered-emails.md`](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
+  - **Fix direction (smallest blast radius):** wrap the log-row `save()` + id read (+ attachment
+    loop) in `_Email::send()` with `_Database::setIsReadHostEnabled(false)`, restoring the prior
+    value in a `finally` (get/set at `Database.php`). Fixes all scripted-API emails and hardens the
+    interceptor path.
+  - **Rejected alternatives:** (a) `commit` right after `save()` before the read — recreates the
+    failure as an *intermittent* replication-lag race AND breaks the request's one-transaction
+    boundary (orphans the row on a later throw, prematurely commits everything else buffered on the
+    connection, strands attachment/API-log writes on a closed lazy transaction). (b) Clear
+    `_model_needsToBeInitialized` after INSERT in `_Model::save()` so `->id` reads from memory
+    (zero queries) — conceptually the *correct* fix but changes every model in the ORM, too broad
+    for a hotfix.
+
 ## Change history
 - 2026-07-29 — Noted that the SES SMTP credentials/settings now have a **non-PHP consumer**: the
   BDR funnel sends via nodemailer against the same SES host (us-west-2, port 587, TLS) using
   `TOGA_SMTP_*` env vars, so a rotation must be coordinated with BDR's `.env.local` + Amplify
   vars or BDR's summary email goes dark. (tcox)
+- 2026-07-29 — Documented the **deterministic read-after-write failure** in `_Email::send()`:
+  reading `$emailLog->id` after INSERT triggers `_Model::__get`→`initialize()`→ a reader-replica
+  SELECT for an uncommitted row → `"exactly 1 row expected, 0"` throw. Only the scripted-API send
+  path fails (CRUD/worker paths suppress the read host via `setIsReadHostEnabled(false)`); this was
+  the root cause of the Compass In-Transit outage. Fix direction = suppress the read host around the
+  log-row save+read in `send()`. (dfranks)
 - 2026-07-28 — Created: documented that `_Email::send()` **queues** (writes a `PENDING`
   `Logs_[Client].Email` row + attachment BLOBs) rather than transmitting, and that the worker2
   `Infrastructure/Email/Send` cron (`Core.CronJobs` `* * * * *`) does the actual SES SMTP send

package/knowledge/2.0/apps/api2/INDEX.md

@@ -11,7 +11,7 @@
 | [Nested FK object embedding is gated by the CHILD record's own ACL](features/nested-fk-acl-embedding.md) | When the V2 JSON engine serializes a foreign-key field into a **nested object** (in `getFullModelData()`, ~V2.php L6016-6060), it re-checks the **child** record | api2/Component/Api/V2/V2.php, dbchanges2/Client_Compass/2026-07-23b - PurchaseOrdersRecordReadAcl.sql |
 | [Nested-relationship writes & child matching (link vs. create)](features/nested-relationship-writes.md) | When a 2.0 API write payload (`POST`/`PUT`) contains a **nested related object** (e.g. | api2/Component/Api/V2/V2.php, _underscore/Model/Client/ContactEmailAddress.php |
 | [Record Scripts (computed/aggregate /v2 endpoints — the authoring contract)](features/record-scripts.md) | In api2 you almost never write a controller. | api2/Component/Api/V2/V2.php, _underscore/Model/Team/Sprint.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Query.php |
-| [V2 Request Logging & Where Requests Land (client vs core log DB)](features/request-logging.md) | The V2 engine logs **every inbound request** — success *and* failure, with response code and payload — and routes each log entry to the **client** log or the ** | api2/Component/Api/V2/V2.php, _underscore/Model/Client/Logs/Api.php, _underscore/Model/Core/Logs/Api.php |
+| [V2 Request Logging & Where Requests Land (client vs core log DB)](features/request-logging.md) | The V2 engine logs **every inbound request** — success *and* failure, with response code and payload — and routes each log entry to the **client** log or the ** | api2/Component/Api/V2/V2.php, api2/Controller/Index.php, _underscore/Model/Client/Logs/Api.php, _underscore/Model/Core/Logs/Api.php |
 | [POST + JSON-body args for scripted APIs](features/scripted-api-post-body-args.md) | The V2 engine can run a Record Script (scripted API) for a **POST** request, and a scripted API can receive its arguments from the **JSON request body** instead | api2/Component/Api/V2/V2.php |
 | [TOGa IQ Sprint Dashboard API (Record Scripts)](features/sprint-dashboard-api.md) | The internal **TOGa IQ sprint dashboard** is served in production by **six api2 Record Scripts** on `_Model_Team_Sprint` (`_underscore/Model/Team/Sprint.php`), | _underscore/Model/Team/Sprint.php, api2/Component/Api/V2/V2.php, dbchanges2/Core/2026-07-24a - SprintDashboardRecordScripts.sql, dbchanges2/Client_True/2026-07-24b - SprintDashboardScriptAcl.sql |
 | [Surface action-state via the surface=<slug> request option (M2M-safe)](features/surface-meta-option.md) | An opt-in V2 engine request option, `surface=<slug>`, that attaches per-record UI action state (`isVisible`/`isEnabled`) to a GET response **under `meta.surface | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Surface.php |

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -7,9 +7,10 @@ client: shared
 type: feature
 status: active
 updated: 2026-07-29
-owners: ["mhammontree"]
+owners: ["mhammontree", "dfranks"]
 files:
   - api2/Component/Api/V2/V2.php
+  - api2/Controller/Index.php
   - _underscore/Model/Client/Logs/Api.php
   - _underscore/Model/Core/Logs/Api.php
 related:

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 14 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 17 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 18 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 10 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.495",
+  "version": "1.0.496",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",