toga-ai 1.0.644 → 1.0.646

package/knowledge/1.0/apps/library/features/cron-execution-monitoring.md

@@ -6,12 +6,13 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-07-06
-owners: [dfranks]
+updated: 2026-08-25
+owners: [dfranks, bala]
 files:
   - library/app/framework.php
 related:
   - ../../worker/architecture.md
+  - ../../worker/workflows/tracing-a-worker-cron-run-in-production.md
 ---
 
 ## Summary
@@ -38,13 +39,28 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   cast silently discards the sub-second precision even when the schema column is `decimal(10,3)`.
 - **Schema:** the table is provisioned by the dbchanges migration
   `dbchanges/Common/DF/2026-5-7 Cron Checkin.sql` (table `CronJobExecutions`,
-  `executionTimeSeconds decimal(10,3)`, no `note` column).
+  `executionTimeSeconds decimal(10,3)`). **Correction (2026-08-25): prod `Common.CronJobExecutions`
+  does have a `note` TEXT column** — verified by writing to it and reading it back. `App_Framework`
+  never populates it, which makes it the practical place to park **temporary** step tracing for a
+  cron you are diagnosing (worker crons have no readable stdout — see
+  [Tracing a worker cron run in production](../../worker/workflows/tracing-a-worker-cron-run-in-production.md)).
+  Remove the tracing when you are done, and never write payloads or credentials into it.
+- **Don't confuse the two `note`s.** `cronInitialization()` writes `note = 'Started execution'` to
+  the **`Log`** table on `db_log` (the older `CRON`/`recordType` row), not to `CronJobExecutions`.
 - **Design decision:** 1.0 records executions by writing **directly to the shared DB**
   (`db_common`), *not* by POSTing from 1.0 to a 2.0 API endpoint. Per Jeff Cardinal, 1.0 code
   must not POST to 2.0 code; the direct shared-DB write is the sanctioned path (TRUE-78182).
 
 ## Gotchas
 
+- **⚠ A run skipped by the overlap guard leaves NO ROW — absence is ambiguous.**
+  `cronInitialization()` calls `exitIfProcessRunning()` **before** the `CronJobExecutions` INSERT,
+  and the `cronFinished(true)` that guard then calls does nothing because `cronLogId` was never
+  set. So a blocked run is indistinguishable from a run that never launched: **a missing row for
+  an expected slot means "skipped", not "hung"**, and a hung run looks different (it has
+  `dtCheckIn` with no `dtCheckOut`). `isProcessRunning()` matches **any** `ps -ef` line containing
+  the script path (it only excludes `/bin/sh` and its own pid), so a stray `tail -f` or editor on
+  that path blocks the cron indefinitely.
 - **`db_log` has no `CronJobExecutions` table.** A `cronFinished()` UPDATE that runs against the
   `db_log` connection throws on **every** 1.0 cron job. Both the INSERT and the UPDATE must
   target `db_common`.
@@ -60,6 +76,12 @@ row in `db_log` and of the per-job Sentry check-in monitors.
   logic. (Fixed 2026-07-06: consolidated back to one `db_common` INSERT/UPDATE pair.)
 
 ## Change history
+- 2026-08-25 — Corrected the schema note (prod `CronJobExecutions` **does** carry a `note` TEXT
+  column, unused by `App_Framework`, usable for temporary cron tracing) and separated it from the
+  `note = 'Started execution'` row `cronInitialization()` writes to `Log` on `db_log`. Recorded
+  that the overlap guard runs **before** the INSERT, so a skipped run writes no row at all and
+  cannot be told apart from "never launched" — and that `isProcessRunning()`'s `ps -ef` substring
+  match lets any process holding the script path block the cron. No code change. (bala)
 - 2026-07-06 — Fixed a merge (`origin/_production`) that reintroduced duplicate
   `CronJobExecutions` writes and pointed one `cronFinished()` UPDATE at `db_log` (where the
   table doesn't exist), which would have thrown on every 1.0 cron. Consolidated to one

package/knowledge/1.0/apps/worker/INDEX.md

@@ -11,9 +11,10 @@
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
 | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
-| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php |
+| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Staples cXML Order Import (SFTP → NetSuite)](features/staples-cxml-order-import.md) | `sync_staples_cxml.php` is an **hourly** cron (runs at **:45**) that imports Staples cXML purchase orders from SFTP into NetSuite as Sales Orders, then writes a | worker/crons/sync/staples/sync_staples_cxml.php |
 | [Diagnosing frozen 1.0 worker cron check-ins (Sentry "missed" flood)](workflows/diagnosing-frozen-cron-checkins.md) | When 1.0 worker cron timestamps freeze and Sentry project `worker1` fills with **`missed`** check-ins, the intuitive diagnosis — a wedged `App_Framework::isProc | worker/.ebextensions/cron.config, library/app/worker.php |
 | [isFulfillable Multi-Client Backfill (all togasupply clients)](workflows/isfulfillable-multi-client-backfill.md) | One-time backfill that catches up `Items.isFulfillable` on **existing** items across **all 17 togasupply clients** (AIG, Broward Sheriff, Canon, Endeavor Health | worker/crons/toga2/netsuite/backfill_isfulfillable_all_clients.php, library/app/api/toga2.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL |
+| [Tracing a 1.0 worker cron run in production (no stdout, silent skips)](workflows/tracing-a-worker-cron-run-in-production.md) | How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where **there is no usable stdout** and **a skipped run leaves no tr | worker/ebs/cron.worker.php, worker/schedules/cron.worker.sync.json, library/app/framework.php |

package/knowledge/1.0/apps/worker/architecture.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-07-29
-owners: [jcardinal, sking]
+updated: 2026-08-24
+owners: [jcardinal, sking, bala]
 files:
   - worker/index.php
   - worker/_/app/framework.php
@@ -20,6 +20,8 @@ files:
 related:
   - ../library/architecture.md
   - ../dbchanges/workflows/authoring-and-shipping-sql-files.md
+  - ./features/oneuptime-worker-uptime-monitoring.md
+  - ../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -44,6 +46,7 @@ which **self-elects a distinct role** (`notification`, `database`, `infrastructu
 under `crons/`; the schedule (cron registration) is the source of truth for what runs — a script
 that isn't scheduled never executes. Client integrations live under `crons/toga2/<client>/`.
 Use prepared statements for all SQL; never interpolate input.
+A worker that fails deploy-time role election runs NO crons while EB still reads healthy.
 
 ## How a job becomes a cron (the dispatch pipeline)
 
@@ -108,6 +111,32 @@ is the most important and least obvious part of the architecture.
 > Net effect: roles are a **claim-the-first-free-slot pool**, and a dead worker's role is
 > reclaimed by its replacement within minutes. There is no static instance→role mapping to edit.
 
+### The failure mode: NO NAME = NO CRONTAB (and it is silent)
+
+The election is the **single point of failure for an entire instance**, and the architecture gives
+it no second chance:
+
+- The claim runs **only at DEPLOY time** — `ebs/cron.worker.php` is written as the EB `appdeploy`
+  enact hook. **It never retries.** An instance that fails to claim stays nameless until the next
+  deploy or until something terminates it.
+- The claim writes `/etc/worker-role` and **only then** appends `cron.worker.<role>.json` to the
+  crontab. So a nameless instance has **no role crontab at all** — it runs *nothing*, not even
+  `worker_heartbeat.php`, while **EB reports the environment healthy**. The box is up; it just
+  does no work.
+- **The claim's DB connect is `@mysqli_connect(...)` — error-suppressed.** A boot-time connect
+  failure to `Vision_Log` therefore leaves the instance nameless **with no log line anywhere**.
+  This is the prime suspect for the 2026-08-24 incident (2 of 7 workers roleless for ~3 hours),
+  and it violates the no-`@`-suppression rule in `rules/toga/common/coding-style.md`. Removing the
+  `@` and logging the failure is the recommended fix.
+- `Workers` **only ever holds instances that SUCCEEDED**, so a failed claim leaves no row —
+  **absence is the only evidence**, and detecting it requires the AWS instance list.
+
+Because every OneUptime monitor on this tier is keyed by **role** rather than by instance, none of
+them can see this (see
+[the blind spot](./features/oneuptime-worker-uptime-monitoring.md)). External coverage now comes
+from the 2.0
+[Worker Fleet Role-Assignment Monitor](../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+
 ## Anatomy of a cron script
 
 Every script is self-contained and follows this boilerplate:
@@ -208,6 +237,10 @@ web face (`mvc/` GET routes for login/logout/404); the tier's real work is the c
 
 ## Conventions & gotchas
 
+- **A worker that fails role election is silently dead, not degraded.** No name → no crontab →
+  the instance runs nothing while EB reads healthy, and the `@`-suppressed `mysqli_connect` in
+  `ebs/cron.worker.php` logs nothing. Never treat "EB is green" as evidence the fleet is working;
+  check `Vision_Log.Workers` against the actual EB instance list.
 - **`dbchanges` auto-apply is branch-gated, and `_production` is NOT auto-applied.**
   `crons/infrastructure/execute_dbchanges.php` runs every 2 minutes but only for branches matching
   `_%`, deriving the env by stripping the leading `_` and requiring `config.<env>.ini` in the

package/knowledge/1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md

@@ -6,12 +6,17 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-13
-owners: ["jcardinal"]
+updated: 2026-08-24
+owners: ["jcardinal", "bala"]
 files:
   - library/app/worker.php
   - worker/crons/worker/worker_heartbeat.php
-related: ["oneuptime-server-monitor-host-hygiene"]
+  - worker/ebs/cron.worker.php
+  - worker/.ebextensions/045_oneuptime_agent.config
+related:
+  - ./oneuptime-server-monitor-host-hygiene.md
+  - ../architecture.md
+  - ../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
 ---
 
 ## Summary
@@ -57,6 +62,31 @@ Database, Infrastructure, TOGa, TOGa Desk, and Catalog, then importing each into
 manually. To add a new worker's monitor, clone an existing monitor export the same way and
 register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
 
+## THE BLIND SPOT — every monitor here is keyed by ROLE, so a ROLELESS instance is invisible
+
+Load-bearing limitation, learned the hard way (2026-08-24: **2 of the 7 production workers ran
+roleless for ~3 hours undetected**). This layer, and the Server monitors in
+[host hygiene](./oneuptime-server-monitor-host-hygiene.md), are keyed by `workerName` — **14
+monitors, none keyed by instance**:
+
+- `worker_heartbeat.php` only pings `App_Worker::$oneUptimeEndpoints[$row['workerName']]` for
+  the row matching **its own** `instanceId`. No row → no ping target → it pushes nothing.
+- The `045_oneuptime_agent.config` Server agent reads `/etc/worker-role` and **exits clean when
+  it is empty.**
+
+So an instance that never claimed a role reports to nothing — and because every role is still
+owned by *someone*, **no role-keyed monitor is missing a ping either.** The fleet reads 100%
+healthy while a box does zero work (role assignment happens **only at deploy time**, and no role
+means the role's `cron.worker.<role>.json` is never appended to the crontab, so the instance runs
+**nothing**).
+
+`Vision_Log.Workers` only ever holds instances that **succeeded** in claiming a role, so the
+failure leaves no row and no log line — **absence is the only evidence**, and detecting an
+absence requires the AWS instance list, which nothing on this tier consults. That gap is what
+the 2.0
+[Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md)
+now closes from outside the fleet. **Do not try to close it with another role-keyed monitor.**
+
 ## Gotchas
 
 - Do **not** hardcode the heartbeat URLs/UUIDs anywhere but `App_Worker::$oneUptimeEndpoints`
@@ -67,6 +97,13 @@ register the resulting heartbeat URL in `App_Worker::$oneUptimeEndpoints`.
   pending cleanup — the live source of truth is the `App_Worker::$oneUptimeEndpoints` map.
 
 ## Change history
+- 2026-08-24 — Documented **the blind spot**: all 14 monitors on this tier are keyed by
+  `workerName`, so an instance that claimed **no** role pings nothing and leaves every role-keyed
+  monitor still green — which is how 2 of 7 production workers ran roleless for ~3 hours
+  undetected. Recorded that `Vision_Log.Workers` only holds successful claims (absence is the
+  only evidence) and that the gap is now covered from outside the fleet by the 2.0
+  [Worker Fleet Role-Assignment Monitor](../../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md).
+  (bala)
 - 2026-07-13 — Added external OneUptime heartbeat push for all 1.0 workers on top of the
   existing internal DB-heartbeat mechanism; endpoints keyed by workerName in
   `App_Worker::$oneUptimeEndpoints`. (jcardinal)

package/knowledge/1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md

@@ -0,0 +1,124 @@
+---
+title: Tracing a 1.0 worker cron run in production (no stdout, silent skips)
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: workflow
+status: active
+updated: 2026-08-25
+owners: ["bala"]
+files:
+  - worker/ebs/cron.worker.php
+  - worker/schedules/cron.worker.sync.json
+  - library/app/framework.php
+related:
+  - ../architecture.md
+  - ./diagnosing-frozen-cron-checkins.md
+  - ../../library/features/cron-execution-monitoring.md
+  - ../../../1.0/standards/backend-php.md
+---
+
+## Summary
+How to answer *"did this cron actually run, and what did it do?"* on the 1.0 `worker` tier, where
+**there is no usable stdout** and **a skipped run leaves no trace at all**. Written from a
+production diagnostic session on
+`worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`; the mechanics
+apply to every cron on the tier.
+
+Use this before you conclude a cron "never fired" — three different situations look identical
+from the outside.
+
+## Step 1 — get the real schedule from `schedules/*.json`, never from the file
+The `.php` file's own header comment is not authoritative and is routinely stale. So are skills,
+docs and tickets. `worker/schedules/cron.<env>[.<role>].json` is the only source of truth.
+
+Worked example: the entry *"Download EDI From S3 Files & Create PO - EDI Office Depot"* in
+`cron.worker.sync.json` is `*/5 * * * *` — **every 5 minutes**. The cron file's header said
+"EVERY HOUR" (corrected 2026-08-25) and at least one skill still says hourly. A wrong assumed
+frequency makes you read a normal gap as an outage.
+
+Also check whether the job is registered in the env you are testing in at all: this one is **not**
+in `cron.beta.json`, so beta never runs it on a schedule. Nothing on beta is evidence about it.
+
+## Step 2 — echo/print goes nowhere useful, so do not debug with it
+The crontab line is assembled in `worker/ebs/cron.worker.php`, and the two blocks that build it
+differ:
+
+| Schedule file | Command written | Where output goes |
+|---|---|---|
+| base `cron.<env>.json` | `php /var/www/html/crons/<script>` | **nowhere** — the ` > /var/www/cache/WORKER_ERROR_$RANDOM` redirect is **commented out** |
+| role `cron.worker.<role>.json` | `php /var/www/html/crons/<script> > /var/www/cache/WORKER_ERROR_$RANDOM` | a **randomly named** file under `/var/www/cache/` on whichever of the 7 instances ran it |
+
+Either way you cannot go and read it: base-schedule output is discarded outright, and a
+role-schedule run leaves an unpredictable filename on a box in an autoscaled fleet. **Treat
+worker cron stdout as write-only.**
+
+## Step 3 — put temporary tracing somewhere you can query
+What worked: write timestamped step markers into the **`note`** column of the run's
+`Common.CronJobExecutions` row (the row `App_Framework::cronInitialization()` already inserted),
+then read them back over the DB tooling from your desk. `dtCheckIn`, `dtCheckOut` and
+`executionTimeSeconds` on the same row give you the shape of the run for free.
+
+Rules: keep it to a few markers, and **remove the tracing once diagnosed** (it was removed in
+this session). Never write payloads or credentials into `note`.
+
+## Step 4 — read `Common.CronJobExecutions` to tell "ran", "hung" and "skipped" apart
+Query the legacy env's `Common.CronJobExecutions` filtering `job LIKE` the script path:
+
+- **`dtCheckIn` + `dtCheckOut` + `executionTimeSeconds`** — it ran and finished; the duration
+  tells you whether it fits inside its schedule slot.
+- **`dtCheckIn` with no `dtCheckOut`** — it started and never completed (crash, timeout, or the
+  instance went away).
+- **no row for a slot you expected** — the run was **skipped by the overlap guard**, not "never
+  fired". A blocked run writes **nothing**: `cronInitialization()` calls
+  `exitIfProcessRunning()` **before** the INSERT, and the `cronFinished(true)` it then calls is a
+  no-op because `cronLogId` is unset. See
+  [Cron Execution Monitoring](../../library/features/cron-execution-monitoring.md).
+
+### The overlap guard is a `ps -ef` substring match — anything can block it
+`App_Framework::isProcessRunning()` returns true for **any** `ps -ef` line containing the script
+path (it only excludes `/bin/sh` lines and its own pid). So a developer's `tail -f`, an editor, or
+any shell holding that path in its command line **blocks the cron indefinitely** while looking
+like a legitimate overlap. Check `ps -ef` for what is actually holding the name before assuming a
+long-running instance of the job itself.
+
+## Step 5 — if runs overlap, look at what the job does before its real work
+A job whose own runtime exceeds its schedule interval silently loses most of its slots to the
+guard. Measure the phases, not the total: in the ODP case an instrumented run showed the S3
+listing alone returning **138,472 objects and taking 32 seconds** before any PO was touched, with
+each PO then costing roughly 40 seconds — comfortably past a 5-minute schedule. Prefixes the job
+skips while processing are still listed, so accumulated `SENT/` and `OUTBOX/` objects inflate
+every run.
+
+## Deploy-time trap: a PHP 8-only syntax kills the whole cron with no visible error
+This tier runs **PHP 7.2**, so a **PHP 8 named argument** (`someFunction(items: $x)`) is a
+**parse error**, not a runtime warning:
+
+```
+PHP Parse error: syntax error, unexpected ':', expecting ',' or ')'
+```
+
+The script then does nothing at all on every tick — and per Steps 2 and 4 you will see no output
+and no `CronJobExecutions` row, i.e. it looks exactly like "the cron never fired". Typed
+parameters and return types (`: void`, `: array`, `: bool`, `?array`, `?object`, `object $x`) are
+PHP 7.0/7.1 and **are** safe here; they are already used in
+`worker/crons/toga2/compass/backfill_items_isfulfillable.php` and
+`update_item_fulfillments_in_netsuite.php`.
+
+`1.0/standards/backend-php.md` already forbids named arguments in 1.0 — but a workspace
+`CLAUDE.md` / `.github/copilot-instructions.md` that says *"use PHP 8 Named Parameters"* applies
+to **2.0 only**. The 1.0 standard wins in `library/`, `worker/` and every other 1.0 app.
+
+**Do not read a PHP 8 function call in existing code as proof the tier is PHP 8.**
+`library/app/systemmonitor/500error.php` uses `str_contains()`; that file is not a version
+signal, it is a latent bug.
+
+## Change history
+- 2026-08-25 — Documented from a prod diagnostic session on the Compass ODP 850 importer: the
+  schedule JSON (not the file header) is authoritative, worker cron stdout is unreadable (base
+  block's `WORKER_ERROR` redirect commented out, role block writes a random filename),
+  `Common.CronJobExecutions.note` is the practical place for temporary tracing, a run skipped by
+  the `ps -ef` overlap guard writes **no row at all** (guard runs before the INSERT), the guard
+  matches any process holding the script path, and a PHP 8 named argument parse-errors the whole
+  cron silently on this PHP 7.2 tier. (bala)

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -19,7 +19,7 @@
 | [Core.Domains — the app host registry (and why you cannot derive a host)](features/core-domains-app-host-registry.md) | `Core.Domains` is the **authoritative registry of which host serves which client's app in which environment**. | _underscore/Model/Core/Domain.php, api2/Component/Api/CrossClient/CrossClient.php, api2/Component/Api/V2/V2.php, worker/config.beta.ini |
 | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
 | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, _underscore/String.php, worker2/Worker/Infrastructure/Email/Send.php, _underscore/Model/Client/Logs/Email.php, _underscore/Model/Client/Logs/EmailAttachment.php |
-| [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
+| [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Quad/ApprovalDecision.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
 | [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Database.php, _underscore/Exception/Business.php, api2/Controller/Index.php, worker2/Controller/Index.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, worker2/Worker/Infrastructure/Errors.php, tools/mvc/errors/issue/get.php, tools/mvc/errors/post.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
 | [FIELD_STORAGE fields — per-row lazy hydration and the platform-wide missing-column 500](features/field-storage-row-hydration.md) | `FIELD_STORAGE` is the 2.0 field type for blob-backed columns (S3 or local folder). | _underscore/Model.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/Invoice.php, dbchanges2/Core/HISTORIC/2024/2024-11b - item-fulfillments.sql |
@@ -37,8 +37,9 @@
 | [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona **names** in multiple languages by adding a per-language **sidecar** table `PersonaTranslations`, reusing the platform's existing metadata-driven | _underscore/Model/Client/PersonaTranslation.php, dbchanges2/Client/2026-07-22b - PersonaTranslations.sql, dbchanges2/Core/2026-07-22a - PersonaTranslationsRecord.sql, dbchanges2/Client/2026-07-22c - PersonaTranslationsAcl.sql, dbchanges2/Client_CompassCanada/2026-07-22 - PersonaTranslationsFrench.sql, toga2-commerce/src/pages/Account/view/MySettingsView.tsx |
 | [Record Change Audit Log (Logs_<Client>.Record / RecordField) — reading a field's history](features/record-change-audit-log.md) | Every 2.0 client schema has a sibling **logs** schema `Logs_<Tenant>` (e.g. | _underscore/Model/Client/Logs/Record.php, _underscore/Model/Client/Logs/RecordField.php, _underscore/Model/Client/Logs/CustomRecordField.php |
 | [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | In a multi-tier supply chain a sales order (SO) spawns a purchase order (PO) that becomes another SO downstream, and so on. | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillmentItem.php, _underscore/Model/Client/ItemFulfillmentItemUnit.php, _underscore/Model/Client/ItemFulfillmentPackage.php, _underscore/Model/Compass/AdvanceShippingNotice.php, dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql, dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql, dbchanges2/Client_Compass/2026-08-18 - FixSA135471HeroItemHalfQuantityFulfillment.sql |
+| [Per-client sales-order status filter (Surface FILTER_SET → table meta `filterOptions`)](features/sales-order-status-filter-surface.md) | The status-filter dropdown on the sales-orders table is **per client**, driven by a Surface `FILTER_SET` rather than by the raw contents of the client's `SalesO | _underscore/Model/Client/TableView.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/SalesOrder.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Prudential/SalesOrder.php, toga-blox/src/components/Table/hooks/useFetchTablePageMeta.ts, toga-blox/src/api/types.ts, dbchanges2/Core/2026-08-24b - SalesOrderStatusFilterSurfaceSeed.sql, dbchanges2/Client_Compass/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_CompassCanada/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_Quad/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_Nychh/2026-08-24b - SalesOrderStatusFilterHides.sql |
 | [_String helpers — ASCII-safe HTML entity encoding (and the parseBetween trap)](features/string-html-entity-helpers.md) | `_String` is the 2.0 framework's static string utility class. | _underscore/String.php |
-| [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | The runtime for the platform-wide **Surface** UI presentation layer: 9 `_underscore` models plus a cached resolver, `_Model_Core_Surface::resolve(&$api, string | dbchanges2/Core/2026-08-21 - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Core/2026-08-24 - RestoreApproveDenyRowActionsVisibility.sql, dbchanges2/Client_Quad/2026-08-24 - ProdPortApprovePoNumberEnabledRule.sql, dbchanges2/Client_Compass/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_CompassCanada/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql, toga25-supply/src/App.tsx, toga25-supply/src/surface/useFetchSurfaceMeta.ts, toga25-supply/src/contexts/AuthContext.tsx, dbchanges2/Core/2026-07-21a - SalesOrderDecisionSummarySurfaceSeed.sql, dbchanges2/Core/2026-07-21b - SalesOrderDecisionActionSurfaceSeed.sql, dbchanges2/Core/2026-08-21a - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Client_Compass/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Quad/2026-07-21c - SalesOrderDecisionSummaryOverride.sql, dbchanges2/Client_CompassCanada/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Compass/2026-08-04a - ApprovalDetailsAssignedManagerPreferredStage.sql, _underscore/Model/Core/Surface.php, _underscore/Model/Client/AclRecordScript.php, _underscore/Model/Core/RecordScript.php, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Quad/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Client_CompassCanada/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Client_Compass/2026-07-15f - SalesOrderRecordActionsRemoveDeadConfigRuleOverrides.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, _underscore/Model/Core/SurfaceElement.php, _underscore/Model/Core/Action.php, _underscore/Model/Core/Vocabulary.php, _underscore/Model/Core/VocabularyTerm.php, _underscore/Model/Core/Message.php, _underscore/Model/Client/SurfaceOverride.php, _underscore/Model/Client/MessageTranslation.php, _underscore/Model/Client/ThemeToken.php, _underscore/Model/Core/Page.php, api2/Component/Api/V2/V2.php |
+| [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | The runtime for the platform-wide **Surface** UI presentation layer: 9 `_underscore` models plus a cached resolver, `_Model_Core_Surface::resolve(&$api, string | _underscore/Model/Client/Language.php, dbchanges2/Core/2026-08-21 - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Core/2026-08-24 - RestoreApproveDenyRowActionsVisibility.sql, dbchanges2/Client_Quad/2026-08-24 - ProdPortApprovePoNumberEnabledRule.sql, dbchanges2/Client_Compass/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_CompassCanada/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql, toga25-supply/src/App.tsx, toga25-supply/src/surface/useFetchSurfaceMeta.ts, toga25-supply/src/contexts/AuthContext.tsx, dbchanges2/Core/2026-07-21a - SalesOrderDecisionSummarySurfaceSeed.sql, dbchanges2/Core/2026-07-21b - SalesOrderDecisionActionSurfaceSeed.sql, dbchanges2/Core/2026-08-21a - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Client_Compass/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Quad/2026-07-21c - SalesOrderDecisionSummaryOverride.sql, dbchanges2/Client_CompassCanada/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Compass/2026-08-04a - ApprovalDetailsAssignedManagerPreferredStage.sql, _underscore/Model/Core/Surface.php, _underscore/Model/Client/AclRecordScript.php, _underscore/Model/Core/RecordScript.php, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Quad/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Client_CompassCanada/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Client_Compass/2026-07-15f - SalesOrderRecordActionsRemoveDeadConfigRuleOverrides.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, _underscore/Model/Core/SurfaceElement.php, _underscore/Model/Core/Action.php, _underscore/Model/Core/Vocabulary.php, _underscore/Model/Core/VocabularyTerm.php, _underscore/Model/Core/Message.php, _underscore/Model/Client/SurfaceOverride.php, _underscore/Model/Client/MessageTranslation.php, _underscore/Model/Client/ThemeToken.php, _underscore/Model/Core/Page.php, api2/Component/Api/V2/V2.php |
 | [Table-View Hyperlink Columns (meta → ACL → computed URL → render)](features/tableview-hyperlink-columns.md) | Any 2.0 table-view column can render its value as a clickable link instead of plain text. | _underscore/Model/Client/TableView.php, _underscore/Model/Client/TrackingNumber.php, api2/Component/Api/V2/V2.php, toga2-supply/src/api/toga.ts, toga2-supply/src/components/ui/Tables/PrimaryTable/helperFunctions/formatTableData.tsx, toga2-supply/src/components/ui/Tables/PrimaryTable/helperFunctions/convertData.tsx, toga2-supply/src/components/ui/Tables/hooks/useDataTableState.tsx, dbchanges2/Client/2026-07-20 - TrackingNumberHyperlinkAndFieldPermission.sql |
 | [TableView joins (TableViewJoins → SQL) — aliasing, chained multi-hop joins, ACL](features/tableview-joins.md) | `Client_*.TableViewJoins` rows are what let a table view show a column from a table other than its base record. | _underscore/Model/Client/TableView.php, dbchanges2/Client/2026-08-11a - ServiceRequestsInventoryUnitsTableViews.sql, dbchanges2/Client_Elite/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql |
 | [TogaIQ Gateway Client (_Component_Api_Togaiq) — AI generate/translate from 2.0](features/togaiq-gateway-client.md) | `_Component_Api_Togaiq` is the 2.0 framework's client for the **TogaIQ** (Talos) AI gateway. | _underscore/Component/Api/Togaiq/Togaiq.php, _underscore/ApiRequest.php |

package/knowledge/2.0/apps/_underscore/features/email-template-sending.md

@@ -6,10 +6,12 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-03
+updated: 2026-08-25
 owners: ["jcardinal", "bala", "mhammontree", "apeterson"]
 files:
   - _underscore/Model/Client/EmailTemplate.php
+  - _underscore/Model/Quad/ApprovalDecision.php
+  - _underscore/Model/Quad/SalesOrder.php
   - _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php
   - _underscore/Email.php
 related:
@@ -105,9 +107,31 @@ can keep using `sendEmail($api, ...)`.
   → `$args = ['userName'=>$name, ...]`). Passing the map as a single positional argument
   (`sendEmail($api, $uuid, $to, [], [], $vars)`) makes `$args = [0 => [...]]` — a numeric key
   whose value is an *array*. `replaceTemplateVariables()` then runs `str_replace('{0}', <array>, …)`
-  → `TypeError: str_replace(): Argument #2 must be of type string, array given`, surfacing as a
-  production 500 / **EO-1**. `Model/Compass/*` gets this right (spread); the four `Model/Quad/*`
+  → `TypeError: str_replace(): Argument #2 ($replace) must be of type string when argument #1
+  ($search) is a string`, surfacing as a production 500 / **EO-1** — it fails the **whole request**,
+  not just one placeholder. `Model/Compass/*` gets this right (spread); the four `Model/Quad/*`
   call sites were copied from Compass but dropped the `...` — fixed 2026-08-18.
+  - **This contract has regressed TWICE.** `c22999d9` (2026-05-19) **removed** the spread and broke
+    Quad order-approval emails; `2b0c1470` (2026-08-18) restored it. `sendEmail` has been variadic
+    since **2024-12-24**, so the spread has **always** been the correct form and the May commit was
+    the regression, not a style change. The contract and both regressions are now recorded on
+    `sendEmail()`'s docblock, with a marker comment at each of the four Quad call sites — because
+    that is where the mistake actually gets made.
+  - **Hardened 2026-08-25 — a mis-spread caller now degrades instead of 500-ing.**
+    `replaceTemplateVariables()` (now typed) **skips non-scalar values**, logging the key and the
+    **type only — never the value** (template vars carry user names and order numbers, so logging
+    them would be a PII violation). Result: a bad caller leaves an **unreplaced placeholder** in the
+    email rather than taking down order approval. Verified behaviourally: the old code fatals on the
+    no-spread shape, the new code degrades, and the correct spread path is **byte-identical**. The
+    spread is still the contract — the guard is a safety net, not a licence to pass a bare array.
+  - **⚠ Still seeing this error in an environment does NOT mean the code is wrong — check the deploy
+    FIRST.** Verified 2026-08-25: Quad order placement was still 500-ing in the **commerce
+    sandbox-client** environment (`EmailTemplate.php:82` via `_Model_Quad_SalesOrder::postPost`)
+    even though the fix `2b0c1470` was already committed on `_sandbox-client`. `_underscore` is
+    **cloned at build time from a moving per-environment branch**, so the running artifact simply
+    predated the commit. That is a **deploy gap, not a code bug** — a redeploy fixes it, and no code
+    change should be made. Same known-issue as
+    [api2 environment-variable-drives-underscore-branch](../../api2/features/environment-variable-drives-underscore-branch.md).
 - **`sendEmail()`'s signature is load-bearing for scripted APIs** — the Record Script engine
   (`api2/Component/Api/V2/V2.php`, ~line 3594) calls the method with `api` as a named
   argument, so the first param must stay `&$api`. Do not "clean it up" by removing it.
@@ -161,6 +185,19 @@ worker method) in-process instead.
 
 ## Change history
 
+- 2026-08-25 — Added the **deploy-gap** caveat to the spread gotcha: Quad order placement was still
+  500-ing in the commerce **sandbox-client** environment after `2b0c1470` landed, because
+  `_underscore` is cloned at build from a moving per-environment branch and the running artifact
+  predated the commit. Diagnose the deploy before touching the code. (apeterson)
+- 2026-08-25 — Hardened `replaceTemplateVariables()` (signature typed) to **skip non-scalar values**,
+  logging key + **type only** (never the value — template vars carry user names and order numbers), so
+  a mis-spread caller degrades to an unreplaced placeholder instead of a 500 that kills the whole
+  request. Verified the old code fatals on the no-spread shape, the new code degrades, and the correct
+  spread path is byte-identical. Recorded that the spread contract has regressed **twice** —
+  `c22999d9` (2026-05-19) removed it and broke Quad order-approval emails, `2b0c1470` (2026-08-18)
+  restored it — and that `sendEmail` has been variadic since 2024-12-24, so the spread was always
+  correct and May was the regression. Documented the contract on `sendEmail()`'s docblock plus a marker
+  comment at each of the four Quad call sites. (apeterson)
 - 2026-08-18 — Fixed the Quad order-approval/rejection emails' production 500
   (`str_replace(): Argument #2 must be string`, EO-1): the four `Model/Quad/ApprovalDecision.php`
   + `Model/Quad/SalesOrder.php` call sites passed the template-vars map as a bare positional

package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
-owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
+updated: 2026-08-24
+owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi", "bala"]
 files:
   - _underscore/Database.php
   - _underscore/Model.php
@@ -177,6 +177,16 @@ here — they live in `Config/*.ini`.)
 - Related 1.0 analogue: the legacy `App_` worker has the same hazard writing to `Logs.API`
   (`db_logs`) — the laptop trap there is documented separately in the worker NetSuite bootstrap
   notes.
+- **`_Database::register()` is LAZY — it stores connection config and never opens a socket.**
+  The connect happens on first use. So adding a static alias registration to an app's `_.php`
+  bootstrap is **safe across every environment**, even one whose config group points at a
+  localhost that lacks the schema: nothing fails until something actually queries that alias.
+  This is what makes "register the alias globally, use it in one cron" a two-line change rather
+  than a per-environment config exercise (worked example: `DB_VISION_LOGS` in
+  [the Worker Fleet Role-Assignment Monitor](../../worker2/features/worker-fleet-role-assignment-monitor.md)).
+  The corollary is the trap the rest of this doc describes: because registration is free and
+  silent, a **bad** registration also stays silent until the request that needs it blows up with
+  `Unknown database`.
 
 ## WITHDRAWN — the "alias-keyed `$_modelCache` cross-tenant leak" hypothesis (2026-08-17)
 
@@ -210,6 +220,11 @@ also hits `_modules/<module>/`** — is recorded in
 
 ## Change history
 
+- 2026-08-24 — Recorded that **`_Database::register()` is lazy** (stores config, opens no
+  socket; the connect happens on first use), so a static alias registration in an app's `_.php`
+  is safe in every environment including a localhost config that lacks the schema — and that
+  this is also why a *bad* registration stays silent until first query. Verified while adding
+  `DB_VISION_LOGS` to worker2. (bala)
 - 2026-08-17 (later pass) — **WITHDRAWN, supersedes the entry below.** The alias-keyed
   `$_modelCache` cross-tenant-leak hypothesis is **not supported** and is no longer a live security
   concern. The six `c_` columns are **AIG's own**: `_modules/netsuite/2026-07-10a -