toga-ai 1.0.540 → 1.0.542

package/knowledge/1.0/apps/library/INDEX.md

@@ -9,6 +9,7 @@
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
 | [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
+| [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |
 | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
 | [isFulfillable from NetSuite during Item Sync (Phase 1)](features/netsuite-item-isfulfillable-sync.md) | This is the **1.0 (Phase 1)** half of the `isFulfillable` feature: reading the NetSuite `isfulfillable` flag during item sync and stamping it onto the **Agilant | library/app/netsuite.php, library/app/api/toga2.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/backfill_isfulfillable_jul5.php |
 | [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem, test/@dave/Junk Drawer/nsq.php |

package/knowledge/1.0/apps/library/features/http-500-error-monitor.md

@@ -0,0 +1,169 @@
+---
+title: "HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-10
+owners: ["bala"]
+files:
+  - library/app/systemmonitor/500error.php
+  - worker/crons/infrastructure/system_monitors.php
+  - api2/Controller/Index.php
+  - _underscore/Error.php
+related:
+  - ./error-capture-1-0.md
+  - ./netsuite-sync-alert-monitor.md
+  - ./cron-execution-monitoring.md
+  - ../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
+  - ../../../../2.0/apps/api2/features/v2-api-error-codes.md
+---
+
+## Summary
+
+`App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title
+**"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` for
+`responseCode = 500`** and raises a ClickUp task + a `devteam@togatech.com` email when it finds
+any. It is the alert most api2 production incidents arrive through.
+
+> **⚠ The "Error Type" line in the alert is almost always the literal string
+> `No message found in response payload`, and that string carries ZERO diagnostic signal.** It is
+> generated **by this monitor**, not by the API, and it is the *expected* value for every genuine
+> production api2 500. Do not spend time on it, do not search for it, and do not treat two alerts
+> that share it as the same bug. **The real diagnostic is `error.id` in the logged response
+> payload** — see the lookup recipe below.
+
+## How it works
+
+- **Schedule.** `minuteFrequency = 5`, `minHourToRun = 8`, `maxHourToRun = 17`,
+  `alertRemindFrequency = 240`. Registered in the shared dispatcher
+  `worker/crons/infrastructure/system_monitors.php` alongside the other `App_SystemMonitor`
+  classes (same 300s shared budget — see
+  [NetSuite Sync Alert Monitor](./netsuite-sync-alert-monitor.md) for how a monitor can silently
+  die inside that budget).
+- **Query.** `getStatus()` selects `id, requestHeaders, instanceId, route, method, dtStamp,
+  queryString, responsePayload` from **`Logs.Api`** where `responseCode = 500` and
+  `dtStamp >= NOW() - INTERVAL 5 MINUTE`, on the **`db_toga2_logs`** connection. Zero rows = OK.
+  **The window is 5 minutes and the monitor keeps no cursor**, so its view of an incident is only
+  ever the last 5 minutes of it.
+- **Non-production rows are filtered by a referer heuristic** — a `referer:` header containing
+  `beta`/`gamma`, or lacking `https://` or `.com`, marks the row non-prod and skips it.
+- **Grouping.** `groupErrors()` keys each group on `referer . ' HTTP 500 Error: ' . $errorMessage`
+  and creates/updates one ClickUp task per key in list `901110669877` (or the current dev-sprint
+  list when it resolves one). Two ClickUp custom fields are set from substring matches on the
+  request headers: a **stakeholder** map (`compass`, `prudential`, `quad`, `spglobal`, `nychh`,
+  `growrk`) and a **platform** map (`togasupply`, `togacommerce`, `togadesk`), each falling back
+  to a default uuid.
+
+### Where the "Error Type" string comes from
+
+`groupErrors()` reads `$respPayload->messages[0]->identifiers->message` out of the logged
+`responsePayload` and falls back to the literal `'No message found in response payload'` when that
+path is absent (`500error.php:93-99`).
+
+**api2 leaves those identifiers empty on every production 500 by design.**
+`api2/Controller/Index.php` (~L271-285) populates `message` and `trace` **only** when
+`_Environment::isDebugMode()` is true, because `getTrace()` returns frame **arguments** verbatim
+and any stack with `_Database::register()` on it would hand the database username and password to
+anyone who could force a 500. See the api2
+[error-codes doc](../../../../2.0/apps/api2/features/v2-api-error-codes.md) for that decision.
+
+So the fallback is not a failure — it is the **normal** state in production. Its only real meaning
+is "this 500 came from production api2."
+
+## The real diagnostic: `error.id` (verified lookup recipe)
+
+The api2 envelope carries `error: { code, id }`. **`id` is the quotable Core Logs reference**, in
+the form `<Logs.Issue.reference>-<Logs.Event.eventNumber>` (e.g. `78-859`), produced by
+`_Error::captureException()` (`_underscore/Error.php:334+`), which resolves/creates a `Logs.Issue`
+by fingerprint, writes a `Logs.Event`, and returns the reference. Three queries get you from an
+alert to the actual exception:
+
+```sql
+# 1. Read error.id out of the logged envelope (ids come from the alert body)
+SELECT id, route, method, dtStamp, responsePayload
+FROM Logs.Api
+WHERE id IN (<ids from the alert>);
+
+# 2. The exception itself + its full framework stack (reference = the part before the dash)
+SELECT id, reference, status, errorMessage, trace, totalOccurrences, dtLastOccurred
+FROM Logs.Issue
+WHERE reference = '78';
+
+# 3. The specific occurrence — and WHICH TENANT it belongs to
+SELECT clientId, dtCreated, errorMessage
+FROM Logs.Event
+WHERE issueId = <issue.id> AND eventNumber = <n>;
+```
+
+- `Logs.Issue.trace` is the full framework stack — this is the thing the alert deliberately does
+  not contain.
+- `Logs.Event.clientId` is how you find out **which client** was affected; the monitor's
+  stakeholder guess is a header substring match and is not authoritative.
+- **api2's own catch block is the ONLY capture point for the inbound-API 500 surface.** api2
+  handles `Throwable` in its front controller, so `_Error::exceptionHandler` never fires for API
+  traffic. If api2's catch did not capture it, nothing did.
+
+Full mechanics of the Issue/Event model, the reference encoding, and its failure modes:
+[error reporting](../../../../2.0/apps/_underscore/features/error-reporting-issue-event.md).
+
+## ⚠ Known defect (2026-08-10, NOT fixed) — all unrelated 500s collapse into ONE ClickUp task
+
+Because the grouping key is `referer + errorMessage` and `$errorMessage` is the **constant**
+`No message found in response payload` for every production 500, the key degenerates to the
+referer alone. Consequences:
+
+- Distinct, unrelated defects sharing a referer are **merged into one ClickUp task**, whose
+  description accumulates mixed routes and mixed incidents.
+- The responder cannot tell from the task whether it is one bug or five, and the task's occurrence
+  count is meaningless as an impact signal.
+
+**Proposed fix (recorded, not implemented):** parse the `Logs.Issue` reference out of `error.id`
+and **group on that** — it is exactly the platform's own deduplicated identity for a distinct
+problem — and surface `error.id` in the task description so the responder can skip the manual
+lookup entirely. Falling back to the current key only when `error.id` is null keeps graceful
+rejections working.
+
+## Gotchas / known issues
+
+- **A 500 whose request has NO `referer:` header passes the production filter and groups under an
+  empty referer.** The `$isProd = false` assignment only happens *inside* the referer branch
+  (`500error.php:55-74`), so a header set with no referer line at all keeps `$isProd = true` and
+  `$referer = ''`. Server-to-server inbound API traffic has no referer, so all of it lands in one
+  unnamed group — compounding the collapse defect above.
+- **Repeat count in the alert is NOT user impact.** apiproxy retries a 5xx across regions, so one
+  user action routinely produces several near-identical `Logs.Api` 500 rows within seconds, on
+  different `instanceId`s, **sharing one `transactionId`**. Check the transaction id before
+  reading a burst as a burst of users.
+- **The 5-minute window means the alert is a sample, not the incident.** Always go to `Logs.Api` /
+  `Logs.Issue` for the full extent and the first occurrence — `Issue.dtLastOccurred` +
+  `totalOccurrences` are the real timeline.
+- **`Logs.Issue.status = RESOLVED` does not mean fixed** — it means *stopped firing* (the worker2
+  frequency-decay cron auto-resolves on consecutive quiet windows). A live defect on a low-traffic
+  tenant reads as RESOLVED the moment that tenant goes quiet. See
+  [error-issue auto-resolution](../../../../2.0/apps/worker2/features/error-issue-auto-resolution.md).
+- **Hardcoded ClickUp API token in source.** `CLICK_UP_API_ACCESS_TOKEN` is a literal class
+  constant in this file at line 14 (pre-existing, same pattern as the NetSuite Sync Alert monitor).
+  It belongs in config/secrets; treat the in-repo value as compromised. (Location only — the value
+  is never copied into the KB.)
+- **Stakeholder/platform attribution is a substring guess.** `str_contains($headers, 'growrk')`
+  style matching mis-attributes any request whose headers happen to mention another tenant, and
+  silently defaults when nothing matches. Use `Logs.Event.clientId` for the real tenant.
+
+## Change history
+
+- 2026-08-10 — Created while investigating a production api2 500 alert for GroWrk. Recorded that
+  the alert's "Error Type" is the monitor's **own fallback literal** (`500error.php:93-99`) and is
+  the expected value for **every** production 500 — because api2 gates `message`/`trace` behind
+  `_Environment::isDebugMode()` to avoid returning `getTrace()` frame arguments (DB credentials) to
+  external callers — so it carries no diagnostic signal at all. Documented the **`error.id`
+  three-query lookup recipe** (`Logs.Api.responsePayload` → `Logs.Issue.reference` →
+  `Logs.Event.eventNumber`/`clientId`) as the actual diagnostic path, and that api2's catch block is
+  the only capture point for the inbound-API 500 surface. Recorded the **open grouping defect**
+  (constant `errorMessage` collapses every unrelated 500 sharing a referer into one ClickUp task;
+  proposed grouping on the `Logs.Issue` reference from `error.id` instead — not implemented), the
+  no-referer rows that pass the prod filter into an empty group, that a burst of near-identical 500s
+  sharing one `transactionId` is apiproxy's cross-region retry rather than user impact, and that
+  `Issue.status = RESOLVED` means "stopped firing", not "fixed". (bala)

package/knowledge/2.0/apps/_underscore/features/acl-permission-chain.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-04
-owners: ["jcardinal", "mhammontree", "tcox"]
+updated: 2026-08-10
+owners: ["jcardinal", "mhammontree", "tcox", "bala"]
 files:
   - api2/Component/Api/V2/V2.php
   - _underscore/Model/Core/Page.php
@@ -135,6 +135,29 @@ its `CustomRecordFields` row was deleted, so the pending `AclCustomFieldPermissi
 (never hardcode), and clone a sibling field's per-role rows — e.g. `Client/2026-07-02c -
 TrackingNumberNeedsReturnLabelFieldPermission.sql` is the precedent shape.
 
+### A custom (`c_`) field has THREE parts, and each missing part fails DIFFERENTLY
+
+The custom-field counterpart of the four-requirement standard-field recipe below. Knowing which
+symptom maps to which missing part is what turns a 500 into a one-line diagnosis:
+
+| Missing piece | Symptom |
+|---|---|
+| the **physical column** on the client table | MySQL **1054 "Unknown column"** → **HTTP 500 / `EO-1`** on **every** load of that record |
+| the **`CustomRecordFields`** row (client DB) | the field is **not requestable** through the v2 API at all |
+| the **`AclCustomFieldPermissions`** grant | **`EZ-2` / HTTP 403 for the WHOLE request** when the field is named in `fields=` — one unreadable field fails the entire GET |
+
+The fourth part is the **declaration**: a `c_` property on the client model **or on a trait it
+`use`s** (e.g. `_Model_Growrk_Unit` is nothing but `extends _Model_Client_Unit` +
+`use _Trait_Netsuite_Unit`, and the trait declares all seven `c_` fields). That declaration is what
+makes a missing column **fatal**: `_Model` builds its SELECT from every declared property, so it
+asks for the column whether or not the caller requested it.
+
+**Practical consequence for an audit:** "which columns must this client have?" cannot be answered by
+reading the model class — follow its `use` statements. And a `CustomRecordFields` row that exists
+with **no** grant is not automatically a defect; compare against a healthy client on the same module
+first (see
+[client schema-drift audit](../../dbchanges2/workflows/client-schema-drift-audit.md)).
+
 ## Where ACL rows live: Core vs. Client database
 
 `Core.Records.aclDatabase` decides which database holds the ACL rows for that record:
@@ -335,6 +358,15 @@ hardcoded `Core.RecordFields` id literals instead of a subselect.
 
 ## Change history
 
+- **2026-08-10** — Added the **custom (`c_`) field THREE-part anatomy** as the counterpart of the
+  standard-field recipe — physical column (missing → MySQL 1054 → 500/`EO-1` on every load),
+  `CustomRecordFields` row (missing → not requestable), `AclCustomFieldPermissions` grant (missing →
+  `EZ-2`/403 for the **whole** request) — plus the fourth, code-side part: the `c_` declaration may
+  live on a **trait**, not the model class (`_Model_Growrk_Unit` declares nothing; all seven `c_`
+  fields come from `_Trait_Netsuite_Unit`), which is why `_Model` demands the column regardless of the
+  requested `fields=`. Noted that a `CustomRecordFields` row with no grant can be the normal
+  cross-client state, so compare with a healthy same-module client before repairing. Surfaced by the
+  production `Client_Growrk.Units` `c_lastModified` outage. (bala)
 - **2026-08-06** — TRUE-80282 testing: recorded that **`appId` + `indirectRecordId` both NULL is a
   grant with no scoping at all**, that **`Base` is not a low-privilege role** (every portal customer
   and API service user carries it), and that the dangerous combination to audit for is *unscoped

package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-06
-owners: ["dfranks", "jcardinal", "mhammontree", "ajean"]
+updated: 2026-08-10
+owners: ["dfranks", "jcardinal", "mhammontree", "ajean", "bala"]
 files:
   - _underscore/Error.php
   - _underscore/Database.php
@@ -34,6 +34,7 @@ related:
   - ../../../1.0/apps/library/features/error-capture-1-0.md
   - ../../api2/features/v2-api-error-codes.md
   - ../../dbchanges2/architecture.md
+  - ../../../../1.0/apps/library/features/http-500-error-monitor.md
 ---
 
 ## Summary
@@ -330,6 +331,42 @@ work side by side. Accepted cost: 10 of the 68 existing issues keep ambiguous re
 > `IssueFingerprint`, `IssueEmailAddress`, `IssueClickupTask` and `Event` all reference
 > `Issue.id` — they must be cleared in the **same** operation or they orphan onto reused ids.
 
+### Looking up an `error.id` — from an alert to the actual exception (3 queries)
+
+`error.id` in the api2 envelope is **`<Issue.reference>-<Event.eventNumber>`** (e.g. `78-859`).
+That string is the *only* handle a production 500 gives you: api2 returns **no message and no
+trace** outside debug mode (`api2/Controller/Index.php` ~L271-285) because `getTrace()` would leak
+`_Database::register()`'s frame arguments. So an alert that says nothing useful is working as
+designed — resolve the reference instead:
+
+```sql
+# 1. error.id lives in the logged response envelope
+SELECT id, route, method, dtStamp, responsePayload
+FROM Logs.Api
+WHERE id IN (<ids from the alert>);
+
+# 2. the exception + full framework stack (reference = the part BEFORE the dash)
+SELECT id, reference, status, errorMessage, trace, totalOccurrences, dtLastOccurred
+FROM Logs.Issue
+WHERE reference = '78';
+
+# 3. the specific occurrence, and WHICH TENANT it was
+SELECT clientId, dtCreated, errorMessage
+FROM Logs.Event
+WHERE issueId = <issue.id> AND eventNumber = <n>;
+```
+
+- `Issue.trace` is the full stack — the thing the response deliberately omits.
+- `Event.clientId` is the authoritative tenant, not whatever a monitor guessed from headers.
+- **`Issue.status = RESOLVED` does NOT mean fixed** — it means *stopped firing*. A live defect on a
+  tenant that has gone quiet auto-resolves; check `dtLastOccurred` against the tenant's traffic
+  before believing it (see
+  [error-issue auto-resolution](../../worker2/features/error-issue-auto-resolution.md)).
+- The **1.0 alert surface** that fires on these is
+  [App_SystemMonitor_500Error](../../../../1.0/apps/library/features/http-500-error-monitor.md) — its
+  "Error Type" is a fallback literal with **zero** diagnostic value, which is why this recipe
+  exists.
+
 ### Shutdown handler
 
 `register_shutdown_function` is registered with a **pre-allocated memory reserve**. Before this,
@@ -580,6 +617,14 @@ clientUserId). **Neither was built.** As built instead:
 
 ## Change history
 
+- 2026-08-10 — Added the **`error.id` lookup recipe** (`Logs.Api.responsePayload` → `Logs.Issue`
+  by `reference` → `Logs.Event` by `eventNumber` for the tenant), because in production `error.id`
+  is the **only** handle an api2 500 gives you — `message`/`trace` are debug-mode-gated so the
+  `Logs.Issue.trace` row is the only stack. Recorded that `Issue.status = RESOLVED` means *stopped
+  firing*, not *fixed* (a live defect auto-resolves once its tenant goes quiet — a prod GroWrk
+  defect read RESOLVED with `dtLastOccurred` 4 days stale while still broken), and linked the 1.0
+  alert surface [App_SystemMonitor_500Error](../../../../1.0/apps/library/features/http-500-error-monitor.md),
+  whose "Error Type" is a fallback literal with no diagnostic value. (bala)
 - 2026-08-06 — Added two debugging/security notes found while unjamming the Quad PO/ASN importer:
   (1) a `varchar(255)`-truncated `errorMessage` severs a dumped api2 envelope exactly at
   `status`/`error.code`/`messages[]`, so escalate to the **client-scoped** `Logs_<Client>.Api`

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-08-04
+updated: 2026-08-10
 owners: ["mhammontree", "dfranks"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -106,6 +106,28 @@ fact AIG **does** post live entitlements — the writes were **failing (HTTP 500
 core `Logs`**, not `Logs_Aig`. See
 [AIG entitlement intake → Intake is a live API feed](../../../../clients/aig/features/entitlement-intake.md).
 
+## Querying prod `Logs.Api` without timing out (narrow first, fetch payloads last)
+
+Prod `Logs.Api` is large enough that **any `LIKE` over `requestPayload` / `responsePayload`
+(both `mediumtext`) times out the `toga_query` MCP call at 300 s — even scoped to a single day.**
+There is no usable index on those columns. Do not start from a payload search.
+
+The working pattern, in order:
+
+1. **Narrow with indexed / non-text columns only** — a `dtStamp` range plus `route`,
+   `method`, `responseCode`, `hostname`. A
+   `GROUP BY hostname, route, method, responseCode` over one day is cheap and is the fastest way
+   to spot a failing route (this is what surfaced the AIG entitlement 500s).
+2. **Collect the `id`s** from that narrowed set.
+3. **Then fetch payloads by `id`**, a few rows at a time, using `LEFT(...)` / `SUBSTRING(...)`
+   rather than selecting the whole `mediumtext`.
+
+Volume reference for sizing a query: **~8,151 rows in `Logs.Api` for a single day** (2026-05-27).
+
+Note this is in tension with the payload-based techniques in the next section — the
+`JSON_EXTRACT` dedupe/vendor-split works because it runs over an **already-narrowed** row set
+(one route, one response code, one date range), not over the raw table.
+
 ## Recovering an intake backlog from logged payloads
 
 Because the core/writer `Logs.Api` stores **every** inbound request — including failed writes —
@@ -134,6 +156,11 @@ re-send.
 
 ## Change history
 
+- 2026-08-10 — Added **Querying prod `Logs.Api` without timing out**: any `LIKE` over the
+  `mediumtext` `requestPayload`/`responsePayload` times out the `toga_query` MCP call at 300 s even
+  for one day. Narrow first on indexed/non-text columns (`dtStamp` + `route` + `responseCode`, or a
+  `GROUP BY hostname, route, method, responseCode` to spot the failing route), collect `id`s, then
+  fetch payloads by `id` with `LEFT()`/`SUBSTRING()`. Volume reference ~8,151 rows/day. (mhammontree)
 - 2026-08-04 — Added the limit of the commit-logs contract: an **uncaught PHP fatal** (a
   `Call to undefined method` from an interceptor dispatch) produced **zero rows in
   `Logs_Compass.Api`** — no 5xx row for the whole day and no row for the failing request — while

package/knowledge/2.0/apps/api2/features/v2-api-error-codes.md

@@ -7,7 +7,7 @@ client: shared
 type: feature
 status: active
 updated: 2026-08-10
-owners: [mhammontree, tcox, jcardinal, ajean]
+owners: [mhammontree, tcox, jcardinal, ajean, bala]
 files:
   - api2/Component/Api/V2/V2.php
   - api2/Component/Api/V2/Response/Response.php
@@ -19,6 +19,8 @@ files:
 related:
   - ../../_underscore/features/error-reporting-issue-event.md
   - ../../_underscore/features/acl-permission-chain.md
+  - ../../dbchanges2/workflows/client-schema-drift-audit.md
+  - ../../../../1.0/apps/library/features/http-500-error-monitor.md
   - scripted-api-post-body-args.md
   - v2-rest-query-contract.md
   - ../architecture.md
@@ -43,6 +45,7 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
 | **EV-9** | No permission to write field | The field exists but the caller's role has no **`AclFieldPermissions`** grant (`isWritable=1`) in the CLIENT DB | Add the `AclFieldPermissions` row for the role (clone a writable sibling field's grant) |
 | **EO-1** | Operation failed — identifier "There is no field called 'X' in the '_Model_Client_Y' model" | The DB column **and** `Core.RecordFields` exist, but the **generated model class** `_underscore/Model/Client/<Name>.php` doesn't declare the field. This is the **4th** requirement beyond the 3-file migration — and most often it's a **cross-repo git branch mismatch** (`_underscore` on a branch whose generated model lacks a field the DB/RecordFields already carry) | Declare the field in the generated model class (`public $field = self::FIELD_*`) and put all related repos (`_underscore`, `api2`, `dbchanges2`, `toga2-supply`) on the **same** feature branch — see the ACL doc's writable-field recipe |
 | **EO-1** | Operation failed — **surfaced from a PHP warning/notice, not a real op error** (e.g. "Attempt to read property 'id' on bool", undefined variable) | A latent PHP warning escalates to a 500 because **Sentry's `ErrorHandler` in api2 promotes warnings/notices into thrown exceptions** (see diagnosis note 5). The known instance: in `V2.php::processRoutePairs()` an **unresolved route** leaves the local `$record = false`, and the post-processing payload interceptor layer then dereferenced `$record->id` → warning → 500 | Guard before dereferencing an unresolved record. The fix added a guard clause `if (!$record) return [$rawRequestedRouteName => $outData];` **before** the interceptor/logging layer (so a bad route returns a clean envelope, not a fatal), and initializes `$record = false;` at the top of the `foreach ($lookupByRouteNames ...)` loop so the invalid-HTTP-method / null-`$action` branch can't leave `$record` undefined (an undefined-variable warning would itself escalate to a 500) |
+| **EO-1** | Operation failed — MySQL **1054 "Unknown column 'X' in 'field list'"** | The **exact reverse** of the row above: the generated model (or a **trait** it `use`s) *declares* the field but the client table has **no such column**. `_Model` builds its SELECT from **every declared property**, so this fires on **any** load of that record on **any** route — including when the caller never asked for that field. Usual cause: a client DB whose module migration only **partially** applied | Add the missing column to that client DB — plus its `CustomRecordFields` row and `AclCustomFieldPermissions` grant if it is a `c_` field. Audit the rest of the drift first: [client schema-drift audit](../../dbchanges2/workflows/client-schema-drift-audit.md) |
 | **EZ-1** | Unauthorized record/script dispatch | Missing **`AclRecordScripts`** (scripted APIs) or the **`AclRecordPermissions`** four-table chain (records) for the caller's role | Grant `AclRecordScripts` (scripts) or complete the record-CRUD chain |
 | **EZ-2** | Field-level authorization denied (READ) | The field is registered (`Core.RecordFields` present → no `EV-8`) but the caller's role has no **`AclFieldPermissions`** grant to **read** it. The read-side counterpart of `EV-9` (write). **This applies to the `id` field too:** fetching a record by its numeric `id` (`GET /v2/<record>?fields=id,...`) 403s `EZ-2` if `id` has no read grant — the `id` field is ACL-gated like any other, not implicitly readable | Add the `AclFieldPermissions` row for the role (`isWritable=0` if the field is server-written). For a **custom** `c_` field use `AclCustomFieldPermissions` instead — see the ACL doc's standard-vs-custom table |
 | **EV-5** | Duplicate `transactionId` | The globally-unique `transactionId` was reused | Send a fresh unique `transactionId` per request |
@@ -137,6 +140,24 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
     a payload server-side will **create a duplicate** rather than match the existing record.
     When replaying or backfilling, rewrite every nested reference to use `uuid`.
 
+12. **Why a production 500 can only be diagnosed through note 9 — and what NOT to conclude.**
+    `message` and `trace` are gated behind `_Environment::isDebugMode()` (see the security fix
+    below), so a production envelope carries `error.code` + `error.id` and nothing else. Two
+    consequences worth knowing before you start:
+    - The 1.0 [HTTP 500 monitor](../../../../1.0/apps/library/features/http-500-error-monitor.md)
+      alert's "Error Type" is therefore **always** *"No message found in response payload"* — the
+      monitor's own fallback literal, identical for every unrelated production 500. It is not a
+      diagnosis and two alerts sharing it are not the same bug.
+    - `error.id` is `<Issue.reference>-<Event.eventNumber>`: note 9 resolves the `Issue` for the
+      stack; the **`Event`** half also gives you `Logs.Event.clientId` — the authoritative **tenant**
+      (full recipe:
+      [error reporting](../../_underscore/features/error-reporting-issue-event.md)).
+
+    **Never conclude "timeout" / "deep traversal" / "memory" from the request shape.** Read the
+    logged `executionTime` on the `Logs.Api` rows first: 0.19-2.3s 500s on a `depth=6` request were a
+    **missing DB column**. And several near-identical 500s sharing **one `transactionId`** are
+    apiproxy's cross-region retry, not repeated user impact.
+
 ## BREAKING (2026-07-30): `error` is now an object, not a bare string
 
 The V2 envelope's `error` changed from a bare string (`"EO-1"`) to an **object**:
@@ -222,6 +243,17 @@ Full mechanics:
 
 ## Change history
 
+- 2026-08-10 — Added a **third `EO-1` case — MySQL 1054 "Unknown column"**: the reverse of the
+  known model-declaration gap. When the model (or a **trait** it `use`s) declares a field the client
+  table lacks, `_Model` still SELECTs it — it builds its SELECT from **every declared property**,
+  ignoring the caller's `fields=` — so **every** load of that record 500s on **every** route. Root
+  cause in the observed case was a **partially applied** module migration on one client DB
+  (`Client_Growrk.Units` missing `c_lastModified` from `_Trait_Netsuite_Unit`). Added diagnosis note
+  12: in production the envelope carries only `error.code` + `error.id`, so the 1.0 HTTP 500 monitor
+  alert always reads *"No message found in response payload"* (its own fallback literal) — resolve
+  `error.id` against `Logs.Issue`/`Logs.Event` instead (note 9's lookup, plus `Logs.Event.clientId`
+  for the tenant), and check the logged `executionTime` before believing a timeout theory (a
+  0.19-2.3s 500 on `depth=6` was a missing column). (bala)
 - 2026-08-10 — TRUE-80575: added diagnosis notes 9-11 from debugging a Rate entitlement
   replay — (9) `error.id`'s prefix before the dash **is** `Logs.Issue.reference`, so an `EO-1`
   500 becomes a real stack trace via `SELECT * FROM Logs.Issue WHERE reference = '…'`, and

package/knowledge/2.0/apps/api2/features/v2-rest-query-contract.md

@@ -6,8 +6,8 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-07-29
-owners: [tcox]
+updated: 2026-08-10
+owners: [tcox, bala]
 files:
   - api2/Component/Api/V2/V2.php
 related:
@@ -114,6 +114,13 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
 
 ## Gotchas / known issues
 
+- **⚠ `fields=` narrows the RESPONSE, never the SQL SELECT.** `_Model` builds its SELECT from
+  **every property declared on the model class and on the traits it `use`s**, so a column that is
+  declared but physically missing from that client's table raises MySQL **1054 → HTTP 500 (`EO-1`)**
+  even when the caller asked only for one unrelated field. Two consequences: (a) a 500 naming a field
+  you never requested is expected, not a red herring; (b) **every** route whose traversal reaches that
+  record fails identically, so narrowing `fields=` cannot work around it. The fix is schema-side —
+  see [client schema-drift audit](../../dbchanges2/workflows/client-schema-drift-audit.md).
 - **A 404 on a route you know exists ⇒ suspect an unrecognized query param** (LIST → READ flip)
   before suspecting the route.
 - **A 500 (`EO-1`) on a filter ⇒ suspect a `where` reference to an unjoined table.**
@@ -122,6 +129,11 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
   invisible; re-request with `fields=` to force `EZ-2` and see the denied list.
 
 ## Change history
+- 2026-08-10 — Recorded that **`fields=` narrows the response but not the SQL SELECT**: `_Model`
+  selects every property declared on the class **and its traits**, so a declared-but-missing column
+  500s (`EO-1`, MySQL 1054) any request touching that record — including one that named only an
+  unrelated field — and no `fields=` list can avoid it. Found via the production `Client_Growrk.Units`
+  `c_lastModified` outage. (bala)
 - 2026-07-29 — Created from a source read of `api2/Component/Api/V2/V2.php` while building the
   BDR funnel's call-status reads. Recorded: routes-as-data (`Core.Records`, kebab → camel
   envelope key); the recognized-param list and the **unrecognized-param LIST→READ flip** that

package/knowledge/2.0/apps/dbchanges2/INDEX.md

@@ -5,5 +5,6 @@
 | [Database Changes (dbchanges2) Repository Architecture](architecture.md) | `dbchanges2` is the **schema-migration / SQL change-set repository** for the entire 2.0 platform. | Core/, Client/, Client_<Tenant>/, Logs/, Logs_Client/, _modules/ |
 | [Surface Layer Schema (UI presentation/config tables)](features/surface-layer-schema.md) | The persistent schema for the platform-wide **Surface** UI presentation/configuration layer (see the `_underscore` [surface-resolver](../../_underscore/features | dbchanges2/Client/2026-06-25a - SurfaceClientTables.sql, dbchanges2/Client_Compass/2026-06-25d - SalesOrderSurfaceClientSeed.sql, _underscore/Model/Client/ThemeToken.php, toga25-supply/src/themeConfig.json, dbchanges2/Core/2026-06-25a - SurfaceCoreTables.sql, dbchanges2/Core/2026-06-25b - SurfaceRecordsAndFields.sql, dbchanges2/Core/2026-06-25c - SalesOrderLoginSurfaceSeed.sql, dbchanges2/Core/2026-06-29a - ItemsSurfaceSeed.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Core/2026-06-29d - VendorItemsSurfaceSeed.sql, dbchanges2/Core/2026-06-29e - InventorySurfaceSeed.sql, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Compass/2026-06-30a - SalesOrderDisplaySectionManagerOverrides.sql, dbchanges2/Client_CompassCanada/2026-06-30a - SalesOrderSurfaceManagerOverrides.sql, dbchanges2/Client_Quad/2026-06-30a - SalesOrderSurfaceClientOverrides.sql, dbchanges2/Client/2026-06-25a - SurfaceClientTables.sql, dbchanges2/Client/2026-06-25b - SurfaceClientSeed.sql, dbchanges2/Client/2026-06-25c - SurfaceClientAcl.sql, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Core/2026-07-20a - Update - HideAdminNotesSectionByDefault.sql, dbchanges2/Core/2026-07-20b - Update - NotesSectionFieldElements.sql, dbchanges2/Client_Compass/2026-07-20a - AdminNotesSectionVisibilityOverride.sql, dbchanges2/Client_Compass/2026-07-20b - NotesSectionFieldsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-20a - AdminNotesSectionVisibilityOverride.sql, dbchanges2/Client_CompassCanada/2026-07-20b - NotesSectionFieldsOverride.sql, dbchanges2/Client_Quad/2026-07-20a - NotesSectionFieldsOverride.sql, dbchanges2/Core/2026-07-20c - Update - VendorItemsToggleSurfaceSeed.sql, dbchanges2/Client_Compass/2026-07-20c - ItemRecordEditButtonEnable.sql, dbchanges2/Client_Compass/2026-07-20d - ItemRecordVendorItemsEnable.sql, dbchanges2/Client_CompassCanada/2026-07-20c - ItemRecordEditButtonEnable.sql, dbchanges2/Client_CompassCanada/2026-07-20d - ItemRecordVendorItemsEnable.sql, dbchanges2/Core/2026-07-20e - RestoreApproveDenyRowActions.sql, dbchanges2/Client_Compass/2026-07-20e - RowActionsApprovalWorkflowAdminEnable.sql, dbchanges2/Client_CompassCanada/2026-07-20e - RowActionsApprovalWorkflowAdminEnable.sql, dbchanges2/Core/2026-07-17 - README - RUN ORDER.md, dbchanges2/Client_Compass/2026-07-21a - SalesOrderApproveEnabledRuleOverride.sql, dbchanges2/Client_Compass/2026-07-21b - SalesOrderApprovalsGateEnable.sql, dbchanges2/Client_CompassCanada/2026-07-21a - SalesOrderApproveEnabledRuleOverride.sql, dbchanges2/Client_CompassCanada/2026-07-21b - SalesOrderApprovalsGateEnable.sql, dbchanges2/Client_Quad/2026-07-21a - SalesOrderApproveEnabledRuleOverride.sql, dbchanges2/Client_Quad/2026-07-21b - SalesOrderApproveDisabledTooltipTranslation.sql, dbchanges2/Core/2026-07-21a - SalesOrderDecisionSummarySurfaceSeed.sql, dbchanges2/Core/2026-07-21b - SalesOrderDecisionActionSurfaceSeed.sql, dbchanges2/Client_Quad/2026-07-21c - SalesOrderDecisionSummaryOverride.sql, dbchanges2/Client_Compass/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_CompassCanada/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Core/2026-07-23a - PoNumberDetailFieldValueKey.sql |
 | [2.0 New-Client Onboarding (manual process)](workflows/client-onboarding.md) | > **A local browser wizard now automates this.** Steps 2–9 below (create DBs, generate Core/API > inserts, append to `Clients_Db.txt`) — plus the dbchanges2 bla | Client/, Client_<Tenant>/, Core/, Logs_Client/ |
+| [Auditing a client DB that drifted from its models (partially applied module migration)](workflows/client-schema-drift-audit.md) | A recurring 2.0 failure mode: **one client's database drifts from what the PHP models declare**, usually because a `_modules/<module>/` migration was applied to | dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql, dbchanges2/Client_Growrk/_modules.txt |
 | [Local vs prod MySQL config parity — why “it passed locally” is not evidence](workflows/local-vs-prod-mysql-config-parity.md) | Several migration failures that look like "prod-only bugs" are actually **per-machine MySQL server-configuration differences**. |  |
 | [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini |

package/knowledge/2.0/apps/dbchanges2/workflows/client-schema-drift-audit.md

@@ -0,0 +1,135 @@
+---
+title: Auditing a client DB that drifted from its models (partially applied module migration)
+framework: "2.0"
+repo: dbchanges2
+project: Database Changes
+client: shared
+type: workflow
+status: active
+updated: 2026-08-10
+owners: ["bala"]
+files:
+  - dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql
+  - dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql
+  - dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql
+  - dbchanges2/Client_Growrk/_modules.txt
+related:
+  - ../architecture.md
+  - ./nonprod-metadata-drift-repair.md
+  - ../../_underscore/features/acl-permission-chain.md
+  - ../../api2/features/v2-api-error-codes.md
+  - ../../../../clients/growrk/features/units-netsuite-custom-fields.md
+---
+
+## Summary
+
+A recurring 2.0 failure mode: **one client's database drifts from what the PHP models declare**,
+usually because a `_modules/<module>/` migration was applied to that client only **partially**.
+The `_Model` layer then SELECTs a column that does not exist and the API 500s on every route that
+touches the record.
+
+> **This is the mirror image of
+> [non-prod metadata drift](./nonprod-metadata-drift-repair.md).** There the environment is behind
+> (works in prod, broken in beta). Here **production is the outlier** and beta/dev-sandbox is
+> correct — so "it works on beta" is not evidence the schema is fine. Check *which* side is behind
+> before assuming.
+
+Written from repairing `Client_Growrk.Units` (missing `c_lastModified`, 6 of 7 columns and 1 of 6
+metadata rows applied) — see
+[GroWrk Units custom fields](../../../../clients/growrk/features/units-netsuite-custom-fields.md).
+
+## Why it 500s: `_Model` ignores the caller's `fields=`
+
+`_Model` builds its SELECT from **every declared property on the class and its traits**, not from
+the fields the API request asked for. So a single missing column fails **any** traversal that
+reaches that record, on **any** route, even when the request only asked for one unrelated field.
+MySQL raises **1054 "Unknown column 'X' in 'field list'"**, api2 answers **HTTP 500 / `EO-1`**.
+
+Corollary: **the field named in the error has nothing to do with the field the request wanted.**
+Do not narrow the investigation to the requested field list.
+
+## The three parts of a 2.0 custom (`c_`) field — each fails differently
+
+| Missing piece | Symptom |
+|---|---|
+| the **physical column** on the client table | MySQL **1054** → **HTTP 500 / `EO-1`** on every load of the record |
+| the **`CustomRecordFields`** row (client DB) | the field is **not requestable** through the v2 API at all |
+| the **`AclCustomFieldPermissions`** grant | **`EZ-2` / HTTP 403 for the WHOLE request** — one unreadable field fails the entire GET; it is *not* silently omitted when named in `fields=` |
+
+(The declaration side — the `c_` property on the client model or a trait it `use`s — is the fourth
+part, and it is what makes a missing column fatal. Standard, non-`c_` fields use
+`Core.RecordFields` + `AclFieldPermissions` instead; see
+[ACL permission chain](../../_underscore/features/acl-permission-chain.md).)
+
+## Steps
+
+1. **Get the real exception first.** Resolve `error.id` from the response envelope against
+   `Logs.Issue` / `Logs.Event` — the stack tells you which model's `load()` failed and which column
+   MySQL rejected. Recipe:
+   [error reporting](../../_underscore/features/error-reporting-issue-event.md). Also read
+   `Logs.Api.executionTime`: sub-second 500s **disprove** the timeout / deep-traversal / memory
+   theories a high `depth=` invites.
+2. **Find the client's module membership** — `dbchanges2/Client_<Name>/_modules.txt` (one module per
+   line; e.g. Growrk = `netsuite`). That tells you which `_modules/<module>/` change-sets *should*
+   have been applied to it.
+3. **Diff `information_schema.COLUMNS` against a HEALTHY client on the SAME module**, scoped to the
+   tables in the failing request's traversal path. This is the cheap, complete check — it answers
+   "is there a second missing column?" in one query instead of guessing field by field.
+   `Client_Aig` served as the healthy `netsuite` reference for `Client_Growrk`; the diff returned
+   exactly one row before the fix and zero after.
+4. **Corroborate from the code side.** Extract every declared **stored** field from the client's
+   model classes (`_Model_Client_*` and `_Model_<Client>_*`) **plus every trait they `use`**, and
+   test each against `information_schema`. **Exclude `FIELD_SQL` and underscore-prefixed calculated
+   fields — those are computed, not columns**, and treating them as missing columns produces false
+   positives.
+5. **Check the metadata rows, not just the columns.** For the record in question, count
+   `CustomRecordFields` rows and their `AclCustomFieldPermissions` grants, and compare the counts
+   with the healthy client. A partial run leaves an arbitrary subset of both.
+6. **Confirm which environment is the outlier** by running the same audit against dev-sandbox
+   (beta). If beta already has the correct state, the missing work is a **production-only** catch-up
+   — do not re-run the whole module change-set.
+7. **Write the catch-up migration in `dbchanges2/Client_<Name>/`** (naming contract:
+   `YYYY-MM-DD<letter> - <Description>.sql`, letter mandatory). Two rules:
+   - **Reuse the module migration's fixed UUIDs** for the metadata rows it failed to insert, so the
+     repaired client ends up **identical** to every other client on that module.
+   - **Grant ACL for the rows that already existed too.** An insert-then-grant migration typically
+     grants only what *it* inserted, so any `CustomRecordFields` row that survived the partial run
+     keeps its missing grant and needs its own follow-up file.
+8. **Verify the end state explicitly:** every custom field on that `recordId` has **exactly one**
+   ACL grant, and the healthy-client column diff across all tables in the request path is **empty**.
+   Then re-drive the failing request.
+
+## Gotchas / known issues
+
+- **Partial means arbitrary — never assume all-or-nothing.** The observed run landed **6 of 7**
+  columns and **1 of 6** metadata rows from one file. "The migration ran" is not a fact about any
+  individual statement in it.
+- **A `CustomRecordFields` row with no ACL grant is not automatically a defect.** Compare with the
+  healthy client before "fixing" it — `Client_Growrk` recordId 312
+  (`c_netsuiteInternalSalesOrderId`, `TransferOrders`) has no grant, and `Client_Aig` is identical,
+  so that is the normal cross-client state. Repairing it would be a change, not a fix.
+- **A client can be fine for months and then break on onboarding.** Before a client joins the
+  module, its records resolve to the plain `_Model_Client_*` class with none of the `c_` fields, so
+  the missing columns are unreachable. The 500s begin the day the client is switched onto the
+  module-specific model — which is why the "it has always worked" evidence is misleading. Prove it
+  with `Logs.Api`: successful 200s for the exact same request shape right up to the onboarding
+  timestamp, 500s immediately after.
+- **`ALTER TABLE … ADD COLUMN` was INSTANT here** (MySQL 8.0.39, ~5.4k rows) — a small tenant table
+  needs no maintenance window, but confirm the server version and row count rather than assuming.
+- **Re-running the whole module change-set is the wrong repair** when only one environment drifted:
+  the other statements are already applied and some are not idempotent. Write a scoped catch-up
+  file instead.
+
+## Change history
+
+- 2026-08-10 — Created from the production `Client_Growrk.Units` incident: `_Trait_Netsuite_Unit`
+  declared `c_lastModified` but the column had never been added, so **every** traversal reaching a
+  Growrk Unit 500'd (`EO-1`, MySQL 1054) regardless of the requested `fields=`. Recorded the
+  audit playbook (diff `information_schema.COLUMNS` against a healthy same-module client scoped to
+  the failing traversal path; corroborate from the model + trait declarations, excluding `FIELD_SQL`
+  and underscore-prefixed calculated fields), the **three-part anatomy** of a 2.0 custom field and
+  how each part fails differently, and the catch-up-migration rules (reuse the module's fixed UUIDs;
+  grant ACL for pre-existing rows too). Noted that this is the **inverse** of non-prod metadata
+  drift — here prod was the outlier and beta was already correct — and that a `CustomRecordFields`
+  row with no ACL grant can be the normal cross-client state (recordId 312, identical in
+  `Client_Aig`). (bala)

package/knowledge/2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md

@@ -6,8 +6,8 @@ project: Database Changes
 client: shared
 type: workflow
 status: active
-updated: 2026-08-03
-owners: ["mhammontree"]
+updated: 2026-08-10
+owners: ["mhammontree", "bala"]
 files:
   - dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql
   - dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql
@@ -15,6 +15,7 @@ files:
   - api2/Config/beta.ini
 related:
   - ../architecture.md
+  - ./client-schema-drift-audit.md
   - ../../api2/features/api-payload-interceptors.md
   - ../../api2/workflows/environment-configuration-and-provisioning.md
   - ../../_underscore/features/acl-permission-chain.md
@@ -32,6 +33,10 @@ environments receive those rows by someone remembering to run a migration there,
 This workflow is the repair procedure, written from bringing dev-sandbox up to date so beta could
 exercise TOGa Supply Fulfill & Ship (TRUE-80494).
 
+> **Wrong doc?** When the drift runs the other way — **production** is the outlier and beta is
+> correct, or one **client's** DB is behind the models (a partially applied `_modules/` migration) —
+> use [client schema-drift audit](./client-schema-drift-audit.md) instead.
+
 ## Steps
 
 1. **Identify which cluster the environment actually reads.** `api.beta.togahub.com` reads the
@@ -140,6 +145,10 @@ blocked by the id-divergence hazard in item 2 above, which is why both go to jca
 
 ## Change history
 
+- 2026-08-10 — Added a disambiguation pointer to the new
+  [client schema-drift audit](./client-schema-drift-audit.md) for the **inverse** case (production or
+  a single client's DB is the outlier, e.g. a partially applied `_modules/` migration). No change to
+  this procedure. (bala)
 - 2026-08-03 — TRUE-80494: created from repairing dev-sandbox so beta could exercise Fulfill & Ship —
   completed `Client_Growrk` measure grants (2476/2477/2478, role 1), applied the missing
   `ApiPayloadInterceptors` PRE/POST row for recordId 28, registered

package/knowledge/2.0/apps/worker2/features/error-issue-auto-resolution.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-05
-owners: ["jcardinal"]
+updated: 2026-08-10
+owners: ["jcardinal", "bala"]
 files:
   - worker2/Worker/Infrastructure/Errors.php
   - worker2/Worker/Clickup/ErrorTask.php
@@ -171,6 +171,13 @@ console's own conventions.)
 
 ## Gotchas / known issues
 
+- **⚠ For a DEBUGGER, `status = RESOLVED` means "stopped firing" — NOT "fixed".** Resolution keys on
+  quiet time alone, so a still-broken defect on a low-traffic tenant auto-resolves as soon as that
+  tenant goes quiet. Observed 2026-08-10: a production GroWrk 500 (missing DB column, entirely
+  unfixed) presented as a RESOLVED Issue with `dtLastOccurred` **4 days** stale, purely because
+  GroWrk traffic had stopped. **Never close an investigation on Issue status** — read
+  `dtLastOccurred` against that tenant's traffic, and confirm the root cause is gone in the code or
+  schema.
 - **Only the frequency-decay cron may write `Issue.status = RESOLVED`.** The ClickUp webhook
   (`ErrorTask.php`) closes episodes and resets counters but must never touch `Issue.status`.
 - **`autoResolveIssue` must stay DB-first.** Closing the episode and committing `RESOLVED` before
@@ -190,6 +197,10 @@ console's own conventions.)
 
 ## Change history
 
+- 2026-08-10 — Added the reader-facing warning that **RESOLVED means "stopped firing", not
+  "fixed"** — a live production GroWrk 500 (missing DB column) read as RESOLVED with
+  `dtLastOccurred` 4 days stale simply because that tenant's traffic had gone quiet. No behaviour
+  change; this is a diagnosis trap for anyone reading Issue status during an incident. (bala)
 - 2026-08-05 — **Built:** Issue-level auto-resolution and reopen by frequency decay. New
   `Issue.status enum('OPEN','RESOLVED')` + `dtAutoResolved`, a durable cron-maintained firing
   baseline (`baselineGapSeconds` EWMA + `baselineSampleCount`/`dtBaselineAnchor`/

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 15 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 16 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 21 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -18,10 +18,10 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 47 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 48 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 42 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 22 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
-- **dbchanges2** (Database Changes) _(framework core)_ — 7 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
+- **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 5 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
 - **saml** (SAML SSO Gateway) — 3 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
 - **toga2-view** (TOGa View Frontend) — 7 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)

package/knowledge/clients/aig/INDEX.md

@@ -3,5 +3,5 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [AIG Contract Reconciliation & Dealer Programs (STS_001 / SA_001)](features/contract-reconciliation.md) | 2.0 | How to reconcile an **AIG contract sales sheet** against the `Client_Aig` tenant, and the durable finding that the **Staples Advantage.com (`SA_001`) program do | _underscore/Model/Client/Entitlement.php, _underscore/Model/Aig/Entitlement.php, _underscore/Model/Client/Contact.php, _underscore/Model/Aig/Contact.php |
-| [AIG Entitlement Intake & SaleItem Code Resolution](features/entitlement-intake.md) | 2.0 | The `_Model_Aig_Entitlement::prePost` interceptor below is the code path for AIG protection-plan entitlements arriving as `api2` V2 JSON POSTs — and that path * | _underscore/Model/Aig/Entitlement.php, dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql |
+| [AIG Entitlement Intake & SaleItem Code Resolution](features/entitlement-intake.md) | 2.0 | The `_Model_Aig_Entitlement::prePost` interceptor below is the code path for AIG protection-plan entitlements arriving as `api2` V2 JSON POSTs — and that path * | _underscore/Model/Aig/Entitlement.php, api2/Component/Api/V2/V2.php, dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql |
 | [AIG (Staples Protection Plan)](profile.md) | 2.0 | AIG is the warranty underwriter behind the **Staples Protection Plan** retail program. |  |

package/knowledge/clients/aig/features/contract-reconciliation.md

@@ -5,7 +5,7 @@ project: API
 client: aig
 type: client-feature
 status: active
-updated: 2026-07-29
+updated: 2026-08-10
 owners: ["mhammontree"]
 files:
   - _underscore/Model/Client/Entitlement.php
@@ -27,7 +27,7 @@ durable finding that the **Staples Advantage.com (`SA_001`) program does not rou
 | DealerId | Program | Status |
 |---|---|---|
 | `STS_001` | **Staples Technology Solutions** | live api2 feed; intake gap **RESOLVED 2026-07-29** (self-healed) |
-| `SA_001` | **Staples Advantage.com** | **never reaches api2** — separate feed; scope decision is the one open item |
+| `SA_001` | **Staples Advantage.com** | **never reaches api2** — SA.com is not sending them; **AIG owns and is working it**, not a TOGa defect (2026-08-10) |
 
 `STS_001` arrives as a **live API feed**; its 156-contract gap from the now-fixed intake bug
 **self-healed** (the feed re-sent and all 156 succeeded — see below). `SA_001` never reaches our
@@ -106,6 +106,28 @@ because their `SA_001` contract was never ingested.
 into `Client_Aig`, and if so via what feed/integration? That is a scope/integration decision for
 AIG / the team — not a back-fill of a dropped load. Until decided, `SA_001` stays out-of-pipeline.
 
+#### CLOSED (2026-08-10): missing `SA_001` entitlements are AIG's issue, not a TOGa defect
+
+**SA.com is not sending the new entitlements to us. AIG owns this and is working it.** Stop
+working "missing SA_001 entitlement" escalations as a TOGa bug — the correct response is to route
+them to AIG.
+
+Confirmed on a concrete case from AIG's own Warranty Administration Portal: contract
+**CN 1000046237886 / CCN 7680760752-2** (Staples Advantage.com program, plan *"2YR Printer Onsite
+Next Day: ($500-$999.99)"*, purchased 2026-05-27) exists in AIG's portal and is **completely absent
+on our side**:
+
+- zero rows in `Client_Aig.Entitlements` — searched by `number`, `vendorEntitlementNumber`,
+  `datePurchase`, **and** contract-holder name;
+- zero rows in `Client_Aig.Contacts`;
+- **zero trace in `Logs_Aig.Api` or the core/writer `Logs.Api`.**
+
+No log row anywhere means the contract was **never delivered** — nothing was rejected, nothing was
+partially created, and there is no dropped payload to replay. That is the distinguishing signal:
+a TOGa-side intake defect always leaves a failed-write row in the core `Logs` (see
+[request logging](../../../2.0/apps/api2/features/request-logging.md)); **total absence points
+upstream.**
+
 ## How to load a missing program / sheet (back-fill)
 
 If `SA_001` is ever confirmed in-scope for `Client_Aig` and AIG cannot route it through api2, it
@@ -118,6 +140,13 @@ feed re-send over a manual load whenever the contracts do reach api2.
 
 ## Change history
 
+- 2026-08-10 — Read-only prod investigation. **Closed the `SA_001` question as NOT a TOGa defect:**
+  SA.com is not sending the entitlements; **AIG owns and is working it.** Evidenced by portal
+  contract CN 1000046237886 / CCN 7680760752-2 (purchased 2026-05-27) being absent from
+  `Client_Aig.Entitlements` *and* `Contacts` *and* from both `Logs_Aig.Api` and core `Logs.Api` —
+  never delivered, so nothing was rejected or partially created. Recorded the general
+  discriminator: an intake defect leaves a failed-write row in core `Logs`; **total log absence
+  points upstream.** Future "missing SA_001 entitlement" escalations route to AIG. (mhammontree)
 - 2026-07-29 — TRUE-80179 / TRUE-79534: **156 active `STS_001` RESOLVED (self-healed).** The next
   scheduled batch re-sent all 156 outstanding contracts and every one succeeded (201 × 156, zero
   failures; `Client_Aig.Entitlements` +156) — no manual back-fill needed, because the feed re-sends

package/knowledge/clients/aig/features/entitlement-intake.md

@@ -5,10 +5,11 @@ project: API
 client: aig
 type: client-feature
 status: active
-updated: 2026-08-03
+updated: 2026-08-10
 owners: ["mhammontree"]
 files:
   - _underscore/Model/Aig/Entitlement.php
+  - api2/Component/Api/V2/V2.php
   - dbchanges2/Client_Aig/2026-06-18a - TRUE-79534 AIG SaleItem codes.sql
 related:
   - clients/aig/features/contract-reconciliation.md
@@ -122,6 +123,32 @@ Loading the catalog fixed intake, and because the feed re-sends outstanding cont
 **backlog of dropped contracts self-healed on the next scheduled run** — no manual back-fill was
 needed (see [contract reconciliation](contract-reconciliation.md)).
 
+## Verified prod state (2026-08-10) — the catalog IS loaded; do NOT re-diagnose as "missing item ID"
+
+Re-verified read-only against prod. **The TRUE-79534 catalog load did run and is present**, so a
+1064/500 seen from here on is **not** the missing-catalog problem:
+
+- **`Client_Aig.Items` = 2,212 rows** — 1,110 `ASI-*`, 558 `SM-*`, 37 legacy `SP-*`.
+- The exact part number in the historically-failing payloads, **`ASI-2YG2`, IS present.**
+- **`Client_Aig.Entitlements` = 621 rows**, latest `dtCreated` **2026-08-04**, latest
+  `datePurchase` **2026-07-31** — entitlements are still being created.
+
+So the remaining exposure at `Entitlement.php:48` is the **input-empty** variant (an *empty*
+`itemId` interpolated), not an unknown-code variant. Anyone re-opening this should read the actual
+interpolated value before assuming a missing AIG item ID — the earlier ticket already ruled that
+out.
+
+**Open question (not answered this session):** the historical failure window in the core/writer
+`Logs.Api` shows **exactly 1,200 `POST /v2/entitlements` requests per day, all HTTP 500, zero
+successes on that route**, every day from at least **2026-05-20 through 2026-07-28**, with the
+*same* stale batch replayed daily — log id 996641 (05-27) and id 1877726 (07-28) carry
+**byte-identical** payloads (`number: 1000045980287`), as do ids 998804 / 1881349
+(`number: 1000045972141`). No entitlement POSTs are logged in core `Logs` after 07-28. Because
+**successful** writes log to `Logs_Aig`, not core `Logs`, the most likely reading is that the route
+recovered on 07-29 exactly as the resolution above describes. But the creating path for the newest
+rows (through 2026-08-04) was **not positively identified** — confirm it in `Logs_Aig.Api` before
+asserting either "intake is entirely down" or "intake is healthy."
+
 ## Business Unit (`UserDefined2`) — answered and DROPPED (nothing to build)
 
 AIG (Alissa, 2026-05-04) confirmed the **Business Unit / Value** field needs **nothing returned to
@@ -294,7 +321,17 @@ this interceptor or use this dual-purpose Items pattern.
   `Apis_RecordFields` override row that stripped `name` as an identifier — see
   [api2 nested-relationship writes → Per-API overrides](../../../2.0/apps/api2/features/nested-relationship-writes.md).
 - **A lookup miss yields an empty `$itemId` that crashes the next query (confirmed in prod,
-  TRUE-79534).** When `saleItem.partNumber` is **valid but absent** from `Client_Aig.Items`, the
+  TRUE-79534).** The unguarded line is **`_underscore/Model/Aig/Entitlement.php:48`** in `prePost`,
+  reached from **`api2/Component/Api/V2/V2.php:2601`**. The malformed statement it emits is:
+  ```sql
+  SELECT EntitlementFulfillmentTypes.name
+  FROM Items_EntitlementFulfillmentTypes
+  INNER JOIN EntitlementFulfillmentTypes ON ...
+  WHERE Items_EntitlementFulfillmentTypes.itemId =      -- nothing after the '='
+  ```
+  → MySQL **ERROR 1064** → HTTP 500. The fix is a **guard clause / early return** on an
+  empty-or-failed lookup *before* this query runs. When `saleItem.partNumber` is
+  **valid but absent** from `Client_Aig.Items`, the
   `SELECT id … WHERE partNumber = '{$partNumber}' LIMIT 1` returns nothing → `$itemId` is empty →
   the **fulfillment query interpolates it unquoted** → `WHERE …itemId = ` → **ERROR 1064** → HTTP
   500 (the mechanism behind the whole intake outage). A separate **input-empty** variant also
@@ -320,6 +357,16 @@ this interceptor or use this dual-purpose Items pattern.
 
 ## Change history
 
+- 2026-08-10 — Read-only prod investigation (no code/SQL changed). Added **Verified prod state**:
+  the TRUE-79534 catalog **is** loaded (`Client_Aig.Items` = 2,212 rows — 1,110 `ASI-*` / 558
+  `SM-*` / 37 `SP-*`; `ASI-2YG2` present), so a future 1064 here is the **empty-input** variant,
+  **not** a missing AIG item ID — do not re-diagnose it as TRUE-79534. Pinned the unguarded lookup
+  to **`_underscore/Model/Aig/Entitlement.php:48`** (from `api2/Component/Api/V2/V2.php:2601`) and
+  recorded the exact malformed fulfillment query, so the fix is a plain guard clause. Quantified
+  the historical outage from core `Logs.Api` (**1,200 × HTTP 500/day, zero successes on the route,
+  2026-05-20 → 2026-07-28**, the same batch replayed byte-identically — corroborates the
+  scheduled-re-send model). Flagged the **open question**: 621 entitlements exist with `dtCreated`
+  up to 2026-08-04 and the creating path was not positively identified. (mhammontree)
 - 2026-08-03 (later pass) — **Two corrections.** (1) The `ERROR 1267` temp-table collation trap is a
   **per-machine server-config property, not a "prod-only" / version difference**: measured prod Core
   8.0.39 with the MySQL 8 default `utf8mb4_0900_ai_ci` vs local Core_2 8.0.44 with an explicit

package/knowledge/clients/growrk/INDEX.md

@@ -3,4 +3,5 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [Transfer Order Flow (NetSuite SalesOrder workaround → dedicated TransferOrders)](features/transfer-order-flow.md) | 2.0 | GroWrk moves inventory between their own locations (warehouses, distribution points). | _underscore/Model/Client/TransferOrder.php, _underscore/Model/Client/TransferOrderItem.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemReceipt.php, _underscore/Model/Client/ItemReceiptItem.php, _underscore/Trait/Netsuite/TransferOrder.php, dbchanges2/Client_Growrk/, dbchanges2/Client/ |
+| [GroWrk Units — NetSuite-module custom fields (the c_lastModified 500 incident)](features/units-netsuite-custom-fields.md) | 2.0 | GroWrk is on the dbchanges2 **`netsuite`** module (`dbchanges2/Client_Growrk/_modules.txt`), so its Units carry the module's seven `c_` custom fields. | _underscore/Trait/Netsuite/Unit.php, _underscore/Model/Growrk/Unit.php, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql, dbchanges2/Client_Growrk/_modules.txt |
 | [GroWrk](profile.md) | 2.0 | GroWrk is a 2.0 client using TOGA for inventory and order management. |  |

package/knowledge/clients/growrk/features/units-netsuite-custom-fields.md

@@ -0,0 +1,122 @@
+---
+title: "GroWrk Units — NetSuite-module custom fields (the c_lastModified 500 incident)"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: growrk
+type: client-feature
+status: active
+updated: 2026-08-10
+owners: ["bala", "mhammontree", "rgirish"]
+files:
+  - _underscore/Trait/Netsuite/Unit.php
+  - _underscore/Model/Growrk/Unit.php
+  - dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql
+  - dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql
+  - dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql
+  - dbchanges2/Client_Growrk/_modules.txt
+related:
+  - ../profile.md
+  - ../../../2.0/apps/_underscore/features/units-for-items-for-purchase-orders.md
+  - ../../../2.0/apps/_underscore/features/acl-permission-chain.md
+  - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+  - ../../../2.0/apps/dbchanges2/workflows/client-schema-drift-audit.md
+  - ../../../1.0/apps/library/features/http-500-error-monitor.md
+---
+
+## Summary
+
+GroWrk is on the dbchanges2 **`netsuite`** module (`dbchanges2/Client_Growrk/_modules.txt`), so its
+Units carry the module's seven `c_` custom fields. `_Model_Growrk_Unit` is *only*
+`extends _Model_Client_Unit` + `use _Trait_Netsuite_Unit` — **every one of those `c_` fields is
+declared by the trait**, and each must exist as a physical column in `Client_Growrk.Units`, as a
+`CustomRecordFields` row (recordId **31** = Units), and as an `AclCustomFieldPermissions` grant.
+
+On **2026-08-10** production GroWrk 500'd on `GET /v2/item-fulfillments` because one of those
+columns — `c_lastModified` — had never been created: the module migration
+`_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` had applied to `Client_Growrk` only
+**partially**. Two catch-up migrations fixed it in production; the general playbook lives in
+[client schema-drift audit](../../../2.0/apps/dbchanges2/workflows/client-schema-drift-audit.md).
+
+## The seven fields (all from `_Trait_Netsuite_Unit`)
+
+`c_netsuiteInternalInventoryAssignmentId` (integer), `c_itemDescription`, `c_lastModified`,
+`c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`, `c_grade` (char).
+
+Because they come from a **trait**, grepping `Model/Growrk/Unit.php` alone shows nothing — the model
+body is empty. Any audit of "what columns must this client have?" has to follow the `use` statements.
+
+## What happened (2026-08-10 production incident)
+
+**Symptom.** 6 logged 500s in ~26 seconds across 5 EC2 instances (`Logs.Api` ids 2052548-2052569),
+all `GET /v2/item-fulfillments`, `depth=6`, one fulfillment uuid — and **all sharing one
+`transactionId`**.
+
+**Root cause.** `Client_Growrk.Units` had no `c_lastModified` column. `_Model` builds its SELECT from
+**every** declared property, so MySQL threw **1054 "Unknown column 'c_lastModified' in 'field
+list'"** and api2 answered `EO-1` / HTTP 500. The trace bottomed out at `_Model->load()` from
+`getChildOptionRecords()` — the FK hop to Unit.
+
+**Trigger.** GroWrk was onboarded to NetSuite on **2026-08-05**, which matches the `Logs.Issue`
+first-seen timestamp of 18:03:22 exactly. Before onboarding, Units resolved through the plain
+`_Model_Client_Unit` (no `c_` fields at all), which is why the same request shape had **1,613
+successful 200s** right up to 2026-08-05 15:37:51.
+
+**Extent of the drift.** From `_modules/netsuite/2026-07-10a - UnitInventoryFields.sql`:
+**6 of its 7 Units columns** landed (`c_lastModified` did not) and **5 of its 6
+`CustomRecordFields` rows** never inserted.
+
+## Remediation (applied to production and verified)
+
+| Migration | What it does |
+|---|---|
+| `dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql` | `ALTER TABLE Units ADD c_lastModified VARCHAR(255) NULL AFTER c_itemDescription`; the 5 missing `CustomRecordFields` rows (recordId 31) **reusing the module migration's fixed UUIDs** so every client stays identical; and their 5 `AclCustomFieldPermissions` grants (roleId 1 = Base, `isWritable = 1`) |
+| `dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql` | The `c_itemDescription` ACL grant — missed by the first file because that `CustomRecordFields` row already existed from the partial run, and the first file only granted ACL for rows **it** inserted |
+
+**Verified end state:** all 7 recordId-31 custom fields have exactly one ACL grant; the
+`Client_Aig`-vs-`Client_Growrk` column diff across all 20 tables in the request path is empty; no
+further 500s. The `ALTER` was an **instant** operation (MySQL 8.0.39, 5,415 `Units` rows).
+
+> **⚠ Two follow-ups on these files.** (1) The first file is named `2026-08-10 - …` with **no letter
+> suffix** — the dbchanges2 contract requires `YYYY-MM-DD<letter>` (it should be `2026-08-10a`), and
+> that rule is hook-enforced. (2) At capture time **neither file was present in the `dbchanges2`
+> checkout or on any branch**, while both are applied to production — i.e. production is ahead of the
+> repo. Commit them, or the next client/environment provisioned from `dbchanges2` re-creates exactly
+> this drift.
+
+## Gotchas / known issues
+
+- **The failing field is unrelated to the requested field.** The request asked only for
+  `…itemFulfillmentItemUnits.unit.serialNumber`, yet failed on `c_lastModified`, because `_Model`
+  loads the whole row. **Any** traversal reaching a Growrk Unit failed, on **any** route — not just
+  this field list.
+- **Execution times were 0.19s-2.3s, which DISPROVES the timeout / deep-traversal /
+  memory-exhaustion theories** that a `depth=6` request invites. Read `Logs.Api.executionTime`
+  before theorising about depth.
+- **6 near-identical hits sharing ONE `transactionId` is apiproxy's cross-region 5xx retry**, not 6
+  user actions. Do not read the repeat count as user impact.
+- **`Logs.Issue.status` said RESOLVED with `dtLastOccurred` 4 days earlier while the defect was
+  still live.** Issues auto-resolve on consecutive quiet windows, so RESOLVED means "stopped
+  firing" — GroWrk traffic had simply gone silent. Never close an investigation on Issue status.
+- **A missing `AclCustomFieldPermissions` grant fails the WHOLE request** (`EZ-2` / 403) when the
+  field is named in `fields=` — it is not silently dropped. So an ACL gap and a column gap present
+  very differently: 403 vs 500.
+- **`Client_Growrk` recordId 312 (`c_netsuiteInternalSalesOrderId`, `TransferOrders`) has a
+  `CustomRecordFields` row with no ACL grant — and that is NORMAL**, not a defect: `Client_Aig` is
+  identical. Do not "repair" it on a future audit.
+
+## Change history
+
+- 2026-08-10 — **Fixed** the production GroWrk 500s on `GET /v2/item-fulfillments`: `Units` was
+  missing the `c_lastModified` column declared by `_Trait_Netsuite_Unit`, so `_Model`'s
+  SELECT-all-declared-fields behaviour raised MySQL 1054 on **every** load of a Growrk Unit
+  (`EO-1`/500) — triggered by the 2026-08-05 NetSuite onboarding, which switched Units from
+  `_Model_Client_Unit` onto the module model. Root cause was a **partially applied**
+  `_modules/netsuite/2026-07-10a - UnitInventoryFields.sql` (6 of 7 columns, 1 of 6
+  `CustomRecordFields` rows). Applied two catch-up migrations in `Client_Growrk/` (column + the 5
+  missing metadata rows with the module's fixed UUIDs + 5 role-1 grants; then the `c_itemDescription`
+  grant that the insert-then-grant pattern had skipped) and verified the end state against
+  `Client_Aig`. Recorded the diagnostic traps: the failing column is unrelated to the requested
+  field, sub-second execution times disprove the timeout theory, one shared `transactionId` means
+  apiproxy retried, `Issue.status = RESOLVED` only means the tenant went quiet, and recordId 312's
+  grant-less row is the normal cross-client state. (bala)

package/knowledge/clients/growrk/profile.md

@@ -7,15 +7,17 @@ apps:
   - worker2
   - toga2-supply
   - dbchanges2
+  - library
 project: _Underscore
 client: growrk
 type: profile
 status: active
-updated: 2026-07-14
-owners: ["rgirish", "mhammontree"]
+updated: 2026-08-10
+owners: ["rgirish", "mhammontree", "bala"]
 files: []
 related:
   - clients/growrk/features/transfer-order-flow.md
+  - clients/growrk/features/units-netsuite-custom-fields.md
 ---
 
 ## Summary
@@ -24,6 +26,23 @@ GroWrk is a 2.0 client using TOGA for inventory and order management. Their prim
 
 GroWrk is (as far as told) the **only client on TOGa Supply's Fulfill & Ship** flow — it is the client used to drive that feature and its return-label increment (TRUE-79191) green. See the [Fulfill & Ship feature](../../2.0/apps/toga2-supply/features/fulfill-and-ship.md) and [tracking-number bridges](../../2.0/apps/_underscore/features/tracking-number-bridges.md).
 
-## Local dev gotchas
+## Identity
 
-- **`_Model_Growrk_Unit` declares runtime custom-field columns that must physically exist on `Client_Growrk.Units`.** The model declares `c_` fields (`c_itemDescription`, `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`, `c_grade`); these are runtime custom-field columns, so a locally-drifted `Client_Growrk` DB missing them **1054s (unknown column) on any Unit load**. Add the columns to bring a local DB current.
+`Core.Clients` id **33**, `clientIdentifier` **`Growrk`**, row title **"Growrks"**; databases
+`Client_Growrk` / `Logs_Growrk`; site `growrk.togasupply.com`. Written **GroWrk** in prose and
+`growrk` in slugs/config — all four spellings refer to this one client.
+
+GroWrk was **onboarded to NetSuite on 2026-08-05**, and it is on the dbchanges2 **`netsuite`**
+module (`dbchanges2/Client_Growrk/_modules.txt`).
+
+## Gotchas
+
+- **`_Model_Growrk_Unit`'s `c_` columns must physically exist on `Client_Growrk.Units` — in EVERY
+  environment, production included.** The model body is empty: it is only
+  `extends _Model_Client_Unit` + `use _Trait_Netsuite_Unit`, and **the trait** declares all seven
+  custom fields (`c_netsuiteInternalInventoryAssignmentId`, `c_itemDescription`, `c_lastModified`,
+  `c_chargingBrick`, `c_deviceStatus`, `c_chargingCable`, `c_grade`). `_Model` SELECTs every
+  declared field regardless of the API's requested `fields=`, so a DB missing any one of them
+  **1054s (unknown column) on any Unit load, on any route**. This is not only a local-drift
+  problem — it took production down for GroWrk on 2026-08-10; see
+  [Units — NetSuite-module custom fields](features/units-netsuite-custom-fields.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.540",
+  "version": "1.0.542",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",