toga-ai 1.0.911 → 1.0.913

package/knowledge/1.0/apps/library/INDEX.md

@@ -4,6 +4,7 @@
 |-----|---------|
 | [Library (1.0 Framework) Architecture](architecture.md) | The 1.0 `App_` framework core: bootstrap/autoloader, the `app/` (`App_`) + `browser/` (`Browser_`) class trees, and vendored packages; open for how any 1.0 app |
 | [Address Validation Gateway (App_Api_OfficeDepot::validateAddress + USPS fallback)](features/address-validation-gateway.md) | The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before sav |
+| [API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)](features/api-request-logging-and-retention.md) | Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open befo |
 | [Where a new App_ class goes — the app/ folder IS a behavioral contract](features/app-class-placement-base-contracts.md) | Where a new `App_` class goes — each `app/` folder's base class carries real inherited behavior; open before creating any class under `app/model|client|api`. |
 | [App_Sso — Reusable 1.0 SSO Initiation (SP-initiated SAML via saml.togahub.com)](features/app-sso-initiation.md) | Reusable 1.0 SP-initiated SAML SSO via `saml.togahub.com` (`App_Sso::initiate`); the canonical doc to follow for any "add SSO to `<1.0 app>`" request. |
 | [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | 1.0 cron check-in/out lifecycle via `App_Framework::cronInitialization()`/`cronFinished()` → `Common.CronJobExecutions`; open to monitor a 1.0 cron or read its |

package/knowledge/1.0/apps/library/features/api-request-logging-and-retention.md

@@ -0,0 +1,89 @@
+---
+title: "API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-09-29
+owners: ["mhammontree"]
+files:
+  - library/app/apitransaction.php
+  - library/app/model/logs/api.php
+  - library/app/model/toga2/logs/api.php
+  - worker/crons/maintenance/maintenance_daily.php
+related:
+  - error-capture-1-0.md
+  - ../architecture.md
+---
+
+Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open before changing log routing, retention, or debugging past-window history.
+
+## Summary
+
+`App_ApiTransaction` writes one row per outbound API call. It has **two** storage paths — the legacy 1.0 `Logs`.`API` table and the 2.0 per-tenant `Logs_<Tenant>` cluster — and picks between them per run. Retention is enforced only on the 1.0 side, by a daily worker cron, at **30 days**.
+
+## How it works
+
+### Which log DB a call lands in
+`library/app/apitransaction.php:377`:
+```php
+if (App_Registry::varIsSet('toga2LogsDatabase')) { /* App_Model_Toga2_Logs_Api */ }
+else { /* App_Model_Logs_API  → legacy Logs.API */ }
+```
+- 2.0 routing is **opt-in per cron**. Only 10 crons set it, all Prudential, each hardcoding
+  `App_Registry::set('toga2LogsDatabase', 'Logs_Prudential')`.
+- Every other `crons/toga2/*` cron falls through to the 1.0 table — **this is why 2.0 client traffic
+  lands in the 1.0 `Logs.API`**.
+
+### Retention (1.0 only)
+`worker/crons/maintenance/maintenance_daily.php` trims the log tables nightly.
+- `$daysToKeep_APILog` = **30** (was 90; cut 2026-09-28, TRUE-82049, commit `89ef44f35`).
+- The `DELETE FROM Logs.API` is **chunked**: 25,000 rows per statement in a do/while loop with a
+  1s sleep, still bounded overall by `$maxLimitForDeletes`. Every *other* table this cron trims is
+  a single statement — `Logs.API` is the exception because it is ~12x the next biggest table
+  (`Vision_Log.Log`, 45.85 GB), and a single 5,000,000-row DELETE holds locks for minutes and lags
+  read replicas.
+- **The one-time backlog purge (~20M rows) is separate and not done** as of 2026-09-29.
+
+### Storage facts (measured 2026-09-28)
+- 1.0 `Logs`.`API`: **565.27 GB / 34,464,186 rows**, growing **~6.3 GB/day**. `dtStamp` is the
+  indexed date column; `sourceJob` names the writing cron.
+- Top writers by bytes (2-day sample): `sync_togasupply_compass_usa.php` 2.53 GB (128 KB/row),
+  `periodic_forecast_discrepancy_fix.php` 1.63 GB, `sync_togasupply_hh.php` 0.84 GB,
+  `sync_togasupply_elite.php` 0.53 GB, `compass/workflow/5_create_netsuite_sales_order.php`
+  1,581 KB/row, `infrastructure/system_monitors.php` 3,040 KB/row.
+- 2.0 logs cluster (`writer.logs.database.togahub.com`): **~1.67 TB** across 36 `Logs_<Tenant>`
+  DBs, oldest rows **2025-08-24** (~13 months — *not* the 90 days believed), and **no scheduled
+  DB events trimming it**. Largest: `Logs_Wje` 372.80 GB, `Logs_Elite` 228.99 GB,
+  `Logs_Compass` 185.88 GB, `Logs_Nychh` 129.71 GB, `Logs_Prudential` 125.32 GB.
+  `Logs_Northwell` is 43.59 GB from only 194,643 rows = 224 KB/row.
+
+### Decision
+**`Logs.API` is forensics-only; 30 days is enough** (Jeff, 2026-09-17). Debugging that needs
+older history is no longer possible — slow-burn bugs must be caught by **alerting**, not
+reconstructed after the fact.
+
+## Gotchas
+
+- **DELETE does not return space to disk on InnoDB.** After the backlog purge the file stays
+  ~565 GB with `DATA_FREE` growing. Reclaiming disk needs `OPTIMIZE TABLE` (~565 GB spare
+  required) or a copy-and-swap — a separate decision, not part of the retention change.
+- **Do NOT move 2.0 client logging to the 2.0 cluster without solving retention there first.**
+  It would shift ~7 GB/day onto a cluster that is already bigger and has **no working retention**.
+  Deliberately descoped from TRUE-82049. Nobody has found the 2.0 archiving script — it is not in
+  `worker2`, `api2`, or `_underscore`.
+- **`App_Model_Toga2_Logs_Api` has NO `save()` override.** The 1.0 `App_Model_Logs_API::save()`
+  deliberately wraps `parent::save()` in try/catch because a long-running cron can outlive its
+  `db_logs` connection (MySQL "server has gone away"), which once aborted whole imports mid-run.
+  Routing traffic to the 2.0 model without adding that guard reintroduces that failure.
+- **The 2.0 model has a `source` column the 2.0 branch never sets** — moving a cron to 2.0 logging
+  loses the writing-cron name (`sourceJob`'s equivalent).
+- **Cap payload size when adding new logging.** Team precedent: worker2's
+  `MAX_LOG_PAYLOAD_LENGTH = 8000` with a `...[truncated]` marker
+  (`Worker/Netsuite/Opportunity.php:148`, `Worker/Clickup/Opportunity.php:45`). Rows of
+  1,500–3,000 KB are what built the 565 GB table.
+
+## Change history
+- 2026-09-29 — Created: `Logs.API` retention cut 90→30 days and chunked DELETE (TRUE-82049); dual 1.0/2.0 routing and 2.0-cluster storage facts recorded. (mhammontree)

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -44,6 +44,7 @@
 | [Legacy page meta (Page::meta) & context-scoped ClientRecordFieldSettings](features/page-meta-context-field-settings.md) | Legacy `Page::meta()` page-meta resolver + context-scoped `ClientRecordFieldSettings`; open when a grid column header is wrong/camelCase, or a `/pages/meta` pag |
 | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | `_underscore` opens three per-client DB connections (data/logs/archive) per request; open when a local or prod request 500s with `Unknown database`, when resolv |
 | [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona names in multiple languages via a `PersonaTranslations` sidecar (copy of the assortment-translation pattern); open when adding/running persona tr |
+| [Persona nav-visibility gate + user nav pins (keyed by Core.SurfaceElements uuid)](features/persona-nav-visibility-gate.md) | Sidebar-nav persona gate in `_Model_Core_Surface` (`Client.PersonaNavVisibility`) and user nav pins (`Client.UserNavPreferences`), both keyed by Core `SurfaceEl |
 | [Record Change Audit Log (Logs_<Client>.Record / RecordField) — reading a field's history](features/record-change-audit-log.md) | Field-level change audit trail per tenant; open to answer "who changed this field, to what, when" or to tell a direct SQL edit from an in-app change. |
 | [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | Engine that mirrors a downstream ItemFulfillment up the SO→PO→SO chain onto every customer-facing SO; open when touching IF creation, bundle scaling, or upstrea |
 | [Sales-order denial reason (`ApprovalDecisions.note`) — capture, the denial-details card, and the read-only workflow mode](features/sales-order-denial-reason.md) | Map of the SO approval denial reason (`ApprovalDecisions.note`) — where it is written, read, and displayed; open when building any denial-reason display, banner |
@@ -53,6 +54,7 @@
 | [Shipping Carriers & Carrier Account Numbers (records, matching, duplication, deletion)](features/shipping-carriers-and-accounts.md) | Per-client `ShippingCarriers` / `ShippingCarrierAccountNumbers` model, how the syncs match/create carrier rows, and the duplication + blank-carrier + pagination |
 | [_String helpers — ASCII-safe HTML entity encoding (and the parseBetween trap)](features/string-html-entity-helpers.md) | `_String` static string utilities: use `encodeToAsciiHtmlEntities` for any dynamic text going into a 2.0 email; open when escaping email text or parsing substri |
 | [Surface FIELD ↔ RecordField binding (maxLength / validationType / isWritable fold-in)](features/surface-field-recordfield-binding.md) | Resolver capability that folds `Core.RecordFields` input-contract metadata (maxLength/validationType/isWritable) onto a surface FIELD element; open when a 2.5 s |
+| [DEFERRED — re-key Client.SurfaceOverrides from numeric Core ids to uuids](features/surface-overrides-uuid-rekey.md) | Deferred: `Client.SurfaceOverrides.surfaceId`/`surfaceElementId` still point at Core rows by numeric id; open before a client's first override rows on an env wh |
 | [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | Runtime for the platform Surface UI layer (`_Model_Core_Surface::resolve`/`metaGroup`, replaces `Page::meta`); open for surface resolution, the override cascade |
 | [Table-View Hyperlink Columns (meta → ACL → computed URL → render)](features/tableview-hyperlink-columns.md) | How a 2.0 table-view column renders its value as a clickable link (`hyperlinkRecordFieldId` → ACL field-read → computed URL → render); open when a configured hy |
 | [TableView joins (TableViewJoins → SQL) — aliasing, chained multi-hop joins, ACL](features/tableview-joins.md) | How `TableViewJoins` rows become a view's join SQL — aliasing, chained multi-hop joins, ACL gating, and the three ways a view breaks; open before adding/rebasin |

package/knowledge/2.0/apps/_underscore/features/persona-nav-visibility-gate.md

@@ -0,0 +1,51 @@
+---
+title: "Persona nav-visibility gate + user nav pins (keyed by Core.SurfaceElements uuid)"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-09-29
+owners: [jcardinal]
+files:
+  - _underscore/Model/Core/Surface.php
+  - _underscore/Model/Client/PersonaNavVisibility.php
+  - _underscore/Model/Client/UserNavPreference.php
+  - dbchanges2/Client/2026-09-29a - NavItemSurfaceElementUuidColumns.sql
+  - dbchanges2/Client/2026-09-29b - DropNavItemSurfaceElementIdColumns.sql
+  - dbchanges2/Core/2026-09-29a - UserNavPreferencesNavItemUuidRecordField.sql
+related:
+  - surface-resolver.md
+  - surface-overrides-uuid-rekey.md
+  - ../../dbchanges2/features/surface-layer-schema.md
+---
+
+Sidebar-nav persona gate in `_Model_Core_Surface` (`Client.PersonaNavVisibility`) and user nav pins (`Client.UserNavPreferences`), both keyed by Core `SurfaceElements.uuid`; open before touching nav visibility, pins, or any Client row that points at a Core Surface element.
+
+## Summary
+
+- Gate runs only when the surface `config.renderAs === 'nav'`. Default-deny: a nav element survives only if one of the caller's personas has a `PersonaNavVisibility` grant for it in the request app.
+- Grants and pins store `navItemSurfaceElementUuid` (CHAR(36), lowercase) = `Core.SurfaceElements.uuid`. The numeric `navItemSurfaceElementId` column was dropped 2026-09-29.
+- Rule: never key a Client row to a Core Surface/SurfaceElement by numeric id. Core ids shift when Core is rebuilt or copied to another env; seeded uuids are the same everywhere.
+
+## How it works
+
+1. `_buildBundle` step 5b: `$isNavGated` = `config.renderAs === 'nav'`. False → nothing runs, no element dropped (user menu uses `renderAs='menu'`). Do not widen this check.
+2. App id = `$api->app->id`, else `Surfaces.appId`.
+3. `_grantedNavItemSurfaceElementUuids($personaIds, $appId)`: one query on the client DB (`personaId IN (...) AND appId = ... AND navItemSurfaceElementUuid IS NOT NULL`) → set keyed by lowercase uuid. No personas or no app → empty set → all nav elements hidden.
+4. Per element: drop if `strtolower($element->uuid)` is not in the set.
+5. Cluster-safe: grant uuids are read on the client connection and matched in PHP against the Core elements. No cross-database SQL.
+
+**Pins (`Client.UserNavPreferences`, Core.Records 349):** RecordField 2645 = `navItemSurfaceElementUuid` (2487 = old id field, retired). `prePost` interceptor stamps `userId` from the JWT and `appId` (`$api->app->id`, fallback 34), checks uuid format, lowercases it.
+
+## Gotchas
+
+- Model field is `FIELD_CHAR`, not `FIELD_CHAR_UUID`. `FIELD_CHAR_UUID` auto-generates a value, which would overwrite the FK.
+- Symptom of id drift: sidebar-nav returns only the theme (all elements dropped). Seen on qa-alpha 2026-09-29: Core rebuilt from prod, grants pointed at ids 114-134, local elements were 241-261 (+127).
+- `Client/2026-09-29b` deletes old id-only grants only for a persona+app that already has uuid grants, then sets the uuid NOT NULL. A client never re-keyed fails loud (`Invalid use of NULL`) instead of losing its whole sidebar. Fix: write that client's uuid grant seed (pattern: `Client_True/2026-09-29a`), then re-run.
+- Run order: `Client/2026-09-29a` + client grant seeds → code live → `Client/2026-09-29b` → `Core/2026-09-29b` (removes RecordField 2487).
+- `Client.SurfaceOverrides` still uses numeric Core ids — same risk, deferred: [surface-overrides-uuid-rekey](surface-overrides-uuid-rekey.md).
+
+## Change history
+- 2026-09-29 — Grants + pins re-keyed from numeric SurfaceElements id to uuid; `_grantedNavItemSurfaceElementIds` renamed `_grantedNavItemSurfaceElementUuids` (_underscore `8e299b1b`, dbchanges2 `66ba056`) (jcardinal)

package/knowledge/2.0/apps/_underscore/features/surface-overrides-uuid-rekey.md

@@ -0,0 +1,37 @@
+---
+title: "DEFERRED — re-key Client.SurfaceOverrides from numeric Core ids to uuids"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: deferred
+updated: 2026-09-29
+owners: [jcardinal]
+files:
+  - _underscore/Model/Core/Surface.php
+  - _underscore/Model/Client/SurfaceOverride.php
+related:
+  - persona-nav-visibility-gate.md
+  - surface-resolver.md
+---
+
+Deferred: `Client.SurfaceOverrides.surfaceId`/`surfaceElementId` still point at Core rows by numeric id; open before a client's first override rows on an env whose Core can be rebuilt.
+
+## Summary
+
+Not built. `_Model_Core_Surface::_loadOverrides` matches `SurfaceOverrides.surfaceId` / `surfaceElementId` to Core numeric ids. Same failure as the nav gate before 2026-09-29 ([persona-nav-visibility-gate](persona-nav-visibility-gate.md)): once a client has override rows and Core is rebuilt/copied with different AUTO_INCREMENT ids, overrides silently miss or land on the wrong element.
+
+## How it works
+
+Planned fix (do before first client use on such an env):
+1. Add `surfaceUuid` / `surfaceElementUuid` CHAR(36) to `Client.SurfaceOverrides` (`FIELD_CHAR`, not `FIELD_CHAR_UUID`).
+2. `_loadOverrides`: match by uuid on the client connection, join to Core elements in PHP (no cross-database SQL).
+3. Forward-only re-key seeds per client, then drop the id columns with a fail-loud NOT NULL step (same shape as `Client/2026-09-29b`).
+
+## Gotchas
+
+- Elements seeded with reserved id literals are stable across envs; the risk is elements that took AUTO_INCREMENT ids (e.g. the `navigation` surface's elements). The framework-rules "Bounded exemption — the Surface tables" stays as-is until this is built.
+
+## Change history
+- 2026-09-29 — Recorded as deferred after the nav-gate uuid fix exposed the same numeric-id risk (jcardinal)

package/knowledge/2.0/apps/_underscore/features/surface-resolver.md

@@ -36,6 +36,8 @@ related:
   - sales-order-status-filter-surface.md
   - surface-field-recordfield-binding.md
   - ../../toga25-supply/features/side-navigation-and-default-route.md
+  - persona-nav-visibility-gate.md
+  - surface-overrides-uuid-rekey.md
 ---
 
 Runtime for the platform Surface UI layer (`_Model_Core_Surface::resolve`/`metaGroup`, replaces `Page::meta`); open for surface resolution, the override cascade, vocab/theme/message resolution, client seeding, caching, and 403/empty-bundle debugging.

package/knowledge/2.0/apps/api2/features/health-check-endpoint.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-29
 owners: [jcardinal]
 files:
   - api2/health
@@ -32,6 +32,7 @@ The EB/ALB liveness probe (`/health`) is served two ways, and the **load-bearing
 - **`api2/health`** — a one-line static file (`OK`). Apache serves it directly.
 - **`api2/.platform/httpd/conf.d/health_probe.conf`** — comment-only documentation of why the probe is static (no directives; explains the decision at the Apache layer).
 - **Why it works without a rewrite fight:** `api2/.htaccess` only rewrites to `index.php` when the request target is **not a real file** (`RewriteCond %{REQUEST_FILENAME} !-f`). Because `health` is a real file, the rewrite is skipped and Apache serves it straight.
+- **`api2/.ebextensions/health-check.config`** (2026-09-29) — pins the EB default process health check: `aws:elasticbeanstalk:environment:process:default` → `HealthCheckPath: /health`, `MatcherHTTPCode: 200`. Without it EB checks `/`, which returns 405 → env Red/Severe, `Target.ResponseCodeMismatch`. Applies on the next deploy of each env.
 - **Why `enforce_https` doesn't redirect it:** EB health checks arrive on **HTTP:80 with no `X-Forwarded-Proto`**, and `enforce_https.conf` only redirects requests carrying the forwarded-proto marker — so the probe is not bounced to HTTPS.
 
 **PHP short-circuit (for `/v2/health` and direct hits).** At the top of `Index.php::api()`, before host dispatch and before the Core/Logs DB bootstrap. Matches against a class constant:
@@ -56,7 +57,14 @@ So **all** these probe variants return 200 and never touch routing or the DB: `/
 - **Do not** move `/health` back into a PHP-served route or rename/remove `api2/health` — that re-couples EB liveness to PHP-FPM worker availability, the exact failure this fixed (see [api2 architecture — EB "Degraded" health](../architecture.md)).
 - Health probes must degrade to a 200 short-circuit, not enter routing. Add new probe paths to `HEALTH_CHECK_ROUTES` in normalized form, never as a new exact-string branch.
 
+- Env Red/Severe with `Target.ResponseCodeMismatch` → check the target group health-check path is `/health`, not `/`. The `.ebextensions` file sets it; an env not yet redeployed still has the old path.
+- Git Bash path-converts AWS CLI args (`/health` → `C:/Program Files/Git/health`). Prefix the command with `MSYS_NO_PATHCONV=1`.
+
 ## Related
 
 - [api2 architecture — EB "Degraded" health](../architecture.md)
 - [V2 API error codes](v2-api-error-codes.md)
+
+## Change history
+- 2026-09-29 — Added `.ebextensions/health-check.config` (HealthCheckPath `/health`, matcher 200); api-qa-alpha was Red on `/` 405 (api2 `885d114`) (jcardinal)
+- 2026-08-28 — `/health` moved to a static Apache file, off PHP-FPM (jcardinal)

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 28 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 29 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 40 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 92 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 80 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/knowledge/clients/nycdoe/features/hold-status-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-09-25
+updated: 2026-09-29
 owners: [mhammontree, sking]
 files:
   - worker/crons/sync/nycdoe/send_ticket_updates.php
@@ -16,6 +16,7 @@ files:
   - library/app/model/common/nycdoeticket.php
   - dbchanges/Common/MH/2026-09-16.sql
   - library/app/api/nycdoev2.php
+  - worker/crons/sync/nycdoe/import_inc.php
   - togadesk/desk/includes/classes/class.repair.php
 related:
   - servicenow-integration.md
@@ -101,6 +102,60 @@ Current selection query:
 - The standalone assignment-group block was **deleted** — the main payload already diffs `assignment_group`, so it was a duplicate GET, a second code path, and the source of the lockout stamp.
 - Debug pin already in the file: the commented `#AND NYCDOETickets.ticketNumber = '...'` line scopes a pass to one ticket.
 
+### Assignment-group scoping — what counts as OUR ticket (rule, Mark 2026-09-28)
+**We only care about an INC or RITM while it is assigned to one of OUR ServiceNow assignment
+groups. If it is not, drop it and never re-check it. NEVER add another vendor's group to
+`Common.NYCDOETicketsDepartments`** to "make it sync" — a group that is not ours means DOE
+reassigned the incident away from ASI.
+
+**⚠ `Common.NYCDOETickets.isValidDepartmentName` means the OPPOSITE of its name.** `1` =
+**EXCLUDED from processing**; `0`/`NULL` = processed. Every write of `1` is on a skip path
+(invalid site code, unmapped assignment group, or SD QA — deliberately skipped at
+`process_tickets.php:392`), and all four DOE selections pick up only `= 0 OR IS NULL`. The
+companion column `departmentName` carries the human reason (`INVALID_SITE_CODE`,
+`Invalid Address Resolution`, or the actual ServiceNow group name). Reading the column name
+literally gives you the exact opposite of the truth. Writers: `process_tickets.php` (348, 392,
+1520), `import_asn.php:104`, `import_inc.php:85`, `send_ticket_updates.php:235`,
+`send_request_item_updates.php:56`. A pure rename across 13 sites would be safe, but must not be
+bundled with a behavior change.
+
+**The drop is self-healing — which is why no new column is needed.** `import_inc.php:82-86` loads
+the existing row by `referenceSysId` and unconditionally resets
+`dtProcessed = null; isValidDepartmentName = null; departmentName = null;` on every re-import
+(same in `import_asn.php:148`). While a ticket sits in a non-ASI group it never matches the
+importer filter, so it stays dropped at zero cost; if DOE reassigns it back to ASI its
+`sys_updated_on` moves, it re-imports, and the flag clears automatically. A purpose-built new
+column would NOT be reset by either import path and would make the drop irreversible — reusing
+`isValidDepartmentName` is the correct choice, not a compromise.
+
+**⚠ OPEN DEFECT (backfilled by hand, CODE FIX NOT DONE — needs its own ticket).**
+`send_ticket_updates.php` (~L375) treats an unmapped group as "someone forgot a mapping" and
+emails devteam@togatech.com hourly saying *"Add the group to that table to let them sync."* That
+advice is wrong. Measured cost: 15 tickets across 9 other-vendor groups (NOC Field, ATLAS Tier III
+Support, SD Nagarro Queens, OASIS L3 Support, NGSIS/GAM Support, Central User Support, Field Ops
+Queens, Field Ops Bronx, IET Support) re-GET every 6 minutes = **~3,600 wasted ServiceNow calls/day
+and ~236 emails since 2026-09-18**. Those 15 INCs were set by hand in production to
+`isValidDepartmentName = 1` with the real group in `departmentName`, and confirmed excluded.
+**Agreed approach for the fix ticket (CTO-reviewed) — an ordered predicate:**
+1. Group **is** in `NYCDOETicketsDepartments` → process normally (this is what keeps SD QA working).
+2. Group is not in the table but **starts with `ASI`** → keep the throttled email; this is the only
+   case the current advice is right for.
+3. Neither → **drop silently.**
+Do **not** close the repair orders — write one `repair_order_notes` line and send a one-time digest
+to DOE ops. `send_request_item_updates.php` needs the same treatment for RITMs.
+
+**⚠ Three competing definitions of "our DOE assignment groups" — they disagree.**
+- **(a) `Common.NYCDOETicketsDepartments`** — 14 rows: ASI Bronx, ASI Brooklyn, ASI Manhattan,
+  ASI Queens, ASI Staten Island, ASI General, ASI Calls, ASI Reconnect, ASI Research, ASI DOA,
+  ASI TM, SD QA (`isImportEnabled=0`), ASI Disposal, ASI PCS Delivery. **This one is correct.**
+- **(b) `download_tickets_incidents.php:80`** — a hardcoded 12-group ServiceNow query list that
+  includes "ASI Enhanced", which exists nowhere else. The file is referenced by **no**
+  `schedules/*.json` → **dead code; delete it in its own commit.**
+- **(c) `App_Api_NYCDOEV2::listIncItems()` (`library/app/api/nycdoev2.php:912`)** — the **live**
+  importer filter, a prefix match `assignment_groupSTARTSWITHASI^stateIN1,2,3,10`. This is what
+  `import_inc.php` actually uses.
+**Risk:** a future group named "ASI Something" imports under (c) then is dropped under (a).
+
 ### Batch loop, run caps & per-ticket isolation (TRUE-82129 current shape)
 Before this the cron ran the selection query **once per ticket under `LIMIT 1`** with a `sleep(1)` after every ticket — one ticket per 6-minute run, ~10 days for a full pass over the queue.
 - **One `LIMIT 250` (`BATCH_SIZE`) read into a PHP array, consumed with `array_shift`, refilled only when empty.** The query is `type=ALL`, ~35,021 rows, `Using temporary; Using filesort` (it drives off `managed_service_orders`, so `Common.NYCDOETickets.idx_dtLastChecked` is unusable at this join order). A filesort already sorts every surviving row before returning the top N, **so a bigger LIMIT costs nothing extra**.
@@ -138,6 +193,19 @@ There is **no sandbox for the SNOW integration**; verification happens in prod.
 2. INC2087375's `u_eta` moving from 2026-09-07 to 2026-09-17.
 3. `SELECT COUNT(*) FROM Common.NYCDOETickets WHERE dtLastChecked IS NOT NULL` climbing.
 
+**Answering "did the alert fire, and did the mail actually send?" — two tables, not an inbox.**
+An inbox check alone is misleading; the DB is decisive.
+- **Did the alert fire?** marker rows in `Core.AdvanceShippingNoticeQueue`, `dedupeKey`
+  `system|ticket_sync_patch_failure_alert` and `system|ticket_sync_invalid_group_alert`.
+- **Did the mail send?** `Logs.Emails` — columns `dtSent`, `isSuccess`, `error`, `subject`, `to`,
+  `sourceJob`. ⚠ the table is **`Emails`**, not `Email`, and the timestamp is **`dtSent`**, not
+  `dtStamp`.
+
+**TRUE-82027 verified healthy in production 2026-09-28:** `dtLastChecked` + `idx_dtLastChecked`
+present; ~2,784 active tickets cycling in under an hour at `RECHECK_INTERVAL_MINUTES=6`; PATCH
+volume ~44,000/day at 443,706 x HTTP 200 against only 6 non-200 over 10 days (3x 502, 1x 429);
+reproduction case INC2087375 back in rotation and syncing.
+
 ### RITM cron — deliberately unchanged by TRUE-81044
 `send_request_item_updates.php` already pushes native `state:"8"` (On Hold) and **never wrote** `u_status_task`, so TRUE-81044's ask was already satisfied there. It carries a defensive `HOLD_*` guard on the scheduled-update path so it can't push `state:"-14"` while held; its reverse-hold logic still **reads** `u_status_task == 'On Hold'`.
 > **OPEN QUESTION:** ServiceNow's communication was scoped to **Incident** state. We did **not** extend `hold_reason` / `u_eta` / `u_other_reason_on_hold` to the RITM payload without confirmation. Whether the RITM On Hold transition carries the same mandatory-set requirement is unconfirmed — ask before changing the RITM payload.
@@ -175,7 +243,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
   - **Query shape that works (TRUE-82332):** `endpoint LIKE '%nycd3%...'` and broad `requestPayload LIKE` scans time out (>300s) on the MCP. Use exact matches: PATCH `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/data/tasks/INCxxxx'`; GET `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/v2/data/tasks?domain=diit%2Calm&filter=sys_class_name%3Dincident%5Enumber%3DINCxxxx'`. For sweeps: `FORCE INDEX (dtStamp)`, windows ≤2 days, plus `sourceJob` / `method` filters. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
 - **`NYCDOETickets` (`App_Model_Common_NYCDOETicket`, `db_common` / legacy `Common`)** stores the inbound SNOW record in `rawData`, a **`FIELDTYPE_BLOBSTORAGE`** field (S3 bucket `asifiles`, key `prod/Common/NYCDOETickets/rawData/{id}`) — **NOT a queryable DB column.**
 - **⚠ `repair_orders.dtUpdated` is skewed — never build a selection query on it.** It is written exactly **240 minutes behind** the clock used by `repair_order_notes.notesdate` and `NYCDOETickets.dtSynced`. **Do NOT compensate with `+240 MINUTE`** — 240 is almost certainly UTC-vs-Eastern and becomes 300 at the DST change (2026-11-01), so a hardcoded offset breaks twice a year. TRUE-82027 sidestepped it by removing `dtUpdated` from the query. Mechanism + the second (720-minute) skew: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
-- **⚠ `Logs.API` history is capped at 30 days from 2026-09-17** (Jeff, sprint review) and existing logs are being purged. This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0. Agreed: hold the purge until TRUE-82027 is verified in production.
+- **⚠ `Logs.API` history is capped at 30 days — SHIPPED 2026-09-28 (TRUE-82049).** This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`. Mechanics, storage facts and the not-yet-run backlog purge: [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md).
 - **Worker prod code cannot run on a Windows dev box.** The DB/S3 data layer fetches EC2 instance-metadata (IMDSv2) credentials and fails off-EC2. Investigate with the toga DB MCP and the `Logs.API` table instead of running the crons locally.
 
 ## Gotchas
@@ -202,6 +270,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
 - `qqStatus()` collapses all recognized hold sub-statuses to base `HOLD` in `repair_orders.status`; the specific hold **sub-reason persists only in `repair_order_notes.newstatus`**. Any status badge / report / filter reading `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
 
 ## Change history
+- 2026-09-29 — Assignment-group scoping rule + inverted `isValidDepartmentName` + the three competing group definitions; TRUE-82027 verified in prod (marker rows + `Logs.Emails` method). (mhammontree)
 - 2026-09-25 — TRUE-82332: hold-release no longer overwrites the close `state`; close re-sent when SN not closed; stuck-ticket re-queue recipe; `Logs.API` query shapes. (mhammontree)
 - 2026-09-18 — TRUE-82129: fixed the by-reference `fetchOne()` notice that killed every run; batch loop + run caps replaced LIMIT 1 + per-ticket sleep; alert emails de-PII'd and escaped; unmapped-group email now grouped per run. (mhammontree)
 - 2026-09-17 — TRUE-82027: outbound push runs once per order per pass; `dtLastChecked` split from `dtSynced`; failure alerting instead of a throw. (mhammontree)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-09-25
+updated: 2026-09-29
 owners: [mhammontree, sking]
 files:
   - worker/crons/sync/nycdoe/import_asn.php
@@ -62,7 +62,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
 | Table | Purpose |
 |---|---|
 | `NYCDOETickets` | Local ticket mirror (keyed `referenceSysId`; `type` = INCIDENT / RITM) |
-| `NYCDOETicketsDepartments` | Allowlist of valid assignment groups — others are not processed |
+| `NYCDOETicketsDepartments` | Allowlist of OUR assignment groups — others are dropped and never re-checked. ⚠ Two other places define this list differently, and `NYCDOETickets.isValidDepartmentName` is **inverted** (`1` = excluded): see [Hold-Status Sync](hold-status-sync.md) → *Assignment-group scoping* |
 | `NYCDOELocations` | Site-id → address mirror (backs ASN ship-to resolution) |
 | `AdvanceShippingNoticeQueue` | Stage-1 landing zone; **UNIQUE index `idx_dedupeKey`** |
 | `AdvanceShippingNotices` / `…Items` / `…Units` | ASN header (vendor+PO) → per-part items (`qtyOrder`) → individual units |
@@ -130,7 +130,7 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘      [UNIQUE ded
 
 ### Debugging & DB topology (DOE 1.0 sync)
 The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API` instead of running prod code locally.
-- **⚠ `Logs.API` retention is 30 days from 2026-09-17** (Jeff, sprint review) and existing logs are being purged — so it is no longer a *historical* debugging avenue, only a recent-window one. Bugs that burn slowly (a hold not re-pushed for 22 days; 9,914 PATCHes traced back to 2026-08-20) can no longer be reconstructed after the fact and must be caught by **alerting** instead. NYCDOE stays 1.0, so its logs remain in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0.
+- **⚠ `Logs.API` retention is 30 days — SHIPPED 2026-09-28 (TRUE-82049)**; mechanics, storage facts and the not-yet-run backlog purge in [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md) — so it is no longer a *historical* debugging avenue, only a recent-window one. Bugs that burn slowly (a hold not re-pushed for 22 days; 9,914 PATCHes traced back to 2026-08-20) can no longer be reconstructed after the fact and must be caught by **alerting** instead. NYCDOE stays 1.0, so its logs remain in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0.
 - **`repair_orders.dtUpdated` and `.lastupdateddate` are both time-skewed in `TOGaDeskSupport`** — never select or diff on them, and never compensate with a fixed offset. Detail: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
 - **`dbchanges` does not describe the real 1.0 schema** — some live columns were created outside the repo. Verify indexes/columns against the database, not the repo: [Authoring & Shipping a 1.0 dbchanges SQL File](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
 - **`db_core` alias → legacy (V1) environment, schema `Core`** — holds the FLAT tables `AdvanceShippingNotices` / `AdvanceShippingNoticeItems` / `AdvanceShippingNoticeUnits` (columns incl. `togadeskRepairOrderId`, `serialNumber`, `netSuiteInternalSalesOrderId`, `netSuiteInternalPurchaseOrderId`, `customerPurchaseOrder`). The 1.0 cron reads **legacy/Core**.
@@ -262,5 +262,6 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-09-29 — Assignment-group scoping corrected (inverted `isValidDepartmentName`, three competing group lists) with detail in hold-status-sync; `Logs.API` 30-day retention shipped. (mhammontree)
 - 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
 - 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.911",
+  "version": "1.0.913",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",