toga-ai 1.0.398 → 1.0.400

package/knowledge/2.0/apps/_underscore/features/assortment-name-translation.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-26
+updated: 2026-07-21
 owners: ["bala"]
 files:
   - _underscore/Model/Client/AssortmentTranslation.php
@@ -25,9 +25,15 @@ Serves Assortment (product-grouping) **names** in multiple languages by adding a
 English stays in `Assortments.name`; the sidecar holds only non-English overlays. Because the
 translation layer is fully metadata-driven (see
 [Language Translation Layer](../../api2/features/language-translation-layer.md)), wiring this up
-required **no api2 code change** — only a new table, model, Core record registration, ACL grant,
-and a single field link. Compass Canada's French (fr-CA) assortment names are seeded as the first
-consumer.
+required only a new table, model, Core record registration, ACL grant, and a single field link.
+Compass Canada's French (fr-CA) assortment names are seeded as the first consumer.
+
+> **Correction (2026-07-21):** the original "no api2 code change needed" claim was only true for the
+> **FK/top-level** read paths. Assortment names are surfaced on the storefront **category browse** via a
+> `join=` request (`Assortments.name`), and the joined-select-fields read path did **not** consult the
+> sidecar — it copied the raw SQL column straight into the response, so the seeded French names stayed
+> English there. That required a real api2 change (the 5th serialization site) — see the
+> [Language Translation Layer](../../api2/features/language-translation-layer.md) 2026-07-21 change history.
 
 ## Key files / entry points
 
@@ -51,8 +57,10 @@ The mechanism itself is the data-driven translation layer documented in
 4. **The field link (what makes it translate automatically).** The translatable `Assortments.name`
    RecordField gets its `Core.RecordFields.translationRecordFieldId` set to the sidecar's `name`
    RecordField. The api2 V2 layer reads `translationRecordFieldId` for every field and automatically
-   returns the sidecar value for the caller's resolved language (falling back to English when absent) —
-   so assortment names now translate **without any api2 code change**.
+   returns the sidecar value for the caller's resolved language (falling back to English when absent).
+   This works with no api2 code change for FK/top-level reads; the storefront's `join=` (joined-select)
+   read path additionally required the 5th-serialization-site fix landed 2026-07-21 (see the correction
+   note above and the Language Translation Layer doc).
 5. **ACL.** A full ACL chain grants the **Base** role read/write on the new record. Because the
    record's `aclDatabase = 'CLIENT'`, the grant lives in each client DB — see
    [ACL Permission Chain](./acl-permission-chain.md).
@@ -75,6 +83,11 @@ The mechanism itself is the data-driven translation layer documented in
 
 ## Change history
 
+- 2026-07-21 — Correction: the "no api2 code change needed" claim held only for FK/top-level reads. The
+  storefront category browse surfaces `Assortments.name` via a `join=` request, and the joined-select read
+  path in api2 did not consult the sidecar (raw column copied through), so seeded fr-CA names stayed English
+  there. Fixed by the 5th-serialization-site change in the Language Translation Layer (V2.php); no change to
+  this sidecar/model/metadata was required. (bala)
 - 2026-06-26 — Initial build: `AssortmentTranslations` sidecar table + `_Model_Client_AssortmentTranslation`
   model + `Core.Records` 332 registration + `translationRecordFieldId` link on `Assortments.name` + Base-role
   ACL chain. Reuses the existing metadata-driven translation layer, so no api2 code change was needed.

package/knowledge/2.0/apps/api2/features/language-translation-layer.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-07-14
+updated: 2026-07-21
 owners: ["jcardinal", "bala"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -110,8 +110,11 @@ RecordField. `buildLookups`/RecordFields-load resolves this into
 `lookupTranslatableFieldByRecordFieldId[sourceRecordFieldId] => {sidecarRecordId, sidecarField}`.
 
 **Read path (PHP `_Model::load()`, NOT a SQL JOIN — deliberate choice).** At every response
-serialization site (top-level full-model, custom-fields, FK child, both inherent-child paths)
-`getTranslatedFieldValue($record, $field, $sourceModel, $defaultValue)` is called. When a non-base
+serialization site `getTranslatedFieldValue($record, $field, $sourceModel, $defaultValue)` is called.
+There are now **five** serialization sites, all of which must route through the helper: (1) top-level
+full-model fields, (2) custom-field options, (3) foreign-key child objects, (4) inherent-child arrays
+(both the specific-fields and the all-fields/expand-everything branches), and (5) **joined-select
+fields** (fields requested from a `join=` table, e.g. `Assortments.name`). When a non-base
 language is active and the field is translatable, it loads the sidecar row for that source row +
 `languageId` (cached per row so multiple fields = one load) and returns the sidecar value if non-null;
 otherwise it returns the English default and queues a fallback warning. The warning now carries the
@@ -119,6 +122,21 @@ missing record's `uuid` in its `identifiers` and dedupes per-field-**and**-per-r
 (key `recordFieldId:uuid`) rather than once-per-field — so every record lacking a translation for a
 field is reported individually.
 
+**Joined-select fields (site 5) — how the source row is reached.** A `join=` field's value arrives as
+a flat SQL column (`$row->{Alias_field}`), so the helper needs the joined row's identity to load its
+sidecar. In the LIST/GET branch of `processRoutePairs` (joined-select SQL build ~L3700-3800,
+serialization ~L4038-4060): when `shouldTranslate()` is true and a selected joined field is translatable
+(`lookupRecordFieldByRecordIdAndField` + `lookupTranslatableFieldByRecordFieldId`), the SQL build also
+SELECTs that joined table's primary key (`` `Alias`.`id` AS `Alias_id` ``), tracked in
+`$joinedTableRecordsByAlias` and `$joinedTablePrimaryKeyAliasByAlias`; the hidden PK column is only
+appended when not already requested (`in_array` guard). At serialization the joined source model is
+instantiated by that PK (`new $joinedSourceModelName((int)$row->$alias)`) and each joined value routes
+through `getTranslatedFieldValue()`. English and API-credential/public traffic is byte-identical (all
+new logic gated behind `shouldTranslate()`, which is false for en / API / public — no extra SQL column,
+no extra model load, output unchanged). This is why a metadata-only sidecar (e.g. `AssortmentTranslations`)
+still needed an api2 code change to translate on the `join=` read path even though FK/top-level reads
+worked with no code change.
+
 **Write path.** On create and update, `extractTranslationWrites()` pulls translatable fields out of
 the write set for a non-base language (so the English source is never overwritten), and after the
 source row saves, `saveTranslationWrites()` upserts them into the sidecar for the current language.
@@ -191,6 +209,25 @@ None — uniform across all clients. The sidecar table + ACL ship via `dbchanges
   (matching `ItemTranslations`) while the source fields are `utf8mb4_0900_ai_ci`. Ad-hoc queries that
   compare a sidecar string against a source string must add `COLLATE utf8mb4_bin` or MySQL throws a
   collation-mismatch error.
+- **GROUP BY gotcha on translatable joined fields (site 5).** The sibling joined-field code pushes each
+  selected joined field into `$groupBy` under aggregates; the hidden PK column added for translation must
+  **also** be pushed to `$groupBy` when `!empty($aggregateFunctionFields)`, or a non-English list request
+  combining a translatable `join=` with an aggregate/`group=` throws MySQL 1055 under `ONLY_FULL_GROUP_BY`.
+  Mirror the sibling `$groupBy[]` push for the PK column.
+- **DISTINCT granularity (site 5).** The list path always runs `SELECT DISTINCT`, so adding the hidden PK
+  to the SELECT can in theory change DISTINCT granularity. Only reachable with distinct + join + a
+  translatable joined field whose rows share values but differ by PK; verified harmless for the real
+  storefront query (a `WHERE` pinning one assortment yields the same 4 rows). Worth checking if a
+  translatable join is added to a query that relies on DISTINCT collapsing duplicate rows.
+- **Per-row cost on translatable-join lists (site 5).** `getTranslatedFieldValue` dereferences the source
+  model key via `_Model __get`, forcing a full row load, so ~2 queries per row (source + sidecar) for
+  non-English translatable-join lists — consistent with the feature's per-row-load tradeoff. Candidate
+  future optimization: let the helper accept a known id instead of a live `_Model`.
+- **Pre-existing (not introduced by site 5): `join=`/`ojoin=` identifiers are unvalidated.** Table/alias/
+  ON-clause identifiers flow from the query string into SQL with no validation/escaping (`V2.php` ~L3416-3463,
+  `parseOptionsJoin` ~L7874-7920). The new PK-select line reuses the same already-tainted `$tableAlias` as the
+  adjacent pre-existing lines, so it adds no new injection class — but the underlying tainted-identifier path
+  is a standing concern flagged for separate follow-up.
 - Reading a possibly-absent magic field off a generic `_Model` needs `array_key_exists(...)` +
   `__get` — `isset()`/`?? null` always read false/null. See
   [_Model magic-field access](../../_underscore/features/model-magic-field-access.md); this was the
@@ -202,6 +239,20 @@ None — uniform across all clients. The sidecar table + ACL ship via `dbchanges
 
 ## Change history
 
+- 2026-07-21 — Wired translation into the **joined-select-fields** path (the 5th and final read-path
+  serialization site) in `V2.php` `processRoutePairs` (joined-select SQL build ~L3700-3800, serialization
+  ~L4038-4060). `join=` fields (e.g. `Assortments.name`) previously copied the raw SQL column straight into
+  the response and never consulted the sidecar, so Compass Canada's already-seeded French assortment/category
+  names stayed English on the storefront category browse. Fix (gated behind `shouldTranslate()`, so en/API/
+  public output is byte-identical): SELECT the joined table's PK as a hidden `` `Alias`.`id` AS `Alias_id` ``
+  (tracked in `$joinedTableRecordsByAlias`/`$joinedTablePrimaryKeyAliasByAlias`, `in_array`-guarded), then at
+  serialization instantiate the joined source model by that PK and route the value through
+  `getTranslatedFieldValue()`. Also fixed a GROUP BY 1055 bug (the hidden PK must be pushed to `$groupBy` when
+  aggregates are present). Noted the DISTINCT-granularity edge, the ~2-queries-per-row cost, and that the
+  pre-existing unvalidated `join=` identifier path is unchanged (no new injection class). No toga2-commerce or
+  DB/metadata change needed — sidecar rows, the `Assortments.name` `translationRecordFieldId` link, and the
+  fr-CA storefront request all already existed. cso: SAFE TO SHIP; php-reviewer: safe after the GROUP BY fix
+  (applied), remaining items non-blocking follow-ups. (bala)
 - 2026-07-13 — Extended the translation layer to item **feature** text: three new sidecar tables
   (`FeatureTranslations`, `ItemCategoryFeatureGroupTranslations`, `ItemFeatureTranslations`; Records
   343/344/345, RecordFields 2442–2457) with `translationRecordFieldId` wired to the source fields,

package/knowledge/clients/nycdoe/features/hold-status-sync.md

@@ -6,8 +6,8 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-07-07
-owners: [mhammontree]
+updated: 2026-07-20
+owners: [mhammontree, sking]
 files:
   - worker/crons/sync/nycdoe/send_ticket_updates.php
   - worker/crons/sync/nycdoe/process_tickets.php
@@ -76,29 +76,55 @@ revert or never reach ServiceNow. It is the status-sync companion to the broader
 - `Hold` has **statusRank 6**, beating `Work in Progress` (rank 5) in the downgrade-guard
   added by TRUE-76812 — so a downstream sync cannot silently demote a held order.
 
-### Status field (ServiceNow side)
-- The SNOW **native `state` field** (human-readable: `On Hold`, `In Progress`, `Assigned`,
-  `Resolved`, …) is the **SLA-bearing, authoritative status**. Read/write this for hold.
-- The custom **`u_status_task`** field is a *separate concept* with a different vocabulary;
-  it can lag/disagree with `state` (observed: native `state="In Progress"` while
-  `u_status_task="Open"`). **Never use `u_status_task` for hold detection.**
-- Numeric `state` codes seen on the wire: INC `On Hold` = **3**, `In Progress` = **2**;
-  RITM `On Hold` = **8**, the scheduled-update state = **"-14"**.
-
-### Outbound (TOGaDesk → ServiceNow)
-- `send_ticket_updates.php` (INC) pushes the hold as `state:3`. The "Assigned → In Progress"
-  auto-start block is **guarded** so it never pushes `state:2` when the local repair order is
-  in any `HOLD_*` status (this is what caused the self-clobber — see Change history).
-- `send_request_item_updates.php` (RITM) pushes `state:"8"` (On Hold) and carries a
-  defensive `HOLD_*` guard on the scheduled-update path so it can't push `state:"-14"` while
-  held.
+### Status fields (ServiceNow side) — CORRECTED 2026-07-20
+> **The earlier "native `state` is authoritative; never use `u_status_task`" model was WRONG**
+> (see Change history 2026-07-20). Both fields exist and both matter, with distinct roles:
+- **`u_status_task` (custom) is the CLIENT-FACING, authoritative incident status** — the field
+  DOE's own ServiceNow views/reporting read (values `Open`, `In Progress`, `On Hold`, …). **A hold
+  MUST be written here or the client never sees it.** It is **writable from any active state**; a
+  controlled test PATCH of `{u_status_task:'On Hold'}` returned success and PERSISTED for days
+  (INC2192842 held 3+ days). **The TOGaDesk inbound sync keys off `u_status_task`**
+  (`process_tickets.php:206` reads `$data->u_status_task`), and the RITM cron checks
+  `u_status_task == 'On Hold'` (`send_request_item_updates.php:224`) — so `u_status_task` **is**
+  the correct field for hold detection.
+- The native **`state`** field is ServiceNow's INTERNAL, SLA-bearing state machine. Setting
+  `state=3` (On Hold) pauses the SNOW SLA clock but is **invisible to the DOE client**. ServiceNow
+  only permits the native On Hold transition **FROM In Progress (`state=2`)** — pushing `state=3`
+  from any other state (Assigned, or already Resolved/Closed/Canceled) is silently rejected (HTTP
+  200, empty body — see the empty-body gotcha). `hold_reason=10` is not a valid choice on this
+  instance and is stored empty/dropped, but native On Hold still sticks without it.
+- Numeric `state` codes seen on the wire: INC `On Hold`=**3**, `In Progress`=**2**;
+  RITM `On Hold`=**8**, the scheduled-update state=**"-14"**.
+
+### Outbound (TOGaDesk → ServiceNow) — INC hold push (worker #1676 + #1677)
+When a DOE repair order is on hold, `send_ticket_updates.php` pushes:
+- **Always set `u_status_task='On Hold'`** — the client-facing field, writable from any active
+  state; on release set it back to `'In Progress'`. This is what actually surfaces the hold to the
+  client. Until the local status stops reverting (the library `qqStatus()` fix below) the outbound
+  never sees `orderIsOnHold=true` and never pushes `u_status_task` at all — hence the deploy
+  dependency.
+- **Set native `state=3` (On Hold) only when the incident is currently `In Progress`** — an SLA
+  pause. Because ServiceNow rejects the transition from any other state (empty-body 200), the cron
+  **defers without advancing `dtSynced`** when the incident is neither On Hold nor In Progress, and
+  re-pushes only once it reaches In Progress.
+- **Closed-incident reconcile:** if the incident is already `Resolved`/`Closed`/`Canceled` on the
+  SNOW side it can no longer be held — the cron **advances `dtSynced`** to accept the remote
+  closure instead of retrying forever. (A canceled incident, INC2190846, had generated **2,392**
+  failed empty-body PATCHes before this.)
+- The "Assigned → In Progress" auto-start block is still guarded so it never pushes `state:2` over
+  a held order (TRUE-79922). Named constants: `SN_STATE_IN_PROGRESS` / `SN_STATE_ON_HOLD` /
+  `SN_HOLD_REASON_DEFAULT` in `send_ticket_updates.php`, plus `SN_TASK_STATUS_ON_HOLD` /
+  `SN_TASK_STATUS_IN_PROGRESS` for the `u_status_task` writes (#1677).
+- `send_request_item_updates.php` (RITM) pushes `state:"8"` (On Hold) and carries a defensive
+  `HOLD_*` guard on the scheduled-update path so it can't push `state:"-14"` while held; its
+  reverse-hold logic keys off `u_status_task == 'On Hold'`.
 
 ### Inbound (ServiceNow → TOGaDesk)
-- `process_tickets.php` determines local status from the native **`$data->state`** field
-  (NOT `u_status_task`): after the `u_status_task` switch — before the completion-preserve
-  step and the `statusRank` downgrade-guard — if native `state` is `On Hold`
-  (case-insensitive) it sets `$newStatus = STATUS_HOLD`. Both inbound "Assigned-force"
-  `patchINC(state:2)` blocks are guarded against a local hold.
+- `process_tickets.php` reads the client-facing **`u_status_task`** field
+  (`$status = $data->u_status_task`, L206) — **NOT** native `state` — and maps `On Hold` to the
+  bare **`STATUS_HOLD`** on the local repair order (this is exactly why the library `qqStatus()`
+  fix below had to recognize bare `HOLD`). Both inbound "Assigned-force" `patchINC(state:2)` blocks
+  are guarded against a local hold.
 
 ### RITM "ETA-in-hold"
 - `send_request_item_updates.php` syncs `scheduled_date` (ETA) to ServiceNow **inside the
@@ -182,9 +208,19 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
   auto-start fought the hold push — `state:3` then `state:2` pushed seconds apart, every run,
   so SNOW never stayed On Hold. Any future auto-start/auto-status block on these crons MUST
   short-circuit when the local RO is in a `HOLD_*` status.
-- **The wrong-field read (root cause #2 of TRUE-79922):** reading `u_status_task` instead of
-  native `state` meant a hold set on the SNOW side never reached TOGaDesk. Hold detection is
-  on native `state` only.
+- **⚠ SUPERSEDED (see Change history 2026-07-20):** TRUE-79922 originally concluded that reading
+  `u_status_task` (instead of native `state`) was a bug and that hold detection should be on native
+  `state` only. That was **backwards.** `u_status_task` is the client-facing authoritative field
+  and **is** the correct hold-detection field — the live inbound cron reads it at
+  `process_tickets.php:206`. Native `state` is the internal SLA state, pushed only as a secondary
+  pause when the incident is In Progress.
+- **The empty-body 200 (nycd3 `data/tasks` PATCH):** `App_Api_NYCDOEV2::patchINC`/`patchRITM`
+  (both hit `/api/nycd3/data/tasks/{number}`) return **HTTP 200 with an EMPTY body (no `result`
+  object)** when a requested native `state` transition is illegal — e.g. pushing `state=3` from a
+  non-In-Progress state. This is silent: no exception, no error field. A canceled incident
+  (INC2190846) generated **2,392** such failed PATCHes before the closed-incident reconcile was
+  added. Gate native `state=3` on `snState == 'In Progress'`, and reconcile
+  Resolved/Closed/Canceled by advancing `dtSynced`.
 - Holds are **never auto-released** — if a SNOW user manually changes status, the outbound
   cron correctly re-asserts On Hold; that is intended, not a bug. Clear the hold in TOGaDesk.
 - **Diagnosing which writer dropped a hold:** `repair_order_history`
@@ -217,6 +253,25 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
   `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
 
 ## Change history
+- 2026-07-20 — **Corrected the authoritative-field model and shipped the INC hold push.**
+  Controlled test PATCHes proved the earlier "native `state` is authoritative; never use
+  `u_status_task`" conclusion (TRUE-79922) **backwards**: the custom **`u_status_task`** is the
+  client-facing authoritative incident status (a `{u_status_task:'On Hold'}` PATCH persisted 3+
+  days, INC2192842), the inbound cron reads it (`process_tickets.php:206`), and the RITM cron keys
+  off it (L224). Native `state` is the internal SLA state and only accepts On Hold **from In
+  Progress** — any other transition returns a silent HTTP-200 empty body (a canceled incident,
+  INC2190846, had racked up 2,392 failed PATCHes). Fix: outbound `send_ticket_updates.php` now
+  always writes `u_status_task='On Hold'` (client-facing; `'In Progress'` on release), sets native
+  `state=3` only when In Progress (SLA pause), and reconciles already-closed incidents by advancing
+  `dtSynced` instead of retrying forever; added `SN_STATE_*` / `SN_HOLD_REASON_DEFAULT` /
+  `SN_TASK_STATUS_*` constants. worker PRs **#1676** (merged — retry-storm/closed reconcile + native
+  state gate) and **#1677** (open — `u_status_task` write). (sking)
+- 2026-07-20 — library **#845**: `App_Model_TogaDesk_RepairOrder::qqStatus()` now recognizes all
+  six `HOLD_*` constants in both "on hold" IN-clauses. The inbound sync writes SNOW On Hold as the
+  bare `STATUS_HOLD` ('HOLD'), which the prior 2-of-6 match dropped, silently recomputing the order
+  to `ORDER_ASSIGNED_AWAITING_SCHEDULING` (the local "hold revert"). Additive only. Must ship with/
+  before worker #1677 — until the local status stops reverting, the outbound never sees
+  `orderIsOnHold=true`. (sking)
 - 2026-07-07 — TRUE-80060 (togadesk portion): closed the **third** DOE hold-failure path — a
   hold set via a note/comment (`Repair::addNotes`) never reaching SNOW. `addNotes` recorded
   status only on `repair_order_notes.newstatus` (never the `repair_orders.status` cache the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.398",
+  "version": "1.0.400",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",