toga-ai 1.0.557 → 1.0.559

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -32,7 +32,7 @@
 | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
 | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
 | [NYCHH Asset-Tag Backfill (worker2)](features/nychh-asset-tag-backfill.md) | Keeps NYC Health & Hospitals (`Client_Nychh`) unit asset tags and MAC addresses synced from NetSuite. | worker2/Worker/Client/Nychh.php, worker2/Worker/Client/Nychh/AssetTagBackfill.php, worker/crons/toga2/netsuite/verify_fulfillment_asset_tag_sync_nychh.php, worker/crons/toga2/netsuite/backfill_all_asset_tags_from_netsuite_nychh.php |
-| [OneUptime Incident → ClickUp Task Sync (Monitor/Oneuptime/SyncIncidents)](features/oneuptime-incident-clickup-sync.md) | An internal/shared TOGA ops feature: worker2 polls the OneUptime API every 15 minutes for **currently-open** incidents and ensures a ClickUp task exists for eac | worker2/Component/Api/Oneuptime/Oneuptime.php, worker2/Worker/Monitor/Oneuptime.php, worker2/Config/production.ini, dbchanges2/Team/2026-08-10a - OneUptime Incident ClickUp Tasks.sql, dbchanges2/Core/2026-08-10b - OneUptime Incident Sync CronJob.sql |
+| [OneUptime Incident → ClickUp Task Sync (Monitor/Oneuptime/SyncIncidents)](features/oneuptime-incident-clickup-sync.md) | An internal/shared TOGA ops feature: worker2 polls the OneUptime API every 15 minutes for **currently-open** incidents and ensures a ClickUp task exists for eac | worker2/Component/Api/Oneuptime/Oneuptime.php, worker2/Worker/Monitor/Oneuptime.php, worker2/Config/production.ini, dbchanges2/Team/2026-08-10a - OneUptime Incident ClickUp Tasks.sql, dbchanges2/Core/2026-08-10b - OneUptime Incident Sync CronJob.sql, dbchanges2/Team/2026-08-12a - OneUptime Incident Tasks Closed Marker.sql |
 | [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php, worker2/Worker/Monitor/Operations.php |
 | [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` owns maintenance of the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-client-data- | worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |

package/knowledge/2.0/apps/worker2/features/oneuptime-incident-clickup-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-10
+updated: 2026-08-12
 owners: ["jcardinal"]
 files:
   - worker2/Component/Api/Oneuptime/Oneuptime.php
@@ -14,6 +14,7 @@ files:
   - worker2/Config/production.ini
   - dbchanges2/Team/2026-08-10a - OneUptime Incident ClickUp Tasks.sql
   - dbchanges2/Core/2026-08-10b - OneUptime Incident Sync CronJob.sql
+  - dbchanges2/Team/2026-08-12a - OneUptime Incident Tasks Closed Marker.sql
 related:
   - ./oneuptime-worker2-monitoring.md
   - ./error-escalation-cron.md
@@ -41,7 +42,7 @@ reporter, smart monitor"; this feature adds the reader half.
   open incidents (filters out resolved **in PHP**).
 - `worker2/Worker/Monitor/Oneuptime.php` — `_Worker_Monitor_Oneuptime::SyncIncidents(): string`,
   the cron action. Returns a structured summary string on every path
-  (`incidentsPolled` / `created` / `skipped` / `failed`).
+  (`incidentsPolled` / `created` / `skipped` / `failed`, plus `closed` / `closeFailed`).
 
 ## How it works
 
@@ -68,6 +69,36 @@ worker2 way to prevent duplicate ClickUp tasks under overlapping cron runs — p
 from `_Worker_Infrastructure_Errors::createClickupTask`. `INSERT … WHERE NOT EXISTS` is
 **insufficient** under REPEATABLE READ; the UNIQUE key is the concurrency arbiter.
 
+### Close pass (auto-close on incident resolution)
+
+The sync was originally **create-only**; a ClickUp task opened for an incident was never
+closed once the incident resolved, so a recurrence (a *new* OneUptime incident with the same
+monitor title) opened a second task that **looked like a duplicate** but was not (see the
+gotcha below). `SyncIncidents()` now runs a **close pass first** each cycle, independent of how
+many incidents are currently open, via `closeResolvedIncidents()`:
+
+- SELECT ledger rows `WHERE clickupIdentifier IS NOT NULL AND dtClosed IS NULL`.
+- For each, call `_Component_Api_Oneuptime::getIncidentResolvedState(string $incidentId): ?bool`
+  — `true` = resolved, `false` = confirmed still open, `null` = unknown. **Only a definite
+  `true` closes a task**; `null` means "leave alone" so an ambiguous read never closes a live
+  incident's ticket. This lookup uses a new per-incident route (`ROUTE__INCIDENT_ITEM`,
+  `/api/incident/{id}/get-item`, still a PLACEHOLDER) because resolved incidents no longer
+  appear in `getActiveIncidents()` and must be re-checked individually. A 2xx response missing
+  `currentIncidentState.isResolvedState` logs a warning (incident id only, no body/PII) so a
+  wrong placeholder shape is detectable rather than silently returning `null` forever.
+- To close, PUT `/task/{id}` with the list's **closed-type status NAME** (ClickUp has no
+  generic close verb — see gotcha), resolved per-list via `GET /list/{id}` `statuses[]` where
+  `type == 'closed'`, cached per run in `$closedStatusByList`. Then stamp `dtClosed = NOW()`.
+- `dtClosed` is stamped **only after a successful close**, so a failure retries next run.
+  Per-incident failures are caught (`Throwable`) + `error_log` + counted, not fatal; whole-run
+  failures (e.g. no sprint list) still re-throw for queue retry.
+- All new ClickUp calls in the close path are throttled with
+  `usleep(_Worker_Clickup_Project::RATE_LIMIT_DELAY_US)` (120ms = 500/min), matching
+  `_Worker_Infrastructure_Errors`.
+
+Helpers added: `closeResolvedIncidents()`, `closeClickupTask()`, `resolveClosedStatusForList()`,
+`markClosed()`, and the `$closedStatusByList` per-run cache.
+
 ### Target list = current sprint list
 
 The ClickUp task lands in the **current sprint** list, resolved by `GET /folder/{id}` and
@@ -80,8 +111,11 @@ id is read from `[clickup] sprint_folder_id` with a constant fallback (`90020432
 
 `OneUptimeIncidentTasks` (CREATE on the **Team** cluster — chosen over Logs;
 `dbchanges2/Team/2026-08-10a`): `id`, `uuid char(36)`, `dtCreated`,
-`oneuptimeIncidentId varchar(64)`, `clickupIdentifier varchar(64) NULL`. The **UNIQUE KEY on
+`oneuptimeIncidentId varchar(64)`, `clickupIdentifier varchar(64) NULL`, `dtClosed DATETIME NULL` (added
+`dbchanges2/Team/2026-08-12a`, `AFTER clickupIdentifier`). The **UNIQUE KEY on
 `oneuptimeIncidentId`** is the dedup contract, same rationale as `IssueClickupTasks`.
+`dtClosed` marks a ledger row whose ClickUp task has been closed following incident
+resolution (NULL = still open / not yet closed).
 
 ## Provisioning / deploy
 
@@ -104,13 +138,30 @@ id is read from `[clickup] sprint_folder_id` with a constant fallback (`90020432
 
 ## Gotchas / known issues
 
-- **Incident-list route/fields are PLACEHOLDER** (`/api/incident/get-list`) pending
-  confirmation against the live OneUptime instance. Do not enable the cron (`isActive = 1`)
-  until the route and returned fields are verified.
+- **Same-title ClickUp tickets from OneUptime are expected on incident recurrence.** Dedup is
+  **per-incident-id, not per-title**: the `oneuptimeIncidentId` UNIQUE key + claim-first insert
+  only prevents duplicates for the *same* incident id. Two tasks with the same monitor title
+  minutes apart are two *different* OneUptime incidents, not a dedup failure. The symptom that
+  looked like duplication was actually the missing auto-close: incident #1 resolved but its
+  task never closed, then a fresh incident opened a second task. Auto-close (above) closes
+  #1 on resolution so a recurrence (#2) is the only open ticket.
+- **ClickUp has no generic "close" verb.** A task is closed by PUTting the exact NAME of the
+  status whose `type == 'closed'`, which is **list-specific** — resolved per-list from
+  `GET /list/{id}` `statuses[]` and cached per run.
+- Throttle every ClickUp call in the close path with `usleep(RATE_LIMIT_DELAY_US)` (120ms) or
+  rate-limit failures degrade into `closeFailed` counts.
+- **Both OneUptime routes remain PLACEHOLDERS** — the incident-list route
+  (`/api/incident/get-list`) *and* the new per-incident route (`/api/incident/{id}/get-item`,
+  `ROUTE__INCIDENT_ITEM`) are unverified against the live OneUptime API. Auto-close **fails
+  safe** (only a definite resolved `true` closes anything; `null` leaves the task alone) but
+  won't fire until the route/field shape is confirmed. Do not enable the cron (`isActive = 1`)
+  until the routes and returned fields are verified.
 - **OneUptime cannot filter open-only server-side here** — resolution filtering happens in
   PHP, so the returned field set must actually carry the resolved/open state.
 - Ship the worker2 code deploy and both dbchanges2 migrations **together**; the cron row is
   inert until the action is deployed and the ledger table exists.
+- **Deploy order for the auto-close change:** run `dbchanges2/Team/2026-08-12a` (the `dtClosed`
+  column) **before** the worker2 code deploys — the new close-pass queries reference `dtClosed`.
 
 ## Change history
 - 2026-08-10 — Built: first OneUptime API *reader* (`_Component_Api_Oneuptime`) + the
@@ -118,5 +169,13 @@ id is read from `[clickup] sprint_folder_id` with a constant fallback (`90020432
   ClickUp task in the current sprint list; new `OneUptimeIncidentTasks` ledger (Team cluster,
   UNIQUE on incident id) and a seeded-inactive `*/15 * * * *` cron row. Incident-list route
   left PLACEHOLDER pending confirmation. (jcardinal)
+- 2026-08-12 — Built: auto-close of ClickUp tasks when the OneUptime incident resolves (the
+  sync was create-only). Added `dtClosed` marker to the ledger
+  (`dbchanges2/Team/2026-08-12a`), `_Component_Api_Oneuptime::getIncidentResolvedState()` +
+  PLACEHOLDER per-incident route, and a close-first pass in `SyncIncidents()` (resolves each
+  list's `type=='closed'` status name, PUTs it, stamps `dtClosed` only on success, throttled
+  120ms; return string now carries `closed`/`closeFailed`). Discovered: apparent duplicate
+  same-title tickets were two distinct incidents — dedup is per-incident-id, not per-title; the
+  real gap was the missing close path. (jcardinal)
 </content>
 </invoke>

package/knowledge/sessions/2026-08-11-TRUE-80282-Toga2-View-Cancel-button-mhammontree.md

@@ -0,0 +1,107 @@
+---
+type: session
+slug: TRUE-80282-Toga2-View-Cancel-button
+title: Wire and verify the TOGa View cancel-subscription button for Rate
+author: mhammontree
+repos: [_underscore, api2, toga2-view, worker2, dbchanges2, test]
+framework: "2.0"
+client: rate
+status: active
+created: 2026-08-11
+updated: 2026-08-11
+---
+
+# Session: TRUE-80282-Toga2-View-Cancel-button
+**Date:** 2026-08-11
+**Project/Repo:** _underscore · api2 · toga2-view · worker2 · dbchanges2 · test (2.0)
+**Task:** Wire the already-shipped TOGa View "Cancel Subscription" button to a real backend so a Rate customer can self-cancel a tech-support subscription — stopping future PayPal charges while keeping access to the end of the period already paid for — then prove it end-to-end on beta. Supersedes the 2026-08-04 and 2026-08-06 saves of this ticket.
+
+---
+
+## What WORKED
+
+- **End-to-end cancellation verified on beta (api.beta.togahub.com), 2026-08-06.** `POST /v2/subscriptions/cancel` with `{"entitlementUuid":"9c827c5c-7779-b260-b0cb-1d047a937a03"}` returned **200** with `data.subscriptions.cancel = {isCancelled: true, dateCancel: "2026-08-27", message: "Your subscription will be canceled on August 27, 2026. You'll have access until then."}`. A real PayPal **sandbox** agreement was cancelled.
+- **Database state verified directly** (`Client_Rate.Subscriptions` id 84, sandbox-dev): `isActive = 1`, `dateEnd = 2026-08-27` (untouched), `dateCancelled = 2026-08-27`, `dtRequestToCancel = 2026-08-06 12:02:40`. That is the business rule proven — the customer keeps what they already paid for.
+- **UI verified after refresh:** card renders "Monthly · $9.97/mo · **Service ends by Aug 27, 2026**", the Cancel button is gone, and the "you can cancel anytime" note goes with it.
+- **Verification script passes 34/34** — `test/@Mark/Rate/verify_cancel_subscription.php`. Covers the effective-date logic (incl. past/missing `dateEnd`), warranty-vs-tech discrimination, the borrower/item/subscription join, all six `cancelApi` pre-PayPal rejections (each asserting nothing was written), the idempotent double-click return, and the NULL-rollback regression. Never calls PayPal. Scratch rows self-clean (`leftover rows: 0` every run).
+- **Field grants confirmed present, so worker2's wind-down guard will work.** `Subscriptions.dateEnd` (1696), `dateCancelled` (1697), `dtRequestToCancel` (1698) are all granted to role `Base` in **both** sandbox-dev and production. Decisive link: `Client_Rate.Apis` uuid `fe4df2e2-f685-4c03-a2d7-2015502dadc9` — exactly `_Worker_Rate::API_UUID_RATE` — carries **both** `Base` and `API`, so the worker inherits Base's grants.
+- **Hardcoded `Core.RecordFields` id literals verified** — 1696/1697/1698 on `recordId 253` (route `subscriptions`) are correct per environment, so the migration's use of literals is sound.
+- **All three migrations landed on beta:** `CustomRecordScripts` id 2 (`POST` / route `cancel` / `cancelApi`, granted to `Base`), and `Core.CronJobs` id 27 (`Rate/ExpireCancelledSubscriptions`, `0 3 * * *`, active). `2026-08-04a` was a confirmed no-op (grants already existed).
+- **Unblocked Paulina's re-test** by fast-forwarding the wind-down on beta: `UPDATE Subscriptions SET isActive = 0 WHERE id = 84 AND dtRequestToCancel IS NOT NULL AND dateCancelled IS NOT NULL`. Verified result `isActive = 0` with both cancellation dates intact, so the audit trail survives. Chose this over deleting the rows — see Decisions.
+- **All TRUE-80282 branches pushed** (verified 2026-08-11, each 0 ahead / 0 behind its origin): `_underscore` `2fc8e450`, `dbchanges2` `df48181`, `toga2-view` `76d7d49`, `api2` `47a4060`. `worker2`'s TRUE-80282 work is `6de7fa7` on its own branch.
+- **Knowledge captured and pushed** — `529a5d5` on `origin/_main` (9 docs incl. the elevated `2.0/standards/backend-testing.md`), plus the 08-04 capture `54fe73b`.
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **`empty()` / `isset()` on a `_Model` field.** `_Model` defines `__get`/`__set` but **no `__isset`** (Model.php:464,481). Proven empirically: `$s->dateCancelled` returns `'2026-08-26'` while `empty($s->dateCancelled)` returns **true** and `isset()` returns **false**. This silently broke the cancel idempotency guard — `$isRequested` was permanently false, so an already-cancelled subscription never short-circuited, every click re-called PayPal, and the crash-recovery branch wiped the earlier attempt's intent instead of preserving it. **Use `trim((string) $field) !== ''`.** ⚠ Already documented in `2.0/apps/_underscore/features/model-magic-field-access.md` — reading it first would have prevented the bug.
+- **`_Config::<group>('key', false)` to survive a missing group.** It still throws. In `_Config::__callStatic` the missing-**group** branch tests `$properties[0]` *before* `array_shift()` has run, so it sees the property NAME (truthy) rather than the `false` flag and falls through to the throw at Config.php:85. The optional-property form only rescues a missing key inside a group that **exists**. **Guard with `_Config::isGroupSet('<group>')` first.** Symptom before the fix: EO-1 with a stack trace to a logged-in customer.
+- **Reading a scripted endpoint's response as `data.<script>`.** The real shape is **`data.<recordRoute>.<scriptRoute>`** — ours is `data.subscriptions.cancel` — and the key is the script's **route** (`cancel`), not its phpMethod (`cancelApi`). Reading `data.cancel` made the result `undefined`, so the frontend threw and told the customer the cancellation had **failed** while the backend had already cancelled at PayPal and written correct state. Precedent that had it right: `toga2-view/src/pages/ZipValidation/api/zipValidation.ts:88` → `response?.data?.entitlements?.["warranty-availability"]`. Local tests structurally cannot catch this — they call the model method directly and never traverse the envelope.
+- **Merging `_underscore` to `_sandbox-dev` and expecting api.beta.togahub.com to have it.** That environment runs **`ENVIRONMENT=beta`**, so `.ebextensions/git.php` clones `_underscore` branch **`_beta`** (`'_' . strtolower($env)`), because there is **no `.ebextensions/git.beta.json`** to override it. A `Config/sandbox-dev.ini` and a `git.sandbox-dev.json` (which pins `branch: _sandbox-dev`) both exist and point at the same host, apparently from a rename — they are **not** what that environment reads. **This cost two deploy cycles.** Diagnostic rule: `Call to undefined method _Model_Client_Subscription::cancelApi()` means the **client override class is not on the deployed server** — `class_exists` fails so V2 declines the override at V2.php:3155-3162 and falls back to the base model. It is not a code bug. ⚠ Also already documented, in `2.0/apps/api2/features/environment-variable-drives-underscore-branch.md`.
+- **Assuming a merge to `_underscore` is a deploy.** api2 and worker2 clone `_underscore` at **their own build time**, so `_underscore` reaching a branch changes nothing until the consuming tier is redeployed.
+- **Raw SQL `UPDATE` then reload a model to assert** (in the test script). Returned the **stale pre-update row**: `_Model::load()` consults `_Database::$_modelCache`, which is a **separate cache from `useQueryCache(false)`** and is flushed **only** by `_Model::save()`. Two assertions failed against correct code. Fix: mutate the loaded model **in memory** when testing a pure function.
+- **Defaulting the test script's Core schema to `'Core'`.** Locally `Core` is the **1.0** database and `Core_2` is the 2.0 one — verified: `Core` has 0/2 of `Records`+`RecordFields`, `Core_2` has 2/2. The fallback does not error; it silently registers the wrong schema and every Core lookup returns nothing. Fix: probe for the metadata tables and **assert** the result.
+- **Cross-schema JOIN from `Client_Rate` to `Core` on the prod cluster.** Refused outright: `Unknown database 'Core'`. Live confirmation of why `Client_*` migrations cannot subselect `Core`.
+- **Redacting a JSON *array of objects* by filtering top-level keys.** The filter keyed on `password`/`username`, but the top level was numeric indices, so nothing matched and **a GitHub PAT was printed into the session transcript**. Extract the specific keys you want instead of filtering a shape you have not inspected. The PAT lives in `api2/.ebextensions/git.json` (committed) — **treat as compromised and rotate**.
+- **Expecting a mid-session MCP reconnect to restore tools.** The TOGa database tools showed "connected" on the developer's side but were not present in the running session; `ToolSearch` found no match. The tool list resolves when the session picks up the server, so a session restart is needed.
+
+## Not tried yet (candidates for next session)
+
+- **The entire worker2 half — production-only.** `handleSubscriptionDeactivation`'s wind-down guard, the webhook `dateCancelled` reconciliation stamp, and `Rate/ExpireCancelledSubscriptions`. Non-prod receives **no** PayPal webhooks (no SQS on non-prod worker2; PayPal delivers to one configured URL) and has **no cron scheduler**, so none of it is reachable on beta. Gated on **TRUE-80575**.
+- **The expiry cron's selection logic.** The manual `isActive = 0` update reproduced the cron's **end state** only. The cron would **not** have selected row 84 at that point — its `WHERE` requires `dateEnd < today` and `dateEnd` is 2026-08-27, so the wind-down was fast-forwarded ahead of schedule on purpose.
+- The idempotency guard against the real stack. Now unreachable through the UI (hiding the Cancel button is what prevents the second click) — would need a direct API call. Covered by section 6 of the local script.
+- The `Logs.Issue.clickupPriority` migration (offered, not written).
+- Committing `verify_cancel_subscription.php` to the `test` repo.
+- Porting the Core-DB auto-detect fix into `test/@Mark/Rate/verify_wholehome_per_address_guard.php`, which still has the `?: 'Core'` latent bug (harmless only because nothing in it reads Core metadata).
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `_underscore/Component/Api/Paypal/Paypal.php` | committed `2fc8e450`, **pushed** | `isGroupSet` guard; 422 narrowed to `SUBSCRIPTION_STATUS_INVALID` |
+| `_underscore/Model/Rate/Subscription.php` | committed `2fc8e450`, **pushed** | two-phase write; `empty()` replaced with cast-string compare |
+| `worker2/Worker/Rate.php` | committed `6de7fa7` on branch `TRUE-80282` | wind-down guard, webhook confirmation, extension guard, expiry cron, unconfirmed reporting |
+| `dbchanges2` × 3 | committed `df48181`, **pushed** | field grants (no-op where present), record script, cron |
+| `api2/Config/beta.ini` | committed `47a4060`, **pushed** | `[paypal]` sandbox section — correct file, since that env reads `beta.ini` |
+| `toga2-view` × 11 | committed `6db0aec`, **pushed** | button wiring, "Service ends by", error surfacing |
+| `toga2-view` GetSupport api + viewModel | committed `76d7d49`, **pushed** | envelope path fix (`data.subscriptions.cancel`) |
+| `test/@Mark/Rate/verify_cancel_subscription.php` | **untracked** | 34/34 passing; not committed anywhere |
+| `_underscore` 3 vendored CSS files | modified, deliberately uncommitted | someone else's `-webkit-` prefix cleanup — not this ticket |
+| `worker2/Config/dev-markhammontree-laptop.ini` | modified, deliberately uncommitted | personal local config |
+| `toga2-view/src/styles/index.css` | modified, deliberately uncommitted | line-endings only |
+| `C:\WWW\TRUE-80282-Cancel-Plan.md` | rev 2 | team review doc |
+| `Client_Rate.Subscriptions` id 84 (beta data) | `isActive = 0` | manually fast-forwarded so Paulina can repurchase; cancellation dates preserved |
+
+**Note on `worker2`:** the working tree has moved to branch **`TRUE-80575`** (HEAD `9001413` "Stop the ACTIVATED handler duplicating the…", 1 commit unpushed). That is the next ticket's work, not TRUE-80282's.
+
+## Decisions made
+
+- **Two-phase write: `dtRequestToCancel` before the PayPal call, `dateCancelled` after.** The two columns carry different facts (asked vs confirmed), making "requested but unconfirmed" an honest, recoverable state. Rejected: stamping both up front, which reports "cancelled" the moment anything fails after the commit while PayPal keeps billing.
+- **The webhook confirms `dateCancelled` if api2 didn't** (Rohan's suggestion). PayPal sends the event regardless of what happened to our request, so it doubles as the reconciliation channel. Rejected: a second api2 scripted endpoint callable by the API role — new authenticated surface on a destructive action for a millisecond-wide window.
+- **The expiry cron requires `dateCancelled IS NOT NULL`.** An unconfirmed cancellation may still be billing, so deactivating it would take a customer's money *and* their access. Leaving it active only costs us money.
+- **Deactivate rather than delete row 84 on beta.** A single reversible field change, semantically identical to what the cron does, and it preserves the only end-to-end evidence that the cancel worked. Rejected: deleting the Entitlement/Subscription rows — destructive, needs children-first ordering around `EntitlementSalesOrders`/`SalesOrders`, and loses the audit trail. Also noted the premise was probably wrong: nothing blocks a second *tech* purchase, since the duplicate guard is warranty-only.
+- **Unwrap the response envelope in the api layer, not the viewModel.** Keeps envelope knowledge in one place and gives the caller a single result to check — matching `zipValidation.ts`.
+- **Did not promote the `_Model` `__isset` trap to a standard.** `model-magic-field-access.md` already owns the subject; a second home would fragment the rule. Cross-linked to the `save()` field-diff trap as a sibling.
+- **Kept `dbchanges2/Client_Rate/2026-08-04a` even though it is a no-op** where the grants exist — harmless, re-runnable, documents the dependency, and still protects a fresh client blank.
+- **Reported rather than acted on the ACL exposure.** Pre-existing, and fixing permissions platform-wide is outside this ticket.
+- **Production deployment order** (established this session): TRUE-80575 first → merge `_underscore` → redeploy **worker2** → redeploy **api2** (no config change needed; `production.ini` already has `[paypal]` with the live pair and the matching `client_id`) → `dbchanges2 Client_Rate/2026-08-04a` then **`04b`** → `Core/2026-08-04a` → `toga2-view`. worker2 must precede `04b` because in production the webhook genuinely fires, so an endpoint live against old worker2 code truncates the first customer's coverage to today. Code must precede `04b` or the route resolves to a non-existent `phpMethod`.
+
+## Blockers
+
+1. **TRUE-80575 is the gate on the remaining verification** (in progress — `worker2` branch `TRUE-80575`, HEAD `9001413`, 1 commit unpushed). Until `Rate/Webhook` jobs stop dying on the watchdog, `BILLING.SUBSCRIPTION.CANCELLED` is not processed in production, so the reconciliation channel — the safety net for a crash between the PayPal call and the `dateCancelled` write — is dead there.
+2. **⚠ Security, needs its own ticket.** In `Client_Rate` (sandbox-dev **and** production) role `Base` has `AclRecordPermissions` on recordId 253 with allowCreate/Read/Update/Delete all = 1 and **`appId`/`indirectRecordId` both NULL — no scoping** — plus `isWritable = 1` on `isActive`, `dateEnd`, `dateCancelled`, `dtRequestToCancel`, `token`. Every portal customer carries `Base`. Per the ACL data that permits extending one's own `dateEnd` (free coverage), un-cancelling, overwriting the PayPal `token`, deleting rows, and possibly acting on another customer's subscription. **Not exploited — permission tables were read only.** It directly undermines this feature's server-side date computation.
+3. **Error capture is broken on sandbox-dev.** `Logs.Issue` lacks `clickupPriority`, so the capture INSERT fails (MySQL 1054) and **no error is persisted on that environment** — it only appears inline as `identifiers.captureFailure`. From `_underscore` `4ebe12fe` shipping without its migration. jcardinal's 2026-08-05 `Logs` migration added `status`/`dtAutoResolved`/`baseline*` but **not** `clickupPriority`, so this is still open. Anyone debugging on beta is flying blind.
+4. **Two credentials need rotating**: the live PayPal `client_secret` (committed in `api2/Config/production.ini` and `worker2/Config/production.ini`) and the GitHub PAT in `api2/.ebextensions/git.json`.
+5. **Branch-target question for Rohan.** He specified `_sandbox-dev` for api2 and `_underscore`, but this session proved api.beta.togahub.com runs `ENVIRONMENT=beta` and therefore reads `_beta`. Settle which is intended before opening PRs — either merge to `_beta`, or add a `.ebextensions/git.beta.json` pinning `branch: _sandbox-dev` (a deploy-config change, not a merge).
+
+## Exact next step
+
+> The developer is on **TRUE-80575** (`worker2` branch `TRUE-80575`, HEAD `9001413`, 1 commit unpushed) — finish and deploy that first, because it is blocker 1 for trusting TRUE-80282 in production.
+>
+> To close out TRUE-80282 afterwards:
+> 1. Settle blocker 5 with Rohan, then open the PRs against the branch each environment actually reads. `dbchanges2` → `_main`.
+> 2. Follow the production deployment order in Decisions — **worker2 before `dbchanges2 Client_Rate/2026-08-04b`**, which is what switches the endpoint on.
+> 3. Decide whether `test/@Mark/Rate/verify_cancel_subscription.php` is committed to the `test` repo (it is the regression test for both bugs fixed in `2fc8e450`, which `2.0/standards/backend-testing.md` requires).
+> 4. File tickets for blockers 2–4.
+
+---
+_Saved by /session-save on 2026-08-11_

package/knowledge/sessions/2026-08-11-TRUE-80824-Fulfill-And-Ship-mhammontree.md

@@ -0,0 +1,112 @@
+---
+type: session
+slug: TRUE-80824-Fulfill-And-Ship
+title: GroWrk Fulfill & Ship production outage, NetSuite field prefill, unreadable FedEx errors
+author: mhammontree
+repos: [dbchanges2, _underscore, toga2-supply, test]
+framework: "2.0"
+client: growrk
+status: active
+created: 2026-08-11
+updated: 2026-08-11
+---
+
+# Session: TRUE-80824-Fulfill-And-Ship
+**Date:** 2026-08-11
+**Project/Repo:** dbchanges2 + _underscore + toga2-supply (2.0)
+**Task:** Fulfill & Ship went live for GroWrk and returned EO-1/500 on every save — find and fix the production cause, then prefill Reference 1/2 and Phone # from NetSuite.
+
+---
+
+## What WORKED
+
+- **Root-caused the production 500.** `POST`/`PUT /v2/item-fulfillments` failed with `EO-1`, error id `95-24`. Traced via `Logs.Event` (issueId 166, clientId 33) to MySQL **1054 Unknown column 'ServiceRequests.c_externalReferenceId'**, raised inside `_Model_Growrk_ItemFulfillment::sendGrowrkTrackingInfo()` called from `postPost()`. Evidence: 6 events, ids 38967–38971, sub-second execution times.
+- **Found the actual cause: an entire commit never executed.** `dbchanges2/Client_Growrk/2026-05-28.sql` is committed on `_main` as **af30e63** (rgirish97, 2026-06-02) but none of its statements exist in production. Verified: ServiceRequests `c_` columns 0 of 2; `CustomRecordFields` recordId 35 → 0 of 2; `ServiceRequestTypes` 2 of 19; `UnitDispositions` 1 of 5; **none of the file's 7 fixed UUIDs present**. Its two siblings from the same commit (`Core/` and `Client/2026-06-01 - UnitTypes.sql`) are also absent — `Core.Records` 315 and `Client_Growrk.UnitTypes` do not exist.
+- **This corrected an existing knowledge doc.** `clients/growrk/features/units-netsuite-custom-fields.md` blamed the same-day Units `c_lastModified` 500 on a *partially applied* `_modules/netsuite/2026-07-10a`. Wrong — lines 26–31 of the never-executed `2026-05-28.sql` add those same five Units columns. One skipped commit explains both incidents.
+- **Explained why it surfaced when it did.** The `postPost`/`postPut` `ApiPayloadInterceptors` (recordId 28, both `isActive=1`) landed on 2026-07-31, and the 2026-08-05 NetSuite onboarding switched GroWrk onto `_Model_Growrk_*`, which declare the `c_` fields. `_Model` SELECTs every declared field, so the missing column became a hard failure.
+- **Confirmed no data cleanup was needed.** `ItemFulfillments MAX(id)` stayed at 7645 (2026-08-05) across ~36 failed attempts — the exception rolls the insert back. Only auto-increment was burned.
+- **Fixed production.** Mark ran the safe subset of `2026-05-28.sql` by hand (lines 1–3, 5–6, 8–24, 39, 49–52) plus a lookup-based ACL insert. Verified end state: 2 columns; `CustomRecordFields` ids 33/34 with the original fixed UUIDs; **exactly one** ACL grant per field (roleId 1, writable); `ServiceRequestTypes` 19; `UnitDispositions` 5; `AclFieldPermissions` 2087 present.
+- **Save confirmed working in production.** ItemFulfillment **7676** created 2026-08-10 14:41:39 with `shipToAddressId` 31834 and `dtSubmitted` NULL (correctly pending). Zero new `Logs.Issue` 166 events afterwards.
+- **Found and fixed missing UPS service codes.** `ShippingMethods.code` → `shippingMethodCode` → `$shipmentRequest->service` → UPS `Service.Code`. Both GroWrk UPS rows held `''` → UPS **120500**. The dropdown renders `name`, so the UI looked normal. Set id 3 = `03` (Ground), id 4 = `01` (Next Day Air); verified in prod.
+- **Built the NetSuite prefill.** Reference 1 = `SalesOrders.number`; Reference 2 = NetSuite `otherRefNum` via the `PurchaseOrders_SalesOrders` bridge; Phone = NetSuite `addrPhone`. Verified on beta: Reference 1 `281144` and Reference 2 `TEST` both populated (`sales-orders` response, transactionId 7f75cc89).
+- **Fixed unreadable FedEx errors.** Added `inflateIfCompressed()` to `Fedex.php`. Regression suite `test/@Mark/true-80824-fedex-inflate-test.php` — **18/18 pass**, drives the private method by reflection against the real class, no DB/network/bootstrap.
+- **Validated the dbchanges2 migrations for real.** All four files run **3×** against a throwaway local MySQL 8.0.44 DB with production table shapes: exit 0, no errors, **zero duplicate rows**, correct end state. Scratch DB dropped.
+- **Verified the ACL prerequisite empirically.** `internalApiRequest()` runs on the caller's session, so ACL applies, and the sync is roleId 3 (`API`). `Addresses.phoneNumber` = `Core.RecordFields` **990** had no role-3 grant on 20 of the 22 netsuite-module clients (Compass and Prudential already had it).
+- All three repos committed and pushed on branch `TRUE-80824`; verified clean, 0 ahead / 0 behind.
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **Looking the failing request up by the envelope's `transactionId` in `Logs.Api`** — returned 0 rows. `Logs.Api.transactionId` stores a **timestamp string** (e.g. `2026-08-10 12:54:08.752`), not the API transactionId. Pivot on `error.id` instead.
+- **Reading `error.id` `95-24` as `Logs.Issue.id` 95** — wrong. Issue id 95's newest event is `eventNumber` 6 from 2026-08-05. The format is `Issue.issueNumber`-`Event.eventNumber`; issueNumber 95 is **Issue id 166**.
+- **Searching `Logs_Growrk.Api` / `Logs.Api` for the outbound FedEx call** — 0 rows. The FedEx ship call runs `$api->setLogging(false)` ([Fedex.php:280](_underscore/Component/Library/Carriers/Fedex/Fedex.php#L280)). Its own docblock already notes this as a dead end. Deliberate: success responses carry the full base64 label. UPS *is* logged (direction `OUT`, `hostname LIKE '%ups.com%'`) — FedEx is not.
+- **Reconstructing the gzipped FedEx body from the pasted JSON** — abandoned. `json_decode` failed with *"Control character error, possibly incorrectly encoded"*; the CP1252→bytes round trip could not be reproduced through the tooling, and the stream was truncated anyway. Do not retry — deploy the decode fix and re-trigger instead.
+- **Theory: the executor skips files whose name lacks the `YYYY-MM-DD<letter>` suffix.** Disproved — other suffix-less files in the same folder (`2024-11-13.sql`, `2026-02-23.sql`) did apply.
+- **Theory: the file was skipped because it was back-dated** (dated 2026-05-28, committed 2026-06-02, i.e. behind the executor's watermark). Disproved — its two siblings dated **2026-06-01** in the same commit are *also* missing. The whole commit was skipped.
+- **Theory: a date-window gap in the executor.** Disproved — `Client_Growrk/2026-06-17` and `2026-07-31` both applied, straddling the skipped file. Isolated to commit af30e63.
+- **Using `Client_Rate` as a reference for correct UPS codes** — every code there is `NULL`, FedEx included. `Client_Compass` has UPS Ground = `'GND'`, which is **not** a valid UPS REST service code (looks like a NetSuite ship-method abbreviation in the wrong column). No client in production had a known-good UPS code.
+- **Editing `Client_Rate.ShippingMethods` instead of `Client_Growrk`** — the row ids collide across client databases (in `Client_Rate` id 1 is *FedEx* Ground), so UPS numeric codes `03`/`01` were written onto FedEx rows. Reset afterwards. Scope every such `UPDATE` by `shippingCarrierId` and `name`, not id alone.
+- **A single combined `ALTER TABLE ServiceRequests ADD a, ADD b`** — fails as a whole with **1060** if either column already exists (both are already in production), adding neither and aborting the file before its inserts.
+- **`INSERT … WHERE NOT EXISTS` alone for ACL rows** — treats an existing `isWritable = 0` row as success, leaving a read grant with no write: a "successful" migration that still fails the write with `EV-9`. Needs `UPDATE`-then-`INSERT`.
+- **CodeRabbit's request to remove the UTF-8 guard from `inflateIfCompressed()`** (attempt `gzuncompress`/`gzinflate` on every non-gzip payload, "first non-empty result wins"). **Rejected on measurement:** fuzzing **100,000** valid-UTF-8 bodies, `gzinflate()` returned a **non-false, NON-EMPTY** result on **257** (~1 in 390) — one 64-byte JSON error became 403 bytes of binary junk. The non-empty rule does not help because the junk is non-empty. Four of those inputs are now `fp:` regression cases. **Do not remove that guard.**
+- **`Glob` with pattern `Client_Growrk/*.sql`** returned no files (forward slash), which briefly suggested the migration didn't exist. It did. Use `**/name*.sql`.
+- **My first validation harness** treated any stdout from a migration as an error, misreading the guard's `SELECT 1` output as a failure on pass 2. It isn't — exit 0, zero occurrences of "error".
+
+## Not tried yet (candidates for next session)
+
+- **Deploy and read the real FedEx 400.** Everything about that failure is still unknown.
+- **Rule out the 36-char `reference2`** as the 400's cause — the failing call sent `Vincent Reynoso NGnyAkJ9pS12j3x8HEkr` (36 chars, one over the carrier cap). Now truncated to 35.
+- **Carrier prefill from NetSuite** (the third of Skyler's asks, not built). Lead: `ShippingMethods.c_netsuiteInternalShipMethodId`, which holds real values in GroWrk (`318627`, `318628`, `421872`). Would need the SO sync to import that id — not yet traced.
+- **Harden `sendGrowrkTrackingInfo()`** so a side-effect notification in `postPost` cannot 500 the primary write. The webhook call is already in a try/catch; its two raw queries are not.
+- **`CURLOPT_ENCODING` in `_ApiRequest`** — the systemic fix for the whole gzip class, across every integration. Shared HTTP client, needs its own review. The UPS client has the same latent gap.
+- **GroWrk UPS methods 11/12** (`Overnight Standard`/`Overnight Priority`) — still `NULL`, FedEx names on UPS rows duplicating Next Day Air. Needs a data-owner decision: real codes or delete the rows.
+- **Duplicate `SalesOrders`** — 2,787 of 7,776 NetSuite ids in `Client_Growrk` (36%), worst 139 rows; historic, none since 2026-08-01. Causes `EV-12` on save and an arbitrary `salesOrders[0]` address/PO pick. Own ticket: dedupe + decide whether the code should fail loudly.
+- **Unregistered `reprint` RecordScript** — absent from production `Core.RecordScripts` (no `tracking-numbers` scripts at all); reprint is broken for every client. Own ticket.
+- **`Logs.Issue` 124** — `worker/crons/toga2/netsuite/sync_togasupply_growrk.php` → `App_Api_Toga2::syncItemReceiptFromNetsuite` failing every 5 minutes since 2026-08-06 (~288/day), untracked. 1.0 stack (`worker` + `library`).
+- **Duplicate `getAddress` call** — fires twice per load with identical params. Harmless but doubles the call in production too.
+- **Audit every client's `_modules.txt`** — 22 clients compose `_Trait_Netsuite_SalesOrder`; only GroWrk's module membership was confirmed. A client using the trait but not on the `netsuite` module would miss the module-scoped ACL migration.
+- **Commit the two production-applied files** `2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql` and `2026-08-10b` — applied to prod, in no branch. Prod is still ahead of the repo.
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `dbchanges2/Client_Growrk/2026-08-10c - GrowrkServiceRequestCustomFieldsCatchUp.sql` | Created, pushed | Outage fix. Per-column `INFORMATION_SCHEMA`+`PREPARE` guards; ACL by lookup; `UPDATE`-then-`INSERT`. Effects already in prod by hand |
+| `dbchanges2/Client_Growrk/2026-08-10d - GrowrkServiceRequestTypeAndDispositionSeeds.sql` | Created, pushed | 17 types + 4 dispositions, `NOT EXISTS`-guarded on name. **Not yet applied to prod** |
+| `dbchanges2/Client_Growrk/2026-08-10e - GrowrkUpsServiceMethodCodes.sql` | Created, pushed | Records the hand-applied UPS `03`/`01`. Scoped by `shippingCarrierId` + `name` |
+| `dbchanges2/_modules/netsuite/2026-08-10a - AddressPhoneNumberApiRoleAcl.sql` | Created, pushed | Role-3 write on `RecordFields` 990. **Runs on 22 clients. Must apply BEFORE the _underscore deploy** |
+| `_underscore/Trait/Netsuite/SalesOrder.php` | Modified, pushed | `phoneNumber` from `addrPhone`, normalised via `_String::cleanPhoneNumber()`, plus the change-detection clause |
+| `_underscore/Component/Library/Carriers/Fedex/Fedex.php` | Modified, pushed | `inflateIfCompressed()` + applied to error and success paths. **Not deployed** |
+| `toga2-supply/src/pages/EditShipment/api/UpdateShipmentApi.ts` | Modified, pushed | `getAddress` also requests `number` + `purchaseOrderSalesOrders.purchaseOrder.number` |
+| `toga2-supply/src/pages/EditShipment/view/EditShipmentPage.tsx` | Modified, pushed | Maps `salesOrderNumber` / `purchaseOrderNumber` onto the transformed address |
+| `toga2-supply/src/pages/EditShipment/view/components/forms/EditShipmentForm.tsx` | Modified, pushed | `toCarrierReference()` 35-char cap at defaults, edit reset, and submit; `phoneNumber` in defaults; references added to `reset()` |
+| `toga2-supply/src/pages/EditShipment/types.ts` | Modified, pushed | Declared `phoneNumber` on `ShipmentData` (was missing) |
+| `test/@Mark/true-80824-fedex-inflate-test.php` | Created | 18-case regression suite. `test` is not branched, outside TRUE-80824 |
+| `~/toga-tech/knowledge/**` (7 docs + growrk profile) | **Written, staged, NOT pushed** | `/capture` withheld publish pending one elevated approval — see Blockers |
+
+## Decisions made
+
+- **Split the fix across four migration files** instead of re-running `2026-05-28.sql`. Rationale: production had moved on — its `ALTER Units` and recordId-31 metadata were already applied, and its hardcoded `customRecordFieldId` 27–33 now point at the *Units* fields, so it would have written wrong ACL grants. Rejected: running the original whole (errors mid-file, stale ids), and editing the original (the contract forbids editing a committed migration, and it must stay valid for a fresh client).
+- **Did not delete or edit `2026-05-28.sql`.** It stays correct for a newly provisioned client; the catch-up files handle the drifted production.
+- **Normalise the phone with `cleanPhoneNumber()` on the way in**, not just at comparison time. Rationale: `_Model` applies the same transform on write, so sending the raw NetSuite string would store a different value than was sent, making the comparison permanently unequal and re-`PUT`ting the address on **every** sync for **every** client on the shared trait. Rejected: normalising only inside the comparison (leaves payload and storage disagreeing).
+- **No backfill of historic addresses** (Mark). Phone appears only on orders that re-sync from NetSuite. Consequence: the fix is unverifiable on an existing order.
+- **Module-scoped ACL migration, not per-client.** The trait is shared by 22 clients, all of which start sending the field the moment the code deploys. Compass already had the grant and set the pattern.
+- **Kept the UTF-8 guard in `inflateIfCompressed()` against CodeRabbit's advice**, on the 257/100,000 measurement, and encoded four of the fuzz-found inputs as regression cases so the guard cannot be silently removed. Accepted CodeRabbit's *other* point — dropped the zlib header allowlist, which was too narrow (legal zlib CMF varies with window size) — netting fewer branches than before.
+- **Left the rehydrate effect's dep array as `[editShipment]`.** Adding the reference values would look more correct to a linter but would re-run `reset()` and discard a user's in-progress edits.
+- **Left `setLogging(false)` on the FedEx ship call.** A successful response carries the full base64 label; logging it would write megabytes per shipment.
+- **Truncate carrier references to 35 chars** for consistency with typed input. Noted as a product question — a clipped PO may not match GroWrk's records; still Skyler's call.
+- **Kept `CURLOPT_ENCODING` out of this change.** It is the framework's shared HTTP client and deserves its own review rather than riding in a hotfix.
+- **Trimmed all comments ~60%** (194 comment lines → 79) at Rohan's request, keeping only what prevents reintroducing a known bug. Full reasoning preserved in the commit bodies.
+
+## Blockers
+
+- **The FedEx 400 cause is still unknown.** It cannot be diagnosed until `_underscore` is deployed, because the body is gzip-encoded and the ship call is unlogged. This blocks end-to-end verification with Skyler.
+- **`/capture` has not published.** 7 knowledge docs are written and staged in `~/toga-tech` but unpushed, because the capture agent is holding one ⚠ ELEVATED item: two new re-run-safety rules (10 and 11) for `2.0/apps/dbchanges2/architecture.md`, which is senior-owned. **A wording fix was identified before it lands:** the draft says the combined `ALTER` *"would have no-op'd the entire file"* — it would have **failed with 1060 and aborted**, the opposite of a no-op. Decide approve-with-fix / approve-as-is / skip, then re-invoke `session-capture` with `mode: apply-approved`.
+- **Nothing is deployed.** All three branches are pushed but no environment has the code.
+- Beta almost certainly still has the ServiceRequests drift — yesterday's hand-fix went to production only, so saving on beta may still 500 until `2026-08-10c` runs there.
+
+## Exact next step
+
+> Resolve the held `/capture` approval first (it takes seconds and is blocking a push): decide on the two `2.0/apps/dbchanges2/architecture.md` rules — recommended is *approve with the wording fix*, replacing "would have no-op'd the entire file" with "would have failed on the first `ALTER` and aborted the file before its inserts ran" — then re-invoke the `session-capture` agent with `mode: apply-approved` so all 7 docs publish in one push. Immediately after, begin the deployment: apply `dbchanges2/_modules/netsuite/2026-08-10a - AddressPhoneNumberApiRoleAcl.sql` to **beta** and confirm `SELECT roleId, isWritable FROM Client_Growrk.AclFieldPermissions WHERE recordFieldId = 990` returns a role-3 writable row **before** deploying `_underscore` — that ordering is the one hard constraint in this ticket.
+
+---
+_Saved by /session-save on 2026-08-11_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.557",
+  "version": "1.0.559",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",