toga-ai 1.0.774 → 1.0.776

package/knowledge/1.0/apps/webhook/INDEX.md

@@ -2,4 +2,4 @@
 
 | Doc | Summary | Files |
 |-----|---------|-------|
-| [NetSuite → ClickUp Opportunity Sync](features/netsuite-clickup-opportunity-sync.md) | NetSuite Opportunities are pushed into the ClickUp **Opportunities** list (list id `901111987449`, in the Opportunity/RFP Hub space `90113928591`) as tasks name | webhook/_/webhook/netsuite.php, webhook/api.php, api2/Webhook/Netsuite.php, worker2/Worker/Netsuite.php, api2/Controller/Index.php |
+| [NetSuite → ClickUp Opportunity Sync](features/netsuite-clickup-opportunity-sync.md) | > **⚠ CORRECTED 2026-09-04 — parts of this doc were factually wrong and actively misleading.** > Two claims below have been retracted in place: that > `worker2/ | webhook/_/webhook/netsuite.php, webhook/api.php, api2/Webhook/Netsuite.php, worker2/Worker/Netsuite.php, api2/Controller/Index.php |

package/knowledge/1.0/apps/webhook/features/netsuite-clickup-opportunity-sync.md

@@ -6,17 +6,33 @@ project: Webhook
 client: shared
 type: feature
 status: active
-updated: 2026-07-17
-owners: ["dfranks"]
+updated: 2026-09-04
+owners: ["dfranks", "ajean"]
 files:
   - webhook/_/webhook/netsuite.php
   - webhook/api.php
   - api2/Webhook/Netsuite.php
   - worker2/Worker/Netsuite.php
   - api2/Controller/Index.php
-related: []
+related:
+  - ../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md
 ---
 
+> **⚠ CORRECTED 2026-09-04 — parts of this doc were factually wrong and actively misleading.**
+> Two claims below have been retracted in place: that
+> `worker2/Worker/Netsuite/Opportunity.php` and `maybeCreateClickupTask()` **do not exist**, and that
+> the integration has **"no dedup of any kind."** Both files exist and are git-active, and the 2.0
+> path carries an **atomic DB claim** on `Forecast.Opportunities.clickupTaskId` plus folder-wide
+> dedup. Those two claims are why a later session opened with a contradiction about which app owns
+> the receiver. Canonical detail:
+> [NetSuite ↔ ClickUp Opportunity Sync (worker2)](../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
+>
+> **Scope caveat on everything else here:** the 1.0 `webhook` repo is **not checked out** on the
+> machine used for the 2026-09-04 review and could not be found anywhere on disk, so the claims in
+> this doc about the *live 1.0 receiver* are **unverified as of that date**. The NetSuite-side
+> content (scheduled script **2412**, UserEvent **2493**, the `OPP <id>` stub root cause and its fix)
+> was **not** in question and stands.
+
 ## Summary
 NetSuite Opportunities are pushed into the ClickUp **Opportunities** list (list id `901111987449`,
 in the Opportunity/RFP Hub space `90113928591`) as tasks named `<oppNumber> — <customerNumber>
@@ -74,23 +90,32 @@ the assignment fires 2493 in `edit` (full-record) context → proper name (see G
 None — single internal Agilant/True integration.
 
 ## Gotchas / known issues
-- **Create-only, NO dedup of any kind (confirmed 2026-07-17).** `Webhook_NetSuite::post()`
-  json-decodes the payload and calls `createTask()`, which POSTs a **new** task to ClickUp
-  list `901111987449` **unconditionally on every delivery** (writing link `db_client`). There
-  is no idempotency check whatsoever — so it double-creates a task on **any** repeat or
-  concurrent delivery, not only concurrent ones.
+- **Create-only with no dedup *in this 1.0 handler* — but the claim "NO dedup of any kind" was
+  WRONG and is retracted (2026-09-04).** As originally written (2026-07-17) this bullet asserted the
+  integration as a whole had "no idempotency check whatsoever." That is false at the platform level:
+  the **2.0** path (`worker2/Worker/Netsuite/Opportunity.php`) carries a **race-safe atomic claim** —
+  a guarded `UPDATE Opportunities SET clickupTaskId=… WHERE … AND clickupTaskId IS NULL` — plus
+  folder-wide dedup by `Opportunity #`. What remains true, scoped correctly: **`Webhook_NetSuite::post()`
+  in the 1.0 `webhook` repo** json-decodes the payload and calls `createTask()`, which POSTs a new
+  task **unconditionally on every delivery** (writing link `db_client`), so *that* handler
+  double-creates on any repeat or concurrent delivery. Unverified since 2026-09-04 (repo not checked
+  out).
 - **Live payload carries no immutable opportunity identifier (confirmed 2026-07-17).** The
   inbound body has **only**: `name`, `description`, `presalesLead`, `salesRepText`, `token`.
   It has **no NetSuite opportunity internal id and no opportunity/customer number**, so there
   is no stable key available in this handler to dedup on today. Any dedup fix must first get
   NetSuite (script 2493) to include an immutable id in the payload, or derive one another way.
-- **Where the swapped Opp#/Cust# mapping actually lives.** The
-  `OPPORTUNITY_NUMBER ← customerNumber` / `CUSTOMER_NUMBER ← opportunityNumber` swap is in the
-  **dormant 2.0 port** `worker2/Worker/Netsuite.php` (class `_Worker_NetSuite`, ~lines 116–121),
-  which is **NOT live**. The live `webhook/_/webhook/netsuite.php` payload doesn't even carry
-  those number fields (see above). There is **no** `worker2/Worker/Netsuite/Opportunity.php` and
-  **no** `maybeCreateClickupTask()` method anywhere — earlier plans referencing those paths were
-  citing files that do not exist.
+- **Where the swapped Opp#/Cust# mapping lives — with the "these files don't exist" claim
+  RETRACTED (2026-09-04).** The `OPPORTUNITY_NUMBER ← customerNumber` /
+  `CUSTOMER_NUMBER ← opportunityNumber` swap is in the **dormant 2.0 port**
+  `worker2/Worker/Netsuite.php` (class `_Worker_NetSuite`, ~lines 116–121). That part stands.
+  **What was wrong:** this bullet claimed there is "**no** `worker2/Worker/Netsuite/Opportunity.php`
+  and **no** `maybeCreateClickupTask()` method anywhere," and that plans citing them were citing
+  fictional files. **Both exist and are git-active.** `_Worker_Netsuite_Opportunity` is the current
+  2.0 handler — it does the Forecast upsert, holds the atomic `clickupTaskId` claim, and
+  `maybeCreateClickupTask()` is its ClickUp entry point. It also **fixed** the swap (2026-06-16).
+  Do not use this bullet to conclude the 2.0 path is vapour; read
+  [the worker2 doc](../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
 - **`OPP <internalId>` degenerate tasks (root-caused & confirmed 2026-06-15).** A task named just
   `OPP 7147699` with no description/fields. The name is built by NetSuite and relayed verbatim;
   the opps are **fully populated** in NetSuite (so it's not missing data, not a TOGA bug). Cause:
@@ -148,6 +173,16 @@ Done 2026-06-15 for opps 7147699/7147698/7147472/6382879 → 4 proper tasks crea
 (task ids `868k0tr41`, `868k0tr16`, `868k0rwtp`, `868k0h44m`) were archived in ClickUp afterward.
 
 ## Change history
+- 2026-09-04 — **Corrected two factually wrong claims (retracted in place; no code changed).** This
+  doc asserted that `worker2/Worker/Netsuite/Opportunity.php` and `maybeCreateClickupTask()` "do not
+  exist" and that the integration has "no dedup of any kind." Both files exist and are git-active,
+  and the 2.0 path carries an **atomic DB claim** on `Forecast.Opportunities.clickupTaskId` plus
+  folder-wide dedup; the create-only/no-dedup statement is true only of the **1.0 `webhook` handler**
+  and has been rescoped to it. These two claims were the reason a later session opened with a
+  contradiction about which app owns the receiver. Added a scope caveat: the 1.0 `webhook` repo is
+  **not checked out** on the reviewing machine and could not be found on disk, so this doc's claims
+  about the live 1.0 receiver are **unverified as of 2026-09-04**; the NetSuite-side script 2412/2493
+  content was never in question and stands. (ajean)
 - 2026-07-17 — Confirmed against live code: `createTask()` POSTs to list `901111987449`
   **unconditionally with no dedup** (double-creates on any repeat/concurrent delivery); the
   live payload carries only `name`/`description`/`presalesLead`/`salesRepText`/`token` — **no

package/knowledge/1.0/apps/worker/INDEX.md

@@ -8,7 +8,7 @@
 | [Compass Manager Approval Reminder Emails (1.0 worker crons)](features/compass-manager-approval-reminder-emails.md) | Two 1.0 worker crons nag approvers about sales orders still waiting on a decision — one per Compass tenant. | worker/crons/toga2/compass/compass_email_reminders.php, worker/crons/toga2/compasscanada/compass_email_reminders.php, worker/schedules/cron.worker.sync.json, worker1.5/crons/toga2/compass/compass_email_reminders.php, worker1.5/schedules/cron.worker.json |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compass/send_delivered_email.php, worker/crons/toga2/compasscanada/update_salesorder_status_from_g&t.php, worker/crons/toga2/compasscanada/send_delivered_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_in_transit_email.php, worker/crons/toga2/compasscanada/workflow/test_partial_delivered_email.php, library/app/client/compasscanada.php, worker/schedules/cron.worker.sync.json, worker2/Worker/Monitor/Compass.php, dbchanges2/Client_Compass/2026-09-03 - compass_intransit_email_backlog_backfill.sql |
 | [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
 | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/query.php, library/app/database.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-28
-owners: [dfranks, jcardinal, kyalamarthi]
+updated: 2026-09-04
+owners: [dfranks, jcardinal, kyalamarthi, ajean]
 files:
   - test/@dave/checker.php
   - worker2/Component/Forecast/SaleImport/SaleImport.php
@@ -32,6 +32,7 @@ files:
   - test/@dave/probe_profit_gap.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
   - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php
+  - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php
   - worker/crons/toga2/forecast2/import_open_orders.php
   - worker/schedules/cron.worker.infrastructure.json
 related:
@@ -373,6 +374,50 @@ have broken **every insert the cron makes**. It was caught only by rendering the
 stub harness before any run. **`php -l` cannot catch malformed SQL string assembly**; when you edit a
 query *builder*, print the built query and read it.
 
+### ⚠ The opportunities discrepancy-fix cron DELETES rows that carry a ClickUp link (2026-09-04)
+
+**This is a live data-loss path — treat it as the highest-priority item in this doc.**
+`worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_opportunities.php` (209 lines,
+`0 4 * * *`, **active**) and the same code path in `tools/bin/forecast/fixer.php`
+(`fixOpportunities()`) both build their delete set **without ever selecting or checking
+`clickupTaskId`**.
+
+**Blast radius, measured 2026-09-04: 746 of 26,766 in-window rows carried a ClickUp task id**, and
+one row was already stranded at the `PENDING` sentinel. Deleting such a row **orphans the ClickUp
+task** and, because the 2.0 webhook's atomic claim keys on `Forecast.Opportunities.clickupTaskId`,
+the next NetSuite event for that opportunity finds no claim and **creates a duplicate task**. The
+trigger does not require anything exotic: **a single truncated `listOpportunities()` page makes every
+unreturned opportunity look stale**, so an API hiccup is enough to mass-delete live rows.
+Claim semantics and the duplicate-creation consequence:
+[NetSuite ↔ ClickUp Opportunity Sync](../../../../2.0/apps/worker2/features/netsuite-opportunity-sync.md).
+
+Three further defects found in the same file:
+
+1. **⚠ `MAINTENANCE_MODE = true` gates LOGGING ONLY — it does NOT stop writes.** Its sibling
+   `periodic_forecast_discrepancy_fix_open_orders.php` uses the same-named constant as a genuine
+   **dry run**. So the reasonable assumption — "same constant, same meaning, safe to leave on while I
+   watch it" — is **wrong here, and it deletes**. This is a genuine trap, not a style inconsistency:
+   verify per file before relying on it.
+2. **The INSERT omits seven columns:** `forecastCategoryId`, `salesStageId`,
+   `percentToCloseStatusId`, `projectedProfit`, `leadSource`, `clickupTaskId`, and now
+   `endCustomerName`. Rows the cron recreates therefore come back stripped of all seven — the same
+   delete-and-reinsert failure shape as the `locationId` case above.
+3. **Empty `sql_mode` in prod turns omission #2 into silent corruption.** `projectedProfit` is
+   NOT NULL with **no default**, so instead of rejecting the INSERT, MySQL silently stores `0.00`.
+   **This is the root cause of the perpetual checker↔fixer churn** this doc has tracked for months:
+   the checker flags a profit delta, the fixer "repairs" the row, the cron zeroes it again the next
+   night.
+
+**And the reconciler cannot compute `projectedProfit` at all.** It fetches **no line items** — zero
+references to `itemList` or `costEstimate` anywhere in the file — while opportunity profit is
+Σ(amount − costEstimate) over qualifying lines (the formula recorded above). So it cannot fix the
+column it breaks, and there is no version of "just add `projectedProfit` to the INSERT" that works
+without adding line-item fetching first.
+
+**Agreed direction (2026-09-04): the reconciler should STOP INSERTing and report instead.** A
+component that cannot compute a NOT-NULL column should not be authoring rows. Detail in
+[the bidirectional opportunity-sync design](../../../../2.0/apps/worker2/features/netsuite-clickup-opportunity-bidirectional-design.md).
+
 ## Data model
 
 `Forecast.Sales`, `Forecast.OpenOrderItems` on the **core2** cluster
@@ -654,6 +699,21 @@ None — Forecast2 is a single shared dataset.
   sequence the rotation against whatever else reads that replica.
 
 ## Change history
+- 2026-09-04 — **Investigation only; no code changed (a prototype fix was written and deliberately
+  reverted).** Recorded a **live data-loss path**: the nightly
+  `periodic_forecast_discrepancy_fix_opportunities.php` (`0 4 * * *`, active) and
+  `tools/bin/forecast/fixer.php::fixOpportunities()` build their delete set **without checking
+  `clickupTaskId`** — **746 of 26,766 in-window rows carried a task id** (one already stranded at
+  `PENDING`), so a delete **orphans the ClickUp task and causes duplicate task creation**, and a
+  single truncated `listOpportunities()` page is enough to make every unreturned opportunity look
+  stale. Three more defects in the same file: **`MAINTENANCE_MODE = true` gates logging only, NOT
+  writes** (unlike its `..._open_orders.php` sibling, where it *is* a dry run — a genuine trap); the
+  INSERT **omits** `forecastCategoryId`, `salesStageId`, `percentToCloseStatusId`, `projectedProfit`,
+  `leadSource`, `clickupTaskId` and `endCustomerName`; and because prod `sql_mode` is **empty**,
+  `projectedProfit` (NOT NULL, no default) silently lands as **`0.00`** — the **root cause of the
+  long-standing checker↔fixer churn**. Also established that the reconciler **cannot compute
+  `projectedProfit` at all** (it fetches no line items — zero `itemList`/`costEstimate` references),
+  so the agreed direction is that it **stops INSERTing and reports** instead. (ajean)
 
 - 2026-08-28 — **⚠ Security finding (not fixed):** the `tools` repo copies of these scripts —
   `tools/bin/forecast/fixer.php:1424` and `tools/bin/forecast/checker.php:678` — hardcode **live

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -7,7 +7,7 @@
 | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, dbchanges2/Core/2026-08-27a - TransferOrderUuidSearchableIdentifier.sql, _underscore/Model/Core/Page.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, dbchanges2/Client_Quad/2026-08-24 - Quad Multi Currency Item Pricing.sql, dbchanges2/Client/2026-08-29 - ItemReceiptTransferOrderIdFieldPermission.sql, dbchanges2/Client/2026-08-29b - ItemReceiptItemTransferOrderItemIdFieldPermission.sql, dbchanges2/Client/2026-09-01a - PurchaseOrderItemQtyFieldsApiRoleRead.sql |
 | [Address Uniqueness Normalization (unit identifier + 5-digit ZIP comparison)](features/address-uniqueness-normalization.md) | When a business rule says *"only one X per physical address"*, comparing address rows field-for-field does **not** work: the same dwelling is spelled many diffe | _underscore/Model/Rate/Entitlement.php |
 | [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php, _underscore/Component/Library/Carriers/Usps/Usps.php |
-| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php |
+| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, _underscore/Component/Api/Clickup/Clickup.php |
 | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
 | [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | `_Query` can run a **write** query asynchronously so a long/slow write does not hold a request-scoped DB connection open long enough to hit **"MySQL server has | _underscore/Query.php, worker2/Worker/Infrastructure/Database.php, worker2/Worker/Team/Transcripts.php |
 | [FIELD_AUTOINCREMENT record numbering (SA / TA prefixes, NULL-only trigger, no padding)](features/autoincrement-record-numbering.md) | A human-facing record number (`SA100001`, `TA100000`) is generated by the ORM, not by MySQL. | _underscore/Model.php, _underscore/Model/Client/SalesOrder.php, _underscore/Model/Client/TransferOrder.php |

package/knowledge/2.0/apps/_underscore/architecture.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: architecture
 status: active
-updated: 2026-08-26
-owners: ["jcardinal", "rgirish", "mhammontree"]
+updated: 2026-09-04
+owners: ["jcardinal", "rgirish", "mhammontree", "ajean"]
 files:
   - _underscore/_underscore.php
   - _underscore/Loader.php
@@ -31,7 +31,10 @@ Minimum PHP 8.1 (enforced in `_underscore.php`). The current architecture is
 **Critical rules:** framework classes use a leading-underscore prefix (`_Model_*`, `_Controller_*`,
 `_Worker_*`). Always use parameterized queries / the `_Db` layer — never interpolate input into SQL.
 Beware the **lazy-transaction gotcha**: writes issued outside an explicitly committed transaction
-can be silently dropped — confirm commit semantics before relying on a write.
+can be silently dropped — confirm commit semantics before relying on a write. Two corollaries:
+**never hold a DB transaction open across an external HTTP call**, and **an empty log table is not
+proof a code path never ran** — uncommitted rows vanish, so absence of log rows means "no committed
+write", not "no execution".
 The Surface presentation layer replaces `Page::meta()` but `meta()` must NOT be deleted until
 every page is cut over (per-surface, with a parity diff); config describes, PHP decides —
 business logic never moves into Surface config.
@@ -369,6 +372,18 @@ multi-file UI components (`.php`/`.html`/`.css`/`.js`) invoked as `<_ComponentNa
   pin is `Addresses.isValidated` and `Entitlements.serviceAddressId` both-or-neither. Relevant code:
   `_underscore/Model/Rate/Entitlement.php`, `api2/Component/Api/V2/V2.php::getFullModelData`.)
 - **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026, commit `fa7835ed`).** Any code that calls `register()` and then writes to that DB must call `_Database::transactionCommit()` before the request ends — otherwise MySQL silently rolls back all writes when the connection closes. Lazy transactions only materialise on the first write, so read-only callers are unaffected. See `_underscore/Database.php:48`. First discovered when Rate SAML user provisioning silently discarded all new user INSERTs (Jun 2026).
+  - **Never hold a DB transaction open across an external HTTP round-trip.** Beyond pinning a
+    connection for the duration of a third-party call, it means the log/telemetry rows you are
+    writing *about* that call are still uncommitted while it runs. Live instance found 2026-09-04:
+    `_Worker_Clickup_Opportunity::logged()` opens `DB_LOGS`, never commits, and holds the
+    transaction across every ClickUp/NetSuite call.
+  - **An empty log table is NOT proof that a code path never ran.** This is the diagnostic trap the
+    gotcha creates, and it has already produced one wrong conclusion in this knowledge base: a
+    zero-row `Logs.Api` census was read as "this integration has been dead since it shipped," when
+    the writes were simply never committed. Before concluding a path is dead from absent rows,
+    confirm the writer commits. Correct pattern:
+    `worker2/Worker/Netsuite/SalesOrder.php:441`. Case detail:
+    [netsuite-opportunity-sync](../worker2/features/netsuite-opportunity-sync.md).
 - **`_Model::load()` with a collapsed WHERE clause silently full-table-scans → OOM (guarded 2026-08-26, `Model.php`).** `_Model::search()` builds `SELECT <pk> FROM <table> [WHERE …] ORDER BY <pk>` and emits the WHERE **only if at least one search term survives**. `buildSqlFieldValue()` **drops** a term (returns false) when a set field's value is `null` and its type is a plain `FIELD_CHAR` — i.e. not one of the special-cased types (`AUTOINCREMENT`, `DATETIME_CREATED`, `DATETIME_UPDATED`, `SQL`, `STORAGE`, `CHAR_UUID`). If the only set search field collapses this way, the WHERE vanishes and `search()` selects the **whole table**, constructing one model per row → memory exhaustion on a large table. Because `load()` calls `search()` first and only then checks `count() == 1`, it **OOMs building the result array before it ever counts** — the empty-WHERE case never reaches the "not exactly one" branch. **`load()` means "find exactly one by criteria"; an empty WHERE is always a bug, never a full-table scan.** **Fix (commit `0b944a51`, `_production`):** `load()` now guards at the top — if no set search field yields a WHERE term (via `buildSqlFieldValue`), it `error_log`s and returns `false` (or throws when `throwExceptionIfNotFound`) instead of scanning. `search()` is unchanged, so intentional "list all rows" callers of `search()` are unaffected. **Triage:** a PHP OOM stack trace of `Model.php search() → __construct` that is *shallow* (a single search→construct, not deeply repeated) is a runaway result set from a collapsed WHERE, **not** recursion — confirm by checking whether the single set search field is a `FIELD_CHAR` whose value is null. (Root cause of a recurring prod OOM: `Logs.Issue` ref `2L` (#38), first seen 2026-08-03, 136 occurrences, trace `Model.php search()→__construct`.)
 - **PHP "Unclosed '{'" parse errors report a MISLEADING line number.** When a `.php` file loaded by the autoloader (`Loader.php`) has a dropped/unbalanced brace, PHP reports `Unclosed '{' on line N` where N is the **outermost `class X {` line** and fails at EOF — NOT at the true location of the missing `}`. Worse, because the file loads lazily via the SPL autoloader, the runtime trace points at the **caller** that triggered the autoload (e.g. a `new _Email()` call site), not the broken file. **Triage rule:** for an "Unclosed '{'" error, the real culprit is a missing `}` somewhere between the reported line and EOF of the file that failed to load — run `php -l <file>` (it reports the EOF line) and scan the whole file. **Merge-conflict resolutions are a common source of a single dropped brace** — review the entire merge, not just the one file the error appears to name. (First hit: production 500 EO-1, Jul 2026 — a `}` dropped from `_Email::send()` during merge `685e4a14` surfaced as a trace pointing at the `new _Email()` caller.)
 

package/knowledge/2.0/apps/_underscore/features/apirequest-json-content-type.md

@@ -6,10 +6,12 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-26
+updated: 2026-09-04
 owners: ["jcardinal", "ajean"]
 files:
   - _underscore/ApiRequest.php
+  - _underscore/Model/Client/Logs/Api.php
+  - _underscore/Component/Api/Clickup/Clickup.php
 related:
   - ../../worker2/features/oneuptime-worker2-monitoring.md
   - ../../worker2/features/creating-worker-actions.md
@@ -90,10 +92,46 @@ case-insensitive scan of already-set headers, so a caller-supplied `Content-Type
   **hardcodes** its token in committed source — flagged for a rotate-and-move-to-config ticket; 11
   worker2 files use it). When adding a new component, decide deliberately: log and accept the header
   exposure, strip/redact the auth header before the call is logged, or `setLogging(false)`.
+- **Multipart/file upload ALREADY WORKS — do not add an `ENCODE__MULTIPART` case (2026-09-04).**
+  The request-encoding switch (`ApiRequest.php:205-228`) has **no `default:` case**, so
+  `payloadEncoding: null` passes the payload through **untouched** and sets no `Content-Type`; `:236`
+  hands it straight to `CURLOPT_POSTFIELDS`, where **cURL itself** builds the multipart body and the
+  boundary. So the working recipe needs no framework change at all: pass an array such as
+  `['attachment' => new CURLFile($path, $mime, $name)]` with `payloadEncoding` left null, and cURL
+  **streams the file from disk** (memory stays flat regardless of file size). Add `setLogging(false)`
+  (you do not want file bytes in `Logs.Api`) and an `Expect: ''` header (suppresses cURL's
+  100-continue round-trip).
+  **Why an `ENCODE__MULTIPART` case is the wrong fix:** `$payloadEncoding` drives **both** request
+  encoding **and** response decoding (`:358-372`). A multipart-**request** / JSON-**response** call
+  cannot be expressed in one enum value — the response would fall through to `default:` and come back
+  as a **raw string instead of a decoded object**, breaking the caller. `new _ApiRequest` appears at
+  **97 sites across 35 files** in `_underscore` + `worker2`, so this would be an expensive and buggy
+  change to shared core for no gain. (Independent `/cto` review: DISAGREE-WITH-ALTERNATIVE — the
+  alternative being exactly the pass-through recipe above.)
+- **⚠ `_ApiRequest` FATALS on a non-string request payload — latent for all 97 callers
+  (found 2026-09-04, NOT fixed).** `ApiRequest.php:268` assigns
+  `$log->requestPayload = $this->requestPayload;` into a **`FIELD_CHAR`** column, and that value
+  reaches `_Database::escape()`. Hand it an **array** — which is exactly what the multipart recipe
+  above requires, since the array holds a `CURLFile` — and you get a **fatal error**, not a bad log
+  row. The failure lands in the *logging* path, so it looks nothing like a payload problem. One-line
+  guard: `is_string($this->requestPayload) ? $this->requestPayload : '[non-scalar payload omitted]'`.
+  Until that lands, any array payload **must** use `setLogging(false)`.
 - **`Logs.Api` is 1.4M+ rows and `source` is NOT indexed.** Any diagnostic query must be bounded by
   `dtStamp` (indexed) or it times out — `WHERE source = '…'` alone will not return.
 
 ## Change history
+- 2026-09-04 — **Investigation only; no framework change made (deliberately).** Established that
+  `_ApiRequest` **already supports multipart/file upload** with no core change: the request switch
+  (`:205-228`) has no `default:` case, so `payloadEncoding: null` passes the payload through and
+  `:236` lets cURL build the multipart body + boundary — pass
+  `['attachment' => new CURLFile($path,$mime,$name)]` and it streams from disk with flat memory
+  (add `setLogging(false)` + `Expect: ''`). Recorded **why not to add an `ENCODE__MULTIPART` case**:
+  `$payloadEncoding` drives request encoding *and* response decoding (`:358-372`), so
+  multipart-request/JSON-response is inexpressible and the response would return a raw string; 97
+  `new _ApiRequest` sites across 35 files would be at risk (`/cto`: DISAGREE-WITH-ALTERNATIVE). Also
+  found a **latent fatal**: `:268` assigns `requestPayload` into a `FIELD_CHAR` column that reaches
+  `_Database::escape()`, so a **non-string payload is a fatal error, not a bad log row** — live for
+  all 97 callers the moment anyone passes an array. (ajean)
 - 2026-08-26 — Documented the **manual-insert column contract** (`isAuthRequest`, `instanceId`,
   `hostname` are NOT NULL with no defaults, and `_ApiRequest` sets them at `ApiRequest.php:260-264`),
   after a hand-rolled `logged()` helper in two worker2 handlers threw on every insert and silently

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-23
-owners: [dfranks]
+updated: 2026-09-04
+owners: [dfranks, ajean]
 files:
   - worker2/Component/Forecast/SaleImport/SaleImport.php
   - worker2/Component/Forecast/Db/Db.php
@@ -167,6 +167,20 @@ uses) and then writes the line — instead of the cron's silent skip. Failure se
   the job for retry.
 - `itemGroup` items carry no line revenue and are skipped.
 
+**⚠ This makes the catch clause a hard constraint on the whole codebase (2026-09-04).** The engine
+catches **only `RuntimeException`**, and the "infrastructure failures propagate" half of the contract
+above works *only because* `_Component_Api_Netsuite::send()` throws a **plain `Exception`**. So:
+
+- **Any new exception type raised on this path must extend `Exception` directly.** Give it a
+  `RuntimeException` parent and a **429 gets swallowed as "unmappable item"** — the line is skipped
+  and logged as bad data, the job reports success, and the sale is silently missing. This is why
+  `_Exception_Database` extends plain `Exception` too; see
+  [v2 request retry — scope](../../api2/features/v2-deadlock-retry.md).
+- **One case is already misclassified today:** `_Component_Api_Netsuite::fetchRecord()` **does** throw
+  `RuntimeException` on a 200-with-non-object body, so a **malformed** NetSuite response reaches this
+  catch and is booked as an unmappable item rather than an infrastructure failure. See
+  [NetSuite REST client](./netsuite-rest-client.md).
+
 ## JournalEntry import (the fifth transaction type)
 JournalEntry extends the same engine to capture **GL revenue/cost adjustments posted
 directly as journal entries**. It is **structurally different from the four sale types** — a
@@ -589,6 +603,13 @@ success from a `Forecast.Sales` row alone.
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-09-04 — Recorded (no code change) that the `RuntimeException`-only catch in the item self-heal
+  is a **constraint on new code**: because "infrastructure failure" is signalled by
+  `_Component_Api_Netsuite::send()` throwing a **plain `Exception`**, any new exception type on this
+  path must extend `Exception` directly or **API 429s get swallowed as "unmappable item"** (line
+  skipped, job green, sale silently missing). Also noted one case already misclassified:
+  `fetchRecord()` throws `RuntimeException` on a 200-with-non-object body, so a malformed NetSuite
+  response is treated as bad data. (ajean)
 - 2026-07-23 — **Broadened the AMQ event-capture blind spot beyond inline line-edits** (TRUE-80262
   planning; no code shipped, dfranks). Recorded that **bulk/mass updates and CSV imports** fire no UE
   (CSV runs server SuiteScript only when "Run Server SuiteScript and Trigger Workflows" is on — OFF by

package/knowledge/2.0/apps/_underscore/features/netsuite-rest-client.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-27
-owners: ["dfranks", "jcardinal", "bala", "kyalamarthi"]
+updated: 2026-09-04
+owners: ["dfranks", "jcardinal", "bala", "kyalamarthi", "ajean"]
 files:
   - _underscore/Component/Api/Netsuite/Netsuite.php
   - _underscore/ApiRequest.php
@@ -176,6 +176,16 @@ Surfaced building the NYCHH asset-tag backfill (see
 
 ## NetSuite transaction status sourcing (2.0) — and the 2026.2 REST `status.id` change
 
+> **`fetchRecord()` throws `RuntimeException` on a 200-with-non-object body — so a MALFORMED
+> NetSuite response is already misclassified as bad data (confirmed 2026-09-04).** The distinction
+> matters to every caller that separates "NetSuite gave me something I can't map" (a data problem, do
+> not retry) from "NetSuite is unhealthy" (infrastructure, retry). A 200 whose body isn't an object
+> is the second thing wearing the costume of the first. Concretely,
+> `worker2/Component/Forecast/SaleImport/SaleImport.php` catches **only `RuntimeException`** and will
+> swallow it as an unmappable item — see
+> [Forecast.Sales import](./forecast-sale-import.md) and the retry-scope note in
+> [v2 deadlock retry](../../api2/features/v2-deadlock-retry.md).
+
 **`fetchRecord()` returns the REST record UNTOUCHED — there is no central status
 normalization in the shared client.** `fetchRecord()` (`Netsuite.php` ~L215-223) does
 `send('GET', route)`, json-decodes, and hands the raw record back; it never reads or maps
@@ -235,6 +245,15 @@ Two consequences, one good and one expensive:
 Note that `Authorization` headers are persisted in plaintext by that same auto-logging path — see
 [`_ApiRequest` JSON/logging behavior](./apirequest-json-content-type.md).
 
+**⚠ `authenticate()` persists the signed OAuth JWT to `Logs.Api` on every token refresh
+(found 2026-09-04, NOT fixed).** The `:100` construction noted above is the auth call, and it never
+calls `setLogging(false)` — so the signed **`client_assertion` JWT** is written into the logs DB
+every time the token is refreshed. That is a credential-bearing artifact sitting in a table with
+1.4M+ rows and broad read access. **`setLogging(false)` on that one call is a pure win with no
+visibility cost:** the auth request carries no business data, and an auth failure still surfaces as
+a thrown exception from `send()`/`execute()`. Unlike the general "logging is expensive" question,
+there is no tradeoff to weigh here.
+
 ## Gotchas / known issues
 
 - **No `Location` header / wrong header accessor.** `_ApiRequest` has `responseHeaders` (raw
@@ -253,6 +272,13 @@ Note that `Authorization` headers are persisted in plaintext by that same auto-l
 
 ## Change history
 
+- 2026-09-04 — Recorded two findings from a read-only investigation (no code changed).
+  **(1)** `authenticate()` never calls `setLogging(false)`, so the signed OAuth **`client_assertion`
+  JWT** is persisted to `Logs.Api` on **every token refresh**; `setLogging(false)` there costs no
+  diagnostic visibility (auth failures still throw) and should be added. **(2)** `fetchRecord()`
+  throws **`RuntimeException`** on a 200-with-non-object body, which means a **malformed NetSuite
+  response is already misclassified as bad data rather than infrastructure failure** — significant
+  because `SaleImport` catches only `RuntimeException` and will book it as an unmappable item. (ajean)
 - 2026-08-27 — **CORRECTED the Logging section: NetSuite REST calls ARE logged to `Logs.Api`, always.**
   The previous claim (`send()` calls `setLogging(false)`, bodies not captured) is false — there are
   **zero** `setLogging` occurrences in `_underscore/Component/Api/Netsuite/Netsuite.php`, and all three

package/knowledge/2.0/apps/api2/features/v2-deadlock-retry.md

@@ -6,8 +6,8 @@ project: API
 client: shared
 type: feature
 status: draft
-updated: 2026-09-01
-owners: [jcardinal, bala]
+updated: 2026-09-04
+owners: [jcardinal, bala, ajean]
 files:
   - api2/Controller/Index.php
   - api2/Component/Api/V2/V2.php
@@ -154,6 +154,31 @@ every branch. See
 [ENVIRONMENT decides the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
 and [V2 request logging](./request-logging.md).
 
+## ⚠ Scope: this protocol is MySQL-statement-scoped and has NO HTTP dimension (2026-09-04)
+
+Two retry mechanisms now sit next to each other in 2.0 and are easy to confuse. Keep them apart:
+
+| Concern | Mechanism | Retries what |
+|---|---|---|
+| MySQL deadlock / lock-wait / unique race | this doc — `wasDeadlock()`, `wasRetriableUniqueRace()`, `ERRNO_DEADLOCK` 1213, `ERRNO_LOCK_WAIT_TIMEOUT` 1205, `ERRNO_DUPLICATE_ENTRY` 1062, request-level replay | a **SQL statement / request** against the DB |
+| Outbound HTTP 429 / transient API failure | **`_ApiRequest::setAutoRetry()`** | an **HTTP call** |
+
+**Do not reach for this protocol for an API 429.** Nothing here inspects HTTP status; the errno
+consts are MySQL error numbers and `wasDeadlock()` reads `_Database::$lastErrorNumber`. Use
+`setAutoRetry()` (and mind its flat-delay / retry-every-non-2xx behavior — see
+[`_ApiRequest`](../../_underscore/features/apirequest-json-content-type.md)).
+
+**Two consequences of the typed exception worth knowing:**
+
+- **`_Exception_Database` extends `Exception` deliberately — not `RuntimeException`.** That is a
+  design choice, not an oversight: callers that catch `RuntimeException` to mean "bad data, skip this
+  row" must **not** accidentally swallow a DB failure. It also means any *new* exception type added
+  near this code must extend `Exception` directly, or existing catch blocks change meaning — a live
+  case in [Forecast.Sales import](../../_underscore/features/forecast-sale-import.md).
+- **Every SQL failure in worker2 is now `_Exception_Database`.** So any code that compares
+  `get_class($e)` or matches on DB error **text** changes behaviour under this protocol. Audit for
+  those before assuming the change is transparent.
+
 ## Gotchas / known issues
 
 - Detection is built on the **MySQL errno** (1213/1205), not on matching the response text —
@@ -164,6 +189,14 @@ and [V2 request logging](./request-logging.md).
   not a substitute for removing the deadlock at its source.
 
 ## Change history
+- 2026-09-04 — Added a **scope** section (no code change): this retriable protocol is
+  **MySQL-statement-scoped and has no HTTP dimension** — for outbound API 429s use
+  `_ApiRequest::setAutoRetry()`, not `wasDeadlock()`/`wasRetriableUniqueRace()`. Recorded that
+  `_Exception_Database` extends **plain `Exception`** deliberately (so a `RuntimeException`
+  "bad-data, skip" catch cannot swallow a DB failure), and that because **every** worker2 SQL failure
+  is now `_Exception_Database`, any code comparing `get_class()` or DB error text changes behaviour.
+  Surfaced while separating DB retry from API retry during the NetSuite↔ClickUp opportunity
+  investigation. (ajean)
 - 2026-09-01 — Recorded the **deployment reach** of this work's `_underscore` primitives: commit
   `6355d0f0` (`ERRNO_DUPLICATE_ENTRY`, `$lastErrorNumber`, `$retriableUniqueRace`,
   `wasRetriableUniqueRace()`) is on **`_production` and `_sandbox-client` only** — **absent from

package/knowledge/2.0/apps/dbchanges2/architecture.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: architecture
 status: active
-updated: 2026-08-18
+updated: 2026-09-04
 owners: [jcardinal, mhammontree, bala, ajean]
 files:
   - Core/
@@ -96,6 +96,7 @@ capital, rest lower-case (e.g. `Compass`, `Compasscanada`).
 |---|---|---|
 | `Core/` | `Core` (shared platform DB) | single DB |
 | `Logs/` | `Logs` (framework-level logs, not tenant-scoped) | single DB |
+| `Forecast/` | `Forecast` (the core2 forecast DB) | single DB |
 | `Client/` | **every** `Client_<Tenant>` database | fan-out to all active clients |
 | `Client_<Name>/` | the **one** `Client_<Name>` DB only | single tenant |
 | `Logs_Client/` | **every** `Logs_<Tenant>` database | fan-out to all client-log DBs |
@@ -347,7 +348,7 @@ its own header.)
 
 ## Adding a new change (the rules)
 
-1. Pick the right folder for the target DB: `Core/`, `Logs/`, the shared `Client/` (every
+1. Pick the right folder for the target DB: `Core/`, `Logs/`, `Forecast/`, the shared `Client/` (every
    tenant), a specific `Client_<Name>/`, `Logs_Client/` (every tenant log), or a module under
    `_modules/<module>/`.
 2. Name the file `YYYY-MM-DD<letter> - <ShortDescription>.sql` using **today's date** and a

package/knowledge/2.0/apps/saml/architecture.md

@@ -6,7 +6,7 @@ project: SAML SSO Gateway
 client: shared
 type: architecture
 status: active
-updated: 2026-08-27
+updated: 2026-09-04
 owners: ["rgirish", "jcardinal"]
 files:
   - saml/index.php
@@ -49,7 +49,7 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
 2. **Version + time check** — RelayState JSON: `v` (currently `1`), `time`, `domain` UUID, optional `urlParameters`. Validated against `MAX_TIME_TO_LIVE` (300s).
 3. **Parse SAMLResponse** — base64-decoded, deserialized via LightSaml.
 4. **Status check** — proceeds only on `STATUS_SUCCESS`.
-5. **Assertion decryption** — TOGa SP credential from `_underscore/Assets/ssl/`.
+5. **Assertion decryption (dual-credential during cert overlap).** `/acs` builds two `X509Credential`s from the SP encryption keys in `_underscore/Assets/ssl/` (new + old) and calls `$reader->decryptMultiAssertion([$credentialNew, $credentialOld], $ctx)`. LightSaml's `decryptMulti()` tries each credential and only re-throws once all fail (fails closed), so an assertion encrypted to either the new or old SP key decrypts during a rotation overlap. See the SP cert-rotation workflow.
 6. **Resolve client + environment** — `_Model_Core_Domain` by RelayState `domain` UUID → `_Model_Core_Client` + `_Model_Core_Environment`. Dev on real AWS instance downgrades to beta.
 7. **Dynamic DB registration** — SQL joins `Clients`, `Databases`, `DatabaseHosts`, `Environments` for the environment slug. Calls `_Database::register()` + `connect()` for `DB_CLIENT`, `DB_CLIENT_LOGS`, `DB_CLIENT_ARCHIVE`.
 8. **Identity mapping** — `_Model_<ClientIdentifier>_ClientAuthentication::getAuthenticatedSsoUser($assertion)`. Wrapped in try/catch; exceptions sent to Sentry.
@@ -59,6 +59,10 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
 ### Failure responses
 `Invalid SAMLResponse (Code = 1–4)`, `SAML Authentication Failed`, `User Authentication Failed`.
 
+## /meta SP credential (signing vs encryption)
+
+`/meta` publishes TWO independent `KeyDescriptor`s: `use="signing"` and `use="encryption"`. They can carry **different** certs, which lets the SP encryption cert rotate without touching the signing cert. The private key that signs our **outbound** AuthnRequests lives in `_underscore` (`_Model_Core_ClientAuthentication::PATH_TO_PRIVATE_KEY`) and is **global — one key for all client IdPs**, so the signing cert cannot flip until every IdP has loaded the new metadata. Cert paths are set via class constants in `saml/Controller/Index.php` (`PATH_TO_SIGNING_CERTIFICATE`, `PATH_TO_ENCRYPTION_CERTIFICATE_NEW/OLD` + matching private-key constants — paths only, no key material). Full procedure: SP cert-rotation workflow.
+
 ## Per-client SSO extension pattern
 
 Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentication.php`.
@@ -90,6 +94,7 @@ To onboard a new SSO client, add the class in `_underscore` — not in this repo
 - **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. The `/acs` handler calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
 - **SAML signature verification is commented out — this is the repo's top security priority.** The IdP X509 cert verification block in the `/acs` flow is disabled, so the gateway accepts any well-formed, status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat model: an attacker who can craft or replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user, since nothing binds the assertion to a trusted signing key. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix for this repo.
 - **Plaintext secrets in source — move to AWS SSM Parameter Store.** `saml/Config/production.ini` stores credentials in plaintext, and a Sentry DSN is hardcoded in `saml/Controller/Index.php`. Both are checked into the repo. Migrate these to AWS SSM Parameter Store (SecureString) and load at runtime; the source tree should reference parameter names only, never literal credential values.
+- **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; the cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both the signing and encryption blocks.
 - **SLS not implemented.** `/sls` falls through to the default banner.
 - **`set_exception_handler(null)` at top of `saml()`.** Unhandled exceptions output raw PHP errors. All exceptions from `getAuthenticatedSsoUser()` are caught and sent to Sentry.
 
@@ -97,3 +102,4 @@ To onboard a new SSO client, add the class in `_underscore` — not in this repo
 - 2026-06-11 — Added `transactionCommit()` before redirect; wrapped `getAuthenticatedSsoUser()` in try/catch with Sentry; echo+exit on auth failure (rgirish)
 - 2026-06-25 — Sharpened signature-verification gotcha with explicit threat model and flagged it as the repo's top security priority; documented plaintext secrets in `production.ini` + hardcoded Sentry DSN in `Controller/Index.php` with SSM Parameter Store recommendation (jcardinal)
 - 2026-08-27 — Documented non-prod shared-ALB self-registration postdeploy hook (`060_...sh` + `ebs/register_instance_to_shared_application_load_balancer.php`); skips `production`, requires `aws/aws-sdk-php` + two IAM actions; cross-linked worker2 ALB auto-registration feature (jcardinal)
+- 2026-09-04 — `/acs` now dual-credential decrypt (`decryptMultiAssertion([new, old])`) for SP encryption-cert overlap; documented `/meta`'s two KeyDescriptors (signing vs encryption) and the global outbound-signing key; added the CRLF `/meta` cert-line `trim()` gotcha. (jcardinal)