toga-ai 1.0.256 → 1.0.258

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-06-25
+updated: 2026-06-30
 owners: ["dfranks"]
 files:
   - library/app/api/netsuite/rest.php
@@ -330,9 +330,23 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
   (PHP 8.0) is for running probes on the laptop only, **not** for compat checking.
 - Field/relationship availability varies by NetSuite account **and** record type — probe the live
   account before assuming a column exists.
+- **`IS NOT NULL` on a custom `transactionline` column inside an aggregate query is a ~15× planner
+  trap — prefer `> 0` (or `NVL(col,0) <> 0`).** Filtering a custom employee-ref column
+  (`custcol_sales_rep_line`) with `AND tl.custcol_sales_rep_line IS NOT NULL` inside a
+  `GROUP BY` / power-sum aggregate (`MOD(SUM(MOD(id*id,P)),P)`) made the planner scan
+  pathologically: **~122s** vs **~8s** for the semantically identical `AND tl.custcol_sales_rep_line
+  > 0` (employee internalIds are always positive) — a ~15× regression purely from the predicate form.
+  Measured against the Forecast2 reconciliation checker (TRUE-79968). Use `> 0` / `NVL(col,0) <> 0`
+  for a custom ref column you only need to test for presence in an aggregate; reserve `IS NOT NULL`
+  for non-aggregate row filters.
 
 ## Change history
 
+- 2026-06-30 — **Recorded the `IS NOT NULL` vs `> 0` planner trap for custom `transactionline`
+  columns in aggregate queries** (TRUE-79968): a custom employee-ref column filtered with
+  `IS NOT NULL` inside a power-sum `GROUP BY` aggregate scanned ~15× slower (~122s) than the
+  identical `> 0` predicate (~8s). Prefer `> 0` / `NVL(col,0) <> 0` for presence tests in
+  aggregates. (dfranks)
 - 2026-06-25 — **Documented Journal Entry line structure + the JE→Forecast.Sales deferral.** JEs
   invert the `mainline` rule — **all** JE `transactionline` rows are `mainline='T'` (no `'F'` detail
   split), so a `mainline='F'` revenue query over a JE returns empty. Also: `tl.salesrep` does not

package/knowledge/1.0/apps/tools/features/saml-sso-auth.md

@@ -36,10 +36,13 @@ auto-created.
 ## How it works
 
 ### Wiring overview
-- **`/sso/initiate`** (`mvc/sso/initiate/get.php`) — guards `isLoggedIn`, reads `config[saml]`,
-  calls `App_Sso::initiate([...])` and `App_MVC::routeTo($url); exit;`. Initiation now uses the
-  reusable 1.0 [`App_Sso`](../../library/features/app-sso-initiation.md) library class (no longer a
-  placeholder).
+- **`/sso/initiate`** (`mvc/sso/initiate/get.php`) — guards `isLoggedIn`, reads `config[saml]`
+  for the IdP/ACS urls + `domain_uuid`, sources the **rotating shared secret from `Core.Parameters`
+  via `App_Auth::currentApiSecret()`** (NOT from config — see below), calls `App_Sso::initiate([...])`
+  passing `'apiSecretAccessToken' => App_Auth::currentApiSecret()`, then `App_MVC::routeTo($url); exit;`.
+  Initiation uses the reusable 1.0 [`App_Sso`](../../library/features/app-sso-initiation.md) library
+  class (no longer a placeholder). The secret encrypts the RelayState — see the non-empty-token
+  gotcha below.
 - **`/sso`** (`mvc/sso/get.php`) — the handoff consumer (below).
 - **Core.Domains** — a fixed-uuid row registers this app's return domain so the gateway resolves
   client/environment/return URL: `uuid 2927bc15-e347-4358-a430-fb28f9446d27`, `clientId 1` (True),
@@ -59,18 +62,29 @@ auto-created.
    and that both decrypted values match a UUID regex.
 4. Load the active Client_True user: `WHERE uuid = ? AND isActive = 1`. No match → fail closed.
 
-### Handoff key sourcing (`handoffKeys()` → `Core.Parameters`)
-The shared secret is **rotated regularly**, so the static `config.*.ini [saml]` token goes stale —
+### Handoff key sourcing (`handoffKeys()` → `Core.Parameters`) — used by BOTH SSO legs
+The shared secret is **rotated regularly**, so any static `config.*.ini [saml]` copy goes stale —
 that staleness was exactly why `decryptHandoffValue()` returned `false` and production login failed
 (the payload decoded to the correct iv+base64 shape; it was a **key** mismatch, not a format bug).
-- `App_Auth::handoffKeys()` (new private helper) `SELECT \`key\`,\`value\` FROM Parameters WHERE
+**Both SSO legs must source the secret from `Core.Parameters` at runtime** — the initiate leg
+encrypts the RelayState with it, the consumer leg decrypts the handoff with it. A config-only copy
+silently breaks SSO on the next rotation.
+- `App_Auth::handoffKeys()` (private helper) `SELECT \`key\`,\`value\` FROM Parameters WHERE
   \`key\` IN ('API_SECRET_ACCESS_TOKEN','API_SECRET_ACCESS_TOKEN_PREVIOUS')` over the **`db_toga2core`**
   connection (2.0 PROD Core DB; `const CORE_DB = 'db_toga2core'`). Result is **cached per request**.
   `\`key\`` and `\`value\`` are SQL reserved words → **must be backticked**.
-- `decryptHandoffValue()` then tries the current key, falling back to previous (rotation window).
-  Its **public signature is unchanged** — only `mvc/sso/get.php` calls it.
-- The old `[saml] api_secret_access_token` / `_previous` config values are now **dead** (removable).
-  `[saml] true_client_uuid` is **still used** (`sso/get.php:80-84`) and stays.
+- **Consumer leg** (`mvc/sso/get.php`): `decryptHandoffValue()` tries the current key, falling back
+  to previous (rotation window). Its **public signature is unchanged**.
+- **Initiate leg** (`mvc/sso/initiate/get.php`): calls **`App_Auth::currentApiSecret()`** (public;
+  reuses the cached `handoffKeys()` lookup, returns the current `API_SECRET_ACCESS_TOKEN`) and passes
+  it to `App_Sso::initiate(['apiSecretAccessToken' => …])` to encrypt the RelayState.
+- **Do NOT remove `[saml] api_secret_access_token` / `_previous` from config.** ⚠ An earlier capture
+  this session wrongly called these "dead/removable" — that guidance **broke SSO**: the initiate leg
+  still read the config token, so removing it made `App_Sso::initiate()` throw on an empty token
+  (`/login?error=1`). The fix moved the initiate leg to `Core.Parameters` too; the config copies are
+  now genuinely unused, but the durable rule is that the **only source of truth is `Core.Parameters`**.
+- **`[saml] true_client_uuid` is still read from config** (`sso/get.php:80-84`) and **must NOT be
+  removed**. The IdP/ACS urls and `domain_uuid` also stay in config.
 - See [Core.Parameters key/value store](#coreparameters-key-value-store) for the table shape and
   which DB cluster it lives on.
 
@@ -115,6 +129,19 @@ removed.
   preloader and runs `App_Page::flushCapture()` (`ob_end_flush + flush`, `library/app/page.php:213`)
   **before** `body()/App_MVC::loadFile()` runs the mvc page. Fixed with the same
   `if (!headers_sent()) { http_response_code(401); }` guard (matches `App_MVC::routeTo`'s own check).
+- **`App_Sso::initiate()` hard-requires a non-empty `apiSecretAccessToken`** (`library/app/sso.php:64-65`,
+  `empty()` guard → `InvalidArgumentException`; the token encrypts the RelayState). An empty/absent token
+  surfaces as a redirect to **`/login?error=1` ("Sign in failed. Please try again.")** — the initiate leg
+  catches the exception (`sso/initiate/get.php:30-34`) and routes to `/login?error=1`. This fails
+  **before** the IdP round-trip, so it never reaches the consumer's **"Authentication failed"**
+  (`tools_ssoFail`) page. **The two error UIs tell you which leg failed:** `/login?error=1` = initiate
+  (RelayState encryption / token), "Authentication failed" = consumer (decrypt). This was the
+  2026-06-30 regression — removing the config token per the (wrong) earlier note made initiate throw.
+- **Both legs depend on `db_toga2core` + a populated `Core.Parameters` token.** If `[database_toga2core]`
+  is not configured, `currentApiSecret()`/`handoffKeys()` runs `App_Database::query` against an
+  unregistered alias and **throws**. Because initiate only catches `InvalidArgumentException`/`RuntimeException`,
+  that surfaces as a **500** — not the login banner. **Diagnostic:** a 500 at initiate = missing
+  `db_toga2core`; `/login?error=1` = empty/absent `API_SECRET_ACCESS_TOKEN`.
 - **No app-side replay defense.** The handoff token carries no nonce/timestamp the app verifies.
   Recommend the gateway embed `iat` + `jti`.
 
@@ -140,8 +167,11 @@ from the 1.0 prod-cluster and from the 2.0 client cluster that backs `db_true`/`
 `db_toga2core`, dbname `Core`, the prod core cluster host (`production-core-cluster…us-west-2…`),
 needed in **each** `config.*.ini` so the handoff keys can be read at runtime (developer is adding
 creds). `[saml]`: `true_client_uuid` (still used), `client_authentication_uuid`, `domain_uuid`,
-plus the IdP/ACS urls used by initiation; `[internal]` `dev_mode`. The old `[saml]
-api_secret_access_token` / `_previous` are now **dead** (superseded by `Core.Parameters`).
+plus the IdP/ACS urls used by initiation; `[internal]` `dev_mode`. `[saml] api_secret_access_token` /
+`_previous` are **superseded by `Core.Parameters`** (the only source of truth for BOTH legs) — but do
+**not** treat them as freely removable: removing the config token while the initiate leg still read it
+is what caused the 2026-06-30 `/login?error=1` regression. `[saml] true_client_uuid` is **still read
+from config** and must stay.
 
 **Secret location.** Rotating secrets are now sourced from `Core.Parameters` **at runtime** rather
 than static config — an improvement over the old committed `config.production.ini [saml]` tokens.
@@ -150,6 +180,7 @@ remains (already noted in tools knowledge) — **flag for rotation.** Document *
 never the values.
 
 ## Change history
+- 2026-06-30 — Fixed SSO "Sign in failed" (`/login?error=1`) regression and **corrected the earlier wrong "config tokens are removable" note**. The prior fix moved only the consumer/decrypt leg to `Core.Parameters` and advised the `[saml] api_secret_access_token` was removable — but the **initiate** leg (`mvc/sso/initiate/get.php`) still read it from config, so removing it made `App_Sso::initiate()` throw on an empty token (`library/app/sso.php:64-65`) → `/login?error=1`. Fix: **both** legs now source the rotating secret from `Core.Parameters` — added public `App_Auth::currentApiSecret()` (reuses cached `handoffKeys()`), and `sso/initiate/get.php` passes it instead of the config value. Documented the non-empty-token requirement, the `/login?error=1` (initiate) vs "Authentication failed" (consumer) leg distinction, the `true_client_uuid`-stays-in-config rule, and the `db_toga2core`-missing→500 diagnostic. `php -l` clean on both files. (jcardinal)
 - 2026-06-30 — Fixed prod SSO login failure: handoff decrypt keys now sourced from `Core.Parameters` (rotated `API_SECRET_ACCESS_TOKEN`/`_PREVIOUS`) via new `App_Auth::handoffKeys()` over `db_toga2core` (per-request cached, current→previous fallback), instead of the stale committed `config [saml]` tokens — the stale key was the root cause. Added `[database_toga2core]` connection requirement to every config and documented the AES-256-CBC crypto format. Fixed `tools_ssoFail()` 500→intended 401: `http_response_code(401)` fataled on headers-already-sent (preloader flushes before mvc page runs); guarded with `!headers_sent()`. (jcardinal)
 - 2026-06-26 — Wired up real SSO initiation via the new 1.0 `App_Sso` library class (`/sso/initiate`), replacing the `initiation_url` placeholder; registered the fixed-uuid Core.Domains return row (dbchanges2 `2026-06-26a`); fixed the `session_regenerate_id` headers-already-sent fatal (guarded with `!headers_sent()`) and documented the mid-render failure-path limitation; collapsed home → `/login` to one-step sign-in; noted prod secrets live in `config.production.ini [saml]` (jcardinal)
 - 2026-06-25 — Built App_Auth: SAML `?saml=` handoff consumer with dual-key decrypt, hash_equals client-uuid check, fail-closed 401, persona-cached session (HttpOnly+SameSite=Lax), and a double-gated dev bypass. Initiation + replay defense left as open items (jcardinal)

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -233,6 +233,22 @@ by reconciling a chosen tranDate range directly against NetSuite.
   are correctly excluded **by design** (e.g. JE 7213407 = a Bank↔Equity reclass, 0 allowlist lines → never
   imported — not a miss). An accrual JE and its NetSuite auto-reversal are **separate transactions**, each
   reconciled independently by its own `internalId`.
+- **The rep-only-NS-side / unfiltered-FC-side asymmetry is DELIBERATE and load-bearing — do NOT add
+  a rep filter to any Forecast-side query (TRUE-79968, dfranks 2026-06-30).** The importer now keeps
+  **only JE lines that carry a raw `custcol_sales_rep_line`** (see the sale-import doc). To make the
+  fixer be the **cleanup mechanism** for any pre-existing no-rep JE rows (there is no separate delete
+  script), the **NS-side** JE SuiteQL in both `checker.php` (`journalEntryNsTotals` arm of the sales
+  `nsSub`) and `fixer.php` (`journalEntryNsTotals` FIND + `fixJournalEntries` FIX) is scoped to
+  **rep-bearing lines** (mirroring the importer), while the **Forecast-side** reads in both tools stay
+  **UNFILTERED** (still SUM all JE rows, incl. any `salesRepEmployeeId IS NULL`). That asymmetry is
+  what surfaces the stale rows: NS (rep-only) **<** FC (still holds no-rep rows) → FIND flags the JE →
+  the existing per-`'journalEntry'` orphan-delete in `fixJournalEntries` removes them. **If the FC side
+  were ALSO rep-filtered, both sides would match and the stale rows would be invisible / never purged.**
+  A blanket `DELETE WHERE salesRepEmployeeId IS NULL` is the wrong fix — locally-null also means
+  "rep present in NetSuite but unmapped to `Forecast.Employees`," which must survive; the fixer sweep
+  correctly re-derives "no raw rep" from NetSuite. One-time cleanup =
+  `fixer.php --commit --prod --category sales` over the JE window, then `checker.php` to confirm
+  (in current prod this was a no-op — 0 no-rep JE rows — so the importer filter is preventative).
 - **JE item dimension is NULL today; here is how to enable it later.** `itemId` is null on JE Sales rows
   because the JE import path reads item **only** via the configurable `JE_LINE_ITEM_FIELD` (currently null)
   — it does **not** read native `transactionline.item`, so even a populated native `tl.item` would NOT flow
@@ -468,11 +484,32 @@ None — Forecast2 is a single shared dataset.
   invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
   OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
   will overwrite the importer's value to NULL on its next run.
+- **`IS NOT NULL` on a CUSTOM `transactionline` column inside a SuiteQL aggregate is a ~15× planner
+  trap — use `> 0` instead.** Filtering `custcol_sales_rep_line` (an employee-ref custom column) with
+  `AND tl.custcol_sales_rep_line IS NOT NULL` inside checker's UNION'd moment-fingerprint (the
+  `MOD(SUM(MOD(id*id,P)),P)` power-sum aggregates) made the planner scan pathologically: a full-window
+  checker run measured **~122s** with `IS NOT NULL` vs **~8s** with the semantically identical
+  `AND tl.custcol_sales_rep_line > 0` (employee internalIds are always positive). Applied to
+  `checker.php` (JE arm) and `fixer.php` (both JE queries) with inline comments warning against
+  reverting. See the SuiteQL API reference for the general rule. (Use `> 0`, or `NVL(col,0) <> 0`,
+  never `IS NOT NULL`, for a custom employee-ref `transactionline` column in a SuiteQL aggregate.)
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-30 — **Made the fixer the cleanup mechanism for no-rep JE rows + killed a ~15× SuiteQL
+  planner trap (TRUE-79968, dfranks).** The importer now keeps only rep-bearing JE lines (see the
+  sale-import doc); to let `fixer.php` purge any pre-existing no-rep `Forecast.Sales` JE rows with no
+  separate delete script, the **NS-side** JE SuiteQL in both `checker.php` and `fixer.php` was scoped
+  to rep-bearing lines while the **FC-side reads stay UNFILTERED** — a deliberate, load-bearing
+  asymmetry (NS rep-only < FC still-has-no-rep → FIND flags → existing per-`'journalEntry'` orphan
+  delete removes them; filtering the FC side too would hide them forever). A blanket
+  `DELETE WHERE salesRepEmployeeId IS NULL` is wrong (locally-null also = NS-present-but-unmapped rep,
+  which must survive). **Perf:** replaced `AND tl.custcol_sales_rep_line IS NOT NULL` with
+  `AND ... > 0` in checker (JE arm) + fixer (both JE queries) — a clean full-window checker run dropped
+  from **~122s to ~8s** purely from the predicate form, with inline anti-revert comments. Current prod
+  had 0 no-rep JE rows, so the cleanup sweep was a no-op (filter is preventative). (dfranks)
 - 2026-06-30 — **Folded JOURNAL ENTRIES into the SALES reconciliation in `fixer.php` + `checker.php`
   (TRUE-79862).** Architecture decision (dfranks): a JE is **not its own category** — JE rows live in the
   same `Forecast.Sales` table (`netsuiteTransactionType='journalEntry'`) as the four sale types, so the

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-29
+updated: 2026-06-30
 owners: [dfranks]
 files:
   - _underscore/Component/Forecast/SaleImport/SaleImport.php
@@ -163,6 +163,19 @@ location, memo). Handler `_Worker_Netsuite_JournalEntry`: `post`/`put` → `sync
 item-line upsert/reconcile machinery (`syncLines`/`guardedInsert`/`deleteRows`).
 
 **Mapping model (durable design):**
+- **Only JE lines that carry a sales rep are imported (TRUE-79968, dfranks 2026-06-30).**
+  `buildJournalEntryRows` applies a **PER-LINE** filter **before** the (salesRep,item)
+  bucketing: a GL line whose **raw** `custcol_sales_rep_line` is empty is **skipped**
+  (`continue`). Consequences — a JE with no rep on any line yields **zero** `Forecast.Sales`
+  rows; a mixed JE keeps only its rep-bearing groups; offset/balancing lines (which omit the
+  custom column) drop out naturally. **The test is on the RAW NetSuite line field, NOT the
+  mapped local `salesRepEmployeeId`** — so a rep present in NetSuite but **not yet mapped** to
+  a local `Forecast.Employees` row **still qualifies** (its group keeps a null
+  `salesRepEmployeeId`). This is deliberate: it keeps no-rep JE noise out of Forecast revenue
+  while not silently dropping a real-but-unmapped rep's revenue. (A blanket
+  `DELETE WHERE salesRepEmployeeId IS NULL` would be **wrong** for the same reason — locally
+  null also means "rep present in NS but unmapped"; any cleanup must re-derive "no raw rep"
+  from NetSuite. See the reconciliation doc for how the fixer sweep does exactly that.)
 - **Sales rep is now LIVE on JE lines** as the custom column **`custcol_sales_rep_line`**
   (`JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'`, wired 2026-06-29 after live Apr–May
   2026 probing). The value is a NetSuite **employee REFERENCE object** `{links, id, refName}`;
@@ -429,6 +442,20 @@ record is deleted in NetSuite.)
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-06-30 — **JE import now keeps ONLY rep-bearing lines** (TRUE-79968, dfranks).
+  `buildJournalEntryRows` skips any GL line whose **raw** `custcol_sales_rep_line` is empty,
+  applied **per-line before** the (salesRep,item) bucketing: a no-rep JE → 0 Sales rows, a
+  mixed JE keeps only rep-bearing groups. The filter tests the **RAW NS field**, not the
+  mapped local `salesRepEmployeeId`, so a rep present in NetSuite but unmapped to
+  `Forecast.Employees` still qualifies (group stays null `salesRepEmployeeId`) — which is also
+  why a blanket `DELETE WHERE salesRepEmployeeId IS NULL` is wrong; cleanup must re-derive
+  "no raw rep" from NetSuite. No-rep cleanup is handled by the existing `fixer.php` orphan-
+  delete sweep (see the reconciliation doc), not a separate delete script — in current prod
+  there were 0 no-rep JE rows so the filter is preventative going forward.
+  **Confirmed the JE webhook handler `_Worker_Netsuite_JournalEntry`
+  (worker2/Worker/Netsuite/JournalEntry.php) holds no business logic** — post/put delegate to
+  `_Component_Forecast_SaleImport::syncJournalEntry`, delete to `removeAllJournalEntry`; editing
+  the importer in `_underscore` is the single correct entry point for JE import changes. (dfranks)
 - 2026-06-29 — **Wired JE sales-rep ingestion live** (TRUE-79862): set
   `JE_LINE_SALESREP_FIELD = 'custcol_sales_rep_line'` after live Apr–May 2026 probing
   confirmed the rep is now present on JE lines as that custom column — a NetSuite employee

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.256",
+  "version": "1.0.258",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",