toga-ai 1.0.207 → 1.0.209

package/knowledge/1.0/apps/library/INDEX.md

@@ -7,7 +7,7 @@
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
 | [1.0 MVC Page Pattern & New-App Skeleton](features/mvc-page-pattern-and-app-skeleton.md) | This is the **reusable recipe for standing up a new 1.0 (`App_`) application** and for adding pages to one — the folder-based MVC routing, the page lifecycle, t | library/app/framework.php, library/app/frameworkindex.php, library/app/mvc.php, library/app/database.php, library/app/model.php, library/app/config.php |
-| [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem |
+| [NetSuite SuiteQL/REST API Reference](features/netsuite-suiteql-api-reference.md) | General working reference for the Agilant NetSuite integration: how to authenticate, how SuiteQL behaves, and the confirmed schema of the tables/columns/codes w | library/app/api/netsuite/rest.php, library/ssl/netsuite_ec_key.pem, test/@dave/Junk Drawer/nsq.php |
 | [NetSuite SuiteQL/REST Shim — Field Semantics](features/netsuite-suiteql-rest-shim.md) | `App_Api_Netsuite_Rest` is the REST/SuiteQL replacement for the deprecated NetSuite SOAP toolkit. | library/app/api/netsuite/rest.php |
 | [Startech PC Matic B2B Sync (library)](features/startech-pcmaticb2b-sync.md) | `library/app/api/toga2.php` handles bidirectional ticket sync for PC Matic B2B between TOGaDesk 1.0 and TOGA 2.0. | library/app/api/toga2.php |
 | [App_Api_Toga2 — TOGa2 API Client & 1.0↔2.0 Sync Bridge](features/toga2-api-client-and-bridge.md) | `App_Api_Toga2` (`library/app/api/toga2.php`, ~8400 lines) is the **1.0-side client for the TOGa 2 (`_underscore`/api2) public API** *and* the home of the cross | library/app/api/toga2.php, worker/crons/toga2/aig/sync_togasupply_aig.php, worker/crons/toga2/wje/sync_togasupply_wje.php |

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -6,11 +6,12 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-06-11
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - library/app/api/netsuite/rest.php
   - library/ssl/netsuite_ec_key.pem
+  - test/@dave/Junk Drawer/nsq.php
 related:
   - netsuite-suiteql-rest-shim.md
   - ../architecture.md
@@ -183,6 +184,30 @@ and custom fields (`CUSTBODY_END_CUSTOMER`, `CUSTBODY_STOCKING_ORDER`, …).
 - **Open-order (in-flight) product revenue:** `((-tl.quantity) - NVL(tl.quantitybilled,0)) * tl.rate`;
   shipping term `SUM(-tl.foreignamount)` over `tl.itemtype = 'ShipItem'`.
 
+### Journal Entry lines — the `mainline` rule INVERTS
+
+Journal Entries break the `mainline='F' = detail / mainline='T' = header` rule above — **all JE
+`transactionline` rows have `mainline='T'`** (there is no header/`mainline='F'` split). A revenue
+query that filters `mainline='F'` over a JE returns **empty**. When querying JE lines, do **not**
+filter `mainline='F'`. Confirmed by probing accrual JE #6962 (internalId 7128398), reclass JE #6967
+(7128548), and 15 recent revenue JEs.
+
+Other JE line facts (same probes):
+- **`tl.salesrep` does NOT exist** on `transactionline` — selecting it is an HTTP 500. (There is no
+  line-level sales-rep dimension on a JE.)
+- **`tl.item` is NEVER populated on JE revenue/cost lines** (`COUNT(item) = 0` across every sampled
+  JE) — JEs post to accounts, not items.
+- The only line-level dimensions present on a JE are **`class`** and **`entity`** (customer), and
+  only when the preparer fills them in.
+- Revenue line = `accttype IN ('Income','OthIncome')`; cost line = `accttype = 'COGS'`.
+- An auto-created **reversing JE** (`isreversal='T'`) fires **NO webhook** — if a sync consumes JEs it
+  must synthesize the reversal locally.
+
+> **Deferral decision (2026-06-25):** adding Journal Entries as a 5th `Forecast.Sales` transaction
+> type is **DEFERRED**. The intended `(salesRep, item)` grouping is **impossible** on JE lines until a
+> NetSuite custom field supplying those dimensions is created and populated (no `salesrep` column, no
+> `item`). Re-evaluate only once that custom field exists — don't re-probe this from scratch.
+
 ## `previousTransactionLineLink` table
 
 The bridge between a child line (e.g. invoice) and its originating parent line (e.g. SO).
@@ -308,6 +333,15 @@ numbers. Budget hours for multi-year runs and launch them under `nohup`/`tmux`.
 
 ## Change history
 
+- 2026-06-25 — **Documented Journal Entry line structure + the JE→Forecast.Sales deferral.** JEs
+  invert the `mainline` rule — **all** JE `transactionline` rows are `mainline='T'` (no `'F'` detail
+  split), so a `mainline='F'` revenue query over a JE returns empty. Also: `tl.salesrep` does not
+  exist (HTTP 500), `tl.item` is never populated on JE lines (post to accounts, not items), the only
+  line dims are `class`/`entity` (when filled), revenue = `accttype IN ('Income','OthIncome')` / cost
+  = `'COGS'`, and an auto-reversing JE (`isreversal='T'`) fires no webhook. Recorded the decision to
+  **defer** adding JEs as a 5th `Forecast.Sales` type — the intended `(salesRep, item)` grouping is
+  impossible until a NetSuite custom field supplies those dims. Probed via `test/@dave/Junk
+  Drawer/nsq.php` against accrual JE #6962, reclass #6967, and 15 recent revenue JEs. (dfranks)
 - 2026-06-11 — Initial reference captured from live-probe notes (TRUE-79183): SuiteQL mechanics,
   ET session timezone, transaction/transactionline/previousTransactionLineLink/shippingAddress/item
   schemas, type & status codes, custom-field map, REST GET-only fields, `list*()` cost model and

package/knowledge/1.0/apps/tools/INDEX.md

@@ -6,3 +6,4 @@
 | [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
 | [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php |
 | [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php |
+| [Deploying Tools to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the **Tools** 1.0 app boots on Elastic Beanstalk running `PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. | tools/.ebextensions/004_http_to_https.config, tools/.ebextensions/006_mount-s3fs.config, tools/.ebextensions/007_setup_export_cache_folders.config, tools/.ebextensions/008_setup_ldap.config, tools/.ebextensions/009_setup_phpini.config, tools/.ebextensions/020_setup_git_libraries.config, tools/.ebextensions/050_register_instance_to_shared_application_load_balancer.config, tools/ebs/git.json |

package/knowledge/1.0/apps/tools/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -0,0 +1,126 @@
+---
+title: Deploying Tools to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)
+framework: "1.0"
+repo: tools
+project: Tools
+client: shared
+type: workflow
+status: active
+updated: 2026-06-26
+owners: [jcardinal]
+files:
+  - tools/.ebextensions/004_http_to_https.config
+  - tools/.ebextensions/006_mount-s3fs.config
+  - tools/.ebextensions/007_setup_export_cache_folders.config
+  - tools/.ebextensions/008_setup_ldap.config
+  - tools/.ebextensions/009_setup_phpini.config
+  - tools/.ebextensions/020_setup_git_libraries.config
+  - tools/.ebextensions/050_register_instance_to_shared_application_load_balancer.config
+  - tools/ebs/git.json
+related:
+  - ../architecture.md
+---
+
+## Summary
+
+How the **Tools** 1.0 app boots on Elastic Beanstalk running
+`PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. The EB platform was upgraded from
+Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023), which changes package names, the proxy
+(nginx instead of Apache), the PHP ini scan directory, and how credentials reach the
+instance. The `.ebextensions/*.config` files run alphabetically and are merged by EB; getting
+them wrong leaves the app booting without `/var/www/library` and serving 500s on every
+request. This doc is the migration checklist and the durable AL2023 gotchas learned doing it.
+
+## AL2023 package migration checklist
+
+When migrating any 1.0 app's `.ebextensions` from AL2 to AL2023, rename/remove these yum
+packages (AL2023 uses different names for the same libraries):
+
+- `libstdc++48` → `libstdc++`
+- `php73-ldap` → `php-ldap` (in `008_setup_ldap.config`)
+- **Remove `libcurl: []` entirely** — AL2023 ships `libcurl-minimal` pre-installed, and adding
+  `libcurl` to the yum packages list causes a package-conflict error that fails the deploy.
+
+## How it works
+
+### PHP ini — write the file directly, do not run a script
+
+`009_setup_phpini.config` writes `/etc/php.d/application.ini` with a cfn-init **`files:`**
+block, **not** a container_command that runs a PHP script. On AL2023, `/etc/php.d/` is the
+PHP ini scan directory, and cfn-init `files:` runs before container_commands and before
+PHP-FPM starts, so the file is guaranteed present at startup with no PHP execution needed.
+
+The old approach ran `ebs/setup_phpini.php` (now superseded) which called
+`php_ini_scanned_files()`, parsed it to find the scan dir, and wrote there. On AL2023 that
+failed silently (it was wrapped in `ignoreErrors: true`), so `include_path` never picked up
+`/var/www/library`, producing fatal `require_once('_.php')` errors at runtime.
+
+The `application.ini` sets `include_path = ".:/var/www/library"`, upload/post limits (16M),
+`memory_limit = 2G`, `display_errors`, the `redis.so` extension, and the rediscluster session
+handler (`session.save_path` points at the ElastiCache cluster cfg endpoint). Session GC
+maxlifetime is 4 days.
+
+### Cloning the libraries — bash, not the CodePipeline PHP script
+
+`020_setup_git_libraries.config` clones the framework libraries onto the instance. The old
+`setup_git_libraries.php` (downloaded from S3) requires a DB connection to the `Logs` database
+and a matching row in the `CodePipelineEnvironments` table; for a new app with **no**
+CodePipeline it exits silently without cloning anything, so the app boots without
+`/var/www/library` and 500s on every request.
+
+The replacement is pure bash via a cfn-init `files:` block that writes
+`/tmp/clone_git_repos.sh` (mode `000755`) plus a `container_commands` entry that runs
+`bash /tmp/clone_git_repos.sh`. It clones:
+
+- `agilantsolutions/library`   branch `_production` → `/var/www/library`
+- `agilantsolutions/resources` branch `_production` → `/var/app/ondeck/resources`
+
+`ebs/git.json` was corrected to match: branch `_production` for both repos and the resources
+path set to `/var/app/ondeck/resources`.
+
+### Duplicate config neutralization
+
+EB merges every `.config` alphabetically; two configs with **identical** `container_commands`
+keys silently overwrite each other, causing unpredictable failures (the redis build ran twice;
+an http→https collision obscured failure diagnosis). Two duplicate pairs were neutralized
+(replaced with a comment), keeping the canonical one of each:
+
+- `004_enforce_http_to_https.config` (dup of `004_http_to_https.config`) — neutralized
+- `010_setup_redis_sessions.config` (dup of `010_setup_redis.config`) — neutralized
+
+### ignoreErrors on non-fatal infrastructure commands
+
+Several `container_commands` write to system paths that do not exist on AL2023 (e.g.
+`/etc/httpd/conf.d/` — AL2023 EB uses **nginx**, not Apache, as the reverse proxy). These were
+marked `ignoreErrors: true` in `004`, `006`, `007`, `009`, and `050` so a non-fatal failure
+does not block the deploy. This also covers `006`'s `01_credentials` step: modern EB uses IAM
+roles, so `AWS_SECRET_KEY` / `AWS_ACCESS_KEY_ID` are not injected into `$_SERVER` and the
+s3fs credential write fails cleanly. Use `ignoreErrors` only on genuinely non-fatal infra
+setup — the git clone (`020`) deliberately does **not** use it (see gotchas).
+
+## Gotchas (durable AL2023 / EB rules)
+
+- **Write new-deploy files to `/var/app/ondeck/<path>`, never `/var/www/html/<path>` during
+  `container_commands`.** While container_commands run, `/var/www/html` still symlinks to the
+  **old** app. After the deploy swap, `/var/app/ondeck` becomes `/var/app/current`
+  (= `/var/www/html`). Writing to `/var/www/html` during container_commands writes into the
+  currently-running old app.
+- **Only `/var/log/eb-activity.log` (and `cfn-init.log`) are captured in EB bundle snapshots.**
+  Custom log files under `/var/log/` are invisible unless you SSH the instance. Deploy scripts
+  should log to `/var/log/eb-activity.log`.
+- **CodePipeline-dependent scripts are no-ops for apps without a pipeline.**
+  `setup_git_libraries.php` silently does nothing when the DB has no `CodePipelineEnvironments`
+  row — fatal for a brand-new app, since nothing gets cloned and the failure is silent.
+- **Make clone failures block the deploy.** The bash clone script does `exit 1` on clone
+  failure so the deploy fails visibly, rather than succeeding and serving 500s.
+- **PATH is unreliable in the cfn-init execution environment** — resolve binaries defensively,
+  e.g. `GIT=$(which git || echo /usr/bin/git)`.
+- **cfn-init `files:` runs before `container_commands`** — the correct pattern for writing a
+  script and then executing it in the same config.
+
+## Change history
+- 2026-06-26 — Documented the AL2 → AL2023 (PHP 8.5) EB migration for Tools: package renames
+  (`libstdc++48`→`libstdc++`, `php73-ldap`→`php-ldap`, drop `libcurl`), php ini via cfn-init
+  `files:` writing `/etc/php.d/application.ini`, bash-based library clone replacing the
+  CodePipeline PHP script, duplicate-config neutralization, `ignoreErrors` on non-fatal infra
+  commands, and the ondeck-path / eb-activity.log / pipeline-noop gotchas (jcardinal)

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -10,6 +10,7 @@ updated: 2026-06-25
 owners: [dfranks]
 files:
   - test/@dave/reconcile_netsuite_totals.php
+  - test/@dave/fixer.php
   - test/@dave/analyze_netsuite_forecast_diff.php
   - test/@dave/trueup_sales.php
   - test/@dave/trueup_open_orders.php
@@ -49,6 +50,15 @@ by reconciling a chosen tranDate range directly against NetSuite.
   bucket. The diffs are **gated** because they materialize every transaction/order in the window
   into PHP arrays and OOM on multi-year windows — default is summary-only; positional `[from] [to]`
   work alongside the flag.
+- `fixer.php [--commit --prod]` — **self-contained** all-in-one delta finder + corrector across
+  **Sales + OpenOrderItems + Opportunities** in one tool (bootstraps the 1.0 framework; no shelling
+  out to the per-table trueups). **FIND** phase aggregates per-id diffs of all three categories,
+  NetSuite (SuiteQL) vs the prod `Forecast` reader, over a **fixed window** (current date back
+  through 2025-01-01); paginated `GROUP BY` made stable with `ORDER BY t.id`. **FIX** phase
+  (`fixSales`/`fixOpenOrders`/`fixOpportunities`) emulates the existing trueup logic **scoped to
+  only the discrepant ids**. **Default dry-run**; `--commit` requires `--prod` (both must be passed
+  to write). See the two baked-in gotchas (lazy self-reconnecting `fcReader()`; `buildDoubleFieldArray`
+  by-ref) and the **NS-deleted-invoice stale Sales row** failure mode below.
 - `analyze_netsuite_forecast_diff.php [from] [to]` — decomposes the delta **per transaction** into
   NS_ONLY (missing from FC), FC_ONLY (stale/extra), DRIFT (value differs). Read-only.
 - `trueup_sales.php --from --to [--chunk-days N] [--prod] [--dry-run]` — makes `Forecast.Sales`
@@ -143,6 +153,22 @@ None — Forecast2 is a single shared dataset.
 
 ## Gotchas / known issues
 
+- **An NS-DELETED invoice leaves STALE cost-only `Forecast.Sales` rows the add/update-only import can
+  never remove.** The 5-min `import_sales.php` cron is **add/update-only — it has no delete path** (only
+  the nightly discrepancy-fix deletes). So when an invoice is **deleted in NetSuite**, its already-imported
+  `Forecast.Sales` lines survive as orphans — and they survive in a telltale shape: **`revenue = 0`,
+  `profit = -cost`** (the revenue side is gone but the cost line lingers), inflating a **profit-only**
+  delta with no matching revenue delta. `fixer.php` detects and **deletes** them. Real example: closed a
+  **$12,838.84 profit delta** on 2026-06-18 — 5 NS-deleted invoices, 33 stale lines. When a profit-only
+  gap localizes to invoices that **no longer exist in NetSuite**, this is the cause; deletion (not trueup
+  re-insert) is the fix.
+- **`fixer.php` baked-in gotchas (apply to any long NS-bulk + Forecast-read tool):**
+  - **The Forecast reader connection idles out mid-run (Aurora idle-drop) during long NetSuite bulk
+    fetches.** Don't hold one long-lived handle across the NS calls — use a **lazy self-reconnecting
+    `fcReader()`** (ping-or-reconnect) called **immediately before each FC query**, not once up front.
+  - **`App_Database::buildDoubleFieldArray(App_Database::query(...))` throws "Only variables should be
+    passed by reference."** `buildDoubleFieldArray` takes its argument by reference, so a function-call
+    result can't be passed inline — **assign `query(...)` to a temp variable first**, then pass the temp.
 - **An audit must apply the sync's own inclusion rules or it manufactures phantom deltas.**
   `reconcile`'s NetSuite Sales SUM/diff originally had **no transaction-status filter**, but the
   sync (`trueup_sales` / `common_import_sales_from_netsuite.php`) deliberately **excludes** Invoice

package/knowledge/2.0/apps/_underscore/features/netsuite-rest-client.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - _underscore/Component/Api/Netsuite/Netsuite.php
@@ -48,10 +48,20 @@ through `_ApiRequest` directly (mirroring `send()`'s auth/endpoint/header setup)
 - `_ApiRequest` exposes **`responseCode` + `responseHeaders`** (a raw header string — there is **no
   `getHeader()`** accessor). `_Component_Api_Netsuite::send()` does **not** surface either.
 - Robust parse: `preg_match_all('/^Location:\s*(\S+)/im', $responseHeaders, ...)` and take the
-  **LAST** match (skips any 100-Continue / redirect block), then `#/(\d+)(?:[?#]|$)#` to pull the
-  trailing numeric id tolerating a query string or fragment.
+  **LAST** match (skips any 100-Continue / redirect block), then a trailing-id regex to pull the
+  numeric id tolerating a query string or fragment.
 - Throws on non-2xx, on no `Location` header, and on no id parsed.
 
+> **⚠ BUG (live, not yet fixed):** the trailing-id regex was written as `'#/(\d+)(?:[?#]|$)#'`,
+> which uses `#` as the PCRE **delimiter** *and* puts `#` inside the `[?#]` character class — so PHP
+> reads the class-`#` as a premature closing delimiter and the whole pattern **always throws
+> `preg_match(): Unknown modifier ']'`**. This fires *after* the record is already created, so the
+> create succeeds in NetSuite but `createRecord()` raises and the caller never gets the new id —
+> it breaks the entire outbound create/push path. **Fix direction:** change the delimiter so `#`
+> isn't both delimiter and class member, e.g. `~/(\d+)(?:[?#]|$)~`. Interim workaround used in
+> probes: recover the new id by regex-parsing the thrown exception message (the `Location` value
+> is in it). Confirmed live 2026-06-25.
+
 ### Update — reuse `send('PATCH', $route, $body)`
 
 Updates do **not** need a new helper. A NetSuite record PATCH returns 204 with no body, and
@@ -85,9 +95,17 @@ doc.)
 - **204-with-empty-body is success, not failure.** Both create (204 + `Location`) and update (204,
   no body) return no payload; treat a 2xx with empty body as success and key off the status, not the
   body.
+- **`createRecord()`'s trailing-id regex is a live `#`-delimiter bug** (`'#/(\d+)(?:[?#]|$)#'` →
+  always `Unknown modifier ']'`), thrown *after* the record is created — see the ⚠ note under
+  *Create* for the cause and the `~…~`-delimiter fix.
 
 ## Change history
 
+- 2026-06-25 — **Recorded a live bug in `createRecord()`'s trailing-id regex.** The pattern
+  `'#/(\d+)(?:[?#]|$)#'` uses `#` as both the PCRE delimiter and a class member, so it always throws
+  `Unknown modifier ']'` — *after* the record is created, breaking the outbound create/push path.
+  Fix direction: re-delimit (e.g. `~/(\d+)(?:[?#]|$)~`); interim workaround is to parse the new id
+  out of the thrown exception message. (dfranks)
 - 2026-06-24 — **Added `createRecord()` — the first REST write method in the framework.** Posts
   through `_ApiRequest` directly (since `send()` hides headers) and parses the new internalId from
   the 204 `Location` header (last match, trailing-id regex). Update path documented as reuse of

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -9,7 +9,7 @@
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
-| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
 | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -21,6 +21,8 @@ files:
   - test/@dave/clickup/backfill_opportunity_numbers.php
   - test/@dave/clickup/probe_opportunity_fields.php
   - test/@dave/probe_clickup_desc_match.php
+  - test/@dave/test_model_load_behavior.php
+  - "dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql"
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
 related:
   - ./netsuite-salesorder-open-orders-sync.md
@@ -163,7 +165,15 @@ Shared helpers: `buildCustomFields()` (the field array, used by both create and
 
 Writes `Forecast.Opportunities` (header) + `Forecast.OpportunityItems` (children), faithful to
 the legacy cron's OPPORTUNITIES section:
-- Upsert header by `netsuiteOpportunityInternalId`; items insert/update/delete by `line`; no
+- Upsert header by `netsuiteOpportunityInternalId` via a **business-key resolve**, NOT
+  `_Model::load()` — `SELECT id FROM Opportunities WHERE netsuiteOpportunityInternalId=? ORDER BY
+  id`, take the first row (or new), save, then **delete any extra duplicate rows** (and their
+  `OpportunityItems` children) inline so the handler is **self-healing** against pre-existing
+  duplicates (see the `load()` gotcha below for why `load()` cannot be used here).
+- A **UNIQUE index on `Opportunities.netsuiteOpportunityInternalId`** (migration
+  `dbchanges2/Forecast/2026-06-25a …`) is the only race-proof guard against concurrent/redelivered
+  webhooks creating a duplicate. The table was confirmed duplicate-free before the index was added.
+- items insert/update/delete by `line`; no
   header delete; children deleted before the header on DELETE.
 - Lookups: `customerId`/`salesRepEmployeeId` → null on miss; `forecastCategoryId`/`salesStageId`/
   `percentToCloseStatusId` → **create-on-miss** (+name update); unknown line item → **throws**
@@ -182,6 +192,19 @@ None — platform-wide Forecast sync.
 
 ## Gotchas / known issues
 
+- **`_Model::load()` returns TRUE only on an EXACTLY-ONE match — a duplicate-row self-amplifier in
+  any load-then-upsert handler.** `_Model::load()` (`_underscore/Model.php:788–808`) returns FALSE
+  for **both** 0 rows **and** 2+ rows — it succeeds only when the lookup matches exactly one row.
+  The old inbound opportunity handler used `load()` to find the existing header before its upsert, so
+  the instant an `netsuiteOpportunityInternalId` had **two** rows (from any earlier race/redelivery),
+  `load()` could never re-find it and **every subsequent webhook INSERTed yet another row** — a
+  self-amplifying duplicate (opp 7192265 reached **4 rows**, a $1,275 reconcile delta). This is a
+  **platform-wide footgun**: any `load()`-by-business-key-then-upsert pattern silently flips to
+  insert-only once a duplicate exists. **Fix:** resolve the id with a plain
+  `SELECT id … ORDER BY id` (take the first) instead of `load()`, and have the handler delete the
+  extra rows so it self-heals; back it with a UNIQUE index so duplicates can't form. The exactly-1
+  semantics is proven empirically by `test/@dave/test_model_load_behavior.php` (seeds 2 sentinel
+  rows, asserts `load()` → FALSE; self-cleaning local fixture).
 - **A User Event can't trigger another User Event → the immediate trigger lives in the enqueuer.**
   The enqueuer (a UE) creating the queue row does NOT fire any UE on the queue record (only the
   *scheduled* drainer's `submitFields` does). So a drain-UE on the queue record never runs for
@@ -364,6 +387,16 @@ same **skip-if-unchanged** compare on the extracted values, and **actor-identity
 trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
 
 ## Change history
+- 2026-06-25 — **Fixed the Opportunity duplicate-row amplification + added a unique-index guard.**
+  Root cause: `_Model::load()` returns TRUE only on an **exactly-one** match (FALSE for 0 *and* 2+),
+  so once an nsId had 2 rows the `load()`-then-upsert handler could never re-find it and every
+  webhook INSERTed another row (opp 7192265 hit 4 rows / $1,275 delta). Replaced `load()` with a
+  business-key `SELECT id … ORDER BY id` resolve + inline delete of extra duplicate rows and their
+  `OpportunityItems` children (self-healing), and added a **UNIQUE index on
+  `Opportunities.netsuiteOpportunityInternalId`** (`dbchanges2/Forecast/2026-06-25a …`; table
+  confirmed duplicate-free first) as the race-proof guard. Recorded the `load()` exactly-1 semantics
+  as a platform-wide footgun for any load-then-upsert handler; proven by
+  `test/@dave/test_model_load_behavior.php`. (dfranks)
 - 2026-06-25 — **Characterized the AMQ enqueuer scope + a live drainer TypeError.** The enqueuer is
   generic over all record types via `RECORD_TYPE_MAP` (no context filter, prod `DEV_OVERRIDE.enabled=false`)
   but probe-confirmed `Released` on **only Opportunity (`customdeploy1`) + Sales Order (`customdeploy2`)** —

package/knowledge/INDEX.md

@@ -12,7 +12,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
 - **test** (Test) — 11 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
 - **toga** (TOGa) — 2 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
-- **tools** (Tools) — 4 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
+- **tools** (Tools) — 5 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
 
 ## 2.0 framework
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.207",
+  "version": "1.0.209",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",