toga-ai 1.0.213 → 1.0.215

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -6,7 +6,7 @@
 | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Page.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql |
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
-| [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, test/@dave/test_invoice_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
+| [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
 | [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php |
 | [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
 | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php |

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -15,9 +15,15 @@ files:
   - worker2/Worker/Netsuite/CashSale.php
   - worker2/Worker/Netsuite/CreditMemo.php
   - worker2/Worker/Netsuite/CashRefund.php
+  - worker2/Worker/Netsuite/JournalEntry.php
   - worker2/Worker/Netsuite/Opportunity.php
   - worker2/Worker/Netsuite/SalesOrder.php
+  - dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql
   - test/@dave/test_invoice_lifecycle.php
+  - test/@dave/test_je_lifecycle.php
+  - test/@dave/test_creditmemo_lifecycle.php
+  - test/@dave/test_cashsale_lifecycle.php
+  - test/@dave/test_cashrefund_lifecycle.php
   - test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js
   - test/@dave/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js
 related:
@@ -30,13 +36,14 @@ related:
 ## Summary
 Real-time importer that takes a NetSuite **sale** record and writes its lines into
 `Forecast.Sales` (the Forecast2 revenue table). Covers the four NetSuite sale record
-types: **invoice, cashSale, creditMemo, cashRefund**. It is the webhook-driven replacement
-for the SALES section of the legacy 5-minute pull cron
+types: **invoice, cashSale, creditMemo, cashRefund**, plus **journalEntry** (GL
+revenue/cost adjustments posted directly as journal entries — see *JournalEntry import*).
+It is the webhook-driven replacement for the SALES section of the legacy 5-minute pull cron
 (`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php`), and mirrors the
 already-shipped SalesOrder (open-orders) and Opportunity webhook handlers.
 
-The shared engine is `_Component_Forecast_SaleImport`; four thin worker2 handlers
-(`_Worker_Netsuite_{Invoice,CashSale,CreditMemo,CashRefund}`) just delegate to it.
+The shared engine is `_Component_Forecast_SaleImport`; thin worker2 handlers
+(`_Worker_Netsuite_{Invoice,CashSale,CreditMemo,CashRefund,JournalEntry}`) just delegate to it.
 
 **Critical, non-obvious facts (read before touching this):**
 - **Sign convention differs from the cron** — the raw REST record is all-positive, so the
@@ -50,7 +57,9 @@ The shared engine is `_Component_Forecast_SaleImport`; four thin worker2 handler
 |---|---|
 | `_underscore/Component/Forecast/SaleImport/SaleImport.php` | the engine (shared body for all four types) |
 | `_underscore/Component/Forecast/Db/Db.php` | shared NS-REST + sublist-pagination helpers |
-| `worker2/Worker/Netsuite/Invoice.php` etc. | four thin handlers; `post`/`put` → `sync`, `delete` → `removeAll` |
+| `worker2/Worker/Netsuite/Invoice.php` etc. | four sale-type thin handlers; `post`/`put` → `sync`, `delete` → `removeAll` |
+| `worker2/Worker/Netsuite/JournalEntry.php` | thin JE handler; `post`/`put` → `syncJournalEntry`, `delete` → `removeAllJournalEntry` |
+| `test/@dave/test_je_lifecycle.php` | JE lifecycle harness (direct-engine create/update/delete) |
 
 Each handler is `abstract class _Worker_Netsuite_<Type> implements
 _Interface_Static_Webhook_Netsuite`. The router (`_Worker_NetSuite::Webhook`) PascalCases
@@ -65,7 +74,9 @@ Per-record flow in `sync`:
 1. REST GET the record (`expandSubResources=true`) via `_Component_Forecast_Db::fetchRecord`.
 2. **Per-type status gate** → if excluded, call `removeAll` and stop. Exclusions: invoice
    `Voided`/`Rejected`; cashSale `Unapproved Payment`; creditMemo `Voided`; cashRefund has
-   no gate.
+   no gate. **A cashSale created via REST lands in status `Deposited`** (NOT the excluded
+   `Unapproved Payment`), so REST-created cash sales pass the gate and import — relevant when
+   building cashSale test fixtures.
 3. If `shippingCost != 0`, append a **synthetic SHIPPING line at lineNumber 0** using
    NetSuite item internalId **13500** (a real `Forecast.Items` row).
 4. Per line: resolve the item (with self-heal — see below); skip `itemGroup` items
@@ -117,9 +128,56 @@ uses) and then writes the line — instead of the cron's silent skip. Failure se
   the job for retry.
 - `itemGroup` items carry no line revenue and are skipped.
 
+## JournalEntry import (the fifth transaction type)
+JournalEntry extends the same engine to capture **GL revenue/cost adjustments posted
+directly as journal entries**. It is **structurally different from the four sale types** — a
+JE has **no item lines**; each line is a raw GL posting (account, debit/credit, entity,
+location, memo). Handler `_Worker_Netsuite_JournalEntry`: `post`/`put` → `syncJournalEntry`,
+`delete` → `removeAllJournalEntry`. The router PascalCases `journalEntry` → this class, so
+**no router change** is needed (same as the sale types). The engine's
+`syncJournalEntry`/`buildJournalEntryRows`/`importOneJournalEntry`/etc. reuse the existing
+item-line upsert/reconcile machinery (`syncLines`/`guardedInsert`/`deleteRows`).
+
+**Mapping model (durable design):**
+- A JE line carries **no native item and no salesRep** (confirmed against the live account:
+  0 JEs carry an item on a revenue/cost line). The fields the business rule needs **do not
+  exist in NetSuite yet** — they are stubbed behind constants `JE_LINE_ITEM_FIELD` /
+  `JE_LINE_SALESREP_FIELD` (currently **null**), so until the fields are added every kept
+  line groups under `(null, null)` into **one aggregate row**.
+- Lines hitting a **revenue** or **cost** GL account are grouped by **(salesRep, item)**:
+  `revenue = Σ(credit − debit)` over revenue lines, `cost = Σ(debit − credit)` over cost
+  lines, `profit = revenue − cost`.
+- **Account classification is done on the fly by NetSuite `accttype`** — SuiteQL
+  `SELECT id, accttype FROM account WHERE id IN (...)`, cached per run, bucketed via a tunable
+  `JE_ACCTTYPE_BUCKETS` map. It is **NOT** a hardcoded account-id list. PROVISIONAL default:
+  `Income → revenue`, `COGS → cost` (pending sales-team sign-off; open questions: whether
+  `OthIncome`/`Expense` count, and the Income-typed but "Cost"-named accounts 430/433).
+- Group key uses `json_encode([salesRep, item])` (not a string-join) to stay collision-safe
+  once the fields go live — an empty-string separator would let `"5|" + null` collide.
+
+**Reversal handling (a JE-only pattern):**
+- Setting a reversal date makes NetSuite auto-create a **paired reversing JE**, but **only
+  one webhook fires** (for the original). The engine resolves the reversal from the original
+  via SuiteQL `transaction.reversal` (the reversal's id) and imports it too, keyed by its
+  **own** internalId, so each posting period nets correctly (the reversal carries opposite
+  debit/credit → the natural negation). Helper: `resolveJournalEntryReversal`.
+- The reversal's REST `createdFrom` = the original entry, so its `Forecast.Sales` rows carry
+  `createdFromNetsuiteTransactionInternalId = original`. A **delete on the original therefore
+  cascades to the reversal rows locally** via `deleteRowsCreatedFrom` with **no NetSuite
+  call** — important because the record may already be gone on a delete event.
+- The partner-reversal import is wrapped so a reversal **deleted in NetSuite before the
+  webhook fires** (`fetchRecord` 404) does **not** abort the original entry's
+  already-committed import (it is logged + skipped).
+
+**Additive + dormant.** The enum add (`'journalEntry'` in `Sales.netsuiteTransactionType`)
+is backward-compatible; with the prod JE enqueuer deployment **off**, no JE webhook fires so
+the handler never runs. Safe to merge and deploy the NS enqueuer later. The
+`'journalentry':'journalEntry'` entry was added to both enqueuer `RECORD_TYPE_MAP`s.
+
 ## Data model — Forecast.Sales schema facts
 Prod `Forecast.Sales` writable columns: `netsuiteTransactionType`
-(enum `'invoice','cashSale','creditMemo','cashRefund'`), `netsuiteTransactionInternalId`,
+(enum `'invoice','cashSale','creditMemo','cashRefund','journalEntry'` — `journalEntry`
+added by `dbchanges2/Forecast/2026-06-26a`), `netsuiteTransactionInternalId`,
 `tranDate`, `tranNumber`, `customerId`, `leadSource`, `salesRepEmployeeId`, `itemId`,
 `lineNumber`, `revenue`/`profit` `decimal(14,2)`, `createdFromNetsuiteTransactionInternalId`.
 - **No `amountDue` and no `dtPendingBilling` column** in prod (no staged dbchanges2 adds
@@ -142,8 +200,19 @@ events therefore require this one script deployed on **Invoice, Cash Sale, Credi
 Cash Refund**. The drainer (`customscript_ss_amq_drain`) is record-type-agnostic and needs
 no per-type deployment.
 
+**To make a UE enqueuer fire for a REST/M2M-driven record, the deployment's AUDIENCE must
+include that integration user/role.** The **"Execute As Role"** field does NOT control
+triggering — it only sets the run-as role; changing it has no effect on whether the webhook
+fires. Adjusting the audience is what made create/edit/delete events fire for REST-created
+test records. Symptom of a misconfigured audience (or a missing per-type deployment): **no
+AMQ queue row is created at all** — SuiteQL the AMQ custom record shows 0, nothing at ngrok,
+nothing at prod `Logs.Webhook`.
+
 The enqueuer's `RECORD_TYPE_MAP` (lowercase NS `record.type` → camelCase `recordType` the
-worker2 router PascalCases) already maps the four sale types 1:1. Note: NetSuite emits item
+worker2 router PascalCases) maps the four sale types 1:1, plus
+`'journalentry':'journalEntry'` (added this session to both the prod and dev enqueuers); a
+JE webhook therefore also requires this script deployed on the **Journal Entry** record
+type. Note: NetSuite emits item
 events as specific **subtypes** (`inventoryItem`, `nonInventoryResaleItem`, `kitItem`,
 `itemGroup`, …) with no generic `item` type — collapsing those subtype VALUES to `'item'`
 would be needed only for a future real-time item webhook, **not** for this Sales importer
@@ -166,6 +235,11 @@ Actions: `check | create | get | update | delete | sync | recent | amq | deploym
   [REST client doc](./netsuite-rest-client.md)); the framework was **not** edited this session.
 - `amq` action SuiteQLs the AMQ custom record; `deployments` SuiteQLs `scriptdeployment`+`script`
   to show which enqueuer fires per record type (used to find the dual-deployment trap below).
+- **Per-type siblings** `test_creditmemo_lifecycle.php`, `test_cashsale_lifecycle.php`,
+  `test_cashrefund_lifecycle.php` (and `test_je_lifecycle.php`) mirror this harness, each with its
+  own `create()` POSTing via `_ApiRequest` with the `~`-delimited Location regex workaround.
+  **cashRefund and creditMemo require a header-level location** on create. The create webhook can
+  lag **~20s** before the row appears (read-after-write/processing) — wait before asserting.
 
 ## Gotchas / known issues
 - **Testing locally pollutes PROD unless you UNDEPLOY the prod AMQ enqueuer first.** Each sale
@@ -182,6 +256,16 @@ Actions: `check | create | get | update | delete | sync | recent | amq | deploym
   for the test id). The **dev** enqueuer's `debugWrap` envelope
   `{action,parameters:{payload,headers}}` is required because a local worker2 has no
   `WebhookIngestion` Lambda to wrap the raw body.
+- **The legacy 5-min pull cron can leak a live test record into PROD `Forecast.Sales`
+  regardless of whether the new webhook code is deployed.** Because all testing creates real
+  (tiny, ±$0.01) transactions in the *single* prod NetSuite account, if a test record is
+  still alive when `worker/crons/toga2/forecast2/import_sales.php` runs (every 5 min), the
+  cron pulls it into prod `Forecast.Sales` — this is independent of the webhook (prod has no
+  Sales handler deployed). Observed repeatedly across test batches; each leaked row was an
+  exact test match (customerId 2 / item 25 / revenue ±0.01) and was removed from the prod
+  writer with a guarded `DELETE`. **Mitigations:** keep the create→delete window short (most
+  records slip between cron runs), or pause `import_sales` during a batch; the daily
+  discrepancy-fix cleans any straggler anyway since the record is deleted in NetSuite.
 - **No local app logs for debug-path webhooks — verify arrival via the ngrok inspector, and decode
   the body.** The legacy `{action,parameters}` debug path writes **no** `WorkerJobs`/`Logs.Webhook`
   locally. Confirm inbound arrival at the ngrok request inspector
@@ -203,6 +287,31 @@ Actions: `check | create | get | update | delete | sync | recent | amq | deploym
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-06-26 — **Hardened + verified the full webhook path for all four sale types end-to-end**
+  (create→insert, edit→update, delete→removeAll) with sign conventions reconfirmed (invoice/cashSale
+  +, creditMemo/cashRefund −). Recorded durable gotchas: a UE enqueuer fires for a REST/M2M user only
+  when the deployment's **AUDIENCE** includes that user — **"Execute As Role" does NOT control
+  triggering** (no-AMQ-row is the symptom); a **REST-created cashSale lands in status `Deposited`**, not
+  the excluded `Unapproved Payment`, so it imports; the create webhook can lag **~20s**. Sharpened the
+  prod-leak note: the legacy 5-min `import_sales.php` cron pulls a live test record into prod
+  `Forecast.Sales` independent of webhook deployment (exact ±$0.01 test matches deleted from the prod
+  writer; mitigate by short create→delete windows / pausing the cron). Added per-type harnesses
+  `test_{creditmemo,cashsale,cashrefund}_lifecycle.php`. JE php-reviewer hardening applied
+  (partner-reversal fetch isolated in try/catch, null-safe SuiteQL `items[0]`). (dfranks)
+- 2026-06-26 — **Extended the importer to a fifth transaction type, `journalEntry`** (GL
+  revenue/cost adjustments). New thin handler `_Worker_Netsuite_JournalEntry`
+  (post/put→`syncJournalEntry`, delete→`removeAllJournalEntry`); engine gained
+  `TYPE_JOURNAL_ENTRY` + JE methods reusing the existing line upsert/reconcile machinery;
+  `'journalEntry'` added to the `Sales.netsuiteTransactionType` enum (dbchanges2 + local
+  ALTER) and `'journalentry'` to both enqueuer maps. Durable design recorded: JEs have **no
+  item lines** (lines are GL postings), grouped by (salesRep, item) with item/salesRep
+  **stubbed null** until the NetSuite fields exist; **on-the-fly accttype classification**
+  (`JE_ACCTTYPE_BUCKETS`, provisional Income→revenue / COGS→cost); and the **reversal
+  pattern** (resolve via `transaction.reversal`, import keyed by its own id, delete cascades
+  through `createdFrom` locally, 404-on-reversal tolerated). Verified e2e via the direct
+  engine against prod NetSuite with prod isolated (JE enqueuer undeployed → 0 prod rows);
+  built harness `test/@dave/test_je_lifecycle.php`. Additive + dormant until the NS enqueuer
+  deploys. (dfranks)
 - 2026-06-26 — **Verified the full create→update→delete webhook lifecycle end-to-end on a local box**
   (ngrok tunnel → local worker2 → local `Forecast.Sales`), driving a real **$0** NetSuite invoice
   through create→insert, edit→update (tracked-column change-detection), delete→`removeAll`; confirmed

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.213",
+  "version": "1.0.215",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",