toga-ai 1.0.893 → 1.0.894

package/knowledge/sessions/2026-09-25-TRUE_80519-mhammontree.md

@@ -0,0 +1,112 @@
+---
+type: session
+slug: TRUE_80519
+title: NYCDOE 2.0 ASN Phase 2 parsing + worker2 Codeception pilot
+author: mhammontree
+repos: [worker2, _underscore, dbchanges2]
+framework: "2.0"
+client: nycdoe
+status: active
+created: 2026-09-25
+updated: 2026-09-25
+---
+
+# Session: TRUE_80519
+**Date:** 2026-09-25
+**Project/Repo:** worker2, _underscore, dbchanges2 (2.0)
+**Task:** Build and verify Phase 2 (ASN file parsing → Case/ServiceRequest creation) of the NYCDOE ServiceNow/ASN/NetSuite 2.0 rewrite (TRUE-80519), then set up worker2 as 2.0's first Codeception/PHPUnit test pilot and codify the Phase 2 verification as regression tests.
+
+---
+
+## What WORKED
+
+- **Phase 2 parsing complete for all 4 manufacturers**, each verified end-to-end via manual Postman testing against real sample files, then codified as automated tests:
+  - Lenovo: real file → 66 Cases, 746 ServiceRequests; re-run confirmed dedupe (`serviceRequestsCreated: 0`).
+  - Lexmark: real file → 4 Cases, 14 ServiceRequests; dedupe confirmed.
+  - Apple: real file → 56 Cases, 402 ServiceRequests; dedupe confirmed.
+  - Acer: real file → 12 Cases, 1145 ServiceRequests; dedupe confirmed.
+  - `Worker/Sftp/Parse.php` implements one `_Worker_Sftp_Parse::<Manufacturer>()` static method per manufacturer, header-validated against a hardcoded expected-column-names constant (LENOVO_HEADER, LEXMARK_HEADER, ACER_HEADER, APPLE_HEADER), rejecting the whole file loudly on any mismatch.
+- **Case grain resolved**: one `Case` per customer PO (via new `c_customerPurchaseOrder` field), not one per file — confirmed from real sample data containing multiple POs per file.
+- **NetSuite linkage design corrected** per Rohan: SO/PO header stamp belongs on `SalesOrders`/`PurchaseOrders` (`SalesOrders.serviceRequestId`, `SalesOrders_PurchaseOrders`, `SalesOrderItems.quantity`/`.qtyCommitted`), not as `Case` custom fields as originally assumed. This narrowed Phase 2's own `c_` field list to RITM linkage + `filesId` provenance only.
+- **`_Model_Nycdoe_Case` / `_Model_Nycdoe_ServiceRequest`** (`_underscore/Model/Nycdoe/`) — client model overrides declaring the `c_` fields as real properties, required for direct `_Model` writes (same constraint as Phase 1's Files model).
+- **dbchanges2 migrations applied and confirmed working**: `ServiceRequestStages` seeded (New/Scheduled/On Hold/Delivered/Cancelled), `ServiceRequests` gained `c_partNumber`/`c_serialNumber`/`c_ritmNumber`/`c_filesId`, `Cases` gained `c_customerPurchaseOrder` and the missing `longDescription` column (schema/model drift fix).
+- **Codeception 5 / PHPUnit 11 set up in worker2** as 2.0's test pilot, CTO-reviewed before building (AGREE + 5 required changes, all applied) and php-reviewer-reviewed after building (2 real bugs found and fixed — see below). Dev-only via `composer require-dev`; confirmed `composer.lock`'s non-dev `packages` block is byte-identical before/after, and `--no-dev --dry-run` cleanly removes every test package (matches worker2's real deploy hook `composer install --no-dev`).
+- **`tests/Integration/ParseCest.php`** — 6 tests (one per manufacturer + header-mismatch rejection + Lenovo comma-repair regression), all passing, confirmed repeatable across multiple runs, confirmed clean fixture teardown via a separate verification script.
+- **Two hard safety gates** in `tests/Support/Helper/Integration.php`, both run before any write: refuses to run unless resolved `ENVIRONMENT` is `dev-*` (read from gitignored `tests/local.env.php`, never hardcoded); refuses to run against a non-local `Client_Nycdoe` database host. Verified both actually fire (temporarily pointed `local.env.php` at `'production'`, confirmed the suite refused to run, then restored it).
+- **Lenovo comma-in-description bug fixed and tested**: ported 1.0's exact fix (`library/app/asnprocessor/lenovo.php`) for Lenovo's `Product Long Description` column sometimes containing an unquoted, unescaped comma that shifts every later column. Added `repairLenovoRowColumnCount()`. Regression test (`lenovoRepairsAnUnescapedCommaInTheDescription`) — confirmed it actually fails without the fix (serial number lookup breaks once columns shift) before confirming it passes with the fix.
+- **php-reviewer caught 2 real bugs in the test harness itself, both fixed and re-verified**:
+  1. `commitDatabaseTransaction()` was a no-op — `_Database::transactionCommit(null)` only acts when the global `-1` flag is set, which is only set by the parameterless `_Database::transactionStart()` (called by `_Route` in production, never by our test). Fixed by calling `_Database::transactionStart()` in `_before()`. This also proved an earlier workaround (`_Database::setIsReadHostEnabled(false)`) was unnecessary once real commits happened — removed it after 3 clean re-runs confirmed tests still pass without it.
+  2. Unnecessary `ReflectionClass`/`setAccessible` into `_Database::$_registers`, which is actually a `public static` property — simplified to direct access.
+- **Everything captured and published to the team knowledge base** (commit `146c68d7` on `origin/_main`): 9 docs written (2 new: `clients/nycdoe/features/asn-sftp-ingestion-2-0.md`, `2.0/apps/worker2/features/codeception-integration-tests.md`; 5 updated: `servicenow-integration.md`, `cli-script-bootstrap.md`, `model-magic-field-access.md`, `item-classification-itemclasses.md`, nycdoe `profile.md`; 2 elevated standards updated with developer approval: `framework-rules.md`, `backend-testing.md`).
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **Assuming Lenovo's file layout matched 1.0's stale `asnprocessor/lenovo.php` code (47 columns)** — wrong. The real, current Lenovo file is a genuinely different 30-column layout, confirmed against 3 real live sample files. The header-name validation correctly rejected the file when tested against the wrong assumed header — this was the design working as intended, not a bug, but the underlying `LENOVO_HEADER` constant itself had to be rewritten from scratch based on real file content, not from reading old code.
+- **A "stale FIELD_STORAGE file due to id reuse" theory** — raised mid-investigation of the Lenovo header mismatch (suspecting local folder-mode storage served old content for a reused row id). Proven wrong: a fresh row showed the same "wrong" header, which turned out to be the real, current file format all along. Retracted for the record so it isn't mistaken for a real finding later.
+- **Assuming Lexmark's `Order_Date` was `M/D/YYYY`** — wrong; it's `YYYY/MM/DD`. Caught by re-reading the real sample file before testing (direct lesson from the Lenovo mistake above), not by a failed test in production.
+- **Assuming Acer/Lexmark share identical header column names** (only positions) — wrong; 3 column names differ (`First_Name_Shipping`/`Last_Name_Shipping`/`SerialNo` for Acer vs `Contact_Name_Shipping`/`Email_Shipping`/`Svc_Tag_ID` for Lexmark). A single shared `LEXMARK_ACER_HEADER` constant would have rejected every real Acer file. Split into `LEXMARK_HEADER`/`ACER_HEADER`.
+- **Assuming a 2-digit year never appears in Apple's `order_date`** — wrong; a newer sample file had `10/23/25`, which the original `parseDate()` would have turned into year `0025`. Fixed with `expandYear()`.
+- **Running `vendor/bin/codecept bootstrap` fully non-interactively** — it stopped and waited on an interactive `(y/n)` prompt ("composer.json updated. Do you want to run 'composer update'?") that the Bash tool couldn't answer; had to `TaskStop` the backgrounded process and finish the setup manually (`generate:suite`, manual `composer update` with the lock-diff safety check) instead of relying on the bootstrap wizard end-to-end.
+- **The original `commitDatabaseTransaction()`/`setIsReadHostEnabled(false)` combination** — see "What WORKED" above for the real root cause and fix; the original diagnosis (separate read connection can't see an uncommitted write) was directionally right but incomplete — the actual missing piece was the global `transactionStart()` call, not just connection routing.
+
+## Not tried yet (candidates for next session)
+
+- **Acer/Lenovo/Lexmark date-format edge cases beyond what real samples showed** — Acer's `YYYY/MM/DD` format is now confirmed (not just guessed), but only against one sample file; worth reconfirming if a 2-digit-year Acer file ever surfaces (same class of bug already found for Apple).
+- **Re-confirm the Lenovo comma-in-description fix against a real occurrence** — no real sample file with this exact issue was available this session; the fix mirrors 1.0's proven logic exactly but has not been tested against real Lenovo data that actually has the bug.
+- **Deploy/rollout planning for Phase 1 + 2** (discussed conceptually, not started):
+  - Confirm whether Phase 1's SFTP fetch is read-only / non-destructive relative to what 1.0 already consumes from the same SFTP (must not interfere with 1.0's live pipeline).
+  - Decide a "shadow mode" plan: run 2.0 ingestion+parsing against the same real files 1.0 processes, writing into the isolated `Client_Nycdoe` 2.0 tables, purely for validation — low risk since Phase 3 (NetSuite/ServiceNow outbound calls) doesn't exist yet, so nothing can double-fire.
+  - Decide what "shadow mode is good enough to cut over" means concretely (e.g., compare 2.0 Case/ServiceRequest counts per manufacturer per day against 1.0's `AdvanceShippingNotices` counts) — no comparison method defined yet.
+  - No monitoring/alerting exists yet for Phase 1/2 failures (header rejections, comma-repair firing, etc.) — needs a decision on whether/how to alert (Logs.Issue entry? tie into Mark's separate Monitoring System Design project?).
+  - Branch/PR/deploy process not started: needs `php-reviewer` + `cso` on the diff, Rohan's deploy window for `_underscore`/`dbchanges2` changes, and Mark's own pseudocode/proposal to Jeff (Claude's output is reference material only — Mark authors the actual submission).
+- **Follow-up with Rohan**: whether other clients' `Cases` tables are also missing the `longDescription` column (found only for `Client_Nycdoe` this session).
+- **Low-priority housekeeping**: stale `DOE-OPEN-LENOVO-*.CSV` files still sit on the real Lenovo SFTP and aren't cleaned up by anything (per Skyler/Eric, safe to delete, retired process) — not blocking, not scheduled.
+- **INC (Incident) ticket flow** — a separate flow from the ASN/RITM pipeline this session covered, not yet redesigned for 2.0 (flagged in the main plan as "next session" from an earlier date, still open).
+- **Phase 3** (NetSuite Sales Order/PO dispatch, ServiceNow RITM linkage, outbound sync) — explicitly out of scope for Phase 2, not started.
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `worker2/Worker/Sftp/Parse.php` | Modified, uncommitted | Phase 2 parsing for all 4 manufacturers + Lenovo comma-repair fix. Lint-clean, all tests pass. |
+| `worker2/tests/Support/Helper/Integration.php` | New, uncommitted | Codeception module: framework boot, dev/local safety gates, fixture + transaction helpers. |
+| `worker2/tests/Integration/ParseCest.php` | New, uncommitted | 6 integration tests, all passing. |
+| `worker2/tests/local.env.php.example` | New, uncommitted | Template for the gitignored per-developer env file. |
+| `worker2/tests/local.env.php` | New, gitignored (not committed by design) | This machine's value: `dev-markhammontree-laptop`. |
+| `worker2/codeception.yml`, `worker2/tests/Integration.suite.yml` | New, uncommitted | Codeception config. |
+| `worker2/.gitattributes` | New, uncommitted | `export-ignore` for `tests/` and `codeception.yml`. |
+| `worker2/.htaccess` | Modified, uncommitted | Added deny rules for `tests/` and `*.yml`. |
+| `worker2/.gitignore` | Modified, uncommitted | Added `/tests/_output/`, `/tests/Support/_generated/`, `/tests/local.env.php`. |
+| `worker2/composer.json`, `worker2/composer.lock` | Modified, uncommitted | Added `codeception/codeception:^5`, `phpunit/phpunit:^11`, `codeception/module-asserts` as `require-dev` only. Confirmed non-dev `packages` block unchanged. |
+| `_underscore/Model/Nycdoe/Case.php` | New, uncommitted | Declares `c_customerPurchaseOrder`. |
+| `_underscore/Model/Nycdoe/ServiceRequest.php` | New, uncommitted | Declares `c_partNumber`, `c_serialNumber`, `c_ritmNumber`, `c_filesId`. |
+| `dbchanges2/Client_Nycdoe/2026-09-22a - Seed ServiceRequestStages for DOE installs.sql` | New, uncommitted | Confirmed applied to local dev DB. |
+| `dbchanges2/Client_Nycdoe/2026-09-22b - Add ASN parsing fields to ServiceRequests.sql` | New, uncommitted | Confirmed applied. |
+| `dbchanges2/Client_Nycdoe/2026-09-22c - Add customerPurchaseOrder field to Cases.sql` | New, uncommitted | Confirmed applied. |
+| `dbchanges2/Client_Nycdoe/2026-09-25a - Add missing longDescription column to Cases.sql` | New, uncommitted | Confirmed applied; fixes real schema/model drift against shared `_Model_Client_Case`. |
+| `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` | Continuously updated throughout | Full change history of every finding this session, kept current. |
+| Team KB (`~/toga-tech/knowledge/`) | Published | Commit `146c68d7` on `origin/_main`. 9 docs (2 new, 7 updated), see "What WORKED". |
+
+**None of the code changes above have been committed or pushed to their respective repos yet** — draft commit titles/descriptions were prepared for worker2 (2 commits), `_underscore` (1 commit), and `dbchanges2` (1 commit), but no `git commit` has been run per the standing "Claude never commits without explicit ask" rule.
+
+## Decisions made
+
+- **One `Case` per customer PO, not per file.** Rationale: real sample data (Lexmark, Apple) showed multiple distinct customer POs in a single file, so "one Case per file" was never workable. No alternative seriously considered once real data was checked.
+- **NetSuite SO/PO stamp lives on `SalesOrders`/`PurchaseOrders`, not as `Case` custom fields.** Rationale: Rohan's direct correction, verified against real schema. Rejected: the original assumption from the main plan's earlier design session.
+- **Lenovo kept as its own header/parse path, never shared with Lexmark/Acer**, even though both `Lenovo()` and `runLexmarkAcerFamily()` looked similar early on. Rationale (Mark's explicit instruction): Lenovo is not actually a CDW supplier (only Acer/Lexmark are); sharing code risks a future Lenovo-only format change breaking two unrelated suppliers. Rejected: a single shared parsing function keyed by manufacturer name.
+- **`DOE-OPEN-LENOVO` files are retired and must never be parsed**, confirmed by Skyler/Eric. Rationale: an old, retired process Lenovo still sends to the SFTP; Phase 1's existing filename pattern already excludes them, so no code change was needed, just confirmation.
+- **Lenovo's comma-in-description bug is repaired, not rejected** (unlike a genuine header-shape mismatch, which stays fatal). Rationale: Mark's direct confirmation that, per 1.0's history, this is the ONLY field Lenovo ever breaks this way — a well-understood, narrow quirk worth auto-fixing rather than bouncing every affected file for manual resend.
+- **2.0 test framework: Codeception 5 + PHPUnit 11, `worker2` as the sole pilot repo.** Rationale: Jeff's direction; CTO-reviewed first given it's a framework-adoption decision. Explicitly rejected: copying 1.0's `api` repo pattern of running tests against production on every deploy via an `.ebextensions` postdeploy hook — tests stay a manual/local step here, never a deploy-time gate, unless explicitly changed later.
+- **`integration` suite naming, not `unit`/`functional`.** Rationale (from the CTO review): `Parse.php` needs a real DB, so it isn't a pure unit test; Codeception reserves "functional" for framework-module suites (Symfony/Laravel-style), which doesn't apply here — `integration` avoids both misnomers.
+- **Test-only workarounds must match production's real transaction/connection behavior, not just "make the test pass."** Concretely: once `php-reviewer` proved the transaction-commit workaround was a no-op, the correct fix was to make it a REAL commit (matching what `_Route` does), not to just silence the symptom differently. This also let an earlier workaround (`setIsReadHostEnabled(false)`) be removed as unneeded once the real fix was in — verified empirically (3 clean re-runs without it) rather than assumed.
+
+## Blockers
+
+None for continuing Phase 2 work itself. For **rollout** (the open question this session ended on): no blocker yet identified, but the next session needs to (1) confirm Phase 1's SFTP read is non-destructive to 1.0's own consumption, and (2) decide the shadow-mode comparison method, before recommending an actual rollout plan to Mark for his own submission to Jeff.
+
+## Exact next step
+
+> Start the next session by reading `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` in full (it has the complete, current design + change history), then investigate whether `worker2/Worker/Sftp/Import.php`'s SFTP fetch is read-only/non-destructive relative to 1.0's own consumption of the same DOE SFTP — this is the first concrete fact needed before proposing a "shadow mode" parallel-run plan to Mark.
+
+---
+_Saved by /session-save on 2026-09-25_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.893",
+  "version": "1.0.894",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",