toga-ai 1.0.892 → 1.0.894
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/_underscore/features/cli-script-bootstrap.md +8 -2
- package/knowledge/2.0/apps/_underscore/features/item-classification-itemclasses.md +8 -3
- package/knowledge/2.0/apps/_underscore/features/model-magic-field-access.md +9 -4
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/codeception-integration-tests.md +51 -0
- package/knowledge/2.0/standards/backend-testing.md +7 -5
- package/knowledge/2.0/standards/framework-rules.md +18 -12
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/nycdoe/INDEX.md +2 -1
- package/knowledge/clients/nycdoe/features/asn-sftp-ingestion-2-0.md +63 -0
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +8 -3
- package/knowledge/clients/nycdoe/profile.md +9 -5
- package/knowledge/sessions/2026-09-25-TRUE_80519-mhammontree.md +112 -0
- package/package.json +1 -1
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["bala"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["bala", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Database.php
|
|
13
13
|
- _underscore/Environment.php
|
|
@@ -53,6 +53,9 @@ For DB-**free** unit tests of interceptor logic, use the PHPUnit scaffold instea
|
|
|
53
53
|
raised. This is the CLI face of the lazy-transaction behavior documented on
|
|
54
54
|
`_underscore/architecture.md` — `_Database::register()` starts a lazy transaction that MySQL rolls
|
|
55
55
|
back when the connection closes uncommitted.
|
|
56
|
+
- **Keep the CWD at the app root for the whole run, not just the `require`.** `Loader::autoloader`
|
|
57
|
+
resolves project classes (e.g. `_Worker_Sftp_Parse`) against `./` lazily at first use, so a
|
|
58
|
+
harness that `chdir`s back after booting breaks autoloading on the first call into app code.
|
|
56
59
|
- **`ENVIRONMENT` must be set before the framework loads**, not after — `_Environment::initialize`
|
|
57
60
|
fatals on a missing value rather than defaulting.
|
|
58
61
|
- **Local MySQL and PROD both run `sql_mode = ''`**, so `ONLY_FULL_GROUP_BY` is **not** enforced in
|
|
@@ -74,3 +77,6 @@ For DB-**free** unit tests of interceptor logic, use the PHPUnit scaffold instea
|
|
|
74
77
|
- [Client Email Template Sending](./email-template-sending.md) — its local test-harness section.
|
|
75
78
|
- [French (fr-CA) Order Email Localization (Compass Canada)](../../../clients/compass-canada/features/french-order-email-localization.md)
|
|
76
79
|
— the work these traps surfaced in.
|
|
80
|
+
|
|
81
|
+
## Change history
|
|
82
|
+
- 2026-09-25 — Added lazy-CWD autoloader gotcha (found building the worker2 Codeception harness) (mhammontree)
|
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["rgirish"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Client/ItemClass.php
|
|
13
13
|
- _underscore/Model/Client/Item.php
|
|
@@ -120,7 +120,9 @@ Queried live via SuiteQL (2026-09-11): **311 classifications**, columns
|
|
|
120
120
|
- **A hierarchy path column must be `TEXT`, not `varchar(512)`.** NetSuite `fullname` values are
|
|
121
121
|
long, and the 2.0 standard caps varchar at 255.
|
|
122
122
|
- **The table is already `utf8mb4_0900_ai_ci`**, so new columns need no explicit `COLLATE`.
|
|
123
|
-
- **`c_` custom fields are NOT declared as properties on the 2.0 Model class
|
|
123
|
+
- **`c_` custom fields are NOT declared as properties on the 2.0 Model class** when written
|
|
124
|
+
through the /v2 API (a direct `_Model` write does need a declaration — see
|
|
125
|
+
[_Model magic-field access](model-magic-field-access.md)). The API engine
|
|
124
126
|
reads them from `CustomRecordFields` (`ItemClasses` is `Core.Records` id **105**). Confirmed:
|
|
125
127
|
no `public $c_netsuite*` property exists on any `Model/Client` class. Copy the pattern in
|
|
126
128
|
`dbchanges2/_modules/netsuite/2026-08-18 - TransferOrdersNetsuiteStatus.sql`.
|
|
@@ -136,3 +138,6 @@ Queried live via SuiteQL (2026-09-11): **311 classifications**, columns
|
|
|
136
138
|
- [Prudential ItemClasses device-tier data](../../../../clients/prudential/features/itemclasses-device-tier-data.md)
|
|
137
139
|
- [Client schema-drift audit](../../dbchanges2/workflows/client-schema-drift-audit.md)
|
|
138
140
|
- [NetSuite item-class sync (1.0)](../../../../1.0/apps/library/features/netsuite-item-class-sync.md)
|
|
141
|
+
|
|
142
|
+
## Change history
|
|
143
|
+
- 2026-09-25 — Scoped the "c_ not declared" note to /v2 API writes (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["jcardinal", "mhammontree", "apeterson"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Core/Model.php
|
|
@@ -79,9 +79,11 @@ class does not declare. Two things follow immediately, and they cut the search s
|
|
|
79
79
|
|
|
80
80
|
1. **It is never a schema problem.** No SQL ran. Do not check `information_schema`, do not suspect a
|
|
81
81
|
missing migration on *that* column — check the model class.
|
|
82
|
-
2. **
|
|
83
|
-
|
|
84
|
-
declare the property on the model
|
|
82
|
+
2. **It can be a `c_` column too.** The /v2 API engine handles `c_` columns dynamically, but
|
|
83
|
+
`_Model` itself does not: a direct model read/write (`__get`/`__set`) of an undeclared `c_`
|
|
84
|
+
field throws this same message. The fix is binary: declare the property on the model (a
|
|
85
|
+
client override such as `_Model_Nycdoe_ServiceRequest`, or a `_Trait_Netsuite_*`), or stop
|
|
86
|
+
using it.
|
|
85
87
|
|
|
86
88
|
**The trap that produces it: two similarly-named models, one of which lacks the column.** Swapping
|
|
87
89
|
one for another at a call site **lints clean and only dies at runtime** — nothing in PHP or CI
|
|
@@ -135,3 +137,6 @@ wrong: **1054** = the database is behind the model (see
|
|
|
135
137
|
- [Surface FIELD ↔ RecordField binding](./surface-field-recordfield-binding.md)
|
|
136
138
|
- [Framework branch ahead of schema](../../dbchanges2/workflows/framework-branch-ahead-of-schema.md)
|
|
137
139
|
- [Rate subscription cancellation](../../../../clients/rate/features/subscription-cancellation.md)
|
|
140
|
+
|
|
141
|
+
## Change history
|
|
142
|
+
- 2026-09-25 — Corrected: undeclared `c_` fields also throw on direct `_Model` access; dynamic `c_` handling is the /v2 API engine only (mhammontree)
|
|
@@ -19,6 +19,7 @@
|
|
|
19
19
|
| [ClickUp Task Description Fluffer (Talos KB Investigation)](features/clickup-task-fluffer-talos.md) | Async worker that "fluffs" a new ClickUp task by running its name+description through the Talos `dev-core` KB agent and writing the report back; open for the fl |
|
|
20
20
|
| [ClickUp Webhook Health Monitor → OneUptime (Monitor/Clickup/CheckWebhook)](features/clickup-webhook-health-monitor.md) | worker2 cron that checks the team's ClickUp webhook subscriptions and pages via a OneUptime Incoming-Request monitor; open when tuning the ClickUp webhook alarm |
|
|
21
21
|
| [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | Auto-maintains each ClickUp task's Work Type (Committed/Conditional/Stretch) from its dependencies, status, and due date; open when work-type promotion/demotion |
|
|
22
|
+
| [worker2 Codeception integration tests (2.0 test-framework pilot)](features/codeception-integration-tests.md) | First 2.0 app-level automated test suite: Codeception 5 / PHPUnit 11 in worker2 (pilot, Jeff's direction), real-DB integration tests with hard local-only safety |
|
|
22
23
|
| [worker2 — Committed Secrets & Config Exposure (debt register)](features/committed-secrets-and-config-debt.md) | Standing list of credentials committed to the worker2 repo (locations only); open before adding a credential to worker2 or assuming a config value is private. |
|
|
23
24
|
| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | worker2 action that ingests Compass's quarterly VIP spreadsheet and sets each VIP's support tech (`Users.c_supportedByUserId` in `Client_Compass`); open when wo |
|
|
24
25
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). |
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: worker2 Codeception integration tests (2.0 test-framework pilot)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/codeception.yml
|
|
13
|
+
- worker2/tests/Integration.suite.yml
|
|
14
|
+
- worker2/tests/Support/Helper/Integration.php
|
|
15
|
+
- worker2/tests/Integration/ParseCest.php
|
|
16
|
+
- worker2/tests/local.env.php.example
|
|
17
|
+
- worker2/composer.json
|
|
18
|
+
- worker2/.gitattributes
|
|
19
|
+
- worker2/.htaccess
|
|
20
|
+
related:
|
|
21
|
+
- ../../../standards/backend-testing.md
|
|
22
|
+
- ../../_underscore/features/cli-script-bootstrap.md
|
|
23
|
+
- ../../_underscore/features/model-interceptor-unit-testing.md
|
|
24
|
+
- ../../../../clients/nycdoe/features/asn-sftp-ingestion-2-0.md
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
First 2.0 app-level automated test suite: Codeception 5 / PHPUnit 11 in worker2 (pilot, Jeff's direction), real-DB integration tests with hard local-only safety gates; open before adding or running worker2 tests.
|
|
28
|
+
|
|
29
|
+
## Summary
|
|
30
|
+
|
|
31
|
+
worker2 is the **pilot repo** for a 2.0 test framework. One suite, `Integration` (not `unit`/`functional` — the code under test needs a real DB), boots the real `_underscore` framework and calls worker actions in-process against a **local** client DB. First coverage: `Sftp/Parse` (6 tests). **Tests are a manual/local step — never a deploy-time gate.** 1.0's pattern (`codecept.phar` + `.ebextensions/040_run_tests.config` running tests against production on every deploy) was deliberately NOT copied.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
1. **Dev-only install.** Packages are `require-dev` only (`codeception/codeception ^5`, `phpunit/phpunit ^11`, `codeception/module-asserts`). The deploy hook `.platform/hooks/postdeploy/015_install_composer.sh` runs `composer install --no-dev`, so nothing ships; the non-dev block of `composer.lock` was byte-identical before/after.
|
|
36
|
+
2. **Never deployed.** `.gitattributes` `export-ignore`s `/tests` + `/codeception.yml`; `.htaccess` denies `^tests/`.
|
|
37
|
+
3. **Per-dev environment.** Copy `tests/local.env.php.example` → `tests/local.env.php` (gitignored) returning your Apache `SetEnv ENVIRONMENT` value (e.g. `dev-<name>-laptop`). Or export `ENVIRONMENT`.
|
|
38
|
+
4. **Run** from the worker2 root with PHP 8: `vendor/bin/codecept run Integration`.
|
|
39
|
+
5. **Safety gates (run before any write):** helper refuses unless `ENVIRONMENT` starts `dev-`; `assertUsingLocalClientDatabaseOnly($clientId)` refuses unless the registered `DB_CLIENT` `hostname`/`writeHost`/`readHost` are all `127.0.0.1`/`localhost`/`::1`. Call it before every action call.
|
|
40
|
+
6. **Boot + transaction.** `_before()` boots once (same two lines as `index.php`) then calls parameterless `_Database::transactionStart()` — the call `_Route` makes for a real request. `commitDatabaseTransaction()` then really commits.
|
|
41
|
+
7. **Fixtures** mirror production field-by-field (`createFileFixture` mirrors `Import.php` `processFile()`); each test cleans up its own rows.
|
|
42
|
+
|
|
43
|
+
## Gotchas
|
|
44
|
+
|
|
45
|
+
- **Calling a worker action directly bypasses `_Route`, so there is no open transaction** — without the explicit `_Database::transactionStart()` the commit is a silent no-op and reads miss writes. Do not work around the symptom (e.g. disabling read-host connections); start the transaction. See [CLI script bootstrap](../../_underscore/features/cli-script-bootstrap.md).
|
|
46
|
+
- **Do not `chdir` back after booting.** `Loader::autoloader` resolves project classes against the CWD lazily at first use, so restoring the old CWD breaks autoloading of `_Worker_*` classes mid-test.
|
|
47
|
+
- **`_Database::$_registers` is public static** — read it directly; no Reflection.
|
|
48
|
+
- **Prove a regression test fails without the fix** before trusting it (done for the Lenovo comma repair: serial mismatch once columns shift).
|
|
49
|
+
|
|
50
|
+
## Change history
|
|
51
|
+
- 2026-09-25 — Created: Codeception/PHPUnit pilot, Integration suite for Sftp/Parse; cto-reviewed design, php-reviewer fixes to the harness (mhammontree)
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: 2.0 Backend Testing (
|
|
2
|
+
title: 2.0 Backend Testing (ad-hoc client-pinned scripts; worker2 Codeception pilot)
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
project: _Underscore
|
|
5
5
|
client: shared
|
|
6
6
|
type: standard
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-09-
|
|
8
|
+
updated: 2026-09-25
|
|
9
9
|
owners: [mhammontree]
|
|
10
10
|
files:
|
|
11
11
|
- _underscore/Query.php
|
|
@@ -14,17 +14,19 @@ files:
|
|
|
14
14
|
- _underscore/Model.php
|
|
15
15
|
- test/@Mark/Rate/verify_cancel_subscription.php
|
|
16
16
|
- test/@Mark/Rate/verify_aig_contract_identifier_persistence.php
|
|
17
|
+
- worker2/tests/Support/Helper/Integration.php
|
|
17
18
|
related:
|
|
18
19
|
- ../apps/_underscore/architecture.md
|
|
19
20
|
- ../apps/_underscore/features/per-client-database-connections.md
|
|
20
21
|
- ../apps/test/architecture.md
|
|
22
|
+
- ../apps/worker2/features/codeception-integration-tests.md
|
|
21
23
|
---
|
|
22
24
|
|
|
23
|
-
How to verify 2.0 backend behavior
|
|
25
|
+
How to verify 2.0 backend behavior — ad-hoc `test`-repo scripts that pin a real client DB (everywhere), or the worker2 Codeception pilot suite; open before writing any 2.0 backend verification.
|
|
24
26
|
|
|
25
27
|
## Summary
|
|
26
28
|
|
|
27
|
-
The 2.0 backend
|
|
29
|
+
The 2.0 backend has **no shared PHPUnit harness** — the only automated suite is the **worker2 Codeception/PHPUnit pilot** ([codeception-integration-tests](../apps/worker2/features/codeception-integration-tests.md)), local-only and never a deploy gate. Everywhere else, verify backend behavior with **ad-hoc scripts in the `test` repo** (per-dev folders, e.g. `@Mark`) that bootstrap `_underscore` from a 2.0 app and pin a real client database explicitly.
|
|
28
30
|
|
|
29
31
|
## Bootstrapping a verification script
|
|
30
32
|
|
|
@@ -131,7 +133,7 @@ Also design around what a direct-call script **cannot** see: calling a model met
|
|
|
131
133
|
|
|
132
134
|
## Rules
|
|
133
135
|
|
|
134
|
-
-
|
|
136
|
+
- Outside worker2, do not add or assume a PHPUnit suite; write a `test`-repo script. In worker2, add tests to the Codeception `Integration` suite (local-only; never wire it into deploy).
|
|
135
137
|
- Always set `ENVIRONMENT` before bootstrap (PowerShell: `$env:ENVIRONMENT='...'`, or `putenv`).
|
|
136
138
|
- Always pin the concrete `Client_<Id>` schema to `DB_CLIENT` (and logs/archive if the code logs).
|
|
137
139
|
- **Commit seeded rows before you read them back** — the read connection cannot see uncommitted writes; commit cleanup too so it persists.
|
|
@@ -5,8 +5,8 @@ project: _Underscore
|
|
|
5
5
|
client: shared
|
|
6
6
|
type: standard
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-09-
|
|
9
|
-
owners: [jcardinal, ajean, apeterson]
|
|
8
|
+
updated: 2026-09-25
|
|
9
|
+
owners: [jcardinal, ajean, apeterson, mhammontree]
|
|
10
10
|
files: []
|
|
11
11
|
related:
|
|
12
12
|
- ../apps/_underscore/architecture.md
|
|
@@ -19,7 +19,7 @@ related:
|
|
|
19
19
|
- framework-rules-cases.md
|
|
20
20
|
---
|
|
21
21
|
|
|
22
|
-
Core 2.0 `_underscore` framework rules — class naming, the worker contract & dispatch, the api2 `{success,data,errors}` envelope, dbchanges2 schema/isolation rules, and `c_` dynamic
|
|
22
|
+
Core 2.0 `_underscore` framework rules — class naming, the worker contract & dispatch, the api2 `{success,data,errors}` envelope, dbchanges2 schema/isolation rules, and `c_` columns (dynamic in /v2, declared for direct `_Model` use); open when writing/reviewing any 2.0 code.
|
|
23
23
|
|
|
24
24
|
These rules apply to all 2.0 repos: `_underscore` (core), `worker2`, and `api2`. They are
|
|
25
25
|
loaded on-demand by `/kickoff` when a 2.0 repo is in scope (not always-on) — see the
|
|
@@ -317,18 +317,24 @@ continue. Data accuracy outranks job "completion," even when the failure is not
|
|
|
317
317
|
ACL error on one record) — a developer reconciling a stalled job is the intended outcome, not a
|
|
318
318
|
job that reports success having quietly skipped records.
|
|
319
319
|
|
|
320
|
-
## `c_`-prefixed custom columns are
|
|
320
|
+
## `c_`-prefixed custom columns are dynamic in the /v2 API — not in `_Model`
|
|
321
321
|
|
|
322
|
-
Columns prefixed `c_` are **custom/dynamic** columns
|
|
322
|
+
Columns prefixed `c_` are **custom/dynamic** columns. The **/v2 API engine** handles them
|
|
323
323
|
**without any model-class declaration**:
|
|
324
324
|
|
|
325
|
-
-
|
|
326
|
-
|
|
325
|
+
- Through /v2 they are read, written, and filtered **without** being declared as properties on
|
|
326
|
+
the `_Model_*` subclass. Raw `_Query` result rows also return them as-is.
|
|
327
327
|
- They require **no `Core.RecordFields` entry and no ACL entry**.
|
|
328
328
|
- Precedent examples (`c_clickupUserIdentifier`, the 2026-07 transcript-routing columns): see framework-rules-cases.md.
|
|
329
329
|
|
|
330
|
-
**Consequence:**
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
330
|
+
**Consequence:** for data that only flows through /v2 (or raw `_Query`), adding a `c_` column
|
|
331
|
+
is a **schema-only** change (a `dbchanges2` migration). Do not add `RecordFields`/ACL rows for a
|
|
332
|
+
`c_` column. Use a non-`c_` column when you *do* want the full declared, ACL-governed field
|
|
333
|
+
treatment.
|
|
334
|
+
|
|
335
|
+
**Exception — direct `_Model` access.** `_Model` itself is **not** dynamic: `__get`/`__set` of an
|
|
336
|
+
undeclared field throws `There is no field called 'c_…'`. Code that reads or writes a `c_` column
|
|
337
|
+
through a model instance (worker2 actions, crons, scripts) **must declare it as a property** on a
|
|
338
|
+
client model override (e.g. `_Model_Nycdoe_ServiceRequest`) or a `_Trait_*`. Precedent:
|
|
339
|
+
`_underscore/Model/Nycdoe/{File,Case,ServiceRequest}.php` (TRUE-80519). Symptom and fix:
|
|
340
|
+
[model-magic-field-access](../apps/_underscore/features/model-magic-field-access.md).
|
package/knowledge/INDEX.md
CHANGED
|
@@ -20,7 +20,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
20
20
|
## 2.0 framework
|
|
21
21
|
|
|
22
22
|
- **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
23
|
-
- **worker2** (Worker) —
|
|
23
|
+
- **worker2** (Worker) — 77 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
24
24
|
- **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
25
25
|
- **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
26
26
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary |
|
|
4
4
|
|-----|-----------|---------|
|
|
5
|
+
| [NYCDOE 2.0 ASN SFTP Ingestion + Parsing (Sftp/Import → Sftp/Parse → Case/ServiceRequest)](features/asn-sftp-ingestion-2-0.md) | 2.0 | 2.0 rewrite of DOE's vendor-SFTP ASN intake (TRUE-80519 Phases 1–2): per-manufacturer fetch to `Files`, header-validated parse into one Case per customer PO + o |
|
|
5
6
|
| [NYCDOE EDI 850 Receive & 997 Acknowledgement](features/edi-850-receive-and-997-ack.md) | 1.0 | DOE's EDI 850-in → NetSuite SO + inline 997-out ack; open for the 997 build, its "always accept" stub, or a DOE ack complaint. |
|
|
6
7
|
| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, and the outbound cron's sel |
|
|
7
8
|
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | Full DOE ServiceNow/ASN integration map — ticket mirror, ASN → NetSuite SO/TOGa Desk pipeline, outbound status/POD, EDI; open for any DOE cron/pipeline work or |
|
|
8
|
-
| [New York City Department of Education](profile.md) | 1.0 | NYC DOE client profile (1.0 worker tier, ServiceNow + EDI
|
|
9
|
+
| [New York City Department of Education](profile.md) | 1.0 | NYC DOE client profile (1.0 worker tier live; 2.0 ASN rewrite in progress), ServiceNow + EDI; open for DOE's platform scope, endpoints, API clients, and where i |
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NYCDOE 2.0 ASN SFTP Ingestion + Parsing (Sftp/Import → Sftp/Parse → Case/ServiceRequest)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: nycdoe
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Sftp/Import.php
|
|
13
|
+
- worker2/Worker/Sftp/Parse.php
|
|
14
|
+
- _underscore/Model/Nycdoe/File.php
|
|
15
|
+
- _underscore/Model/Nycdoe/Case.php
|
|
16
|
+
- _underscore/Model/Nycdoe/ServiceRequest.php
|
|
17
|
+
- dbchanges2/Client_Nycdoe/2026-09-22b - Add ASN parsing fields to ServiceRequests.sql
|
|
18
|
+
- dbchanges2/Client_Nycdoe/2026-09-22c - Add customerPurchaseOrder field to Cases.sql
|
|
19
|
+
- worker2/tests/Integration/ParseCest.php
|
|
20
|
+
related:
|
|
21
|
+
- servicenow-integration.md
|
|
22
|
+
- ../profile.md
|
|
23
|
+
- ../../../2.0/apps/worker2/features/codeception-integration-tests.md
|
|
24
|
+
- ../../../2.0/apps/_underscore/features/cli-script-bootstrap.md
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
2.0 rewrite of DOE's vendor-SFTP ASN intake (TRUE-80519 Phases 1–2): per-manufacturer fetch to `Files`, header-validated parse into one Case per customer PO + one ServiceRequest per line; open for any 2.0 DOE ASN/SFTP work.
|
|
28
|
+
|
|
29
|
+
## Summary
|
|
30
|
+
|
|
31
|
+
The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [ServiceNow / ASN Integration](servicenow-integration.md) for the 1.0 pipeline, still live). **ASN-first:** nothing is created in 2.0 except from an ingested ASN file. Phase 1 (fetch) and Phase 2 (parse) are built and verified locally against real sample files for all 4 manufacturers; Phase 3 (NetSuite SO, freeze/growth lineage) is not built. Not yet deployed/scheduled in prod.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
1. **`Sftp/Import/Run`** — one `Core.CronJobs` row per client+manufacturer (`dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE <Mfr>.sql`). Connects (secret by reference via `_Component_Secrets::resolve`), lists files matching `fileNamePattern`, dedupes (size/mtime, then checksum), saves each new file as a `Files` row (`Files.data` is `FIELD_STORAGE` → auto S3), `c_status='pending'`, then `_Worker::runTask('Sftp/Parse/<adapter>', …)`. Client-agnostic; the client's `Files` model class is passed per CronJobs row. `RunLocalTest()` swaps SFTP listing for a local folder (local SFTP is blocked: dev AWS SSO cannot assume WorkloadsRuntime).
|
|
36
|
+
2. **`Sftp/Parse/{Lenovo,Lexmark,Acer,Apple}`** — loads the `Files` row, **validates the header by exact name match** (mismatch → `c_status='failed'` + throw; never a column-count guess — the direct fix for 1.0's "Lenovo Off-Layout ASN File"), parses rows, then:
|
|
37
|
+
- **Case = one per customer PO** (`Cases.c_customerPurchaseOrder`; `Cases.number` is `FIELD_AUTOINCREMENT`, not settable). One file can hold several POs.
|
|
38
|
+
- **ServiceRequest = one per line**, `serviceRequestTypeId` = `Onsite`, stage = `New` (both looked up by name, never hardcoded ids). Fields: `c_partNumber`, `c_serialNumber`, `c_ritmNumber`, `c_filesId`.
|
|
39
|
+
- **Dedupe on re-run:** serialized → match `c_serialNumber`; non-serialized → `(caseId, c_partNumber)` — one record per part per Case regardless of qty. Re-running a file creates nothing (verified all 4).
|
|
40
|
+
- Fail-loud: a bad row aborts the job; no catch-and-skip.
|
|
41
|
+
|
|
42
|
+
| Mfr | Layout | Header const | Date column / format |
|
|
43
|
+
|---|---|---|---|
|
|
44
|
+
| Lenovo | own 30-col layout (not CDW) | `LENOVO_HEADER` | Ship Date `M/D/YYYY H:MM:SS AM/PM` (date part) |
|
|
45
|
+
| Lexmark | CDW 47-col | `LEXMARK_HEADER` | `Order_Date` `YYYY/MM/DD` |
|
|
46
|
+
| Acer | CDW 47-col, 3 names differ from Lexmark | `ACER_HEADER` | `YYYY/MM/DD` |
|
|
47
|
+
| Apple | own | `APPLE_HEADER` | `order_date` `M/D/YY` or `M/D/YYYY` |
|
|
48
|
+
|
|
49
|
+
Schema (`dbchanges2/Client_Nycdoe/`): `2026-09-14a` Files `c_status`/`c_manufacturer`/`c_dtProcessed`; `2026-09-22a` ServiceRequestStages seed (New/Scheduled/On Hold/Delivered/Cancelled); `2026-09-22b` ServiceRequests fields; `2026-09-22c` Cases `c_customerPurchaseOrder`; `2026-09-25a` Cases `longDescription`.
|
|
50
|
+
|
|
51
|
+
Tests: `worker2/tests/Integration/ParseCest.php` (all 4 mfrs + header rejection + comma repair) — see [worker2 Codeception integration tests](../../../2.0/apps/worker2/features/codeception-integration-tests.md).
|
|
52
|
+
|
|
53
|
+
## Gotchas
|
|
54
|
+
|
|
55
|
+
- **Direct `_Model` writes need every `c_` field declared as a real property** — `_Model::__set()` throws `There is no field called 'c_…'` otherwise. The "c_ needs no declaration" behavior is the /v2 API engine's, not `_Model`'s. Hence `_Model_Nycdoe_File` / `_Model_Nycdoe_Case` / `_Model_Nycdoe_ServiceRequest`.
|
|
56
|
+
- **1.0 `asnprocessor/lenovo.php` is not a source of truth for today's Lenovo layout.** Its 47-col mapping is stale; the real file is 30 cols (confirmed on 3 live samples). Verify every header and date format against a **real current sample file**, never the 1.0 code — wrong Lenovo header, Lexmark date, and Apple 2-digit year (`'25'` → year `0025`) were all caught this way.
|
|
57
|
+
- **Acer and Lexmark share column positions, not header names** (Acer `First_Name_Shipping`/`Last_Name_Shipping`/`SerialNo` vs Lexmark `Contact_Name_Shipping`/`Email_Shipping`/`Svc_Tag_ID`). Keep separate header constants. Lenovo stays its own path (not a CDW supplier).
|
|
58
|
+
- **Lenovo unescaped comma in `Product Long Description` (col 18) is repaired, not rejected** — `repairLenovoRowColumnCount()` merges the extra fragments back (ported from 1.0; the only field Lenovo breaks this way). A true header mismatch is still fatal. No real occurrence seen in 2.0 yet — re-confirm when Lenovo next sends one.
|
|
59
|
+
- **`DOE-OPEN-LENOVO-*` files are a retired process — never parse; delete on sight** (Skyler/Eric). Lenovo's `fileNamePattern` `DOE-Lenovo-ASN-*.csv` already excludes them. SFTP cleanup of old ones is deferred.
|
|
60
|
+
- **Schema drift: `_Model_Client_Case` declares `longDescription` but `Client_Nycdoe.Cases` lacked the column** → any direct-model Case load/save 1054s. Fixed for DOE (`2026-09-25a`). Other tenants' `Cases` may have the same gap — unchecked (ask Rohan).
|
|
61
|
+
|
|
62
|
+
## Change history
|
|
63
|
+
- 2026-09-25 — Phase 2 parsing built for all 4 mfrs; header validation, comma repair, real-sample header/date corrections; integration tests added (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/import_asn.php
|
|
@@ -43,6 +43,7 @@ related:
|
|
|
43
43
|
- edi-850-receive-and-997-ack.md
|
|
44
44
|
- ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
|
|
45
45
|
- hold-status-sync.md
|
|
46
|
+
- asn-sftp-ingestion-2-0.md
|
|
46
47
|
- ../../../1.0/apps/worker/architecture.md
|
|
47
48
|
- ../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md
|
|
48
49
|
---
|
|
@@ -145,7 +146,7 @@ The five 1.0 `App_SystemMonitor_ServiceNow*` monitors scrape logs for **arrived-
|
|
|
145
146
|
- **Any structured HTTP response below 500 is a PASS** — it proves DNS, TLS, the load balancer, routing, and the auth layer are alive. A `401` is a *healthy* answer, not a failure.
|
|
146
147
|
- **Accepted coverage gap: token validity is NOT monitored.** The OAuth client `App_Api_NYCDOEV2` lives in the 1.0 `library` repo, which worker2 cannot reach. Re-implementing it in worker2 would duplicate DOE's credentials into a second repo and race the shared access token. An expired/revoked token will surface via the 1.0 failure monitors, not this probe — do not "fix" this by forking the client.
|
|
147
148
|
|
|
148
|
-
|
|
149
|
+
All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progress ASN rewrite on the `Client_Nycdoe` tenant — see [2.0 ASN SFTP Ingestion](asn-sftp-ingestion-2-0.md). See [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md).
|
|
149
150
|
|
|
150
151
|
### Operating rules when changing this integration
|
|
151
152
|
1. **Trace the full pipeline downstream before fixing** — the constraint that makes an edit unsafe usually lives downstream (the freeze point and the consumer/display query).
|
|
@@ -170,7 +171,8 @@ This probe is the client's **only 2.0 footprint**. DOE is **1.0 by default** —
|
|
|
170
171
|
- **Asset-tag repair is a SEPARATE, TWO-STORE backfill** — see the derived-`assets.tag` gotcha below. Generate the SQL **mechanically** from the customer's serial→tag CSV (never hand-transcribe), match serials through `App_NYCDOE::serialKey()` (the leading-`S` gotcha below), scope the statements to the ASN/MSO, and fill **only still-empty** tags so the file is re-runnable. Done this way for ASN `26720` / PO `S202648137` / MSO `329924` (87 units) on 2026-08-13, PR dbchanges#265.
|
|
171
172
|
- **Known remaining damage from this defect** (each needs its own customer serial→tag list): ASN `26641` / PO `S202668066` — **162 of 363** units still tagless; ASN `26706` / PO `WR270003276` — **1 of 7**.
|
|
172
173
|
- **A fixture-based regression test is owed with the parser fix**: assert that serial, quantity and assetTag each land in the right field for **both** layouts (do not detect by column count alone). ⚠ Do not quote constants out of `asnprocessor/lenovo.php` into a doc or ticket — that file contains hardcoded FTP credentials.
|
|
173
|
-
-
|
|
174
|
+
- **`DOE-Open-Lenovo` is a retired Lenovo process — it should never be parsed; delete on sight** (Skyler/Eric, 2026-09-25). The 1.0 parser still picks it up (unchanged).
|
|
175
|
+
- **Recommended 1.0 code fixes (proposed, NOT approved, NOT implemented):** map columns by **header name** and hard-reject an unknown layout with a loud alert; stop building the dedupe key from a field that can be a constant; widen `AdvanceShippingNoticeItems.partNumber`. The 2.0 rewrite already does exact header validation and excludes `DOE-Open-Lenovo` by filename — see [2.0 ASN SFTP Ingestion](asn-sftp-ingestion-2-0.md).
|
|
174
176
|
- **UNRESOLVED.** On ASN `26720`, items `54672`/`54673` carried 31 and 25 units with **correct** serials in `serialNumber` and `assetTag` NULL — which the three collapsed queue rows cannot account for. The run that created those units was **not identified**. Still open; do not assume it is explained.
|
|
175
177
|
- **Ruled out:** the recent **Quad** ASN import work is unrelated — Quad lives under `worker/crons/toga2/**` against the toga2 `AdvanceShippingNoticeItemUnits` schema, while the DOE path is `Core.AdvanceShippingNoticeUnits` via `asnprocessor/lenovo.php`.
|
|
176
178
|
|
|
@@ -257,3 +259,6 @@ This probe is the client's **only 2.0 footprint**. DOE is **1.0 by default** —
|
|
|
257
259
|
- [NYCDOE EDI 850 Receive & 997 Acknowledgement](edi-850-receive-and-997-ack.md) — 850-in/997-out
|
|
258
260
|
- [Worker (1.0) architecture](../../../1.0/apps/worker/architecture.md) — cron dispatch, role election, schedule files
|
|
259
261
|
- Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
|
|
262
|
+
|
|
263
|
+
## Change history
|
|
264
|
+
- 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)
|
|
@@ -7,26 +7,29 @@ apps:
|
|
|
7
7
|
- togadesk
|
|
8
8
|
- dbchanges
|
|
9
9
|
- worker2
|
|
10
|
+
- _underscore
|
|
11
|
+
- dbchanges2
|
|
10
12
|
project: Worker
|
|
11
13
|
client: nycdoe
|
|
12
14
|
type: profile
|
|
13
15
|
status: active
|
|
14
|
-
updated: 2026-09-
|
|
16
|
+
updated: 2026-09-25
|
|
15
17
|
owners: [mhammontree, sking]
|
|
16
18
|
files: []
|
|
17
19
|
related:
|
|
18
20
|
- features/servicenow-integration.md
|
|
19
21
|
- features/edi-850-receive-and-997-ack.md
|
|
22
|
+
- features/asn-sftp-ingestion-2-0.md
|
|
20
23
|
---
|
|
21
24
|
|
|
22
|
-
NYC DOE client profile (1.0 worker tier, ServiceNow + EDI
|
|
25
|
+
NYC DOE client profile (1.0 worker tier live; 2.0 ASN rewrite in progress), ServiceNow + EDI; open for DOE's platform scope, endpoints, API clients, and where its ticket/ASN/EDI logic lives.
|
|
23
26
|
|
|
24
27
|
## Summary
|
|
25
|
-
NYC DOE — TOGA client whose
|
|
28
|
+
NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/sync/nycdoe/` and `worker/crons/notifications/nycdoe/`, on the `sync` and `notification` worker roles). ServiceNow is DOE's system of record for tickets (Incidents and RITMs); TOGA mirrors those locally, drives fulfillment through TOGa Desk and NetSuite, and pushes status / ETA / proof-of-delivery back. A parallel EDI relationship (850 POs in, invoices out) runs over DOE's SFTP. A **2.0 rewrite (TRUE-80519)** of the ASN intake is being built against the `Client_Nycdoe` tenant — not yet live.
|
|
26
29
|
|
|
27
30
|
## Platforms & data
|
|
28
|
-
- **1.0 worker tier** — all business logic. Local mirror tables in `db_common` (`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).
|
|
29
|
-
-
|
|
31
|
+
- **1.0 worker tier** — all live business logic. Local mirror tables in `db_common` (`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).
|
|
32
|
+
- **2.0 (`worker2` + `_underscore` + `dbchanges2`, tenant `Client_Nycdoe`)** — (1) the OneUptime liveness probe `Monitor/Nycdoe/ServiceNowEndpointHealth` (`worker2/Worker/Monitor/Nycdoe.php`, TRUE-80587, live); (2) the ASN rewrite `Sftp/Import` → `Sftp/Parse` → Case/ServiceRequest (Phases 1–2 built, not deployed) — see [2.0 ASN SFTP Ingestion](features/asn-sftp-ingestion-2-0.md). Entity map: `Case` = 1.0 managed_service_order, `ServiceRequest` = 1.0 repair_order; DOE does not use `Ticket`. Client models: `_underscore/Model/Nycdoe/`.
|
|
30
33
|
- **TOGa Desk:** DOE is client id **16** (`db_togadesk`) — repair orders, managed service orders, the "#received / #ordered" Receiving Summary UI.
|
|
31
34
|
- **NetSuite:** Sales Orders, PO reconciliation, item receipts, Item Fulfillments, invoices (SuiteTalk toolkit in `library`).
|
|
32
35
|
- **`dbchanges` (1.0):** DOE schema + **data-repair** changes ship here (`dbchanges/Core/` targets the legacy/V1 `Core` schema). Stranded-ASN recoveries are data-only repairs — see the [1.0 dbchanges authoring workflow](../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
|
|
@@ -46,3 +49,4 @@ NYC DOE — TOGA client whose entire integration runs in the **1.0 worker tier**
|
|
|
46
49
|
|
|
47
50
|
## Key features (this client)
|
|
48
51
|
- [ServiceNow / ASN Integration](features/servicenow-integration.md) — full integration map: ticket sync, the ASN pipeline (core flow), outbound status sync, EDI, SME-confirmed business rules.
|
|
52
|
+
- [2.0 ASN SFTP Ingestion + Parsing](features/asn-sftp-ingestion-2-0.md) — the TRUE-80519 2.0 rewrite (fetch → header-validated parse → Case/ServiceRequest).
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: TRUE_80519
|
|
4
|
+
title: NYCDOE 2.0 ASN Phase 2 parsing + worker2 Codeception pilot
|
|
5
|
+
author: mhammontree
|
|
6
|
+
repos: [worker2, _underscore, dbchanges2]
|
|
7
|
+
framework: "2.0"
|
|
8
|
+
client: nycdoe
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-09-25
|
|
11
|
+
updated: 2026-09-25
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: TRUE_80519
|
|
15
|
+
**Date:** 2026-09-25
|
|
16
|
+
**Project/Repo:** worker2, _underscore, dbchanges2 (2.0)
|
|
17
|
+
**Task:** Build and verify Phase 2 (ASN file parsing → Case/ServiceRequest creation) of the NYCDOE ServiceNow/ASN/NetSuite 2.0 rewrite (TRUE-80519), then set up worker2 as 2.0's first Codeception/PHPUnit test pilot and codify the Phase 2 verification as regression tests.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
|
|
23
|
+
- **Phase 2 parsing complete for all 4 manufacturers**, each verified end-to-end via manual Postman testing against real sample files, then codified as automated tests:
|
|
24
|
+
- Lenovo: real file → 66 Cases, 746 ServiceRequests; re-run confirmed dedupe (`serviceRequestsCreated: 0`).
|
|
25
|
+
- Lexmark: real file → 4 Cases, 14 ServiceRequests; dedupe confirmed.
|
|
26
|
+
- Apple: real file → 56 Cases, 402 ServiceRequests; dedupe confirmed.
|
|
27
|
+
- Acer: real file → 12 Cases, 1145 ServiceRequests; dedupe confirmed.
|
|
28
|
+
- `Worker/Sftp/Parse.php` implements one `_Worker_Sftp_Parse::<Manufacturer>()` static method per manufacturer, header-validated against a hardcoded expected-column-names constant (LENOVO_HEADER, LEXMARK_HEADER, ACER_HEADER, APPLE_HEADER), rejecting the whole file loudly on any mismatch.
|
|
29
|
+
- **Case grain resolved**: one `Case` per customer PO (via new `c_customerPurchaseOrder` field), not one per file — confirmed from real sample data containing multiple POs per file.
|
|
30
|
+
- **NetSuite linkage design corrected** per Rohan: SO/PO header stamp belongs on `SalesOrders`/`PurchaseOrders` (`SalesOrders.serviceRequestId`, `SalesOrders_PurchaseOrders`, `SalesOrderItems.quantity`/`.qtyCommitted`), not as `Case` custom fields as originally assumed. This narrowed Phase 2's own `c_` field list to RITM linkage + `filesId` provenance only.
|
|
31
|
+
- **`_Model_Nycdoe_Case` / `_Model_Nycdoe_ServiceRequest`** (`_underscore/Model/Nycdoe/`) — client model overrides declaring the `c_` fields as real properties, required for direct `_Model` writes (same constraint as Phase 1's Files model).
|
|
32
|
+
- **dbchanges2 migrations applied and confirmed working**: `ServiceRequestStages` seeded (New/Scheduled/On Hold/Delivered/Cancelled), `ServiceRequests` gained `c_partNumber`/`c_serialNumber`/`c_ritmNumber`/`c_filesId`, `Cases` gained `c_customerPurchaseOrder` and the missing `longDescription` column (schema/model drift fix).
|
|
33
|
+
- **Codeception 5 / PHPUnit 11 set up in worker2** as 2.0's test pilot, CTO-reviewed before building (AGREE + 5 required changes, all applied) and php-reviewer-reviewed after building (2 real bugs found and fixed — see below). Dev-only via `composer require-dev`; confirmed `composer.lock`'s non-dev `packages` block is byte-identical before/after, and `--no-dev --dry-run` cleanly removes every test package (matches worker2's real deploy hook `composer install --no-dev`).
|
|
34
|
+
- **`tests/Integration/ParseCest.php`** — 6 tests (one per manufacturer + header-mismatch rejection + Lenovo comma-repair regression), all passing, confirmed repeatable across multiple runs, confirmed clean fixture teardown via a separate verification script.
|
|
35
|
+
- **Two hard safety gates** in `tests/Support/Helper/Integration.php`, both run before any write: refuses to run unless resolved `ENVIRONMENT` is `dev-*` (read from gitignored `tests/local.env.php`, never hardcoded); refuses to run against a non-local `Client_Nycdoe` database host. Verified both actually fire (temporarily pointed `local.env.php` at `'production'`, confirmed the suite refused to run, then restored it).
|
|
36
|
+
- **Lenovo comma-in-description bug fixed and tested**: ported 1.0's exact fix (`library/app/asnprocessor/lenovo.php`) for Lenovo's `Product Long Description` column sometimes containing an unquoted, unescaped comma that shifts every later column. Added `repairLenovoRowColumnCount()`. Regression test (`lenovoRepairsAnUnescapedCommaInTheDescription`) — confirmed it actually fails without the fix (serial number lookup breaks once columns shift) before confirming it passes with the fix.
|
|
37
|
+
- **php-reviewer caught 2 real bugs in the test harness itself, both fixed and re-verified**:
|
|
38
|
+
1. `commitDatabaseTransaction()` was a no-op — `_Database::transactionCommit(null)` only acts when the global `-1` flag is set, which is only set by the parameterless `_Database::transactionStart()` (called by `_Route` in production, never by our test). Fixed by calling `_Database::transactionStart()` in `_before()`. This also proved an earlier workaround (`_Database::setIsReadHostEnabled(false)`) was unnecessary once real commits happened — removed it after 3 clean re-runs confirmed tests still pass without it.
|
|
39
|
+
2. Unnecessary `ReflectionClass`/`setAccessible` into `_Database::$_registers`, which is actually a `public static` property — simplified to direct access.
|
|
40
|
+
- **Everything captured and published to the team knowledge base** (commit `146c68d7` on `origin/_main`): 9 docs written (2 new: `clients/nycdoe/features/asn-sftp-ingestion-2-0.md`, `2.0/apps/worker2/features/codeception-integration-tests.md`; 5 updated: `servicenow-integration.md`, `cli-script-bootstrap.md`, `model-magic-field-access.md`, `item-classification-itemclasses.md`, nycdoe `profile.md`; 2 elevated standards updated with developer approval: `framework-rules.md`, `backend-testing.md`).
|
|
41
|
+
|
|
42
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
43
|
+
|
|
44
|
+
- **Assuming Lenovo's file layout matched 1.0's stale `asnprocessor/lenovo.php` code (47 columns)** — wrong. The real, current Lenovo file is a genuinely different 30-column layout, confirmed against 3 real live sample files. The header-name validation correctly rejected the file when tested against the wrong assumed header — this was the design working as intended, not a bug, but the underlying `LENOVO_HEADER` constant itself had to be rewritten from scratch based on real file content, not from reading old code.
|
|
45
|
+
- **A "stale FIELD_STORAGE file due to id reuse" theory** — raised mid-investigation of the Lenovo header mismatch (suspecting local folder-mode storage served old content for a reused row id). Proven wrong: a fresh row showed the same "wrong" header, which turned out to be the real, current file format all along. Retracted for the record so it isn't mistaken for a real finding later.
|
|
46
|
+
- **Assuming Lexmark's `Order_Date` was `M/D/YYYY`** — wrong; it's `YYYY/MM/DD`. Caught by re-reading the real sample file before testing (direct lesson from the Lenovo mistake above), not by a failed test in production.
|
|
47
|
+
- **Assuming Acer/Lexmark share identical header column names** (only positions) — wrong; 3 column names differ (`First_Name_Shipping`/`Last_Name_Shipping`/`SerialNo` for Acer vs `Contact_Name_Shipping`/`Email_Shipping`/`Svc_Tag_ID` for Lexmark). A single shared `LEXMARK_ACER_HEADER` constant would have rejected every real Acer file. Split into `LEXMARK_HEADER`/`ACER_HEADER`.
|
|
48
|
+
- **Assuming a 2-digit year never appears in Apple's `order_date`** — wrong; a newer sample file had `10/23/25`, which the original `parseDate()` would have turned into year `0025`. Fixed with `expandYear()`.
|
|
49
|
+
- **Running `vendor/bin/codecept bootstrap` fully non-interactively** — it stopped and waited on an interactive `(y/n)` prompt ("composer.json updated. Do you want to run 'composer update'?") that the Bash tool couldn't answer; had to `TaskStop` the backgrounded process and finish the setup manually (`generate:suite`, manual `composer update` with the lock-diff safety check) instead of relying on the bootstrap wizard end-to-end.
|
|
50
|
+
- **The original `commitDatabaseTransaction()`/`setIsReadHostEnabled(false)` combination** — see "What WORKED" above for the real root cause and fix; the original diagnosis (separate read connection can't see an uncommitted write) was directionally right but incomplete — the actual missing piece was the global `transactionStart()` call, not just connection routing.
|
|
51
|
+
|
|
52
|
+
## Not tried yet (candidates for next session)
|
|
53
|
+
|
|
54
|
+
- **Acer/Lenovo/Lexmark date-format edge cases beyond what real samples showed** — Acer's `YYYY/MM/DD` format is now confirmed (not just guessed), but only against one sample file; worth reconfirming if a 2-digit-year Acer file ever surfaces (same class of bug already found for Apple).
|
|
55
|
+
- **Re-confirm the Lenovo comma-in-description fix against a real occurrence** — no real sample file with this exact issue was available this session; the fix mirrors 1.0's proven logic exactly but has not been tested against real Lenovo data that actually has the bug.
|
|
56
|
+
- **Deploy/rollout planning for Phase 1 + 2** (discussed conceptually, not started):
|
|
57
|
+
- Confirm whether Phase 1's SFTP fetch is read-only / non-destructive relative to what 1.0 already consumes from the same SFTP (must not interfere with 1.0's live pipeline).
|
|
58
|
+
- Decide a "shadow mode" plan: run 2.0 ingestion+parsing against the same real files 1.0 processes, writing into the isolated `Client_Nycdoe` 2.0 tables, purely for validation — low risk since Phase 3 (NetSuite/ServiceNow outbound calls) doesn't exist yet, so nothing can double-fire.
|
|
59
|
+
- Decide what "shadow mode is good enough to cut over" means concretely (e.g., compare 2.0 Case/ServiceRequest counts per manufacturer per day against 1.0's `AdvanceShippingNotices` counts) — no comparison method defined yet.
|
|
60
|
+
- No monitoring/alerting exists yet for Phase 1/2 failures (header rejections, comma-repair firing, etc.) — needs a decision on whether/how to alert (Logs.Issue entry? tie into Mark's separate Monitoring System Design project?).
|
|
61
|
+
- Branch/PR/deploy process not started: needs `php-reviewer` + `cso` on the diff, Rohan's deploy window for `_underscore`/`dbchanges2` changes, and Mark's own pseudocode/proposal to Jeff (Claude's output is reference material only — Mark authors the actual submission).
|
|
62
|
+
- **Follow-up with Rohan**: whether other clients' `Cases` tables are also missing the `longDescription` column (found only for `Client_Nycdoe` this session).
|
|
63
|
+
- **Low-priority housekeeping**: stale `DOE-OPEN-LENOVO-*.CSV` files still sit on the real Lenovo SFTP and aren't cleaned up by anything (per Skyler/Eric, safe to delete, retired process) — not blocking, not scheduled.
|
|
64
|
+
- **INC (Incident) ticket flow** — a separate flow from the ASN/RITM pipeline this session covered, not yet redesigned for 2.0 (flagged in the main plan as "next session" from an earlier date, still open).
|
|
65
|
+
- **Phase 3** (NetSuite Sales Order/PO dispatch, ServiceNow RITM linkage, outbound sync) — explicitly out of scope for Phase 2, not started.
|
|
66
|
+
|
|
67
|
+
## Current file state
|
|
68
|
+
|
|
69
|
+
| File | Status | Notes |
|
|
70
|
+
|------|--------|-------|
|
|
71
|
+
| `worker2/Worker/Sftp/Parse.php` | Modified, uncommitted | Phase 2 parsing for all 4 manufacturers + Lenovo comma-repair fix. Lint-clean, all tests pass. |
|
|
72
|
+
| `worker2/tests/Support/Helper/Integration.php` | New, uncommitted | Codeception module: framework boot, dev/local safety gates, fixture + transaction helpers. |
|
|
73
|
+
| `worker2/tests/Integration/ParseCest.php` | New, uncommitted | 6 integration tests, all passing. |
|
|
74
|
+
| `worker2/tests/local.env.php.example` | New, uncommitted | Template for the gitignored per-developer env file. |
|
|
75
|
+
| `worker2/tests/local.env.php` | New, gitignored (not committed by design) | This machine's value: `dev-markhammontree-laptop`. |
|
|
76
|
+
| `worker2/codeception.yml`, `worker2/tests/Integration.suite.yml` | New, uncommitted | Codeception config. |
|
|
77
|
+
| `worker2/.gitattributes` | New, uncommitted | `export-ignore` for `tests/` and `codeception.yml`. |
|
|
78
|
+
| `worker2/.htaccess` | Modified, uncommitted | Added deny rules for `tests/` and `*.yml`. |
|
|
79
|
+
| `worker2/.gitignore` | Modified, uncommitted | Added `/tests/_output/`, `/tests/Support/_generated/`, `/tests/local.env.php`. |
|
|
80
|
+
| `worker2/composer.json`, `worker2/composer.lock` | Modified, uncommitted | Added `codeception/codeception:^5`, `phpunit/phpunit:^11`, `codeception/module-asserts` as `require-dev` only. Confirmed non-dev `packages` block unchanged. |
|
|
81
|
+
| `_underscore/Model/Nycdoe/Case.php` | New, uncommitted | Declares `c_customerPurchaseOrder`. |
|
|
82
|
+
| `_underscore/Model/Nycdoe/ServiceRequest.php` | New, uncommitted | Declares `c_partNumber`, `c_serialNumber`, `c_ritmNumber`, `c_filesId`. |
|
|
83
|
+
| `dbchanges2/Client_Nycdoe/2026-09-22a - Seed ServiceRequestStages for DOE installs.sql` | New, uncommitted | Confirmed applied to local dev DB. |
|
|
84
|
+
| `dbchanges2/Client_Nycdoe/2026-09-22b - Add ASN parsing fields to ServiceRequests.sql` | New, uncommitted | Confirmed applied. |
|
|
85
|
+
| `dbchanges2/Client_Nycdoe/2026-09-22c - Add customerPurchaseOrder field to Cases.sql` | New, uncommitted | Confirmed applied. |
|
|
86
|
+
| `dbchanges2/Client_Nycdoe/2026-09-25a - Add missing longDescription column to Cases.sql` | New, uncommitted | Confirmed applied; fixes real schema/model drift against shared `_Model_Client_Case`. |
|
|
87
|
+
| `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` | Continuously updated throughout | Full change history of every finding this session, kept current. |
|
|
88
|
+
| Team KB (`~/toga-tech/knowledge/`) | Published | Commit `146c68d7` on `origin/_main`. 9 docs (2 new, 7 updated), see "What WORKED". |
|
|
89
|
+
|
|
90
|
+
**None of the code changes above have been committed or pushed to their respective repos yet** — draft commit titles/descriptions were prepared for worker2 (2 commits), `_underscore` (1 commit), and `dbchanges2` (1 commit), but no `git commit` has been run per the standing "Claude never commits without explicit ask" rule.
|
|
91
|
+
|
|
92
|
+
## Decisions made
|
|
93
|
+
|
|
94
|
+
- **One `Case` per customer PO, not per file.** Rationale: real sample data (Lexmark, Apple) showed multiple distinct customer POs in a single file, so "one Case per file" was never workable. No alternative seriously considered once real data was checked.
|
|
95
|
+
- **NetSuite SO/PO stamp lives on `SalesOrders`/`PurchaseOrders`, not as `Case` custom fields.** Rationale: Rohan's direct correction, verified against real schema. Rejected: the original assumption from the main plan's earlier design session.
|
|
96
|
+
- **Lenovo kept as its own header/parse path, never shared with Lexmark/Acer**, even though both `Lenovo()` and `runLexmarkAcerFamily()` looked similar early on. Rationale (Mark's explicit instruction): Lenovo is not actually a CDW supplier (only Acer/Lexmark are); sharing code risks a future Lenovo-only format change breaking two unrelated suppliers. Rejected: a single shared parsing function keyed by manufacturer name.
|
|
97
|
+
- **`DOE-OPEN-LENOVO` files are retired and must never be parsed**, confirmed by Skyler/Eric. Rationale: an old, retired process Lenovo still sends to the SFTP; Phase 1's existing filename pattern already excludes them, so no code change was needed, just confirmation.
|
|
98
|
+
- **Lenovo's comma-in-description bug is repaired, not rejected** (unlike a genuine header-shape mismatch, which stays fatal). Rationale: Mark's direct confirmation that, per 1.0's history, this is the ONLY field Lenovo ever breaks this way — a well-understood, narrow quirk worth auto-fixing rather than bouncing every affected file for manual resend.
|
|
99
|
+
- **2.0 test framework: Codeception 5 + PHPUnit 11, `worker2` as the sole pilot repo.** Rationale: Jeff's direction; CTO-reviewed first given it's a framework-adoption decision. Explicitly rejected: copying 1.0's `api` repo pattern of running tests against production on every deploy via an `.ebextensions` postdeploy hook — tests stay a manual/local step here, never a deploy-time gate, unless explicitly changed later.
|
|
100
|
+
- **`integration` suite naming, not `unit`/`functional`.** Rationale (from the CTO review): `Parse.php` needs a real DB, so it isn't a pure unit test; Codeception reserves "functional" for framework-module suites (Symfony/Laravel-style), which doesn't apply here — `integration` avoids both misnomers.
|
|
101
|
+
- **Test-only workarounds must match production's real transaction/connection behavior, not just "make the test pass."** Concretely: once `php-reviewer` proved the transaction-commit workaround was a no-op, the correct fix was to make it a REAL commit (matching what `_Route` does), not to just silence the symptom differently. This also let an earlier workaround (`setIsReadHostEnabled(false)`) be removed as unneeded once the real fix was in — verified empirically (3 clean re-runs without it) rather than assumed.
|
|
102
|
+
|
|
103
|
+
## Blockers
|
|
104
|
+
|
|
105
|
+
None for continuing Phase 2 work itself. For **rollout** (the open question this session ended on): no blocker yet identified, but the next session needs to (1) confirm Phase 1's SFTP read is non-destructive to 1.0's own consumption, and (2) decide the shadow-mode comparison method, before recommending an actual rollout plan to Mark for his own submission to Jeff.
|
|
106
|
+
|
|
107
|
+
## Exact next step
|
|
108
|
+
|
|
109
|
+
> Start the next session by reading `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` in full (it has the complete, current design + change history), then investigate whether `worker2/Worker/Sftp/Import.php`'s SFTP fetch is read-only/non-destructive relative to 1.0's own consumption of the same DOE SFTP — this is the first concrete fact needed before proposing a "shadow mode" parallel-run plan to Mark.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
_Saved by /session-save on 2026-09-25_
|
package/package.json
CHANGED