toga-ai 1.0.916 → 1.0.917
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: TRUE_80519
|
|
4
|
+
title: NYCDOE 2.0 ASN activation — gate the 1.0 SFTP delete on 2.0 (design)
|
|
5
|
+
author: mhammontree
|
|
6
|
+
repos: [worker2, dbchanges2, _underscore, worker, library]
|
|
7
|
+
framework: "both"
|
|
8
|
+
client: nycdoe
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-09-29
|
|
11
|
+
updated: 2026-09-29
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: TRUE_80519
|
|
15
|
+
**Date:** 2026-09-29 (work 2026-09-25 → 2026-09-29)
|
|
16
|
+
**Project/Repo:** worker2 / dbchanges2 / _underscore (2.0) + worker / library (1.0, read-only)
|
|
17
|
+
**Task:** Plan how to turn on the already-built NYCDOE 2.0 Phase 1 + 2 (SFTP import + parse) in production next to the live 1.0 pipeline, so 1.0 no longer deletes a vendor ASN file until 2.0 has imported and parsed it. Design only — for Mark to review with Jeff.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
- **Traced how 1.0 deletes vendor files (code read):** `worker/crons/sync/nycdoe/legacy_import_asn.php` (*/5) → `App_ASNProcessor_{Apple,Lexmark,Lenovo}::processFiles()` read over plain FTP (`App_FTP`), write `Logs.FileLog`, parse, cache to disk, email, then queue `App_Worker::runTask('Utilities/Sftp/DeleteFile/DeleteSftpFile', …)` — `lenovo.php:176-184`, `apple.php:105-113`, `lexmark.php:102-110`. Gated only on `$filePutContentsResults !== false` (disk cache), NOT on a good parse. `worker2/Worker/Utilities/Sftp/DeleteFile.php` deletes over SFTP. Password travels in the SQS message.
|
|
23
|
+
- **Confirmed 2.0 `worker2/Worker/Sftp/Import.php` `Run` is read-only on the SFTP** (only `nlist`, `stat`, `get`).
|
|
24
|
+
- **Acer proven via data:** legacy `Logs.FileLog` has 458 `ASN_LEXMARK` rows with `acer` in `fileName` (2025-08-18 → 2026-09-25), 0 `ASN_ACER` rows. 1.0 `acer.php` is dead code; Mark confirmed Acer files arrive through CDW (shared CDW/Lexmark account), so 1.0 processes them as Lexmark.
|
|
25
|
+
- **`Logs.FileLog` index check (legacy, read-only):** only `PRIMARY(id)`, `FileLog_job(job)`, `dtStamp`. EXPLAIN of a job+fileName+fileTimestamp lookup uses `FileLog_job`, ~1,899 rows for `ASN_LENOVO` (954 Lexmark, 113 Apple). Fine if it selects only `id`.
|
|
26
|
+
- **CTO review (2026-09-25) of the first (polling) design: AGREE** + 3 fixes (checksum match, patterns from all CronJobs rows by account, one job per run). Those ideas were carried into the rewrite.
|
|
27
|
+
- **4 CronJobs migrations set to `isActive = 0`** (`dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE *.sql`, line 4). Mark committed it as `ed0acb8` (2026-09-28) on `TRUE-80519`, in sync with `origin/TRUE-80519`.
|
|
28
|
+
- **Plan rewritten** — `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` §6.1–6.5 (Mark's idea: 1.0 triggers 2.0, 2.0 deletes after a good parse).
|
|
29
|
+
- **/capture published** KB commit `8377d069` on `origin/_main` (both NYCDOE docs updated).
|
|
30
|
+
|
|
31
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
32
|
+
- **"2.0 decides, 1.0 keeps asking" polling design (2026-09-25):** replaced, not wrong but worse. 1.0 re-queued a gated delete every 5 min; 2.0 had to download + sha256 every waiting file each run; delete waited on 2.0's `*/15` cron; needed a decision on whether inactive CronJobs count; rollback was a two-step change (kill switch AND crons off). Mark's trigger design removes all of these.
|
|
33
|
+
- **Matching the gated delete on `Files.name` or name+size:** unsafe. A same-content re-send under a new name has no `Files` row of its own (`Import.php:171` dedupes on checksum) → never deleted; a same-name re-send with new content matches the old `succeeded` row → deleted before import. Use checksum.
|
|
34
|
+
- **Trusting a manufacturer sent from 1.0:** 1.0 Lexmark hands over Acer files too (shared folder), so matching only the Lexmark pattern would treat an Acer file as junk and delete it. Resolve patterns by SFTP username (CDW returns Lexmark + Acer rows).
|
|
35
|
+
- **Earlier "1.0 copies the file to a temp SFTP subfolder" bridge (2026-08-28):** dropped — the file stays in the real vendor folder until 2.0 is done.
|
|
36
|
+
- **Assuming the 2026-09-14a CronJobs rows were inactive:** they were written `isActive = 1`; now `0`.
|
|
37
|
+
|
|
38
|
+
## Not tried yet (candidates for next session)
|
|
39
|
+
- Fresh `cto` pass on the rewritten design (§6.2) — the 2026-09-25 AGREE was on the polling version.
|
|
40
|
+
- Review §6 with Jeff (Mark authors his own proposal; plan is reference material only).
|
|
41
|
+
- Add `'released'` to the `c_status` ENUM in `dbchanges2/Client_Nycdoe/2026-09-14a - Add SFTP import status fields to Files.sql` (not shipped yet) — for manual release of `failed` files.
|
|
42
|
+
- Build: `Sftp/Import/RunForFile(clientId, username, filePath)`, `Sftp/Import/DeleteSource(clientId, filesUuid)`, `deleteSourceOnSuccess` flag on the 4 `Sftp/Parse/*` methods (+ later on `Run`), `mtime` minimum-age guard in `Run`.
|
|
43
|
+
- 1.0 changes in `lenovo.php` / `apple.php` / `lexmark.php`: replace `DeleteSftpFile` runTask with `RunForFile`; skip files already in `Logs.FileLog` but re-queue `RunForFile`; `[nycdoe] asn_delete_mode = gated|immediate` (missing = immediate). Lint with PHP 7.2 (xampp_7_2_33).
|
|
44
|
+
- Codeception tests for the delete decision logic (keep it in a plain method with no SFTP calls, like `ParseCest`).
|
|
45
|
+
- SFTP file-age monitor (OneUptime, phpseclib3 pattern) + alert on `c_status = 'failed'`.
|
|
46
|
+
- Check whether `Core.WorkerJobs` stores the SFTP password from today's `DeleteSftpFile` parameters (security).
|
|
47
|
+
- Older open items still open: Acer/Lenovo date + comma-repair re-checks on real files; ask Rohan about other tenants' `Cases.longDescription`; INC flow redesign; Phase 3.
|
|
48
|
+
|
|
49
|
+
## Current file state
|
|
50
|
+
| File | Status | Notes |
|
|
51
|
+
|------|--------|-------|
|
|
52
|
+
| `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` | Updated (local, not in a repo) | §6 Activation: 6.1 how 1.0 deletes, 6.2 trigger design, 6.3 9-step activation order, 6.4 risks, 6.5 cutover; change history entries 2026-09-25 + 2026-09-29 |
|
|
53
|
+
| `dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE {Lenovo,Apple,Lexmark,Acer}.sql` | Committed `ed0acb8`, pushed to `origin/TRUE-80519` | `isActive = 0` |
|
|
54
|
+
| worker2 `TRUE-80519` | Clean, in sync with origin (`70f1061`) | Phase 1/2 + Codeception committed; no gated-delete code yet |
|
|
55
|
+
| _underscore `TRUE-80519` | In sync with origin (`1223e2a9`) | 3 unrelated CSS files show as modified (`Selectize` scss css, `underCSS.css`) — not from this work |
|
|
56
|
+
| `knowledge/clients/nycdoe/features/servicenow-integration.md`, `asn-sftp-ingestion-2-0.md` | Published `8377d069` | 1.0 delete path, Acer via CDW, FileLog index, activation design (marked not built) |
|
|
57
|
+
|
|
58
|
+
## Decisions made
|
|
59
|
+
- **1.0 must not delete an ASN file until 2.0 has imported and parsed it** (Mark). Rejected: shadow run racing 1.0's delete; temp-subfolder copy.
|
|
60
|
+
- **1.0 triggers 2.0, 2.0 deletes after a good parse** (Mark, 2026-09-29). 1.0 replaces its `DeleteSftpFile` runTask with `Sftp/Import/RunForFile` (no password in SQS); Parse queues `Sftp/Import/DeleteSource` on `succeeded`; `DeleteSource` re-checks status + remote checksum before deleting. Rejected: the polling design (see DO NOT RETRY).
|
|
61
|
+
- **2.0 CronJobs stay `isActive = 0` for all of shadow mode** — used only as config, so 1.0 is the only trigger and 2.0 can never delete a file 1.0 hasn't read. Accepted trade-off: 2.0's own listing (`Run`) is only exercised at cutover.
|
|
62
|
+
- **Patterns are resolved by SFTP username across all CronJobs rows**, never from a 1.0-supplied manufacturer (Acer via CDW).
|
|
63
|
+
- **Files matching no 2.0 pattern are deleted as today** (junk, retired `DOE-OPEN-LENOVO-*`).
|
|
64
|
+
- **Kill switch defaults to `immediate`**; rollback is one config change. 1.0 ships in `immediate` first, then flips to `gated`.
|
|
65
|
+
- **Cutover end state = same path**: turn crons on with `deleteSourceOnSuccess = true`, remove 1.0, delete `RunForFile` + the 1.0 bridge code.
|
|
66
|
+
- Vendors upload and forget — longer file lifetime on the SFTP has no impact (Mark).
|
|
67
|
+
|
|
68
|
+
## Blockers
|
|
69
|
+
none — waiting on Jeff's review of §6 before building.
|
|
70
|
+
|
|
71
|
+
## Exact next step
|
|
72
|
+
> Open `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` §6 and review it with Jeff; then run a fresh `cto` review of §6.2 (RunForFile / DeleteSource) before writing any code.
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
_Saved by /session-save on 2026-09-29_
|
package/package.json
CHANGED