toga-ai 1.0.686 → 1.0.688

package/knowledge/2.0/standards/framework-rules.md

@@ -5,13 +5,15 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-07-29
-owners: [jcardinal, ajean]
+updated: 2026-08-28
+owners: [jcardinal, ajean, apeterson]
 files: []
 related:
   - ../apps/_underscore/architecture.md
   - ../apps/worker2/architecture.md
   - ../apps/api2/architecture.md
+  - ../apps/dbchanges2/architecture.md
+  - ../apps/dbchanges2/features/surface-layer-schema.md
 ---
 
 # Framework 2.0 (_underscore) Rules
@@ -137,6 +139,14 @@ Enforced mechanically by the `PreToolUse` hook
 examples, and the approved rewrites: `2.0/apps/dbchanges2/architecture.md` → *Database
 isolation*. **Applies to `dbchanges2` only — the 1.0 `dbchanges` repo is exempt.**
 
+**Verified empirically in production, 2026-08-28:** the Core reader exposes only `Core`,
+`Forecast`, `Forecast_Archive` and `Team`; the client reader exposes only `Client_*`. This is a
+**physical** boundary, not a permissions setting to be widened — a `Client_*` file naming `Core.*`
+does not fail an authorization check, it cannot resolve at all. Corollary for **reviewers**: "it
+ran locally" and "it ran on beta" are both worthless as evidence of production-runnability, because
+every non-prod environment is all-in-one. The only meaningful checks are the hook, or a
+connection-scoped run against a user granted just the one database.
+
 ### `Core.Records` / `Core.RecordFields` — the only hardcoded `id`s on the platform
 
 **These two tables, and only these two, have team-maintained primary keys.** Their `id` values are
@@ -161,6 +171,58 @@ This is also what makes the `dbchanges2` isolation rule above workable: a `Clien
 never has to read `Core` to resolve a `recordFieldId`. Full detail and examples:
 `2.0/apps/dbchanges2/architecture.md` → *`Core.Records` / `Core.RecordFields`*.
 
+#### ⚠ Bounded exemption — the Surface tables
+
+**`Core.Surfaces`, `Core.SurfaceElements` and `Core.Messages` also carry team-reserved literal `id`
+blocks.** This is a **specific exemption for the Surface layer, not a general precedent** — do not
+cite it to justify hardcoding ids for any other feature. Reserving them was forced by the cluster
+split above: `SurfaceOverrides` lives in the **Client** DB while the elements it overrides live in
+**Core**, so a `Client_*` override file must reference a Core id it is forbidden to look up at run
+time. Without a stable literal there is nothing for that file to hardcode.
+
+When seeding into these three tables:
+
+- **Verify the block is actually free** — run both `SELECT MAX(id)` *and*
+  `SELECT id FROM … WHERE id BETWEEN <lo> AND <hi>`. Confirm whether a number you were handed means
+  **"next free"** or **"highest used"**; reading one as the other has already caused a live
+  collision.
+- **Use a plain `INSERT`, never `INSERT IGNORE`.** `IGNORE` turns a primary-key collision — the one
+  error that must halt the migration — into a silent mis-wiring. Put idempotency in a `NOT EXISTS`
+  guard on a *semantic* key, or in a paired `UPDATE`.
+
+Blocks in use, the authoring house style, and the collision incident:
+[surface-layer-schema](../apps/dbchanges2/features/surface-layer-schema.md).
+
+### A committed migration records INTENT, not deployed state
+
+**Never infer what an environment contains by reading `dbchanges2`. Query the environment.**
+
+There is no automated executor. A human applies each `.sql` per environment, and where the
+isolation rule bites — a client-side file that needs a Core id — the operator resolves the id
+against the target's Core and **hand-writes the literal statement at run time**. Those
+transpositions have historically **not been written back to the repo**. The consequence is
+permanent and structural, not a backlog item:
+
+- A file in the repo may never have run anywhere.
+- A file that *did* run may have run in an edited form that exists nowhere in version control.
+- Deployed rows can therefore have no committed file that would reproduce them.
+
+So before authoring anything whose correctness depends on current state — an override, a
+correction, an `INSERT … NOT EXISTS` guard, an id block — **establish that state with read-only
+`SELECT`s against the target environment's read replicas.** Connect using the `readhost` keys of
+the `[database]` and `[databaseClient]` groups in `api2/Config/production.ini`; never copy
+credential values into a doc, a ticket, a commit or a chat message.
+
+**The hook cannot save you here.** `dbchanges2-cluster-isolation.js` is a static check on the text
+of a file, so it catches a cross-database *reference* at authoring time. It cannot know whether a
+file ever ran, ran in edited form, or matches the target environment. Cluster isolation is
+mechanically enforced; state verification is **on you**.
+
+Worked examples of both failure modes — a file unrunnable in prod, and prod rows with no
+corresponding committed file — are in
+[surface-layer-schema](../apps/dbchanges2/features/surface-layer-schema.md) and
+[surface-resolver](../apps/_underscore/features/surface-resolver.md).
+
 ## Checking dependencies before touching shared code
 
 `dependsOn` in `knowledge/registry.json` means a repo extends or depends on another repo's classes. Before modifying a class in a dependency repo (e.g. `_underscore` core):
@@ -230,6 +292,22 @@ and not how the convention works. Use a non-`c_` column when you *do* want the f
 ACL-governed field treatment.
 
 ## Change history
+- 2026-08-28 — Extended § Schema migrations with the production-state rules. Recorded that cluster
+  isolation is **physical** (verified: the prod Core reader exposes only Core/Forecast/
+  Forecast_Archive/Team, the client reader only `Client_*`), so a cross-database reference cannot
+  resolve rather than merely being denied — and that neither a local nor a beta run is evidence of
+  prod-runnability, since every non-prod environment is all-in-one. Added a **bounded exemption**
+  to the hardcoded-id rule for the Surface tables only (`Core.Surfaces`/`SurfaceElements`/
+  `Messages` carry reserved literal blocks because `Client_*` override files must reference Core
+  ids across the cluster split and cannot resolve them at run time) — **explicitly not a general
+  precedent**; the platform rule that only `Core.Records`/`Core.RecordFields` have team-maintained
+  ids is unchanged. Attached the seeding safeguards to that exemption (verify the block is free
+  with `MAX(id)` *and* a `BETWEEN` check; "next free" vs "highest used"; plain `INSERT`, never
+  `INSERT IGNORE`). New subsection: **a committed migration records intent, not deployed state** —
+  there is no automated executor, operators hand-transpose Core ids at run time and historically
+  did not write them back, so current state must be established with read-only queries against the
+  target environment, and the cluster-isolation hook catches cross-database references but
+  **cannot** catch state drift. (apeterson)
 - 2026-07-29 — Documented the `c_`-prefixed framework-dynamic column convention (schema-only,
   no model-class/RecordFields/ACL declaration), generalized from the transcript AI-model routing
   work. (ajean)

package/knowledge/sessions/2026-08-28-nycdoe-phase1-sftp-mhammontree.md

@@ -0,0 +1,134 @@
+---
+type: session
+slug: nycdoe-phase1-sftp
+title: NYCDOE 2.0 Phase 1 — SFTP Ingestion Code-Level Plan (TRUE-80519)
+author: mhammontree
+repos: [dbchanges2, _underscore, worker2, worker]
+framework: "both"
+client: nycdoe
+status: active
+created: 2026-08-28
+updated: 2026-08-28
+---
+
+# Session: nycdoe-phase1-sftp
+**Date:** 2026-08-28
+**Project/Repo:** dbchanges2, _underscore, worker2 (2.0) + worker (1.0)
+**Task:** Following Jeff's scoping decision (Phase 1 = the SFTP ingestion portion of the NYCDOE
+2.0 rewrite, TRUE-80519), produced a code-level Phase 1 design document for Mark's review meeting
+with Jeff, then resolved repo/branch scope and local database setup requirements to start
+implementation against `Client_Nycdoe` (the only current client receiving Lenovo ASN files, with
+real production examples to test against). Continues directly from the
+`2026-08-28-nycdoe-2-0-integration-mhammontree` session saved earlier today.
+
+---
+
+## What WORKED
+
+- **Produced `C:\Users\mhammontree\.claude\plans\phase-1-sftp-ingestion-plan.md`** — a full
+  code-level design doc: real table DDL (`SftpCredentials`, `SftpManufacturerImports`), a worker2
+  class/method sketch (`_Worker_Sftp_Discover::ForManufacturer`/`::ForClient`) grounded in the
+  confirmed-real `Startech.php` credential-resolution pattern, cron registration SQL, dbchanges2
+  module scope, the 1.0→2.0 SFTP bridge plan, explicit security requirements (secrets by reference,
+  never raw columns), and a scale section for the confirmed 1000+ client target.
+- **Confirmed via code read**: `_underscore\Model\Client\File.php` already exists as an established
+  Model — meaning `Files` is very likely already a registered `/v2` API Record, so adding the new
+  `c_status`/`c_partnerId`/`c_dtProcessed` custom fields is a `dbchanges2` data change, not an
+  `api2` code change. This directly answered whether `api2` needed a working branch (it doesn't,
+  for Phase 1).
+- **Confirmed via code read**: existing S3-upload precedent already lives in `worker2`
+  (`Worker/Team/Transcripts.php`, `Worker/Client/Compass/VipSupport.php`) — real code to reuse/adapt
+  for the SFTP-to-S3 backup step, not a new S3 component to invent from scratch.
+- **Confirmed the §2.5 open question from the Phase 1 doc**: every client has its own genuinely
+  separate SFTP server and credentials — no shared-server-with-per-client-subfolder topology.
+  Settles the `SftpCredentials` table as `Client_<Name>`-scoped (as originally designed) and
+  confirms the real scale problem is up to 1000+ genuinely separate SFTP connections per
+  manufacturer cron tick — updated the plan doc's §2.5 and §7 accordingly (previously "open
+  question," now "confirmed design constraint").
+- **Resolved which repos need working branches for Phase 1**, reasoned from the plan doc's actual
+  scope, not guessed: `dbchanges2`, `_underscore`, `worker2`, and 1.0 `worker` (for the temporary
+  bridge script change) — NOT `api2` (see above), NOT 1.0 `dbchanges` (the bridge change is pure
+  PHP logic in an existing 1.0 script, not a schema/data migration).
+- **Resolved local database requirements for starting `Client_Nycdoe` work**: `Core_2` (local-
+  renamed 2.0 Core) + `Client_Nycdoe` + **`Logs_Nycdoe` (confirmed necessary, not optional)** +
+  `Archive_Nycdoe` (parity, unexercised by Phase 1). The `Logs_Nycdoe` requirement is directly tied
+  to a real platform gotcha: `_ApiRequest` logging silently no-ops (or errors "Unknown database")
+  unless the Logs database is registered as `DB_CLIENT_LOGS` — load-bearing because this whole
+  design's rationale for writing through the API instead of direct model writes is to get logging
+  for free, so it has to actually be verifiable locally.
+- **Located the exact local-db-refresh workflow doc** for confirming all four databases are present
+  and consistent: `C:\Users\mhammontree\toga-tech\knowledge\2.0\apps\_underscore\workflows\local-db-refresh-from-beta.md`
+  (in the **team knowledge repo**, not inside the `_underscore` codebase itself — worth being
+  precise about this, since the two are easy to conflate).
+- Both plan-file updates (Phase 1 doc §2.5/§7 revision, new §8 "Getting started" section) were made
+  **before** this save, per this session's own established discipline of writing durable decisions
+  to disk before compacting/running low on context.
+
+## What did NOT work — DO NOT RETRY THESE
+
+- None this session — this was a clean continuation with no rejected approaches. (The prior
+  session, `2026-08-28-nycdoe-2-0-integration-mhammontree`, has the real "did not work" list from
+  the broader design work — S3-vs-SFTP-bridge confusion, the ledger-table-vs-custom-fields
+  correction, the SO/PO grain flip-flop. Nothing from that list was revisited or retried here.)
+
+## Not tried yet (candidates for next session)
+
+- Actually verify `files` is a registered `Core.Records` route in `api2` (the plan doc treats this
+  as "very likely true, worth a quick sanity check" — not yet directly confirmed).
+- Read `Worker/Team/Transcripts.php` and `Worker/Client/Compass/VipSupport.php` in full to extract
+  the actual reusable S3-upload pattern (only confirmed they exist via grep, not yet read in depth).
+- Determine the exact secrets-resolution mechanism for `SftpCredentials.secretRef` (Secrets Manager
+  vs. SSM Parameter Store, and which helper/component — if any — already exists for reading either
+  from PHP; the plan doc's `_Component_Secrets::resolve()` call is a placeholder name, not confirmed
+  real).
+- Confirm `phpseclib`'s SFTP client supports a server-side `rename()`/move (for the 1.0 bridge
+  script) vs. requiring a full download+reupload — still an open item carried from the prior session.
+- Run the `local-db-refresh-from-beta.md` workflow against `Client_Nycdoe` and confirm all four
+  local databases (`Core_2`, `Client_Nycdoe`, `Logs_Nycdoe`, `Archive_Nycdoe`) actually exist and
+  are current — Mark suspected he already has them all but had not yet confirmed at end of session.
+- Everything else still open from the prior session's "not tried yet" list remains open and
+  unaffected by this session's work (SO/PO grain confirmation with NetSuite/accounting, ASN file
+  format consistency across manufacturers/clients, §4.1 PO-suffix problem, Incidents/INCs entirely
+  deferred, worker-tier concurrency load estimate for 1000+ simultaneous SFTP connections).
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `C:\Users\mhammontree\.claude\plans\phase-1-sftp-ingestion-plan.md` | Created and revised this session | New file: full code-level Phase 1 design (table DDL, worker2 class sketch, cron SQL, dbchanges2 scope, security requirements, scale analysis). Revised §2.5/§7 from "open question" to "confirmed" once the per-client-SFTP-topology fact was confirmed. New §8 "Getting started" section added (repos/branches, local DB setup) just before this save. |
+| `C:\Users\mhammontree\.claude\plans\true-80519-this-is-a-sunny-riddle.md` | Unchanged this session | Still pending the update noted as "next step" in the prior session's save (Case→ServiceRequest model, `Core.Partners` mechanism, Phase 1/scale requirement) — not done in this session either; carries forward as outstanding. |
+| `C:\Users\mhammontree\.claude\plans\true-80519-file-mapping.md` | Unchanged this session | Same — still pending the no-Ticket-model revision noted previously. |
+| `C:\Users\mhammontree\.claude\plans\2.0-generic-sftp-netsuite-integration-pattern.md` | Unchanged this session | Same — still pending the `Core.Partners`/`PartnerApiIdentities` correction noted previously (still references the earlier speculative "Integrations table" idea). |
+| No code files | Unchanged | Still pure planning/design; no implementation started yet. Branches are cut but empty of new work as of this session. |
+
+## Decisions made
+
+- **Start Phase 1 implementation with `Client_Nycdoe`** — rationale: the only current client
+  receiving Lenovo ASN files, with abundant real production examples already reviewed this session
+  to test against. No alternative client seriously considered for the starting point.
+- **Branch `dbchanges2`, `_underscore`, `worker2`, and 1.0 `worker`; do NOT branch `api2`** —
+  rationale: traced directly against the plan doc's actual scope rather than assumed; `api2`
+  excluded because adding custom fields to an already-existing `File` Model is a data change, not
+  an API code change. Rejected: branching `api2` "just in case" (Mark's original instinct/question)
+  — deferred until a real need surfaces.
+- **`SftpCredentials` stays `Client_<Name>`-scoped** (not `Core`-scoped) — rationale: confirmed
+  every client has genuinely separate SFTP servers/credentials per manufacturer, no shared-server
+  topology to design around. This was the single open question flagged as most likely to reshape
+  the Phase 1 doc, and it resolved in favor of the design as originally written.
+- **`Logs_Nycdoe` is a required local database, not optional** — rationale: the design's core
+  reason for writing through the API (vs. direct model writes) is to get logging for free; that
+  can't be verified locally without the Logs database registered and present.
+
+## Blockers
+
+None.
+
+## Exact next step
+
+> Verify `phpseclib`'s SFTP client supports a server-side `rename()` for the 1.0 bridge script
+> (carried over, still unconfirmed), then run the `local-db-refresh-from-beta.md` workflow to
+> confirm all four local databases (`Core_2`, `Client_Nycdoe`, `Logs_Nycdoe`, `Archive_Nycdoe`) are
+> present and current before writing the first `dbchanges2` migration for `_modules/sftp-import/`.
+
+---
+_Saved by /session-save on 2026-08-28_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.686",
+  "version": "1.0.688",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",