toga-ai 1.0.1165 → 1.0.1167

package/knowledge/1.0/apps/api/INDEX.md

@@ -4,4 +4,4 @@
 |-----|---------|
 | [API 1.0 (api) Architecture](architecture.md) | The 1.0 REST `api` (clients, suppliers, Talos Voice): single front controller `api.php`, route classes in `_/app/route/`, no sessions, error handling from libra |
 | [Talos Ticket API (1.0 api POST/PUT /ticket) — Desk tickets + file uploads from Talos Voice](features/talos-ticket-endpoint.md) | 1.0 `api` route that Talos Voice calls to create (`POST /ticket`) and update (`PUT /ticket/{ticketNumber}`) TOGa Desk tickets and attach call files; open before |
-| [Deploying API 1.0 to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — new prod env + pipeline, s3fs 1.97 mount over IMDSv2, cross-account `asifiles`, php-fpm ini ru |
+| [Deploying API 1.0 to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — prod env + pipeline, the nonprod api sandbox (= beta), s3fs 1.97 mount over IMDSv2, cross-acco |

package/knowledge/1.0/apps/api/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -6,9 +6,11 @@ project: API 1.0
 client: shared
 type: workflow
 status: active
-updated: 2026-10-22
-owners: ["jcardinal"]
+updated: 2026-10-27
+owners: ["jcardinal", "rgirish"]
 files:
+  - api/api.php
+  - api/config.beta.ini
   - api/.ebextensions/006_mount-s3fs.config
   - api/.ebextensions/009_setup_phpini.config
   - api/.ebextensions/020_setup_git_libraries.config
@@ -25,9 +27,10 @@ related:
   - ../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md
   - ../../togadesk/features/s3fs-asifiles-file-storage.md
   - ../../library/features/php-8-5-runtime-compatibility.md
+  - ../../test/workflows/legacy-1-0-web-apps-in-toga-prod.md
 ---
 
-How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — new prod env + pipeline, s3fs 1.97 mount over IMDSv2, cross-account `asifiles`, php-fpm ini rules, cutover/rollback; open before touching `.ebextensions`/`.platform` or deploying api.
+How the 1.0 `api` deploys on EB PHP 8.5 / AL2023 (Apache proxy) — prod env + pipeline, the nonprod api sandbox (= beta), s3fs 1.97 mount over IMDSv2, cross-account `asifiles`, php-fpm ini rules, cutover/rollback; open before touching `.ebextensions`/`.platform` or deploying api.
 
 ## Summary
 
@@ -51,6 +54,16 @@ TRUE-83054 moved `api` off EB "PHP 7.2 / 64bit Amazon Linux 2.8.9" (AL1, Apache
 11. Load balancer (new env had only HTTP:80 and an LB security group open on 80): add an HTTPS:443 listener, policy `ELBSecurityPolicy-TLS13-1-2-2021-06`, ACM certs `*.agilantsolutions.com` (default) + `*.togatechnology.com` (SNI) in us-east-1 / 654654170868; open 443 on the LB security group; ALB `idle_timeout` 900. Plain HTTP gets a 302 to https from `enforce_https.conf` (ALB sends `X-Forwarded-Proto: http`).
 12. Cutover: old and new env are in different accounts and regions, so EB "Swap environment URLs" is impossible. Before DNS, test with `curl --resolve <host>:443:<ALB IP> https://<host>/health/`. Then Route 53 alias UPSERT to the new ALB (alias zone `Z35SXDOTRQ7X7K`): `api.agilantsolutions.com` (zone in legacy account) and `api.togatechnology.com` (zone in new account). Rollback = alias both back to `agilant-api-prod.us-west-2.elasticbeanstalk.com` (zone `Z38NKT9BP95V3O`).
 
+### Non-prod: api sandbox (= beta, TRUE-83310)
+
+Rebuild, not a move: old `agilant-api-beta` (legacy account) was terminated under TRUE-82983; only saved template `pre-terminate-TRUE-82983-agilant-api-beta` remains (PHP 7.2 — do not reuse; it holds a plaintext AWS key env property).
+
+- Nonprod account 975050298201, us-east-1. EB app `Legacy API` (EB app `API` in nonprod is api2), env `togatech-legacy-api-sandbox`: 1 x `t4g.small` arm64, platform 4.13.13, private subnets of `vpc-0c5319e7df6f736c5`, instance profile `aws-elasticbeanstalk-ec2-role`. Env properties `ENVIRONMENT=beta`, `GITHUB_TOKEN` (same as prod env; needed by `020`).
+- Host `https://api.sandbox.togatechnology.com`: shared ALB `true-alb-sandbox-client` (listener 80 disabled, 443), nonprod ACM cert by SNI, manual host-header rule priority 11 → env target group. Process matcher `200,301,302,401,403,404` on `/`. DNS alias + validation CNAME in `togatechnology.com` zone (toga-prod).
+- `ENVIRONMENT=beta` → `config.beta.ini` → legacy `beta1-cluster` (us-west-2, 502614707982) via transit gateway — chosen on purpose, not the sandbox/stage cluster.
+- Pipeline `Legacy-API-Sandbox`: connection "Test Environments", `agilantsolutions/api` `_beta`, push trigger → Deploy. **No Approval stage — every push to api `_beta` deploys.**
+- Smoke test: `/health/` 200; bad key → 401 "Authentication failed"; a signed request reaching a route (400 validation) proves DB auth on beta1. `/` 403 and `/accounts` 405 are normal (prod too). Sign test requests on the instance so no secret leaves AWS.
+
 **Cross-account `asifiles`** (bucket in us-west-2, legacy account 502614707982): bucket policy allows role `arn:aws:iam::654654170868:role/ElasticBeanstalk-EC2-Instance-Profile` ListBucket / GetBucketLocation + Get/Put/Delete/PutObjectAcl; that shared role has inline policy `asifiles-readwrite`. Bucket has no ownership controls, so `public-read` ACLs work.
 
 ## Gotchas
@@ -66,9 +79,13 @@ TRUE-83054 moved `api` off EB "PHP 7.2 / 64bit Amazon Linux 2.8.9" (AL1, Apache
 - **Same `container_commands` key in two configs = one silently wins.** `035`'s graceful-restart cron was already dead this way (overwritten by `050`'s duplicate key).
 - **`iam simulate-principal-policy` with several resources reports for the first one only.** Simulate object actions against an object ARN.
 - **Disable the old account's pipeline before merging api into `_production`.** It still deploys `_production` to the AL1 rollback env and would push AL2023 config there, breaking the rollback.
+- **`020` hardcodes `BRANCH="_production"`** for library + dbchanges (toga uses `_<ENVIRONMENT>`). So the sandbox runs library `_production`, not `_beta`. `030` dbchanges on beta is a silent no-op — see [dbchanges](../../dbchanges/workflows/authoring-and-shipping-sql-files.md) gotchas.
+- **api `_beta` drifts behind `_production`.** On 2026-10-27 it was 73 commits behind (no AL2023 files); merged `_production` into `_beta` (`a88c819`). Check before a sandbox deploy.
+- **New SNI cert is not served for a few seconds.** curl exit 60 right after `add-listener-certificates`; fine ~20s later. Do not treat it as a bad cert.
 - Prefork tuning (old `001`) does nothing under event MPM + php-fpm. Removed: `010` redis (api has `useSessions=false`; `$_SESSION` is per-request only), `035` cron, `040` tests, `codecept.phar`, `tests/`, `ebs/git.json`, old `setup_git_libraries.php` path (needed a `CodePipelineEnvironments` row).
 - **Security, open (values not recorded):** `config.prod.ini` in the repo holds DB / SMTP / cert passwords (now blocked from the web by the deny rule; still need rotating); an old GitHub token is in git history (`ebs/git.json`, also tools `020`); `test.php` has a hardcoded client token. Developer will rotate.
 
 ## Change history
+- 2026-10-27 — api sandbox env (= beta) in nonprod, `_beta` drift, `020` hardcoded `_production`, SNI delay (TRUE-83310) (rgirish)
 - 2026-10-22 — Live deploy: `/etc/php.d` direct write, `/health/` health check, HTTPS:443 listener, Route 53 cutover + rollback, GET-body / `Logs.API` debugging (TRUE-83054) (jcardinal)
 - 2026-10-22 — Created: AL1 → PHP 8.5 / AL2023 with Apache proxy, new prod account env + pipeline, s3fs 1.97 over IMDSv2, cross-account `asifiles` (TRUE-83054) (jcardinal)

package/knowledge/1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md

@@ -6,10 +6,11 @@ project: Database Changes
 client: shared
 type: workflow
 status: active
-updated: 2026-10-22
+updated: 2026-10-27
 owners: [sking, jcardinal, rgirish]
 files:
   - dbchanges/index.php
+  - dbchanges/config.beta.ini
   - dbchanges/Core/
   - dbchanges/TOGaDeskSupport/
   - worker/crons/infrastructure/execute_dbchanges.php
@@ -109,7 +110,7 @@ Consequence: **a folder whose database is not reachable from that host is skippe
 | Elastic Beanstalk container command (`worker/.ebextensions/030_dbchanges.config`) | worker **deploy** | on next deploy |
 
 The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, therefore:
-- **Auto-apply within ~2 min of a push:** `_alpha`, `_beta`, `_demo`, `_hotfix`, `_stage`.
+- **Auto-apply within ~2 min of a push:** `_alpha`, `_beta`, `_demo`, `_hotfix`, `_stage` — but `_beta` reaches nothing (see Gotchas: `beta-cluster` is gone).
 - **`_production` does NOT auto-apply** — it would need `config.production.ini`, which does not exist (the shipped file is `config.prod.ini`). Production applies via the EB deploy container command instead, i.e. **on the next worker deploy**. Plan comms and expectations accordingly.
 
 ## Gotchas
@@ -119,6 +120,7 @@ The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, there
 - **No `DELIMITER`, no stored routines, no explicit `COMMIT`.** See the guard-pattern section.
 - **Never write `*/` inside a `/* */` comment** — e.g. a cron `*/5` ends the comment early → syntax error → the file is stuck (Core `SK/2026-07-30`). Use `--` comments.
 - **Find stuck files:** per database folder, compare the repo's `.sql` files with `<DB>._dbchanges.versionFileName`. Found on the PHP 8.5 move (2026-10-22, TRUE-83055) and marked applied by hand: Common `MH/2024-04-25` (Battery Issue duplicate), `MH/2024-06-06`, `MH/2025-04-02`, `MH/2026-09-16`, `MH/2026-09-30`; Core `SK/2026-07-30`, `SK/2026-08-04`, `SK/2026-08-06`, `SK/2026-08-13`. **Two were NOT really applied:** Common `MH/2024-06-06` (EmailAddresses UNIQUE — 3 duplicate pairs block it) and Core `SK/2026-08-13` (asset-tag backfill). Marking a file applied is not applying it — re-check the data.
+- **⚠ Beta dbchanges is a silent no-op.** `config.beta.ini` points at `beta-cluster`, which no longer exists (live beta DB is `beta1-cluster`). Every `ENVIRONMENT=beta` run logs "Skipping <db> Database... unable to connect" for every folder and changes nothing. So beta1 is ~140 files behind (2026-10-27: Common 6, Core 8, TOGA_* ~9 each, Vision 5, TOGaDeskSupport 4, …); stage1 ~12 behind (mostly hand-applied or prod-only repairs). **Before pointing a deploy's dbchanges at a legacy non-prod cluster, count pending files per DB first** (repo files vs `_dbchanges.versionFileName`). The runner stops on the first failing file (e.g. stage1 Common `MH/2024-04-25` → 1062 Duplicate entry).
 - **Error 1093** is why the preflight subquery must be wrapped in a derived table.
 - **⚠ The repo does NOT describe the whole live schema — never conclude "that column/index does not exist" from `dbchanges` alone.** `Core.AdvanceShippingNoticeQueue.dedupeKey` and `.alertSentAt` exist in production (`dedupeKey` confirmed UNIQUE) but **no migration for them exists anywhere in this repo** — they were created from a `library` commit (TRUE-77676). Found 2026-09-16 because a reviewer read the repo and reached the wrong conclusion. Verify columns and indexes against the **database**.
 - **A merged PR is not an applied change.** Confirmed live 2026-08-13 (PR dbchanges#265): after the merge into `_production`, both target tables were still 100% unchanged. Never tell a customer or a ticket "applied" off a merge — always re-read the data.
@@ -127,6 +129,7 @@ The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, there
 - **⚠ SECURITY — hardcoded GitHub token.** `worker/crons/infrastructure/execute_dbchanges.php` contains a **plaintext GitHub personal access token** in the small config block near the top of the file (~L14–18), and interpolates it into a `shell_exec()` git-clone command line — which also exposes it in the worker host's process listing. Treat the token as **compromised**: rotate it, move it to config/env (`config.<env>.ini`), and pass it without embedding it in an argv string. No credential value is recorded here by design.
 
 ## Change history
+- 2026-10-27 — `config.beta.ini` → dead `beta-cluster`: beta runs skip every DB; count pending files before pointing at a non-prod cluster (TRUE-83279) (rgirish)
 - 2026-10-22 — PHP 8.1+: `mysqli_report(MYSQLI_REPORT_OFF)` keeps PHP 7 behavior; legacy runner exited 0 on error and hid stuck files; `*/` in block comments; stuck-file check (TRUE-83055) (rgirish)
 - 2026-10-22 — TRUE-83064 (dbchanges#270, CodeRabbit review): one-time data-repair rules — one CASE UPDATE with exact-count preflight, chained after-state guards, exact-id forward + pinned rollback, no SQL error on mismatch, no-op still commits `_dbchanges`. (sking)
 - 2026-10-19 — Comment lines ending in `;` also split the file; verify SELECTs in the header without `;` (TRUE-82935) (jcardinal)

package/knowledge/1.0/apps/test/workflows/legacy-1-0-web-apps-in-toga-prod.md

@@ -6,7 +6,7 @@ project: Test
 client: shared
 type: workflow
 status: active
-updated: 2026-10-22
+updated: 2026-10-27
 owners: ["rgirish"]
 files:
   - test/team/aws/create_elastic_beanstalk.php
@@ -14,7 +14,11 @@ files:
   - togadesk/desk/includes/functions.php
   - togaview/_/app/frameworkindex.php
   - toga/_/app/framework.php
+  - toga/.ebextensions/020_setup_git_libraries.config
+  - toga/ebs/mount-s3fs-folders.php
 related:
+  - ../../dbchanges/workflows/authoring-and-shipping-sql-files.md
+  - ../../../../clients/office-depot/profile.md
   - ../features/create-elastic-beanstalk.md
   - retiring-unused-legacy-elastic-beanstalk-environments.md
   - ../../store/workflows/deploy-to-elastic-beanstalk-al2023.md
@@ -22,9 +26,10 @@ related:
   - ../../walmart-activate/workflows/deploy-to-elastic-beanstalk-al2023.md
   - ../../togadesk/features/per-client-host-restriction.md
   - ../../../../clients/prudential/workflows/store-saml-deploy-topology.md
+  - ../../api/workflows/deploy-to-elastic-beanstalk-al2023.md
 ---
 
-Where the 1.0 web apps (store, togadesk, togaview, toga, walmart-activate) run since 2026-10-22 (toga-prod 654654170868, us-east-1), their env/pipeline names and settings, how a domain is cut over with "1" test names, and how each app maps host → client; open before any 1.0 web-app deploy, DNS, cert, or domain change.
+Where the 1.0 web apps (store, togadesk, togaview, toga, walmart-activate) run since 2026-10-22 (toga-prod 654654170868, us-east-1), their env/pipeline names and settings, the TOGa Retail sandbox (nonprod), how a domain is cut over with "1" test names, and how each app maps host → client; open before any 1.0 web-app deploy, DNS, cert, or domain change.
 
 ## Summary
 
@@ -40,6 +45,18 @@ On 2026-10-22 (TRUE-83055) the five legacy 1.0 web apps moved from legacy accoun
 
 Pipelines: source GitHub `agilantsolutions/<repo>` branch `_production` → **Approval** → Deploy. A merge does not deploy until someone approves.
 
+### Non-prod: TOGa Retail sandbox (ODP, TRUE-83279)
+
+ODP calls it "sandbox"; internally it **is beta** (replaces the old shared `agilant-toga-beta`, saved config `pre-terminate-TRUE-82983-agilant-toga-beta`).
+
+- Nonprod account 975050298201, us-east-1. EB app `TOGa Retail`, env `togatech-legacy-togaretail-sandbox`: 1 x `t4g.small` arm64, PHP 8.5 AL2023, private web subnets of `vpc-0c5319e7df6f736c5`, instance profile `aws-elasticbeanstalk-ec2-role`.
+- Host `odp.toga.sandbox.togatechnology.com`: shared ALB `true-alb-sandbox-client` — EB rule on the env CNAME plus a manual host-header rule (priority 9) → EB target group; its ACM cert is on the shared listener by SNI. DNS alias + cert-validation CNAME live in the `togatechnology.com` zone in toga-prod.
+- `ENVIRONMENT=beta` → `toga/config.beta.ini` → legacy `beta1-cluster` (us-west-2, account 502614707982). Deploy clones `_beta` of `library`, `resources`, `dbchanges` (`.ebextensions/020_setup_git_libraries.config`). Nonprod VPC reaches legacy `172.30.0.0/16` through transit gateway `tgw-0d697161b3380ad81`.
+- Pipeline `Legacy-TOGaRetail-Sandbox`: connection "Test Environments", `agilantsolutions/toga` `_beta`, trigger on push → Deploy. **No Approval stage — every push to toga `_beta` deploys.**
+- S3: all non-prod toga envs mount the same legacy `asifiles` + `agilant-catalog` buckets (us-west-2) via s3fs. The nonprod EC2 role needs **both** the bucket-policy statements `NonprodSandboxListBucket` / `NonprodSandboxObjects` and the inline role policy `LegacyS3FsBuckets`.
+- api has the same setup (`togatech-legacy-api-sandbox`, host `api.sandbox.togatechnology.com`) — see [api deploy](../../api/workflows/deploy-to-elastic-beanstalk-al2023.md).
+- ODP SSO in test mode uses ODP's sandbox Azure tenant. SP entityId/ACS are built from the request host, so a new host needs ODP to add `https://<host>/?acs` as a Reply URL.
+
 ## Steps
 
 ### Create an env for a 1.0 web app
@@ -71,7 +88,7 @@ Pipelines: source GitHub `agilantsolutions/<repo>` branch `_production` → **Ap
 | store | `_/app/framework.php` | strips known suffixes from `SERVER_NAME` → `Store.Catalogs.domain LIKE <host>` (`*.togahub.com` → `togahub`); many `$_SESSION['host'] == 'spglobal'`-style branches |
 | togadesk | `desk/includes/functions.php` `getRestrictedClient()` | exact host switch — see [per-client host restriction](../../togadesk/features/per-client-host-restriction.md) |
 | togaview | `_/app/frameworkindex.php` `getRestrictedClient()` | same idea; SAML client chosen by **substring** of the host |
-| toga | `_/app/framework.php` | subdomain = host labels minus the last two, `.` → `-`, matched to `Clients.subdomain`; plus a `SERVER_NAME == 'chat.togahub.com'` check |
+| toga | `_/app/framework.php` (~L101) | subdomain = host labels minus the last two, `.` → `-`, env suffixes stripped (`-alpha`, `-beta`, `-stage`, `-sandbox`, …), matched to `Clients.subdomain` (`odp.toga.sandbox.togatechnology.com` → `odp-toga`); plus a `SERVER_NAME == 'chat.togahub.com'` check. A new env suffix not in the strip list = "Invalid configuration" |
 | walmart-activate | — | no host logic |
 
 SAML ACS URLs are hard-coded (store: `prudential.togahub.com/?acs`, then redirect to `https://prudential.togacommerce.com`). A new host name needs an IdP change, not only DNS.
@@ -81,7 +98,11 @@ SAML ACS URLs are hard-coded (store: `prudential.togahub.com/?acs`, then redirec
 - **ACM validation in toga-prod can take 30–60+ minutes** even with correct CNAMEs. Attach already-issued wildcard certs first; add the slow cert by SNI later.
 - **`ENVIRONMENT` must be `prod`, not `production`** (see step 2). Wrong value = no config, every request fails.
 - **The Approval stage is the only brake on prod deploys.** The legacy `Worker` pipeline (still live, account 502) also fires on `library` pushes; it got an Approval stage on 2026-10-22 so a library merge no longer auto-deploys the worker. Every app clones library `_production` at deploy, so a library merge ships with the next approved deploy of any app.
+- **S3 mount fails the non-prod deploy.** Missing either S3 grant (sandbox section) → `mount-s3fs-folders.php` exits 1 → deploy fails at `02_mount_s3fs`.
+- **`_beta` branches drift behind `_production`.** On 2026-10-27 they were missing the AL2023/PHP 8.5 deploy files (toga 13, library 50, dbchanges 123, resources 1 commits behind); `_production` was merged into `_beta` in all four. Check before a beta deploy. Beta dbchanges does not apply — see [dbchanges](../../dbchanges/workflows/authoring-and-shipping-sql-files.md) gotchas.
+- **Unused `_sandbox` branches** exist in toga, library, dbchanges (abandoned for `_beta`). Do not deploy from them.
 - **Old env names in other docs are history.** `prudential-store-prod`, `true-togacommerce-prod`, `spglobal-store-prod`, `*-togadesk-prod` and pipeline `Store-Production` no longer exist.
 
 ## Change history
+- 2026-10-27 — TOGa Retail sandbox for ODP (= beta internally), `-sandbox` host strip, nonprod S3 grants, `_beta` drift (TRUE-83279) (rgirish)
 - 2026-10-22 — Created: 5 legacy 1.0 web apps moved to toga-prod us-east-1, naming, env settings, "1"-host cutover, host → client lookup (TRUE-83055) (rgirish)

package/knowledge/1.0/apps/worker1.5/INDEX.md

@@ -4,5 +4,6 @@
 |-----|---------|
 | [Worker 1.5 (1.0 Framework) Architecture](architecture.md) | The single-server 1.0 cron tier that replaced worker's 7-role fleet; open for how the crontab is built from schedules/*.json, the overlap check, and what the cr |
 | [worker1.5 — Committed Secrets & Credentials in Logs (debt register)](features/committed-secrets-and-log-exposure.md) | Open security debt in worker1.5: a GitHub token committed in `ebs/git.json` and a cron that writes OptimumDesk passwords into the cron log; open before touching |
+| [worker1.5 cron overlap guard (isProcessRunning, crontab shell, manual runs)](features/cron-overlap-guard.md) | How the 1.0 "already running" check decides a cron exits, why the crontab must use /bin/sh, and how to run a cron by hand on worker15 without it exiting; open w |
 | [worker1.5 PHP ini overrides on EB AL2023 (memory_limit 11G, zz-application.ini, cron log_errors=0)](features/php-ini-memory-limit-eb.md) | How worker1.5 sets PHP ini values (memory_limit 11G, cron `log_errors=0`) on EB `worker15-production`; open before changing any PHP ini setting or the cron comm |
 | [Diagnosing worker1.5 crons that stopped running (single-server 1.0 tier)](workflows/diagnosing-crons-not-running.md) | Steps to tell "worker1.5 crons are not running" from "a downstream system is failing", find the cause on the box, and recover, plus how to triage failing crons |

package/knowledge/1.0/apps/worker1.5/features/cron-overlap-guard.md

@@ -0,0 +1,46 @@
+---
+title: worker1.5 cron overlap guard (isProcessRunning, crontab shell, manual runs)
+framework: "1.0"
+repo: worker1.5
+project: Worker 1.5
+client: shared
+type: feature
+status: active
+updated: 2026-10-23
+owners: ["rgirish"]
+files:
+  - worker1.5/ebs/cron.worker.php
+  - library/app/framework.php
+related: ["../architecture.md", "../../worker/architecture.md", "../../../../clients/nycdoe/features/servicenow-integration.md"]
+---
+
+How the 1.0 "already running" check decides a cron exits, why the crontab must use /bin/sh, and how to run a cron by hand on worker15 without it exiting; open when a job is missing from CronJobExecutions or a manual run does nothing.
+
+## Summary
+
+`App_Framework::cronInitialization($time, $readCluster, $exitIfProcessAlreadyRunning = true)` (`library/app/framework.php` ~L689) calls `exitIfProcessRunning()` → `isProcessRunning()` (~L833). It exits **before** the `Common.CronJobExecutions` row is written, silently (no log, exit 0). Jobs passing `false` as the 3rd arg skip the check.
+
+## How it works
+
+- `isProcessRunning()` runs `ps -ef` and returns true when a row contains `$_SERVER['SCRIPT_FILENAME']` as a **substring**, is not its own pid, and does **not** contain `/bin/sh`.
+- The `/bin/sh` skip exists to ignore cron's own wrapper (`/bin/sh -c php /var/www/html/crons/...`). Any other shell's wrapper matches → the job sees "itself" and exits every run.
+- `ebs/cron.worker.php` writes root's crontab with **no `SHELL=` line** (cron default `/bin/sh`); a code comment says why. Never add `SHELL=/bin/bash`.
+- Match is on the **exact path string**, so the same script started by another path is not seen:
+  - cron runs `/var/www/html/crons/<path>` (`/var/www/html` → `/var/app/current`).
+  - manual `php /var/www/html/crons/<path>` → exits at once while a cron run is active.
+  - manual `php ./<script>.php` → exits only if another `./` run is active.
+  - manual `php /var/app/current/crons/<path>` → never blocked by the cron run.
+
+### Manual run on worker15 (SSM, as root)
+
+1. Pick a path the cron is not using (`/var/app/current/crons/...` or `cd` + `./`) for a long `*/5` job that is almost always running.
+2. Run one at a time; parallel runs on the same path exit with no output.
+3. Bypassing the guard means the cron and your run can work the same record — verify the result afterwards (each job's doc names its check).
+
+## Gotchas
+- **Fingerprint of a broken guard:** a job is **missing entirely** from `Common.CronJobExecutions` (no row at all, not a failed row) while other jobs on the same box log fine → check the job's `cronInitialization` 3rd arg and the crontab `SHELL`.
+- **2026-10-22 evening → 2026-10-23 10:35 CDT outage (TRUE-83121):** crontab had `SHELL=/bin/bash`; ~200 of 232 legacy crons never ran; only the 16 jobs with `exitIfProcessAlreadyRunning=false` worked. Fixed by removing the `SHELL` line (worker1.5 `ce0f0b2` / `ecd1bc2`). worker15-production's CodePipeline (Source `_main` → Deploy) has **no approval stage** — a merge to `_main` goes live.
+- **Open, not traced (seen 2026-10-23 on worker15):** PHP 8.5 deprecation "Using null as an array offset is deprecated" at `library/app/database.php:591` (does not stop the run); `worker-cron.log` "Allowed memory size of 268435456 bytes exhausted ... _underscore/Query.php line 430" from an unknown job. Each needs its own ticket.
+
+## Change history
+- 2026-10-23 — TRUE-83126/TRUE-83121: crontab back to /bin/sh after bash broke the overlap guard (~19 h, ~200 crons); guard path-matching and manual-run rules (rgirish)

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -33,7 +33,7 @@
 | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. |
 | [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide 2.0 error reporting on an Issue/Event aggregation model in the shared Core Logs DB (capture, fingerprinting, references); open before touching err |
-| [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` publishes a record-changed JSON message to the `toga-events` SQS queue after a V2 write; open when wiring or debugging the real-time event f |
+| [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` sends a record-changed JSON message to the event SQS queue after a V2 write (role-based creds, no keys); open when wiring or debugging the r |
 | [FIELD_STORAGE fields — per-row lazy hydration and the platform-wide missing-column 500](features/field-storage-row-hydration.md) | `FIELD_STORAGE` columns are hydrated lazily per returned row, so a missing column 500s only when a query returns rows; open when auditing storage-field schema d |
 | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time webhook importer that writes NetSuite sale records (invoice/cashSale/creditMemo/cashRefund/journalEntry) into `Forecast.Sales`; open before touching s |
 | [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | api2 interceptor that propagates `Items.isFulfillable` UP the SO↔PO chain onto client-facing item rows; open when the storefront Qty-Fulfilled gate is wrong or |

package/knowledge/2.0/apps/_underscore/features/config-group-access.md

@@ -6,10 +6,11 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["mhammontree", "bala"]
+updated: 2026-10-26
+owners: ["mhammontree", "bala", "rgirish"]
 files:
   - _underscore/Config.php
+  - _underscore/Event.php
   - _underscore/Component/Api/Paypal/Paypal.php
   - _underscore/Worker.php
   - api2/Config/beta.ini
@@ -126,6 +127,9 @@ the reasoning generalises:
 
 - **`_Config::<group>('prop', false)` is NOT a null-safe read.** It is property-optional only.
   Group-optional requires `_Config::isGroupSet('<group>')`.
+- **`_Config::<group>('key') ?: null` is NOT an optional read** — the one-argument form throws on a
+  missing key before `?: null` runs. Optional key → `_Config::<group>('key', false) ?: null`. Live
+  case: [`_Event::publish`](./event-publish-sqs.md) sent 0 prod events for 3 months (TRUE-83223).
 - **A missing group is an uncaught throw on the user's request**, not a silent null — so the
   blast radius is a visible 500/`EO-1`, not a quiet misbehavior. Guard before you read.
 - **Never guard by catching the exception broadly.** Wrap the `isGroupSet()` check in a
@@ -134,9 +138,13 @@ the reasoning generalises:
 - **Do not record credential values anywhere.** Document only which ini and which group a
   credential lives in (e.g. `api2/Config/beta.ini` `[paypal]`).
 
+## Change history
+- 2026-10-26 — Gotcha: `_Config::<group>('key') ?: null` throws before `?: null`; optional keys need the `false` flag (`_Event::publish`, TRUE-83223) (rgirish)
+
 ## Related
 
 - [CLI script bootstrap](./cli-script-bootstrap.md)
+- [Event publishing to SQS](./event-publish-sqs.md)
 - [Model magic field access](./model-magic-field-access.md)
 - [ENVIRONMENT drives the branch and Config file](../../api2/features/environment-variable-drives-underscore-branch.md)
 - [Rate subscription cancellation](../../../../clients/rate/features/subscription-cancellation.md)

package/knowledge/2.0/apps/_underscore/features/event-publish-sqs.md

@@ -6,64 +6,75 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-10-12
+updated: 2026-10-26
 owners: [rgirish, jcardinal]
 files:
   - _underscore/Event.php
+  - _underscore/Cloud.php
+  - api2/Config/production.ini
+  - api2/Config/beta.ini
 related:
   - ../../../../standalone/apps/websocket/features/sqs-fanout.md
+  - ../../../../standalone/apps/websocket/architecture.md
+  - ./cloud-s3-helpers.md
+  - ./config-group-access.md
+  - ../../api2/features/request-logging.md
 ---
 
-`_Event::publish()` publishes a record-changed JSON message to the `toga-events` SQS queue after a V2 write; open when wiring or debugging the real-time event fan-out.
+`_Event::publish()` sends a record-changed JSON message to the event SQS queue after a V2 write (role-based creds, no keys); open when wiring or debugging the real-time event fan-out.
 
 ## Summary
 
-`_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event
-pipeline. After a successful V2 write (POST/PUT/PATCH), it publishes a single JSON message
-to the `toga-events` SQS queue describing what changed. The `websocket` standalone server
-consumes that queue and fans out `record:changed` to browsers. The whole call is wrapped in
-a `try/catch (\Throwable)` so an SQS failure logs but never blocks or errors the API
-response.
+`_Event::publish()` (`_underscore/Event.php`) is the PHP side of the real-time event pipeline.
+After a successful V2 write (POST/PUT/PATCH) it sends one JSON message to the env's event queue
+(prod: `toga-events` in 654654170868; non-prod: `WebsocketEvents` in 975050298201). The
+`websocket` server consumes it and fans out `record:changed`. The whole call sits in
+`try/catch (\Throwable)` → a failure only `error_log`s; it never blocks or errors the response.
 
 ## How it works
 
 `publish(object &$api, string $httpMethod, array $routeVars)`:
 
-1. Reads `aws_event_queue_url` / `aws_event_queue_region` from `_Config::cloud()` (no
-   default — see Gotchas).
-2. Determines the affected record's `recordRoute`, `recordId`, `uuid`.
-3. Derives `parentRoute` / `parentUuid` from `$routeVars` when the URL is a nested resource
-   (`count($routeVars) >= 3` → `parentRoute = $routeVars[0]`, `parentUuid = $routeVars[1]`).
-4. JSON-encodes `{ clientId, clientUuid, recordRoute, recordId, uuid, httpMethod,
-   parentRoute, parentUuid, dtEvent }` and enqueues via `_Cloud::addSqsMessage()`.
+1. Reads `[cloud] aws_event_queue_url` / `aws_event_queue_region` (required reads — throw if
+   missing; see Gotchas). Empty URL → return.
+2. `recordRoute` = `$routeVars[0]` (fallback `$api->record->route`); `recordId` / `uuid` from the
+   first key of `$api->response->data`.
+3. Nested URL (`count($routeVars) >= 3`) → `parentRoute = $routeVars[0]`, `parentUuid = $routeVars[1]`.
+4. JSON `{ clientId, clientUuid, recordRoute, recordId, uuid, httpMethod, parentRoute, parentUuid, dtEvent }`
+   → `_Cloud::addSqsMessage()`.
 
-SQS credentials come from `_Config::cloud('aws_event_queue_access_key_id' /
-'..._secret_access_key')` — a static key, only in `api2/Config/beta.ini` (open: replace with
-the `WorkloadsRuntime` provider, see [_Cloud credentials](./cloud-s3-helpers.md#credentials)).
-Never put the values in a doc.
+**Credentials.** Optional keys `aws_event_queue_access_key_id` / `..._secret_access_key` are read
+with `_Config::cloud('<key>', false) ?: null`. No keys (prod, by design) → `_Cloud::addSqsMessage`
+assumes `WorkloadsRuntime` in the account named in the queue URL — see
+[_Cloud credentials](./cloud-s3-helpers.md). Static event-queue keys still sit in
+`api2/Config/beta.ini`: the code no longer needs them; remove them and rotate the key (never
+record values).
 
 ## Gotchas
 
-- **⚠ Event publishing to SQS silently fails in every env except beta.** `aws_event_queue_*`
-  is read with no default, and only `beta.ini` sets it, so `_Config::cloud()` throws; the
-  `try/catch (\Throwable)` logs and moves on. Known (2026-10-12, TRUE-82705); developer chose to
-  leave it. Before relying on `record:changed` outside beta, add the keys or a default.
-
+- **Optional config keys need the `false` flag.** `_Config::cloud('key') ?: null` throws on a
+  missing key before `?: null` runs. That bug sent 0 events to prod `toga-events` from its creation
+  (2026-07-28) to TRUE-83223 (2026-10-26). Rule lives in [_Config group access](./config-group-access.md).
+- **Failures are invisible by design.** The catch only `error_log`s `[_Event::publish] SQS publish
+  failed: ...`. On prod that lands in `/var/log/php-fpm/www-error.log` — not in the EB log bundle,
+  not in CloudWatch (see [request logging](../../api2/features/request-logging.md)). Check there
+  first when `record:changed` never arrives.
+- **Open (not fixed, 2026-10-26) — remaining prod failures:**
+  - `Attempt to read property "id" on null` (77 on one instance) — likely `$api->client` is null
+    (request with no client context); `clientId`/`clientUuid` read it unguarded.
+  - `Variable 'data' cannot be retrieved because it has not been set.` (15).
+  - `aws_event_queue_url` / `_region` are required reads — an env without them throws on every write.
 - **`recordRoute` must come from `$routeVars[0]`, not `$api->record->route`.** Post-`POST`
-  interceptors can swap `$api->record` to a *different* model before `publish()` runs (e.g.
-  `service-requests`' `postPost` switches `$api->record` to the `service-request-bundles`
-  model), so `$api->record->route` would point at the wrong route. Read the original URL
-  segment `$routeVars[0]`, falling back to `$api->record->route` only when `routeVars` is
-  empty.
-- **`$api->response->data` is a PHP array, not a stdClass object.** `processRoutePairs()`
-  in V2.php returns e.g. `['serviceRequests' => $outData]`. Using object access
-  (`$data->{$topKey}`) on it silently returns null, so `uuid`/`recordId` were always null in
-  published events. Cast first: `$data = (array) $api->response->data;` then
-  `$topRecord = (object) $data[array_key_first($data)];` before reading `->id` / `->uuid`.
+  interceptors can swap `$api->record` to another model (e.g. `service-requests` `postPost` →
+  `service-request-bundles`).
+- **`$api->response->data` is a PHP array, not stdClass.** Cast: `$data = (array) $api->response->data;`
+  then `(object) $data[array_key_first($data)]` before reading `->id` / `->uuid`.
 
 ## Change history
+- 2026-10-26 — Fixed prod publishing: optional queue keys read with `false` flag, prod now uses the WorkloadsRuntime role path (verified MessageId + websocket-production fanout); recorded remaining prod failures and the php-fpm log location (TRUE-83223) (rgirish)
 - 2026-10-12 — Recorded that publishing throws (caught) outside beta: `aws_event_queue_*` has no default; static queue key still open (TRUE-82705) (jcardinal)
 
 ## Related
 
 - [websocket SQS fan-out](../../../../standalone/apps/websocket/features/sqs-fanout.md)
+- [websocket architecture](../../../../standalone/apps/websocket/architecture.md)

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -230,11 +230,13 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **Never copy logged rows into committed artifacts** — they carry live Bearer tokens and PII.
 - **Timing: `Logs_*.Api.dtStamp` is the request START time; `executionTimeSeconds` misses part of the cost.** Measured 2026-10-16 (Prudential, TRUE-82596): every GET ~1.1s `executionTimeSeconds` (even `/announcements` with 3 rows), plus ~1.5–1.9s more between back-to-back calls outside that timer; PUT/POST ~0.2s. Likely api2 bootstrap and/or the synchronous Logs write — open follow-up. To measure a caller's real wait, diff consecutive `dtStamp`s, not `executionTimeSeconds`.
 - **⚠ A row under a user's `userId` is not proof that user's browser sent it.** View-as / delegator sessions (`POST /v2/auth/delegator`) and staff password logins as the user all log under the TARGET user's `userId`. To trace the user's own browser, filter on their `X-Forwarded-For` value inside `requestHeaders`, inside a `dtStamp` range (a week-wide `LIKE` timed out at 300 s), and check User-Agent before concluding "they logged in". `dtStamp` is server-local time (-05:00).
+- **Prod PHP `error_log()` output goes to `/var/log/php-fpm/www-error.log` on each instance** — incl. errors that a `catch` logs and swallows (e.g. `_Event::publish`). It is NOT in the EB log bundle and NOT in CloudWatch; read it through SSM on each instance (each one has its own file).
 - **A browser "Network Error" leaves no row at all** — the request never reached api2. See [commerce Sentry signals](../../toga2-commerce/features/error-codes-and-sentry-signals.md).
 - **`Logs_<Client>.Api` is pruned (once TRUE-81902 deploys)** — GET rows after 30 days, writes after 90. A missing older row is not proof the request never happened. See [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md).
 
 ## Change history
 - 2026-10-27 — Corrected: failed client-scoped writes (non-2xx) are logged in `Logs_<Client>.Api`, not core `Logs` (verified on Prudential 404 `EV-14` PUTs, TRUE-83276). (jcardinal)
+- 2026-10-26 — Gotcha: prod `error_log()` output is only in `/var/log/php-fpm/www-error.log` (not EB bundle / CloudWatch); read via SSM (TRUE-83223). (rgirish)
 - 2026-10-23 — `Logs_Prudential.Api` `(method, dtStamp)` index; `ORDER BY id DESC` + payload `LIKE` = full backward PK scan; REGEXP error 3699 (TRUE-83157). (bala)
 - 2026-10-23 — A multi-MB response OOMs in the log INSERT (`Query.php` L430) and leaves no Api row or Issue (TRUE-83138). (bala)
 - 2026-10-20 — Gotcha: local `Logs.Api.sourceIp` NOT NULL (blank Core Logs template) vs nullable prod (TRUE-82934). (jcardinal)

package/knowledge/INDEX.md

@@ -7,7 +7,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **library** (Library) _(framework core)_ — 41 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
-- **worker1.5** (Worker 1.5) — 7 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
+- **worker1.5** (Worker 1.5) — 8 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 19 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 9 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -7,8 +7,12 @@ client: nycdoe
 type: client-feature
 status: active
 updated: 2026-10-27
-owners: [mhammontree, sking, bala, jcardinal]
+owners: [mhammontree, sking, bala, jcardinal, rgirish]
 files:
+  - worker1.5/crons/sync/nycdoe/3_create_installation_ticket.php
+  - worker1.5/schedules/cron.worker.json
+  - worker1.5/ebs/cron.worker.php
+  - library/app/framework.php
   - worker/crons/sync/nycdoe/import_asn.php
   - worker/crons/sync/nycdoe/import_inc.php
   - worker/crons/sync/nycdoe/legacy_import_asn.php
@@ -50,6 +54,7 @@ related:
   - hold-status-sync.md
   - asn-sftp-ingestion-2-0.md
   - ../../../1.0/apps/worker/architecture.md
+  - ../../../1.0/apps/worker1.5/architecture.md
   - ../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md
 ---
 
@@ -61,7 +66,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
 
 ## How it works
 
-> **Schedule files are the source of truth for what runs** (`worker/schedules/cron.worker.sync.json`, `cron.worker.notification.json`). Header comments inside the scripts are frequently stale about cadence. A script on disk but absent from the schedule never runs.
+> **Schedule files are the source of truth for what runs.** Since TRUE-83078 (2026-10-23) the live 1.0 crons run from `worker1.5/schedules/cron.worker.json` on EB `worker15-production` — see [Worker 1.5 architecture](../../../1.0/apps/worker1.5/architecture.md). The old `worker/schedules/cron.worker.sync.json` / `cron.worker.notification.json` rows are all `active: 0`. Header comments inside the scripts are frequently stale about cadence. A script on disk but absent from the schedule never runs.
 
 ### Core local tables (`db_common` unless noted)
 | Table | Purpose |
@@ -83,7 +88,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
 | Queue → ASN records (Stage 2) | `legacy_process_asn_queue.php` | every 15 min |
 | NetSuite SO create (Stage 3, **freeze**) | `1_send_asn_to_netsuite.php` | every 2 h at :07 |
 | NetSuite PO/serials (Stage 4) | `2_send_serials_to_netsuite.php` | every 2 h at :27 |
-| TOGa Desk repair orders (Stage 5) | `3_create_installation_ticket.php` | every 5 min |
+| TOGa Desk repair orders (Stage 5; schedule name "DOE/NS Integration - Step 3") | `3_create_installation_ticket.php` | `*/5`, but each run lasts ~590 s (own `$stopTime`) + overlap guard → effectively every ~10 min |
 | NetSuite Item Fulfillment (Stage 6) | `4_create_ns_item_fulfillment.php` (**paused**, `active: 0`); `5_DOA_…` unscheduled (moving to `crons/sync/DOA/`, see below) | not running |
 | ETA → ServiceNow | `send_ticket_updates.php` | every 6 min |
 | RITM state → ServiceNow | `send_request_item_updates.php` | every 5 min |
@@ -127,6 +132,8 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘      [UNIQUE ded
 - **Stage 5:** for units `WHERE togadeskRepairOrderId IS NULL`, reads serials from NetSuite item receipts (so TOGa Desk RO serials are **driven by NetSuite**, not the ASN unit), creates/reuses TOGa Desk manufacturers/models/assets/serial-numbers/MSO and creates **a new repair_orders row unconditionally per loop iteration** — one RO per Unit (a 50-cable line = 1 unit = 1 RO). Existing-serial lookup is scoped **per item row** (`$asnItemId`). Item receipts are resolved **Sales-Order-wide** via `App_NetSuite::listItemReceiptsCreatedFromSalesOrder($soId)` (fans across ALL POs from `listPurchaseOrdersCreatedFromSalesOrder`, dedupes receipts by `internalId`), cached **per sales-order internal id** (`$itemReceiptsBySo`, keyed by `netSuiteInternalSalesOrderId`) — a shared SO is still fetched once — with the `getItemReceipt` detail cache kept per (globally-unique) receipt `internalId`. Both the inner item query **and** the outer ASN-candidate query require `netSuiteInternalSalesOrderId <> 0` (a sentinel guard mirroring the PO guard) so a null/zero-SO row cannot poison the cache or match the wrong SO's receipts. Self-healing: units previously stuck under an unstamped PO clear on subsequent 5-min cron runs.
   - **Receipt resolution is SO-ONLY — the stamped PO id never finds a receipt** (traced 2026-07-29, `3_create_installation_ticket.php`: candidate query ~L248-278, per-item ~L311-341, receipt resolution ~L354-360, part-number match ~L418-422, per-unit gate ~L511-535). The chain is strictly `AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId` → `listPurchaseOrdersCreatedFromSalesOrder` (all POs created from that SO) → `listItemReceiptsCreateFromPurchaseOrder` for each → dedupe (this is what `listItemReceiptsCreatedFromSalesOrder` wraps, `library/app/netsuite.php` ~L739 / ~L762 / ~L797, details via `getItemReceipt` ~L1697). **`netSuiteInternalPurchaseOrderId` is NEVER used to look up receipts** — it is only a non-null / non-zero **eligibility gate**. The schema is misleading here: the PO column *looks* authoritative and is not.
   - **Item receipts are never persisted.** They are fetched **live from NetSuite on every run** (no table, no bridge row), and the receipt-line → ASN-line linkage is derived **at runtime by part-number string match** after stripping the `DOE-` prefix and `_` characters. So NetSuite is the only source of truth for what was received, and a part-number formatting drift breaks the match silently.
+  - **Candidate pick is `ORDER BY RAND() LIMIT 1` ON PURPOSE — do not change it to a fixed order** (rgirish, 2026-10-23). Newest-first was tried and the script stuck for days on one ASN with bad data. Cost: ~1,698 pending candidate ASNs (oldest 2023-10-02, most will never complete) and only a handful finish per run, so a new, fully received ASN can wait hours or days by chance (2026-10-22: 4 ASNs completed in 7.5 h). Symptom "received in NetSuite, no repair order in Desk" with correct SO/PO stamps = this wait, not a stamp mismatch → run the ASN by hand (below).
+  - **Manual one-ASN run** (optional argv `<asnId>`, int-cast, non-numeric throws "Invalid ASN id"; pins the candidate query to `AdvanceShippingNoticeItems.advanceShippingNoticeId`; no arg = normal cron): as root on worker15 via SSM, `php /var/app/current/crons/sync/nycdoe/3_create_installation_ticket.php <asnId>` (or `cd` + `./`). Never use the cron's own `/var/www/html/...` path — the overlap guard exits it at once, silently ([path rules](../../../1.0/apps/worker1.5/features/cron-overlap-guard.md)). Run ASNs **one after another**. Verify after: per MSO, `COUNT(repair_orders) = COUNT(DISTINCT serialNumId)`.
   - **MSO reuse is safe on re-run.** `doeCreateInstallationRepairOrder()` looks the managed service order up by `advanceShippingNoticeId`, so re-running Stage 5 against an ASN that already has an MSO **reuses** it rather than creating a duplicate.
   - **The per-SO (not per-ASN) receipt cache is load-bearing.** An ASN can carry lines from more than one NetSuite sales order; caching the receipt set once per ASN off the *first* line's SO matches every other line against the wrong SO's receipts (FIXED 2026-07-22 — see the multi-SO gotcha). One level *above* the multi-PO-per-SO fix.
 - **"#/N received" UI** (`togadesk/desk/template/pages/managedserviceorders/view.php`): denominator = `SUM(qtyOrder)` per part (committed + unsent); numerator = Units with `togadeskRepairOrderId IS NOT NULL`.
@@ -161,6 +168,7 @@ The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API`
 - **`Bridge_NetSuite`** (legacy) has `SalesOrders` / `SalesOrderItems` / `SerialNumbers` but **no `PurchaseOrders` table** — POs are pulled live from the NetSuite API.
 - **Stuck-vs-received check, per ASN item:** compare `listItemReceiptsCreatedFromSalesOrder($stampedSO)` against `listItemReceiptsCreateFromPurchaseOrder($stampedPO)`, dumping each receipt line's `itemName` plus its `inventoryAssignment` serials. If the SO-wide call returns 0 but the stamped PO returns the receipt with the matching serials, it is the cross-wired-stamp regression below.
 - **Legacy `Logs.FileLog` (~47k rows) has no `fileName` index** — only `PRIMARY(id)`, `FileLog_job(job)`, `dtStamp`. Filter by `job` first (uses `FileLog_job`; `ASN_LENOVO` 1,899 / `ASN_LEXMARK` 954 / `ASN_APPLE` 113 rows as of 2026-09-25), then `fileName`. Select `id` only — never `fileData` (longblob) in a scan.
+- **Legacy `Common.CronJobExecutions` (~120M rows) has only `PRIMARY(id)`** — filter by an id range (`id > MAX(id) − N`) plus `job LIKE '%3_create_installation_ticket%'`, never a date scan. Job value is the absolute path (`/var/www/html/crons/sync/nycdoe/<script>.php`).
 - **`App_Model_Core_AdvanceShippingNotice` does NOT map `dtCreatedInstallationTicket`** — its `__get` throws "No field or quick query defined for key …". Read that column via **raw SQL**, not the model.
 
 ### Monitoring the ServiceNow endpoint (worker2 liveness probe — TRUE-80587)
@@ -182,6 +190,12 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 ## Gotchas
 - **Re-running a missed DOE report by hand** (steps in [Diagnosing worker1.5 crons](../../../1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md)): safe to repeat, but each run emails DOE again (no duplicate guard). A late open/closed run moves its 24 h window, so closures between 1:00 AM and the late run drop out of the "closed" tabs. Both report crons are clean on PHP 8.5 since library TRUE-83159 (byte-identical xlsx on 7.2 / 8.1 / 8.5).
 - **⚠ OUTAGE 2026-10-22 → 27 (fixed TRUE-82940, not deployed) — PHP 8.5 stopped the whole Stage 1b import.** `Logs.Issue` 16U "Undefined offset: 2" at `library/app/asnprocessor/apple.php:74` (column 18 contact split on `/` with 2 parts). On worker1.5 (PHP 8.5) a warning ends the cron; Apple runs first in `legacy_import_asn.php`, so Lexmark and Lenovo never ran either. Signs: no `Logs.FileLog` `ASN_APPLE`/`ASN_LENOVO`/`ASN_LEXMARK` rows after the cutover, no new `Core.WorkerJobs` `Sftp/Import/RunForFile`, while other SFTP jobs (Staples, Compass) run fine (= not a ban). **worker1.5 cron errors land in prod V2 `Logs.Issue`, not legacy `Vision_Log.ErrorLog` / `Logs.Errors`.** Fix: `?? ''` on `$data[2]`, `$cols[45]`, `$cols[47]`; also declared `App_ASNRaw::$_dedupeKey` and passed `$escape` to `str_getcsv()` in lexmark/lenovo (8.5 deprecations). A manual run 2026-10-27 cleared the backlog (11 files; 1124 Apple, 80 Acer, 80 Lexmark, 10 Lenovo service requests in 2.0). PHP 8.5 rules: [Running library on PHP 8.4 / 8.5](../../../1.0/apps/library/features/php-8-5-runtime-compatibility.md).
+- **Stage 5 did not run 2026-10-22 16:50 → 2026-10-23 10:35 CT** (worker1.5 cutover, TRUE-83078: last legacy run 16:50 on the old fleet, worker 1.0 schedules `active: 0` in worker `22ce50cf`; worker1.5's crontab ran under `/bin/bash`, so the overlap guard made Stage 5 exit every run until the TRUE-83121 fix at 10:35 — see [cron overlap guard](../../../1.0/apps/worker1.5/features/cron-overlap-guard.md)). ASNs received in that window got repair orders late; 12 were run by hand (TRUE-83140, see change history). Check gaps in legacy `Common.CronJobExecutions` (Debugging section).
+- **Manual Stage 5 runs — four traps** (`3_create_installation_ticket.php`, overlap guard `App_Framework::isProcessRunning()` in `library/app/framework.php` ~L833):
+  1. The guard matches the exact path string ([rules](../../../1.0/apps/worker1.5/features/cron-overlap-guard.md)). The cron (`*/5`, runs ~590 s) is almost always active, so a manual run on `/var/www/html/...` **exits 0 with no output and no work**; parallel manual runs on the same path do the same. Run one at a time.
+  2. A run on another path (`/var/app/current/...` or `./`) runs next to the cron — small duplicate-repair-order risk if both pick the same ASN (1 of ~1,698). Check per MSO `COUNT(repair_orders) = COUNT(DISTINCT serialNumId)` afterwards.
+  3. With an ASN filter the do/while re-picks the **same** ASN until 590 s while any line is unreceived (e.g. cable `DOE-A3L980-10-S` on its own NetSuite PO still Pending Receipt) — each such ASN takes ~10 min. Not a hang.
+  4. An EB deploy mid-run swaps `/var/app/current`; later runs fail "Could not open input file: ./3_create_installation_ticket.php". `cd` into the folder fresh for each ASN.
 - **⚠ OPEN BUG (found 2026-10-19, not fixed) — 1.0 Lexmark/CDW dates save as `1969-12-31`.** CDW sends `YYYY/MM/DD`, but `library/app/asnprocessor/lexmark.php` reads dates with `App_Date::convertMDYYYYtoSQLDate`. All 5,244 of 5,244 `App_ASNProcessor_Lexmark` queue rows since 2026-09-01 have `dateShipped` (and other dates) = 1969-12-31. 2.0 Parse reads `YYYY/MM/DD` correctly. Never trust a 1.0 Lexmark/Acer ship date.
 - **⚠ OUTAGE (fixed TRUE-82735, not deployed 2026-10-14) — the RITM and POD crons never finished for months.** Signs: `Common.CronJobExecutions.dtCheckOut` NULL on every run, zero ServiceNow calls. RITM cron: Feb–Mar 2026 refactors dropped the per-MSO inner loop → `Logs.Issue` 1R *"Undefined variable: row2"* every run since at least 2026-05-07 (last sane version `ce98cd2b5`). POD cron: the file query had **no `ORDER BY`**, so every run hit the same poison file (RITM1387575, already closed → empty-body 200 on `-7` → `Logs.Issue` 1S *"Empty response body from API (HTTP 200)"*), died, never set `dtTransmitted`, and re-uploaded that POD every 5 min; 6,718 files stuck, last sent 2026-09-04. Lesson: an unordered work queue + one poison row + a run-killing throw = a silent total outage; order the queue, isolate per row, and capture. **Still broken on worker1.5 (2026-10-23, TRUE-83163):** its `crons/sync/nycdoe/send_request_item_updates.php` (copied in `58dab19`) has undefined `$row2` / `$locationSysId` / `$sendLocationUpdate` / `$ritmNumber` — a git bug, not PHP 8; left unfixed by choice.
 
@@ -254,10 +268,11 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
     Note this only confirms the row is an eligible *candidate* for the `*/5 min` Stage 5 cron (which then picks ONE ASN at random via `ORDER BY RAND() LIMIT 1` across all pending ASNs) — it does not by itself confirm a ticket was created; wait for the cron and check `togadeskRepairOrderId`/`dtInstallTicketStatusChanged` to confirm ticketing completed.
   - Unlike instances 1–2 below, no duplicate/replacement SO was involved here — the mismatch is a same-ASN sibling PO the warehouse had already been using for other lines on this ASN.
 
-- **⚠ OPEN BUG (TRUE-83064, 2026-10-22; data repaired, root-cause code fix NOT done) — one customer PO split into two ASNs (two vendors) → wrong manufacturer + stranded cross-received units.** Customer PO `WR270089579` came in as two ASNs, one under the Acer vendor and one under the Lexmark vendor, for the same Acer part `DOE-NX.JHWAA.004-NYCDOE` (likely the Acer-as-Lexmark SFTP path, Stage 1b above).
+- **⚠ OPEN BUG (TRUE-83064, 2026-10-22; data repair COMPLETE 2026-10-23, root-cause code fix NOT done) — one customer PO split into two ASNs (two vendors) → wrong manufacturer + stranded cross-received units.** Customer PO `WR270089579` came in as two ASNs, one under the Acer vendor and one under the Lexmark vendor, for the same Acer part `DOE-NX.JHWAA.004-NYCDOE` (likely the Acer-as-Lexmark SFTP path, Stage 1b above).
   - **Manufacturer comes from the ASN vendor, not the part.** Stage 5 (`3_create_installation_ticket.php`) sets the TOGa Desk manufacturer from the ASN's `Vendors.name` and finds the MSO by `advanceShippingNoticeId`. Two ASNs = two MSOs, and the Lexmark-ASN line was ticketed as Lexmark.
   - **Cross-received serials dead-end silently.** Stage 5 matches an item's units only against receipts of that item's **own** SO. The warehouse received serials onto the **other** PO of the same customer PO, so they never match, and the customer-PO-scoped duplicate-serial guard refuses to add them as new units. No error; the cron retries forever. Evidence: legacy `Logs.API` `getItemReceipt` rows with `sourceJob LIKE '%3_create_installation_ticket%'` (30-day retention).
   - **Repair (data-only, dbchanges#270, branch `fix/repair-wr270089579-consolidate`):** `Core/SK/2026-10-22-data-repair-wr270089579-consolidate.sql` swaps the 102+102 cross-received `AdvanceShippingNoticeUnits` onto the item whose SO received them and moves item `55578` onto ASN `27180`; `TOGaDeskSupport/SK/2026-10-22-data-repair-wr270089579-consolidate.sql` moves the 24 Lexmark ROs onto MSO `334826` as Acer (assets → manufacturer `10` / model `23668`) and deletes the empty MSO `334820`. Two databases — verify each. Guard rules: [dbchanges workflow](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md) → *Rules for a one-time data repair*.
+  - **Result (TRUE-83126, 2026-10-23):** after the repair the 204 remaining units matched their own SO receipts (item `55573` SO `7626155` / receipt `7637002` qty 499; item `55578` SO `7629904` / receipt `7635916` qty 126) and only lacked ROs because Stage 5 was not running (see the outage gotcha). Manual run on ASN `27180` → **625/625 units ticketed on MSO `334826`, 625 distinct serials, no duplicates.** No further data fix needed.
   - **⚠ OUTSTANDING:** TOGa Desk Lexmark model `23782` still holds **152 mislabeled assets** on other POs (`WR270055266` MSO `332014`, `S202652963` MSO `332217`) — not fixed. Any customer PO split across vendors can repeat both failures until Stage 5 is fixed.
 
 - **⚠ OPEN BUG (investigated 2026-06-19, not yet fixed) — manufacturer PO-suffix format variance causes DUPLICATE fulfillment.** Stage 2's serialized dedupe is scoped to the *exact* PO string (`lookupAsnUnitIdByAsnPartNumberAndSerialNumber[vendorId][purchaseOrder][part][serial]`, `legacy_process_asn_queue.php:272`), and the PO flows **verbatim** end to end — never trimmed/normalized except Lenovo's `trim()` in `asnprocessor/lenovo.php`. So when a manufacturer re-sends the *same physical unit* under a different suffix spelling (e.g. Lenovo `WR260150152-RPL` then `WR260150152-RPLE`, same serial `V60095WK`), the two PO strings differ → the serial guard misses → a full duplicate ASN→SO→MSO→RO chain is created for one device (two NetSuite SOs, two repair orders, two onsite tickets).
@@ -295,13 +310,16 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 ## Related
 - [NYC DOE client profile](../profile.md)
 - [NYCDOE EDI 850 Receive & 997 Acknowledgement](edi-850-receive-and-997-ack.md) — 850-in/997-out
-- [Worker (1.0) architecture](../../../1.0/apps/worker/architecture.md) — cron dispatch, role election, schedule files
+- [Worker (1.0) architecture](../../../1.0/apps/worker/architecture.md) — old cron fleet (schedules now inactive)
+- [Worker 1.5 architecture](../../../1.0/apps/worker1.5/architecture.md) — live cron server, schedule, server access
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
 - 2026-10-27 — TRUE-82940: `apple.php:74` PHP 8.5 outage (10-22 → 27) fixed, not deployed; errors land in V2 `Logs.Issue`; Stage 1b cadence is 15 min (worker1.5); gated mode sends one `RunForFiles` per account per run. (mhammontree)
 - 2026-10-23 — Daily open/closed report row; manual re-run gotcha (duplicate email, 24 h window shift) (TRUE-83159) (bala)
 - 2026-10-23 — RITM cron still broken in the worker1.5 copy (undefined vars); not fixed on TRUE-83163 (jcardinal)
+- 2026-10-23 — TRUE-83126: Stage 5 outage root cause = worker1.5 bash crontab vs overlap guard (TRUE-83121); argv `<asnId>` manual run + safe path; ASN 27180 / `WR270089579` fully ticketed (625/625) (rgirish)
+- 2026-10-23 — TRUE-83140: Stage 5 gap during worker1.5 cutover; ran Stage 5 by hand for ASNs 27280, 27281, 27276, 27270, 27275, 27271, 27274, 27267, 27228, 27231, 27230, 27269 (12 customer POs, e.g. `S202787667`, `WR270111083`), ROs in 855056–855271, no serial has two installation ROs. Cable lines `DOE-A3L980-10-S` (and `DOE-50G0802-NYCDOE` on `WR270111083`) open until received. Random pick confirmed intentional; manual-run gotchas. No code change. (rgirish)
 - 2026-10-22 — Open: new `apple.php:74` Undefined offset (`Logs.Issue` #1115), seen during TRUE-82983 (jcardinal)
 - 2026-10-22 — TRUE-83064: repaired customer PO `WR270089579` (two ASNs, Acer + Lexmark vendor) via dbchanges#270; open-bug gotcha — Stage 5 takes the manufacturer from the ASN vendor and keys the MSO on the ASN; cross-received serials dead-end silently; 152 mislabeled assets on Lexmark model `23782` remain. No code change. (sking)
 - 2026-10-19 — Gated delete live since ~10-11 and its re-release login storm (worker2 SFTP IP ban); file names matter to 1.0 only for FileLog/dedupeKey (rename = re-email); open Lexmark 1969-12-31 date bug. (bala)

package/knowledge/clients/nycdoe/profile.md

@@ -15,7 +15,7 @@ client: nycdoe
 type: profile
 status: active
 updated: 2026-10-23
-owners: [mhammontree, sking, bala]
+owners: [mhammontree, sking, bala, rgirish]
 files: []
 related:
   - features/servicenow-integration.md

package/knowledge/clients/office-depot/profile.md

@@ -18,7 +18,7 @@ project: TOGa Voice
 client: office-depot
 type: profile
 status: active
-updated: 2026-10-23
+updated: 2026-10-27
 owners: [akhokhani, mhammontree, jcardinal, snaredla, rgirish]
 files: []
 related:
@@ -40,7 +40,7 @@ Office Depot is the production support client of the standalone Python/LiveKit T
 ## Platforms and knowledge
 
 - [ODP support agent](features/odp-tech-support.md) owns ODP prompt, settings, billing tenant and deployment target.
-- Talos call files reach Desk tickets (TOGaDesk clientid 3) through the 1.0 `api`: [Talos ticket API](../../1.0/apps/api/features/talos-ticket-endpoint.md). Since 2026-10-22 ODP's Desk, store and toga hosts run on the shared `togatech-legacy-*-production` envs in toga-prod (old `odp-*` envs terminated): [legacy 1.0 web apps in toga-prod](../../1.0/apps/test/workflows/legacy-1-0-web-apps-in-toga-prod.md).
+- Talos call files reach Desk tickets (TOGaDesk clientid 3) through the 1.0 `api`: [Talos ticket API](../../1.0/apps/api/features/talos-ticket-endpoint.md). Since 2026-10-22 ODP's Desk, store and toga hosts run on the shared `togatech-legacy-*-production` envs in toga-prod (old `odp-*` envs terminated): [legacy 1.0 web apps in toga-prod](../../1.0/apps/test/workflows/legacy-1-0-web-apps-in-toga-prod.md). ODP's TOGa Retail sandbox `odp.toga.sandbox.togatechnology.com` (internally beta, auto-deploys on toga `_beta` push) is in the same doc's sandbox section.
 - [Monthly call transcript report](features/odp-call-transcript-monthly-report.md) — worker2 cron, built from S3 session reports.
 - [Voice architecture](../../standalone/apps/voice-to-voice/architecture.md) maps SIP, LiveKit, shared services and operational checks.
 - [Ticket API](../../standalone/apps/voice-to-voice/features/ticket-api-integration.md) owns signing and client/department routing.

package/knowledge/sessions/2026-10-27-nychh-jcardinal.md

@@ -0,0 +1,85 @@
+---
+type: session
+slug: nychh
+title: H&H (NYCHH) PO stock reconciliation — Supply vs NetSuite (TRUE-82604)
+author: jcardinal
+repos: [_underscore, library, worker1.5, dbchanges2]
+framework: both
+client: hh
+status: active
+created: 2026-10-27
+updated: 2026-10-27
+---
+
+# Session: nychh
+**Date:** 2026-10-27
+**Project/Repo:** _underscore (2.0), library + worker1.5 (1.0)
+**Task:** Make TOGa Supply's per-PO-line stock for H&H (Client_Nychh) match NetSuite, explain/fix every gap the client raised, and prepare Ops/Dev fix lists (ticket TRUE-82604).
+
+---
+
+## The agreed H&H stock rule (from dev + Ops, final)
+- Stock IN = an IA linked (Stock Adjustment field) to the fulfillment of a stocking order ("stocking order" ticked). Location does NOT matter.
+- Stock OUT = the fulfillment of a transfer order (non-stocking SO). Location does NOT matter.
+- An IA on a TO's fulfillment adjusts the stock out (return to stock).
+- "Stocking order" not ticked = drop ship, no stock.
+- One TO depletes one PO (SO 161465 with PO # "EIT0009416/0010893" was a one-off).
+- Old IAs with a blank Stock Adjustment are pre-process history; Ops will NOT edit them (keep item history).
+- On-hand / available never show below 0 (all clients).
+
+## What WORKED
+- `_underscore` Model/Client/PurchaseOrderItem.php: on-hand = received − fulfilled + IA shares on linked TO fulfillments (EIT0010365 P2422HE netted to 0). Committed 45098c9c.
+- `_underscore` floors: `_qtyOnHand`/`_qtyAvailable` wrapped in GREATEST(...,0) in Client/Item, Client/PurchaseOrderItem, Client/SalesOrderItem (available only), Elite/Item, Nychh/Item. Committed c187d25b.
+- `_underscore` location removed from Nychh stock math (Nychh/PurchaseOrderItem `_qtyFulfilled` override deleted → inherits; `_qtyCommitted` no location; Nychh/Item same; Nychh/Location.php back to original). Committed c187d25b.
+- `worker1.5` sync_togasupply_hh.php: removed `inventoryAdjustmentStockLocationNetsuiteIds` (missing key = all locations). Committed d0c1f1a. Verified: every linked IA line in prod is at location 117, so no shares change and no share-runner re-run is needed.
+- `library` app/api/toga2.php (syncSalesOrderFromNetsuite, existing-PO branch): SO line's own item wins over an item-group sibling when the PO has a line for it. Committed 715e7ecc. Re-synced SO 154937 (ns 4230372) and SO 154738 (ns 4223786): EIT0010893 L31 210-BFZY and L72 210-BGCU now linked and 0 on hand; "-A" lines 136/144-146 have no links.
+- EIT0009416 fully matches NetSuite after Ops IAs #3141/#3142 (+IF #154769 on #3142), #3143/#3144 (210-AZBM), #3136/#3137 (210-BEQW-HHC).
+- EIT0012615 matches NetSuite only when New York (4) is counted (1000 each of O20-017-NHH, 30-0361300, O10-052-NHH1 physically sit at New York; Plainview shows −10).
+- EIT0011189: IA #1611 unlinked; allocation runner left HPCLJEM528Z at ±2 (whole TO lines only).
+- Excel for agreeing lines built: D:\TEMP\H&H\HH-Stock-Reconciliation-2026-10-26.xlsx (v1) and -v2.xlsx (adds ELM-resolved lines). Builder script lived in the session scratchpad (build_hh_xlsx.py) — dev said hold off on the Excel until discrepancies are settled.
+
+## What did NOT work — DO NOT RETRY THESE
+- Filtering TOs by origin location (only 117/173 count as stock out): hid real data problems (SVX-VX-RPY licences) and Ops confirmed location is not part of their process. Removed.
+- IA location rule in the sync split (IA line outside list = share 0): same reason. Removed.
+- Adding HQ (6), New York (4), New York Warehouse (1) to the location list: those warehouses hold OTHER customers' stock, so they pollute both Supply math and the NetSuite comparison (e.g. CR2702-100-A271-C34-MB6 shows 157 at HQ, 210-BPPL 12, 210-AZBM 11 — not H&H).
+- Treating 7/28/2026 inventory transfers (#316–#325) as stock leaving H&H: they moved Plainview (117) → ELM (202), still H&H stock. Not a Supply problem.
+- Assuming invoice 236040 offsets credit memo 8124100: invoices 236040/238475 are NOT inventory-affecting; credit memo 8124100 IS (+128 DTKN/64GB_ at 117). NetSuite is high by 128; Supply (28) is right.
+- Giving Ops actions for items where Supply and NetSuite already match (dev called this overcomplicating). Rule: if totals match, nothing to do.
+- SuiteQL `custbody_stock_adjustment IS NULL` on the multi-select does not detect blank IAs — use BUILTIN.DF(...) per IA instead.
+- Comparing NetSuite on-hand per item including "_" twin items separately (C31CJ57012 vs C31CJ57012_): add twins together before calling a mismatch.
+
+## Not tried yet (candidates for next session)
+- Re-run the item-level comparison (Supply total across all POs vs NetSuite on-hand at 117+173+202 only) now that location is out of the math, and give the dev a short list: only items that still differ, one fix each.
+- Rebuild the client Excel (receipts + deployments) once discrepancies are settled; reuse the receipts/deployments queries (filter by agreeing PO lines).
+- Decide with dev: blank-Stock-Adjustment IAs → option 1 (list as known exceptions) vs option 2 (new IA pair +X linked / −X unlinked, old IA untouched). Dev's pick pending.
+- Decide with dev: floor Nychh item-level `_qtyCommitted` at 0 (Model/Nychh/Item.php subtracts shipped from committed and can go negative, inflating available).
+- Ops answers pending: SVX-VX-RPY is set up as InvtPart in NetSuite though Ops says licences are not inventory.
+- Ops items open: tick "stocking order" on SO 159067 (vendor PO 107117, 1000 × AB678031); IA −128 DTKN/64GB_ unlinked (reverse credit memo 8124100 stock return); remove IF #274364 from IA #2136 (EIT0012614 −1103 vs EIT0012372 +1057); SOs 169639/169656 stocking flag; transfer 1000 each NHH items from New York to Plainview (EIT0012615).
+- Still unexplained: 520-BBNF (Supply 20 vs NetSuite 60); T-LBK462-BSB-R −25 on EIT0011108 / 210-BCTU −5 on EIT0010893 (cross-PO TO use → Ops IA pair, need the TO numbers).
+
+## Current file state
+| File | Status | Notes |
+|------|--------|-------|
+| _underscore Model/Client/PurchaseOrderItem.php | committed c187d25b (+45098c9c) | TO-IF IA term; floors |
+| _underscore Model/Client/Item.php, Client/SalesOrderItem.php, Elite/Item.php | committed c187d25b | floors at 0 |
+| _underscore Model/Nychh/Item.php, Nychh/PurchaseOrderItem.php, Nychh/Location.php | committed c187d25b | no location; floors |
+| worker1.5 crons/toga2/netsuite/sync_togasupply_hh.php | committed d0c1f1a | IA location key removed |
+| library app/api/toga2.php | committed 715e7ecc | own item before group sibling |
+| dbchanges2 Client_Nychh/2026-10-16a, 2026-10-20a, 2026-10-20b | committed earlier | one-time cleanup SQL (ran) |
+| All three app repos | pushed on TRUE-82604 | deploy status: dev deploys; re-syncs done after deploy |
+
+## Decisions made
+- Location never decides H&H stock (dev + Ops). Rejected: hard-coded warehouse list; per-location "client stock" flag (not needed once location is out).
+- On-hand/available floored at 0 for all clients (dev). Accepted side effect: on-order changes for lines that were negative; negatives are hidden on screen but raw math still shows gaps for reconciliation.
+- SO→PO line matcher must never prefer a different item when a same-item PO line exists (dev). Rejected: fall-through to a sibling when own item has no qty/price match.
+- When comparing to NetSuite, use only H&H-only warehouses (117 Plainview, 173 Processing, 202 ELM); name H&H stock sitting in shared warehouses instead of calling it a mismatch.
+- Ops text style: only the items asked about, one short line each, "currently X → Y", no actions where totals match.
+
+## Blockers
+- Waiting on Ops answers (SVX-VX-RPY item type; open Ops items above) and the dev's pick on blank-IA handling (option 1 vs 2).
+
+## Exact next step
+> Re-run the item comparison for the client's 12 POs (EPC0003177, EIT0012148, EIT0012673, EIT0012615, EIT0012548, EIT0012107, EIT0011563, COF0015668, EIT0011733, EIT0011108, EIT0010893, EIT0012129): Supply item total across all PO lines (no location filter) vs NetSuite inventorybalance at locations 117,173,202 (add "_" twins), and report only items that still differ, one fix each.
+
+---
+_Saved by /session-save on 2026-10-27_

package/knowledge/standalone/apps/websocket/features/sqs-fanout.md

@@ -6,7 +6,7 @@ project: WebSocket Server
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-26
 owners: [rgirish]
 files:
   - websocket/fanout.js
@@ -14,6 +14,7 @@ files:
   - websocket/rooms.js
 related:
   - ../../../../2.0/apps/_underscore/features/event-publish-sqs.md
+  - ../architecture.md
 ---
 
 Fan-out logic turning one SQS `toga-events` message into `record:changed` emits to list/instance/child/parent Socket.io rooms; open for `fanOut()` room resolution + the subscribe pattern.
@@ -24,7 +25,7 @@ The fan-out logic that turns one SQS `toga-events` message (published by api2 af
 
 ## How it works
 
-`pollLoop()` long-polls SQS (`MaxNumberOfMessages: 10`, `WaitTimeSeconds: 20`), parses each message body to an `event`, calls `fanOut(event)`, then deletes the message. Malformed bodies are logged and deleted (not retried); fan-out errors are logged without deleting fan-out from blocking the loop. On a receive error it backs off 5s before retrying.
+`pollLoop()` long-polls SQS (`MaxNumberOfMessages: 10`, `WaitTimeSeconds: 20`), parses each message body to an `event`, calls `fanOut(event)`, then deletes the message. Malformed bodies are logged and deleted. A `fanOut()` throw is logged and the message is **still deleted** (no retry, see Gotchas). On a receive error it backs off 5s before retrying. A successful fan-out logs `[fanout] clientUuid=<uuid> route=<route> method=<method>`.
 
 `fanOut(event)` reads `{ clientUuid, recordRoute, uuid, httpMethod, dtEvent, parentRoute, parentUuid }` and emits to up to four kinds of rooms (all keyed by `clientUuid`):
 
@@ -45,10 +46,18 @@ socket.on('record:changed', (payload) => { /* refresh the matching list/detail v
 
 ## Gotchas
 
+- **⚠ Non-prod: one queue, 19 consumers (open, 2026-10-26).** Every non-prod websocket env in 975050298201 polls the same `WebsocketEvents` queue, and every non-prod api2 config publishes to it. SQS gives each message to ONE consumer, so a non-prod event lands on a random env — usually the wrong one. "My browser got no `record:changed` on beta" is expected until each env gets its own queue (or SNS fan-out). See [architecture → Deployment](../architecture.md).
+- **⚠ Prod at 2 instances splits events.** No shared Socket.io adapter: an event is emitted only on the instance that pulled it, so a browser on the other instance misses it. ALB stickiness is off, so the polling transport also breaks. Keep prod at 1 instance, or add a shared adapter + stickiness, before relying on it.
+- **A `fanOut()` throw still deletes the message** — the event is lost, not retried.
+- No `record:changed` at all? Check the publisher first — see [_Event::publish gotchas](../../../../2.0/apps/_underscore/features/event-publish-sqs.md#gotchas).
 - Room keys are built from **`clientUuid`**, never `clientId` — using `clientId` cross-wires tenants or misses rooms entirely.
 - The children-room lookup is a nested scan of `routeMap` (recordId → route) per child; it is correct but O(routes × children) — fine at current route counts, watch if routes grow.
 - Children notification depends on `Core.InherentRecordChildren` being populated and the metadata cache being fresh (refreshes every 5 min) — a brand-new parent/child relation will not fan out until the next refresh.
 
+## Change history
+- 2026-10-26 — Gotchas: shared non-prod `WebsocketEvents` queue (wrong-env delivery), prod 2-instance event split, message deleted on `fanOut()` throw (TRUE-83223) (rgirish)
+
 ## Related
 
-- `2.0/apps/_underscore/features/event-publish-sqs.md`
+- [_Event::publish to SQS](../../../../2.0/apps/_underscore/features/event-publish-sqs.md)
+- [websocket architecture](../architecture.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1165",
+  "version": "1.0.1167",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",