toga-ai 1.0.1177 → 1.0.1179

package/knowledge/1.0/apps/togadesk/features/email-to-ticket-intake.md

@@ -6,8 +6,8 @@ project: TOGa Desk
 client: shared
 type: feature
 status: active
-updated: 2026-10-27
-owners: ["jcardinal", "dfranks", "mhammontree"]
+updated: 2026-10-28
+owners: ["jcardinal", "dfranks", "mhammontree", "rgirish"]
 files:
   - crons/tickets.php
   - crons/tickets_prod.php
@@ -21,6 +21,7 @@ related:
   - 1.0/apps/library/features/app-cloud-aws-credentials.md
   - 1.0/apps/worker/architecture.md
   - 1.0/apps/worker1.5/architecture.md
+  - 1.0/apps/worker1.5/workflows/diagnosing-crons-not-running.md
   - 1.0/apps/library/features/php-8-5-runtime-compatibility.md
   - 2.0/apps/worker2/features/inbound-email-ticket-importer.md
 ---
@@ -38,12 +39,15 @@ TOGa Desk 1.0 ingests support email into tickets through one live cron-driven IM
 
 ### Flow
 1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `$scriptpath` = `togadesk/desk`; autoload = `desk/vendor/autoload.php` (**no AWS SDK**) + `vendorClassAutoload` (`desk/vendor/classes/`). `tickets_prod.php` additionally performs **OAuth2/MSAL** token acquisition for the O365 mailbox.
-2. **Poll:** scheduled on worker1.5 every 15 min (:05/:20/:35/:50, `worker1.5/schedules/cron.worker.json:204`); connects to `outlook.office365.com:993` INBOX and long-polls up to 780s, processing each message then moving it to the IMAP folder **"Trash"** (see Gotchas).
-3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes). Thread match, in order: **plus-addressing** `+<ticketnum>@` in To, then `Ticket #` in the subject. Matched ⇒ `addReply()`; otherwise ⇒ `add()`. `In-Reply-To` is captured but **never read**; there is **no Message-ID dedupe**.
+2. **Poll:** scheduled on worker1.5 (EB `worker15-production`) every 15 min (:05/:20/:35/:50, `worker1.5/schedules/cron.worker.json:204`); connects to `outlook.office365.com:993` INBOX and long-polls up to 780s, processing each message then moving it to the IMAP folder **"Trash"** (see Gotchas).
+3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes). Thread match, in order: **plus-addressing** `+<ticketnum>@` in To, then `Ticket #` in the subject. Matched ⇒ `addReply()`; otherwise ⇒ `add()`. `tickets.php` passes `Message-ID` + `In-Reply-To` into `emailToTicket()`: Message-ID is only stored (`emailMessageIdentifier`), In-Reply-To is **never read**; there is **no Message-ID dedupe**.
    - **Noise suppression (before any ticket/attachment is created):** `Ticket::isSuppressedNoiseEmail(?string $from, ?string $subject): bool` (static, `class.ticket.php`) is true only when the sender matches `/^no-?reply@/` (domain-agnostic) **AND** the subject starts with `staples po ` — both required so a real person is never blocked. `crons/tickets.php` adds it as one more `&&`-clause in the intake guard (with the delivery-status / `MAILER-DAEMON` / Compass auto-reply filters) and logs a "suppressed" line. (TRUE-79975 — ~815 such alerts since Jun 2024, mostly closed by hand in "Support Desk".)
 4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments **straight into `desk/uploads`** — on the worker a symlink to the s3fs mount (`/var/app/asifiles/togadesk-prod/uploads`), so the write is the S3 write. Before writing it checks `is_link` + written size. **`App_Cloud` is not used** (PR #625, TRUE-82935). Blocked names (`TICKET_IMPORT_BLOCKED_FILE_PATTERN` in `crons/tickets.php`: php*/cgi/sh/asp/jsp, html/htm, xht/xhtm/xhtml, shtm/shtml, stm, ssi, svg/svgz, xml, xsl/xslt, mht/mhtml, `.htaccess`, `.user.ini`) are saved with `.txt` added (widened TRUE-82935, PR #626). Optionally generate Talos summaries.
 5. **Post-pass:** `Ticket::processRules()` (escalation) and `Ticket::autoClose()` (SLA auto-close) run after the mailbox sweep.
-6. **On failure:** alert email to `serviceops@togatech.com`, also logged in legacy `Logs.Emails` (`subject LIKE 'TOGaDesk email import failed%'`) — the fastest way to read the cron's error (the cron has no stdout).
+6. **On failure** (all in `crons/tickets.php`):
+   - Import error, first time: email is flagged `\Flagged`, left in INBOX, run ends; next run retries once, then moves it to IMAP folder **"Import Failed"** (`TICKET_IMPORT_FAILED_FOLDER`). A save failure (`Ticket::EMAIL_IMPORT_SAVE_FAILED`) parks at once (retry could duplicate).
+   - Email that cannot even be read (`$imap->getMessages()` throws): parked in "Import Failed" + alerted. More than `TICKET_IMPORT_MAX_UNREADABLE_PER_RUN` (3) in one run rethrows and stops the run — that is a code/connection fault, not a bad email (TRUE-83364).
+   - Alerts go to `serviceops@togatech.com` (`TICKET_IMPORT_ALERT_EMAIL`), also logged in legacy `Logs.Emails` (`subject LIKE 'TOGaDesk email import failed%'`). Anything thrown outside these paths sends **no** alert — see Gotchas (diagnosis).
 
 ### Data model
 Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `config` table (`tickets_defaultdepartment`, `tickets_defaultclient`, `auto_close_tickets`, `email_from_address`, `timezone`). Department routing keys on a `tickets_departments.email` `LIKE` match against the To/CC addresses; **anything unmatched falls through to config `tickets_defaultdepartment` (= dept 1 "Support Desk" catch-all) and `tickets_defaultclient`** — which is why unrouted/automated mail piles into "Support Desk". There is **no allowlist/blocklist config framework**: noise filtering is ad-hoc inline guards in the crons (the hardcoded Compass "Automatic reply:" filter and now `isSuppressedNoiseEmail`).
@@ -74,7 +78,13 @@ Ruled out as user-creation sources (do not re-chase):
 ## Gotchas
 - **`$processName` guard:** `tickets.php` throws if executed directly instead of via an env wrapper — and `ENVIRONMENT`/`$env` must be set or it loads the wrong config.
 - **On worker1.5 (PHP 8.5, no ext-imap) intake depends on library's php-imap2 patches.** Before `4a2c363f` the prod cron died at `desk/vendor/classes/class.imap.php:228` (`imap2_headerinfo` → `Class "Javanile\Imap2\Polyfill" not found`). There the cron is `php /var/www/html/crons/../ontrack/crons/tickets_prod.php`; a library fix reaches it only after a worker1.5 redeploy. See [PHP 8.5 § IMAP without ext-imap](../../library/features/php-8-5-runtime-compatibility.md#imap-without-ext-imap-php-imap2) (TRUE-83148).
-- **A togadesk merge does NOT update the cron.** The cron runs on the 1.0 **worker** (`/var/www/html/ontrack`, cloned from togadesk at worker deploy — NOT `/var/www/ontrack`). The `Worker` CodePipeline sources only `worker` + `library`; a togadesk merge runs `TOGaDesk-Production` (web envs) only. After any `crons/` change, **release the `Worker` pipeline by hand**. TRUE-82935: PR #625 merged 10-16 16:15, reached the cron only 10-19 13:10. See [worker architecture](../../worker/architecture.md).
+- **A togadesk merge does NOT update the cron.** Since TRUE-83078 the cron runs on worker1.5 (EB `worker15-production`, toga-prod account 654654170868, us-east-1) at `/var/www/html/ontrack`. The prebuild hook clones togadesk `_production` as `ontrack` only at a worker1.5 deploy. A togadesk merge runs `TOGaDesk-Production` (web envs) only. After any `crons/` or `desk/vendor/classes/` change, **release the `worker15-production` CodePipeline** (source worker1.5 `_main`). `Worker-Production` deploys worker2 and does NOT touch this cron. See [worker1.5 architecture](../../worker1.5/architecture.md).
+- **Diagnosing a stuck intake on worker1.5** (TRUE-83364):
+  - Output: `/var/log/worker-cron.log`, mixed with every other cron; rotated, ~2 days kept. Grep `tickets.php(` or `ontrack/`.
+  - `Common.CronJobExecutions` rows with `job LIKE '%tickets_prod%'` and `dtCheckOut` NULL = crashed run OR a run that hit the 780s limit (the `exit` at the stop-time check skips `cronFinished`). NULL rows every 15 min + no new EMAIL tickets = blocked INBOX.
+  - No `App_Error` handler here (it calls `App_Config::initialize` + `cronInitialization`, not `App_Framework::initialize`): PHP warnings print and continue. Only a thrown error stops the run, through `set_exception_handler`, which writes to STDERR and exits with **no alert**.
+  - More on the box / crontab: [Diagnosing worker1.5 crons](../../worker1.5/workflows/diagnosing-crons-not-running.md).
+- **One email with an unknown charset blocked all intake (2026-10-28, 08:05 → 10:50 CDT, TRUE-83364).** On PHP 8.5 `mb_convert_encoding()` throws `ValueError` for a charset mbstring does not know (here `windows-1256`); `@` cannot stop an exception. The read (`Imap::getMessages` → `formatMessage` → `getBody` → `get_part`) was outside any try, so the email stayed first in INBOX, every run crashed, no alert; 36 emails waited. Fix (PR #630, `a7464886`): `Imap::convertCharsetToUtf8()` uses mbstring only for charsets it knows (names + aliases), else `iconv`, else reads as ISO-8859-1 — body is always valid UTF-8; plus the unreadable-email park (Flow step 6). Any new charset conversion must go through this helper.
 - **Never load a second composer autoloader in the cron as a fallback** (e.g. root `vendor/autoload.php` for the AWS SDK) — CTO decision TRUE-82935. Keep the cron off `App_Cloud`; any `App_Cloud` call there throws `Error` ([App_Cloud](../../library/features/app-cloud-aws-credentials.md)).
 - **Attachment outage 2026-10-12 → 10-19 (TRUE-82935).** 10-12 15:00 → 10-16 15:00: `files` rows written, S3 object missing (`copyFileToS3` threw). 10-16 15:16 → 10-19 13:10: PR #624 deleted the row on failure → no attachments at all. First good run 10-19 13:20 (files 615609-615611). Attachments lost 2026-10-12 15:00 → 2026-10-19 13:10 were **NOT recovered** (developer decision, TRUE-82935); the original emails remain in IMAP "Trash" if ever needed.
 - **Processed mail is moved, never purged.** `desk/vendor/classes/class.imap.php` `getTrash()` matches only `trash` / `inbox.trash` / `papierkorb`, else creates a custom folder **"Trash"** — originals stay recoverable there.
@@ -89,6 +99,7 @@ Ruled out as user-creation sources (do not re-chase):
 - **`adminid` clobber on reply routing:** after `emailToTicket()` sets `$data['adminid']` from the FROM address (~line 1407), an asset lookup (~line 1437-1442) overwrites it with the replying user's asset's assigned admin. On a matched-ticket reply this makes a user's email look like a staff reply in `addReply()` (`isAdminReply=true`), so `Awaiting User` etc. never transition. The asset-derived adminid is for NEW-ticket auto-assign in `add()`, not reply classification. See the ticket-lifecycle doc.
 
 ## Change history
+- 2026-10-28 — Unknown-charset email blocked intake: `convertCharsetToUtf8()` + park unreadable email (PR #630); documented "Import Failed" / retry-once / Message-ID store; worker1.5 diagnosis gotcha; deploy pipeline corrected to `worker15-production` (TRUE-83364) (rgirish)
 - 2026-10-27 — Corrected: threads on `+<ticket#>` then subject `Ticket #` only (In-Reply-To unread), no Message-ID dedupe, worker1.5 every 15 min with 780s long-poll; pipe.php/alpha/beta dead; new gotchas (sender check, CC dept override, unchecked move); linked 2.0 rewrite (TRUE-82933) (jcardinal)
 - 2026-10-23 — worker1.5 / PHP 8.5 gotcha: intake needs library php-imap2 patches (TRUE-83148) (jcardinal)
 - 2026-10-19 — Attachments saved straight into `desk/uploads` (s3fs), no `App_Cloud`; lost attachments not recovered; worker-pipeline deploy gotcha; corrected live `class.imap.php` copy; wider blocked-file pattern (TRUE-82935) (jcardinal)

package/knowledge/2.0/apps/api2/INDEX.md

@@ -30,3 +30,4 @@
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | Why a reverse hasMany collection (a parent's child records that FK back to it) is absent unless named in the fetch `fields` list; open when a related collection |
 | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | How 2.0 apps deploy (CodePipeline → CodeConnections → EB), the per-env `_underscore` branch resolution, the production auto-deploy + Rolling/batch-1 policy, and |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | The three things that must be right for a new 2.0 API environment to serve requests (config conventions, `Core.DatabaseHosts` data, DB seeding incl. |
+| [api2 instance frozen by php-fpm out of memory (triage and recovery)](workflows/php-fpm-out-of-memory-triage.md) | How to spot, confirm and recover an api2 EB instance that the kernel OOM-killer has frozen (every call 60 s then 504, EB "No Data", yet EC2 checks pass and CPU |

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-10-27
+updated: 2026-10-28
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -16,6 +16,7 @@ files:
   - api2/Config/
 related:
   - features/cxml-shipnotice-gateway.md
+  - workflows/php-fpm-out-of-memory-triage.md
 ---
 
 Backend for the public TOGa 2.0 API — metadata/ACL-driven REST JSON (`/v2`) + a cXML gateway, both over `_underscore` `_Model`; open for the response-envelope contract, request lifecycle, multi-DB/EB deploy, and the EB-health/committed-secret traps.
@@ -160,7 +161,9 @@ Two paging facts confirmed 2026-09-22 on `api-production-3`: (1) **EB native not
 2. Group `Logs_<Client>.Api` by `sourceIp, hostname` to find who moved.
 3. ALB access logs are OFF on api-production-1 — use CloudWatch + `Logs_<Client>.Api`.
 
-**Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout (worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing in-flight requests, or a PHP worker crash). ELB-generated 5xx never reach PHP → **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB "% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
+**A ninth cause — a single-instance env goes dark from php-fpm out of memory.** Every call hangs 60 s then ELB 504, the ALB target shows `Target.Timeout`, EB health is Grey "No Data", yet EC2 status checks pass and CPU is low. api2 sets no php-fpm pool cap (no `pm.max_children` in `.platform`/`.ebextensions`), so the platform default worker count runs at ~70 MB each, and a normal burst can exceed instance memory. Seen on `api-sandbox-client` (one t4g.small, 2 GB) during ordinary Compass QA, TRUE-83366, 2026-10-28. Unlike #3 the instance does not recover by itself. Triage and recovery: [php-fpm out-of-memory triage](workflows/php-fpm-out-of-memory-triage.md).
+
+**Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout (worker saturation, or an instance frozen by php-fpm out of memory); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing in-flight requests, or a PHP worker crash). ELB-generated 5xx never reach PHP → **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB "% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
 
 ### Known issues / accepted risks
 

package/knowledge/2.0/apps/api2/features/request-logging.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-10-27
+updated: 2026-10-28
 owners: ["mhammontree", "dfranks", "bala", "jcardinal", "rgirish"]
 files:
   - api2/Component/Api/V2/V2.php
@@ -214,6 +214,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 
 ## Gotchas
 
+- **Who sent a traffic burst: group `Logs_<Client>.Api` by `sourceIp`, route, and `queryString` with the `transactionId` part stripped.** GET calls carry `transactionId` inside `queryString`, so a raw `GROUP BY queryString` gives one row per call. Core `Logs.Api` holds only auth / pre-scope rows, so it cannot answer this (TRUE-83366).
 - **Failed client-scoped writes (non-2xx) ARE in `Logs_<Client>.Api`** (verified 2026-10-27, TRUE-83276). A client-log-only audit still misses pre-scope (401/`EN-*`) failures, apiproxy 5xx retry attempts, file-based logs, and non-API loads.
 - **A throw from a PRE interceptor leaves NO Api-log row** — 400 with null `transactionId`/`authority`/`audience`, real message in `messages[0].identifiers.message`. The only durable record is a `Logs.Issue` from `_Exception_Business`; a plain `_Exception_Validation` leaves no trace anywhere.
 - **An uncaught PHP fatal can leave NO log row anywhere** (incl. GETs, which return an empty 500 body). Empty 500 body + missing Api row = a fatal thrown before response assembly. Diagnose from the data (orphan sweeps), not the log.
@@ -235,6 +236,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - **`Logs_<Client>.Api` is pruned (once TRUE-81902 deploys)** — GET rows after 30 days, writes after 90. A missing older row is not proof the request never happened. See [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md).
 
 ## Change history
+- 2026-10-28 — Gotcha: find who sent a burst by grouping `Logs_<Client>.Api` on `sourceIp` + route + `queryString` minus `transactionId` (TRUE-83366). (bala)
 - 2026-10-27 — Corrected: failed client-scoped writes (non-2xx) are logged in `Logs_<Client>.Api`, not core `Logs` (verified on Prudential 404 `EV-14` PUTs, TRUE-83276). (jcardinal)
 - 2026-10-26 — Gotcha: prod `error_log()` output is only in `/var/log/php-fpm/www-error.log` (not EB bundle / CloudWatch); read via SSM (TRUE-83223). (rgirish)
 - 2026-10-23 — `Logs_Prudential.Api` `(method, dtStamp)` index; `ORDER BY id DESC` + payload `LIKE` = full backward PK scan; REGEXP error 3699 (TRUE-83157). (bala)
@@ -246,8 +248,6 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
 - 2026-09-25 — Linked the `Logs_<Client>.Api` retention job (GET 30d / writes 90d). (rgirish)
 - 2026-09-22 — Log-save now saves/restores `_Database` error state in `try/finally`; its own 1062s were disarming the unique-race retry on `/v2/service-requests`. Also recorded the `_production`-branch missing-method hazard and the rejected null-`transactionId` finding. (rgirish)
 
-- 2026-09-21 — Added `Logs_<Client>.Api.originalUserId` (delegation audit: NULL = none, never equal to `userId`) and the rule that outbound `direction = OUT` rows stamp neither user column. (bala)
-
 ## Related
 
 - [api2 architecture](../architecture.md)

package/knowledge/2.0/apps/api2/workflows/php-fpm-out-of-memory-triage.md

@@ -0,0 +1,57 @@
+---
+title: "api2 instance frozen by php-fpm out of memory (triage and recovery)"
+framework: "2.0"
+repo: api2
+project: API
+client: shared
+type: workflow
+status: active
+updated: 2026-10-28
+owners: [bala]
+files:
+  - api2/.platform/
+  - api2/.ebextensions/
+related:
+  - ../architecture.md
+  - ../features/request-logging.md
+  - ../features/health-check-endpoint.md
+  - ../../../../clients/compass-usa/profile.md
+---
+
+How to spot, confirm and recover an api2 EB instance that the kernel OOM-killer has frozen (every call 60 s then 504, EB "No Data", yet EC2 checks pass and CPU is low); open when a whole API env stops answering.
+
+## Summary
+
+api2 sets **no php-fpm pool cap** (no `pm.max_children` anywhere in `.platform` or `.ebextensions`), so php-fpm runs the EB platform default worker count. Each worker costs ~70 MB (anon-rss ~52 MB + shmem ~19 MB). On a small instance a normal burst of concurrent requests needs more memory than the box has. The kernel then kills php-fpm workers in a loop and the instance stops answering. There is **no memory metric** on api2 instances (no CWAgent `mem_used_percent`), so nothing warns before it happens.
+
+Fingerprint (all at once):
+- Every call hangs 60 s, then ELB **504**. After recovery, a short burst of fast ELB **502** while php-fpm restarts.
+- ALB target unhealthy with reason `Target.Timeout`.
+- EB health **Grey / "No Data"** ("None of the instances are sending data").
+- SSM `PingStatus` = `ConnectionLost`.
+- But EC2 status checks **pass** and CPU stays low (~10-25%). The box looks alive.
+
+Differs from api2 architecture EB cause #3 (worker starvation): there the ALB stays healthy and EB only flips Degraded. Here the instance goes dark.
+
+## Steps
+
+1. **Read the console log.** `aws ec2 get-console-output --instance-id <id> --latest`. Look for `Out of memory: Killed process <pid> (php-fpm)` repeated, and `oom-kill ... task_memcg=/system.slice/php-fpm.service`. On Windows set `PYTHONUTF8=1` first, or the CLI fails with a `charmap` error.
+2. **Size the burst.** CloudWatch RDS `DatabaseConnections` on the env's writer (each api2 request holds ~4 connections) and ALB `RequestCount`.
+3. **Find who sent the traffic.** Client calls are in `Logs_<Client>.Api`, not Core `Logs.Api` (that holds only auth / pre-scope rows). Group by `sourceIp`, route, and `queryString` with `transactionId` stripped. See [request logging](../features/request-logging.md).
+4. **Recover: reboot the EC2 instance.** On an OOM-frozen box `reboot` takes ~1 min (soft, then hard). The instance state stays `running` the whole time, so confirm the reboot by the **kernel uptime reset** in the console output, not by the state.
+5. **Confirm api2 answers** (an authenticated client call, not only `/health`, which is a static Apache file and can answer while php-fpm is down).
+
+## Gotchas
+
+- **Worked instance (TRUE-83366, 2026-10-28).** `api-sandbox-client` (account toga-test `975050298201`, us-east-1, ALB `true-alb-sandbox-client`, target group `api-sandbox-client`) is **one t4g.small (2 GB), MinSize 1 / MaxSize 1**, so there is no second instance to fail over to. Down ~15:09 to 15:26 UTC. ~30-35 concurrent requests (`sandbox-client-writer` connections 1 to 136) exhausted memory. Fixed by a reboot at 15:24 UTC.
+- **The traffic was normal QA, not a bot.** All calls came from one IP, `99.179.145.25` (the team's own office network), through TOGa Commerce and TOGa Supply. 100-200 calls/min. Busiest hour 1,686 logged calls vs 1,390 (21 Oct) and 1,272 (23 Oct). The box is simply too small for ordinary Compass QA. Side observation: one Compass test user's Commerce page called `/v2/locations?c_erpSystemEntityId=4930` 171 times in 20 min at ~1.85 s each. Extra load, not the cause.
+- **Open, not done (recorded on TRUE-83366):** (a) `api-sandbox-client` t4g.small to t4g.medium (4 GB); (b) a `pm.max_children` cap in `api2/.platform` sized to memory. It ships to every api2 env including prod, so it needs a `cto` review; (c) a memory alarm on sandbox API servers.
+
+## Change history
+- 2026-10-28 — Created from the api-sandbox-client php-fpm OOM outage (TRUE-83366). (bala)
+
+## Related
+
+- [api2 architecture — EB health causes](../architecture.md)
+- [Request logging](../features/request-logging.md)
+- [Health-check endpoint](../features/health-check-endpoint.md)

package/knowledge/INDEX.md

@@ -24,7 +24,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 - **_underscore** (_Underscore) _(framework core)_ — 117 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 100 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
-- **api2** (API) — 28 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
+- **api2** (API) — 29 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 24 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
 - **saml** (SAML SSO Gateway) — 10 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)

package/knowledge/standalone/apps/claude/workflows/harness-distribution.md

@@ -6,8 +6,8 @@ project: Claude Harness
 client: shared
 type: workflow
 status: active
-updated: 2026-10-14
-owners: ["jcardinal"]
+updated: 2026-10-27
+owners: ["jcardinal", "rgirish"]
 files:
   - claude/scripts/install.js
   - claude/scripts/hooks/harness-version-check.js
@@ -50,6 +50,7 @@ There are **two copies** of `knowledge.js` on a developer's machine, and the ski
 - **`npx toga-ai` can silently fail to update the knowledge repo** (open issue, not yet fixed). On a non-zero `git pull --ff-only` exit, `tryGitPull()` returns `{ pulled: false, newDocs: 0, message: null }` — the git stderr is discarded and **nothing is printed to the developer**. A teammate whose pull is refused (e.g. local edits to a tracked file the incoming commits also touch — `.claude/settings.json` is the usual culprit) will believe they are up to date while still running old tooling. Suggested fix, not applied: surface the git stderr in the `pulled: false` path so a refused pull is visible. Same defect class as the swallowed-stderr bug in the publish pipeline.
 - Until that is fixed, if a harness fix "didn't take", check the pull manually rather than trusting `npx toga-ai`'s output.
 
+- **Run `npx toga-ai@latest` from the project (workspace root) folder, never from inside `~/toga-tech`.** It installs into the cwd: run there, it writes `.claude/` (agents, skills, rules, hooks, knowledge) and rewrites `CLAUDE.md` inside the KB repo. Cleanup: never commit those stray files; `/finish` stages only `knowledge/`.
 - **The harness `.claude` lives only at the workspace root, never in a sub-repo.** Hook commands must anchor on `$CLAUDE_PROJECT_DIR` (`node "$CLAUDE_PROJECT_DIR/.claude/hooks/toga/X.js"`), not relative paths: relative paths resolve against cwd, so a sub-repo cwd gives MODULE_NOT_FOUND. A hook error naming a sub-repo `.claude` path is a path-resolution bug, not a folder to create. The settings file ships from the bundle, so the fix must live upstream or `npx toga-ai` reverts it.
 - **Publish Action (`.github/workflows/publish.yml`, on push to `_main`: bump patch, `npm publish`).** Devs get a change only after a version bump (the SessionStart version-check hook tells them to run `npx toga-ai@latest`).
   - `npm publish` **E404** ("could not be found or you do not have permission") = auth failure, not a missing package: the `NPM_TOKEN` secret in the repo's **Actions environment** expired. Fix: new npm Automation token, update the secret, re-run only the latest failed run.
@@ -59,6 +60,7 @@ There are **two copies** of `knowledge.js` on a developer's machine, and the ski
 
 ## Change history
 
+- 2026-10-27 — Added gotcha: `npx toga-ai@latest` run from inside `~/toga-tech` installs the harness into the KB repo (TRUE-83322). (rgirish)
 - 2026-10-14 — Added `@latest` rule, SessionStart version-check hook, warn-not-auto-update decision, Windows npm.cmd gotcha; sync-team-skills retired (TRUE-82829). (jcardinal)
 - 2026-10-09 — Added permission-retire (`RETIRED_TOGA_PERMISSIONS`) and auto-mode settings-edit gotchas. (jcardinal)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1177",
+  "version": "1.0.1179",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",