toga-ai 1.0.864 → 1.0.866

package/knowledge/2.0/apps/_underscore/architecture.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: ["jcardinal", "rgirish", "mhammontree", "ajean"]
 files:
   - _underscore/_underscore.php
@@ -253,12 +253,28 @@ const DATABASE = _underscore::DB_CLIENT;  // or DB_CORE
 const TABLE    = 'TableName';
 ```
 
-Columns are declared as `public` properties set to field-type constants. Numeric
-(`FIELD_PRIMARYKEY`, `FIELD_FOREIGNKEY`, `FIELD_INTEGER`, `FIELD_BOOLEAN`,
-`FIELD_DECIMAL`…), string (`FIELD_CHAR`, `FIELD_CHAR_CAPS`, `FIELD_CHAR_UUID`,
-`FIELD_LIST`…), date/time (`FIELD_DATE`, `FIELD_DATETIME`, `FIELD_DATETIME_CREATED`
-auto-set on insert, `FIELD_DATETIME_UPDATED` auto-set on save), binary
-(`FIELD_BLOB`, `FIELD_STORAGE` for S3/filesystem), and calculated (`FIELD_SQL`).
+Columns are declared as `public` properties set to field-type constants:
+
+- **Numeric:** `FIELD_PRIMARYKEY`, `FIELD_FOREIGNKEY` (array form, see below), `FIELD_INTEGER`,
+  `FIELD_AUTOINCREMENT` (see [autoincrement-record-numbering](./features/autoincrement-record-numbering.md)),
+  `FIELD_BOOLEAN`, `FIELD_DECIMAL`.
+- **String — several apply an on-write/on-read transform, so pick the specific type instead of a
+  plain `FIELD_CHAR` plus manual formatting:** `FIELD_CHAR` (plain), `FIELD_CHAR_CAPS` (forced
+  uppercase), `FIELD_CHAR_UCWORDS` (title-case), `FIELD_CHAR_HTMLENTITY` (HTML-escaped —
+  XSS-relevant), `FIELD_CHAR_PASSWORD` (hashed — never store a plaintext password in a plain
+  `FIELD_CHAR`), `FIELD_CHAR_PHONE` (phone-formatted), `FIELD_CHAR_SERIALIZED` (PHP-serialized),
+  `FIELD_CHAR_UUID`, `FIELD_CHAR_ZIPCODE` (ZIP-formatted), `FIELD_LIST` (enum/list value — see the
+  `FIELD_LIST` enum-validation caveat in `rules/common/security.md`).
+- **Date/time:** `FIELD_DATE`, `FIELD_DATETIME`, `FIELD_DATETIME_CREATED` (auto-set on insert),
+  `FIELD_DATETIME_UPDATED` (auto-set on save), `FIELD_TIME`, `FIELD_YEAR`.
+- **Binary/file:** `FIELD_BLOB`, `FIELD_STORAGE` (S3 or filesystem — see the storage-write gotcha
+  below and [field-storage-row-hydration](./features/field-storage-row-hydration.md)).
+- **Calculated:** `FIELD_SQL` (see below).
+
+**Field options** (the `FIELDOPT_*` array keys that tune a field): `FIELDOPT_DEFAULT` (default
+value), `FIELDOPT_FOREIGNKEY_MODEL` (target model class), `FIELDOPT_AUTOINCREMENT_PREFIX` /
+`FIELDOPT_AUTOINCREMENT_FIRSTNUMBER` / `FIELDOPT_AUTOINCREMENT_SUFFIX` (numbering scheme, see the
+autoincrement feature), and `FIELDOPT_SQL_STORED` / `FIELDOPT_SQL_TYPE` (calculated fields, below).
 
 **Foreign keys** are arrays with the target model:
 

package/knowledge/2.0/apps/api2/architecture.md

@@ -62,9 +62,9 @@ third party → apiproxy (region failover) → EB ALB → Apache → index.php
 
 One ~2,000-line `execute()` then `processRoutePairs()`:
 
-1. **Parse/normalize** — reads "now" from Core (`SELECT NOW()`) for token expiry; parses method/headers/route (strips `/v2`)/body; builds Core-metadata lookups (`Parameters`, `Records`, `InherentRecordChildren`, `RecordFields`).
+1. **Parse/normalize** — reads "now" from Core (`SELECT NOW()`) for token expiry; parses method/headers/route (strips `/v2`)/body; builds Core-metadata lookups (`Parameters`, `Records`, `InherentRecordChildren`, `RecordFields`). A legacy **`apiKey` compatibility patch** rewrites a payload `apiKey` field into `api`/`secret` before auth, so old integrations that send one `apiKey` value still authenticate.
 2. **Options** (`fields`, `where`, `join`/`ojoin`, `sort`, `group`, `distinct`, `page`, `recordsPerPage` default 25 / max 10000, `depth` default 3, `calcDepth` default 1) — query params or one base64(JSON) `_` blob.
-3. **Authentication** — `transactionId` required and **globally unique** (checked vs Core + client logs; reuse → `EV-5`). Auth routes under `/v2/auth/*`: `api`, `refresh`, `oauth`, `oauth/refresh`, `login` (email/pw; AD via LDAP bind for AD-configured clients), `public`, `delegator`/`encrypted`. JWTs HS256, `sub` access/refresh (access 3600s, refresh 30d); signing secret **rotated** with current+previous accepted across the boundary. The JWT `id` claim carries the full identity used for ACL. **`/v2/auth/oauth` credential transport:** OAuth 2.0 `client_credentials`. `client_id` + `client_secret` may arrive as **HTTP Basic** *or* POST body fields — **Basic takes priority** when both present. `scope` is **required**; `grant_type` is **optional**, defaults to `client_credentials` (accepted: `client_credentials`, `password`, `refresh_token`). Mints a JWT; subsequent requests carry `Authorization: Bearer`. (`V2.php` L414-460.)
+3. **Authentication** — `transactionId` required and **globally unique** (checked vs Core + client logs; reuse → `EV-5`). Auth routes under `/v2/auth/*`: `api`, `refresh`, `oauth`, `oauth/refresh`, `login` (email/pw; AD via LDAP bind for AD-configured clients), `public`, `delegator`/`encrypted`. JWTs HS256, `sub` access/refresh (access 3600s, refresh 30d); signing secret **rotated** with current+previous accepted across the boundary. The JWT `id` claim carries the full identity used for ACL — shape: `id.core.roles[]` (core role ids); `id.client.{uuid, slug, roles[], personas[]}`; `id.client.user.uuid` / `id.client.api.uuid` (whichever authenticated); `id.app.uuid` (from the `Domain`); `id.delegator` (present only when acting on behalf of another client/user). **`/v2/auth/oauth` credential transport:** OAuth 2.0 `client_credentials`. `client_id` + `client_secret` may arrive as **HTTP Basic** *or* POST body fields — **Basic takes priority** when both present. `scope` is **required**; `grant_type` is **optional**, defaults to `client_credentials` (accepted: `client_credentials`, `password`, `refresh_token`). Mints a JWT; subsequent requests carry `Authorization: Bearer`. (`V2.php` L414-460.)
 4. **Response envelope** — `{transactionId, timestamp, authority, audience, isSuccess, status, error, messages[], meta{}, data{}}`; `isSuccess` = 2xx status.
 5. **Transaction logging** — every request logged (to client/core Logs DB, or a JSONL line shipped by CloudWatch when `[api] log_filepath` is set).
 

package/knowledge/2.0/apps/apiproxy/INDEX.md

@@ -0,0 +1,5 @@
+# apiproxy (API Proxy) — 2.0 knowledge
+
+| Doc | Summary |
+|-----|---------|
+| [API Proxy (apiproxy) Architecture](architecture.md) | Thin, stateless reverse-proxy / failover router that fronts every public TOGa API tier. |

package/knowledge/2.0/apps/apiproxy/architecture.md

@@ -0,0 +1,150 @@
+---
+title: API Proxy (apiproxy) Architecture
+framework: "2.0"
+repo: apiproxy
+project: API Proxy
+client: shared
+type: architecture
+status: active
+updated: 2026-09-22
+owners: [jcardinal]
+files:
+  - apiproxy/Controller/Index.php
+  - apiproxy/_.php
+  - apiproxy/index.php
+  - apiproxy/Config/production.ini
+  - apiproxy/.htaccess
+  - apiproxy/.platform/httpd/conf.d/long_gateway_timeout.conf
+related: []
+---
+
+Thin, stateless reverse-proxy / failover router that fronts every public TOGa API tier. All external API traffic hits apiproxy first; it forwards the request verbatim to a configured backend and retries the next backend on a 5xx, so integrators never see a single-region/single-tier failure. Built on `_underscore` but DB-free in the request path. Open for the config-group resolution, the 5xx-only failover contract, and the two-tier timeout ceiling.
+
+## Summary
+
+`apiproxy` is the public edge for the TOGa 2.0 API. A request (e.g. `GET https://api.togahub.com/sales-orders/<uuid>`) lands here; Apache rewrites all to `index.php`, `_underscore` boots and — because no real controller matches the incoming URI — dispatch falls through to the default `['_Controller_Index','api']`. `api()` **is** the entire proxy: it picks a config group from hostname + region, walks an ordered list of `attempts_*` from the INI, and cURLs `endpoint + REQUEST_URI` forwarding the original method/headers/body. The first non-5xx response is relayed back unchanged; a 5xx moves to the next attempt. `/health` short-circuits to `200 OK` without proxying.
+
+Runtime: PHP 8.1+, AWS Elastic Beanstalk (Amazon Linux, Apache/httpd → PHP-FPM). `_underscore` is **pulled at deploy, not vendored** (cloned by a prebuild hook). **No database is used in the request path** despite the framework supporting one — keep it that way.
+
+**Critical rules:** (1) **Failover fires on HTTP 5xx only** (`floor(code/100) == 5`). Any 2xx/3xx/4xx is final and returned immediately, including the last attempt's 5xx if all fail — changing this trigger changes what counts as "try the next backend." (2) **Every fronted hostname/region needs a matching config group** (`<SERVER_NAME>:<region>` preferred, `<SERVER_NAME>` fallback); a missing group makes `api()` **throw**. (3) **`/health` must keep returning 200 without proxying** — it is the EB/LB probe. (4) **Keep the proxy stateless and DB-free in the request path.**
+
+## Dependencies
+
+- **`_underscore` (framework core)** — apiproxy is a minimal app over the framework boot/route/config layers (`_Config`, `_Environment`, `_Cloud`, `_Route`). Read its architecture first. Cloned at build from branch `_<ENVIRONMENT>`, not vendored.
+- **`api2` / cXML tiers (downstream)** — apiproxy forwards to the real API backends (`api1/api2/api3.togahub.com`, `api-writer.togahub.com`, `cxml*.togahub.com`, `webhook.togahub.com`, etc.). api2's own architecture doc names apiproxy as the front edge; the timeout chain is shared across both tiers.
+
+## Request lifecycle
+
+```
+client → LB (3600s) → apiproxy [Apache → PHP-FPM] → cURL → api2/cxml [Apache → PHP-FPM] → DB
+```
+
+1. **`.htaccess`** — `RewriteRule ^(.*) index.php` sends every non-file/non-dir request to the front controller; direct access to `.php`/`.ini` is denied.
+2. **`index.php`** → `require '_underscore.php'` — version check, autoloader, `_Config`, `_Environment`, `_underscore::initialize()`, `_Route`.
+3. **Routing (key fact)** — `_underscore::DEFAULT_CONTROLLER_METHOD = ['_Controller_Index','api']` (in `_.php`). The incoming URI (`/sales-orders/...`) matches no real controller/method, so `_Route` falls back to this default. The proxy has **no per-route controllers on purpose** — everything funnels into `Index::api()`, which is why the full original `REQUEST_URI` survives to forward downstream.
+4. **`_Controller_Index::api()`** — the whole proxy lives here (`Controller/Index.php`), plus the `/health` short-circuit.
+
+## Config-group resolution
+
+`api()` selects which backend list to use by building a config-group name and looking it up in `Config/<environment>.ini` (`<environment>` = `_Environment::$name`). It tries two names, region-specific first:
+
+```
+1. "<SERVER_NAME>:<region>"   e.g. api.togahub.com:us-east-1   (preferred)
+2. "<SERVER_NAME>"            e.g. api.togahub.com             (fallback)
+```
+
+- `SERVER_NAME` = the Host the request arrived on.
+- `region` = `_Cloud::getRegion()`: in **production** it reads `storage/region.txt` (written at deploy by `.platform` hook `041-write-region.sh` from IMDS); in **debug mode** (`debug_mode = 1`) it returns `_Environment::$name` itself.
+- Locally the dev INI uses `[apiproxy:dev-<name>]` because `SERVER_NAME` is `apiproxy` and the debug "region" resolves to the env name. Production defines real groups: `[api.togahub.com:us-east-1]`, `[cxml.togahub.com:us-west-2]`, `[webhook.togahub.com]`, etc.
+- **Neither group found → `api()` throws.** Adding a fronted host or region means adding a matching group to every relevant `Config/*.ini`.
+
+## Retry / failover mechanism
+
+Each config group is a flat, ordered list of attempts keyed with underscore-numbered keys:
+
+```ini
+attempts_0_read_endpoint  = https://api1.togahub.com
+attempts_0_write_endpoint = https://api-writer.togahub.com
+attempts_0_delay          = 0        ; microseconds to usleep BEFORE this attempt
+attempts_1_read_endpoint  = https://api1.togahub.com
+attempts_1_write_endpoint = https://api-writer.togahub.com
+attempts_1_delay          = 2000000  ; 2 seconds
+```
+
+How `api()` consumes them:
+
+- **Read vs write:** `GET` uses `*_read_endpoint`; every other method (`POST`/`PUT`/`DELETE`/...) uses `*_write_endpoint`. Reads spread across read replicas (`api1`/`api2`/`api3`); writes always go to the dedicated writer.
+- The loop reads `attempts_0_*`, `attempts_1_*`, ... until a missing key stops it. **Empty list → throws.**
+- Before each attempt, if `delay > 0` it `usleep()`s (back-off; the first attempt's delay is always `0`).
+- Builds `destinationUrl = endpoint + REQUEST_URI` and cURLs it, forwarding the original **method, body (`php://input`), and headers**.
+- **Failover trigger: a 5xx only.** Non-5xx (2xx/3xx/4xx) is final and returned at once. If the last attempt is still 5xx, that 5xx is returned.
+- Region-ordered endpoint lists differ per region so each region prefers its local tier first (e.g. `us-east-1` prefers `api1`, `us-west-2` prefers `api2`).
+
+### Header handling
+
+- **Request → backend:** all incoming headers forwarded except `EXCLUDE_HEADERS_REQUEST` (`Host`).
+- **Backend → client:** all returned headers relayed except `EXCLUDE_HEADERS_RESPONSE` (`Transfer-Encoding`, `Content-Length`, `Connection`, `Server`, `Date`) — dropped so the proxy/Apache set them correctly for the relayed body.
+
+### Health check
+
+`GET /health` (any path reducing to `health`) short-circuits to `200 OK` without proxying — the EB/ALB liveness probe. Keep it working.
+
+## Timeouts (read before debugging 504s)
+
+A request crosses **two** Apache→PHP-FPM tiers, each with its own timeout ceiling:
+
+| Layer | Setting | Where |
+|---|---|---|
+| Load balancer idle timeout | 3600s | EB / ALB config |
+| apiproxy Apache→FPM proxy | `ProxyTimeout 1800` / `Timeout 1800` | `.platform/httpd/conf.d/long_gateway_timeout.conf` |
+| apiproxy PHP max execution | `time_limit = 900` | `Config/*.ini` `[_underscore]` |
+| apiproxy → backend cURL | `CURLOPT_TIMEOUT = 900` (connect 10s) | `Controller/Index.php` |
+| backend (api2/cxml) Apache→FPM | `ProxyTimeout 1800` / `Timeout 1800` | `api2/.platform/httpd/conf.d/long_gateway_timeout.conf` |
+
+- **The "exactly 60 seconds" 504:** EB's Amazon Linux PHP platform proxies to PHP-FPM via `mod_proxy_fcgi`, whose default `ProxyTimeout` is **60s**. If `long_gateway_timeout.conf` is missing for a tier, that tier 504s at exactly 60s regardless of LB/PHP/cURL limits. This file must exist in **every** httpd-platform tier handling long requests (apiproxy, api2, saml, worker2). On any 60s 504, check for this file first.
+- **Effective single-attempt ceiling is ~900s** — the cURL `CURLOPT_TIMEOUT`, not 1800s. Apache allows 1800 but cURL aborts the upstream call at 900. Raise `CURLOPT_TIMEOUT` in `Index.php` if a single backend call legitimately needs longer.
+- **Retry amplification:** because a 5xx triggers failover, a slow request that *eventually* 5xxs can burn several multi-minute attempts back-to-back. A slow-but-successful request does not retry (non-5xx is final), so normal large-record reads are safe.
+
+## Deployment (Elastic Beanstalk)
+
+`.platform/` hooks and `.ebextensions/` configure the instance:
+
+- **`.ebextensions/php_include_underscore.config`** — sets PHP `include_path` to `.:/var/www/html/_underscore` so the framework is importable.
+- **`.ebextensions/git.php` + `.platform/hooks/prebuild/git.sh`** — at build, clones the shared `_underscore` repo (branch `_<ENVIRONMENT>`, e.g. `_production`) into place, strips its `.git`, swaps it in. **Framework source is NOT vendored — pulled at deploy.** Env-specific overrides come from `git.<ENVIRONMENT>.json` if present.
+- **`.platform/hooks/postdeploy/015_install_composer.sh`** — installs Composer and runs `composer install` (currently no runtime deps; `composer.json` `require` is empty).
+- **`.platform/hooks/_shared/040-write-instance-id.sh` & `041-write-region.sh`** — write `storage/instance-id.txt` and `storage/region.txt` from IMDSv2. `_Cloud::getRegion()`/`getInstanceId()` read these. Symlinked into `prestart` and `postdeploy`.
+- **`.platform/httpd/conf.d/long_gateway_timeout.conf`** — the timeout fix above.
+
+## Local development
+
+- Config chosen by `_Environment::$name`. The dev file (`Config/dev-<name>.ini`) sets `debug_mode = 1` and points its attempts at local backends (`http://api2`, `http://cxml2`).
+- `cache_folder` / `model_storage_folder` in the dev INI use Windows paths (`C:\CACHE`, `C:\STORAGE`).
+- To front a new host/region locally, add a `[<SERVER_NAME>:<env-name>]` (or bare `[<SERVER_NAME>]`) group with `attempts_*` keys.
+
+## File map
+
+| Path | Purpose |
+|---|---|
+| `index.php` | Front controller — just requires the framework |
+| `_.php` | Project class `_underscore` — namespace, defaults, `DEFAULT_CONTROLLER_METHOD` |
+| `Controller/Index.php` | **The entire proxy logic** (`api()`) + `/health` |
+| `Controller/Error.php` | Error route controller (sets HTTP code) |
+| `Config/production.ini` | Production config groups (real hostnames/regions/endpoints) |
+| `Config/dev-*.ini` | Per-developer local config |
+| `.htaccess` | Rewrite-all-to-`index.php` + deny direct `.php`/`.ini` |
+| `.platform/`, `.ebextensions/` | EB deploy configuration (see Deployment) |
+| `Layout/`, `View/`, `Theme/`, `Component/`, `Model/`, `Assets/` | Framework scaffolding — largely unused; placeholder `.gitkeep`s / boilerplate |
+
+## When making changes here
+
+- **Adding a fronted hostname/region:** add a matching config group (`<host>:<region>` and/or `<host>`) with `attempts_*` keys to every relevant `Config/*.ini`. Missing group → `api()` throws.
+- **Changing failover behavior:** the only retry trigger is a 5xx (`floor(code/100) == 5`). That check is the lever for what counts as "try the next backend."
+- **Long-running requests / 504s:** verify `long_gateway_timeout.conf` exists for the tier, then check `CURLOPT_TIMEOUT` (900s) and PHP `time_limit` (900s). The LB is at 3600s and is rarely the cause.
+- **Don't break `/health`** — the LB health probe.
+- **Keep it stateless and DB-free in the request path** unless there is a strong reason not to.
+
+## Security notes
+
+- **No DB and no per-request secrets in the proxy path** — apiproxy forwards the client's own `Authorization` header downstream unchanged; it does not mint or hold tokens. Auth stays a backend (api2) concern.
+- **Deploy-time credential exposure to check:** the `git.php` / `git.<ENVIRONMENT>.json` clone config supplies whatever credential `prebuild/git.sh` uses to pull `_underscore` at build. Sibling 2.0 tiers (api2) ship a **committed plaintext GitHub PAT** in their per-env `git.*.json`; if apiproxy's clone config carries a token the same way, treat it as compromised, rotate it, and move it to SSM / EB env properties. Verify before touching deploy config. (Location/remediation only — never record a token value.)
+- **`_underscore` is cloned from a moving branch** (`_<ENVIRONMENT>`), not pinned to a commit, so two deploys of the same apiproxy commit can behave differently — check the framework branch state when triaging an "it worked yesterday" regression.

package/knowledge/2.0/apps/saml/architecture.md

@@ -6,7 +6,7 @@ project: SAML SSO Gateway
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: ["rgirish", "jcardinal"]
 files:
   - saml/index.php
@@ -65,6 +65,13 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
 
 Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentication.php`. Each class extends `_Model_Core_ClientAuthentication` + uses `_Trait_ClientAuthentication`. To onboard a new SSO client, add the class in `_underscore` — not in this repo.
 
+## Runtime & config
+
+- **PHP 8.1+ in production**, on Amazon Linux with **Apache/httpd fronting PHP-FPM**. `composer.json` pins `platform.php` to **`7.2.33` for dependency resolution only** — that is not the runtime version; do not treat it as the target.
+- **Config files are `Config/<environment>.ini`, selected by `_Environment::$name`.** Production sets `time_limit = 300`, `debug_mode = 0`. Many per-developer/per-stage configs exist (`alpha`, `beta`, `gamma`, `dev-*`, `rohan-mac`).
+- **SP host per environment is a framework constant**, not local config: `_Model_Core_ClientAuthentication::SAML_ASSERTION_CONSUMER_SERVICE_HOST_URL_BY_ENVIRONMENT` — `dev → http://saml`, `beta`/`production → https://saml.togahub.com`. This is the ACS host IdPs POST to; it must match what `/meta` advertises.
+- **Apache `.platform` configs (keep these):** `enforce_https.conf` redirects any request with `X-Forwarded-Proto != https` to HTTPS — **SAML must run over TLS only; do not weaken it**. `long_gateway_timeout.conf` sets `ProxyTimeout 1800` / `Timeout 1800` (same override as `apiproxy`). `php_include_underscore.config` adds `/var/www/html/_underscore` to PHP `include_path`.
+
 ## Deployment
 
 - **EB env:** `saml-production` (us-east-1). DNS: `saml.togahub.com → saml-production.us-east-1.elasticbeanstalk.com`.
@@ -82,8 +89,10 @@ Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentica
 ## Gotchas
 
 - **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. `/acs` calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
-- **SAML signature verification is commented out — repo's top security priority.** The IdP X509 cert verification block in `/acs` is disabled, so the gateway accepts any well-formed status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat: an attacker who can craft/replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix.
-- **Plaintext secrets in source — move to AWS SSM Parameter Store.** `saml/Config/production.ini` stores credentials in plaintext, and a Sentry DSN is hardcoded in `saml/Controller/Index.php`. Both checked into the repo. Migrate to AWS SSM Parameter Store (SecureString), load at runtime; source tree should reference parameter names only, never literal credential values.
+- **SAML signature verification is commented out — repo's top security priority.** The IdP X509 cert verification block in `/acs` is disabled, so the gateway accepts any well-formed status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat: an attacker who can craft/replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix. The per-client IdP verification certs already exist in source as base64 blobs (Compass, Office Depot, Prudential, RUMCSI, Agilant) — they are what a re-enabled verifier would load; verification is per-client and each IdP's signing cert must match.
+- **Plaintext secrets in source — move to AWS SSM Parameter Store.** Checked-in credentials, all live and to be rotated: `saml/Config/production.ini` holds a **DB password, an AWS access key id + secret, and NetSuite credentials** in cleartext; `saml/Controller/Index.php` hardcodes the **Sentry DSN**; `.ebextensions/git.json` carries a **GitHub PAT** (same leak class as `apiproxy`). Migrate to AWS SSM Parameter Store (SecureString), load at runtime; source tree should reference parameter names only, never literal credential values.
+- **Stale test scaffolding in `/acs` — remove it.** A large commented-out block in `saml/Controller/Index.php` hardcodes captured real `SAMLResponse` payloads and a synthetic `RelayState` for Office Depot / Compass testing. It is inert (commented out) but captured assertions living in source are a confidentiality/hygiene risk — delete it.
+- **RelayState time check bounds the future, not the past.** `MAX_TIME_TO_LIVE` (300s) sanity-checks how far in the *future* a RelayState timestamp may be. It does not clearly bound replay of an *old* captured RelayState — review before relying on it as replay protection.
 - **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both signing and encryption blocks.
 - **SP encryption certs load from Secrets Manager at request time.** `/acs` holds no encryption key material on disk (`_underscore/Assets/ssl/` no longer carries the SP encryption key). Rotation = edit the `ENCRYPTION_SECRET_NAMES` list + the secrets themselves, not a redeploy of on-disk `.crt`/`.key` files.
 - **Committed AWS key must be rotated.** A hardcoded AWS key was removed from `_underscore/Cloud.php`; because it was committed to git history it is compromised — rotate it. Still open: plaintext creds in `production.ini` and the hardcoded Sentry DSN.

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 26 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 37 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 38 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
@@ -25,6 +25,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
 - **saml** (SAML SSO Gateway) — 6 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
+- **apiproxy** (API Proxy) — 1 doc(s) → [2.0/apps/apiproxy/INDEX.md](2.0/apps/apiproxy/INDEX.md)
 - **toga2-view** (TOGa View Frontend) — 12 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)
 - **toga2-hub** (TOGa Hub) — 2 doc(s) → [2.0/apps/toga2-hub/INDEX.md](2.0/apps/toga2-hub/INDEX.md)
 - **talos** (TOGa IQ) — 8 doc(s) → [2.0/apps/talos/INDEX.md](2.0/apps/talos/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -18,6 +18,7 @@
 | [Compass ODP EDI 850 Line-Item Resolution (VA part number to IN SKU fallback)](features/odp-edi-850-item-resolution.md) | 1.0 | How each PO1 line on an inbound Office Depot EDI 850 resolves to a real Compass catalog item before cron 3a writes anything; open for ODP 850 import failures, u |
 | [Compass Office Depot EDI 855 Acknowledgement + Over-Quantity PO Guard](features/odp-edi-855-acknowledgement-and-overquantity-guard.md) | 1.0 | How Compass acknowledges ODP EDI 850 POs with an X12 855 and rejects a duplicate/over-quantity PO before import; open for 855 accept/reject behaviour, over-quan |
 | [Compass ODP EDI File Retention & Import Retry (cron 3a)](features/odp-edi-file-retention-and-retry.md) | 1.0 | When cron 3a keeps vs deletes an ODP EDI file in S3, and why an api2 outage used to silently lose orders; open for lost ODP imports, retry behaviour, or the sti |
+| [Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)](features/odp-strategic-systems-asn-csv-import.md) | 1.0 | The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.7 |
 | [Compass: MA refresh-order monitor (Monitor/Compass/MaRefreshOrderExceptions)](features/oneuptime-ma-refresh-order-monitor.md) | 2.0 | worker2 OneUptime push monitor counting Compass MA employee-refresh orders whose ODP SO has no NetSuite order yet; open when working on Compass MA monitoring or |
 | [Compass USA — Fulfilled vs Partially Fulfilled (order total kept; per-line rule built then rejected)](features/order-fulfillment-status-per-line.md) | 2.0 | Why Compass USA shows Fulfilled on partly-shipped orders and why the per-line fix was built, measured, then reverted; open before touching Compass USA `_status` |
 | [Compass PEOPLE-File User Lifecycle (duplicate accounts, reactivation grace window, raw-SQL deactivation)](features/people-file-user-lifecycle.md) | 2.0 | The nightly Compass PEOPLE-file cron's Users-row lifecycle (insert/reactivate/deactivate), why it creates duplicate accounts, and its raw-SQL/index traps; open |

package/knowledge/clients/compass-usa/features/odp-strategic-systems-asn-csv-import.md

@@ -0,0 +1,84 @@
+---
+title: "Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)"
+framework: "1.0"
+repo: worker
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-09-22
+owners: ["rgirish"]
+files:
+  - worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php
+  - worker/crons/toga2/compass/DOA_Scripts/DOA_compass_statuses.php
+related:
+  - ./tracking-number-origin-and-asn-link.md
+  - ./asn-to-item-fulfillment.md
+  - ./multi-tier-tracking-and-serial-mirroring.md
+  - ../../compass-canada/features/grand-and-toy-asn-import.md
+  - ../../quad/features/po-asn-email-import.md
+  - ../../../1.0/apps/worker/architecture.md
+---
+The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.77536E+11` with its digits already gone. Open before touching tracking-number parsing in this file or its DOA copy.
+
+## Summary
+
+Office Depot (ODP) and Strategic Systems email Compass USA ASN files as **CSV attachments**. The 1.0 worker cron `crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php` reads them and writes `Client_Compass.TrackingNumbers` rows. `DOA_Scripts/DOA_compass_statuses.php` is a **near-identical copy** of the same parsing logic — every change here must be made in both files.
+
+Two facts drive most of the trouble in this file:
+
+1. **The file is routed by COLUMN COUNT, not by filename.** 25 columns → ODP parser, 15 → Strategic Systems, anything else throws.
+2. **It is a CSV, so a tracking number that arrives as `8.77536E+11` is unrecoverable.** The characters in the file *are* `8.77536E+11`. There is no underlying cell value to read back. No PHP change can restore the digits.
+
+## How it works
+
+### Key files / entry points
+- `worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php` — the scheduled importer. Reads attachments with `fgetcsv` / `str_getcsv`.
+- `worker/crons/toga2/compass/DOA_Scripts/DOA_compass_statuses.php` — copy of the same logic, same parsing, same guards. No shared helper exists; the two files drift.
+- `PhpSpreadsheet` is imported at the top of the importer but is used only to **write the validation-error report workbook** (`setCellValueExplicit` / `createWriter('Xlsx')`). It does **not** read the incoming feed. Do not read the import as a spreadsheet path.
+
+### File routing is by column count
+`3b_import_..._advance_shipping_notices.php` line ~266 counts the header's columns and dispatches:
+
+| Header column count | Constant | Parser |
+|---|---|---|
+| 25 | `ODP_HEADER_COLUMN_COUNT` | `processOfficeDepotFile()` |
+| 15 | `SS_HEADER_COLUMN_COUNT` | `processStrategicSystems()` |
+| anything else | — | throws `"Unsupported file format: $filename"` |
+
+Filename and extension are ignored. **If a vendor adds or drops one column, the whole file silently reroutes to the wrong parser** (25 ↔ 15 is a wide gap today, but the check is exact equality, so any drift toward the other constant is a real risk). This mirrors the Compass Canada G&T importer's 24-column gate — see [Grand & Toy ASN Import](../../compass-canada/features/grand-and-toy-asn-import.md).
+
+### Tracking-number parsing (4 guard sites)
+Both files parse tracking numbers in two branches each — ODP and Strategic Systems — so there are **four** copies of the same block. The correct order, now used in all four:
+
+1. Replace the multi-value separators `<BR>`, `<br>`, `;` with commas.
+2. `explode(',')`, trim, strip noise characters (`"`, `'`, `VYLI`, `#`), `array_filter` the empties.
+3. **Then** test each element for `e+` and, on a match, append to `$validationErrors`.
+
+`$validationErrors` already feeds an unprocessed-rows report emailed to the team, so a rejected row is visible, not silent.
+
+## Gotchas
+
+- **⚠ A scientific-notation tracking number is a DATA bug, not a display bug — and the digits are gone before our code runs.** `Client_Compass.TrackingNumbers.number` literally stores the 11-character string `8.77536E+11` (e.g. id 80112, `shippingCarrierId` NULL). 44 such rows exist, Sep 2025 → 2026-09-22. The cause is upstream: a 12-digit number like `877536271001` becomes `8.77536E+11` when a spreadsheet saves the column as a number rather than text. Only 6 significant digits survive. Reported on order **SA137640**.
+
+- **⚠ NEVER "recover" a scientific-notation number with `number_format()` — it pads zeros and fabricates a believable-but-wrong tracking number.** A guard added 2026-10-17 (commit `be8ff1d14`) ran `number_format($n, 0, '', '')` on any value containing `e+`. Verified by running it: `number_format("8.77536E+11")` returns **`877536000000`**, while the real value was **`877536271001`**. The result is a 12-digit number that looks perfectly valid, so nobody can spot it as fake — strictly worse than failing the row. Removed 2026-09-22; all four sites now reject instead. **A silent wrong tracking number is worse than a loud rejected row.**
+
+- **⚠ Guard-before-split threw a PHP fatal on multi-value rows.** The Strategic Systems branch tested the **whole raw string** for `e+` *before* splitting on `<BR>` / `;`. Input `"8.77536E+11<BR>8.77517E+11"` reached `number_format()` as a non-numeric string: `TypeError: number_format(): Argument #1 ($num) must be of type float, string given`. The ODP branch in the same file split first and was correct — the two branches disagreed. Always **split, clean, then guard each element**.
+
+- **CSV vs XLSX is the deciding fact for any fix here.** The feed is a CSV, which has no cell types, so the file itself contains only the rounded characters — nothing to recover. If the sender switched to `.xlsx`, the cell could be read as a string and the full number would survive. **Decided 2026-09-22:** ask Office Depot for `.xlsx` as the primary fix; fallback ask is CSV with the tracking column formatted as text (weaker — breaks again the moment anyone re-opens the file). **BLOCKED as of 2026-09-22 waiting on ODP to confirm the format**; no further integration work until they answer. The four reject-guards are worth keeping either way.
+
+- **The 44 corrupted rows cannot be repaired in SQL.** The digits do not exist anywhere in our data. They must be re-sourced from ODP or the carrier.
+
+- **One corrupted number on many orders is mirroring, not a rounding collision.** `8.77517E+11` appears on 69 distinct orders, `8.77536E+11` on 21, `8.73895E+11` on 18. Each was written once at the Agilant tier and then mirrored up the 3-tier drop-ship chain — see [multi-tier tracking & serial mirroring](./multi-tier-tracking-and-serial-mirroring.md). Do not read a high order count as many separate bad reads.
+
+- **⚠ OPEN — a third, unidentified writer creates TrackingNumbers rows.** Five corrupted rows landed **2026-09-22 06:22–06:32 with NULL `shippingCarrierId` AND NULL `trackingUrl`**. Both CSV importers always set a carrier (ODP from the `ODP_SHIPPING_CARRIER` column; Strategic Systems from the tracking number or the file), so neither wrote them. The fulfillments they attached to were created 2026-09-19, three days earlier. **Unresolved: find what job runs at 06:22.** By contrast the July 2025 corrupted rows carry Convey/Veyer URLs (`track.getconvey.com/veyer/us/odn/...`, Office Depot's tracking service), confirming the CSV path genuinely produced some of them. Note this is a *third* writer on top of the two already documented — see [where a TrackingNumbers row comes from](./tracking-number-origin-and-asn-link.md).
+
+- **⚠ OPEN — wider tracking-number data quality problem (168 bad rows).** Of **73,909** rows in `Client_Compass.TrackingNumbers`, **168** fail `^[A-Za-z0-9]{8,40}$`: 44 scientific notation, 79 with leading spaces, 45 shorter than 8 characters, plus keyboard typos such as `!Z8696XA0140259245` (`!` instead of `1` — a shift-key slip on a UPS number). Max observed length is 114. A CHECK constraint on `TrackingNumbers.number` is the only fix that covers **every** write path including the unknown 06:22 one, but the existing rows must be cleaned first and the upper bound validated against carriers that use longer formats.
+
+- **Only two files in the whole `crons/toga2/compass/` tree create TrackingNumbers rows**, and they are copy-paste siblings with no shared helper. When you fix one guard, grep the other file for the same block.
+
+- The Compass **Canada** G&T importer already rejects scientific-notation tracking as a row-level guard; its read-only inbox inspector reports it as a dry-run category. That is the pattern Compass USA now matches.
+
+## Change history
+- 2026-09-22 — Removed the lossy `number_format()` "recovery" from all 4 guard sites in both files; now split-then-guard, appending to `$validationErrors` instead. Fixes the fabricated-tracking-number bug and the multi-value PHP fatal. Written and linted (`php -l` passes), **not deployed**. (rgirish)
+- 2026-09-22 — Documented CSV-vs-XLSX as unrecoverable-at-source, column-count file routing, the unknown 06:22 writer, and the 168-row data-quality survey. (rgirish)

package/knowledge/clients/compass-usa/features/tracking-number-origin-and-asn-link.md

@@ -6,18 +6,20 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - library/app/api/toga2.php
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
   - worker/crons/toga2/netsuite/sync_togasupply_compass_usa.php
   - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
+  - worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php
   - worker2/Worker/Monitor/Compass.php
 related:
   - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
   - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
   - ./asn-to-item-fulfillment.md
+  - ./odp-strategic-systems-asn-csv-import.md
   - ./contact-email-resolution.md
   - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
 ---
@@ -26,6 +28,8 @@ Why a Compass TrackingNumbers row is born from the invoice sync (not the ASN) an
 ## Summary
 For Compass USA a `TrackingNumbers` row is **born from the NetSuite invoice sync**, not from the ASN. The ASN turns up days or weeks later and is the **only** thing that ties that tracking number to a sales order, and therefore to a customer's email address.
 
+The invoice sync is the **main** origin, but it is **not the only** one — see *Who else writes a TrackingNumbers row* below. At least three writers exist and one is still unidentified.
+
 Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is the invoice date.** It is the wrong clock to time any shipping notification, backlog age, or SLA from. Anything that asks *"when did we get this tracking number?"* and means *"when could we first have told the customer?"* is wrong for Compass by up to two weeks.
 
 ## How it works
@@ -52,6 +56,17 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
    ```
 4. Only once that chain exists can the in-transit email be built or sent.
 
+### Who else writes a `TrackingNumbers` row (added 2026-09-22)
+The invoice sync is the main writer, not the only one. Known and unknown writers:
+
+| Writer | Sets carrier? | Notes |
+|---|---|---|
+| NetSuite invoice sync (`syncInvoiceFromNetsuite`) | yes | the normal origin described above |
+| ODP / Strategic Systems ASN **CSV email** cron + its `DOA_Scripts` copy | **always** | [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md) |
+| **UNKNOWN job running ~06:22** | **no** — NULL `shippingCarrierId` *and* NULL `trackingUrl` | ⚠ unidentified, see gotcha |
+
+So **"which writer made this row?" is answerable from the carrier columns**: both CSV importers always set a carrier, so a row with NULL `shippingCarrierId` and NULL `trackingUrl` did **not** come from them.
+
 ### How big the gap is (measured on prod)
 - **2026-09-03, 11:15:56–11:40:20** — `Logs_Compass.Api` shows **98** `POST /v2/invoice-tracking-numbers` from the worker instances. **100** `TrackingNumbers` rows created in that window, while only **4** ASNs were created in the whole of 09-03.
 - Since **2026-08-01**, **53** tracking rows had their ASN land **more than 7 days** after the tracking row, with a **maximum gap of 13 days**.
@@ -66,6 +81,8 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
 ## Gotchas
 - **⚠ Never window, age, or alarm a Compass shipping feature on `TrackingNumbers.dtCreated`.** Use `AdvanceShippingNotices.dtCreated` for a window, and `GREATEST(TrackingNumbers.dtCreated, AdvanceShippingNotices.dtCreated)` when you need "how long has this been sendable". Getting this wrong produced both a real missed-email gap and two false HIGH alarms — see [Compass in-transit emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
 - **⚠ Correction to an earlier belief.** It was previously recorded that invoice-created tracking numbers "never touch the ASN chain" and are therefore out of scope for in-transit email. That is wrong. They are the *normal* origin for Compass tracking numbers; the ASN chain simply attaches to them later. The narrower true statement: a tracking number that **never** gets an ASN can never be emailed, and a naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count (with no ASN join) reads those as a backlog that never drains.
+- **⚠ OPEN — an unidentified job writes TrackingNumbers rows around 06:22.** Five rows landed 2026-09-22 06:22–06:32 with NULL `shippingCarrierId` and NULL `trackingUrl`, attached to fulfillments created three days earlier (2026-09-19). Neither CSV importer wrote them (both always set a carrier) and the shape does not match the invoice sync either. Do not assume a Compass tracking row came from one of the two known paths until this is identified.
+- **⚠ Tracking-number values are not validated anywhere — 168 of 73,909 rows are malformed.** Scientific notation, leading spaces, sub-8-character values, and typos all persist. Because an unknown writer exists, a **CHECK constraint on `TrackingNumbers.number` is the only fix that covers every path**. Details and counts: [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md).
 - **A tracking number with no ASN has no customer.** There is no fallback route from invoice to contact, because `Invoices.salesOrderId` is NULL. Waiting for the ASN is the design, not a bug.
 - Credentials for the sync ride on `App_Api_Toga2` class constants (e.g. `App_Api_Toga2::CLIENT_UUID_COMPASS`) — reference the constant, never the value.
 
@@ -73,5 +90,9 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
 - [Compass partial in-transit/delivered emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md)
 - [NetSuite TOGaSupply per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)
 - [ASN → ItemFulfillment](./asn-to-item-fulfillment.md)
+- [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md)
 - [Contact email resolution](./contact-email-resolution.md)
 - [OneUptime worker2 monitoring](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md)
+
+## Change history
+- 2026-09-22 — Recorded that the invoice sync is not the only writer: added the ASN CSV cron and an unidentified ~06:22 writer, plus the tracking-number data-quality gap. (rgirish)

package/knowledge/clients/compass-usa/profile.md

@@ -37,6 +37,7 @@ related:
   - features/oneuptime-ma-refresh-order-monitor.md
   - workflows/catalog-fold-reversal.md
   - features/asn-to-item-fulfillment.md
+  - features/odp-strategic-systems-asn-csv-import.md
   - features/multi-tier-tracking-and-serial-mirroring.md
   - features/order-fulfillment-status-per-line.md
   - features/cost-centers.md

package/knowledge/registry.json

@@ -73,6 +73,13 @@
     "role": "app",
     "dependsOn": []
   },
+  {
+    "repo": "apiproxy",
+    "project": "API Proxy",
+    "framework": "2.0",
+    "role": "app",
+    "dependsOn": []
+  },
   {
     "repo": "toga2-view",
     "project": "TOGa View Frontend",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.864",
+  "version": "1.0.866",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",